Interviewed for ZEISS eXtended Data by PRONEWS Written Article Vol 2

https://www.pronews.jp/column/20191030110011.html

Translation for the article Exploring ZEISS eXtended Data Vol.02

While the camera department shoots, the VFX department shoots HDRI to record lightings at the locations.

For the shooting of MORPHO, the camera records the videos and lens distortion information automatically at the same time, so it doesn’t require shooting the lens grid footage. However, they shot the grid just in case. Also HDRI was shot to recreate the lighting in CG. It records 360° panorama as HDRI and is used as environment maps. It captures buildings at the location as environment data, and by using the HDRI and bringing the environment in lighting and reflection, you can achieve photorealistic images.

In addition, because live action footage with drastic camera movement will be composited with CG in this project, the camera department consulted the VFX department about camera settings and movements during the shoot.

Mr. Adachi from the VFX department shoots HDRI. He says he shoots with a 8mm fisheye lens for front, back, left, right, and sometimes even top.

03 Post Production Preparation part - Prepare plates and ZEISS eXtended Data used in Post Production.

Export EXR sequence from R3D file with all metadata included.

From here, preparation process for Post Production will be explained. These steps are also presented in the Youtube tutorial video “Shot on ZEISS CP.3 XD Lenses Tutorial”, so you can check that out as well.

VFX for this project was done by Finitto, the CG production company that works on CG, VFX, VR, AR content. Their speciality is seamless compositing live action footage with CG, and they have been actively working mainly on CG/VFX for commercials, films, tv, and so on.

Compositing for MORPHO is done by Nuke at Finitto. Correcting distortion and Vignette is also done in Nuke. The growing plants on ground and buildings are done by GrowFX in 3DS Max.

Mr. Adachi in charge of VFX process.

(01) Introducing steps to prepare eXtended Data from R3D files in order to hand them over to Post Production.

ZEISS developed and published “zeiss-xdcp”, a free command line tool used for eXtended Injection. zeiss-xdcp allows eXtended Data to be injected into EXR sequence. It doesn’t need Zilverstack, an expensive onset data management software that used to be required in this process. This is a great news because Silverstack is an expensive software.

Download and install “Injection Tool” and “Compositing Software Plugins” from Zeiss website.

(02) eXtended Data is recorded in R3D file, but this doesn’t allow you to read lens data as it is. First, in order to work on the plate in Nuke, you export EXR sequence from R3D file.

Open Nuke, create Read node. Then import R3D file that you want to extract metadata from.

(03) In the Write node, choose EXR file and all metadata in metadata dropdown, and then export sequence.

(04) Execute Render.

(05) Open PDF “ZEISS eXtended Data workflow with RED DSMC2 cameras” and copy the command in the PDF page 5 onto any text editor (Windows Accesories notepad used in the screen below)

zeiss-xdcp -s example_R3D/example.R3D -d example_EXR/example_*.exr

(06) In the command, replace the directory and file name of the R3D file and replace the directory and file name of the EXR sequence.

(07) Open Command Prompt if using Windows and Terminal if using MacOS. In Windows10, go to Start Menu “Windows System Tool” → Click Command Prompt. In macOS, go to Application → Utility → Double click Terminal.

(08) Copy the command you just replaced, paste it in Command Prompt (Windows) or Terminal (Mac) and execute.

(09) by using zeiss-xdcp, lens data is injected into the EXR sequence that will be used in post production.

By the way, the resolution for MONSTRO footage is too big to work on, so they decrease the resolution and hand them over to the compositors so that they can composite the plates that are easy to work with.

04 Post Production Part - Utilize ZEISS eXtended Data in Post Production

Distortion Correction and Vignette Correction in Nuke

You can correct distortion and vignette in the footage by reading EXR with lens metadata. Check out the steps below.

(01) Correct distortion and vignette automatically by reading EXR sequence injected with eXtended Data and by using Zeiss plugin in Nuke.

First, open Nuke and read EXR sequence.

(02) In Nuke, choose ZEISS Lens Shading Correction node and Zeiss Lens Distortion correction node.

(03) When you plug it to the read node, you can correct vignette and distortion in the footage. You do shading correction first, and then you do distortion correction after that.

Steps to do tracking in Nuke

Here are the steps to track. Tracking analyzes the camera movement from the live action footage. By using tracking data, the camera in CG world can match the actual camera movement in the footage.

(01) Open CameraTracker property. Type in the sensor size in Film Back Size and focal length in the length section. In this case, Film Back Size 40.96mmx21.60mm is set for MONSTRO’s sensor size.

(02) Choose Track. It will start tracking the undistorted footage in 3D tracking.

(03) Because the footage is precisely corrected for distortion and vignette, you can achieve the high quality 3D tracking.

(04) After compositing in Nuke is done, Zeiss plugins allows you to bring back the distortion and vignette that you corrected before. By doing this, you can realize more realistic compositing.

(05) The fact that the entire screen is evenly shaded means that it helps for camera projection and keying as well. Also the newly added 3D objects can be seamlessly composited. For example, if the texture of the camera mapping is dark around the screen, it can look unnatural, but by correcting the vignette, it can solve the problem.

eXtended Data is much more efficient and can achieve higher quality matchmove and VFX than the conventional workflow. Regarding details of Nuke workflow, PDF “Apply lens characteristics using ZEISS eXtended Data plugin for NUKE” is published in ZEISS web page, so please check it out as well.

Tool to control focus work.

The plugins for eXtended Data are not just ZEISS.

In Nukepedia, the Nuke user information website, Max van Leeuwen developed “readZEISSdata”, a plugin that reads eXtended Data file in Nuke.

With this tool, you can set it up to automate the center of the depth for z lens blur. By setting up the lens focus data with Z depth render, you can match the complicated rack focus with the live action footage. It will be possible to do the focus work that is difficult to achieve by hand animating keyframes.