Professional post production workflow for immersive media

This article details a VR production studio’s process for editing and delivering immersive media. These tools and practices were developed for a fully staffed team but can be adapted to meet the needs of smaller producers.

This article presents an example of a high end advanced workflow for working with immersive media footage. It uses third-party stitching software and is intended to achieve a level of quality on par with some of the top video experiences available in Meta Quest TV. For this level of quality requirements would be: working with high resolution stereoscopic footage, minimizing stitching artifacts, and removing the tripod or other camera support system.

Not every production will have the resources to utilize this workflow in its entirety, but these steps should be considered by any creator when planning and producing a high quality immersive experience. A top level overview of the post production process can be found here.

1. Back up media to two separate hard drives. Software such as Hedge or Shot Put Pro is helpful, as these will perform a data validity check on the media to ensure files were not corrupted during the copying process.
   
   
2. Create a ‘Takes’ folder on one of the drives and organize the media files so that each take folder contains the correct media file from each of the several cards. Some tools exist that automatically do this; if you are using a Z Cam camera the Z Cam VideoConcatenator utility will organize the media as well as join clips that were separated by the camera for being over 5 minutes in length.
   
   
3. Quick stitch all of the footage using a preset template in a stitching tool such as Mistika VR. Try not to spend too much time on any one shot, but make a note of potentially difficult stitch issues to solve later in case rotoscoping or compositing is needed to layer multiple stitching passes into a single video.
   
   Rendering with optical flow on and in the same resolution/format as the delivery format will help to spot potential stereo and stitching issues to fix. To save space, render to a codec with a lower bitrate, such as Prores 422LT, but not to Prores Proxy because the compression is too high. It’s important to render the entire duration of the clip so a reference to its frame range can be used later to work on final frames when fine stitching.
   
   Review the quick stitch footage to create a list of “Selects’ which will be the shots approved for inclusion in the edit. Not all takes will become selects.
   
   
4. Create proxy files from the offline media using your preferred non-linear editor (NLE) to make editing faster.
   
   
5. Edit using the proxy media until ready to lock picture. Locking picture means that none of the visual shots or timing will change.
   
   
6. Once you have a locked picture, make a note of the used frame ranges of each clip. To help organize and name each clip, it’s helpful to assign each clip a unique name such as “VFX_001” “VFX_002” and so forth. Use a spreadsheet and make a note of the starting frame and end frame. It’s a good idea to add at least 2 seconds on either side in case small changes need to be made later.
   
   
7. If the footage requires pre-stitch processing of individual clips to denoise or increase the resolution, the entire clip should be processed, to preserve frame count further down the pipeline. As the pre-stitched clips are rectilinear at this stage, 360 aware tools are not required. This permits the use of software such as Topaz Video AI or Neat Video denoiser.
   
   
8. Modify the stitching on the select clips, carefully placing edge points and adjusting parameters until you are happy with the stitch results. Select only the used range of the clips and render out full resolution, high quality stitched versions. ProRes 422HQ is a good codec for this as it preserves visual quality at the same high level as ProRes 4444 but for 4:2:2 sources.
   
   
9. Import and place the high quality stitched media on top of your original edit (a process known as ‘reconforming’), toggling on and off the visibility of the clip to make sure the edit matches beneath.
   
   
10. If needed, the next step is to open the high quality stitched media in tools such as After Effects or Mocha Pro to remove any tripod or rigging, and perform other VFX work. Render out a new version using a lossless format that includes the remove operations and with any VFX applied.
    
    
11. If VFX and other compositing work was done, replace the footage in the timeline with these cleaned up versions, again checking to make sure nothing has changed. NLE software often has a replace clip function which can be useful here in making swaps to the footage.
    
    
12. Color correct and color grade the footage. This can be done either in the NLE, or by working with an external tool such as DaVinci Resolve or Misktika Boutique. If working with an external tool, often the individual clips are placed in order for color grading, then once color grading is done they are rendered out as individual clips to be replaced in the NLE.
    
    
13. If receiving audio from a professional post audio team, duplicate the sequence (so that the original version is saved) and replace the audio with either the ambisonic or stereo mix.
    
    
14. The sequence should now contain the audio tracks from the mix team, the final video files, and any graphic elements (titles, picture in picture, etc). Export a ProRes 422HQ version to use as your mastering file. It’s a good idea to also render out a version without text or graphics in case you want to change those elements later and don’t want to re-render any immersive effects.
    
    
15. Encode the ProRes 422HQ master video to the delivery platform specifications. This article on encoding lists details of recommended codecs and dimensions.
    
    
16. Mux (merge) the encoded video file and the spatial audio mix using FB360 Encoder.