Version 2 (modified by jakubk, 7 years ago) (diff)


Using ffmpeg in a VFX pipeline

Every VFX pipeline needs a way of converting still frames into motion sequences that can be played back on a large screen or projector (and/or directly on artists' workstations) for the purposes of doing a review or just seeing the resulting work in motion. It is perfectly possible to play back high resolution frames directly, but such a setup requires an incredible amount of throughput bandwidth (a 2K sequence will need 300MB/s for seamless playback) and an even larger amount of storage space (the average size for a 2K frame is about 12MB for a 10bit DPX, and a 16bit EXR is about 14MB in size). Encoding these frames into a single compressed video file provides the option to quickly preview the work, makes it portable and is much more suited for quick daily reviews of the work. Full resolution frames should always be used for final reviews and color correct final grades, but that would be performed on speciality hardware/software such as a Mistika suite or a Nucoda Filmmaster attached to a high performance SAN.

The basic requirements for generating movie clips in a VFX pipeline can be summed up into the following points:

  • Resulting clip should be playable on all three major OS's used in VFX: Mac OS X, Windows and most of all, Linux
  • The codec and review player should allow for frame-by-frame scrubbing of the clip
  • There should be some level of compression. This is obviously based on each studios capability in terms of storage space available.
  • Portable devices such as iPads are becoming increasingly more popular for doing film reviews. The clip encoding part of the VFX pipeline should be ideally capable of producing videos playable on these devices
  • The color must be as close as possible to the original source frames
  • The clip should play back on a reasonable hardware setup (you should not need a SAN to do daily reviews)
  • There should be a possibility of creating a mono and a stereo (3D) version of the clip

Out of all the codecs that are supported by ffmpeg, only some of them are usable in a VFX pipeline and possibly none of them meet all of the criteria outlined above, at the time of writing. Some areas will require compromise, but in general it is perfectly possible to successfully use ffmpeg in a such an environment.

Input The default image file format for a high-end VFX workflow is Lucasfilm's OpenEXR (.exr). In addition to the standard .exr file, Weta Digital has developed a stereo (3D) extension to the EXR standard called SXR, which is basically a container for both the left and the right eyes within one file. (Saves the data managers having to worry about 2 sequences of files per each stereo stream). Unfortunately, at the time of this writing, OpenEXR is not very well supported in ffmpeg (despite there being claims of OpenEXR support) and SXR is definitely not supported. This means that the first part of the workflow must be a file conversion from EXR to either DPX or TIFF (or lossless JPEG).

Describing this conversion process is outside the scope of this guide, but there are many ways to skin this cat. The easiest one is to use Nuke to create a reader node for the SXRs, attach a writer node to it and have it write out a sequence of converted frames. The one thing to keep in mind is that many VFX workflows work with a color LUT to manipulate the look of the resulting images. FFmpeg does not have a way of applying a text-based LUT to its inputs so the LUT must be applied during the conversion process. Once we have a sequence of DPX/TIFF/JPEG/whatever images, we can proceed with encoding them into a movie clip.

One thing to note is that the high dynamic range of EXR (and 16bits per channel) will be "flattened" once the frames are converted to DPX or some other format, but this is OK because the best most video codecs can do is a 10bit depth per channel anyway.