Should retain pixel density metadata from HiDPI/Retina screen recordings

[moubry]

Summary of the bug:
Whenever I use ffmpeg to encode a HiDPI/Retina screen recording, the video plays at 2x the size, so it looks fuzzy, because the pixel density is not retained. I expect videos encoded with ffmpeg to retain whatever metadata is used by players (like QuickTime Player) to play HiDPI/Retina videos at the expected size.

I'm experiencing this issue on a Mac, but it might be an issue across other platforms with HiDPi displays as well.

$ ffmpeg version
ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.39.2)

How to reproduce:

1. Use QuickTime Player to create a Screen Recording on a Retina Mac.
2. Play the video you recorded in QuickTime Player using the ⌘1 Actual Size view. Notice that it’s playing 2:1 on your Retina Display, so the video looks sharp. It’s playing in half the space of the actual recorded pixels.
3. Use ffmpeg to encode the video using a command like this:

   $ ffmpeg -i haha.mov -c:v libx264 -crf 23 haha-lg.mov
   

4. Play the new ffmpeg-compressed video in QuickTime Player using the ⌘1 Actual Size view. Notice that it’s playing 1:1, so the video looks fuzzy.

To clarify, the video does not look blurry because it was compressed. Rather, it looks blurry because the video is being played twice as big as it should be, at a 1:1 pixel density, instead of the required 2:1 pixel density, presumably because some metadata is being discarded when encoding.

Here is the detailed information ffmpeg shows for the original screen recording:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'haha.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 0
    compatible_brands: qt  
    creation_time   : 2018-02-26T16:46:00.000000Z
    com.apple.quicktime.make: Apple
    com.apple.quicktime.model: iMac18,3
    com.apple.quicktime.software: Mac OS X 10.13.3 (17D102)
    com.apple.quicktime.creationdate: 2018-02-26T10:45:50-0600
  Duration: 00:00:04.35, start: 0.000000, bitrate: 10947 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709), 1396x928 [SAR 1:1 DAR 349:232], 10701 kb/s, 60 fps, 60 tbr, 6k tbn, 12k tbc (default)
    Metadata:
      creation_time   : 2018-02-26T16:46:00.000000Z
      handler_name    : Core Media Data Handler
      encoder         : H.264

And here is the information for the ffmpeg-compressed version:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'haha-lg.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 512
    compatible_brands: qt  
    encoder         : Lavf57.83.100
  Duration: 00:00:04.35, start: 0.000000, bitrate: 1923 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1396x928 [SAR 1:1 DAR 349:232], 1783 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    Metadata:
      handler_name    : DataHandler
      encoder         : Lavc57.107.100 libx264

[moubry]

For the record, VLC plays both videos too big (blurry). So being able to play HiDPI videos at the expected pixel density seems to be a feature of QuickTime Player.

[Carl Eugen Hoyos]

Please test current FFmpeg git head and provide the command line you tested together with the complete, uncut console output and upload an input file to make this a valid ticket.

[moubry]

Original screen recording from QuickTime Player (this correctly plays at 2x)

[moubry]

Compressed by ffmpeg/lix264 (this plays at 1x instead of the expected 2x)

[moubry]

Replying to cehoyos:

Please test current FFmpeg git head and provide the command line you tested together with the complete, uncut console output and upload an input file to make this a valid ticket.

I've added the things you've asked for. I reinstalled ffmpeg from head, compressed a sample video, and attached the terminal output as well as the before and after videos. I hope this helps!

[Carl Eugen Hoyos]

Is the issue also reproducible with the following command line?

$ ffmpeg -i retina_sample.mov -vcodec copy out.mov

[hughes]

This seems to be caused by dropping the com.apple.quicktime.pixeldensity metadata tag. It's present in the movie atoms prior to conversion, but absent after.

[hughes]

Specifically, it has a value of 0x30 prior to conversion. Confirmed that setting it to any other value seems to revert the lower DPI during playback.

This metadata is actually part of the trak atom meta, not the moov meta.

[Daniel Kaiser]

I recently implemented this feature to generate mov files for HiDPI screens. And unlike mentioned in the previous comment a bit more data is required for this feature to work properly, but it also offers more possibilities than simply showing videos in half scale.

As already mentioned the com.apple.quicktime.pixeldensity metadata tag is required in the trak meta atom followed by a value of 0x30. However 0x30 is not the actual value of this metadata tag but the size of the following ilst atom which (among other things) includes 16 Bytes of data representing 4 uint32_t. The first pair of these values seems to specify the actual pixel resolution of the video stream, the second specifies the desired playback resolution (unfortunately, I wasn't able to find any documentation from Apple proving this, but regarding my tests it is almost certainly the case). This allows arbitrary playback resolutions to be defined through the metadata.

Attached is a diff based on ffmpeg version 4.2.1 adding this feature to mov encoding using the write_pixeldensity flag. This is a quick-and-dirty implementation with the primary aim to keep the patch simple and small setting the playback resolution to a fixed half scale. This code is far from production ready and containing a lot of hard-coded values but it might be a help for someone who wants to implement this properly.

[Carl Eugen Hoyos]

Please send your patch - made with git format-patch - to the FFmpeg development mailing list where it can be reviewed.