ffmpeg fails at jpeg EXIF orientation (test included)

[Johnny]

Summary of the bug:

If you try viewing these test images in any other image viewer, they'll all display them correctly (all in the same orientation). Ffmpeg on the other hand fails to display them correctly.

This is a major issue when using ffmpeg-based players as an image viewer, because most images these days (from digital cameras and phones) use EXIF orientation to tell the viewer how to display the image.

Trying to use ffmpeg as an image viewer currently leads to a sad experience where images are rotated and flipped into crazy angles, rather than being displayed correctly. ;-)

How to reproduce:

• Download EXIF orientation test images from this repository (it covers all available JPG orientations): ​https://github.com/recurser/exif-orientation-examples

• Try viewing/converting the JPEG with ffmpeg:

ffmpeg -i Landscape_2.jpg test.png

• The result will be badly flipped/rotated.

• All images are supposed to look identical (as they will in a correct viewer).

What needs fixing:

• 4 years ago (2014), ffmpeg added EXIF metadata reading: ​http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=bb4e1b4cf910af0de2bc884c75544603c40010cc

• 3 years ago (2015), ffmpeg added video autorotation: ​http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=f5b26fbc2f564831b337f1de9faaaf6753a03871

• According to discussions (​https://trac.ffmpeg.org/ticket/4149 and ​https://trac.ffmpeg.org/ticket/515), ffmpeg's stance is that it should always autorotate/autoflip the input by default to respect display-orientation flags. So ffmpeg missing the "orientation fix" for JPEG is just an oversight.

• So the fix would be: "If input format == jpeg/exif with orientation flag, apply the orientation filters to the output".

There are 8 JPEG EXIF orientations. They represent various combinations of rotate + vflip + hflip. And all of the code for rotate/flip filters is already available in ffmpeg. Just look at the video autorotate code above. It seems to be very easy to add this JPEG fix by just looking at that commit and the EXIF commit.

I don't know this codebase at all and would just screw it up if I attempt this fix. Hopefully someone here wants to fix ffmpeg's broken JPEG support. It's important these days, since most camera sensors capture pixels in a single orientation and then add an EXIF orientation tag to assign the display-orientation. By lacking that feature, ffmpeg currently fails badly at almost all JPEG input.

[Johnny]

Replying to cehoyos:

Hey cehoyos. Would you mind saying what this is a duplicate of? I did a lot of searching before opening this ticket.

[Johnny]

Replying to cehoyos:

Hey cehoyos. Would you mind saying what this is a duplicate of? I did a lot of searching before opening this ticket.

[Carl Eugen Hoyos]

I thought you mentioned this is a duplicate of ticket #4149, is it not?

[Johnny]

Replying to cehoyos:

I thought you mentioned this is a duplicate of ticket #4149, is it not?

Hehe, no. Please re-open this ticket.

I apologize if I wasn't clear. I was only referring to those old tickets since they contained discussions about "ffmpeg's stance" on auto-rotation of input material.

Number 4149 is someone who asked about "passthrough" writing (output) of video rotation data into ffmpeg's generated mjpeg snapshots from videos. It's a very muddy ticket actually. You might even want to close that one or ask for more info in it, to see what they really need (and it's 3 years old, so it seems dead). I only linked to it because it contained some related statements.

This ticket on the other hand is about something that's close to "critical" level: ffmpeg's reading of (input) jpeg files is totally incorrect because it unfortunately doesn't insert vflip/hflip/rotate video transformation filters to the output graph as-required by the the jpeg format's EXIF orientation value. This means that ffmpeg is unable to understand large amounts of modern jpegs taken from digital cameras, which very frequently use EXIF orientation to assign the final display-orientation. You can try out ffmpeg with the test files in my original message above. The result is very sad. :-\ ffmpeg will decode those jpegs as their raw pixels without any of their assigned transformations, which means that ffmpeg's output will just be randomly rotated and flipped (mirrored) rather than their intended orientation.

It seems like the appropriate fix would be to combine the techniques from the two linked commits; the one that handles video autorotate and the one that handles jpeg EXIF reading.

[James]

Implemented in e93c9986027d17917c3b4f533b28ee4a2ce7cd4c.

[U. Artie Eoff]

This fix breaks vaapi jpeg 422 decoding:

$ ffmpeg -hwaccel vaapi -init_hw_device vaapi=hw:/dev/dri/renderD128 \
 -hwaccel_output_format vaapi -v verbose -i some-422.jpg \
 -vf 'hwdownload,format=yuv422p' -pix_fmt yuv422p \
 -f rawvideo -vframes 1 -y output.yuv

ffmpeg version N-103752-g59719a905c5e Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (GCC)
  configuration: --prefix=/home/uaeoff/Work/workspace/media/install --disable-static --enable-shared --enable-libdrm --enable-vaapi --enable-libmfx --disable-amf --disable-audiotoolbox --disable-cuda --disable-cuda-sdk --disable-cuvid --disable-d3d11va --disable-dxva2 --disable-libnpp --disable-mmal --disable-nvdec --disable-nvenc --disable-omx --disable-omx-rpi --disable-rkmpp --disable-v4l2-m2m --disable-vdpau --disable-videotoolbox --enable-gpl --enable-libx264 --enable-libx265
  libavutil      57.  6.100 / 57.  6.100
  libavcodec     59.  9.100 / 59.  9.100
  libavformat    59.  5.100 / 59.  5.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8.  9.100 /  8.  9.100
  libswscale      6.  1.100 /  6.  1.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
[AVHWDeviceContext @ 0xc9c040] libva: VA-API version 1.13.0
[AVHWDeviceContext @ 0xc9c040] libva: User environment variable requested driver 'iHD'
[AVHWDeviceContext @ 0xc9c040] libva: Trying to open /home/uaeoff/Work/workspace/media/install/lib/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0xc9c040] libva: Found init function __vaDriverInit_1_13
[AVHWDeviceContext @ 0xc9c040] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0xc9c040] Initialised VAAPI connection: version 1.13
[AVHWDeviceContext @ 0xc9c040] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 21.3.4 ().
[AVHWDeviceContext @ 0xc9c040] Driver not found in known nonstandard list, using standard behaviour.
Input #0, image2, from 'some-422.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 217499 kb/s
  Stream #0:0: Video: mjpeg (Baseline), 1 reference frame, yuvj422p(pc, bt470bg/unknown/unknown, center), 2048x1536, 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> rawvideo (native))
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 0xeb4b40] w:2048 h:1536 pixfmt:vaapi tb:1/25 fr:25/1 sar:0/1
[auto_scale_0 @ 0xeb8d40] w:iw h:ih flags:'' interl:0
[Parsed_hwdownload_0 @ 0xe51a40] auto-inserting filter 'auto_scale_0' between the filter 'transpose' and the filter 'Parsed_hwdownload_0'
Impossible to convert between the formats supported by the filter 'transpose' and the filter 'auto_scale_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
[AVIOContext @ 0xcec300] Statistics: 0 seeks, 0 writeouts
[AVIOContext @ 0xce4380] Statistics: 1087496 bytes read, 0 seeks
Conversion failed!

[U. Artie Eoff]

See regression reported in #9432