Opened 4 years ago

Closed 4 weeks ago

Last modified 4 weeks ago

#6945 closed enhancement (fixed)

ffmpeg fails at jpeg EXIF orientation (test included)

Reported by: Johnny Owned by:
Priority: wish Component: avcodec
Version: git-master Keywords: mjpeg rotate
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no


Summary of the bug:

If you try viewing these test images in any other image viewer, they'll all display them correctly (all in the same orientation). Ffmpeg on the other hand fails to display them correctly.

This is a major issue when using ffmpeg-based players as an image viewer, because most images these days (from digital cameras and phones) use EXIF orientation to tell the viewer how to display the image.

Trying to use ffmpeg as an image viewer currently leads to a sad experience where images are rotated and flipped into crazy angles, rather than being displayed correctly. ;-)

How to reproduce:

  • Try viewing/converting the JPEG with ffmpeg:
ffmpeg -i Landscape_2.jpg test.png
  • The result will be badly flipped/rotated.
  • All images are supposed to look identical (as they will in a correct viewer).

What needs fixing:

  • So the fix would be: "If input format == jpeg/exif with orientation flag, apply the orientation filters to the output".

There are 8 JPEG EXIF orientations. They represent various combinations of rotate + vflip + hflip. And all of the code for rotate/flip filters is already available in ffmpeg. Just look at the video autorotate code above. It seems to be very easy to add this JPEG fix by just looking at that commit and the EXIF commit.

I don't know this codebase at all and would just screw it up if I attempt this fix. Hopefully someone here wants to fix ffmpeg's broken JPEG support. It's important these days, since most camera sensors capture pixels in a single orientation and then add an EXIF orientation tag to assign the display-orientation. By lacking that feature, ffmpeg currently fails badly at almost all JPEG input.

Change History (9)

comment:1 by Carl Eugen Hoyos, 4 years ago

Keywords: exif jpg jpeg orientation removed
Priority: importantnormal
Resolution: duplicate
Status: newclosed

in reply to:  1 comment:2 by Johnny, 4 years ago

Replying to cehoyos:

Hey cehoyos. Would you mind saying what this is a duplicate of? I did a lot of searching before opening this ticket.

in reply to:  1 comment:3 by Johnny, 4 years ago

Replying to cehoyos:

Hey cehoyos. Would you mind saying what this is a duplicate of? I did a lot of searching before opening this ticket.

comment:4 by Carl Eugen Hoyos, 4 years ago

I thought you mentioned this is a duplicate of ticket #4149, is it not?

in reply to:  4 comment:5 by Johnny, 4 years ago

Replying to cehoyos:

I thought you mentioned this is a duplicate of ticket #4149, is it not?

Hehe, no. Please re-open this ticket.

I apologize if I wasn't clear. I was only referring to those old tickets since they contained discussions about "ffmpeg's stance" on auto-rotation of input material.

Number 4149 is someone who asked about "passthrough" writing (output) of video rotation data into ffmpeg's generated mjpeg snapshots from videos. It's a very muddy ticket actually. You might even want to close that one or ask for more info in it, to see what they really need (and it's 3 years old, so it seems dead). I only linked to it because it contained some related statements.

This ticket on the other hand is about something that's close to "critical" level: ffmpeg's reading of (input) jpeg files is totally incorrect because it unfortunately doesn't insert vflip/hflip/rotate video transformation filters to the output graph as-required by the the jpeg format's EXIF orientation value. This means that ffmpeg is unable to understand large amounts of modern jpegs taken from digital cameras, which very frequently use EXIF orientation to assign the final display-orientation. You can try out ffmpeg with the test files in my original message above. The result is very sad. :-\ ffmpeg will decode those jpegs as their raw pixels without any of their assigned transformations, which means that ffmpeg's output will just be randomly rotated and flipped (mirrored) rather than their intended orientation.

It seems like the appropriate fix would be to combine the techniques from the two linked commits; the one that handles video autorotate and the one that handles jpeg EXIF reading.

comment:6 by Carl Eugen Hoyos, 4 years ago

Component: undeterminedavcodec
Keywords: mjpeg rotate added
Priority: normalwish
Resolution: duplicate
Status: closedreopened
Type: defectenhancement
Version: unspecifiedgit-master

comment:7 by James, 4 weeks ago

Resolution: fixed
Status: reopenedclosed

comment:8 by U. Artie Eoff, 4 weeks ago

This fix breaks vaapi jpeg 422 decoding:

$ ffmpeg -hwaccel vaapi -init_hw_device vaapi=hw:/dev/dri/renderD128 \
 -hwaccel_output_format vaapi -v verbose -i some-422.jpg \
 -vf 'hwdownload,format=yuv422p' -pix_fmt yuv422p \
 -f rawvideo -vframes 1 -y output.yuv

ffmpeg version N-103752-g59719a905c5e Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (GCC)
  configuration: --prefix=/home/uaeoff/Work/workspace/media/install --disable-static --enable-shared --enable-libdrm --enable-vaapi --enable-libmfx --disable-amf --disable-audiotoolbox --disable-cuda --disable-cuda-sdk --disable-cuvid --disable-d3d11va --disable-dxva2 --disable-libnpp --disable-mmal --disable-nvdec --disable-nvenc --disable-omx --disable-omx-rpi --disable-rkmpp --disable-v4l2-m2m --disable-vdpau --disable-videotoolbox --enable-gpl --enable-libx264 --enable-libx265
  libavutil      57.  6.100 / 57.  6.100
  libavcodec     59.  9.100 / 59.  9.100
  libavformat    59.  5.100 / 59.  5.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8.  9.100 /  8.  9.100
  libswscale      6.  1.100 /  6.  1.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
[AVHWDeviceContext @ 0xc9c040] libva: VA-API version 1.13.0
[AVHWDeviceContext @ 0xc9c040] libva: User environment variable requested driver 'iHD'
[AVHWDeviceContext @ 0xc9c040] libva: Trying to open /home/uaeoff/Work/workspace/media/install/lib/dri/
[AVHWDeviceContext @ 0xc9c040] libva: Found init function __vaDriverInit_1_13
[AVHWDeviceContext @ 0xc9c040] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0xc9c040] Initialised VAAPI connection: version 1.13
[AVHWDeviceContext @ 0xc9c040] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 21.3.4 ().
[AVHWDeviceContext @ 0xc9c040] Driver not found in known nonstandard list, using standard behaviour.
Input #0, image2, from 'some-422.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 217499 kb/s
  Stream #0:0: Video: mjpeg (Baseline), 1 reference frame, yuvj422p(pc, bt470bg/unknown/unknown, center), 2048x1536, 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> rawvideo (native))
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 0xeb4b40] w:2048 h:1536 pixfmt:vaapi tb:1/25 fr:25/1 sar:0/1
[auto_scale_0 @ 0xeb8d40] w:iw h:ih flags:'' interl:0
[Parsed_hwdownload_0 @ 0xe51a40] auto-inserting filter 'auto_scale_0' between the filter 'transpose' and the filter 'Parsed_hwdownload_0'
Impossible to convert between the formats supported by the filter 'transpose' and the filter 'auto_scale_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
[AVIOContext @ 0xcec300] Statistics: 0 seeks, 0 writeouts
[AVIOContext @ 0xce4380] Statistics: 1087496 bytes read, 0 seeks
Conversion failed!

comment:9 by U. Artie Eoff, 4 weeks ago

See regression reported in #9432

Note: See TracTickets for help on using tickets.