Autorotate wrong direction for some android mp4 videos

[Taylor Hughes]

Summary of the bug: For some mp4 uploads from native android apps (probably using a weird and/or broken encoder), ffmpeg interprets the rotation differently from Chrome/Quicktime and autorotates the video incorrectly, despite the rotation/display matrix appearing correct via ffprobe.

How to reproduce:
Given video in this unusual format (attached) with rotation = 90

ffmpeg -i upsidedown-original.mp4 -y upsidedown-out.mp4

Upside down video results with rotation = 0

ffprobe output for original video:

[STREAM]
index=1
codec_name=h264
...
[SIDE_DATA]
side_data_type=Display Matrix
displaymatrix=
00000000:            0      -65536           0
00000001:        65536           0           0
00000002:            0           0  1073741824

rotation=90
[/SIDE_DATA]
[/STREAM]

but ffmpeg -debug log indicates cclock instead of clock rotation for 90deg:

[transpose @ 0x149f0a260] Setting 'dir' to value 'cclock'

resulting output video thus appears 180deg from expectation

[Taylor Hughes]

mp4 from android encoder that is probably somehow invalid

[Taylor Hughes]

log of encoding this mp4, note the rotation = 90 yet dir = cclock (rather than clock)

[Carl Eugen Hoyos]

Chrome and WMP do what FFmpeg does, vlc and (old) Firefox rotate the video differently.

[mkver]

The rotation degree of ffprobe is always counterclockwise.

[quinkblack]

There is a matrix in the track to rotate -90 degrees (counterclockwise), and a matrix in movie header to rotate -180 degrees. The second one is applied after the first one, so it's -270 degrees, which is the same as 90 degrees.

VLC failed to handle the second matrix, FFmpeg does which the specification required.