av_display_rotation_set() has no effect in android_camera

[juha-h]

I'm using ffmpeg android_camera device as video source in my open source baresip VoIP app.

If I make a call by holding the device upright (portrait), video from the front facing camera of the device has been rotated clockwise 90 degrees from portrait orientation.

libavdevice/android_camera.c has this piece of code related to rotation:

av_display_rotation_set(display_matrix, ctx->sensor_orientation);

if (ctx->lens_facing == ACAMERA_LENS_FACING_FRONT) {

av_display_matrix_flip(display_matrix, 1, 0);

}

I added debug to it and found that ctx->sensor_orientation has value 270.0. It is correct value, since according to Android reference CameraCharacteristics.SENSOR_ORIENTATION means:

The orientation of the camera image. The value is the angle that the
camera image needs to be rotated clockwise so it shows correctly on
the display in its natural orientation. It should be 0, 90, 180, or
270.

If av_display_rotation_set(display_matrix, ctx->sensor_orientation) call would work correctly, 90 + 270 = 360, and rotation of the video from the camera would be as it should be, i.e., upright. Now it is 90 degrees from portrait, i.e., av_display_rotation_set() has no effect.

I made more tests by replacing ctx->sensor_orientation argument with various constant values, eg. 0.0, 90.0, 180.0, etc, and noticed that the call has no effect: video is always rotated 90 degrees from portrait.

So, I'm suspecting that there is a bug, which causes av_display_rotation_set() call to have no effect in android_camera.c.

If needed I'm ready to make more tests and try possible patches.

[Carl Eugen Hoyos]

Is the issue you see reproducible with current FFmpeg git head, the only version supported here?

Is the issue reproducible with ffmpeg, the command line interface? If yes, please provide the command line you tested together with the complete, uncut console output to make this a valid ticket. Please post code that can be compiled and tested if the issue is not reproducible with ffmpeg.

[juha-h]

I just build ffmpeg libs using today's git master and the issue still exists.

I don't currently have ffmpeg cli commands for Android. I'll check if there exists an application that includes them.

I'm not familiar with ffmpeg cli commands. So it would help if someone would tell, what kind of cli command I need to give in order to get stream from Android camera rotated by a given amount of degrees.

[juha-h]

I installed termux app and its ffmpeg package and then gave 'ffmpeg -devices' command. It only showed libavfilter input device. So I'm afraid that I cannot reproduce the issue using ffmpeg cli command.

[juha-h]

I also checked that 'rotate' filter is in "Enabled filters" of libavfilter.

[Carl Eugen Hoyos]

I just compiled current FFmpeg for android and the camera indev gets compiled / configure line missing.

[juha-h]

Sorry, but I don't understand what you mean by "configure line missing".

[Carl Eugen Hoyos]

You wrote above that you built FFmpeg: To do this, a configure line is necessary. If you show it, I may be able to understand why the indev was not built.

[juha-h]

Sorry about the confusion. I have built ffmpeg libraries myself for use in my Android app and in that build android_camera video source exists.

I then tried to find an Android app that would allow execution of ffmpeg cli commands. and discovered termux Android shell app. It has ffmpeg extension package, but unfortunately without android_camera video source.

[Carl Eugen Hoyos]

Why don’t you build an ffmpeg binary?

[juha-h]

I now built also ffmpeg binary, but it is dynamically linked:

$ file ffmpeg
ffmpeg: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, stripped

I think it must be static so that I could try to run it as is under termux.

[Carl Eugen Hoyos]

Android binaries can be dynamically linked, this is only an issue on iOS.

[juha-h]

OK. Unfortunately termux does not allow access to Android camera. I installed termux api package ​https://wiki.termux.com/wiki/Termux:API that has camera related commands, but it never asks for CAMERA permission, which means that those commands fail. So I need to find some other means to execute ffmpeg cli commands on Android. If you know how, let me know.

[farstars]

ffmpeg is really easy to use on Android, but it appears you are dealing with an XY problem here: coming from windows or linux you were looking for a way to call ffmpeg like you would on those OS.
However, on Android you usually use wrappers for these kinds of tasks (namely executing command line calls) or include the binaries directly into your app. Luckily there is such a wrapper for ffmpeg so you don't have to do it the hard way.

There are plenty of good tutorials if you google 'android ffmpeg'. I can recommend this one:
​https://medium.com/wolox/https-medium-com-wolox-driving-innovation-the-power-of-ffmpeg-on-android-ef6e0c01d59f

[juha-h]

I'm using ffmpeg libs from native code, not from Java/Kotlin. In cpp/CMakeLists.txt I have

target_link_libraries(baresip

android
EGL
GLESv2
GLESv1_CM
OpenSLES
lib_baresip
lib_rem
lib_re
lib_ssl
lib_crypto
lib_opus
lib_avdevice
lib_avfilter
lib_swscale
lib_avformat
lib_avcodec
lib_x264
lib_vpx
lib_avutil
lib_swresample
lib_postproc
lib_spandsp
lib_g722_1
lib_ilbc
lib_amrnb
lib_webrtc
lib_zrtp
lib_bn
z
log
camera2ndk
mediandk)

What #include file do I need to add in native code file order to be able to execute ffmpeg cli commands?

Looks like this library ​http://writingminds.github.io/ffmpeg-android-java/ has not been updated for many years.

[juha-h]

I'll try with this: ​https://github.com/tanersener/mobile-ffmpeg.

[juha-h]

I installed mobile-ffmpeg and used it to record video from android_camera like this:

ffmpeg -video_size hd720 -f android_camera -i 0 -camera_index 1 -r 10 -t 5 -y <path>

Resulting video always has this side data no matter in which position I hold the device:

Input #0, android_camera, from '0':
Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc
05-13 18:59:41.746 18715 18715 I Baresip : Side data:
05-13 18:59:41.746 18715 18715 I Baresip : displaymatrix: rotation of -90.00 degrees

I haven't yet figured out how I can set android_camera add_display_matrix() function ctx->sensor_orientation value from a CLI option.

Suggestions on how to test this issue using ffmpeg CLI options would be appreciated.

[juha-h]

Tried also using front front camera:

ffmpeg -video_size hd720 -f android_camera -camera_index 1 -i anything -r 10 -t 5 -y <path>

No matter if I hold the device portrait or landscape, the result is the same:

05-13 22:49:32.181 27037 27037 I Baresip : [android_camera @ 0x7823e1cc00] Android camera capture session is active.
05-13 22:49:32.181 27037 27037 I Baresip : Input #0, android_camera, from 'anything':
05-13 22:49:32.181 27037 27037 I Baresip : Duration: N/A, start: 645873.465432, bitrate: N/A
05-13 22:49:32.181 27037 27037 I Baresip : Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc
05-13 22:49:32.181 27037 27037 I Baresip : Side data:
05-13 22:49:32.181 27037 27037 I Baresip : displaymatrix: rotation of 90.00 degrees
05-13 22:49:32.181 27037 27037 I Baresip : Stream mapping:
05-13 22:49:32.181 27037 27037 I Baresip : Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
05-13 22:49:32.181 27037 27037 I Baresip : Output #0, mp4, to '/storage/emulated/0/Download/video.mp4':
05-13 22:49:32.181 27037 27037 I Baresip : Metadata:
05-13 22:49:32.181 27037 27037 I Baresip : encoder : Lavf58.42.100
05-13 22:49:32.181 27037 27037 I Baresip : Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 720x1280, q=2-31, 200 kb/s, 10 fps, 10240 tbn, 10 tbc
05-13 22:49:32.181 27037 27037 I Baresip : Metadata:
05-13 22:49:32.181 27037 27037 I Baresip : encoder : Lavc58.78.102 mpeg4
05-13 22:49:32.181 27037 27037 I Baresip : Side data:
05-13 22:49:32.181 27037 27037 I Baresip : cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
05-13 22:49:32.181 27037 27037 I Baresip : displaymatrix: rotation of -0.00 degrees

VLC shows portrait correctly, but landscape is rotated 90 clockwise degrees.

[Carl Eugen Hoyos]

Please provide the command line you tested together with the complete, uncut console output to make this a valid ticket. And please try hard to remove the useless additions to the console output.

[juha-h]

Command that is executed to record video:

ffmpeg -video_size hd720 -f android_camera -camera_index 1 -i anything -r 10 -t 5 -y <path>

Output when defaultDisplay.rotation of Android device is Surface.ROTATION_0 (= portrait):

05-14 09:13:03.247  4589  4589 I Baresip : [android_camera @ 0x7823e3fc00] Android camera capture session is active.
05-14 09:13:03.247  4589  4589 I Baresip : Input #0, android_camera, from 'anything':
05-14 09:13:03.247  4589  4589 I Baresip :   Duration: N/A, start: 683284.573975, bitrate: N/A
05-14 09:13:03.247  4589  4589 I Baresip :     Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc
05-14 09:13:03.247  4589  4589 I Baresip :     Side data:
05-14 09:13:03.247  4589  4589 I Baresip :       displaymatrix: rotation of 90.00 degrees
05-14 09:13:03.247  4589  4589 I Baresip : Stream mapping:
05-14 09:13:03.247  4589  4589 I Baresip :   Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
05-14 09:13:03.247  4589  4589 I Baresip : Press [q] to stop, [?] for help
05-14 09:13:03.247  4589  4589 I Baresip : [graph 0 input from stream 0:0 @ 0x7823c6b140] sws_param option is deprecated and ignored
05-14 09:13:03.247  4589  4589 I Baresip : Output #0, mp4, to '/storage/emulated/0/Download/video.mp4':
05-14 09:13:03.247  4589  4589 I Baresip :   Metadata:
05-14 09:13:03.247  4589  4589 I Baresip :     encoder         : Lavf58.42.100
05-14 09:13:03.247  4589  4589 I Baresip :     Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 720x1280, q=2-31, 200 kb/s, 10 fps, 10240 tbn, 10 tbc
05-14 09:13:03.247  4589  4589 I Baresip :     Metadata:
05-14 09:13:03.247  4589  4589 I Baresip :       encoder         : Lavc58.78.102 mpeg4
05-14 09:13:03.247  4589  4589 I Baresip :     Side data:
05-14 09:13:03.247  4589  4589 I Baresip :       cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
05-14 09:13:03.247  4589  4589 I Baresip :       displaymatrix: rotation of -0.00 degrees

Output when defaultDisplay.rotation of Android device is Surface.ROTATION_90 (= landscape):

05-14 09:15:09.294  4589  4589 I Baresip : [android_camera @ 0x7823e40e00] Android camera capture session is active.
05-14 09:15:09.294  4589  4589 I Baresip : Input #0, android_camera, from 'anything':
05-14 09:15:09.294  4589  4589 I Baresip :   Duration: N/A, start: 683410.615418, bitrate: N/A
05-14 09:15:09.294  4589  4589 I Baresip :     Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc
05-14 09:15:09.294  4589  4589 I Baresip :     Side data:
05-14 09:15:09.294  4589  4589 I Baresip :       displaymatrix: rotation of 90.00 degrees
05-14 09:15:09.294  4589  4589 I Baresip : Stream mapping:
05-14 09:15:09.294  4589  4589 I Baresip :   Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
05-14 09:15:09.294  4589  4589 I Baresip : Press [q] to stop, [?] for help
05-14 09:15:09.294  4589  4589 I Baresip : [graph 0 input from stream 0:0 @ 0x7813e430c0] sws_param option is deprecated and ignored
05-14 09:15:09.294  4589  4589 I Baresip : Output #0, mp4, to '/storage/emulated/0/Download/video.mp4':
05-14 09:15:09.294  4589  4589 I Baresip :   Metadata:
05-14 09:15:09.294  4589  4589 I Baresip :     encoder         : Lavf58.42.100
05-14 09:15:09.294  4589  4589 I Baresip :     Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 720x1280, q=2-31, 200 kb/s, 10 fps, 10240 tbn, 10 tbc
05-14 09:15:09.294  4589  4589 I Baresip :     Metadata:
05-14 09:15:09.294  4589  4589 I Baresip :       encoder         : Lavc58.78.102 mpeg4
05-14 09:15:09.294  4589  4589 I Baresip :     Side data:
05-14 09:15:09.294  4589  4589 I Baresip :       cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
05-14 09:15:09.294  4589  4589 I Baresip :       displaymatrix: rotation of -0.00 degrees

The issue is that in both cases input stream displaymatrix side data has the same rotation of 90.0 degrees.