Opened 18 months ago

Last modified 18 months ago

#8659 new defect

av_display_rotation_set() has no effect in android_camera

Reported by: juha-h Owned by:
Priority: normal Component: avdevice
Version: git-master Keywords: android
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

I'm using ffmpeg android_camera device as video source in my open source baresip VoIP app.

If I make a call by holding the device upright (portrait), video from the front facing camera of the device has been rotated clockwise 90 degrees from portrait orientation.

libavdevice/android_camera.c has this piece of code related to rotation:

av_display_rotation_set(display_matrix, ctx->sensor_orientation);

if (ctx->lens_facing == ACAMERA_LENS_FACING_FRONT) {

av_display_matrix_flip(display_matrix, 1, 0);

}

I added debug to it and found that ctx->sensor_orientation has value 270.0. It is correct value, since according to Android reference CameraCharacteristics.SENSOR_ORIENTATION means:

The orientation of the camera image. The value is the angle that the
camera image needs to be rotated clockwise so it shows correctly on
the display in its natural orientation. It should be 0, 90, 180, or
270.

If av_display_rotation_set(display_matrix, ctx->sensor_orientation) call would work correctly, 90 + 270 = 360, and rotation of the video from the camera would be as it should be, i.e., upright. Now it is 90 degrees from portrait, i.e., av_display_rotation_set() has no effect.

I made more tests by replacing ctx->sensor_orientation argument with various constant values, eg. 0.0, 90.0, 180.0, etc, and noticed that the call has no effect: video is always rotated 90 degrees from portrait.

So, I'm suspecting that there is a bug, which causes av_display_rotation_set() call to have no effect in android_camera.c.

If needed I'm ready to make more tests and try possible patches.

Change History (19)

comment:1 by Carl Eugen Hoyos, 18 months ago

Keywords: android_camera removed
Version: 4.2unspecified

Is the issue you see reproducible with current FFmpeg git head, the only version supported here?

Is the issue reproducible with ffmpeg, the command line interface? If yes, please provide the command line you tested together with the complete, uncut console output to make this a valid ticket. Please post code that can be compiled and tested if the issue is not reproducible with ffmpeg.

comment:2 by juha-h, 18 months ago

Version: unspecifiedgit-master

I just build ffmpeg libs using today's git master and the issue still exists.

I don't currently have ffmpeg cli commands for Android. I'll check if there exists an application that includes them.

I'm not familiar with ffmpeg cli commands. So it would help if someone would tell, what kind of cli command I need to give in order to get stream from Android camera rotated by a given amount of degrees.

comment:3 by juha-h, 18 months ago

I installed termux app and its ffmpeg package and then gave 'ffmpeg -devices' command. It only showed libavfilter input device. So I'm afraid that I cannot reproduce the issue using ffmpeg cli command.

comment:4 by juha-h, 18 months ago

I also checked that 'rotate' filter is in "Enabled filters" of libavfilter.

comment:5 by Carl Eugen Hoyos, 18 months ago

I just compiled current FFmpeg for android and the camera indev gets compiled / configure line missing.

comment:6 by juha-h, 18 months ago

Sorry, but I don't understand what you mean by "configure line missing".

comment:7 by Carl Eugen Hoyos, 18 months ago

You wrote above that you built FFmpeg: To do this, a configure line is necessary. If you show it, I may be able to understand why the indev was not built.

comment:8 by juha-h, 18 months ago

Sorry about the confusion. I have built ffmpeg libraries myself for use in my Android app and in that build android_camera video source exists.

I then tried to find an Android app that would allow execution of ffmpeg cli commands. and discovered termux Android shell app. It has ffmpeg extension package, but unfortunately without android_camera video source.

comment:9 by Carl Eugen Hoyos, 18 months ago

Why don’t you build an ffmpeg binary?

comment:10 by juha-h, 18 months ago

I now built also ffmpeg binary, but it is dynamically linked:

$ file ffmpeg
ffmpeg: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, stripped

I think it must be static so that I could try to run it as is under termux.

comment:11 by Carl Eugen Hoyos, 18 months ago

Android binaries can be dynamically linked, this is only an issue on iOS.

comment:12 by juha-h, 18 months ago

OK. Unfortunately termux does not allow access to Android camera. I installed termux api package https://wiki.termux.com/wiki/Termux:API that has camera related commands, but it never asks for CAMERA permission, which means that those commands fail. So I need to find some other means to execute ffmpeg cli commands on Android. If you know how, let me know.

comment:13 by farstars, 18 months ago

ffmpeg is really easy to use on Android, but it appears you are dealing with an XY problem here: coming from windows or linux you were looking for a way to call ffmpeg like you would on those OS.
However, on Android you usually use wrappers for these kinds of tasks (namely executing command line calls) or include the binaries directly into your app. Luckily there is such a wrapper for ffmpeg so you don't have to do it the hard way.

There are plenty of good tutorials if you google 'android ffmpeg'. I can recommend this one:
https://medium.com/wolox/https-medium-com-wolox-driving-innovation-the-power-of-ffmpeg-on-android-ef6e0c01d59f

comment:14 by juha-h, 18 months ago

I'm using ffmpeg libs from native code, not from Java/Kotlin. In cpp/CMakeLists.txt I have

target_link_libraries(baresip

android
EGL
GLESv2
GLESv1_CM
OpenSLES
lib_baresip
lib_rem
lib_re
lib_ssl
lib_crypto
lib_opus
lib_avdevice
lib_avfilter
lib_swscale
lib_avformat
lib_avcodec
lib_x264
lib_vpx
lib_avutil
lib_swresample
lib_postproc
lib_spandsp
lib_g722_1
lib_ilbc
lib_amrnb
lib_webrtc
lib_zrtp
lib_bn
z
log
camera2ndk
mediandk)

What #include file do I need to add in native code file order to be able to execute ffmpeg cli commands?

Looks like this library http://writingminds.github.io/ffmpeg-android-java/ has not been updated for many years.

comment:16 by juha-h, 18 months ago

I installed mobile-ffmpeg and used it to record video from android_camera like this:

ffmpeg -video_size hd720 -f android_camera -i 0 -camera_index 1 -r 10 -t 5 -y <path>

Resulting video always has this side data no matter in which position I hold the device:

Input #0, android_camera, from '0':
Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc
05-13 18:59:41.746 18715 18715 I Baresip : Side data:
05-13 18:59:41.746 18715 18715 I Baresip : displaymatrix: rotation of -90.00 degrees

I haven't yet figured out how I can set android_camera add_display_matrix() function ctx->sensor_orientation value from a CLI option.

Suggestions on how to test this issue using ffmpeg CLI options would be appreciated.

comment:17 by juha-h, 18 months ago

Tried also using front front camera:

ffmpeg -video_size hd720 -f android_camera -camera_index 1 -i anything -r 10 -t 5 -y <path>

No matter if I hold the device portrait or landscape, the result is the same:

05-13 22:49:32.181 27037 27037 I Baresip : [android_camera @ 0x7823e1cc00] Android camera capture session is active.
05-13 22:49:32.181 27037 27037 I Baresip : Input #0, android_camera, from 'anything':
05-13 22:49:32.181 27037 27037 I Baresip : Duration: N/A, start: 645873.465432, bitrate: N/A
05-13 22:49:32.181 27037 27037 I Baresip : Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc
05-13 22:49:32.181 27037 27037 I Baresip : Side data:
05-13 22:49:32.181 27037 27037 I Baresip : displaymatrix: rotation of 90.00 degrees
05-13 22:49:32.181 27037 27037 I Baresip : Stream mapping:
05-13 22:49:32.181 27037 27037 I Baresip : Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
05-13 22:49:32.181 27037 27037 I Baresip : Output #0, mp4, to '/storage/emulated/0/Download/video.mp4':
05-13 22:49:32.181 27037 27037 I Baresip : Metadata:
05-13 22:49:32.181 27037 27037 I Baresip : encoder : Lavf58.42.100
05-13 22:49:32.181 27037 27037 I Baresip : Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 720x1280, q=2-31, 200 kb/s, 10 fps, 10240 tbn, 10 tbc
05-13 22:49:32.181 27037 27037 I Baresip : Metadata:
05-13 22:49:32.181 27037 27037 I Baresip : encoder : Lavc58.78.102 mpeg4
05-13 22:49:32.181 27037 27037 I Baresip : Side data:
05-13 22:49:32.181 27037 27037 I Baresip : cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
05-13 22:49:32.181 27037 27037 I Baresip : displaymatrix: rotation of -0.00 degrees

VLC shows portrait correctly, but landscape is rotated 90 clockwise degrees.

comment:18 by Carl Eugen Hoyos, 18 months ago

Keywords: android added

Please provide the command line you tested together with the complete, uncut console output to make this a valid ticket. And please try hard to remove the useless additions to the console output.

comment:19 by juha-h, 18 months ago

Command that is executed to record video:

ffmpeg -video_size hd720 -f android_camera -camera_index 1 -i anything -r 10 -t 5 -y <path>

Output when defaultDisplay.rotation of Android device is Surface.ROTATION_0 (= portrait):

05-14 09:13:03.247  4589  4589 I Baresip : [android_camera @ 0x7823e3fc00] Android camera capture session is active.
05-14 09:13:03.247  4589  4589 I Baresip : Input #0, android_camera, from 'anything':
05-14 09:13:03.247  4589  4589 I Baresip :   Duration: N/A, start: 683284.573975, bitrate: N/A
05-14 09:13:03.247  4589  4589 I Baresip :     Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc
05-14 09:13:03.247  4589  4589 I Baresip :     Side data:
05-14 09:13:03.247  4589  4589 I Baresip :       displaymatrix: rotation of 90.00 degrees
05-14 09:13:03.247  4589  4589 I Baresip : Stream mapping:
05-14 09:13:03.247  4589  4589 I Baresip :   Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
05-14 09:13:03.247  4589  4589 I Baresip : Press [q] to stop, [?] for help
05-14 09:13:03.247  4589  4589 I Baresip : [graph 0 input from stream 0:0 @ 0x7823c6b140] sws_param option is deprecated and ignored
05-14 09:13:03.247  4589  4589 I Baresip : Output #0, mp4, to '/storage/emulated/0/Download/video.mp4':
05-14 09:13:03.247  4589  4589 I Baresip :   Metadata:
05-14 09:13:03.247  4589  4589 I Baresip :     encoder         : Lavf58.42.100
05-14 09:13:03.247  4589  4589 I Baresip :     Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 720x1280, q=2-31, 200 kb/s, 10 fps, 10240 tbn, 10 tbc
05-14 09:13:03.247  4589  4589 I Baresip :     Metadata:
05-14 09:13:03.247  4589  4589 I Baresip :       encoder         : Lavc58.78.102 mpeg4
05-14 09:13:03.247  4589  4589 I Baresip :     Side data:
05-14 09:13:03.247  4589  4589 I Baresip :       cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
05-14 09:13:03.247  4589  4589 I Baresip :       displaymatrix: rotation of -0.00 degrees

Output when defaultDisplay.rotation of Android device is Surface.ROTATION_90 (= landscape):

05-14 09:15:09.294  4589  4589 I Baresip : [android_camera @ 0x7823e40e00] Android camera capture session is active.
05-14 09:15:09.294  4589  4589 I Baresip : Input #0, android_camera, from 'anything':
05-14 09:15:09.294  4589  4589 I Baresip :   Duration: N/A, start: 683410.615418, bitrate: N/A
05-14 09:15:09.294  4589  4589 I Baresip :     Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc
05-14 09:15:09.294  4589  4589 I Baresip :     Side data:
05-14 09:15:09.294  4589  4589 I Baresip :       displaymatrix: rotation of 90.00 degrees
05-14 09:15:09.294  4589  4589 I Baresip : Stream mapping:
05-14 09:15:09.294  4589  4589 I Baresip :   Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
05-14 09:15:09.294  4589  4589 I Baresip : Press [q] to stop, [?] for help
05-14 09:15:09.294  4589  4589 I Baresip : [graph 0 input from stream 0:0 @ 0x7813e430c0] sws_param option is deprecated and ignored
05-14 09:15:09.294  4589  4589 I Baresip : Output #0, mp4, to '/storage/emulated/0/Download/video.mp4':
05-14 09:15:09.294  4589  4589 I Baresip :   Metadata:
05-14 09:15:09.294  4589  4589 I Baresip :     encoder         : Lavf58.42.100
05-14 09:15:09.294  4589  4589 I Baresip :     Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 720x1280, q=2-31, 200 kb/s, 10 fps, 10240 tbn, 10 tbc
05-14 09:15:09.294  4589  4589 I Baresip :     Metadata:
05-14 09:15:09.294  4589  4589 I Baresip :       encoder         : Lavc58.78.102 mpeg4
05-14 09:15:09.294  4589  4589 I Baresip :     Side data:
05-14 09:15:09.294  4589  4589 I Baresip :       cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
05-14 09:15:09.294  4589  4589 I Baresip :       displaymatrix: rotation of -0.00 degrees

The issue is that in both cases input stream displaymatrix side data has the same rotation of 90.0 degrees.

Note: See TracTickets for help on using tickets.