Opened 4 years ago
Last modified 4 years ago
#8659 new defect
av_display_rotation_set() has no effect in android_camera
Reported by: | juha-h | Owned by: | |
---|---|---|---|
Priority: | normal | Component: | avdevice |
Version: | git-master | Keywords: | android |
Cc: | Blocked By: | ||
Blocking: | Reproduced by developer: | no | |
Analyzed by developer: | no |
Description
I'm using ffmpeg android_camera device as video source in my open source baresip VoIP app.
If I make a call by holding the device upright (portrait), video from the front facing camera of the device has been rotated clockwise 90 degrees from portrait orientation.
libavdevice/android_camera.c has this piece of code related to rotation:
av_display_rotation_set(display_matrix, ctx->sensor_orientation);
if (ctx->lens_facing == ACAMERA_LENS_FACING_FRONT) {
av_display_matrix_flip(display_matrix, 1, 0);
}
I added debug to it and found that ctx->sensor_orientation has value 270.0. It is correct value, since according to Android reference CameraCharacteristics.SENSOR_ORIENTATION means:
The orientation of the camera image. The value is the angle that the
camera image needs to be rotated clockwise so it shows correctly on
the display in its natural orientation. It should be 0, 90, 180, or
270.
If av_display_rotation_set(display_matrix, ctx->sensor_orientation) call would work correctly, 90 + 270 = 360, and rotation of the video from the camera would be as it should be, i.e., upright. Now it is 90 degrees from portrait, i.e., av_display_rotation_set() has no effect.
I made more tests by replacing ctx->sensor_orientation argument with various constant values, eg. 0.0, 90.0, 180.0, etc, and noticed that the call has no effect: video is always rotated 90 degrees from portrait.
So, I'm suspecting that there is a bug, which causes av_display_rotation_set() call to have no effect in android_camera.c.
If needed I'm ready to make more tests and try possible patches.
Change History (19)
comment:1 by , 4 years ago
Keywords: | android_camera removed |
---|---|
Version: | 4.2 → unspecified |
comment:2 by , 4 years ago
Version: | unspecified → git-master |
---|
I just build ffmpeg libs using today's git master and the issue still exists.
I don't currently have ffmpeg cli commands for Android. I'll check if there exists an application that includes them.
I'm not familiar with ffmpeg cli commands. So it would help if someone would tell, what kind of cli command I need to give in order to get stream from Android camera rotated by a given amount of degrees.
comment:3 by , 4 years ago
I installed termux app and its ffmpeg package and then gave 'ffmpeg -devices' command. It only showed libavfilter input device. So I'm afraid that I cannot reproduce the issue using ffmpeg cli command.
comment:4 by , 4 years ago
I also checked that 'rotate' filter is in "Enabled filters" of libavfilter.
comment:5 by , 4 years ago
I just compiled current FFmpeg for android and the camera indev gets compiled / configure line missing.
comment:7 by , 4 years ago
You wrote above that you built FFmpeg: To do this, a configure line is necessary. If you show it, I may be able to understand why the indev was not built.
comment:8 by , 4 years ago
Sorry about the confusion. I have built ffmpeg libraries myself for use in my Android app and in that build android_camera video source exists.
I then tried to find an Android app that would allow execution of ffmpeg cli commands. and discovered termux Android shell app. It has ffmpeg extension package, but unfortunately without android_camera video source.
comment:10 by , 4 years ago
I now built also ffmpeg binary, but it is dynamically linked:
$ file ffmpeg
ffmpeg: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, stripped
I think it must be static so that I could try to run it as is under termux.
comment:11 by , 4 years ago
Android binaries can be dynamically linked, this is only an issue on iOS.
comment:12 by , 4 years ago
OK. Unfortunately termux does not allow access to Android camera. I installed termux api package https://wiki.termux.com/wiki/Termux:API that has camera related commands, but it never asks for CAMERA permission, which means that those commands fail. So I need to find some other means to execute ffmpeg cli commands on Android. If you know how, let me know.
comment:13 by , 4 years ago
ffmpeg is really easy to use on Android, but it appears you are dealing with an XY problem here: coming from windows or linux you were looking for a way to call ffmpeg like you would on those OS.
However, on Android you usually use wrappers for these kinds of tasks (namely executing command line calls) or include the binaries directly into your app. Luckily there is such a wrapper for ffmpeg so you don't have to do it the hard way.
There are plenty of good tutorials if you google 'android ffmpeg'. I can recommend this one:
https://medium.com/wolox/https-medium-com-wolox-driving-innovation-the-power-of-ffmpeg-on-android-ef6e0c01d59f
comment:14 by , 4 years ago
I'm using ffmpeg libs from native code, not from Java/Kotlin. In cpp/CMakeLists.txt I have
target_link_libraries(baresip
android
EGL
GLESv2
GLESv1_CM
OpenSLES
lib_baresip
lib_rem
lib_re
lib_ssl
lib_crypto
lib_opus
lib_avdevice
lib_avfilter
lib_swscale
lib_avformat
lib_avcodec
lib_x264
lib_vpx
lib_avutil
lib_swresample
lib_postproc
lib_spandsp
lib_g722_1
lib_ilbc
lib_amrnb
lib_webrtc
lib_zrtp
lib_bn
z
log
camera2ndk
mediandk)
What #include file do I need to add in native code file order to be able to execute ffmpeg cli commands?
Looks like this library http://writingminds.github.io/ffmpeg-android-java/ has not been updated for many years.
comment:16 by , 4 years ago
I installed mobile-ffmpeg and used it to record video from android_camera like this:
ffmpeg -video_size hd720 -f android_camera -i 0 -camera_index 1 -r 10 -t 5 -y <path>
Resulting video always has this side data no matter in which position I hold the device:
Input #0, android_camera, from '0':
Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc
05-13 18:59:41.746 18715 18715 I Baresip : Side data:
05-13 18:59:41.746 18715 18715 I Baresip : displaymatrix: rotation of -90.00 degrees
I haven't yet figured out how I can set android_camera add_display_matrix() function ctx->sensor_orientation value from a CLI option.
Suggestions on how to test this issue using ffmpeg CLI options would be appreciated.
comment:17 by , 4 years ago
Tried also using front front camera:
ffmpeg -video_size hd720 -f android_camera -camera_index 1 -i anything -r 10 -t 5 -y <path>
No matter if I hold the device portrait or landscape, the result is the same:
05-13 22:49:32.181 27037 27037 I Baresip : [android_camera @ 0x7823e1cc00] Android camera capture session is active.
05-13 22:49:32.181 27037 27037 I Baresip : Input #0, android_camera, from 'anything':
05-13 22:49:32.181 27037 27037 I Baresip : Duration: N/A, start: 645873.465432, bitrate: N/A
05-13 22:49:32.181 27037 27037 I Baresip : Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc
05-13 22:49:32.181 27037 27037 I Baresip : Side data:
05-13 22:49:32.181 27037 27037 I Baresip : displaymatrix: rotation of 90.00 degrees
05-13 22:49:32.181 27037 27037 I Baresip : Stream mapping:
05-13 22:49:32.181 27037 27037 I Baresip : Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
05-13 22:49:32.181 27037 27037 I Baresip : Output #0, mp4, to '/storage/emulated/0/Download/video.mp4':
05-13 22:49:32.181 27037 27037 I Baresip : Metadata:
05-13 22:49:32.181 27037 27037 I Baresip : encoder : Lavf58.42.100
05-13 22:49:32.181 27037 27037 I Baresip : Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 720x1280, q=2-31, 200 kb/s, 10 fps, 10240 tbn, 10 tbc
05-13 22:49:32.181 27037 27037 I Baresip : Metadata:
05-13 22:49:32.181 27037 27037 I Baresip : encoder : Lavc58.78.102 mpeg4
05-13 22:49:32.181 27037 27037 I Baresip : Side data:
05-13 22:49:32.181 27037 27037 I Baresip : cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
05-13 22:49:32.181 27037 27037 I Baresip : displaymatrix: rotation of -0.00 degrees
VLC shows portrait correctly, but landscape is rotated 90 clockwise degrees.
comment:18 by , 4 years ago
Keywords: | android added |
---|
Please provide the command line you tested together with the complete, uncut console output to make this a valid ticket. And please try hard to remove the useless additions to the console output.
comment:19 by , 4 years ago
Command that is executed to record video:
ffmpeg -video_size hd720 -f android_camera -camera_index 1 -i anything -r 10 -t 5 -y <path>
Output when defaultDisplay.rotation of Android device is Surface.ROTATION_0 (= portrait):
05-14 09:13:03.247 4589 4589 I Baresip : [android_camera @ 0x7823e3fc00] Android camera capture session is active. 05-14 09:13:03.247 4589 4589 I Baresip : Input #0, android_camera, from 'anything': 05-14 09:13:03.247 4589 4589 I Baresip : Duration: N/A, start: 683284.573975, bitrate: N/A 05-14 09:13:03.247 4589 4589 I Baresip : Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc 05-14 09:13:03.247 4589 4589 I Baresip : Side data: 05-14 09:13:03.247 4589 4589 I Baresip : displaymatrix: rotation of 90.00 degrees 05-14 09:13:03.247 4589 4589 I Baresip : Stream mapping: 05-14 09:13:03.247 4589 4589 I Baresip : Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native)) 05-14 09:13:03.247 4589 4589 I Baresip : Press [q] to stop, [?] for help 05-14 09:13:03.247 4589 4589 I Baresip : [graph 0 input from stream 0:0 @ 0x7823c6b140] sws_param option is deprecated and ignored 05-14 09:13:03.247 4589 4589 I Baresip : Output #0, mp4, to '/storage/emulated/0/Download/video.mp4': 05-14 09:13:03.247 4589 4589 I Baresip : Metadata: 05-14 09:13:03.247 4589 4589 I Baresip : encoder : Lavf58.42.100 05-14 09:13:03.247 4589 4589 I Baresip : Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 720x1280, q=2-31, 200 kb/s, 10 fps, 10240 tbn, 10 tbc 05-14 09:13:03.247 4589 4589 I Baresip : Metadata: 05-14 09:13:03.247 4589 4589 I Baresip : encoder : Lavc58.78.102 mpeg4 05-14 09:13:03.247 4589 4589 I Baresip : Side data: 05-14 09:13:03.247 4589 4589 I Baresip : cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A 05-14 09:13:03.247 4589 4589 I Baresip : displaymatrix: rotation of -0.00 degrees
Output when defaultDisplay.rotation of Android device is Surface.ROTATION_90 (= landscape):
05-14 09:15:09.294 4589 4589 I Baresip : [android_camera @ 0x7823e40e00] Android camera capture session is active. 05-14 09:15:09.294 4589 4589 I Baresip : Input #0, android_camera, from 'anything': 05-14 09:15:09.294 4589 4589 I Baresip : Duration: N/A, start: 683410.615418, bitrate: N/A 05-14 09:15:09.294 4589 4589 I Baresip : Stream #0:0: Video: rawvideo (NV21 / 0x3132564E), nv21, 1280x720, 30 fps, 30 tbr, 1000000000.00 tbn, 1000000000.00 tbc 05-14 09:15:09.294 4589 4589 I Baresip : Side data: 05-14 09:15:09.294 4589 4589 I Baresip : displaymatrix: rotation of 90.00 degrees 05-14 09:15:09.294 4589 4589 I Baresip : Stream mapping: 05-14 09:15:09.294 4589 4589 I Baresip : Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native)) 05-14 09:15:09.294 4589 4589 I Baresip : Press [q] to stop, [?] for help 05-14 09:15:09.294 4589 4589 I Baresip : [graph 0 input from stream 0:0 @ 0x7813e430c0] sws_param option is deprecated and ignored 05-14 09:15:09.294 4589 4589 I Baresip : Output #0, mp4, to '/storage/emulated/0/Download/video.mp4': 05-14 09:15:09.294 4589 4589 I Baresip : Metadata: 05-14 09:15:09.294 4589 4589 I Baresip : encoder : Lavf58.42.100 05-14 09:15:09.294 4589 4589 I Baresip : Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 720x1280, q=2-31, 200 kb/s, 10 fps, 10240 tbn, 10 tbc 05-14 09:15:09.294 4589 4589 I Baresip : Metadata: 05-14 09:15:09.294 4589 4589 I Baresip : encoder : Lavc58.78.102 mpeg4 05-14 09:15:09.294 4589 4589 I Baresip : Side data: 05-14 09:15:09.294 4589 4589 I Baresip : cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A 05-14 09:15:09.294 4589 4589 I Baresip : displaymatrix: rotation of -0.00 degrees
The issue is that in both cases input stream displaymatrix side data has the same rotation of 90.0 degrees.
Is the issue you see reproducible with current FFmpeg git head, the only version supported here?
Is the issue reproducible with
ffmpeg
, the command line interface? If yes, please provide the command line you tested together with the complete, uncut console output to make this a valid ticket. Please post code that can be compiled and tested if the issue is not reproducible withffmpeg
.