FFmpeg should do framerate convert before scaling if (and only if) that is expected to be faster

[kaifu]

Summary of the bug:
How to reproduce:

%time ffmpeg -i test.mp4 -vcodec mjpeg -s 960x640 -r 0.2 -y out.mov

we may compare the real time with result of running scripts that transcoding to other video format, such as msmpeg4, x264,...

ffmpeg version 2.0.1
built on linux/MINGW32

Details see below:

$ time ./ffmpeg -i test.mp4 -vcodec mjpeg -s 960x640 -r 0.2 -an -y out.mov
ffmpeg version 2.0 Copyright (c) 2000-2013 the FFmpeg developers

built on Oct 1 2013 00:27:45 with gcc 4.3.2 (4.3.2-tdm-1 for MinGW)
configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-nonfree --disable-ffserver --enable-memalign-hack --enable-pthreads --disable-txtpages
libavutil 52. 46.100 / 52. 46.100
libavcodec 55. 33.101 / 55. 33.101
libavformat 55. 18.104 / 55. 18.104
libavdevice 55. 3.100 / 55. 3.100
libavfilter 3. 88.100 / 3. 88.100
libswscale 2. 5.100 / 2. 5.100
libswresample 0. 17.103 / 0. 17.103
libpostproc 52. 3.100 / 52. 3.100

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':

Metadata:

major_brand : mp42
minor_version : 0
compatible_brands: isommp42
creation_time : 2013-04-09 11:10:32
copyright :
copyright-eng :

Duration: 00:03:53.87, start: 0.000000, bitrate: 897 kb/s

Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x960, 846 kb/s, 24 fps, 24 tbr, 24k tbn, 48 tbc (default)
Metadata:

creation_time : 2013-04-09 11:10:30
handler_name : GPAC ISO Video Handler

Stream #0:1(und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 47 kb/s (default)
Metadata:

creation_time : 2013-04-09 11:10:32
handler_name : GPAC ISO Audio Handler

[swscaler @ 03783020] deprecated pixel format used, make sure you did set range correctly
Output #0, mov, to 'out.mov':

Metadata:

major_brand : mp42
minor_version : 0
compatible_brands: isommp42
copyright-eng :
copyright :
encoder : Lavf55.18.104
Stream #0:0(und): Video: mjpeg (jpeg / 0x6765706A), yuvj420p, 960x640, q=2-31, 200 kb/s, 16384 tbn, 0.20 tbc (default)
Metadata:

creation_time : 2013-04-09 11:10:30
handler_name : GPAC ISO Video Handler

Stream mapping:

Stream #0:0 -> #0:0 (h264 -> mjpeg)

Press [q] to stop, ? for help
frame= 50 fps=0.7 q=0.0 Lsize= 4130kB time=00:04:10.00 bitrate= 135.3kbits/s dup=0 drop=5561
video:4129kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.023129%

real 1m9.687s
user 0m0.031s
sys 0m0.015s

$ time ./ffmpeg -i test.mp4 -vcodec msmpeg4 -s 960x640 -r 0.2 -an -y out.avi
ffmpeg version 2.0 Copyright (c) 2000-2013 the FFmpeg developers

built on Oct 1 2013 00:27:45 with gcc 4.3.2 (4.3.2-tdm-1 for MinGW)
configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-nonfree --disable-ffserver --enable-memalign-hack --enable-pthreads --disable-txtpages
libavutil 52. 46.100 / 52. 46.100
libavcodec 55. 33.101 / 55. 33.101
libavformat 55. 18.104 / 55. 18.104
libavdevice 55. 3.100 / 55. 3.100
libavfilter 3. 88.100 / 3. 88.100
libswscale 2. 5.100 / 2. 5.100
libswresample 0. 17.103 / 0. 17.103
libpostproc 52. 3.100 / 52. 3.100

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':

Metadata:

major_brand : mp42
minor_version : 0
compatible_brands: isommp42
creation_time : 2013-04-09 11:10:32
copyright :
copyright-eng :

Duration: 00:03:53.87, start: 0.000000, bitrate: 897 kb/s

Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x960, 846 kb/s, 24 fps, 24 tbr, 24k tbn, 48 tbc (default)
Metadata:

creation_time : 2013-04-09 11:10:30
handler_name : GPAC ISO Video Handler

Stream #0:1(und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 47 kb/s (default)
Metadata:

creation_time : 2013-04-09 11:10:32
handler_name : GPAC ISO Audio Handler

Output #0, avi, to 'out.avi':

Metadata:

major_brand : mp42
minor_version : 0
compatible_brands: isommp42
copyright-eng :
ICOP :
ISFT : Lavf55.18.104
Stream #0:0(und): Video: msmpeg4v3 (msmpeg4) (MP43 / 0x3334504D), yuv420p, 960x640, q=2-31, 200 kb/s, 0.20 tbn, 0.20 tbc (default)
Metadata:

creation_time : 2013-04-09 11:10:30
handler_name : GPAC ISO Video Handler

Stream mapping:

Stream #0:0 -> #0:0 (h264 -> msmpeg4)

Press [q] to stop, ? for help
frame= 49 fps=1.7 q=1.6 Lsize= 3972kB time=00:04:05.00 bitrate= 132.8kbits/s dup=0 drop=5562
video:3965kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.169484%

real 0m29.644s
user 0m0.015s
sys 0m0.031s

I tested the benchmark of ffmpeg, and I can not find any issue in codec's performance, but I profiled the transcoder and find the performance issue is in the below call-chain, it is avfilter issue.
main()->transcode()->transcode_step()->process_input()->output_packet()->decode_video()->av_buffersrc_add_frame_flags()->av_buffersrc_add_frame_internal()->ctx->output_pads[0].request_frame(ctx->outputs[0])

[Elon Musk]

Why are you using different containers?
If you want to do real benchmark use '-f null -'

[kaifu]

test sample for this issue

[kaifu]

Replying to richardpl:

Why are you using different containers?
If you want to do real benchmark use '-f null -'

I tested it again following your advice with attached stream, but this issue reproduced yet.
$ time ./ffmpeg -i test.mp4 -vcodec mjpeg -r 0.2 -f null -
ffmpeg version 2.0 Copyright (c) 2000-2013 the FFmpeg developers

built on Oct 1 2013 00:27:45 with gcc 4.3.2 (4.3.2-tdm-1 for MinGW)
configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-nonfree --disable-ffserver --enable-memalign-hack --enable-pthreads -- disable-txtpages
libavutil 52. 46.100 / 52. 46.100
libavcodec 55. 33.101 / 55. 33.101
libavformat 55. 18.104 / 55. 18.104
libavdevice 55. 3.100 / 55. 3.100
libavfilter 3. 88.100 / 3. 88.100
libswscale 2. 5.100 / 2. 5.100
libswresample 0. 17.103 / 0. 17.103
libpostproc 52. 3.100 / 52. 3.100

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':

Metadata:

major_brand : mp42
minor_version : 0
compatible_brands: isommp42
creation_time : 2013-04-09 11:10:32
copyright :
copyright-eng :

Duration: 00:03:53.87, start: 0.000000, bitrate: 89 kb/s

Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709) , 1280x960, 846 kb/s, 24 fps, 24 tbr, 24k tbn, 48 tbc (default)
Metadata:

creation_time : 2013-04-09 11:10:30
handler_name : GPAC ISO Video Handler

Stream #0:1(und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 47 kb/s (default)
Metadata:

creation_time : 2013-04-09 11:10:32
handler_name : GPAC ISO Audio Handler

[swscaler @ 02f309e0] deprecated pixel format used, make sure you did set range correctly
Output #0, null, to 'pipe:':

Metadata:

major_brand : mp42
minor_version : 0
compatible_brands: isommp42
copyright-eng :
copyright :
encoder : Lavf55.18.104
Stream #0:0(und): Video: mjpeg, yuvj420p, 1280x960, q=2-31, 200 kb/s, 90k tb n, 0.20 tbc (default)
Metadata:

creation_time : 2013-04-09 11:10:30
handler_name : GPAC ISO Video Handler

Stream #0:1(und): Audio: pcm_s16le, 44100 Hz, stereo, s16, 1411 kb/s (defaul t)
Metadata:

creation_time : 2013-04-09 11:10:32
handler_name : GPAC ISO Audio Handler

Stream mapping:

Stream #0:0 -> #0:0 (h264 -> mjpeg)
Stream #0:1 -> #0:1 (aac -> pcm_s16le)

Press [q] to stop, ? for help
frame= 3 fps=0.0 q=0.0 size=N/A time=00:00:05.00 bitrate=N/A dup=0 drop=25 frame= 4 fps=3.9 q=0.0 size=N/A time=00:00:10.00 bitrate=N/A dup=0 drop=57 frame= 4 fps=2.6 q=0.0 size=N/A time=00:00:10.00 bitrate=N/A dup=0 drop=90 frame= 4 fps=2.0 q=0.0 size=N/A time=00:00:10.00 bitrate=N/A dup=0 drop=123 frame= 4 fps=1.6 q=0.0 size=N/A time=00:00:10.00 bitrate=N/A dup=0 drop=158 frame= 5 fps=1.6 q=0.0 size=N/A time=00:00:15.00 bitrate=N/A dup=0 drop=193 frame= 5 fps=1.4 q=0.0 size=N/A time=00:00:15.00 bitrate=N/A dup=0 drop=226 frame= 5 fps=1.2 q=0.0 size=N/A time=00:00:15.00 bitrate=N/A dup=0 drop=260 frame= 5 fps=1.1 q=0.0 size=N/A time=00:00:15.00 bitrate=N/A dup=0 drop=293 frame= 6 fps=1.2 q=0.0 size=N/A time=00:00:20.00 bitrate=N/A dup=0 drop=325 frame= 6 fps=1.1 q=0.0 size=N/A time=00:00:20.00 bitrate=N/A dup=0 drop=358 frame= 6 fps=1.0 q=0.0 size=N/A time=00:00:20.00 bitrate=N/A dup=0 drop=393 frame= 7 fps=1.1 q=0.0 size=N/A time=00:00:25.00 bitrate=N/A dup=0 drop=425 frame= 7 fps=1.0 q=0.0 size=N/A time=00:00:25.00 bitrate=N/A dup=0 drop=461 [h264 @ 034243c0] AVC: nal size 19894

Last message repeated 1 times

[h264 @ 034243c0] no frame!
frame= 7 fps=1.0 q=0.0 Lsize=N/A time=00:00:35.00 bitrate=N/A dup=0 drop=474
video:506kB audio:3368kB subtitle:0 global headers:0kB muxing overhead -100.0005 55%

real 0m7.620s
user 0m0.000s
sys 0m0.030s

$ time ./ffmpeg -i test.mp4 -vcodec msmpeg4 -r 0.2 -f null -
ffmpeg version 2.0 Copyright (c) 2000-2013 the FFmpeg developers

built on Oct 1 2013 00:27:45 with gcc 4.3.2 (4.3.2-tdm-1 for MinGW)
configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-nonfree --disable-ffserver --enable-memalign-hack --enable-pthreads -- disable-txtpages
libavutil 52. 46.100 / 52. 46.100
libavcodec 55. 33.101 / 55. 33.101
libavformat 55. 18.104 / 55. 18.104
libavdevice 55. 3.100 / 55. 3.100
libavfilter 3. 88.100 / 3. 88.100
libswscale 2. 5.100 / 2. 5.100
libswresample 0. 17.103 / 0. 17.103
libpostproc 52. 3.100 / 52. 3.100

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':

Metadata:

major_brand : mp42
minor_version : 0
compatible_brands: isommp42
creation_time : 2013-04-09 11:10:32
copyright :
copyright-eng :

Duration: 00:03:53.87, start: 0.000000, bitrate: 89 kb/s

Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709) , 1280x960, 846 kb/s, 24 fps, 24 tbr, 24k tbn, 48 tbc (default)
Metadata:

creation_time : 2013-04-09 11:10:30
handler_name : GPAC ISO Video Handler

Stream #0:1(und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 47 kb/s (default)
Metadata:

creation_time : 2013-04-09 11:10:32
handler_name : GPAC ISO Audio Handler

Output #0, null, to 'pipe:':

Metadata:

major_brand : mp42
minor_version : 0
compatible_brands: isommp42
copyright-eng :
copyright :
encoder : Lavf55.18.104
Stream #0:0(und): Video: msmpeg4v3 (msmpeg4), yuv420p, 1280x960, q=2-31, 200 kb/s, 90k tbn, 0.20 tbc (default)
Metadata:

creation_time : 2013-04-09 11:10:30
handler_name : GPAC ISO Video Handler

Stream #0:1(und): Audio: pcm_s16le, 44100 Hz, stereo, s16, 1411 kb/s (defaul t)
Metadata:

creation_time : 2013-04-09 11:10:32
handler_name : GPAC ISO Audio Handler

Stream mapping:

Stream #0:0 -> #0:0 (h264 -> msmpeg4)
Stream #0:1 -> #0:1 (aac -> pcm_s16le)

Press [q] to stop, ? for help
frame= 4 fps=0.0 q=2.0 size=N/A time=00:00:20.00 bitrate=N/A dup=0 drop=149 frame= 5 fps=5.0 q=2.0 size=N/A time=00:00:25.00 bitrate=N/A dup=0 drop=288 frame= 7 fps=4.6 q=2.0 size=N/A time=00:00:35.00 bitrate=N/A dup=0 drop=420 [h264 @ 037540e0] AVC: nal size 19894

Last message repeated 1 times

[h264 @ 037540e0] no frame!
frame= 7 fps=4.2 q=2.0 Lsize=N/A time=00:00:35.00 bitrate=N/A dup=0 drop=474
video:458kB audio:3368kB subtitle:0 global headers:0kB muxing overhead -100.0005 61%

real 0m1.985s
user 0m0.000s
sys 0m0.016s

[Carl Eugen Hoyos]

Please test (and report) all issues with current git head.

Could you elaborate a little on the problem you have? Iiuc, you are reporting that the msmpeg4 encoder is faster than the mjpeg encoder but I am not sure that this is a useful explanation / that they can be compared like this.
An enhancement request for a faster mjpeg encoder is probably valid but I wonder if this is everything this issue is about...

Note that you cannot avoid the scaling step (the scaler is also responsible for colour-space conversion), your h264 input sample has a different colour space than mjpeg (while msmpeg4 has the same as your h264 sample). So if this is your problem, I don't think there is anything that can be fixed about it.

[Michael Niedermayer]

jpeg does not support yuv420, it uses yuvj420 which needs colorspace convertion that msmpeg4 does not need.
and scaling is done before framerate handling currently.
You can force it to be done the other way around with -vf fps=0.2,scale

[Michael Niedermayer]

this isnt really mjpeg related, except by chance

[kaifu]

Replying to michael:

jpeg does not support yuv420, it uses yuvj420 which needs colorspace convertion that msmpeg4 does not need.
and scaling is done before framerate handling currently.
You can force it to be done the other way around with -vf fps=0.2,scale

Thank you! michael.
I try testing it with your advice. This issue is resolved when frame rate set be low. The colorspace conversion or scaling will be performance issue if frame rate is unchange. Is it right?