FFmpeg -progress url Isn't working

[BlackDream]

According to documentation:

-progress url (global)

Send program-friendly progress information to url.

Progress information is written approximately every second and at the end of the encoding process. It is made of "key=value" lines. key consists of only alphanumeric characters. The last key of a sequence of progress information is always "progress".

However, ffmpeg does NOT post the data every 1 second, but ONLY one time at the end of the encoding process.

If instead of URL, we specify a local file, it is working without any problem.

[Carl Eugen Hoyos]

Please provide the command line that allows to reproduce the issue together with the complete, uncut console output to make this a valid ticket.

[BlackDream]

root@3kkz05gk:~# ffmpeg -progress "http://127.0.0.1:8000/test.php" -i "18og.mp4" -c copy -f matroska /tmp/test.mkv

ffmpeg version N-78216-g9554420 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.3.0 (Ubuntu 5.3.0-3ubuntu1~14.04) 20151204
  configuration: --prefix=/root/FFmpeg_old/target --extra-cflags='-I/root/FFmpeg_old/target/include -static' --extra-ldflags='-L/root/FFmpeg_old/target/lib -lm -static' --disable-debug --disable-shared --enable-libx265 --enable-static --extra-cflags=--static --disable-ffplay --disable-ffserver --disable-doc --enable-librtmp --enable-gpl --enable-pthreads --enable-postproc --enable-libass --enable-gray --enable-runtime-cpudetect --enable-gnutls --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libx264 --enable-libxvid --enable-bzlib --enable-zlib --enable-version3 --enable-libxavs --extra-libs='-lrtmp -lnettle -lhogweed -lgmp -lssl -lcrypto -lz -lc -ldl'
  libavutil      55. 16.101 / 55. 16.101
  libavcodec     57. 24.100 / 57. 24.100
  libavformat    57. 23.101 / 57. 23.101
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 27.100 /  6. 27.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '18og.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    creation_time   : 1970-01-01 00:00:00
    encoder         : Lavf52.64.2
  Duration: 00:22:41.40, start: 0.000000, bitrate: 4572 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 4416 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 47.95 tbc (default)
    Metadata:
      creation_time   : 1970-01-01 00:00:00
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 151 kb/s (default)
    Metadata:
      creation_time   : 1970-01-01 00:00:00
      handler_name    : SoundHandler
Output #0, matroska, to '/tmp/test.mkv':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.23.101
    Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p, 1920x1080, q=2-31, 4416 kb/s, 23.98 fps, 23.98 tbr, 1k tbn, 24k tbc (default)
    Metadata:
      creation_time   : 1970-01-01 00:00:00
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) ([255][0][0][0] / 0x00FF), 44100 Hz, stereo, 151 kb/s (default)
    Metadata:
      creation_time   : 1970-01-01 00:00:00
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=32641 fps=7037 q=-1.0 Lsize=  759819kB time=00:22:41.36 bitrate=4572.2kbits/s speed= 293x
video:734037kB audio:25124kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.086602%

And in my NGINX Access log you can see only one HTTP Request and it was at the end of the encoding.

root@3kkz05gk:/var/log/nginx# cat access.log
127.0.0.1 - - [04/Mar/2016:17:12:29 -0500] "POST /test.php HTTP/1.1" 404 162 "-" "Lavf/57.23.101"

[Carl Eugen Hoyos]

What happens if you reencode instead of -vcodec copy? Is progress sent if the command takes longer?

[BlackDream]

Exactly the same happened buddy. Only one and at the end of the encoding.

[Carl Eugen Hoyos]

Could you also test with a very old version like 7b521c52?

[BlackDream]

I compiled 7b521c52 & the latest stable where the -progress introduced (v1.0.0) and still same results.. :-(

I tried encoding using -re too (thought it can be timestamp related), same results.

I repeat though that using a local file, it works OK.

[c-14]

The "problem" as such is that ffmpeg uses a single continuous chunked post to post the progress information to the server and the HTTP server sees this as a single request and therefore only processes it once when the request finishes.

The only way to "fix" this (besides not using http) would be to create a new http muxer that creates a new post request every time ffurl_write is called on it (and then force that muxer for use with progress), or close and reopen the AVIOContext when writing progress information (hacky, but works). Neither of those options seems particularly clean though.

[Cigaes]

Another solution would be to have the HTTP server pass the data to the script as it arrives, not just at the end of the request. I have no idea if this an option with existing HTTP server, especially with the PHP interpreter, but there is no reason that it would not be possible.

[balooshy]

close and reopen the AVIOContext when writing progress information (hacky, but works).

Im not C DEV and i have same problem can you send us such hack ?

[Jay Caines-Gooby]

This is actually an issue with the way the web server receiving the -progress URL is configured.

ffmpeg is sending the progress data as it happens, using a chunked POST body.

You can test this with netcat; use the netcat url as the destination for the -progress option:

# start "server"
nc -l 7000 > ffmpeg.http.log &

# render 30 seconds of blank h264 sending progress to the server
ffmpeg -y -progress http://127.0.0.1:7000 -t 30 -f lavfi -i color=c=black:s=640x480 -c:v libx264 -tune stillimage -pix_fmt yuv420p black.mp4

# cat the "server" log
cat ffmpeg.http.log

You'll see the chunked POST headers, followed by the data:

POST / HTTP/1.1
Transfer-Encoding: chunked
User-Agent: Lavf/58.29.100
Accept: */*
Connection: close
Host: 127.0.0.1:7000
Icy-MetaData: 1

c5
frame=346
fps=0.00
stream_0_0_q=28.0
bitrate=   0.0kbits/s
total_size=48
out_time_us=11840078
out_time_ms=11840078
out_time=00:00:11.840078
dup_frames=0
drop_frames=0
speed=23.7x
progress=continue

c7
frame=708
fps=704.69
stream_0_0_q=28.0
bitrate=   0.0kbits/s
total_size=48
out_time_us=26320078
out_time_ms=26320078
out_time=00:00:26.320078
dup_frames=0
drop_frames=0
speed=26.2x
progress=continue

c5
frame=750
fps=680.10
stream_0_0_q=-1.0
bitrate=   7.7kbits/s
total_size=28599
out_time_us=29880078
out_time_ms=29880078
out_time=00:00:29.880078
dup_frames=0
drop_frames=0
speed=27.1x
progress=end

0