Blackdetect filter PTS return limited to six digits

[morgan]

Summary of the bug: blackdetect filter is returning a maximum of six digits for pts values. This doesn't give enough precision for frame-accurate results. You'll notice that all detected blacks have a maximum of six digits returned for the pts time. The early detected blacks have good precision. As the file progresses the precision is reduced and numbers are rounded. I.e. towards the end of the file precision is lowered to two digits to keep the total number of digits to six.

How to reproduce:

% ffmpeg -loglevel info -i input.mp4 -vf blackdetect=d=0.2:pix_th=0.2 -f null -y /dev/null
ffmpeg version 4.2.3

macbook-pro:~ $ ffmpeg -loglevel info -i ABC_R1.mp4 -vf blackdetect=d=0.2:pix_th=0.2 -f null -y /dev/null
ffmpeg version 4.3.2 Copyright (c) 2000-2021 the FFmpeg developers
  built with Apple clang version 12.0.0 (clang-1200.0.32.29)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.3.2-with-options --enable-shared --cc=clang --host-cflags= --host-ldflags= --enable-gpl --enable-libaom --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-libsnappy --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-demuxer=dash --disable-libjack --disable-indev=jack --enable-opencl --enable-videotoolbox --disable-htmlpages --enable-libtesseract
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'ABC_R1.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.45.100
  Duration: 01:06:37.43, start: 0.000000, bitrate: 460 kb/s
    Stream #0:0(eng): Video: h264 (High 4:2:2) (avc1 / 0x31637661), yuv422p10le, 640x480 [SAR 4:3 DAR 16:9], 410 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 47.95 tbc (default)
    Metadata:
      handler_name    : Apple Video Media Handler
      timecode        : 00:57:30:00
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 44 kb/s (default)
    Metadata:
      handler_name    : Apple Sound Media Handler
    Stream #0:2(eng): Data: none (tmcd / 0x64636D74)
    Metadata:
      handler_name    : Apple Video Media Handler
      timecode        : 00:57:30:00
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, null, to '/dev/null':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.45.100
    Stream #0:0(eng): Video: wrapped_avframe, yuv422p, 640x480 [SAR 4:3 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn, 23.98 tbc (default)
    Metadata:
      handler_name    : Apple Video Media Handler
      timecode        : 00:57:30:00
      encoder         : Lavc58.91.100 wrapped_avframe
    Stream #0:1(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s (default)
    Metadata:
      handler_name    : Apple Sound Media Handler
      encoder         : Lavc58.91.100 pcm_s16le
[blackdetect @ 0x7fc080c0f7c0] black_start:0 black_end:20.02 black_duration:20.02
frame= 1426 fps=1425 q=-0.0 size=N/A time=00:01:00.09 bitrate=N/A speed=  60x   frame= 2190 fps=1459 q=-0.0 size=N/A time=00:01:31.94 bitrate=N/A speed=61.2x   frame= 2953 fps=1475 q=-0.0 size=N/A time=00:02:03.77 bitrate=N/A speed=61.8x   [blackdetect @ 0x7fc080c0f7c0] black_start:90.09 black_end:150.275 black_duration:60.1851
frame= 3690 fps=1475 q=-0.0 size=N/A time=00:02:34.51 bitrate=N/A speed=61.8x   [blackdetect @ 0x7fc080c0f7c0] black_start:155.489 black_end:156.114 black_duration:0.625625
[blackdetect @ 0x7fc080c0f7c0] black_start:164.873 black_end:167.417 black_duration:2.54421
frame= 4413 fps=1470 q=-0.0 size=N/A time=00:03:04.68 bitrate=N/A speed=61.5x   frame= 5121 fps=1462 q=-0.0 size=N/A time=00:03:34.20 bitrate=N/A speed=61.2x   [blackdetect @ 0x7fc080c0f7c0] black_start:227.519 black_end:229.396 black_duration:1.87687
frame= 5836 fps=1458 q=-0.0 size=N/A time=00:04:04.01 bitrate=N/A speed=  61x   frame= 6568 fps=1459 q=-0.0 size=N/A time=00:04:34.56 bitrate=N/A speed=  61x   frame= 7305 fps=1460 q=-0.0 size=N/A time=00:05:05.30 bitrate=N/A speed=  61x   frame= 8000 fps=1454 q=-0.0 size=N/A time=00:05:34.27 bitrate=N/A speed=60.7x   frame= 8719 fps=1452 q=-0.0 size=N/A time=00:06:04.26 bitrate=N/A speed=60.7x   frame= 9432 fps=1450 q=-0.0 size=N/A time=00:06:34.00 bitrate=N/A speed=60.6x   frame=10173 fps=1452 q=-0.0 size=N/A time=00:07:04.91 bitrate=N/A speed=60.7x   frame=10899 fps=1452 q=-0.0 size=N/A time=00:07:35.18 bitrate=N/A speed=60.6x   frame=11630 fps=1453 q=-0.0 size=N/A time=00:08:05.67 bitrate=N/A speed=60.7x   frame=12370 fps=1454 q=-0.0 size=N/A time=00:08:36.54 bitrate=N/A speed=60.7x   frame=13117 fps=1456 q=-0.0 size=N/A time=00:09:07.71 bitrate=N/A speed=60.8x   frame=13867 fps=1459 q=-0.0 size=N/A time=00:09:38.98 bitrate=N/A speed=60.9x   frame=14612 fps=1460 q=-0.0 size=N/A time=00:10:10.04 bitrate=N/A speed=  61x   frame=15356 fps=1461 q=-0.0 size=N/A time=00:10:41.08 bitrate=N/A speed=  61x   frame=16076 fps=1460 q=-0.0 size=N/A time=00:11:11.12 bitrate=N/A speed=  61x   [blackdetect @ 0x7fc080c0f7c0] black_start:679.095 black_end:686.853 black_duration:7.75775
frame=16809 fps=1461 q=-0.0 size=N/A time=00:11:41.69 bitrate=N/A speed=  61x   frame=17540 fps=1461 q=-0.0 size=N/A time=00:12:12.18 bitrate=N/A speed=  61x   frame=18278 fps=1461 q=-0.0 size=N/A time=00:12:42.96 bitrate=N/A speed=  61x   frame=18977 fps=1459 q=-0.0 size=N/A time=00:13:12.10 bitrate=N/A speed=60.9x   frame=19705 fps=1459 q=-0.0 size=N/A time=00:13:42.48 bitrate=N/A speed=60.9x   [blackdetect @ 0x7fc080c0f7c0] black_start:835.126 black_end:838.546 black_duration:3.42008
frame=20432 fps=1458 q=-0.0 size=N/A time=00:14:12.80 bitrate=N/A speed=60.9x   frame=21172 fps=1459 q=-0.0 size=N/A time=00:14:43.66 bitrate=N/A speed=60.9x   frame=21855 fps=1456 q=-0.0 size=N/A time=00:15:12.14 bitrate=N/A speed=60.8x   [blackdetect @ 0x7fc080c0f7c0] black_start:925.258 black_end:925.508 black_duration:0.25025
frame=22560 fps=1454 q=-0.0 size=N/A time=00:15:41.54 bitrate=N/A speed=60.7x   frame=23286 fps=1454 q=-0.0 size=N/A time=00:16:11.84 bitrate=N/A speed=60.7x   frame=23992 fps=1453 q=-0.0 size=N/A time=00:16:41.28 bitrate=N/A speed=60.6x   frame=24720 fps=1453 q=-0.0 size=N/A time=00:17:11.63 bitrate=N/A speed=60.6x   frame=25422 fps=1452 q=-0.0 size=N/A time=00:17:40.92 bitrate=N/A speed=60.6x   frame=26143 fps=1451 q=-0.0 size=N/A time=00:18:10.98 bitrate=N/A speed=60.6x   frame=26869 fps=1451 q=-0.0 size=N/A time=00:18:41.28 bitrate=N/A speed=60.6x   frame=27601 fps=1452 q=-0.0 size=N/A time=00:19:11.80 bitrate=N/A speed=60.6x   frame=28320 fps=1451 q=-0.0 size=N/A time=00:19:41.80 bitrate=N/A speed=60.6x   frame=29053 fps=1452 q=-0.0 size=N/A time=00:20:12.37 bitrate=N/A speed=60.6x   frame=29780 fps=1452 q=-0.0 size=N/A time=00:20:42.68 bitrate=N/A speed=60.6x   frame=30516 fps=1452 q=-0.0 size=N/A time=00:21:13.38 bitrate=N/A speed=60.6x   frame=31255 fps=1453 q=-0.0 size=N/A time=00:21:44.21 bitrate=N/A speed=60.6x   [blackdetect @ 0x7fc080c0f7c0] black_start:1301.05 black_end:1303.93 black_duration:2.87787
frame=31981 fps=1453 q=-0.0 size=N/A time=00:22:14.48 bitrate=N/A speed=60.6x   frame=32722 fps=1453 q=-0.0 size=N/A time=00:22:45.39 bitrate=N/A speed=60.6x   frame=33455 fps=1454 q=-0.0 size=N/A time=00:23:15.96 bitrate=N/A speed=60.7x   frame=34165 fps=1453 q=-0.0 size=N/A time=00:23:45.57 bitrate=N/A speed=60.6x   frame=34891 fps=1453 q=-0.0 size=N/A time=00:24:15.87 bitrate=N/A speed=60.6x   frame=35619 fps=1453 q=-0.0 size=N/A time=00:24:46.22 bitrate=N/A speed=60.6x   frame=36336 fps=1452 q=-0.0 size=N/A time=00:25:16.13 bitrate=N/A speed=60.6x   frame=37056 fps=1452 q=-0.0 size=N/A time=00:25:46.15 bitrate=N/A speed=60.6x   frame=37771 fps=1452 q=-0.0 size=N/A time=00:26:15.97 bitrate=N/A speed=60.6x   frame=38507 fps=1452 q=-0.0 size=N/A time=00:26:46.67 bitrate=N/A speed=60.6x   frame=39200 fps=1451 q=-0.0 size=N/A time=00:27:15.58 bitrate=N/A speed=60.5x   frame=39888 fps=1450 q=-0.0 size=N/A time=00:27:44.27 bitrate=N/A speed=60.5x   frame=40551 fps=1447 q=-0.0 size=N/A time=00:28:11.92 bitrate=N/A speed=60.4x   frame=41291 fps=1448 q=-0.0 size=N/A time=00:28:42.79 bitrate=N/A speed=60.4x   frame=41997 fps=1447 q=-0.0 size=N/A time=00:29:12.23 bitrate=N/A speed=60.4x   frame=42710 fps=1447 q=-0.0 size=N/A time=00:29:41.97 bitrate=N/A speed=60.4x   frame=43421 fps=1446 q=-0.0 size=N/A time=00:30:11.62 bitrate=N/A speed=60.3x   [blackdetect @ 0x7fc080c0f7c0] black_start:1815.19 black_end:1817.44 black_duration:2.25225

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[morgan]

Some clean blackdetect results to make it more clear.
[blackdetect @ 0x7ff28e004980] black_start:0 black_end:1.20954 black_duration:1.20954
[blackdetect @ 0x7ff28e004980] black_start:4.5045 black_end:6.21455 black_duration:1.71004
[blackdetect @ 0x7ff28e004980] black_start:123.999 black_end:125.417 black_duration:1.41808
[blackdetect @ 0x7ff28e004980] black_start:185.394 black_end:186.603 black_duration:1.20954
[blackdetect @ 0x7ff28e004980] black_start:2049.92 black_end:2053.01 black_duration:3.08642
[blackdetect @ 0x7ff28e004980] black_start:2583.88 black_end:2585.96 black_duration:2.08542
[blackdetect @ 0x7ff28e004980] black_start:2652.49 black_end:2653.57 black_duration:1.08442

[Carl Eugen Hoyos]

Please test if this issue is reproducible with current FFmpeg git head, the only version supported on this bug tracker.

[morgan]

This is the issue:

snprintf(buf, AV_TS_MAX_STRING_SIZE, "%.6g", av_q2d(*tb) * ts)

in ​https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/timestamp.h

was truncating the PTS return to six significant digits. Should be

snprintf(buf, AV_TS_MAX_STRING_SIZE, "%.6f", av_q2d(*tb) * ts)

Haven't done full regression testing yet, but this affects all pts/ts returns for things like scene detect, black detect, etc.?

[morgan]

This is the issue:

snprintf(buf, AV_TS_MAX_STRING_SIZE, "%.6g", av_q2d(*tb) * ts)

in ​https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/timestamp.h

was truncating the PTS return to six significant digits. Should be

snprintf(buf, AV_TS_MAX_STRING_SIZE, "%.6f", av_q2d(*tb) * ts)

Haven't done full regression testing yet, but this affects all pts/ts returns for things like scene detect, black detect, etc.?

[Balling]

On latest:

[blackdetect @ 00000292367c4040] black_start:22.089 black_end:23.49 black_duration:1.401
[blackdetect @ 00000292367c4040] black_start:46.046 black_end:48.549 black_duration:2.503

Looks like it truncates zeroes.

Also with mp4:

[blackdetect @ 000001fd2252d4c0] black_start:0 black_end:1.08442 black_duration:1.08442

but with mkv:

[blackdetect @ 000001917179b840] black_start:0 black_end:1.001 black_duration:1.001

I can confirm the issue on latest git master.

[blackdetect @ 000001917179b840] black_start:216.258 black_end:216.633 black_duration:0.375
[blackdetect @ 000001917179b840] black_start:305.764 black_end:310.143 black_duration:4.379
[blackdetect @ 000001917179b840] black_start:459.376 black_end:459.668 black_duration:0.292
[blackdetect @ 000001917179b840] black_start:473.014 black_end:474.641 black_duration:1.627
[blackdetect @ 000001917179b840] black_start:664.622 black_end:665.081 black_duration:0.459
[blackdetect @ 000001917179b840] black_start:684.058 black_end:684.684 black_duration:0.626
[blackdetect @ 000001917179b840] black_start:685.059 black_end:686.477 black_duration:1.418
[blackdetect @ 000001917179b840] black_start:703.036 black_end:704.37 black_duration:1.334
[blackdetect @ 000001917179b840] black_start:707.874 black_end:710.335 black_duration:2.461
[blackdetect @ 000001917179b840] black_start:719.052 black_end:722.138 black_duration:3.086
[blackdetect @ 000001917179b840] black_start:740.531 black_end:740.74 black_duration:0.209
[blackdetect @ 000001917179b840] black_start:746.996 black_end:747.455 black_duration:0.459
[blackdetect @ 000001917179b840] black_start:747.789 black_end:752.585 black_duration:4.796
[blackdetect @ 000001917179b840] black_start:866.908 black_end:868.409 black_duration:1.501
[blackdetect @ 000001917179b840] black_start:870.453 black_end:870.703 black_duration:0.25
[blackdetect @ 000001917179b840] black_start:873.247 black_end:874.332 black_duration:1.085
[blackdetect @ 000001917179b840] black_start:911.327 black_end:912.745 black_duration:1.418
[blackdetect @ 000001917179b840] black_start:914.08 black_end:914.705 black_duration:0.625
[blackdetect @ 000001917179b840] black_start:915.206 black_end:915.79 black_duration:0.584
[blackdetect @ 000001917179b840] black_start:915.832 black_end:916.582 black_duration:0.75
[blackdetect @ 000001917179b840] black_start:916.833 black_end:917.667 black_duration:0.834
[blackdetect @ 000001917179b840] black_start:917.834 black_end:920.253 black_duration:2.419
[blackdetect @ 000001917179b840] black_start:923.214 black_end:929.929 black_duration:6.715
[blackdetect @ 000001917179b840] black_start:932.515 black_end:933.432 black_duration:0.917
[blackdetect @ 000001917179b840] black_start:938.646 black_end:939.188 black_duration:0.542
[blackdetect @ 000001917179b840] black_start:944.861 black_end:948.99 black_duration:4.129
[blackdetect @ 000001917179b840] black_start:951.2 black_end:955.58 black_duration:4.38
[blackdetect @ 000001917179b840] black_start:957.415 black_end:960.585 black_duration:3.17
[blackdetect @ 000001917179b840] black_start:963.671 black_end:964.005 black_duration:0.334
[blackdetect @ 000001917179b840] black_start:967.091 black_end:968.551 black_duration:1.46
[blackdetect @ 000001917179b840] black_start:969.427 black_end:971.345 black_duration:1.918
[blackdetect @ 000001917179b840] black_start:974.515 black_end:975.308 black_duration:0.793
[blackdetect @ 000001917179b840] black_start:980.438 black_end:980.855 black_duration:0.417
[blackdetect @ 000001917179b840] black_start:983.149 black_end:984.233 black_duration:1.084
[blackdetect @ 000001917179b840] black_start:988.696 black_end:991.949 black_duration:3.253
[blackdetect @ 000001917179b840] black_start:993.534 black_end:995.036 black_duration:1.502
[blackdetect @ 000001917179b840] black_start:995.161 black_end:1000.88 black_duration:5.714
[blackdetect @ 000001917179b840] black_start:1002.34 black_end:1003.46 black_duration:1.126
[blackdetect @ 000001917179b840] black_start:1004.84 black_end:1006.92 black_duration:2.086
[blackdetect @ 000001917179b840] black_start:1008.97 black_end:1009.63 black_duration:0.668
[blackdetect @ 000001917179b840] black_start:1012.14 black_end:1013.1 black_duration:0.959
[blackdetect @ 000001917179b840] black_start:1025.9 black_end:1027.07 black_duration:1.168
[blackdetect @ 000001917179b840] black_start:1027.57 black_end:1027.9 black_duration:0.334
***
[blackdetect @ 000001917179b840] black_start:1457 black_end:1458.25 black_duration:1.251

[Balling]

This can fix it, I think.

​https://patchwork.ffmpeg.org/project/ffmpeg/patch/1149006136.243626.1590899918044@mail.yahoo.com/

[morgan]

Thanks Balling!

Yesterday I applied the change I mentioned above to ​​https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/timestamp.h

The returns from detectblack and "select='gt(scene,0.1)',showinfo" (using these in my current project) are now frame accurate due to the retained six digits of precision throughout the program.

The truncation of precision digits to 3, 2, and eventually 1 or zero as timestamps increment in the current code does not return frame accurate results.

[Balling]

Can somebody fix this? It is a very bad issue.

[Gyan]

It would be better to just print the raw ts and use that, rather than make a ffmpeg-wide change to the formatting.

[Balling]

Replying to Gyan:

It would be better to just print the raw ts and use that, rather than make a ffmpeg-wide change to the formatting.

This is not a change in formatting, there IS a bug in the formatting and not just in this one filter. What kind of formatting is that after decimal amount of digits depends on before the decimal, so that max digits are 6, but the zeroes after decimal can be ommitted?? Is that some kind of iputils (ping) formatting? But it makes no sense, you cannot do this because you need to get the PRECISE frame.