pngdec output frame that has double image

[akoryagin]

How to reproduce:

% ./ffmpeg-4.4 -init_hw_device qsv=hw -filter_hw_device hw -c:v h264_qsv -i sailboat.ts -filter_complex "movie=alogo.png:loop=0,setpts=N/FRAME_RATE/TB[mv];[mv]hwupload=extra_hw_frames=10,format=qsv[watermark]; [0:v][watermark]overlay_qsv=x=(W-w):y=(H-h)/2:shortest=1,scale_qsv=w=1280:h=720" -c:v h264_qsv -b:v 3M -maxrate 3M -c:a copy -y -movflags +faststart test-ffmpeg44.mp4
ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --prefix=/root/ffmpeg44/build --ld=g++ --pkg-config-flags=--static --disable-shared --enable-gpl --enable-nonfree --enable-libass --enable-libfreetype --enable-libx264 --enable-libx265 --enable-libfdk-aac --enable-libmp3lame --enable-libopus --enable-libvpx --enable-libtheora --enable-libvorbis --enable-version3 --enable-libopenjpeg --enable-zlib --enable-libdav1d --enable-libvmaf --disable-decoder=jpeg2000 --extra-cflags=-I/root/ffmpeg44/build/include --extra-ldflags=-L/root/ffmpeg44/build/lib --extra-ldflags=-pthread --enable-libmfx --enable-vaapi --enable-libdrm --extra-cflags=-I/root/ffmpeg44/build/opt/intel/mediasdk/include
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
[h264 @ 0x5558a9c86e40] non-existing SPS 0 referenced in buffering period
    Last message repeated 1 times
Input #0, mpegts, from 'sailboat.ts':
  Duration: 00:00:45.74, start: 1.433367, bitrate: 7143 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
  Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
  Stream #0:1[0x101](eng): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 131 kb/s
Stream mapping:
  Stream #0:0 (h264_qsv) -> overlay_qsv:main
  scale_qsv -> Stream #0:0 (h264_qsv)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[Parsed_overlay_qsv_4 @ 0x5558a9daedc0] Mixing hardware and software pixel formats is not supported.
[Parsed_overlay_qsv_4 @ 0x5558a9daedc0] Failed to configure output pad on Parsed_overlay_qsv_4
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

I tried version 4.1 and it worked without problems:

% ffmpeg-4.1 -init_hw_device qsv=hw -hwaccel qsv -c:v h264_qsv -i sailboat.ts \
-filter_complex "movie=alogo.png:loop=0,setpts=N/FRAME_RATE/TB[mv];[mv]hwupload=extra_hw_frames=10,format=qsv[watermark]; \
[0:v][watermark]overlay_qsv=x=(W-w):y=(H-h)/2:shortest=1,scale_qsv=w=1280:h=720" -c:v h264_qsv -b:v 3M -maxrate 3M -c:a copy -y -movflags +faststart test-ffmpeg41.mp4

[Carl Eugen Hoyos]

Please confirm that the issue is reproducible with current FFmpeg git head and please find the change introducing the regression.

[wenbin,chen]

Hi akoryagin:
I don't know why 4.1 works bug 4.4 failed, but according to your command line and error log, I think you should change your command line like this:

ffmpeg-4.4 -init_hw_device qsv=hw -filter_hw_device hw -c:v h264_qsv -i sailboat.ts -filter_complex "movie=alogo.png:loop=0,setpts=N/FRAME_RATE/TB[mv];[mv]hwupload=extra_hw_frames=10,format=qsv[watermark]; [0:v]format=nv12,hwupload=extra_hw_frames=10[temp1];[temp1][watermark]overlay_qsv=x=(W-w):y=(H-h)/2:shortest=1,scale_qsv=w=1280:h=720" -c:v h264_qsv -b:v 3M -maxrate 3M -c:a copy -y -movflags +faststart test-ffmpeg44.mp4

[akoryagin]

Thank you wenbinCHEN, technically it works but the animation leaves artefacts of the previous frames. Please see output of 4.1 and 4.4 for comparison:
​https://drive.google.com/drive/folders/1JAGfhsWhqwBpnVB0OL-MHFD4qEnj3CJM?usp=sharing

[wenbin,chen]

Sorry for the late response.
I can reproduce this issue, and I found that it is png decoder's issue. If you run command

ffplay -i alogo.png

you can see the same problem. I am working on it but I am not sure if I can fix it as I am not familiar with this part.

[Carl Eugen Hoyos]

Replying to Carl Eugen Hoyos:

Please confirm that the issue is reproducible with current FFmpeg git head and please find the change introducing the regression.

Please note that this ticket is still missing all necessary information.

[Marton Balint]

The corruption changes slightly because of recent developments in the png decoder, but it looks to me the first corruption was caused by 5663301560d77486c7f7c03c1aa5f542fab23c24.

And yes, it is reproducible with git master.

[Balling]

Replying to Marton Balint:

The corruption changes slightly because of recent developments in the png decoder, but it looks to me the first corruption was caused by 5663301560d77486c7f7c03c1aa5f542fab23c24.

And yes, it is reproducible with git master.

There was all kind of nagative commentory on that patch. I suppose you cannot just revert or we will lose gAMA and stuff in sidedata?

​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210216202416.12116-5-anton@khirnov.net/

[mkver]

It was actually 5663301560d77486c7f7c03c1aa5f542fab23c24 that broke exporting side data; it was restored later in 8d74baccff59192d395735036cd40a131a140391. Reverting 5663301560d77486c7f7c03c1aa5f542fab23c24 and the fixup commits on top of it will instead reopen tickets #9017 and #8972.

[Balling]

Replying to mkver:

#8972.

But #8972 was introduced before the targeted commit that you aimed the fix for. ​https://trac.ffmpeg.org/ticket/8972#comment:6

[Balling]

Will fix this ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210508111303.27757-1-anton@khirnov.net/

[Balling]

And... fixed in aa726eaed9848bcf6f615d2d9e6e5dd6174ad570.

Hope that is all and no other problems with pngdec, please test yourself too.

Does this allow for extra release? 🤔 haha

[Pljushevij]

Hello
My FFMPEG version is: ffmpeg version N-103023-gc866a099b2 Copyright (c) 2000-2021 the FFmpeg developers built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)

Code from this comment has bug - artefact like line bottom on the video:
​https://trac.ffmpeg.org/ticket/9184#comment:2

I was trying h264_qsv (with movie clip H.264) and hevc_qsv (with movie clip HEVC) - problem is the same. Can you please fix it?

[Pljushevij]

Hello
My FFMPEG version is: ffmpeg version N-103023-gc866a099b2 Copyright (c) 2000-2021 the FFmpeg developers built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)

Code from this comment has bug - artefact like line bottom on the video:
​https://trac.ffmpeg.org/ticket/9184#comment:2

I was trying h264_qsv (with movie clip H.264) and hevc_qsv (with movie clip HEVC) - problem is the same. Can you please fix it?

[Pljushevij]

I was founding where is the bug - problem with scale_qsv.
Without this parameter all work perfect, but I need this scale_qsv...
It's filter bug?

[Pljushevij]

I was replacing scale_qsv like this and now all works perfect:

ffmpeg-4.4 -init_hw_device qsv=hw -filter_hw_device hw -c:v h264_qsv -i sailboat.ts -filter_complex "movie=alogo.png:loop=0,setpts=N/FRAME_RATE/TB[mv];[mv]hwupload=extra_hw_frames=10,format=qsv[watermark]; [0:v]format=nv12,hwupload=extra_hw_frames=10,scale_qsv=w=1280:h=720[temp1];[temp1][watermark]overlay_qsv=x=(W-w):y=(H-h)/2:shortest=1" -c:v h264_qsv -b:v 3M -maxrate 3M -c:a copy -y -movflags +faststart test-ffmpeg44.mp4

But may you can help me make this by VAAPI?

[wenbin,chen]

Sorry for missing your last comment, glad that you make cmd work. I will give you a VAAPI cmd.

[wenbin,chen]

ffmpeg -init_hw_device vaapi=va -filter_hw_device va -hwaccel vaapi -hwaccel_output_format vaapi -i 1080P_THE_HOBIT3_1920x1080_2700frames.264 -filter_complex "movie=alogo.png:loop=100,setpts=N/FRAME_RATE/TB[mv];[mv]hwupload=extra_hw_frames=10,format=vaapi[watermark]; [0:v]scale_vaapi=w=1280:h=720[temp1];[temp1][watermark]overlay_vaapi=x=1180:y=355:alpha=1" -c:v h264_vaapi -b:v 3M -maxrate 3M -c:a copy -y -movflags +faststart output.mp4

Hi Pljushevij. Here is the command for vaapi. There is no "shortest" option so you need to set "loop" for movie. And overlay_vaapi seems doesn't process alpha channel, so you need to set "alpha" option for overlay_vaapi. And somehow the command cannot work on "sailboat.ts" but can work on other video. Please use qsv as vaapi only have limited function.

[Pljushevij]

Replying to wenbin,chen:

ffmpeg -init_hw_device vaapi=va -filter_hw_device va -hwaccel vaapi -hwaccel_output_format vaapi -i 1080P_THE_HOBIT3_1920x1080_2700frames.264 -filter_complex "movie=alogo.png:loop=100,setpts=N/FRAME_RATE/TB[mv];[mv]hwupload=extra_hw_frames=10,format=vaapi[watermark]; [0:v]scale_vaapi=w=1280:h=720[temp1];[temp1][watermark]overlay_vaapi=x=1180:y=355:alpha=1" -c:v h264_vaapi -b:v 3M -maxrate 3M -c:a copy -y -movflags +faststart output.mp4

Hi Pljushevij. Here is the command for vaapi. There is no "shortest" option so you need to set "loop" for movie. And overlay_vaapi seems doesn't process alpha channel, so you need to set "alpha" option for overlay_vaapi. And somehow the command cannot work on "sailboat.ts" but can work on other video. Please use qsv as vaapi only have limited function.

Hello. Thank you for you answer, but this command show error: No such filter: 'overlay_vaapi'
P.S. i5-7400+Ubuntu 18.04.5+QSV

[Pljushevij]

VAINFO

[Pljushevij]

ffmpeg -filters

[Pljushevij]

A was trying to apply this patch:

​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20200226055334.15280-1-xinpeng.sun@intel.com/

but got the error:

curl https://patchwork.ffmpeg.org/series/407/mbox/ | git am -3


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 22931  100 22931    0     0  49313      0 --:--:-- --:--:-- --:--:-- 49208
Applying: avfilter: add overlay vaapi filter
Using index info to reconstruct a base tree...
M       configure
M       doc/filters.texi
M       libavfilter/Makefile
M       libavfilter/allfilters.c
Falling back to patching base and 3-way merge...
Auto-merging libavfilter/allfilters.c
CONFLICT (content): Merge conflict in libavfilter/allfilters.c
Auto-merging libavfilter/Makefile
Auto-merging doc/filters.texi
Auto-merging configure
CONFLICT (content): Merge conflict in configure
error: Failed to merge in the changes.
Patch failed at 0001 avfilter: add overlay vaapi filter
Use 'git am --show-current-patch' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

[wenbin,chen]

Hi Pljushevij. Sorry, I forget that there are patches not accepted. You can visit the repo: ​https://github.com/intel-media-ci/cartwheel-ffmpeg and apply all the patches in "patches" directory. That is the latest update for ffmpeg-qsv and ffmpeg-vaapi.

[Pljushevij]

Replying to wenbin,chen:

Hi Pljushevij. Sorry, I forget that there are patches not accepted. You can visit the repo: ​https://github.com/intel-media-ci/cartwheel-ffmpeg and apply all the patches in "patches" directory. That is the latest update for ffmpeg-qsv and ffmpeg-vaapi.

Thank you. All works fantastic, but...
If I am using concat function, when first video finishing and starting second video ffmpeg showing error:

Impossible to convert between the formats supported by the filter 'Parsed_overlay_vaapi_5' and the filter 'auto_scaler_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0

Both video in HEVC format, but resolution is small different.
When I was using SW overlay (CPU) this problem not exits. Command (SW overlay) is here:

ffmpeg -hwaccel qsv -hwaccel_output_format qsv -qsv_device /dev/dri/renderD128 -re -f concat -safe 0 -i "files.txt" -loop 1 -i "logo.png" -filter_complex "[1]lut=a=val*0.5[a];[0:v]scale=1920:-1:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2[bg];[bg][a]overlay=(main_w-overlay_w-1):(main_h-overlay_h-1)" -map 0:v:0 -map 0:a:0 -c:v hevc_qsv -preset veryfast -b:v 7M -maxrate 7M -c:a eac3 -sn -f hls -hls_list_size 5 -hls_flags delete_segments -y stream.m3u8

But this command more load CPU. Your first code with QSV has the same problem with concat.
There is the link to ffmpegTest.zip ​https://downloader.disk.yandex.ru/disk/7fcadf659ac622e7664f302adf7f562c9f899770ca6afe73d243c18f3b78a056/61005ae4/PDINqw0ZLeM8KTjoBoI3Xma9eGEWjvl-veqdDOpZs1jsuKiaPNHl5ClnY4rTf2sg7P08ZPIst9nKOQWr5UNWcQ%3D%3D?uid=0&filename=ffmpegTest.zip&disposition=attachment&hash=YapiblbgaprvvrU2e14Ozi7naz2IKOS4qTq%2BNz/FYfVbwZUCkEug6gmZ1nU%2BxY8tq/J6bpmRyOJonT3VoXnDag%3D%3D%3A&limit=0&content_type=application%2Fzip&owner_uid=1394258617&fsize=28956872&hid=1a20996605bd1c947f8b527fbdf6db5b&media_type=compressed&tknv=v2. There is files for testing. Two video, logo, and files.txt (~30Mb).
Can you please test it on your side?

[wenbin,chen]

Hi Pljushevij, I cannot download the zip file.
I will try on my side by using two video with different resolution.

[wenbin,chen]

I don't know the detail of your qsv command, but according to your description you can try to add "-autoscale 0" after filter string.

[Pljushevij]

Replying to wenbin,chen:

Hi Pljushevij, I cannot download the zip file.
I will try on my side by using two video with different resolution.

There is the new link for my test video:
​https://drive.google.com/file/d/1QAsAtBzU782KLBKoDe1CLbu-8DXjvMU1/view?usp=drivesdk

[wenbin,chen]

Here is my command and it works on my side:

ffmpeg -init_hw_device qsv=hw:/dev/dri/renderD128 -filter_hw_device hw -re -f concat -safe 0 -i "files.txt" -loop 1 -i "logo.png" -filter_complex "[1]lut=a=val*0.5,hwupload=extra_hw_frames=10,format=qsv[a];[0:v]hwupload=extra_hw_frames=10,scale_qsv=1920:-1[bg];[bg][a]overlay_qsv=x=(W-w-1):y=(H-h-1):shortest=1,hwdownload,format=nv12,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" -map 0:v:0 -map 0:a:0 -c:v hevc_qsv -preset veryfast -b:v 7M -maxrate 7M -c:a eac3 -sn -f hls -hls_list_size 5 -hls_flags delete_segments -y stream.m3u8

I adjust the order of filter to make cmd work. I don't know if it still meet your requirement.

[Pljushevij]

Replying to wenbin,chen:

Here is my command and it works on my side:

ffmpeg -init_hw_device qsv=hw:/dev/dri/renderD128 -filter_hw_device hw -re -f concat -safe 0 -i "files.txt" -loop 1 -i "logo.png" -filter_complex "[1]lut=a=val*0.5,hwupload=extra_hw_frames=10,format=qsv[a];[0:v]hwupload=extra_hw_frames=10,scale_qsv=1920:-1[bg];[bg][a]overlay_qsv=x=(W-w-1):y=(H-h-1):shortest=1,hwdownload,format=nv12,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" -map 0:v:0 -map 0:a:0 -c:v hevc_qsv -preset veryfast -b:v 7M -maxrate 7M -c:a eac3 -sn -f hls -hls_list_size 5 -hls_flags delete_segments -y stream.m3u8

I adjust the order of filter to make cmd work. I don't know if it still meet your requirement.

Ok. But with this command CPU load is very high. You can see it on attach. picture.

[Pljushevij]

High CPU load

[wenbin,chen]

Hi Pljushevij
CPU High because it use cpu to decode. Here is the command using qsv to decode.(I add "-c:v hevc_qsv" option)

ffmpeg -init_hw_device qsv=hw:/dev/dri/renderD128 -filter_hw_device hw -re -f concat -safe 0 -c:v hevc_qsv -i "files.txt" -loop 1 -i "logo.png" -filter_complex "[1]lut=a=val*0.5,hwupload=extra_hw_frames=10,format=qsv[a];[0:v]hwupload=extra_hw_frames=10,scale_qsv=1920:-1[bg];[bg][a]overlay_qsv=x=(W-w-1):y=(H-h-1):shortest=1,hwdownload,format=nv12,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" -map 0:v:0 -map 0:a:0 -c:v hevc_qsv -preset veryfast -b:v 7M -maxrate 7M -c:a eac3 -sn -f hls -hls_list_size 5 -hls_flags delete_segments -y stream.m3u8

There are upload and download parts which still reduces the efficiency. If we remove pad filter we can remove hwdownload filter too. This makes encoder to encode directly in device memory.

As for upload part, we need to submit patch to improve it. overlay_qsv need two input frames to be in the same device context, but ffmpeg-qsv cannot specify device context for each decoder, so I use hwupload to make sure they are in the same device context.

[Pljushevij]

Replying to wenbin,chen:

Hi Pljushevij
CPU High because it use cpu to decode. Here is the command using qsv to decode.(I add "-c:v hevc_qsv" option)

ffmpeg -init_hw_device qsv=hw:/dev/dri/renderD128 -filter_hw_device hw -re -f concat -safe 0 -c:v hevc_qsv -i "files.txt" -loop 1 -i "logo.png" -filter_complex "[1]lut=a=val*0.5,hwupload=extra_hw_frames=10,format=qsv[a];[0:v]hwupload=extra_hw_frames=10,scale_qsv=1920:-1[bg];[bg][a]overlay_qsv=x=(W-w-1):y=(H-h-1):shortest=1,hwdownload,format=nv12,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" -map 0:v:0 -map 0:a:0 -c:v hevc_qsv -preset veryfast -b:v 7M -maxrate 7M -c:a eac3 -sn -f hls -hls_list_size 5 -hls_flags delete_segments -y stream.m3u8

There are upload and download parts which still reduces the efficiency. If we remove pad filter we can remove hwdownload filter too. This makes encoder to encode directly in device memory.

As for upload part, we need to submit patch to improve it. overlay_qsv need two input frames to be in the same device context, but ffmpeg-qsv cannot specify device context for each decoder, so I use hwupload to make sure they are in the same device context.

With this comand I has the errors when starts second video:

[hevc_qsv @ 0x560ee0848340] Error initializing the MFX video decoder: invalid video parameters (-15)
Error while decoding stream #0:0: Invalid argument

[Pljushevij]

Replying to wenbin,chen:

ffmpeg -init_hw_device vaapi=va -filter_hw_device va -hwaccel vaapi -hwaccel_output_format vaapi -i 1080P_THE_HOBIT3_1920x1080_2700frames.264 -filter_complex "movie=alogo.png:loop=100,setpts=N/FRAME_RATE/TB[mv];[mv]hwupload=extra_hw_frames=10,format=vaapi[watermark]; [0:v]scale_vaapi=w=1280:h=720[temp1];[temp1][watermark]overlay_vaapi=x=1180:y=355:alpha=1" -c:v h264_vaapi -b:v 3M -maxrate 3M -c:a copy -y -movflags +faststart output.mp4

Hi Pljushevij. Here is the command for vaapi. There is no "shortest" option so you need to set "loop" for movie. And overlay_vaapi seems doesn't process alpha channel, so you need to set "alpha" option for overlay_vaapi. And somehow the command cannot work on "sailboat.ts" but can work on other video. Please use qsv as vaapi only have limited function.

Also about this command - logo/watermark shows in black square, but with QSV command - normal, without square.

[Carl Eugen Hoyos]

Please open a new ticket if there still is a bug, don't forget to test current FFmpeg git head and to provide the command line you tested together with the complete, uncut console output.