Opened 3 months ago

Last modified 3 months ago

#6212 new defect

Memory leak on Ubuntu 16.04

Reported by: alqemyst Owned by:
Priority: normal Component: undetermined
Version: git-master Keywords: leak
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

I'm trying to run ffmpeg on Ubuntu 16.04, libc6-dev 2.23-0ubuntu5 amd64.

There is a memory leak that is according to my tests related to pthreads. If I fully transcode video and audio process can go up to 1GB of memory usage in an hour.

Tested with following command:
ffmpeg -loglevel error -i "udp://239.250.15.1:5004?fifo_size=1000000&overrun_nonfatal=1" -ignore_unknown -sn -c:v copy -c:a copy -f null -

Valgrind with --leak-check=full --show-leak-kinds=all doesn't show too much in my opinion.

When ffmpeg is compiled with --disable-pthreads memory is stable for hours.

Change History (12)

comment:1 Changed 3 months ago by cehoyos

  • Keywords leak added
  • Priority changed from critical to normal

Complete, uncut console output is required for all bug reports, valgrind console output is required when reporting leaks.
Note that if valgrind does not report a leak, there is no memory leak but maybe another issue.

comment:2 Changed 3 months ago by alqemyst

Sure. This is ffmpeg version:

./ffmpeg -version
ffmpeg version N-83708-g4bed066 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609
configuration: --prefix=/opt/siqnal --enable-gpl --enable-nonfree --enable-version3 --enable-libx264 --enable-libfdk_aac --enable-libmp3lame --enable-openssl --enable-libfreetype --enable-fontconfig --enable-vaapi --enable-nvenc --enable-cuda --enable-cuvid --enable-libnpp --disable-ffplay --disable-doc --disable-ffserver
libavutil      55. 47.100 / 55. 47.100
libavcodec     57. 81.101 / 57. 81.101
libavformat    57. 66.102 / 57. 66.102
libavdevice    57.  3.100 / 57.  3.100
libavfilter     6. 74.100 /  6. 74.100
libswscale      4.  3.101 /  4.  3.101
libswresample   2.  4.100 /  2.  4.100
libpostproc    54.  2.100 / 54.  2.100

NVENC command:

valgrind ./ffmpeg_g -y -loglevel error -hwaccel cuvid -c:v mpeg2_cuvid -deint 2 -i "udp://239.250.14.12:5004?fifo_size=1000000&overrun_nonfatal=1" -ignore_unknown -sn -an -c:v nvenc_h264 -profile:v high -b:v 2000k -maxrate:v 1M -bufsize:v 500k -r 25 -g 50 -aspect 16:9 -vf scale_npp=640:360 -f hls -hls_time 6 -hls_list_size 6 -use_localtime 1 -hls_segment_filename "hls/0-%s.ts" "hls/0.m3u8"
==21588== Memcheck, a memory error detector
==21588== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==21588== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==21588== Command: ./ffmpeg_g -y -loglevel error -hwaccel cuvid -c:v mpeg2_cuvid -deint 2 -i udp://239.250.14.12:5004?fifo_size=1000000&overrun_nonfatal=1 -ignore_unknown -sn -an -c:v nvenc_h264 -profile:v high -b:v 2000k -maxrate:v 1M -bufsize:v 500k -r 25 -g 50 -aspect 16:9 -vf scale_npp=640:360 -f hls -hls_time 6 -hls_list_size 6 -use_localtime 1 -hls_segment_filename hls/0-%s.ts hls/0.m3u8
==21588== 
[mpeg2video @ 0xc4f7040] Invalid frame dimensions 0x0.
==21588== Warning: noted but unhandled ioctl 0x30000001 with no size/direction hints.
==21588==    This could cause spurious value errors to appear.
==21588==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==21588== Warning: noted but unhandled ioctl 0x27 with no size/direction hints.
==21588==    This could cause spurious value errors to appear.
==21588==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==21588== Warning: noted but unhandled ioctl 0x7ff with no size/direction hints.
==21588==    This could cause spurious value errors to appear.
==21588==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==21588== Warning: noted but unhandled ioctl 0x25 with no size/direction hints.
==21588==    This could cause spurious value errors to appear.
==21588==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==21588== Warning: noted but unhandled ioctl 0x17 with no size/direction hints.
==21588==    This could cause spurious value errors to appear.
==21588==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==21588== Warning: set address range perms: large range [0x200000000, 0x700000000) (noaccess)
==21588== Warning: set address range perms: large range [0x900000000, 0xc00000000) (noaccess)
==21588== Warning: noted but unhandled ioctl 0x19 with no size/direction hints.
==21588==    This could cause spurious value errors to appear.
==21588==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==21588== Warning: noted but unhandled ioctl 0x21 with no size/direction hints.
==21588==    This could cause spurious value errors to appear.
==21588==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==21588== Warning: noted but unhandled ioctl 0x1b with no size/direction hints.
==21588==    This could cause spurious value errors to appear.
==21588==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==21588== Conditional jump or move depends on uninitialised value(s)
==21588==    at 0x1C7E078F: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7F44BC: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7F26BE: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E14A1: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E91D6: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E6ECA: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E4BD6: cuvidCreateDecoder (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x4235DC: cuvid_test_dummy_decoder (cuvid.c:670)
==21588==    by 0x4235DC: cuvid_decode_init (cuvid.c:855)
==21588==    by 0xAF71A9: avcodec_open2 (utils.c:1609)
==21588==    by 0x49CA47: init_input_stream (ffmpeg.c:2758)
==21588==    by 0x49CA47: transcode_init (ffmpeg.c:3478)
==21588==    by 0x4A2C28: transcode (ffmpeg.c:4364)
==21588==    by 0x483823: main (ffmpeg.c:4598)
==21588== 
==21588== Conditional jump or move depends on uninitialised value(s)
==21588==    at 0x1C7E079F: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7F44BC: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7F26BE: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E14A1: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E91D6: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E6ECA: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E4BD6: cuvidCreateDecoder (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x4235DC: cuvid_test_dummy_decoder (cuvid.c:670)
==21588==    by 0x4235DC: cuvid_decode_init (cuvid.c:855)
==21588==    by 0xAF71A9: avcodec_open2 (utils.c:1609)
==21588==    by 0x49CA47: init_input_stream (ffmpeg.c:2758)
==21588==    by 0x49CA47: transcode_init (ffmpeg.c:3478)
==21588==    by 0x4A2C28: transcode (ffmpeg.c:4364)
==21588==    by 0x483823: main (ffmpeg.c:4598)
==21588== 
==21588== Conditional jump or move depends on uninitialised value(s)
==21588==    at 0x1C7E07AF: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7F44BC: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7F26BE: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E14A1: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E91D6: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E6ECA: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E4BD6: cuvidCreateDecoder (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x4235DC: cuvid_test_dummy_decoder (cuvid.c:670)
==21588==    by 0x4235DC: cuvid_decode_init (cuvid.c:855)
==21588==    by 0xAF71A9: avcodec_open2 (utils.c:1609)
==21588==    by 0x49CA47: init_input_stream (ffmpeg.c:2758)
==21588==    by 0x49CA47: transcode_init (ffmpeg.c:3478)
==21588==    by 0x4A2C28: transcode (ffmpeg.c:4364)
==21588==    by 0x483823: main (ffmpeg.c:4598)
==21588== 
==21588== Warning: noted but unhandled ioctl 0x42 with no size/direction hints.
==21588==    This could cause spurious value errors to appear.
==21588==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==21588== Warning: noted but unhandled ioctl 0x22 with no size/direction hints.
==21588==    This could cause spurious value errors to appear.
==21588==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==21588== Conditional jump or move depends on uninitialised value(s)
==21588==    at 0x1C7E0CED: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7F077F: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E8D4A: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x79B0C3: cuvid_handle_picture_decode (cuvid.c:300)
==21588==    by 0x1C7DEE47: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C845598: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C845FAB: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7DE970: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x79B322: cuvid_decode_packet (cuvid.c:393)
==21588==    by 0xAF2FCA: avcodec_send_packet (utils.c:2866)
==21588==    by 0x4A1CE2: decode (ffmpeg.c:2052)
==21588==    by 0x4A1CE2: decode_video (ffmpeg.c:2248)
==21588==    by 0x4A38E5: process_input_packet (ffmpeg.c:2491)
==21588==    by 0x4A38E5: process_input (ffmpeg.c:4251)
==21588==    by 0x4A38E5: transcode_step (ffmpeg.c:4339)
==21588==    by 0x4A38E5: transcode (ffmpeg.c:4393)
==21588== 
==21588== Conditional jump or move depends on uninitialised value(s)
==21588==    at 0x1C7E0D15: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7F077F: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7E8D4A: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x79B0C3: cuvid_handle_picture_decode (cuvid.c:300)
==21588==    by 0x1C7DEE47: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C845598: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C845FAB: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x1C7DE970: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==21588==    by 0x79B322: cuvid_decode_packet (cuvid.c:393)
==21588==    by 0xAF2FCA: avcodec_send_packet (utils.c:2866)
==21588==    by 0x4A1CE2: decode (ffmpeg.c:2052)
==21588==    by 0x4A1CE2: decode_video (ffmpeg.c:2248)
==21588==    by 0x4A38E5: process_input_packet (ffmpeg.c:2491)
==21588==    by 0x4A38E5: process_input (ffmpeg.c:4251)
==21588==    by 0x4A38E5: transcode_step (ffmpeg.c:4339)
==21588==    by 0x4A38E5: transcode (ffmpeg.c:4393)
==21588== 
    Last message repeated 10 times
==21588== 
==21588== HEAP SUMMARY:
==21588==     in use at exit: 187,676,954 bytes in 240,734 blocks
==21588==   total heap usage: 1,101,312 allocs, 860,578 frees, 3,892,090,398 bytes allocated
==21588== 
==21588== LEAK SUMMARY:
==21588==    definitely lost: 26,108 bytes in 200 blocks
==21588==    indirectly lost: 2,868,060 bytes in 579 blocks
==21588==      possibly lost: 54,614,106 bytes in 28,034 blocks
==21588==    still reachable: 130,168,680 bytes in 211,921 blocks
==21588==         suppressed: 0 bytes in 0 blocks
==21588== Rerun with --leak-check=full to see details of leaked memory
==21588== 
==21588== For counts of detected and suppressed errors, rerun with: -v
==21588== Use --track-origins=yes to see where uninitialised values come from
==21588== ERROR SUMMARY: 127 errors from 5 contexts (suppressed: 0 from 0)

VAAPI command:

valgrind ./ffmpeg_g -y -loglevel error -vaapi_device /dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -i "udp://239.250.14.12:5004?fifo_size=1000000&overrun_nonfatal=1" -ignore_unknown -sn -an -c:v h264_vaapi -b:v 1M -maxrate:v 1M -bufsize:v 500k -r 25 -g 50 -aspect 16:9 -vf "format=nv12|vaapi,hwupload,deinterlace_vaapi,scale_vaapi=w=640:h=360" -f hls -hls_time 6 -hls_list_size 6 -use_localtime 1 -hls_segment_filename "hls/1-%s.ts" "hls/1.m3u8"
==21586== Memcheck, a memory error detector
==21586== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==21586== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==21586== Command: ./ffmpeg_g -y -loglevel error -vaapi_device /dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -i udp://239.250.14.12:5004?fifo_size=1000000&overrun_nonfatal=1 -ignore_unknown -sn -an -c:v h264_vaapi -b:v 1M -maxrate:v 1M -bufsize:v 500k -r 25 -g 50 -aspect 16:9 -vf format=nv12|vaapi,hwupload,deinterlace_vaapi,scale_vaapi=w=640:h=360 -f hls -hls_time 6 -hls_list_size 6 -use_localtime 1 -hls_segment_filename hls/1-%s.ts hls/1.m3u8
==21586== 
libva info: VA-API version 0.39.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
[mpeg2video @ 0xc504860] Invalid frame dimensions 0x0.
    Last message repeated 3 times
==21586== 
==21586== HEAP SUMMARY:
==21586==     in use at exit: 12,940,216 bytes in 3,919 blocks
==21586==   total heap usage: 3,916,874 allocs, 3,912,955 frees, 265,677,253,039 bytes allocated
==21586== 
==21586== LEAK SUMMARY:
==21586==    definitely lost: 9,714,600 bytes in 290 blocks
==21586==    indirectly lost: 3,014,596 bytes in 626 blocks
==21586==      possibly lost: 48,648 bytes in 166 blocks
==21586==    still reachable: 162,372 bytes in 2,837 blocks
==21586==         suppressed: 0 bytes in 0 blocks
==21586== Rerun with --leak-check=full to see details of leaked memory
==21586== 
==21586== For counts of detected and suppressed errors, rerun with: -v
==21586== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

After 10 minutes NVENC was using 800 MB and VAAPI 400 MB, but usage was rising when I stopped it.

comment:3 Changed 3 months ago by alqemyst

Now when I try wihout pthreads I have following:

./ffmpeg -version
ffmpeg version N-83708-g4bed066 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609
configuration: --prefix=/opt/siqnal --enable-gpl --enable-nonfree --enable-version3 --enable-libx264 --enable-libfdk_aac --enable-libmp3lame --enable-openssl --enable-libfreetype --enable-fontconfig --enable-vaapi --enable-nvenc --enable-cuda --enable-cuvid --enable-libnpp --disable-ffplay --disable-doc --disable-ffserver --disable-pthreads
libavutil      55. 47.100 / 55. 47.100
libavcodec     57. 81.101 / 57. 81.101
libavformat    57. 66.102 / 57. 66.102
libavdevice    57.  3.100 / 57.  3.100
libavfilter     6. 74.100 /  6. 74.100
libswscale      4.  3.101 /  4.  3.101
libswresample   2.  4.100 /  2.  4.100
libpostproc    54.  2.100 / 54.  2.100

NVENC

valgrind ./ffmpeg_g -y -loglevel error -hwaccel cuvid -c:v mpeg2_cuvid -deint 2 -i "udp://239.250.14.12:5004?fifo_size=1000000&overrun_nonfatal=1" -ignore_unknown -sn -an -c:v nvenc_h264 -profile:v high -b:v 1M -maxrate:v 1M -bufsize:v 500k -r 25 -g 50 -aspect 16:9 -vf scale_npp=640:360 -f hls -hls_time 6 -hls_list_size 6 -use_localtime 1 -hls_segment_filename "hls/0-%s.ts" "hls/0.m3u8"
==1753== Memcheck, a memory error detector
==1753== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==1753== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==1753== Command: ./ffmpeg_g -y -loglevel error -hwaccel cuvid -c:v mpeg2_cuvid -deint 2 -i udp://239.250.14.12:5004?fifo_size=1000000&overrun_nonfatal=1 -ignore_unknown -sn -an -c:v nvenc_h264 -profile:v high -b:v 1M -maxrate:v 1M -bufsize:v 500k -r 25 -g 50 -aspect 16:9 -vf scale_npp=640:360 -f hls -hls_time 6 -hls_list_size 6 -use_localtime 1 -hls_segment_filename hls/0-%s.ts hls/0.m3u8
==1753== 
[mpeg2video @ 0xc4f6d60] Invalid frame dimensions 0x0.
==1753== Warning: noted but unhandled ioctl 0x30000001 with no size/direction hints.
==1753==    This could cause spurious value errors to appear.
==1753==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1753== Warning: noted but unhandled ioctl 0x27 with no size/direction hints.
==1753==    This could cause spurious value errors to appear.
==1753==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1753== Warning: noted but unhandled ioctl 0x7ff with no size/direction hints.
==1753==    This could cause spurious value errors to appear.
==1753==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1753== Warning: noted but unhandled ioctl 0x25 with no size/direction hints.
==1753==    This could cause spurious value errors to appear.
==1753==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1753== Warning: noted but unhandled ioctl 0x17 with no size/direction hints.
==1753==    This could cause spurious value errors to appear.
==1753==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1753== Warning: set address range perms: large range [0x200000000, 0x700000000) (noaccess)
==1753== Warning: set address range perms: large range [0x900000000, 0xc00000000) (noaccess)
==1753== Warning: noted but unhandled ioctl 0x19 with no size/direction hints.
==1753==    This could cause spurious value errors to appear.
==1753==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1753== Warning: noted but unhandled ioctl 0x21 with no size/direction hints.
==1753==    This could cause spurious value errors to appear.
==1753==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1753== Warning: noted but unhandled ioctl 0x1b with no size/direction hints.
==1753==    This could cause spurious value errors to appear.
==1753==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1753== Conditional jump or move depends on uninitialised value(s)
==1753==    at 0x10C9478F: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CA84BC: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CA66BE: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C954A1: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C9D1D6: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C9AECA: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C98BD6: cuvidCreateDecoder (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x422F8C: cuvid_test_dummy_decoder (cuvid.c:670)
==1753==    by 0x422F8C: cuvid_decode_init (cuvid.c:855)
==1753==    by 0xAEF898: avcodec_open2 (utils.c:1609)
==1753==    by 0x49C007: init_input_stream (ffmpeg.c:2758)
==1753==    by 0x49C007: transcode_init (ffmpeg.c:3478)
==1753==    by 0x4A21E8: transcode (ffmpeg.c:4364)
==1753==    by 0x483073: main (ffmpeg.c:4598)
==1753== 
==1753== Conditional jump or move depends on uninitialised value(s)
==1753==    at 0x10C9479F: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CA84BC: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CA66BE: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C954A1: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C9D1D6: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C9AECA: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C98BD6: cuvidCreateDecoder (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x422F8C: cuvid_test_dummy_decoder (cuvid.c:670)
==1753==    by 0x422F8C: cuvid_decode_init (cuvid.c:855)
==1753==    by 0xAEF898: avcodec_open2 (utils.c:1609)
==1753==    by 0x49C007: init_input_stream (ffmpeg.c:2758)
==1753==    by 0x49C007: transcode_init (ffmpeg.c:3478)
==1753==    by 0x4A21E8: transcode (ffmpeg.c:4364)
==1753==    by 0x483073: main (ffmpeg.c:4598)
==1753== 
==1753== Conditional jump or move depends on uninitialised value(s)
==1753==    at 0x10C947AF: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CA84BC: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CA66BE: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C954A1: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C9D1D6: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C9AECA: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C98BD6: cuvidCreateDecoder (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x422F8C: cuvid_test_dummy_decoder (cuvid.c:670)
==1753==    by 0x422F8C: cuvid_decode_init (cuvid.c:855)
==1753==    by 0xAEF898: avcodec_open2 (utils.c:1609)
==1753==    by 0x49C007: init_input_stream (ffmpeg.c:2758)
==1753==    by 0x49C007: transcode_init (ffmpeg.c:3478)
==1753==    by 0x4A21E8: transcode (ffmpeg.c:4364)
==1753==    by 0x483073: main (ffmpeg.c:4598)
==1753== 
==1753== Warning: noted but unhandled ioctl 0x42 with no size/direction hints.
==1753==    This could cause spurious value errors to appear.
==1753==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1753== Warning: noted but unhandled ioctl 0x22 with no size/direction hints.
==1753==    This could cause spurious value errors to appear.
==1753==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1753== Conditional jump or move depends on uninitialised value(s)
==1753==    at 0x10C94CED: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CA477F: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C9CD4A: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x797A23: cuvid_handle_picture_decode (cuvid.c:300)
==1753==    by 0x10C92E47: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CF9598: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CF9FAB: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C92970: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x797C82: cuvid_decode_packet (cuvid.c:393)
==1753==    by 0xAEB7DA: avcodec_send_packet (utils.c:2866)
==1753==    by 0x4A12A2: decode (ffmpeg.c:2052)
==1753==    by 0x4A12A2: decode_video (ffmpeg.c:2248)
==1753==    by 0x4A2DD4: process_input_packet (ffmpeg.c:2491)
==1753==    by 0x4A2DD4: process_input (ffmpeg.c:4251)
==1753==    by 0x4A2DD4: transcode_step (ffmpeg.c:4339)
==1753==    by 0x4A2DD4: transcode (ffmpeg.c:4393)
==1753== 
==1753== Conditional jump or move depends on uninitialised value(s)
==1753==    at 0x10C94D15: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CA477F: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C9CD4A: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x797A23: cuvid_handle_picture_decode (cuvid.c:300)
==1753==    by 0x10C92E47: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CF9598: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10CF9FAB: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x10C92970: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==1753==    by 0x797C82: cuvid_decode_packet (cuvid.c:393)
==1753==    by 0xAEB7DA: avcodec_send_packet (utils.c:2866)
==1753==    by 0x4A12A2: decode (ffmpeg.c:2052)
==1753==    by 0x4A12A2: decode_video (ffmpeg.c:2248)
==1753==    by 0x4A2DD4: process_input_packet (ffmpeg.c:2491)
==1753==    by 0x4A2DD4: process_input (ffmpeg.c:4251)
==1753==    by 0x4A2DD4: transcode_step (ffmpeg.c:4339)
==1753==    by 0x4A2DD4: transcode (ffmpeg.c:4393)
==1753== 
    Last message repeated 1 times
==1753== 
==1753== HEAP SUMMARY:
==1753==     in use at exit: 188,936,314 bytes in 241,022 blocks
==1753==   total heap usage: 1,416,881 allocs, 1,175,859 frees, 5,150,659,127 bytes allocated
==1753== 
==1753== LEAK SUMMARY:
==1753==    definitely lost: 33,820 bytes in 261 blocks
==1753==    indirectly lost: 4,087,020 bytes in 807 blocks
==1753==      possibly lost: 54,646,874 bytes in 28,035 blocks
==1753==    still reachable: 130,168,600 bytes in 211,919 blocks
==1753==         suppressed: 0 bytes in 0 blocks
==1753== Rerun with --leak-check=full to see details of leaked memory
==1753== 
==1753== For counts of detected and suppressed errors, rerun with: -v
==1753== Use --track-origins=yes to see where uninitialised values come from
==1753== ERROR SUMMARY: 127 errors from 5 contexts (suppressed: 0 from 0)

VAAPI

valgrind ./ffmpeg_g -y -loglevel error -vaapi_device /dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -i "udp://239.250.14.12:5004?fifo_size=1000000&overrun_nonfatal=1" -ignore_unknown -sn -an -c:v h264_vaapi -b:v 1M -maxrate:v 1M -bufsize:v 500k -r 25 -g 50 -aspect 16:9 -vf "format=nv12|vaapi,hwupload,deinterlace_vaapi,scale_vaapi=w=640:h=360" -f hls -hls_time 6 -hls_list_size 6 -use_localtime 1 -hls_segment_filename "hls/1-%s.ts" "hls/1.m3u8"
==1752== Memcheck, a memory error detector
==1752== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==1752== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==1752== Command: ./ffmpeg_g -y -loglevel error -vaapi_device /dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -i udp://239.250.14.12:5004?fifo_size=1000000&overrun_nonfatal=1 -ignore_unknown -sn -an -c:v h264_vaapi -b:v 1M -maxrate:v 1M -bufsize:v 500k -r 25 -g 50 -aspect 16:9 -vf format=nv12|vaapi,hwupload,deinterlace_vaapi,scale_vaapi=w=640:h=360 -f hls -hls_time 6 -hls_list_size 6 -use_localtime 1 -hls_segment_filename hls/1-%s.ts hls/1.m3u8
==1752== 
libva info: VA-API version 0.39.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
[mpeg2video @ 0xc504580] Invalid frame dimensions 0x0.
Unrepairable overflow!ted 10 times
    Last message repeated 10 times
==1752== 
==1752== HEAP SUMMARY:
==1752==     in use at exit: 14,172,856 bytes in 4,230 blocks
==1752==   total heap usage: 5,505,497 allocs, 5,501,267 frees, 375,443,749,639 bytes allocated
==1752== 
==1752== LEAK SUMMARY:
==1752==    definitely lost: 9,723,940 bytes in 362 blocks
==1752==    indirectly lost: 4,069,220 bytes in 836 blocks
==1752==      possibly lost: 217,404 bytes in 197 blocks
==1752==    still reachable: 162,292 bytes in 2,835 blocks
==1752==         suppressed: 0 bytes in 0 blocks
==1752== Rerun with --leak-check=full to see details of leaked memory
==1752== 
==1752== For counts of detected and suppressed errors, rerun with: -v
==1752== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

After 20 minutes nvenc takes 574 MB and vaapi 162 MB and these values aren't changing over time.

comment:4 Changed 3 months ago by cehoyos

Your original command line looks very different than the one you are testing now: Is the issue only reproducible with hardware acceleration? Is it only reproducible with hardware acceleration using filters?
Is network input required to reproduce the problem you see?

comment:5 Changed 3 months ago by alqemyst

I can reproduce it with libx264 also. Reason why I was testing here with nvidia and vaapi is because leak is faster that way.

It seems with http input leak is slower and I don't have enough data, also udp without cirrcular buffer. When I append cirrcular buffer I can very quickly see memory increasing. Of course when ffmpeg is compiled with --disable-pthreads fifo_sze is omitted but I kept same command anyway.

I will continue investigation and let you know if I find something more useful. Same ffmpeg and same commands, at least QSV, VAAPI and libx264 work on Debian without leaks.

Last edited 3 months ago by alqemyst (previous) (diff)

comment:6 Changed 3 months ago by stevenliu

https://patchwork.ffmpeg.org/patch/2706/

test with this patch please.

comment:7 Changed 3 months ago by alqemyst

Thanks for the patch, now this is really strange. Over 30 min vaapi is stable at 218 MB of resident size. During this period nvidia is constantly increasing, but slower than before this patch, currently it's takinng 600 MB. Well I'm not really sure what to do now. Let's see if I remove libnpp filters.

Last edited 3 months ago by alqemyst (previous) (diff)

comment:8 Changed 3 months ago by alqemyst

What I can tell now is when cuvid and libnpp are removed, nvidia looks stable over 30 min. Resident size 577 MB. Here is valgrind:

valgrind ./ffmpeg_g -y -loglevel error -i "udp://239.250.14.12:5004?reuse=1&timeout=3000000&fifo_size=1000000&overrun_nonfatal=1&buffer_size=16777216" -ignore_unknown -sn -an -c:v nvenc_h264 -profile:v high -b:v 2M -maxrate:v 2M -bufsize:v 1M -f hls -hls_time 6 -hls_list_size 6 -use_localtime 1 -hls_segment_filename "hls/0-%s.ts" "hls/0.m3u8"
==2052== Memcheck, a memory error detector
==2052== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==2052== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==2052== Command: ./ffmpeg_g -y -loglevel error -i udp://239.250.14.12:5004?reuse=1&timeout=3000000&fifo_size=1000000&overrun_nonfatal=1&buffer_size=16777216 -ignore_unknown -sn -an -c:v nvenc_h264 -profile:v high -b:v 2M -maxrate:v 2M -bufsize:v 1M -f hls -hls_time 6 -hls_list_size 6 -use_localtime 1 -hls_segment_filename hls/0-%s.ts hls/0.m3u8
==2052== 
[mpeg2video @ 0xc4f6800] Invalid frame dimensions 0x0.
==2052== Warning: noted but unhandled ioctl 0x30000001 with no size/direction hints.
==2052==    This could cause spurious value errors to appear.
==2052==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==2052== Warning: noted but unhandled ioctl 0x27 with no size/direction hints.
==2052==    This could cause spurious value errors to appear.
==2052==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==2052== Warning: noted but unhandled ioctl 0x7ff with no size/direction hints.
==2052==    This could cause spurious value errors to appear.
==2052==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==2052== Warning: noted but unhandled ioctl 0x25 with no size/direction hints.
==2052==    This could cause spurious value errors to appear.
==2052==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==2052== Warning: noted but unhandled ioctl 0x17 with no size/direction hints.
==2052==    This could cause spurious value errors to appear.
==2052==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==2052== Warning: set address range perms: large range [0x200000000, 0x700000000) (noaccess)
==2052== Warning: set address range perms: large range [0x900000000, 0xc00000000) (noaccess)
==2052== Warning: noted but unhandled ioctl 0x19 with no size/direction hints.
==2052==    This could cause spurious value errors to appear.
==2052==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==2052== Warning: noted but unhandled ioctl 0x21 with no size/direction hints.
==2052==    This could cause spurious value errors to appear.
==2052==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==2052== Warning: noted but unhandled ioctl 0x1b with no size/direction hints.
==2052==    This could cause spurious value errors to appear.
==2052==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==2052== Conditional jump or move depends on uninitialised value(s)
==2052==    at 0x2421978F: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2422D4BC: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2422B6BE: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2421A4A1: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2429D873: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x23FE2F82: ??? (in /usr/lib/nvidia-378/libnvidia-encode.so.378.13)
==2052==    by 0x23FF221C: ??? (in /usr/lib/nvidia-378/libnvidia-encode.so.378.13)
==2052==    by 0x46D441: nvenc_open_session (nvenc.c:168)
==2052==    by 0x46F950: nvenc_check_device (nvenc.c:362)
==2052==    by 0x46F950: nvenc_setup_device (nvenc.c:449)
==2052==    by 0x46F950: ff_nvenc_encode_init (nvenc.c:1270)
==2052==    by 0xAF86C9: avcodec_open2 (utils.c:1609)
==2052==    by 0x49ED5F: init_output_stream.constprop.23 (ffmpeg.c:3404)
==2052==    by 0x4A3427: reap_filters (ffmpeg.c:1437)
==2052== 
==2052== Conditional jump or move depends on uninitialised value(s)
==2052==    at 0x2421979F: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2422D4BC: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2422B6BE: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2421A4A1: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2429D873: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x23FE2F82: ??? (in /usr/lib/nvidia-378/libnvidia-encode.so.378.13)
==2052==    by 0x23FF221C: ??? (in /usr/lib/nvidia-378/libnvidia-encode.so.378.13)
==2052==    by 0x46D441: nvenc_open_session (nvenc.c:168)
==2052==    by 0x46F950: nvenc_check_device (nvenc.c:362)
==2052==    by 0x46F950: nvenc_setup_device (nvenc.c:449)
==2052==    by 0x46F950: ff_nvenc_encode_init (nvenc.c:1270)
==2052==    by 0xAF86C9: avcodec_open2 (utils.c:1609)
==2052==    by 0x49ED5F: init_output_stream.constprop.23 (ffmpeg.c:3404)
==2052==    by 0x4A3427: reap_filters (ffmpeg.c:1437)
==2052== 
==2052== Conditional jump or move depends on uninitialised value(s)
==2052==    at 0x242197AF: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2422D4BC: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2422B6BE: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2421A4A1: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x2429D873: ??? (in /usr/lib/nvidia-378/libnvcuvid.so.378.13)
==2052==    by 0x23FE2F82: ??? (in /usr/lib/nvidia-378/libnvidia-encode.so.378.13)
==2052==    by 0x23FF221C: ??? (in /usr/lib/nvidia-378/libnvidia-encode.so.378.13)
==2052==    by 0x46D441: nvenc_open_session (nvenc.c:168)
==2052==    by 0x46F950: nvenc_check_device (nvenc.c:362)
==2052==    by 0x46F950: nvenc_setup_device (nvenc.c:449)
==2052==    by 0x46F950: ff_nvenc_encode_init (nvenc.c:1270)
==2052==    by 0xAF86C9: avcodec_open2 (utils.c:1609)
==2052==    by 0x49ED5F: init_output_stream.constprop.23 (ffmpeg.c:3404)
==2052==    by 0x4A3427: reap_filters (ffmpeg.c:1437)
==2052== 
    Last message repeated 4 times
==2052== Warning: noted but unhandled ioctl 0x42 with no size/direction hints.
==2052==    This could cause spurious value errors to appear.
==2052==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==2052== Warning: noted but unhandled ioctl 0x22 with no size/direction hints.
==2052==    This could cause spurious value errors to appear.
==2052==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==2052== 
==2052== HEAP SUMMARY:
==2052==     in use at exit: 497,122 bytes in 3,060 blocks
==2052==   total heap usage: 3,205,074 allocs, 3,202,014 frees, 8,895,453,494 bytes allocated
==2052== 
==2052== LEAK SUMMARY:
==2052==    definitely lost: 2,556 bytes in 128 blocks
==2052==    indirectly lost: 80 bytes in 2 blocks
==2052==      possibly lost: 3,088 bytes in 22 blocks
==2052==    still reachable: 491,398 bytes in 2,908 blocks
==2052==         suppressed: 0 bytes in 0 blocks
==2052== Rerun with --leak-check=full to see details of leaked memory
==2052== 
==2052== For counts of detected and suppressed errors, rerun with: -v
==2052== Use --track-origins=yes to see where uninitialised values come from
==2052== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)

What I can do next is to add cuvid and libnpp in 2 separate tests and let you know which of these two leaks.

comment:9 Changed 3 months ago by stevenliu

Can you test is only write ts, don't write hls ?

comment:10 Changed 3 months ago by alqemyst

Sure, I was going to test mpegts and rtmp outputs anyway while I'm grinding. What I can tell you now is that nvidia memory seems stable when I removed -deint 2, both cuvid (328 MB) and cuvid + libnpp (523 MB) setups.

I will switch soon to .ts.

Thanks,

comment:11 Changed 3 months ago by alqemyst

I need more uptime to verify this, but your patch fixed hls muxer, while removing -deint 2 from cuvid fixed nvidia issue. I wonder is this related to:

https://patchwork.ffmpeg.org/patch/2520/

I will provide more infos after I apply this patch.

comment:12 Changed 3 months ago by alqemyst

I confirm that deint 2 (refering to adaptive) was causing second leak. Using deint 1 (bob) seems fine. From what I read adaptive deinterlacing is not fully performed inside nvdec chip. It also uses cuda cores. Anyway I hope this can help somehow.

Ticket subject is now misleading, maybe I should change it.

Thanks to all.

Note: See TracTickets for help on using tickets.