pp=linblenddeint failure

[markfilipak]

Summary of the bug:

In Windows 1803

This:
ffmpeg -i IN -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)
,5)
,3))'[C],[B]select='eq(mod((n+1)
,5)
,3)'[D],[C][D]interleave" OUT
succeeds.

But this:
ffmpeg -i IN -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)
,5)
,3))'[C],[B]select='eq(mod((n+1)
,5)
,3)',pp=linblenddeint[D],[C][D]interleave" OUT
fails.

Due to the delayed nature of the failure, I wonder whether its cause might be the same as '​https://trac.ffmpeg.org/ticket/8612'.

Without pp=linblenddeint:
CPU utilization is steady between 85% & 95%. Output is about 13 GB.
Playback: Normal.

With pp=linblenddeint:
CPU utilization is erratic, generally below 35%, sometimes below 10%. Output is about 1 GB.
Playback: Total time is not correct. It begins at zero and advances, keeping just ahead of running time. Video freezes when total time reaches 19:01.

I have tested this with 2 M2TS files and 2 VOB files.

Thank you for making ffmpeg. Thank you for maintaining ffmpeg.

How to reproduce:

ffmpeg -i IN -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)\\,5)\\,3))'[C],[B]select='eq(mod((n+1)\\,5)\\,3)',pp=linblenddeint[D],[C][D]interleave" OUT

ffmpeg version git-2020-04-20-cacdac8

[markfilipak]

Sorry for the attachment mixup. Zip files are apparently preferred here, but apparently verboten in the mailing list. I can't find a way to delete the 1st attachment.

[Carl Eugen Hoyos]

Please provide the input file 00000.m2ts

[markfilipak]

It's 23 GB and copyrighted.

[Carl Eugen Hoyos]

Feel free to reopen this ticket if you can provide an input file that allows to reproduce the issue you see.

[markfilipak]

I didn't need to supply a file for ticket 8612. Why do I need to supply one for this ticket? The file would need to be a p24 with a running time of 10 minutes or more. Do you have such a file? Can you give it to me? I would retest using that file.

[pdr0]

You can use an internal generator, such as testsrc2

e.g. a 1800 sec clip

ffmpeg -f lavfi -i testsrc2=s=720x480:rate=24000/1001:duration=1800

ffmpeg -f lavfi -i testsrc2=s=720x480:rate=24000/1001:duration=1800 -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1),5),3))'[C],[B]select='eq(mod((n+1),5),3)',pp=linblenddeint[D],[C][D]interleave" -c:v libx264 -preset:v faster -crf 20 -an -aspect 16/9 test.mkv -y

That encode hangs at 29 min 59sec , but that's expected with select and interleave. Otherwise it "works" - that suggests that it's source related

[markfilipak]

I'm testing with "Sintel" right now. "Sintel" is 14:48. The transcode has hung at 11:50. I don't think that can be select+interleave related. I'm waiting for it to either complete -- I had a BD that did complete after a long delay -- or reach 14:48 (at which time I will 'q'uit).

Addendum: I didn't realize it, but the version of the batch file that invokes ffmpeg had '-ss 2:58'. That explains the missing running time.

[markfilipak]

I'm rerunning "Sintel". I noticed something that may be important.

In the 1st run, the memory allocated at that time was 29 GB. The task had a couple dozen threads, but was using only 0.06% CPU time. During the test, CPU usage was about 30%.

In the 2nd run (in progress as I write this), the memory allocated was 2 GB (going up and down), but is now 22 GB and climbing. CPU usage was 92%, but has now fallen to 0.06%.

Addendum: The run had finished. Of course, I had to 'q'uit because of the select+interleave hang. I'm analyzing the transcode now.

[markfilipak]

2nd run analyzed. The symptoms are different.

Instead of the video hanging, the new ("Sintel") symptom is that transcode framerate drops from 60fps to 12fps at 12:12 running time -- of course, the "Sintel" source is 24fps to the end. In order to confirm this as a real symptom, I will repeat the transcode, this time without the 'pp' filter to determine whether this new symptom results from 'pp' or is present in a transcode without 'pp'. Please stand by...

[markfilipak]

Okay, I submitted this:

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "M:
Test Videos
Sintel.2010.1080p.mkv" -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)\,5)\,3))'[C],[B]select='eq(mod((n+1)\,5)\,3)'[D],[C][D]interleave" -map 0 -c:v libx264 -crf 20 -codec:a copy -codec:s copy "C:\AVOut\Sintel.2010.1080p.3.MKV"

(note: no 'pp' filter)

and the transcode does drop from 60fps to 12fps at 12:12 running time. So that symptom is not a result of 'pp=linblenddeint'.

Have I encounted one issue? Two related issues? Two unrelated issues? I don't know.

What should I do?
A, Attach the report for this 2nd symptom?
B, Create a new ticket for this 2nd symptom?
C, Submit all this information to ffmpeg-users and ask for help/confirmation?

[pdr0]

The 29GB, 22GB memory usage suggest a memory leak - but it should be reproducible using testsrc2.

I repeated the earlier testsrc2 test at 1920x1080 and have no issues. I set end signal with -t 00:20:00 and encode completes gracefully

I repeated with a retail BD for 20min limited by -t 00:20:00, same - no problems. All tests used pp=linblenddeint. Windows x64

Memory usage ok (fluctuates ~500-520MB for 1920x1080) and encoding fps stays about the same in all tests

If you didn't mention the memory usage, I would suggest checking temps, looking at CPU throttling next.

When debugging, I suggest using a faster encoding preset such as -preset:v superfast. You will encode and get feedback faster

[markfilipak]

Hey, pdr0,

I so appreciate your comments. I find them to be quite valuable.

I made 2 originals of testsrc2: a 720x480 and a 1920x1080. I then transcoded each of them to 55-telecine and 55-telecine with pp lineblend. That's 8 combinations.

None of the telecine processing went above about 700 MB of process memory. So, is 29 GB excessive? I think so. Does that indicate that when I do the same sort of processing on an M2TS, something causes a memory leak? I guess so, but I don't know of any way to troubleshoot it further. Does that warrant a "memory leak" trac ticket? I don't know.

Carl Eugen asked me to attach the M2TS, but I can't do it because the video is copyrighted material. I can copy it as a backup (fair use), but I can't post it.

My CPU is not throttled. The computer is an Acer Predator gaming laptop, but it's not throttled (or overclocked).

[markfilipak]

Replying to cehoyos:

Please provide the input file 00000.m2ts

Carl Eugen, do you have a blu-ray of "2001: A Space Odyssey"?

[Carl Eugen Hoyos]

No but I saw it a year and a half ago in 70mm.

[markfilipak]

Replying to cehoyos:

No but I saw it a year and a half ago in 70mm.

Oh, lucky you!

Well, it appears there's something about that M2TS (and others) that's causing ffmpeg's memory usage to hit 29 GB. pdr0 thinks it may be a memory leak. I can't duplicate the behavior with "Sintel" -- a different symptom appears with "Sintel" that appears to be unrelated because memory usage doesn't explode -- and I can't post a copyrighted work. I'm willing to run/rerun more tests if you're willing to work along. Shall I continue with more tests?

[Carl Eugen Hoyos]

If you are generally unwilling to provide input files, I am not sure how useful additional tests are: We have a very large number of open, reproducible tickets, I don’t think we need more reports of bugs that we cannot reproduce.

[Balling]

Only 23 GB?? Haha, on torrents there is a 80.17 GB version of whole Bly-ray disk. Lol.

[markfilipak]

Replying to Balling:

Only 23 GB?? Haha, on torrents there is a 80.17 GB version of whole Bly-ray disk. Lol.

Really? Where? That must be the never seen, 8 hour version with the Martian subtitles.

I want it!

[markfilipak]

Replying to cehoyos:

If you are generally unwilling to provide input files, I am not sure how useful additional tests are: We have a very large number of open, reproducible tickets, I don’t think we need more reports of bugs that we cannot reproduce.

I agree.

I would try to find public domain videos that expose the flaws I'm seeing. I remembered there's public domain videos at archive.org.

Also, I'd run tests with blu-ray & DVD movies that I can't post in order to find any patterns or commonalities that might lead me to better testing.

I just don't know until I try, but I'll need information from you and others regarding memory leaks (and other things that I'm sure I'm not anticipating).

I just need to 'hear' you 'say', "Mark, we'll stand behind you."

[pdr0]

One difference is I used MakeMKV . My BD stream was packaged in a MKV container.

The MakeMKV beta is free. I suggest you retest the disc using MakeMKV, then using that MKV as the input for ffmpeg. If that works, maybe there is some transport stream related issue, maybe it's a decryption issue (how did you extract the transport stream?) . And maybe some other related issue with VOB

[markfilipak]

Replying to pdr0:

One difference is I used MakeMKV . My BD stream was packaged in a MKV container.

What did you use MakeMKV for, pdr0? There have been 3 videos in this case: Your 23.976p.mp4 (i.e., "MOVE" "TEXT"), my "2001: A Space Odyssey", and "Sintel". Did I miss something?

The MakeMKV beta is free. I suggest you retest the disc using MakeMKV, then using that MKV as the input for ffmpeg. If that works, maybe there is some transport stream related issue, maybe it's a decryption issue (how did you extract the transport stream?) . And maybe some other related issue with VOB

I didn't extract the transport stream. I submitted the M2TS directly to ffmpeg.

I have found that running a video through MKVToolNix fixes some problems with streams -- I presume it fixes some PTS problems. But MKVToolNix doesn't transcode and I'd prefer to use ffmpeg in a single command line rather than adding the MKVToolNix extra step.

However, as a test, I will try running "2001: A Space Odyssey" through MKVToolNix to make an MKV stream, then run that through ffmpeg. Stay tuned...

[pdr0]

Replying to markfilipak:

Replying to pdr0:

One difference is I used MakeMKV . My BD stream was packaged in a MKV container.

What did you use MakeMKV for, pdr0? There have been 3 videos in this case: Your 23.976p.mp4 (i.e., "MOVE" "TEXT"), my "2001: A Space Odyssey", and "Sintel". Did I miss something?

For my retail BD test that had no problems. You used a m2ts, but how did you get it off the disc ? (How did you decrypt it? AnyDVD HD etc... )

[Balling]

Replying to markfilipak:

Replying to Balling:

Only 23 GB?? Haha, on torrents there is a 80.17 GB version of whole Bly-ray disk. Lol.

Really? Where? That must be the never seen, 8 hour version with the Martian subtitles.

I want it!

magnet:?xt=urn:btih:40a238fc4be2aa35cd53582ab2d05627167395e8&dn=2001.A.Space.Odyssey.1968.COMPLETE.UHD.BLURAY-COASTER
this is 80 GB version, this is
magnet:?xt=urn:btih:45507afcbf0d3ee87cdbb79711c2174dda8b14a2&dn=2001.A.Space.Odyssey.1968.2160p.BluRay.HEVC_DV.mp4
60 GB version with better quality: dual-layer (changed to one layer) Dolby Vision. Dunno if you will manage to download it, only one sid and you are in the different autonomous system region.

[markfilipak]

Replying to pdr0:

Replying to markfilipak:

Replying to pdr0:

One difference is I used MakeMKV . My BD stream was packaged in a MKV container.

What did you use MakeMKV for, pdr0? There have been 3 videos in this case: Your 23.976p.mp4 (i.e., "MOVE" "TEXT"), my "2001: A Space Odyssey", and "Sintel". Did I miss something?

For my retail BD test that had no problems.

How did you do that? Didn't the BD have encryption?

You used a m2ts, but how did you get it off the disc ? (How did you decrypt it? AnyDVD HD etc... )

AnyDVD HD.

[pdr0]

Replying to markfilipak:

Replying to pdr0:

Replying to markfilipak:

Replying to pdr0:

One difference is I used MakeMKV . My BD stream was packaged in a MKV container.

What did you use MakeMKV for, pdr0? There have been 3 videos in this case: Your 23.976p.mp4 (i.e., "MOVE" "TEXT"), my "2001: A Space Odyssey", and "Sintel". Did I miss something?

For my retail BD test that had no problems.

How did you do that? Didn't the BD have encryption?

You used a m2ts, but how did you get it off the disc ? (How did you decrypt it? AnyDVD HD etc... )

AnyDVD HD.

makemkv decrypts it too

[markfilipak]

After spending about 20 hours troubleshooting, there appears to be 2 problems here:
1, PTS errors: "Starting new cluster due to timestamp" when memory usage reaches 8 GB at approximately 1:12.
2, Memory leak: "Error while filtering: Cannot allocate memory" when memory usage exceeds 120 GB at 18:46.32.
Disclaimer: I don't know for sure, but PTS errors & memory leak are my best guesses.

First things, first: Fix PTS errors.

FFREPORT=file=00000.1.LOG:level=32
ffmpeg -report -ss 2:58 -to 4:15 -analyzeduration 5000000000 -probesize 5000000000 -i G:\BDMV\STREAM\00000.m2ts -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)\,5)\,3))'[C],[B]select='eq(mod((n+1)\,5)\,3)',pp=linblenddeint[D],[C][D]interleave" -map 0 -c:v libx264 -crf 20 -codec:a copy -codec:a:1 pcm_s16be -codec:s copy "C:\AVOut\00000.1.MKV"

I would attach 00000.1.MKV (84,480,629 bytes) & 00000.1.LOG (76,586 bytes) but there's a 2.5 MB limit. Granted that 00000.1.MKV is not 00000.m2ts, but maybe it would help.

I now also have 00000.2.LOG (95,519,111 bytes), which is at debug level.

[mkver]

You can upload big files ​here; make sure to select FFmpeg and set the track ticket. (But actually output files are pretty much useless, we want the input files that cause the issues.)

Is your subtitle track sparse? How much memory does the process take if you leave it out?

[markfilipak]

Replying to mkver:

... we want the input files that cause the issues.)

Yeah, I figured that would be the case.

Should I trim 00000.m2ts and save the trimmed file as an MKV without transcoding from VC1 to H264 or would the result still be useless?

[markfilipak]

Idea: What if I used a hex editor to clip the first 100 MB of 00000.m2ts?

[Balling]

Replying to markfilipak:

Idea: What if I used a hex editor to clip the first 100 MB of 00000.m2ts?

Yep, totally fine.

[markfilipak]

I've tried several large M2TS files. These symptoms appear to be hard.

Computer:
Windows 10 1803.
Physical memory: 32 GB.
Disk: 1 TB, SS RAID-0, 562 GiB free.

Key:
[PE] is Windows Sysinternals, Process Explorer
[MPV] is MPV player (Windows)

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "I:\BDMV\STREAM\00001.m2ts" -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)\,5)\,3))'[C],[B]select='eq(mod((n+1)\,5)\,3)',pp=linblenddeint[D],[C][D]interleave" -map 0 -c:v libx264 -crf 20 -codec:a copy -codec:s copy "C:\AVOut\00001.3.MKV"

Processing Events:
[PE] Initial memory use is approx. 650 MiB.
[PE] Memory usage climbs from there.

• Private Bytes: Rises steadily to 124.1 GiB (Note 1).
• Working Set: Stabilizes in 23 to 25 GiB range, up-down-up-down (Note 2).

[PE] ffmpeg.exe runs 33 threads.
[ffmpeg] Displays continuous series of "Starting new cluster due to timestamp" when [PE] memory usage reaches 8.8 GiB.
[ffmpeg] Displays "Error while filtering: Cannot allocate memory" when [PE] Private Bytes reaches 124.1 GiB.

(Note 1) Since these allocations exceed the amount of physical memory, and since I saw a process named "Memory Compression" open-close-open-close, I assume that Private Bytes means virtual memory.
(Note 2) I assume Working Set is physical memory.

Playback of 00001.3.MKV:
[MPV] Total time starts at zero (instead of 2:51:57) and rises by about 2x(Running time).
[MPV] When Running time reaches about 3:30, video freezes while audio continues.
[MPV] Total time stops rising when it reaches 8:24.
[MPV] When Running time reaches 8:24, Total time begins rising again, keeping 1 second ahead of Running time.
[MPV} Abruptly closes, but I didn't catch the Running time when it closed -- it was greater than 10:00.

[Carl Eugen Hoyos]

Please reopen this ticket if you can provide an input file that allows to reproduce the issue you see.

[markfilipak]

Hey, Carl Eugen,

I'm working on it. I split the source into 158, 250MiB chunks and am concatenating them to make the smallest file that displays the symptoms. I'm currently up to 32 chunks (8 GB).

[Elon Musk]

This happens naturally because interleave filter picks frames from queued inputs with lowest pts. Also because one input constantly feeds more frames to interleave than another it will consume unlimited memory.

Feel free to consult graphmonitor filter and it will confirm this findings.

[markfilipak]

I hope this report helps.

Needed: Some way to upload 00001.018.m2ts (4,505,960,448 bytes).
(Upload to '​https://streams.videolan.org/upload/' failed because there's a 1 GB limit.)

ffmpeg version git-2020-04-20-cacdac8

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "C:\AVOut\foo\00001.018.m2ts" -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)\,5)\,3))'[C],[B]select='eq(mod((n+1)\,5)\,3)',pp=linblenddeint[D],[C][D]interleave" -map 0 -c:v libx264 -crf 20 -codec:a copy -codec:s copy "C:\AVOut\00001.018.1.MKV"

Source file:
00001.018.m2ts (4,505,960,448 bytes)
-- Initial 18:35 of "Patton" blu-ray --
fails as outlined below.

(Note that this file: 00001.017.m2ts (4,255,629,312 bytes) -- Initial 17:23 of "Patton" blu-ray -- does not malfunction.)

Attached:

• 00001.018.1.LOG
• 00001.018.m2ts.Startingnewclusterduetotimestamp.PNG -- [PE] Screen shot at "Starting new cluster due to timestamp".
• 00001.018.m2ts.Cannotallocatememory.PNG -- [PE] Screen shot at "Cannot allocate memory".

=====

Computer:
Windows 10 1803.
Physical memory: 32 GB.
Disk: 1 TB, SS RAID-0, 562 GiB free.

Key:
[PE] is Windows Sysinternals, Process Explorer
[MPV] is MPV player (Windows)

Processing Events:
[PE] Initial memory use is approx. 650 MiB.
[PE] Memory usage climbs from there.

Private Bytes: Rises steadily to approximately 123 GiB (Note 1).
Working Set: Stabilizes in 23 to 25 GiB range, up-down-up-down (Note 2).

[PE] ffmpeg.exe runs 33 threads.
[ffmpeg] Displays continuous series of "Starting new cluster due to timestamp" when [PE] Working Set reaches 8.8 GiB.
[ffmpeg] Displays "Error while filtering: Cannot allocate memory" when [PE] Private Bytes reaches 123 GiB.

(Note 1) Since these allocations exceed the amount of physical memory, and since I saw a process named "Memory Compression" open-close-open-close, I assume that Private Bytes means virtual memory.
(Note 2) I assume Working Set is physical memory.

Playback of 00001.3.MKV:
[MPV] 00:00 - Total time starts at zero (instead of 18:35) and rises by about 2x(Running time).
[MPV] 03:24 - Video freezes while audio continues. (Total time reads 8:24 at this time.)
[MPV] 08:24 - Running time reaches Total time. Total time begins rising again, keeping 1 second ahead of Running time.
[MPV} 16:37 - Abruptly closes (early).

[markfilipak]

Replying to richardpl:

I was up all night creating a video fragment that was smaller than the whole video but that nonetheless provokes the error.

You cannot close this and mark it 'invalid' without a reason.

[markfilipak]

I have uploaded the following to '​​https://streams.videolan.org/upload/' in order to reproduce the issue.

00001.018.m2ts.001
00001.018.m2ts.002
00001.018.m2ts.003
00001.018.m2ts.004
00001.018.m2ts.005
00001.018.m2ts.006
00001.018.m2ts.007
00001.018.m2ts.CRC

​https://trac.ffmpeg.org/ticket/8626#comment:36 is the core of the issue, and you needn't look further than there.

Thanks,
Mark

[Carl Eugen Hoyos]

You did not successfully upload 00001.018.m2ts.002

[Elon Musk]

As already mentioned, "bug" is in interleave filter queueing bunch of frames because unequal selection of frames. Your filtergraph is simply useless anyway.

[pdr0]

Replying to richardpl:

As already mentioned, "bug" is in interleave filter queueing bunch of frames because unequal selection of frames. Your filtergraph is simply useless anyway.

The queue should be released every 5 frames. That's not a large memory footprint (if there was no memory leak)

And if that's the explanation, then why does that filtergraph work ok with testsrc2 , or on my BD that used makemkv ? The memory consumption hovers around 500-520MB. Something else is going on

Would you suggest a better filtergraph format ? Such as 0,1,2,3,4... (5 equal inputs) then interleaving those ?

[markfilipak]

Replying to pdr0:
-snip-

The queue should be released every 5 frames. That's not a large memory footprint (if there was no memory leak)

And if that's the explanation, then why does that filtergraph work ok with testsrc2 , or on my BD that used makemkv ? The memory consumption hovers around 500-520MB. Something else is going on

Would you suggest a better filtergraph format ? Such as 0,1,2,3,4... (5 equal inputs) then interleaving those ?

That's a very good suggestion. I will try that.

[markfilipak]

Replying to cehoyos:

You did not successfully upload 00001.018.m2ts.002

May I explain?
I may have forgotten to select

(_)VideoLAN (o)FFmpeg (_)mplayer

You see, my laptop has a 17-inch UHD display. Those radio buttons (e.g.,
<input id="ffmpegproject" name="project" value="ffmpeg" type="radio">
) are not styled. They are so tiny... That they are only slightly darker gray against a gray background doesn't help, either.

I uploaded 00001.018.m2ts.002 again. I hope that worked.

[markfilipak]

I submitted this:
ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "D:\D\foo\patton\00001.018.m2ts" -filter_complex "telecine=pattern=5,split=5[A][B][C][D][E],[A]select='eq(mod(n+1\,5)\,1)'[F],[B]select='eq(mod(n+1\,5)\,2)'[G],[C]select='eq(mod(n+1\,5)\,3)',pp=linblenddeint[H],[D]select='eq(mod(n+1\,5)\,4)'[I],[E]select='eq(mod(n+1\,5)\,0)'[J],[F][G][H][I][J]interleave=nb_inputs=5" -map 0 -c:v libx264 -crf 20 -codec:a copy -codec:s copy "C:\AVOut\00001.018.2.MKV"

Playback of 00001.018.2.MKV:
It's better. The total time is correct now. But skipping forward & back in MPV reveals that the PTSs are still screwed up. I'm attaching the log.

[mkver]

This doesn't reveal anything. If you had used a higher loglevel it might have revealed something (e.g. the Matroska muxer outputs a bit of info about the packets it is writing).

[markfilipak]

Replying to mkver:

This doesn't reveal anything. If you had used a higher loglevel it might have revealed something (e.g. the Matroska muxer outputs a bit of info about the packets it is writing).

I wish I could, but you see, the debug log is too big.

20-04-22 23:06 9,000,365 00001.018.3.LOG.ZIP

You have the command line. You have (or can get) the source video. I'm sorry but you'll have to make your own run. I would like to know what you get.

Oh, by the way, there's nothing special about the 1st 18:35 of "Patton". I get the same results with every blu-ray.

[markfilipak]

Well, it now appears that this problem is entirely an 'interleave' fault, not 'pp'.

The following freezes video at 3:24, and skipping forward or back in MPV causes audio to die.

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "D:\D\foo\patton\00001.018.m2ts" -filter_complex "telecine=pattern=5,split=5[A][B][C][D][E],[A]select='eq(mod(n+1\,5)\,1)'[F],[B]select='eq(mod(n+1\,5)\,2)'[G],[C]select='eq(mod(n+1\,5)\,3)'[H],[D]select='eq(mod(n+1\,5)\,4)'[I],[E]select='eq(mod(n+1\,5)\,0)'[J],[F][G][H][I][J]interleave=nb_inputs=5" -map 0 -c:v libx264 -crf 20 -codec:a copy -codec:s copy "C:\AVOut\00001.018.4.MKV"

PS: To make things easier I should have noted that the above filter complex does not include 'pp=linblenddeint'.

[Elon Musk]

There may be gaps between timestamps. You still can not grasp the fact that interleave picks the least timestamp in each case and not some N-th frame.

If you ever bothered to read interleave filter documentation you would notice that it needs strictly monotonic increasing timestamps.

[markfilipak]

Replying to richardpl:

There may be gaps between timestamps. You still can not grasp the fact that interleave picks the least timestamp in each case and not some N-th frame.

If you ever bothered to read interleave filter documentation you would notice that it needs strictly monotonic increasing timestamps.

Paul,

A week ago in my very first "ffmpeg architecture question" I wrote: "I assume that frame numbers are assigned at the input of the filter chain as frames are encountered ... Is what I've written correct? Authoritative confirmation or correction of this architectural detail is desired."

Later that day I wrote: "When (and where in the filter complex) does ffmpeg assign frame numbers? I would guess that ffmpeg assigns frame numbers (0 1 2 3...) at the input, based on the PTSs of the arriving frames. The alternative would be that ffmpeg defers assigning the 'next' frame number (n+1) to a frame only when the frame succeeds to the output (or a queue), but I don't think that's what ffmpeg does. Again, this is a very important architectural detail, and I seek confirmation from a developer (Paul?)."

Note: "based on the PTSs of the arriving frames". Note: "I seek confirmation from a developer (Paul?)".

Instead of supplying the necessary information, you attacked me and wrote that I didn't know anything and that I am a troll.

[markfilipak]

Paul,

Will you answer Michael's question?

"How would you solve the problem?
"Given is a sequence of frames 1, 2, 3, 4, ...
"Wanted is a sequence 1, 1, 1+2, 2, 2, 3, 3, 3+4, 4, 4, ...
"where "+" stands for a mix of two frames.

"Michael"

[markfilipak]

IT'S THE SUBTITLES! The subtitle packets are screwing up 'interleave' (I think) -- look at the number of frames.

Replicating the problem:

Take the '00001.018.m2ts' that I uploaded (as parts) to ​​https://streams.videolan.org/ffmpeg/incoming/8626/. Rename it 'Patton_18m35s.m2ts'.

Run:

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "Patton_18m35s.m2ts" -filter_complex "telecine=pattern=5,split=5[A][B][C][D][E],[A]select='eq(mod((n+1)\,5)\,1)'[F],[B]select='eq(mod((n+1)\,5)\,2)'[G],[C]select='eq(mod((n+1)\,5)\,3)'[H],[D]select='eq(mod((n+1)\,5)\,4)'[I],[E]select='eq(mod((n+1)\,5)\,0)'[J],[F][G][H][I][J]interleave=nb_inputs=5" -map 0 -codec:v libx264 -crf 50 -codec:a copy -codec:s copy -dn "Patton_18m35s.1.MKV"

-- 58836 frames (58836 = 66851*0.8801065 ?)
-- video freezes at 3:24, audio dies when seeking

Sanity checks:

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "Patton_18m35s.m2ts" -filter_complex "telecine=pattern=5,split=5[A][B][C][D][E],[A]select='eq(mod((n+1)\,5)\,1)'[F],[B]select='eq(mod((n+1)\,5)\,2)'[G],[C]select='eq(mod((n+1)\,5)\,3)'[H],[D]select='eq(mod((n+1)\,5)\,4)'[I],[E]select='eq(mod((n+1)\,5)\,0)'[J],[F][G][H][I][J]interleave=nb_inputs=5" -map 0 -codec:v libx264 -crf 50 -codec:a copy -sn -dn "Patton_18m35s.1.MKV"

-- 66851 frames (66851 = int(26741*2.5)-1)
-- plays fine

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "Patton_18m35s.m2ts" -map 0 -codec:v libx264 -crf 50 -codec:a copy -codec:s copy -dn "Patton_18m35s.1.MKV"

-- 26741 frames
-- plays fine

PS: You have to let the output MKV run to 3:24. If you skip to 3:24, it doesn't freeze (but the audio dies and never returns).

[markfilipak]

I assume you are all familiar with "Sintel". It is available on the net, and it's in the public domain.

Transcoding with 55-transcode & split & select & interleave without subtitles works.

Transcoding with 55-transcode & split & select & interleave with subtitles doesn't work.

There's something about the interleave filter that breaks with subtitles. I think subtitles can span GOPS. Is the interleave filter handling those PTSs properly?

[Balling]

IT'S THE SUBTITLES!

Lol

[markfilipak]

Continuing my exploration, I have news.

I transcoded the video & audio. Then I extracted the subs separately. Then I merged them.

The result DOES NOT exhibit the problem. The subs lead the audio by 1:14, but otherwise, the merged version works correctly.

I hope this info helps. If you want any of the logs, I saved them. The command lines follow.

Step 1: Transcode 2001av.MKV.
ffmpeg -report -ss 2:57 -analyzeduration 5000000000 -probesize 5000000000 -i "G:\BDMV\STREAM\00000.m2ts" -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)\,5)\,3))'[C],[B]select='eq(mod((n+1)\,5)\,3)',pp=linblenddeint[D],[C][D]interleave" -map 0 -codec:v libx264 -crf 20 -codec:a copy -codec:a:1 pcm_s16be -sn -dn "2001av.MKV"

Step 2: Extract subtitles to 2001s.MKV.
ffmpeg -report -ss 2:57 -analyzeduration 5000000000 -probesize 5000000000 -i "G:\BDMV\STREAM\00000.m2ts" -map 0 -vn -an -codec:s copy -dn "2001s.MKV"

Step 3: Merge 2001s.MKV & 2001av.MKV to 2001.MKV.
ffmpeg -report -analyzeduration 5000000000 -probesize 5000000000 -i 2001av.MKV -i 2001s.MKV -map 0 -codec:v copy -codec:a copy -map 1:s -codec:s copy 2001.2.MKV

[markfilipak]

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "Patton_18m35s.m2ts" -map 0 -codec:v libx264 -crf 50 -codec:a copy -codec:s copy -dn "Patton_18m35s.1.MKV"

-- Patton_18m35s.1.MKV plays properly.

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "Patton_18m35s.m2ts" -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)\,5)\,3))'[C],[B]select='eq(mod((n+1)\\,5)\\,3)',pp=linblenddeint[D],[C][D]interleave" -map 0 -codec:v libx264 -crf 50 -codec:a copy -sn -dn "Patton_18m35s.2.MKV"

-- Patton_18m35s.2.MKV plays properly.

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "Patton_18m35s.m2ts" -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)\,5)\,3))'[C],[B]select='eq(mod((n+1)\\,5)\\,3)',pp=linblenddeint[D],[C][D]interleave" -map 0 -codec:v libx264 -crf 50 -codec:a copy -codec:s copy -dn "Patton_18m35s.3.MKV"

-- Patton_18m35s.3.MKV video freezes at 3m24s.

The sole differences (1 v. 2 v. 3) are '-codec:s copy' v. '-filter_complex'+'-sn' v. '-filter_complex'+'-codec:s copy'.

The finger points, not to '-codec:s copy', but to 'interleave': Either to 'interleave' itself, or to the PTSs that 'interleave' depends on.

[markfilipak]

I may have another clue.

I used the 'eac3to D: -demux' application to chop the Patton video I provided into pieces:
00001 - 12 - Subtitle (PGS), Spanish, 27 captions.sup
00001 - 13 - Subtitle (PGS), French, 139 captions.sup
00001 - 10 - Subtitle (PGS), Chinese, 1608 captions.sup
00001 - 11 - Subtitle (PGS), English, 112 captions.sup
00001 - 9 - Subtitle (PGS), Chinese, 1481 captions.sup
00001 - 5 - AC3, Spanish, 2.0 channels, 224kbps, 48kHz.ac3
00001 - 6 - AC3, French, 2.0 channels, 224kbps, 48kHz.ac3
00001 - 7 - AC3, English, 2.0 channels, 224kbps, 48kHz.ac3
00001 - 8 - Subtitle (PGS), English, 1639 captions.sup
00001 - 2 - h264, 1080p24.h264
00001 - 3 - DTS Master Audio, English, 5.1 channels, 24 bits, 48kHz.dtsma
00001 - 4 - AC3, English, 5.0 channels, 448kbps, 48kHz.ac3
00001 - Chapters.txt

They all seem okay except for the DTS MA. It exhibits the MPV Total Time symptom noted previously, to wit: Total Time starts at zero and then keeps ahead of running time by about 1 second.

Is this related to the problem that's cured by excluding subtitles (-sn)? I don't know, but maybe it's a valuable clue.

[markfilipak]

Major new symptom: For the file: '2001 1m58s -ss 1188 -to 1294 p24.mkv', at exactly 48 seconds and thereafter, FPS drops from 59.940 (+/- 2 FPS) to 11.988. In some cases, the video freezes a few seconds later.

In the logs: "[matroska @ 00000248fae72880] Starting new cluster due to timestamp". Is the running time 48 seconds when those notices begin? I think so, but, heck, I don't know. What is the point of "[matroska @ 00000248fae72880] Starting new cluster due to timestamp" in log files when it doesn't provide any informative metrics? Wouldn't providing the 'timestamp @ running time' be helpful?

Filters tested:
yadif essentially turns the 55-telecine into 46-telecine.
yaepblur worsens the combing of the frames that flow through it (i.e., 1 frame in 5). After 48 seconds, every frame is combed.
pp=lowpass5 is somewhat effective. After 48 seconds, every frame is combed.
pp=ffmpegdeint essentially turns the 55-telecine into 46-telecine.

At this point, I reran the tests, but with subtitles turned off (-sn).

The 48-second phenomenon does not happen with subtitles off (-sn) and "[matroska @ 00000248fae72880] Starting new cluster due to timestamp" does not appear in logs.

I can't attach the source video: '2001 1m58s -ss 1188 -to 1294 p24.mkv' so I'll try to add it via ​​https://streams.videolan.org/ffmpeg/incoming/8626/.

Edit, later.
I uploaded '2001 1m58s -ss 1188 -to 1294 p24.mkv' with the Description: "Provokes shift from 59.940 FPS to 11.988 FPS at 48 second running time when transcode includes subtitles." but it's not there. Oh, well.

PS: I should note that I 55-telecined the source. It doesn't appear that 55-telecine causes the problems. It appears that either 'select' or 'interleave' causes the problems. Also note that I tried 'mix' in place of 'interleave' and the problem was there, too.

[markfilipak]

I just tested 'Patton_18m35s.m2ts' with ffmpeg version git-2020-05-04-5767a2e with subtitles enabled.

ffmpeg -report -ss 0:00 -analyzeduration 5000000000 -probesize 5000000000 -i "M:\Test Videos\Patton_18m35s.m2ts" -filter_complex "telecine=pattern=5,split[A][B],[A]select='not(eq(mod((n+1)\,5)\,3))'[C],[B]select='eq(mod((n+1)\,5)\,3)',pp=linblenddeint[D],[C][D]interleave" -map 0 -codec:v libx264 -crf 18 -codec:a copy -codec:s copy -dn "C:\AVOut\[M][Videos]Patton_18m35s.01.MKV"

I see that the end-of-stream problem ('select'? with 'interleave'?) has been fixed. Thank you.

The PTS & video freeze issues are still there, though.