mxf_set_audio_pts creates incorrect PTS for compressed audio

[Markus P Schumann]

Summary of the bug:
How to reproduce: ffmpeg -i AVC_AAC.mxf AVC_AAC.mp4
MP4 output file will have incorrect timestamps, plays with A/V sync issues and scrubbing is not possible.

The bug is in mxf_set_audio_pts.

FFMEPG uses the element size (KLV) to keep a cumulative count and uses the count as PTS.

For PCM audio the element size is proportional to the sample count – for compressed audio (like AAC) it is not.

So - FFMPEG MXF audio PTS calculation works only for PCM correctly.

I verified my assumption by fixing FFMPEG MXF audio PTS calculation. Now I am counting the AAC packets in the KLV and then increment PTS += AAC_frame_count * 1024;

Now the resulting transcoded MP4 files for fine.

[Carl Eugen Hoyos]

Please provide the command line you tested together with the complete, uncut console output and point us to the input file.

[Markus P Schumann]

Command line: ffmpeg -i AVC_AAC.mxf AVC_AAC.mp4

Console output:

y:\bugs>ffmpeg_g.exe -i AVC_AAC.mxf AVC_AAC.mp4
ffmpeg version N-85950-g8ef2c79 Copyright (c) 2000-2017 the FFmpeg developers
  built with Microsoft (R) C/C++ Optimizing Compiler Version 18.00.40629 for x86
  configuration: --optflags=-Od --disable-asm --disable-yasm --arch=i386 --disable-avdevice --disable-ffserver --disable-doc --enable-static --disable-bzlib --disable-libopenjpeg --disable-iconv --dis
able-zlib --prefix=/c/ffmpeg --toolchain=msvc
  libavutil      55. 61.100 / 55. 61.100
  libavcodec     57. 94.101 / 57. 94.101
  libavformat    57. 72.101 / 57. 72.101
  libavfilter     6. 88.100 /  6. 88.100
  libswscale      4.  7.101 /  4.  7.101
  libswresample   2.  8.100 /  2.  8.100
[aac @ 0040B4B0] Multiple frames in a packet.
Input #0, mxf, from 'AVC_AAC.mxf':
  Metadata:
    uid             : 6b18ead6-cf32-1440-9d51-faaae544a9b0
    generation_uid  : 27516c27-70cd-134f-85bf-1a05bb1e4666
    company_name    : Telestream, LLC
    product_name    : Telestream Media Framework - Local
    product_version : 99.99.999999
    modification_date: 2018-08-18T06:15:56.000000Z
    product_uid     : 11350ec5-148a-1010-aaca-d2c1000f040a
    material_package_umid: 0x060A2B340101010501010D1213000000BC1C85B7AB1F194A8EBF42FFCD0077DE
    timecode        : 00:00:00:00
  Duration: 00:00:37.27, start: 0.000000, bitrate: 850 kb/s
    Stream #0:0: Video: h264 (Main), yuv420p(tv, smpte170m, progressive), 640x360 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 29.97 tbn, 60k tbc
    Metadata:
      file_package_umid: 0x060A2B340101010501010D1213000000E1BE093CFF5A1A47AC54CFC171237109
    Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp
    Metadata:
      file_package_umid: 0x060A2B340101010501010D1213000000E1BE093CFF5A1A47AC54CFC171237109

[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 1024, current: 8; changing to 1025. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 1032, current: 16; changing to 1033. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 1033, current: 20; changing to 1034. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 1044, current: 140; changing to 1045. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 1045, current: 205; changing to 1046. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 1229, current: 336; changing to 1230. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 1230, current: 401; changing to 1231. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 1425, current: 532; changing to 1426. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 1556, current: 925; changing to 1557. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 1557, current: 1039; changing to 1558. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2063, current: 1174; changing to 2064. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2064, current: 1236; changing to 2065. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2260, current: 1369; changing to 2261. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2261, current: 1435; changing to 2262. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2459, current: 1567; changing to 2460. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2591, current: 1697; changing to 2592. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2592, current: 1759; changing to 2593. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2783, current: 1896; changing to 2784. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2784, current: 1964; changing to 2785. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2988, current: 2089; changing to 2989. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 2989, current: 2158; changing to 2990. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 3182, current: 2283; changing to 3183. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 3183, current: 2359; changing to 3184. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 3383, current: 2482; changing to 3384. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 3506, current: 2616; changing to 3507. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 3507, current: 2682; changing to 3508. This may result in incorrect timestamps in the output file.
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 3706, current: 2814; changing to 3707. This may result in incorrect timestamps in the output file.
[aac @ 03C1FEC0] Queue input is backward in time
[mp4 @ 03B30460] Non-monotonous DTS in output stream 0:1; previous: 3707, current: 2874; changing to 3708. This may result in incorrect timestamps in the output file.

[snip]

[Carl Eugen Hoyos]

Is the issue reproducible with current FFmpeg git head?

[Markus P Schumann]

Yes:

Y:\bugs>ffmpeg_g.exe -i AVC_AAC.mxf AVC_AAC.mp4
ffmpeg version 3.4.git Copyright (c) 2000-2018 the FFmpeg developers
  built with Microsoft (R) C/C++ Optimizing Compiler Version 18.00.40629 for x86
  configuration: --optflags=-Od --disable-asm --disable-yasm --arch=i386 --disable-avdevice --disable-doc --enable-static --disable-zlib --disable-encoder=png --disable-decoder=png --disable-bzlib --d
isable-libopenjpeg --disable-iconv --prefix=/Y/formats/ffmpeg/ffmpeg-head/FFmpeg --toolchain=msvc
  libavutil      56. 19.100 / 56. 19.100
  libavcodec     58. 23.100 / 58. 23.100
  libavformat    58. 17.103 / 58. 17.103
  libavfilter     7. 26.100 /  7. 26.100
  libswscale      5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
[aac @ 01E39B70] Multiple frames in a packet.
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 0 to 1
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 1 to 2
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 2 to 3
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 3 to 4
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 4 to 5
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 5 to 6
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 6 to 7
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 7 to 8
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 8 to 9
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 9 to 10
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 10 to 11
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 11 to 12
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 12 to 13
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 13 to 14
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 14 to 15
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 15 to 16
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 16 to 17
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 17 to 18
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 18 to 19
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 19 to 20
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 20 to 21
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 21 to 22
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 22 to 23
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 23 to 24
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 24 to 25
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 25 to 26
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 26 to 27
[mxf @ 01E32FB0] edit unit sync lost on stream 1, jumping from 27 to 28

[Markus P Schumann]

The implementation of mxf_set_audio_pts hasn't changed between FFmpeg 3.4.1 and head. mxf_set_audio_pts still uses the packet size to determine the sample count which works well for PCM but not for compressed audio.

[Carl Eugen Hoyos]

How did you produce this version number?
Why can’t you test a sane binary?

[Markus P Schumann]

I cloned FFmpeg's head on GitHub.
I made debug build on my machine.
I reproduced the issue.

What would you like me to do instead?
Please send me a link to a sane binary and I am happy to post the console output - thanks.

[Carl Eugen Hoyos]

Replying to go4shoe:

I cloned FFmpeg's head on GitHub.

That sounds unlikely.

The console output of your comment:2 and comment:5 look very different: Did the issue really not change? It's not easy to see A/V desync for the given sample...

[Markus P Schumann]

The issue is in FFmpeg's MXF reader - creating the PTS timestamp.
mxf_set_audio_pts - hasn't changed in a fairly long time.

I am using FFmpeg's transcode from MXF to MP4 as a test invoking FFmpeg's MXF reader.

So the issue is in the mxf_set_audio_pts function that is in libavformat/mxfdec.c

[Carl Eugen Hoyos]

Is the issue reproducible with the sample you attached here?

[Markus P Schumann]

Yes - the attached sample file AVC_AAC.mxf reproduces the issue.

Although some of the surrounding code has changed - the function mxf_set_audio_pts in libavformat/mxfdec.c has been unchanged for some time and seems faulty.

[Carl Eugen Hoyos]

I tested transcoding the input file you attached with FFmpeg 3.4 and the output file has severe A/V synchronization issues. I repeated the test with current FFmpeg git head and the output file seems to play in-sync but this is difficult to verify because of the file content (small desync would not be visible afaict). If there still is an issue with current FFmpeg git head and the attached input file, please explain how I can reproduce / see it.

Please understand that I am not asking you to analyze the issue using the source code: If you are able to do that, you should not have opened a ticket but instead send a patch - made with git format-patch - to the development mailing list. In a ticket on this bug tracker, no analysis is required (or asked for), just explain how we can reproduce the problem you see.

Attaching screen-shots of FFmpeg source code is definitely not appreciated.

[Markus P Schumann]

Thanks for reproducing the issue in FFmpeg 3.4.

I am using ffmpeg.exe as test bed to show the timestamp issue in the FFmpeg's MXF parser.
The timestamp issue still exists - see console output of my comment 5.

But it looks like that in FFmpeg git head - other parts of FFmpeg make up for the faulty timestamps coming out of the FFmpeg's MXF reader.

So that now a transcode may create a file with A/V sync.

But for a player like VLC that uses libavformat/mxfdec.c the issue is likely unchanged.

[Carl Eugen Hoyos]

Replying to go4shoe:

But for a player like VLC that uses libavformat/mxfdec.c the issue is likely unchanged.

Which version of vlc did you test?

[Markus P Schumann]

I am using VLC 3.0.3 Vetinari - May 29 2018 17:37:08

[Carl Eugen Hoyos]

I suspect you will have to test a nightly.

[Carl Eugen Hoyos]

Replying to cehoyos:

I suspect you will have to test a nightly.

Unfortunately, you either have to compile yourself (after bumping the FFmpeg version in contrib/src/ffmpeg/rules.mak) or wait until videolan updates its FFmpeg components.

[Carl Eugen Hoyos]

The messages were silenced in 6aaf1b504c6a0e0e5e2f4c97712b0f83a4115e10, afaiu, there is no other issue.