Audio ticks and video/audio out-of-sync when using MXF files with pre-charge

[Henk Demper]

Summary of the bug:
When we convert a MXF file that contains pre-charge frames (Long GOP codec), then the there will be audio ticks at the beginning and/or ending of the resulting output file. As a related problem, the video/audio can be a few frames out-of-sync.

Workflow where this is used:
With our Primestream Production software users make subclips from XDCAM MXF files out of Sony broadcast cameras or recorded with our Ingest software from feeds. These subclips are then (lossless) re-wrapped (with our own tool currently using bmx/libMXF as MXF writer) to MXF files and kept on the storage. Because the source MXF is Long GOP, the MXF subclips will have pre-charge by definition. For parts of our software we then create low-res QuickTime mov or MP4 proxies using ffmpeg command line.

Background on pre-charge in MXF:
As developer of MXF parsers, I believe this has to do because of the way that MXF files stores pre-charge: in MXF files, each frame is stored in a Content Package (CP), both video and audio. If we have pre-charge for Long GOP codecs, then each pre-charge frame is thus stored in it's own CP, together with some mandatory audio frames. These audio frames are never to be heard, they are just added as 'stuffing' to make complete CP packages to satisfy SMPTE ST377. Depending on how the MXF was created, these 'pre-charge audio frames' might contain silence, some original audio (belonging to the never shown pre-charge frames) or could even be random/garbage. The 'Origin' parameter for both the video and audio tracks in the File Source Package are set to the size of the pre-charge. Note that for instance in QuickTime .mov/MP$ files you have (/need) no audio samples for pre-charge video frames, as they are separately addressed & stored per track.

Likely cause of the bug:
What I think is happening is that the ffmpeg MXF parser is correctly handling the video with the pre-charge frames, but when the calling layer requests audio for t=0, it returns from the beginning of the first CP, which is actually the never-to-be-heard/used pre-charge audio CP. Instead, it should take into consideration the Origin parameter of the File Source Package for the audio tracks and start reading audio from that CP instead. With our MXF files, these pre-charge audio frames are silent, hence you hear a tick when the waveform spikes to the real values after the pre-charge audio frames.
Because the MXF starts to read audio at the wrong place ('too early'), video and audio will be out of sync with a difference of the pre-charge: somewhere between 0 and GOP Size frames, depending on the GOP layout.

Quick look at ffmpeg source code:
I see in libavformat\mxfdec.c that MXFSequence::origin is populated, and forwarded to av_dict_set_int() for material/source tracks. I also see some sample_time manipulations inside mxf_read_seek(), but I don't think the origin value is taken into account for audio here... I'm not the author of this code, and not familiar with the design, but happy to further discuss/dive in as needed...

How to reproduce:

% ffmpeg -i input.mxf -vcodec h264 -acodec copy -report output.mov
ffmpeg version N-95635-gcae7f6658c-tessus
built on: SnapShot from November 5th 2019

I have attached:

1. Input MXF file with pre-charge
2. Excel sheet for this file showing GOP structure with 12 frames pre-charge
3. Screenshot of MXF Analyser showing the 12 frames Origin for the 1st audio track
4. Resulting output file QuickTime .mov that has a tick at start and end

[Henk Demper]

Excel sheet for MXF file showing GOP structure with 12 frames pre-charge

[Henk Demper]

Screenshot of MXF Analyser showing the 12 frames Origin for the 1st audio track

[Henk Demper]

Report file for ffmpeg command line

[Henk Demper]

I also added 5th attachment:

5. Report file for ffmpeg command line

Due to attachment upload size limitations, I have uploaded:

1. Input MXF file with pre-charge
2. Resulting output file QuickTime .mov that has a tick at start and end

to ​https://we.tl/t-IX2TfBaBHA instead, available for 1 week (but can re-upload as needed)

[Tomas Härdin]

Could you upload to a less obnoxious host? Maybe ​http://0x0.st/. I think FFmpeg's FTP still allows anonymous upload, could also be worth a try. Or just dd off the first couple of MiB or so and upload here.

Could you paste the output of mxfdump as well?

This sounds like an OP3a file, despite what the header says, since it's cutting off the start/end of a FilePackage to make the MaterialPackage. Adding full OP3a support is a tall order of course. Setting start_time on the relevant AVStreams might be enough in this case. libavformat/mov.c does something similar for edit lists, but frankly it's a horrible hack that doesn't belong in a demuxer..!

Any patch that attempts to fix this should take care that edit lists are added both to remuxed MOVs and MXFs, and that such remuxed MXFs are marked as OP3a.

[Henk Demper]

Replying to Tjoppen:

Could you upload to a less obnoxious host? Maybe ​http://0x0.st/. I think FFmpeg's FTP still allows anonymous upload, could also be worth a try. Or just dd off the first couple of MiB or so and upload here.

I re-uploaded a zip with all files including input.mxf and output.mov to ​http://0x0.st/zg7l.zip
Good tip for alternative to WeTransfer !

Could you paste the output of mxfdump as well?

My bmx build on Mac doesn't include the mxfdump binary for some reason, likely because we created Universal Binaries (incl. 64 bits) and the mxfdump part failed on that or so...

But I did include an alternative in the .zip and separate attachment: the textual output for input.mxf (input.txt) of a my own MXF Analyser (the one from the PNG screenshot), which I think contains equally usable info...

This sounds like an OP3a file, despite what the header says, since it's cutting off the start/end of a FilePackage to make the MaterialPackage. Adding full OP3a support is a tall order of course. Setting start_time on the relevant AVStreams might be enough in this case. libavformat/mov.c does something similar for edit lists, but frankly it's a horrible hack that doesn't belong in a demuxer..!

Using the Origin field for (only) pre-charge purposes does not make the MXF file OP3a.
Logically (let's say viewable) it is still SingleItem-SinglePackage and categorised as OP1a.
I agree technically it 'misuses' a bit of the editlist methods.
I have sample files from Sony (Japan camera division) also with PreCharge in XDCAM (from their camera's/Servers) labelled as OP1a.

Any patch that attempts to fix this should take care that edit lists are added both to remuxed MOVs and MXFs, and that such remuxed MXFs are marked as OP3a.

As mentioned I think that 'only' the Origin offset for audio must be added/taken into account somewhere when audio samples are fetched, such that for t = 0 in the MXF file the Edit Unit with value 'Origin' is used instead of 0.

[Henk Demper]

MXF Analyser textual output for input.mxf

[Tomas Härdin]

Replying to HenkDemper:

Replying to Tjoppen:

This sounds like an OP3a file, despite what the header says, since it's cutting off the start/end of a FilePackage to make the MaterialPackage. Adding full OP3a support is a tall order of course. Setting start_time on the relevant AVStreams might be enough in this case. libavformat/mov.c does something similar for edit lists, but frankly it's a horrible hack that doesn't belong in a demuxer..!

Using the Origin field for (only) pre-charge purposes does not make the MXF file OP3a.

Yes, it does. To quote S377m:

Single item: The file contains only one item. There is a single material package SourceClip which is the same duration as the top-level file package(s).

The MXF Book backs this up (page 27):

In some applications, it may be desirable to play out only the central contribution portion of our example file. In this case a material package is used that describes only this central portion of the file, and not the color bars, slate, or black portion of the stored content. The operational pattern that describes this functionality is OP3a.

Replying to HenkDemper:

Logically (let's say viewable) it is still SingleItem-SinglePackage and categorised as OP1a.

The file has Duration = 287 in the MP clips, but 299 in the FP clips. Therefore it is not SingleItem per 7.3.1 in S377m.

I have sample files from Sony (Japan camera division) also with PreCharge in XDCAM (from their camera's/Servers) labelled as OP1a.

This wouldn't be the first time a vendor was wrong.

Any patch that attempts to fix this should take care that edit lists are added both to remuxed MOVs and MXFs, and that such remuxed MXFs are marked as OP3a.

As mentioned I think that 'only' the Origin offset for audio must be added/taken into account somewhere when audio samples are fetched, such that for t = 0 in the MXF file the Edit Unit with value 'Origin' is used instead of 0.

No, both audio and video have Origin > 0. It you only cut the audio then the output will be horribly out of sync :)

I don't know how the ffmpeg CLI deals with start times on streams. Can it be told to decode but discard the first Duration number of edit units? What if Origin is very large? I do not want the same kind of hack that libavformat/mov.c does for edit lists, that crap doesn't belong in a demuxer.

[Henk Demper]

Hello Tjoppen,

I have & read the MXF Book, know & implemented ST377, but I think the original design idea about OP1a was to have a single logical clip from start-to-end. Anyhow all places I can find these simple pre-charged files (so only including pre-charge and roll-out) they are marked as OP1a. There is just no other technical way to achieve this and when exchanging files we (broadcast industry) don't want to label them (in this case) as anything other than OP1a, exactly because what you state: there are not so much MXF readers available that (claim) to support anything beyond OP1a...

Examples AMWA AS-03, RDD25:2014 (Harmonic), RDD9:2013 (Sony), EBU ​https://ebu.io/qc/items/0115W, ...

Let's keep it that the early documents are not very conclusive about how to handle the pre-charge case, but the broadcast industry is standardising/labelling them as OP1a.

If we would like to be 100% sure, I might reach out to 'Mr MXF' (Bruce Devlin) or others on the original MXF design team (that came up with the OP's) and ask him/them for his/their advise on the subject... but given that these files are actually frequently used/seen in practice it won't change the use case, so maybe wasted time... In the end the goal should be to let FFmpeg (/libavformat/VLC/...) cope with as many files as possible. Agree ?

No, both audio and video have Origin > 0. It you only cut the audio then the output will be horribly out of sync :)

Well, the video Origin is already used correctly in FFmpeg: the pre-charge video frames (amount: Origin) are used used to produce the first visible frame, but never displayed itself. So it would 'only' need handling for the audio case, which is now out-of-sync with video if there is pre-charge...

What if Origin is very large? I do not want the same kind of hack that libavformat/mov.c does for edit lists, that crap doesn't belong in a demuxer.

We (as in our software) placed this pre-charge logic in the MXF file reader (because it is specific to MXF files and handled differently in for instance QuickTime files as mentioned). We first 'ask' the MXF reader which 'media frame' (Source Package, Index Table) to fetch for 'display frame' (viewable, Material Package) and then ask it to fetch that 'media frame' (for audio that is. for video we need to dig back to fill the decoder with pre-charge video frames). That would seem the proper 2-step way supporting any media (single/multi-item) in a MXF (or any other wrapper) file, with a split responsibility between the wrapper (MXF/QuickTime/...) and upper layer I would say ?

As mentioned, I'm not intimate with how this (timing, single/multiple items in editlist/timeline, media vs display frames, pre-charge, ..) should/is exactly be properly handled in FFmpeg, but my guess would be the fix for the problem on-hand would be in libavformat\mxfdec.c. Happy to study that code more (as a possible next step), but do not want (as you mention) add 'crap in the wrong places'... So open for alternative suggestions...

[Tomas Härdin]

Replying to HenkDemper:

Hello Tjoppen,

I have & read the MXF Book, know & implemented ST377, but I think the original design idea about OP1a was to have a single logical clip from start-to-end. Anyhow all places I can find these simple pre-charged files (so only including pre-charge and roll-out) they are marked as OP1a. There is just no other technical way to achieve this and when exchanging files we (broadcast industry) don't want to label them (in this case) as anything other than OP1a, exactly because what you state: there are not so much MXF readers available that (claim) to support anything beyond OP1a...

This may well be the intent, but it's also not what the spec says :) But, OperationalPattern is just informative, the truth for any particular file is in how its MPs and FPs are laid out.

If we would like to be 100% sure, I might reach out to 'Mr MXF' (Bruce Devlin) or others on the original MXF design team (that came up with the OP's) and ask him/them for his/their advise on the subject... but given that these files are actually frequently used/seen in practice it won't change the use case, so maybe wasted time... In the end the goal should be to let FFmpeg (/libavformat/VLC/...) cope with as many files as possible. Agree ?

FFmpeg can certainly be made to support at least this class of files. At some point it's better to let things like libmelt take over

No, both audio and video have Origin > 0. It you only cut the audio then the output will be horribly out of sync :)

Well, the video Origin is already used correctly in FFmpeg: the pre-charge video frames (amount: Origin) are used used to produce the first visible frame, but never displayed itself. So it would 'only' need handling for the audio case, which is now out-of-sync with video if there is pre-charge...

I didn't notice this when looking at the file, but perhaps ffplay does sync differently from ffmpeg.

What if Origin is very large? I do not want the same kind of hack that libavformat/mov.c does for edit lists, that crap doesn't belong in a demuxer.

We (as in our software) placed this pre-charge logic in the MXF file reader (because it is specific to MXF files and handled differently in for instance QuickTime files as mentioned). We first 'ask' the MXF reader which 'media frame' (Source Package, Index Table) to fetch for 'display frame' (viewable, Material Package) and then ask it to fetch that 'media frame' (for audio that is. for video we need to dig back to fill the decoder with pre-charge video frames). That would seem the proper 2-step way supporting any media (single/multi-item) in a MXF (or any other wrapper) file, with a split responsibility between the wrapper (MXF/QuickTime/...) and upper layer I would say ?

As mentioned, I'm not intimate with how this (timing, single/multiple items in editlist/timeline, media vs display frames, pre-charge, ..) should/is exactly be properly handled in FFmpeg, but my guess would be the fix for the problem on-hand would be in libavformat\mxfdec.c. Happy to study that code more (as a possible next step), but do not want (as you mention) add 'crap in the wrong places'... So open for alternative suggestions...

Well, exporting this information so that the ffmpeg CLI could make use of it for all streams probably wouldn't hurt. But there are some use cases to consider:

• remuxing MXF -> MXF
• remuxing MXF -> MOV
• remuxing MOV -> MXF
• transcoding MXF, including all frames and including EDL information (only MOV and MXF out)
• transcoding MXF, cutting off the start and end, no EDL (any output format)

I don't think there's any other containers that support EDL stuff, but I may be wrong. Which of the bottom two to pick is sort of business logic..

If support for this were added then FFmpeg could claim support for "single clip OP3a" perhaps

[Henk Demper]

Well, exporting this information so that the ffmpeg CLI could make use of it for all streams probably wouldn't hurt. But there are some use cases to consider:

Exactly, we are on the same page.
I ran into the exact same issue when making our lossless (/remuxing) rewrapper tool (which we use to make a subclip or partial file restore from MOV/MXF directly to any MOV/MXF combination): depending where you make the cut in the MXF/MOV source file (which may already be pre-charged itself !) and which source/destination MXF/MOV wrapper you use, you might end up using in that destination file:

• For audible audio frames in MXF/MOV destination: always audible audio frames from source
• For pre-charge (/roll-out) frames (non-audible) in MXF destination either:
  • audible (non-pre-charge/rollout) audio frames from source (if possible)
  • pre-charge/rollout (non-audible) audio frames from source (if possible)
  • create own zero (or even garbage ?) audio samples (always possible, I would say only after above)

I have attached a slide I made at that time to visualise things...
That was from some years ago, but I do recall it was challenging to get it right, such that it was all frame-accurate & in-sync and not taking too little/many frames in the resulting destination file including edge cases like cutting on a B-frame (in encoding/file order) just after an I-frame of an Open GOP (in which case you need to add an addition GOP at the beginning).

Having said that (and thinking agile), having perfect re-wrapping/remux and transcoding support for Long GOP in FFmpeg might be a bit higher goal than the more limited case of this ticket: 'just' transcoding (not remux) to MP4/MOV (not MXF), in which case many of mentioned complexities do not (happen to) apply.

What would be your suggestion to move forward on this ticket (and/or the more generic case which might many parts/files in FFmpeg) ?

[Henk Demper]

Rewrapping between MOV and MXF with Pre-Charge and/or RollOut frames

[ngaullie]

I have analyzed your files and I have another reading :

• every picture sample is encoded to your MOV file, and every audio sample is rewrapped, there is no loss
• the loss of sync (ffmpeg bug) does not seem to be related to the MXF wrapper itself. At the output, the MOV file has a video track with an edit list that shift video of 0.0667s (precisely=2/29.97) - but not the audio track, and this breaks sync.

It you disables B-frames when encoding (-bf 0), there is no precharge signalization in the output, that should fix you problem, can you confirm ?

---

The input MXF file has a precharge value which is consistent amongst the tracks (video/audios/tc), and I agree that, in this case, you can claim its's Op1a because this is the state of the art, broadly implemented, despite absence of clarification in actual texts.
Support for MXF having non-consistent precharges values does not seem necessary in my opinion.
Handling MXF files having a consistent precharge value like yours is no big deal with a first pass analysis and a basic trim clip switch; but of course this is not ideal and could be enhanced.

[Tomas Härdin]

Replying to ngaullie:

I have analyzed your files and I have another reading :

• every picture sample is encoded to your MOV file, and every audio sample is rewrapped, there is no loss
• the loss of sync (ffmpeg bug) does not seem to be related to the MXF wrapper itself. At the output, the MOV file has a video track with an edit list that shift video of 0.0667s (precisely=2/29.97) - but not the audio track, and this breaks sync.

I think I missed the part where it's only different after transcoding/muxing to MOV. output.mov looks in sync to me, but maybe my brain's tolerance for desync is too high? The MOV has higher stream durations, which is definitely shady as hell. A file with a larger Offset would make the effect more noticable. Maybe I can cook something up tomorrow after having some sleep.

The input MXF file has a precharge value which is consistent amongst the tracks (video/audios/tc), and I agree that, in this case, you can claim its's Op1a because this is the state of the art, broadly implemented, despite absence of clarification in actual texts.

The spec is clear, and the MXF book reaffirms it. It's only OP1a if the MP and FP have the same duration. FFmpeg can still implement this limited form of OP3a of course, but let's stick to correct terminology.

Support for MXF having non-consistent precharges values does not seem necessary in my opinion.
Handling MXF files having a consistent precharge value like yours is no big deal with a first pass analysis and a basic trim clip switch; but of course this is not ideal and could be enhanced.

If you get the trim right, then you'll automagically handle different trimming for each stream right. There are storage benefits to getting it right as well.

HenkDemper's use case is equivalent to grabbing the same interval out of the original XDCAM files. Getting that right is not trivial. We can get some funny cases where a malicious file reports as being 1 second long (MP length), but with a 100 hour long heavily compressed FP. A naïve trim implementation will process all that. Hence why I don't like the hacks in mov.c, because this belongs in the ffmpeg CLI and in ffplay, or somewhere between those two and libavformat. But that also gets into libmelt territory.

[Henk Demper]

Based on your both replies I decided to make a perfect test sample to see what is going on in more detail:
I constructed with QuickTime Player 7 Pro (always handy in these cases) an Intra codec file that contains:

1. 13 frames of football match with horn sounds
2. 10 seconds bouncing ball A/V sync test with burned-in timecode starting at 00:00:03:00

All samples in 1080i50 (so 25 fps) to keep things simple with 1920 audio samples per frame without any NTSC audio 1601/1602 sequence misery.
I then rendered above file to XDCAM codec as QuickTime file, which then has no pre-charge (fresh render).
This file was then rewrapped (with timecode matching the burned-in version) to SyncTest.mxf, which I zipped and uploaded to ​http://0x0.st/z66D.zip
I also uploaded in the same .zip an Excel file of the GOP structure of the test source file.

Next I made a 5-second sub-clip out of this file on the second B-frame = 14th (display order) frame in the 2nd GOP (worst case for pre-charge, it will need the last P-frame out of the 1st GOP as well), resulting in a pre-charge aka File Package - Origin in MXF file of 13 frames.
I made the sub-clip both in Sony flavour and generic OP1a (OP3a if you like ;-)) flavour, which turns out not to influence below results, just some slight metadata differences and extra System Metadata Packs & Package Metadata Sets in each Content Package (aka frame).
In an ideal MXF player, the video and audio for this MXF subclip should start exactly at the 14th frame of the source file, start with the bouncing ball at burned-in timecode 00:00:03:00, have no traces of either football or blowing horns and last for exactly 5 seconds.
Note this happens OK in Mac QuickView, Switch Player, Premiere, FCPX, ... etc.
These 2 resulting MXF sub-clips together with matching GOP Analysis Excels have been uploaded as ​http://0x0.st/z66d.zip

Then I transcoded these 2 sub-clips using the earlier mentioned ffmpeg command line

ffmpeg -i input.mxf -vcodec h264 -acodec copy -report output.mov

I uploaded the resulting output.mov files (with log and GOP analysis) to ​http://0x0.st/z66n.zip
I then played these files back and guess what ?
I must admit that my earlier conclusion 'out of sync' was wrong, it was based on bad example files in combination with me believing that the audio started earlier (in the pre-charge, as I heard that) in combination with the video looked fine...
Because what I see is that ffmpeg seems to completely ignore the Origin (pre-charge), both for video and for audio:
The resulting .mov files start with the football match with horn (both in pre-charge area), are perfectly in sync and play for 5 seconds and 13 frames.
Note the output.mov files itself have no pre-charge (as expected).
This is the same issue as you see in VLC playing back the MXF subclips that where used as source for the ffmpeg command line:
VLC starts with the 1st pre-charge frame as well instead of skipping the pre-charge.

Looking at libavformat/mxfdec.c I see:

if (material_track->sequence->origin) {
	av_dict_set_int(&st->metadata, "material_track_origin", material_track->sequence->origin, 0);
}
if (source_track->sequence->origin) {
	av_dict_set_int(&st->metadata, "source_track_origin", source_track->sequence->origin, 0);
}

For these files, material_track->sequence->origin is 0 and source_track->sequence->origin is 13.
Apart from these lines, it appears sequence->origin is not used and I doubt that the av_dict with the string-int mapping is used up-stream ?

So the revised request/issue is to add pre-charge support for video/audio in MXF altogether.
Note that with no doubt there might be some option in ffmpeg command line to skip the first 13 frames, but:

1. the amount of 13 frames is unknown to the (non-technical) user
2. will differ for each subclip (depending where the cuts were made in the GOP)
3. is not the proper fix
4. also still the issue remains in VLC etc

[Tomas Härdin]

So the revised request/issue is to add pre-charge support for video/audio in MXF altogether

This is looking more like a three-part thing:

• set st->start_time to MP SourceClip StartPosition + FP Track Origin (+ FP SourceClip StartPosition)
• have fftools respect st->start_time for more than just subtitles
• have fftools respect st->duration (MP SourceClip Duration) or else crap after the cut gets included in the output

The first one can probably done without too much trouble, but the other two requires digging into ffmpeg.c and ffplay.c, and dealing with any sync issues that inevitably arise..

A hackish way might be to set -ss and -t based on st->start_time and st->duration, but then every stream would have to have identical start_time and duration.

[Carl Eugen Hoyos]

Replying to Tjoppen:

• have fftools respect st->duration

I believe that is not generally wanted as other containers do not enforce their specification as strictly as mxf (pun intended)

[ngaullie]

If I dare add a related use case : when stream-copying, you may expect the precharge/postroll to be reported as a new precharge/postroll in the output wrapper (actual mxf precharge or mov editlist).

[Henk Demper]

@ngaullie

If I dare add a related use case : when stream-copying, you may expect the precharge/postroll to be reported as a new precharge/postroll in the output wrapper (actual mxf precharge or mov editlist)

Correct, when not re-encoding video pre-charge figures would need to be preserved (or adjusted depending if you specify an offset when copying) between any combination of MOV/MXF wrapper. It is not that simple, see the attachment 'Rewrap with PreCharge and/or RollOut.pdf' that is part of this ticket for a bit of clarification (taken from our own non-ffmpeg implementation of a rewrapper that does that).

@Tjoppen and @cehoyos

Do I understand correct from your replies that -given that by no means I have knowledge of the internals of ffplay/fftools etc- to solve this issue is not going to be 'an easy one' and I should adjust my expectations for any quick fix (if ever) accordingly ?

[Tomas Härdin]

Replying to HenkDemper:

@Tjoppen and @cehoyos

Do I understand correct from your replies that -given that by no means I have knowledge of the internals of ffplay/fftools etc- to solve this issue is not going to be 'an easy one' and I should adjust my expectations for any quick fix (if ever) accordingly ?

I don't really know enough to say, but based on Carl's comment I'd say it would at least require a bunch of flags in addition to the potential sync issues I mentioned.

This also ties into dealing with gapless stuff, where we'd have to cut off the end of an audio stream in certain cases, and possibly also carrying over the "actual duration" metadata.

This means we have three classes of containers that need to deal with this:

• MXF
• MOV
• anything that supports gapless playback of codecs like MP3