Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#1924 closed enhancement (invalid)

ASS subtitles are demuxed inconsistently

Reported by: gjdfgh Owned by:
Priority: normal Component: avformat
Version: unspecified Keywords: ass
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

There's a difference how subtitles are demuxed with the "matroska" and the "ass" demuxers.

Consider an even like this in the source file:

Dialogue: 0,0:00:00.00,0:00:01.00,,,0000,0000,0000,,text

"matroska" outputs as packet:

0,0,,,0000,0000,0000,,text

while "ass" outputs:

Dialogue: 0,0:00:00.00,0:00:01.00,,,0000,0000,0000,,text

Matroska produces output that libass' ass_process_chunk() can consume. That means an application using the ffmpeg demuxer interface to read ASS subtitles (possibly for uniformity with the matroska case) has to add extra workarounds. Further, the "ass" output doesn't include the event index, while "matroska" does. (The event index is the first number, called "ReadOrder?" in libass.) Additionally, the "ass" output does not seem to seem to set the packet duration correctly.

The "ass" decoder in libavcodec is indifferent to that, but libass can't handle it. This makes the "ass" demuxer useless. Please fix.

Attachments (1)

fffail.ass (1.1 KB) - added by gjdfgh 5 years ago.
failing example

Download all attachments as: .zip

Change History (13)

comment:1 in reply to: ↑ description Changed 5 years ago by gjdfgh

  • Type changed from defect to enhancement

Nevermind the comments about the "matroska" demuxer. That was accidentally mplayer's internal mkv demuxer. Actually "ass" and "matroska" behave the same. PEBKAC.

It's still not nice that libavformat outputs packets incompatible to libass' ass_process_chunk() function. Changing to an enhancement request,

comment:2 follow-up: Changed 5 years ago by gjdfgh

OK, found out that libass provides a separate function ass_process_data(), which can takes "complete" ASS events as libavformat outputs them. While it's a bit annoying that you can't get the length of the subtitle event in the AVPacket duration/convergence_duration fields, these are not really needed when using libass (or if you need them, use the libavcodec ass decoder).

You actually can't get the Matroska ReadOrder? field, which is used yb libass for something. I don't know if this is a bug or a caveat, or if it's actually handled in some way.

Can't close this myself as invalid. Sorry for the noise.

comment:3 in reply to: ↑ 2 Changed 5 years ago by cehoyos

  • Keywords ass added
  • Resolution set to invalid
  • Status changed from new to closed

Replying to gjdfgh:

Can't close this myself as invalid. Sorry for the noise.

Don't worry!

comment:4 Changed 5 years ago by Cigaes

Please do not apologize, your input is very valuable. We have been considering changing the format of the demuxed ASS packets for some time, the main concern being doing so without compatibility break. Another concern was the status of the ReadOrder? field, and this tickets shows that it is useful and should be kept.

comment:5 Changed 5 years ago by gjdfgh

About the packet format: it's certainly a bit odd that SRT subs don't have time codes inline, while ASS subs do. Note that there's also the "layer" field before the time codes (the first number). The Matroska-internal format eliminates the redundant time codes, and adds the ReadOrder? field. On the other hand, the current ffmpeg packet format is a bit easier to understand and more intuitive at first.

Here's a suggestion: at least, ASS packets could perhaps have the duration field set, and the ReadOrder? (matroska files) / event index (ASS files) could be separately exported. (Maybe an additional packet field?)

I have no idea what the ReadOrder? field is about. Looking at libass, it looks like a fully demuxed ASS file might behave differently from muxed subs that are fed per-packet as they are demuxed from an interleaved stream.

Aside from using the ReadOrder? field to eliminate duplicate events (i.e. packets that have been fed to libass more than once, as all events are stored permanently), this is the only use of the ReadOrder? field I could find: http://repo.or.cz/w/libass.git/blob/c207000c7cdde335679b53a8cffc22bef61ffa45:/libass/ass_render.c#l2274

Also, note that ass_process_chunk() might be faster than ass_process_data(), as the latter has to search for plain text rather than comparing the ReadOrder? field to eliminate duplicate events. ReadOrder? acts as unique packet identifier here.

comment:6 Changed 5 years ago by plorkyeran

I have no idea what the ReadOrder?? field is about. Looking at libass, it looks like a fully demuxed ASS file might behave differently from muxed subs that are fed per-packet as they are demuxed from an interleaved stream.

ASS subtitles don't have to be sorted by time, but an interleaved stream obviously needs to be. Since the order of the lines effects the rendering, the original index of the line is stored in the ReadOrder? field so that the renderer can reconstruct the unsorted order.

Changed 5 years ago by gjdfgh

failing example

comment:7 Changed 5 years ago by gjdfgh

I attached a failing example.

There are two green rectangles visible: the first one closer to the upper/left corner, and the second one right/below that. The first one should be less transparent than the second. If the second does not appear transparent, whatever you're using is buggy.

Mux with mkvmerge -o test.mkv arbitrary_input.mkv fffail.ass

mplayer2 is good enough to test this with its native mkv demuxer. mplayer proper doesn't seem to handle transparency correctly?

Sorry about the filename.

comment:8 Changed 5 years ago by reimar

(Sorry, this comment is all off-topic)

mplayer proper doesn't seem to handle transparency correctly?

Congratulations you found a recently introduced bug in vo_gl :-)
Fixed in r35539, using -vf ass or a vo different from gl is a workaround.

comment:9 Changed 5 years ago by reimar

I do not think there is a bug in the ASS demuxer.
After enabling it to use subtitle-only demuxers in r35541, MPlayer displays the subtitles the same way no matter whether you specify the subtitle normally, play it via the muxed stream or play it though a separate FFmpeg demuxer.
To use MPlayer with the FFmpeg demuxer use this command-line:
mplayer somevideo -subfile fffail.ass -ass -sid 0
So please provide the exact steps to reproduce your issue.

comment:10 Changed 5 years ago by gjdfgh

This is not an issue with the ASS demuxer, but an issue with the ffmpeg ASS packet format. It loses information. You can't distinguish multiple identical events from each other. This happens always when getting ASS subtitles from a ffmpeg demuxer.

It should happen when seeking back, so that the same subtitle events are demuxed and displayed again. This assumes that mplayer doesn't remove all events from to the ASS_Track on seeking.

Turns out it does.

Look at mplayer commit 31293, where this was explicitly "fixed", in exchange for a worse user experience: you lose the nice behavior that subtitles will display immediately if you seek back to somewhere you already have watched, but it also compensates for ffmpeg's not so great packet format.

So you can't demonstrate this issue with mplayer. In fact, ffmpeg pretty much forces you to use ass_flush_events() on seeking to delete all existing subtitle events, because otherwise subtitle events would accumulate.

(Note: depending on how the file is muxed, subtitles can appear immediately anywaqy when seeking back. Just increase the duration of the subtitle events until it breaks.)

comment:11 Changed 5 years ago by gjdfgh

PS: I would not agree that this means that the bug is not really a bug, but a "caveat". What really happened is that aurel noticed his mistake (or something like this), and made the mplayer user experience worse by adding the ass_flush_events() call on seeking.

comment:12 Changed 5 years ago by reimar

I would consider this a feature request to allow handling seeking in different, possibly better ways.
I don't really agree that not flushing is better: It results in random behaviour after seeking, entirely dependent on what parts of the video you watched before.
The fully working way to support this would be to index the file and mark "key frames", i.e. positions where you can start playback exactly because none of the subtitles before that point influence the rendering of those after.
This is the same way we handle audio and video.
On second thought, I think I kind of agree it is a bug that we do not do it like this.
The ReadOrder? method on the other hand to me seems like a really messy hack that doesn't work properly half of the time and due to its random behaviour gives a rather bad experience.
I admit it would be easier to add than 100% perfect seeking support though.

Note: See TracTickets for help on using tickets.