the concat filter does not respect the audio delay of the second segment

[redneb]

Suppose that you have two files, each having a video and an audio stream, and also that the second video's audio starts with a delay. If you try to concatenate them with the concat filter, then the resulting video will have sync problems with the audio portion that came from the second file.

Here's a quick way to reproduce this:

ffmpeg -y \
	-filter_complex '
		color=c=black:d=10[v1];
		color=c=white:d=10[v2];
		sine=d=10:f=220[a1];
		sine=d=5,asetpts=PTS+5/TB[a2];
		[v1][a1][v2][a2]concat=a=1
	' \
	test.mkv

When I play the resulting file, the first 10 seconds are just like what I expected (black background with low pitched tone), but the remaining 10 seconds are not. I was expecting that the higher pitched tone would start at 15s but it doesn't. Instead it starts at 10s. So basically, the concat filter simply concatenated the two audio streams without taking into consideration the 5 sec delay of the second one.

I am not sure if this is a bug or the intended behavior, but I think it would be more reasonable if concat inserted silence at the beginning audio streams if necessary, just like it does with the end of such streams.