Dropped video frames from iOS 11 Screen Record Videos lead to an Unsynced video

[Jon]

It seems like ffmpeg does not know how to handle videos that were recorded on iOS 11 through the native screen recording functionality.

We were taking the video stream and copying the second audio stream (from the microphone) and stitch it together so web players would be able to handle the audio portion properly.

We discovered that frames were dropped at the beginning of the converted video, but the audio still began on time. It seems like everything before the 2nd keyframe is dropped, as it can't locate the first keyframe. We believe this occurs due to a negative start timestamp on the video.

How to reproduce:

ffmpeg -i input.mp4 -acodec copy -vcodec copy -map 0:v:0 -map 0:a:1? -f mp4 output.mp4

Output from ffmpeg -i input.mp4

ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.39.2)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.4.2 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --disable-jack --enable-gpl --enable-libmp3lame --enable-libx264 --enable-libxvid --enable-opencl --enable-videotoolbox --disable-lzma
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp41mp42isom
    creation_time   : 2018-03-16T18:56:20.000000Z
  Duration: 00:03:54.07, start: -0.015941, bitrate: 2456 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc), 1080x1920, 2319 kb/s, 23.03 fps, 60 tbr, 600 tbn, 1200 tbc (default)
    Metadata:
      creation_time   : 2018-03-16T18:56:21.000000Z
      handler_name    : Core Media Video
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 2 kb/s (default)
    Metadata:
      creation_time   : 2018-03-16T18:56:21.000000Z
      handler_name    : Core Media Audio
    Stream #0:2(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 103 kb/s (default)
    Metadata:
      creation_time   : 2018-03-16T18:56:21.000000Z
      handler_name    : Core Media Audio

Output of this video has dropped video frames (approximately 10 seconds worth), while keeping the audio. This leads to an unsynced video. Even with flags like -copyts and other fixes we found online, we still ended up with dropped frames at the start of the video, i.e. a black screen.

We tried a fork of ffmpeg called libav, which seems to handle this file properly and doesn’t drop the frames at the start of the output video, between the first and second keyframe.

Output from avconv -i input.mp4

avconv version 12.3, Copyright (c) 2000-2018 the Libav developers
  built on Feb 13 2018 11:14:42 with Apple LLVM version 9.0.0 (clang-900.0.39.2)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp41mp42isom
    creation_time   : 2018-03-16 18:56:20
  Duration: 00:03:56.66, start: 0.062381, bitrate: 2429 kb/s
    Stream #0:0(und): Video: h264 (High) [avc1 / 0x31637661]
      yuvj420p, pc, 1080x1920, 2319 kb/s
      46.92 fps, 600 tbn (default)
    Metadata:
      creation_time   : 2018-03-16 18:56:21
      handler_name    : Core Media Video
    Stream #0:1(und): Audio: aac (LC) [mp4a / 0x6134706D]
      44100 Hz, stereo, fltp, 2 kb/s (default)
    Metadata:
      creation_time   : 2018-03-16 18:56:21
      handler_name    : Core Media Audio
    Stream #0:2(und): Audio: aac (LC) [mp4a / 0x6134706D]
      44100 Hz, stereo, fltp, 103 kb/s (default)
    Metadata:
      creation_time   : 2018-03-16 18:56:21
      handler_name    : Core Media Audio

Notice the start timestamps for the same video is different.

ffmpeg start timestamp: -0.015941
avconv start timestamp: 0.062381

[llogan]

Please test a recent build from the git master branch (see the FFmpeg download page for links to macOS builds if you can't compile) and provide a link to the input file.

[Carl Eugen Hoyos]

Please reopen this ticket if you can add the missing information.