attachments mjpeg are discovered wrongly as video stream and therefore h264 encoding is broken

[TheoMeister]

Summary of the bug: Encoding with libx264 broken since ​https://github.com/FFmpeg/FFmpeg/commit/511585ce7f7272e5069ef011d6be5f073d268901 - reverting this patch makes it working again as before as the attachments are just copied and not h264 encoded (which is not desired).

How to reproduce: Take the sample ​https://sourceforge.net/projects/matroska/files/test_files/cover_art.mkv and issue the following command:

% ffmpeg -i cover_art.mkv -map 0:0 -map 0:1 -map 0:2 -map 0:3 -map 0:4 -map 0:5 -c:a copy -c:s copy -c:t copy -c:v libx264 -profile:v high -level 4.1 cover_art.mkv

ffmpeg version 2.6.3 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.8 (SUSE Linux)
  configuration: --shlibdir=/usr/lib64 --prefix=/usr --mandir=/usr/share/man --libdir=/usr/lib64 --enable-shared --disable-static --enable-debug --disable-stripping --extra-cflags='-fmessage-length=0 -grecord-gcc-switches -fstack-protector -O2 -Wall -D_FORTIFY_SOURCE=2 -funwind-tables -fasynchronous-unwind-tables -g' --enable-pic --optflags='-fmessage-length=0 -grecord-gcc-switches -fstack-protector -O2 -Wall -D_FORTIFY_SOURCE=2 -funwind-tables -fasynchronous-unwind-tables -g' --enable-gpl --enable-x11grab --enable-version3 --enable-pthreads --datadir=/usr/share/ffmpeg --enable-avfilter --enable-libpulse --enable-libwebp --enable-libvpx --enable-libopus --enable-libmp3lame --enable-libvorbis --enable-libtheora --enable-libspeex --enable-libxvid --enable-libx264 --enable-libx265 --enable-libschroedinger --enable-libgsm --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-postproc --enable-libdc1394 --enable-librtmp --enable-libfreetype --enable-avresample --enable-libtwolame --enable-libvo-aacenc --enable-gnutls --enable-libass --enable-frei0r --enable-libcelt --enable-libcdio --enable-ladspa
  libavutil      54. 20.100 / 54. 20.100
  libavcodec     56. 26.100 / 56. 26.100
  libavformat    56. 25.101 / 56. 25.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 11.102 /  5. 11.102
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, matroska,webm, from 'cover_art.mkv':
  Metadata:
    encoder         : libebml2 v0.12.1 + libmatroska2 v0.11.1
    creation_time   : 2010-09-23 19:01:57
    SAMPLE/TITLE    : Trailer
    COPYRIGHT       : ShowTime
    COPYRIGHT/URL   : http://www.sho.com/
    track           : 5
    DATE_RELEASE    : 2010
    SAMPLE/PART_NUMBER: 0
    TITLE           : Dexter Season 5 trailer
    ORIGINAL/URL    : http://www.youtube.com/watch?v=CUbCMbW-BRE
  Duration: 00:02:36.00, start: 0.000000, bitrate: 2024 kb/s
    Stream #0:0: Audio: aac (LC), 44100 Hz, stereo, fltp (default)
    Stream #0:1: Video: h264 (High), yuv420p, 1272x720 [SAR 1:1 DAR 53:30], 24 fps, 24 tbr, 1k tbn, 48 tbc (default)
    Stream #0:2: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown), 600x882 [SAR 300:300 DAR 100:147], 90k tbr, 90k tbn, 90k tbc
    Metadata:
      filename        : cover.jpg
      mimetype        : image/jpeg
    Stream #0:3: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown), 120x176 [SAR 300:300 DAR 15:22], 90k tbr, 90k tbn, 90k tbc
    Metadata:
      filename        : small_cover.jpg
      mimetype        : image/jpeg
    Stream #0:4: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown), 1067x600 [SAR 96:96 DAR 1067:600], 90k tbr, 90k tbn, 90k tbc
    Metadata:
      filename        : cover_land.jpg
      mimetype        : image/jpeg
    Stream #0:5: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown), 213x120 [SAR 96:96 DAR 71:40], 90k tbr, 90k tbn, 90k tbc
    Metadata:
      filename        : small_cover_land.jpg
      mimetype        : image/jpeg
No pixel format specified, yuvj444p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.                                                                                                                    
    Last message repeated 3 times
[libx264 @ 0x137bbe0] using SAR=1/1
[libx264 @ 0x137bbe0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2
[libx264 @ 0x137bbe0] profile High, level 4.1
[libx264 @ 0x137bbe0] 264 - core 142 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=24 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
x264 [error]: high profile doesn't support 4:4:4
[libx264 @ 0x1373060] Error setting profile high.
[libx264 @ 0x1373060] Possible profiles: baseline main high high10 high422 high444
Output #0, matroska, to '/tmp/cover_art1.mkv':
  Metadata:
    encoder         : libebml2 v0.12.1 + libmatroska2 v0.11.1
    ORIGINAL/URL    : http://www.youtube.com/watch?v=CUbCMbW-BRE
    SAMPLE/TITLE    : Trailer
    COPYRIGHT       : ShowTime
    COPYRIGHT/URL   : http://www.sho.com/
    track           : 5
    DATE_RELEASE    : 2010
    SAMPLE/PART_NUMBER: 0
    TITLE           : Dexter Season 5 trailer
    Stream #0:0: Audio: aac, 44100 Hz, stereo (default)
    Stream #0:1: Video: h264 (libx264), yuv420p, 1272x720 [SAR 1:1 DAR 53:30], q=-1--1, 24 fps, 24 tbn, 24 tbc (default)
    Metadata:
      encoder         : Lavc56.26.100 libx264
    Stream #0:2: Video: h264, none, q=2-31, 128 kb/s, SAR 300:300 DAR 0:0, 90k fps
    Metadata:
      filename        : cover.jpg
      mimetype        : image/jpeg
      encoder         : Lavc56.26.100 libx264
    Stream #0:3: Video: h264, none, q=2-31, 128 kb/s, SAR 300:300 DAR 0:0, 90k fps
    Metadata:
      filename        : small_cover.jpg
      mimetype        : image/jpeg
      encoder         : Lavc56.26.100 libx264
    Stream #0:4: Video: h264, none, q=2-31, 128 kb/s, SAR 96:96 DAR 0:0, 90k fps
    Metadata:
      filename        : cover_land.jpg
      mimetype        : image/jpeg
      encoder         : Lavc56.26.100 libx264
    Stream #0:5: Video: h264, none, q=2-31, 128 kb/s, SAR 96:96 DAR 0:0, 90k fps
    Metadata:
      filename        : small_cover_land.jpg
      mimetype        : image/jpeg
      encoder         : Lavc56.26.100 libx264
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (h264 (native) -> h264 (libx264))
  Stream #0:2 -> #0:2 (mjpeg (native) -> h264 (libx264))
  Stream #0:3 -> #0:3 (mjpeg (native) -> h264 (libx264))
  Stream #0:4 -> #0:4 (mjpeg (native) -> h264 (libx264))
  Stream #0:5 -> #0:5 (mjpeg (native) -> h264 (libx264))
Error while opening encoder for output stream #0:2 - maybe incorrect parameters such as bit_rate, rate, width or height

[Hendrik]

Exporting Cover Art as pseudo video streams is the correct way to export them. They have appropriate metadata set to detect that they are single-frame pseudo streams.

They should not be used for transcoding automatically AFAIK, but if you force them to be used with -map, then thats your own doing.

[TheoMeister]

mediainfo detects the jpegs correctly as attachment consisting of cover.jpg / small_cover.jpg / cover_land.jpg / small_cover_land.jpg whereas ffmpeg detects them as video stream which seams to be wrong to me, because it's not a video but a image attachment.

Anyway: I want ffmpeg to encode video stream 0:1 and copy all the other streams. How can I tell ffmpeg to do so all at the same time? Or maybe I want to ignore some existing streams completely? I don't know of any other option than -map to select streams.

[Carl Eugen Hoyos]

For future tickets: Please remember to test current FFmpeg git head before reporting issues here.

[Michael Niedermayer]

Replying to TheoMeister:

Anyway: I want ffmpeg to encode video stream 0:1 and copy all the other streams. How can I tell ffmpeg to do so all at the same time? Or maybe I want to ignore some existing streams completely? I don't know of any other option than -map to select streams.

./ffmpeg -i in.mkv -map 0:0 -map 0:1 -map 0:2 -map 0:3 -map 0:4 -map 0:5 -c:a copy -c:s copy -c:t copy -c:v copy -c:v:0 libx264 -profile:v:0 high -level 4.1 out.mkv

Does this resolve the issue ?

[Michael Niedermayer]

also heres a shorter command line that should do the same

./ffmpeg -i in.mkv -map 0 -c:a copy -c:s copy -c:t copy -c:v copy -c:v:0 libx264 -profile:v:0 high -level 4.1 out.mkv

[TheoMeister]

ffmpeg doesn't try to encode it anymore - anyway, the attachments are not copied but changed to video streams.

ffmpeg breaks the existing attachments because they are not detected as attachments but as video streams, and therefore they are handled as if they would be video streams. During copy(!!), the attachments are changed(!!) to video streams.

mediainfo before using ffmpeg (original file):

General
Unique ID : 9206505738904495848916047746595705552 (0x6ED1BC737C3FC4CE5492CAFFFC252D0)
Complete name : cover_art.mkv
Format : Matroska
Format version : Version 4 / Version 2
File size : 2.15 MiB
Duration : 10s 425ms
Overall bit rate : 1 734 Kbps
Movie name : Dexter Season 5 trailer
Writing application : Lavf56.25.101
Writing library : Lavf56.25.101
ORIGINAL/URL : ​http://www.youtube.com/watch?v=CUbCMbW-BRE
SAMPLE/TITLE : Trailer
COPYRIGHT : ShowTime
COPYRIGHT/URL : ​http://www.sho.com/
PART_NUMBER : 5
DATE_RELEASE : 2010
SAMPLE/PART_NUMBER : 0
FILENAME : small_cover_land.jpg
MIMETYPE : image/jpeg
Attachements : cover.jpg / small_cover.jpg / cover_land.jpg / small_cover_land.jpg
...

mediainfo after using ffmpeg according your suggestions:

Video #2
ID : 3
Format : V_MJPEG
Codec ID : V_MJPEG
Duration : 10s 424ms
Width : 600 pixels
Height : 882 pixels
Display aspect ratio : 0.680
Frame rate mode : Constant
Frame rate : 90 000.898 fps
Default : No
Forced : No
FILENAME : cover.jpg
MIMETYPE : image/jpeg

Video #3
ID : 4
Format : V_MJPEG
Codec ID : V_MJPEG
Duration : 10s 424ms
Width : 120 pixels
Height : 176 pixels
Display aspect ratio : 0.682
Frame rate mode : Constant
Frame rate : 90 000.898 fps
Default : No
Forced : No
FILENAME : small_cover.jpg
MIMETYPE : image/jpeg

Video #4
ID : 5
Format : V_MJPEG
Codec ID : V_MJPEG
Duration : 10s 424ms
Width : 1 067 pixels
Height : 600 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 90 000.898 fps
Default : No
Forced : No
FILENAME : cover_land.jpg
MIMETYPE : image/jpeg

Video #5
ID : 6
Format : V_MJPEG
Codec ID : V_MJPEG
Duration : 10s 424ms
Width : 213 pixels
Height : 120 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 90 000.898 fps
Default : No
Forced : No
FILENAME : small_cover_land.jpg
MIMETYPE : image/jpeg

..

[julian]

this change is also extremely inconvenient to us. we don't want to see covers as video streams, which they are not. can we have at least a flag so ignore these streams?

[Marton Balint]

You can use -map 0:V to use only the real video streams.

So the only valid issue I see here is that when ffmpeg copies the attachemnts, it loses the attached_pic dispositon, right?

[mitar]

Is there some workaround? I cannot believe that after all these years, running:

ffmpeg -i input.ogg -map 0 -c copy output.ogg

does not produce a copy of the input if the file contains covers (tested with both MJPEG and PNG covers). I have tried to run this on some ogg files to fix minor muxing issues, but then it failed on covers. Can I change this command line to get a full copy of everything (metadata, streams, etc.)?

[mitar]

​Example:

ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, ogg, from 'KR369.ogg':
  Duration: 01:32:53.80, start: 0.000000, bitrate: 164 kb/s
    Stream #0:0: Audio: vorbis, 44100 Hz, stereo, fltp, 152 kb/s
    Metadata:
      ALBUM           : Küchenradio.org
      ENCODED-BY      : auphonic.com
      ARTIST          : Philip Banse
      TITLE           : KR369 Hotel Berlin
      PUBLISHER       : Küchenstud.io
      URL             : http://kuechenstud.io/
      DATE            : 2014
      GENRE           : Podcast
      RIGHTS-DATE     : 2014
      RIGHTS          : 2014 CC BY SA
      LICENSE         : http://creativecommons.org/licenses/by-sa/3.0/de/
      RIGHTS-URI      : http://creativecommons.org/licenses/by-sa/3.0/de/
      ENCODED_BY      : auphonic.com
      comment         : DocPhil, Cindy, Frau Katja und Onkel Andi zu Besuch bei Susanne DeOcampo-Herrmann, der Chefköchin im Hotel Berlin. Wir waren lange nicht mehr zusammen und hatten wie gewohnt alle auf einmal viele Fragen zur: Küchenplanung, Hotelgeschichte, Filetier-Tech
    Stream #0:1: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 500x500 [SAR 1:1 DAR 1:1], 90k tbr, 90k tbn, 90k tbc (attached pic)
    Metadata:
      comment         : Cover (front)
[ogg @ 0x560a7d33be80] Unsupported codec id in stream 0
Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
[ogg @ 0x560a7d33be80] Unsupported codec id in stream 0
Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
    Last message repeated 1 times

[winterlint_]

Still an issue. Reproduce:

# Fetch
yt-dlp --embed-info-json --embed-thumbnail --merge-output-format mkv --output 'in.%(ext)s' https://youtu.be/C0DPdy98e4c
# Mux
ffmpeg -i in.mkv -map 0 -c copy out.mkv

--- $ mediainfo in.mkv
+++ $ mediainfo out.mkv
 General
 Complete name                            : in.mkv
 Format                                   : Matroska
-Attachments                              : info.json / cover.jpg
+Attachments                              : info.json
 
-Video
+Video #1
 ID                                       : 1
 Format                                   : AV1

+Video #2
+ID                                       : 3
+Format                                   : V_MJPEG
+Matrix coefficients                      : BT.470 System B/G
+FILENAME                                 : cover.jpg
+MIMETYPE                                 : image/jpeg

 Audio
 ID                                       : 2
 Format                                   : Opus

This breaks MKV as an archival format. Issue is caused by ffmpeg inappropriately treating an attachment with mimetype image/jpeg as a video track.

$ ffmpeg -hide_banner -i in.mkv
Input #0, matroska,webm, from 'in.mkv':
  Metadata:
    MAJOR_BRAND     : dash
    MINOR_VERSION   : 0
    COMPATIBLE_BRANDS: iso6av01mp41
    ENCODER         : Lavf60.16.100
  Duration: 00:00:17.59, start: 0.000000, bitrate: 167 kb/s
  Stream #0:0: Video: av1 (Main), yuv420p(tv, bt709), 640x480, 25 fps, 25 tbr, 1k tbn (default)
    Metadata:
      HANDLER_NAME    : ISO Media file produced by Google Inc.
      VENDOR_ID       : [0][0][0][0]
      DURATION        : 00:00:16.600000000
  Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
    Metadata:
      DURATION        : 00:00:17.588000000
  Stream #0:2: Attachment: none
    Metadata:
      filename        : info.json
      mimetype        : application/json
  Stream #0:3: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 480x360 [SAR 1:1 DAR 4:3], 90k tbr, 90k tbn (attached pic)
    Metadata:
      filename        : cover.jpg
      mimetype        : image/jpeg

Suggestion, ffmpeg should never convert attachments into tracks without explicit user action, no matter the mimetype.

External page with a person with same issue:
​https://superuser.com/questions/1858881/ffmpeg-copy-attached-cover-image-cover-jpg-of-video-file-during-encode

No solution exists.