Non-ASCII chars in -passlogfile on Windows

[Thierry Lelegard]

This report describes a problem with ffmpeg on Windows with the -passlogfile option using non-ASCII characters in the file path.

The problem was originally presented in the FFmpeg-user mailing list as seen at ​http://ffmpeg-users.933282.n4.nabble.com/Weird-characters-in-created-passlogfile-on-Windows-td4661845.html

I have not tested it yet on Linux but a reply in the mailing list suggests that the problem does not exist on FreeBSD. Given the nature of the problem, it is probably Windows-specific.

Consider a file path containing non-ASCII characters like è (e grave) in a file or directory name. There are (at least) three different use cases for a file name: input file, output file, pass log file.

In the first two cases, the non-ASCII characters in file names are processed correctly. But in the case of the pass log file, the file name is transformed into some weird sequence (typical untranslated UTF-8 sequence). The strange thing is that ffmpeg creates two files, the pass log file and the output file, but processes the characters in the corresponding file names quite differently. The way the two files are created must be completely different.

See the following example. The host system is Windows 7 64-bit with some Unix-like command line tools.

> ls -l x*
-rw-rw-rw-   1 user     group     9400000 Oct 16 17:08 xè.ts

> ffmpeg -i xè.ts -pass 1 -passlogfile xè xè.mpg
ffmpeg version N-57176-g64b3aaf Copyright (c) 2000-2013 the FFmpeg developers
  built on Oct 15 2013 21:47:01 with gcc 4.8.1 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 46.101 / 52. 46.101
  libavcodec     55. 37.100 / 55. 37.100
  libavformat    55. 19.102 / 55. 19.102
  libavdevice    55.  4.100 / 55.  4.100
  libavfilter     3. 88.101 /  3. 88.101
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.103 /  0. 17.103
  libpostproc    52.  3.100 / 52.  3.100
[h264 @ 000000000249e820] non-existing PPS referenced
[h264 @ 000000000249e820] non-existing PPS 0 referenced
[h264 @ 000000000249e820] decode_slice_header error
[h264 @ 000000000249e820] no frame!
[h264 @ 000000000249e820] non-existing PPS referenced
[h264 @ 000000000249e820] non-existing PPS 0 referenced
[h264 @ 000000000249e820] decode_slice_header error
[h264 @ 000000000249e820] no frame!
[h264 @ 000000000249e820] non-existing PPS referenced
[h264 @ 000000000249e820] non-existing PPS 0 referenced
[h264 @ 000000000249e820] decode_slice_header error
[h264 @ 000000000249e820] no frame!
[h264 @ 000000000249e820] non-existing PPS referenced
[h264 @ 000000000249e820] non-existing PPS 0 referenced
[h264 @ 000000000249e820] decode_slice_header error
[h264 @ 000000000249e820] no frame!
[h264 @ 000000000249e820] mmco: unref short failure
    Last message repeated 2 times
[mpegts @ 00000000024aebe0] PES packet size mismatch
    Last message repeated 2 times
Input #0, mpegts, from 'xè.ts':
  Duration: 00:00:37.04, start: 42851.202589, bitrate: 2030 kb/s
  Program 4006
    Metadata:
      service_name    : ARTE
      service_provider: Harmonic
    Stream #0:0[0x424]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25
 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x425](fra): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, stereo, s16p, 128 kb/s
    Stream #0:2[0x426](eng): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, stereo, s16p, 128 kb/s
    Stream #0:3[0x427](deu): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, mono, s16p, 64 kb/s
    Stream #0:4[0x42b](qad): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, mono, s16p, 64 kb/s (visual impaired)
    Stream #0:5[0x42c](fra): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)
[mpeg @ 00000000048e9980] VBV buffer size not set, muxing may fail
Output #0, mpeg, to 'xè.mpg':
  Metadata:
    encoder         : Lavf55.19.102
    Stream #0:0: Video: mpeg1video, yuv420p, 720x576 [SAR 16:11 DAR 20:11], q=2-31, pass 1, 200 kb/s, 90k tbn, 25 tbc
    Stream #0:1(fra): Audio: mp2, 48000 Hz, stereo, s16, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> mpeg1video)
  Stream #0:1 -> #0:1 (mp2 -> mp2)
Press [q] to stop, [?] for help
[h264 @ 0000000004bb2280] reference picture missing during reorder
    Last message repeated 1 times
[h264 @ 0000000004bb2280] Missing reference picture, default is 0
[h264 @ 0000000004bb2280] decode_slice_header error
[h264 @ 00000000048cf8a0] mmco: unref short failure
[h264 @ 0000000004bb2280] mmco: unref short failure
[mpegts @ 00000000024aebe0] PES packet size mismatch0:34.60 bitrate= 451.3kbits/s dup=18 drop=0
[h264 @ 00000000048d01c0] error while decoding MB 22 28, bytestream (-5).8kbits/s dup=18 drop=0
[mp2 @ 000000000249ec80] incomplete frame
[h264 @ 00000000048d01c0] concealing 387 DC, 387 AC, 387 MV errors in B frame
Error while decoding stream #0:1: Invalid data found when processing input
frame=  923 fps=186 q=31.0 Lsize=    2046kB time=00:00:36.88 bitrate= 454.5kbits/s dup=20 drop=0
video:1455kB audio:571kB subtitle:0 global headers:0kB muxing overhead 0.990720%

> ls -l x*
-rw-rw-rw-   1 user     group      128580 Oct 16 17:15 xA"-0.log
-rw-rw-rw-   1 user     group     2095104 Oct 16 17:15 xè.mpg
-rw-rw-rw-   1 user     group     9400000 Oct 16 17:08 xè.ts

The command contains "-passlogfile xè xè.mpg" but the "xè" in the two created files resulted in two different encodings.

Of course, Windows is subject to the old "character code page" issues. But this cannot be the sole explanation. FFmpeg seems to correctly interpret the command line arguments since it successfully opens the input file and creates the output file with the right encoding. There is no rational reason that the arguments "-passlogfile xè" would not be read the same way. However, when it comes to creating the log file, the name "xè" is specifically transformed into "xA"". When creating the output file "xè.mpg", such name transformation does not occur.

The problem is that the specified file name on the command line does NOT correspond to the actual file name in the file system. One may say that we don't care since this is a temporary log file.

But there is a real functional problem when the non-ASCII character is in the name of a directory. In that case, ffmpeg transforms the directory name and the resulting path is invalid since the transformed path is a non-existent directory. And ffmpeg fails...

See this example:

> mkdir xèdir

> ls -ld x*
-rw-rw-rw-   1 user     group     9400000 Oct 16 17:08 xè.ts
drwxrwxrwx   1 user     group           0 Oct 16 17:15 xèdir

> ffmpeg -i xè.ts -pass 1 -passlogfile xèdir\log xè.mpg
ffmpeg version N-57176-g64b3aaf Copyright (c) 2000-2013 the FFmpeg developers
  built on Oct 15 2013 21:47:01 with gcc 4.8.1 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 46.101 / 52. 46.101
  libavcodec     55. 37.100 / 55. 37.100
  libavformat    55. 19.102 / 55. 19.102
  libavdevice    55.  4.100 / 55.  4.100
  libavfilter     3. 88.101 /  3. 88.101
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.103 /  0. 17.103
  libpostproc    52.  3.100 / 52.  3.100
[h264 @ 000000000031aa40] non-existing PPS referenced
[h264 @ 000000000031aa40] non-existing PPS 0 referenced
[h264 @ 000000000031aa40] decode_slice_header error
[h264 @ 000000000031aa40] no frame!
[h264 @ 000000000031aa40] non-existing PPS referenced
[h264 @ 000000000031aa40] non-existing PPS 0 referenced
[h264 @ 000000000031aa40] decode_slice_header error
[h264 @ 000000000031aa40] no frame!
[h264 @ 000000000031aa40] non-existing PPS referenced
[h264 @ 000000000031aa40] non-existing PPS 0 referenced
[h264 @ 000000000031aa40] decode_slice_header error
[h264 @ 000000000031aa40] no frame!
[h264 @ 000000000031aa40] non-existing PPS referenced
[h264 @ 000000000031aa40] non-existing PPS 0 referenced
[h264 @ 000000000031aa40] decode_slice_header error
[h264 @ 000000000031aa40] no frame!
[h264 @ 000000000031aa40] mmco: unref short failure
    Last message repeated 2 times
[mpegts @ 000000000031a4a0] PES packet size mismatch
    Last message repeated 2 times
Input #0, mpegts, from 'xè.ts':
  Duration: 00:00:37.04, start: 42851.202589, bitrate: 2030 kb/s
  Program 4006
    Metadata:
      service_name    : ARTE
      service_provider: Harmonic
    Stream #0:0[0x424]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25
 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x425](fra): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, stereo, s16p, 128 kb/s
    Stream #0:2[0x426](eng): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, stereo, s16p, 128 kb/s
    Stream #0:3[0x427](deu): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, mono, s16p, 64 kb/s
    Stream #0:4[0x42b](qad): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, mono, s16p, 64 kb/s (visual impaired)
    Stream #0:5[0x42c](fra): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)
Cannot write log file 'xèdir\log-0.log' for pass-1 encoding: No such file or directory

>

In this case, ffmpeg fails since the log file cannot be created.

Obvious workarounds exist, including using the old DOS 8.3 "short path name" for the directory parts which contain non-ASCII characters.

Example:

> dir/x x*

2013-10-16  17:08         9 400 000 X2691~1.TS   xè.ts
2013-10-16  17:15    <DIR>          XDIR~1       xèdir

> ffmpeg -i xè.ts -pass 1 -passlogfile xdir~1\log xè.mpg
ffmpeg version N-57176-g64b3aaf Copyright (c) 2000-2013 the FFmpeg developers
  built on Oct 15 2013 21:47:01 with gcc 4.8.1 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 46.101 / 52. 46.101
  libavcodec     55. 37.100 / 55. 37.100
  libavformat    55. 19.102 / 55. 19.102
  libavdevice    55.  4.100 / 55.  4.100
  libavfilter     3. 88.101 /  3. 88.101
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.103 /  0. 17.103
  libpostproc    52.  3.100 / 52.  3.100
[h264 @ 00000000003aaa40] non-existing PPS referenced
[h264 @ 00000000003aaa40] non-existing PPS 0 referenced
[h264 @ 00000000003aaa40] decode_slice_header error
[h264 @ 00000000003aaa40] no frame!
[h264 @ 00000000003aaa40] non-existing PPS referenced
[h264 @ 00000000003aaa40] non-existing PPS 0 referenced
[h264 @ 00000000003aaa40] decode_slice_header error
[h264 @ 00000000003aaa40] no frame!
[h264 @ 00000000003aaa40] non-existing PPS referenced
[h264 @ 00000000003aaa40] non-existing PPS 0 referenced
[h264 @ 00000000003aaa40] decode_slice_header error
[h264 @ 00000000003aaa40] no frame!
[h264 @ 00000000003aaa40] non-existing PPS referenced
[h264 @ 00000000003aaa40] non-existing PPS 0 referenced
[h264 @ 00000000003aaa40] decode_slice_header error
[h264 @ 00000000003aaa40] no frame!
[h264 @ 00000000003aaa40] mmco: unref short failure
    Last message repeated 2 times
[mpegts @ 00000000003aa4a0] PES packet size mismatch
    Last message repeated 2 times
Input #0, mpegts, from 'xè.ts':
  Duration: 00:00:37.04, start: 42851.202589, bitrate: 2030 kb/s
  Program 4006
    Metadata:
      service_name    : ARTE
      service_provider: Harmonic
    Stream #0:0[0x424]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25
 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x425](fra): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, stereo, s16p, 128 kb/s
    Stream #0:2[0x426](eng): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, stereo, s16p, 128 kb/s
    Stream #0:3[0x427](deu): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, mono, s16p, 64 kb/s
    Stream #0:4[0x42b](qad): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz, mono, s16p, 64 kb/s (visual impaired)
    Stream #0:5[0x42c](fra): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)
[mpeg @ 0000000004b662c0] VBV buffer size not set, muxing may fail
Output #0, mpeg, to 'xè.mpg':
  Metadata:
    encoder         : Lavf55.19.102
    Stream #0:0: Video: mpeg1video, yuv420p, 720x576 [SAR 16:11 DAR 20:11], q=2-31, pass 1, 200 kb/s, 90k tbn, 25 tbc
    Stream #0:1(fra): Audio: mp2, 48000 Hz, stereo, s16, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> mpeg1video)
  Stream #0:1 -> #0:1 (mp2 -> mp2)
Press [q] to stop, [?] for help
[h264 @ 0000000004a074c0] reference picture missing during reorder
    Last message repeated 1 times
[h264 @ 0000000004a074c0] Missing reference picture, default is 0
[h264 @ 0000000004a074c0] decode_slice_header error
[h264 @ 0000000004a07de0] mmco: unref short failure
[h264 @ 0000000004a074c0] mmco: unref short failure
[mpegts @ 00000000003aa4a0] PES packet size mismatch0:35.36 bitrate= 453.2kbits/s dup=18 drop=0
[mp2 @ 000000000249ec40] incomplete frame
Error while decoding stream #0:1: Invalid data found when processing input
[h264 @ 0000000004a08700] error while decoding MB 22 28, bytestream (-5)
[h264 @ 0000000004a08700] concealing 387 DC, 387 AC, 387 MV errors in B frame
frame=  923 fps=432 q=31.0 Lsize=    2046kB time=00:00:36.88 bitrate= 454.5kbits/s dup=20 drop=0
video:1455kB audio:571kB subtitle:0 global headers:0kB muxing overhead 0.990720%

> ls -ld x*
-rw-rw-rw-   1 user     group     2095104 Oct 16 17:40 xè.mpg
-rw-rw-rw-   1 user     group     9400000 Oct 16 17:08 xè.ts
drwxrwxrwx   1 user     group           0 Oct 16 17:40 xèdir

> ls -l xèdir
total 126
-rw-rw-rw-   1 user     group      128580 Oct 16 17:40 log-0.log

> 

I understand that this kind of Windows oddity may not be too interesting to FFmpeg developers but it can be quite annoying for Windows users.

As far as I am concerned, I use ffmpeg on Windows mainly through the GUI front-end named QtlMovie. All ffmpeg commands are generated by the front-end and I implemented a specific workaround for this problem. In the specific case of -passlogfile on Windows, I first translate the directory path of the log file with its DOS 8.3 counterpart. But this is just a non-portable workaround, pushing the problem into the ffmpeg user's code.

[Michael Niedermayer]

should be fixed by d7ac887cd62814394b4e1617154f6d3c3bffa7a3