[Windows] Non-ASCII characters in "FFREPORT" may produce garbled filename or fail

[m.feriati]

Summary of the bug
this has been around for a long time, maybe years.
on windows, if the filename set in FFREPORT environment variable contains accented characters, the resulting log won't match the expected filename, although the log contains the correct filename.

ffplay started on 2024-10-11 at 16:22:37
Report written to "00 - hétérogénéisé.log"
Log level: 40
Command line:
ffplay -hide_banner -noborder -i "00 - h\xe9t\xe9rog\xe9n\xe9is\xe9.mp4" -vf "scale=1280:-2,setsar=1"
Initialized direct3d renderer.

also tried different console code pages with no success

chcp 1252
chcp 65001 :: my default console code page
chcp 437

expected behavior: produce 00 - hétérogénéisé.log
actual behavior: produces 00 - hétérogénéisé.log

hint
the target name can be recovered by reading the log filename bytes in code page 1252 and then rewritten as utf8

fyi
this applies to all ffmpeg binaries.

How to reproduce:

> set "FFREPORT=file=00 - hétérogénéisé.log:level=40"
> ffplay -hide_banner -noborder -i "in\00 - hétérogénéisé.mp4" -vf scale=1280:-2,setsar=1

>ffmpeg -version
ffmpeg started on 2024-10-11 at 16:15:29
Report written to "00 - hétérogénéisé.log"
Log level: 40
ffmpeg version 7.1-full_build-www.gyan.dev Copyright (c) 2000-2024 the FFmpeg developers

[MasterQuestionable]

͏    Likely Windows problem.
͏    See also: ​https://trac.ffmpeg.org/ticket/11241#comment:5

͏    Windows is known to be buggy with Unicode support.
͏    I guess this one is probably caused by the shell and OS's handling.
͏    Likely not addressable from FFmpeg.

[Marton Balint]

Replying to MasterQuestionable:

͏    Likely Windows problem.
͏    See also: ​https://trac.ffmpeg.org/ticket/11241#comment:5

͏    Windows is known to be buggy with Unicode support.
͏    I guess this one is probably caused by the shell and OS's handling.
͏    Likely not addressable from FFmpeg.

No, ffmpeg has full unicode filename support even on windows. The problem is likely that the ffreport file is opened using fopen() instead fopen_utf8().

[m.feriati]

Replying to Marton Balint:

Replying to MasterQuestionable:

͏    Likely Windows problem.
͏    See also: ​https://trac.ffmpeg.org/ticket/11241#comment:5

͏    Windows is known to be buggy with Unicode support.
͏    I guess this one is probably caused by the shell and OS's handling.
͏    Likely not addressable from FFmpeg.

No, ffmpeg has full unicode filename support even on windows. The problem is likely that the ffreport file is opened using fopen() instead fopen_utf8().

thanks marton,
this is furthermore confirmed by the content of the generated log file:
Report written to "00 - hétérogénéisé.log"
which means that the ffreport environment variable is properly read by the ffmpeg binary.
ffmpeg just fails when it comes to create the log file with the proper character encoding.

see ​https://github.com/FFmpeg/FFmpeg/blob/db7b4fc89fb18d5ff0a1426bd433c234555a3fff/fftools/opt_common.c#L1210
report_file = fopen(filename.str, "w");
located in method int init_report(const char *env, FILE **file)

[MasterQuestionable]

͏    Would you clarify somewhat... what went wrong exactly?
͏    Which part of the output log had unexpected content?

͏    Is it only the output log's filename wrong?
͏    (I thought it was the filenames in log's content...)

[m.feriati]

Replying to MasterQuestionable:

͏    Would you clarify somewhat... what went wrong exactly?
͏    Which part of the output log had unexpected content?

͏    Is it only the output log's filename wrong?
͏    (I thought it was the filenames in log's content...)

indeed it is only the output log's filename that is encoded in the wrong code page.

[MasterQuestionable]

͏    Then this should be addressable.
͏    And likely just the cause mentioned by Marton.

͏    Note Windows does not necessarily support full-Unicode for the filename. (UTF-16 no surrogate)
͏    See also: ​https://github.com/exiftool/exiftool/issues/253#issuecomment-2063406000

͏    ----

͏    The purpose is to hint the theoretical boundary:
͏    UTF-16 no surrogate cannot represent every possibility of UTF-8.

[m.feriati]

Replying to MasterQuestionable:

͏    Then this should be addressable.
͏    And likely just the cause mentioned by Marton.

͏    Note Windows does not necessarily support full-Unicode for the filename. (UTF-16 no surrogate)
͏    See also: ​https://github.com/exiftool/exiftool/issues/253#issuecomment-2063406000

for sure, it doesn't.
the issue's scope is to create a log file whose name matches an already existing input file.

[MasterQuestionable]

͏    I believe it should also be reproducible specifying the filename in whatsoever manner.
͏    Also nothing specific to "accented". (apparently any non-ASCII would trap)

͏    Worth notice: title length matters.

[MasterQuestionable]

͏    It appears "-report" only supports custom options via "FFREPORT" environment variable:
͏    https://ffmpeg.org/ffmpeg.html#Generic-options
͏    (yes, I don't use it...)

͏    Specifically tried below:
(Windows CMD)
͏    SET "FFREPORT=file=中.txt" & ffmpeg -i "中.mp4"
͏    SET "FFREPORT=file=é.txt" & ffmpeg -i "中.mp4"
͏    SET "FFREPORT=file=А.txt" & ffmpeg -i "中.mp4"
͏    .
͏    “Failed to open report "中.txt": Invalid argument”
͏    Garbled filename for else.
͏    Note "А" is Cyrillic.
͏    "é" is "e" + "́". (not "é")
͏    [ ​https://github.com/MasterInQuestion/Markup/blob/main/AAA.htm ]

[m.feriati]

it is worth to mention that the environment variable is read as an utf8 string

env = getenv_utf8("FFREPORT");

​https://github.com/FFmpeg/FFmpeg/blob/3330b733d3eb912ee60a90a163ef8ee5f44ff4c0/fftools/cmdutils.c#L566

[m.feriati]

hello, any updates to share ?
the fix is provided in the current discussion, is there anyone to apply it ?

[MasterQuestionable]

͏    Didn't notice any apparent patch..?
͏    Try submit a clear one?
͏    ​https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls

[Zhao Zhili <quink@noreply.code.ffmpeg.org>]

In 2a1d5dd7/ffmpeg:

fftools: use fopen_utf8 to open FFREPORT

Should fix #11240