drawtext fails to render non-Latin characters

[KenSharp]

Summary of the bug:
drawtext fails to render non-Latin characters.

How to reproduce:

$ ffmpeg -f lavfi -i nullsrc -to 5 -vf "drawtext=text=ＡＴ 05 ーポン 攢ð:fontcolor=white:fontfile=FreeSans" test.mpg

FreeSans? has the relevant glyphs, but it is the same result no matter which font I use.
Using a UTF-8 encoded text file makes no difference.

$ ffmpeg version 3.3.2-static http://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 5.4.1 (Debian 5.4.1-11) 20170519
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-5 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gray --enable-libass --enable-libfreetype --enable-libfribidi --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libzimg
libavutil      55. 58.100 / 55. 58.100
libavcodec     57. 89.100 / 57. 89.100
libavformat    57. 71.100 / 57. 71.100
libavdevice    57.  6.100 / 57.  6.100
libavfilter     6. 82.100 /  6. 82.100
libswscale      4.  6.100 /  4.  6.100
libswresample   2.  7.100 /  2.  7.100
libpostproc    54.  5.100 / 54.  5.100

A line from the log that I find interesting is:

ffmpeg -v 9 -loglevel 99 -f lavfi -i nullsrc -to 5 -vf "drawtext=text='\xef\xbc\xa1\xef\xbc\xb4 05 \xe3\x83\xbc\xe3\x83\x9d\xe3\x83\xb3 \xc3\xa6\xe2\x80\x9d\xc2\xa2\xc3\xb0':fontcolor=white:fontfile=FreeSans" -report test.mpg

as it appears to be doing a conversion before sending that data to drawtext. The log suggests that this is understood:

Reading option '-vf' ... matched as option 'vf' (set video filters) with argument 'drawtext=text='ＡＴ 05 ーポン 攢ð':fontcolor=white:fontfile=FreeSans'.

I find it interesting because the terminal only shows the UTF characters, not this "conversion" - that only shows up in the log.

I had a skip through the code but I don't really know what I'm looking at. vf_drawtext.c uses GET_UTF8 which seems to convert the text into UTF-32, however the reverse PUT_UTF8 does not seem to be used. Is the text being sent as UTF-32?

Reference: ​https://unix.stackexchange.com/questions/374046/cannot-render-non-ascii-characters-with-ffmpegs-drawtext

[KenSharp]

Verbose log

[KenSharp]

Output example

[cehoyos]

You have to choose another font: TrueType fonts never contain all UTF-8 glyphs.

[KenSharp]

“FreeSans?? has the relevant glyphs, but it is the same result no matter which font I use.”

Right there: like four lines down.

[richardpl]

Try to reproduce with latest master.

[KenSharp]

I am using the latest master. Stop trying to bury your head in the sand!

[KenSharp]

Good work.

[cehoyos]

​https://github.com/byrongibson/fonts/blob/master/truetype/wqy/wqy-microhei.ttc