hls_enc_key does not work as advertised

[Mark Karpeles]

Summary of the bug: hls_enc_key is not properly handled as hex data but is instead truncated and not used as binary data as it should

How to reproduce:

% ffmpeg -i input -f hls -hls_enc_key 42424242424242424242424242424242 ... output
ffmpeg version 5.1.3

master.key will contain the actual string "4242424242424242" instead of containing "BBBBBBBBBBBBBBBB" (B has ascii value 0x42).

---

The hls_enc_key option has the following description: hex-coded 16 byte key to encrypt the segments

Setting it will change the value of hls->key, however libavformat/hlsenc.c on line 773 a simple memcpy() is performed instead of reading the string, decoding its hex value and setting key appropriately.

• this reads data that might not be allocated, since it will always read 16 bytes even if "key" length is lower than that (unlikely)
• encryption is weaker because the key can only contain alphanumeric characters and will typically only contain 0-9 a-f

This does not break anything as is because ffmpeg will output the key it used in the right file (unless ones ignores the outputted file, generates the key file separately and spends hours wondering why nothing works).

libavformat/hlsenc.c:773 should be modified to decode the passed value as hex-encoded data. I do not know if keeping the existing behavior is needed.