Opened 6 years ago

Last modified 8 months ago

#5375 open enhancement

Can't download a m3u playlist stream that uses AES encryption

Reported by: viv1d-shadows Owned by:
Priority: wish Component: avformat
Version: git-master Keywords: hls
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug: I am trying to download a video using a m3u8 playlist but I get the error saying "Sample-AES encryption is not yet supported; Failed to open segment of playlist 0". Then at the end I get "Error loading first segment: <very long url is here> Invalid data when processing input
How to reproduce:

% ffmpeg -i "https://once.unicornmedia.com/now/od/aut
o/e62f13e1-cab4-4fca-bdbb-eaa84d793f41/523587ee-47d2-403b-a3d0-e5db30f6929c/cim
44emqp001i0fo5daa1yd3o/content.m3u8" -c copy test.ts

ffmpeg version: 3.0
built on: Unknown

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

Change History (20)

comment:1 by Carl Eugen Hoyos, 6 years ago

Keywords: hls added

Please test current FFmpeg git head and provide the failing command line together with the complete, uncut console output to make this a valid ticket.

comment:2 by viv1d-shadows, 6 years ago

$ ffmpeg -i "https://once.unicornmedia.com/now/od/auto/e62f13e1-cab4-4fca-bdbb-eaa84d793f41/523587ee-47d2-403b-a3d0-e5db30f6929c/cim44emqp001i0fo5daa1yd3o/content.m3u8" -c copy test.ts
ffmpeg version 3.0 Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 7.0.2 (clang-700.1.81)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.0 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libxvid --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libspeex --enable-openssl --enable-libvidstab --enable-libx265 --enable-nonfree --enable-vda
  libavutil      55. 17.103 / 55. 17.103
  libavcodec     57. 24.102 / 57. 24.102
  libavformat    57. 25.100 / 57. 25.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 31.100 /  6. 31.100
  libavresample   3.  0.  0 /  3.  0.  0
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
[hls,applehttp @ 0x7fd4da810600] SAMPLE-AES encryption is not supported yet
[hls,applehttp @ 0x7fd4da810600] Failed to open segment of playlist 0
[hls,applehttp @ 0x7fd4da810600] SAMPLE-AES encryption is not supported yet
[hls,applehttp @ 0x7fd4da810600] Failed to open segment of playlist 0

...

[hls,applehttp @ 0x7fd4da810600] SAMPLE-AES encryption is not supported yet
[hls,applehttp @ 0x7fd4da810600] Failed to open segment of playlist 0
[hls,applehttp @ 0x7fd4da810600] SAMPLE-AES encryption is not supported yet
[hls,applehttp @ 0x7fd4da810600] Failed to open segment of playlist 0
[hls,applehttp @ 0x7fd4da810600] Error when loading first segment 'https://api002-phx.unicornmedia.com/now/media/segment/e62f13e1-cab4-4fca-bdbb-eaa84d793f41/523587ee-47d2-403b-a3d0-e5db30f6929c/74ba4d0b-a347-11e4-bfdb-005056837bc7/d4ddcab5-a383-4e99-af17-9e16c7be623b/0/0/270/content.ts?visitguid=6442f8e8-b6d5-4c77-a7b8-9e82789b1ce7&baseguid=d4ddcab5-a383-4e99-af17-9e16c7be623b&streamDuration=3297&umx=cAw7b/V1pNitZ2XV9RnbyA==&startsegmentseconds=0&endsegmentseconds=10'
https://once.unicornmedia.com/now/od/auto/e62f13e1-cab4-4fca-bdbb-eaa84d793f41/523587ee-47d2-403b-a3d0-e5db30f6929c/cim44emqp001i0fo5daa1yd3o/content.m3u8: Invalid data found when processing input
Last edited 6 years ago by Carl Eugen Hoyos (previous) (diff)

comment:3 by Carl Eugen Hoyos, 6 years ago

Component: undeterminedavformat
Priority: normalwish
Status: newopen
Type: defectenhancement
Version: unspecifiedgit-master

For future tickets: Please remember to test current FFmpeg git head before reporting issues.

comment:4 by viv1d-shadows, 6 years ago

I did try to use the current git snapshot, but after I compiled it on my ubuntu based machine I get the error saying "error while loading shared libraries: libx264.so.148" hence why I didn't use the git snapshot because I don't know how to get rid of that error.

Do I need to do anything else to this ticket, other than trying to test the git build?

Last edited 6 years ago by viv1d-shadows (previous) (diff)

in reply to:  4 comment:5 by Carl Eugen Hoyos, 6 years ago

Replying to viv1d-shadows:

I did try to use the current git snapshot, but after I compiled it on my ubuntu based machine I get the error saying "error while loading shared libraries: libx264.so.148"

This error can never occur for a default compilation which should allow to reproduce this issue.

comment:6 by viv1d-shadows, 6 years ago

Ok I will try it again later today and get back to you

in reply to:  6 comment:7 by Carl Eugen Hoyos, 6 years ago

Replying to viv1d-shadows:

Ok I will try it again later today and get back to you

When I wrote For future tickets I meant that you should follow my advice when reporting your next issue.

comment:8 by viv1d-shadows, 6 years ago

Yeah I understand. So I am required to do anything else to this ticket, or just wait until the developers come around and look at this issue?

Cheers

comment:9 by Carl Eugen Hoyos, 6 years ago

Ideally you would fix the issue and send a patch to the development mailing list.

in reply to:  8 comment:10 by James, 6 years ago

Replying to viv1d-shadows:

Yeah I understand. So I am required to do anything else to this ticket, or just wait until the developers come around and look at this issue?

No, the ticket is fine as is. But keep in mind this is a feature request. It being implemented depends on someone wanting to give this a go.

comment:11 by fitz123, 6 years ago

+1 for this feature.
Version 3.1.3 and it's still not here =(

comment:12 by marantz, 6 years ago

sub

comment:13 by kvasa, 5 years ago

Hi, I'm no expert, but isn't this pretty much what is needed? https://github.com/selsta/hlsdl/blob/master/src/hls.c#L292

It is under MIT license and seems to be doing SAMPLE-AES decryption using libavformat and libavcodec. Unfortunately, I do not have the skills to implement this directly into FFmpeg. :(

comment:14 by CoRoNe, 5 years ago

Today the NPO (Dutch Public Broadcast) has started using the SAMPLE-AES encryption which FFmpeg sadly enough still doesn't support:

D:\Binaries>FOR /F "delims=" %A IN ('xidel.exe -s "https://www.npo.nl/nos-journaal/27-10-2017/POW_03375731" -e "let $prid:=extract($url,'.+/(.+)',1) return json(concat('https://start-player.npo.nl/video/',$prid,'/streams?profile=hls&quality=npo&tokenId=',x:request({'data':'https://www.npo.nl/player/'||$prid,'method':'POST'})/json/token))//src"') DO ffmpeg-N-88268-g75bd010_hls-test.exe -v 40 -i "%A"

D:\Binaries>ffmpeg-N-88268-g75bd010_hls-test.exe -v 40 -i "https://nl-ams-p4-am5.cdn.streamgate.nl/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDkzNDU3NjMsImNsaWVudF9pcCI6IjgwLjEwMS40OS41IiwidXJpIjoiXC92b2RcL25wb1wvZnBzXC9URVNUXC9ucG9cL2hsc1wvUE9XXzAzMzc1NzMxIn0.8TcJZK13155360BpwACs3YJepA-TAFkoD4RjcdsSXT8/vod/npo/fps/TEST/npo/hls/POW_03375731/playlist.m3u8"
ffmpeg version N-88268-g75bd010 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.2.0 (GCC)
  configuration: --arch=x86 --target-os=mingw32 --cross-prefix=/cygdrive/m/ffmpeg-windows-build-helpers-master/native_build/windows/ffmpeg_local_builds/sandbox/cross_compilers/mingw-w64-i686/bin/i686-w64-mingw32- --pkg-config=pkg-config --pkg-config-flags=--static --enable-gray --enable-version3 --disable-debug --disable-doc --disable-htmlpages --disable-manpages --disable-podpages --disable-schannel --disable-txtpages --disable-w32threads --disable-ffprobe --disable-hwaccels --disable-muxers --disable-bsfs --disable-filters --enable-gnutls --extra-libs=-lcrypt32 --enable-gpl --extra-cflags='-march=pentium3' --extra-cflags=-O2 --extra-cflags='-mfpmath=sse' --extra-cflags=-msse --enable-static --disable-shared --prefix=/cygdrive/m/ffmpeg-windows-build-helpers-master/native_build/windows/ffmpeg_local_builds/sandbox/cross_compilers/mingw-w64-i686/i686-w64-mingw32
  libavutil      56.  0.100 / 56.  0.100
  libavcodec     58.  1.100 / 58.  1.100
  libavformat    58.  0.102 / 58.  0.102
  libavdevice    58.  0.100 / 58.  0.100
  libavfilter     7.  0.101 /  7.  0.101
  libswscale      5.  0.101 /  5.  0.101
  libswresample   3.  0.100 /  3.  0.100
  libpostproc    55.  0.100 / 55.  0.100
[hls,applehttp @ 02539440] Opening 'https://nl-ams-p4-am5.cdn.streamgate.nl/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDkzNDU3NjMsImNsaWVudF9pcCI6IjgwLjEwMS40OS41IiwidXJpIjoiXC92b2RcL25wb1wvZnBzXC9URVNUXC9ucG9cL2hsc1wvUE9XXzAzMzc1NzMxIn0.8TcJZK13155360BpwACs3YJepA-TAFkoD4RjcdsSXT8/vod/npo/fps/TEST/npo/hls/POW_03375731/a1/a1.m3u8' for reading
[hls,applehttp @ 02539440] Opening 'https://nl-ams-p4-am5.cdn.streamgate.nl/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDkzNDU3NjMsImNsaWVudF9pcCI6IjgwLjEwMS40OS41IiwidXJpIjoiXC92b2RcL25wb1wvZnBzXC9URVNUXC9ucG9cL2hsc1wvUE9XXzAzMzc1NzMxIn0.8TcJZK13155360BpwACs3YJepA-TAFkoD4RjcdsSXT8/vod/npo/fps/TEST/npo/hls/POW_03375731/v2/v2.m3u8' for reading
[hls,applehttp @ 02539440] Opening 'https://nl-ams-p4-am5.cdn.streamgate.nl/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDkzNDU3NjMsImNsaWVudF9pcCI6IjgwLjEwMS40OS41IiwidXJpIjoiXC92b2RcL25wb1wvZnBzXC9URVNUXC9ucG9cL2hsc1wvUE9XXzAzMzc1NzMxIn0.8TcJZK13155360BpwACs3YJepA-TAFkoD4RjcdsSXT8/vod/npo/fps/TEST/npo/hls/POW_03375731/v3/v3.m3u8' for reading
[hls,applehttp @ 02539440] Opening 'https://nl-ams-p4-am5.cdn.streamgate.nl/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDkzNDU3NjMsImNsaWVudF9pcCI6IjgwLjEwMS40OS41IiwidXJpIjoiXC92b2RcL25wb1wvZnBzXC9URVNUXC9ucG9cL2hsc1wvUE9XXzAzMzc1NzMxIn0.8TcJZK13155360BpwACs3YJepA-TAFkoD4RjcdsSXT8/vod/npo/fps/TEST/npo/hls/POW_03375731/v4/v4.m3u8' for reading
[hls,applehttp @ 02539440] Opening 'https://nl-ams-p4-am5.cdn.streamgate.nl/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDkzNDU3NjMsImNsaWVudF9pcCI6IjgwLjEwMS40OS41IiwidXJpIjoiXC92b2RcL25wb1wvZnBzXC9URVNUXC9ucG9cL2hsc1wvUE9XXzAzMzc1NzMxIn0.8TcJZK13155360BpwACs3YJepA-TAFkoD4RjcdsSXT8/vod/npo/fps/TEST/npo/hls/POW_03375731/v5/v5.m3u8' for reading
[hls,applehttp @ 02539440] HLS request for url 'https://nl-ams-p4-am5.cdn.streamgate.nl/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDkzNDU3NjMsImNsaWVudF9pcCI6IjgwLjEwMS40OS41IiwidXJpIjoiXC92b2RcL25wb1wvZnBzXC9URVNUXC9ucG9cL2hsc1wvUE9XXzAzMzc1NzMxIn0.8TcJZK13155360BpwACs3YJepA-TAFkoD4RjcdsSXT8/vod/npo/fps/TEST/npo/hls/POW_03375731/a1/fileSequence.aac', offset 0, playlist 0
[hls,applehttp @ 02539440] SAMPLE-AES encryption is not supported yet
[hls,applehttp @ 02539440] Failed to open segment of playlist 0
[hls,applehttp @ 02539440] HLS request for url 'https://nl-ams-p4-am5.cdn.streamgate.nl/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDkzNDU3NjMsImNsaWVudF9pcCI6IjgwLjEwMS40OS41IiwidXJpIjoiXC92b2RcL25wb1wvZnBzXC9URVNUXC9ucG9cL2hsc1wvUE9XXzAzMzc1NzMxIn0.8TcJZK13155360BpwACs3YJepA-TAFkoD4RjcdsSXT8/vod/npo/fps/TEST/npo/hls/POW_03375731/a1/fileSequence.aac', offset 131089, playlist 0
[hls,applehttp @ 02539440] SAMPLE-AES encryption is not supported yet
[hls,applehttp @ 02539440] Failed to open segment of playlist 0
[...]
[hls,applehttp @ 02539440] HLS request for url 'https://nl-ams-p4-am5.cdn.streamgate.nl/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDkzNDU3NjMsImNsaWVudF9pcCI6IjgwLjEwMS40OS41IiwidXJpIjoiXC92b2RcL25wb1wvZnBzXC9URVNUXC9ucG9cL2hsc1wvUE9XXzAzMzc1NzMxIn0.8TcJZK13155360BpwACs3YJepA-TAFkoD4RjcdsSXT8/vod/npo/fps/TEST/npo/hls/POW_03375731/a1/fileSequence.aac', offset 24451598, playlist 0
[hls,applehttp @ 02539440] SAMPLE-AES encryption is not supported yet
[hls,applehttp @ 02539440] Failed to open segment of playlist 0
Format aac detected only with low score of 1, misdetection possible!
[hls,applehttp @ 02539440] No longer receiving playlist 0
https://nl-ams-p4-am5.cdn.streamgate.nl/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDkzNDU3NjMsImNsaWVudF9pcCI6IjgwLjEwMS40OS41IiwidXJpIjoiXC92b2RcL25wb1wvZnBzXC9URVNUXC9ucG9cL2hsc1wvUE9XXzAzMzc1NzMxIn0.8TcJZK13155360BpwACs3YJepA-TAFkoD4RjcdsSXT8/vod/npo/fps/TEST/npo/hls/POW_03375731/playlist.m3u8: Invalid data found when processing input

Compiling this hlsdl didn't go so well either.

comment:15 by intermediary, 3 years ago

A current git compile of ffmpeg still has trouble handling streams with sample-aes.
HLSDL is also a bit troublesome with getting the right key material in.
Hopefully someone will come around and implement this in ffmpeg proper.

comment:17 by Balling, 9 months ago

The patches are applied now. Please test. ff958b38463f6ebd17471c40001287c71698f639

-c copy does work with https://pw.pc.cdn.bitgravity.com/b7826d86-612d-4ef2-b17c-cc4f91d9584c/master.m3u8 but you still need a key, alas.

-allowed_extensions ALL does not help either.

Last edited 9 months ago by Balling (previous) (diff)

comment:18 by Daniel Cantarín, 8 months ago

@Balling

I'm trying to test this new functionality, using my own encrypted streams (I know the encryption parameters). Yet, I'm failing to get the proper ffmpeg syntax for it.

This command is able to download and mix the file, yet of course with no decryption:
ffmpeg -hide_banner -y -i "https://server/master-playlist.m3u8" -map 0:v:0 -c:v copy -t 20 video.mp4

However, I can't find the proper way to enable it. Would you please provide an example ffmpeg command for HLS decryption using the new patches?

Last edited 8 months ago by Daniel Cantarín (previous) (diff)

comment:19 by Balling, 8 months ago

However, I can't find the proper way to enable it.

echo "0123456789ABCDEF" > keyfile
dd if=keyfile bs=16 count=1 of=inputkey

mediafilesegmenter --encrypt-key-file=myKey.bin --encrypt-key-url=http://127.0.0.1/keyOnly.bin --stream-encrypt --iso-fragmented bbb_sunflower_native_60fps_normal.mp4

From a different terminal:

./ffmpeg -i http://127.0.0.1/prog_index.m3u8

in reply to:  19 comment:20 by Daniel Cantarín, 8 months ago

Replying to Balling:

However, I can't find the proper way to enable it.

echo "0123456789ABCDEF" > keyfile
dd if=keyfile bs=16 count=1 of=inputkey

mediafilesegmenter --encrypt-key-file=myKey.bin --encrypt-key-url=http://127.0.0.1/keyOnly.bin --stream-encrypt --iso-fragmented bbb_sunflower_native_60fps_normal.mp4

From a different terminal:

./ffmpeg -i http://127.0.0.1/prog_index.m3u8

Thank you very much for your example.

I understand your example has an encryption key URL in the playlists, and so ffmpeg is able to get that key this way. But my setup involves FairPlay DRM, so I get this kind of errors:

Unable to open key file skd://REDACTED-URI-HERE

What happens next is that ffmpeg actually do download the segments, and creates the mp4 container with the h264 copied from the input, but does not decrypt it. The end video is noise.

I was hoping to be able to feed the decryption key somehow to ffmpeg by command line, as I know all the encryption parameters applied to my streams. Yet, I didn't find a proper syntax so far. Are you aware of any way of doing that?

Thanks again.

(For the record: I DO NOT want ffmpeg to implement fairplay, but just to test the currently implemented SAMPLE-AES decryption using my own already encrypted streams)

Note: See TracTickets for help on using tickets.