FFmpeg RTMPE not working

[RiCON]

Summary of the bug:
Native RTMPE is complaining about "Signature mismatch" with both OpenSSL and GnuTLS/GMP while librtmp's is working fine.

How to reproduce:

% youtube-dl -j "http://www.crunchyroll.com/the-perfect-insider/episode-4-the-rainbow-colored-past-686451"
<to obtain url and playpath>

% ffmpeg-rtmp-gnutls -rtmp_playpath "mp4:c2/s/ve2017005/video.mp4" -i "rtmpe://cp150757.edgefcs.net/ondemand/?auth=daEa1bmajcLctbqcEaSbpcYaQc5bcdea_ai-bwp2MS-dHa-oBFwusOCGvA&aifp=0009&slist=c2/s/ve2017005/video.mp4"
ffmpeg version N-76516-ged45b45 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (Rev4, Built by MSYS2 project)
  configuration: --prefix=/local64 --bindir=/local64/bin-video --enable-static --disable-shared --pkg-config-flags=--static --enable-gnutls --enable-gpl --enable-librtmp
  libavutil      55.  5.100 / 55.  5.100
  libavcodec     57. 15.100 / 57. 15.100
  libavformat    57. 14.100 / 57. 14.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 14.101 /  6. 14.101
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.100 /  2.  0.100
  libpostproc    54.  0.100 / 54.  0.100
HandShake: Type mismatch: client sent 6, server answered 8
Metadata:
  duration              1381.05
  moovPosition          32.00
  width                 656.00
  height                368.00
  videocodecid          avc1
  audiocodecid          mp4a
  avcprofile            77.00
  avclevel              30.00
  aacaot                2.00
  videoframerate        23.98
  audiosamplerate       44100.00
  audiochannels         2.00
trackinfo:
  length                16555000.00
  timescale             11988.00
  language              und
sampledescription:
  sampletype            avc1
  length                60904448.00
  timescale             44100.00
  language              und
sampledescription:
  sampletype            mp4a
Input #0, flv, from 'rtmpe://cp150757.edgefcs.net/ondemand/?auth=daEa1bmajcLctbqcEaSbpcYaQc5bcdea_ai-bwp2MS-dHa-oBFwusOCGvA&aifp=0009&slist=c2/s/ve2017005/video.mp4':
  Metadata:
    moovPosition    : 32
    avcprofile      : 77
    avclevel        : 30
    aacaot          : 2
    videoframerate  : 24
    audiochannels   : 2
    length          : 60904448
    timescale       : 44100
    sampletype      : mp4a
  Duration: 00:23:01.05, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p, 656x368 [SAR 368:369 DAR 16:9], 24.42 fps, 23.98 tbr, 1k tbn, 47.95 tbc
    Stream #0:1: Audio: aac (LC), 44100 Hz, stereo, fltp
At least one output file must be specified

% ffmpeg-gnutls -rtmp_playpath "mp4:c2/s/ve2017005/video.mp4" -i "rtmpe://cp150757.edgefcs.net/ondemand/?auth=daEa1bmajcLctbqcEaSbpcYaQc5bcdea_ai-bwp2MS-dHa-oBFwusOCGvA&aifp=0009&slist=c2/s/ve2017005/video.mp4"
ffmpeg version N-76516-ged45b45 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (Rev4, Built by MSYS2 project)
  configuration: --prefix=/local64 --bindir=/local64/bin-video --enable-static --disable-shared --pkg-config-flags=--static --enable-gnutls --enable-gpl
  libavutil      55.  5.100 / 55.  5.100
  libavcodec     57. 15.100 / 57. 15.100
  libavformat    57. 14.100 / 57. 14.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 14.101 /  6. 14.101
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.100 /  2.  0.100
  libpostproc    54.  0.100 / 54.  0.100
[rtmpe @ 0000003605adf440] Signature mismatch
rtmpe://cp150757.edgefcs.net/ondemand/?auth=daEa1bmajcLctbqcEaSbpcYaQc5bcdea_ai-bwp2MS-dHa-oBFwusOCGvA&aifp=0009&slist=c2/s/ve2017005/video.mp4: Input/output error

% ffmpeg-openssl -rtmp_playpath "mp4:c2/s/ve2017005/video.mp4" -i "rtmpe://cp150757.edgefcs.net/ondemand/?auth=daEa1bmajcLctbqcEaSbpcYaQc5bcdea_ai-bwp2MS-dHa-oBFwusOCGvA&aifp=0009&slist=c2/s/ve2017005/video.mp4"
ffmpeg version N-76516-ged45b45 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.2.0 (Rev4, Built by MSYS2 project)
  configuration: --prefix=/local64 --bindir=/local64/bin-video --enable-static --disable-shared --pkg-config-flags=--static --enable-openssl --enable-gpl --enable-nonfree
  libavutil      55.  5.100 / 55.  5.100
  libavcodec     57. 15.100 / 57. 15.100
  libavformat    57. 14.100 / 57. 14.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 14.101 /  6. 14.101
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.100 /  2.  0.100
  libpostproc    54.  0.100 / 54.  0.100
[rtmpe @ 000000d3a97cee00] Signature mismatch
rtmpe://cp150757.edgefcs.net/ondemand/?auth=daEa1bmajcLctbqcEaSbpcYaQc5bcdea_ai-bwp2MS-dHa-oBFwusOCGvA&aifp=0009&slist=c2/s/ve2017005/video.mp4: Input/output error

[Eugene Yunak]

i've hit the same problem with crunchyroll

[RiCON]

Tested with patches submitted to libav's mailing list by Martin Storsjö it seems to be fixed.

[Carl Eugen Hoyos]

Isn't this a duplicate of ticket #2907?

[RiCON]

It's about RTMPE too but I don't think it's the same problem.

To be more precise, this seems a problem with type 8 handshakes, looking at the patches that fix it. Ticket #2907's server is using type 6, so it's not the same issue.

[RiCON]

Fixed with e12f4036786136734b917d5bad641f2f89c3f9d2