RTMP handshake problem

[Yukun Guo]

Summary of the bug:

When FFmpeg connects to RTMP server via digest ("complex") handshake, FFmpeg takes inconsistent ways at the step of sending C2: If it is a publisher, it sends C2 as an echo of S1, OTOH if it's a subscriber, it creates a new packet of C2 consisting of 1504B random-data and 32B signature. In my option the latter one is the expected behavior, while the formal one is only correct in the context of simple handshake. The code shows that publishing and subscribing fall into different branches from this line: ​https://github.com/FFmpeg/FFmpeg/blob/release/3.4/libavformat/rtmpproto.c#L1300. And RTMP publishing feature was introduced in ​https://github.com/FFmpeg/FFmpeg/commit/6bf22e18d1357f11048902e2c5ac9f814cd123fa.

Although the official RTMP specification obviously lacks a section on digest handshaking, I have checked several open source implementations like nginx-rtmp, Red5 and SRS, and they all take the random-data-and-signature way.

[Carl Eugen Hoyos]

Please consider sending a patch made with git format-patch to the development mailing list.