SChannel TLS implementation bug

[Adion]

There's a bug in tls_schannel.c that prevents rtmps to work correctly when using this tls implementation. (It hangs in an infinite loop in tcp_read)
It is present in ffmpeg v4.1.3 and at this time is also still present in master.

The problem is in tls_read, when only a subset of the data returned by the server is requested.
First data is read from the server into c->enc_buf.
Next the data is decoded, but data is only encoded until enough data is decoded to fulfill the read request.
In the case of SECBUFFER_EXTRA, there may still be undecoded data in enc_buf when the loop stops and the data is returned.

On the next call to tls_read, there is not enough data in the decoded buffer, so it requests more data from the tcp layer, but this never comes as the data is already there, just not decoded yet.

I fixed it by decoding all the data before returning. Another way would be to check if there is still undecoded data in the encode data on the next request and decode it at that time.

I changed

while (c->enc_buf_offset > 0 && sspi_ret == SEC_E_OK && c->dec_buf_offset < len) {

to

while (c->enc_buf_offset > 0 && sspi_ret == SEC_E_OK) {

Related to this, I found it odd that by default there is no timeout specified, is this normal?

[Carl Eugen Hoyos]

Please send your patch - made with git format-patch - to the FFmpeg development mailing list.

[Carl Eugen Hoyos]

Fixed by Jan Ekström in 39977fff20048f1798a95c593d6034a0e73ebbe5