Opened 23 months ago

Last modified 23 months ago

#8561 new defect

"URL read error: End of file" when http_persistent enabled

Reported by: vschweitzer Owned by:
Priority: normal Component: avformat
Version: git-master Keywords: http
Cc: liuqi05@kuaishou.com Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description (last modified by Carl Eugen Hoyos)

When streaming via HLS to a HTTP server while http_persistent
is enabled, some segments will fail to upload with the error

[http @ 0000023db7ff2140] URL read error: End of file
[hls @ 0000023db85f0200] upload segment failed, will retry with a new http session.

The command line used to reproduce the output above is

ffmpeg -y -f lavfi -i color=c=black:s=1920x1080:r=25 -vcodec libx264 -f hls -http_persistent 1 http://127.0.0.1:45000/streams/index.m3u8

The error should occur after approximately 50 requests.

This behavior was tested with both Apache and NGINX.
NGINX was configured in the following way:

In front of "server {...}":

map $request_method $my_proxy_method {
  default $request_method;
  POST PUT;
}

In "server {...}":

location /streams/ {
    proxy_pass http://127.0.0.1:45000/test/;
    proxy_method $my_proxy_method;
    proxy_ignore_client_abort on;
}

location /test/ {
    root   html;
    index  index.html index.htm;
    dav_methods PUT DELETE MKCOL COPY MOVE;
    dav_access all:rw;
    create_full_put_path on;
    client_max_body_size 10000m;
}

As FFmpeg sometimes uses POST requests even if PUT is explicitly
specified, POST requests are mapped to PUT. This requires a
new request, which is why the server creates a new request
to itself to a new directory when receiving a request to
/streams/.

On a possibly related note, the NGINX log reports a strange
looking request at the time of the FFmpeg error:

127.0.0.1 - - [09/Mar/2020:10:48:26 +0000] "PUT /streams/index54.ts HTTP/1.1" 201 0 "-" "Lavf/58.35.104"
127.0.0.1 - - [09/Mar/2020:10:48:26 +0000] "PUT /test/index54.ts HTTP/1.0" 201 0 "-" "Lavf/58.35.104"
127.0.0.1 - - [09/Mar/2020:10:48:26 +0000] "0" 400 157 "-" "-"

Although possibly related, this bug is not the same as
ticket #8546.

Attachments (1)

ffmpeg_log.txt (14.3 KB ) - added by vschweitzer 23 months ago.
The console output of the FFmpeg command listed in the report.

Download all attachments as: .zip

Change History (4)

by vschweitzer, 23 months ago

Attachment: ffmpeg_log.txt added

The console output of the FFmpeg command listed in the report.

comment:1 by Carl Eugen Hoyos, 23 months ago

Component: undeterminedavformat
Description: modified (diff)

comment:2 by Steven Liu, 23 months ago

http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests
said;
Default:
keepalive_requests 100;

Syntax:	keepalive_requests number;
Default:	
keepalive_requests 100;
Context:	http, server, location
This directive appeared in version 0.8.0.
Sets the maximum number of requests that can be served through one keep-alive connection. After the maximum number of requests are made, the connection is closed.

Closing connections periodically is necessary to free per-connection memory allocations. Therefore, using too high maximum number of requests could result in excessive memory usage and not recommended.

So FFmpeg want fix the problem of https://trac.ffmpeg.org/ticket/7975.
But I think if retry put the files, must give a warning before retry.

comment:3 by Steven Liu, 23 months ago

Cc: liuqi05@kuaishou.com added
Note: See TracTickets for help on using tickets.