Opened 2 weeks ago

Closed 9 days ago

#6642 closed defect (invalid)

GPU hang in decoding hevc video by using ffmpeg

Reported by: hsyrja Owned by:
Priority: important Component: undetermined
Version: git-master Keywords: deadlock vaapi
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

GPU hang occurred when hw accelerated decoding is issued by ffmpeg and video codec is Hevc encoded.

ffmpeg command (decode only)
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -I hang.vid -f null -

input video cam be downloaded:
http://www.datafilehost.com/d/adf64b8a

Tested in Skylake as wel Kabyake i7.

Behaviour is random hang won't happen every time, but when it's happens it is always in ffmpeg start.

I don't know if this is intel-vaapi-driver issue or FFmpeg issue so I will file bug to both projects.

Attachments (1)

hang.zip (529.8 KB) - added by hsyrja 2 weeks ago.
Small video which should reproduce the hang

Download all attachments as: .zip

Change History (5)

Changed 2 weeks ago by hsyrja

Small video which should reproduce the hang

comment:1 Changed 2 weeks ago by hsyrja

From Libya team.

I saw lots of error messages when using a tool to parse hang.vid
I can't reproduce this issue with gstreamer-vaapi, it seems ffmpeg and gstreamer handle the element errors in different way so they pass different picture parameters and slice parameters to the driver. You may enable VA tracer by setting LIBVA_TRACE in your environment then check the tracer log.
e.g. in gstreamer

[48642.979908][ctx 0x02000000] pcm_sample_bit_depth_luma_minus1 = 0
[48642.979909][ctx 0x02000000] pcm_sample_bit_depth_chroma_minus1 = 0
...
[48642.979952][ctx 0x02000000] slice_parsing_fields = 15246

however in FFmpeg

[48794.782034][ctx 0x02000000] pcm_sample_bit_depth_luma_minus1 = 255
[48794.782034][ctx 0x02000000] pcm_sample_bit_depth_chroma_minus1 = 255
...
[48794.782054][ctx 0x02000000] slice_parsing_fields = 910

According to the tracer log, I think this is a FFmpeg specific issue.

driver(HW) provides error handling, but it can't cover all of errors.

comment:2 Changed 2 weeks ago by jkqxz

I can reproduce this here with VAAPI on Intel. It works fine with VAAPI on AMD/Mesa and matches software decode output, so I think it has to be a bug in the Intel driver.

The pcm_sample_bit_depth_*_minus1 is irrelevant, because pcm_enabled_flag is 0.

On slice_parsing_fields, for the first picture it is:

[57463.395908][ctx 0x0000001b]  slice_parsing_fields = 910
[57463.395909][ctx 0x0000001b]  lists_modification_present_flag = 0
[57463.395909][ctx 0x0000001b]  long_term_ref_pics_present_flag = 1
[57463.395910][ctx 0x0000001b]  sps_temporal_mvp_enabled_flag = 1
[57463.395910][ctx 0x0000001b]  cabac_init_present_flag = 1
[57463.395911][ctx 0x0000001b]  output_flag_present_flag = 0
[57463.395911][ctx 0x0000001b]  dependent_slice_segments_enabled_flag = 0
[57463.395912][ctx 0x0000001b]  pps_slice_chroma_qp_offsets_present_flag = 0
[57463.395913][ctx 0x0000001b]  sample_adaptive_offset_enabled_flag = 1
[57463.395913][ctx 0x0000001b]  deblocking_filter_override_enabled_flag = 1
[57463.395914][ctx 0x0000001b]  pps_disable_deblocking_filter_flag = 1
[57463.395914][ctx 0x0000001b]  slice_segment_header_extension_present_flag = 0
[57463.395915][ctx 0x0000001b]  RapPicFlag = 0
[57463.395915][ctx 0x0000001b]  IdrPicFlag = 0
[57463.395916][ctx 0x0000001b]  IntraPicFlag = 0

which matches the picture parameter set:

[AVBSFContext @ 0x5606a8962b40] Picture Parameter Set
[AVBSFContext @ 0x5606a8962b40] 0           forbidden_zero_bit                                          0 = 0
[AVBSFContext @ 0x5606a8962b40] 1           nal_unit_type                                          100010 = 34
[AVBSFContext @ 0x5606a8962b40] 7           nuh_layer_id                                           000000 = 0
[AVBSFContext @ 0x5606a8962b40] 13          nuh_temporal_id_plus1                                     001 = 1
[AVBSFContext @ 0x5606a8962b40] 16          pps_pic_parameter_set_id                                    1 = 0
[AVBSFContext @ 0x5606a8962b40] 17          pps_seq_parameter_set_id                                    1 = 0
[AVBSFContext @ 0x5606a8962b40] 18          dependent_slice_segments_enabled_flag                       0 = 0
[AVBSFContext @ 0x5606a8962b40] 19          output_flag_present_flag                                    0 = 0
[AVBSFContext @ 0x5606a8962b40] 20          num_extra_slice_header_bits                               000 = 0
[AVBSFContext @ 0x5606a8962b40] 23          sign_data_hiding_enabled_flag                               0 = 0
[AVBSFContext @ 0x5606a8962b40] 24          cabac_init_present_flag                                     1 = 1
[AVBSFContext @ 0x5606a8962b40] 25          num_ref_idx_l0_default_active_minus1                        1 = 0
[AVBSFContext @ 0x5606a8962b40] 26          num_ref_idx_l1_default_active_minus1                        1 = 0
[AVBSFContext @ 0x5606a8962b40] 27          init_qp_minus26                                             1 = 0
[AVBSFContext @ 0x5606a8962b40] 28          constrained_intra_pred_flag                                 0 = 0
[AVBSFContext @ 0x5606a8962b40] 29          transform_skip_enabled_flag                                 0 = 0
[AVBSFContext @ 0x5606a8962b40] 30          cu_qp_delta_enabled_flag                                    1 = 1
[AVBSFContext @ 0x5606a8962b40] 31          diff_cu_qp_delta_depth                                    011 = 2
[AVBSFContext @ 0x5606a8962b40] 34          pps_cb_qp_offset                                            1 = 0
[AVBSFContext @ 0x5606a8962b40] 35          pps_cr_qp_offset                                            1 = 0
[AVBSFContext @ 0x5606a8962b40] 36          pps_slice_chroma_qp_offsets_present_flag                    0 = 0
[AVBSFContext @ 0x5606a8962b40] 37          weighted_pred_flag                                          0 = 0
[AVBSFContext @ 0x5606a8962b40] 38          weighted_bipred_flag                                        0 = 0
[AVBSFContext @ 0x5606a8962b40] 39          transquant_bypass_enabled_flag                              0 = 0
[AVBSFContext @ 0x5606a8962b40] 40          tiles_enabled_flag                                          0 = 0
[AVBSFContext @ 0x5606a8962b40] 41          entropy_coding_sync_enabled_flag                            0 = 0
[AVBSFContext @ 0x5606a8962b40] 42          pps_loop_filter_across_slices_enabled_flag                  1 = 1
[AVBSFContext @ 0x5606a8962b40] 43          deblocking_filter_control_present_flag                      1 = 1
[AVBSFContext @ 0x5606a8962b40] 44          deblocking_filter_override_enabled_flag                     1 = 1
[AVBSFContext @ 0x5606a8962b40] 45          pps_deblocking_filter_disabled_flag                         1 = 1
[AVBSFContext @ 0x5606a8962b40] 46          pps_scaling_list_data_present_flag                          0 = 0
[AVBSFContext @ 0x5606a8962b40] 47          lists_modification_present_flag                             0 = 0
[AVBSFContext @ 0x5606a8962b40] 48          log2_parallel_merge_level_minus2                            1 = 0
[AVBSFContext @ 0x5606a8962b40] 49          slice_segment_header_extension_present_flag                 0 = 0
[AVBSFContext @ 0x5606a8962b40] 50          pps_extension_present_flag                                  0 = 0
[AVBSFContext @ 0x5606a8962b40] 51          rbsp_stop_one_bit                                           1 = 1
[AVBSFContext @ 0x5606a8962b40] 52          rbsp_alignment_zero_bit                                     0 = 0
[AVBSFContext @ 0x5606a8962b40] 53          rbsp_alignment_zero_bit                                     0 = 0
[AVBSFContext @ 0x5606a8962b40] 54          rbsp_alignment_zero_bit                                     0 = 0
[AVBSFContext @ 0x5606a8962b40] 55          rbsp_alignment_zero_bit                                     0 = 0

and the content of the first packet:

[AVBSFContext @ 0x5606a8962b40] Packet: 660 bytes, pts 0, dts 0, duration 1800.
[AVBSFContext @ 0x5606a8962b40] Access Unit Delimiter
[AVBSFContext @ 0x5606a8962b40] 0           forbidden_zero_bit                                          0 = 0
[AVBSFContext @ 0x5606a8962b40] 1           nal_unit_type                                          100011 = 35
[AVBSFContext @ 0x5606a8962b40] 7           nuh_layer_id                                           000000 = 0
[AVBSFContext @ 0x5606a8962b40] 13          nuh_temporal_id_plus1                                     001 = 1
[AVBSFContext @ 0x5606a8962b40] 16          pic_type                                                  010 = 2
[AVBSFContext @ 0x5606a8962b40] 19          rbsp_stop_one_bit                                           1 = 1
[AVBSFContext @ 0x5606a8962b40] 20          rbsp_alignment_zero_bit                                     0 = 0
[AVBSFContext @ 0x5606a8962b40] 21          rbsp_alignment_zero_bit                                     0 = 0
[AVBSFContext @ 0x5606a8962b40] 22          rbsp_alignment_zero_bit                                     0 = 0
[AVBSFContext @ 0x5606a8962b40] 23          rbsp_alignment_zero_bit                                     0 = 0
[AVBSFContext @ 0x5606a8962b40] Slice Segment Header
[AVBSFContext @ 0x5606a8962b40] 0           forbidden_zero_bit                                          0 = 0
[AVBSFContext @ 0x5606a8962b40] 1           nal_unit_type                                          000001 = 1
[AVBSFContext @ 0x5606a8962b40] 7           nuh_layer_id                                           000000 = 0
[AVBSFContext @ 0x5606a8962b40] 13          nuh_temporal_id_plus1                                     001 = 1
[AVBSFContext @ 0x5606a8962b40] 16          first_slice_segment_in_pic_flag                             1 = 1
[AVBSFContext @ 0x5606a8962b40] 17          slice_pic_parameter_set_id                                  1 = 0
[AVBSFContext @ 0x5606a8962b40] 18          slice_type                                                010 = 1
[AVBSFContext @ 0x5606a8962b40] 21          slice_pic_order_cnt_lsb                      0000000000100101 = 37
[AVBSFContext @ 0x5606a8962b40] 37          short_term_ref_pic_set_sps_flag                             0 = 0
[AVBSFContext @ 0x5606a8962b40] 38          num_negative_pics                                         010 = 1
[AVBSFContext @ 0x5606a8962b40] 41          num_positive_pics                                           1 = 0
[AVBSFContext @ 0x5606a8962b40] 42          delta_poc_s0_minus1[i]                                      1 = 0
[AVBSFContext @ 0x5606a8962b40] 43          used_by_curr_pic_s0_flag[i]                                 1 = 1
[AVBSFContext @ 0x5606a8962b40] 44          num_long_term_pics                                          1 = 0
[AVBSFContext @ 0x5606a8962b40] 45          slice_temporal_mvp_enabled_flag                             1 = 1
[AVBSFContext @ 0x5606a8962b40] 46          slice_sao_luma_flag                                         1 = 1
[AVBSFContext @ 0x5606a8962b40] 47          slice_sao_chroma_flag                                       1 = 1
[AVBSFContext @ 0x5606a8962b40] 48          num_ref_idx_active_override_flag                            0 = 0
[AVBSFContext @ 0x5606a8962b40] 49          cabac_init_flag                                             1 = 1
[AVBSFContext @ 0x5606a8962b40] 50          five_minus_max_num_merge_cand                           00100 = 3
[AVBSFContext @ 0x5606a8962b40] 55          slice_qp_delta                                      000010000 = 8
[AVBSFContext @ 0x5606a8962b40] 64          deblocking_filter_override_flag                             1 = 1
[AVBSFContext @ 0x5606a8962b40] 65          slice_deblocking_filter_disabled_flag                       0 = 0
[AVBSFContext @ 0x5606a8962b40] 66          slice_beta_offset_div2                                      1 = 0
[AVBSFContext @ 0x5606a8962b40] 67          slice_tc_offset_div2                                        1 = 0
[AVBSFContext @ 0x5606a8962b40] 68          slice_loop_filter_across_slices_enabled_flag                0 = 0
[AVBSFContext @ 0x5606a8962b40] 69          alignment_bit_equal_to_one                                  1 = 1
[AVBSFContext @ 0x5606a8962b40] 70          alignment_bit_equal_to_zero                                 0 = 0
[AVBSFContext @ 0x5606a8962b40] 71          alignment_bit_equal_to_zero                                 0 = 0

slice_parsing_fields = 15246 would require the first slice to be an IDR frame (IdrPicFlag? and IntraPicFlag? both set), but it isn't. Maybe gstreamer isn't a useful comparison here because it has thrown away some of the frames at the start of the stream?

comment:3 Changed 2 weeks ago by jkqxz

To make clearer that other platforms are fine:

$ ./ffmpeg_g -v 0 -y -i hang.vid out_sw.yuv
$ ./ffmpeg_g -v 0 -y -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -i hang.vid -vf format=yuv420p out_amd.yuv
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/local/lib/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0
$ ls -l out_*.yuv
-rw-r--r-- 1 mrt mrt 388454400 Sep  8 20:23 out_amd.yuv
-rw-r--r-- 1 mrt mrt 388454400 Sep  8 20:23 out_sw.yuv
$ cmp out_sw.yuv out_amd.yuv
$ 

i.e. output is bit-identical, so VAAPI is not the problem.

Then:

$ ./ffmpeg_g -v 0 -y -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -i hang.vid -vf format=yuv420p out_intel.yuv
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/local/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0

and the GPU hangs immediately.

(renderD128 is an Intel Skylake GT2, renderD129 is an AMD Polaris 11.)

comment:4 Changed 9 days ago by cehoyos

  • Component changed from ffmpeg to undetermined
  • Keywords deadlock vaapi added
  • Priority changed from normal to important
  • Resolution set to invalid
  • Status changed from new to closed

Assuming this has to be fixed by Intel (invalid data should never hang a driver), I suggest to close this ticket.

Note: See TracTickets for help on using tickets.