Opened 5 years ago

Closed 5 years ago

#635 closed defect (invalid)

Incorrect Memory Layout when using avpicture_layout with PIX_FMT_NV12

Reported by: jkersch Owned by:
Priority: normal Component: avcodec
Version: unspecified Keywords: NV12
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

when decoding (e.g. a h264 stream) with output format PIX_FMT_NV12, the AVFrame contains 3 data buffers with Luma and Cb and Cr planes.

when using avpicture_layout to copythe AVPicture to a previously allocated memory buffer, the bytes are layouted incorrectly.
NV12 is all Luma bytes first, and after that the Cb and Cr bytes interleaved (see http://msdn.microsoft.com/en-us/library/windows/desktop/dd206750(v=vs.85).aspx#nv12)

i´ve worked around this manually, but it would be great if this incorrect behavior could be fixed within libavcodec itself.

Change History (4)

comment:1 in reply to: ↑ description Changed 5 years ago by cehoyos

  • Keywords PIX_FMT_NV12 layout removed

Replying to jkersch:

when decoding (e.g. a h264 stream) with output format PIX_FMT_NV12

I don't think this is possible with libavcodec.

The only decoder that supports NV12 is rawvideo.
A rawvideo with pix_fmt nv12 produced with FFmpeg (that can be decoded correctly with FFmpeg) plays also fine with mplayer -vo vdpau (which supports displaying NV12).

Last edited 5 years ago by cehoyos (previous) (diff)

comment:2 follow-up: Changed 5 years ago by jkersch

the Luma plane is correct, however, for Cb and Cr there are two different buffers instead of an interleaved one where the byte are merged in the form (Cb)(Cr)(Cb)(Cr).
linesize however correctly displays only 2 planes.
any chance to get this layouted directly within avpicture_layout?

i want to pack those buffers into a media sample to use avcodec as a Media Foundation Transform decoder. (which works great, except the hack to output NV12)

comment:3 in reply to: ↑ 2 Changed 5 years ago by cehoyos

Replying to jkersch:

the Luma plane is correct, however, for Cb and Cr there are two different buffers instead of an interleaved one where the byte are merged in the form (Cb)(Cr)(Cb)(Cr).

That is because libavcodec's H.264 decoder can only output to yuv420p (if the original sample is 8bit yuv420 which I assume). It cannot output to nv12, no matter what the input is.

libswscale can convert yuv420p to nv12.

comment:4 Changed 5 years ago by cehoyos

  • Resolution set to invalid
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.