Opened 5 years ago

Closed 3 years ago

Last modified 3 years ago

#2452 closed defect (needs_more_info)

decoding yuv420p to rgb shifts luminance down

Reported by: burek Owned by:
Priority: normal Component: undetermined
Version: git-master Keywords:
Cc: glopes Blocked By:
Blocking: Reproduced by developer: yes
Analyzed by developer: no

Description

Original post on the forum: http://ffmpeg.gusari.org/viewtopic.php?f=11&t=884

Summary of the bug:
"I'm recording video from a grayscale camera into an MPEG4 video file using FFMPEG 1.2. Recently I noticed a weird effect: if I decode the file using FFMPEG or windows media player, the output frames are noticeably darker (by about 10/12 brightness values) than the original source."

How to reproduce:

ffmpeg -ss 0.5 -i video.avi -vframes 1 -t 1 -s 1280x680 -pix_fmt gray gray.bmp

ffmpeg version 1.2 Copyright (c) 2000-2013 the FFmpeg developers
  built on Apr  4 2013 12:40:58 with gcc 4.6.2 (GCC)
  configuration: --enable-w32threads
  libavutil      52. 18.100 / 52. 18.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 42.103 /  3. 42.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
Input #0, avi, from 'video.avi':
  Metadata:
    encoder         : Lavf53.32.100
  Duration: 00:00:07.47, start: 0.000000, bitrate: 16532 kb/s
    Stream #0:0: Video: mpeg4 (Simple Profile) (FMP4 / 0x34504D46), yuv420p, 128
0x720 [SAR 1:1 DAR 16:9], 30 tbr, 30 tbn, 30 tbc
Output #0, image2, to 'gray.bmp':
  Metadata:
    encoder         : Lavf54.63.104
    Stream #0:0: Video: bmp, gray, 1280x680 [SAR 17:18 DAR 16:9], q=2-31, 200 kb
/s, 90k tbn, 30 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 -> bmp)
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.03 bitrate=N/A
video:851kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.002524%

"I thought the encoding step was doing this until I opened the same file in VLC and it gave me back the correct result. I played around with FFMPEG command lines to decode a single frame and realized that if I decode a frame into GRAY8 pixel format, the brightness/luminance values are preserved."

"This happens with every single grayscale video that I encode to MPEG4 with FFMPEG. My best guess so far is it has to do with how the container pixel format gets converted to/from. Since I'm using MPEG4, the file pixel format is YUV420P. I have no idea how ffmpeg encodes from GRAY8 to YUV420P, but maybe it stores just the luminance values in Y... if this happens, then decoding from this to RGB could produce darker pixels due to the shift factor that is applied to the luminance on decoding?

To sum it up:

1) How can encoding a grayscale video to YUV420P with FFMPEG and decoding back produce wrong (darker) brightness values when decoding to RGB versus GRAY8? Presumably once the frames are in YUV420P format it shouldn't matter whether the source is actually grayscale or not so the result should be equivalent, no?

2) How does VLC avoid this situation? I was under the impression that VLC used FFMPEG as well for video decoding, but somehow they managed to figure out how to produce the correct values without requiring me to indicate explicitly that the video was grayscale."

Change History (19)

comment:1 Changed 5 years ago by michael

How can this be reproduced ?
I mean like "here is a file, when run with that command here and viewing the output file with that player it is showing this artifact compared to viewing this file with that player"
You say alot but piecing together from that how to reproduce the bug is not as easy as it may seem to you.

comment:2 Changed 5 years ago by glopes

Sorry for that. You can download an example video from here:
https://dl.dropboxusercontent.com/u/3907539/video.avi

Running the following commandline produces a bitmap with the correct luminance values:

ffmpeg -ss 0.5 -i video.avi -vframes 1 -t 1 -s 1280x680 -pix_fmt gray gray.bmp

Running the following commandline produces a bitmap in which the luminance values are shifted ~10 values darker:

ffmpeg -ss 0.5 -i video.avi -vframes 1 -t 1 -s 1280x680 -pix_fmt bgr24 rgb.bmp

Also playing this video file in VLC produces the correct luminance values.

The video was acquired from a grayscale camera and encoded as grayscale pixel format to mpeg4 yuv420p using FFMPEG (as you can see from the commandline output shown in the original ticket).

Hope this helps to clarify the issue.

comment:3 Changed 4 years ago by glopes

Additional note: as of version 2.0.7 (at least) VLC is no longer able to correctly reproduce the grayscale brightness values.

comment:4 Changed 4 years ago by cehoyos

  • Cc glopes added

Is the problem still reproducible with current FFmpeg?

comment:5 Changed 4 years ago by glopes

Thanks for asking, actually! I just tried with the latest build (ffmpeg version N-55159-gf118b41) and it turns out now it's not even possible to get the correct luminance values by using the first command line!!

I don't know what changed about pix_fmt gray decoding from yuv420p, but now things are even worse as I have currently no way of recovering the correct luminance from my recorded videos... data loss is now a serious possibility...

Last known working version is ffmpeg version N-51639-g7775992.

Here's the output from the two commands now:

gray pix_fmt:

ffmpeg -ss 0.5 -i video.avi -vframes 1 -t 1 -s 1280x680 -pix_fmt gray gray.bmp
ffmpeg version N-55159-gf118b41 Copyright (c) 2000-2013 the FFmpeg developers
  built on Aug  1 2013 18:08:30 with gcc 4.7.3 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvo-aacenc --enable-libvo-
amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --
enable-libxvid --enable-zlib
  libavutil      52. 40.100 / 52. 40.100
  libavcodec     55. 19.100 / 55. 19.100
  libavformat    55. 12.102 / 55. 12.102
  libavdevice    55.  3.100 / 55.  3.100
  libavfilter     3. 82.100 /  3. 82.100
  libswscale      2.  4.100 /  2.  4.100
  libswresample   0. 17.103 /  0. 17.103
  libpostproc    52.  3.100 / 52.  3.100
Input #0, avi, from 'video.avi':
  Metadata:
    encoder         : Lavf53.22.0
  Duration: 00:00:13.59, start: 0.000000, bitrate: 5596 kb/s
    Stream #0:0: Video: mpeg4 (Simple Profile) (FMP4 / 0x34504D46), yuv420p, 128
0x680 [SAR 1:1 DAR 32:17], 120 fps, 120 tbr, 120 tbn, 120 tbc
File 'gray.bmp' already exists. Overwrite ? [y/N] y
Output #0, image2, to 'gray.bmp':
  Metadata:
    encoder         : Lavf55.12.102
    Stream #0:0: Video: bmp, gray, 1280x680 [SAR 1:1 DAR 32:17], q=2-31, 200 kb/
s, 90k tbn, 120 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 -> bmp)
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=-1.0 Lsize=N/A time=00:00:00.00 bitrate=N/A
video:851kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.002524%

bgr24 pix_fmt:

ffmpeg -ss 0.5 -i video.avi -vframes 1 -t 1 -s 1280x680 -pix_fmt bgr24 rgb.bmp
ffmpeg version N-55159-gf118b41 Copyright (c) 2000-2013 the FFmpeg developers
  built on Aug  1 2013 18:08:30 with gcc 4.7.3 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvo-aacenc --enable-libvo-
amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --
enable-libxvid --enable-zlib
  libavutil      52. 40.100 / 52. 40.100
  libavcodec     55. 19.100 / 55. 19.100
  libavformat    55. 12.102 / 55. 12.102
  libavdevice    55.  3.100 / 55.  3.100
  libavfilter     3. 82.100 /  3. 82.100
  libswscale      2.  4.100 /  2.  4.100
  libswresample   0. 17.103 /  0. 17.103
  libpostproc    52.  3.100 / 52.  3.100
Input #0, avi, from 'video.avi':
  Metadata:
    encoder         : Lavf53.22.0
  Duration: 00:00:13.59, start: 0.000000, bitrate: 5596 kb/s
    Stream #0:0: Video: mpeg4 (Simple Profile) (FMP4 / 0x34504D46), yuv420p, 128
0x680 [SAR 1:1 DAR 32:17], 120 fps, 120 tbr, 120 tbn, 120 tbc
File 'rgb.bmp' already exists. Overwrite ? [y/N] y
Output #0, image2, to 'rgb.bmp':
  Metadata:
    encoder         : Lavf55.12.102
    Stream #0:0: Video: bmp, bgr24, 1280x680 [SAR 1:1 DAR 32:17], q=2-31, 200 kb
/s, 90k tbn, 120 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg4 -> bmp)
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=-1.0 Lsize=N/A time=00:00:00.00 bitrate=N/A
video:2550kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.000843
%

comment:6 Changed 4 years ago by cehoyos

How did you produce the input video?
Ticket #2684 was about incorrect encoding of gray input, this should now be fixed.

comment:7 Changed 4 years ago by glopes

Thanks for the reference. I have to admit I'm not entirely sure I understand what the problem was with encoding gray input...

More importantly, did the fix change anything about the way gray format is decoded?

The video was produced using the AForge framework, which underneath uses some version of FFMPEG prior to 2012. I did not reencode the video with the new version, nor would I want to, since I have 1.5 years of these videos to analyze that just can't be reencoded...

In my case, it turns out I am absolutely sure that the gray encoding is actually correct, since I checked the frame grabber's pixel values myself in real-time while the data was coming through precisely to make sure that it wasn't an encoder problem, back when I found out about this.

This decoding issue now places me at risk of either losing data since the shift actually reduces precious contrast or having to stop updating FFMPEG from now on; neither of them I am particularly fond of...

comment:8 Changed 4 years ago by cehoyos

How did you check the mpeg4 frames encoded by FFmpeg, that sounds difficult?
(And I wonder if the conversion from gray input to yuv420p was correct.)

Please provide the command line including console output of the original encoding, that might help understanding the actual problem.

comment:9 Changed 4 years ago by cehoyos

If you look at the gray input: does it contain values from 0-255 or 16-235?

comment:10 Changed 4 years ago by glopes

Yes, I didn't check the actual mpeg4 frames (sorry if I wasn't clear).

What I did check was the raw grayscale pixel intensities of the data coming in through the camera before it was encoded and then compared it to the pixel intensities of the reconstructed frames after decoding.

So if the previous gray encoder/decoder pair was buggy it's interesting why they allowed me to reconstruct the pixel intensities perfectly in the first place.

As to the last question, I just checked and I can find values ranging from 8-255 without much effort. Not sure if there's absolute zeros in there because of lighting conditions, but it definitely looks like I'm full range (which doesn't happen with this current version, where pixels everywhere from [8-14] get squashed to 0).

Don't have access to the setup right now, will try to make do. If I manage to reproduce anything meaningful about encoder differences, I'll let you know.
I'm still learning about pixel format specifications, but I guess what I mean to say is that somehow the bytes that were encoded in the file contain information about the correct pixel intensities. I just want to make sure I can read them back...

comment:11 Changed 4 years ago by cehoyos

Could you confirm that the following patch (against current git head) "fixes" decoding of your samples?

diff --git a/libavcodec/h263dec.c b/libavcodec/h263dec.c
index bf9e072..85ca4b2 100644
--- a/libavcodec/h263dec.c
+++ b/libavcodec/h263dec.c
@@ -770,7 +770,7 @@ const enum AVPixelFormat ff_h263_hwaccel_pixfmt_list_420[] = {
 #if CONFIG_VDPAU
     AV_PIX_FMT_VDPAU,
 #endif
-    AV_PIX_FMT_YUV420P,
+    AV_PIX_FMT_YUVJ420P,
     AV_PIX_FMT_NONE
 };

comment:12 follow-up: Changed 4 years ago by richardpl

  • Resolution set to needs_more_info
  • Status changed from new to closed

Provide way to reproduce this bug.
Where is input sample?

comment:13 in reply to: ↑ 12 Changed 4 years ago by cehoyos

comment:14 follow-up: Changed 4 years ago by glopes

  • Resolution needs_more_info deleted
  • Status changed from closed to reopened

I'm really sorry for not being able to retry the example with the new modified FFmpeg sources, but to say that there is no input sample is not true!

It's in the first comment, I'll repost below. If someone could please just run this video through the patch would be super helpful...

https://dl.dropboxusercontent.com/u/3907539/video.avi

Running the following commandline produces a bitmap with the correct luminance values:
ffmpeg -ss 0.5 -i video.avi -vframes 1 -t 1 -s 1280x680 -pix_fmt gray gray.bmp
Running the following commandline produces a bitmap in which the luminance values are shifted ~10 values darker:
ffmpeg -ss 0.5 -i video.avi -vframes 1 -t 1 -s 1280x680 -pix_fmt bgr24 rgb.bmp
Also playing this video file in VLC produces the correct luminance values.
The video was acquired from a grayscale camera and encoded as grayscale pixel format to mpeg4 yuv420p using FFMPEG (as you can see from the commandline output shown in the original ticket).
Hope this helps to clarify the issue.

I can't seem to modify the original description of the ticket or else I would include it in there...

comment:15 Changed 4 years ago by richardpl

  • Reproduced by developer set
  • Version changed from unspecified to git-master

Video indeed use full color range.

Workaround is to use '-vf extractplanes=y'

comment:16 in reply to: ↑ 14 Changed 4 years ago by cehoyos

Replying to glopes:

I'm really sorry for not being able to retry the example with the new modified FFmpeg sources

Could you test if I am correct that the patch from comment:11 fixes the issue?

comment:17 follow-up: Changed 3 years ago by michael

  • Resolution set to needs_more_info
  • Status changed from reopened to closed

The input file is correctly decoded, the fault is in what / how created the input file. I would guess that the wrong pixel format was used mixing jpeg and mpeg luma ranges

comment:18 in reply to: ↑ 17 ; follow-up: Changed 3 years ago by glopes

Replying to michael:

The input file is correctly decoded, the fault is in what / how created the input file. I would guess that the wrong pixel format was used mixing jpeg and mpeg luma ranges

Well, in regards to "what" created the input file, it was FFmpeg itself, so if there is anything wrong in the encoder it is a bug in FFmpeg as well.

Also, ffmpeg version N-51639-g7775992 was correctly decoding the file, so clearly something changed, would at least be helpful to know what.

Did anyone try the patch in comment:11?

I'll try encoding/decoding grayscale in the latest ffmpeg and report what I find there.

comment:19 in reply to: ↑ 18 Changed 3 years ago by cehoyos

Replying to glopes:

Replying to michael:

The input file is correctly decoded, the fault is in what / how created the input file. I would guess that the wrong pixel format was used mixing jpeg and mpeg luma ranges

Well, in regards to "what" created the input file, it was FFmpeg itself, so if there is anything wrong in the encoder it is a bug in FFmpeg as well.

Also, ffmpeg version N-51639-g7775992 was correctly decoding the file, so clearly something changed, would at least be helpful to know what.

(7775992 is from April 2013.)

FFmpeg behaviour wrt to gray was changed in 1ba01d3d46e8c200a2527ebafc71d5651084586b in July 2013, see ticket #2684. To get the old behaviour, either set input color_range or use the extractplanes filter. The patch in comment:11 is another option.

Note: See TracTickets for help on using tickets.