DirectShow, wrong pixel format

[Michael Koch]

There is a problem with a ASI178MM (monochrome) camera and its DirectShow driver. When I check the options with this command

ffmpeg -list_options 1 -f dshow -i video="ASI178MM Camera (ZWO Design)"

I get this output:

[dshow @ 00000203f5b315c0] pixel_format=rgb8 min s=3096x2080 fps=1.0842e-12 max s=3096x2080 fps=30
[dshow @ 00000203f5b315c0] pixel_format=rgb8 min s=1280x960 fps=1.0842e-12 max s=1280x960 fps=30
[dshow @ 00000203f5b315c0] pixel_format=rgb8 min s=1280x800 fps=1.0842e-12 max s=1280x800 fps=30
... (there are more sizes listed, all with the same pixel format rgb8)

When FFmpeg gets a frame from the camera, the detected pixel format is rgb8 (as defined in FFmpeg: 2 bits red, 3 bits green, 3 bits blue). Of course that's wrong, because it's a monochrome camera and the data is 8-bit gray.

My first idea was that it must be a bug in the DirectShow driver. The data is obviously correct 8-bit gray (I did verify that), but it's incorrectly labeled as "rgb8" instead of "gray".

When you look at the specifications of DirectShow

​https://learn.microsoft.com/en-us/windows/win32/directshow/uncompressed-rgb-video-subtypes

you see that there exists no "gray" pixel format, and "RGB8" is a different pixel format described as "RGB, 8 bpp, palettized". It seems an uncompressed 8-bit gray frame must be encoded with "RGB8" pixel format, together with a palette which contains 256 levels of gray.

"RGB8" in DirectShow should become "pal8" or "gray" in FFmpeg, and not "rgb8".

By the way, in line 64 of dshow.c there is a comment that this pixel format is untested. So this might be the first test.

[Balling]

gray

That is YCbCr, except it has Y plane only.

[Michael Koch]

I suspect the problem is in line 70 of dshow.c, but I can't make a test because I can't compile FFmpeg under Windows.

"RGB8" is in the Directshow documentation defined as "RGB, 8 bpp, palettized".
"AV_PIX_FMT_RGB8" in FFmpeg is defined in line 86 of pixfmt.h as "packed RGB 3:3:2, 8bpp, (msb)2R 3G 3B(lsb)".
These are two different pixel formats.

Shouldn't line 70 in dshow.c be changed to AV_PIX_FMT_PAL8, and the palette must be loaded from the DirectShow driver?

The same problem is also in line 68 for RGB4, and in line 66 for RGB1 (here it should work, but the colors from the palette are lost and replaced by black and white).

[Balling]

Is not this a dup of #6279?

[Michael Koch]

I don't think it's a duplicate because #6279 is about RGB24.

[Balling]

No, it is a dup, because avpriv_find_pix_fmt() does not find RGB formats.

[Balling]

dshow.c, but I can't make a test because I can't compile FFmpeg under Windows.

Just use github actions pointing to your FFmpeg fork to compile. For that clone ​https://github.com/BtbN/FFmpeg-Builds enable actions and press build and publish a release. Do not forget to change the repo of FFmpeg.

[Balling]

It seems an uncompressed 8-bit gray frame must be encoded with "RGB8" pixel format, together with a palette which contains 256 levels of gray.

Where is the code that will decode palette?? How does it look?

[Michael Koch]

Replying to Balling:

It seems an uncompressed 8-bit gray frame must be encoded with "RGB8" pixel format, together with a palette which contains 256 levels of gray.

Where is the code that will decode palette?? How does it look?

"For palettized formats, the color of each pixel is specified as an index into a palette. The palette must be included in the format block, following the BITMAPINFOHEADER structure."

Source: ​https://learn.microsoft.com/en-us/windows/win32/directshow/uncompressed-rgb-video-subtypes

and some more details when you click on BITMAPINFOHEADER.

[Michael Koch]

Screenshot of GraphStudioNext with the properties of the video renderer.

[Michael Koch]

With GraphStudioNext the output is correct 8-bit grayscale. A screenshot with the properties of the video renderer is attached. I think "Extradata" is the palette.

[Balling]

So you need to parse

biClrUsed

Specifies the number of color indices in the color table that are actually used by the bitmap. See Remarks for more information.

biClrImportant

Specifies the number of color indices that are considered important for displaying the bitmap. If this value is zero, all colors are important.

[Michael Koch]

Replying to Balling:

dshow.c, but I can't make a test because I can't compile FFmpeg under Windows.

Just use github actions pointing to your FFmpeg fork to compile. For that clone ​https://github.com/BtbN/FFmpeg-Builds enable actions and press build and publish a release. Do not forget to change the repo of FFmpeg.

Is this somewhere explained in more detail step by step? I don't understand anything of this github stuff.

[Balling]

Replying to Michael Koch:

Replying to Balling:

dshow.c, but I can't make a test because I can't compile FFmpeg under Windows.

Just use github actions pointing to your FFmpeg fork to compile. For that clone ​https://github.com/BtbN/FFmpeg-Builds enable actions and press build and publish a release. Do not forget to change the repo of FFmpeg.

Is this somewhere explained in more detail step by step? I don't understand anything of this github stuff.

No, that is what I use.

[Michael Koch]

A simple bugfix could be in line 70 of dshow.c to replace AV_PIX_FMT_RGB8 by AV_PIX_FMT_GRAY8. I know that it's a hack and the correct way is to use AV_PIX_FMT_PAL8 and read the palette.
But for this camera the hack should work because the palette does anyway contain only 256 shades of gray. Because nobody noticed this issue earlier, that probably means that there exist no color cameras that use the RGB8 pixel format with a non-grayscale palette.

[Balling]

hack should work

So you did not test it?

With GraphStudioNext the output is correct 8-bit grayscale

Are you sure it does not read palette?

Also can you paste all 1040 bytes of the palette here?

[Michael Koch]

Replying to Balling:

hack should work

So you did not test it?

As said before, I can't compile FFmpeg under Windows.

With GraphStudioNext the output is correct 8-bit grayscale

Are you sure it does not read palette?

I'm quite sure that GraphStudioNext does read the palette.

Also can you paste all 1040 bytes of the palette here?

00 00 00 00 01 01 01 00 02 02 02 00 03 03 03 00 04 04 04 00 05 05 05 00 06 06 06 00 07 07 07 00 08 08 08 00 09 09 09 00 0A 0A 0A 00 0B 0B 0B 00 0C 0C 0C 00 0D 0D 0D 00 0E 0E 0E 00 0F 0F 0F 00 10 10 10 00 11 11 11 00 12 12 12 00 13 13 13 00 14 14 14 00 15 15 15 00 16 16 16 00 17 17 17 00 18 18 18 00 19 19 19 00 1A 1A 1A 00 1B 1B 1B 00 1C 1C 1C 00 1D 1D 1D 00 1E 1E 1E 00 1F 1F 1F 00 20 20 20 00 21 21 21 00 22 22 22 00 23 23 23 00 24 24 24 00 25 25 25 00 26 26 26 00 27 27 27 00 28 28 28 00 29 29 29 00 2A 2A 2A 00 2B 2B 2B 00 2C 2C 2C 00 2D 2D 2D 00 2E 2E 2E 00 2F 2F 2F 00 30 30 30 00 31 31 31 00 32 32 32 00 33 33 33 00 34 34 34 00 35 35 35 00 36 36 36 00 37 37 37 00 38 38 38 00 39 39 39 00 3A 3A 3A 00 3B 3B 3B 00 3C 3C 3C 00 3D 3D 3D 00 3E 3E 3E 00 3F 3F 3F 00 40 40 40 00 41 41 41 00 42 42 42 00 43 43 43 00 44 44 44 00 45 45 45 00 46 46 46 00 47 47 47 00 48 48 48 00 49 49 49 00 4A 4A 4A 00 4B 4B 4B 00 4C 4C 4C 00 4D 4D 4D 00 4E 4E 4E 00 4F 4F 4F 00 50 50 50 00 51 51 51 00 52 52 52 00 53 53 53 00 54 54 54 00 55 55 55 00 56 56 56 00 57 57 57 00 58 58 58 00 59 59 59 00 5A 5A 5A 00 5B 5B 5B 00 5C 5C 5C 00 5D 5D 5D 00 5E 5E 5E 00 5F 5F 5F 00 60 60 60 00 61 61 61 00 62 62 62 00 63 63 63 00 64 64 64 00 65 65 65 00 66 66 66 00 67 67 67 00 68 68 68 00 69 69 69 00 6A 6A 6A 00 6B 6B 6B 00 6C 6C 6C 00 6D 6D 6D 00 6E 6E 6E 00 6F 6F 6F 00 70 70 70 00 71 71 71 00 72 72 72 00 73 73 73 00 74 74 74 00 75 75 75 00 76 76 76 00 77 77 77 00 78 78 78 00 79 79 79 00 7A 7A 7A 00 7B 7B 7B 00 7C 7C 7C 00 7D 7D 7D 00 7E 7E 7E 00 7F 7F 7F 00 80 80 80 00 81 81 81 00 82 82 82 00 83 83 83 00 84 84 84 00 85 85 85 00 86 86 86 00 87 87 87 00 88 88 88 00 89 89 89 00 8A 8A 8A 00 8B 8B 8B 00 8C 8C 8C 00 8D 8D 8D 00 8E 8E 8E 00 8F 8F 8F 00 90 90 90 00 91 91 91 00 92 92 92 00 93 93 93 00 94 94 94 00 95 95 95 00 96 96 96 00 97 97 97 00 98 98 98 00 99 99 99 00 9A 9A 9A 00 9B 9B 9B 00 9C 9C 9C 00 9D 9D 9D 00 9E 9E 9E 00 9F 9F 9F 00 A0 A0 A0 00 A1 A1 A1 00 A2 A2 A2 00 A3 A3 A3 00 A4 A4 A4 00 A5 A5 A5 00 A6 A6 A6 00 A7 A7 A7 00 A8 A8 A8 00 A9 A9 A9 00 AA AA AA 00 AB AB AB 00 AC AC AC 00 AD AD AD 00 AE AE AE 00 AF AF AF 00 B0 B0 B0 00 B1 B1 B1 00 B2 B2 B2 00 B3 B3 B3 00 B4 B4 B4 00 B5 B5 B5 00 B6 B6 B6 00 B7 B7 B7 00 B8 B8 B8 00 B9 B9 B9 00 BA BA BA 00 BB BB BB 00 BC BC BC 00 BD BD BD 00 BE BE BE 00 BF BF BF 00 C0 C0 C0 00 C1 C1 C1 00 C2 C2 C2 00 C3 C3 C3 00 C4 C4 C4 00 C5 C5 C5 00 C6 C6 C6 00 C7 C7 C7 00 C8 C8 C8 00 C9 C9 C9 00 CA CA CA 00 CB CB CB 00 CC CC CC 00 CD CD CD 00 CE CE CE 00 CF CF CF 00 D0 D0 D0 00 D1 D1 D1 00 D2 D2 D2 00 D3 D3 D3 00 D4 D4 D4 00 D5 D5 D5 00 D6 D6 D6 00 D7 D7 D7 00 D8 D8 D8 00 D9 D9 D9 00 DA DA DA 00 DB DB DB 00 DC DC DC 00 DD DD DD 00 DE DE DE 00 DF DF DF 00 E0 E0 E0 00 E1 E1 E1 00 E2 E2 E2 00 E3 E3 E3 00 E4 E4 E4 00 E5 E5 E5 00 E6 E6 E6 00 E7 E7 E7 00 E8 E8 E8 00 E9 E9 E9 00 EA EA EA 00 EB EB EB 00 EC EC EC 00 ED ED ED 00 EE EE EE 00 EF EF EF 00 F0 F0 F0 00 F1 F1 F1 00 F2 F2 F2 00 F3 F3 F3 00 F4 F4 F4 00 F5 F5 F5 00 F6 F6 F6 00 F7 F7 F7 00 F8 F8 F8 00 F9 F9 F9 00 FA FA FA 00 FB FB FB 00 FC FC FC 00 FD FD FD 00 FE FE FE 00 FF FF FF 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

[Balling]

Okay, so palette AF AF AF 00 is R, G, B, A. Hm...

[Michael Koch]

Replying to Balling:

Okay, so palette AF AF AF 00 is R, G, B, A. Hm...

"A" makes no sense for a camera. I think the 4th byte is always 00.

[Balling]

Replying to Michael Koch:

Replying to Balling:

Okay, so palette AF AF AF 00 is R, G, B, A. Hm...

"A" makes no sense for a camera. I think the 4th byte is always 00.

It is RGBQUAD, i.e. it has 4 bytes, where 4th is "Alpha or Don't Care".