When decoding a multilayer DWAA compressed .exr, I get error decode_block()

[Sean Devonport]

Summary of the bug:

When decoding a multilayer DWAA compressed .exr, the DWAA decompression in ffmpeg doesn't take into account multilayer. It returns decode_block() failed.

I've search the code base and see the issue is happening dwa_uncompress function in lavc/exr exr.c line 200 (SHA 53d0f9afb46ac811269252c9e3be000fc7c3b2cc in git-master). When trying to add in support for multilayer, I am able to get close to a solution, however there is still banding and it is not quite right.

I have a attached source multilayer DWAA compressed files to test with as well as the output of the video with my attempted bug fix. Any help or guidance to fix this would be greatly appreciated!

How to reproduce:

% ./ffmpeg_g -layer Composite.Combined -i "./outputRGBMultilayerEXR1080p/frame_%04d.exr" -profile:v 2 -pix_fmt yuv422p10le -c:v prores_ks out.mov
ffmpeg version N-116393-g001df47a56
built with Apple clang version 15.0.0 (clang-1500.3.9.4)

[Sean Devonport]

I was unable to attach files in Trac so have added them as a gdrive link here

​https://drive.google.com/drive/folders/1WptOFDufzO0zaECJlozo1LhLsyR3SaBW?usp=sharing

[Sean Devonport]

ffmpeg log that results in decode_block() failed

[Balling]

Doeds Davinci decode correct? But yes, ​https://github.com/AcademySoftwareFoundation/openexr is reference

[Sean Devonport]

Yes, Davinci does seem to be able to decode and I can select the various layers. Any ideas of how to get it to work for ffmpeg would be great. I'll check out the reference against ffmpeg's implementation

[Balling]

Yes, I see that davinci decodes with layers available in Fusion Panel of Davinci. But are the strange pixels expected? Is it linear transfer BTW,

[Sean Devonport]

Ah yes, sorry, the strange pixels in the example are from a non-denoised, low sampled path traced image. I did that for testing speed. I can rerender with a correctly denoised image if it helps.

It is linear Rec 709 from blender.

I've been digging through the exr.c code and I think I can see where the root of the problem is.

It seems that ExrThreadData has DCT block array defined by [3][64]. So that's 3 channel I would assume.

typedef struct EXRThreadData {
    uint8_t *uncompressed_data;
    int uncompressed_size;

    uint8_t *tmp;
    int tmp_size;

    uint8_t *bitmap;
    uint16_t *lut;

    uint8_t *ac_data;
    unsigned ac_size;

    uint8_t *dc_data;
    unsigned dc_size;

    uint8_t *rle_data;
    unsigned rle_size;

    uint8_t *rle_raw_data;
    unsigned rle_raw_size;

    float block[3][64];    // this would need to be [14][64] for my example

    int ysize, xsize;

    int channel_line_size;

    int run_sym;
    HuffEntry *he;
    uint64_t *freq;
    VLC vlc;
} EXRThreadData;

As well as this, the dwa_uncompress function only has support for 3 channels, and the loop that does the DCT decoding is only doing it for 3 channels, and filling the RGB pixels as 3, but in my example it would need 14 channels (that's 2x layers that are RGB and 2x layers that are RGBA).

I see that s->nb_channels = 14 so I would imagine we need to run the inverse DCT on all 14 channels, and then for each layer, take the decoded DCT block and fill in the pixels. I've tried doing this but it's still not quite working. I get missing information in the final pixels.

// dwa_uncompress function in exr.c
            for (int j = 0; j < 3; j++) // This should be s->nb_channels which is 14 in my case
            {
                float *block = td->block[j];
                const int idx = (x >> 3) + (y >> 3) * dc_w + dc_w * dc_h * j;
                uint16_t *dc = (uint16_t *)td->dc_data;
                union av_intfloat32 dc_val;

                dc_val.i = half2float(dc[idx], &s->h2f_tables);

                block[0] = dc_val.f;
                ac_uncompress(s, &agb, block);
                dct_inverse(block);
            }