prores_aw : Encode Prores 4444 with alpha fail in high resolution

[mvignali]

Summary of the bug:
Following bug report request by blender project

Prores_aw fail to encode high resolution with alpha
Issue probably similar to ​https://trac.ffmpeg.org/ticket/9173#ticket

How to reproduce:

./ffmpeg -f lavfi -i "color=c=red:size=5500x3500" -t 1 -pix_fmt yuva444p10le -vcodec prores_aw test.mov

./ffmpeg -f lavfi -i "color=c=red:size=5500x3500" -t 1 -pix_fmt yuva444p10le -vcodec prores_aw test.mov
ffmpeg version N-119106-gc1a2da72cc Copyright (c) 2000-2025 the FFmpeg developers
  built with Apple clang version 16.0.0 (clang-1600.0.26.6)
  configuration: --prefix=../build
  libavutil      60.  0.100 / 60.  0.100
  libavcodec     62.  0.100 / 62.  0.100
  libavformat    62.  0.100 / 62.  0.100
  libavdevice    62.  0.100 / 62.  0.100
  libavfilter    11.  0.100 / 11.  0.100
  libswscale      9.  0.100 /  9.  0.100
  libswresample   6.  0.100 /  6.  0.100
Input #0, lavfi, from 'color=c=red:size=5500x3500':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, yuv420p, 5500x3500 [SAR 1:1 DAR 11:7], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> prores (prores_aw))
Press [q] to stop, [?] for help
[prores_aw @ 0x7fb584907700] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb584908000] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb5849088c0] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb5849091c0] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb584909ac0] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490a3c0] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490ac80] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490b580] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490be80] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490c780] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490d040] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490d940] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490e240] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490eb40] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490f400] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb58490fd00] encoding with ProRes 4444+ (ap4h) profile
[prores_aw @ 0x7fb584e05d40] encoding with ProRes 4444+ (ap4h) profile
Output #0, mov, to 'test.mov':
  Metadata:
    encoder         : Lavf62.0.100
  Stream #0:0: Video: prores (4444) (ap4h / 0x68347061), yuva444p10le(tv, progressive), 5500x3500 [SAR 1:1 DAR 11:7], q=2-31, 200 kb/s, 25 fps, 12800 tbn
    Metadata:
      encoder         : Lavc62.0.100 prores_aw
[prores_aw @ 0x7fb584907700] Underestimated required buffer size.
[vost#0:0/prores_aw @ 0x7fb584e05580] [enc:prores_aw @ 0x7fb584e05c80] Error submitting video frame to the encoder
[vost#0:0/prores_aw @ 0x7fb584e05580] [enc:prores_aw @ 0x7fb584e05c80] Error encoding a frame: Internal bug, should not have happened
[vost#0:0/prores_aw @ 0x7fb584e05580] Task finished with error code: -558323010 (Internal bug, should not have happened)
[vost#0:0/prores_aw @ 0x7fb584e05580] Terminating thread with return code -558323010 (Internal bug, should not have happened)
[out#0/mov @ 0x7fb584e04500] video:0KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
[out#0/mov @ 0x7fb584e04500] Output file is empty, nothing was encoded(check -ss / -t / -frames parameters if used)
frame=    0 fps=0.0 q=0.0 Lsize=       0KiB time=N/A bitrate=N/A speed=N/A    
[prores_aw @ 0x7fb584908000] Underestimated required buffer size.
[prores_aw @ 0x7fb5849088c0] Underestimated required buffer size.
Conversion failed!