| 295 | | **Description:** The recent libswscale rewrite is transforming the library into a flexible and reliable architecture to handle scaling, with the logic being decoupled from the DSP code. This simplifies writing backends with all the benefits and deterministic behaviour of the reference C implementation. In order to achieve the goal of deprecating and phasing out the "legacy" code paths from FFmpeg, the new architecture must not introduce performance regressions for platforms with widespread use, such as x86_64 and AArch64, for Linux, macOS, and Windows. This project will extend the new libswscale architecture by implementing an AArch64 NEON backend, ensuring that ARM-based platforms (including mobile devices, servers, and single-board computers) benefit from the same deterministic and optimized pipeline as x86_64. This backend will be used to generate highly optimized NEON SIMD assembly code at runtime for platforms where just-in-time code execution (JIT) is supported, and it will also be used to generate static (non-JIT) assembly files that match the Continuation-Passing Style (CPS) code of the reference C and x86_64 implementations for platforms where JIT is not supported. |
| 296 | | |
| 297 | | **Expected results:** libswscale AArch64 NEON backend, both JIT and non-JIT, with feature parity to the reference C and x86_64 implementations. |
| | 295 | **Description:** The recent libswscale rewrite is transforming the library into a flexible and reliable architecture to perform pixel format conversion and image scaling. In order to allow transitioning away from the "legacy" code paths from FFmpeg, the new architecture must not introduce performance regressions for platforms with widespread use, such as x86_64 and AArch64, for Linux, macOS, and Windows. This project will extend the new libswscale architecture by implementing an AArch64 NEON backend, ensuring that ARM-based platforms (including mobile devices, servers, and single-board computers) benefit from the same optimized pipeline as x86_64. |
| | 296 | |
| | 297 | **Expected results:** libswscale AArch64 NEON backend, with feature parity to the reference C and x86_64 implementations. |
| 307 | | ==== Milestones |
| 308 | | |
| 309 | | ===== '''asmjit''' validation backend |
| 310 | | |
| 311 | | * **Description**: A JIT validation backend will be made using the '''asmjit''' library. This will allow us to validate the generated assembly code and obtain performance metrics. |
| 312 | | |
| 313 | | * **Deliverables**: An initial '''asmjit''' JIT validation backend in a separate git repository. An RFC with a performance report posted to the FFmpeg-devel mailing list. |
| 314 | | |
| 315 | | * **Compensation:** 6,000 EUR |
| 316 | | |
| 317 | | ===== Assembly generation abstraction layer |
| 318 | | |
| 319 | | * **Description**: The '''asmjit''' validation backend will be split in two parts: an abstract assembly generation layer (with no '''asmjit''' dependency), and the '''asmjit''' code generator based on the abstraction layer. This assembly generation layer should include a register allocator. It will then be possible to add other assembly output generators. |
| 320 | | |
| 321 | | * **Deliverables**: An update on the RFC on the FFmpeg-devel mailing list with the code split. |
| 322 | | |
| 323 | | * **Compensation:** 6,000 EUR |
| 324 | | |
| 325 | | ===== Static non-JIT backend |
| 326 | | |
| 327 | | * **Description**: Using the assembly abstraction layer and the AArch64 NEON SIMD code from the asmjit validation backend, it will be possible to generate static assembly files that match the CPS code of the reference C and x86_64 implementations, even though they come with a certain overhead compared to the JIT code. A tool (program or script) will be created to generate the static assembly files. This tool will be integrated into the FFmpeg codebase, and the static assembly files will be generated either at build-time or integrated into the FFmpeg codebase as well (to be determined). |
| 328 | | |
| 329 | | * **Deliverables**: Series merged to git master, effectively providing an initial AArch64 NEON backend for the new libswscale architecture. |
| 330 | | |
| 331 | | * **Compensation:** 6,000 EUR |
| 332 | | |
| 333 | | ===== Evaluate other JIT implementations |
| 334 | | |
| 335 | | * **Description**: The '''asmjit''' validation backend, after having been used to bootstrap the static non-JIT backend, will be revisited. Other JIT generation backends will be evaluated, with a focus on either finding or creating a framework that can be cleanly integrated with FFmpeg's C codebase. This dependency should be lightweight and easy to integrate into FFmpeg itself, guaranteeing the maintainability of the new codebase. |
| 336 | | |
| 337 | | * **Deliverables**: An RFC with a report on the findings posted to the FFmpeg-devel mailing list. The community will help select the best implementation. |
| 338 | | |
| 339 | | * **Compensation:** 6,000 EUR |
| 340 | | |
| 341 | | ===== Final JIT backend |
| 342 | | |
| 343 | | * **Description**: The final AArch64 NEON JIT backend will be integrated into the FFmpeg codebase, based on the implementation selected by the community. Support will be added incrementally for each platform (Linux, macOS, Windows), only after they are each properly tested an validated. This JIT implementation will considerably reduce the overhead compared to the non-JIT code, ensuring that there are no performance regressions between the legacy code paths and the new architecture for AArch64 platforms. |
| 344 | | |
| 345 | | * **Deliverables**: A pull request with the final AArch64 NEON JIT backend, is either merged or has received at least one merge approval. |
| 346 | | |
| 347 | | * **Compensation:** 6,000 EUR |
| | 307 | ==== Milestone 1: AArch64 NEON static (non-JIT) backend |
| | 308 | |
| | 309 | **Description**: Implement AArch64 NEON SIMD code that matches the reference C and x86_64 implementations of the new libswscale architecture. |
| | 310 | |
| | 311 | **Compensation:** 18,000 EUR |
| | 312 | |
| | 313 | **Time estimate:** 6 weeks = 240 hours (full-time) |
| | 314 | |
| | 315 | **Delivery date:** 3 months after kick-off |
| | 316 | |
| | 317 | **Deliverables in this milestone**: |
| | 318 | - Implement the kernels for the new libswscale architecture in AArch64 NEON SIMD. |
| | 319 | - Create a tool (program or script) to generate the static assembly files that match the Continuation-Passing Style (CPS) code of the reference C and x86_64 implementations. |
| | 320 | - Series merged to git master, effectively providing an initial AArch64 NEON backend for the new libswscale architecture. |
| | 321 | |
| | 322 | **Completion of this milestone will improve the project in the following way(s):** |
| | 323 | - The new libswscale will be performant enough to allow deprecating and phasing out the legacy code paths on ARM-based platforms. |
| | 324 | |
| | 325 | ==== Milestone 2: AArch64 NEON JIT backend |
| | 326 | |
| | 327 | **Description**: The overhead introduced from Continuation-Passing Style (CPS) code can be considerably reduced by generating just-in-time (JIT) assembly code instead of static assembly files. In this milestone, JIT support will be implemented building upon the previous non-JIT backend. |
| | 328 | |
| | 329 | **Compensation:** 18,000 EUR |
| | 330 | |
| | 331 | **Time estimate:** 6 weeks = 240 hours (full-time) |
| | 332 | |
| | 333 | **Delivery date:** 6 months after kick-off |
| | 334 | |
| | 335 | **Deliverables**: |
| | 336 | - An AArch64 NEON SIMD JIT generator, using the kernels previously implemented in Milestone 1. |
| | 337 | - Support will be added incrementally for each platform (Linux, macOS, Windows). |
| | 338 | - A pull request with the AArch64 NEON JIT backend, is either merged or has received at least one merge approval. |
| | 339 | |
| | 340 | **Completion of this milestone will improve the project in the following way(s):** |
| | 341 | - The new libswscale will outperform the legacy code paths on ARM-based platforms that support JIT. |
