avcodec/vmnc: Simplify "24bit" support
This also makes the code more robust, removing potential out of array writes.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>