avcodec/dvbsubdec: Return proper error codes from more functions
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>