FFMPEG crashes when trying to convert from PIX_FMT_YUV420P for specfic size of images

[Kiran Kumar]

Summary of the bug:

Hi I am trying to convert frames from PIX_FMT_YUYV422 to PIX_FMT_YUV420P, for some resolution it crashes, but works for other resolutions of frame.

We have compiled Windows Version using instruction from ​http://ffmpeg.zeranoe.com, and are using the latest build.

I managed to debug with windows using MinGW and used GDB, & below isthe stack when it crashes.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1792.0x14fc]
(gdb) bt
#0 0x6ab246f7 in extract_odd2avg_MMXEXT (count=-44, dst1=0x1c5c502c <Address 0x1c5c502c out of bounds>, dst0=0x1c5af10c "",

src1=0x1db90761 "}"%"\200"\210"|c+c|"+ª{ª+"\200"+«~«ƒ¬x«,«{«,¬{¬+¬{¬,-\177¬ƒ-ƒ¬\177¬\200r|rƒ-xrƒ_{r,r,-|¬,-|¬,¬{r,_{-._vr._{_,_}°,_}rƒ±{_,_{r._|_\200_\177°ƒ_}±,r}_.r~r\201_|°ƒ_{_ƒ_{_._|°._~r.-

{r\201-|_\200_{r,-}_,r~-.rzrƒ-y-,-{-\201-}¬\200«{«,«z«.c{¬,¬"...,

src0=0x1db90411 "~".§~c,c}c+"xc,c}«+«\177«+c\177«+ª{-,-}-,¬|r+rx-ƒ-{r.-|_,r}-ƒ_|r.-{_,_|_.-zr.¬}«,-|¬,«\201_{_,_{_ƒ_|r,°\177°ƒ°{°.±{°._}°,_}°\200°z_,_{_,_|°ƒ_\200°,°~°ƒ°~°ƒr|_\201r|_,r\200r,r{r\

201r{r\200-zr,r\177_\200-~¬,-{_,r{-\201ry-ƒ-y«ƒ¬{¬,¬z¬+¬|¬+¬"...) at libswscale/x86/rgb2rgb_template.c:2348
#1 yuyvtoyuv420_MMXEXT (ydst=<optimized out>,

udst=0x1c5af038 "

|z}|\177|~}x{}|{y{z

{{zy}{y{}{{}\177}~|{\177{z{~}{{zxzz|{yzzy{zy|zxzzyzzy~z

}zz{|}|xx{ywwxzzxxuyyzwxxxwxvyywzyyxwyzyxxzzzzy{y~wwzyx}yxyxzy}vxzwxyy{|wzyxy{{xy{yzz{}{|}xxyxwwy{"

,
vdst=0x1c5c4f58 ".,+,+++,ƒ,ƒ+_.,.ƒ\201\201_ƒ\201\200,ƒ,ƒƒƒ,\201\201\201\201_,\201ƒ,._.....+.,.,ƒ,ƒ,\201..++++\210<SSS\210++S<+SS<\210SO\215\210<\210<<<%<<\210%<\210\210\210+S%OS%%\210\
210%O%\210%SZS%SS%<%<<SSSO\215<O%SSSS%<OS\210S%O%S<S%+%%SS\210SOS%S" <Address 0x1c5c5000 out of bounds>, src=<optimized out>, width=424, height=240, lumStride=424, chromStride=212, srcStride=848)
at libswscale/x86/rgb2rgb_template.c:2363
#2 0x6ab1c25a in yuyvToYuv420Wrapper (c=0x9f3a470, src=0x1d99fb60, srcStride=0x1d99fb80, srcSliceY=0, srcSliceH=240, dstParam=0x1d99fb70, dstStride=0x1d99fb90)
at libswscale/swscale_unscaled.c:235
#3 0x6ab192f7 in sws_scale (c=0x9f3a470, srcSlice=0x1d99fbf4, srcStride=0x1d99fbfc, srcSliceY=0, srcSliceH=240, dst=0x1d99fbe8, dstStride=0x1d99fbdc)
at libswscale/swscale.c:928

How to reproduce:

Code to reproduce the issue is :

BOOL ConvertFromYUY2ToI420(BYTE* dataIn, DWORD inWidth, DWORD inHeight, DWORD InLen)
{

struct SwsContext *img_convert_ctx = NULL;


uint8_t *downSample = new BYTE[InLen]; //new BYTE[InLen];
if (!downSample)
{
return FALSE;
}


memset(downSample, 0 , InLen);

int outstride[3];

int instride[2] ;
instride[0] = inWidth *2;
instride[1] = 0;





uint8_t *buf[2];
uint8_t *out[2];

out[0] = (uint8_t *) downSample;
out[1] = (uint8_t *) (downSample + (inWidth * inWidth)/2);
out[2] = (uint8_t *) (out[1] + ((inWidth * inWidth)/2));

//Strides
outstride[0] = inWidth;
outstride[1] = inWidth/2;
//outstride[2] = inWidth/4;



buf[0] = (uint8_t *) dataIn;


if(img_convert_ctx == NULL) {

img_convert_ctx = sws_getContext(inWidth, inHeight,
PIX_FMT_YUYV422,
inWidth,
inHeight,
PIX_FMT_YUV420P,
SWS_BICUBIC,
NULL,
NULL,
NULL);
}
if(img_convert_ctx == NULL) {
return FALSE;
}

sws_scale(img_convert_ctx,
buf,
instride,
0,
inHeight,
out,
outstride);

memset(dataIn, 0, InLen);

CopyMemory(dataIn, out[0], inWidth *inHeight);
CopyMemory((dataIn + (inWidth *inHeight)), out[1], (inWidth *inHeight/4));
CopyMemory( (dataIn + (inWidth *inHeight) + ((inWidth * inHeight)/4) ) , out[2], ((inWidth * inHeight)/4) );

sws_freeContext(img_convert_ctx);
img_convert_ctx = NULL;

if (downSample)
{
delete[] downSample;
downSample = NULL;
}

return TRUE;

}


The above code works for all Width X Height expect when the input resoultion is the fallowing combinaton :
424 x 240
800 x 448
960 x 544
1280 x 720


ffmpeg version : 0.11.0
built on: Windows

Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker.

[Hari]

Tried to reproduce the issue using test application libswscale/swscale-test. It was executed as follows in Ubuntu:
make libswscale/swscale-test
./libswscale/swscale-test -src yuyv422 -dst yuv420p

Didnt see any crash for the above mentioned resolutions.

Have you tried verifying the erroneous resolutions using swscale-test test app?

[Carl Eugen Hoyos]

Your code does not compile here, please add a c file as attachment, ideally with compilation instructions.

[Carl Eugen Hoyos]

Please reopen this ticket if you can add information on how this can be reproduced.