2009.02.19 17:17 "[Tiff] Bug in tiff_jpeg.c dealing with pre-subsampled ycbcr data passed to TIFFWriteEncodedXxx", by

Folks,

I've found a bug passing pre-subsampled ycbcr data to TIFFWriteEncodedXxx when using JPEG compression (i.e. not using the troublesome JPEG_COLORMODE anomaly). The result looks like it's been in the blender. It seems that the bug occurs only when vertical subsampling factor is different from 1, horizontal subsampling factor does not seem to have any impact.

The exact same data and input sequence works fine if compression is NONE or FLATE, in that such files do decode back to the original input with LibTiff decoder, LibTiff RGBA decoder, and my proprietary TIFF decoder. The decoding result of the faulty files is consistent between LibTiff, LibTiff RGBA, and the various modes of my proprietary codec. Therefore, the problem must be in tif_jpeg.c, it cannot be a bug in the way the data is prepared prior to passing it to LibJpeg, nor can it be a bug in the decoding end of the encoded result.

I think I've tracked down the bug to JPEGEncodeRaw in tif_jpeg.c. Near the bottom of that procedure, it reads

buf+=sp->bytesperline;

I guess that is incorrect. The tif->tif_row and nrows are incremented/decremented by sp->v_sampling lines, correctly indicating that sp->v_sampling lines are being dealt with per iteration of the main loop in this procedure. But, sp->bytesperline does not reflect sp->v_sampling lines, rather it reflects a single line in as far as such a concept exists for subsampled data. Near the top of the procedure, bytesperclumpline is calculated, and that does accurately reflect the size of sp->v_sampling lines in buf. Therefore, the line should change to

buf+=bytesperclumpline;

This fix seems to totally cure the problem on my end. It seems to make sense and correspond to the buggy results I was getting, suggesting that this was the exactly problem. Indeed, if vertical subsampling factor is 1 and only in that case, sp->v_sampling is 1, and sp->bytesperline equals bytesperclumpline.

Frank, are you comfortable with this fix? It seems very unlikely any calling code can be depending on this bug. Anyone else have any remarks? How about I renew my CVS write access, so that I can do this this fix and others I proposed recently, myself, directly, again?

Best regards,

Joris