LibTiff Mailing List

TIFF and LibTiff Mailing List Archive
February 2009

Previous Thread
Next Thread

Previous by Thread
Next by Thread

Previous by Date
Next by Date

The TIFF Mailing List Homepage
This list is run by Frank Warmerdam
Archive maintained by AWare Systems

2009.02.19 17:17 "Bug in tiff_jpeg.c dealing with pre-subsampled ycbcr data passed to TIFFWriteEncodedXxx", by Joris Van Damme

Folks,

I've found a bug passing pre-subsampled ycbcr data to TIFFWriteEncodedXxx 
when using JPEG compression (i.e. not using the troublesome JPEG_COLORMODE 
anomaly). The result looks like it's been in the blender. It seems that the 
bug occurs only when vertical subsampling factor is different from 1, 
horizontal subsampling factor does not seem to have any impact.

The exact same data and input sequence works fine if compression is NONE or 
FLATE, in that such files do decode back to the original input with LibTiff 
decoder, LibTiff RGBA decoder, and my proprietary TIFF decoder. The decoding 
result of the faulty files is consistent between LibTiff, LibTiff RGBA, and 
the various modes of my proprietary codec. Therefore, the problem must be in 
tif_jpeg.c, it cannot be a bug in the way the data is prepared prior to 
passing it to LibJpeg, nor can it be a bug in the decoding end of the 
encoded result.

I think I've tracked down the bug to JPEGEncodeRaw in tif_jpeg.c. Near the 
bottom of that procedure, it reads

buf+=sp->bytesperline;

I guess that is incorrect. The tif->tif_row and nrows are 
incremented/decremented by sp->v_sampling lines, correctly indicating that 
sp->v_sampling lines are being dealt with per iteration of the main loop in 
this procedure. But, sp->bytesperline does not reflect sp->v_sampling lines, 
rather it reflects a single line in as far as such a concept exists for 
subsampled data. Near the top of the procedure, bytesperclumpline is 
calculated, and that does accurately reflect the size of sp->v_sampling 
lines in buf. Therefore, the line should change to

buf+=bytesperclumpline;

This fix seems to totally cure the problem on my end. It seems to make sense 
and correspond to the buggy results I was getting, suggesting that this was 
the exactly problem. Indeed, if vertical subsampling factor is 1 and only in 
that case, sp->v_sampling is 1, and sp->bytesperline equals 
bytesperclumpline.

Frank, are you comfortable with this fix? It seems very unlikely any calling 
code can be depending on this bug. Anyone else have any remarks? How about I 
renew my CVS write access, so that I can do this this fix and others I 
proposed recently, myself, directly, again?


Best regards,

Joris