2009.12.07 18:01 "Re: [Tiff] Full strip allocation in TIFFWriteScanline", by Frank Warmerdam

TIFFWriteScanline always allocates memory for a complete strip. When writing large TIFF files in a single strip, this can lead to memory problems quickly.

I am aware that the whole point of dividing the image data into strips is to be able to read and write in manageable chunks of memory. However, some applications (e.g. many fax applications) seem to have problems with TIFF images with more than a single strip.

I am aware of libtiff's strip-based API, which is capable of writing strip data incrementally with TIFFWriteRawStrip. Unfortunately, it expects already compressed data and is therefore not making use of the built-in compressors. TIFFWriteEncodedStrip on the other hand compresses data prior to writing it, but it doesn't work incrementally, i.e. it expects the full data of each strip.

I assume that this is because libtiff's implementation of the compression algorithms cannot operate on streamed data? I think it'd make sense to support this, as the data which needs to be buffered by most compression schemes should be substantially less (and never more) than the smallest strip size feasible for each algorithm.

What are you general thoughts on changing the compression implementations in this regard? Do you think it is doable or am I overlooking something? Is somebody already working on it or planning to?

Lars,

I inspected the tif_packbits.c code (because it is one of the simplier encoders) and it *appears* to me that it ought to support writing a scanline at a time without buffering the whole output strip. I PackBitsEncode appears to be structured to handle a scanline at a time and it calls TIFFFlushData1() periodically - presumably to flush out the data compressed so far.

I haven't looked any further, and I must confess to some confusion about the assumptions and expectations for encoders and decoders despite having fiddled with them for years. But it seems to me that at least some encoders support scanline oriented encoding and writing to disk. So if the fax encoders do not, we can consider it a task local to the fax encoder to improve it to do the same.

Of course, I may be missing lots of things.

Best regards,
--
---------------------------------------+--------------------------------------

I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam

and watch the world go round - Rush    | Geospatial Programmer for Rent