2004.03.15 02:56 "Re: [Tiff] Tiff question", by Joris Van Damme
I have a question. I have huge TIFF files and with my software I cant handle them, they are simply too large. I am a programmer as well, is it possible to cretae something to handle those files.
There's two distinct parts to this problem: the file, and the internal decoded raster (object) representation of your imaging library. First part doesn't really matter that much, though, of course it's simply sensible to store such huge TIFF pages tiled. It's mainly the second part that is actually difficult, and that really matters.
The problem with allocating/managing a single internal raster representation that big is that one single continuous chunk of memory is needed that is way too big for comfort. Such a chunck is a serious headache to the operating system, if allocating it is at all possible, and causes constant swapping activity. That is why this problem is best solved by splitting up this raster into more managable parts. You could split it up in bands or in tiles. Bands are pieces of same width as total raster, but limited height. Tiles are square or almost square regions. Of these two techniques, tiling comes most recommended. Here's a comparison of both techniques, that may also be helpfull in giving you a feel of why exactly big images are a problem and why exactly splitting the raster is a solution:
http://www.awaresystems.be/techtalk/techtalks/002_tiling_vs_banding.html
A good tiling solution can handle rasters of the size you've mentioned. It's best implemented in a raster object, managing an array of tiles, that are in turn arrays of pixel color data, solving your need from big images to the huge ones from that point on. Such a raster object is best designed to hide this complication to the world of code outside of the object. Thing is, of course, this raster does not to be read or exposed at some stage. That need in particular, I find, is best solved by a dedicated 'scanline stream object', that kinda 'serves' the raster, line by line. Such a scanline stream object should be viewed as a chain of individual steps. Next, it's only a matter of designing the actual TIFF encoder/decoder to work as a step in such a chain, for instance. This way, you can easily and independently solve stuff like putting together TIFF strips/tiles of any size or decoding a single-strip TIFF on the one hand, or writing out the result, line by line, into your raster tile array object, or resampling, buffering only the minimum amount of lines needed (eg 2 in a linear interpolation resampler), or filtering, or... etc. As all of these steps become implemented as single independent steps you can link together in a 'scanline stream' chain, reading and writing from and to tile array rasters, you've build yourself an imaging library that withstands the big image problem.
This whole thing really is quite a bit of work. I can tell you that it's not a dead end, but a definite solution, but I cannot guarantee that it's actually worth the effort if you percieve this as an isolated need. In that case, hacking together something that reads such a TIFF tile by tile, writing out each tile to a destination TIFF file, as soon it is processed, and forgetting about it before reading in the next tile, for example, may be more worth your while. Of course, such a quick hack is no longer 'universal', depends on particular properties of the particular files and the particular task and such, which may perhaps not be a problem.
Joris Van Damme
info@awaresystems.be
http://www.awaresystems.be
Download your free TIFF tag viewer for windows here:
http://www.awaresystems.be/free_downloads/astifftools.html