2011.02.18 22:06 "[Tiff] Deferring loading of strip/tile offsets and sizes", by Frank Warmerdam

Folks,

In a recent performance analysis of the MapServer application with TIFF files with overviews (each as a distinct directory) it was determinated that a disproportionate amount of time was being spent loading the arrays of offsets and sizes of tiles while reading directories.

In order to establish what overviews are available the applications reads through all the directories in the file to get the sizes. The file in question is (at base) 131000 x 150000 and has nearly 900000 tiles. With 64bit offsets and sizes that is around 14MB of reading just to get the offsets and sizes for the tiles.

In order to optimize my application, I have incorporated a new feature into libtiff (CVS head / aka libtiff 4.0). This is a compile-time option to defer loading the strip/tile offsets and sizes until such time as the are actually needed to access imagery data.

This feature is enabled with the --enable-defer-strile-load configure option which enables the DEFER_STRILE_LOAD macro. Appropriate code is surrounded by #if defined(DEFER_STRILE_LOAD) and it is my hope that there is no risk when the feature is disabled.

When enabled, any part of libtiff that might need to use the offset/size arrays from the directory call _TIFFFillStriles(TIFF*) in advance to ensure the values are loaded.

This has dramatically improved performance for my particular application in one use context. I also think it could be helpful to others - particularly when dealing with files that have:

  1. a lot of tiles or strips - ie. large files.
  2. multiple image directories
  3. only needing to grab a bit of image data from one of the directories.

The change is now in libtiff CVS head. I consider it experimental and a bit risky so it is off by default, though if we get comfortable with it at some point in the future it could be changed to always-on.

Best regards,
--

---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent