LibTiff Mailing List

Folks,

In a recent performance analysis of the MapServer application with TIFF
files with overviews (each as a distinct directory) it was determinated that
a disproportionate amount of time was being spent loading the arrays of
offsets and sizes of tiles while reading directories.

In order to establish what overviews are available the applications reads
through all the directories in the file to get the sizes.  The file in
question is (at base) 131000 x 150000 and has nearly 900000 tiles.  With
64bit offsets and sizes that is around 14MB of reading just to get the
offsets and sizes for the tiles.

In order to optimize my application, I have incorporated a new feature into
libtiff (CVS head / aka libtiff 4.0).  This is a compile-time option to
defer loading the strip/tile offsets and sizes until such time as the are
actually needed to access imagery data.

This feature is enabled with the --enable-defer-strile-load configure
option which enables the DEFER_STRILE_LOAD macro.  Appropriate code is
surrounded by #if defined(DEFER_STRILE_LOAD) and it is my hope that there
is no risk when the feature is disabled.

When enabled, any part of libtiff that might need to use the offset/size
arrays from the directory call _TIFFFillStriles(TIFF*) in advance to ensure
the values are loaded.

This has dramatically improved performance for my particular application
in one use context.  I also think it could be helpful to others - particularly
when dealing with files that have:

a) a lot of tiles or strips - ie. large files.
b) multiple image directories
c) only needing to grab a bit of image data from one of the directories.

The change is now in libtiff CVS head.  I consider it experimental and
a bit risky so it is off by default, though if we get comfortable with it
at some point in the future it could be changed to always-on.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent


AWARE [SYSTEMS]		Imaging expertise for the Delphi developer
		TIFF and LibTiff Mailing List Archive
	LibTiff Mailing List TIFF and LibTiff Mailing List Archive February 2011 Previous Thread Next Thread Previous by Thread Next by Thread Previous by Date Next by Date Contact The TIFF Mailing List Homepage This list is run by Frank Warmerdam Archive maintained by AWare Systems	2011.02.18 22:06 "Deferring loading of strip/tile offsets and sizes", by Frank Warmerdam Folks, In a recent performance analysis of the MapServer application with TIFF files with overviews (each as a distinct directory) it was determinated that a disproportionate amount of time was being spent loading the arrays of offsets and sizes of tiles while reading directories. In order to establish what overviews are available the applications reads through all the directories in the file to get the sizes. The file in question is (at base) 131000 x 150000 and has nearly 900000 tiles. With 64bit offsets and sizes that is around 14MB of reading just to get the offsets and sizes for the tiles. In order to optimize my application, I have incorporated a new feature into libtiff (CVS head / aka libtiff 4.0). This is a compile-time option to defer loading the strip/tile offsets and sizes until such time as the are actually needed to access imagery data. This feature is enabled with the --enable-defer-strile-load configure option which enables the DEFER_STRILE_LOAD macro. Appropriate code is surrounded by #if defined(DEFER_STRILE_LOAD) and it is my hope that there is no risk when the feature is disabled. When enabled, any part of libtiff that might need to use the offset/size arrays from the directory call _TIFFFillStriles(TIFF*) in advance to ensure the values are loaded. This has dramatically improved performance for my particular application in one use context. I also think it could be helpful to others - particularly when dealing with files that have: a) a lot of tiles or strips - ie. large files. b) multiple image directories c) only needing to grab a bit of image data from one of the directories. The change is now in libtiff CVS head. I consider it experimental and a bit risky so it is off by default, though if we get comfortable with it at some point in the future it could be changed to always-on. Best regards, -- ---------------------------------------+-------------------------------------- I set the clouds in motion - turn up \| Frank Warmerdam, warmerdam@pobox.com light and sound - activate the windows \| http://pobox.com/~warmerdam and watch the world go round - Rush \| Geospatial Programmer for Rent

LibTiff Mailing List

2011.02.18 22:06 "Deferring loading of strip/tile offsets and sizes", by Frank Warmerdam