2016.01.25 18:25 "[Tiff] OpenMP enabled libtiff", by Aaron Boxer

2016.01.26 09:52 "Re: [Tiff] OpenMP enabled libtiff", by Even Rouault

Le mardi 26 janvier 2016 10:36:16, Mat Maher a écrit:

Hello List,

Is anyone interested in having OpenMP support in libtiff?

Is this feasible? planning on adding this feature for another I am project I am working on, so I am trying to gauge interest.

libTiff performance is governed by two factors:

  1. . Performance of disk read/writes
  2. . Performance of compression/decompression routines (eg CPU)

The latter would benefit enormously from multi-threaded implementation, BUT, will ultimately be throttled by the former.

There are a few simple things which could be done to improve matters:

  1. . Buffer the entire file in memory for processing. HOWEVER, this would only work with a user switch input, as it's always possible the application TIFFs would be larger than available memory. 'Blasting' an entire file to/from disk maximises the transfer rate due to seek-times and DMA-setup latencies. In this scenario, you will end up reading/writing at the line-rate of the disk: which about as fast as you can get (and actually, for write operations, shows SATA as the faster medium for file sizes > SSD cache)
  2. . Implement thread-per-strip CPU processing.

For these changes, I actually wouldn't use OpenMP. The TIFF strip arrangement is a perfect division for multi-threading utilisation. Trying to multi-thread at a lower granularity than this is likely to yield lower performance due to the compression/decompression libraries which are largely sequential in operation.

For what is worth, I've experimented some time ago in the GDAL TIFF driver, multi-treaded compression. This works like this:

  1. Uncompressed strips/tiles to write are sent to a queue
  2. Worker threads fetch that uncompressed data from the queue, creates a in- memory TIFF file of the dimension of the tile/strip with the desired compression options, and extract the resulting compressed data from the writen in-memory TIFF file.
  3. The main thread collects the compressed data and write it to the main TIFF file with TIFFWriteRawTile / TIFFWriteRawStrip
  4. This gives performance boost on slow compression methods such as Deflate.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com