2018.01.24 00:33 "Re: [Tiff] Strategies for multi-core speedups", by Larry Gritz

Update: I have more or less final results, having implemented my scheme for both reading and writing, both scanline and tiled images. And also refined the output technique it a bit more since my preliminary report.

As an example of the performance gain, one use case I come across frequently is in our "texture creation", wherein we take a source scanline TIFF file, read it, compute a bunch of image statistics on the pixels, then output it tiled and multi-resolution (successively halving the resolution and appending it to make a multi-subimage file). The filtered resize that generates the successive lower-res images was already parallelized in my implementation, so by far the biggest portion of runtime was writing the output, and my intuition was that the majority of that was the deflate compression, which has no good reason to be serialized.

Previously, doing this on a TIFF file of 8192 x 8192 x 4 channel x 16 bit unsigned integer values took 28.2s total, of which 1.13s was reading the source image, and 23.3s writing all the subimages of the result.

With my changes, on my 16 core workstation (32 hyperthreads), the same operation takes 1.6s total (0.13s read, 1.0s write).

(Those times also incorporate the fact that I found another item on my side of the code, accounting for ~3 seconds of the original total, that I'd failed to parallelize, so I did that, too.)

That's a particularly good case. For some other use cases (different operations, similar amount of I/O), I'm only getting 6-10x improvement.

> On Jan 16, 2018, at 6:00 PM, Larry Gritz <lg@larrygritz.com <mailto:lg@larrygritz.com>> wrote:

Status report: I've implemented this, thanks to Scott Ribe's suggestion of not trying to struggle with libtiff's codec internals, but just calling zlib directly, then TIFFWriteRawStrip (this strategy only works for deflate compression, but that's my common case anyway). The strips' data are compressed in parallel.

I'm getting a 6x improvement in total time for writing large TIFF files.

--
Larry Gritz

lg@larrygritz.com <mailto:lg@larrygritz.com>