AWARE SYSTEMS
TIFF and LibTiff Mail List Archive

Thread

2017.11.20 17:04 "[Tiff] TIFF tag and compression registration", by Kemp Watson
2017.11.20 17:34 "Re: [Tiff] TIFF tag and compression registration", by Bob Friesenhahn
2017.11.20 21:01 "Re: [Tiff] TIFF tag and compression registration", by Kemp Watson
2018.01.16 20:08 "Re: [Tiff] Strategies for multi-core speedups", by
2018.01.16 00:15 "Re: [Tiff] Strategies for multi-core speedups", by Bob Friesenhahn
2018.01.15 21:41 "Re: [Tiff] Strategies for multi-core speedups", by Larry Gritz
2018.01.15 19:17 "[Tiff] Strategies for multi-core speedups", by Larry Gritz
2018.01.15 20:23 "Re: [Tiff] Strategies for multi-core speedups", by Bob Friesenhahn
2018.01.15 22:02 "Re: [Tiff] Strategies for multi-core speedups", by Olivier Paquet
2018.01.15 22:47 "Re: [Tiff] Strategies for multi-core speedups", by Craig Bruce
2018.01.15 22:38 "Re: [Tiff] Strategies for multi-core speedups", by Craig Bruce
2018.01.15 22:43 "Re: [Tiff] Strategies for multi-core speedups", by scott ribe
2018.01.15 21:46 "Re: [Tiff] Strategies for multi-core speedups", by scott ribe
2018.01.15 21:51 "Re: [Tiff] Strategies for multi-core speedups", by Larry Gritz
2018.01.16 01:09 "Re: [Tiff] Strategies for multi-core speedups", by Craig Bruce
2018.01.16 18:36 "Re: [Tiff] Strategies for multi-core speedups", by
2018.01.15 23:29 "Re: [Tiff] Strategies for multi-core speedups", by Larry Gritz
2018.01.16 00:24 "Re: [Tiff] Strategies for multi-core speedups", by Bob Friesenhahn
2018.01.16 00:43 "Re: [Tiff] Strategies for multi-core speedups", by Paul Hemmer
2018.01.16 00:58 "Re: [Tiff] Strategies for multi-core speedups", by scott ribe
2018.01.16 07:31 "Re: [Tiff] Strategies for multi-core speedups", by Paavo Helde
2018.01.17 02:00 "Re: [Tiff] Strategies for multi-core speedups", by Larry Gritz
2018.01.24 00:33 "Re: [Tiff] Strategies for multi-core speedups", by Larry Gritz
2018.01.16 18:48 "Re: [Tiff] Strategies for multi-core speedups", by scott ribe
2018.01.16 19:01 "Re: [Tiff] Strategies for multi-core speedups", by Kemp Watson
2018.01.16 19:17 "Re: [Tiff] Strategies for multi-core speedups", by
2018.01.16 19:34 "Re: [Tiff] Strategies for multi-core speedups", by Paavo Helde
2018.01.16 19:52 "Re: [Tiff] Strategies for multi-core speedups", by Kemp Watson
2018.01.16 19:57 "Re: [Tiff] Strategies for multi-core speedups", by
2018.01.16 20:18 "Re: [Tiff] Strategies for multi-core speedups", by Kemp Watson
2018.01.16 20:20 "Re: [Tiff] Strategies for multi-core speedups", by Kemp Watson
2018.01.16 20:23 "Re: [Tiff] Strategies for multi-core speedups", by

2018.01.15 19:17 "[Tiff] Strategies for multi-core speedups", by Larry Gritz

I have applications in which the I/O for TIFF files dominate the runtime. The big expense is the decompression, not the raw read/write. Almost always there are cores to burn, but I'm finding it hard to figure out how to leverage that with libtiff.

As a point of comparison, the OpenEXR library libIlmImf has API calls that will read multiple adjacent scanlines or tiles at once. The library maintains a thread pool, so those multi-scanline reads or writes actually will compress/decompress the scanlines/strips/tiles in parallel. This dramatically speeds up reads and writes of whole files when using the multi-scanline/tile API (though obviously does not come into play when the app reads or writes a single scanline or tile at once).

I tried to do this myself using libtiff -- for the common case of needing to read an entire image, my aim was read all the raw strips (TIFFReadRawStrip, serially), then dole them out to different threads to decompress in parallel (and when writing, compress in parallel, then TIFFWriteRawStrip each one serially). If libIlmImf is any indication, doing this on our typical 12 or 16 core machines ought to speed up TIFF I/O by an order of magnitude, easily.

But my plan was thwarted because it sure seems that the API for the codecs is inherently stateful and non-reentrant.

Has anybody tried something like this? Any advice on how to compress/decompress multiple strips or tiles simultaneously? Is there some workaround, or perhaps I just don't understand the libtiff internals enough to see the solution?

--
Larry Gritz
lg@larrygritz.com