2020.12.31 02:38 "Re: [Tiff] Multithreading support?", by Larry Gritz

Yeah, the back compatibility could be tricky, but I think it's possible to add a few new API calls that are thread-safe, even while keeping the old calls and workflows and documenting them as not threadsafe.

I agree with you about the internal thread pool -- FWIW, that's a bit of an albatross around the neck of OpenEXR. A better strategy is to allow an interface for specifying how to submit tasks to the application's thread pool, somewhat akin for how the application can supply its own low-level I/O functions. Or to just punt on any internal threading knowledge at all and just have a clean separation between the inherently serial and stateful parts of the API and the parts amenable to concurrency. The biggest sin currently is that TIFFReadScanline/Tile/etc present a stateful serial interface, but most of the internal expense is code that doesn't need to be.

Are you (and others) ready and willing to do most of the work?

Not at all! Unfortunately, my libtiff internals knowledge is thin and I'm already the maintainer of two large and widely used open source packages (hence, I do fully understand the frustration of somebody just laying a call for a major overhaul at your doorstep but being unable to send a PR, sorry). I'm afraid that the biggest contribution that I'm able to make in practice is just to outline in some detail what the downstream packages really need from libtiff in order to get an order of magnitude speedup.

I think that the minimal step forward would be to expose a stateless/reentrant TIFF{Encode,Decode}{Strip,Tile}, which would be a simple way to access libtiff's implementations of the compression codecs (e.g. TIFFReadEncodedStrip would be equivalent to TIFFReadRawStrip followed immediately by TIFFDecodeStrip) -- that would allow apps to do all the thread policy stuff by using libtiff to serially perform the I/O and then the app side could make concurrent calls for the decoding. That's basically what I'm doing in OIIO now to speed up TIFF I/O, except that I only have it working for a few major cases (deflate compression only, horizontal or no predictor, uint8 or uint16 types only, contiguous planarconfig only). A simple libtiff API call that handled raw strips/tiles <--> pixels for all possible cases supported by the library would be tremendously helpful. It's possible (I don't know) that these entry points already more or less exist internal to the library, and all they really need is to be cleaned up and exposed as part of the API?

-- lg

Larry Gritz
lg@larrygritz.com