- 2020.12.30 15:26 "Re: [Tiff] Multithreading support?", by John
2020.12.31 01:48 "Re: [Tiff] Multithreading support?", by Bob Friesenhahn
But it would dramatically improve libtiff to have any or all of the following:
- Add API calls for reading/writing a range of scanlines or tiles at a time.
- Eschew any "statefulness" between calls -- i.e. a single call should be able to specify begin/end scanline, and the TIFF directory within the file to read from (so there is not hidden state between TIFFSetDirectory and TIFFReadScanline that currently requires a lock on the application side).
- Reentrancy/thread safety of multiple read calls happening simultaneously on the same open TIFF handle.
- Ideally, using an internal thread pool (or at least the ability to optionally use TBB or an external thread pool passed in, etc.) so that libtiff could parallelize everything but the raw I/O that has to be serialized. Or, alternately, ...
- Exposing through the libtiff API a simple thread-safe "decompress this raw strip or tile that I already read" and "compress this pixel buffer to a raw strip buffer" that implements all of the codecs supported, thus allowing apps to split the raw read/write from the [de]compression, so they can more easily use their own threading to parallelize at least the [de]compression.
Many applications could see an order of magnitude improvement in the I/O performance for TIFF files.
This sounds great to me, as long as it can be layered in without impacting existing usages. Existing apps need to continue to compile and run and without any background cycles consumed that they did not request. I expect that there would need to be a lot of internal changes.
Things like an internal thread pool would never be something that libtiff would itself decide to instantiate (given that the usage you describe is not the norm), but it could provide APIs which allow a using app to side-load a thread pool.
Others have said that support for async-I/O (offloading several I/O requests to the kernel and being notified when data is available) is the solution to perceived problems, but async-I/O is not as uniformly supported as threads are. Async-I/O does not provide any assistance for compression/decompression requirements.
Libtiff is on GitLab now and the common way things work is that someone does the work, pushes a branch, and then submits a merge request. Sometimes branches stay open/pending for a long time as development proceeds. Are you (and others) ready and willing to do most of the work?
Bob
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt