2021.12.16 16:47 "Re: [Tiff] Ensuring buffer alignment when creating tiff files", by Milian Wolff

Milian, are these images for geomapping or digital pathology? I am guessing so since you mentioned mipmap subIFDs.

It's for quantitative imaging, so digital pathology is probably the right category for us.

If so, there is a good 20-25 years of experience in how to efficiently write and read these large files, and the solution has inevitably come down to tiling the data, and absolutely not mmapping.

I agree that tiling is ideal. I don't see why tiling would mean "no mmapping" - one could mmap the tiles, no? Furthermore, it also has nothing to do with mipmapping.

Furthermore, to reiterate this once more - we are interacting with a thirdparty library which requires contiguous buffers. There is (sadly!) no easy way to change that to deal with discontiguous, potentially compressed, tiled image data.

In fact, when you say the size of the image leaves you no alternative, I think perhaps you really should really think that the size of the images means mmapping should be out of the question.

I don't understand why that is, can you please elaborate? See above: Without mmapping, I would have to copy the full image into a contiguous heap memory buffer, which is stricly worse than using mmap where the kernel can evict the data as needed, instead of having to write the data to swap.

Are you generating the TIFFs yourself, or are they being generated as some kind of well-known TIFF derivative?

Both. For this question here it's about those parts where we are actually writing the data ourselves.

If so, there may well be SDKs or well-known approaches - also, if you are not generating the data, can you really ensure that the files meet all your criteria for mmapping?

We can, and are, checking the criteria for mmapping. If they are all met, we can continue directly. Otherwise our software will create a mmappable buffer on disk in a cache diretly that can be mmapped directly.

If not, you will need to read/write in memory anyway... similarly, if your other software stacks are not all on the same machine, or if you are using GPUs with limited memory.

None of this applies to us.

Thanks

Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts