2017.09.17 11:45 "Re: [Tiff] TIFF tile size limit", by Roger Leigh

On 17/09/17 00:56, Bob Friesenhahn wrote:

On 16/09/2017 19:50, Bob Friesenhahn wrote:

Does anyone have an opinion on the maximum tile size which should be allowed by a general purpose TIFF reader?

For example is 4096x4096 large enough to handle any practical TIFFs?

I'd venture to say any size up to and including the full image size.

Technically larger, since the tile size can be larger than the image size. is nothing to prevent a one pixel image from claiming a   There huge tile size.

Yes. Though if it's not rounded up to the next multiple of 16, and/or the size is excessive, I think you would have reasonable grounds for warning or erroring out on suspect-but-valid sizes.

(CL_DEVICE_IMAGE2D_MAX_HEIGHT/WIDTH). I think it's fair to say that   So a practical TIFF reader should be capable of at least 8k x 8k for today's basic needs, and likely significantly more to be future proof.

256 MB per tile seems like quite a lot of memory to require from a client.

It's a lot from one point of view. But if you have images in the range of multiple gigabytes to terabytes, it's a drop in the ocean, and you'd have hardware to match to effectively process it. Now we can easily get workstations with 128 GiB and up, it's a reasonable chunk size, particularly given that the hardware to deal with data of this size is today's off the shelf mid-range hardware.

One thing which might be useful to explore is how best to store data of this size in TIFF. For example, is compression and the need for temporary buffers to store the decompressed data worth it? Or is it better to memory map and use transparent filesystem compression at the block level, e.g. ZFS with lz4/gzip. This lets zlib and the application use zero-copy I/O if all you're doing is handing over tiles, and it means there's no real constraint to the tile size other than virtual memory and available disc bandwidth.

I've started to benchmark some of this stuff, and I will certainly report back the findings.

Regardless, the problem I am seeing here is that the client uses TIFFScanlineSize(), TIFFStripSize(), TIFFTileSize(), or TIFFGetField(tiff,TIFFTAG_IMAGEWIDTH,&width) to determine the amount of memory to allocate. can return huge sizes and it is difficult for   These client software to know what sizes are valid.

That is certainly a tough problem to solve. At least for tiles you can do an initial sanity check that the size is a multiple of 16. It would be nice if such policies were configurable so the safeties can be turned off or tuned. Maybe you could factor in the use of memory mapping and compression so that if it's uncompressed and mapped you allow arbitrarily large sizes?

Tile/strip sizes gratuitously larger than the image size/width seem potential candidates for being "too big". Image width/height are tough ones--we already have examples where the sizes ar