2007.07.05 16:02 "Re: [Tiff] 16-Bit-Per-Channel Lossless Compression", by Bob Friesenhahn

As I don't speak gdalese, what size do tiles have in your test? It's my opinion ideally efficient tiles over a broad range of compression modes, that fit workflow on today's machines best, are in the order of about 1 to 3 megabyte, not the legacy 8 kilobyte (uncompressed, of course). I believe that for some compression modes, tile size does have a considerable impact on efficienty.

Joris,

You make many assumptions in the above statement, and particularly
about how applications store/access their pixels. I would not
encourage anyone to write such huge tiles.

There are many major factors which influence performance:

• Application pixel storage organization
• CPU cache
• Disk cache
• Main memory bandwidth
• Disk filesystem implementation
• Operating system I/O implementation
• Network filesystem implementation

I have done considerable testing and benchmarking in various scenarios
and results usually show that data access sizes of 8K or 16K are best,
with performance diminishing past 32K.

Your huge tiles have these problems:

• They are larger than the cache on anything but the most exotic RISC systems. Regardless, the first level cache is usually fairly small.
• They are larger than many disk drive caches.
• They don't mesh well with the typical 8K or 16K filesystem block sizes.
• They don't mesh well with network filesystem buffering ("block") sizes.
• The application is blocked on large I/O when it could be simultaneously processing data (parsing/storing pixels).
• They require the application to allocate more memory.
• They defeat the operating system's built in "read ahead" logic for sequential file access. The operating system is only going to be willing to buffer a limited amount of unrequested data in anticipation that it might be used. Only a rather foolish OS will gamble on reading 3MB of data in advance with the hope that it will be used. If the OS is only willing to do read ahead by 8K or 16K, then performance is best if the application does not need more than that in order to proceed.

Bob
======================================
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/