LibTiff Mailing List
TIFF and LibTiff Mailing List Archive
July 2007
Previous Thread
Next Thread
Previous by Thread
Next by Thread
Previous by Date
Next by Date
Contact
The TIFF Mailing List Homepage
This list is run by Frank Warmerdam
Archive maintained by AWare Systems
Thread
2007.07.05 16:02 "Re: 16-Bit-Per-Channel Lossless Compression", by Bob Friesenhahn
On Thu, 5 Jul 2007, Joris wrote:
> As I don't speak gdalese, what size do tiles have in your test? It's my
> opinion ideally efficient tiles over a broad range of compression modes, that
> fit workflow on today's machines best, are in the order of about 1 to 3
> megabyte, not the legacy 8 kilobyte (uncompressed, of course). I believe that
> for some compression modes, tile size does have a considerable impact on
> efficienty.
Joris,
You make many assumptions in the above statement, and particularly
about how applications store/access their pixels. I would not
encourage anyone to write such huge tiles.
There are many major factors which influence performance:
o Application pixel storage organization
o CPU cache
o Disk cache
o Main memory bandwidth
o Disk filesystem implementation
o Operating system I/O implementation
o Network filesystem implementation
I have done considerable testing and benchmarking in various scenarios
and results usually show that data access sizes of 8K or 16K are best,
with performance diminishing past 32K.
Your huge tiles have these problems:
o They are larger than the cache on anything but the most exotic
RISC systems. Regardless, the first level cache is usually fairly
small.
o They are larger than many disk drive caches.
o They don't mesh well with the typical 8K or 16K filesystem block
sizes.
o They don't mesh well with network filesystem buffering ("block")
sizes.
o The application is blocked on large I/O when it could be
simultaneously processing data (parsing/storing pixels).
o They require the application to allocate more memory.
o They defeat the operating system's built in "read ahead"
logic for sequential file access. The operating system is only
going to be willing to buffer a limited amount of unrequested
data in anticipation that it might be used. Only a rather
foolish OS will gamble on reading 3MB of data in advance with
the hope that it will be used. If the OS is only willing to
do read ahead by 8K or 16K, then performance is best if the
application does not need more than that in order to proceed.
Bob
======================================
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/