LibTiff Mailing List
TIFF and LibTiff Mailing List Archive
July 2007
Previous Thread
Next Thread
Previous by Thread
Next by Thread
Previous by Date
Next by Date
Contact
The TIFF Mailing List Homepage
This list is run by Frank Warmerdam
Archive maintained by AWare Systems
Thread
2007.07.05 16:30 "Re: 16-Bit-Per-Channel Lossless Compression", by Joris Van Damme
Bob,
Bob Friesenhahn wrote:
> > As I don't speak gdalese, what size do tiles have in your test?
> > It's my opinion ideally efficient tiles over a broad range of
> > compression modes, that fit workflow on today's machines best, are
> > in the order of about 1 to 3 megabyte, not the legacy 8 kilobyte
> > (uncompressed, of course). I believe that for some compression
> > modes, tile size does have a considerable impact on efficienty.
>
> You make many assumptions in the above statement, and particularly
> about how applications store/access their pixels. I would not
> encourage anyone to write such huge tiles.
>
> There are many major factors which influence performance:
>
> o Application pixel storage organization
> o CPU cache
> o Disk cache
> o Main memory bandwidth
> o Disk filesystem implementation
> o Operating system I/O implementation
> o Network filesystem implementation
>
> I have done considerable testing and benchmarking in various scenarios
> and results usually show that data access sizes of 8K or 16K are best,
> with performance diminishing past 32K.
>
> Your huge tiles have these problems:
>
> o They are larger than the cache on anything but the most exotic
> RISC systems. Regardless, the first level cache is usually fairly
> small.
> o They are larger than many disk drive caches.
> o They don't mesh well with the typical 8K or 16K filesystem block
> sizes.
> o They don't mesh well with network filesystem buffering ("block")
> sizes.
> o The application is blocked on large I/O when it could be
> simultaneously processing data (parsing/storing pixels).
> o They require the application to allocate more memory.
> o They defeat the operating system's built in "read ahead"
> logic for sequential file access. The operating system is only
> going to be willing to buffer a limited amount of unrequested
> data in anticipation that it might be used. Only a rather
> foolish OS will gamble on reading 3MB of data in advance with
> the hope that it will be used. If the OS is only willing to
> do read ahead by 8K or 16K, then performance is best if the
> application does not need more than that in order to proceed.
Though a number of your points are based on LibTiff's design, which is not
ideal in that it buffers compressed and decompressed striles (=strips or
tiles, whatever applies), I do think you make sense given that LibTiff
design and a number of your points probably apply to AsTiff design as well.
Likely my estimate of 1 to 3 megabyte for ideal strile size is a bit much.
Best regards,
Joris Van Damme
info@awaresystems.be
http://www.awaresystems.be/
Download your free TIFF tag viewer for windows here:
http://www.awaresystems.be/imaging/tiff/astifftagviewer.html