2018.05.11 19:25 "Re: [Tiff] LZ4 compression", by Roger Leigh
On 11/05/18 13:50, Bob Friesenhahn wrote:
I'm new at libtiff mailinglist.
If LZ4 compression was discussed before, sorry for bothering you guys. But, I couldn't find LZ4 discussion in mailing list archive.
I'm wondering there is any plan to officially support LZ4 compression for contone images.
LZ4 is getting popular in file system. It improves decoding performance significantly. That means if TIFF is compressed in LZ4, any TIFF viewer can decode the images significantly faster than flate or LZW compression. Downside is LZ4 is little bit worse in compression ratio. How does it sound?
Have you tested its effectiveness with image data?
Development libtiff now supports Facebook's Zstandard (zstd) compression which is pretty fast and does seem to be effective with image data.
One other alternative is use of filesystem-level compression. I've been using ZFS with no compression, lz4 and gzip compression of various levels. The advantages of having it at the filesystem level are complete transparency at the application level, including memory mapping, with the writer and reader not needing to take any action to do the compression. This gives you all the benefits of LZ4 compression but without the need to alter libtiff or any of your own code. It's also very fast for lz4 (not so much for gzip; you'll need a decent CPU to avoid slowness).
I've tested a heterogeneous dataset consisting of 273226 files. The vast majority are TIFF files, about 50% uncompressed and 50% using LZW. You can see the full uncompressed size is 356GB. With lz4 it's 273GB and with gzip-6 166GB. For comparison, lzma and xz compression of the files (no filesystem compression) resulted in sizes of 130GB and 123GB respectively, but these aren't usable without explicit decompression of the archive.
% zfs list -r red/scratch
NAME USED AVAIL REFER MOUNTPOINT
red/scratch 1.02T 1.85T 104K /export/scratch
red/scratch/new-gzip6 166G 1.85T 166G /export/scratch/new-gzip6
red/scratch/new-lz4 273G 1.85T 273G /export/scratch/new-lz4
red/scratch/new-uncompressed 356G 1.85T 356G
red/scratch/orig-lzma 130G 1.85T 130G /export/scratch/orig-lzma
red/scratch/orig-xz 123G 1.85T 123G /export/scratch/orig-xz
% zfs get -r -t filesystem
NAME PROPERTY VALUE SOURCE
red/scratch logicalreferenced 36K -
red/scratch referenced 104K -
red/scratch compression off default
red/scratch refcompressratio 1.00x -
red/scratch/new-gzip6 logicalreferenced 355G -
red/scratch/new-gzip6 referenced 166G -
red/scratch/new-gzip6 compression gzip received
red/scratch/new-gzip6 refcompressratio 2.21x -
red/scratch/new-lz4 logicalreferenced 355G -
red/scratch/new-lz4 referenced 273G -
red/scratch/new-lz4 compression lz4 received
red/scratch/new-lz4 refcompressratio 1.32x -
red/scratch/new-uncompressed logicalreferenced 355G -
red/scratch/new-uncompressed referenced 356G -
red/scratch/new-uncompressed compression off received
red/scratch/new-uncompressed refcompressratio 1.00x -
red/scratch/orig-lzma logicalreferenced 130G -
red/scratch/orig-lzma referenced 130G -
red/scratch/orig-lzma compression off received
red/scratch/orig-lzma refcompressratio 1.00x -
red/scratch/orig-xz logicalreferenced 123G -
red/scratch/orig-xz referenced 123G -
red/scratch/orig-xz compression off received
red/scratch/orig-xz refcompressratio 1.00x -
As always, the compression performance will depend upon the source data, and the numbers here are only meaningful for the test data (several plates from drug screens).