2018.05.11 01:43 "[Tiff] LZ4 compression", by fx HAYAKAWA MICHIO

2018.05.14 05:13 "Re: [Tiff] LZ4 compression", by fx HAYAKAWA MICHIO

> One other alternative is use of filesystem-level compression.

This is a good experiment.

It shows LZ4 doesn't have good compression ratio comparted to gzip or other compression method for archiving.

LZ4 is focused on encoding and decoding speed. That's why LZ4 is adapted on ZFS file system. The better compression method for TIFF should have good balancing between decoding performance and transferring performance, which equals to compression ratio. Because TIFF is always decoded and viewed on viewer applications.

Anyway, it looks like Adobe is the key person. If Adobe implements LZ4 compression on Photoshop, LZ4 TIFF will become a common format.

BTW, TIFF is still common in the printing industry since TIFF supports CMYK images that can represent printing device output. That's why I'm seeking LZ4 TIFF option.

Thanks,
Michio

-----Original Message-----

From: tiff-bounces@lists.maptools.org [mailto:tiff-bounces@lists.maptools.org] On Behalf Of Roger Leigh

On 11/05/18 13:50, Bob Friesenhahn wrote:

I'm new at libtiff mailinglist.

If LZ4 compression was discussed before, sorry for bothering you guys. But, I couldn't find LZ4 discussion in mailing list archive.

I'm wondering there is any plan to officially support LZ4 compression for contone images.

LZ4 is getting popular in file system. It improves decoding performance significantly. That means if TIFF is compressed in LZ4, any TIFF viewer can decode the images significantly faster than flate or LZW compression. Downside is LZ4 is little bit worse in compression ratio. How does it sound?

Have you tested its effectiveness with image data?

Development libtiff now supports Facebook's Zstandard (zstd) compression which is pretty fast and does seem to be effective with image data.

One other alternative is use of filesystem-level compression. I've been using ZFS with no compression, lz4 and gzip compression of various levels. The advantages of having it at the filesystem level are complete transparency at the application level, including memory mapping, with the writer and reader not needing to take any action to do the compression. This gives you all the benefits of LZ4 compression but without the need to alter libtiff or any of your own code. It's also very fast for lz4 (not so much for gzip; you'll need a decent CPU to avoid slowness).

I've tested a heterogeneous dataset consisting of 273226 files. The vast majority are TIFF files, about 50% uncompressed and 50% using LZW. You can see the full uncompressed size is 356GB. With lz4 it's 273GB and with gzip-6 166GB. For comparison, lzma and xz compression of the files (no filesystem compression) resulted in sizes of 130GB and 123GB respectively, but these aren't usable without explicit decompression of the archive.

% zfs list -r red/scratch

NAME                           USED  AVAIL  REFER  MOUNTPOINT
red/scratch                   1.02T  1.85T   104K  /export/scratch
red/scratch/new-gzip6          166G  1.85T   166G  /export/scratch/new-gzip6
red/scratch/new-lz4            273G  1.85T   273G  /export/scratch/new-lz4
red/scratch/new-uncompressed   356G  1.85T   356G
/export/scratch/new-uncompressed
red/scratch/orig-lzma          130G  1.85T   130G  /export/scratch/orig-lzma
red/scratch/orig-xz            123G  1.85T   123G  /export/scratch/orig-xz

% zfs get -r -t filesystem

logicalreferenced,referenced,compression,refcompressratio red/scratch
NAME                          PROPERTY           VALUE     SOURCE
red/scratch                   logicalreferenced  36K       -
red/scratch                   referenced         104K      -
red/scratch                   compression        off       default
red/scratch                   refcompressratio   1.00x     -
red/scratch/new-gzip6         logicalreferenced  355G      -
red/scratch/new-gzip6         referenced         166G      -
red/scratch/new-gzip6         compression        gzip      received
red/scratch/new-gzip6         refcompressratio   2.21x     -
red/scratch/new-lz4           logicalreferenced  355G      -
red/scratch/new-lz4           referenced         273G      -
red/scratch/new-lz4           compression        lz4       received
red/scratch/new-lz4           refcompressratio   1.32x     -
red/scratch/new-uncompressed  logicalreferenced  355G      -
red/scratch/new-uncompressed  referenced         356G      -
red/scratch/new-uncompressed  compression        off       received
red/scratch/new-uncompressed  refcompressratio   1.00x     -
red/scratch/orig-lzma         logicalreferenced  130G      -
red/scratch/orig-lzma         referenced         130G      -
red/scratch/orig-lzma         compression        off       received
red/scratch/orig-lzma         refcompressratio   1.00x     -
red/scratch/orig-xz           logicalreferenced  123G      -
red/scratch/orig-xz           referenced         123G      -
red/scratch/orig-xz           compression        off       received
red/scratch/orig-xz           refcompressratio   1.00x     -

As always, the compression performance will depend upon the source data, and the numbers here are only meaningful for the test data (several plates from drug screens).