2018.05.11 01:43 "[Tiff] LZ4 compression", by fx HAYAKAWA MICHIO

2018.06.08 00:02 "Re: [Tiff] LZ4 compression", by fx HAYAKAWA MICHIO

I successfully implemented LZ4 compression in libtiff on Windows platform. LZ4 frame format is used for compression scheme. LZ4 frame is widely used for various applications.

Does anyone have any test images to see LZ4 compression characteristics?

I also modified tiffcp to support LZ4, so that any tiff files can be converted into LZ4 tiff.

From my small experiments, its compression ratio is better than LZW, but worse than ZIP. Predictor is nicely working with LZ4. My tiff viewer shows LZ4 tiff very fast compared to ZIP especially large tiff files. If someone want the code, I'd like to pick up two TAG values before releasing it to avoid confusion.

#define COMPRESSION_LZ4         34926   /* LZ4 */

#define TIFFTAG_LZ4QUALITY              65564   /* LZ4 acceleration factor */

Are there any objections to pick up those values?


-----Original Message-----

From: tiff-bounces@lists.maptools.org [mailto:tiff-bounces@lists.maptools.org] On Behalf Of fx HAYAKAWA MICHIO

Sent: Tuesday, May 15, 2018 8:51 AM

To: Joe.Maniaci@ricoh-usa.com; Bob Friesenhahn <bfriesen@simple.dallas.tx.us>

Cc: tiff@lists.maptools.org
Subject: Re: [Tiff] LZ4 compression

> My company uses Adobe PDF print engine, which generates TIFFs and we're going to a print engine that will create TIFFs consistently outside the regular TIFF spec. And it turns out that if size warrants, the APPE will create a bigTIFF.

Exactly. PDF itself doesn't help to print. PDF doesn't represent video data or raster image. Thus, some applications are necessary to convert PDF to Raster Images and APPE is the one. In general, TIFF generating performance is a big issue at Ripping when RIP design is based on TIFF. So that LZ4 TIFF helps such applications and usages.


-----Original Message-----

From: Joe.Maniaci@ricoh-usa.com [mailto:Joe.Maniaci@ricoh-usa.com]

Sent: Monday, May 14, 2018 11:05 PM
To: Bob Friesenhahn <bfriesen@simple.dallas.tx.us>
Cc: fx HAYAKAWA MICHIO <michio.hayakawa@fujixerox.co.jp>; tiff@lists.maptools.org; tiff-bounces@lists.maptools.org
Subject: Re: [Tiff] LZ4 compression

> BTW, TIFF is still common in the printing industry since TIFF supports
> CMYK images that can represent printing device output.
> That's why I'm seeking LZ4 TIFF option.

Adobe promotes PDF for such applications.

My company uses Adobe PDF print engine, which generates TIFFs and we're going to a print engine that will create TIFFs consistently outside the regular TIFF spec. And it turns out that if size warrants, the APPE will create a bigTIFF. So internally, Adobe has implemented bigTIFF into at least some of their products, but won't do anything to address it in an official capacity. I'm sure that goes for other extensions as well.

Joseph Maniaci
Staff Software Engineer

From: Bob Friesenhahn <bfriesen@simple.dallas.tx.us>

To:        fx HAYAKAWA MICHIO <michio.hayakawa@fujixerox.co.jp> 
Cc:        "tiff@lists.maptools.org" <tiff@lists.maptools.org> 

Date: 05/14/2018 07:46 AM

Subject:        Re: [Tiff] LZ4 compression [EXTERNAL] 
Sent by:        tiff-bounces@lists.maptools.org 


On Mon, 14 May 2018, fx HAYAKAWA MICHIO wrote:

>> One other alternative is use of filesystem-level compression.
> This is a good experiment.
> It shows LZ4 doesn't have good compression ratio comparted to gzip or other compression method for archiving.
> LZ4 is focused on encoding and decoding speed. That's why LZ4 is adapted on ZFS file system. The better compression method for TIFF should have good balancing between decoding performance and transferring performance, which equals to compression ratio. Because TIFF is always decoded and viewed on viewer applications.

While LZ4 file compression performance in zfs is interesting, it is not very indicative of how well it would do on raster data as implemented in libtiff. The reason why it is not very very indicative is that the filesystem is a general-purpose mechanism without any knowlege of the content (e.g. TIFF strips/tiles) and with an underlying block size not related to the image data.

It is easy to implement LZ4 inside the operating system kernel space but ZStd is likely too complex to port into such an environment.
Regardless, ZStd compresses much faster than gzip while achieving a better compression ratio on raster data.

> Anyway, it looks like Adobe is the key person. If Adobe implements
> LZ4 compression on Photoshop, LZ4 TIFF will become a common format.

Adobe is no longer pushing or promoting TIFF and it is unlikely to
implement any more extensions to TIFF. It is not clear to me that
Adobe's Photoshop is as dominant as it used to be.

Bob Friesenhahn
bfriesen@simple.dallas.tx.us, https://urldefense.proofpoint.com/v2/url?u=http-3A__www.simplesystems.org_users_bfriesen_&d=DwICAg&c=5hYF0Zu0Yz-C6S-kaHDItw&r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&m=faxnTKuowEJsfJ8pe-vQomwFTEOpfInS_z3i4LvxgMM&s=WkAgKWVZNzBYk5aJr3Zlwu2cjHW3wqhx5Bjd1ycHfsg&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.simplesystems.org_users_bfriesen_&d=DwICAg&c=5hYF0Zu0Yz-C6S-kaHDItw&r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&m=faxnTKuowEJsfJ8pe-vQomwFTEOpfInS_z3i4LvxgMM&s=WkAgKWVZNzBYk5aJr3Zlwu2cjHW3wqhx5Bjd1ycHfsg&e=>

GraphicsMagick Maintainer, https://urldefense.proofpoint.com/v2/url?u=http-3A__www.GraphicsMagick.org_&d=DwICAg&c=5hYF0Zu0Yz-C6S-kaHDItw&r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&m=faxnTKuowEJsfJ8pe-vQomwFTEOpfInS_z3i4LvxgMM&s=ymzhagmq6MOhTcFXBY9XKZW12ae_vxewcAV7sK1eH18&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.GraphicsMagick.org_&d=DwICAg&c=5hYF0Zu0Yz-C6S-kaHDItw&r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&m=faxnTKuowEJsfJ8pe-vQomwFTEOpfInS_z3i4LvxgMM&s=ymzhagmq6MOhTcFXBY9XKZW12ae_vxewcAV7sK1eH18&e=>

Tiff mailing list: Tiff@lists.maptools.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.maptools.org_mailman_listinfo_tiff&d=DwICAg&c=5hYF0Zu0Yz-C6S-kaHDItw&r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&m=faxnTKuowEJsfJ8pe-vQomwFTEOpfInS_z3i4LvxgMM&s=1k0gTb7qiT40DyH9_JhmEaBtUA92-OXA7I90nBfqk_I&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.maptools.org_mailman_listinfo_tiff&d=DwICAg&c=5hYF0Zu0Yz-C6S-kaHDItw&r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&m=faxnTKuowEJsfJ8pe-vQomwFTEOpfInS_z3i4LvxgMM&s=1k0gTb7qiT40DyH9_JhmEaBtUA92-OXA7I90nBfqk_I&e=>
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.remotesensing.org_libtiff_&d=DwICAg&c=5hYF0Zu0Yz-C6S-kaHDItw&r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&m=faxnTKuowEJsfJ8pe-vQomwFTEOpfInS_z3i4LvxgMM&s=pewtP1QW7BeHumqqbTXlm7gEsKZOL4WHz5rRAVHC3oI&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.remotesensing.org_libtiff_&d=DwICAg&c=5hYF0Zu0Yz-C6S-kaHDItw&r=HnVw9YJBS4ST0tOBucT3NvWUJRRGhDxXu4zHnS5AkgY&m=faxnTKuowEJsfJ8pe-vQomwFTEOpfInS_z3i4LvxgMM&s=pewtP1QW7BeHumqqbTXlm7gEsKZOL4WHz5rRAVHC3oI&e=>