2020.10.13 19:49 "Re: [Tiff] faster Zip/Deflate compression/decompression with libdeflate", by Even Rouault

Although we have better alternative than Deflate now with ZSTD, it is > still heavily used as being far more supported than ZSTD or other > alternatives. As an example of this, very recently, in the geospatial > domain, Sentinel2 L2A scenes have been reprocessed as Deflate- compressed > TIFF and published in https://registry.opendata.aws/sentinel-2-l2a-cogs/

<https://registry.opendata.aws/sentinel-2-l2a-cogs/ >. This represents a > volume of 200 million TIFF files for 10.5 petabytes.

> The libdeflate work looks really great. I’ll try to find time to review it in more detail tomorrow.

Thanks

One thought: GitLab supports Git LFS for storing “large files”. It might we worth using it for the TIFF sample data; same would apply to the other image data already in the repository. Doesn’t need doing for this merge request, but might be worth considering migrating over to. It would permit adding more test data to the repository in the future as well without affecting repository clone time (but it would download the branch-specific large file content on demand when you check out a branch, if you didn’t already pull it down explicitly).

Are you talking about the content in test/images/? It is just 1 MB large.

So I don't think we have a critical need for that feature for now. We don't necessarily need big images to test the various aspects of the code, and the smaller the better for "make check" purposes too.

(I've been ennoyed on the github side of things with LFS where there's a monthly bandwidth, that includes people forking the repo... and beyond that checkout no longer works and you've to buy credits. Perhaps things are different on gitlab.com)

> Regarding ZSTD, what’s the current situation with regard to the Adobe tag registry? Do we have a number for ZSTD yet?

We're still living with the number we have attributed to ourselves. And at that point, it would be a pain to have to change it.

Even

Spatialys - Geospatial professional services
http://www.spatialys.com