2008.08.22 20:02 "Re: [Tiff] creating sparse files......", by Phillip Crews

It is likely that some popular filesystems will fail to seek past the end of the file or will return random bytes (or parts of some previously deleted file) for the uninitialized portions.

This was my immediate objection to this scheme. Even if it were a platform-specific or configuration-controlled option, there will be cases where one O/S is writing to a filesystem on another O/S (i.e., over the network), resulting the same random garbage or simply ignored seeks.

If this is for Windows, the simplest solution for the OP may be to store the files in an NTFS compressed folder, or modify TIFFFDOpen to accept an additional flag which would set the file as compressed on compatible filesystems. Note that the DeviceIoControl call will fail on unsupported file systems. No further changes should be required (i.e., non-standard seeks or checking for zeros). Similar minimal changes should be possible for other operating systems as well.

(NB: I apologize for the HTML mail but I thought it much more readable for code; I'm sending in both, hoping that the plaintext version will still be readable!)

In *tif_win32.c* (/untested /code follows, my changes in red):

|#include <winioctl.h>
...
TIFF*
TIFFFdOpen(thandle_t ifd, PCTSTR name, const char *mode)
{
  TIFF* tif;
  BOOL fSuppressMap = (mode[1] == 'u' || (mode[1] != 0 && mode[2] == 'u'));
  static const USHORT compmode = COMPRESSION_FORMAT_DEFAULT;
  DWORD retbytes = 0; /* Not used but necessary per MS docs */

  if (mode[0] == 'w' && (mode[1] == 'c' || mode[2] == 'c') ) {
    /* Compress new files only, otherwise no feedback on slow compression */
    DeviceIoControl(ifd, FSCTL_SET_COMPRESSION, &compmode, sizeof comp,
NULL,
        0, &retbytes, NULL);
    /* Will fail on unsupported file systems */
  }
  tif = TIFFClientOpen(name, mode, ifd,

    _tiffReadProc, _tiffWriteProc,
    _tiffSeekProc, _tiffCloseProc, _tiffSizeProc,

     fSuppressMap ? _tiffDummyMapProc : _tiffMapProc,
     fSuppressMap ? _tiffDummyUnmapProc : _tiffUnmapProc);

  if (tif)
    tif->tif_fd = ifd;
  return (tif);
}
...
|

This should greatly reduce disk consumption and I/O on holey files or files that are simply compressed.

Reference: http://msdn.microsoft.com/en-us/library/aa364592.aspx