LibTiff Mailing List
TIFF and LibTiff Mailing List Archive
August 2008
Previous Thread
Next Thread
Previous by Thread
Next by Thread
Previous by Date
Next by Date
Contact
The TIFF Mailing List Homepage
This list is run by Frank Warmerdam
Archive maintained by AWare Systems
Thread
2008.08.22 13:11 "Re: creating sparse files......", by Toby Thain
On 22-Aug-08, at 5:56 AM, Rogier Wolff wrote:
>
> Hi,
>
> I'm stitching kind of large panoramas. This results in big
> intermediate files. On my last run, which took overnight to stitch,
> I thought 42 Gb of free disk space would be enough. Wrong!
>
> I got over thrity files of over 1.2Gbytes, filling up the disk.
>
> It turns out that most of the files contain lots of zeroes. On Unix
> this can be stored effciently by not issueing a "write" with a buffer
> full of zeroes, but by seeking over the area. The operating system
> will act as if the area was filled with zeroes.
>
> I'v modified "tif_unix.c" to read:
>
>
> static int isallzero (tdata_t buf, tsize_t size)
> {
> int i;
> for (i=0;i<size;i++)
> if (buf[i]) return 0;
> return 1;
> }
This appears unnecessarily inefficient. It's going to be much cheaper
to test whole longwords (or whatever data alignment will allow) than
byte by byte. (I wonder if something clever can be done with MMX/SSE
on newer chips?)
>
> static tsize_t
> _tiffWriteProc(thandle_t fd, tdata_t buf, tsize_t size)
> {
> if (isallzero (buf, size))
> return (lseek ((int) fd, (off_t)size, SEEK_CUR));
> else
> return ((tsize_t) write((int) fd, buf, (size_t)
> size));
> }
This penalises all writes to optimise a very special case. Not saying
it can't be rationalised, but pros and cons can be debated.
just my 2c
--Toby
>
> static toff_t
> _tiffSeekProc(thandle_t fd, toff_t off, int whence)
> {
> return ((toff_t) lseek((int) fd, (off_t) off, whence));
> }
>
> static int
> _tiffCloseProc(thandle_t fd)
> {
> toff_t curoffset;
> toff_t fsize;
>
> curoffset = _tiffSeekProc (fd, 0, SEEK_CUR);
> fsize = _tiffSeekProc (fd, 0, SEEK_END);
> if (curoffset > fsize) {
> ftruncate((int)fd, curoffset);
> }
>
> return (close((int) fd));
> }
>
>
> I haven't tested it yet. Maybe someone here has the same problem I
> have , and is willing to do a test-run. I'll make a proper patch later
> on. This is just to allow you guys know and to invite comments....
>
> While writing this Email message, I've found a bug: If an area is
> written, and then later zeroed by the application program the new code
> won't work. I'll have to think about how to get this case right. In
> practise a modified libtiff should be easy to plug into "hugin" and
> the results should be visible, the bug-situation will probably not
> trigger.
>
> (I used to be able to hit the "compress" checkbox inside hugin, and it
> would work. But it seems to have gone "dead" in the version currently
> installed on my machine.)
>
> Roger.
>
>
> --
> ** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ **
> +31-15-2600998 **
> ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK:
> 27239233 **
> *-- BitWizard writes Linux device drivers for any device you may
> have! --*
> Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous
> statement.
> Does it sit on the couch all day? Is it unemployed? Please be
> specific!
> Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
> _______________________________________________
> Tiff mailing list: Tiff@lists.maptools.org
> http://lists.maptools.org/mailman/listinfo/tiff
> http://www.remotesensing.org/libtiff/