2001.03.29 22:00 "TIFF 3.5.6 Beta Bugs", by Scott Marovich

2001.03.29 22:59 "Re: TIFF 3.5.6 Beta Bugs", by Frank Warmerdam

Scott,

I have submitted your report into BugZilla at:

http://bugzilla.remotesensing.org/show_bug.cgi?id=40

I am also cc:ing this reply to the libtiff mailing list in the hopes that Sam, Mike and others can given their opinion on some of the points I am less than enthusiastic about changing.

If you're familiar with "xv(1)", you may know that whenever something like the TIFF Library generates a message on a UNIX user's standard error stream, this usually-terse text is displayed under "X Windows" using a small pop-up window. For a naive user who's innocently trying display an image downloaded from the Web, because "xv(1)" gets invoked from Netscape Navigator via "metamail(1)" and a "mailcap" file, the source and significance of these messages is often mysterious, hard to diagnose, and very annoying. We have therefore engaged in a local campaign to help stamp out TIFF Library (and other) spurious diagnostic messages whenever this is reasonably possible. As you may know, a common cause of spurious messages is when a user's TIFF file contains a vendor-specific TIFF record tag that isn't described in the TIFF Library's "tiffFieldInfo[]" array, in the "tif_dirinfo.c" file. In general, I think that this array should strive to describe "all known" TIFF tags, even if they are vendor-specific and even if the associated (default) parsing action is "FIELD_IGNORE". I realize updating this array could be a never-ending job, since there are probably thousands of these things, and Adobe Systems doesn't seem to publish a complete list of registered tags on its Web site. So, as a small step in that direction, we've added definitions for a couple of extra Adobe tags that their documents state (and our testing confirms) may appear in PageMaker or Photoshop output. This is the reason for my suggested "tiff.h" and "tif_dirinfo.c" changes, shown below.

As you say, having this list be complete is pretty difficult, and I am not sure that it's in the spirit of what Sam intended for the list. The "traditional" approach to quieting warnings from applications like xv is to examine the warnings emitted, and ignore those that are about unknown tags. Lord knows I have found xv's habit of reporting unknown tags in one popup per tag very annoying.

The "tif_thunder.c" change is an attempt to solve a different, but related problem: Without it, when "xv(1)" tries to display the "text.tif" file in Sam's test suite, we get the TIFF Library error message:

ThunderDecode: Not enough data at scanline 320 (0 != 1512)

I have also been through various iterations on the issue of what a decompressor should do if the input is corrupt. A number of folks have written "validation" tools that depend on libtiff generating a fatal error on corrupt data rather than attempting to recover. However, the different decompressors are inconsistent in how they handle this. A previous discussion on the list seemed to be inconclusive as to what should be done, so I have been avoiding doing anything about the general issue. The ThunderScan code is not the only one that will produces a fatal error on short data. The Packbits does this as well.

and our "xv(1)" fails to display the image because of this error. Upon investigation, it appears that this file represents a 768x359 pixel image for which the last 3 rows (2268 pixels exactly) are missing from the compressed- image data. I'm not very familiar with the details of "Thunder" compression or with how this file was generated, so I don't know whether this discrepancy represents a defective test file or a defective decoding algorithm in the TIFF Library, but as a "hack" solution, I changed the "ThunderDecode()" subroutine to issue a warning message (not a fatal error) and automatically pad the image with rows of all-zero pixels. Our "xv(1)" then displays this image and I can barely see anything wrong on my CRT screen. Notice that the old- and new code might also similarly complain if a file's "Thunder" compressed image data encodes *too many* pixels, rather than too few. Although it would be possible to issue a warning message and truncate the image, I've chosen to leave the TIFF Library's fatal-error response intact here. Frankly, I'm not sure what the "best" answer is in this case.

Concerning an unrelated subject, I had a bunch of problems getting the auto- configure and build process to work for this software under HP-UX, and I severely hacked many of the shell scripts and "Makefile.in" templates in your distribution. I don't know whether you're interested in the details of these changes, but I could make up an additional set of "diff(1)" outputs if you care. The basic problems that I found were:

1. HP-UX Version 10+ uses an SVR4-style file-directory structure, which has no "/bin" directory. Hence, all references to paths like "/bin/sh" must be changed to "/usr/bin/sh", etc.

Your kidding! Even shell scripts can't start with "#!/bin/sh"? What is the universally safe way of starting out a bourne shell script? What exactly is going to be required to get things working out of the box on HP/UX 11? I have hesitant to really butcher things for what I would feel is an unnecessarily difficult build environment.

2. Shell environment variables in the "configure" script should NOT be unconditionally initialized to the null string or any other value; instead, they should be *conditionally* initialized so that they can be potentially over- ridden by a developer. For example, instead of initializing:

JPEG=

a more helpful syntax would be something like:

JPEG=${JPEG:=no}

This seems like a reasonable suggestion, but I believe the intent is that all overrides be done in the config.site file (invoked later in configure) rather than by pre-defining environment variables before running configure. Since the namespace for the configuration environment variables is not very distinct (ie. JPEG, HTML, ZIP, LIBGL, DSO) I am hesitant to assume that if they are set they are meant to provide an override. There is a non-trivial danger of adding configuration problems for people with "rich" environments.

3. In "Makefile.in" templates, recursive invocations of "make(1)" should always include the MAKEFLAGS macro (e.g., "${MAKE} -${MAKEFLAGS}... ") so that command-line flags are properly inherited.

Absolutely. I would like to correct this.

4. "configure" and the "Makefile.in" templates should support an OPTIMIZE parameter for a "C" compiler's optimization flags, since "-O" isn't always a developer's preferred choice.

I agree. I will try to apply this.

5. Finally, a bunch of HP-UX specific compiler and linker command flags have changed between Version 9 (when these templates were apparently built) and Version 11. I could send more details if you're interested.

I would be pleased to have an update to the configure for HP/UX 11.

Finally, to summarize, please do not intepret this long message as an attempt to trash the TIFF Library. On the contrary: it's great and I love it! But I hope that perhaps these suggestions might help to make it better.

I really appreciate your contribution. I feel back that I am dragging my feet on a number of your suggestions, but I will try to apply at least the ones I consider non-controversial.

Finally, would you like CVS commit access so you can do some of the tweaking directly (on points we agree upon at least)?

Best regards,

---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent