2004.01.14 12:01 "[Tiff] COLORMAP and byte padding", by Stephan Assmus

# 2004.01.14 19:36 "Re: [Tiff] COLORMAP and byte padding", by Marti Maria

Hi,

Here is a small program to check the error by using both methods, shifting 8, (same as *, / by 256) and the "good" method, which is by 257. But this should be not so surprising. If you are scaling a value with range 0.. 255 to a range 0..65535, the factor would be 65535 / 255, and that equals to 257 :-)

--- Cut ---

#include <stdio.h>
#include <stdlib.h>

int main()
{
int i;
unsigned char a8_by256, a8_by257;
unsigned short a16_by256, a16_by257;
int error256, error257;
int max256 = 0, max257 = 0;
double sum256=0., sum257=0.;

for (i=0; i < 0xFFFF; i++) {

a8_by256 = (unsigned char) (i >> 8);
a8_by257 = (unsigned char) (double) floor(i / 257. + .5);

a16_by256 = ((unsigned short) a8_by256 << 8);
a16_by257 = ((unsigned short) a8_by257 << 8) | a8_by257;

error256 = abs(a16_by256 - i);
error257 = abs(a16_by257 - i);

if (error256 > max256) max256 = error256;
if (error257 > max257) max257 = error257;

sum256 += error256;
sum257 += error257;
}

printf("Max error:  by 256 = %d, by 257 = %d\n", max256, max257);
printf("Mean error: by 256 = %g, by 257 = %g\n", sum256 / 65535., sum257 / 65535.);

return 0;
}

--- Cut --