AWARE [SYSTEMS] Imaging expertise for the Delphi developer
AWare Systems, Imaging expertise for the Delphi developer, Home TIFF and LibTiff Mailing List Archive

LibTiff Mailing List

TIFF and LibTiff Mailing List Archive
September 2004

Previous Thread
Next Thread

Previous by Thread
Next by Thread

Previous by Date
Next by Date

Contact

The TIFF Mailing List Homepage
This list is run by Frank Warmerdam
Archive maintained by AWare Systems



Valid HTML 4.01!



Thread

2004.09.16 19:12 "BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.16 19:23 "Re: BigTIFF Tag Value Count issue", by Frank Warmerdam
2004.09.16 19:36 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.16 19:38 "Re: BigTIFF Tag Value Count issue", by Frank Warmerdam
2004.09.16 20:08 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 01:14 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.16 22:39 "Re: BigTIFF Tag Value Count issue", by Ian Ameline
2004.09.17 00:09 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 00:32 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 01:54 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 02:29 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 02:58 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 03:08 "Re: BigTIFF Tag Value Count issue", by Chris Cox
2004.09.17 04:20 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 04:39 "Re: BigTIFF Tag Value Count issue", by Chris Cox
2004.09.17 11:25 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 14:31 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 14:40 "Re: BigTIFF Tag Value Count issue", by Frank Warmerdam
2004.09.17 15:02 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 21:41 "Re: BigTIFF Tag Value Count issue", by Chris Cox
2004.09.17 14:53 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 15:06 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 21:36 "Re: BigTIFF Tag Value Count issue", by Chris Cox
2004.09.17 05:46 "Re: BigTIFF Tag Value Count issue", by Frank Warmerdam
2004.09.17 11:40 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 13:11 "Re: BigTIFF Tag Value Count issue", by Frank Warmerdam
2004.09.17 13:34 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 15:10 "Re: BigTIFF Tag Value Count issue", by Fernando Loygorri
2004.09.17 15:10 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 15:55 "Re: BigTIFF Tag Value Count issue", by Fernando Loygorri
2004.09.17 16:00 "Re: BigTIFF Tag Value Count issue", by Frank Warmerdam
2004.09.17 16:33 "Re: BigTIFF Tag Value Count issue", by Fernando Loygorri
2004.09.17 16:13 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 16:34 "Re: BigTIFF Tag Value Count issue", by Fernando Loygorri
2004.09.17 16:43 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 21:48 "Re: BigTIFF Tag Value Count issue", by Ross A Finlayson
2004.09.17 22:15 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn
2004.09.17 21:41 "Re: BigTIFF Tag Value Count issue", by Chris Cox
2004.09.17 11:38 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 06:35 "Re: BigTIFF Tag Value Count issue", by Rob Tillaart
2004.09.17 11:43 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 12:46 "Re: BigTIFF Tag Value Count issue", by Rob Tillaart
2004.09.17 12:55 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 13:20 "Re: BigTIFF Tag Value Count issue", by Rob Tillaart
2004.09.17 13:47 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.17 13:08 "Re: BigTIFF Tag Value Count issue", by Ross A Finlayson
2004.09.19 12:31 "Re: BigTIFF Tag Value Count issue", by Andrey Kiselev
2004.09.19 16:43 "Re: BigTIFF Tag Value Count issue", by Joris Van Damme
2004.09.20 16:18 "Re: BigTIFF Tag Value Count issue", by Ed Grissom
2004.09.20 22:05 "Re: BigTIFF Tag Value Count issue", by Steve Carlsen
2004.09.20 23:09 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn

2004.09.17 02:29 "Re: BigTIFF Tag Value Count issue", by Bob Friesenhahn

I have attached source for a small test program I used to see how 
systems behave under load when using memory mapped files.  Naturally, 
documentation is poor and the code is meant for hacking. :-)

The program creates a 6GB file on disk (adjustable at line 39) and 
then proceeds to update the content of the file using a stride (size 
of the chunk of memory mapped and updated at one time).  The msync 
call is used to tell the OS which modified pages are no longer needed 
(and should be be saved to disk).

The results of test runs were very interesting.  Solaris did 
exceedingly well with any size stride and was done with the file very 
quickly without excess memory usage or thrashing.  FreeBSD 5.0 did 
very well if the stride was over a certain size (8 or 16K I think), 
but under that magic size, it suddenly took 20X longer.  Linux 2.6 did 
not seem to observe the request to purge dirty memory so it thrashed 
away.

There is a chunk of code which is #ifdeffed out which sees how well 
the system works if the file uses one huge mapping.  The test filesize 
may need to be reduced in order to use this.

Bob
======================================
Bob Friesenhahn
bfriesen@simple.dallas.tx.us
http://www.simplesystems.org/users/bfriesen

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <limits.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

int main (int argc, char *argv[])
{
  char testfile[] = "mmap.tmp";
  int fd;
  off_t filelen;
  off_t index;
  size_t stride;
  unsigned char *mapped;
  size_t pagesize;

  if (argc != 2)
    {
      printf("Usage: mmaptest stride\n");
      exit(1);
    }

  pagesize = (size_t) sysconf(_SC_PAGESIZE);

  stride=pagesize*atoi(argv[1]);

  fd = open(testfile,O_RDWR|O_CREAT|O_TRUNC,S_IRUSR|S_IWUSR);
  if (fd == -1)
    {
      perror("Failed to open file");
      exit(1);
    }

  filelen=2147483647ULL*3;
  printf("File length = %lld\n", (long long)filelen);
  //filelen=8096;
  index = 0;
  if ((ftruncate(fd,filelen)) == -1)
    {
      perror("ftruncate failed");
      exit(1);
    }
#if 1
  {
    // stride = pagesize*16;
    printf("Stride      = %lld\n", (long long)stride);
    
    for (index = 0; index+stride < filelen; index += stride)
      {
        mapped = (unsigned char *) mmap((caddr_t) 0,stride,PROT_READ|PROT_WRITE,
                                        MAP_SHARED,fd,index);
        if (mapped == MAP_FAILED)
          {
            perror("Failed to map file");
            exit(1);
          }

        // madvise(mapped, stride, MADV_SEQUENTIAL);

        // Uncomment the following line for a display of progress.
        // printf("index       = %lld\r", (long long)index);
        memset((void *) mapped, 64, stride);

        if ((msync(mapped,stride,MS_ASYNC)) == -1) // MS_ASYNC or MS_SYNC
          {
            perror("\nmsync failed");
            exit(1);
          }

        if((munmap(mapped,stride)) == -1)
          {
            perror("\nmunmap failed");
            exit(1);
          }
        mapped=0;
      }
  }
#else
  {
    off_t off;
    unsigned char *p;
    off = 0;
    stride = pagesize*16;
    printf("Stride      = %d\n", stride);
    mapped = (unsigned char *) mmap((caddr_t) 0,filelen,PROT_READ|PROT_WRITE,
                                    MAP_SHARED,fd,off);
    if (mapped == MAP_FAILED)
      {
        perror("Failed to map file");
        exit(1);
      }

    for (p = mapped, index=0; index+stride < filelen; index += stride, p += stride)
      {
        memset((void *) p, 64, stride);
      }

    if((munmap(mapped,filelen)) == -1)
      {
        perror("munmap failed");
        exit(1);
      }
  }
#endif

  printf("\n");
  close(fd);
  unlink(testfile);
  return 0;
}