2002.08.22 15:59 "RE: OT: large memory allocation in Windows", by Ed Grissom
I was thinking more along the lines of using the operating system level memory mapping offered by Windows. This is supposed to be pretty efficient and work very much like virtual memory with frequently used areas kept in RAM. What I do not know is what kind of limitations Windows puts on memory mapped files :-)
A 1 GB file occupies a 1 GB address range (within the 2 GB virtual user address space area).
If you map the whole file at once, sure.
A more reasonable approach is to map the current region of the file you are interested in. We typically map a few hundred K of the file at a time. I probably ought to revisit this and map a few meg at a time.
I actually hide this from the rest of my library in a file i/o layer that I wrote that mimics fopen/fseeek/fread with mmap'ed calls using MapViewOfFile. If the current position (set by fseek or fread) is outside the current mapping, then toss the old mapping and make a new map including the new area of interest.