2002.08.22 22:09 "RE: OT: large memory allocation in Windows", by Andreas R. Kleinert

A 1 GB file occupies a 1 GB address range (within the 2 GB virtual user address space area).

If you map the whole file at once, sure.

A more reasonable approach is to map the current region of the file you are interested in. We typically map a few hundred K of the file at a time. I probably ought to revisit this and map a few meg at a time.

Well, of course. But this again means a tiled/segmented memory architecture - with a slide of the address window and/or its content, which needs to be covered by the algorithm that processes the data.

I actually hide this from the rest of my library in a file i/o layer that I wrote that mimics fopen/fseeek/fread with mmap'ed calls using MapViewOfFile.

I like the idea, but you (functionally) also could achieve this with normal fopen/fseek/fread on a fixed malloc'ed buffer. No fancy 4K mmu-page wise "write back" support but anyway...

If I want linear addressing of a file's content, I need to fully map it to memory. If not, all sorts of elegant/appropriate algorithms certainly come in handy ;-)

And a "linear pointer space" was mentioned at the start. Here's the quote again:

>>>How about a memory mapped file? Could that possibly give you a
>>>linear "pointer space" of 2GB or maybe even more? Of course the file
>>>would not be completely mapped at all times, but maybe it would do the
>>>trick... (Just a crazy idea. I don't know if it would work at all...).

Andreas_Kleinert@t-online.de  | http://www.ar-kleinert.de             |
Freelance Consultant & Writer | Software Engineering                  |
 *** PerSuaSiVe SoftWorX ***  | x86 Win/Linux, 68k/PPC Amiga and more |