Delphi - writing a large dynamic array to disk using stream

Question

In a Delphi program, I have a dynamic array with 4,000,000,001 cardinals. I'm trying to write (and later read) it do a drive. I used the following:

const Billion = 1000000000;

stream := tFileStream.Create( 'f:\data\BigList.data', fmCreate);
stream.WriteBuffer( Pointer( BigArray)^, (4 * billion + 1) * SizeOf( cardinal));
stream.free;

It bombed out with: ...raised exception class EWriteError with message 'Stream write error'.

The size of the file it wrote is only 3,042,089KB.

Am I doing something wrong? Is there a limit to the size that can be written at once (about 3GB)?

Answer 1

The Count parameter of WriteBuffer is a 32 bit integer so you cannot pass the required value in that parameter. You will need to write the file with multiple separate calls to WriteBuffer, where each call passes a count that does not exceed this limit.

I suggest that you write it something like this.

var
  Count, Index, N: Int64;
.... 
Count := Length(BigArray);
Index := 0;
while Count > 0 do begin
  N := Min(Count, 8192);
  stream.WriteBuffer(BigArray[Index], N*SizeOf(BigArray[0]));
  inc(Index, N);
  dec(Count, N);
end;

An additional benefit is that you can readily display progress.