Perl: Efficiently store/get 2D array of constrained integers in file

Question

This is an attempt to improve my Perl: seek to and read bits, not bytes by explaining more thoroughly what I was trying to do.

I have x, a 9136 x 42 array of integers that I want to store super-efficiently in a file. The integers have the following constraints:

• All of the 9136 integers in x[0..9135][0] are between -137438953472 and 137438953471, and can therefore be stored using 38 bits.

• All of the 9136 integers in x[0..9135][1] are between -16777216 and 16777215, and can therefore be stored using 25 bits.

• And so on... (the integer bit constraints are known in advance; Perl doesn't have to compute them)

Question: Using Perl, how do I efficiently store this array in a file?

Notes:

• If an integer can be stored in 25 bits, it can also be stored in 4 bytes (32 bits), if you're willing to waste 7 bits. In my situation, however, every bit counts.

• I want to use file seek() to find data quickly, not read sequentially through the file.

• The array will normally be accessed as x[i]. In other words, I'll want the 42 integers corresponding to a given x[i], so these 42 integers should be stored close to each other (ideally, they should be stored adjacent to each other in the file)

• My initial approach was to just lay down a bitstream, and then find a way to read it back and change it back into an integer. My original question focused on that, but perhaps there's a better solution to the bigger problem that I'm not seeing.

Far too much detail on what I'm doing:

• https://github.com/barrycarter/bcapps/blob/master/ASTRO/bc-read-cheb.m
• https://github.com/barrycarter/bcapps/blob/master/ASTRO/bc-read-cheb.pl
• https://github.com/barrycarter/bcapps/blob/master/ASTRO/playground.pl

Answer 1

I'm not sure I should be encouraging you, but it loks like Data::BitStream will do what you ask.

The program below writes a 38-bit value and a 25-bit value to a file, and then opens and retrieves the values intact.

#!/usr/bin/perl

use strict;
use warnings;

use Data::BitStream;

{
   my $bs_out = Data::BitStream->new(
      mode => 'w',
      file => 'bits.dat',
   );

   printf "Maximum %d bits per word\n", $bs_out->maxbits;

   $bs_out->write(38, 137438953471);
   $bs_out->write(25, 16777215);

   printf "Total %d bits written\n\n", $bs_out->len;
}

{
   my $bs_in = Data::BitStream->new(
      mode => 'ro',
      file => 'bits.dat',
   );

   printf "Total %d bits read\n\n", $bs_in->len;
   print "Data:\n";

   print $bs_in->read(38), "\n";
   print $bs_in->read(25), "\n";
}

output

Maximum 64 bits per word
Total 63 bits written

File size 11 bytes
Total 63 bits read

Data:
137438953471
16777215

38 and 25 is 63 bits of data written, which the module confirms. But there is clearly some additional housekeeping data involved as the total size of the resulting file is eleven bytes, and not just the eight that would be the minimum necessary. Note that, when reopened, the data remembers that it is 63 bits long. However, it is shorter than the sixteen bytes that a file would have to be to contain two simple 64-bit integers.

What you do with this information is up to you, but remember that data packed in this way will be extremely difficult to debug with a hex editor. You may be shooting yourself in the foot if you adopt something like this.