perl gunzip to buffer and gunzip to file have different byte orders

Question

I'm using Perl v5.22.1, Storable 2.53_01, and IO::Uncompress::Gunzip 2.068.

I want to use Perl to gunzip a Storable file in memory, without using an intermediate file.

I have a variable $zip_file = '/some/storable.gz' that points to this zipped file.

If I gunzip directly to a file, this works fine, and %root is correctly set to the Storable hash.

gunzip($zip_file, '/home/myusername/Programming/unzipped');
my %root = %{retrieve('/home/myusername/Programming/unzipped')};

However if I gunzip into memory like this:

my $file;
gunzip($zip_file, \$file);
my %root = %{thaw($file)};

I get the error

Storable binary image v56.115 more recent than I am (v2.10)`

so the Storable's magic number has been butchered: it should never be that high.

However, the strings in the unzipped buffer are still correct; the buffer starts with pst which is the correct Storable header. It only seems to be multi-byte variables like integers which are being broken.

Does this have something to do with byte ordering, such that writing to a file works one way while writing to a file buffer works in another? How can I gunzip to a buffer without it ruining my integers?

Answer 1

That's not related to unzip but to using retrieve vs. thaw. They both expect different input, i.e. thaw expect the output from freeze while retrieve expects the output from store. This can be verified with a simple test:

$ perl -MStorable -e 'my $x = {}; store($x,q[file.store])'
$ perl -MStorable=freeze -e 'my $x = {}; print freeze($x)' > file.freeze

On my machine this gives 24 bytes for the file created by store and 20 bytes for freeze. If I remove the leading 4 bytes from file.store the file is equivalent to file.freeze, i.e. store just added a 4 byte header. Thus you might try to uncompress the file in memory, remove the leading 4 bytes and run thaw on the rest.