Perl Tk app sometimes crashes after exceeding 4GB ram usage

Question

I have a Perl Tk GUI application that sometimes crashes after it exceeds 4GB of RAM usage. I can exceed 4GB of RAM usage in some cases using Perl Tk, and I have no issues exceeding 4GB when running tests in a console application.

• Operating system: Microsoft Windows [Version 10.0.19044.2006]
• Perl version: v5.30.3
• Tk version: 804.036 (latest available on CPAN)

Perl spits out this error almost every time it crashes, but sometimes it crashes without an error:

Free to wrong pool 678ea0 not e228dd0 at .\common\GUI_TESTS\test_memory_hog_gui.pl line 41.

When searching for this error, everything I could find was multi-threading related, and our application does not use multi-threading.

I thought it may be because we have something configured as 32-bit instead of 64-bit, so I followed the instructions in this question and found that everything is configured as 64-bit.

perl -V:ivsize          # ivsize='8';
perl -V:ptrsize         # ptrsize='8';
perl -V:archname        # archname='MSWin32-x64-multi-thread';

Below is an example GUI application that crashes after the memory exceeds 4GB. I have boiled this down from our application and the crashing behavior is the same. The data structure that we use is obviously much larger, so I am cloning a simplified version of ours many times to pass the 4GB threshold.

use strict;
use warnings;

use Tk;
use Tk::LabFrame;

use Clone;

my $MAIN_WINDOW = MainWindow->new;

$MAIN_WINDOW->minsize(400, 400);

my @dataStructureClones = ();
my $textBox;
my $button_frame = $MAIN_WINDOW->LabFrame(-label => "Test", -relief => 'groove', -borderwidth => 2)->pack();

$button_frame->Button(
    -text    => 'Run Crashing Operation',
    -command => sub {

        my $dataStructureThatCrashes = {
            NETLIST_INFO => {
                EXTRA_PROPERTIES => {
                    C_SIGNAL => {},
                    NET      => {},
                },
                NET_LIST => [
                    # omitting this call will allow the program to exceed 4GB until after it finishes the loop
                    { NL_INDEX => 0, }
                ]
            },
        };

        my $lastUpdate = time();

        push @dataStructureClones, $dataStructureThatCrashes;
        for (1 .. 5000000) {
            if (time() - $lastUpdate > 1) {
                # omitting this call will allow the program to exceed 4GB
                $textBox->insert("end", "Cloning hash ($_)...\n");
                $MAIN_WINDOW->update();
                $lastUpdate = time();
            }
            push @dataStructureClones, Clone::clone($dataStructureThatCrashes);
        }
    }
)->grid(-row => 0, -column => 0);

$textBox = $MAIN_WINDOW->Scrolled(
    'Text',
    -relief     => 'groove',
    -background => 'light grey',
    -foreground => 'black',
    -wrap       => 'char',
    -scrollbars => 'osoe',
    -width      => 110,
    -height     => 24,
)->pack(-side => 'top', -fill => 'both', -expand => 1);

MainLoop;

Things to note:

1. Commenting out line 28 makes the program run properly.
2. Commenting out line 38 makes the cloning loop finish executing, but then it crashes with a similar error about 15 seconds after the cloning has finished.

The similar error:

Free to wrong pool 1008ea0 not fcedf7a8 at C:/Strawberry/perl/site/lib/Tk.pm line 424.

I tried this on a Linux VM that we have (CentOS 7) and the issue does not happen.

Answer 1

The first problem is that the underlying Tk library has internal memory allocators that can only handle memory allocations up to around 2GB (due to an ABI botch long ago that forced the memory object size parameter to be an int; fixing that is messy and breaks many things). The internal data structures mean that you don't necessarily blow things up as soon as you go over 2GB, but you're playing with fire, or maybe with nitroglycerin would be a better analogy.

The second problem, that you've not yet hit per se but will soon, is that 4GB of text data is a lot to actually work with in a GUI! It simply overwhelms users to have all the detail at once.

The fix, which I can only outline here, is to use the fact that that much data will never show all on the screen at once. Instead, you will need to be scrolling through the data... but you can connect up the scrolling mechanisms to scroll a window through the data itself instead of moving the widget's view into the data. When a scrolling event happens, you delete the contents of the window and replace it from the underlying data source. (This is only an outline because I don't know how to do this in Perl/Tk... and would have to look up the details for even Tcl/Tk despite knowing that much better.) I've read about people doing this to visualise very large result sets from DB queries, and suspect something similar would be useful for you.

Of course, if you go with putting all that data in a DB (for their winnowing support) then you might find better ways of querying and updating things that doesn't involve looking at huge amounts of data at one time...