Large xml file parsing failed using XML::TWIG

Question

I am trying parse a large xml file (around 100,000 records) using XML::Twig but perl parsing fails with error:

perl.exe - Application Error: 
The instruction at "0x28086920" referenced memory at "0x00000004". The memory could not be "written"...

I read that XML::Twig parses large xml files without any issues, but in my case it fails with the above error.

My .pl file has a forloop code that rotates 100,000 times as below:

foreach my $d1(@detailData)   {
    if ($d1->first_child('a')->getElementsByTagName('b')) {
        $id= $d1->first_child('a')->first_child('x')->field('b');
    }
    ....
    .....
    ......

}

Inside forloop I have around 20 if loops as above. Is it causing the memory issues?

Can anyone suggest me how to overcome this memory issues?

Answer 1

After googling for perl "The memory could not be written" I would guess that the problem comes either from loading the file all at once in memory, see for example http://www.perlmonks.org/?node_id=457265 or (less likely) from mixing modules compiled with different compilers (eg using Activestate packages with a Cygwin perl, see http://cygwin.com/ml/cygwin/2006-12/msg00798.html).

For XML::Twig to deal with huge files, you need to tell it at which level to process parts of the file (usually by defining handlers that process a sub-tree, then discard it), see the docs of the module.