Read File in chunks and then read line by line using LineNumberReader. Repeat this activity

Question

I have a file containing some 6.5 lakh lines. Now I wish to read every line of this file using LineNumberReader.

However I am encountering an outofMemoryError on adding these many number of lines to another 3rd party library..

What I intend to do is, read 200000 lines of a file at a time and add these lines to 3rd party library.

I am using LineNumberReader but I think the entire file is being read although I have provided condition that when line count reaches 200000 break the loop and add these to 3rd party library..

A code snippet for the same:

    LineNumberReader lnr=new LineNumberReader(new FileReader(file));
    String line=null;
    int i=0;      
    while(flags)
              {
     while( null != (line = lnr.readLine()) ){  
    i++;   
     3rdPartyLibrary.add(line.trim());
    if(i==200000)
                {      
                    System.out.println("Breaking");
                    lnr.mark(i);
                    break;
                }  
                if(i==400000)
                {
                    System.out.println("" );
                    lnr.mark(i);
                    break;
                }
                if(i==600000)
                {
                    System.out.println("BREAKING " );
                    lnr.mark(i);
                    break;
                }
     }
    if(line==null)
              {
                  System.out.println(" FLAG");
                  flags=false;
              }
lnr.reset();
    }

What I am intending to do here is read file from 0-200000 in first iteration. Then read each individual line and add to 3rd party lib.. Once this is done, read another 200000 lines from (200001-400000) and then repeat the same activity.

Need help..Can someone please guide..

Answer 1

As far as I think, this error arises when the JVM fails to allocate more objects (String in your case) due to the lack of memory as your program has already occupied a lot of it. Please place a call System.gc() somewhere inside the loop and check whether it works.

Answer 2

Here is the example that i tried a long time back, http://www.javacodegeeks.com/2013/01/processing-huge-files-with-java.html This solve out of memory exception.