Reading a large log file in C#

Question

For my project, I need to extract message types from a log file. I have a 700 MB log file which contains about 4.7 million lines and I need to read each entry line by line and extract the message field. I need to find the size of message in each entry(which is the event size) and store it along with that message in a dictionary. There can be multiple messages for same event size. But I get a OutOfMemoryException when I use the below logic.

Dictionary<Int32,List<String>> dt=new Dictionary<Int32,List<String>>();
List<String> entries=new List<String>();
StreamReader sr=new StreamReader("Bluegene.log");
String s;
while((s=sr.readLine())!=null)
{
    eventsize=s.length - 9; //size of only the message field
    entries.Add(s);
    if (!dt.ContainsKey(eventsize))
    {
        dt.Add(eventsize, entries);
    }
    else
    {
       dt.Remove(eventsize);
       dt.Add(eventsize, entries);
     }
  }

Will using MemoryMappedFile help?

Answer 1

The problem is your list is ever growing.
So, you can try the following:

Dictionary<Int32, List<String>> dt = new Dictionary<Int32, List<String>>();           
            int eventsize;
            StreamReader sr = new StreamReader("Bluegene.log");           
            string s;
            while ((s = sr.ReadLine()) != null)
            {
                eventsize = s.Length - 9; //size of only the message field      
                if (!dt.ContainsKey(eventsize))
                {
                    List<String> entries = new List<String>();
                    entries.Add(s);
                    dt.Add(eventsize, entries);
                }
                else
                {
                    dt[eventsize].Add(s);
                }
            }