File.AppendText() Slow

Question

I have one big file that is structured something like this:

 Report 1  section 1
 Report 2  section 1
 Report 1  section 2
 Report 3  section 1
 Report 2  section 2
 and so on....

I have to put all the 1s together, all the 2s together, etc, into Report 1, Report 2, Report 3. I have no choice but to go line by line. The problem is that it is very slow. Here is the code I am using to write the files:

        using (StreamWriter sw = File.AppendText(newFileName))
                    { sw.WriteLine(line); }

I think the problem is that the File.AppendText() is slowing down this process. I'm wondering if anyone has any ideas about how to speed this up.

Answer 1

It seems you are opening that file for each iteration. Try this:

using (StreamWriter sw = File.AppendText(path))
{
    while (condition)
    {
        sw.WriteLine("write your line here");
    }
}

As Chris Berger has commented, you can nest usings like this

using (StreamWriter sw1 = File.AppendText(path1))
{
    using (StreamWriter sw2 = File.AppendText(path2))
    {
        while (condition)
        {
            if(writeInFile1)
                sw1.WriteLine("write your line here");
            else
                sw2.WriteLine("write your line here");
        }
    }
}

Answer 2

As you mentioned in Facundo's answer

This is a good solution however I will have five or six report files coming from one file...

You can open all 6 files at once by using multiple using statements.

using (StreamReader sr = File.OpenText(source)
using (StreamWriter sw1 = File.AppendText(path1))
using (StreamWriter sw2 = File.AppendText(path2))
using (StreamWriter sw3 = File.AppendText(path3))
using (StreamWriter sw4 = File.AppendText(path4))
using (StreamWriter sw5 = File.AppendText(path5))
using (StreamWriter sw6 = File.AppendText(path6))
{
    string line;
    while ((line = sr.ReadLine()) != null)
    {
        if(line.StartsWith("Report 1")
        {
            sw1.WriteLine(line);
        }
        else if(line.StartsWith("Report 2")
        {
            sw2.WriteLine(line);
        }
        else if(line.StartsWith("Report 3")
        {
            sw3.WriteLine(line);
        }
        else if(line.StartsWith("Report 4")
        {
            sw4.WriteLine(line);
        }
        else if(line.StartsWith("Report 5")
        {
            sw5.WriteLine(line);
        }
        else if(line.StartsWith("Report 6")
        {
            sw6.WriteLine(line);
        }
        else
        {
            throw new InvalidDataException($"Line does not start with a report number: \n{line}");
        }
    }
}