preserve special characters when writing all lines to text file

Question

I have a few lines that contain special characters like these lines:

CommunautéFinancièreAfricaineBEACFranc(XAF) CommunautéFinancièreAfricaineBCEAOFranc(XOF)

But, when I write those lines into a text file, I get this as a result:

CFACommunaut�Financi�reAfricaineBEACFranc CFACommunaut�Financi�reAfricaineBCEAOFranc

This is how I write the lines:

File.WriteAllLines(@"c:\file5.txt", lines);

I also tried the 3rd parameter of File.WriteAllLines() by passing an Encoding. But that didn't help either.

File.WriteAllLines(@"c:\file5.txt", lines, Encoding.UTF8); File.WriteAllLines(@"c:\file5.txt", lines, Encoding.ASCII);

This is how I read all lines:

File.ReadAllLines(@"C:\file4.txt").ToList()
        .ForEach(g => 
            lines.Add(g.ToString()
            .Replace("/", string.Empty)
            .Replace("(", string.Empty)
            .Replace(")", string.Empty))
        );

The crazy thing is, the characters are displayed perfectly fine in another text file (file4.txt) where I read everything in.

Answer 1

Notepad does not recognize UTF-8 unless you add a BOM.

You have three choices:

1. Decide notepad is dumb and ignore the issue. Better editors will recognize UTF-8.
2. Add a BOM. Use var utf8WithBom = new Utf8Encoding(true, true) as second argument.

3. Use the legacy encoding. Specify Encoding.Default as second argument.

   The resulting file will corrupt characters not present in the current code page. It won't display correctly on systems using in certain other countries.

   IMO this one is a bad idea, but I still mention it for completeness.