C# MathNet Fourier Transform seems to behave in strange ways, with unexpected results

Question

I have been exprimenting with FFTs in C#. I have written the following function to process a given audio file in such a way that it will fill a List of double arrays with the frequency data. The ultimate goal is to create a spectrogram from this data.

private static List<double[]> ProcessAudio(int fftSize) {
    List<double[]> spectrogram = new List<double[]>();
    int hopSize = fftSize / 2;

    using (var reader = new AudioFileReader(FILE)) {
    sampleRate = reader.WaveFormat.SampleRate;
    byte[] buffer = new byte[fftSize];

    Complex[] complexBuffer = new Complex[fftSize];

     while (reader.Position < reader.Length) {
        // Read the next chunk of the audio into the buffer
        int bytesRead = reader.Read(buffer, 0, buffer.Length);
        bool eof = bytesRead < buffer.Length;

        // Convert the byte buffer to a double array
        double[] audioBuffer = buffer.Select(b => (double) b).ToArray();

        // Calculate FFT for the audio buffer
        for (int i = 0; i < fftSize; i++) {
         complexBuffer[i] = new Complex(audioBuffer[i], 0);
        }
        Fourier.Forward(complexBuffer);

        // Calculate magnitude and add to the spectrogram
        var magnitudes = new double[fftSize];
        for (int i = 0; i < fftSize; i++) {
        magnitudes[i] = complexBuffer[i].Magnitude;
    }
    if (!eof) spectrogram.Add(magnitudes);

    // Advance the position by overlap to process the next frame
        reader.Position += hopSize;
    }
}

I have already implemented the visual output. I have attached an image to demonstrate the issues I'm experiencing. As you can see, the Fourier.Forward() apparently produces several different spectrograms layered above each other, at different scales and partially mirrored. The audio file I tested the algorithm with is just a sine wave sliding down, so there should just be a single white curve in the image.

The result my algorithm produces

Upon doing some research, I learned that it's normal for the FFT algorithm to produce mirrored results, as it also calculates "negative" frequencies. I have tried working around this by only iterating over the first half of the contents of the complexBuffer array, as such:

var magnitudes = new double[fftSize / 2];
for (int i = 0; i < fftSize / 2; i++) {
    magnitudes[i] = complexBuffer[i].Magnitude;
}
if (!eof) spectrogram.Add(magnitudes);

However, this only results in the final image being cropped to half the height, but there are still negative curves in the image.