c++ writing and reading text file is very slow, any alternatives?

Question

I am currently writing code for a game and I'm a little bit stuck on saving and loading the level. For writing I use this piece of code:

    bool WorldGen::GenerateNewWorld(unsigned int seed, int width)
{
    std::cout << "World generating..." << std::endl;
    int heigth = 1; //2D perlin noise instead of 3D
    m_WorldSizeX = width;
    m_WorldSizeY = 1800; //add a int height if implementing different world sizes

    // Create a PerlinNoise object with a random permutation vector generated with seed
    PerlinNoise(seed);

    std::vector<byte> arrMaxHeight;

    // looping through all the x locations and deciding the y value
    for (unsigned int i = 0; i < heigth; ++i) {     // y
        for(unsigned int j = 0; j < width; ++j) {  // x
            double x = (double)j / ((double)width);
            double y = (double)i / ((double)heigth);

            // Typical Perlin noise
            double n = noise(10 * x, 10 * y, 0.8);

            //n is the ground added on top of the base layer (n = highest peak at point j)
            arrMaxHeight.push_back((int)(n * 255));
        }
    }

    std::wofstream fileStream;
    fileStream.open(L"GameSave/world/World.txt");

    if (fileStream.fail())
    {
        return false;
    }

    //fileStream << L"[I could put something up here but that's still in development!]" << std::endl;

    byte blockType = 0;
    std::vector<byte> arrBlockType;

    for (int i = 0; i < m_WorldSizeX; i++)
    {
        for (int j = 0; j < m_WorldSizeY; j++)
        {
            if (j > arrMaxHeight.at(i))
            {
                //block is not air
                blockType = 1;
            }
            else
            {
                //block is air
                blockType = 0;
            }

            arrBlockType.push_back(blockType);
            fileStream << blockType << "/n";
        }
    }

    fileStream.close();

    return true;
}

Now this not too bad, generates the world in around 5 minutes and sends it to world.txt without any issues, my loading(reading world.txt line per line) however takes ages. Around 30+ minutes to fully read all the lines from the text file using std::wifstream and its getline() function. It reads all the lines and adds them to a std::vector and later creates "blocks" from this vector. The creation of the blocks is done in a few seconds but the wifstream is really slow.

Here's the code for worldLoad:

    std::wifstream ifileStream;

ifileStream.open("GameSave/world/World.txt");
if (ifileStream.fail())
{
    std::cout << "Could not open World.txt" << std::endl;
    return;
}

std::wcout << "LoadWorld Started";
std::wstring extractedLine;
while (!ifileStream.eof())
{
    std::getline(ifileStream, extractedLine);

    m_ArrBlockData.push_back(StringToByte(extractedLine));
    std::wcout << m_ArrBlockData.size() << "\n";
}


DOUBLE2 location;

for (size_t i = 0; i < m_ArrBlockData.size(); i++)
{

    location.y = (i % m_WorldSizeY) * 16;
    if (location.y == 0)
    {
        location.x += 16;
    }

    Block *block = new Block(location, m_ArrBlockData.at(i));
    m_ArrBlocks.push_back(block);

    std::wcout << "Bock Created" << std::endl;
}

Any idea on how to optimise this? I was thinking about only reading blockTypes around the player but that would still require me putting all the blocks into a vector before being able to operate on them.

Kind regards, Jannes

PS: DOUBLE2 is a custom variable holding 2 doubles DOUBLE2(double x, double y)

Answer 1

Attach a profiler to your code to see what exactly is very slow, I suspect that it is related to the streams (those tend to be slow), and the vectors. (They re-allocate / move data around a lot when you are inserting data).

If you know the size beforehand, reserve the size in the vector.

Answer 2

What is arrBlockType for? You likely have a lot of reallocations there as it grows (because you do not preallocate any space inside it).... and then you never actually use it once it's filled. Take that out and your function will be faster.

Answer 3

If you can use boost, I would suggest using its serialization library. It is simple to use, flexible, and relatively high-performance when using the binary archive. There are, of course, many alternatives.