How to read properly binary (raw) .pbm file in C++?

Question

I want to read a binary .pbm file. This is so far what I have:

void readPBMFile(const std::string& filePath, std::vector<std::vector<int>>& pixels) {
    std::ifstream file(filePath, std::ios::binary);

    std::string header;
    std::getline(file, header);

    if (header != "P4") {
        throw std::runtime_error("File is not in PBM format.");
    }

    int width, height;
    file >> width >> height;

    pixels.resize(height, std::vector<int>(width));

    // This is the part where I read the file data after the header
    for (int y = 0; y < height; ++y) {
        for (int x = 0; x < width; ++x) {
            unsigned char byte;
            file.read(reinterpret_cast<char*>(&byte), 1);
            int pixelValue = byte ? 1 : 0;
            pixels[y][x] = pixelValue;
        }
    }
}

And the save function:

void savePBMFile(const std::string& filePath, const std::vector<std::vector<int>>& pixels) {
    std::ofstream file(filePath, std::ios::binary);

    file << "P4\n";

    int width = pixels[0].size();
    int height = pixels.size();
    file << width << " " << height << "\n";

    for (const auto& row : pixels) {
        for (const auto& pixel : row) {
            unsigned char byte = pixel ? 1 : 0;
            file.write(reinterpret_cast<const char*>(&byte), 1);
        }
    }
}

But I ended with a strange image with correct white spaces and black vertical lines. The black areas are ignored. What am I doing wrong?