I have found a few strange things while using C++ Thread

Question

I realized the salsa20 algorithm on my computer(windows10) yesterday. This salsa20 function is a function that receives fileName and key and converts a file at a very high speed.

The main part of the code is here:

void salsa20(const char *fileName, const char *key) {
    uint8_t temp[BN]; //40KByte memory

    int currentpos = 0;
    int step = BN * 100;
    
    int file = open(fileName, O_RDWR | O_BINARY, S_IRUSR | S_IWUSR);

    int readSize = 0;
    int bufsec = 0;
    int i;

    while (readSize = read(file, temp, BN)) {
        if (readSize < 0) break;
        for (i = 0; i < readSize; i += 64) {
            salsa_encrypt((uint8_t*)key, bufsec++, (uint8_t*)temp + i, readSize - i);
        }
        lseek(file, -readSize, SEEK_CUR);
        write(file, temp, readSize);
        lseek(file, readSize, SEEK_CUR);
    }
    close(file);
}

int main() {
    salsa20("1.iso", "PASSWORDTYPE1___!@#$%^&*()!@#$%^");
}

This function worked well and the memory usage was very small (<1Mbyte).

And today I wanted to convert several files at the same time by using this function with several threads.

int main() {
    thread t1(salsa20, "1.iso", "PASSWORDTYPE1___!@#$%^&*()!@#$%^");
    thread t2(salsa20, "2.iso", "PASSWORDTYPE2___!@#$%^&*()!@#$%^");
    thread t3(salsa20, "3.iso", "PASSWORDTYPE2___!@#$%^&*()!@#$%^");
    t1.join();
    t2.join();
    t3.join();
}

By the way, the speed was a bit faster, but I found that the usage of momory suddenly increased by more than 700Mbyte(12.3G->13.1G) and it gradually decreased, and even though the program was terminated, it was completely recovered after about 30 seconds.

It is thought to be a problem related to the operation system and file management, but I have not yet had an accurate understanding of it. I would like to know the countermeasures to prevent memory increase while using threads.

I need a safe program that uses threads but does not use memory.

Answer 1

You can change the code like this on windows.

void salsa20(const char* fileName, const char* key) {
    uint8_t temp[BN]; //40KByte memory
    FILE* file;
    int currentpos = 0;
    int step = BN * 100;

    HANDLE hFile = CreateFileA(fileName, GENERIC_READ|GENERIC_WRITE, FILE_SHARE_READ|FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_FLAG_NO_BUFFERING, NULL);
    DWORD readSize = 0, writeSize;
    int bufsec = 0;
    int i;
    
    DWORD dwSizeHi;
    DWORD dwSizeLo = GetFileSize(hFile, &dwSizeHi);

    while (true) {
        ReadFile(hFile, temp, BN, (DWORD*) &readSize, NULL);
        for (i = 0; i < readSize; i += 64) {
            salsa_encrypt((uint8_t*)key, bufsec++, (uint8_t*)temp + i, readSize - i);
        }
        SetFilePointer(hFile, -(LONG)readSize, NULL, FILE_CURRENT);
        WriteFile(hFile, temp, BN, (DWORD*) & writeSize, NULL);
        if (BN != readSize)
            break;
    }
    CloseHandle(hFile);
    hFile = CreateFileA(fileName, GENERIC_READ|GENERIC_WRITE, FILE_SHARE_READ|FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);
    SetFilePointer(hFile, dwSizeLo, (LONG*)&dwSizeHi, FILE_BEGIN);
    SetEndOfFile(hFile);
    CloseHandle(hFile);
}

On linux system, you can use posix_fadvise function.

void salsa20_256(const char *fileName, const char *key) {
    uint8_t temp[BN]; //40KByte memory
    int file = open(fileName, O_RDWR | O_BINARY, S_IRWXU);

    fdatasync(file);
    posix_fadvise(file, 0,0,POSIX_FADV_DONTNEED);

    int readSize = 0;
    int bufsec = 0;
    int i;

    long long totalSize = 0;

    while (readSize = read(file, temp, BN)) {
        if (readSize < 0) break;
        for (i = 0; i < readSize; i += 64) {
            salsa_encrypt((uint8_t*)key, bufsec++, (uint8_t*)temp + i, readSize - i);
        }
        lseek(file, -readSize, SEEK_CUR);
        write(file, temp, readSize);
        lseek(file, 0, SEEK_CUR);
        posix_fadvise(file, 0,0,POSIX_FADV_DONTNEED);
    }
    posix_fadvise(file, 0,0,POSIX_FADV_DONTNEED);
    close(file);
}

Answer 2

This program is safe. The operating system has made a decision to keep the data written in memory. This may be to avoid having the program wait until the write completes. This may be in case another program reads the data soon.

Unless you think the operating system's decision is bad -- which it can only be if the memory was needed for some other purpose -- then this is not a problem. This is the system being efficient.

Note that you are not measuring the program's memory usage. You are measuring the operating system's memory usage while the program runs. The operating system may, for example, decide to cache more data from disk because of what the program is doing. It's usually going to benefot from those kinds of decisions and likely here it lets the program complete much faster and thus makes its results available to other programs faster.

Answer 3

In the Windows system, when using Thread, if the processing speed of the Harddisk does not keep up, cache memory is used. Of course, if Thread is not used, this phenomenon does not appear. Perhaps if you reduce the number of threads to 1 or 2, such a phenomenon may not appear. However, the influence of other programs that are thoroughly executed on the operating system is also related.

You can probably try running this program on a Linux system as well. In such a case, you can see that the operating system uses cache memory when reading and writing files. This memory use can be initialized by using the posix_fadvise function.