open versus fopen in C program (with multi-threading)

Question

For an unknown reason, it seems that fopen fails to open text files when it is called repeated times before closing.

My C program uses multithreading, and each thread handles one type of output text files (eleven per type), each type in a separated folder. I need to mantain these files opened during a long execution and at the same thread in order to write a lot of data.

To explain it better, the process is the following:

1- Thread #1 starts and creates and writes 11 files in one folder.

2- Without closing previous files, Thread #2 starts and creates and writes another 11 files in a different folder.

3- The same with other two Threads.

4- In the end, when all the files needed have been created and all the Threads have finished unless the main one, all the files are closed.

However and surprisingly, open is do able to handle all this files at the same time. Does anybody have a tip about this issue with fopen? I don't know if it is related with the multi-threading or with a max number of files that fopen can handle at the same time.

I am in Windows platform with Borland compiler.

Answer 1

It is very unlikely that the problem is with fopen. fopen uses open underneath and the maximum number of open files most probably has the same limit as open.

My guess is that you have an error somewhere else, most likely going out of an array boundaries.

If you are not using any Windows specific functions, you could always valgrind it on Linux.

If you can't port the code, you could binary search the error. For example, comment out half the code that writes to files and see if the problem persists. If it doesn't, the problem is likely on the part that is commented out. If not, the problem is on the part that is not commented.

Note the term "likely" above. If your code results in undefined behavior, you can't rely on the behavior. Even buggy code could behave correctly.

Answer 2

The C Run Time (CRT) library is thread safe from Windows XP and above. Also, if you believe some of the comments in the CRT include files, it has a limit of 20 open files.

So, in your application. What is the reason for a single thread to have 11 files open? You are NOT going to get a measurable performance gain by writing them simultaneously. In fact, you could make it worse because the C drive will thrash as it moves around trying to write each file.

Depending on how your application is designed: open one file, write to it until full, close it, open second file, etc.

Now, multiply the possibility of disk thrashing by four as each thread is trying to do I/O to the same C drive! So, serialize disk accesses if possible. to get a real improvement increase the buffer sizes.

Answer 3

AFAIK the f-functions aren't thread-safe. open may be the foundation for fopen but underneath it all are basically the (originally Win32) functions CreateFile, ReadFile, WriteFile and CloseHandle. They are thread-safe and I suggest you use them rather than using critical sections around your f-calls.