Filesystem VS Raw disk benchmarking in C

Question

I am doing some benchmarking (on OS X) to see how the use of file system influences the bandwidth etc. I am using concurrency with the hope to create fragmentation in the FS.

However, it looks like using the FS is more efficient than raw disk accesses. Why?

Here is my code:

#include <pthread.h>
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>

#define NO_THREADS (2)
#define PACKET_SIZE (1024 * 4)
#define SIZE_TO_WRITE (1024 * 1024 * 1024)

void write_buffer(void *arg) {
    int *p_start = arg;
    int start = *p_start;
    char buffer[PACKET_SIZE];


    char path[50];
    sprintf(path, "file%d", start);
    int fd = open(path, O_CREAT | O_WRONLY | O_APPEND);

    //int fd = open("/dev/rdisk0s4", O_WRONLY);

    if (fd < 0) {
        fprintf(stderr, "Cound not open.\n", stderr);
        goto end;
    }

    //lseek(fd, start * SIZE_TO_WRITE, SEEK_SET);

    int current;
    for (current = start; current < start + SIZE_TO_WRITE; current += PACKET_SIZE) {

        int i;
        for (i = 0; i < PACKET_SIZE; ++i) {
            buffer[i] = i + current;
        }

        if (PACKET_SIZE != write(fd, buffer, PACKET_SIZE)) {
            fprintf(stderr, "Could not write packet %d properly.", current);
            goto close;
        }
    }

    fsync(fd);

  close:
    close(fd);
  end:
    pthread_exit(0);
}

void flush(void) {
    fflush(stdout);
    fflush(stderr);
}

int main(void) {
    pthread_t threads[NO_THREADS];
    int starts[NO_THREADS];
    int i;

    atexit(flush);

    for (i = 0; i < NO_THREADS; ++i) {

        starts[i] = i;

        if(pthread_create(threads + i, NULL, (void *) &write_buffer, (void *)(starts + i))) {
            fprintf(stderr, "Error creating thread no %d\n", i);
            return EXIT_FAILURE;
        }
    }

    for (i = 0; i < NO_THREADS; ++i) {
        if(pthread_join(threads[i], NULL)) {
            fprintf(stderr, "Error joining thread\n");
            return EXIT_FAILURE;
        }
    }

    puts("Done");

    return EXIT_SUCCESS;
}

With the help of the FS, the 2 threads write the file in 31.33 seconds. Without, it is achieved after minutes...

Answer 1

When you use /dev/rdisk0s4 instead of /path/to/normal/file%d, for every write you perform the OS will issue a disk I/O. Even if that disk is an SSD, that means that the round-trip time is probably at least a few hundred microseconds on average. When you write to the file instead, the filesystem isn't actually issuing your writes to disk until later. The Linux man page describes this well:

A successful return from write() does not make any guarantee that data has been committed to disk. In fact, on some buggy implementations, it does not even guarantee that space has successfully been reserved for the data. The only way to be sure is to call fsync(2) after you are done writing all your data.

So, the data you wrote is being buffered by the filesystem, which only requires that a copy be made in memory -- this probably takes on the order of a few microseconds at most. If you want to do an apples-to-apples comparison, you should make sure you're doing synchronous I/O for both test cases. Even running fsync after the whole test is done will probably allow the filesystem to be much faster, since it will batch up the I/O into one continuous streaming write, which could be faster than what your test directly on the disk can achieve.

In general, writing good systems benchmarks is incredibly difficult, especially when you don't know a lot about the system you're trying to test. I'd recommend using an off-the-shelf Unix filesystem benchmarking toolkit if you want high quality results -- otherwise, you could spend literally a lifetime learning about performance pathologies of the OS and FS you're testing... not that that's a bad thing, if you're interested in it like I am :-)