OverlayFS on a single large file

555 Views Asked by At

I would like to solve the following set of constraints:

  1. I want to be able to mount a copy of a large (16gb) remote file
  2. if a part of the file is written to by the application, it is written to the local copy and not synced over the network
  3. if a part of the file is read, if it was previously written to by the application, it will read the local copy. if it was never written to, it will first copy from the remote to local, then read from local
  4. parts of the file that are never read before being written to should never be transmitted over the network (this is the most important constraint)
  5. the file will always be the same size, so there is never ambiguity about what should happen when we read a specific byte from the file.

The reason for these constraints is that the vast majority of a single file will never be read, there are many such files (at least a small portion of each file is read), and network bandwidth is extremely limited.

OverlayFS comes very close to what I want. If I was able to apply overlayfs at the file level instead of the directory level, I would use the (perhaps nfs-mounted) remote file as the lower_file and an empty, sparse file as the upper_file.

Is there something that would allow me to do the above?

0

There are 0 best solutions below