I'm trying to use a zarr store as a fixed-size buffer (i.e. new data is appended to the end, and the same amount of data is removed from the beginning when a certain size is reached).
The store is huge (20 TB), and contains a 2D matrix (positions over time).
Writing to zarr is handled by xarray.
However, I'm not sure whether zarr supports this.
I can think of two solutions:
- create a new xarray object from the first, eliminating the older data. However, writing that to disk will either append (
"a"), leaving the older data intact, or overwrite ("w"), in which case I'm afraid the whole thing is rewritten which would not be performant for 20 TB. - use zarr.core.Array.resize, but this does not seem to allow dropping data at the start
Maybe zarr does not support this and I have to think of another solution, or writing my own store specifically aimed at this type of problem.