Pytables Undo vs Pandas hdf5 write

189 Views Asked by At

I am processing large amounts of financial data from csv files and storing them to pytables. This process is very slow, I think because of converting the panda dataframe a row at a time into a pytable row and appending.

But, in order to keep the pytable on disk from corruption, I utilize the mark and undo feature of pytables. Each time I'm certain the writes are all completed for a days worth of data, I mark the table. Each time the app is run, I undo to the latest mark.

I'd like to experiment with using pandas write to hdf5 file capability as it may be substantially faster.

Is there a way to still use pytables mark and undo feature, if I use pandas to write directly to an hdf5 file?

0

There are 0 best solutions below