I am processing large amounts of financial data from csv files and storing them to pytables. This process is very slow, I think because of converting the panda dataframe a row at a time into a pytable row and appending.
But, in order to keep the pytable on disk from corruption, I utilize the mark and undo feature of pytables. Each time I'm certain the writes are all completed for a days worth of data, I mark the table. Each time the app is run, I undo to the latest mark.
I'd like to experiment with using pandas write to hdf5 file capability as it may be substantially faster.
Is there a way to still use pytables mark and undo feature, if I use pandas to write directly to an hdf5 file?