Py-tables vs Blaze vs S-Frames

Question

I am working on an exploratory data analysis using python on a huge Dataset (~20 Million records and 10 columns). I would be segmenting, aggregating data and create some visualizations, I might as well create some decision trees liner regression models using that dataset.

Because of the large data set I need to use a data-frame that allows out of core data storage. Since I am relatively new to Python and working with large data-sets, i want to use a method which would allow me to easily use sklearn on my data-sets. I'm confused weather to use Py-tables, Blaze or s-Frame for this exercise. If someone could help me understand what are their pros and cons. What are the factors that are important in this kind of decision making that would be much appreciated.

Answer 1

good question! one option you may consider is to not use any of the libraries aformentioned, but instead read and process your file chunk-by-chunk, something like this:

csv="""\path\to\file.csv"""

pandas allows to read data from (large) files chunk-wise via a file-iterator:

it = pd.read_csv(csv, iterator=True, chunksize=20000000 / 10)

for i, chunk in enumerate(it): ...