How to import a whole folder of CSVs in python (pandas) from UCI ML Repo?

Question

this is the link from which I want the csv files:http://archive.ics.uci.edu/ml/datasets/selfBACK

My approach right now is to download it locally, by simply clicking it. But, this folder has a lot of different folders with many CSVs in it. How I do i import it in an efficient manner?

I know how to do it one by one but I feel there has to be a more efficient way.

Answer 1

You can first read all paths in that folder, and filter for csv files (or add other filters e.g. for specific file names). After that combine the files, here i use pandas if the data is tabular and structured in the same way.

import os
import pandas as pd
path = 'your_folder_path'
dfs = [pd.read_csv(f) for f in os.listdir(path) if f.endswith('.csv')]

# combine them (if they have the same format) like this:
df = pd.concat(dfs)

Note: you could also make a dictionary instead (key=filename, value=dataframe) and then access the data by using the filename.