How to append data of all Parquet files from a folder of s3 bucket into a single dataframe in python

1k Views Asked by At

In my folder there are around 10 parquet files with same column names. I want to append all 10 parquet files data into one dataframe. I used below code but dataframe is having only first files data.

s3 = s3fs.S3FileSystem()
complete_path="s3://s3bucket/folder_path
df=pq.ParquetDataset(complete_path,filesystem=s3).read_pandas().
           to_pandas()

Can someone please help me.

0

There are 0 best solutions below