Facing Memory issues While Creating DataFrame Object using Marketorestpython api Bulk Lead Data Extraction Method

Question

I am using the below method for Bulk Lead Data Extraction and I am calling this method and storing data extracted from server for last 31 days to one object name data. But as soon as it fetches data for 31 days it will throw an error, I cannot fetch for less than 31 days as Historical Data needs to be backed up and sync in tables.

Any advice on how we can parse this memory issue while creating dataframe out of data.

   def get_lead_data_Bulk(self, fields, startTime, endTime):
        print("Creating a job")
        new_export_job_details = self.mc.execute(method='create_leads_export_job', fields=fields,
                                                 filters={
                                                     'createdAt': {'endAt': str(endTime), 'startAt': str(startTime)}})
        exportId = new_export_job_details[0]['exportId']
        print("Export Id :", exportId)
        print("Enqueing a job")
        enqueued_job_details = self.mc.execute(method='enqueue_leads_export_job', job_id=exportId)
        status = ''
        while (status != 'Completed'):
            export_job_status = self.mc.execute(method='get_leads_export_job_status', job_id=exportId)
            status = export_job_status[0]['status']
            if status != 'Completed':
                time.sleep(30)
        print("Fetching the file")
        data = self.mc.execute(method='get_leads_export_job_file', job_id=exportId)
        data = data.decode("utf-8")
        data = StringIO(data)
        data = pd.read_csv(data,chunksize=1000)
       data = pd.concat(chunk for chunk in data)
        print("Length of Data from Marketo which need to be Upserted",len(data))
        return data

Edit-1:-

I have Changed the date Cycle to 15 Days instead of 31 days, and defined chunk size=1000 while reading data from csv but but still ran into same issues of memory Error at Line data = self.mc.execute(method='get_leads_export_job_file', job_id=exportId)