I am using myworkbench 8.0 to import data to MySql 8.0.32 from a 16G csv file. I get "Error Code: 5. Out of memory (Needed 784334881 bytes)". Machine spec: Win 10, 8Gb ram. CSV is | delimited.
Import statement used:
LOAD DATA INFILE "C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/IT14_Prov20230412P.csv" INTO TABLE _original CHARACTER SET latin1 FIELDS TERMINATED BY '|' IGNORE 1 ROWS;
My.ini was set as follows:
innodb_buffer_pool_size=1G innodb_buffer_pool_instances=8
I tried changing to innodb_buffer_pool_size=2G. The problem still persist.
To test the file content, I tried looping through the file using Python. The content looks fine but the loop always stops at line 411452 and all of a sudden, the machine freezes with no error. I don't know if stopping of the loop, machine freezing and the error above are related.
Here is my Python test code:
with open(file_name, "r", encoding='latin-1') as file: csv_dict_reader = csv.DictReader(file, delimiter='|') for row in csv_dict_reader: print(csv_dict_reader.line_num - 1, row)