rows missing in mysql

Question

I am trying to import data from CSV with rows 1.7 million approx in MYSQL using bulk insert after importing I get only 1.04 million rows for all the tables.

I used

select count (*) from table_name;

command to check the number of rows.

for crosschecking the rows I used the pandas' module in python.

Answer 1

For an import job of that size, you'll be wise to break up your CSV into chunks of about 50,000 rows each, then import them one by one. Watch for errors on each import job, and COUNT(*) the rows after each one.

You will probably find some rows in your data did not load correctly. Some reasons:

• Letters where there should be numbers
• Inconsistently formatted dates
• Text fields longer than allowed by the column in the table
• Too many, or not enough, commas
• Bad field escaping: In CSV when you want to write Stephen "The Rifleman" Flemmi you must write Stephen ""The Rifleman"" Flemmi doubling the quote characters.