Using CsvBulkLoader when no header row in the csv file in Silverstripe #silverstripe

Question

For CSV file import, Silverstripe provides a good class CsvBulkLoader . But it requires the first row in the file to be column names. In my case, the customer files dont have header row, and the first row is also data. Is there an easy way to do this in this case? The problem is it wants to know which column will go to which field in the database.

Answer 1

Actually after doing it, I found out that Silverstripe CsvBulkLoader class is not efficent. If file is big it runs out of memory. Also, it doesnt allow ploading zipfile but csv file is too large to upload. I wrote my own custom class which is very fast and can read cvs file contents from zip. In silverstripe to find duplicates, they search in list whether an object exists. It means each time it needs to check a record existance, it traverses the list from start. The csv file I have has near one million records. For silverstripe takes several ours to run it. But my class with checks duplicates by isset array key and also pulls existing records by raw sql. Silverstripe datamodel when looped makes another query for each loop which means near one million queries. Raw sql runs very fast.

Answer 2

You can set public $hasHeaderRow = false; in your custom CsvBulkUploader class. Example:

class MyCsvBulkUploader extends CsvBulkUploader {
    public $hasHeaderRow = false;
    //custom code here
}

Note that you'll need to write your own custom CsvBulkUploader for anything beyond the simplest case. There's some helpful tutorials on how to write a custom uploader at the bottom of the page at http://doc.silverstripe.org/framework/en/trunk/howto/csv-import