How to force server-side insert on Meteor's collection.insert

Question

I've got a simple CSV file with 40,000 rows which I'm processing browser-side with papa-parse.

I'm trying to insert them one-by-one into a collection using the techniques in Discover Meteor and other 101 posts I find when Googling.

40000 insert browser-side pretty quickly - but when I check mongo server side it's only got 387 records.

Eventually (usually after 20 seconds or so) it starts to insert server-side. But if I close or interrupt the browser, the already-inserted records disappear obviously.

How do I force inserts to go server-side, or at least monitor so I know when to notify the user of success?

I tried Tracker.flush() no difference.

I'd go server-side inserts in a Meteor.method, but all the server-side CSV libraries are more complex to operate than client-side (I'm a beginner to pretty much everything programming :)

Thanks!

---

This is the main part of my code (inside client folder):

Template.hello.events({

    "submit form": function (event) {

        event.preventDefault();

        var reader = new FileReader();

        reader.onload = function (event) {

            var csv = Papa.parse(this.result, {header: true});

            var count = 0;

            _.forEach(csv.data, function (csvPerson) {
                count++;
                Person.insert(csvPerson);
                console.log('Inserting: ' + count + ' -> ' + csvPerson.FirstName);
            });

        };

        reader.readAsText(event.target[0].files[0]);
    }
});

---

The last few lines of console output:

Inserting: 39997 -> Joan 
Inserting: 39998 -> Sydnee 
Inserting: 39999 -> Yael 
Inserting: 40000 -> Kirk 

---

The last few lines of CSV (random generated data):

Jescie,Ayala,27/10/82,"P.O. Box 289, 5336 Tristique Road",Dandenong,7903,VI,[email protected]
Joan,Petersen,01/09/61,299-1763 Aliquam Rd.,Sydney,1637,NS,[email protected]
Sydnee,Oliver,30/07/13,Ap #648-5619 Aliquam Av.,Albury,1084,NS,[email protected]
Yael,Barton,30/12/66,521 Auctor. Rd.,South Perth,2343,WA,[email protected]
Kirk,Camacho,25/09/08,"Ap #454-7701 A, Road",Stirling,3121,WA,[email protected]

---

The hello template is a simple form obviously, just file select and submit. Client code is under client directory. Person defined in a file in application root. CSV parsed as strings for now, to avoid complexity. The records inserted look fine, retrieve by name, whatever.

Person.find().count() browser-side in console results in 40000.

Happy to send the file, which is only 1.5MB and it's random data - not sensitive.

Answer 1

I think call() should work as follows:

On client side

Meteor.call("insertMethod",csvPerson);

And method on server side

insertMethod: function(csvPerson){
Person.insert(csvPerson);
}

Answer 2

In Meteor, on some scenarios, if you don't pass a callback the operation will sync.

If you run the code Person.insert(csvPerson); on the server, the operation will be sync not async. Depending on what you want to do, you might have serious problems in the future. On the client, it won't be sync but async.

Since node.js is an event-based server, a single sync operation can halt the entire system. You've to be really about your sync operations.

For importing data, the best option is to do at server-side inside Meteor.startup(function(){ //import code goes here}).

The solution propose by Sindis works but it slow and if the browser closes (for some reason), you're not keeping a track of the already inserted records. If you use Meteor.call("insertMethod",csvPerson);, this operation will be sync on the client.

The best option on your beginner scenario (not optimal) is to:

1- While (You have record to insert)

2- Call Meteor.call without a callback

3- Count all the inserted fields in the Collection

4- Save this value to localStorage

5- Go back to step 1

This works assuming that the order of insertion is the same on every insert attempt. If you browser fails, you can always get the value from localStorage and skip that number of records.