firebase-queue, there is a delay in the rate at which the worker receives task

Question

I push 100 tasks to the queue on the client side like this.

var ref = firebase.database().ref('queue/tasks');
for(var i = 0;i<100;i++){
 ref.push({'foo': 'bar',i:i})
};

At the server, my worker looks like this

var queue = new Queue(ref, function(data, progress, resolve, reject){
  console.log(data);
  resolve();
}

The issue is that to complete processing all the tasks, it takes about 60 seconds, which is way to slow. Is there any way my worker can receive the tasks faster. I want to use the queue for the my client to send a request to the server. But with the current speed of the queue, I will not be able to support many concurrent users. I am looking to support 50k concurrent users.

Answer 1

One thing I noticed is that you don't have any options being passed in, and you can set a number of workers that will pull simultaneously from your Queue.

options = {
  'numWorkers': 5,
  'sanitize': false,
  'suppressStack': true,
}    
var queue = new Queue(ref, options, function(data, progress, resolve, reject){
  console.log(data);
  resolve();
})

However, while I did notice a speed increase on my machine, with 1 worker my process ran through 100 in about 9 seconds, but with 5 concurrent workers it finished in about 5 seconds.

I'm on a brand new 13" MacBookPro with the i7 chip.

I used performance-now to perform my benchmarks by doing this before the Queue is set up:

var start = now();

and this after the resolve() inside the Queue

console.log(((now()-start)/1000).toFixed(3));