Run multiple puppeteer executions with puppeteer-cluster

986 Views Asked by At

How do you run multiple concurrent executions with puppeteer-cluster?

I have 5 as max concurrency, but doing await cluster.execute() only runs 1 at a time.

const cluster = await Cluster.launch({
   monitor: false,
   concurrency: Cluster.CONCURRENCY_BROWSER,
   maxConcurrency: 5,
})

for(let url of urls) {
   await cluster.execute(url)
}

I want all 5 to be executed concurrent. The alternative is await cluster.queue(), but after a while, the memory is being eaten up, because there is no way to check for current queue length/size.

1

There are 1 best solutions below

1
On

change concurrency: Cluster.CONCURRENCY_BROWSER to concurrency: Cluster.CONCURRENCY_CONTEXT