How to modularize puppeteer-cluster code?

190 Views Asked by At

I'm pretty new to this module. I heard it's better than just using puppeteer because it can run tasks in parallel. Anyway, I need help modularizing my code. Here's the example program in the npm page:

const { Cluster } = require('puppeteer-cluster');
 
(async () => {
  const cluster = await Cluster.launch({
    concurrency: Cluster.CONCURRENCY_CONTEXT,
    maxConcurrency: 2,
  });
 
  await cluster.task(async ({ page, data: url }) => {
    await page.goto(url);
    const screen = await page.screenshot();
    // Store screenshot, do something else
  });
 
  cluster.queue('http://www.google.com/');
  cluster.queue('http://www.wikipedia.org/');
  // many more pages
 
  await cluster.idle();
  await cluster.close();
})();

What I want is something like this:

main.js initializes the cluster

(async () => {
  const cluster = await Cluster.launch({
    concurrency: Cluster.CONCURRENCY_CONTEXT,
    maxConcurrency: 2,
  });
})();

functions.js has all the tasks

async function screenshot(){
// ...
}

async function getTitle(){
// ...
}

so I would then be able to do something like this:

functions.getTitle('http://www.google.com/');
functions.screenshot('http://www.wikipedia.org/');

instead of doing everything in one file. how can i do something like that?

0

There are 0 best solutions below