ProtocolError: Protocol error (Runtime.callFunctionOn): Target closed. - express.js and puppeteer-cluster

Question

I want to iterate over key-pairs of data.extractRules and get elements data from the page.

This snippet inside forEach loop is causing app crashes. I tried hardcoding key and cssSelector and tried this outside of forEach loop and it worked.

    const extractContent = {};
    if (data.extractRules !== null) {
      Object.entries(data.extractRules).forEach(async ([key, cssSelector]) => {
        extractContent[key] = await page.$$eval(cssSelector, (elements) =>
          elements.map((element) => element.outerHTML)
        );
      });
    }

Answer 1

I figured out solution

async function getSelectorContent(page, cssSelector) {
  return page.$$eval(cssSelector, (elements) =>
    elements.map((element) => element.outerHTML)
  );
}

const extractContent = {};
if (data.extractRules !== null) {
    await Object.entries(data.extractRules).reduce(
        async (item, [key, cssSelector]) => {
            await item;
            extractContent[key] = await getSelectorContent(page, cssSelector);
        },
        Promise.resolve()
    );
}

Answer 2

I Just had this issue, to be honest i couldn't find a solution in any of these comments or elsewhere, so i managed to find my own way around. Here is an overview of my understanding and solution.

By the way my javascript solution is inspired by @stramanu's solution in typscript.

Why this error?

The error (TargetCloseError: Protocol error (Runtime.callFunctionOn): Target closed) is typically associated with attempting to interact with a target (page or context) that has been closed. Generally the script works well but yet fails(like in my case), so it's possible that there might be an asynchronous operation scheduled after the browser or page is closed.

Solution:

Ensure all promises are resolved, make sure that all promises are resolved before closing the browser or page. You can use Promise.all to await multiple promises simultaneously.

like in my case, i just had to wrap my call in a Promise.all

await Promise.all([
  // Your Puppeteer operations
]);

// Close the page
await page.close();

// Close the browser
await browser.close();

In simple words, the idea is to wrap whatever iteration you are doing in a promise and make sure it is all resolved before you close the page and browser.

Works like charm✨