The goal of my script is to retrieve all links of a website using the PuppetterCrawler from Crawlee. I was wondering how to skip the links who have rel="nofollow" has attribute. I have tried to exclude some urls on the enqueueLinks() function using that condition :
request.url.startsWith(url) && !request.url.includes('#') && !extensions.some(ext => request.url.endsWith(ext));
I expect to not have the urls who redirect to nothing.