Hi I am trying to crawl a website, specifically www.cvedetails.com and all the CVEs and after about 600 CVEs I got this error:
Maximum execution time of 30 seconds exceeded in phar:///root/PHP/VULDRONE/goutte.phar/vendor/symfony/dom-crawler/Symfony/Component/DomCrawler/Crawler.php on line 173
I have changed the max_execution_time value in the php.ini to 900 seconds but apparently it doesn't work fine.
I am using also a sleep after X requests but no success, have tried to change the user-agent and the same.
I have no idea what can I do not for being blocked from the crawling.
Thank you in advance