I am using Crawlera as a proxy manager, but it has a limited number of concurrent requests. When a single crawler is running (using scrapyd), I stay under that limit, so I get no errors. However, when multiple scrapy instances are running at the same time using scrapyd, I get 429 responses by Crawlera.
Is there a way to limit the overall number of concurrent requests across scrapyd so that I still use the maximum I can when a spider is running alone, but limit when I have multiple at the same time?
I know I could set the maximum concurrent requests of each scrapy spider to be crawlera_max/max_num_of_spiders, but that would be too limiting when a spider runs alone.
Thank you in advance, Aimilios