I use scrapy-splash v3. I have 4 scrapy and 4 scrapy-splash total 8 crawlers. They start at intervals of 3 hours once a day.
01:00 PM > crawler-splash1
04:00 PM > crawler-scrapy1
07:00 PM > crawler-splash2
...
About 40 hours (1.5 day) after the first crawler starts, the server crashes or splash fails like
http://172.18.0.10:8050/render.html> (failed 1 times): Connection was refused by other side: 111: Connection refused.
When I restart the server, the connection is fixed and it starts working normally up to 2 days again.
What could go wrong? Are the server properties low or another problem? I have no idea. I tried to get the log, but it is empty, no error in log. But I have to restart the server every day.
My vp server : 2 vCore(s) 2.4 GHz 8 GB RAM 40 GB SSD Linux
Edit Docker:
docker run -d --restart always --net pabedanetwork --ip 172.18.0.10 scrapinghub/splash -v3
Crawler
venv/bin/scrapy crawl crawler -a retailer=RetailerName