So I recently got in a project that essentially scrapes tweets using twint library, saves the tweets in elastic search and on top of that there are some dashboards in Kibana for showing several metrics. This project has been running for the past one year and imagine that we have collected around 5M tweets.
Suddenly, yesterday the same code without any changes throws the above error.
Has anyone managed to solve this ? Or does anyone know if there is the possibility that somehow the twint scraper is banned or even if I have been banned? Are there any recommended scrapers doing the same job without this issues?
Thank you in advance.
- Python version is 3.8;
- Installed twint with pip install git+https://github.com/twintproject/twint.git@e7c8a0c764f6879188e5c21e25fb6f1f856a7221
- I have also tried to upgrade it with pip install --upgrade git+https://github.com/twintproject/twint.git@e7c8a0c764f6879188e5c21e25fb6f1f856a7221
I have searched a lot about the issue, I have seen that the repository for twint is no longer active and I have not figured it out yet. The most relevant issue that I found is #1257, which has not been solved.