Python twint library is not working in Colab environment

Question

I am trying to run a code using Python's twint library (Twitter scraper) in Colab.

My code is:

!pip install twint
!pip install nest_asyncio
!pip install pandas

import twint
import nest_asyncio
nest_asyncio.apply()
import time
import pandas as pd
import os
import re

timestr = time.strftime("%Y%m%d")

c = twint.Config()
c.Limit = 1000
c.Lang = "en"
c.Store_csv = True
c.Search = "apple"
c.Output = timestr + "_en_apple.csv"
twint.run.Search(c)

The above code works perfectly in Jupyter on my machine and fetches tweets. However, the same code in Colab results in the following:

 CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
 sleeping for 1.0 secs
 CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
 sleeping for 8.0 secs
 CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
 sleeping for 27.0 secs
 CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
 sleeping for 64.0 secs
 CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
 sleeping for 125.0 secs
 CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
 sleeping for 216.0 secs

How can this be fixed in Colab?

Answer 1

I got the following to work in Google Colab. Installing from requirements.txt is less hassle.

!git clone --depth=1 https://github.com/twintproject/twint.git
!cd /content/twint && pip3 install . -r requirements.txt
import twint
import nest_asyncio
nest_asyncio.apply()
import time
import pandas as pd
import os
import re
timestr = time.strftime("%Y%m%d")

c = twint.Config()
c.Limit = 1000
c.Lang = "en"
c.Store_csv = True
c.Search = "apple"
c.Output = timestr + "_en_apple.csv"
twint.run.Search(c)

Answer 2

for those who are failing the build, edit your requirements.txt like so;

aiohttp==3.7.0
aiogram==2.2
aiodns
beautifulsoup4
cchardet
dataclasses
elasticsearch
pysocks
pandas>=0.23.0
aiohttp_socks<=0.4.1
schedule
geopy
fake-useragent
googletransx

Answer 3

well, here is an updated code to run on google Colab.

!git clone --depth=1 https://github.com/twintproject/twint.git
!cd /content/twint && pip3 install . -r requirements.txt
!pip install aiohttp==3.7.0
!pip install nest_asyncio
# Import Libraries
import twint
import nest_asyncio
nest_asyncio.apply()
import time
import pandas as pd
import os
import re
timestr = time.strftime("%Y%m%d")
#configuration
c = twint.Config()   
c.Limit = 1000       
c.Lang = "en"        # Language
c.Store_csv = True
c.Search = "robin i love you"  # key words to look for.
c.Output = timestr + "_en_robin.csv"
twint.run.Search(c)