I want to scrape google as fast as possible. Google in browser loads progressively in which loading results is divided into Waiting(TTFB)
and Content Downlaod
.
I am using python requests
to scrape google. The timing of requests
is only Waiting(TTFB)
and a little Content Download
.
Code is as below:
google_url = 'https://www.google.com/search?q={}&num={}&start={}'.\
format(escaped_search_term, size, offset)
response = self.session.get(google_url, verify=False,
headers={'User-Agent': self.USER_AGENT})
return response.text
I want to load result as fast as possible, so i have to download google results progressively like browser. How can i do this?