how to scrape google result progressively?

185 Views Asked by At

I want to scrape google as fast as possible. Google in browser loads progressively in which loading results is divided into Waiting(TTFB) and Content Downlaod.

enter image description here

I am using python requests to scrape google. The timing of requests is only Waiting(TTFB) and a little Content Download.

enter image description here

Code is as below:

google_url = 'https://www.google.com/search?q={}&num={}&start={}'.\
        format(escaped_search_term, size, offset)
response = self.session.get(google_url, verify=False,
                            headers={'User-Agent': self.USER_AGENT})
return response.text

I want to load result as fast as possible, so i have to download google results progressively like browser. How can i do this?

0

There are 0 best solutions below