Google search results limits

Question

I tried to search in google search engine the word "sunday".

The total number of results is 1.390.000.000. However I can see only the first 420 results, until 42 page of results.

Is there any way to take all the results of google search?

Answer 1

You can use the google module to control the number of results from your query.

Install it using pip:

pip install google

Usage:

from google import search

n = 10  # number of results
query = 'sunday'
results = google.search(query, stop=n)  # returns a generator

for result in results:
    print(result)

Before using this method, keep this in mind about the pause parameter:

pause (float) - Lapse to wait between HTTP requests. A lapse too long will make the search slow, but a lapse too short may cause Google to block your IP. Your mileage may vary!

Docs

Answer 2

This 'feature' of google - and other search engines - applies to the web interface too. The claimed number of matches on the first page is not reflected in the actual number of results returned.

For example, if you search Google for "Systematic Literature Review" it will claim a few milion results on the first page, but if you go to page 3 (at 100 results/page) it will 'revise' the estimate to 200-300 results.

This seems too high in the initial estimate but may be a problem with filtering large datasets and is possibly forgivable. However it is definitely too small a number of actual results returned for this topic. Bing and DuckDuckGo are similar. Google Scholar claims 11k or so results but returns a server error if you try going beyond 1000.

Speculating: this might be in order to encourage people to change their search terms, rather than return a huge number of results that are then filtered programmatically locally (which is what I would like to do!).

This isn't an answer to getting more results I'm afraid, but it is an explanation that the problem is not the library, it's the search engine(s).