What I am trying to do is search for specific term, get the list of websites that come up with when searching for that term, and store them as a list for future reference and analysis. Here is the code that I have so far that is working:
from googlesearch import search
print("Googlesearch package installed successfully!")
# set query to search for in Google
query = "Lysotracker"
# execute query and store search results
results = search(query, tld="com", lang="en", stop=25, pause=2)
# iterate over all search results and print them
for result in results:
print(result)
So this just gets the list of URLs and prints the results. What I then try to do is remove all of the extraneous address text and input the website title to a list with this code:
# Initialize a list to store website titles
website_titles = []
# Iterate over all search results
for result in results:
# Use regular expressions to extract the website title
title = re.search('(?<=://)(.*?)(?=/|$)', result).group(1)
# Append the website title to the website_titles list
website_titles.append(title)
# Display the list of website titles
print(website_titles)
However, my list just winds up having no entries. I am sure there has to be a way to do this. I have tried web scraping tutorials but I just get error after error when running even first few steps, so I tried this method. I can at least get websites, but now I can't make a list.
Thanks for you time.