I'm trying to scrape some URLs from Bing for a project. Once the Bing search has been found, I try to get all the links that were on the bing page into a list for further analysis. However, when using random keywords as the search, the list of URLs Found either work as expected or there are no links found. The keywords that no links change every time I use it.
def getBingLink(query):
l=[]
count = 0
num = random.random()
time.sleep(num)
target_url="https://www.bing.com/search?q=" +query
headers = {'User-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Norton/121.0.0.0'}
resp=requests.get(target_url, headers )
print(resp.status_code)
soup = BeautifulSoup(resp.text, 'html.parser')
completeData = soup.find_all("li",{"class":"b_algo"})
if (len(completeData) == 0):
return l
for i in range(0, len(completeData)):
o=completeData[i].find("a").get("href")
l.append(o)
count = count +1
return l
Today, when I use the word "doctor" as the keyword I get : []
When I use the words "doctor who" as the keyword I get:
[[https://en.wikipedia.org/wiki/Doctor_Who,...]
and other links which is what is expected. What am I doing wrong?