I am using aiohttp to web scrape different websites. For most websites, my code works just fine, but for https://bestbuy.com/, my code is unable to even print a response code. Does anyone know why that is?
My Code
import asyncio
import aiohttp
head = {"User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'}
async def myDriver():
await httpBestBuy("https://www.amazon.com/")
await httpBestBuy("https://www.bestbuy.com/")
async def httpBestBuy(URL):
async with aiohttp.ClientSession(headers = head) as session:
async with session.get(URL) as page:
print(page.status)
asyncio.run(myDriver())
Output
200
I'm just trying to understand why it doesn't throw any errors for Best Buy; it just doesn't print out anything. Thank you in advance for the help :)
Notes
- Python 3.7.9
- Aiohttp 3.6.3
- I have tried with headers and without headers, there is no difference