I am trying to parse individual product pages for information on each product and I am constructing the page link from the categories API,
def parse_categories(self, response):
resps = json.loads(response.body)
prods = resps.get("data").get("attributes").get(
"main-content")[1].get("records")
for prod in prods:
product_link = prod.get("pdp-url")
yield scrapy.Request(f"https://shop.lululemon.com/api{product_link}", callback=self.parse, meta={
'pdp-url': product_link})
my problem is that my function is not returning a JSON object for my following function to parse, whenever I run it all I get is a 'NONE' value.
The start of my second function is
def parse(self, response):
resp = json.loads(response.body)
print(resp)
Anyone have an idea how to load the json response that I should be getting with the first scrapy request?