Loading Multiple Json responses from API

167 Views Asked by At

I am trying to parse individual product pages for information on each product and I am constructing the page link from the categories API,

def parse_categories(self, response):
    resps = json.loads(response.body)
    prods = resps.get("data").get("attributes").get(
        "main-content")[1].get("records")
    for prod in prods:
        product_link = prod.get("pdp-url")
        yield scrapy.Request(f"https://shop.lululemon.com/api{product_link}", callback=self.parse, meta={
            'pdp-url': product_link})

my problem is that my function is not returning a JSON object for my following function to parse, whenever I run it all I get is a 'NONE' value.

The start of my second function is

def parse(self, response):
    resp = json.loads(response.body)
    print(resp)

Anyone have an idea how to load the json response that I should be getting with the first scrapy request?

0

There are 0 best solutions below