I'm scraping a page that have 36 <@hrefs in the same <@div (bold) xpath, so when i try to get those, even on scrapy shell, it gets only the same 12 <@hrefs all the time, and it's not in order.
i'm using this way: response.xpath('/html/body/div[1]/div[2]/section/div/div[3]/div[2]/div/div[2]//div//article//div[1]// a[re:test(@href,"pd")]//@href').getall()
it's from this following page: https://www.lowes.com/pl/Bottom-freezer-refrigerators-Refrigerators-Appliances/4294789499?offset=36
Seems that part of the html is dynamically loaded, so scrapy cannot see it. The data itself is present in a json-structure within the html. You can try to get it like this: