Why is scrapy shell returning an empty list when my XPath selector works as it should in the “Elements” tab of my Chrome browser?

Question

The XPath selector in Scrapy shell response.xpath('//div[@class="chr-lot-header__bid-details"]//span[@class="chr-lot-header__value-field"] returns an empty list while the same XPath selector selects the right html tag in the "Elements" tab of my Chrome browser. Here's the website the XPath selector is intended for: https://www.christies.com/en/lot/lot-5973059

The output I want the XPath selector to produce is "GBP 11,282,500".

Answer 1

I just checked the website you mentioned in your mentioned is getting the required information dynamically loaded, which means it can not be scrapped directly. Because scrapy only scrap statically available data not dynamically loaded data. To scrap dynamically loaded data either you need to mimic real time browser like you can use selenium/playwright and integrate there library inside your scrapy code or you can try to find API calls in network tab, which this required data is being loaded/fetched.