I'm learning to scrape search results from e-commerce platforms(Ebay, etc...) with nodeJS.
The problem I'm facing is that there are sponsored products in these platforms; such that these sponsored items will appear together with other non-sponsored but keyword-relevant items in the search result page.
When I use Postman to check the API responsible for the search results,
it appears that only the non-sponsored but relevant items can be retrieved from that API called.
As a result, simply calling API is not workable in this case as I also wanna scrape those sponsored items as well.
I would like to ask: using nodeJS, how to scrape both sponsored and non-sponsored items appeared in the search result page? I'm thinking about using certain packages such as jsdom or puppeteer, may I ask if my thought is on the right track? Thx a lots!
I think you should send GET requests with axios or etc. and then parse the whole webpage using puppeteer or REGEX , you are on the right path