How to use Pyquery with scrapy?

Question

My objective is to use pyquery with scrapy, apparently from scrapy.selector import PyQuerySelector returns ImportError: cannot import name PyQuerySelector when I crawl the spider.

I followed this specific gist https://gist.github.com/joehillen/795180 to implement pyquery.

Any suggestions or tutorials that can help me get this job done?

Answer 1

You declare a class and make your rules and in the callback attribute of rule extractor give parse_item by default the scrapy goes parse() function

def parse_item(self, response):
    pyquery_obj = PyQuery(response.body)
    header = self.get_header(pyquery_obj)
    return {
        'header': header,
    }


def get_header(self, pyquery_obj):
    return pyquery_obj('#page_head').text()