How can I use scrapy middleware in the scrapy Shell?

Question

In a scrapy project one uses middleware quite often. Is there a generic way of enableing usage of middleware in the scrapy shell during interactive sessions as well?

Answer 1

Although, Middlewares set up in setting.py are enabled by default in scrapy shell. You can see it on the logs when running scrapy shell.

So to answer your question, yes you can do so using this command.

scrapy shell -s DOWNLOADER_MIDDLEWARES='<<your custom middleware>>' 

You can override settings using the -s parameters.

Remember, just running scrapy shell inside a folder that contains a scrapy project. It will load the default settings from settings.py.

Happy Scraping :)