Scrapy-Splash how to add return variables to response

Question

i would like to run a splash script that does some things in lua and then returns those results to my scrapy bot. However, i can only receive the html body, but none of the variables inside the return statement

splash:go(...)
--lua/splash stuff
test = 500
return {
-- another SO thread stated that these have to be in JSON format? doesn't work either way though
test = test
}

i tried using the JSON-endpoint, but the result is the same. The Scrapy-Splash documentation doesn't really explain how to get these variables either.

So my question is - how can i receive arbitrary return variables from my lua script through the scrapy_plash.SplashRequest call?

this is my current splash request:

yield SplashRequest(url, self.parse,
    args={'lua_source': QuotesSpider.SPLASH_SCRIPT, 'wait': 0.5})

Answer 1

The default setting for the splashRequest is render.html, which only returns the html and doesn't execute the lua script. Adding the execute endpoint properly runs the lua script and thus returns the desired variables

yield SplashRequest(url, self.parse,endpoint='execute'
    args={'lua_source': QuotesSpider.SPLASH_SCRIPT, 'wait': 0.5})