How can I write tests for code using twisted.web.client.Agent and its subclasses?

Question

I read the official tutorial on test-driven development, but it hasn't been very helpful in my case. I've written a small library that makes extensive use of twisted.web.client.Agent and its subclasses (BrowserLikeRedirectAgent, for instance), but I've been struggling in adapting the tutorial's code to my own test cases.

I had a look at twisted.web.test.test_web, but I don't understand how to make all the pieces fit together. For instance, I still have no idea how to get a Protocol object from an Agent, as per the official tutorial

Can anybody show me how to write a simple test for some code that relies on Agent to GET and POST data? Any additional details or advice is most welcome...

Many thanks!

Answer 1

How about making life simpler (i.e. code more readable) by using @inlineCallbacks.

In fact, I'd even go as far as to suggest staying away from using Deferreds directly, unless absolutely necessary for performance or in a specific use case, and instead always sticking to @inlineCallbacks—this way you'll keep your code looking like normal code, while benefitting from non-blocking behavior:

from twisted.internet import reactor
from twisted.web.client import Agent
from twisted.internet.defer import inlineCallbacks
from twisted.trial import unittest
from twisted.web.http_headers import Headers
from twisted.internet.error import DNSLookupError


class SomeTestCase(unittest.TestCase):
    @inlineCallbacks
    def test_smth(self):
        ag = Agent(reactor)
        response = yield ag.request('GET', 'http://example.com/', Headers({'User-Agent': ['Twisted Web Client Example']}), None)
        self.assertEquals(response.code, 200)

    @inlineCallbacks
    def test_exception(self):
        ag = Agent(reactor)
        try:
            yield ag.request('GET', 'http://exampleeee.com/', Headers({'User-Agent': ['Twisted Web Client Example']}), None)
        except DNSLookupError:
            pass
        else:
            self.fail()

Trial should take care of the rest (i.e. waiting on the Deferreds returned from the test functions (@inlineCallbacks-wrapped callables also "magically" return a Deferred—I strongly suggest reading more on @inlineCallbacks if you're not familiar with it yet).

P.S. there's also a Twisted "plugin" for nosetests that enables you to return Deferreds from your test functions and have nose wait until they are fired before exiting: http://nose.readthedocs.org/en/latest/api/twistedtools.html

Answer 2

How about this? Run trial on the following. Basically you're just mocking away Agent and pretending it does as advertised, and using FakeAgent to (in this case) fail all requests. If you actually want to inject data into the transport, that would take "more doing" I guess. But are you really testing your code, then? Or Agent's?

from twisted.web import client
from twisted.internet import reactor, defer

class BidnessLogik(object):
    def __init__(self, agent):
        self.agent = agent
        self.money = None

    def make_moneee_quik(self):
        d = self.agent.request('GET', 'http://no.traffic.plz')
        d.addCallback(self.made_the_money).addErrback(self.no_dice)
        return d

    def made_the_money(self, *args):
        ##print "Moneeyyyy!"
        self.money = True
        return 'money'

    def no_dice(self, fail):
        ##print "Better luck next time!!"
        self.money = False
        return 'no dice'

class FailingAgent(client.Agent):
    expected_uri = 'http://no.traffic.plz'
    expected_method = 'GET'
    reasons = ['No Reason']
    test = None

    def request(self, method, uri, **kw):
        if self.test:
            self.test.assertEqual(self.expected_uri, uri)
            self.test.assertEqual(self.expected_method, method)
            self.test.assertEqual([], kw.keys())
        return defer.fail(client.ResponseFailed(reasons=self.reasons,
                                                response=None))

class TestRequest(unittest.TestCase):
    def setUp(self):
        self.agent = FailingAgent(reactor)
        self.agent.test = self

    @defer.inlineCallbacks
    def test_foo(self):
        bid = BidnessLogik(self.agent)
        resp = yield bid.make_moneee_quik()
        self.assertEqual(resp, 'no dice')
        self.assertEqual(False, bid.money)

Answer 3

This is similar to what mike said, but attempts to test response handling. There are other ways of doing this, but I like this way. Also I agree that testing things that wrap Agent isn't too helpful and testing your protocol/keeping logic in your protocol is probably better anyway but sometimes you just want to add some green ticks.

class MockResponse(object):
    def __init__(self, response_string):
        self.response_string = response_string

    def deliverBody(self, protocol):
        protocol.dataReceived(self.response_string)
        protocol.connectionLost(None)


class MockAgentDeliverStuff(Agent):

    def request(self, method, uri, headers=None, bodyProducer=None):
        d = Deferred()
        reactor.callLater(0, d.callback, MockResponse(response_body))
        return d

class MyWrapperTestCase(unittest.TestCase):

    def setUp:(self):
        agent = MockAgentDeliverStuff(reactor)
        self.wrapper_object = MyWrapper(agent)

    @inlineCallbacks
    def test_something(self):
        response_object = yield self.wrapper_object("example.com")
        self.assertEqual(response_object, expected_object)