How can I apply iter() to a pagination api?

Question

I watched Raymond Hettinger's Idiomatic Python talk, and learned about the sentinel argument to iter(). I'd like to try to apply it to a piece of code I'm working on iterating over an API that uses pagination (it's Twilio, but not relevant to my question).

I have an API that returns: a list of data, and a next page URL. When the pagination is exhausted, the next page URL returns as an empty string. I wrote the fetching function as a generator and looks roughly like this:

def fetch(url):
    while url:
        data = requests.get(url).json()
        url = data['next_page_uri']
        for row in data[resource]:
            yield row

This code works fine, but I'd like to try to remove the while loop and replace it with a call to iter() using the next_page_uri value as the sentinel argument. Alternately, could this be written with a yield from?

Answer 1

I think this might be what you mean… but as stated in the comments, it doesn't help much:

def fetch_paged(url):
    while url:
        res = requests.get(url)
        res.raise_for_status()
        data = res.json()
        yield data
        url = data['next_page_uri']

def fetch(url):
    for data in fetch_paged(url):
        yield from data[resource]

(I've taken the opportunity to put in a call to raise_for_status() which will raise for non-successful, i.e. res.status_code < 400, responses)

not sure if it's any "better", but possibly if you're going to be reusing the fetch_paged functionality a lot

Note: lots of other APIs put this next_page_uri into the response headers in standard ways which the requests library knows how to deal with and exposes via the res.links attribute