Is there a more functional way to do this sequence of requests?

Question

There is a REST api which returns a json with a list of jsons (named results) and a url to the next batch of results (this url is equal to null for the last "page"). I want to make an aggregate of the whole results (concatenate all the lists of jsons into a single one). I am using spray-client to do the GET requests and this is what I came up with:

  val request: HttpRequest => Future[MyResponse] = sendReceive ~> unmarshal[MyResponse]
  def getCandidatesStartingFrom(url: String): Future[List[Candidate]] =
      request(Get(url)).flatMap {
        response =>
            val next = response.next match {
              case Some(nextUrl) => getCandidatesStartingFrom(nextUrl)
              case None => Future.successful(Nil)
            }
            next.map(response.results ++ _)
      }

My question is: is there a way to make this more functional? (avoid the recursion or maybe make it tail recursive?). Or even, does spray support idiomatically this kind of aggregations?

Answer 1

request(Get(url)).flatMap {
        response =>
            val next = response.next match {
              case Some(nextUrl) => getCandidatesStartingFrom(nextUrl)
              case None => Future.successful(Nil)
            }
            next.map(response.results ++ _)
      }

can be written as the following. But it's not that much of an improvement.

for{   
  response <- request(Get(url))   
  nextUrl <- response.next
               .map(getCandidatesStartingFrom)
               .getOrElse(Future.successful(Nil)) 
} yield response.results ++ nextUrl

Answer 2

It's not currently recursive. Your "recursive" call to getCandidatesStartingFrom is not actually in the call-stack of the parent call to getCandidatesStartingFrom, but rather in the anonymous lambda passed to flatMap.

That is, getCandidatesStartingFrom:

1. asks spray to begin an HTTP request (which immediately returns a future)
2. calls flatMap on that future with an anonymous function to run when it completes. this, too, immediately returns a future.
3. returns that future to the caller