Let's start with a straightforward definition of foldRight:
def foldRight[T, U](base: U)(f: (T, => U) => U)(as: Seq[T]): U = {
as match {
case Nil => base
case head +: next => f(head, foldRight(base)(f)(next))
}
}
One of the advantages of such combinator is that it allows us to write something like (I use an if to make the short-circuiting behaviour of || more explicit):
def containsElement[T](e: T)(as: Seq[T]): Boolean = {
foldRight(false)((el: T, acc) => if (el == e) true else acc)(as)
}
Which then works with infinite structures:
val bs = 0 #:: 1 #:: 2 #:: 3 #:: LazyList.continually(1)
containsElement(3)(bs)
However, it doesn't work with very looooong sequences, because we are blowing up the stack:
val veryLongList = List.fill(1_000_000)(0) :+ 3
containsElement(3)(veryLongList)
... would result in a java.lang.StackOverflowError.
Enter the scala.util.control.TailCalls. We can write a very specialised implementation of containsElement that takes advantage of TCO, such as:
def containsElement[T](e: T)(as: Seq[T]) = {
import scala.util.control.TailCalls._
def _rec(as: Seq[T]): TailRec[Boolean] = {
as match {
case Nil => done(false)
case head +: next => if (head == e) done(true) else _rec(next)
}
}
_rec(as).result
}
But now I want to generalize this to foldRight. The following code was as far as I got via incremental refactoring, but if I keep following this path, I will bump into the fact that I would need to change the f signature to f: (T, => TailRec[U]) => U which is not what I wanted at all:
def containsElement[T](e: T)(as: Seq[T]) = {
import scala.util.control.TailCalls._
val base = false
def f(head: T, next: => TailRec[Boolean]): TailRec[Boolean] = if (head == e) done(true) else next
def _rec(as: Seq[T]): TailRec[Boolean] = {
as match {
case Nil => done(base)
case head +: next => f(head, _rec(next))
}
}
_rec(as).result
}
Question: how can we create an implementation of foldRight that (a) preserves the [T, U](base: U)(f: (T, => U) => U)(as: Seq[T]): U signature, (b) works on infinite structures, and (c) doesn't blow up with a StackOverflowError in very long structures?
It can't be done (at least not on a single JVM thread with limited stack).
TL;DR
The combination of requirements (a), (b), (c) inevitably leads to pathological solutions (see "Inter-Thread Recursion" below as well as the Addendum).
In the signature
the type
(T, => U) => Umeans thatfcannot return until all the subsequent invocations offhave returned;fmust be able to hold some data in its stack frame from the beginning to the end;fthat coexist simultaneouslyNo amount of tricks with trampolines will help, because you cannot change the way causality works.
The above argument leads to a working implementation (see "Inter-thread recursion" section below). However, spawning thousands of threads as mere containers for stack frames is very unnatural, and indicates that the signature should be changed.
Overview
Instead of just giving the code snippet with the ready solution, we want to explain how to arrive at it. Indeed, we want to arrive at two different solutions:
TailRec/Eval.The rest of this answer is structured as follows:
We first take a look at some failed approaches, just to understand the problem better, and to collect some requirements:
The "Inter-Thread Recursion" section presents the solution that formally satisfies all three requirements ((a), (b), (c)) of the question, but spawns and unbounded number of threads. Since spawning multiple threads is not a viable solution, this forces us to give up the original signature.
Then we investigate alternative signatures:
TailReccan be replaced bycats.Eval.Naive Recursion
The naive recursion attempted in the first paragraph of the question (and also mentioned in numerous blogs and articles, such as here) has two seemingly contradictory properties:
There is no paradox here. The first property means that if the solution can be determined by looking just at the few elements at the beginning of a sequence, the naive-recursive
foldRightcan skip the infinite number of elements in the tail. However, the second property means that the solution cannot look at too many elements at the beginning of the sequence.The first property is the good part of this solution: we definitely should give
fthe possibility to skip the rest of the sequence once it has processed the data that it actually needed.Failed
TailRecIn the comments to the question this
TailRec-based solution was mentioned:The problem here is that it does not actually work on infinite streams (because it doesn't give
fthe opportunity to decide whether it wants to continue or not, so it cannot "stop early", before "seeing the end of an infinite stream" - which never happens).It is "avoiding" the problem with an unbounded number of simultaneously coexisting stack frames of
fby refusing thek-th invocation the possibility to decide whether the(k+1)-th is required or not. For finite lists, this allows to dofto begin with, this scheme falls apart.The good thing about this solution is the idea with
TailRectrampolining. While it's insufficient on its own, it will come in handy later on (in "TailRec Done Properly").Inter-Thread Recursion
Suppose that we don't want to make the same mistake as in the previous section: we definitely want to let
fdecide whether the rest of the sequence should be looked at or not.In the introduction, we claimed that this leads to the requirement that an unbounded number of
fstack frames must be able to simultaneously coexist in the memory.To see this, we need just a single counterexample that makes it obvious that we must be able to keep an unbounded number of stack frames in memory.
Let's specialize the signature of
f: (T, => U) => UforT = BooleanandU = Unit, and consider a list consisting of one milliontrues, followed by a singlefalse.Suppose that
fis implemented as follows:The very first invocation of
fx;u, invokingffor the second, third, fourth... millionth time.fs have exited, it must still have access to thexin its stack frame.Therefore, one million random values must be stored in one million local variables
xin one million stack frames, all of which must be active at the same time.On the JVM, there is no way to take a stack frame and convert it into something else (a heap-allocated object, say).
Unless you tweak the JVM settings and allow the stack to grow indefinitely, you must limit the height of the stack. Having an unbounded number of stack frames while respecting the maximum stack height means that you need multiple stacks. In order to have multiple stacks, you need to start multiple threads.
This indeed leads to a solution that formally satisfies all three of your requirements:
It keeps your
foldRightsignature exactly as it was (a), and it works for both your test-cases (the infinite stream withoutbase-case (b), as well as the list with a million entries (c)).But creating an unbounded number of threads as mere containers for stack-frames is clearly insane. Therefore, if we want to keep (b) and (c), we are forced to abandon the signature (a).
No Frameworks: Returning
Either[U => U, U]How can we modify the signature so that we can solve (b) and (c), but without creating multiple threads?
Here is one simple and illustrative solution that gets by with a single thread, and also doesn't rely any pre-existing stack-safety/trampolining frameworks:
It works on both of your examples. The helper-method
recis obviously tail-recursive, it doesn't need any difficult-to-grasp libraries.The change in the signature is that
fis allowed to look atTand then to return anEither[U => U, U], with theRight[U]corresponding to the final result of currentrec-invocation, andLeft[U => U]corresponding to the case where it needs to look at the result of the tail and postprocess it in some way.The reason why this solution works is that it creates heap-allocated closures
U => Uwhich are stashed in an ordinaryList. The information that was previously held in the stack frames is moved to the heap, so there is no potential for stack overflows.This solution has at least one drawback, though: for simple functions like
containsElement, it would create a very longList[U => U]that holds just identity functions that don't do anything. Even the GC complains about it:Can we get rid of this list somehow? (let's call this new requirement (d)).
No Postprocessing
The reason why we needed a
List[U => U]in the previous section is thatfneeds a way to "post-process" the results returned from the tail of the sequence. Since simple functions likecontainsElementdon't actually need this, we might be tempted to experiment with simpler recursion schemes, such as the following:Unfortunately, even though it's sufficient for
containsElement, it's not as expressive as a truefoldRight(seenestBracketsandfoldNonassocOpin the "Full Code" section for concrete examples of functions that are not expressible throughcollectFirst).Back to
TailRec:TailRecDone RightHaving looked at a few other failed attempts, we are returning to
TailRec:which can be used like this:
This satisfies (b), (c) and (d). Also note how similar it is to the naive recursion with nested helper method (see Full Code).
Full code
Requires Scala 3.
For the
cats.Eval, you need a basic Cats project that can be generated withOutput for all experiments in one big table: