Why Scala runtime reflection no longer works on lambda?

Question

The following simple code:

import org.scalatest.FunSpec

class RuntimeMirrorSpike extends FunSpec {

  import org.apache.spark.sql.catalyst.ScalaReflection.universe._

  it("can reflect lambda") {

    val ll = { v: String =>
      v.toInt
    }

    val clazz = ll.getClass
    val mirror = runtimeMirror(clazz.getClassLoader)

    val sym = mirror.classSymbol(clazz)

    print(sym)
  }
}

used to work perfectly on Scala 2.11. But now it breaks on Scala 2.12:

assertion failed: no symbol could be loaded from class <...>.spike.RuntimeMirrorSpike$$Lambda$124/78204644 in package spike with name RuntimeMirrorSpike$$Lambda$124/78204644 and classloader sun.misc.Launcher$AppClassLoader@18b4aac2
java.lang.AssertionError: assertion failed: no symbol could be loaded from class <...>.spike.RuntimeMirrorSpike$$Lambda$124/78204644 in package spike with name RuntimeMirrorSpike$$Lambda$124/78204644 and classloader sun.misc.Launcher$AppClassLoader@18b4aac2
    at scala.reflect.internal.SymbolTable.throwAssertionError(SymbolTable.scala:184)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.classToScala1(JavaMirrors.scala:1061)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.$anonfun$classToScala$1(JavaMirrors.scala:1019)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.$anonfun$toScala$1(JavaMirrors.scala:130)
    at scala.reflect.runtime.TwoWayCaches$TwoWayCache.$anonfun$toScala$1(TwoWayCaches.scala:50)
    at scala.reflect.runtime.TwoWayCaches$TwoWayCache.toScala(TwoWayCaches.scala:46)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.toScala(JavaMirrors.scala:128)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.classToScala(JavaMirrors.scala:1019)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.classSymbol(JavaMirrors.scala:231)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.classSymbol(JavaMirrors.scala:68)

What is going on here? What kind of object doesn't have a runtime class?

Answer 1

The encoding of lambdas changed in 2.12 relative to 2.11. See the 2.12.0 release notes.

Before, lambdas were encoded in a way that's basically equivalent to Dmytro Mitin's answer as objects that extend a FunctionN and have an apply method.

In Java 8, lambdas were introduced, but in a slightly different way from Scala. The java.lang.invoke.LambdaMetaFactory.metafactory bootstrap method is called which will be create a class at runtime with one method which calls a method in another class.

In Scala 2.12, the encoding of Scala lambdas moved to match the Java implementation (likely because the JVM (especially HotSpot) recognizes that the result of the LambdaMetaFactory.metafactory result is a lambda and does more optimization than it otherwise would). However, the resulting class at runtime appears to not be mappable into a Scala class, causing Scala's reflection to blow up.

EDIT: the only fix apparent to me would be to manually encode lambdas as instances of FunctionN, after Dmytro Mitin's comment:

def delambdafy[Result](ll: () => Result): Function0[Result] = new Function0[Result] { def apply(): Result = ll() }

def delambdafy[In, Result](ll: In => Result): Function1[In, Result] =
  new Function1[In, Result] { def apply(in: In): Result = ll(in) }

And so forth for the necessary arities...

Adapting your code for a REPL, this works:

// Entering paste mode (ctrl-D to finish)

import scala.reflect.runtime.universe._         // Effectively what is being imported with your Spark import

def delambdafy[In, Result](ll: In => Result): Function1[In, Result] = new Function1[In, Result] { def apply(in: In): Result = ll(in) }

val ll: String => Int = { v: String => v.toInt }

val clazz = delambdafy(ll).getClass

val symbol = runtimeMirror(clazz.getClassLoader).classSymbol(clazz)

// Exiting paste mode, now interpreting.

import scala.reflect.runtime.universe._
delambdafy: [In, Result](ll: In => Result)In => Result
ll: String => Int = $$Lambda$5162/1405722527@114a083b
clazz: Class[_ <: String => Int] = class $anon$1
symbol: reflect.runtime.universe.ClassSymbol = $anon

Whether $anon is useful to you depends on what you were actually trying to do which led you to this question.