could not access AnyRef in package Scala

Question

I'm using apache toree (version from github). When i'm trying to execute a query against a postgresql table, i'm getting intermittent scala compiler errors (when i run the same cell twice, the errors are gone and the code runs fine).

I am looking for advice on how to debug these errors. The errors look weird (they appear in the notebook nog on stdout).

error: missing or invalid dependency detected while loading class file 'QualifiedTableName.class'.
Could not access type AnyRef in package scala,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'QualifiedTableName.class' was compiled against an incompatible version of scala.
error: missing or invalid dependency detected while loading class file 'FunctionIdentifier.class'.
Could not access type AnyRef in package scala,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'FunctionIdentifier.class' was compiled against an incompatible version of scala.
error: missing or invalid dependency detected while loading class file 'DefinedByConstructorParams.class'.
...

The code is simple: extract a dataset from a postgres table:

%AddDeps org.postgresql postgresql 42.1.4 --transitive
val props = new java.util.Properties();
props.setProperty("driver","org.postgresql.Driver");
val df = spark.read.jdbc(url = "jdbc:postgresql://postgresql/database?user=user&password=password", 
                table = "table", predicates = Array("1=1"), connectionProperties = props)
df.show()

i checked for the obvious (both toree and apache spark use scala 2.11.8, i built apache toree with APACHE_SPARK_VERSION=2.2.0 which is the same as the spark i donwloaded)

For reference, this is the part of the Dockerfile i used to set up toree and spark:

RUN wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz && tar -zxf spark-2.2.0-bin-hadoop2.7.tgz && chmod -R og+rw /opt/spark-2.2.0-bin-hadoop2.7 && chown -R a1414.a1414 /opt/spark-2.2.0-bin-hadoop2.7
RUN (curl https://bintray.com/sbt/rpm/rpm > /etc/yum.repos.d/bintray-sbt-rpm.repo)
RUN yum -y install --nogpgcheck sbt
RUN (unset http_proxy; unset https_proxy; yum -y install --nogpgcheck java-1.8.0-openjdk-devel.i686)
RUN (git clone https://github.com/apache/incubator-toree && cd incubator-toree && make clean release APACHE_SPARK_VERSION=2.2.0 ; exit 0)
RUN (. /opt/rh/rh-python35/enable; cd /opt/incubator-toree/dist/toree-pip ;python setup.py install)
RUN (. /opt/rh/rh-python35/enable; jupyter toree install --spark_home=/opt/spark-2.2.0-bin-hadoop2.7 --interpreters=Scala)

Answer 1

I had a similar issue, but it appeared to resolve itself by merely reevaluating the cell in the Jupyter notebook, or by restarting the kernel and then reevaluating the cell. Annoying.

Answer 2

As said in cchantep's comment, you are probably using a different Scala version than the one used to build Spark.

The easiest solution is to check which one is used by Spark, and switch to this one, eg on Mac:

brew switch scala 2.11.8