Scala script engine not initialized when starting java -jar file. javax.script.ScriptException: Failed to compile ctx

Question

i am trying to compile scala code in runtime in java programm. I am using jsr232 api and my code looks like:

ScriptEngineManager manager=new ScriptEngineManager();
Scripted engine = (Scripted) manager.getEngineByName("scala");
engine.compile(sourceCode);

My pom looks like:

<properties>
   <scala.version>2.13.10</scala.version>
</properties>

<dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-compiler</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-reflect</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
       </dependency>
</dependencies>

When i am running such code in IDE all is ok and i have no problems (all dependencies jars are in -classpath option) But after i try to package to jar with copy dependencies (so i have small jar with fat manifest and put all dependencies to lib folder).

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.2.0</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <outputDirectory>${project.build.directory}/lib</outputDirectory>
                    <overWriteReleases>false</overWriteReleases>
                    <overWriteSnapshots>false</overWriteSnapshots>
                    <overWriteIfNewer>true</overWriteIfNewer>
                </configuration>
</plugin>
<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.2.2</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <classpathPrefix>lib/</classpathPrefix>
                            <mainClass>$$$MYCLASS$$$</mainClass>
                        </manifest>
                        <manifestEntries>
                            <Class-Path>.</Class-Path>
                        </manifestEntries>
                    </archive>
                </configuration>
            </plugin>

and run my application i got null on (Scripted) manager.getEngineByName("scala");

After some research i found that javax.script.ScriptEngineManager contains scala engine factory; But 'javax.script.ScriptEngineManager#getEngineByName(String)' returns null if something go wrong during method execution(without printstacktrace). And my problem is on ScriptEngine engine = spi.getScriptEngine();

I manually try to create Scala script engine the same way as manager and Scala.Factory(Engine factory) do this with some debug settings:

if(engine==null) {
    log.warn("No scala script engine exists. Try load again");
    try {
        Settings settings = new Settings();
        settings.usejavacp().value_$eq(true);//same as scala code 
        settings.usemanifestcp().value_$eq(true); //same as scala code
                settings.Yreplclassbased().value_$eq(true); //same as scala code
                settings.verbose().value_$eq(true); //for more logs
                settings.debug().value_$eq(true); //for more logs
        Scripted.Factory fact = new Scripted.Factory();
        engine= Scripted.apply(fact,settings, ReplReporterImpl.defaultOut());
    }catch (Exception e){
        log.error("Scala compiler wasn't initialized",e);
        return;
    }
}

I got next errors. Top level error is:

javax.script.ScriptException: Failed to compile ctx
        at scala.tools.nsc.interpreter.shell.Scripted.<init>(Scripted.scala:89) ~[scala-compiler-2.13.10.jar:?]
        at scala.tools.nsc.interpreter.shell.Scripted$.apply(Scripted.scala:278) ~[scala-compiler-2.13.10.jar:?]
        at scala.tools.nsc.interpreter.shell.Scripted.apply(Scripted.scala) ~[scala-compiler-2.13.10.jar:?]

and some other errors from compiler:

java.lang.NullPointerException
        at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
        at scala.reflect.io.ManifestResources$$anon$3.close(ZipArchive.scala:434)
        at scala.tools.nsc.symtab.classfile.ReusableDataReader.reset(ReusableDataReader.scala:85)
......
java.io.IOException: class file 'file:..../target/lib/scala-library-2.13.10.jar(scala/Predef.class)' is broken
(class java.lang.NullPointerException/null)
        at scala.tools.nsc.symtab.classfile.ClassfileParser.scala$tools$nsc$symtab$classfile$ClassfileParser$$handleError(ClassfileParser.scala:126)
        at scala.tools.nsc.symtab.classfile.ClassfileParser$$anonfun$scala$tools$nsc$symtab$classfile$ClassfileParser$$parseErrorHandler$1.applyOrElse(ClassfileParser.scala:134)
        at scala.tools.nsc.symtab.classfile.ClassfileParser$$anonfun$scala$tools$nsc$symtab$classfile$ClassfileParser$$parseErrorHandler$1.applyOrElse(ClassfileParser.scala:132)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
......
java.lang.NullPointerException
        at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
        at scala.reflect.io.ManifestResources$$anon$3.close(ZipArchive.scala:434)
        at scala.tools.nsc.symtab.classfile.ReusableDataReader.reset(ReusableDataReader.scala:85)
        at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$2(ClassfileParser.scala:161)
......
java.io.IOException: class file 'file:..../target/lib/scala-library-2.13.10.jar(scala/Unit.class)' is broken
(class java.lang.NullPointerException/null)
        at scala.tools.nsc.symtab.classfile.ClassfileParser.scala$tools$nsc$symtab$classfile$ClassfileParser$$handleError(ClassfileParser.scala:126)
        at scala.tools.nsc.symtab.classfile.ClassfileParser$$anonfun$scala$tools$nsc$symtab$classfile$ClassfileParser$$parseErrorHandler$1.applyOrElse(ClassfileParser.scala:134)
        at scala.tools.nsc.symtab.classfile.ClassfileParser$$anonfun$scala$tools$nsc$symtab$classfile$ClassfileParser$$parseErrorHandler$1.applyOrElse(ClassfileParser.scala:132)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
        at scala.tools.nsc.symtab.classfile.ClassfileParser.parse(ClassfileParser.scala:144)

It looks like scala compiler for some reason can not read jar source files.

Java version: openjdk version "11" 2018-09-25 OpenJDK Runtime Environment 18.9 (build 11+28) OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

Updated after Dmytro answer

It helps. Thanks alot.

But still got 1 issue with runtime classloading. I want user be able to add new dependency jar runtime.

On first version i do it like: engine.intp().addUrlsToClassPath(new ArraySeq.ofRef<>(url)). And it works well(i mean well when run inside IDEA)

Now i try do it like:

private final ScalaCompilerClassLoader urlClassLoader = new ScalaCompilerClassLoader(new URL[]{},ClassLoader.getSystemClassLoader());

private static class ScalaCompilerClassLoader extends URLClassLoader {
    public ScalaCompilerClassLoader(URL[] urls, ClassLoader parent) {
           super(urls, parent);
    }

    public void add(URL url){
        super.addURL(url);
    }
}

and i use your code with my classLoader.

JavaUniverse.JavaMirror mirror = universe.runtimeMirror(urlClassLoader);

The issue is that i can add new jar to my urlClassLoader before first call to toolBox.eval(toolBox.parse(sourceCode)). After first toolBox.eval call newly injected dependencies are not recognized by compiler.

And now i have to create new toolBox after each new jar dependency injecting.

Answer 1

Could you try to replace

ScriptEngineManager manager=new ScriptEngineManager();
Scripted engine = (Scripted) manager.getEngineByName("scala");
engine.compile(sourceCode);

with

import scala.reflect.api.JavaUniverse;
import scala.tools.reflect.ToolBox;

// ...

scala.reflect.runtime.package$ runtime = scala.reflect.runtime.package$.MODULE$;
JavaUniverse universe = runtime.universe();
JavaUniverse.JavaMirror mirror = universe.runtimeMirror(this.getClass().getClassLoader());
scala.tools.reflect.package$ toolsReflect = scala.tools.reflect.package$.MODULE$;
ToolBox<?> toolBox = toolsReflect.ToolBox(mirror).mkToolBox(toolsReflect.mkSilentFrontEnd(), "");
int res = (int) toolBox.eval(toolBox.parse("1 + 1")); // 2

?

Does this change anything for you?

Using scala macro from java

How can I run generated code during script runtime?

How to compile and execute scala code at run-time in Scala3?

"eval" in Scala

---

Regarding your original code with JSR232 scripting. Could you check whether anything changes if you add fork := true (sbt syntax) to the build file? I guess in Maven this should be <fork>true</fork>.

Regarding your question with classloading. One option is to use protected method addURL

ClassLoader classLoader = this.getClass().getClassLoader();
URLClassLoader urlClassLoader = (URLClassLoader) classLoader;
Method addUrlMethod = URLClassLoader.class.getDeclaredMethod("addURL", URL.class);
addUrlMethod.setAccessible(true);
String url = "file:///home/dmitin/.cache/coursier/v1/https/repo1.maven.org/maven2/com/chuusai/shapeless_2.13/2.3.10/shapeless_2.13-2.3.10.jar";
addUrlMethod.invoke(urlClassLoader, new URL(url));
Object res = toolBox.eval(toolBox.parse("import shapeless._; 1 :: \"a\" :: HNil"));
// 1 :: a :: HNil

Another is to create a child class loader, new runtime mirror and toolbox

ClassLoader classLoader1 = new URLClassLoader(new URL[]{new URL(url)}, classLoader);
JavaUniverse.JavaMirror mirror1 = universe.runtimeMirror(classLoader1);
ToolBox<?> toolBox1 = toolsReflect.ToolBox(mirror1).mkToolBox(toolsReflect.mkSilentFrontEnd(), "");
Object res = toolBox1.eval(toolBox1.parse("import shapeless._; 1 :: \"a\" :: HNil"));
// 1 :: a :: HNil

---

Every toolbox has two mirrors (and two class loaders). The first is the one it was created with, i.e. the mirror of toolbox dependencies. The second is its own mirror toolbox.mirror, where new definitions live (the class loader is mirror.classloader).

See the following example, where I create a new class, instantiate it, add new JAR, and use previous definition

import scala.reflect.api.JavaUniverse;
import scala.reflect.api.Symbols;
import scala.reflect.api.Trees;
import scala.tools.reflect.ToolBox;
import java.net.URL;
import java.net.URLClassLoader;

public class App {
    public static void main(String[] args) throws Exception {
        new App().run();
    }

    private static scala.reflect.runtime.package$ runtime = scala.reflect.runtime.package$.MODULE$;
    private static JavaUniverse universe = runtime.universe();
    private static ClassLoader classLoader = App.class.getClassLoader();
    private static JavaUniverse.JavaMirror mirror = universe.runtimeMirror(classLoader);
    private static scala.tools.reflect.package$ toolsReflect = scala.tools.reflect.package$.MODULE$;
    private static ToolBox<?> toolBox = toolsReflect.ToolBox(mirror).mkToolBox(toolsReflect.mkSilentFrontEnd(), "");
    public static Symbols.SymbolApi symbol = toolBox.define((Trees.ClassDefApi) toolBox.parse("case class A(i: Int, s: String)"));

    public static Object a = toolBox.eval(toolBox.parse(
            "import scala.reflect.runtime.universe._;" +
            " q\"\"\"new ${App.symbol.asInstanceOf[Symbol]}(1, \"a\")\"\"\" "
    ));

    public void run() throws Exception {
        JavaUniverse.JavaMirror mirror1 = (JavaUniverse.JavaMirror) toolBox.mirror();
        ClassLoader classLoader1 = mirror1.classLoader();

        String url = "file:///home/dmitin/.cache/coursier/v1/https/repo1.maven.org/maven2/com/chuusai/shapeless_2.13/2.3.10/shapeless_2.13-2.3.10.jar";
        ClassLoader classLoader2 = new URLClassLoader(new URL[]{new URL(url)}, classLoader1);
        JavaUniverse.JavaMirror mirror2 = universe.runtimeMirror(classLoader2);
        ToolBox<?> toolBox2 = toolsReflect.ToolBox(mirror2).mkToolBox(toolsReflect.mkSilentFrontEnd(), "");
        Object res = toolBox2.eval(toolBox2.parse("import shapeless._; 1 :: \"a\" :: App.a :: HNil"));
        System.out.println(res); // 1 :: a :: A(1,a) :: HNil
    }
}