Error when trying to use dynamically loaded libraries

Question

I'm trying to dynamically load libraries instead of shading them into my JAR to reduce file size. It's a standalone application with a Bootstrap class and a Main class. The main class is responsible for loading libraries and calling the bootstrap class' execute(...) method.

The downloading of the artifacts and loading of the library classes using Class.forName(class, true, libraryLoader) works, but when I try to load the Bootstrap class it can't find the class definitions for the library classes.

Here is how I'm currently trying to achieve this:

//////// Main.java /////////
/* ... public static void main(String[] args) { */

// download libraries
// these URLs reference the local JAR files downloaded
// i'm not going to get into detail on how i'm doing this right now, as this works
URL[] libraryUrls = ...;

// create parent loader for libraries
final ClassLoader rootLoader = Main.class.getClassLoader(); // app class loader
ClassLoader libraryLoader = new DelegatingClassLoader(new URLClassLoader(
        libraryUrls,
        rootLoader
), rootLoader);

/* this passes, so the libraries are successfully loaded onto the loaders ucp */
Class.forName("org.eclipse.jgit.transport.CredentialsProvider", true, libraryLoader);

// find URL for JAR of bootstrap class
ProtectionDomain domain = Main.class.getProtectionDomain();
CodeSource source = domain.getCodeSource();
URL file = source.getLocation();

// create child loader with our JAR file
ClassLoader appLoader = new DelegatingClassLoader(new URLClassLoader(
        new URL[] { file }, libraryLoader), libraryLoader);

// load bootstrap class and execute
Class<?> bootstrap = Class.forName( // the error occurs here
        "my.package.Bootstrap",
        /* initialize */ true,
        appLoader
);

bootstrap.getMethod("execute", String[].class)
        .invoke(null, (Object) args);

/* } ... */

The DelegatingClassLoader first tries to load a class through the delegate and if that fails delegates to the parent. Here is the exact source.

The specific error I get:

Exception in thread "main" java.lang.NoClassDefFoundError: org/eclipse/jgit/transport/CredentialsProvider 
    at java.base/java.lang.Class.forName0(Native Method) 
    at java.base/java.lang.Class.forName(Class.java:467) 
    at my.package.Main.main(Main.java:108)
        // line 108 is the line where i call Class.forName("my.package.Bootstrap, ...) 

I was expecting the references to library classes to be resolved correctly because the class loader of the Bootstrap class should delegate to the library class loader, which I confirmed to have successfully 'loaded' the libraries.

Answer 1

A NoClassDefFoundError indicates that a class, who was successfully loaded from some ClassLoader, is statically referring to another class, whose definition that ClassLoader can not find.

That is, if a class A refers to a class B, the runtime does

A.class.getClassLoader().loadClass("the.fully.qualified.name.of.B");

Suppose you have the following ClassLoader hierarchy:

+---------+
| parent  |
+---------+
| B.class |
+---------+
     ^
     |
+---------+
| child   |
+---------+
| A.class |
| C.class |
+---------+

where A refers to B, and B refers to C, and we do child.loadClass("A"). Then, the following happens:

• child is asked to load A, finds its definition, defines the class, and then proceecds to resolve its references:
  • child is asked to load B, and can not find its definition. Therefore,

    • parent is asked to load B, finds its definition, defines the class, and then proceeds to resolve its references

      • parent is asked to load C, can not find its definition, and throws a ClassNotFoundException

      since the reference to C could not be resolved, a NoClassDefFoundError for C is thrown

That is, loading A from child fails with a NoClassDefFoundError for C, even though child has a definition for C.

I was expecting the references to library classes to be resolved correctly because the class loader of the Bootstrap class should delegate to the library class loader, which I confirmed to have successfully 'loaded' the libraries.

As we've just seen, that doesn't follow. Each class resolves its dependencies using its defining class loader, not the class loader that originally triggered the loading attempt.

So, how could this have happened? You have set up the following class loader hierarchy:

+-----------------+
| rootLoader      |
+-----------------+
| Main.class      |
| Bootstrap.class |
| Y.class         |
+-----------------+
         ^
         |
+---------------------------+
| libraryLoader             |
+---------------------------+
| CredentialsProvider.class |
| X.class                   |
+---------------------------+
         ^
         |
+-----------------+
| appLoader       |
+-----------------+
| Bootstrap.class |
| Y.class         |
+-----------------+

That is, your Bootstrap classes are on the classpath twice, in two different ClassLoaders. You're probably hoping that the extra classes in rootLoader do no harm, because the appLoader prefers its own definition over that of its ancestors. However, the libraryLoader doesn't see the definitions in the appLoader, and will delegate to the rootLoader.

Suppose that Bootstrap uses X, who uses Y, who uses CredentialsProvider. Then, the following happens:

• Bootstrap is defined by appLoader
• X is defined by libraryLoader
• Y is defined by rootLoader (!)
• CredentialsProvider is not found by rootLoader

That is, the extra class definitions obscure the origin of dependency problems. If a library class incorrectly depends on an application class, you don't get a NoClassDefFoundError for that application class, but a NoClassDefFoundError for one of its library dependencies. And if there are no library dependencies, the class will be loaded a second time, and you will have separate classes with the same name in the system, which can cause really hard to diagnose issues (for instance: why is that static field having different values depending on who uses it?).

For the sake of your sanity, and those who come after you, I therefore recommend that you do not put your application classes into two different class loaders in the hierarchy.

PS: I can't tell you the identity of X and Y, but you should be able to find them as follows:

1. determine the runtime type of rootLoader
2. create a conditional breakpoint in its loadClass() implementation that triggers if the argument ends with "CredentialsProvider"
3. once your program hits the breakpoint, inspect the stack trace. You should see the recursive calls to loadClass, and can check their local variables to identify the classes in the dependency chain and the class loaders involved.

(note that conditional breakpoints in library code can be a bit brittle. For instance, when I just tested this in a toy example, I had to briefly halt execution with a normal breakpoint, to give the conditional breakpoint enough time to hook into the ClassLoader code)

Alternatively, you could write your own class for libraryLoader, which throws, logs, or halt in a breakpoint whenever it is asked for classes that should come from appLoader.

Answer 2

So I ended up figuring it out by meritons answer. It wasn't a problem with the hyarchy, rather it was the fact that the URLClassLoader for some reason prefers ancestors for findClass. I ended up on this question which had an answer I ended up copying (the ParentLastURLClassLoader) and it worked.

Essentially the problem was that, since the URLClassLoader for appLoader delegated to the root class loader first, the Bootstrap class was loaded by the root loader. This meant it was not under the library class loader in the class loading chain, not giving it access to those classes.