Java 10 Spring Boot Infinispan org.jgroups.logging.Slf4jLogImpl not found

Question

I have a Spring Boot application which I'm building and running with Java 10. If I run the app using

java -jar

Everything works fine. The app starts just OK.

But if I put my app inside a Docker container with the exactly same Java version, my app throws this exception:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.jgroups.logging.Slf4jLogImpl
    at org.jgroups.logging.LogFactory.getLog(LogFactory.java:101)
    at org.jgroups.conf.XmlConfigurator.<clinit>(XmlConfigurator.java:33)
    at org.jgroups.conf.ConfiguratorFactory.getStackConfigurator(ConfiguratorFactory.java:62)
    at org.jgroups.JChannel.<init>(JChannel.java:122)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.buildChannel(JGroupsTransport.java:591)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.initChannel(JGroupsTransport.java:405)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:389)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
    ... 104 common frames omitted

I'm using this version of Java:

java version "10.0.2" 2018-07-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.2+13)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.2+13, mixed mode)

Docker version is:

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     true

My Docker is using an Alpine base image alpine:latest. I'm installing java in my container from this link:

curl -jksSLH "Cookie: oraclelicense=accept-securebackup-cookie" -o /tmp/java.tar.gz \
      http://download.oracle.com/otn-pub/java/jdk/10.0.2+13/19aef61b38124481863b1413dce1855f/jdk-10.0.2_linux-x64_bin.tar.gz

I'm really confused because from outside a docker container my app works fine, but inside a docker container it doesn't. Either case I'm using the same Java version.

UPDATE

We tried Oracle JDK and OpenJDK, same behavior

UPDATE 2

We even tried java -jar from inside the container, no luck

Answer 1

TL;DR

There are three options to resolve.

1. Upgrade to version 4.0.16 of JGroups, currently in SNAPSHOT. Edit: Now released here.
2. Make sure java properties for "user.language" and "user.country" are set.
3. Force JDKLogImpl with -Djgroups.use.jdk_logger=true. (mentioned by Perimosh)

Explanation

Ran into this issue in the following scenario.

Apache Camel + JGroups worked fine in a local environment. Deployed it elsewhere in a Docker instance, where we got the following stacktrace:

2018-11-19 13:38:03.063  INFO 582 --- [           main] o.a.camel.spring.boot.RoutesCollector    : Loading additional Camel XML routes from: classpath:camel/*.xml
2018-11-19 13:38:03.064  INFO 582 --- [           main] o.a.camel.spring.boot.RoutesCollector    : Loading additional Camel XML rests from: classpath:camel-rest/*.xml
2018-11-19 13:38:03.107  INFO 582 --- [           main] o.a.camel.spring.SpringCamelContext      : Apache Camel 2.22.2 (CamelContext: camel-1) is starting
2018-11-19 13:38:03.111  INFO 582 --- [           main] o.a.c.m.ManagedManagementStrategy        : JMX is enabled
2018-11-19 13:38:03.480  INFO 582 --- [           main] o.a.camel.spring.SpringCamelContext      : StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at     http://camel.apache.org/stream-caching.html
2018-11-19 13:38:03.597  INFO 582 --- [           main] o.a.camel.spring.SpringCamelContext      : Apache Camel 2.22.2 (CamelContext: camel-1) is shutting down
2018-11-19 13:38:03.616  WARN 582 --- [           main] o.a.camel.spring.SpringCamelContext      : Error occurred while shutting down service: org.apache.camel.component.jgroups.cluster.    JGroupsLockClusterService@10fa5af5. This exception will be ignored.
java.lang.NullPointerException: null
    at org.apache.camel.component.jgroups.cluster.JGroupsLockClusterView.doStop(JGroupsLockClusterView.java:109)
    at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
    at org.apache.camel.impl.cluster.AbstractCamelClusterService.lambda$doStop$2(AbstractCamelClusterService.java:134)
    at org.apache.camel.util.concurrent.LockHelper.doWithReadLockT(LockHelper.java:54)
    at org.apache.camel.impl.cluster.AbstractCamelClusterService.doStop(AbstractCamelClusterService.java:130)
    at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
    at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:142)
    at org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:205)
    at org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:3663)
    at org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:3688)
    at org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:3676)
    at org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:3567)
    at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
    at org.apache.camel.impl.DefaultCamelContext.stop(DefaultCamelContext.java:3220)
    at org.apache.camel.spring.SpringCamelContext.stop(SpringCamelContext.java:148)
    ...

2018-11-19 13:38:03.679  INFO 582 --- [           main] o.a.camel.spring.SpringCamelContext      : Apache Camel 2.22.2 (CamelContext: camel-1) uptime 0.570 seconds
2018-11-19 13:38:03.680  INFO 582 --- [           main] o.a.camel.spring.SpringCamelContext      : Apache Camel 2.22.2 (CamelContext: camel-1) is shutdown in 0.082 seconds
2018-11-19 13:38:03.716  INFO 582 --- [           main] org.mongodb.driver.connection            : Closed connection [connectionId{localValue:2, serverValue:2}] to localhost:43115 because the pool has been closed.

As you can see, Apache Camel attempts to start, but never does and ends up shutting down. Thus, JGroups gets a NPE because it expects Camel to be up. After debugging the code, it appeared that there was an exception being thrown during the Camel start up process which was getting eaten. From there, discovered that the creation of an instance of Slf4jLogImpl in org.jgroups.logging.LogFactory#getLog(java.lang.Class<?>) (new Slf4jLogImpl(clazz)) was a problem Method threw 'java.lang.ExceptionInInitializerError' exception.:

java.lang.NullPointerException: null
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.jgroups.logging.LogFactory.getLog(LogFactory.java:101)
at org.jgroups.conf.XmlConfigurator.<clinit>(XmlConfigurator.java:33)
at org.jgroups.conf.ConfiguratorFactory.getXmlConfigurator(ConfiguratorFactory.java:210)
at org.jgroups.conf.ConfiguratorFactory.getStackConfigurator(ConfiguratorFactory.java:91)
at org.jgroups.JChannel.<init>(JChannel.java:130)
...

Running (new Slf4jLogImpl(clazz)) the second time onward in the debugger results in the following stacktrace that mirrors the original posted issue:

java.lang.NoClassDefFoundError: Could not initialize class org.jgroups.logging.Slf4jLogImpl
at org.jgroups.logging.LogFactory.getLog(LogFactory.java:101)
at org.jgroups.conf.XmlConfigurator.<clinit>(XmlConfigurator.java:33)
at org.jgroups.conf.ConfiguratorFactory.getXmlConfigurator(ConfiguratorFactory.java:210)
at org.jgroups.conf.ConfiguratorFactory.getStackConfigurator(ConfiguratorFactory.java:91)
at org.jgroups.JChannel.<init>(JChannel.java:130)

This difference in results is due to the class loader caching the result of the Class.forName() call previously to determine that the class definition was not found.

Finally, we tracked the previous NPE to be thrown from java.util.Locale#Locale(java.lang.String, java.lang.String, java.lang.String), since country was null. This is because JGroup's org.jgroups.logging.Slf4jLogImpl is definine a LOCALE field using java properties for "user.language" and "user.country". The former was not set in our docker instance, so Locale.java threw the NPE. Adding both of these java properties should fix this issue. Alternatively you can force using the JDKLogImpl so that the Slf4jLogImpl is never attempted to be instantiated. This was mentioned in the previous answer, by passing in -Djgroups.use.jdk_logger=true.

Edit: Fixed in the latest version released here.

Now, it looks like this will be fixed in the upcoming JGroup release of 4.0.16.Final (https://github.com/belaban/JGroups/commit/61578c657138f02178c32a564ac9eae7c3976093#diff-93eb0f6a8a4953312098be459bd7ce76). Until then, you can get the snapshot version with the fix at https://repository.jboss.org/nexus/content/repositories/snapshots/org/jgroups/jgroups/4.0.16-SNAPSHOT/.

Answer 2

This is not a real solution, but since it unblocked us, I will share it. Also, maybe somebody can think about the real problem by looking at this workaround. We added this JVM argument to by pass SLF4J for jgroups and use JDKLogImpl

-Djgroups.use.jdk_logger=true