AWS Lambda using incorrect classfiles from a Multi-Release JAR?

Question

I've had a lambda running for a few years under Java8, and I just updated it to Java 11. It immediately broke, giving me errors like:

Caused by: java.lang.ExceptionInInitializerError
    at com.mycompany.rest.providers.JsonProvider.writeTo(JsonProvider.java:80)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:242)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:227)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
    at org.glassfish.jersey.client.ClientRequest.doWriteEntity(ClientRequest.java:461)
    at org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:443)
    at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:367)
    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:265)
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297)
    ... 15 more
Caused by: java.lang.UnsupportedOperationException: No class provided, and an appropriate one cannot be found.
    at org.apache.logging.log4j.LogManager.callerClass(LogManager.java:571)
    at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:596)
    at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:583)
    at com.mycompany.rest.util.NonClosingOutputStream.<clinit>(NonClosingOutputStream.java:11)
    ... 25 more

The class in question isn't particularly exciting, and has a straightforward static initialization that is common in my classes:

public class NonClosingOutputStream extends ProxyOutputStream {
    private static final Logger log = LogManager.getLogger(); // Line 11

    public NonClosingOutputStream(final OutputStream proxy) {
        super(proxy);
    }

    ...

I've seen problems like this before, when I switched my (non-Lambda) java servers from 8 to 11; I needed to flag my jar's manifest as Multi-Release: true, because the ApacheLog4j artifact that I depend on provides alternate implementations for the org.apache.logging.log4j.util.StackLocator class in Java 8- and 9+. However, I kind of expect the JVM to just pick up the appropriate version of the class. Is there some configuration that I have to set somewhere? Is it possible that switching my Lambda from Java 8 -> Java 11 confused something, somewhere?

jar/META-INF/versions:

versions/
├── 11
│   └── org
│       └── glassfish
│           └── jersey
│               └── internal
│                   └── jsr166
│                       ├── JerseyFlowSubscriber$1.class
│                       ├── JerseyFlowSubscriber.class
│                       ├── SubmissionPublisher$1.class
│                       ├── SubmissionPublisher$2.class
│                       ├── SubmissionPublisher$3.class
│                       ├── SubmissionPublisher$4.class
│                       ├── SubmissionPublisher$5.class
│                       ├── SubmissionPublisher$6.class
│                       ├── SubmissionPublisher.class
│                       └── SubmissionPublisherFactory.class
└── 9
    ├── module-info.class
    └── org
        └── apache
            └── logging
                └── log4j
                    ├── core
                    │   └── util
                    │       └── SystemClock.class
                    └── util
                        ├── Base64Util.class
                        ├── ProcessIdUtil.class
                        ├── StackLocator.class
                        └── internal
                            └── DefaultObjectInputFilter.class

Edit: I am finding some references indicating that, when AWS Lambda extracts a JAR, they don't extract the META-INF directory, which contains the MANIFEST.MF file that tells the JVM that the JAR is a Muli-Release JAR. Do Lambdas support Multi-Release JARs at all?

Answer 1

As others have mentioned, AWS Lambda doesn't seem to support multi-release JARS. (See JEP 238.)

Log4J uses a org.apache.logging.log4j.util.Stackwalker to walk up the stack. The old version uses sun.reflect.Reflection.getCallerClass(int), but I infer that the new version that works with Java 9+ doesn't work with earlier versions, so they created a multi-release JAR. (See LOG4J2-2537: Log4j 2 Performance issue with Java 11 for more back-story.) Unfortunately even if you use <Multi-Release>true</Multi-Release> in your Maven Shade ManifestResourceTransformer configuration, AWS Lambda doesn't know how to deal with multi-release JARs.

Basically in the multi-release JAR there is a META-INF/versions/9/ directory, under which is kept the Java 9+ version of Stackwalker and related classes, which don't use sun.reflect.Reflection.getCallerClass. If you're targeting Java 9+ you can do what the JVM would do and use these classes rather than the ones not inside META-INF/. The Maven Shade Plugin doesn't seem to have a way to do this manually, but it appears you can brute-force the issue by (mis?)using the Maven Shade Plugin <relocations> facility to copy over the version under META-INF/versions/9/, overwriting the old ones that use sun.reflect.Reflection.getCallerClass. One other trick: specify a fuller path to the classes, so you won't copy over the metadata file under META-INF/versions/9/ itself.

Thus it appears (after limited testing) that adding the following to your Maven Shade Plugin <configuration> section will produce a JAR using the Java 9+ version of Stackwalker, which doesn't even look for sun.reflect.Reflection.getCallerClass, thus preventing the warning:

<relocations>
  <relocation>
    <pattern>META-INF/versions/9/org/apache/logging/log4j/</pattern>
    <shadedPattern>org/apache/logging/log4j/</shadedPattern>
  </relocation>
</relocations>

Using this technique, you can remove the <Multi-Release>true</Multi-Release> of your ManifestResourceTransformer, because there's no point in making a multi-release JAR anymore, as you're manually creating a single-release JAR with the Java 9+ content.

See aws-lambda-java-libs Issue #204: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance where I first reported this workaround.

Note that this answer is only a workaround if you insist in using the Maven Shade Plugin! A better approach would be to use the Maven Assembly Plugin as Maarten Brak suggested, or the the Spring Boot Maven Plugin after they address Issue #36101, so you wouldn't need a workaround such as this.

Answer 2

According to my AWS account rep, AWS Lambdas do not support Multi-Release JARs at this time (2021-06-14). I will need to reconfigure my pom to build multiple artifacts, instead.

Answer 3

Not exactly an answer to your question but I hope this may help.

Your analysis is correct - AWS lambda extracts the entire JAR file. Then the JVM running the lambda function doesn't recognize the code as a JAR file anymore and effectively the entire META-INF directory is ignored.

In my case, I was using the maven-shade-plugin to create an "uber"-jar containing all the dependencies of my lambda function. This approach is recommended in the official AWS documentation. Now - and this is important - the maven-shade-plugin extracts all jar file dependencies and repackages them into a single, flat jar file. If one of your dependencies is a multi-release jar (as is log4j2), then you can configure the maven-shade-plugin to reconstruct an appropriate META-INF directory and if you run the jar as a jar file then everything still works. But because AWS Lambda extracts the jar, the META-INF directory is not "seen" by the JVM anymore, and anything that was in META-INF/versions is ignored.

To resolve this, I switched to the maven-assembly-plugin. It allows creating a ZIP file with the code of your lambda, and add the dependencies as JAR files. Now when AWS Lambda extracts this ZIP file, the JAR remain intact and everything works fine.

To configure this, create a file assembly.xml like this:

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
    <id>zip</id>
    <!-- Make sure the ZIP contents are not nested in a subdirectory -->
    <includeBaseDirectory>false</includeBaseDirectory>

    <formats>
        <format>zip</format>
    </formats>
    <fileSets>
        <fileSet>
            <directory>${project.basedir}/conf</directory>
        </fileSet>
        <!-- Include the compiled classes as-is and put them in the root of the ZIP -->
        <fileSet>
            <directory>${project.build.directory}/classes</directory>
            <outputDirectory>/</outputDirectory>
        </fileSet>
    </fileSets>
    <dependencySets>
        <!-- Include all dependencies in the lib/ directory -->
        <dependencySet>
            <outputDirectory>lib</outputDirectory>
            <excludes>
                <exclude>${project.groupId}:${project.artifactId}:jar:*</exclude>
            </excludes>
        </dependencySet>
    </dependencySets>
</assembly>

Then you need to configure the maven-assembly-plugin in your pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-assembly-plugin</artifactId>
    <version>3.3.0</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
            <configuration>
                <appendAssemblyId>false</appendAssemblyId>
                <descriptors>
                    <descriptor>assembly.xml</descriptor>
                </descriptors>
                <finalName>${project.artifactId}</finalName>
            </configuration>
        </execution>
    </executions>
</plugin>

Now just deploy the resulting zip file to AWS Lambda as usual and voila!

As an aside - whereas the shaded JAR file contained thousands of individual .class files, the assembled ZIP file contains only a handful of JAR files. Even though the overall size (in bytes) is bigger, the number of files will be much smaller and thereby reducing your cold start times. I haven't tested this on the AWS Cloud, but on my LocalStack the cold start went down from about 1 minute to 6 seconds - definitely a great booster for development.