Java on AWS/EKS fails with resolving correct version of org.osgi.framework.launch.FrameworkFactory

Question

We're lift/shift a K8 installation into AWS/EKS and in one cluster works, other cluster it fails to resolve the correct Loaded FrameworkFactory. We have a third party .JAR and Eclipse config so all we do is build a maven/docker build with an image for K8/EKS.
I need some ideas in terms of possible reasons why the startup fails to launch the correct Eclipse OSGI vs Core. OSGI - the working example correctly unbundles and creates the lib/configuration files and workspace folders. The bad example fails to do that and main /libs are okay but the configuration and workspaces cause the main thread to throw right at startup.

    Good - [Loaded org.osgi.framework.launch.FrameworkFactory from file:/usr/share/pm.health.rating.engine/lib/org.eclipse.osgi-3.9.1.jar]
vs
Bad - [Loaded org.osgi.framework.launch.FrameworkFactory from file:/usr/share/pm.health.rating.engine/lib/org.osgi.core-4.3.1.jar]

And

POM.XML

<dependencies>
    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>${avro.version}</version>
    </dependency>
    <dependency>
        <groupId>net.logstash.log4j</groupId>
        <artifactId>jsonevent-layout</artifactId>
        <version>1.7</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-s3</artifactId>
        <version>${aws.version}</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-sts</artifactId>
        <version>${aws.version}</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>${slf4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>${slf4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-server</artifactId>
        <version>${jetty.version}</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-servlet</artifactId>
        <version>${jetty.version}</version>
    </dependency>
    <dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <version>${protobuf.version}</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.eclipse.platform/org.eclipse.osgi -->
    <dependency>
        <groupId>org.eclipse.osgi</groupId>
        <artifactId>org.eclipse.osgi</artifactId>
        <version>3.9.1</version>
    </dependency>
    <dependency>
        <groupId>com.vendor.pm</groupId>
        <artifactId>pmrs</artifactId>
        <version>${pmrs.version}</version>
    </dependency>
    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>19.0</version>
    </dependency>
    <!-- Eos Runtime Services dependencies -->
    <dependency>
        <groupId>com.vendor.pm</groupId>
        <artifactId>eos-rs</artifactId>
        <version>4.2.9.4</version>
        <exclusions>
            <exclusion>
                <groupId>com.vendor.pm</groupId>
                <artifactId>pmrs</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>net.sourceforge.jexcelapi</groupId>
        <artifactId>jxl</artifactId>
        <version>2.6.12</version>
    </dependency>
    <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-core-asl</artifactId>
        <version>1.9.0</version>
    </dependency>
    <dependency>
        <groupId>com.rackspace.apache</groupId>
        <artifactId>xerces2-xsd11</artifactId>
        <version>2.11.2</version>
    </dependency>
    <!-- ADDED TO AVOID THE SAX PARSER WARNINGS -->
    <dependency>
        <groupId>xalan</groupId>
        <artifactId>xalan</artifactId>
        <version>2.7.1</version>
        <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.poi/poi -->
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi</artifactId>
        <version>3.15</version>
        <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml -->
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi-ooxml</artifactId>
        <version>3.15</version>
        <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/javax.xml.parsers/jaxp-api -->
    <dependency>
        <groupId>javax.xml.parsers</groupId>
        <artifactId>jaxp-api</artifactId>
        <version>1.4.5</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.vendor.eos.groupref</groupId>
        <artifactId>com.fja.eos.groupref.formula.lib</artifactId>
        <version>1.0.0.202005041335</version>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>2.7.0</version>
    </dependency>
    <dependency>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-avro-serializer</artifactId>
        <version>5.5.1</version>
    </dependency>

openjdk version "1.8.0_402" OpenJDK Runtime Environment (build 1.8.0_402-b06) OpenJDK 64-Bit Server VM (build 25.402-b06, mixed mode)

NAME="Red Hat Enterprise Linux" VERSION="9.3 (Plow)" ID="rhel" ID_LIKE="fedora" VERSION_ID="9.3" PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.3 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.3" 5.15.145 x86_64 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes:
48 bits physical, 48 bits virtual Byte Order:
Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Vendor ID:
AuthenticAMD Model name: AMD EPYC 7R13 Processor CPU family: 25 Model:
1 Thread(s) per core: 2 Core(s) per socket:
4 Socket(s): 1 Stepping:
1

Answer 1

You should not be using osgi.core jar at runtime. It should not be on your classpath. org.eclipse.osgi will contain the necessary OSGi Core packages for use at runtime.

You may need to determine which dependency is pulling osgi.core in and exclude osgi.core from that dependency. It is possible (and likely) a dependency mistakenly used scope=compile instead of scope=provided for osgi.core.

Answer 2

Hi thank you for your response. I was able to run verbose: class and module to see which module was pulling in the org.osgi and add an exclusion as you mentioned to exclude osgi.core from those libraries and it worked! While not a java person I'm reading that the exclude will override any dependency issues that might be implied at runtime with maven builds? Maybe the code author with the IDE and tools could have verified that dependency. To me this should be more deterministic that it did not have a problem with certain CPU architectures on AWS/EKS but was exacerbated by AWS Bottlerocket CPU architecture.

Here's the exclusion:

<exclusion>
    <groupId>org.osgi</groupId>
    <artifactId>org.osgi.core</artifactId>
</exclusion>