Tomcat Service not starting with elastic APM java agent

Question

I am trying to start tomcat service in linux with elastic APM agent (java). But every time I start the service, getting these errors

SEVERE: ContainerBase.addChild: start:
    org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mis-all-1.0.0]]
    at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1007)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:983)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:639)
    at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1296)
    at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2037)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.VerifyError: (class: javax/servlet/GenericServlet, method: init signature: (Ljavax/servlet/ServletConfig;)V) invokedynamic bytecode is not supported in this class file version
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2509)
    at java.lang.Class.getDeclaredFields(Class.java:1819)
    at org.apache.catalina.util.Introspection.getDeclaredFields(Introspection.java:106)
    at org.apache.catalina.startup.WebAnnotationSet.loadFieldsAnnotation(WebAnnotationSet.java:270)
    at org.apache.catalina.startup.WebAnnotationSet.loadApplicationServletAnnotations(WebAnnotationSet.java:138)
    at org.apache.catalina.startup.WebAnnotationSet.loadApplicationAnnotations(WebAnnotationSet.java:70)
    at org.apache.catalina.startup.ContextConfig.applicationAnnotationsConfig(ContextConfig.java:417)
    at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:891)
    at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:388)
    at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5536)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    ... 10 more
    
May 20, 2021 5:48:15 AM org.apache.catalina.startup.HostConfig deployDirectory
SEVERE: Error deploying web application directory []
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mis-all-1.0.0]]
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1011)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:983)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:639)
    at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1296)
    at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2037)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
    at java.lang.Thread.run(Thread.java:745)

I am using below commands to set Environmental variables for APM agent

export CATALINA_OPTS="$CATALINA_OPTS -javaagent:/app/elastic-apm-agent-1.23.0.jar"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.service_name=Stage-pricing"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.application_packages=org,com,com.qr.jadu"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.server_url=http://ip:8200"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.global_labels='env=Stage'"

and logback-classic version 1.0.6

Please suggest a solution or any steps I am missing, thanks in advance.

---

Update by kriegaex, copied from the OP's comment:

• Server version: Apache Tomcat/7.0.106 Server
  • built: Sep 16 2020 08:33:41 UTC
  • Server number: 7.0.106.0 OS
• OS: Linux OS Version: 4.14.35-2047.502.4.1.el7uek.x86_64
• Architecture: amd64
• JVM Version: 1.7.0_301-b09
• JVM Vendor: Oracle Corporation APM
• Server version: 7.12.1
• APM Agent language and version: java, 1.23. Also apm agent supports 7u60+, 8u40+, 9, 10, 11.

Answer 1

Caused by: java.lang.VerifyError: (
  class: javax/servlet/GenericServlet,
  method: init
  signature: (Ljavax/servlet/ServletConfig;)V)
invokedynamic bytecode is not supported in this class file version

Piotr is right. Your Elastic APM Agent - whatever that thing does - transforms the byte code of void GenericServlet.init(ServletConfig) in such a way as to utilise invokedynamic byte code instructions, not checking or expecting that the code it weaves into is from an ancient Java version. Invokedynamic was introduced in Java 7, therefore your JAR must contain older versions. Either exclude it from transformation, if the agent can be configured that way, or upgrade the JAR, either using a more recent version or recompiling the API sources to a more recent (Java 7+) byte code version. Most agents probably rather expect Java 8+, but I am not sure about yours.