Clone of AWS ElasticBeanstalk giving tomcat error on startup The AJP Connector is configured with secretRequired="true"

Question

Using AWS Elastic Beanstalk environment with Tomcat 7 with Java 7 running on 64bit Amazon Linux/3.4.8, a retired platform but one i have used for a number of years without problem.

Cloned the environment, didn't make any changes to the environment or configuration,and now no longer working and catalina.out log file shows

Nov 28, 2023 11:01:36 AM org.apache.catalina.core.StandardService startInternal
SEVERE: Failed to start connector [Connector[AJP/1.3-8009]]
org.apache.catalina.LifecycleException: service.getName(): "Catalina";  Protocol handler start failed
    at org.apache.catalina.connector.Connector.startInternal(Connector.java:1065)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.StandardService.startInternal(StandardService.java:460)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:744)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:712)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:607)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:284)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:415)
Caused by: java.lang.IllegalArgumentException: The AJP Connector is configured with secretRequired="true" but the secret attribute is either null or "". This combination is not valid.
    at org.apache.coyote.ajp.AbstractAjpProtocol.start(AbstractAjpProtocol.java:211)
    at org.apache.catalina.connector.Connector.startInternal(Connector.java:1058)
    ... 12 more

Nov 28, 2023 11:01:36 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 2437 ms

I'm guessing this is some sort of security fix:

• but why am I getting now when using same configuration?
• whats the easiest way to resolve?

Update

So have a modified server.xml in my application war in my ebtenssions subfolder) and edited that rebuilt the war and uploaded it as a new application version.

My problem is I use the blue/green method. I have a working ElasticBeanstalk server, but Im trying to deploy a new server using Clone I hit the original problem so I then deploy the new application version but eventually after 35 minutes it complains that it failed to deploy new application version

December 1, 2023 13:37:07 (UTC) 
WARN    Environment health has transitioned from Info to Degraded. Incorrect application version found on all instances. Expected version "October2021" (deployment 1). Application update failed 1 second ago and took 35 minutes.
December 1, 2023 13:36:00 (UTC) 
ERROR   During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
December 1, 2023 13:36:00 (UTC) 
ERROR   Failed to deploy application.
December 1, 2023 13:35:59 (UTC) 
ERROR   Unsuccessful command execution on instance id(s) 'i-052f596f7f42a2f99'. Aborting the operation.
December 1, 2023 13:35:59 (UTC) 
INFO    Command execution completed on all instances. Summary: [Successful: 0, TimedOut: 1].
December 1, 2023 13:35:59 (UTC) 
WARN    The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-052f596f7f42a2f99].
December 1, 2023 13:02:10 (UTC) 
INFO    Environment health has transitioned from Ok to Info. Application update in progress (running for 4 seconds).
December 1, 2023 13:01:54 (UTC) 
INFO    Deploying new version to instance(s).

Now, if deploying app version is similar to original deployment it does take a while to deploy server because as part of the deployment it does have to copy over some large index files and unzip them

commands:
   01_install_cli:
    command: wget https://s3.amazonaws.com/aws-cli/awscli-bundle-1.18.223.zip; unzip awscli-bundle-1.18.223.zip;  ./awscli-bundle/install -b ~/bin/aws

   02_get_index:
     command:
       aws s3 cp --region eu-west-1 s3://jthink/release_index_21.tar.gz /home/ec2-user/release_index.tar.gz;
       aws s3 cp --region eu-west-1 s3://jthink/artist_index_21.tar.gz /home/ec2-user/artist_index.tar.gz;
       cd /usr/share/tomcat7;
       tar -xvf /home/ec2-user/release_index.tar.gz;
       rm /home/ec2-user/release_index.tar.gz;
       tar -xvf /home/ec2-user/artist_index.tar.gz;
       rm /home/ec2-user/artist_index.tar.gz;

but this works okay when clone, so I dont see why it would cause a problem when updating application version

I also have an increase timeout script for this

option_settings:
    - namespace: aws:elasticbeanstalk:command
      option_name: Timeout
      value: 1800

So my problem is for some reason trying to update application version isn't working.

I looked to see if you could clone an existing environment but change the application version but this doesn't seem to be possible.

So then I look at creating a new eb instance without cloning, then i could specify the new application version at start and that should hopefully work. However the platform version I use is no longer available and I don't think my code works with the newer platform.

Any ideas ?

Answer 1

The error message indicates that secretRequired is set to true, but no secret is provided. That is a security measure to prevent unauthorized access to the AJP Connector.

Check if this is similar to "The AJP Connector is configured with secretRequired="true" but the secret attribute is either null or "" after upgrade to 2.2.5".

The same configuration with an upgraded Tomcat could trigger that error.

In which case, adding secretRequired="false" to the Tomcat conf/server.xml should help.
The server.xml file might typically be located at /etc/tomcat7/server.xml from the Elastic Beanstalk environment configuration.

---

If you are not using the AJP Connector, you can disable it by commenting it out in the server.xml.

<!-- <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

---

Why is it happening now, when I have cloned an existing server and haven't changed anything? I don't knowingly use AJP it was just configured in the default server.xml, what would I use it for?

Why now

AWS Elastic Beanstalk environments might have automatically applied minor version updates or patches during cloning or redeployment processes. While your main configuration (Tomcat 7 on Amazon Linux 3.4.8) remains the same, the underlying components, like Tomcat, might have been updated to include recent security patches.
Recent Tomcat versions introduced stricter security measures for the AJP Connector due to known vulnerabilities. Even if the major version remains the same (Tomcat 7), minor version updates or patches might introduce these security enhancements.

The default server.xml in Tomcat includes an AJP Connector configuration. In earlier versions or setups, secretRequired might not have been set to true by default, but newer patches could have changed this, leading to the error you are encountering.

Why AJP? What for?

AJP (Apache JServ Protocol) is a protocol used to communicate between an Apache HTTP server and a Tomcat server. It is typically used in a setup where you have both an Apache server and a Tomcat server, and you want Apache to handle static content and pass dynamic content requests (like Servlets and JSPs) to Tomcat.

If you are not using an Apache server in front of Tomcat, or if you are not specifically relying on the AJP protocol for communication between Apache and Tomcat, it is likely you are not using AJP.

Given that you are not knowingly using AJP, the simplest solution would be to disable the AJP Connector in your server.xml file, as mentioned above.

---

So my problem is for some reason trying to update application version isn't working.

Try and address one issue at a time, starting with the most straightforward (like increasing timeout) and gradually moving to more complex solutions (like platform update or custom AMI).

First, the deployment process is timing out: that might be due to the large index files you are transferring and unzipping as part of the deployment. For testing, try increasing the timeout value even more, if the current 1800 seconds (30 minutes) is not enough. If that does not help, consider optimizing the file transfer and unzipping process. For example, check if the files can be compressed more efficiently to reduce transfer time. Also, explore parallelizing the unzipping process, if feasible.

The warning about an "Incorrect application version found on all instances" suggests a discrepancy between the expected version and what is actually deployed on the instances.
Make sure the version label you are deploying is consistent across all configurations and scripts.

For the platform limitation, if possible, consider updating your application to be compatible with a newer, supported platform version. That would allow you to create new instances without relying on cloning.

Create a custom Amazon Machine Image (AMI) from your current working environment and use this AMI for new instances. That method retains your exact configuration, including the platform version.

As an alternative, instead of cloning, try performing an in-place upgrade of your existing environment with the new application version. That has risks, but it might bypass the issues encountered with cloning.

Also, consider deploying the new application version to a subset of instances first, rather than all at once. That can help isolate issues and reduce the impact of a failed deployment.

Use AWS CloudWatch logs and Elastic Beanstalk monitoring tools to gain insights into where the deployment process is failing.