How to prevent Apache Tomcat JNDI Realm configured for AD authentication redirecting to cloud-based controllers

Question

I am using a vendor-provided configuration of Apache Tomcat that utilizes JNDI Realms to connect to Active Directory to perform user authentication. Part of the configuration includes specifying the connectionUrl and an optional alternateUrl for targets to use for user authentication.

With the latest iteration of the software bundled using Apache Tomcat v9.0.68.0 running with Eclipse Adoptium 11.0.17+8, the connection attempts are being directed to our cloud-based AD domain controllers to service the authentication request. There are no firewall rules to allow this traffic in or out of the corporate network. When we look at packet captures using WireShark we can see traffic outbound from the application server with a destination port of 636 (LDAPS) and a destination server of either of the local domain controllers defined in the connectionUrl and/or alternateUrl realm options.

When we look at the firewall logs using Panorama, we see a source of the application server and a destination server of either our AWS domain controllers or Azure domain controllers on port 636. These correspond to the logs in the application where we see these:

Aug 03, 2023 2:55:27 PM org.apache.catalina.realm.JNDIRealm authenticate
SEVERE: Exception performing authentication
javax.naming.PartialResultException [Root exception is javax.naming.CommunicationException: cpr.ca:636 [Root exception is java.net.SocketTimeoutException: connect timed out]]

I have worked with the vendor's R&D team as well as our Domain Admins and network/firewall team and no one has been able to come up with a reason or a resolution to prevent this from happening. If nothing else, I'm looking to try to get more information from the process, whether it is through debug logging, other options for the connector, engine, realm, etc. or black magic!

This is an excerpt of the server.xml file's JNDI Realm definition:

      <Realm className="com.bmc.bcan.catalina.realm.BNALockOutRealm"
             failureCount="5"
             lockOutTime="86400"
             cacheSize="1000"
             cacheRemovalWarningTime="3600">
        <Realm className="com.bmc.bcan.catalina.realm.BNAJNDIRealm"
               connectionURL="ldaps://<FQDN1>:636"
               alternateURL="ldaps://<FQDN2>:636"
               connectionName="CN=*****,OU=*****,OU=*****,OU=*****,OU=*****,OU=*****,DC=**,DC=**"
               connectionPassword="<password>"
               userBase="DC=**,DC=**"
               userSearch="(sAMAccountName={0})"
               userSubtree="true"
               referrals="follow"
         />
      </Realm>
      <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="false">
        <Valve className="org.apache.catalina.valves.ErrorReportValve" showReport="false" showServerInfo="false" />
      </Host>
</Engine>```

Answer 1

Active Directory is often configured with a round-robin DNS. You can verify by doing nslookup a few times in a row. You'll see that the order of the IP address returned to you change every time.

This is usually not a problem for software using Microsoft's API, as they will automatically pick the next one the list. Java does not work that way, and their reasoning is valid. Think about it:

1. A Java application does a DNS lookup.
2. It succeeds, but the IP address returned is not reachable. We don't know that yet
3. The application sends a TCP SYN packet on port 636 at that IP address
4. The connection times out

The problem here is that there is no way for the application to know that querying the DNS again will return a different address. As far as it is concerned, the lookup step was successful. The connection step failed.

This problem also manifests itself when the DC is patched. The DNS will return an IP that will be valid in a few minutes, when the server is done rebooting. I'm sure you've heard of impossible to reproduce connection problems? Look no further.

A quick fix is to add an entry that points to a DC that you can reach in /etc/hosts of the "computer" where your application is running. But it does not scale and it does not fix the patch-then-reboot issue.

The solution is to replace this poor man's load balancing solution (round robin DNS) with a proper load-balancer. But you (and your AD admin) won't want to mess with such a sensitive part of the infrastructure.

This worked for me in the past:

1. Get a new hostname, like load-balanced-dc.cpr.ca
2. Configure your load-balancer with that address
   • Put the domain controllers you can reach "behind" that load-balancer
   • Do not intercept TLS traffic, but stick to the selected server.
3. Issue a new certificate with the name load-balanced-dc.cpr.ca in the subject alt names, along with cpr.ca (or whatever hostname your DC has)
4. Replace your domain controller's certificate with the one you just issued.
5. Configure your application to use load-balanced-dc.cpr.ca

Active Directory is a number of things in the Microsoft environment, but it is also a plain old (but very good) LDAP server. That's all Tomcat needs. Depending on your load-balancer configuration and ability to load-balance itself, you might not need two entries in your application's configuration file.

+As a bonus, you'll get kudos from every owner of an application that is not Microsoft native for setting that up!