Spring Boot app can't connect to Cassandra cluster, driver returning "AllNodesFailedException: Could not reach any contact point"

Question

i've updated my spring-boot to v3.0.0 and spring-data-cassandra to v4.0.0 which resulted in unable to connect to cassandra cluster which is deployed in stg env and runs on IPv6 address having different datacenter rather DC1

i've added a config file which accepts localDC programatically

   `@Bean(destroyMethod = "close")
    public CqlSession session() {
        CqlSession session = CqlSession.builder() 
            .addContactPoint(InetSocketAddress.createUnresolved("[240b:c0e0:1xx:xxx8:xxxx:x:x:x]", port))
                .withConfigLoader(
                        DriverConfigLoader.programmaticBuilder()
                                .withString(DefaultDriverOption.LOAD_BALANCING_LOCAL_DATACENTER, localDatacenter)                              
                                .withString(DefaultDriverOption.AUTH_PROVIDER_PASSWORD,password)
                                .withString(DefaultDriverOption.CONNECTION_INIT_QUERY_TIMEOUT,"10s")
                                .withString(DefaultDriverOption.CONNECTION_CONNECT_TIMEOUT, "20s")
                                .withString(DefaultDriverOption.REQUEST_TIMEOUT, "20s")
                                .withString(DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT, "20s")
                                .withString(DefaultDriverOption.SESSION_KEYSPACE,keyspace)
                                .build())
                //.addContactPoint(InetSocketAddress.createUnresolved(InetAddress.getByName(contactPoints).getHostName(), port))
                .build();
        }
        return session;`

and this is my application.yml file

spring:
  data:
    cassandra:
      keyspace-name: xxx
      contact-points: [xxxx:xxxx:xxxx:xxx:xxx:xxx]
      port: xxx
      local-datacenter: xxxx
      use-dc-aware: true
      username: xxxxx
      password: xxxxx
      ssl: true
      SchemaAction: CREATE_IF_NOT_EXISTS

So locally I was able to connect to cassandra (by default it is pointing to localhost) , but in stg env my appplication is not able to connect to that cluster

logs in my stg env

caused by: com.datastax.oss.driver.api.core.AllNodesFailedException: Could not reach any contact point, make sure you've provided valid addresses (showing first 1 nodes, use getAllErrors() for more): Node (endPoint=/[240b:cOe0:102:xxxx:xxxx:x:x:x]:3xxx,hostId-null,hashCode=4e9ba6a8):[com.datastax.oss.driver.api.core.connection.ConnectionInitException:[s0|controllid:0x984419ed,L:/[240b:cOe0:102:5dd7: xxxx:x:x:xxx]:4xxx - R:/[240b:c0e0:102:xxxx:xxxx:x:x:x]:3xxx] Protocol initialization request, step 1 (OPTIONS: unexpected tarlure com.datastax.oss.driver.apt.core.connection.closedconnectiontxception: Lost connection to remote peer)]

Answer 1

added the certificate in my spring application

    public CqlSession session() throws IOException, CertificateException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
         Resource resource = new ClassPathResource("root.crt");
                InputStream inputStream = resource.getInputStream();
                CertificateFactory cf = CertificateFactory.getInstance("X.509");
                Certificate cert = cf.generateCertificate(inputStream);
                TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
                KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
                keyStore.load(null);
                keyStore.setCertificateEntry("ca", cert);
                trustManagerFactory.init(keyStore);
                SSLContext sslContext = SSLContext.getInstance("TLSv1.3");
                sslContext.init(null, trustManagerFactory.getTrustManagers(), null);
                return CqlSession.builder()
                        .withSslContext(sslContext)
                        .addContactPoint(new InetSocketAddress(contactPoints,port))
                        .withAuthCredentials(username, password)
                        .withLocalDatacenter(localDatacenter)
                        .withKeyspace(keyspace)
                        .build();
}

so added the cert file in the configuration file of the cqlsession builder and this helped me in connecting to the remote cassandra cluster

Answer 2

A hearty +1 to everything @erick-ramirez mentioned above. I would also expand on his answers with an observation or two.

Normally spring-data-cassandra is used to automatically configure a CqlSession and make it available for injection (or for use in CqlTemplate etc.). That's what you'd normally be configuring with your application.yml file. But you're apparently creating the CqlSession directly in code, which means that spring-data-cassandra isn't involved... and therefore what's in your application.yml likely isn't being used.

This analysis strongly suggests that your CqlSession is not being configured to use SSL. My understanding is that your testing sequence went as follows:

• Tested app locally on a local server, everything worked
• Tested app against test environment, observed the errors above

If this sequence is correct and you have SSL enabled in you test environment but not on your local Cassandra instance that could very easily explain the behaviour you're describing.

This explanation could also explain the specific error you cite in the error message. "Lost connection to remote peer" indicates that something is unexpectedly killing your socket connection before any protocol messages are explained... and an SSL issue would cause almost exactly that behaviour.

I would recommend checking the SSL configuration for both servers involved in your testing. I would also suggest consulting the SSL-related documentation referenced by Erick above and confirm that you have all the relevant materials when building your CqlSession.

Answer 3

Network

You appear to have a networking issue. The driver can't connect to any of the nodes because they are unreachable from a network perspective as it states in the error message:

... AllNodesFailedException: Could not reach any contact point ...

You need to check that:

• you have configured the correct IP addresses,
• you have configured the correct CQL port, and
• there is network connectivity between your app and the cluster.

Security

I also noted that you configured the driver to use SSL:

      ssl: true

but I don't see anywhere where you've configured the certificate credentials and this could explain why the driver can't initiate connections.

Check that the cluster has client-to-node encryption enabled. If it does then you need to prepare the client certificates and configure SSL on the driver.

Driver build

This post appears to be a duplicate of another question you posted but is now closed due to lack of clarity and details.

In that question it appears you are running a version of the Java driver not produced by DataStax as pointed out by @absurdface:

Specifically I note that java-driver-core-4.11.4-yb-1-RC1.jar isn't a Java driver artifact released by DataStax (there isn't even a 4.11.4 Java driver release). This could be relevant for reasons we'll get into ...

We are not aware of where this build came from and without knowing much about it, it could be the reason you are not able to connect to the cluster.

We recommend that you switch to one of the supported builds of the Java driver. Cheers!