agents can't connect to cluster after enabling encryption

Question

I have Datastax enterprise 4.5.3 and Opscenter 5.0.1 installed in AWS in a multi-region config. I'm trying to make communication as secure as possible by enabling node-to-node encryption, client-to-node encryption and ssl between opscenter and agents.

I'm having a problem with agents after enabling client-to-node encryption. After enabling encryption in casssandra.yaml, the datastax agent throws an error saying it can't connect to the cluster.

Part of cassandra.yaml:

# enable or disable client/server encryption.
client_encryption_options:
    enabled: true
    keystore: /usr/share/dse/resources/dse/conf/.keystore
    keystore_password: supersecret_password
    # require_client_auth: false
    # Set trustore and truststore_password if require_client_auth is true
    truststore: /usr/share/dse/resources/dse/conf/.truststore
    truststore_password: supersecret_password
    # More advanced defaults below:
    protocol: SSL
    # algorithm: SunX509
    # store_type: JKS
    # cipher_suites [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

The agents can't connect to the cluster after enabling client-to-node encryption in cassandra.yaml. In /var/log/datastax-agent/agent.log, I see this error repeatedly:

ERROR [pdp-loader] 2014-11-10 20:06:18,902 Unable to connect to Cassandra:
me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.
    at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:390)
    at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:244)
    at me.prettyprint.cassandra.service.AbstractCluster.describePartitioner(AbstractCluster.java:255)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
    at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
    at clj_hector.core$partitioner.invoke(core.clj:42)
    at opsagent.cassandra$load_pdps_with_retry$fn__1066.invoke(cassandra.clj:211)
    at opsagent.cassandra$load_pdps_with_retry.invoke(cassandra.clj:210)
    at opsagent.cassandra$setup_cassandra$f__388__auto____1094$fn__1095$f__388__auto____1102.invoke(cassandra.clj:357)
    at clojure.lang.AFn.run(AFn.java:24)
    at java.lang.Thread.run(Thread.java:745)
INFO [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2014-11-10 20:06:28,700 Not checking that 54.86.178.77(54.86.178.77):9160 is a member of the ring since there are no live hosts
WARN [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2014-11-10 20:06:28,701 Downed 54.86.178.77(54.86.178.77):9160 host still appears to be down: Unable to open transport to 54.86.178.77(54.86.178.77):9160 , java.net.ConnectException: Connection refused

Answer 1

Please make sure you have completed your Agent configuration in address.yaml. It should be set up as follows:

stomp_interface: <stomp_interface_ip of opscenterd box>
local_interface: <broadcast or listen_address_ip in cassandra.yaml>
cassandra_conf: "<path>/dse-4.5.1/resources/cassandra/conf/cassandra.yaml"
use_ssl: 1
ssl_keystore: <path to keystore>/.keystore
ssl_keystore_password: <ks password>
thrift_ssl_truststore: <path to truststore>/.truststore
thrift_ssl_truststore_password: <ts password>
hosts: ["<ip address>"]

See other detailed config steps for SSL and Opscenter here for tarball installs and here for package installs