Is there any solution for "nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'"?

Question

In my cassandra cluster , all nodes are available.

But for one node, when I check the status it is showing like:

"nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'."

As I've seen some corrections, I tried restarting cassandra service after setting JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=127.0.0.1". But again it is giving the same status as:

"nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'."

Can someone suggest another solution?

Answer 1

Can you edit your question with the actual nodetool status command being run?

Also are you using the standard, default cassandra-env.sh file? Or has it been changed? Regarding local/remote JMX, there are logic checks like this:

if [ "x$LOCAL_JMX" = "x" ]; then
    LOCAL_JMX=yes
fi

if [ "$LOCAL_JMX" = "yes" ]; then
  JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
else
  JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.remote.port=$JMX_PORT"
  ...

First of all, LOCAL_JMX is not defined by default. So if you want to enable remote JMX, you'll need to adjust the code. Secondly, adjusting the rmi.server.hostname needs to be done in the correct section.

Enabling remote JMX also allows JMX to work from the local machine. But the idea is that the port being used (7199, by default) is bound to the same IP that the Cassandra node itself is.

Essentially, if you were to comment-out the local/remote if constructs, these settings should allow JMX (and nodetool) to function. Note: assume an external IP address of 10.2.3.5.

JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.remote.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.2.3.5"

Basically, the rmi.server.hostname should match the IP used to connect to the node.

If you continue having problems, start by checking your system.log. At startup, each node should have a line like this:

StartupChecks.java:176 - JMX is enabled to receive remote connections on port: 7199

or:

StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.