How to enable JMX in docker swarm for a specific container in swarm cluster?

Question

Problem Statement: I need to connect to a specific container in docker swarm services via JMX. Service is not exposed on any port, so I cannot access JMX by directly hitting the docker substrate on exposed port.

Also, in case service is exposed, then load balancing by swarm does not guarantee which container will I hit.

Answer 1

You can ssh to the worker node running your container and from there on you interact with the container just like you do on your local machine.

If you are using docker-machine to create the worker nodes, you can ssh using:

docker-machine ssh <node-name>

Answer 2

Able to solve the problem via the following approach:

1. Update service to take a inject a separate environment variable, say JMX_OPTS, while starting the application.


2. Update the docker service to add that environment variable

docker service update --env-add JMX_OPTS="-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.rmi.port=7001
-Dcom.sun.management.jmxremote.port=7001 
-Dcom.sun.management.jmxremote.authenticate=true 
-Dcom.sun.management.jmxremote.ssl=false 
-Djava.rmi.server.hostname=<hostname_IP> 
-Dcom.sun.management.jmxremote.password.file=/jmxremote.password" service123

You can choose any port number. There is NO need to update dockerfile to EXPOSE the port. 

Also, in -Djava.rmi.server.hostname, enter the swarm managers IP, and not the hostname.

3. Add another HA Proxy service to connect with the specific container in using swarm network.
docker service create --name proxy-docker-service123 --network swarm-net -p 7001:7001 -e "BACKEND_HOST=<CONTAINER_IP>" -e "BACKEND_PORT=7001" demandbase/docker-tcp-proxy

CONTAINER_IP can be found by inspecting the 
command: docker inspect d87c42441faf | grep IPv4


Make sure that for a docker-tcp-proxy all source, target and backend ports are same.




For JMX with SSL:

Update JMX_OPTS with following parameters
docker service update --env-add JMX_OPTS="-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.rmi.port=7001 
-Dcom.sun.management.jmxremote.port=7002 
-Dcom.sun.management.jmxremote.authenticate=true 
-Djava.rmi.server.hostname=<hostname_ip> 
-Dcom.sun.management.jmxremote.password.file=/jmxremote.password 
-Djavax.net.ssl.keyStore=<location_to_keystore>
-Djavax.net.ssl.keyStorePassword=<keyStore_Password> 
-Dcom.sun.management.jmxremote.ssl.need.client.auth=false
-Dcom.sun.management.jmxremote.registry.ssl=true 
-Dcom.sun.net.ssl.checkRevocation=false" 

Please note in this case, jmxremote and rmi are running on two separate ports. So, we need to deploy two HA proxy services. One for port 7001 and one for 7002.