Cannot ping containers in the same pod in Kubernetes(minikube)

Question

On my local I run a mysql container and then ping it from another container on the same network:

$ docker run -d tutum/mysql
$ docker run -it plumsempy/plum bash
    # ping MYSQL_CONTAINER_ID

      PING 67e35427d638 (198.105.244.24): 56 data bytes
      64 bytes from 198.105.244.24: icmp_seq=0 ttl=37 time=0.243 ms 
      ...

That is good. Then, using Kubernetes(minikube) locally, I deploy tutum/mysql using the following YAML:

...
- name: mysql
  image: tutum/mysql
...

There is nothing else for the mysql container. Then I deploy it, ssh into the minikube pod, spin up a random container and try pinging the mysql container inside the pod this time:

$ kubectl create -f k8s-deployment.yml
$ minikube ssh
    $ docker ps
    $ docker run -it plumsempy/plum bash
        # ping MYSQL_CONTAINER_ID_INSIDE_MINIKUBE

          PING mysql (198.105.244.24): 56 data bytes
          ^C--- mysql ping statistics ---
          10 packets transmitted, 0 packets received, 100% packet loss

        # traceroute MYSQL_CONTAINER_ID_INSIDE_MINIKUBE

          traceroute to aa7f7ed7af01 (198.105.244.24), 30 hops max, 60 byte packets
           1  172.17.0.1 (172.17.0.1)  0.031 ms  0.009 ms  0.007 ms
           2  10.0.2.2 (10.0.2.2)  0.156 ms  0.086 ms  0.050 ms
           3  * * *
           4  * * *
           5  dtr02gldlca-tge-0-2-0-1.gldl.ca.charter.com (96.34.102.201)  16.153 ms  16.107 ms  16.077 ms
           6  crr01lnbhca-bue-200.lnbh.ca.charter.com (96.34.98.188)  18.753 ms  18.011 ms  30.642 ms
           7  crr01mtpkca-bue-201.mtpk.ca.charter.com (96.34.96.63)  30.779 ms  30.523 ms  30.428 ms
           8  bbr01mtpkca-bue-2.mtpk.ca.charter.com (96.34.2.24)  24.089 ms  23.900 ms  23.814 ms
           9  bbr01ashbva-tge-0-1-0-1.ashb.va.charter.com (96.34.3.139)  26.061 ms  25.949 ms  36.002 ms
           10  10ge9-10.core1.lax1.he.net (65.19.189.177)  34.027 ms  34.436 ms  33.857 ms
           11  100ge12-1.core1.ash1.he.net (184.105.80.201)  107.873 ms  107.750 ms  104.078 ms
           12  100ge3-1.core1.nyc4.he.net (184.105.223.166)  100.554 ms  100.478 ms  100.393 ms
           13  xerocole-inc.10gigabitethernet12-4.core1.nyc4.he.net (216.66.41.242)  109.184 ms  111.122 ms  111.018 ms
           14  * * *
           15  * * * 
           ...(til it ends)

the plumsempy/plum can be any container since they are both on the same network and same pod, the pinging should go through. The question is Why can I not reach mysql on minikube and how could I fix that?

Answer 1

From k8s multi-container pod docs:

Pods share fate, and share some resources, such as storage volumes and IP addresses.

Hence the mysql container is reachable from the plum container at the IP address 127.0.0.1.

Also, since mysql runs on port 3306 by default, you probably want telnet 127.0.0.1 3306 to check if it's reachable (ping uses ICMP which doesn't have the concept of ports).

Answer 2

I guess the container ID just don't work with Kubernetes. You can also see, that the container ID resolved to the public IP 198.105.244.24, which looks wrong.

You have multiple ways to contact this pod:

• get the pod IP via kubectl describe -f k8s-deployment.yml
• create a service for that pod and do one of these (assuming the service name is mysql):
  • use environment variables like ping ${MYSQL_SERVICE_HOST}
  • use DNS like ping mysql.default.svc.cluster.local