I have a Pod that occasionally needs to call itself by its own host-name. I have a Deployment set up that ultimately creates the Pod, and a Service in place. I'm able to use the service name from a different Pod within my cluster, but the Pod cannot call itself using its host-name. This is something that works with Minikube v0.17.1 with Kubernetes version 1.5.3, but upgrading either Minikube or the Kubernetes version seems to break things. This also works on our deployed / server version of Kubernetes.
Did something change that I need to take account of in my Pod / Service setup? How do I go about getting past this?
How things should work
Running Minikube version 0.17.1:
Start Minikube:
$ minikube start
Starting local Kubernetes cluster...
Starting VM...
SSH-ing files into VM...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}
$ minikube version
minikube version: v0.17.1
Deploy minimal test images (yaml definition below):
kubectl apply -f deploy_python.yaml
Exec into the python2 image, and verify connection to python image:
$ kubectl exec -it python2-1281934109-k015g bash
root@python2-1281934109-k015g:/# curl python:12345 --connect-timeout 10
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href=".dockerenv">.dockerenv</a></li>
<li><a href="bin/">bin/</a></li>
<li><a href="boot/">boot/</a></li>
<li><a href="dev/">dev/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="home/">home/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="lib64/">lib64/</a></li>
<li><a href="media/">media/</a></li>
<li><a href="mnt/">mnt/</a></li>
<li><a href="opt/">opt/</a></li>
<li><a href="proc/">proc/</a></li>
<li><a href="root/">root/</a></li>
<li><a href="run/">run/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="srv/">srv/</a></li>
<li><a href="sys/">sys/</a></li>
<li><a href="tmp/">tmp/</a></li>
<li><a href="usr/">usr/</a></li>
<li><a href="var/">var/</a></li>
</ul>
<hr>
</body>
</html>
Exec into python image and verify connection to self:
$ kubectl exec -it python-2555691705-5j0f9 bash
root@python-2555691705-5j0f9:/# curl python:12345 --connect-timeout 10
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href=".dockerenv">.dockerenv</a></li>
<li><a href="bin/">bin/</a></li>
<li><a href="boot/">boot/</a></li>
<li><a href="dev/">dev/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="home/">home/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="lib64/">lib64/</a></li>
<li><a href="media/">media/</a></li>
<li><a href="mnt/">mnt/</a></li>
<li><a href="opt/">opt/</a></li>
<li><a href="proc/">proc/</a></li>
<li><a href="root/">root/</a></li>
<li><a href="run/">run/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="srv/">srv/</a></li>
<li><a href="sys/">sys/</a></li>
<li><a href="tmp/">tmp/</a></li>
<li><a href="usr/">usr/</a></li>
<li><a href="var/">var/</a></li>
</ul>
<hr>
</body>
</html>
That's a success. By creating a Deployment and a Service, I'm able to make requests to the referenced Pod from any other Pod in the cluster.
The way it works in newer versions
(Stop and delete running Minikube.)
Start Minikube, specifying Kubernetes version 1.7.0:
$ minikube start --kubernetes-version=v1.7.0
Starting local Kubernetes cluster...
Starting VM...
SSH-ing files into VM...
Downloading localkube binary
137.48 MB / 137.48 MB [============================================] 100.00% 0s
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-30T10:17:58Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Deploy minimal test images (yaml definition below):
$ kubectl apply -f deploy_python.yaml
service "python" created
deployment "python" created
deployment "python2" created
Exec into python 2 image, and verify connection to python image:
$ kubectl exec -it python2-380393367-ztgkq bash
root@python2-380393367-ztgkq:/# curl python:12345 --connect-timeout 10
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href=".dockerenv">.dockerenv</a></li>
<li><a href="bin/">bin/</a></li>
<li><a href="boot/">boot/</a></li>
<li><a href="dev/">dev/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="home/">home/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="lib64/">lib64/</a></li>
<li><a href="media/">media/</a></li>
<li><a href="mnt/">mnt/</a></li>
<li><a href="opt/">opt/</a></li>
<li><a href="proc/">proc/</a></li>
<li><a href="root/">root/</a></li>
<li><a href="run/">run/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="srv/">srv/</a></li>
<li><a href="sys/">sys/</a></li>
<li><a href="tmp/">tmp/</a></li>
<li><a href="usr/">usr/</a></li>
<li><a href="var/">var/</a></li>
</ul>
<hr>
</body>
</html>
Exec into python image and attempt connection to self:
$ kubectl exec -it python-2168884431-gls2j bash
root@python-2168884431-gls2j:/# curl python:12345 --connect-timeout 10
curl: (28) Connection timed out after 10000 milliseconds
Yaml file, deploy_python.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: python
spec:
selector:
app: python
ports:
- port: 12345
targetPort: 12345
name: http
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: python
labels:
app: python
spec:
replicas: 1
template:
metadata:
labels:
app: python
spec:
containers:
- image: python
name: python
command: ["python"]
args: ["-m", "http.server", "12345" ]
ports:
- containerPort: 12345
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: python2
labels:
app: python2
spec:
replicas: 1
template:
metadata:
labels:
app: python2
spec:
containers:
- image: python
name: python2
command: ["python"]
args: ["-m", "http.server", "12345" ]
ports:
- containerPort: 12345
This is a known bug in newer versions of kubernetes / minikube and is being tracked here:
https://github.com/kubernetes/kubernetes/issues/20475
The current accepted workaround (https://github.com/kubernetes/kubernetes/issues/20475#issuecomment-190995739) is to ssh into your minikube vm and run a specific ip link command.
I have tested this workaround and it seems to work for me.