Debugging K8s Connection Refused

Recently i was deploying a python flask application within a docker container and behind a k8s service and i kept on getting connection refused exceptions while connecting to the flask application.

Here is the sample code for the application

from flask import Flask

app = Flask(__name__)


@app.route('/')
def hello():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(debug=True, host='127.0.0.1')

apiVersion: v1
kind: Service
metadata:
  name: flask-python-service
spec:
  selector:
    app: flask-python
  ports:
  - protocol: "TCP"
    port: 8000
    targetPort: 8000
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-python
spec:
  selector:
    matchLabels:
      app: flask-python
  replicas: 1
  template:
    metadata:
      labels:
        app: flask-python
    spec:
      containers:
      - name: flask-python
        image: flask-python:v3
        ports:
        - containerPort: 8000

As you can see that when we were trying to reach the endpoint for flask-python-service from some other pod, we were getting Connection Refused errors.

[ root@curl-test:/ ]$ curl flask-python-service:8000
curl: (7) Failed to connect to flask-python-service port 8000: Connection refused
[ root@curl-test:/ ]$

On looking further, much to our surprise, we were however able to access the localhost:8000 from within the flask python pod without any issues.

[root@flask-python-5fcf7c44d6-gpksc opt]# curl localhost:8000; echo
"Hello, World!"
[root@flask-python-5fcf7c44d6-gpksc opt]#

After many head scratches and wasted many hours of debugging, we figured that the issue was the flask application was being bounded to 127.0.0.1 and not on 0.0.0.0. For some of the folks reading this blog they might be thinking , “What is the difference if a service listens on 127.0.0.1 instead of 0.0.0.0” ???

When the application is bounded to 127.0.0.1 / port, the route table makes sure that if there is any connection request to 127.0.0.1 / port, then the packets get routed to the application listening on a particular port on 127.0.0.1

[root@flask-python-5fcf7c44d6-gpksc opt]# netstat -nap | grep LISTEN
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      149/python
[root@flask-python-5fcf7c44d6-gpksc opt]#

But if the application is bounded to 0.0.0.0 / port, the route tables makes sure that if there is any connection request coming to the IP among any IP Addresses bounded to the host, the OS forwards it to the application listening on the port.

In the context of servers, 0.0.0.0 means "all IPv4 addresses on the local machine". If a host has two ip addresses, 192.168.1.1 and 10.1.2.1, and a server running on the host listens on 0.0.0.0, it will be reachable at both of those IPs.

In our context, K8s server gives each and every pod it’s own IP. In our case, k8s server has given the flask-python service an IP 192.168.53.122 . So the flask python server running within the pod can be reached via 192.168.53.122 as well as 127.0.0.1 within the same pod.

➜  ~ kubectl get pods -o wide
NAME                            READY   STATUS    RESTARTS   AGE     IP               NODE                                          NOMINATED NODE   READINESS GATES
flask-python-6b474476c4-m7k95   1/1     Running   0          2m11s   192.168.53.122   ???   <none>           <none>
➜  ~

Now because the flask python service which is running within the pod has essentially bind itself to 127.0.0.1 . This means that if the connection request is coming for 192.168.53.122 or flask-python-service, then the OS will refuse the connection because there is no service which has bind itself to 192.168.53.122.

However if the service instead of binding itself to 127.0.0.1 had binded itself to 0.0.0.0, then all the connection requests coming to the pods be it to any IP address of the pod, the OS would have created the connection with the flask python service.

• 

Let’s deploy the below python script with binded IP being 0.0.0.0

from flask import Flask

app = Flask(__name__)


@app.route('/')
def hello():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

[ root@curl-test:/ ]$
[ root@curl-test:/ ]$ curl flask-python-service:8000 ; echo
"Hello, World!"
[ root@curl-test:/ ]$
[ root@curl-test:/ ]$

Now’ lets see the output of netstat and see what IP it is actually getting binded to

[root@flask-python-5fcf7c44d6-gpksc opt]# netstat -nap | grep LISTEN
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      8/python3.8
[root@flask-python-5fcf7c44d6-gpksc opt]#

As we can see that in this case, the flask python process is getting binded to 0.0.0.0. This means that any connection request which is coming to this host with whatever IP among those which have been assigned to the host is most welcome and the connection will be routed to the appropriate service accordingly.