Recently i was deploying a python flask application within a docker container and behind a k8s service and i kept on getting
connection refused exceptions while connecting to the flask application.
Here is the sample code for the application
from flask import Flask app = Flask(__name__) @app.route('/') def hello(): return 'Hello, World!' if __name__ == '__main__': app.run(debug=True, host='127.0.0.1')
apiVersion: v1 kind: Service metadata: name: flask-python-service spec: selector: app: flask-python ports: - protocol: "TCP" port: 8000 targetPort: 8000 type: LoadBalancer --- apiVersion: apps/v1 kind: Deployment metadata: name: flask-python spec: selector: matchLabels: app: flask-python replicas: 1 template: metadata: labels: app: flask-python spec: containers: - name: flask-python image: flask-python:v3 ports: - containerPort: 8000
As you can see that when we were trying to reach the endpoint for flask-python-service from some other pod, we were getting
Connection Refused errors.
[ root@curl-test:/ ]$ curl flask-python-service:8000 curl: (7) Failed to connect to flask-python-service port 8000: Connection refused [ root@curl-test:/ ]$
On looking further, much to our surprise, we were however able to access the localhost:8000 from within the flask python pod without any issues.
[root@flask-python-5fcf7c44d6-gpksc opt]# curl localhost:8000; echo "Hello, World!" [root@flask-python-5fcf7c44d6-gpksc opt]#
After many head scratches and wasted many hours of debugging, we figured that the issue was the flask application was being bounded to 127.0.0.1 and not on 0.0.0.0. For some of the folks reading this blog they might be thinking , “What is the difference if a service listens on 127.0.0.1 instead of 0.0.0.0” ???
When the application is bounded to 127.0.0.1 / port, the route table makes sure that if there is any connection request to 127.0.0.1 / port, then the packets get routed to the application listening on a particular port on 127.0.0.1
[root@flask-python-5fcf7c44d6-gpksc opt]# netstat -nap | grep LISTEN tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN 149/python [root@flask-python-5fcf7c44d6-gpksc opt]#
But if the application is bounded to 0.0.0.0 / port, the route tables makes sure that if there is any connection request coming to the IP among any IP Addresses bounded to the host, the OS forwards it to the application listening on the port.
In the context of servers, 0.0.0.0 means "all IPv4 addresses on the local machine". If a host has two ip addresses, 192.168.1.1 and 10.1.2.1, and a server running on the host listens on 0.0.0.0, it will be reachable at both of those IPs.
In our context, K8s server gives each and every pod it’s own IP. In our case, k8s server has given the flask-python service an IP
192.168.53.122 . So the flask python server running within the pod can be reached via
192.168.53.122 as well as
127.0.0.1 within the same pod.
➜ ~ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES flask-python-6b474476c4-m7k95 1/1 Running 0 2m11s 192.168.53.122 ??? <none> <none> ➜ ~
Now because the flask python service which is running within the pod has essentially bind itself to
127.0.0.1 . This means that if the connection request is coming for
192.168.53.122 or flask-python-service, then the OS will refuse the connection because there is no service which has bind itself to
However if the service instead of binding itself to
127.0.0.1 had binded itself to
0.0.0.0, then all the connection requests coming to the pods be it to any IP address of the pod, the OS would have created the connection with the flask python service.
Let’s deploy the below python script with binded IP being
from flask import Flask app = Flask(__name__) @app.route('/') def hello(): return 'Hello, World!' if __name__ == '__main__': app.run(debug=True, host='0.0.0.0')
[ root@curl-test:/ ]$ [ root@curl-test:/ ]$ curl flask-python-service:8000 ; echo "Hello, World!" [ root@curl-test:/ ]$ [ root@curl-test:/ ]$
Now’ lets see the output of netstat and see what IP it is actually getting binded to
[root@flask-python-5fcf7c44d6-gpksc opt]# netstat -nap | grep LISTEN tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 8/python3.8 [root@flask-python-5fcf7c44d6-gpksc opt]#
As we can see that in this case, the flask python process is getting binded to
0.0.0.0. This means that any connection request which is coming to this host with whatever IP among those which have been assigned to the host is most welcome and the connection will be routed to the appropriate service accordingly.