Fix: kubectl the Connection to the Server xxx Was Refused

Posted by Admin on August 23, 2022

Symptom

When using kubectl to administer your Kubernetes cluster, you may receive a message similar to

The connection to the server xxx.xxx.xxx.xxx:xxxx was refused - did you specify the right host or port?

When kubectl returns the connection to the server localhost:8080 was refused, one reason may be that the client certficates for accessing your Kubernetes cluster have expired. How to renew expired client certificates is the topic of this article.

There are other cause for the "connection refused" message, so check a couple of things with your cluster before trying out the below steps to resolve your issue.

Before You Begin

  • Does your Kubernetes cluster use client certificates to authenticate? If yes, then this article may help. If no, then you likely have another problem
  • Do you receive this "connection refused" message when you are tunneled into your Kubernetes master node running a 'kubectl' command? If yes, then this article will likely help you.

My Kubernetes cluster is outdated. It's currently running v1.15.2. As of this writing, the Kubernetes documentation only goes back to v1.20. After I resolve this certificate issue, I'll upgrade my cluster so I'm running the latest version of Kubernetes.

To definitely see if you have an expired certificate run the following command.

$ kubeadm certs check-expiration

If you receive an error message, try running the command for an older version of kubectl.

$ kubeadm alpha certs check-expiration

If it works, you'll see similar output.

CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Aug 21, 2022 19:57 UTC   <invalid>       no      
apiserver                  Aug 21, 2022 19:55 UTC   <invalid>       no      
apiserver-etcd-client      Aug 21, 2022 19:55 UTC   <invalid>       no      
apiserver-kubelet-client   Aug 21, 2022 19:55 UTC   <invalid>       no      
controller-manager.conf    Aug 21, 2022 19:57 UTC   <invalid>       no      
etcd-healthcheck-client    Aug 21, 2022 19:04 UTC   <invalid>       no      
etcd-peer                  Aug 21, 2022 19:04 UTC   <invalid>       no      
etcd-server                Aug 21, 2022 19:04 UTC   <invalid>       no      
front-proxy-client         Aug 21, 2022 19:55 UTC   <invalid>       no      
scheduler.conf             Aug 21, 2022 19:57 UTC   <invalid>       no

Note that the expiration date has passed and the Residual Time says <invalid>. This means my client certificates have expired.

Renew the certificates by running the following command. Note that this command is for v1.15.2 version of kubernetes.

$ kubeadm alpha certs renew all

Now, try a kubectl command.

$ kubectl get pods

If you receive the following error, you may have an old version of kubernetes.

Unable to connect to the server: x509: certificate has expired or is not yet valid

To reconcile, run the following command where xx.xx.xx.xx is your cluster's IP address.

$ kubeadm init phase certs all --apiserver-advertise-address xx.xx.xx.xx

Then run the following.

$ kubeadm init phase kubeconfig all

Then reboot.

Then run the following.

$ cp -i /etc/kubernetes/admin.conf ~/.kube/config

After that you should be able to run kubectl commands.