Kubernetes憑證過期：Unable To Connect To The Server: X509: Certificate Has Expired Or Is Not Yet Valid

👨‍💻簡介

今天早上在下kubectl get pods時，突然跳出了以下錯誤

warning

Unable to connect to the server: x509: certificate has expired or is not yet valid

下了kubeadm alpha certs check-expiration之後才發現原來是憑證過期因此紀錄一下解決過程

Information

environment

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

docker version

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-162.git64e9980.el7.centos.x86_64
 Go version:      go1.10.3
 Git commit:      64e9980/1.13.1
 Built:           Wed Jul  1 14:56:42 2020
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-162.git64e9980.el7.centos.x86_64
 Go version:      go1.10.3
 Git commit:      64e9980/1.13.1
 Built:           Wed Jul  1 14:56:42 2020
 OS/Arch:         linux/amd64
 Experimental:    false

kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:56:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

k8s version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

💡解決步驟

1. 更新憑證

執行以下命令備份現有k8s 憑證

$ mkdir -p $HOME/k8s-old-certs/pki 
$ /bin/cp -p /etc/kubernetes/pki/*.* $HOME/k8s-old-certs/pki 
$ ls -l $HOME/k8s-old-certs/pki/

輸出類似以下內容:

total 56
-rw-r--r-- 1 root root 1090 Jul 27  2022 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Jul 27  2022 apiserver-etcd-client.key
-rw-r--r-- 1 root root 1099 Jul 27  2022 apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Jul 27  2022 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1229 Jul 27  2022 apiserver.crt
-rw------- 1 root root 1675 Jul 27  2022 apiserver.key
-rw-r--r-- 1 root root 1025 Jul 25  2020 ca.crt
-rw------- 1 root root 1675 Jul 25  2020 ca.key
-rw-r--r-- 1 root root 1038 Jul 27  2022 front-proxy-ca.crt
-rw------- 1 root root 1675 Jul 27  2022 front-proxy-ca.key
-rw-r--r-- 1 root root 1058 Jul 27  2022 front-proxy-client.crt
-rw------- 1 root root 1679 Jul 27  2022 front-proxy-client.key
-rw------- 1 root root 1679 Jul 25  2020 sa.key
-rw------- 1 root root  451 Jul 25  2020 sa.pub

執行以下命令備份現有k8s設定檔

/bin/cp -p /etc/kubernetes/*.conf $HOME/k8s-old-certs 
ls -ltr $HOME/k8s-old-certs

輸出類似以下內容:

total 36
-rw------- 1 root root 5506 Jul 27  2022 kubelet.conf
-rw------- 1 root root 5453 Jul 27  2022 admin.conf
-rw------- 1 root root 5489 Jul 27  2022 controller-manager.conf
-rw------- 1 root root 5437 Jul 27  2022 scheduler.conf
drwxrwxr-x 2 root root 4096 Jul 27 12:05 pki

mkdir -p $HOME/k8s-old-certs/.kube 
/bin/cp -p ~/.kube/config $HOME/k8s-old-certs/.kube/. 
ls -l $HOME/k8s-old-certs/.kube/.

輸出類似以下內容:

total 8
-rw------- 1 root root 5454 Jul 27  2022 config

執行以下命令更新所有k8s憑證

kubeadm alpha certs renew all

輸出類似以下內容:

[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

執行以下命令確認憑證是否更新並顯示364天後過期:

kubeadm alpha certs check-expiration

輸出類似以下內容:

[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jul 26, 2024 04:14 UTC   364d                                    no
apiserver                  Jul 26, 2024 04:15 UTC   364d            ca                      no
apiserver-etcd-client      Jul 26, 2024 04:15 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jul 26, 2024 04:15 UTC   364d            ca                      no
controller-manager.conf    Jul 26, 2024 04:15 UTC   364d                                    no
etcd-healthcheck-client    Jul 26, 2024 04:15 UTC   364d            etcd-ca                 no
etcd-peer                  Jul 26, 2024 04:15 UTC   364d            etcd-ca                 no
etcd-server                Jul 26, 2024 04:15 UTC   364d            etcd-ca                 no
front-proxy-client         Jul 26, 2024 04:15 UTC   364d            front-proxy-ca          no
scheduler.conf             Jul 26, 2024 04:16 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jul 23, 2030 03:21 UTC   6y              no
etcd-ca                 Jul 23, 2030 03:21 UTC   6y              no
front-proxy-ca          Jul 24, 2032 03:09 UTC   8y              no

2. 確認kubelet設定

這時候下 kubectl get pods可能會噴以下錯誤

warning

error: You must be logged in to the server (Unauthorized)

此時可以使用指令比對一下家目錄的設定檔是否更新

diff $HOME/.kube/config/  /etc/kubernetes/admin.conf

如果沒顯示任何輸出，原因是家目錄的設定檔沒更新，因此必須複製一份新的設定檔到家目錄

cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

接著需要重起kubelet以及相關的docker服務(master與node都要重啟)

systemctl restart docker 
systemctl restart kubelet

如果以上指令沒有輸出表示成功，最後確認是否新的設定檔成功

kubectl get node

NAME      STATUS   ROLES    AGE      VERSION
master    Ready    <none>   3y2d     v1.18.6
worker1   Ready    <none>   3y2d     v1.18.6
worker2   Ready    <none>   3y2d     v1.18.6

以上就是這次更新憑證的過程。

👨‍💻簡介#

Information#

💡解決步驟#

1. 更新憑證#

2. 確認kubelet設定#

📚Reference#

👨‍💻簡介

Information

💡解決步驟

1. 更新憑證

2. 確認kubelet設定

📚Reference