Kubespray - Upgrade a cluster which is created from master branch

At first the 1st cluster doc: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/setting-up-your-first-cluster.md

No info about that you must change to a release version, so I have initialised my cluster (v1.20.2) from the MASTER branch few months ago.

Now, I want to upgrade to v1.20.5 (at first tried the v1.20.3) but I have already pulled the changes from master and all the upgrades failed.

I tried this:

ansible-playbook  -i inventory/mycluster/hosts.yaml  --become --become-user=root upgrade-cluster.yml -e kube_version=v1.20.5 --extra-vars "upgrade_node_pause_seconds=300"  

The error was:

TASK [kubernetes/control-plane : kubeadm | Upgrade first master] ******************************************************************************************************************************************fatal: [node1]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["timeout", "-k", "600s", "600s", "/usr/local/bin/kubeadm", "upgrade", "apply", "-y", "v1.20.3", "--config=/etc/kubernetes/kubeadm-config.yaml", "--ignore-preflight-errors=all", "--allow-experimental-upgrades", "--etcd-upgrade=false", "--force"], "delta": "0:05:05.154324", "end": "2021-04-09 21:12:40.777249", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2021-04-09 21:07:35.622925", "stderr": "W0409 21:07:35.722848 1981949 common.go:94] WARNING: Usage of the --config flag with kubeadm config types for reconfiguring the cluster during upgrade is not recommended!\nW0409 21:07:35.732860 1981949 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]\n\t[WARNING ControlPlaneNodesReady]: there are NotReady control-planes in the cluster: [node1]\n\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.20.3: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"\n, error: exit status 1\n\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.20.3: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"\n, error: exit status 1\n\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.20.3: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"\n, error: exit status 1\n\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.20.3: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"\n, error: exit status 1\n\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/pause:3.2: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"\n, error: exit status 1\n\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/coredns:1.7.0: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"\n, error: exit status 1\n[upgrade/apply] FATAL: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: timed out waiting for the condition\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["W0409 21:07:35.722848 1981949 common.go:94] WARNING: Usage of the --config flag with kubeadm config types for reconfiguring the cluster during upgrade is not recommended!", "W0409 21:07:35.732860 1981949 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]", "\t[WARNING ControlPlaneNodesReady]: there are NotReady control-planes in the cluster: [node1]", "\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.20.3: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"", ", error: exit status 1", "\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.20.3: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"", ", error: exit status 1", "\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.20.3: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"", ", error: exit status 1", "\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.20.3: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"", ", error: exit status 1", "\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/pause:3.2: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"", ", error: exit status 1", "\t[WARNING ImagePull]: failed to pull image k8s.gcr.io/coredns:1.7.0: output: time="2021-04-09T21:07:39+02:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"", ", error: exit status 1", "[upgrade/apply] FATAL: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: timed out waiting for the condition", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[upgrade/config] Making sure the configuration is correct:\n[preflight] Running pre-flight checks.\n[upgrade] Running cluster health checks\n[upgrade/version] You have chosen to change the cluster version to "v1.20.3"\n[upgrade/versions] Cluster version: v1.19.5\n[upgrade/versions] kubeadm version: v1.20.3\n[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster\n[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection\n[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'\n[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.20.3"...\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-controller-manager-node1 hash: 63ee1f7948603626aeba424b632d84c0\nStatic pod: kube-scheduler-node1 hash: d09bb2d5aebd882f893433064fea3666\n[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests067570520"\n[upgrade/staticpods] Preparing for "kube-apiserver" upgrade\n[upgrade/staticpods] Renewing apiserver certificate\n[upgrade/staticpods] Renewing apiserver-kubelet-client certificate\n[upgrade/staticpods] Renewing front-proxy-client certificate\n[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-04-09-21-07-39/kube-apiserver.yaml"\n[upgrade/staticpods] Waiting for the kubelet to restart the component\n[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1 hash: 01671f394a962c6160319190a8c77176\nStatic pod: kube-apiserver-node1

Now, I have a cluster, which I can't upgrade anymore.

Do you have any ideas how to upgrade/maintain this? Or change to a supported kubespray release.. I've tried to downgrade, but not worked.

I can rebuild my cluster from strach again, but I would try another options before that.

https://stackoverflow.com/questions/67028603/kubespray-upgrade-a-cluster-which-is-created-from-master-branch April 10, 2021 at 05:38AM