Upgrading Rancher TLDR version;

Before you start, understand this part. Rancher is a deployment running on Kubernetes and they are upgraded separately. Since Rancher is depending on Kubernetes you need to first figure out if the version of Rancher you want to upgrade to can run on the current Kubernetes version.

There are also multiple ways of initially setting up a cluster. This article will cover RKE. If you have used RKE2 or K3s you can stop reading now.

Due diligence

Login to the node you used to create the local cluster. It will have the cluster.yaml file in the rancher user’s home directory. In this directory you’ll also have the Kubconfig file for the local cluster. Start by setting the KUBECONFIG environment variable and check out the Kubernetes version of the nodes and my installed Docker version.

# The file name of the Kubeconfig could differ
export KUBECONFIG=~/kube_config_rancher-cluster.yml
kubectl get nodes
# NAME             STATUS   ROLES                      AGE    VERSION
# 192.168.50.232   Ready    controlplane,etcd,worker   480d   v1.21.13
# 192.168.50.233   Ready    controlplane,etcd,worker   480d   v1.21.13
# 192.168.50.234   Ready    controlplane,etcd,worker   480d   v1.21.13
docker --version
# Docker version 20.10.17, build 100c701

The Kubernetes version is v1.21.13. Save this for later in case you need to roll back! Now it’s time to check out if the Rancher version you want to upgrade to can support this version. You can get this information from the Rancher Support Matrix.

Now, their table was confusing to me too and it took embarrassingly long to decipher it.

I want to upgrade to Rancher v2.6.5.
I am running my local cluster on Ubuntu 20.04
My Kubernetes version is v1.21.13 (as determined above)

The table then recommends RKE version v1.3.8:

Scrolling down to the Ubuntu section we can find that 20.04 running Kubernetes v1.21.x and Docker version 20.10.x is supported.

Important, do not upgrade Kubernetes to a version that your current Rancher version does not support! If that is the case you need to upgrade Rancher first!

Upgrading Kubernetes

Before going ahead with this, make sure that you have done and read the previous section, especially the last paragraph.

I don’t really need to do this, but for the sake of this article I’ll cover how I did it anyway. First, download the recommended version of RKE from Github to the same folder as your cluster.yaml. I can highly recommend to name it after the version to keep them apart just in case.

curl https://github.com/rancher/rke/releases/download/v1.3.8/rke_linux-amd64 -o rke-v1.3.8
chmod +x ./rke-v1.3.8

Once you have the file you can check out the supported Kubernetes versions by running this command:

./rke-v1.3.8 config --list-version --all
# v1.23.4-rancher1-1
# v1.18.20-rancher1-3
# v1.20.15-rancher1-2
# v1.19.16-rancher1-4
# v1.21.10-rancher1-1
# v1.22.7-rancher1-1

Pick one that is supported by your intended Rancher version and maybe not the latest unless you have a practical reason to do so. Then edit your cluster.yml file and add a key called kubernetes_version with a value of the Kubernetes version you picked above. In my case the cluster.yaml file looks like this:

kubernetes_version: v1.21.10-rancher1-1
nodes:
  - address: 192.168.50.232
    user: rancher
    role: [controlplane, worker, etcd]
    ssh_key_path: ~/.ssh/id_rsa
  - address: 192.168.50.233
    user: rancher
    role: [controlplane, worker, etcd]
    ssh_key_path: ~/.ssh/id_rsa
  - address: 192.168.50.234
    user: rancher
    ssh_key_path: ~/.ssh/id_rsa
    role: [controlplane, worker, etcd]

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

# Required for external TLS termination with
# ingress-nginx v0.22+
ingress:
  provider: nginx
  options:
    use-forwarded-headers: "true"

Make a backup of your cluster state using the following command:

./rke-v1.3.8 etcd snapshot-save --config rancher-cluster.yaml --name $(date '+%Y%m%d%H%m')

Then you can upgrade your Kubernetes version using the following command:

./rke up -config cluster.yml

You can now follow the progress with:

kubectl get nodes --watch
NAME             STATUS   ROLES                      AGE    VERSION
192.168.50.232   Ready    controlplane,etcd,worker   480d   v1.21.13
192.168.50.233   Ready    controlplane,etcd,worker   480d   v1.21.13
192.168.50.234   Ready    controlplane,etcd,worker   480d   v1.21.13

Once all of the nodes has the intended version it’s time to update Rancher.