Upgrading Kubernetes

Guide on how to upgrade the Kubernetes cluster from Talos Linux.

This guide covers upgrading Kubernetes on Talos Linux clusters.

For a list of Kubernetes versions compatible with each Talos release, see the Support Matrix.

For upgrading the Talos Linux operating system, see Upgrading Talos

Video walkthrough

To see a demo of this process, watch this video:

Automated Kubernetes upgrade

The recommended method to upgrade Kubernetes is to use the talosctl upgrade-k8s command. This will automatically update the components needed to upgrade Kubernetes safely. Kubernetes upgrades are non-disruptive from Talos, but Kubelet upgrades may cause workloads to restart.

To trigger a Kubernetes upgrade, issue a command specifying the version of Kubernetes to upgrade to, such as:

Note that the --nodes parameter specifies the control plane node to send the API call to, but all members of the cluster will be upgraded.

To check what will be upgraded you can run talosctl upgrade-k8s with the --dry-run flag:

You should get an output similar to this:

This command runs in several phases:

1. Images for new Kubernetes components are pre-pulled to the nodes to minimize downtime and test for image availability.
2. Every control plane node machine configuration is patched with the new image version for each control plane component. Talos renders new static pod definitions on the configuration update which is picked up by the kubelet. The command waits for the change to propagate to the API server state.
3. The command updates the kube-proxy daemonset with the new image version.
4. On every node in the cluster, the kubelet version is updated. The command then waits for the kubelet service to be restarted and become healthy. The update is verified by checking the Node resource state.
5. Kubernetes bootstrap manifests are re-applied to the cluster. Updated bootstrap manifests might come with a new Talos version (e.g. CoreDNS version update), or might be the result of machine configuration change.
6. Unneeded resources are pruned from the cluster (Talos v1.13+).

If the command fails for any reason, it can be safely restarted to continue the upgrade process from the moment of the failure.

When using custom/overridden Kubernetes component images, use flags --*-image to override the default image names.

Kubernetes manifest synchronization

A Talos version change might also cause changes in the bootstrap manifests. The updated bootstrap manifests, inline manifests and extra manifests are applied to the cluster as part of the Kubernetes upgrade process.

If any services were deleted or disabled, the resources associated with them will be pruned. For example if kube-proxy was disabled in the machine configuration the resources associated with it would be deleted at this stage. Pruning can be disabled by passing the --manifests-no-prune flag.

Pruning is supported from Talos and talosctl v1.13 onwards.

From Talos v1.13 onwards, all Kubernetes manifests are applied via Kubernetes Server-Side Apply. Talos forces ownership of all fields it applies, even if they have a different field manager.

If you wish to manage a resource previously applied by Talos, you need to take the following steps:

1. Remove the inline manifest or extra manifest entry from the machine configuration. After this step you can run talosctl upgrade-k8s --dry-run --to <in-cluster-k8s-version>. Resources affected will be marked for pruning in the output.
2. Remove the resource entries from the talos tracking inventory configmap data block by running:

kubectl edit cm --namespace kube-system talos-bootstrap-manifests-inventory

3. Remove the config.k8s.io/owning-inventory annotation from the resources.
4. Re-apply the resource with your own field manager or remove the metadata.managedFields manually.

Synchronize declared and deployed configurations

When Kubernetes is upgraded with talosctl upgrade-k8s, the live machine configuration on your nodes is updated with new component image versions (API server, controller manager, scheduler, kube-proxy, etc.).

If you are storing full machine configuration files (controlplane.yaml, worker.yaml) in Git, these versions will drift out of sync. Re-applying those stale files later could unintentionally downgrade components.

That is why we do not recommend storing full machine configurations.

Version numbers for Talos, etcd, Kubernetes components, and add-ons change frequently. Maintaining these across upgrades requires editing many fields by hand.

See the Reproducible Machine Configuration guide for full instructions on handling machine configurations after version bumps.

Manual Kubernetes upgrade

Kubernetes can be upgraded manually by following the steps outlined below. They are equivalent to the steps performed by the talosctl upgrade-k8s command.

Before running the commands below, export the k8s_release variable, replacing the placeholder with the target Kubernetes version:

export k8s_release=<target-k8s-version>

The talosctl patch mc examples use strategic-merge (YAML) patches, which work across supported Talos versions. From Talos v1.13 the machine configuration is a multi-document configuration, for which strategic-merge patches should be used rather than RFC 6902 (op/path/value) patches. A strategic-merge patch also creates the target key if it is not already present.

Kubeconfig

In order to edit the control plane, you need a working kubectl config. If you don’t already have one, you can get one by running:

talosctl --nodes <control-plane-ip> kubeconfig

API server

Patch the machine configuration with the new API server image:

talosctl -n <control-plane-ip> patch mc --mode=no-reboot -p "cluster:
  apiServer:
    image: registry.k8s.io/kube-apiserver:v${k8s_release}"

Capture the new version of kube-apiserver config with:

talosctl -n <control-plane-ip> get apiserverconfig -o yaml

You should get an output similar to this:

node: 172.20.0.2
metadata:
    namespace: controlplane
    type: APIServerConfigs.kubernetes.talos.dev
    id: kube-apiserver
    version: 5
    owner: k8s.ControlPlaneAPIServerController
    phase: running
spec:
    image: registry.k8s.io/kube-apiserver:v${k8s_release}
    cloudProvider: ""
    controlPlaneEndpoint: https://172.20.0.1:6443
    etcdServers:
        - https://localhost:2379
    localPort: 6443
    serviceCIDR:
        - 10.96.0.0/12
    extraArgs: {}
    extraVolumes: []
    environmentVariables: {}
    podSecurityPolicyEnabled: false
    advertisedAddress: $(POD_IP)
    resources:
        requests:
            cpu: ""
            memory: ""
        limits: {}

In this example, the new version is 5.

Wait for the new pod definition to propagate, then confirm the running kube-apiserver pods are using the new image version:

kubectl get pods -n kube-system -l k8s-app=kube-apiserver -o jsonpath='{.items[*].spec.containers[*].image}'

Repeat this process for every control plane node, verifying that state got propagated successfully between each node update.

Controller manager

Patch the machine configuration with the new controller manager image:

talosctl -n <control-plane-ip> patch mc --mode=no-reboot -p "cluster:
  controllerManager:
    image: registry.k8s.io/kube-controller-manager:v${k8s_release}"

Capture new version of kube-controller-manager config with:

talosctl -n <control-plane-ip> get controllermanagerconfig -o yaml

You should see an output similar to this:

node: 172.20.0.2
metadata:
    namespace: controlplane
    type: ControllerManagerConfigs.kubernetes.talos.dev
    id: kube-controller-manager
    version: 3
    owner: k8s.ControlPlaneControllerManagerController
    phase: running
spec:
    enabled: true
    image: registry.k8s.io/kube-controller-manager:v${k8s_release}
    cloudProvider: ""
    podCIDRs:
        - 10.244.0.0/16
    serviceCIDRs:
        - 10.96.0.0/12
    extraArgs: {}
    extraVolumes: []
    environmentVariables: {}
    resources:
        requests:
            cpu: ""
            memory: ""
        limits: {}

In this example, new version is 3.

Wait for the new pod definition to propagate, then confirm the running kube-controller-manager pods are using the new image version:

kubectl get pods -n kube-system -l k8s-app=kube-controller-manager -o jsonpath='{.items[*].spec.containers[*].image}'

Repeat this process for every control plane node, verifying that state propagated successfully between each node update.

Scheduler

Patch the machine configuration with the new scheduler image:

talosctl -n <control-plane-ip> patch mc --mode=no-reboot -p "cluster:
  scheduler:
    image: registry.k8s.io/kube-scheduler:v${k8s_release}"

Capture new version of kube-scheduler config with:

talosctl -n <control-plane-ip> get schedulerconfig -o yaml

You should see an output similar to this:

node: 172.20.0.2
metadata:
    namespace: controlplane
    type: SchedulerConfigs.kubernetes.talos.dev
    id: kube-scheduler
    version: 3
    owner: k8s.ControlPlaneSchedulerController
    phase: running
    created: 2024-11-06T12:37:22Z
    updated: 2024-11-06T12:37:20Z
spec:
    enabled: true
    image: registry.k8s.io/kube-scheduler:v${k8s_release}
    extraArgs: {}
    extraVolumes: []
    environmentVariables: {}
    resources:
        requests:
            cpu: ""
            memory: ""
        limits: {}
    config: {}

In this example, new version is 3.

WWait for the new pod definition to propagate, then confirm the running kube-scheduler pods are using the new image version:

kubectl get pods -n kube-system -l k8s-app=kube-scheduler -o jsonpath='{.items[*].spec.containers[*].image}'

Repeat this process for every control plane node, verifying that state got propagated successfully between each node update.

Proxy

Upgrade kube-proxy through the machine configuration, in the same way as the other control plane components. The kube-proxy DaemonSet is generated from .cluster.proxy.image and is reconciled when the bootstrap manifests are applied in the next step, so its image should be set in the machine configuration rather than by editing the live DaemonSet.

Patch the machine configuration with the new proxy image:

talosctl -n <control-plane-ip> patch mc --mode=no-reboot -p "cluster:
  proxy:
    image: registry.k8s.io/kube-proxy:v${k8s_release}"

Repeat for every control plane node. The updated DaemonSet is rolled out when you apply the bootstrap manifests in the next step.

Bootstrap manifests

Bootstrap manifests can be retrieved in a format which works for kubectl with the following command:

talosctl -n <control-plane-ip> get manifests -o yaml | yq eval-all '.spec | .[] | splitDoc' - > manifests.yaml

Diff the manifests with the cluster:

kubectl diff -f manifests.yaml

Confirm that the kube-proxy DaemonSet image shown in the diff matches the target version (v${k8s_release}). If it shows the previous version, set .cluster.proxy.image in the machine configuration as described in the Proxy step before applying.

Apply the manifests:

kubectl apply -f manifests.yaml

If some bootstrap resources were removed, they have to be removed from the cluster manually.

kubelet

For every node, patch the machine configuration with the new kubelet version, then wait for the kubelet to restart:

talosctl -n <node-ip> patch mc --mode=no-reboot -p "machine:
  kubelet:
    image: ghcr.io/siderolabs/kubelet:v${k8s_release}"

Once kubelet restarts with the new configuration, confirm the upgrade. The VERSION column should show the new Kubernetes version for every node:

kubectl get nodes -o wide