Upgrading Talos Linux

Guide to upgrading a Talos Linux machine.

OS upgrades are effected by an API call, which can be sent via the talosctl CLI utility.

The upgrade API call passes a node the installer image to use to perform the upgrade.

Each Talos version has a corresponding installer image, listed on the release page for the version, for example .

Upgrades use an A-B image scheme in order to facilitate rollbacks. This scheme retains the previous Talos kernel and OS image following each upgrade. If an upgrade fails to boot, Talos will roll back to the previous version. Likewise, Talos may be manually rolled back via API (or talosctl rollback), which will update the boot reference and reboot.

Note An upgrade of the Talos Linux OS will not (since v1.0) apply an upgrade to the Kubernetes version by default. Kubernetes upgrades should be managed separately per upgrading kubernetes.

Supported upgrade paths

Because Talos Linux is image based, an upgrade is almost the same as installing Talos, with the difference that the system has already been initialized with a configuration. The supported configuration may change between versions. The upgrade process should handle such changes transparently, but this migration is only tested between adjacent minor releases. Thus the recommended upgrade path is to always upgrade to the latest patch release of all intermediate minor releases.

For example, if upgrading from Talos 1.0 to Talos 1.2.4, the recommended upgrade path would be:

• upgrade from 1.0 to latest patch of 1.0 - to v1.0.6
• upgrade from v1.0.6 to latest patch of 1.1 - to v1.1.2
• upgrade from v1.1.2 to v1.2.4

Before upgrade to

Migrate the multipath configuration before upgrading.

The multipath-tools system extension now reads /etc/multipath.conf from the Talos host instead of using ExtensionServiceConfig. If the machine configuration still contains an ExtensionServiceConfig document named multipathd, the service waits indefinitely for /etc/multipath.conf after the upgrade. Apply the patch below to every affected machine before upgrading Talos or the extension.

apiVersion: v1alpha1
kind: ExtensionServiceConfig
name: multipathd
$patch: delete
---
apiVersion: v1alpha1
kind: EtcFileConfig
name: multipath.conf
mode: 0o644
contents: |
  defaults {
      user_friendly_names yes
      find_multipaths no
      path_selector "round-robin 0"
  }

Note that the multipath.conf provided is a minimal configuration that works for most setups; if you have a custom configuration, update the contents field accordingly as per your requirements.

The patch removes the old extension service configuration and creates the host file that multipathd waits for and bind-mounts read-only.

Video walkthrough

To see a live demo of an upgrade of Talos Linux, see the video below:

After upgrade to

There are no specific actions to be taken after an upgrade.

talosctl upgrade

To upgrade a Talos node, specify the node’s IP address and the installer container image for the version of Talos to upgrade to.

Starting with Talos 1.14, the ghcr.io/siderolabs/installer image is no longer published: installer images are served by the Image Factory as factory.talos.dev/metal-installer/<schematic-id>:<version>. Use the schematic ID of the image the node was installed from (the one which carries its system extensions), or the empty schematic 376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba for a machine without customizations.

For instance, if your Talos node has the IP address 10.20.30.40 and you want to install the current version, you would enter a command such as:

The --image flag defaults to the installer image of the talosctl version in use, so it can be omitted when upgrading a machine without system extensions to the version matching talosctl.

Note that because Talos Linux reboots via the kexec syscall, the extra reboot adds very little time.

Machine configuration changes

Talos 1.14 continues splitting the monolithic v1alpha1 document into small, focused configuration documents. Every v1alpha1 field listed below is deprecated, but still supported, so a machine configuration which only uses v1alpha1 keeps working after the upgrade. Unless noted otherwise, a deprecated field and its replacement document are mutually exclusive: a configuration which sets both is rejected.

Kubernetes

Kubernetes configuration is now expressed as a set of dedicated documents; new Kubernetes features and settings will only be available in the new documents. kube-proxy is now configured with a configuration file (via KubeProxyConfig) instead of command line arguments.

Deprecated v1alpha1 field	Replacement document
.cluster.clusterName, .cluster.controlPlane.endpoint	KubeClusterConfig
.cluster.apiServer, .cluster.controlPlane.localAPIServerPort	KubeAPIServerConfig, KubeAdmissionControlConfig, KubeAuditPolicyConfig, KubeAuthenticationConfig, KubeAuthorizerConfig
.cluster.ca, .cluster.acceptedCAs	KubeAPIServerCAConfig
.cluster.aggregatorCA	KubeAggregatorCAConfig
.cluster.serviceAccount	KubeServiceAccountConfig
.cluster.controllerManager, .machine.controlPlane.controllerManager	KubeControllerManagerConfig
.cluster.scheduler, .machine.controlPlane.scheduler	KubeSchedulerConfig
.cluster.proxy	KubeProxyConfig
.cluster.network	KubeNetworkConfig, KubeFlannelCNIConfig
.cluster.coreDNS	KubeCoreDNSConfig
.cluster.aescbcEncryptionSecret, .cluster.secretboxEncryptionSecret	KubeEtcdEncryptionConfig
.cluster.inlineManifests	KubeInlineManifestConfig
.cluster.extraManifests, .cluster.extraManifestHeaders	KubeExternalManifestConfig
.cluster.allowSchedulingOnControlPlanes, .machine.nodeLabels, .machine.nodeAnnotations, .machine.nodeTaints, .machine.kubelet.nodeIP, .machine.kubelet.registerWithFQDN, .machine.kubelet.skipNodeRegistration	KubeNodeConfig
.machine.kubelet (remaining fields)	KubeletConfig, KubeCredentialProviderConfig (for .machine.kubelet.credentialProviderConfig)
.machine.pods	KubeStaticPodConfig
.machine.features.kubePrism	KubePrismConfig (remove the document to disable KubePrism)
.machine.features.kubernetesTalosAPIAccess	KubeTalosAPIAccessConfig
.cluster.discovery	DiscoveryServiceConfig
.cluster.id, .cluster.secret	DiscoveryIdentityConfig

Other Kubernetes configuration changes:

• DiscoveryServiceConfig is a multi-document kind, so multiple discovery service endpoints can be configured now.
• The cluster ID encoding in the generated secrets bundle changed from base64.URLEncoding to base64.StdEncoding to align with the rest of Talos.
• The default control plane NoSchedule taint and the control plane node label are now listed explicitly in KubeNodeConfig.
• KubeNetworkConfig adds nodeCIDRMaskSizeIPv4 (default 24) and nodeCIDRMaskSizeIPv6 (default 64) to control the per-node pod CIDR mask size; pod and service subnet sizes are validated against them.
• .machine.kubelet.extraMounts has no equivalent in the new documents, and .machine.kubelet.disableManifestsDirectory is locked to true there.

Kernel and udev

If both the deprecated v1alpha1 field and the new document are used, the new document takes precedence for conflicting settings.

Deprecated v1alpha1 field	Replacement document
.machine.sysctls	SysctlConfig
.machine.sysfs	SysfsConfig
.machine.kernel.modules	KernelModuleConfig
.machine.udev.rules	UdevRulesConfig

Install, files and container runtime

• .machine.install is deprecated in favor of the new UnattendedInstallConfig document, which carries the installer image and a provisioning section with a CEL volumeSelector to match the install disk. talosctl gen config and talosctl cluster create generate this document by default now; .machine.install is still used for older version contracts.
• .machine.files is deprecated in favor of dedicated documents: EtcFileConfig for user-owned files under /etc (the document name is the path relative to /etc), and CRICustomizationConfig for CRI configuration. Talos-managed paths (resolv.conf, hosts, machine-id, CRI and Kubernetes configuration, trust bundles and identity files) are rejected by EtcFileConfig.
• New CRICustomizationConfig document carries a TOML fragment merged into the CRI containerd configuration; fragments are merged in lexicographical order by document name. The legacy /etc/cri/conf.d/20-customization.part machine file is still supported and is exposed under the reserved name customization. CRI configuration changes no longer require a reboot.
• .machine.baseRuntimeSpecOverrides is deprecated in favor of the new CRIBaseRuntimeSpecConfig document.
• .machine.features.imageCache is deprecated in favor of the new ImageCacheConfig document, where localEnabled becomes local.enabled. The deprecated field is still honored, so upgraded clusters keep their image cache enabled, but the field and the document cannot both be set.
• New SecurityProfileConfig document with the workloadIsolation option (enabled by default for new clusters generated with Talos 1.14, absent — and therefore disabled — on upgraded clusters).
• OOMConfig has a new strictCgroupClassOrdering option (defaults to true) to enforce QoS class ordering when picking an OOM victim.

Network

• .machine.features.hostDNS is deprecated: host DNS is now configured with the hostDNS section of the ResolverConfig document.
• ResolverConfig nameservers accept a protocol (DNS over TLS and DNS over HTTPS are supported now) and a tlsServerName setting, configured per nameserver.
• TimeSyncConfig has a new useNTS field to enable Network Time Security for custom time servers (NTS is enabled by default for the default time.cloudflare.com server).
• DHCPv4Config has a new ignoreRoutes option to ignore the default gateway and classless static routes offered by the DHCP server (the connected route for the leased address is still configured). DHCPv4 search domains are now applied to the resolver configuration.
• New VethConfig document to configure virtual Ethernet pairs: both endpoints are created in the host network namespace and support the common link settings, addresses, routes and multicast configuration.
• New HTTPProbeConfig document to configure HTTP-based connectivity checks (probes).
• New BGPInstanceConfig document to run native BGP speakers on the host (local ASN, router-id, optional Linux VRF, advertised interfaces and neighbors), removing the need for the FRR system extension in fabric-facing setups. Routes learned by one instance can be selectively imported into another with importRoutes, and installRoutes: false keeps learned routes out of the Linux routing table.

Volumes and storage

• New FilesystemTrimConfig document to enable periodic filesystem trim (the fstrim equivalent) with a global interval. The default machine configuration generated by Talos 1.14 includes it with a one week interval; upgraded clusters don’t have the document, so trimming stays disabled until it is added. The interval can be overridden or disabled per volume with the new trim section of VolumeConfig, UserVolumeConfig, ExistingVolumeConfig and ExternalVolumeConfig.
• New FilesystemScrubConfig document to enable periodic online filesystem scrub (currently XFS only, via xfs_scrub) with a global interval. Scrubbing is disabled by default, and the generated machine configuration does not include this document, so it stays disabled until it is added. The interval can be overridden or disabled per volume with the new scrub section of VolumeConfig, UserVolumeConfig and ExistingVolumeConfig.
• Volume encryption has a new allowDiscards option (disabled by default) to pass TRIM/discard requests through to the underlying device.
• VolumeConfig and UserVolumeConfig support a new filesystem.xfs.minAllocationGroupSize setting (defaults to 64 GiB, set to zero to restore stock mkfs.xfs behavior); it only applies to volumes formatted by Talos 1.14 or later.
• The ETCD, CRI, KUBELET and LOG system volumes can be placed on dedicated partitions with a VolumeConfig document with provisioning set (optionally encrypted); by default they remain directories under the EPHEMERAL volume. Secure dedicated ETCD and LOG volumes are mounted with noexec; EPHEMERAL, CRI and KUBELET remain executable.
• btrfs can be used as a filesystem type for user and existing volumes (requires the btrfs system extension).
• New LVMVolumeGroupConfig and LVMLogicalVolumeConfig documents to declaratively create and grow LVM volume groups and logical volumes.
• New RAIDArrayConfig document to declaratively create and grow Linux MD (software RAID) arrays; raid1 arrays with metadata: "1.0" can be used as the install disk.

Upgrade sequence

When a Talos node receives the upgrade command, it cordons itself in Kubernetes, to avoid receiving any new workload. It then starts to drain its existing workload.

NOTE: If any of your workloads are sensitive to being shut down ungracefully, be sure to use the lifecycle.preStop Pod spec.

Once all of the workload Pods are drained, Talos will start shutting down its internal processes.

Once all the processes are stopped and the services are shut down, the filesystems will be unmounted. This allows Talos to produce a very clean upgrade, as close as possible to a pristine system. We verify the disk and then perform the actual image upgrade. We set the bootloader to boot once with the new kernel and OS image, then we reboot.

After the node comes back up and Talos verifies itself, it will make the bootloader change permanent, rejoin the cluster, and finally uncordon itself to receive new workloads.

FAQs

Q. What happens if an upgrade fails?

A. Talos Linux attempts to safely handle upgrade failures.

The most common failure is an invalid installer image reference. In this case, Talos will fail to download the upgraded image and will abort the upgrade.

Sometimes, Talos is unable to successfully kill off all of the disk access points, in which case it cannot safely unmount all filesystems to effect the upgrade. In this case, it will abort the upgrade and reboot.

It is possible (especially with test builds) that the upgraded Talos system will fail to start. In this case, the node will be rebooted, and the bootloader will automatically use the previous Talos kernel and image, thus effectively rolling back the upgrade.

Lastly, it is possible that Talos itself will upgrade successfully, start up, and rejoin the cluster but your workload will fail to run on it, for whatever reason. This is when you would use the talosctl rollback command to revert back to the previous Talos version.

Q. Can upgrades be scheduled?

A. Because the upgrade sequence is API-driven, you can easily tie it in to your own business logic to schedule and coordinate your upgrades.

Q. Can the upgrade process be observed?

A. Yes, using the talosctl dmesg -f command. You can also use talosctl upgrade --wait, and optionally talosctl upgrade --wait --debug to observe kernel logs

Q. Are worker node upgrades handled differently from control plane node upgrades?

A. Short answer: no.

Long answer: Both node types follow the same set procedure. From the user’s standpoint, however, the processes are identical. However, since control plane nodes run additional services, such as etcd, there are some extra steps and checks performed on them. For instance, Talos will refuse to upgrade a control plane node if that upgrade would cause a loss of quorum for etcd. If multiple control plane nodes are asked to upgrade at the same time, Talos will protect the Kubernetes cluster by ensuring only one control plane node actively upgrades at any time, via checking etcd quorum.

Q. Can I break my cluster by upgrading everything at once?

A. Possibly - it’s not recommended.

Nothing prevents the user from sending near-simultaneous upgrades to each node of the cluster - and while Talos Linux and Kubernetes can generally deal with this situation, other components of the cluster may not be able to recover from more than one node rebooting at a time. (e.g. any software that maintains a quorum or state across nodes, such as Rook/Ceph)

Q. Which version of talosctl should I use to update a cluster?

A. We recommend using the version that matches the current running version of the cluster.