Cluster Templates

Reference guide to the cluster template schema, document types, and supported fields in Omni.

Omni parses, validates, and converts cluster templates into Omni resources. It then creates or updates these resources via the Omni API. Omni guarantees backward compatibility for cluster templates, so you can use the same template with any future version of Omni.

Store all referenced files in machine configuration patches relative to the current working directory.

Structure

A cluster template is a YAML file consisting of multiple documents. Each document contains a kind field that specifies the document type. Some documents also include a name field that specifies the document ID.

kind: Cluster
name: example
labels:
  my-label: my-value
kubernetes:
  version: v1.26.0
talos:
  version: v1.3.2
features:
  diskEncryption: true
patches:
  - name: kubespan-enabled
    inline:
      machine:
        network:
          kubespan:
            enabled: true
systemExtensions:
  - siderolabs/hello-world-service
kernelArgs:
  - talos.dashboard.disabled=1
---
kind: ControlPlane
machines:
  - 27c16241-96bf-4f17-9579-ea3a6c4a3ca8
  - 4bd92fba-998d-4ef3-ab43-638b806dd3fe
  - 8fdb574a-a252-4d7d-94f0-5cdea73e140a
---
kind: Workers
machines:
  - b885f565-b64f-4c7a-a1ac-d2c8c2781373
  - a54f21dc-6e48-4fc1-96aa-3d7be5e2612b
---
kind: Workers
name: xlarge
machines:
  - 1f721dee-6dbb-4e71-9832-226d73da3841
systemExtensions:
  - siderolabs/hello-world-service
---
kind: Machine
name: 27c16241-96bf-4f17-9579-ea3a6c4a3ca8
---
kind: Machine
name: 4bd92fba-998d-4ef3-ab43-638b806dd3fe
install:
  disk: /dev/vda
---
kind: Machine
name: 8fdb574a-a252-4d7d-94f0-5cdea73e140a
install:
  disk: /dev/vda
---
kind: Machine
name: b885f565-b64f-4c7a-a1ac-d2c8c2781373
install:
  disk: /dev/vda
systemExtensions:
  - siderolabs/hello-world-service
---
kind: Machine
name: a54f21dc-6e48-4fc1-96aa-3d7be5e2612b
locked: true
install:
  disk: /dev/vda
---
kind: Machine
name: 1f721dee-6dbb-4e71-9832-226d73da3841
install:
  disk: /dev/vda
kernelArgs:
  - net.ifnames=0
  - talos.dashboard.disabled=1

Each cluster template must have exactly one document of kind: Cluster, kind: ControlPlane, and any number of kind: Workers with different names. Either a ControlPlane or Workers document must reference every Machine document.

Document types

The following sections describe the specific document kinds available in a cluster template and their configurations.

Cluster

The Cluster document specifies the cluster configuration and labels. It defines the cluster name and base component versions.

kind: Cluster
name: example
labels:
  my-label: my-value
annotations:
  my-annotation: my-value
kubernetes:
  version: v1.26.1
talos:
  version: v1.3.3
features:
  enableWorkloadProxy: true
  diskEncryption: true
  backupConfiguration:
    interval: 1h
patches:
  - file: patches/example-patch.yaml
systemExtensions:
  - siderolabs/hello-world-service
kernelArgs:
  - talos.dashboard.disabled=1

Cluster templates that use machineClass do not support kernelArgs. Defining them at the cluster or machine set level will cause validation errors.

Field	Type	Description
kind	string	Cluster
name	string	Cluster name: only letters, digits and - and _ are allowed. The cluster name is used as a key by all other documents, so if the cluster name changes, Omni creates a new cluster.
labels	map[string]string	Labels to be applied to the cluster.
annotations	map[string]string	Annotations to be applied to the cluster.
kubernetes.version	string	Kubernetes version to use, vA.B.C.
kubernetes.manifests	array[Manifest]	List of Kubernetes manifests to deploy and sync to the cluster. Each entry is applied independently. See Sync Kubernetes Manifests.
kubernetes.healthchecks	array[Healthcheck]	List of advanced healthchecks that gate Talos Linux upgrades. See Gate Talos Upgrades with Healthchecks.
talos.version	string	Talos version to use, vA.B.C.
features.enableWorkloadProxy	boolean	Whether to enable the workload proxy feature. Defaults to false.
features.useEmbeddedDiscoveryService	boolean	Whether to use the embedded discovery service that runs inside the Omni instance instead of the public one (discovery.talos.dev). Defaults to false. It is only valid if the Omni instance has the feature enabled.
features.diskEncryption	boolean	Whether to enable disk encryption. Defaults to false.
features.backupConfiguration.interval	string	Cluster etcd backup interval. Must be a valid Go duration. Zero 0 disables automatic backups.
patches	array[Patch]	List of patches to apply to the cluster.
systemExtensions	array[string]	The list of system extensions to be installed on every machine in the cluster.
kernelArgs	array[string]	The list of kernel args to be set on each machine in this cluster.

ControlPlane

The ControlPlane document specifies the control plane configuration. It defines the number of control plane nodes and the list of machines to use. Because control plane machines run an etcd cluster, use a number of machines that can achieve a stable quorum (e.g., 1, 3, 5).

Changing the set of machines in the control plane triggers a rolling scale-up or scale-down. Configure the control plane with at least a single machine. For high availability, SideroLabs recommends using at least 3 machines.

kind: ControlPlane
labels:
  my-label: my-value
annotations:
  my-annotation: my-value
machines:
  - 27c16241-96bf-4f17-9579-ea3a6c4a3ca8
  - 4bd92fba-998d-4ef3-ab43-638b806dd3fe
  - 8fdb574a-a252-4d7d-94f0-5cdea73e140a
patches:
  - file: patches/example-controlplane-patch.yaml
systemExtensions:
  - siderolabs/hello-world-service

Field	Type	Description
kind	string	ControlPlane
labels	map[string]string	Labels to be applied to the control plane machine set.
annotations	map[string]string	Annotations to be applied to the control plane machine set.
machines	array[string]	List of machine IDs to use for control plane nodes (mutually exclusive with machineClass).
bootstrapSpec.clusterUUID	string	UUID of the cluster to restore from (required when restoring a cluster).
bootstrapSpec.snapshot	string	Snapshot file name to restore from (required when restoring a cluster).
patches	array[Patch]	List of patches to apply to the machine set.
machineClass	MachineClass	Machine Class configuration (mutually exclusive with machines).
systemExtensions	array[string]	The list of system extensions to be installed on every machine in the machine set.
kernelArgs	array[string]	The list of kernel args to be set on each machine in this machine set.

Workers

The Workers document specifies the worker configuration. It defines the number of worker nodes and the list of machines to use.

kind: Workers
name: workers
labels:
  my-label: my-value
annotations:
    my-annotation: my-value
machines:
  - b885f565-b64f-4c7a-a1ac-d2c8c2781373
updateStrategy:
  rolling:
    maxParallelism: 3
upgradeStrategy:
  rolling:
    maxParallelism: 3
deleteStrategy:
  type: Rolling
  rolling:
    maxParallelism: 5
patches:
  - file: patches/example-workers-patch.yaml
systemExtensions:
  - siderolabs/hello-world-service

Field	Type	Description
kind	string	Workers
name	string	Worker machine set name: only letters, digits and - and _ are allowed. Defaults to workers when omitted. Must be unique and not be control-planes.
labels	map[string]string	Labels to be applied to the worker machine set.
annotations	map[string]string	Annotations to be applied to the worker machine set.
machines	array[string]	List of machine IDs to use as worker nodes in the machine set (mutually exclusive with machineClass).
patches	array[Patch]	List of patches to apply to the machine set.
machineClass	MachineClass	Machine Class configuration (mutually exclusive with machines).
updateStrategy	UpdateStrategy	Controls how configuration changes are rolled out across machines in the set (e.g. patches and labels). Defaults to type: Rolling with maxParallelism: 1.
upgradeStrategy	UpdateStrategy	Controls how version upgrades, extensions and kernel args changes are rolled out across machines in the set. Defaults to type: Rolling with maxParallelism: 1.
deleteStrategy	UpdateStrategy	Controls how machines are removed when the machine set is scaled down. Defaults to type: Unset.
systemExtensions	array[string]	The list of system extensions to be installed on every machine in the machine set.
kernelArgs	array[string]	The list of kernel args to be set on each machine in this machine set.

MachineClass

The MachineClass section of ControlPlane or Workers defines the rule for picking the machines in the machine set.

kind: Workers
name: workers
machineClass:
  name: worker-class
  size: 2

Field	Type	Description
name	string	Name of the machine class to use.
size	number | string	Number of machines to pick from the matching machine class, or a keyword (unlimited, infinity, ∞).

The size field supports keyword unlimited|infinity, which makes the machine set pick all available machines from the specified machine class.

UpdateStrategy

The UpdateStrategy section of Workers defines the update, delete, or upgrade strategy for the machine set.

kind: Workers
name: workers
updateStrategy:
  rolling:
    maxParallelism: 3
upgradeStrategy:
  rolling:
    maxParallelism: 3
deleteStrategy:
  type: Rolling
  rolling:
    maxParallelism: 5

Field	Type	Description
type	string	Strategy type. Can be Rolling or Unset. Defaults to Rolling for updateStrategy and upgradeStrategy, and Unset for the deleteStrategy. When Unset, all updates and/or deletes are applied at once.
rolling.maxParallelism	number	Maximum number of machines to update and/or delete in parallel. Only used when the type is Rolling. Defaults to 1.

Machine

The Machine document specifies the install disk and machine-specific configuration patches. They are optional, but either a ControlPlane or Workers document must reference every Machine document.

kind: Machine
name: 27c16241-96bf-4f17-9579-ea3a6c4a3ca8
labels:
  my-label: my-value
annotations:
  my-annotation: my-value
locked: false
install:
  disk: /dev/vda
patches:
  - file: patches/example-machine-patch.yaml
systemExtensions:
  - siderolabs/hello-world-service

Field	Type	Description
kind	string	Machine
name	string	Machine ID.
labels	map[string]string	Labels to be applied to the machine set node.
annotations	map[string]string	Annotations to be applied to the machine set node.
locked	boolean	Whether the machine should be marked as locked. Can be true only if the machine is used as a worker.
install.disk	string	Disk to install Talos on. Matters only for Talos running from ISO or iPXE.
patches	array[Patch]	List of patches to apply to the machine.
systemExtensions	array[string]	The list of system extensions to be installed on the machine.
kernelArgs	array[string]	The list of kernel args to be set on this machine.

When Talos is not installed and you do not specify the install disk, Omni tries to pick the install disk automatically. It finds the smallest disk larger than 5GB.

Common fields

The following configuration fields are shared across multiple document types.

patches

The patches field lists machine configuration patches to apply to a cluster, a machine set, or an individual machine. Config patches modify the configuration before it is applied to each machine in the cluster. Changing configuration patches modifies the machine configuration, which gets automatically applied to the machine.

patches:
  - file: patches/example-patch.yaml
  - name: kubespan-enabled
    inline:
      machine:
        network:
          kubespan:
            enabled: true
  - idOverride: 950-set-env-vars
    labels:
      my-label: my-value
    annotations:
      my-annotation: my-value
    inline:
      machine:
        env:
          MY_ENV_VAR: my-value

Field	Type	Description
file	string	Path to the patch file. Path is relative to the current working directory when executing omnictl. File should contain Talos machine configuration strategic patch.
name	string	Name of the patch. Required for inline patches when idOverride is not set, optional for file patches (default name will be based on the file path).
idOverride	string	Override the config patch ID, so it won’t be generated from the name or file.
labels	map[string]string	Labels to be applied to the config patch.
annotations	map[string]string	Annotations to be applied to the config patch.
inline	object	Inline patch containing Talos machine configuration strategic patch.

A configuration patch may be either inline or file based. Inline patches are useful for small changes, while file-based patches are useful for more complex changes or changes shared across multiple clusters.

kubernetes.manifests

The manifests field under kubernetes in the Cluster document lets you define Kubernetes workloads to deploy directly from the cluster template. Omni stores the manifest data and synchronizes it to the cluster once the Kubernetes API is available and the cluster is healthy.

Field	Type	Description
name	string	Unique name for this manifest entry.
mode	string	Sync mode. one-time applies the manifest once after bootstrapping. full continuously reconciles the manifest state.
inline	array[object]	List of inline Kubernetes resource definitions.
file	string	Path to an external YAML file, relative to the working directory when running omnictl.

For full usage details and examples, see Sync Kubernetes Manifests.

kubernetes.healthchecks

The healthchecks field under kubernetes in the Cluster document defines advanced healthchecks that gate Talos Linux upgrades. Each entry is a Kubernetes Job that Omni runs in the workload cluster before upgrading the next node. While any healthcheck is failing or still running, Omni holds the upgrade.

kind: Cluster
name: example
kubernetes:
  version: v1.35.0
  healthchecks:
    - name: nodes-ready
      interval: 30s
      job: |
        apiVersion: batch/v1
        kind: Job
        spec:
          backoffLimit: 0
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: check
                  image: bitnami/kubectl
                  command: ["kubectl", "wait", "--for=condition=Ready", "nodes", "--all", "--timeout=10s"]

Field	Type	Description
name	string	Unique name for the healthcheck, used to derive the Job name. Required unless idOverride is set.
idOverride	string	Overrides the generated resource ID. When set, name is not required. You normally won’t need this; it’s for when you want a stable resource ID independent of name
job	string	Inline Kubernetes Job manifest. Mutually exclusive with file.
file	string	Path to a file containing the Job manifest, relative to the working directory when running omnictl. Mutually exclusive with job.
interval	string	How often Omni re-checks while it is holding an upgrade. Must be a valid Go duration. Defaults to 30s, minimum 5s.
labels	map[string]string	Labels to apply to the healthcheck resource.
annotations	map[string]string	Annotations to apply to the healthcheck resource.

For full usage details and examples, see Gate Talos Upgrades with Healthchecks.

kernelArgs

You can define kernel args at the cluster, machine set, or individual machine level. Management of kernel args by cluster templates is opt-in: the templates manage them ONLY IF you specify them and they are not null. Otherwise, their lifecycle remains outside of cluster template management.

When defined at multiple levels, the lowest/most specific level takes precedence. For example, a kernel arg defined at the machine level overrides the same kernel arg defined at the cluster or machine set level.

These args are only supported for “static” machines. If a machine is created from a machine class, kernel args management is not supported. This means setting them in a ControlPlane or Workers document that uses machineClass, or in a Cluster document that contains machine sets based on machineClass, will result in an error.

kernelArgs:
  - talos.dashboard.disabled=1
  - net.ifnames=0