Create Etcd Backups

Configure S3 backup storage, create manual backups, and enable automatic etcd backups for Omni-managed clusters.

Etcd is the key-value store that holds all Kubernetes cluster state. If etcd becomes corrupted or unavailable, your cluster will stop functioning. Omni supports creating etcd backups so that you can restore a cluster to a known good state if something goes wrong.

Omni supports two backup storage backends: local and s3.

The backend is selected during Omni initialization.

This guide covers how to configure S3 backup storage, create a manual backup, and enable automatic backups.

To restore a cluster from a backup, see Restore Etcd of a Cluster Managed by Cluster Templates.

Check backup status

Before configuring backups, check the current state of the backup subsystem:

omnictl get etcdbackupoverallstatus

On a freshly created Omni instance, the output will look similar to this:

NAMESPACE   TYPE                      ID                          VERSION   CONFIGURATION NAME   CONFIGURATION ERROR   LAST BACKUP STATUS   LAST BACKUP ERROR   LAST BACKUP TIME   CONFIGURATION ATTEMPT
ephemeral   EtcdBackupOverallStatus   etcdbackup-overall-status   1         s3                   not initialized

The key fields to pay attention to are:

Field	Description
CONFIGURATION NAME	The backup backend currently configured (s3 or local).
CONFIGURATION ERROR	Any error with the current backup configuration. not initialized means the backend has been selected but not yet configured.
LAST BACKUP STATUS	The result of the most recent backup attempt.
LAST BACKUP ERROR	Any error from the most recent backup attempt.
LAST BACKUP TIME	When the most recent backup completed.
CONFIGURATION ATTEMPT	The number of times Omni has attempted to apply the backup configuration.

In the output above, CONFIGURATION NAME is s3 and CONFIGURATION ERROR shows not initialized, which means the S3 backend has been selected but the credentials have not yet been configured. The remaining fields are empty because no backups have been created yet.

Configure S3 backup storage

Omni stores etcd backups in an S3-compatible bucket. Before creating any backups, you need to provide Omni with the credentials and configuration it needs to access your bucket. This section covers compatibility considerations for non-AWS providers and how to apply the S3 configuration.

S3-compatible providers

Omni uses the AWS S3 API for storing etcd backups. Non-AWS providers that advertise S3 compatibility may work when configured with a custom endpoint, but Omni does not maintain a compatibility list for third-party providers.

If you use a non-AWS S3 provider:

• Test backup and restore workflows before relying on them in production.
• Some providers may require additional configuration, such as a region value or provider-specific credentials.
• Ensure the endpoint and credentials are valid and accessible from Omni.

During configuration, Omni validates access by attempting to list objects in the bucket. If validation fails, the configuration is not applied.

Required S3 permissions

The credentials (or IAM role) Omni uses need the following permissions on the bucket:

Permission	Why Omni needs it
s3:ListBucket	Validating the configuration, and listing existing backups.
s3:GetObject	Reading backups, both when listing them and when restoring a cluster.
s3:PutObject	Uploading backups. This also covers the multipart upload operations that are used for larger snapshots.
s3:AbortMultipartUpload	Cleaning up a multipart upload if a backup upload fails partway through.

Omni never deletes objects from the bucket, so no delete permission is required.

Because Omni never deletes old backups, the bucket grows without bound as long as automatic or manual backups keep running. Configure a bucket lifecycle rule (or your provider’s equivalent) to expire objects after a retention period that fits your recovery needs, so you don’t have to clean up the bucket manually.

An example AWS IAM policy that grants the minimum required access to a bucket named mybucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OmniEtcdBackupsListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::mybucket"
    },
    {
      "Sid": "OmniEtcdBackupsObjectAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::mybucket/*"
    }
  ]
}

Apply the S3 configuration

To configure S3 as the backup backend, create an EtcdBackupS3Configs.omni.sidero.dev resource.

The following example configures Omni to use a MinIO S3 instance, edit this example to fit your S3 instance:

cat <<EOF > etcd-backup-s3-conf.yaml
metadata:
  namespace: default
  type: EtcdBackupS3Configs.omni.sidero.dev
  id: etcd-backup-s3-conf
spec:
  bucket: mybucket
  region: us-east-1
  endpoint: http://127.0.0.1:9000
  accesskeyid: access
  secretaccesskey: secret123
  sessiontoken: ""
EOF

The fields in the spec are:

Field	Required	Description
bucket	Yes	The name of the S3 bucket to store backups in.
region	No	The S3 bucket region. Defaults to the AWS default region if not set.
endpoint	No	The S3 endpoint URL. Defaults to the AWS S3 endpoint if not set. Required for non-AWS providers.
accesskeyid	No	The access key ID for the S3 bucket. If not set, Omni assumes it is running on an EC2 instance with an IAM role that has access to the bucket.
secretaccesskey	No	The secret access key for the S3 bucket.
sessiontoken	No	A session token for temporary credentials, if required.

Apply the S3 configuration:

omnictl apply -f etcd-backup-s3-conf.yaml

Omni validates the credentials during resource creation by attempting to list objects in the bucket. If validation fails, the resource will not be created and an error will be returned.

To confirm the configuration was applied successfully, check the backup status again:

omnictl get etcdbackupoverallstatus

The CONFIGURATION ERROR field should now be empty:

NAMESPACE   TYPE                      ID                          VERSION   CONFIGURATION NAME   CONFIGURATION ERROR   LAST BACKUP STATUS   LAST BACKUP ERROR   LAST BACKUP TIME   CONFIGURATION ATTEMPT
ephemeral   EtcdBackupOverallStatus   etcdbackup-overall-status   2         s3

Create a manual backup

To trigger a backup immediately, create and apply an EtcdManualBackups.omni.sidero.dev resource.

The <unix-timestamp> value must be no more than one minute in the future or past. The nanos field must always be 0.

1. Run the following command, replacing <cluster-name> with the name of your cluster:

cat <<EOF > etcd-manual-backup.yaml
metadata:
  namespace: ephemeral
  type: EtcdManualBackups.omni.sidero.dev
  id: <cluster-name>
spec:
  backupat:
    seconds: $(date +%s)
    nanos: 0
EOF

2. Apply the resource:

omnictl apply -f etcd-manual-backup.yaml

3. After a few seconds, check the per-cluster backup status:

omnictl get etcdbackupstatus -o yaml

The output will look similar to this:

metadata:
  namespace: ephemeral
  type: EtcdBackupStatuses.omni.sidero.dev
  id: <cluster-name>
  version: 1
  owner: EtcdBackupController
  phase: running
spec:
  status: 1
  error: ""
  lastbackuptime:
    seconds: 1702166400
    nanos: 985220192
  lastbackupattempt:
    seconds: 1702166400
    nanos: 985220192

4. To check the overall backup subsystem status:

omnictl get etcdbackupoverallstatus -o yaml

Enable automatic backups

Omni supports scheduled automatic backups.

You can configure the backup interval using either cluster templates or by editing the cluster resource directly.

Cluster templates

To enable automatic etcd backups using a cluster template:

1. Add a features.backupConfiguration block to your cluster template:

cat <<EOF > etcd-backup.yaml
kind: Cluster
name: talos-default
kubernetes:
  version: v1.28.2
talos:
  version: v1.5.5
# add the backupConfiguration block to your cluster templates
features:
  backupConfiguration:
    interval: 1h
---
kind: ControlPlane
machines:
  - <controlplane-machine-uuid> # control plane machine uuid
---
kind: Workers
machines:
  - <worker-machine-one-uuid> # worker machine uuid
  - <worker-machine-two-uuid> # worker machine uuid
EOF

Replace <controlplane-machine-uuid>, <worker-machine-one-uuid>, and <worker-machine-two-uuid> with the UUIDs of your control plane and worker machines.

For more information on configuring cluster features with cluster templates, see the Cluster Template reference documentation.

2. Before applying, check whether automatic backups are already enabled on the cluster:

omnictl cluster template -f etcd-backup.yaml diff

If automatic backups are not yet enabled, the output will show the backup configuration being added:

--- Clusters.omni.sidero.dev(default/talos-default)
+++ Clusters.omni.sidero.dev(default/talos-default)
@@ -19,4 +19,7 @@
   features:
     enableworkloadproxy: false
     diskencryption: false
-  backupconfiguration: null
+  backupconfiguration:
+    interval:
+      seconds: 3600
+      nanos: 0

If automatic backups are already enabled, the diff will show an existing backupconfiguration block with a non-null value.

In that case, you can update the interval value in your template and re-run the diff to confirm the change before applying, or skip to step 4 to verify the existing backup schedule is working as expected.

3. Apply the template:

omnictl cluster template -f etcd-backup.yaml sync

If no backups exist for the cluster yet, Omni will create one immediately rather than waiting for the first scheduled interval.

4. Verify that the backup was created by running the following command, replacing <cluster-name> with the name of your cluster:

omnictl get etcdbackup --selector omni.sidero.dev/cluster=<cluster-name>

This command will output something similar to this:

NAMESPACE   TYPE         ID                         VERSION     CREATED AT
external    EtcdBackup   talos-default-1702166400   undefined   {"nanos":0,"seconds":1702166400}

Cluster resource

If your cluster is not managed by cluster templates, you can enable automatic backups by editing the cluster resource directly.

To enable automatic etcd backups by editing the cluster resource directly:

1. Retrieve the current cluster resource, replace the <cluster-name> placeholder with the name of your cluster:

omnictl get cluster <cluster-name> -o yaml > cluster-backup.yaml

2. In the cluster-backup.yaml file, add the backupconfiguration block while preserving all existing fields:

# cluster-backup.yaml
metadata:
  namespace: default
  type: Clusters.omni.sidero.dev
  id: talos-default
  finalizers:
    - KubernetesUpgradeStatusController
    - TalosUpgradeStatusController
    - SecretsController
    - ClusterController
spec:
  installimage: ""
  kubernetesversion: 1.28.2
  talosversion: 1.5.5
  features:
    enableworkloadproxy: false
    diskencryption: false
  # add the backupconfiguration block
  backupconfiguration:
    interval:
      seconds: 3600
      nanos: 0

3. Apply the updated file:

omnictl apply -f cluster-backup.yaml

Next steps

To restore a cluster from an etcd backup, see Restore Etcd of a Cluster Managed by Cluster Templates.