Skip to main content
This guide walks through deploying a 3-node etcd cluster secured with mutual TLS (mTLS). This cluster can be used as an external datastore for Omni on-prem, replacing Omni’s embedded etcd for highly available deployments. Once complete, you will have:
  • A 3-node etcd cluster with mTLS enabled for both client and peer communication
  • An etcd endpoint ready to connect to Omni
A 3-node cluster requires a majority quorum to function. It can tolerate the loss of one node. For a production environment, run each node on a separate physical or virtual machine.

​
When to use external etcd

Omni ships with embedded etcd and works well for most single-instance deployments. Use external etcd when you need:
  • High availability — survive the loss of an Omni host without losing datastore state
  • Operational separation — back up, snapshot, and manage the datastore independently of Omni
  • Shared datastore — run multiple Omni instances against the same etcd cluster
See Run Omni Options for more context on when each mode is appropriate.

​
Prerequisites

You will need 3 Linux hosts, each with:
  • At least 2 vCPUs and 2 GB RAM
  • Docker installed
  • The following ports open between all 3 nodes:
PortProtocolPurpose
2379TCPetcd client API (restrict to etcd node IPs and Omni host IP only)
2380TCPetcd peer communication (restrict to etcd node IPs only)

​
Step 1: Install cfssl (Node 1 only)

cfssl generates the CA and all node certificates. All certificates are created on Node 1 and then distributed to the other nodes in Step 4, so you only need cfssl on Node 1. 
CFSSL_VERSION=$(curl -sI https://github.com/cloudflare/cfssl/releases/latest \
  | grep -i location | awk -F '/' '{print $NF}' | tr -d '\r')

curl -L -o cfssl \
  https://github.com/cloudflare/cfssl/releases/download/${CFSSL_VERSION}/cfssl_${CFSSL_VERSION#v}_linux_amd64
curl -L -o cfssljson \
  https://github.com/cloudflare/cfssl/releases/download/${CFSSL_VERSION}/cfssljson_${CFSSL_VERSION#v}_linux_amd64

chmod +x cfssl cfssljson
sudo mv cfssl cfssljson /usr/local/bin/

​
Step 2: Set environment variables (Node 1 only)

These variables define the IP addresses of all three etcd nodes and the Omni host. They are used when generating certificates and when starting the etcd cluster. Replace the values below with your actual IPs. 
export ETCD1_IP=<etcd-node-1-ip>
export ETCD2_IP=<etcd-node-2-ip>
export ETCD3_IP=<etcd-node-3-ip>
export OMNI_IP=<omni-ip>

​
Step 3: Generate certificates (Node 1 only)

Each etcd node needs its own server certificate with its IP embedded as a Subject Alternative Name (SAN). This allows etcd to verify the identity of peer nodes and clients over mTLS. All certificates are signed by a shared root CA so every node in the cluster trusts the others. Omni also gets a dedicated client certificate, which it uses to authenticate to the cluster.

​
3.1: Create the root CA

This generates the root CA that all etcd nodes and Omni will use as their trust anchor. It produces ca-key.pem (private key), ca.pem (public cert), and ca.csr (signing request). 
cat <<EOF > ca-csr.json
{
  "CN": "etcd Root CA",
  "key": { "algo": "rsa", "size": 4096 },
  "names": [{ "C": "US", "O": "Internal Infrastructure", "OU": "Security" }]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

​
3.2: Create the signing configuration

The server profile is used for etcd node certificates and includes both server auth and client auth usages, since etcd nodes authenticate to each other as both clients and servers during peer communication. The client profile is used for the Omni client certificate. 
cat <<EOF > ca-config.json
{
  "signing": {
    "default": { "expiry": "8760h" },
    "profiles": {
      "server": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      },
      "client": {
        "usages": ["signing", "key encipherment", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

​
3.3: Generate a server certificate for each node

Each node gets its own certificate with its IP address embedded as a SAN. etcd uses these IPs to verify the identity of incoming peer and client connections. Run all three of the following commands on Node 1.
The labels below indicate which node each certificate is for, not where the command runs.
Certificate for etcd-node-1: 
cat <<EOF > etcd1-csr.json
{
  "CN": "etcd-node-1",
  "hosts": ["${ETCD1_IP}", "127.0.0.1"],
  "key": { "algo": "rsa", "size": 4096 }
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=server etcd1-csr.json | cfssljson -bare etcd1
Certificate for etcd-node-2: 
cat <<EOF > etcd2-csr.json
{
  "CN": "etcd-node-2",
  "hosts": ["${ETCD2_IP}", "127.0.0.1"],
  "key": { "algo": "rsa", "size": 4096 }
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=server etcd2-csr.json | cfssljson -bare etcd2
Certificate for etcd-node-3: 
cat <<EOF > etcd3-csr.json
{
  "CN": "etcd-node-3",
  "hosts": ["${ETCD3_IP}", "127.0.0.1"],
  "key": { "algo": "rsa", "size": 4096 }
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=server etcd3-csr.json | cfssljson -bare etcd3

​
3.4: Generate the Omni client certificate

This certificate is mounted into the Omni container and used to authenticate to etcd over mTLS. The OMNI_IP is embedded as a SAN so etcd can verify that connections from Omni are coming from the expected host. 
cat <<EOF > client-csr.json
{
  "CN": "omni-client",
  "hosts": ["${OMNI_IP}"],
  "key": { "algo": "rsa", "size": 4096 },
  "names": [{ "C": "US", "O": "Omni" }]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=client client-csr.json | cfssljson -bare client

chmod 644 client*.pem ca.pem

​
Step 4: Distribute certificates

In Step 3, all certificates were generated on Node 1. In this section, you’ll distribute those certificates to the other etcd and Omni nodes. Each node needs three items: its own certificate and private key (to prove its identity), and the shared CA certificate (to verify other nodes). The Omni host needs the client certificate and the CA certificate to authenticate with the cluster. The commands below use SSH agent forwarding, so your private key is never copied to any server. On your local machine, add your SSH key to the agent, then reconnect to Node 1 with forwarding enabled: 
ssh-add ~/path/to/your-ssh-key
ssh -A <user>@<node-1-address>
Once connected to Node 1, run the following commands to distribute the certificates. If you started a new session, re-export the variables first: 
export ETCD2_IP=<etcd-node-2-ip>
export ETCD3_IP=<etcd-node-3-ip>
export OMNI_IP=<omni-ip>
Create the destination directory on the Omni host: 
ssh <user>@${OMNI_IP} "mkdir -p ~/etcd-certs"
Distribute the certificates to the other nodes: 
scp ca.pem etcd2.pem etcd2-key.pem <user>@${ETCD2_IP}:~/
scp ca.pem etcd3.pem etcd3-key.pem <user>@${ETCD3_IP}:~/
scp ca.pem client.pem client-key.pem <user>@${OMNI_IP}:~/etcd-certs/
Replace <user> with your SSH username. Common values are ubuntu on Ubuntu, ec2-user on Amazon Linux and RHEL, and root on some bare metal setups.
Keep ca-key.pem on Node 1 only. It is not needed on any other host and should not be distributed.

​
Step 5: Start etcd (all nodes)

Each node starts etcd with its own certificate and the shared --initial-cluster flag that tells etcd the addresses of all three members. All three nodes must be started before the cluster can form quorum and become healthy. On each node, set the shared cluster variables: 
export ETCD1_IP=<etcd-node-1-ip>
export ETCD2_IP=<etcd-node-2-ip>
export ETCD3_IP=<etcd-node-3-ip>
Create the data directory where etcd will persist its state: 
mkdir -p $HOME/etcd-data
sudo chmod 700 $HOME/etcd-data
Then run the appropriate command for each node: 
docker run -d \
  --name etcd \
  --restart=unless-stopped \
  -p 2379:2379 \
  -p 2380:2380 \
  -v $HOME/etcd-data:/etcd-data:Z \
  -v $HOME/etcd1.pem:/etcd/server.crt:ro \
  -v $HOME/etcd1-key.pem:/etcd/server.key:ro \
  -v $HOME/ca.pem:/etcd/ca.crt:ro \
  gcr.io/etcd-development/etcd:v3.5.17 \
  etcd \
    --name=etcd-node-1 \
    --data-dir=/etcd-data \
    --listen-client-urls=https://0.0.0.0:2379 \
    --advertise-client-urls=https://${ETCD1_IP}:2379 \
    --listen-peer-urls=https://0.0.0.0:2380 \
    --initial-advertise-peer-urls=https://${ETCD1_IP}:2380 \
    --initial-cluster="etcd-node-1=https://${ETCD1_IP}:2380,etcd-node-2=https://${ETCD2_IP}:2380,etcd-node-3=https://${ETCD3_IP}:2380" \
    --initial-cluster-state=new \
    --cert-file=/etcd/server.crt \
    --key-file=/etcd/server.key \
    --trusted-ca-file=/etcd/ca.crt \
    --client-cert-auth=true \
    --peer-cert-file=/etcd/server.crt \
    --peer-key-file=/etcd/server.key \
    --peer-trusted-ca-file=/etcd/ca.crt \
    --peer-client-cert-auth=true
The :Z volume flag on all mounts ensures compatibility with SELinux. It is safe to use on non-SELinux hosts — Docker silently ignores it.

​
Step 6: Verify the cluster

Once all three nodes are running, confirm they have formed a cluster. Run the following from Node 1. A healthy cluster will list all three members with started status and report each endpoint as healthy. Check that all three members have joined: 
docker exec etcd etcdctl \
  --endpoints=https://${ETCD1_IP}:2379,https://${ETCD2_IP}:2379,https://${ETCD3_IP}:2379 \
  --cacert=/etcd/ca.crt \
  --cert=/etcd/server.crt \
  --key=/etcd/server.key \
  member list
Expected output: 
<id>, started, etcd-node-1, https://10.0.0.1:2380, https://10.0.0.1:2379, false
<id>, started, etcd-node-2, https://10.0.0.2:2380, https://10.0.0.2:2379, false
<id>, started, etcd-node-3, https://10.0.0.3:2380, https://10.0.0.3:2379, false
Check that all endpoints are healthy and accepting writes: 
docker exec etcd etcdctl \
  --endpoints=https://${ETCD1_IP}:2379,https://${ETCD2_IP}:2379,https://${ETCD3_IP}:2379 \
  --cacert=/etcd/ca.crt \
  --cert=/etcd/server.crt \
  --key=/etcd/server.key \
  endpoint health
Expected output: 
https://10.0.0.1:2379 is healthy: successfully committed proposal: took = ...
https://10.0.0.2:2379 is healthy: successfully committed proposal: took = ...
https://10.0.0.3:2379 is healthy: successfully committed proposal: took = ...

​
Step 7: Connect Omni to the etcd cluster

Once the cluster is healthy, connect Omni to it by passing the following flags when starting Omni. Refer to Run Omni On-Prem and use the External etcd tab in Step 7. The --etcd-embedded=false flag disables Omni’s internal etcd instance. The --etcd-endpoints flag points Omni at all three cluster members so it can fail over automatically if one node goes down. 
--etcd-embedded=false \
--etcd-endpoints=https://${ETCD1_IP}:2379,https://${ETCD2_IP}:2379,https://${ETCD3_IP}:2379
Omni will use the client certificate you copied to ~/etcd-certs/ on the Omni host to authenticate to the cluster.

​
Backups

Because etcd is external and not managed by Omni, you are responsible for backing it up independently. See Back Up Omni Database for instructions on taking etcd snapshots.