Run Image Factory On-Prem

The Image Factory is a way for you to dynamically create Talos Linux images. There is a public, hosted version of the Image Factory at factory.talos.dev and it can also be run in your environment.

The Image Factory is a critical component of Omni to generate installation media and update Talos nodes, but it is not required to use Omni to use the Image Factory. It is a web interface and API for the imager command which is used to customize Talos from the command line.

Prerequisites

• Machine to run Image Factory
• crane
• docker or podman

Podman is known to work but has some flags that are different than docker and you may have to translate them for your version of podman.

Container registry

If you already have a container registry available you can export your registry to an environment variable. and skip to create an image cache signing key.

REGISTRY_ENDPOINT=registry.internal:5000

If you don’t have a container registry available to push images to you can temporarily run one with the registry container. We recommend using the official registry:2 registry from docker as some registries do not support all OCI images.

This example doesn’t have persistent storage.

Secure registry

We recommend using certificates for your temporary registry you will need to provide your own certificates and mount them into the container at run time. If you do not have certificates, follow the steps in the Omni air-gapped documentation.

docker run -d \
  --name registry \
  -p 5000:5000 \
  -v ${PWD}/server-key.pem:/certs/server-key.pem:ro \
  -v ${PWD}/server-chain.pem:/certs/server-chain.pem:ro \
  -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
  -e REGISTRY_HTTP_TLS_KEY=/certs/server-key.pem \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server-chain.pem \
  registry:2

Make sure the CA certificate is in your system pki path and docker has restarted to trust the certificate.

Insecure registry

A registry can be run without certificates or encrypted communication. Running this way will require your to add a flag to crane and docker to allow insecure communication.

docker run -d -p 5000:5000 --name registry registry:2

Air-gapped

Without internet access you will need to download the container image, transfer it internally, and load it on the target machine.

docker save -o registry.tar registry:2

Transfer the registry.tar file to an internal system.

docker load -i registry.tar

Run the registry with certificates.

If SELinux is enabled replace :ro with :Z.

docker run -d \
  --name registry \
  -p 5000:5000 \
  -v ${PWD}/server-key.pem:/certs/server-key.pem:ro \
  -v ${PWD}/server-chain.pem:/certs/server-chain.pem:ro \
  -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
  -e REGISTRY_HTTP_TLS_KEY=/certs/server-key.pem \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server-chain.pem \
  registry:2

Image cache signing key

You need to create a Cache Signing Key to sign cached Talos image artifacts, ensuring they haven’t been tampered with before being served.

openssl ecparam -name prime256v1 -genkey -noout -out signing-key.key

Image cache storage (optional)

There are a variety of image cache locations to store built images. Without an image cache, each asset will be built on demand which can consume a high amount of CPU on the image factory machine.

Some supported cache storage options include:

• CDN
• s3 bucket (or compatible API)

Please view the --help output for cache options.

Run Image Factory

There are two supported methods to run the Image Factory:

• Connected to the upstream Sidero container registry
• Using a custom container registry

A custom container registry is required for air-gapped environments or custom Talos builds.

Image factory conneted to upstream

Run with the official, upstream container registry if your machine is connected to the internet and you don’t need custom Talos images.

The official Sidero Labs registry has all of the required Talos installation containers, extensions, and tools. If you want to run image factory connected to the upstream container registry you can do it with:

docker run -p 8080:8080 -d \
  --name image-factory \
  -v $PWD/signing-key.key:/signing-key.key:ro \
  ghcr.io/siderolabs/image-factory:v0.9.0 \
    -cache-signing-key-path /signing-key.key \
    -schematic-service-repository $REGISTRY_ENDPOINT/siderolabs/image-factory/schematic

If your system has SELinux enabled you will need to mount the signing key with the :Z option so the image factory has access to the file.

This will run the image factory on your machine on port 8080 and automatically pull container images from Sidero’s registry. It will also validate image signatures using cosign to validate pulled images.

This will not allow you to create or publish custom system extensions. To do that you will need to run your own container registry with the necessary images. See the disconnected instructions for Image Factory.

Image Factory with custom registry or air-gapped

Run with an internal container registry if your machine is not connected to the internet, or you need custom Talos images and extensions.

Running the image factory in an air-gapped environment has more requirements than running in a connected mode. Make sure you have a registry running from the Internal container registry section. You will need to download container images and seed them into the internal registry and sign the container images.

This is just an example and should not be used in a production environment. If you want to test locally on your machine you can also see the developer documentation in the repository.

Download container images

Starting with Talos 1.12 you can get a list of images needed to seed the image factory directly from talosctl.

Get a list of Talos base images needed for the image factory with:

This will give you a list of all images and extensions for Talos . You will need to repeat this command for each version of Talos you want to download images for. If you don’t need specific extensions you can delete them from the images.txt file.

Push the images to your $REGISTRY_ENDPOINT

Internet available

If your machine can reach the public internet and the internal registry at the same time you can copy the images internally with this command.

for SOURCE_IMAGE in $(cat images.txt)
  do
    IMAGE_WITHOUT_DIGEST=${SOURCE_IMAGE%%@*}
    IMAGE_WITH_NEW_REG="${REGISTRY_ENDPOINT}/${IMAGE_WITHOUT_DIGEST#*/}"
    crane copy \
      $SOURCE_IMAGE \
      $IMAGE_WITH_NEW_REG
done

Air-gapped

If you don’t have direct access to an internal container registry (e.g. air gapped environment) you need to download the container images while connected to the internet with this command:

cat images.txt \
  | talosctl images cache-create \
      --layout flat \
      --image-cache-path ./image-cache \
      --images=-

Move the image-cache folder to an air gapped machine and serve the images on a read only, temporary container registry with:

IP=$(hostname -I | awk '{print $1}')

talosctl image cache-cert-gen \
  --advertised-address $IP

talosctl image cache-serve \
  --address $IP:5000 \
  --image-cache-path ./image-cache \
  --tls-cert-file tls.crt \
  --tls-key-file tls.key

A temporary image registry will run on your local machine IP address port 5000 with self-signed certificates. Copy the images to an internal, permanent container registry.

for SOURCE_IMAGE in $(cat images.txt)
  do
    IMAGE_WITHOUT_DIGEST=${SOURCE_IMAGE%%@*}
    IMAGE_WITH_NEW_REG="${REGISTRY_ENDPOINT}/${IMAGE_WITHOUT_DIGEST#*/}"
    LOCALHOST_IMAGE="localhost:5000/${IMAGE_WITHOUT_DIGEST#*/}"
    crane copy \
      $LOCALHOST_IMAGE \
      $IMAGE_WITH_NEW_REG
done

Sign container images

The Image Factory verifies container image signatures when being used. Generate a cosign singing key and sign each container pushed to the registry.

Image factory currently only supports cosign v2 signatures.

Generate a cosign key

docker run --rm -it \
  -v $PWD:/keys -w /keys \
  -e COSIGN_PASSWORD="" \
  --user $(id -u):$(id -g) \
  ghcr.io/sigstore/cosign/cosign:v2.6.1 \
  generate-key-pair

Sign each image and tag in your internal registry. This will allow the registry to validate images without reaching out to any external services for key validation.

KEY_FILE="cosign.key"
export COSIGN_PASSWORD="" # Leave empty for empty password

Sign all of the images using the images.txt file as a list.

Trusted CA

If your registry is running with a globally trusted certificate (e.g. signed by lets encrypt) you can sign the images with the following command:

for IMAGE in $(cat images.txt)
  do
    NEW_IMAGE="${REGISTRY_ENDPOINT}/${IMAGE#*/}"
    if [[ "$NEW_IMAGE" != *"@sha256:"* ]]; then
      NEW_IMAGE="${NEW_IMAGE}@$(crane digest $NEW_IMAGE)"
    fi
    docker run --rm -it --net=host \
      -v $PWD:/keys -w /keys \
      -e COSIGN_PASSWORD="" \
      -e COSIGN_YES=true \
      --user $(id -u):$(id -g) \
      ghcr.io/sigstore/cosign/cosign:v2.6.1 \
        sign --key /keys/$KEY_FILE \
        --tlog-upload=false \
        $NEW_IMAGE
done

Self-signed CA

If your registry is running with a self-signed CA certificate (i.e. from the Installing Airgapped Omni guide) you need to mount the CA certificate into the cosign container for it to be trusted.

for IMAGE in $(cat images.txt)
  do
    NEW_IMAGE="${REGISTRY_ENDPOINT}/${IMAGE#*/}"
    if [[ "$NEW_IMAGE" != *"@sha256:"* ]]; then
      NEW_IMAGE="${NEW_IMAGE}@$(crane digest $NEW_IMAGE)"
    fi
    docker run --rm -it --net=host \
      -v $PWD:/keys -w /keys \
      -v "$PWD/ca.pem:/etc/ssl/certs/ca-certificates.crt:ro" \
      -e COSIGN_PASSWORD="" \
      -e COSIGN_YES=true \
      --user $(id -u):$(id -g) \
      ghcr.io/sigstore/cosign/cosign:v2.6.1 \
        sign --key /keys/$KEY_FILE \
        --tlog-upload=false \
        $NEW_IMAGE
done

With a populated container registry and signed images you are ready to run the Image Factory.

Set a internal factory endpoint.

FACTORY_URL=https://factory.internal:8080

This guide assumes the container registry and image factory are running on the same machine. Because of this we will run the Image Factory with --net=host which is not recommended for a production, multi-host deployment.

Trusted CA

To run the image factory with a trusted certificate you can use the following command.

docker run -p 8080:8080 -d \
  --name image-factory \
  --net=host \
  -v $PWD/signing-key.key:/signing-key.key:ro \
  -v $PWD/cosign.pub:/cosign.pub:ro \
  ghcr.io/siderolabs/image-factory:v1.0.0 \
    -external-url $FACTORY_URL \
    -image-registry $REGISTRY_ENDPOINT \
    -installer-internal-repository $REGISTRY_ENDPOINT/siderolabs \
    -installer-internal-repository $REGISTRY_ENDPOINT/siderolabs \
    -schematic-service-repository $REGISTRY_ENDPOINT/siderolabs/image-factory/schematic \
    -cache-repository $REGISTRY_ENDPOINT/siderolabs/cache \
    -cache-signing-key-path /signing-key.key \
    -container-signature-pubkey /cosign.pub \
    -cache-cdn-enabled=false \
    -cache-s3-enabled=false

Self-signed CA

To run the image factory with a self-signed CA certificate you need to mount them into the container image at run time.

Red Hat

If you are running on a server with SELinux enabled and enforcing then volumes mounted into the container will not be available unless you append :Z to the volume mounts.

docker run -p 8080:8080 -d \
  --name image-factory \
  --net=host \
  -v $PWD/signing-key.key:/signing-key.key:ro \
  -v $PWD/cosign.pub:/cosign.pub:ro \
  -v $PWD/server-chain.pem:/certs/server-chain.pem:ro \
  -v $PWD/server-key.pem:/certs/server-key.pem:ro \
  -v /etc/pki/ca-trust/source/anchors/:/etc/ssl/certs:ro \
  ghcr.io/siderolabs/image-factory:v1.0.0 \
    -external-url $FACTORY_URL \
    -image-registry $REGISTRY_ENDPOINT \
    -installer-internal-repository $REGISTRY_ENDPOINT/siderolabs \
    -installer-internal-repository $REGISTRY_ENDPOINT/siderolabs \
    -schematic-service-repository $REGISTRY_ENDPOINT/siderolabs/image-factory/schematic \
    -cache-repository $REGISTRY_ENDPOINT/siderolabs/cache \
    -cache-signing-key-path /signing-key.key \
    -container-signature-pubkey /cosign.pub \
    -cache-cdn-enabled=false \
    -cache-s3-enabled=false \
    -http-key-file=/certs/server-key.pem \
    -http-cert-file=/certs/server-chain.pem

Ubuntu

docker run -p 8080:8080 -d \
  --name image-factory \
  --net=host \
  -v $PWD/signing-key.key:/signing-key.key:ro \
  -v $PWD/cosign.pub:/cosign.pub:ro \
  -v $PWD/server-chain.pem:/certs/server-chain.pem:ro \
  -v $PWD/server-key.pem:/certs/server-key.pem:ro \
  -v /usr/local/share/ca-certificates/:/etc/ssl/certs:ro \
  ghcr.io/siderolabs/image-factory:v1.0.0 \
    -external-url $FACTORY_URL \
    -image-registry $REGISTRY_ENDPOINT \
    -installer-internal-repository $REGISTRY_ENDPOINT/siderolabs \
    -installer-internal-repository $REGISTRY_ENDPOINT/siderolabs \
    -schematic-service-repository $REGISTRY_ENDPOINT/siderolabs/image-factory/schematic \
    -cache-repository $REGISTRY_ENDPOINT/siderolabs/cache \
    -cache-signing-key-path /signing-key.key \
    -container-signature-pubkey /cosign.pub \
    -cache-cdn-enabled=false \
    -cache-s3-enabled=false \
    -http-key-file=/certs/server-key.pem \
    -http-cert-file=/certs/server-chain.pem

Insecure

If your image factory and container registry do not have certificates run the following command:

docker run -p 8080:8080 -d \
  --name image-factory \
  --net=host \
  -v $PWD/signing-key.key:/signing-key.key:ro \
  -v $PWD/cosign.pub:/cosign.pub:ro \
  ghcr.io/siderolabs/image-factory:v1.0.0 \
    -insecure-image-registry \
    -insecure-installer-internal-repository \
    -insecure-schematic-service-repository \
    -external-url $FACTORY_URL \
    -image-registry $REGISTRY_ENDPOINT \
    -installer-internal-repository $REGISTRY_ENDPOINT/siderolabs \
    -installer-internal-repository $REGISTRY_ENDPOINT/siderolabs \
    -schematic-service-repository $REGISTRY_ENDPOINT/siderolabs/image-factory/schematic \
    -cache-repository $REGISTRY_ENDPOINT/siderolabs/cache \
    -cache-signing-key-path /signing-key.key \
    -container-signature-pubkey /cosign.pub \
    -cache-cdn-enabled=false \
    -cache-s3-enabled=false

You should now be able to browse to https://registry.internal:8080 and view the Image Factory web interface. If your server or network has any firewall rules you may need to allow TCP traffic to the host.

Run Omni

After the image factory is running you can continue to the Omni Airgapped documentation.