Run Image Factory On-Prem

Run Image Factory on-prem to generate Talos images locally

The Image Factory is a way for you to dynamically create Talos Linux images. There is a public, hosted version of the Image Factory at factory.talos.dev and it can also be run in your environment.

The Image Factory is a critical component of Omni to generate installation media and update Talos nodes, but it is not required to use Omni to use the Image Factory. It is a web interface and API for the imager command which is used to customize Talos from the command line.

Prerequisites

• talosctl
• crane
• docker or podman

Podman is known to work but has some flags that are different than docker and you may have to translate them for your version of podman.

Container registry

If you already have a container registry available you can export your registry to an environment variable. and skip to create an image cache signing key.

REGISTRY_ENDPOINT=registry.internal:5000

If you don’t have a container registry available to push images to you can temporarily run one with the registry container. We recommend using the official registry:2 registry from docker as some registries do not support all OCI images.

This example doesn’t have persistent storage.

Secure registry

We recommend using certificates for your temporary registry you will need to provide your own certificates and mount them into the container at run time. If you do not have certificates, follow the steps in the Omni air-gapped documentation.

docker run -d \
  --name registry \
  -p 5000:5000 \
  -v ${PWD}/server-key.pem:/certs/server-key.pem:ro,Z \
  -v ${PWD}/server-chain.pem:/certs/server-chain.pem:ro,Z \
  -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
  -e REGISTRY_HTTP_TLS_KEY=/certs/server-key.pem \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server-chain.pem \
  registry:2

Make sure the CA certificate is in your system pki path and docker has restarted to trust the certificate.

Insecure registry

A registry can be run without certificates or encrypted communication. Running this way will require your to add a flag to crane and docker to allow insecure communication.

docker run -d -p 5000:5000 --name registry registry:2

Air-gapped

Without internet access you will need to download the container image, transfer it internally, and load it on the target machine.

Transfer the registry.tar and image-factory.tar files to an internal system.

docker load -i registry.tar
docker load -i image-factory.tar

Run the registry with certificates.

docker run -d \
  --name registry \
  -p 5000:5000 \
  -v ${PWD}/server-key.pem:/certs/server-key.pem:ro,Z \
  -v ${PWD}/server-chain.pem:/certs/server-chain.pem:ro,Z \
  -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
  -e REGISTRY_HTTP_TLS_KEY=/certs/server-key.pem \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server-chain.pem \
  registry:2

Image cache signing key

You need to create a Cache Signing Key to sign cached Talos image artifacts, ensuring they haven’t been tampered with before being served.

openssl ecparam -name prime256v1 -genkey -noout -out signing-key.key

Image cache storage (optional)

There are a variety of image cache locations to store built images. Without an image cache, each asset will be built on demand which can consume a high amount of CPU on the image factory machine.

Some supported cache storage options include:

• CDN
• s3 bucket (or compatible API)

Please view the --help output for cache options.

Run Image Factory

There are two supported methods to run the Image Factory:

• Connected to the upstream Sidero container registry
• Using a custom container registry

A custom container registry is required for air-gapped environments or custom Talos builds.

Image factory connected

Run with the official, upstream container registry if your machine is connected to the internet and you don’t need custom Talos images.

The official Sidero Labs registry has all of the required Talos installation containers, extensions, and tools.

Export the URL for where you will be hosting the Image Factory.

export FACTORY_URL="http://factory.internal:8080"

You need to provide a registry where image schematics can be stored. Create a config file.

cat <<EOF > image-factory.yaml
artifacts:
  schematic:
    registry: $REGISTRY_ENDPOINT
cache:
  signingKeyPath: "/signing-key.key"
http:
  externalURL: $FACTORY_URL
EOF

If you want to run image factory connected to the upstream container registry you can do it with:

This will run the image factory on your machine on port 8080 and automatically pull container images from Sidero’s registry. It will also validate image signatures using cosign to validate pulled images.

This will not allow you to create or publish custom system extensions. To do that you will need to run your own container registry with the necessary images. See the disconnected instructions for Image Factory.

Image Factory air-gapped

Run with an internal container registry if your machine is not connected to the internet, or you need custom Talos images and extensions.

Running the image factory in an air-gapped environment has more requirements than running in a connected mode. Make sure you have a registry running from the Internal container registry section. You will need to download container images and seed them into the internal registry and sign the container images.

This is just an example and should not be used in a production environment. If you want to test locally on your machine you can also see the developer documentation in the repository.

Download container images

Starting with Talos 1.12 you can get a list of images needed to seed the image factory directly from talosctl.

Get a list of Talos base images needed for the image factory with:

This will give you a list of all images and extensions for Talos v1.14.5. You will need to repeat this command for each version of Talos you want to download images for. If you don’t need specific extensions you can delete them from the images.txt file.

Push the images to your $REGISTRY_ENDPOINT

Internet available

If your machine can reach the public internet and the internal registry at the same time you can copy the images internally with this command.

for SOURCE_IMAGE in $(cat images.txt)
  do
    IMAGE_WITHOUT_DIGEST=${SOURCE_IMAGE%%@*}
    IMAGE_WITH_NEW_REG="${REGISTRY_ENDPOINT}/${IMAGE_WITHOUT_DIGEST#*/}"
    crane copy \
      $SOURCE_IMAGE \
      $IMAGE_WITH_NEW_REG
done

Air-gapped

If you don’t have direct access to an internal container registry (e.g. air gapped environment) you need to download the container images while connected to the internet.

Download all images from images.txt. Create file names that don’t contain special characters.

cat images.txt \
  | talosctl images cache-create \
      --layout flat \
      --image-cache-path ./image-cache \
      --images=-

Move to a network that has access to the $REGISTRY_ENDPOINT endpoint and push the local images to the registry. This serves the container layers on your local admin machine port :5000 so you can push them with crane. Run this command command from your admin machine so you do not have port conflicts with the $REGISTRY_ENDPOINT.

This method of copying containers is important because Talos releases pin to a container digest which needs to match in your internal registry

export IP=$(hostname -I | awk '{print $1}')

talosctl image cache-cert-gen \
  --advertised-address $IP

talosctl image cache-serve \
  --address $IP:5000 \
  --image-cache-path ./image-cache \
  --tls-cert-file tls.crt \
  --tls-key-file tls.key &

Push the container images with crane.

for SOURCE_IMAGE in $(cat images.txt)
  do
    IMAGE_WITHOUT_DIGEST=${SOURCE_IMAGE%%@*}
    IMAGE_WITH_NEW_REG="${REGISTRY_ENDPOINT}/${IMAGE_WITHOUT_DIGEST#*/}"
    LOCALHOST_IMAGE="${IP}:5000/${IMAGE_WITHOUT_DIGEST#*/}"
    crane copy --insecure \
      $LOCALHOST_IMAGE \
      $IMAGE_WITH_NEW_REG
done

Sign container images

The Image Factory verifies container image signatures when being used. Generate a cosign singing key and sign each container pushed to the registry.

Image factory currently only supports cosign v2 signatures.

Generate a cosign key.

docker run --rm -it \
  -v $PWD:/keys:ro,Z -w /keys \
  -e COSIGN_PASSWORD="" \
  --user $(id -u):$(id -g) \
  ghcr.io/sigstore/cosign/cosign:v2.6.1 \
  generate-key-pair

Sign each image and tag in your internal registry. This will allow the registry to validate images without reaching out to any external services for key validation.

KEY_FILE="cosign.key"
export COSIGN_PASSWORD="" # Leave empty for empty password

Sign all of the images using the images.txt file as a list.

If your registry is running with a self-signed CA certificate (i.e. from the Installing Airgapped Omni guide) you need to mount the CA certificate into the cosign container for it to be trusted.

for IMAGE in $(cat images.txt)
  do
    NEW_IMAGE="${REGISTRY_ENDPOINT}/${IMAGE#*/}"
    if [[ "$NEW_IMAGE" != *"@sha256:"* ]]; then
      NEW_IMAGE="${NEW_IMAGE}@$(crane digest $NEW_IMAGE)"
    fi
    docker run --rm -it --net=host \
      -v $PWD:/keys -w /keys \
      -v "$PWD/ca.pem:/etc/ssl/certs/ca-certificates.crt:ro,Z" \
      -e COSIGN_PASSWORD="" \
      -e COSIGN_YES=true \
      --user $(id -u):$(id -g) \
      ghcr.io/sigstore/cosign/cosign:v2.6.1 \
        sign --key /keys/$KEY_FILE \
        --tlog-upload=false \
        $NEW_IMAGE
done

This guide assumes the container registry and image factory are running on the same machine. Because of this we will run the Image Factory with --net=host which is not recommended for a production, multi-host deployment.

Self-signed CA

Set a internal factory endpoint.

FACTORY_URL=https://factory.internal:8080

Create a configuration file for the Image Factory.

cat <<EOF > image-factory.yaml
artifacts:
  core:
    registry: $REGISTRY_ENDPOINT
  schematic:
    registry: $REGISTRY_ENDPOINT
  installer:
    internal:
      registry: $REGISTRY_ENDPOINT
    external:
      registry: $REGISTRY_ENDPOINT
      repository: "talos"
      namespace: "siderolabs"
containerSignature:
  publicKeyFile: /cosign.pub
  subjectRegExp: ""
cache:
  oci:
    registry: $REGISTRY_ENDPOINT
  signingKeyPath: "/signing-key.key"
http:
  externalURL: $FACTORY_URL
  certFile: "/certs/server-chain.pem"
  keyFile: "/certs/server-key.pem"
EOF

To run the image factory with a self-signed CA certificate you need to mount them into the container image at run time.

You should now be able to browse to https://registry.internal:8080 and view the Image Factory web interface. If your server or network has any firewall rules you may need to allow TCP traffic to the host.

Insecure

Set a internal factory endpoint.

FACTORY_URL=http://factory.internal:8080

Create a configuration for the Image Factory

cat <<EOF > image-factory.yaml
artifacts:
  core:
    registry: $REGISTRY_ENDPOINT
    insecure: true
  schematic:
    registry: $REGISTRY_ENDPOINT
    insecure: true
  installer:
    internal:
      registry: $REGISTRY_ENDPOINT
      insecure: true
    external:
      registry: $REGISTRY_ENDPOINT
      insecure: true
containerSignature:
  publicKeyFile: /cosign.pub
  subjectRegExp: ""
cache:
  oci:
    registry: $REGISTRY_ENDPOINT
    insecure: true
  signingKeyPath: "/signing-key.key"
http:
  externalURL: $FACTORY_URL
EOF

If your image factory and container registry do not have certificates run the following command:

You should now be able to browse to http://registry.internal:8080 and view the Image Factory web interface. If your server or network has any firewall rules you may need to allow TCP traffic to the host.

Run Omni

After the image factory is running you can continue to the Omni Airgapped documentation.