5.3.5 Key Management & HSM Integration

With the HSM provisioned (Set Up Encryption — Provision the HSM) and platform services running, configure each service to use HSM-backed keys instead of software-managed secrets.

How to use this page

This page is procedural: each integration has a configuration section showing the relevant manifests or commands, and a verification section to confirm it is working.

All code blocks are labelled with their file path in the repository. Select your target environment (AWS, Bare Metal, or Proxmox VMs) in any tab group — the choice syncs across the entire page.

• Using the existing rciis-devops repository: All files already exist. Skip the mkdir and git add/git commit commands — they are for users building a new repository. Simply review the files, edit values for your environment, and push.
• Building a new repository from scratch: Follow the mkdir, file creation, and git commands in order.
• No Git access: Apply manifests directly with kubectl apply or talosctl instead of committing to Git.

Prerequisites

• HSM is provisioned, initialised, and network-reachable from worker nodes (Phase 3)
• cert-manager is deployed (Wave 1 — Certificates)
• Keycloak is deployed (Wave 6 — Identity Management)
• PKCS#11 client library is available to pods that need HSM access (via init container or shared volume)

Integration Overview

Integration	What Moves to HSM	Mechanism	Section
Kubernetes PKI	Cluster CA root key	External CA mode in Talos	Kubernetes PKI
cert-manager	TLS CA signing key	PKCS#11 external signer or AWS PCA	cert-manager
Keycloak	Realm JWT signing key	Java PKCS#11 keystore	Keycloak
Kubernetes Secrets	etcd encryption key	KMS v2 plugin (PKCS#11)	KMS v2
SOPS	Age master key	HSM key wrapping	SOPS

---

Kubernetes PKI with HSM-Backed Root CA

Talos manages its own PKI hierarchy: a machine CA (for Talos API mTLS), a Kubernetes CA (for the API server, kubelet, and controller-manager certificates), and an etcd CA (for etcd peer and client certificates). All three CA key pairs are generated by talosctl gen secrets and embedded in the machine configuration.

By default, these CA private keys exist as plaintext within secrets.yaml and the machine config. The external CA pattern replaces the Kubernetes CA root key with one generated inside the HSM, so the root private key never exists in software.

How External CA Mode Works

1. Generate a root CA key pair inside the HSM during the key ceremony (see HSM Provisioning — Bare Metal tab)
2. Export the root CA certificate (public key only) from the HSM
3. Issue an intermediate CA signed by the HSM root — this intermediate's key pair can live in Talos machine config (or also in the HSM for maximum security)
4. Replace the Kubernetes CA in secrets.yaml with the intermediate certificate and key before generating machine configs

# Step 1: Generate root CA in the HSM (example using pkcs11-tool)
pkcs11-tool --module /usr/lib/libCryptoki2.so \
  --login --pin <PIN> \
  --keypairgen --key-type rsa:4096 \
  --label "k8s-root-ca" --id 01

# Step 2: Create a self-signed root CA certificate using the HSM key
# (Requires OpenSSL with PKCS#11 engine or p11-kit)
openssl req -new -x509 -engine pkcs11 \
  -keyform engine -key "pkcs11:token=rciis-production;object=k8s-root-ca;type=private" \
  -sha384 -days 3650 \
  -subj "/CN=RCIIS Kubernetes Root CA/O=EAC" \
  -out k8s-root-ca.pem

# Step 3: Generate an intermediate CA key pair (can be in HSM or software)
openssl genrsa -out k8s-intermediate-ca.key 4096

# Step 4: Create a CSR for the intermediate
openssl req -new -key k8s-intermediate-ca.key \
  -subj "/CN=RCIIS Kubernetes Intermediate CA/O=EAC" \
  -out k8s-intermediate-ca.csr

# Step 5: Sign the intermediate CSR with the HSM root key
openssl x509 -req -engine pkcs11 \
  -keyform engine -CAkey "pkcs11:token=rciis-production;object=k8s-root-ca;type=private" \
  -CA k8s-root-ca.pem \
  -in k8s-intermediate-ca.csr \
  -sha384 -days 1825 \
  -set_serial 01 \
  -extfile <(printf "basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,keyCertSign,cRLSign") \
  -out k8s-intermediate-ca.pem

Inject the External CA into Talos

Edit secrets.yaml before generating machine configs with talosctl gen config:

secrets.yaml (modified)

cluster:
  secret: <cluster-secret>
  id: <cluster-id>
certs:
  k8s:
    # Replace with the intermediate CA certificate (includes chain)
    crt: <base64-encoded concatenation of k8s-intermediate-ca.pem + k8s-root-ca.pem>
    # Replace with the intermediate CA private key
    key: <base64-encoded k8s-intermediate-ca.key>
  etcd:
    crt: <default — or replace with HSM-backed etcd CA>
    key: <default>
  k8saggregator:
    crt: <default>
    key: <default>

Then generate machine configs as normal:

talosctl gen config rciis-kenya https://api.rciis.africa:6443 \
  --with-secrets secrets.yaml \
  --config-patch-control-plane @patches/control-plane.yaml \
  --config-patch-worker @patches/worker.yaml

Talos API vs Kubernetes API

Talos API uses its own machine CA for mTLS between talosctl and nodes. It does not support OIDC or external identity providers — authentication is purely certificate-based via the talosconfig credential.

Kubernetes API is the one that supports OIDC (via Keycloak). See Kubernetes API Server OIDC for the configuration.

These are separate trust domains. Replacing the Kubernetes CA does not affect Talos API authentication.

Certificate Chain Validation

After bootstrapping, verify the certificate chain:

# Get the Kubernetes API server certificate
openssl s_client -connect api.rciis.africa:6443 -showcerts 2>/dev/null | \
  openssl x509 -noout -issuer -subject

# Verify the chain
kubectl get configmap -n kube-public cluster-info -o jsonpath='{.data.kubeconfig}' | \
  grep certificate-authority-data | awk '{print $2}' | base64 -d | \
  openssl x509 -noout -subject -issuer
# Expected issuer: CN=RCIIS Kubernetes Intermediate CA, O=EAC

---

cert-manager with HSM

Use an HSM-backed CA issuer so that the CA private key never exists in software.

PKCS#11 Configuration

Setting	Value
PKCS#11 library path	/usr/lib/libCryptoki2.so (varies by vendor)
Token label	rciis-production
Key label	ca-signing-key
PIN	From SOPS-encrypted Kubernetes Secret

AWSBare MetalProxmox VMs

Use the AWS Private CA issuer plugin for cert-manager, backed by CloudHSM via a Custom Key Store:

helm install aws-privateca-issuer \
  oci://public.ecr.aws/aws-pca/aws-privateca-issuer \
  --namespace cert-manager

aws-pca-issuer.yaml

apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
  name: hsm-backed-issuer
spec:
  arn: arn:aws:acm-pca:af-south-1:123456789012:certificate-authority/xxxxxxxx
  region: af-south-1

The AWS PCA issuer transparently uses CloudHSM for all signing operations when the Private CA is configured with a Custom Key Store. No PKCS#11 library is needed on the Kubernetes nodes — AWS handles the HSM interaction.

Deploy a PKCS#11 signer container alongside cert-manager. The signer mounts the PKCS#11 library (from the HSM vendor) and communicates with the network HSM.

Option A — cert-manager CA issuer with HSM-stored key:

Generate the CA key inside the HSM, export only the certificate, then create a cert-manager CA issuer that references the public certificate. The signing operations are performed by an external signer pod that has PKCS#11 access:

hsm-ca-issuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: hsm-ca-issuer
spec:
  ca:
    secretName: hsm-ca-certificate  # Contains the CA cert (public only)

The private key remains in the HSM. A sidecar or external signer process performs the actual signing via PKCS#11.

Option B — Use a cert-manager external issuer:

Several community projects implement cert-manager external issuers with PKCS#11 backends (e.g., cert-manager-csi-driver-spiffe with a PKCS#11 CA). Evaluate based on your HSM vendor's documentation.

Deploy a PKCS#11 signer container alongside cert-manager. The signer mounts the PKCS#11 library (from the HSM vendor) and communicates with the network HSM.

Option A — cert-manager CA issuer with HSM-stored key:

Generate the CA key inside the HSM, export only the certificate, then create a cert-manager CA issuer that references the public certificate. The signing operations are performed by an external signer pod that has PKCS#11 access:

hsm-ca-issuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: hsm-ca-issuer
spec:
  ca:
    secretName: hsm-ca-certificate  # Contains the CA cert (public only)

The private key remains in the HSM. A sidecar or external signer process performs the actual signing via PKCS#11.

Option B — Use a cert-manager external issuer:

Several community projects implement cert-manager external issuers with PKCS#11 backends (e.g., cert-manager-csi-driver-spiffe with a PKCS#11 CA). Evaluate based on your HSM vendor's documentation.

---

Keycloak with HSM

Configure Keycloak to use the HSM for realm signing keys (JWT signing), so that all tokens issued by Keycloak are signed by a key that never leaves the HSM.

Java PKCS#11 Keystore Configuration

Add the following to the Keycloak CR's additionalOptions:

keycloak.yaml (addition to spec.additionalOptions)

additionalOptions:
  # ... existing options ...
  - name: spi-realm-keys-default-provider
    value: java-keystore
  - name: spi-realm-keys-java-keystore-keystore-type
    value: PKCS11
  - name: spi-realm-keys-java-keystore-keystore-password
    secret:
      name: hsm-pin-secret
      key: pin
  - name: spi-realm-keys-java-keystore-key-alias
    value: rciis-realm-signing

PKCS#11 Provider in the JVM

The Keycloak container must have the PKCS#11 library available and a java.security configuration that registers the PKCS#11 provider. Mount the vendor library and a provider config via an init container:

pkcs11-provider.cfg

name = RCIIS-HSM
library = /usr/lib/libCryptoki2.so
slot = 0

Set the JVM system property to load the provider:

Keycloak CR environment variable

additionalOptions:
  - name: JAVA_OPTS_APPEND
    value: "-Djava.security.properties=/opt/keycloak/conf/pkcs11-provider.cfg"

Token verification uses the public key, which is freely distributable via the OIDC discovery endpoint (/.well-known/openid-configuration). Only the signing operation requires HSM access.

---

Kubernetes Secrets Encryption (KMS v2)

By default, Kubernetes stores Secrets in etcd as base64-encoded plaintext (or using a static aescbc key configured in the Talos machine config). The KMS v2 API enables Kubernetes to delegate encryption/decryption of Secrets to an external key management service — in this case, the HSM via PKCS#11.

How KMS v2 Works

kubectl create secret
        |
        v
  kube-apiserver
        |
        v
  KMS v2 gRPC call ──> k8s-kms-plugin ──> PKCS#11 ──> HSM
        |                                              (encrypt DEK)
        v
  Encrypted Secret stored in etcd

1. The API server generates a random Data Encryption Key (DEK) for each Secret
2. The DEK is sent to the KMS plugin via gRPC
3. The KMS plugin uses the HSM (via PKCS#11) to wrap (encrypt) the DEK with a master key that never leaves the HSM
4. The wrapped DEK + encrypted data are stored in etcd
5. On read, the process reverses — the HSM unwraps the DEK

Deploy the KMS Plugin

The k8s-kms-plugin (by Thales) implements the KMS v2 API with a PKCS#11 backend:

AWSBare MetalProxmox VMs

On AWS, use the native AWS KMS provider instead of the PKCS#11 plugin. When backed by a CloudHSM Custom Key Store, this provides HSM-level protection transparently:

Talos machine config patch — KMS encryption

cluster:
  apiServer:
    extraArgs:
      encryption-provider-config: /etc/kubernetes/encryption-config.yaml
    extraVolumes:
      - name: encryption-config
        hostPath: /etc/kubernetes/encryption-config.yaml
        mountPath: /etc/kubernetes/encryption-config.yaml
        readOnly: true

The encryption config references the AWS KMS key ARN (backed by CloudHSM):

encryption-config.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: aws-kms
          endpoint: unix:///var/run/kms-plugin/socket.sock
      - identity: {}

Deploy the k8s-kms-plugin as a static pod or DaemonSet on control plane nodes:

kms-plugin-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kms-plugin-config
  namespace: kube-system
data:
  config.yaml: |
    pkcs11:
      library: /usr/lib/libCryptoki2.so
      slot: 0
      pin: ${PKCS11_PIN}
      keyLabel: k8s-secrets-kek
    grpc:
      socket: /var/run/kms-plugin/socket.sock

Configure the Talos machine config to mount the KMS plugin socket and enable the encryption provider:

Talos machine config patch — KMS v2

cluster:
  apiServer:
    extraArgs:
      encryption-provider-config: /etc/kubernetes/encryption-config.yaml
    extraVolumes:
      - name: kms-socket
        hostPath: /var/run/kms-plugin
        mountPath: /var/run/kms-plugin
      - name: encryption-config
        hostPath: /etc/kubernetes/encryption-config.yaml
        mountPath: /etc/kubernetes/encryption-config.yaml
        readOnly: true

encryption-config.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: pkcs11-kms
          endpoint: unix:///var/run/kms-plugin/socket.sock
      - identity: {}

Deploy the k8s-kms-plugin as a static pod or DaemonSet on control plane nodes:

kms-plugin-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kms-plugin-config
  namespace: kube-system
data:
  config.yaml: |
    pkcs11:
      library: /usr/lib/libCryptoki2.so
      slot: 0
      pin: ${PKCS11_PIN}
      keyLabel: k8s-secrets-kek
    grpc:
      socket: /var/run/kms-plugin/socket.sock

Configure the Talos machine config to mount the KMS plugin socket and enable the encryption provider:

Talos machine config patch — KMS v2

cluster:
  apiServer:
    extraArgs:
      encryption-provider-config: /etc/kubernetes/encryption-config.yaml
    extraVolumes:
      - name: kms-socket
        hostPath: /var/run/kms-plugin
        mountPath: /var/run/kms-plugin
      - name: encryption-config
        hostPath: /etc/kubernetes/encryption-config.yaml
        mountPath: /etc/kubernetes/encryption-config.yaml
        readOnly: true

encryption-config.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: pkcs11-kms
          endpoint: unix:///var/run/kms-plugin/socket.sock
      - identity: {}

Verify Secrets Encryption

After enabling KMS v2, verify that new Secrets are encrypted:

# Create a test secret
kubectl create secret generic kms-test --from-literal=key=value -n default

# Read the raw etcd data (requires etcd access via talosctl)
talosctl -n <control-plane-node> read /var/lib/etcd/member/snap/db | \
  strings | grep -c "kms-test"
# The secret value should NOT appear in plaintext

# Re-encrypt all existing Secrets
kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -

Talos Limitation

Talos Linux has a read-only root filesystem and manages the API server as a static pod. Adding extra volumes and encryption config requires Talos machine config patches applied at install/upgrade time — this cannot be done at runtime. Plan KMS v2 integration during initial cluster provisioning or during a maintenance window that involves a rolling machine config update.

---

SOPS with HSM Key Wrapping

The current SOPS workflow uses Age keys stored as files. To add HSM protection without changing the SOPS workflow:

1. Generate a wrapping key in the HSM — an AES-256 key labelled sops-age-wrapper

2. Encrypt the Age private key with the HSM wrapping key using pkcs11-tool:

   # Wrap the Age key with the HSM
   pkcs11-tool --module /usr/lib/libCryptoki2.so \
     --login --pin <PIN> \
     --encrypt --mechanism AES-KEY-WRAP \
     --id 02 \
     --input-file age.agekey \
     --output-file age.agekey.wrapped
   

3. At decryption time, unwrap the Age key via the HSM, use it to decrypt SOPS secrets, then discard the unwrapped key from memory

This provides HSM-level protection for the SOPS master key at rest. The Age key file is encrypted and useless without HSM access.

Alternative: SOPS with KMS backend

An alternative is to use SOPS with the --kms backend (AWS) instead of Age, which integrates directly with CloudHSM via the Custom Key Store. This eliminates the Age key wrapping step entirely:

# AWS: Use a KMS key backed by CloudHSM
sops --kms "arn:aws:kms:af-south-1:123456789012:key/xxxxxxxx" \
  --encrypt secret.yaml > secret.enc.yaml

For on-premises deployments, SOPS does not natively support PKCS#11 — the Age key wrapping approach above is the recommended pattern.

---

Integration Checklist

After completing all integrations, verify each component:

Integration	Verification Command	Expected Result
Kubernetes PKI	openssl s_client -connect api.rciis.africa:6443 2>/dev/null \| openssl x509 -noout -issuer	Issuer contains "RCIIS Kubernetes"
cert-manager	kubectl get clusterissuer hsm-ca-issuer -o yaml	Issuer status is Ready
Keycloak	curl -s https://auth.rciis.eac.int/realms/rciis/.well-known/openid-configuration \| jq .jwks_uri	JWKS endpoint returns keys
KMS v2	kubectl create secret generic test --from-literal=x=y && talosctl read /var/lib/etcd/...	Secret value not in etcd plaintext
SOPS	Decrypt a test secret without the plaintext Age key	Requires HSM access to unwrap

See Certificate Rotation for HSM-backed certificate lifecycle procedures and Incident Response for HSM compromise scenarios.

Next Steps

Proceed to DNS and external networking configuration:

6. Configure Cloudflare