Lab 3: CoCo-fy a Workload with cococtl - Confidential Computing Deep Dive

cococtl (kubectl coco) automates the steps from Lab 2: it sets the runtime class, converts Kubernetes Secrets to sealed secrets, generates the initdata annotation (which configures the KBS URL and OPA policy for the Kata VM), and populates KBS — all from a single command.

Prerequisites¶

Lab 2 environment running

Step 1: Create a Sample App¶

cat > myapp.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: quay.io/prometheus/busybox:latest
          command: [sleep, "infinity"]
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: myapp-secret
                  key: password
---
apiVersion: v1
kind: Secret
metadata:
  name: myapp-secret
stringData:
  password: "supersecretpassword"
EOF

myapp.yaml

Step 2: Preview What cococtl Will Change¶

kubectl coco explain -f myapp.yaml

This shows which transformations will be applied: runtime class, sealed secrets, initdata annotation.

Step 3: Apply the Original Manifest¶

The transform step reads the original myapp-secret from the cluster to encrypt it. Create it first:

kubectl apply -f myapp.yaml

Step 4: Transform the Manifest¶

kubectl coco apply \
  -f myapp.yaml \
  --skip-apply

Three files are generated in the same directory as myapp.yaml:

myapp-coco.yaml — transformed deployment (runtimeClass addition, sealed secret ref, initdata annotation)
myapp-sealed-secrets.yaml — Kubernetes Secret containing the sealed token
myapp-trustee-secrets.yaml — KBS upload manifest

Step 5: Upload Secrets to KBS and Deploy¶

# 1. Upload the plaintext secret to KBS (done before sealing)
kubectl coco kbs populate -f myapp-trustee-secrets.yaml

# 2. Create the sealed-secret k8s Secret in the cluster
kubectl apply -f myapp-sealed-secrets.yaml

# 3. Remove the original plaintext secret from the cluster
kubectl delete secret myapp-secret

# 4. Deploy the transformed workload
kubectl apply -f myapp-coco.yaml

# Wait for pod to be Running
kubectl rollout status deployment/myapp

Step 6: Verify Secret Delivery¶

The cococtl initdata annotation includes an OPA policy that blocks kubectl exec (this is intentional security — the workload should not be shell-accessible by cluster operators). Use kubectl logs to verify the secret reaches the workload:

# Replace the long-running pod with one that fetches and prints the secret
kubectl delete deployment myapp

cat > myapp-verify.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: verify-secret
spec:
  runtimeClassName: kata-qemu-coco-dev
  containers:
    - name: verify
      image: quay.io/prometheus/busybox:latest
      command:
        - /bin/sh
        - -c
        - |
          sleep 5
          echo "Fetching from KBS via CDH..."
          wget -qO- http://127.0.0.1:8006/cdh/resource/default/myapp-secret/password
          echo ""
          sleep infinity
  restartPolicy: Never
EOF

myapp-verify.yaml

Transform this pod to get KBS initdata:

kubectl coco apply -f myapp-verify.yaml --skip-apply
kubectl apply -f myapp-verify-coco.yaml
kubectl wait --for=condition=Ready pod/verify-secret --timeout=120s

Secret should appear in the logs:

kubectl logs verify-secret
# supersecretpassword

The sealed secret myapp-secret-sealed in the cluster stores an opaque token, not the plaintext. The actual value is fetched by CDH inside the Kata VM after the Attestation Agent successfully contacts KBS.

What cococtl Automated¶

Manual step (Lab 2)	cococtl equivalent
Set `runtimeClassName`	`apply` adds it automatically
Pass KBS URL to AA	initdata annotation (generated by `apply`)
Set OPA policy	initdata annotation (allow-all-except-exec by default)
Upload secrets to KBS	`kbs populate`
Create sealed k8s Secret	`apply` generates `myapp-sealed-secrets.yaml`

Understanding the Sealed Secret Flow¶

kubectl apply -f myapp.yaml        # plaintext k8s Secret in etcd
kubectl coco apply ...             # cococtl reads plaintext, generates sealed token
kubectl coco kbs populate          # plaintext stored in KBS (encrypted at rest)
kubectl delete secret myapp-secret # plaintext removed from etcd
kubectl apply -f myapp-coco.yaml   # pod uses sealed token as env var reference
                                   # -> CDH decodes token -> AA fetches from KBS
                                   # -> plaintext delivered inside TEE only

Going Further¶

# Add an attestation initContainer that blocks until attestation succeeds
kubectl coco apply -f myapp.yaml --runtime-class kata-qemu-coco-dev --init-container

# Inspect the generated initdata
kubectl coco initdata dump --raw 2>/dev/null || \
  kubectl get pod verify-secret -o jsonpath='{.metadata.annotations.io\.katacontainers\.config\.hypervisor\.cc_init_data}'

Cleanup¶

Follow the Lab 2 cleanup steps to clean the resources.

az group delete --name coco-lab --yes --no-wait
az group show --name coco-lab 2>/dev/null && echo "Still deleting..." || echo "Deleted."