Certificate Renewal

This runbook covers fixing the broken TLS certificate issuance and migrating from HTTP-01 to DNS-01 ACME challenge solver.

Background

The current letsencrypt-prod ClusterIssuer uses HTTP-01 challenges, which fail behind the Cloudflare tunnel because Cloudflare redirects HTTP to HTTPS. The fix is to switch to DNS-01 challenges using the Cloudflare DNS API.

Prerequisites

Access to the Cloudflare dashboard for the relevant domains
kubectl access to the cluster
Cloudflare API token with DNS edit permissions

Step 1: Create Cloudflare API Token

Log in to Cloudflare dashboard: https://dash.cloudflare.com
Go to My Profile > API Tokens > Create Token
Use the Edit zone DNS template or create custom:
- Permissions: Zone > DNS > Edit
- Zone Resources: Include > Specific zone > (select each domain: m3gatr0n.com, stiq.us, corestation.io, idp.pro, kelseyfarmequipment.com)
- Or: Include > All zones (simpler but broader)
Create and copy the token

Step 2: Create Kubernetes Secret

kubectl create secret generic cloudflare-api-token \
  --namespace cert-manager \
  --from-literal=api-token=<YOUR_CLOUDFLARE_API_TOKEN>

Verify:

kubectl get secret cloudflare-api-token -n cert-manager

Step 3: Update ClusterIssuer (Staging First)

Test with the staging issuer first:

# letsencrypt-staging-dns01.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging-key
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

Apply:

kubectl apply -f letsencrypt-staging-dns01.yaml

Verify the issuer becomes ready:

kubectl get clusterissuer letsencrypt-staging
kubectl describe clusterissuer letsencrypt-staging

Step 4: Test with a Staging Certificate

# test-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-dns01-cert
  namespace: default
spec:
  secretName: test-dns01-tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  dnsNames:
  - test.m3gatr0n.com

kubectl apply -f test-cert.yaml

# Watch the certificate progress
kubectl get certificate test-dns01-cert -w
kubectl get challenges -A

# Should become Ready within 1-2 minutes

If successful, clean up the test cert:

kubectl delete -f test-cert.yaml

Step 5: Update Production ClusterIssuer

# letsencrypt-prod-dns01.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

Apply:

kubectl apply -f letsencrypt-prod-dns01.yaml

Verify:

kubectl get clusterissuer letsencrypt-prod
kubectl describe clusterissuer letsencrypt-prod
# Status should show Ready: True

Step 6: Re-Request Failed Certificates

Delete the failed certificates so cert-manager re-creates them with the new solver:

# Harbor
kubectl delete certificate harbor-tls -n registry
# The Certificate resource should be re-created by the Harbor Helm chart
# or you may need to re-apply the Certificate manifest

# Devtron
kubectl delete certificate devtron-tls -n devtroncd

# Monitoring
kubectl delete certificate wildcard-sslip-tls -n monitoring

If the Certificate resources are not automatically recreated (not managed by Helm/operator), recreate them:

# harbor-tls.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: harbor-tls
  namespace: registry
spec:
  secretName: harbor-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - registry.m3gatr0n.com

kubectl apply -f harbor-tls.yaml

Step 7: Verify All Certificates

# Check all certificates
kubectl get certificates -A

# All should show READY = True
# Wait up to 5 minutes for DNS propagation and ACME validation

Expected output:

NAMESPACE       NAME                  READY   SECRET                AGE
default         kuber-ca              True    kuber-ca              ...
default         wildcard-sslip        True    wildcard-sslip-tls    ...
kube-system     traefik-dashboard-tls True    traefik-dashboard-tls ...
registry        harbor-tls            True    harbor-tls            ...
devtroncd       devtron-tls           True    devtron-tls           ...
monitoring      wildcard-sslip-tls    True    wildcard-sslip-tls    ...

Troubleshooting

Challenge stuck in pending

# Check challenge details
kubectl get challenges -A
kubectl describe challenge <name> -n <namespace>

# Common issues:
# - API token lacks permission for the zone
# - DNS propagation delay (wait 2-5 minutes)
# - Rate limiting (check cert-manager logs)

cert-manager logs

kubectl logs -n cert-manager deployment/cert-manager --tail=200 -f

DNS record not created

# Verify Cloudflare API token works
curl -X GET "https://api.cloudflare.com/client/v4/zones" \
  -H "Authorization: Bearer <YOUR_TOKEN>" \
  -H "Content-Type: application/json"

Rate limiting

Let's Encrypt has rate limits:

50 certificates per registered domain per week
5 duplicate certificates per week
Use the staging issuer for testing to avoid hitting limits

Ongoing Maintenance

After migration, certificates will auto-renew via DNS-01. No manual intervention needed unless:

The Cloudflare API token expires or is revoked
A new domain is added (token needs zone access)
cert-manager is upgraded or reinstalled (re-apply ClusterIssuer)

MEGATRON Infrastructure Documentation