Certificate Maintenance Skill

Operations for monitoring, maintaining, and managing SSL/TLS certificates in the Caddy reverse proxy with Let's Encrypt.

Quick Start

Quick certificate status check:

# Check expiry for single domain
echo | openssl s_client -servername pihole.temet.ai -connect pihole.temet.ai:443 2>/dev/null | \
  openssl x509 -noout -dates -issuer

# Check all domains
for domain in pihole jaeger langfuse sprinkler ha code webhook; do
  echo "=== $domain.temet.ai ==="
  echo | openssl s_client -servername $domain.temet.ai -connect $domain.temet.ai:443 2>/dev/null | \
    openssl x509 -noout -dates
  echo
done

# Check Caddy renewal logs
docker logs caddy 2>&1 | grep -E "renewal|renew|certificate obtained"

Table of Contents

1. •When to Use This Skill
2. •What This Skill Does
3. •Instructions
   • •3.1 Check Certificate Expiry
   • •3.2 Monitor Auto-Renewal Status
   • •3.3 Check Certificate Details
   • •3.4 Force Manual Renewal
   • •3.5 Backup Certificates
   • •3.6 Restore Certificates
4. •Supporting Files
5. •Expected Outcomes
6. •Requirements
7. •Red Flags to Avoid

When to Use This Skill

Explicit Triggers:

• •"Check certificate expiry"
• •"Certificate renewal status"
• •"SSL certificate expiring"
• •"Backup certificates"
• •"Force certificate renewal"

Implicit Triggers:

• •Certificate expiring in < 30 days
• •Need to verify auto-renewal working
• •Planning infrastructure maintenance
• •Preparing for disaster recovery

Debugging Triggers:

• •"When does my certificate expire?"
• •"Is auto-renewal working?"
• •"How to backup certificates?"

What This Skill Does

1. •Checks Expiry - Verifies certificate validity dates for all domains
2. •Monitors Renewal - Reviews Caddy logs for renewal activity
3. •Shows Details - Displays certificate issuer, validity, protocols
4. •Forces Renewal - Triggers manual certificate renewal if needed
5. •Backs Up - Creates backup of caddy_data volume
6. •Restores - Restores certificates from backup

Instructions

3.1 Check Certificate Expiry

Check single domain:

echo | openssl s_client -servername pihole.temet.ai -connect pihole.temet.ai:443 2>/dev/null | \
  openssl x509 -noout -dates -issuer

Expected output:

notBefore=Jan 10 12:00:00 2026 GMT
notAfter=Apr 10 12:00:00 2026 GMT
issuer=C = US, O = Let's Encrypt, CN = R3

Check all domains with expiry countdown:

for domain in pihole jaeger langfuse sprinkler ha code webhook; do
  echo "=== $domain.temet.ai ==="

  cert_info=$(echo | openssl s_client -servername $domain.temet.ai -connect $domain.temet.ai:443 2>/dev/null | \
    openssl x509 -noout -dates -issuer 2>&1)

  if echo "$cert_info" | grep -q "notAfter"; then
    echo "$cert_info"

    # Calculate days until expiry
    expiry_date=$(echo "$cert_info" | grep notAfter | cut -d= -f2)
    expiry_epoch=$(date -j -f "%b %d %T %Y %Z" "$expiry_date" +%s 2>/dev/null || \
                   date -d "$expiry_date" +%s 2>/dev/null)
    now_epoch=$(date +%s)
    days_left=$(( ($expiry_epoch - $now_epoch) / 86400 ))

    if [ $days_left -lt 30 ]; then
      echo "⚠️  WARNING: Expires in $days_left days (renewal due)"
    else
      echo "✅ Expires in $days_left days"
    fi
  else
    echo "❌ FAILED to get certificate"
  fi

  echo
done

Alert thresholds:

• •< 30 days: Renewal due (Caddy triggers at 30 days)
• •< 14 days: Check renewal logs for issues
• •< 7 days: Manual intervention may be needed

3.2 Monitor Auto-Renewal Status

Check recent renewal activity:

docker logs caddy 2>&1 | grep -E "renewal|renew|certificate obtained" | tail -20

Expected indicators:

• •certificate obtained successfully - New certificate issued
• •certificate renewed - Auto-renewal succeeded
• •checking certificate renewal - Caddy checking expiry

Check renewal schedule:

Caddy checks renewals every 12 hours and renews 30 days before expiry.

Verify renewal configuration:

# Check Caddy is running
docker ps | grep caddy

# Check Cloudflare DNS plugin loaded
docker exec caddy caddy list-modules | grep cloudflare

# Verify API token set
docker exec caddy env | grep CLOUDFLARE_API_KEY

If no renewal activity and expiry < 30 days:

• •Check Caddy logs for errors (use troubleshoot-https skill)
• •Verify Cloudflare API token valid
• •Consider manual renewal (step 3.4)

3.3 Check Certificate Details

View complete certificate details:

domain="pihole.temet.ai"

echo | openssl s_client -servername $domain -connect $domain:443 2>/dev/null | \
  openssl x509 -noout -text | grep -A5 "Subject:\|Issuer:\|Validity"

Shows:

• •Subject (domain name)
• •Issuer (Let's Encrypt)
• •Validity period (not before/after dates)

Check certificate chain:

echo | openssl s_client -servername pihole.temet.ai -connect pihole.temet.ai:443 -showcerts 2>/dev/null

Check supported protocols:

docker logs caddy | grep -i "protocol\|http/2\|http/3"

Expected: HTTP/2 and HTTP/3 (QUIC) enabled

3.4 Force Manual Renewal

When to force renewal:

• •Certificate expiring in < 7 days with no auto-renewal
• •Testing renewal process
• •After fixing Cloudflare API token

Option A: Reload Caddy (triggers renewal check)

docker exec caddy caddy reload --config /etc/caddy/Caddyfile

Caddy will check expiry and renew if < 30 days remaining.

Option B: Restart Caddy (full renewal check)

docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml restart caddy

Option C: Delete and recreate certificates (last resort)

⚠️ WARNING: Only use if renewal failing and expiry imminent.

# Stop Caddy
docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml down caddy

# Delete certificate volume
docker volume rm network_caddy_data

# Recreate volume
docker volume create network_caddy_data

# Start Caddy (obtains fresh certificates)
docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml up -d caddy

# Monitor certificate issuance
docker logs caddy -f

Watch for: certificate obtained successfully {"identifier": "domain.temet.ai"}

Rate limit warning:

• •Let's Encrypt limit: 50 certificates per domain per week
• •Check usage: https://crt.sh/?q=temet.ai

3.5 Backup Certificates

Why backup:

• •Disaster recovery
• •Infrastructure migration
• •Before risky changes

Note: Certificates can be re-obtained automatically via DNS-01 challenge. Backup not strictly necessary if you have valid Cloudflare API token.

Backup caddy_data volume:

# Create backup directory
mkdir -p /home/dawiddutoit/projects/network/backups

# Backup with date stamp
backup_file="/home/dawiddutoit/projects/network/backups/caddy-backup-$(date +%Y%m%d-%H%M%S).tar.gz"

tar -czf "$backup_file" \
  -C /var/lib/docker/volumes/network_caddy_data/_data .

echo "Backup created: $backup_file"

# Check backup size
ls -lh "$backup_file"

Backup retention:

• •Keep last 3 backups (certificates change every 60 days)
• •Delete backups older than 6 months

Alternative: Backup entire configuration:

backup_dir="/home/dawiddutoit/projects/network/backups/full-backup-$(date +%Y%m%d)"
mkdir -p "$backup_dir"

# Backup configuration files
cp -r /home/dawiddutoit/projects/network/docker-compose.yml "$backup_dir/"
cp -r /home/dawiddutoit/projects/network/caddy "$backup_dir/"
cp -r /home/dawiddutoit/projects/network/config "$backup_dir/"

# Backup .env (SENSITIVE - secure this file)
cp /home/dawiddutoit/projects/network/.env "$backup_dir/.env"

# Backup Docker volumes
tar -czf "$backup_dir/caddy_data.tar.gz" \
  -C /var/lib/docker/volumes/network_caddy_data/_data .

echo "Full backup created: $backup_dir"

3.6 Restore Certificates

Restore from backup:

backup_file="/home/dawiddutoit/projects/network/backups/caddy-backup-20260110.tar.gz"

# Stop Caddy
docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml down caddy

# Delete existing volume
docker volume rm network_caddy_data

# Recreate volume
docker volume create network_caddy_data

# Restore from backup
tar -xzf "$backup_file" \
  -C /var/lib/docker/volumes/network_caddy_data/_data

# Start Caddy
docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml up -d caddy

# Verify certificates loaded
docker logs caddy --tail 50

Disaster recovery scenario:

If complete infrastructure loss:

1. •Restore .env file (contains API tokens)
2. •Restore docker-compose.yml
3. •Restore Caddyfile
4. •Start Caddy (automatically obtains certificates)

No certificate backup needed if Cloudflare API token valid.

Supporting Files

Expected Outcomes

Success:

• •All certificates valid and not expiring soon (> 30 days)
• •Recent renewal activity in logs
• •Backup created successfully
• •Certificates restored and working

Warnings:

• •Certificate expiring in < 30 days (renewal due)
• •No renewal activity in logs (check troubleshoot-https skill)

Failure Indicators:

• •Certificate expired
• •Renewal failing repeatedly
• •No certificate obtained after manual renewal attempt

Requirements

• •Docker running with Caddy container
• •Valid Cloudflare API token for DNS-01 challenge
• •Network connectivity for ACME protocol
• •Sufficient disk space for backups

Red Flags to Avoid

• • Do not delete caddy_data volume without backup (unless can re-obtain quickly)
• • Do not exceed Let's Encrypt rate limits (50 certs/domain/week)
• • Do not force renewal repeatedly (causes rate limiting)
• • Do not restore old certificates if near expiry (let Caddy renew fresh)
• • Do not skip checking Cloudflare API token before manual renewal
• • Do not commit certificate backups to git (includes private keys)
• • Do not backup .env file to insecure location (contains secrets)

Notes

• •Let's Encrypt certificates valid for 90 days
• •Caddy renews automatically 30 days before expiry
• •Renewal checks occur every 12 hours
• •DNS-01 challenge allows internal-only services to get certificates
• •Certificates stored in /var/lib/docker/volumes/network_caddy_data/_data
• •No downtime during renewal (Caddy handles gracefully)
• •OCSP stapling enabled by default (better performance)
• •HTTP/2 and HTTP/3 (QUIC) supported automatically
• •Certificate transparency logs: https://crt.sh/?q=temet.ai