Deploying Ambasdr to DigitalOcean (dev + prod)

Five concerns split across three locations. Every cross-boundary call is either an outbound HTTPS request from the Droplet or an inbound call to a public hostname that Caddy terminates over TLS.

dev vs prod at a glance

Provisioning and deployment are deliberately separated into two repositories. Knowing which one owns a given concern removes most of the ambiguity in the rest of this page.

From the ambasdr-infra repo. The Makefile wires the matching backend-config + var-file from ENV automatically, so you can't apply one environment's config against another's state.

Prerequisites (once)

Apply

# From ambasdr-infra/. State creds in the env.
export AWS_ACCESS_KEY_ID=<spaces_access_id>
export AWS_SECRET_ACCESS_KEY=<spaces_secret_key>

make plan   ENV=dev      # preview
make apply  ENV=dev      # create Droplet + reserved IP + DNS + firewall
make output ENV=dev      # reserved_ip, frontend_fqdn, api_fqdn

What an apply creates: a VPC + cloud firewall (rules target by tag), the Droplet + a reserved IP assigned to it, and the Cloudflare A records for the frontend and API hostnames. For prod it additionally creates the Managed Postgres cluster (with the ambasdr and ambasdr_knowledge_graph databases) and the Spaces bucket. The conditional modules are gated by managed_postgres_enabled / spaces_enabled, so dev provisions nothing it doesn't use.

Four app images plus the off-the-shelf data-store images. All three app Dockerfiles already exist in the repo; CI builds and pushes them to ghcr.io/juelz-ai tagged :dev (plus a SHA-tagged variant for rollback). The Droplet only ever pulls — it never builds.

Each lives at /opt/ambasdr/.env.*, mode 600, never committed. The compose file reads .env.host via --env-file (for ${…} substitutions) and the per-service files via env_file:. The dev-only values that change for prod are called out.

deploy/.env.host (compose-substituted)

Datastore credentials and the Caddy/Cloudflare settings the compose file substitutes. Template: deploy/.env.host.example.

REGISTRY=ghcr.io/juelz-ai
IMAGE_TAG=dev

PG_PASSWORD=...rotated...           # dev only (prod uses Managed PG creds)
NEO4J_PASSWORD=...rotated...
MINIO_USER=...                      # dev only (prod uses Spaces keys)
MINIO_PASSWORD=...rotated...
GRAFANA_ADMIN_PASSWORD=...rotated...

# Caddy / TLS
FRONTEND_DOMAIN=dev.ambasdr.com     # prod: ambasdr.com (or app.ambasdr.com)
API_DOMAIN=api.dev.ambasdr.com      # prod: api.ambasdr.com
ACME_EMAIL=ops@ambasdr.com

# Grafana team access via Cloudflare Tunnel + Access (see §12)
GRAFANA_DOMAIN=grafana-dev.ambasdr.com   # prod: grafana.ambasdr.com
CLOUDFLARED_TUNNEL_TOKEN=...secret...    # per-box tunnel token

deploy/.env.backend

deploy/.env.kg

Frontend (build-time only)

Frontend values are baked into the Vite bundle by CI as --build-args sourced from the dev GitHub Environment variables — not from an env file at runtime: VITE_AUTH0_DOMAIN, VITE_AUTH0_CLIENT_ID, VITE_AUTH0_AUDIENCE, VITE_API_URL (=https://api.dev.ambasdr.com), and the optional VITE_GEMINI_API_KEY / VITE_STRIPE_PAYMENT_LINK.

Two public hostnames per environment, both terminated by one Caddy container. Caddy reads the hostnames and ACME contact from the environment and obtains + renews Let's Encrypt certs automatically over HTTP-01, redirecting HTTP → HTTPS.

# deploy/caddy/Caddyfile  (domains injected from .env.host)
{
	email {$ACME_EMAIL}
}

# Frontend SPA (nginx serves the static bundle on :8080 inside the network).
{$FRONTEND_DOMAIN} {
	encode zstd gzip
	reverse_proxy frontend:8080
}

# Backend API (Gin on :9080 inside the network).
{$API_DOMAIN} {
	encode zstd gzip
	reverse_proxy backend:9080
}

How the request flows: a browser resolves dev.ambasdr.com (a grey-cloud Cloudflare A record from §03) directly to the Droplet's reserved IP → Caddy on port 443 terminates TLS → Caddy reverse-proxies over the internal Docker network to frontend:8080 or backend:9080. backend and frontend are bound only to 127.0.0.1 on the box (for on-box health checks); the public path is exclusively through Caddy.

A per-environment SPA Application (or tenant) is cleanest. Four settings must land before the frontend can complete a login round-trip. Substitute the environment's real frontend/API hostnames below (dev.ambasdr.com / api.dev.ambasdr.com for dev).

The Auth0 CLI scripts this faster than the dashboard:

auth0 apps update <client_id> \
  --callbacks "https://dev.ambasdr.com,https://dev.ambasdr.com/callback" \
  --logout-urls "https://dev.ambasdr.com" \
  --origins "https://dev.ambasdr.com"

auth0 actions update <action_id> \
  --secret PROVISION_URL=https://api.dev.ambasdr.com/internal/users/provision \
  --secret M2M_SHARED_SECRET=<new_secret>
auth0 actions deploy <action_id>

dev stays in Stripe test mode. A webhook endpoint is needed at the dev API hostname:

stripe webhook_endpoints create \
  --url=https://api.dev.ambasdr.com/webhooks/stripe \
  --enabled-events=customer.subscription.created \
  --enabled-events=customer.subscription.updated \
  --enabled-events=customer.subscription.deleted \
  --enabled-events=invoice.payment_succeeded \
  --enabled-events=invoice.payment_failed \
  --description="Ambasdr dev (DigitalOcean)"

# The response's `secret` (whsec_...) is STRIPE_WEBHOOK_SECRET for .env.backend.

prod: switch to live-mode keys and create a fresh live webhook at https://api.ambasdr.com/webhooks/stripe. The signing secret survives URL changes (stripe webhook_endpoints update <we_…> --url=…), but live and test are separate endpoints with separate secrets.

The Python agent in livekit-agent/ runs on LiveKit Cloud, independent of the Droplet. It needs the LiveKit project API key/secret (injected by LiveKit), the public backend URL (to fetch voice prompts), and the AGENT_API_KEY (to authenticate to the backend). Use a separate LiveKit Cloud project per environment.

lk cloud agent secrets set ambasdr-voice-dev \
  AMBASDR_SERVICE_URL=https://api.dev.ambasdr.com \
  AMBASDR_API_KEY=<same value as backend AGENT_API_KEY> \
  DEEPGRAM_API_KEY=<...> \
  ELEVEN_API_KEY=<...>

Order matters: provision, finish the box, place secrets, then bring the stack up. The make do-* targets reach the box over SSH — set DO_HOST to the box (e.g. the ambasdr-dev SSH alias, or root@<reserved-ip>).

make do-bootstrap DO_HOST=root@<reserved-ip> \
  GHCR_USER=<user> GHCR_PAT=<pat> RUNNER_TOKEN=<repo-runner-token>

scp deploy/.env.host    root@<ip>:/opt/ambasdr/.env.host
scp deploy/.env.backend root@<ip>:/opt/ambasdr/.env.backend
scp deploy/.env.kg      root@<ip>:/opt/ambasdr/.env.kg

ssh root@<ip> 'docker run --rm \
  -v /opt/ambasdr/migrations:/migrations \
  --network ambasdr_internal migrate/migrate \
  -path /migrations \
  -database "postgres://ambasdr:$PG_PASSWORD@postgres:5432/ambasdr?sslmode=disable" up'

After the first deploy, pushes to main deploy automatically. .github/workflows/deploy-dev.yml detects which service trees changed, builds + pushes only those images to ghcr.io/juelz-ai, applies migrations when needed, then pulls + recreates the matching containers on the box via the self-hosted do-dev runner, and finally smokes the public URLs.

Manual dispatch

Force-build specific services without a code change:

gh workflow run deploy-dev.yml -R Juelz-AI/ambasdr --ref main -f services=backend
gh workflow run deploy-dev.yml -R Juelz-AI/ambasdr --ref main -f services=frontend,backend

Rollback

Every push also tags :dev-<commit-sha>. Pin a known-good SHA on the box — no compose edit needed:

ssh ambasdr-dev
cd /opt/ambasdr
docker images ghcr.io/juelz-ai/ambasdr-backend           # list SHA tags
IMAGE_TAG=dev-<old-sha> docker compose \
  -f docker-compose.do-dev.yml --env-file .env.host up -d backend

Manual (fallback) deploy

If CI is down, make do-deploy DO_HOST=ambasdr-dev syncs and rolls the whole stack; make do-logs DO_HOST=ambasdr-dev SVC=backend tails one service. Migrations have to be additive (expand/contract) because the backend image rolls right after the migration.

Loki + Promtail (logs), cAdvisor + Prometheus (container metrics), and Grafana run alongside the app. Promtail discovers every container through the Docker socket and ships its stdout/stderr to Loki with labels attached automatically; cAdvisor reads host cgroups for per-container CPU/memory/network and Prometheus scrapes it.

LogQL cheat sheet

# All backend logs
{compose_service="backend"}

# Backend errors only
{compose_service="backend"} | json | level="error"

# A single request flow (paste a request_id)
{compose_service="backend"} | json | request_id="<uuid>"

# All ambasdr services
{compose_project="ambasdr"}

# Knowledge graph errors and exceptions
{compose_service="knowledge-graph"} |~ "(?i)error|exception"

dev keeps its data in-box, so a small daily cron is the safety net. prod leans on managed backups: Managed Postgres has automated daily backups + point-in-time recovery, and Droplet backups are enabled in prod.tfvars; only Neo4j (still in-box on prod) needs the cron dump.

# /etc/cron.daily/ambasdr-backup   (dev box)
#!/bin/bash
set -e
DEST=/data/ambasdr/backups/$(date +%Y%m%d)
mkdir -p $DEST

# Postgres logical dump (both ambasdr and ambasdr_knowledge_graph).
docker compose -f /opt/ambasdr/docker-compose.do-dev.yml \
  exec -T postgres pg_dumpall -U ambasdr | gzip > $DEST/postgres.sql.gz

# Neo4j online dump.
docker compose -f /opt/ambasdr/docker-compose.do-dev.yml \
  exec -T neo4j neo4j-admin database dump neo4j --to-stdout | gzip > $DEST/neo4j.dump.gz

# MinIO bucket mirror (dev only; prod documents live in Spaces).
docker compose -f /opt/ambasdr/docker-compose.do-dev.yml \
  exec -T minio mc mirror --overwrite local/ambasdr-documents $DEST/minio/

# Retention: keep last 14 days.
find /data/ambasdr/backups -maxdepth 1 -mindepth 1 -type d -mtime +14 -exec rm -rf {} +

Day-to-day operation is hands-off. If the Droplet reboots, the full stack comes back automatically: every service declares restart: unless-stopped, Docker's systemd unit is enabled, and cloud-init re-asserts data-dir ownership on each boot.

Manual recovery

ssh ambasdr-dev
sudo systemctl status docker           # is Docker up?
cd /opt/ambasdr
docker compose -f docker-compose.do-dev.yml --env-file .env.host up -d   # idempotent
docker compose -f docker-compose.do-dev.yml --env-file .env.host ps

prod is built later by cloning dev. The infra is ready (prod.tfvars flips on Managed Postgres + Spaces and uses the apex domain); the app side needs a prod compose variant and a prod deploy workflow. Below is the gap list, stated plainly so nothing is assumed done.