Every issue we hit during the staging deploy and every fix we apply going forward. Consult this first when prepping the DigitalOcean cutover — most of these need to be re-verified or re-applied in the production environment, and several are configuration steps that can't be inferred from the code alone.
Each row in §02 captures one issue we hit during the GMKtec staging deploy: what failed, why, the fix we applied for staging, and the action required in production. Three columns matter most when prepping the prod cutover:
§03 collects production-only concerns that don't have a staging analogue — TLS, monitoring, real secrets management, backup strategy. These are net-new for prod and need scoping before cutover, not after.
| Issue | Staging fix | Prod action | Verify |
|---|---|---|---|
go build in the second RUN of backend/Dockerfile intermittently re-fetched github.com/juelz111/gommon and failed because the secret mount + git config rewrite were scoped to the first RUN. BuildKit's cache mount usually carries resolved modules across RUNs but the GHA cache backend occasionally drops entries between layers. |
Mounted the github_token secret in the build RUN as well, with the same git config insteadOf set + unset inside the RUN so the credential never lands on disk in the final image. |
Same Dockerfile in prod; no new prod action. When adding any future build stage that touches modules (e.g. a separate go vet or go test stage), give it the same secret mount + git config rewrite. |
Clean local build with docker buildx build --no-cache succeeds. CI build green on a fresh GHA runner. |
| KG ingestion ran on every agent instruction regeneration even when no underlying content changed. LLM extraction is non-deterministic, so each re-ingest produced slightly different edges; Graphiti's dedup expired the prior edges. Surfaced as visually-isolated nodes in the UI; spent half a session diagnosing the downstream symptoms. | Added content-hash gate to KnowledgeService.IngestPageData (migration 000016_add_kg_content_hash). ComputeKGContentHash takes the exact set of fields that feed buildConsolidatedEpisode, hashes them deterministically (canonical order for docs/links/user_types), and stores the result on agent_configs.kg_content_hash on successful ingest. Subsequent calls short-circuit when the hash matches. ForceIngestPageData bypasses for the operator CLI. Failed ingests preserve the old hash so retries see the mismatch. 14 unit + 6 integration tests cover hash determinism, field sensitivity, the no-agent-config-leak guarantee, and gate behavior. |
Same code shipping to prod. The only operational difference: any new content type added to the consolidated episode body needs a matching entry in ComputeKGContentHash AND a bump of kgContentHashSchemaVersion (currently v1) — otherwise the gate will silently skip ingests that should have run. Both writes covered by the existing unit tests. |
Trigger agent regeneration with no upstream content change; verify backend logs show "KG ingestion skipped — content unchanged" and Graphiti receives no episode. |
Stripe was double-delivering every event because two webhook endpoints existed for the same URL. The backend's single STRIPE_WEBHOOK_SECRET matched one endpoint's signing secret and rejected the other's, producing a spurious 400 alongside every successful 200. Logs filled with "invalid stripe webhook signature" even though billing actually worked. |
Listed endpoints with stripe webhook_endpoints list, identified the stale one by created timestamp + missing event types, deleted it via stripe webhook_endpoints delete <id> (interactive — pipe yes to confirm). The remaining endpoint matches the backend's signing secret. |
Audit Stripe webhook endpoints before every cutover. Either delete every endpoint and create a single fresh one in the target environment, or maintain a per-environment naming convention (e.g. tag with description "ambasdr-prod"). The backend's single-secret design assumes one endpoint per environment. | stripe webhook_endpoints list returns exactly one endpoint per environment with the expected URL. |
Stripe webhook URL in early staging docs was /v1/billing/webhook, but the backend route is actually POST /webhooks/stripe. The staging webhook endpoint was created at the wrong URL and silently 404'd until a doc-audit caught it. |
Updated the Stripe webhook endpoint URL in place via stripe webhook_endpoints update we_… --url https://ambasdr-api.ngrok.app/webhooks/stripe (signing secret preserved). Fixed all stale references in the deployment runbook, production-readiness doc, and architecture page. Added a NOTE callout in the API reference. |
When creating the production webhook, use https://api.ambasdr.com/webhooks/stripe. Always cross-reference webhook URLs against backend/internal/handler/router.go — that file is the only source of truth for the actual mounted paths. |
Trigger a test event from the Stripe dashboard or CLI (stripe trigger customer.subscription.created) and confirm the backend logs the event handler running, not a 404. |
Backend CORS middleware rejected the staging frontend origin with 403 on every preflight. Hardcoded to localhost in dev, ambasdr.com in prod, nothing for anything in between. |
Added ALLOWED_ORIGINS env var (CSV) overlay in cmd/api/main.go, env-driven via ServerConfig.AllowedOriginsCSV. Set ALLOWED_ORIGINS=https://ambasdr.ngrok.dev in .env.backend. |
Don't need ALLOWED_ORIGINS at all if the prod frontend lives at https://ambasdr.com — the existing ENVIRONMENT=production default covers it. Set it explicitly if you also want to allow a www subdomain, a marketing site origin, or a staging promotion. |
curl -sI -X OPTIONS https://api.ambasdr.com/v1/me -H "Origin: https://ambasdr.com" -H "Access-Control-Request-Method: GET" → 204 with Access-Control-Allow-Origin: https://ambasdr.com. |
| Auth0 staging SPA failed with "Client … is not authorized to access resource server" before the post-login Action ran. Missing client-grant to the API. | auth0 api post client-grants with client_id = staging SPA id, audience = https://api.ambasdr.com, empty scopes. Required create:client_grants CLI scope refresh. |
Same approach, different SPA. Every new SPA Application created in Auth0 needs an explicit client-grant to the API before any login can mint an access token. The prod SPA needs the same grant. Do this at SPA creation time as a checklist item. | Visit the prod sign-in URL in an incognito window. Should not see "Client … is not authorized" in the redirect URL or Auth0 dashboard logs. |
lk agent create rejected the agent build with "package livekit-agents version >=1.0.0 is too old, please upgrade to 1.2.0". |
Bumped livekit-agents and all plugin pins to >=1.2.0 in livekit-agent/requirements.txt. Did not bump livekit itself — that package caps at 1.1.x. |
Already merged to main; new prod deploys inherit it. Pin floor will continue to drift; track LiveKit Cloud's "supported version" notes when upgrading agent code. | grep "livekit-agents" livekit-agent/requirements.txt shows a 1.2+ floor on the deploy commit. |
Deploy steps on the self-hosted runner pulled ghcr.io/ambasdr-knowledge-graph:staging (missing the namespace) and 400'd. Cause: the workflow's top-level env: REGISTRY: ghcr.io exported as a job-level env var; docker compose --env-file defers to OS env vars when both define the same key, so ${REGISTRY} in the compose file resolved to ghcr.io (no namespace) instead of ghcr.io/juelz-ai from .env.host. |
Removed the unused REGISTRY from the workflow's top-level env. Inlined the literal ghcr.io in the three docker/login-action blocks. Added an inline comment explaining the precedence trap. |
Same constraint in prod. Never set a workflow env var with the same name as a compose substitution variable, OR override it explicitly inside each deploy step. | A workflow run deploys and the GMKtec pulls the correctly-namespaced image; docker compose config on the GMKtec shows the resolved image strings include the namespace. |
GHCR push from CI failed with 403 Forbidden on ghcr.io/juelz111/ambasdr-backend. Original local pushes used the juelz111 user namespace (because gh auth token belonged to that account), but the GitHub Actions auto-provisioned GITHUB_TOKEN can only push to packages under the repo owner's namespace — juelz-ai. |
Moved package namespace to ghcr.io/juelz-ai/ in the workflow's IMAGE_NAMESPACE env and in the GMKtec's .env.host REGISTRY value. The two old packages under ghcr.io/juelz111/ can be deleted from the GitHub user's packages page once nothing references them. |
Same namespace (ghcr.io/juelz-ai/) in prod — that's the correct pattern (packages live in the same org as the repo). No further action. |
A workflow_dispatch backend build pushes successfully to ghcr.io/juelz-ai/ambasdr-backend:staging and the GMKtec pulls it. |
gh secret set NAME --body - (stdin) silently truncated the GOMMON_PAT to 1 character when piped via gh auth token | tr -d '\n' | gh secret set --body -. Length verification via a temp diagnostic workflow step revealed the secret stored as length=1 (the trailing flag in --body -, not the piped content). |
Re-stored using gh secret set NAME --body "$(gh auth token | tr -d '\n')" — shell command substitution does the right thing. Length and SHA256 prefix now match the local token exactly. |
Avoid the --body - stdin variant entirely. Always use --body "$VALUE" with command substitution. Document this in any prod secrets-rotation runbook. |
Diagnostic workflow step (length + sha256 prefix of the stored secret) compared to the local fingerprint; they match. |
lk agent deploy in CI returned 401 Unauthorized from LiveKit Cloud even with valid project API key/secret. lk agent deploy requires account-level cloud credentials (set up by lk cloud auth, browser flow, user-bound) — the project key/secret pair we use for token signing doesn't have the agent-management scope. No service-account credential exists today. |
Removed the deploy-livekit-agent job from .github/workflows/staging.yml. Agent is deployed manually from a dev machine where lk cloud auth is set up: cd livekit-agent && lk agent deploy. The smoke job's dependency list was updated to match. |
Track LiveKit's roadmap for service-account credentials. If they ship one, re-enable the job and replace LK_CLOUD_API_KEY/LK_CLOUD_API_SECRET with the service-account values. Until then, accept manual agent deploys — they're infrequent and an extra friction-step is fine. |
Periodically check https://docs.livekit.io/cloud/agents for "CI / service account" mentions; or open a feature request. |
docker/build-push-action's multi-line secrets: parser silently mangled the GOMMON_PAT value (auth failed in CI even though the same token worked locally). The parser splits on newlines and the secret value's stored encoding subtly broke the round-trip. |
Switched to secret-files: — workflow writes the secret to /tmp/gommon_pat via printf '%s' (no trailing newline) before the build, then points BuildKit at the path. Bypasses the multi-line parser entirely. |
Apply the same pattern in prod for any sensitive value handed to docker/build-push-action. Treat the secrets: input as "newline-fragile" — prefer secret-files: by default. |
The build-backend job in production runs green end-to-end with secret-files in use; a synthetic test with a newline-padded secret in staging proves the parser isn't tripped. |
GitHub Actions build-backend failed with "Authentication failed for https://github.com/juelz111/gommon/" even though the local build worked with the exact same token. Cause: gh auth token | gh secret set --body - stores the trailing newline that gh auth token emits, and the Dockerfile interpolated the secret directly into the git URL — producing a literal x-access-token:gho_xxx\n@github.com/, which GitHub rejects. Local shell command substitution strips the newline, so the failure only appeared in CI. |
Dockerfile now reads the secret via tr -d '\r\n[:space:]' before substituting, making the build robust to however the secret was stored. The repo secret was also re-stored with gh auth token | tr -d '\n' as a belt-and-suspenders fix. |
Same Dockerfile pattern applies in prod. When storing any token-shaped secret via gh secret set --body -, always pipe through tr -d '\n' first, or use --body "$(...)" which strips trailing newlines via shell command substitution. |
echo -n "" | gh secret list -R <repo> + a manual smoke-build verifies tokens with newlines are tolerated. |
GitHub Actions build-backend job failed with "Repository not found" on the same github.com/juelz111/gommon dependency. The auto-provisioned GITHUB_TOKEN is scoped to Juelz-AI/ambasdr and can't reach a different account's repo. |
Added a GOMMON_PAT repo secret holding a token from the Juelz111 account (currently sourced from gh auth token, which has classic repo scope on that user). Updated the workflow's backend build step to pass GOMMON_PAT as the github_token BuildKit secret instead of the auto token. |
Rotate to a fine-grained PAT before prod. The current secret is a broad-scoped user token tied to a session; if it gets revoked the entire CI pipeline breaks. Create a fine-grained PAT on the Juelz111 account with only Contents: Read on juelz111/gommon, no other repo or scope. Replace the GOMMON_PAT value. Document an annual rotation reminder. |
Triggering a backend build via gh workflow run staging.yml succeeds. The Settings → Secrets page lists GOMMON_PAT with the rotation date noted in its description. |
Backend Docker build failed to download github.com/juelz111/gommon with "could not read Username for 'https://github.com': terminal prompts disabled". Private repository. |
Backend Dockerfile uses BuildKit secret mount for GITHUB_TOKEN + git config url.insteadOf rewrite. Build command sets --secret id=github_token,env=GITHUB_TOKEN. |
Already handled in backend/Dockerfile. The GitHub Actions workflow passes the workflow's GITHUB_TOKEN automatically; for any manual prod build, source the token the same way. |
Re-build the backend image locally with GITHUB_TOKEN=$(gh auth token) docker buildx build --secret id=github_token,env=GITHUB_TOKEN … against a fresh cache; expect success. |
GHCR push failed with insufficient_scope. Default GitHub PAT lacked write:packages. |
Refreshed CLI auth: gh auth refresh -h github.com -s write:packages,delete:packages. The browser flow has a confirmation step that's easy to miss. |
Workflow uses the auto-provisioned GITHUB_TOKEN with permissions: packages: write at the job level — no manual scope management needed in prod. Document the dashboard path "Repository → Settings → Actions → Workflow permissions → Read and write" so future contributors can't silently downgrade it. |
Settings → Actions → General → Workflow permissions shows "Read and write." A no-op manual workflow run with docker login ghcr.io succeeds. |
Frontend Docker build failed with "Cannot find module '@ambasdr/tokens'" when context was ./frontend. |
frontend/Dockerfile built from the repo root with -f frontend/Dockerfile .. The Dockerfile COPYs both frontend/ and packages/. |
Same Dockerfile + same root-context build command for prod. If the prod build is invoked from a different working directory (e.g., a different CI tool), the -f frontend/Dockerfile . pattern must travel. |
Manual docker buildx build -f frontend/Dockerfile -t test:local . from the repo root succeeds. |
Signup flow could hard-fail when the Auth0 Post-Login Action's fetch to /internal/users/provision errored. User would end up with a valid Auth0 session but no Ambasdr user_id custom claim, and every /v1/me call would 401. Manual make dev-reset-user EMAIL=… was the only escape hatch. |
Added UserService.EnsureFromIdentity with a three-step lookup chain (user_id claim → auth0_sub fallback → lazy create from JWT). resolveUser and the M2M provision handler both delegate to it; the Auth0 Action becomes a fast-path optimization rather than a hard dependency. Soft-deleted users are detected and refused (no silent resurrection). |
Already merged to main; ships with the backend image. No prod action needed unless we change the Identity contract — the integration tests in user_handler_test.go cover the lazy-create / stale-id / soft-deleted-refusal cases and would catch a regression. |
A fresh JWT without the custom claim (mint via mintTestJWTWithoutCustomClaims) returns 200 on first /v1/me; a soft-deleted-user JWT returns 404 with "user account has been deleted". |
Auth0 Post-Login Action's AMBASDR_API_URL and AMBASDR_M2M_SECRET secrets are tenant-wide — they protect the call to POST /internal/users/provision. We reused the dev tenant's values for staging because both backends share the same ngrok domain. |
No staging-specific change. Staging reused dev's AUTH0_M2M_SECRET in .env.backend; the Action's secret didn't need to be rotated. |
Different in prod. Production should have a separate Auth0 tenant (or at minimum a distinct Action targeting a distinct PROVISION_URL). Otherwise prod and staging share a single Action with one URL and one secret. Either: (a) create a new tenant for prod and clone the Action there, or (b) make the existing Action conditional on event.client.client_id and branch URL/secret per environment. |
Sign up a fresh email in prod → Auth0 Action logs show only one POST, to the prod URL (not staging's ngrok URL). |
Backend startup log emits unable to load embedded .env, falling back to environment variables as a warning. |
Documented as cosmetic — the embedded .env.example is empty/sample, runtime config flows through env vars overlay. |
Either fix at source (silence the warning when ENVIRONMENT=staging/production) or accept it. Recommendation: silence — it creates noise in monitoring and Datadog/log-search false positives. |
After fix: prod startup logs contain no unable to load embedded entries. |
backend/.env.example was missing KG_SERVICE_URL, KG_TIMEOUT_SECONDS, and the AGENTGEN_* family. Easy for a new engineer to skip them and ship a backend that can't talk to Graphiti or has wrong LLM tuning. |
Added the missing vars to backend/.env.example with documented defaults. |
Already committed. Re-check after each new env var added to config.go — there's no test catching this. |
diff <(grep -oE "env:\"[A-Z_]+" backend/internal/config/config.go | sort -u) <(grep -oE "^[A-Z_]+=" backend/.env.example | sort -u) reports no missing keys. |
curl -sI against /health returns 404 even when the backend is fine — Gin's r.GET() doesn't auto-bind HEAD. |
Documented inline in the deployment doc verification commands. Use curl -s (real GET) for the backend health probe. |
If prod uses an external monitoring tool that issues HEAD probes (most do by default, e.g. UptimeRobot), either point it at the actual GET endpoint or wire a HEAD handler. Cleaner long-term fix: register HEAD explicitly in router.go. |
Monitoring tool reports /health as 200, not 404. If it doesn't, register HEAD. |
Postgres bind-mount under /data/ambasdr/postgres was created by root via sudo mkdir. The postgres container chowns the dir on first init, but a reboot that recreates the path with wrong ownership breaks the container. |
Documented in the recovery section. Manual chown -R 999:999 recipe in §17. |
DigitalOcean Managed Postgres removes the issue entirely. If we keep bind-mounted Postgres on a Droplet, the same recipe applies; if we migrate to managed, drop the postgres/minio-init services from the compose file (per §18). | N/A if migrating to managed DB. |
Dockhand mounts /var/run/docker.sock but shows "No environments" until you explicitly add the host via Settings → Environments. Easy to assume it's broken. |
Documented in §17. Added GMKtec environment via the UI. | If we run Dockhand in prod (recommended for any always-on Droplet), do the same Add Environment step at install time. | Dockhand Containers tab shows the running prod containers. |
Items that staging didn't exercise — either because the local LAN sidesteps them (real DNS, TLS) or because the iteration loop didn't need them (monitoring, real backups). These need scoping before cutover, not after.
| Concern | What it covers | Suggested approach | |
|---|---|---|---|
| Frontend error tracking (Sentry) | Loki captures every container's stdout/stderr in staging, but browser-side JavaScript errors don't reach any container. A user hitting a TypeError in React, an Auth0 callback failure, or a CORS preflight regression today produces no signal we can observe. | Add @sentry/react to the frontend (one Sentry.init in main.tsx, source maps uploaded at build time). Free tier handles staging volume; pay tier (~$26/mo entry) handles prod. Decision before cutover: separate Sentry projects per env, or one project with environment tagging. Also add backend Sentry SDK for unhandled panics that escape the gin recovery middleware. |
Trigger a deliberate frontend error in staging and see it appear in the Sentry issues feed within ~30 s; trigger a backend panic and confirm a Sentry event lands. |
| Real DNS + TLS | Browsers, Stripe, Auth0, LiveKit Cloud all require a stable HTTPS endpoint. ngrok is staging-only. | Register ambasdr.com with the relevant registrar; point A/AAAA at the Droplet (or App Platform CNAME). TLS handled by Let's Encrypt via certbot if hosting nginx ourselves, automatic on App Platform. |
|
| Auth0 production tenant | Dev tenant has the "You are using Auth0 development keys" warning on every login — Google Cloud's OAuth consent screen is in test mode. Prod needs its own tenant with real Google credentials. | Create a new Auth0 tenant for prod. Configure Google connection with production Google Cloud OAuth credentials. Recreate the SPA Application and Post-Login Action. Update all secrets (M2M, provisioning URL). | |
| Stripe live mode | All staging Stripe activity is test mode. Prod needs live keys + live webhook endpoint. | Get live sk_live_… + create live webhook → https://api.ambasdr.com/webhooks/stripe. Activate the Stripe account (bank, business info, identity verification). Backend env on prod uses live values; staging stays test. |
|
| LiveKit Cloud production project | Currently staging reuses the dev LiveKit project. Prod should be isolated — separate billing line, separate quotas. | Create a new LiveKit Cloud project for prod. Deploy a separate agent (lk agent create) with prod's AMBASDR_SERVICE_URL=https://api.ambasdr.com + a freshly rotated AGENT_API_KEY. Keys flow into the prod backend's env. |
|
| Secrets management | Staging puts secrets in plaintext .env files at /opt/ambasdr/.env.* on the GMKtec, mode 600. Workable for one box; doesn't scale and offers no audit trail or rotation. |
DigitalOcean App Platform supports encrypted env vars in the spec. For Droplet, either (a) keep plain .env on disk with stricter ownership, (b) move to Doppler / 1Password Secrets Automation, or (c) GitHub OIDC → AWS Parameter Store / Secrets Manager. Decide before cutover. |
|
| Managed Postgres vs self-hosted | Staging uses a bind-mounted postgres container. Acceptable for one engineer; not acceptable for prod where you want managed backups, PITR, and version upgrade flow. | DigitalOcean Managed PostgreSQL (~$15/mo entry). Backend's existing DB_HOST/DB_PORT/DB_PASSWORD env vars accept any host, so the only change is the connection string. Drop the postgres service + the init script + the bind-mount from the prod compose file. |
|
| Managed Spaces (S3) vs MinIO | Staging uses MinIO container. Prod should use DO Spaces for durability, lifecycle policies, CDN. | Spaces endpoint + access keys. Backend's DO_SPACES_* env vars already model the Spaces interface (we point them at MinIO in staging via DO_SPACES_ENDPOINT=http://minio:9000). Flip endpoint + keys for prod. |
|
| Neo4j hosting | Staging runs Neo4j 5 community in a container with bind-mount. Acceptable for small graphs; vertical-scale ceiling is low. | Either (a) keep containerized on a larger Droplet (cheap, capped), or (b) Neo4j AuraDB (managed, more expensive, scales). Decide based on graph size when we're closer to cutover. | |
| Monitoring + alerting | Staging has zero monitoring beyond docker logs. Prod needs at minimum: uptime check, error-rate alert, DB / queue depth dashboards. |
Minimal viable: Sentry for errors (backend + frontend SDK), Better Uptime / UptimeRobot for liveness, DigitalOcean monitoring for Droplet CPU/disk. Optional layer 2: Datadog or Grafana Cloud once spend justifies it. | |
| Backup + restore strategy | Staging has the cron snippet in §16 (logical Postgres dump, Neo4j dump, MinIO mirror). Never tested for restore. | DigitalOcean Managed Postgres handles daily backups + PITR automatically. For Neo4j and Spaces, automate a weekly snapshot + a documented monthly restore drill. "Backups you haven't restored aren't backups." | |
| Rate limiting + abuse controls | Staging has no rate limiting. The backend's chat / KG endpoints are LLM-backed (cost-sensitive) and the public page can be visited by anyone. | Add rate limiting middleware (per-IP for public endpoints, per-user for authenticated). Either at the application layer (gommon middleware) or upstream (Cloudflare / nginx limit_req). Calibrate against expected legitimate volume. |
|
| CI on PRs (not just deploy on main) | The staging workflow only runs on push to main. PRs don't get tested. | Add a separate .github/workflows/ci.yml that runs go test -tags unit, npm test, frontend tsc --noEmit, and a build-only step (no push) on PR open / sync. Block merge until green. |
|
| Production-grade observability inside the backend | Structured logs already in place via zerolog. No request-level metrics, no traces. | Wire OpenTelemetry SDK to backend (gin middleware + outgoing HTTP client). Export to Datadog / Honeycomb / Grafana Cloud. Frontend RUM via Sentry / Datadog. | |
| Database migration strategy under load | Staging migrations run cold-blooded — backend is bounced after migrate. Prod needs zero-downtime: migrate before code roll, with destructive migrations split into expand / contract. | Discipline + tooling. Document the convention in backend/db/migrations/README.md (additive migrations always safe; destructive ones split into "drop next release"). Workflow already runs migrate before backend roll. |
|
| Disaster recovery runbook | No documented "site is down, what do I do" procedure. | After prod is live, document: who's on call, where the runbook lives, escalation path, communication template (status page), restore-from-backup steps. |
This page is a running list. Anyone fixing a staging-environment issue should add a row to §02 in the same PR that lands the fix. Anyone scoping a production concern should add a row to §03 before assuming it's handled.
The signal that this list is healthy is the same as for the deployment runbook: it shouldn't drift. If you've fixed something during a deploy and didn't add it here, the next person will hit the same trial-and-error you just escaped.
The full staging runbook this checklist tracks against. Section 18 of that doc has the staging-to-prod substitution table.