Accessing the container monitoring tools on the DigitalOcean dev box

Everything below runs on one machine. Pinning down which machine removes most of the ambiguity in the rest of this page.

"The dev box" is a single cloud server — a DigitalOcean Droplet named ambasdr-dev — that hosts the development / shared instance of the entire Ambasdr stack. It is not your laptop and not production; it is the always-on Linux virtual machine the team points at while building. Everything on it runs as Docker containers managed by one docker compose file at /opt/ambasdr/docker-compose.do-dev.yml: the app (backend, frontend, knowledge-graph), the data stores (postgres, neo4j, minio), the TLS entry point (caddy), the log stack (loki, promtail), the monitoring tools this page is about (grafana, prometheus, cadvisor), and the container-management UI (portainer).

Container monitoring is three cooperating programs, each its own container, defined in deploy/docker-compose.do-dev.yml alongside the existing log stack. The clearest way to understand them is by what each one hands to the next.

cAdvisor (short for "Container Advisor", made by Google) is a small program that measures how much of the machine's resources each running container is using. Concretely: the Linux kernel already tracks per-container counters — CPU time consumed, memory in use, network bytes sent and received — in special files under /sys/fs/cgroup. Every time you start a container the kernel creates a "cgroup" for it and keeps live counters there. cAdvisor reads those kernel counters (that is why its container is given read-only access to host paths like /sys, /var/run, and /var/lib/docker), turns them into per-container numbers labelled with each container's name, and publishes them on an HTTP page at cadvisor:8080/metrics. It keeps no history — it only reports the current values whenever it is asked. It does not run your app, route traffic, or change anything; it only observes.

So the full path the data travels is: kernel counters → cAdvisor reads them and exposes /metrics → Prometheus scrapes that page every 30s and stores it with timestamps → Grafana queries Prometheus and draws the graphs. cAdvisor is the first link; without it, Prometheus would have no per-container data to collect and the dashboard would be empty.

As an intuition on top of that literal description: think of cAdvisor as the meter-reader who walks around noting each container's "electricity usage," Prometheus as the logbook that records those readings over time, and Grafana as the wall chart that plots the logbook for you to read.

This is the day-to-day path, and it needs no SSH and no make target — just a browser and your @juelz.ai email.

It defaults to the last 3 hours and auto-refreshes every 30s (which matches the scrape interval — there's no benefit to refreshing faster).

At the top is a container dropdown — a multi-select with an "All" option, populated from the live container names cAdvisor has seen. Use it to focus on one service (e.g. backend, neo4j) or leave it on All. Every panel below filters on this selection.

Logs and metrics live in the same Grafana. You read container logs with LogQL through the Loki datasource — there is no separate Loki UI, and you never query Loki directly.

Open the same Grafana tunnel as above (make do-grafana → http://localhost:3001), then go to Explore (the compass icon) and pick the Loki datasource (it is the default). Promtail discovers every container through the Docker socket and ships its stdout/stderr to Loki with labels attached automatically, so any running container is queryable with no per-service setup.

The labels Promtail attaches — these are what you filter on:

Example LogQL queries (paste into the Explore query box):

# All backend logs
{compose_service="backend"}

# Backend logs containing "error" (case-sensitive substring match)
{compose_service="backend"} |= "error"

# Only error-level lines across backend + knowledge-graph
{compose_service=~"backend|knowledge-graph", level="error"}

# One specific container, dropping noisy health-check lines
{container_name="ambasdr-backend-1"} != "/health"

# Rate of error lines per service over 5m (a metric derived from logs)
sum by (compose_service) (rate({level="error"}[5m]))

Reach for Prometheus directly when you want to run an ad-hoc PromQL query, or to confirm the metrics pipeline itself is healthy.

make do-prometheus DO_HOST=root@<reserved-ip>

# CPU cores per container
sum by (name) (rate(container_cpu_usage_seconds_total[5m]))

# Working-set memory per container (bytes)
sum by (name) (container_memory_working_set_bytes)

Grafana, Prometheus, and cAdvisor tell you how much each container is using; they don't let you act on one. Portainer is the companion for that — a browser UI to see every container's state, read its logs, restart it, or open a shell — reached the same email-gated way as Grafana.

Portainer runs as one more container on the box: the portainer service (image portainer/portainer-ce:lts) in deploy/docker-compose.do-<env>.yml. It is a full Portainer Server, not the older agent: it talks to the box's Docker straight through the mounted socket (-H unix:///var/run/docker.sock), so that box's containers, stacks, volumes, and networks show up automatically as the "local" environment with no wiring. This replaced the previous model — an on-box portainer-agent reached from a Portainer Server on your laptop over an SSH tunnel — so there is now one hosted Server per environment and the laptop Server is gone.

Reaching Portainer means passing two separate logins, owned by two different systems, in order. This is the part that trips people up, so it's worth being explicit — the email PIN is not the Portainer login:

So a teammate's first visit is:

• Config lives in the repo. All of it is under deploy/observability/ (prometheus/config.yml, grafana/provisioning/datasources/prometheus.yml, grafana/provisioning/dashboards/dashboards.yml + containers.json) and the services are in deploy/docker-compose.do-dev.yml.
• The box must use Docker's overlay2 storage driver. cAdvisor cannot read per-container metrics under Docker's newer overlayfs (containerd-snapshotter) image store — the default on Docker 28/29 — because the on-disk layer records it reads don't exist in that layout, so it reports zero containers and the dashboard stays empty. The box is pinned to overlay2 via /etc/docker/daemon.json ({"features":{"containerd-snapshotter":false},"storage-driver":"overlay2"}), which is codified in the ambasdr-infra cloud-init so a rebuilt droplet keeps it. If container panels are empty across all containers, check the storage driver first with docker info | grep Storage.
• Pushing changes to the box. Edit the files, then make do-deploy DO_HOST=... (which rsyncs deploy/ and restarts the stack). Grafana re-reads provisioning on boot; the dashboard provider also re-scans every 30s.
• Tailing a container the dashboard flagged. make do-logs DO_HOST=... SVC=backend tails one service (omit SVC for all).
• Deliberately light. The box is s-2vcpu-4gb and already tight with Neo4j, so cAdvisor runs with most metric families disabled and Prometheus keeps only 7 days / 1 GB. If you need deeper history, raise --storage.tsdb.retention.time / --storage.tsdb.retention.size in the compose file after resizing the Droplet.
• No inbound port was opened. Grafana is reachable at grafana-dev.ambasdr.com only because the cloudflared container dials out to Cloudflare; the box's firewall still accepts nothing new inbound. Prometheus and cAdvisor stay on the loopback, reachable only through the SSH tunnels above. Loki and Prometheus are intentionally not routed through the tunnel — Grafana is the single front door.
• The tunnel token is a per-box secret. CLOUDFLARED_TUNNEL_TOKEN (and GRAFANA_DOMAIN) live in /opt/ambasdr/.env.host (mode 600, never committed). The dev box uses the ambasdr-dev Cloudflare tunnel; the matching public hostname route and the email-PIN Access policy are configured in the Cloudflare Zero Trust account, not in the repo.
• Prod uses the same pattern. A second tunnel ambasdr-prod and Access app already publish grafana.ambasdr.com (one level, so Cloudflare's free Universal SSL covers it). It activates once a prod box runs cloudflared with the prod tunnel token in its own .env.host. The same @juelz.ai email-PIN policy applies.