This page documents the observability stack in docker-compose.observability.yml and its config under observability/. Everything on this page has been run and verified end-to-end (real metrics scraped, real traces linked across HTTP→Redis spans, real nginx/container logs in Loki) — this isn’t a config sketch.

​
Architecture

Design decisions worth knowing:
  • Metrics are pulled, not pushed. Prometheus scrapes the API’s own GET /metrics directly (prom-client), rather than routing metrics through the OTel Collector. This is the standard, simplest split — the collector’s job here is traces only.
  • Traces go through the Collector, not straight to Tempo, even though Tempo has its own OTLP receiver and would accept them directly. Routing through the collector buys batching, a memory limiter (so a traffic spike can’t OOM the trace pipeline), and a place to add processors later (sampling, PII scrubbing) without touching app code.
  • Logs are scraped from Docker, not shipped via OTLP. Promtail uses Docker service discovery (mounts the Docker socket read-only) to tail every container’s stdout — simpler and more battle-tested than OTel’s logs pipeline, which is comparatively new. src/instrumentation.ts’s OTel pino auto-instrumentation still injects trace_id/span_id into every log line, so Loki↔Tempo correlation works without needing logs to physically transit the collector.

​
What each component actually does here

​
Prometheus (observability/prometheus/)

Scrapes 9 targets (prometheus.yml): the API itself, node-exporter (host CPU/memory), cAdvisor (per-container CPU/memory), postgres-exporter, redis-exporter, nginx-exporter, the OTel Collector’s self-telemetry, and itself/Alertmanager. alert_rules.yml defines 10 rules across four groups — availability (FaceTrustApiDown, DependencyDown), errors (HighHttpErrorRate, ElevatedLivenessFailureRate), latency (HighIdentifyLatency, HighVectorSearchLatency), and infra (HighMemoryUsage, EventLoopLagHigh, PostgresDown, RedisDown).
No RabbitMQ or any message broker exists in this system (see Architecture — everything is synchronous HTTP + Postgres + Redis), so there’s no RabbitMQ exporter here despite it being a common pairing with Redis in generic observability-stack checklists.
The API’s own metrics (src/observability/metrics.ts) split into two families:
  • Generic HTTP (facetrust_http_requests_total, facetrust_http_request_duration_seconds) — every request, labeled by method/route/status_code, recorded by src/middleware/metrics.ts. route uses the matched Express route pattern (/users/:id), not the raw URL, so per-user cardinality never explodes the metric.
  • Business-specific — facetrust_otp_requests_total/facetrust_otp_verify_total (by outcome), facetrust_verify_face_total (by registry_type+outcome — the same outcome vocabulary as the API itself: match/face_mismatch/liveness_failed/cross_registry_mismatch), facetrust_link_total, facetrust_identify_total, facetrust_verify_self_total, plus latency histograms for the liveness stage, the HNSW vector search, and face-engine calls specifically — the same three stages Architecture’s per-stage tracing story is about.

​
Loki + Promtail (observability/loki/, observability/promtail/)

Promtail discovers every container via the Docker socket and tails its logs — this is what covers “Docker logs” generically. Two containers get extra structure via pipeline_stages in promtail-config.yaml: the API’s pino JSON logs get level/msg/req.url promoted, and nginx’s combined-format access logs get method/status parsed out as labels.
Promtail’s Docker discovery only sees containerized services. During local development the API normally runs via npm run dev on the host (not the api Docker service, to avoid a port-3000 clash with the two running simultaneously — see Quickstart) — so its logs go to your terminal, not Loki, unless you run the containerized api service instead. nginx’s logs (proxying to whichever API instance is actually listening) always land in Loki regardless, since nginx itself is always containerized.

​
Tempo + OTel Collector (observability/tempo/, observability/otel-collector/)

src/instrumentation.ts is preloaded via -r ./src/instrumentation.ts in package.json’s dev/start scripts — it has to load before Express/pg/ioredis/axios are required, or auto-instrumentation can’t patch them. It auto-instruments HTTP, Express, the Prisma’s underlying pg driver, ioredis, and outbound axios calls (to face-engine and the identity registry), and exports via OTLP/HTTP to the collector. Verified in this session: a single POST /auth/otp/request call produced a 19-span trace — the full Express middleware chain, then seven distinct ioredis spans (evalsha, ttl, multi, set×2, del, exec) each with real latency, corresponding exactly to the cooldown-check → code-generate → multi-set → OTP-send sequence in modules/auth/service.ts. This is the concrete answer to “identifying where a slow request originates” — no guessing which Redis call in a multi-step flow is slow, it’s a span.

​
Grafana (observability/grafana/)

Auto-provisioned (provisioning/datasources/datasources.yml) with all four backends plus Alertmanager — anonymous admin access is enabled for local dev only (see the warning in docker-compose.observability.yml, disable before this runs anywhere reachable outside localhost). Two starter dashboards auto-load into a “FaceTrust” folder:
  • API Overview — request rate by status, 5xx error rate, p50/p95/p99 latency by route, dependency reachability, process memory, event-loop lag, face-engine call latency, vector search latency.
  • Business Metrics — OTP outcomes, verify-face outcomes by registry type, links, /identify outcomes, /verify/self outcomes, liveness check duration by surface/mode.
Loki↔Tempo correlation is wired both directions: a log line’s trace_id becomes a clickable “View Trace” link (Loki datasource’s derivedFields), and a trace can jump to its surrounding logs (Tempo datasource’s tracesToLogsV2).

​
Alertmanager (observability/alertmanager/alertmanager.yml)

Routes severity="critical" alerts to both PagerDuty and a dedicated Slack channel; severity="warning" to Slack only (no page). An inhibit rule suppresses warning-level noise while FaceTrustApiDown is already firing. Every credential in this file is a placeholder (CHANGE_ME) — supply real ones via a .env-style secrets file mounted over the config, or a secrets manager, at actual deploy time; nothing here should ever hold a live SMTP password or Slack webhook URL in version control.

​
Running it

(Deliberately excludes the containerized api service in this command — run the API via npm run dev instead, so it isn’t fighting the compose file’s own api service for host port 3000. See the warning above.) .env’s OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4320 points a host-run API at the collector’s host-mapped OTLP/HTTP port (4320, offset from the container-internal 4318 since Tempo’s own directly-exposed OTLP port already claims host 4318). If the collector isn’t running, the OTel SDK just fails to export silently in the background — it never blocks startup.