Better Stack observability rollout

This doc covers the Better Stack rollout shipped against the AWS dev deployment only. Production stays unchanged until we've proven signal quality. The rollout follows the project plan: app-native direct telemetry (no host collector), managed via Terraform.

What's wired

Concern	Implementation
Server logs	lib/logger/index.ts adds a @logtail/pino transport when BETTER_STACK_SERVER_SOURCE_TOKEN + BETTER_STACK_SERVER_INGESTING_URL are set. Falls back to stdout/CloudWatch in dev or when env is absent.
Browser telemetry (Web Vitals + global errors)	components/observability/better-stack-web-vitals.tsx + app/global-error.tsx send sendBeacon/fetch calls to the browser ingest URL. Both no-op without env.
Server traces (OTLP)	instrumentation.ts register() boots @opentelemetry/sdk-node with the OTLP HTTP trace exporter when BETTER_STACK_OTLP_ENDPOINT + BETTER_STACK_OTLP_SOURCE_TOKEN are set. Node runtime only.
Server error capture	instrumentation.ts onRequestError forwards to the Pino logger so errors land in both stdout and Better Stack.
Health checks	/api/health stays shallow for ALB. /api/health/deep actively probes the DB and returns 503 on dependency failure, plus opportunistically pings BETTER_STACK_WEB_HEARTBEAT_URL.
Worker dead-man switch	worker/worker.ts calls startHeartbeatLoop(process.env.BETTER_STACK_WORKER_HEARTBEAT_URL). Pings every 60s.
Redaction	lib/observability/redact.ts lists Pino redact paths covering auth headers, session tokens, secrets, challenge/passkey values, and obvious PII fields. Applied centrally in lib/logger/index.ts so every sink (stdout + Better Stack) inherits it.

Env contract

Server runtime (set via AWS Secrets Manager)

Variable	Purpose
BETTER_STACK_SERVER_SOURCE_TOKEN	Pino transport auth.
BETTER_STACK_SERVER_INGESTING_URL	Pino transport endpoint.
BETTER_STACK_OTLP_SOURCE_TOKEN	OTLP trace exporter bearer token.
BETTER_STACK_OTLP_ENDPOINT	OTLP base URL (/v1/traces is appended).
BETTER_STACK_WORKER_HEARTBEAT_URL	Worker dead-man switch ping URL.
BETTER_STACK_WEB_HEARTBEAT_URL	Optional. Deep health pings this on success.
BETTER_STACK_LOG_LEVEL	Optional. Overrides the Pino transport level for Better Stack only.

Client/build time (Docker build args + GHA repo vars)

Variable	Purpose
NEXT_PUBLIC_BETTER_STACK_SOURCE_TOKEN	Browser ingest auth (baked into JS bundle).
NEXT_PUBLIC_BETTER_STACK_INGESTING_URL	Browser ingest URL.
NEXT_PUBLIC_BETTER_STACK_LOG_LEVEL	Default info.

Set these as GitHub repository variables. The Docker build threads them through docker/build-push-action build-args, the Dockerfile exposes them as ARG+ENV in the builder stage, and next.config.ts exposes them through env: so next build inlines them into the client bundle.

Terraform

infra/terraform/dev/ owns Better Stack resources:

• versions.tf — declares BetterStackHQ/logtail and BetterStackHQ/better-uptime providers; pnpm infra:tf reads their API tokens from dev.secrets.json.appEnv and passes them through the var.betterstack_*_api_token variables for local plan/apply runs.
• variables.tf — adds betterstack_enabled, region, status-page subdomain, alert emails, and uptime target URLs.
• betterstack.tf — declares:
  • 3 telemetry sources (server logs, browser telemetry, OTLP traces) in Germany/EU region by default.
  • 3 uptime monitors (web_shallow → https://daxcha.in/api/health, web_deep → https://daxcha.in/api/health/deep, auth_login).
  • 1 worker heartbeat with a 180s period + 120s grace.
  • 1 status page (dax-dev-status.betteruptime.com) with Platform and Dependencies sections and 4 components (Web app, API, Authentication, Workers).
• outputs.tf — surfaces every token + ingest URL + heartbeat URL + status-page URL. Sensitive values are marked sensitive = true so they don't leak into terraform output without -raw.

Applying

Store the provider tokens in infra/terraform/dev/dev.secrets.json under appEnv:

{
  "appEnv": {
    "betterstack_telemetry_api_token": "...",
    "betterstack_uptime_api_token": "..."
  }
}

Leave placeholder token fields out of appEnv or keep them blank until real Better Stack Team API tokens are available. pnpm infra:tf skips blank strings so old non-empty values under the terraform object can still be used during the transition, but Better Stack must be disabled or given valid tokens before plan/apply.

pnpm infra:tf validate
pnpm infra:tf plan -out=tfplan
# Review the plan — pay attention to the betterstack_* resource attribute
# names; if the provider schema has drifted, fix `betterstack.tf` to
# match before applying.
pnpm infra:tf apply tfplan

# After apply, pipe the tokens into AWS Secrets Manager / GitHub variables:
pnpm infra:tf output -raw betterstack_server_source_token
pnpm infra:tf output -raw betterstack_server_ingesting_url
pnpm infra:tf output -raw betterstack_client_source_token
pnpm infra:tf output -raw betterstack_client_ingesting_url
pnpm infra:tf output -raw betterstack_otlp_source_token
pnpm infra:tf output -raw betterstack_otlp_endpoint
pnpm infra:tf output -raw betterstack_worker_heartbeat_url
pnpm infra:tf output -raw betterstack_status_page_url

Move the server-side values into AWS Secrets Manager and the browser values into GitHub repo variables (NEXT_PUBLIC_* Docker build args). Do not leave captured token output in local plaintext files.

If dax-dev-status.betteruptime.com is taken, override betterstack_status_page_subdomain to the next available dax-dev-status-* variant and re-apply.

Pushing outputs into AWS Secrets Manager + GitHub

Requires aws CLI with the deploy profile and gh CLI signed in with repo variable permissions.

Local verification

pnpm typecheck
pnpm lint
pnpm test
pnpm build

With the dev deployment running, hit:

• GET /api/health → 200 in < 50 ms (shallow; ALB health check).
• GET /api/health/deep → 200 when DB is reachable; 503 when not.
• Force a client error (e.g. throw inside a page component) → check Better Stack browser source for the redacted error payload.
• Run pnpm worker with BETTER_STACK_WORKER_HEARTBEAT_URL set → watch Better Stack heartbeat go green within 90s.

Confirm the proxy doesn't gate observability traffic:

• curl -i https://daxcha.in/_betterstack/test should pass through the proxy (404 from Next.js is fine — what matters is that it's not a redirect to /login).

Alerting policy

By design, alerts are quiet during rollout:

• Email-only via var.betterstack_alert_emails (empty by default).
• No SMS, no calls, no critical push.

Once a week of clean signal lands, escalate by adding recipients + enabling SMS/push on the heartbeat and web_shallow monitors only.

Redaction policy

The redaction list in lib/observability/redact.ts covers:

• authorization, cookie, set-cookie, x-api-key, x-auth-token header variants (request + response).
• Common credential field names anywhere in the payload (*.password, *.token, *.apiKey, *.secret, *.totpSecret, *.challenge, *.passkey, etc.).
• Auth.js / NextAuth shapes (session.user.id, jwt.token, session tokens).
• Obvious PII fields on request bodies (body.email, body.password).

Anything matched is replaced with [Redacted] before leaving the process. Pino redaction happens at log-record build time, so every configured sink (stdout, Better Stack) sees the redacted value.

If you find a new shape leaking, add the path to REDACTED_PATHS. Errors should never surface raw form values — app/global-error.tsx only sends error.message, error.digest, and window.location.pathname.

Rollback

To disable the integration without uninstalling:

1. Set betterstack_enabled = false in dev.tfvars and apply. Terraform will tear down all betterstack_* resources and clear their outputs.
2. Drop the BETTER_STACK_* and NEXT_PUBLIC_BETTER_STACK_* env vars from AWS Secrets Manager and the GitHub repo. App code no-ops automatically once the env is gone.
3. Set BETTER_STACK_NEXT_DISABLE=1 to skip the @logtail/next wrapper entirely (handy if it ever interferes with a local build).

If Terraform fails with Invalid Team API token while reading existing Better Stack resources, first refresh the two Team API tokens in dev.secrets.json. If the goal is a break-glass removal and no valid tokens are available, set betterstack_enabled = false and create the destroy plan with refresh disabled:

pnpm infra:tf plan -refresh=false -var betterstack_enabled=false -out=tfplan
pnpm infra:tf apply tfplan

Only use the refresh-disabled path after reviewing the AWS side separately; it intentionally skips provider refresh so Terraform can forget the unreachable Better Stack resources.

Assumptions captured by this rollout

• Better Stack data region is Germany/EU for all telemetry sources.
• Status page subdomain is dax-dev-status; if unavailable use the first free dax-dev-status-* variant — no architectural change.
• Tokens are never committed. Terraform produces them; operators inject them through AWS Secrets Manager (server) and GitHub repository variables (Docker NEXT_PUBLIC_* build args).
• Provider attribute names in betterstack.tf reflect the BetterStackHQ/logtail and BetterStackHQ/better-uptime schemas at rollout time. Treat terraform plan as the source of truth if they drift.