▸ prd.vault

vault

12 — Environments

docs/12-Environments.mdtype: architecture-targetupdated: 2026-11-04

Environments

Three environments, each with its own Terraform root over the shared infra/terraform/modules/platform: dev/ serving daxcha.in, beta/ serving beta.daxcha.in, and prod/ serving live.daxcha.in. They share the daxcha.in zone without overlapping any record. Diagrams: 17-System-Design.

Current MVP default AWS region is eu-north-1 (Stockholm). That is a region, not an availability zone; the single-AZ dev posture means one AZ inside eu-north-1.

Local and Dev

Two names that used to share APP_ENV=development and are now kept apart:

  • local is a developer's laptop. It runs dax against the fakes and the devtools console in bullder/daxdev (signed fake webhooks, log-in-as, seed). APP_ENV=local. Local-only conveniences (simulated conversions, mock pricing) key on local and test and nothing else.
  • Dev is the deployed AWS environment at daxcha.in. It runs the production build against the vendors' own test services (Sumsub sandbox, Fireblocks testnet, Mercuryo sandbox, Elliptic, the CEX.IO feed, Resend). APP_ENV=dev. Nothing from daxdev runs in Dev: no fakes, no console, no seed.
APP_ENV Where
local a developer's laptop
test test runs, CI included
dev AWS Dev, daxcha.in
beta AWS Beta, beta.daxcha.in
live AWS Prod, live.daxcha.in

development is the old shared value. It is still accepted while Dev is moved to dev, but it counts as deployed: it never enables local-only behaviour, and env validation warns when it sees it. Replace it with local in .env.local. It will be removed once Dev runs APP_ENV=dev (phase 7).

The three environments

Env Purpose HA DR Provider mode
Dev Integration and product validation on the production build Single AZ None Vendor test services (sandbox, testnet); mock only where a sandbox is not wired yet
Beta Pre-production validation: Live's topology with one of everything, on mainnet, for internal testers Single AZ None Live, production contracts, test funds capped by policy
Prod Live regulated operations Multi-AZ DB DR to secondary region Live

Beta is Prod's architecture at Dev's redundancy: every release is deployed there first and exercised against the real providers before it reaches Live, without paying for a second production estate. See Where each control is validated.

Principles

  1. Dev stays cheap on purpose. Reduced footprint, single AZ, no DR.
  2. Strict isolation. Secrets, provider credentials, data, and network boundaries never cross environments.
  3. Promotion is one-way. Dev → Beta → Prod. No back-promotion of data.
  4. Rehearse in Beta, not in Prod. Anything that depends on the production shape — private subnets, NAT egress, the tag-triggered release — is exercised in Beta first. What Beta cannot hold, because it runs one of everything, is listed explicitly in Where each control is validated rather than assumed away.

Dev

Profile

  • Single Availability Zone.
  • No HA / DR requirement.
  • Smaller compute + DB sizing.
  • Lower logging volume.
  • Reduced redundancy.

What's wired

  • Route 53 + public ALB + regional AWS WAF (lightweight rule set).
  • API Gateway, internal ALB.
  • App + worker containers on EC2 (smaller instance).
  • Postgres single-instance (or simplified topology).
  • S3, Secrets Manager, KMS and CloudWatch. CloudTrail is not provisioned by this stack.

Provider posture

The vendors' own test services, never the daxdev fakes. Providers whose sandbox is not wired yet still run their in-process mock (phase 6):

  • Fireblocks: testnet workspace or simplified CoSigner profile.
  • Elliptic (AML + sanctions): sandbox.
  • Sumsub: sandbox.
  • Mercuryo (acquirer + on/off ramp): sandbox.
  • CEX.IO (rate oracle): public API with a 30 s batch cache.
  • Email: local keeps EMAIL_PROVIDER=mock writing to email_log; Dev uses EMAIL_PROVIDER=resend for live delivery, tagged [DEV].

What Dev is not

  • Final perf testing.
  • Final security sign-off.
  • Final reconciliation validation.
  • Production readiness sign-off.

These are proven in Beta and Prod — see Where each control is validated.

Where each control is validated

Each check has an owner and a place. Dev proves product behaviour, Beta proves the production shape, and a small residue can only be proven in Prod.

What needs validating Dev Beta Prod
Product flows (buy / sell / send / receive / invoice / payout) ✅ with mocks ✅ ✅
/api/v1/* shapes, auth, rate limiting ✅ ✅ ✅
Worker behaviour — payout execution, webhook retry ✅ ✅ ✅
Provider callbacks, retry, signature handling ⚠️ mocks ✅ live, production contracts ✅ live
Release path — migrations once, instance refresh, rollback ❌ ✅ ✅
Private-subnet routing and NAT egress ❌ ✅ ✅
WAF rules and the admin allowlist ❌ ✅ ✅
Image pulls through the free S3 gateway endpoint; other AWS APIs via NAT ❌ ✅ ✅
Multi-AZ failover ❌ ❌ no standby ✅
DR rehearsal (backup + cross-region restore) ❌ ❌ no replica ✅ on a documented cadence
The shared Co-Signer and the signing path ❌ ✅ Beta's API user, same host ✅
Load and perf at realistic volume ❌ ❌ one instance ✅ pre-launch window
Security sign-off + penetration test ❌ partial ✅ pre-launch window

What Beta does not cover

Beta is deliberately one of everything, so anything that depends on a second instance, a standby or a replica — failover, DR timing, capacity — is proven in Prod during a pre-launch window with synthetic data, before signup opens. Each cutover step has a documented rollback.

Beta

Beta is the pre-production validation environment: a full clone of Prod with the redundancy removed, on mainnet, for internal testers. Same module, same release pipeline, same network model, same secret handling, same providers under production contracts. What differs is quantity — one of everything, one AZ, no replica — and reach: it is hidden behind a WAF allowlist and open to nobody outside the team.

Profile

  • One Availability Zone for the workload.
  • One application instance, one worker instance, both t4g.small.
  • Single-AZ db.t4g.small database with 30-day point-in-time recovery, no cross-region replica.
  • No high-availability or recovery objective. Beta may be down.
  • Production provider contracts — Fireblocks mainnet, Sumsub, Resend — with Beta's own API users and webhook endpoints. Company-owned test balances, capped by Fireblocks Transaction Authorization Policy rules on Beta's API user.
  • Internal audience only: the WAF blocks every source outside the tester exit node except webhook and health paths, registration is invite-only, and every response carries X-Robots-Tag: noindex.

What it keeps from Prod

Everything that shapes behaviour rather than capacity:

  • private app subnet, NAT and the free S3 gateway endpoint;
  • regional WAF with managed rule groups and the admin allowlist;
  • ALB with TLS termination and an ACM certificate;
  • Secrets Manager, KMS, IMDSv2, the same instance bootstrap with the same container memory limit;
  • Auto Scaling Groups and the tag-triggered release pipeline, including migrations-run-once and the rollback path;
  • CloudWatch alarms, including the burstable-CPU credit alarm, and ALB access logs;
  • the shared Fireblocks Co-Signer in the custody stack, paired with Beta's own API user.

Because it keeps the release path, a production release is always exercised end to end on Beta before the same tag reaches Live.

Configuration delta from Prod

Variable Prod Beta Why
app_subnet_cidrs two, one per AZ one Pins the ASG to a single AZ
asg_min_size / asg_desired_capacity 2 1 One application instance
asg_max_size 4 2 Lets a release launch the replacement before terminating
asg_target_cpu_percent 60 0 Fixed-size fleet
single_nat_gateway false true One NAT; nothing to keep available
db_multi_az true false Single-AZ database
db_replica_region eu-west-1 "" No cross-region replica
site_allowed_cidrs empty the exit node Hides Beta from everyone else
github_environment production beta Its own OIDC trust; no reviewers, since deploying to Beta is the validation
domain_name / app_domain_name live.daxcha.in beta.daxcha.in Its own delegated zone
betterstack_status_page_subdomain dax-live-status dax-beta-status Globally unique on Better Stack's side

Instance and database sizes are the same in both: t4g.small and db.t4g.small, the size Dev has run on without incident.

"Single AZ" does not mean single subnet

AWS refuses an Application Load Balancer with fewer than two subnets in two Availability Zones, and refuses a DB subnet group that does not span two. public_subnet_cidrs and db_subnet_cidrs therefore keep two entries in Beta. Only the workload is single-AZ.

What Beta cannot prove

  • Failover of any kind. One instance, one AZ, no replica: an AZ outage takes Beta down, which is the accepted trade.
  • Multi-AZ database behaviour — failover time, the connection reset clients see, replica lag.
  • Load and scaling. A single instance says nothing about target-tracking behaviour under real traffic.
  • Customer-scale provider behaviour. The contracts are real, the volumes are not.

Operating Beta

  • Beta carries no availability objective. Recovery is a manual restore from point-in-time backups: RPO about five minutes, RTO hours.
  • Releases reach Beta first, unattended, on every semver tag; Live follows the same tag after Beta succeeds and a reviewer approves.
  • Testers reach Beta through the Tailscale routing peer in the access stack, whose address is the allowlist. Accounts are created by invitation.
  • Beta data is internal test data. Customer data is never copied into it.

Prod

Profile

  • Two Availability Zones.
  • Full HA + private application + data runtime.
  • DB DR to secondary region.
  • Strong observability and audit trail.
  • Least-privilege access is the target; runtime database and application-admin privileges need further separation before launch.
  • Separate admin hostname with an admin-specific rule in the shared regional WAF. IAM Identity Center remains a target outside the current Terraform.

What's wired

  • Everything in Dev, plus:
    • Two Availability Zones with private app and data subnets, NAT egress.
    • The Fireblocks Co-Signer, one Nitro Enclave host shared with Beta in the custody stack, paired with Live's own API user.
    • GuardDuty and Security Hub are target controls, but are not provisioned by the current Terraform.
    • Tighter alert thresholds.
    • Production retention windows on logs/metrics.
    • Production provider contracts (live mode).
    • Production secrets (rotated regularly).

Production priorities

  • Controlled availability.
  • Understandable transaction processing.
  • Provider stability + degraded-mode visibility.
  • Traceable admin behaviour.
  • Recoverable from any single-AZ failure without manual ops.
  • Explicit DR rehearsal cadence.

Production controls

  • Separate admin subdomain.
  • AWS IAM Identity Center (SSO) on admin is a target and is not implemented by this stack.
  • The shared regional WAF blocks the admin hostname outside admin_allowed_cidrs; there is no dedicated admin distribution or web ACL.
  • Private runtime; no direct public ingress to data services.
  • audit_log is append-only; complete privileged-action coverage must be demonstrated before launch.
  • Failure visibility / alerting / escalation paths.

Environment resource inventory

The full inventory — a resource diagram per environment, counts and the variable that turns each resource on — is 22-Environment-Resources.

List of IT Systems

The consolidated systems / asset register — ownership, criticality, environment presence and provider status for every system and vendor the platform depends on — has its own page: 24-IT-Systems.

Cross-environment rules

Strict separation

  • Secrets per environment.
  • Provider credentials per environment.
  • IAM roles per environment.
  • Network boundaries per environment.
  • Data per environment — never copy Prod data into Dev in raw form.

Cost logic

  • Dev: minimise.
  • Prod: resilient and supportable.

Documentation rule

Every environment must be described consistently across:

  • This page (logical profile).
  • The network model (subnets and security groups).
  • The operations runbooks.
  • The roadmap (cutover targets).

DR strategy

Scope Dev Beta Prod
DB backup Best-effort 30-day retention + point-in-time Required, 30-day retention + point-in-time
DB DR to secondary region None None Required — cross-region replica
Application failover Single AZ Single AZ Multi-AZ
RPO target n/a n/a < 15 min
RTO target n/a n/a < 1 h

DR is intentionally DB-first — the rest of the application can be redeployed from artifacts. Critical data continuity is the non-negotiable layer.

DR data flows and interactions

Three replication flows run continuously; nothing about them requires an operator until the day one of them is needed.

flowchart TB subgraph primary["Primary region"] rds[("RDS PostgreSQL<br/>ledger · queues · sessions ·<br/>webhook inbox · halts · audit")] snap["Automated backups<br/>daily snapshot + WAL,<br/>point-in-time recovery"] notrep["NOT replicated: ECR images ·<br/>Secrets Manager · SSM parameters ·<br/>ALB access logs"] end subgraph dr["DR region"] rep[("Read replica<br/>own KMS key · private VPC, no internet route ·<br/>security group empty until promotion")] lag["Replica-lag alarm<br/>a silent replica alarms as breaching"] end rds -->|"asynchronous storage-level replication,<br/>encrypted in transit"| rep rds --> snap rep --> lag
Figure 1 — Continuous DR data flows — what leaves the primary region, and what deliberately does not

Why one arrow is enough. Postgres is the only datastore, so the single replication stream carries the entire system state: the ledger, the pg-boss queues and their cron bookkeeping, sessions, rate-limit windows, the provider webhook inbox and the outbound halts. A promoted replica does not just have the data — it has the queued jobs, and the worker in the DR region picks them up exactly where the primary stopped.

The NOT replicated box is the honest half of the design. Container images, secrets and deploy parameters are regional; a full-region loss means re-provisioning them in the DR region before the fleet can start. That is the main constant in the RTO, and it is a known trade, not an oversight.

Failover sequence

Promotion is a manual, deliberate act — an automatic promotion on a false positive would fork the ledger, which is strictly worse than downtime.

sequenceDiagram autonumber participant Ops as Operator participant DR as DR region participant SM as Secrets and parameters participant Fleet as App and worker fleet participant DNS as DNS Ops->>DR: promote replica — replication stops,<br/>replica becomes a standalone primary Ops->>DR: provision the app stack — same Terraform module,<br/>DR-region values Ops->>SM: recreate secrets and the image-tag parameter,<br/>point DATABASE_URL at the promoted DB Ops->>Fleet: boot from container images Fleet->>DR: migrations are a no-op — schema travelled with the data Ops->>DNS: cut the hostname over Note over Fleet: workers drain the queues the primary left behind
Figure 2 — Failover — the ordered path from regional loss to serving from the DR region

After failover, the system fails closed

The interesting DR behaviour is not the promotion — it is what the application does about the data it may have lost. Async replication means a crash can lose the last seconds of writes, and three mechanisms are built for exactly that window:

Mechanism What it covers after failover
Idempotent custody calls A payout row whose ledger write was lost re-executes safely: the executor looks the transfer up by its deterministic external id first, adopts what already went through, and never double-pays
The 5-minute custody poll Webhooks that fired during the outage are gone; the poll re-fetches every pending transaction and the idempotent inbox absorbs any provider redelivery
The hourly reconciliation halt Lost ledger writes surface as drift between the restored ledger and the actual custody balance — which trips the circuit breaker and freezes outbound flow automatically until the books are verified

Credentials are the exception: a revoked session, API key or password change lost in that window comes back with the restored row. Recovery therefore sets a new SESSION_EPOCH and re-revokes API keys before traffic returns (step 4 of infra/terraform/RECOVERY-AND-LAUNCH.md).

That last row is the designed answer to the RPO question: the platform does not pretend replication is lossless — it detects the loss on the first reconciliation pass and stops moving money until a human clears it. Deposits made during the outage are never lost: funds arrive at custody addresses regardless of platform availability, and are recognised by the poll once the system returns.

What is exercised where

Failover drills run against the DR replica on a documented cadence; the promotion path above is the drill script. What a drill cannot rehearse — provider-side behaviour during a real regional event — is bounded by the idempotency and halt mechanics in the table above.

Drawbacks

  • Beta has no redundancy. One AZ, one instance, no standby, no replica. It reproduces the production shape but not its resilience, so failover, DR timing and capacity are still first exercised in Prod during a pre-launch window.
  • Beta acts on real providers. A Beta bug can move real test funds; the Fireblocks policy limits on Beta's API user cap the exposure.
  • Three environments cost more than two. Beta is sized to keep that difference close to the Dev bill rather than the Prod one.

Open questions

  • The CoSigner profile for non-production environments.
  • Whether Prod eventually runs a second active region rather than DR-only.
  • When to add a second Co-Signer host for signing availability.
  • Final resource sizing per environment.

The target architecture, network model, operations runbooks and glossary are internal documents, available on request.