vault
12 — Environments
Environments
Three environments, each with its own Terraform root over the shared
infra/terraform/modules/platform: dev/ serving daxcha.in, beta/
serving beta.daxcha.in, and prod/ serving live.daxcha.in. They share
the daxcha.in zone without overlapping any record. Diagrams:
17-System-Design.
Current MVP default AWS region is eu-north-1 (Stockholm). That is a region, not an availability zone; the single-AZ dev posture means one AZ inside eu-north-1.
Local and Dev
Two names that used to share APP_ENV=development and are now kept apart:
- local is a developer's laptop. It runs dax against the fakes and the
devtools console in
bullder/daxdev(signed fake webhooks, log-in-as, seed).APP_ENV=local. Local-only conveniences (simulated conversions, mock pricing) key onlocalandtestand nothing else. - Dev is the deployed AWS environment at
daxcha.in. It runs the production build against the vendors' own test services (Sumsub sandbox, Fireblocks testnet, Mercuryo sandbox, Elliptic, the CEX.IO feed, Resend).APP_ENV=dev. Nothing from daxdev runs in Dev: no fakes, no console, no seed.
APP_ENV |
Where |
|---|---|
local |
a developer's laptop |
test |
test runs, CI included |
dev |
AWS Dev, daxcha.in |
beta |
AWS Beta, beta.daxcha.in |
live |
AWS Prod, live.daxcha.in |
development is the old shared value. It is still accepted while Dev is moved
to dev, but it counts as deployed: it never enables local-only behaviour,
and env validation warns when it sees it. Replace it with local in
.env.local. It will be removed once Dev runs APP_ENV=dev
(phase 7).
The three environments
| Env | Purpose | HA | DR | Provider mode |
|---|---|---|---|---|
| Dev | Integration and product validation on the production build | Single AZ | None | Vendor test services (sandbox, testnet); mock only where a sandbox is not wired yet |
| Beta | Pre-production validation: Live's topology with one of everything, on mainnet, for internal testers | Single AZ | None | Live, production contracts, test funds capped by policy |
| Prod | Live regulated operations | Multi-AZ | DB DR to secondary region | Live |
Beta is Prod's architecture at Dev's redundancy: every release is deployed there first and exercised against the real providers before it reaches Live, without paying for a second production estate. See Where each control is validated.
Principles
- Dev stays cheap on purpose. Reduced footprint, single AZ, no DR.
- Strict isolation. Secrets, provider credentials, data, and network boundaries never cross environments.
- Promotion is one-way. Dev → Beta → Prod. No back-promotion of data.
- Rehearse in Beta, not in Prod. Anything that depends on the production shape — private subnets, NAT egress, the tag-triggered release — is exercised in Beta first. What Beta cannot hold, because it runs one of everything, is listed explicitly in Where each control is validated rather than assumed away.
Dev
Profile
- Single Availability Zone.
- No HA / DR requirement.
- Smaller compute + DB sizing.
- Lower logging volume.
- Reduced redundancy.
What's wired
- Route 53 + public ALB + regional AWS WAF (lightweight rule set).
- API Gateway, internal ALB.
- App + worker containers on EC2 (smaller instance).
- Postgres single-instance (or simplified topology).
- S3, Secrets Manager, KMS and CloudWatch. CloudTrail is not provisioned by this stack.
Provider posture
The vendors' own test services, never the daxdev fakes. Providers whose sandbox is not wired yet still run their in-process mock (phase 6):
- Fireblocks: testnet workspace or simplified CoSigner profile.
- Elliptic (AML + sanctions): sandbox.
- Sumsub: sandbox.
- Mercuryo (acquirer + on/off ramp): sandbox.
- CEX.IO (rate oracle): public API with a 30 s batch cache.
- Email: local keeps
EMAIL_PROVIDER=mockwriting toemail_log; Dev usesEMAIL_PROVIDER=resendfor live delivery, tagged[DEV].
What Dev is not
- Final perf testing.
- Final security sign-off.
- Final reconciliation validation.
- Production readiness sign-off.
These are proven in Beta and Prod — see Where each control is validated.
Where each control is validated
Each check has an owner and a place. Dev proves product behaviour, Beta proves the production shape, and a small residue can only be proven in Prod.
| What needs validating | Dev | Beta | Prod |
|---|---|---|---|
| Product flows (buy / sell / send / receive / invoice / payout) | ✅ with mocks | ✅ | ✅ |
/api/v1/* shapes, auth, rate limiting |
✅ | ✅ | ✅ |
| Worker behaviour — payout execution, webhook retry | ✅ | ✅ | ✅ |
| Provider callbacks, retry, signature handling | ⚠️ mocks | ✅ live, production contracts | ✅ live |
| Release path — migrations once, instance refresh, rollback | ❌ | ✅ | ✅ |
| Private-subnet routing and NAT egress | ❌ | ✅ | ✅ |
| WAF rules and the admin allowlist | ❌ | ✅ | ✅ |
| Image pulls through the free S3 gateway endpoint; other AWS APIs via NAT | ❌ | ✅ | ✅ |
| Multi-AZ failover | ❌ | ❌ no standby | ✅ |
| DR rehearsal (backup + cross-region restore) | ❌ | ❌ no replica | ✅ on a documented cadence |
| The shared Co-Signer and the signing path | ❌ | ✅ Beta's API user, same host | ✅ |
| Load and perf at realistic volume | ❌ | ❌ one instance | ✅ pre-launch window |
| Security sign-off + penetration test | ❌ | partial | ✅ pre-launch window |
What Beta does not cover
Beta is deliberately one of everything, so anything that depends on a second instance, a standby or a replica — failover, DR timing, capacity — is proven in Prod during a pre-launch window with synthetic data, before signup opens. Each cutover step has a documented rollback.
Beta
Beta is the pre-production validation environment: a full clone of Prod with the redundancy removed, on mainnet, for internal testers. Same module, same release pipeline, same network model, same secret handling, same providers under production contracts. What differs is quantity — one of everything, one AZ, no replica — and reach: it is hidden behind a WAF allowlist and open to nobody outside the team.
Profile
- One Availability Zone for the workload.
- One application instance, one worker instance, both
t4g.small. - Single-AZ
db.t4g.smalldatabase with 30-day point-in-time recovery, no cross-region replica. - No high-availability or recovery objective. Beta may be down.
- Production provider contracts — Fireblocks mainnet, Sumsub, Resend — with Beta's own API users and webhook endpoints. Company-owned test balances, capped by Fireblocks Transaction Authorization Policy rules on Beta's API user.
- Internal audience only: the WAF blocks every source outside the tester
exit node except webhook and health paths, registration is invite-only,
and every response carries
X-Robots-Tag: noindex.
What it keeps from Prod
Everything that shapes behaviour rather than capacity:
- private app subnet, NAT and the free S3 gateway endpoint;
- regional WAF with managed rule groups and the admin allowlist;
- ALB with TLS termination and an ACM certificate;
- Secrets Manager, KMS, IMDSv2, the same instance bootstrap with the same container memory limit;
- Auto Scaling Groups and the tag-triggered release pipeline, including migrations-run-once and the rollback path;
- CloudWatch alarms, including the burstable-CPU credit alarm, and ALB access logs;
- the shared Fireblocks Co-Signer in the custody stack, paired with Beta's own API user.
Because it keeps the release path, a production release is always exercised end to end on Beta before the same tag reaches Live.
Configuration delta from Prod
| Variable | Prod | Beta | Why |
|---|---|---|---|
app_subnet_cidrs |
two, one per AZ | one | Pins the ASG to a single AZ |
asg_min_size / asg_desired_capacity |
2 | 1 | One application instance |
asg_max_size |
4 | 2 | Lets a release launch the replacement before terminating |
asg_target_cpu_percent |
60 | 0 | Fixed-size fleet |
single_nat_gateway |
false |
true |
One NAT; nothing to keep available |
db_multi_az |
true |
false |
Single-AZ database |
db_replica_region |
eu-west-1 |
"" |
No cross-region replica |
site_allowed_cidrs |
empty | the exit node | Hides Beta from everyone else |
github_environment |
production |
beta |
Its own OIDC trust; no reviewers, since deploying to Beta is the validation |
domain_name / app_domain_name |
live.daxcha.in |
beta.daxcha.in |
Its own delegated zone |
betterstack_status_page_subdomain |
dax-live-status |
dax-beta-status |
Globally unique on Better Stack's side |
Instance and database sizes are the same in both: t4g.small and
db.t4g.small, the size Dev has run on without incident.
"Single AZ" does not mean single subnet
AWS refuses an Application Load Balancer with fewer than two subnets in
two Availability Zones, and refuses a DB subnet group that does not span
two. public_subnet_cidrs and db_subnet_cidrs therefore keep two
entries in Beta. Only the workload is single-AZ.
What Beta cannot prove
- Failover of any kind. One instance, one AZ, no replica: an AZ outage takes Beta down, which is the accepted trade.
- Multi-AZ database behaviour — failover time, the connection reset clients see, replica lag.
- Load and scaling. A single instance says nothing about target-tracking behaviour under real traffic.
- Customer-scale provider behaviour. The contracts are real, the volumes are not.
Operating Beta
- Beta carries no availability objective. Recovery is a manual restore from point-in-time backups: RPO about five minutes, RTO hours.
- Releases reach Beta first, unattended, on every semver tag; Live follows the same tag after Beta succeeds and a reviewer approves.
- Testers reach Beta through the Tailscale routing peer in the access stack, whose address is the allowlist. Accounts are created by invitation.
- Beta data is internal test data. Customer data is never copied into it.
Prod
Profile
- Two Availability Zones.
- Full HA + private application + data runtime.
- DB DR to secondary region.
- Strong observability and audit trail.
- Least-privilege access is the target; runtime database and application-admin privileges need further separation before launch.
- Separate admin hostname with an admin-specific rule in the shared regional WAF. IAM Identity Center remains a target outside the current Terraform.
What's wired
- Everything in Dev, plus:
- Two Availability Zones with private app and data subnets, NAT egress.
- The Fireblocks Co-Signer, one Nitro Enclave host shared with Beta in the custody stack, paired with Live's own API user.
- GuardDuty and Security Hub are target controls, but are not provisioned by the current Terraform.
- Tighter alert thresholds.
- Production retention windows on logs/metrics.
- Production provider contracts (live mode).
- Production secrets (rotated regularly).
Production priorities
- Controlled availability.
- Understandable transaction processing.
- Provider stability + degraded-mode visibility.
- Traceable admin behaviour.
- Recoverable from any single-AZ failure without manual ops.
- Explicit DR rehearsal cadence.
Production controls
- Separate admin subdomain.
- AWS IAM Identity Center (SSO) on admin is a target and is not implemented by this stack.
- The shared regional WAF blocks the admin hostname outside
admin_allowed_cidrs; there is no dedicated admin distribution or web ACL. - Private runtime; no direct public ingress to data services.
audit_logis append-only; complete privileged-action coverage must be demonstrated before launch.- Failure visibility / alerting / escalation paths.
Environment resource inventory
The full inventory — a resource diagram per environment, counts and the variable that turns each resource on — is 22-Environment-Resources.
List of IT Systems
The consolidated systems / asset register — ownership, criticality, environment presence and provider status for every system and vendor the platform depends on — has its own page: 24-IT-Systems.
Cross-environment rules
Strict separation
- Secrets per environment.
- Provider credentials per environment.
- IAM roles per environment.
- Network boundaries per environment.
- Data per environment — never copy Prod data into Dev in raw form.
Cost logic
- Dev: minimise.
- Prod: resilient and supportable.
Documentation rule
Every environment must be described consistently across:
- This page (logical profile).
- The network model (subnets and security groups).
- The operations runbooks.
- The roadmap (cutover targets).
DR strategy
| Scope | Dev | Beta | Prod |
|---|---|---|---|
| DB backup | Best-effort | 30-day retention + point-in-time | Required, 30-day retention + point-in-time |
| DB DR to secondary region | None | None | Required — cross-region replica |
| Application failover | Single AZ | Single AZ | Multi-AZ |
| RPO target | n/a | n/a | < 15 min |
| RTO target | n/a | n/a | < 1 h |
DR is intentionally DB-first — the rest of the application can be redeployed from artifacts. Critical data continuity is the non-negotiable layer.
DR data flows and interactions
Three replication flows run continuously; nothing about them requires an operator until the day one of them is needed.
Why one arrow is enough. Postgres is the only datastore, so the single replication stream carries the entire system state: the ledger, the pg-boss queues and their cron bookkeeping, sessions, rate-limit windows, the provider webhook inbox and the outbound halts. A promoted replica does not just have the data — it has the queued jobs, and the worker in the DR region picks them up exactly where the primary stopped.
The NOT replicated box is the honest half of the design. Container images, secrets and deploy parameters are regional; a full-region loss means re-provisioning them in the DR region before the fleet can start. That is the main constant in the RTO, and it is a known trade, not an oversight.
Failover sequence
Promotion is a manual, deliberate act — an automatic promotion on a false positive would fork the ledger, which is strictly worse than downtime.
After failover, the system fails closed
The interesting DR behaviour is not the promotion — it is what the application does about the data it may have lost. Async replication means a crash can lose the last seconds of writes, and three mechanisms are built for exactly that window:
| Mechanism | What it covers after failover |
|---|---|
| Idempotent custody calls | A payout row whose ledger write was lost re-executes safely: the executor looks the transfer up by its deterministic external id first, adopts what already went through, and never double-pays |
| The 5-minute custody poll | Webhooks that fired during the outage are gone; the poll re-fetches every pending transaction and the idempotent inbox absorbs any provider redelivery |
| The hourly reconciliation halt | Lost ledger writes surface as drift between the restored ledger and the actual custody balance — which trips the circuit breaker and freezes outbound flow automatically until the books are verified |
Credentials are the exception: a revoked session, API key or password
change lost in that window comes back with the restored row. Recovery
therefore sets a new SESSION_EPOCH and re-revokes API keys before traffic
returns (step 4 of infra/terraform/RECOVERY-AND-LAUNCH.md).
That last row is the designed answer to the RPO question: the platform does not pretend replication is lossless — it detects the loss on the first reconciliation pass and stops moving money until a human clears it. Deposits made during the outage are never lost: funds arrive at custody addresses regardless of platform availability, and are recognised by the poll once the system returns.
What is exercised where
Failover drills run against the DR replica on a documented cadence; the promotion path above is the drill script. What a drill cannot rehearse — provider-side behaviour during a real regional event — is bounded by the idempotency and halt mechanics in the table above.
Drawbacks
- Beta has no redundancy. One AZ, one instance, no standby, no replica. It reproduces the production shape but not its resilience, so failover, DR timing and capacity are still first exercised in Prod during a pre-launch window.
- Beta acts on real providers. A Beta bug can move real test funds; the Fireblocks policy limits on Beta's API user cap the exposure.
- Three environments cost more than two. Beta is sized to keep that difference close to the Dev bill rather than the Prod one.
Open questions
- The CoSigner profile for non-production environments.
- Whether Prod eventually runs a second active region rather than DR-only.
- When to add a second Co-Signer host for signing availability.
- Final resource sizing per environment.
Related
- 17-System-Design — the diagram for each environment
- 20-Data-Flows — how value and identity move through the system
- 21-Trust-Boundaries — who is trusted with what, per crossing
- 22-Environment-Resources — what each environment is made of, resource by resource
- 24-IT-Systems — the consolidated systems / asset register
The target architecture, network model, operations runbooks and glossary are internal documents, available on request.