▸ prd.vault

vault

17 — System Design (per environment)

docs/17-System-Design.mdtype: architectureupdated: 2026-08-09

System Design — per environment

One diagram per environment. Each is drawn from the Terraform that defines it, so every box maps to a resource rather than to an intention. Dev is the development estate, Beta the production-shaped rehearsal environment, Prod the regulated one.

Environment Role Terraform Serves
Dev Development and mock-driven iteration infra/terraform/dev daxcha.in, deployed from master
Beta Pre-production validation on mainnet, one of everything, internal infra/terraform/beta beta.daxcha.in, first stop for every tag
Prod Live regulated operations infra/terraform/prod live.daxcha.in, same tag after Beta succeeds
Custody, Access Shared Fireblocks Co-Signer; tester exit node infra/terraform/custody, infra/terraform/access supporting stacks in the production account

How to read the diagrams

Notation Meaning
①②③… A trust zone. The numbers run outward-in: ① is the untrusted internet, higher numbers are progressively more restricted. Crossings between zones are the interesting part
Solid arrow Request or data flow, labelled with protocol, port and how it is authenticated
Dotted arrow Control-plane or configuration read — not customer traffic
⚠ A weakness on that specific edge, stated rather than implied. Encryption is written as TLS, plaintext or at rest, so a reviewer can see the gaps without reading Terraform
SG-* An AWS security group. SG-DB: from SG-APP only, no egress is the literal rule set, not a summary

Every diagram fits its column so the whole graph is visible at once. Use the Expand control on a figure to open it full screen with zoom and pan — the architecture diagrams carry more label text than fits at page scale.

Dev

Everything here maps to a resource in infra/terraform/dev. Split into two views — how a request is served, and how a deploy reaches the host — because one combined graph was unreadable.

Trust boundaries and data flow

Every boundary crossing is annotated with protocol, port and how it is authenticated. Encryption state is stated rather than implied — TLS, plaintext, or at rest — so a reviewer can see the gaps without reading Terraform.

flowchart TB subgraph internet["① Untrusted — public internet"] user["Browser / merchant API client"] end subgraph edge["② Edge — AWS managed, no VPC"] waf["WAF web ACL<br/>rate limit per IP<br/>admin allowlist by Host"] alb["ALB<br/>ACM cert, TLS terminated here"] end subgraph pub["③ Public subnet 10.40.0.0/24, 10.40.1.0/24"] ec2["EC2 t4g.small · IMDSv2 required<br/>SG-APP: ingress from SG-ALB only"] web["web container :3000"] worker["worker container<br/>no listener"] end subgraph priv["④ Private subnet 10.40.10.0/24, 10.40.11.0/24 — no route to IGW"] rds[("RDS PostgreSQL<br/>encrypted at rest, KMS<br/>SG-DB: 5432 from SG-APP only<br/>no egress rules")] end subgraph ctrl["⑤ AWS control plane — IAM authenticated"] sm["Secrets Manager<br/>app env + Fireblocks key"] kms["KMS CMK<br/>rotation enabled"] ssm["SSM Parameter Store<br/>current image tag"] cw["CloudWatch Logs"] end subgraph ext["⑥ Third parties — egress via IGW"] prov["Fireblocks · Sumsub · CEX.IO · Resend<br/>4 live, 5 mocked"] bs["Better Stack<br/>logs, OTLP traces, heartbeat"] end user -->|"TLS 1.2+ :443<br/>session cookie or API key"| waf user -.->|":80 → 301 :443"| waf waf -->|"allowed requests"| alb alb -->|"⚠ HTTP :3000 PLAINTEXT<br/>inside VPC, SG-restricted"| web ec2 --- web ec2 --- worker web -->|"⚠ TLS :5432<br/>certificate validation not proven"| rds worker -->|"⚠ TLS :5432<br/>certificate validation not proven"| rds web -->|"TLS, IAM role, resource-scoped"| sm sm -.->|"decrypt, EncryptionContext pinned"| kms ec2 -.->|"TLS, IAM role, read only"| ssm web -->|"TLS"| cw worker -->|"TLS"| cw worker -->|"TLS, API key from Secrets Manager"| prov web -->|"TLS, source token"| bs
Figure 1 — Dev — trust boundaries and request path, as defined by infra/terraform/dev

Boundary controls, as implemented

Crossing Control Where
① → ② WAF rate limit per IP; admin hostname allowlisted by CIDR modules/platform/waf.tf
② → ③ SG-APP accepts only SG-ALB, by security-group reference network.tf
③ → ④ SG-DB accepts only SG-APP on 5432; no egress rules at all network.tf
③ → ⑤ Instance-profile IAM, scoped to named secret and parameter ARNs iam.tf
⑤ decrypt KMS grant conditioned on kms:EncryptionContext:SecretARN and kms:ViaService iam.tf
③ → ⑥ No inbound; egress via IGW. Provider credentials from Secrets Manager, never baked into the image templates/user-data.sh.tftpl

Three gaps a reviewer should see

  • ALB → instance is plaintext HTTP. The target group is protocol = "HTTP" on port 3000. Traffic is confined to the VPC and the security group, but it is not encrypted in transit inside the VPC. Ending TLS at the instance would need a certificate per instance or an ACM private CA.
  • The HTTPS listener sets no explicit ssl_policy, so it inherits the AWS default rather than a pinned modern policy.
  • Database server identity validation is not currently proven. Treat it as unimplemented until every runtime and migration connection validates the CA and hostname. Exact code evidence is in the controlled review pack.

Release path — supply chain

flowchart TB subgraph gh["GitHub Actions — no long-lived AWS keys"] push["push to master"] test["test · typecheck · unit"] build["build image<br/>tag = commit SHA, immutable"] end subgraph awsauth["AWS trust"] oidc["IAM OIDC provider<br/>token.actions.githubusercontent.com"] role["Deploy role<br/>sub pinned to repo + environment<br/>ref enforced by workflow unless OIDC subject is customised"] end subgraph tgt["Dev target"] ecr["ECR<br/>scan on push, lifecycle policy"] ssmtag["SSM parameter<br/>image tag"] inst["EC2 instance<br/>SSM Run Command"] end push --> test --> build build -->|"OIDC token, no static secret"| oidc oidc -->|"AssumeRoleWithWebIdentity"| role role -->|"ecr:PutImage, scoped to one repo"| ecr role -->|"ssm:PutParameter, scoped to one name"| ssmtag role -->|"ssm:SendCommand, tag-scoped to app/worker"| inst inst -.->|"pull by digest-tagged image"| ecr inst -.->|"read tag"| ssmtag
Figure 2 — Dev release path — how a commit on master reaches the running host

Dev deploys from master. Production is tag-triggered and rolls an ASG instead — see Prod and infra/terraform/prod/README.md.

What Dev deliberately is not

  • No NAT gateway. The instance is in a public subnet with an EIP, which keeps ECR and Secrets Manager reachable without paying for NAT.
  • No app HA. One instance; a restart is downtime.
  • No DR. multi_az = false on RDS; backups only.
  • Shared host. Web and worker are two containers on the same box, so they share CPU, memory and failure domain.
  • PROVIDER_MODE=mock by default. Fireblocks, KYC, the oracle and email have live implementations that can be switched on per provider; chain, acquirer and Elliptic throw if set to live.

Prod

Prod serves live.daxcha.in, sharing the daxcha.in zone with the other environments without overlapping any record.

Multi-AZ, private subnets with NAT egress, the shared Co-Signer in its own custody stack, and a separate identity-aware admin entry path.

flowchart TB subgraph internet["① Untrusted — public internet"] cust["Customers / merchants<br/>live.daxcha.in"] adm["Admin operators"] end subgraph edge["② Edge — AWS managed"] waf["WAF web ACL<br/>managed rule groups · rate limit<br/>admin allowlist: block admin.* unless source in CIDR set"] alb["ALB · ACM cert · TLS terminated"] end subgraph pubp["③ Public subnets — ALB and NAT only, no instances"] nat["NAT gateway per AZ<br/>egress only"] end subgraph appp["④ Private app subnets — no route to IGW"] asg["app ASG · 2 AZ · t4g.small<br/>web only · rolling refresh"] wrk["worker ASG · pinned to 1<br/>pg-boss cron owner"] end subgraph datap["⑤ Private data subnets"] rdsp[("RDS Multi-AZ · db.t4g.small<br/>encrypted at rest, KMS<br/>no egress rules")] s3gw["S3 gateway endpoint<br/>image layers, free"] end subgraph custody["⑧ Custody stack — own VPC, no ingress"] cos["Fireblocks Co-Signer · Nitro Enclave<br/>shared with Beta · egress 443 only"] end subgraph dr["⑥ DR region — separate VPC, no IGW"] rep[("Cross-region replica<br/>encrypted with DR-region CMK<br/>SG with no rules until promotion")] end subgraph ext["⑦ Third parties"] prov["Fireblocks · Elliptic<br/>Mercuryo · Sumsub · CEX.IO · Resend"] end cust -->|"TLS :443"| waf adm -->|"TLS :443, blocked unless allowlisted"| waf waf --> alb alb -->|"⚠ HTTP :3000 plaintext, VPC-internal"| asg asg -->|"⚠ TLS :5432<br/>certificate validation requires code fix"| rdsp wrk -->|"⚠ TLS :5432<br/>certificate validation requires code fix"| rdsp asg -.->|"image layers"| s3gw wrk -.->|"image layers"| s3gw asg -->|"TLS :443 via NAT<br/>AWS APIs, providers"| nat wrk -->|"TLS :443 via NAT"| nat nat --> prov cos -->|"TLS :443, long-poll"| prov rdsp -.->|"AWS-managed replication, encrypted"| rep
Figure 3 — Prod — trust boundaries, private subnets and the DR region

What changes versus Dev, from a control standpoint

Control Dev Prod
Instance reachability Public subnet, Elastic IP Private subnet, no IGW route
Egress Direct via IGW NAT per AZ; image layers via the free S3 gateway endpoint, other AWS APIs through NAT. Interface endpoints are a variable, off until a review asks
Admin restriction None (admin_allowed_cidrs empty) WAF blocks admin.* outside the allowlist
Signing material n/a — Fireblocks mocked Co-Signer in a Nitro Enclave in its own custody VPC, no ingress, prevent_destroy; shared with Beta by API user
Database Single AZ, no DR Multi-AZ + cross-region replica under a DR-region CMK
Migrations On every boot Once per release, from the pipeline
Worker count 1 (same host) 1 (dedicated ASG) — cron must not double-fire
Deploy identity OIDC environment claim; branch enforced by workflow OIDC environment claim; tag enforced by workflow/reviewers unless the repository-wide subject claim is customised

Beta

Beta is Prod's architecture with one of everything, on mainnet, for internal testers: the same module, the same release pipeline, the same network model, the same secret handling and the same providers under production contracts — one AZ, one app instance, one worker, a single-AZ database and no replica. It differs from Prod in variable values and in reach: a WAF site allowlist admits only the tester exit node, except for webhook and health paths, and the application runs invite-only with noindex.

Its purpose is to validate every release before it reaches Live. Dev runs a public-subnet instance with migrations on boot; Prod runs private subnets, an instance refresh and migrations-once. Every tag is deployed to Beta first, unattended, and only a green Beta lets the same tag proceed to Live.

flowchart TB subgraph internet["① Untrusted — public internet"] custb["Testers via the Tailscale exit node<br/>beta.daxcha.in"] admb["Admin operators, same exit node"] other["Everyone else"] end subgraph edge["② Edge — AWS managed"] wafb["WAF web ACL<br/>same rules as Prod<br/>+ site allowlist: block unless exit node,<br/>except /api/webhooks/* and /api/health"] albb["ALB · ACM cert · TLS terminated<br/>⚠ spans 2 AZ — AWS requires it"] end subgraph pubb["③ Public subnets — ALB and NAT only"] natb["NAT gateway ×1<br/>egress only"] end subgraph appb["④ Private app subnet — ONE AZ"] asgb["app ASG · min=desired=1, max=2 · t4g.small<br/>⚠ no redundancy: AZ loss = outage"] wrkb["worker ASG · pinned to 1<br/>pg-boss cron owner"] end subgraph datab["⑤ Private data subnets — 2 AZ subnet group, 1 AZ instance"] rdsb[("RDS single-AZ · db.t4g.small<br/>encrypted at rest, KMS · 30-day PITR<br/>⚠ no standby, no replica")] s3gwb["S3 gateway endpoint<br/>image layers, free"] end subgraph extb["⑥ Third parties — production contracts"] provb["Fireblocks mainnet · Sumsub · Resend<br/>Beta's own API users and webhook URLs"] cosb["Shared Co-Signer (custody stack)<br/>Beta API user, policy-capped vaults"] end custb -->|"TLS :443"| wafb admb -->|"TLS :443"| wafb other -.->|"blocked, unless webhook or health path"| wafb wafb --> albb albb -->|"⚠ HTTP :3000 plaintext, VPC-internal"| asgb asgb -->|"⚠ TLS :5432<br/>certificate validation requires code fix"| rdsb wrkb -->|"⚠ TLS :5432<br/>certificate validation requires code fix"| rdsb asgb -.->|"image layers"| s3gwb wrkb -->|"TLS :443 via NAT"| natb natb --> provb provb -.->|"signing requests"| cosb
Figure 4 — Beta — the production topology with one of everything, behind an allowlist

"Single AZ" is a workload property, not a subnet layout

AWS refuses an ALB with fewer than two subnets in two Availability Zones, and refuses a DB subnet group that does not span two. Beta therefore keeps two public and two database subnets. What is single-AZ is the workload: one app subnet, one instance, one database instance with no standby.

What Beta proves, and what it does not

Question Beta answers it?
Does the tag-triggered release work end to end — migrations once, refresh, rollback? ✅ yes, same pipeline, every tag
Do private subnets, NAT and the S3 gateway let the app reach ECR and Secrets Manager? ✅ yes
Does the WAF admin allowlist actually block? ✅ yes
Do the production providers, webhooks and the shared Co-Signer behave? ✅ yes, with Beta's own API users
Does Multi-AZ failover work, and how long does it take? ❌ no standby exists
Does cross-region promotion meet the RPO/RTO? ❌ no replica exists
Does the fleet scale under load? ❌ one instance

Beta may be down. It carries no availability objective, and an AZ outage takes it out entirely — that is the trade being made for a production-shaped environment at roughly the cost of Dev.

What differs between the three

Dimension Dev Beta Prod
Compute 1 × t4g.small, both containers ASG of 1 + worker ASG of 1, t4g.small, single AZ ASG of 2–4 + worker, t4g.small, 2 AZ
App HA none none — AZ loss is an outage Multi-AZ
Database db.t4g.small, single-AZ db.t4g.small, single-AZ, 30-day PITR db.t4g.small, Multi-AZ
DR backups only backups only cross-region replica
RPO / RTO n/a ~5 min / hours, manual < 15 min / < 1 h
Networking public subnet + EIP, no NAT private subnet + NAT + S3 gateway private subnets + NAT per AZ + S3 gateway
Edge ALB + regional WAF same as Prod, plus a site allowlist ALB + regional WAF, managed rule groups, admin IP allowlist, access logs
Reach public tester exit node only public
Admin entry NextAuth session NextAuth + admin allowlist NextAuth + admin allowlist; IAM Identity Center is an external target
Co-Signer none shared host in the custody stack, Beta API user shared host in the custody stack, Live API user
Migrations on every boot once per release, from the pipeline once per release, from the pipeline
Deploy trigger push to master every version tag, first, unattended same tag, after Beta, with reviewers
Providers mock by default live, production contracts live
Data seeded internal test data, capped balances real customer data

What Beta validates, and what it cannot

Beta closes the shape of the Dev-to-Prod gap but not its scale. It exercises the release path, the network model, the edge controls and the production providers; it says nothing about failover or capacity, because it has no standby and one instance.

  • 12-Environments — the environment model behind these diagrams: profiles, the Beta configuration delta, and where each control is validated
  • 20-Data-Flows — how value and identity move through these components
  • 21-Trust-Boundaries — the same system cut by trust rather than by environment
  • 22-Environment-Resources — the resource inventory behind each environment

The target architecture, network model, deployment runbooks and application architecture are internal documents, available on request.