# Build & Deploy Pipeline

The iec-builder executes a fixed, ordered pipeline for every `tawa deploy`. Understanding each
step — what it does, when it fails hard vs. fails open, and what it writes to each system — is
essential for debugging deploys and hardening the platform.

## Pipeline Overview

```
tawa deploy --prod
     │
     ▼
 iec-builder (builder.ts: executeBuild)
     │
     ├─ PRE-BUILD
     │   ├─ Clone/extract
     │   ├─ Catalog parse
     │   ├─ Dockerfile strategy
     │   ├─ Deploy gate (wallet reserve)
     │   ├─ Config preflight
     │   └─ Convention check / SDK upgrade gate
     │
     ├─ BUILD
     │   ├─ docker build
     │   └─ docker push → registry
     │
     └─ DEPLOY  (ENABLE_K8S_DEPLOY=true)
         ├─ Namespace + Koko namespace registration
         ├─ Database provisioning (Vault or static K8s secrets)
         ├─ Consumed database provisioning
         ├─ Object storage provisioning (Vault, catalog 0.3.0+)
         ├─ Dependency resolution (Koko + Bio-ID scope grants)
         ├─ OAuth client provisioning
         ├─ Scope grant creation
         ├─ Module registration (Bio-ID)
         ├─ userAccess provisioning
         ├─ helm upgrade --install
         ├─ Deploy snapshot (Koko, fire-and-forget)
         ├─ DNS configuration (Cloudflare)
         ├─ Post-deploy pod health check
         ├─ Deploy-gated tests (iec-test, catalog 0.5.0+)
         ├─ Service registration in Koko
         ├─ Queue + schedule binding registration
         ├─ Role + registration metadata (catalog 0.6.0+)
         └─ Uptime monitor (iec-pulse, fire-and-forget)
```

## Status Flow

```
queued → cloning → building → pushing → deploying → testing → completed
                                                            ↘ failed (any step)
```

---

## Step-by-Step Reference

### 1. Clone / Extract

**What:** Clones the repo at the specified `commitSha` using the branch as a hint. If a tarball
was uploaded (via `POST /builds/upload`), it extracts that instead.

**Auth:** Forgejo repos get a PAT injected (`forgejo-token:TOKEN@host`). GitHub SSH URLs are
converted to HTTPS with a GitHub PAT. Both are handled by `injectGitCredentials()`.

**Output:** Actual commit SHA is resolved and backfilled to the build record. Workspace at
`{WORKSPACE_DIR}/{buildId}`.

**Fail behavior:** Hard fail — no source = no build.

---

### 2. Read .iec.yaml (monorepo config)

**What:** Reads `.iec.yaml` from the repo root (or `{service.appPath}/.iec.yaml`). Controls:
- `appPath` — subdirectory containing the service
- `dockerfile` — custom Dockerfile path
- `buildContext` — custom build context (relative to appPath)
- `helmChart` — custom Helm chart path

**Priority for appPath:** `service.appPath` (DB) > `.iec.yaml` > inferred from
`service.dockerfilePath`.

**Fail behavior:** Missing `.iec.yaml` is fine — defaults apply.

---

### 3. Parse catalog-info.yaml

**What:** Reads `catalog-info.yaml` from `{workDir}/{effectiveAppPath}`. Parses all fields:
framework, databases, routes, dependencies, auth, storage, schedules, queues, etc.

**Catalog version:**
- Current: `0.6.0` — supports modules, registration metadata, role IDs
- Minimum: `0.2.0` — old catalogs below this are rejected (when `ENFORCE_MINIMUM_CATALOG_VERSION=true`)
- `0.3.0` — storage buckets
- `0.4.0` — unified `spec.dependencies` with Bio-ID scope grants
- `0.5.0` — deploy-gated tests, `spec.userAccess`, config declarations
- `0.6.0` — `spec.userAccess.roles`, `spec.registration`

**Framework auto-detection:** If no catalog, the builder sniffs `package.json` for Next.js,
Express, Hono, Fastify, etc. and constructs a minimal catalog.

**Fail behavior:** Missing catalog is a warn (not a hard fail) — deploy proceeds with defaults.

---

### 4. Dockerfile Strategy

**Decision tree:**
```
Has explicit Dockerfile?
  (iec.yaml OR service.dockerfilePath != 'Dockerfile')
  OR catalog.customDockerfile == 'true'?
    YES → useCustomDockerfile = true
           ├─ patchCustomDockerfile:
           │   ├─ Replace non-numeric USER → USER 1001 (K8s runAsNonRoot)
           │   └─ Inject Vault entrypoint wrapper (when VAULT_ENABLED=true)
           └─ File not found → fall back to auto-generate
    NO  → ENABLE_AUTO_DOCKERFILE=true?
           YES → generateDockerfile(buildContext, catalog)
           NO  → use project Dockerfile as-is
```

**Important:** Services using a custom Dockerfile must add the Vault entrypoint wrapper
manually unless the builder patches it. See `patchCustomDockerfile()`.

**Non-root validation:** After Dockerfile resolution, `validateNonRootDockerfile()` checks that
the final `USER` directive is a number ≠ 0. Fails hard. Exception: `static` framework (nginx
runs as root internally).

**Package manager:** The generated Dockerfile auto-detects the lockfile and installs with the
matching tool — `yarn.lock` → yarn, `package-lock.json` → npm, `pnpm-lock.yaml` → pnpm. For
pnpm, the builder runs `corepack prepare pnpm@<pinned> --activate` (NOT a bare
`corepack enable pnpm`), because bare enable pulls the latest pnpm (11.x), which requires
Node ≥ 22.13 (`node:sqlite`) and crashes on the Node 20 build image. A repo's own
`packageManager` field in `package.json` still overrides the pinned default — pin one there
if your service needs a specific pnpm version. See `dockerfile-generator.ts` (`PINNED_PNPM`).

---

### 5. Deploy Gate

**When:** `ENABLE_K8S_DEPLOY=true` AND `WALLET_URL` is set AND catalog was parsed.

**Skipped for platform services:** `iec-wallet`, `iec-janus`, `tawa-web`, `koko`, `bio-id`,
`iec-test` (or any service in `DEPLOY_GATE_SKIP_SERVICES` env var).

**Three checks (in order):**

| Check | Fail behavior | Condition |
|-------|--------------|-----------|
| Org allowlist | Hard fail | `catalog.owner` not in `approved_orgs` collection or `ALLOWED_DEPLOY_ORGS` |
| Owner tamper guard | Hard fail | `catalog.owner` ≠ `service.org` (prevents charging a different org's wallet) |
| Gas reserve | Hard fail | Balance < (hosting gas × 3 months) + (storage gas × 3 months) |

**Fail-open:** If the wallet service is unreachable, the gas reserve check returns "sufficient"
and the deploy proceeds.

**Reserve formula:**
```
required = (hosting_gas_per_hour × 24 × 30 × 3) + (storage_gas_per_month × 3)
shortfall = max(0, required - current_balance)
```

---

### 6. Config Preflight

**When:** `catalog.configDeclarations` is non-empty (catalog 0.5.0+).

**What:** Each declared config var with `required: true` must exist in `service.config` or
`service.secrets`. Missing vars → hard fail with a list of what's missing and the `tawa config
set` command to fix it.

**Defaults:** Vars with `default:` are injected into the deploy even if not set by the user
(lowest priority — user config overrides).

---

### 7. Convention Check + SDK Upgrade Gate

**Convention check:** `checkConventions()` runs Koko gate rules against the source files.
- `warn` violations: logged but non-blocking
- `block` violations: hard fail (bypass with `tawa deploy --skip-convention-check --reason "..."`)
- Skip is always logged to Septor audit regardless

**SDK upgrade gate:** For each detected package version bump vs. the previous deploy manifest:
- Checks Koko for a changelog entry at `{package}@{version}`
- No changelog entry + Koko reachable → hard fail
- Koko unreachable → warn and proceed (fail-open)
- Bypassed if `--skip-convention-check` was set

---

### 8. Docker Build + Push

**Build:**
```
docker build --platform linux/amd64 -t {DOCKER_REGISTRY}/{service}:{sha7} -f {dockerfile} {buildContext}
```

**Context size check:** Warns if build context exceeds threshold (avoids accidental large builds).

**Push:** `docker push {imageTag}` to the DigitalOcean registry.

**Fail behavior:** Hard fail on any non-zero exit.

---

### 9. Namespace + Koko Namespace Registration

**Namespace:** `kubectl create namespace {service}-{environment}` (idempotent).

**Koko registration:** Registers namespace with a gas multiplier:
- `sandbox` → `0.1` (90% discount on all gas charges)
- all others → `1.0`

---

### 10. Database Provisioning

**Two modes:**

**Phase 2 (Vault, preferred for MongoDB):**
- Vault feature-flagged via `VAULT_ENABLED=true` + `isVaultHealthy()`
- Creates Vault MongoDB dynamic credentials role per service
- Creates Vault policy + K8s auth binding + ServiceAccount
- Returns Vault Agent pod annotations for sidecar injection
- Pod reads credentials from `/vault/secrets/` at runtime (short-lived, auto-rotated)
- Falls back to Phase 1 on any Vault error

**Phase 1 (static K8s secrets, fallback):**
- Calls `provisionDatabases()` per database type
- MongoDB: creates `svc_{service}_{env}` user with `readWrite` on `{service}-{env}` DB
- Redis: creates ACL user with `svc_{service}_{env}` username
- Neo4j: creates user with `svc_{service}_{env}` username
- Creates K8s Secret `{service}-db-{type}` with connection string
- **Requires:** `DB_MONGODB_ADMIN_URI`, `DB_REDIS_ADMIN_URL`, or `DB_NEO4J_ADMIN_URI`

**Env vars injected:**
| Database | Env var |
|----------|---------|
| MongoDB | `MONGODB_URI` |
| Redis | `REDIS_URL` |
| Neo4j | `NEO4J_URI`, `NEO4J_USERNAME`, `NEO4J_PASSWORD` |

**Koko DB registration:** Stores the connection string in Koko so `koko-db` CLI can discover it.
Only registers when authenticated credentials exist.

**Database sharing:** If `spec.databases[].sharedWith` is declared, stores sharing config on the
service record. Consumer services can then access the DB via `spec.consumesDatabase`.

---

### 11. Consumed Database Provisioning

**When:** `catalog.consumesDatabase` is non-empty (cross-service DB access).

**Scope grant check:**
| Status | Behavior |
|--------|----------|
| `approved` | Proceed |
| `none` | Warn + proceed (fail-open during rollout) |
| `pending` | Warn + proceed (fail-open during rollout) |
| `denied` / `revoked` | Hard fail |
| Koko unreachable | Warn + proceed |

**What:** Creates a `svc_{consumer}_on_{owner}_{env}` MongoDB user with the declared access
level (`readOnly` or `readWrite`) on the owner's database.

---

### 12. Object Storage Provisioning (catalog 0.3.0+)

**Requires:** Vault enabled + healthy. MinIO running at `MINIO_HOST:MINIO_PORT`.

**Steps:**
1. Create bucket via MinIO SDK (idempotent)
2. Set quota via `mc quota set` CLI
3. `ensureVaultRole` — creates Vault MinIO role with an inline IAM policy scoped to the bucket
4. `ensureVaultPolicy` — adds MinIO creds path to the Vault policy
5. `ensureKubeAuthRole` — ensures K8s auth can issue tokens for the service's ServiceAccount
6. Returns Vault Agent annotations for sidecar injection

**Tiers and gas:**
| Tier | Quota | Gas/month |
|------|-------|-----------|
| `s3-sm` | 1 GB | 200 |
| `s3-md` | 5 GB | 800 |
| `s3-lg` | 25 GB | 3,000 |
| `s3-xl` | 100 GB | 10,000 |

**Env vars (injected via Vault Agent to `/vault/secrets/storage`):**
`S3_BUCKET`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_ENDPOINT`, `S3_REGION`

---

### 13. Dependency Resolution (catalog 0.4.0+)

**Two formats in catalog-info.yaml:**

**Legacy `spec.internalDependencies`** (no scopes):
- No Bio-ID scope check
- Always resolves via Koko to K8s or Janus URL
- Injects `{SERVICE}_URL` + `{SERVICE}_CHARGE_MODE`

**Unified `spec.dependencies`** (with scopes):
1. If scopes non-empty: check Bio-ID scope grant status
2. `denied`/`revoked` → block (hard fail after OAuth provisioning pass), no URL injected
3. `pending` → block (awaiting approval), no URL injected
4. `none` → record as a potential blocker **but still resolve + inject the URL**, then let the
   grant-creation pass decide: target has no Bio-ID module → fail-open (proceed); same-org → auto-approved
   (proceed); cross-org with a module → creates a `pending` request that re-blocks. Injecting the URL here
   is what makes fail-open / auto-approved deps actually functional.
5. `unreachable` → warn + proceed (URL injected)
6. `approved` → proceed (URL injected)
7. `transport: gateway` → no URL injection (consumer calls Janus public gateway)
8. `transport: direct` → inject K8s internal URL (same-org only; cross-org silently routes via Janus)
9. `transport: janus` → inject Janus proxy URL (`{JANUS_URL}/i/{service}`)

> URL injection also requires the dependency to be **registered in Koko** — a dep that isn't in the
> registry resolves to no URL (warn), independent of grant status.

**URL formats:**
| Transport | Injected URL |
|-----------|-------------|
| `direct` | `http://{svc}.{svc}-{env}.svc.cluster.local:3000` |
| `janus` | `http://janus.janus-{env}.svc.cluster.local:3000/i/{svc}` |

**Charge mode injected alongside URL:** `{SERVICE}_CHARGE_MODE = callerOrg | service`

---

### 14. OAuth Client Provisioning

**When:** `catalog.authMode == 'sso'` OR `catalog.authMode == 'service-only'` OR service has
any scoped dependencies.

**What:** Calls Bio-ID `POST /api/admin/oauth-clients` (idempotent — upserts).

**Redirect URI registered:**
- Sandbox: `https://{service}.sandbox.tawa.pro/api/auth/callback`
- Prod: `https://{service}.tawa.pro/api/auth/callback`
- Custom domain (if `dnsVerified`): `https://{customDomain}/api/auth/callback`

**Env vars injected:**
- `BIO_CLIENT_ID` — OAuth client ID (`{service}-{environment}`)
- `BIO_CLIENT_SECRET` — OAuth client secret (rotated on each deploy)
- `BIO_ID_URL` — Bio-ID public URL
- `BIO_ID_CALLBACK_URL` — Full callback URL for this service

---

### 15. Scope Grant Creation

**When:** Service has scoped dependencies (`hasScopedDeps = true`).

**For each dep with scopes that doesn't have an approved grant:**
1. `POST /api/admin/scope-grants` in Bio-ID
2. Same-org → auto-approved immediately
3. Cross-org → status `pending` — target org must approve in Console → Permissions → API Access
4. Bio-ID unreachable → fail-open (auto-add to approved set, proceed)
5. Target module not registered → fail-open

**Block enforcement:** After all scope grant creation attempts, any deps still in the `blocked`
list (`pending`/`denied`/`revoked`/`none` that weren't auto-approved or fail-open) cause a hard fail.

> **Important:** A `pending` grant does NOT inject the `{SERVICE}_URL` — the resolver `continue`s
> past URL injection when a dep is blocked. So a stuck-pending grant looks like *"the URL never got
> injected"* at runtime, but the root cause is the unapproved grant. Approve it, then redeploy.

**Surfacing (so you know exactly what to approve):** Each blocking dep is emitted as its own build
log line and persisted to a structured `build.blockedGrants` array on the build record:

```
[blocked] relay [relay:send] — grant pending (grantId: a7d21c07-…). Approve at https://tawa.insureco.io/console/permissions/api-access
```

```jsonc
// build.blockedGrants[]
{ "service": "relay", "scopes": ["relay:send"], "status": "pending",
  "grantId": "a7d21c07-…", "approveUrl": ".../console/permissions/api-access",
  "reason": "Scope grant pending for relay [relay:send] (grantId: …) — approve at …" }
```

The `Deploy blocked: …` error reply also carries the grantId inline. Approve via the console, or
`PATCH /api/admin/scope-grants/{grantId}` with `{ "status": "approved" }` (super_admin or owner of
the target service).

---

### 16. Module Registration (Bio-ID)

**When:** `catalog.modules` is non-empty.

**What:** Registers each module with Bio-ID. Returns a per-service API key injected as
`BIO_INTERNAL_KEY` into the pod (scoped to this service, not the builder's master key).

**Fail behavior:** Non-blocking — logs warning if registration fails.

---

### 17. userAccess Provisioning (catalog 0.5.0+)

**When:** `catalog.userAccess` is declared.

**What:**
- Injects `BIO_USERS_URL = https://bio.tawa.pro/api/v2/users`
- Registers the user access scope in Koko (for tawa-web Console display)
- `scope: cross-org` → lists the service as an allowed consumer in the Koko record

**Note:** `BIO_SERVICE_KEY` is NOT currently injected by the builder — if your service needs
one, provision it separately.

---

### 18. Helm Deploy

**Command pattern:**
```bash
helm upgrade --install {service} {chartPath} \
  --namespace {namespace} \
  --create-namespace \
  --set image.tag={imageTag} \
  --set image.repository={registry}/{service} \
  --set service.port={port} \
  --set health.endpoint={healthEndpoint} \
  --set secretRef={service}-managed-secrets \
  --set env.KEY=VALUE  ... (all provisionedEnvVars + service.config)
  # + Vault Agent annotations if Vault active
  # + NetworkPolicy labels for direct-transport deps
```

**Helm chart priority:**
1. `.iec.yaml` `helmChart` field
2. `service.helmChart` stored in builder DB (set via `tawa services update --helm-chart`)
3. Auto-discovered: `helm/Chart.yaml`, `helm/{service}/Chart.yaml`, `helm/*/Chart.yaml`, `chart/Chart.yaml`
4. Built-in `helm/default-service` chart (fallback)

**Secrets:** Managed secrets decrypted from `service.secrets` → K8s Secret
`{service}-managed-secrets` → mounted via `envFrom.secretRef`.

**NetworkPolicy:** Pods default to receiving traffic only from `tawa.pro/role: janus` namespace
(Janus sidecar). Direct-transport dependencies label the namespace so NetworkPolicy allows
direct pod-to-pod traffic.

---

### 19. DNS Configuration

**When:** `ENABLE_DNS_MANAGEMENT=true`.

**What:** Creates or updates a Cloudflare CNAME record:
- `{service}.{env}.tawa.pro` (sandbox/uat) or `{service}.tawa.pro` (prod) → `INGRESS_TARGET`

**Fail behavior:** DNS errors are non-fatal — logged as `[warn]`, deploy continues.

After DNS: registers domain binding in Koko.

---

### 20. Post-Deploy Pod Health Check

**What:** `kubectl get pods -n {namespace}` + recent events. Logged to build output. Stored
as `build.diagnostics` if issues detected.

**Fail behavior:** Non-fatal — diagnostics only. Does not block subsequent steps.

---

### 21. Deploy-Gated Tests (catalog 0.5.0+)

**When:** `IEC_TEST_URL` is set AND `catalog.tests` is non-empty.

**Target URL:** Internal cluster URL (`http://{service}.{namespace}.svc.cluster.local:{port}`)
— avoids external DNS propagation latency.

**Flow:**
1. `POST {IEC_TEST_URL}/runs` with test specs
2. Poll `GET /runs/{id}` every `TEST_RUNNER_POLL_INTERVAL_MS` (default 5s)
3. Timeout after `TEST_RUNNER_MAX_POLL_MS` (default 10 minutes)

**Suites:**
| Suite | Environments |
|-------|-------------|
| `smoke` | All |
| `e2e` | sandbox, uat |

**Fail behavior:** Hard fail if tests fail. Deployment NOT rolled back.

---

### 22. Koko Service Registration

**What:** `registerOrUpdateServiceInKoko` — registers/updates the service in the service
registry with:
- Routes (path, methods, auth, gas cost)
- Owner (org slug)
- Pod tier + storage tiers
- Direct-transport dependency labels (for NetworkPolicy)

---

### 23. Queue + Schedule Bindings

**What:** `registerQueueBindings` / `registerScheduleBindings` — tells Koko the upstream K8s
URL for iec-queue and iec-cron to call when delivering jobs or firing schedules.

**Fail behavior:** Partial success is logged. Never blocks deploy.

---

### 24. Uptime Monitor (iec-pulse)

**When:** `IEC_PULSE_URL` is set.

**What:** Registers a monitor for `https://{service}.{env}.tawa.pro{healthEndpoint}`.
- prod: 30s poll interval
- sandbox/uat: 60s poll interval

**Fail behavior:** Fire-and-forget — never blocks.

---

## Fail-Open vs Fail-Hard Quick Reference

| System | Scenario | Behavior |
|--------|----------|----------|
| Wallet service | Unreachable | Fail-open (deploy proceeds) |
| Wallet service | Insufficient gas | Hard fail |
| Bio-ID | Unreachable (scope grant check) | Fail-open (warn) |
| Bio-ID | Grant denied/revoked | Hard fail |
| Bio-ID | Grant pending | Hard fail (no URL injected until approved) |
| Bio-ID | Grant none (no module / same-org) | Fail-open / auto-approve — proceed **with** URL injected |
| Bio-ID | Grant none (cross-org, module exists) | Creates pending request → hard fail |
| Koko | Unreachable (dep resolution) | Fail-open (skip dep) |
| Koko | Changelog missing for SDK bump | Hard fail |
| Koko | Unreachable (changelog check) | Fail-open (warn) |
| Vault | Unhealthy | Fall back to static credentials |
| DNS (Cloudflare) | Error | Fail-open (warn) |
| Post-deploy diagnostics | Error | Fail-open (warn) |
| Deploy-gated tests | Fail | Hard fail (pod stays) |
| Docs publish | Error | Fail-open |
| Deploy snapshot (Koko) | Error | Fail-open |
| Uptime monitor | Error | Fail-open |
| Convention check | Blocking violation | Hard fail |
| Non-root Dockerfile | Root user detected | Hard fail |
| Config preflight | Missing required vars | Hard fail |
| Owner tamper guard | Owner mismatch | Hard fail |

---

## Environment Variables

All builder env vars with their defaults and purposes:

| Variable | Default | Purpose |
|----------|---------|---------|
| `PORT` | `3002` | Builder API port |
| `MONGODB_URI` | `mongodb://localhost:27017/builder` | Builder's own DB |
| `REDIS_URL` | `redis://localhost:6379` | Build queue |
| `WORKSPACE_DIR` | `/tmp/iec-builds` | Clone workspace |
| `DOCKER_REGISTRY` | `registry.insureco.io/insureco` | Image push target |
| `DOCKER_REGISTRY_USER` | — | Registry auth username |
| `DOCKER_REGISTRY_TOKEN` | — | Registry auth token |
| `ENABLE_AUTO_DOCKERFILE` | `true` | Auto-generate Dockerfiles |
| `ENABLE_K8S_DEPLOY` | `false` | Enable Kubernetes deployment |
| `KUBECONFIG` | — | K8s cluster credentials |
| `PLATFORM_DOMAIN` | `tawa.pro` | External URL base domain |
| `CLOUDFLARE_API_TOKEN` | — | DNS management |
| `CLOUDFLARE_ZONE_ID` | — | Cloudflare zone |
| `INGRESS_TARGET` | — | K8s ingress IP/hostname for DNS CNAME |
| `ENABLE_DNS_MANAGEMENT` | `false` | Enable Cloudflare DNS updates |
| `FORGEJO_URL` | `https://git.insureco.io` | Git host |
| `FORGEJO_TOKEN` | — | Forgejo PAT for private repos |
| `GITHUB_TOKEN` | — | GitHub PAT for private repos |
| `KOKO_URL` | `https://koko.tawa.pro` | Service registry |
| `BIO_ID_URL` | `https://bio.tawa.pro` | OAuth provider |
| `BIO_INTERNAL_KEY` | — | Builder's Bio-ID internal API key |
| `WALLET_URL` | — | Gas wallet service URL |
| `INTERNAL_SERVICE_KEY` | — | Service-to-service auth key |
| `DB_MONGODB_ADMIN_URI` | — | Admin MongoDB URI for credential provisioning |
| `DB_MONGODB_HOST` | `localhost` | MongoDB host (VPC IP for pods) |
| `DB_MONGODB_PUBLIC_HOST` | — | MongoDB public IP (for whitelist connections) |
| `DB_REDIS_ADMIN_URL` | — | Admin Redis URL for ACL provisioning |
| `DB_REDIS_HOST` | `localhost` | Redis host |
| `DB_NEO4J_ADMIN_URI/USER/PASSWORD` | — | Neo4j admin credentials |
| `DB_NEO4J_HOST` | `localhost` | Neo4j host |
| `MINIO_HOST` | `64.23.181.20` | MinIO host (public IP, builder→MinIO) |
| `MINIO_INTERNAL_HOST` | — | MinIO VPC IP (pods→MinIO, falls back to MINIO_HOST) |
| `MINIO_PORT` | `9000` | MinIO S3 API port |
| `VAULT_ADDR` | — | HashiCorp Vault address |
| `VAULT_TOKEN` | — | Builder's Vault token |
| `VAULT_ENABLED` | `false` | Enable Vault dynamic credentials |
| `ROTATION_TTL_DAYS` | `30` | Days between credential rotations |
| `IEC_TEST_URL` | — | iec-test service URL (empty = skip tests) |
| `IEC_PULSE_URL` | — | iec-pulse URL (empty = skip monitor registration) |
| `BUILD_CONCURRENCY` | `3` | Max parallel builds |
| `ENFORCE_MINIMUM_CATALOG_VERSION` | `false` | Reject catalogs below 0.2.0 |
| `DEPLOY_GATE_SKIP_SERVICES` | (platform services) | Comma-separated services that skip gas gate |
| `ALLOWED_DEPLOY_ORGS` | `insureco` | Fallback org allowlist |
| `CONFIG_ENCRYPTION_KEY` | — | AES-256-GCM key for managed secrets |
| `ANTHROPIC_API_KEY` | — | AI troubleshooter |
| `JCI_BRAIN_URL` | — | Knowledge base for troubleshooter advice |

---

## Auto-Injected Platform Env Vars (Always Available in Pods)

These are injected by the builder for every deploy — services do NOT need to declare them in
`internalDependencies`:

| Var | Value |
|-----|-------|
| `KOKO_URL` | Internal K8s URL for Koko |
| `JANUS_URL` | Internal K8s URL for Janus |
| `SEPTOR_URL` | Internal K8s URL for Septor |
| `IEC_WALLET_URL` | Internal K8s URL for iec-wallet |
| `BIO_ID_URL` | Bio-ID public URL |
| `NODE_ENV` | Always `production` for all deployed environments |
| `ENVIRONMENT` | Deployment environment: `prod`, `sandbox`, or `uat` |

---

## What Gets Written Where

| System | Written | When |
|--------|---------|------|
| **Builder MongoDB** | Build record (status, logs, imageTag, commitSha, diagnostics) | Throughout |
| **Builder MongoDB** | Service record (lastBuildId, credentialRotation, databaseCredentials, catalogSpec) | On success |
| **K8s** | Namespace | Step 18 |
| **K8s** | `{service}-db-{type}` secret (DB connection strings) | Step 10 |
| **K8s** | `{service}-oauth` secret | Step 14 |
| **K8s** | `{service}-managed-secrets` secret (user-defined) | Step 18 (Helm) |
| **K8s** | Deployment, Service, Ingress, ConfigMap, NetworkPolicy | Step 18 (Helm) |
| **Vault** | MongoDB dynamic credential role | Step 10 |
| **Vault** | Policy + K8s auth binding | Step 10 |
| **Vault** | MinIO credential role | Step 12 |
| **MinIO** | Bucket + quota | Step 12 |
| **Koko** | Namespace record (gas multiplier) | Step 9 |
| **Koko** | Database record (connection string) | Step 10 |
| **Koko** | Service record + routes | Step 22 |
| **Koko** | Domain binding | Step 19 |
| **Koko** | Queue + schedule bindings | Step 23 |
| **Koko** | Deploy snapshot manifest | Step 18 (fire-and-forget) |
| **Koko** | Available scopes (sharedWith) | Step 22 |
| **Koko** | Scope grants (auto-seed) | Step 22 |
| **Koko** | Onboarding spec | Step 22 |
| **Koko** | Role records (0.6.0+) | Step 23 |
| **Koko** | Registration metadata (0.6.0+) | Step 23 |
| **Bio-ID** | OAuth client (client ID + secret + redirect URI) | Step 14 |
| **Bio-ID** | Scope grant request | Step 15 |
| **Bio-ID** | Module registration | Step 16 |
| **Cloudflare** | CNAME DNS record | Step 19 |
| **Septor** | `service.deployed` audit event | Step 18 |
| **Septor** | `namespace.created` audit event | Step 9 |
| **Septor** | `build.failed` audit event | On error |
| **iec-pulse** | Uptime monitor | Step 24 (fire-and-forget) |

---

## Common Failure Patterns

### "Deploy gate: insufficient gas reserve"
Org wallet balance is below the 3-month reserve. Run `tawa wallet buy N` to top up, then redeploy.

### "Deploy gate: catalog spec.owner X does not match registered service owner Y"
Someone changed `spec.owner` in catalog-info.yaml. Either revert it, or use
`tawa services update <id> --org <new-org>` if the transfer is intentional.

### "Deploy blocked: Access to X has been denied/revoked"
The scope grant for a `spec.dependencies` entry was explicitly denied or revoked in Bio-ID.
Check Console → Permissions → API Access.

### "Deploy blocked: Scope grant pending for X [scopes] (grantId: …)"
A cross-org `spec.dependencies` entry created a grant request that nobody approved yet. The build
log line `[blocked] X [scopes] — grant pending (grantId: …)` and `build.blockedGrants[]` name the
exact grant. Approve it in Console → Permissions → API Access, or
`PATCH /api/admin/scope-grants/{grantId}` `{ "status": "approved" }` (super_admin / target-service
owner), then redeploy. Note: while pending, `{X}_URL` is **not** injected — fix the grant, don't
hardcode the URL.

### "Convention check failed: N blocking violation(s)"
Source code violates a Koko gate rule (e.g., deprecated SDK usage). Fix the violations or
use `tawa deploy --skip-convention-check --reason "..."`.

### "Config preflight failed: missing required config vars: KEY1, KEY2"
`catalog.configDeclarations` lists required vars that aren't set. Run `tawa config set KEY1=... KEY2=...`.

### "Undocumented upgrade: X Y → Z has no changelog entry"
An SDK was bumped without a Koko changelog entry. Run `tawa changelog --sdk X` to document it.

### "Deploy-gated tests failed"
Post-deploy smoke tests failed. The pod is still running — use `kubectl logs` to debug.
Re-deploy after fixing will re-run tests.

### Pod stuck in CrashLoopBackOff after deploy
Check `build.diagnostics` in the builder DB or run `kubectl describe pod` in the namespace.
Common causes: missing env vars, incorrect health endpoint, DB connection failure on startup.

## Related: Builder Logs

The builder's own PM2 logs are redacted (write-time + read-time), rotated, and exposed through a
platform-admin API. See [Builder Logs — Redaction, Rotation & Admin API](./builder-logs.md) for
the `GET /platform/logs` and `GET /platform/logs/tail` contracts.