# Build & Deploy Pipeline

The iec-builder executes a fixed, ordered pipeline for every `tawa deploy`. Understanding each
step — what it does, when it fails hard vs. fails open, and what it writes to each system — is
essential for debugging deploys and hardening the platform.

## Pipeline Overview

```
tawa deploy --prod
     │
     ▼
 iec-builder (builder.ts: executeBuild)
     │
     ├─ PRE-BUILD
     │   ├─ Clone/extract
     │   ├─ Catalog parse
     │   ├─ Dockerfile strategy
     │   ├─ Deploy gate (wallet reserve)
     │   ├─ Config preflight
     │   └─ Convention check / SDK upgrade gate
     │
     ├─ BUILD
     │   ├─ docker build
     │   └─ docker push → registry
     │
     └─ DEPLOY  (ENABLE_K8S_DEPLOY=true)
         ├─ Namespace + Koko namespace registration
         ├─ Database provisioning (Vault or static K8s secrets)
         ├─ Consumed database provisioning
         ├─ Object storage provisioning (Vault, catalog 0.3.0+)
         ├─ Dependency resolution (Koko + Bio-ID scope grants)
         ├─ OAuth client provisioning
         ├─ Claude access: Anthropic identity + ids (when switched on)
         ├─ Scope grant creation
         ├─ Module registration (Bio-ID)
         ├─ userAccess provisioning
         ├─ helm upgrade --install
         ├─ Deploy snapshot (Koko, fire-and-forget)
         ├─ DNS configuration (Cloudflare)
         ├─ Post-deploy pod health check
         ├─ Vault secret readback (gates re-rolls)
         ├─ Deploy-gated tests (iec-test, catalog 0.5.0+)
         ├─ Service registration in Koko
         ├─ Queue + schedule binding registration + prune
         ├─ Role + registration metadata (catalog 0.6.0+)
         └─ Uptime monitor (iec-pulse, fire-and-forget)
```

## Status Flow

```
queued → cloning → building → pushing → deploying → testing → completed
                                                            ↘ failed (any step)
```

**Two triggers, one build.** A `tawa deploy` for a service + environment whose branch tip is
already being built does not queue a second build: the trigger answers `202` with the
in-flight build's id and `deduplicated: true`, and `tawa deploy --watch` attaches to it. An
in-flight build that has not yet cloned always counts as a build of the tip; one that has
counts only if its commit is still the branch head (one `git ls-remote`). If `main` moved in
between, you get a real new build. The in-flight build's log records who attached (iec-builder#110).

---

## First Deploy: the deploy → config → deploy ordering

**Nothing can be attached to a service until the service record exists — and on a new service,
the first `tawa deploy` is what creates it.**

Every service-keyed thing — config vars, managed secrets, database credentials, whitelist
entries, authorized orgs — hangs off a record in the builder's `services` collection. Until that
record exists, the routes that write those things have nothing to find, and 404.

```
tawa deploy --prod           # 1. creates the service record, then builds
tawa config set KEY=value    # 2. now this works
tawa deploy --prod           # 3. redeploy — this is what puts the value in the pod
```

> Full config reference, including `tawa config set --secret KEY --from-locker lk://insureco/<vault>/<item>` for a
> secret whose plaintext never passes through a terminal: [config](config.md).

**Step 3 is not optional.** `tawa config set` writes to the builder's config store, **not** to
the running pod. Config and secrets are materialized into Kubernetes at deploy time (steps 4b
and 18), so a value set after a deploy reaches the pod only on the **next** deploy.

**Run it in the wrong order** and the builder's config routes answer — not the CLI:

```
Service 'my-new-api' not found. A service record is created by its first deploy,
so config cannot be set before then. …
```

That is not a typo and not a permissions problem: the service has simply never been deployed.

### How the record actually gets created

The record is written by `POST /services` (`servicesRouter.post('/')`), and two things call it:

| Trigger | What happens |
|---------|--------------|
| `tawa deploy` on an unlinked project | Auto-links first: reads `metadata.name` from `catalog-info.yaml`, detects the git remote, registers the service, **then** triggers the build |
| `tawa link` | Registers (or updates) the service and stops — **no build** |

Note the build trigger itself does **not** create anything: `POST /builds/trigger` 404s with
`Service not found` if the record is missing. Registration strictly precedes every build.

**So `tawa link` is the escape hatch** when you want config in place before the first build ever
runs — it creates the record without spending a build:

```
tawa link                    # creates the service record, no build
tawa config set KEY=value    # works immediately
tawa deploy --prod           # first build already has its config
```

### It compounds on a first deploy

A new service with a scoped `spec.dependencies` entry can fail the **scope gate** (steps 13/15),
and then — when you go to set the config it needs — fail again on **config-before-deploy**. Two
unrelated-looking failures, one cause: nothing exists yet. A first deploy that goes in this order
hits neither:

1. `tawa preflight` — catch catalog problems before spending a build.
2. `tawa link` — create the service record (or let the first `tawa deploy` do it).
3. `tawa config set …` — now the service exists, so this works.
4. `tawa deploy --prod` — may still block on a scope grant; the record already exists either way.
5. Approve any blocked grants — the block message names the full grant ID and the exact command
   (`tawa scopes grant <grantId>`). A scope grant is a **deploy-time** gate: approving it is what
   unblocks the build and gets `{SERVICE}_URL` injected. It is not a per-request runtime check.
6. `tawa deploy --prod` — green.

> **Chicken-and-egg with `configDeclarations`.** If the catalog declares a **required** config var
> (catalog 0.5.0+), the config preflight (step 6) hard-fails a first deploy with no value set —
> and you cannot set one before the record exists. Either `tawa link` first (above), or let the
> first deploy fail: registration happens before the build is queued, so the record survives and
> `tawa config set` then works. Declaring a `default:` avoids the round trip entirely.

---

## Step-by-Step Reference

### 1. Clone / Extract

**What:** Clones the repo at the specified `commitSha` using the branch as a hint. If a tarball
was uploaded (via `POST /builds/upload`), it extracts that instead.

**Auth:** Forgejo repos get a PAT injected (`forgejo-token:TOKEN@host`). GitHub SSH URLs are
converted to HTTPS with a GitHub PAT. Both are handled by `injectGitCredentials()`.

**Output:** Actual commit SHA is resolved and backfilled to the build record. Workspace at
`{WORKSPACE_DIR}/{buildId}`.

**Fail behavior:** Hard fail — no source = no build.

---

### 2. Read .iec.yaml (monorepo config)

**What:** Reads `.iec.yaml` from the repo root (or `{service.appPath}/.iec.yaml`). Controls:
- `appPath` — subdirectory containing the service
- `dockerfile` — custom Dockerfile path
- `buildContext` — custom build context (relative to appPath)
- `helmChart` — custom Helm chart path

**Priority for appPath:** `service.appPath` (DB) > `.iec.yaml` > inferred from
`service.dockerfilePath`.

**Fail behavior:** Missing `.iec.yaml` is fine — defaults apply.

---

### 3. Parse catalog-info.yaml

**What:** Reads `catalog-info.yaml` from `{workDir}/{effectiveAppPath}`. Parses all fields:
framework, databases, routes, dependencies, auth, storage, schedules, queues, etc.

**Catalog version:**
- Current: `0.6.0` — supports modules, registration metadata, role IDs
- Minimum: `0.2.0` — old catalogs below this are rejected (when `ENFORCE_MINIMUM_CATALOG_VERSION=true`)
- `0.3.0` — storage buckets
- `0.4.0` — unified `spec.dependencies` with Bio-ID scope grants
- `0.5.0` — deploy-gated tests, `spec.userAccess`, config declarations
- `0.6.0` — `spec.userAccess.roles`, `spec.registration`

**Framework auto-detection:** If no catalog, the builder sniffs `package.json` for Next.js,
Express, Hono, Fastify, etc. and constructs a minimal catalog.

**Fail behavior:** Missing catalog is a warn (not a hard fail) — deploy proceeds with defaults.

---

### 4. Dockerfile Strategy

**Decision tree:**
```
Has explicit Dockerfile?
  (iec.yaml OR service.dockerfilePath != 'Dockerfile')
  OR catalog.customDockerfile == 'true'?
    YES → useCustomDockerfile = true
           ├─ patchCustomDockerfile:
           │   ├─ Replace non-numeric USER → USER 1001 (K8s runAsNonRoot)
           │   └─ Inject Vault entrypoint wrapper (when VAULT_ENABLED=true)
           └─ File not found → fall back to auto-generate
    NO  → ENABLE_AUTO_DOCKERFILE=true?
           YES → generateDockerfile(buildContext, catalog)
           NO  → use project Dockerfile as-is
```

**Important:** Services using a custom Dockerfile must add the Vault entrypoint wrapper
manually unless the builder patches it. See `patchCustomDockerfile()`.

**Next.js `public/`.** The generated Next.js Dockerfile runs `mkdir -p /app/public` in the
builder stage before the runner stage copies it out. `public/` is optional in Next.js —
`create-next-app` does not create it — and until 2026-08-21 the unconditional copy failed the
image build for any service without one, after a full install and compile. Services no longer
need a placeholder `public/.gitkeep`; existing ones are harmless and can be deleted whenever
convenient. Custom Dockerfiles are on their own here, as with the Vault wrapper.

**Static sites: content types.** The generated static Dockerfile appends five mappings to
nginx's `/etc/nginx/mime.types` — `md`, `mjs`, `webmanifest`, `ttf`, `otf`. Stock nginx maps
none of them, so before 2026-09-17 each served as `default_type application/octet-stream`.
`.mjs` was the sharp edge: a browser refuses an ES module with the wrong content type, so a
static site's imports failed outright. Custom Dockerfiles are on their own here too.

If you write your own nginx config, **where** you put a `types { ... }` block decides whether it
adds a mapping or destroys the rest of them:

| Placement | Effect |
|-----------|--------|
| Top level of a `conf.d/*.conf` file (the **http** level, alongside nginx.conf's own `include mime.types`) | **Appends.** `types { text/markdown md; }` there adds `md` and leaves `text/css` alone — this is the working form |
| Inside a `server` or `location` block | **Replaces** the inherited map. The same one line makes every `.css` on the site serve as `application/octet-stream` |

The difference is the inheritance rule: a `types` directive at a *deeper* level redefines what it
inherits, while two at the *same* level merge into one map. Both were checked against a running
container — the server-level version really does take CSS down with it.

A block at either level also cannot re-include `mime.types` to compensate: that file is itself a
`types { ... }` block, and nesting one inside another is a syntax error nginx refuses to start on
(`unexpected "{" in /etc/nginx/mime.types:2`). The generated Dockerfile sidesteps the question by
appending to `mime.types` itself.

**Vault entrypoint wrapper — how dynamic secrets reach `process.env`.** With Vault dynamic
credentials, the Vault Agent renders each secret as a **file** under `/vault/secrets/<name>`
(a block of `KEY=VALUE` lines — e.g. `/vault/secrets/mongodb` → `MONGODB_URI=…`, `/vault/secrets/storage`
→ `S3_*`). The generated `ENTRYPOINT` (`/usr/local/bin/entrypoint.sh`) **sources those files into
the environment and then `exec "$@"`** — so from the app's perspective they are ordinary env vars.
You never read `/vault/secrets/*` from application code.

The wrapper **waits (bounded ~10s) for every secret file to be non-empty before sourcing**, then
sources only non-empty files (`[ -s "$f" ]`). This closes a startup **race**: the `vault-agent`
sidecar starts alongside the app container and **re-renders** the dynamic creds at that instant, so
an entrypoint that sourced immediately (old `[ -f "$f" ]` existence-only check) could read a
half-written/truncated file and boot with `MONGODB_URI`/`S3_*` **unset** — leaving the app with no
database. The wait loop is defined once in `VAULT_ENTRYPOINT` (`dockerfile-generator.ts`) and is
used by both the generated Dockerfiles and `patchCustomDockerfile()`. Reference:
https://tawa.insureco.io/reference/build-pipeline

Because the creds are dynamic (rotated by Vault), they **cannot** be pinned as static
`tawa config set` values — the running app must read the live-rendered files via this entrypoint.

**Non-root validation:** After Dockerfile resolution, `validateNonRootDockerfile()` checks that
the final `USER` directive is a number ≠ 0. Fails hard. Exception: `static` framework (nginx
runs as root internally).

**Package manager:** The generated Dockerfile auto-detects the lockfile and installs with the
matching tool — `yarn.lock` → yarn, `package-lock.json` → npm, `pnpm-lock.yaml` → pnpm. For
pnpm, the builder runs `corepack prepare pnpm@<pinned> --activate` (NOT a bare
`corepack enable pnpm`), because bare enable pulls the latest pnpm (11.x), which requires
Node ≥ 22.13 (`node:sqlite`) and crashes on the Node 20 build image. A repo's own
`packageManager` field in `package.json` still overrides the pinned default — pin one there
if your service needs a specific pnpm version. See `dockerfile-generator.ts` (`PINNED_PNPM`).

---

### 4b. Build-time env injection (`.env.production`)

**When:** Immediately before `docker build`, if the catalog declares build-time vars via
`insureco.io/env-vars`.

The builder materializes the declared vars from the **config store** (`tawa config set` values +
managed secrets) into a **`.env.production`** file at the build-context root. `npm run build`
(CRA, Vite, and Next.js all read `.env*` files at build time) then bakes them into the client
bundle.

**Why this step exists:** `tawa config set` values live in the builder DB, **not the repo**
(whose `.env` is normally gitignored), and `docker build` runs **without `--build-arg`**. Without
this step, `REACT_APP_*` / `NEXT_PUBLIC_*` vars compile to their hardcoded fallback (often
`localhost`) and the deployed **static** bundle calls the wrong API URL. (Symptom: a static/CRA
app whose login/API calls hit the app's own host or `localhost` instead of the real API.)

**What is written:** ONLY the keys listed in `insureco.io/env-vars` that have a value in config.
These are public-by-design (they end up in the client bundle). The **full secret set is never
written here** — non-declared secrets stay runtime-only (injected as K8s secrets at deploy).
Value resolution: `service.config` merged over decrypted `service.secrets`.

**Build log line:** `Wrote N build-time env var(s) to .env.production: KEY1, KEY2` (names only —
values are never logged).

**Belt-and-suspenders:** client apps can also resolve their API base at **runtime** from
`window.location.hostname` so they don't depend on build-time injection at all. Recommended for
static SPAs as a fallback. See `writeBuildEnvFile()` in `builder.ts`.

---

### 5. Deploy Gate

**When:** `ENABLE_K8S_DEPLOY=true` AND `WALLET_URL` is set AND catalog was parsed.

**Skipped for platform services:** `iec-wallet`, `iec-janus`, `tawa-web`, `koko`, `bio-id`,
`iec-test` (or any service in `DEPLOY_GATE_SKIP_SERVICES` env var).

**Three checks (in order):**

| Check | Fail behavior | Condition |
|-------|--------------|-----------|
| Org allowlist | Hard fail | `catalog.owner` not in `approved_orgs` collection or `ALLOWED_DEPLOY_ORGS` |
| Owner tamper guard | Hard fail | `catalog.owner` ≠ `service.org` (prevents charging a different org's wallet) |
| Gas reserve | Hard fail | Balance < (hosting gas × 3 months) + (storage gas × 3 months) |

**Fail-open:** If the wallet service is unreachable, the gas reserve check returns "sufficient"
and the deploy proceeds.

**Reserve formula:**
```
required = (hosting_gas_per_hour × 24 × 30 × 3) + (storage_gas_per_month × 3)
shortfall = max(0, required - current_balance)
```

**Authorized orgs (multi-org collaboration):** The deploy gate keys off `catalog.owner`
(= `spec.owner` = the **primary** org). A service can additionally grant one or more
**authorized orgs** whose devs get full-parity access — deploy, config, secrets, DB — without
being moved into the owning org. The deploy gate is unaffected:

- The authorized-org dev never changes `spec.owner`, so the **owner tamper guard still passes**.
- **Gas is still billed to the primary org's wallet** (`deriveWalletId(catalog.owner)`).
- The authorized org does **not** need to be in `approved_orgs` — the allowlist checks
  `catalog.owner`, not the deploying user.

Authorization is a per-service access check only (`user.org === service.org` OR
`user.org ∈ service.authorizedOrgs`), enforced on the build trigger, build views, and all
service management routes. Manage it as an admin of the **owning** org:

```
GET    /services/:name/authorized-orgs                 # list owner + authorized orgs
POST   /services/:name/authorized-orgs { orgSlug }     # add   (owning-org admin only)
DELETE /services/:name/authorized-orgs/:orgSlug        # remove (owning-org admin only)
```

Only owning-org admins (or super_admin / platform service) may add or remove authorized orgs —
an authorized org cannot add further orgs. A build triggered by an authorized-org dev is stamped
with that dev's org, but build history is resolved through the service so the owning org and every
authorized org share one view.

---

### 6. Config Preflight

**When:** `catalog.configDeclarations` is non-empty (catalog 0.5.0+).

**What:** Each declared config var with `required: true` must exist in `service.config` or
`service.secrets`. Missing vars → hard fail with a list of what's missing and the `tawa config
set` command to fix it.

**Defaults:** Vars with `default:` are injected into the deploy even if not set by the user
(lowest priority — user config overrides).

**Ordering:** `tawa config set` only works once the service record exists, and the value it
writes reaches the pod on the *next* deploy — see
[First Deploy: the deploy → config → deploy ordering](#first-deploy-the-deploy--config--deploy-ordering).

---

### 7. Convention Check + SDK Upgrade Gate

**Convention check:** `checkConventions()` runs Koko gate rules against the source files.
- `warn` violations: logged but non-blocking
- `block` violations: hard fail (bypass with `tawa deploy --skip-convention-check --reason "..."`)
- Skip is always logged to Septor audit regardless

**SDK upgrade gate:** For each detected package version bump vs. the previous deploy manifest:
- Checks Koko for a changelog entry at `{package}@{version}`
- No changelog entry + Koko reachable → hard fail
- Koko unreachable → warn and proceed (fail-open)
- Bypassed if `--skip-convention-check` was set

"The previous deploy manifest" is the one `recordDeploySnapshot` stored after the last
**successful** deploy (Step 18). This step only reads it; it never writes.

> **Correction (2026-09-06):** this gate had never fired. The builder's manifest POST
> lacked `commit`, `sdkVersions`, and `deployedBy` — all three required by Koko — so
> every manifest was rejected `400 VALIDATION_ERROR`, nothing was ever stored, and "the
> previous deploy manifest" was always empty. Because the write is fire-and-forget, no
> build ever warned. Fixed in iec-builder#95: the body now matches Koko's schema, and a
> rejection is written to the build log as `[warn] Deploy snapshot was not stored in
> Koko: …`. There is no backfill — the first deploy after the fix stores a manifest, and
> the gate has something to diff against from the **second** deploy onward.

---

### 8. Docker Build + Push

**Build:**
```
DOCKER_BUILDKIT=1 docker build -t {DOCKER_REGISTRY}/{service}:{sha7} -f {dockerfile} {buildContext}
```

**BuildKit caches (iec-builder#171).** Every generated Dockerfile starts with
`# syntax=docker/dockerfile:1` and uses cache mounts, which only BuildKit understands. BuildKit is
the default on the builder host's Docker 29; `DOCKER_BUILDKIT=1` is set explicitly so a daemon
config change cannot quietly turn every mount into a parse error.

| Mount | Target | Shared? |
|-------|--------|---------|
| `id=npm` | `/root/.npm` | across services (content-addressed, integrity-checked) |
| `id=yarn` | `/usr/local/share/.cache/yarn` | across services, `sharing=locked` (yarn v1's cache is unsafe with concurrent writers) |
| `id=nextjs-{service}` | `/app/.next/cache` (Next.js build step only) | **per service**; one tenant's compiler cache is never readable by another's build |

Layer caching already skips the install when `package.json` and the lockfile are unchanged. The
npm/yarn cache mounts help a little when the lockfile **does** change: packages are re-extracted
from the host cache instead of being downloaded.

**There is no pnpm store mount, on purpose.** pnpm hardlinks packages from its store into
`node_modules`, and a hardlink cannot cross a cache-mount boundary. pnpm falls back to copying
every file, and on the builder host that made a lockfile-change install 20–30s **slower** than
fetching from the registry (measured, iec-builder#171). Do not add one to a custom Dockerfile
either. The
`.next/cache` mount lets `next build` reuse its compiler cache when source changes, where before
every deploy compiled from scratch. None of the mounted content ends up in the image.

**Custom Dockerfiles** get none of this automatically. Add the same lines yourself, using the
same ids so you share the host's caches:

```dockerfile
# syntax=docker/dockerfile:1
RUN --mount=type=cache,id=npm,target=/root/.npm npm ci
RUN --mount=type=cache,id=nextjs-<your-service>,target=/app/.next/cache pnpm build
```

Also give a custom Dockerfile a `.dockerignore` (at least `.git`, `node_modules`, `.next`).
Without one, `COPY . .` sends the whole clone, `.git` included, into the build context. Keep
anything the runtime reads: bio keeps `docs/**` while excluding other `*.md`
([iec-bio#134](https://git.insureco.io/insureco/iec-bio/issues/134)). Never exclude
`.env.production`: Step 4b writes the build-time vars there.

Caches last only as long as the host keeps its BuildKit cache. A host-level prune
(`docker builder prune`) empties them, and the build after that runs cold.

**Context size check:** Warns if build context exceeds threshold (avoids accidental large builds).

**Push:** `docker push {imageTag}` to `registry.insureco.io` (Forgejo's package registry; its
layers live in DO Spaces nyc3). This is the canonical copy, and a failure here is a hard failure.

**Mirror push (optional, off by default; iec-builder#175):** when `MIRROR_REGISTRY` is set, the
builder also pushes the same image to the in-cluster mirror `tawa-mirror` (insureco/tawa-cluster#2)
under the **same repo path**: `registry.insureco.io/insureco/{service}:{sha7}` becomes
`{MIRROR_REGISTRY}/insureco/{service}:{sha7}`. The builder host and the mirror share the VPC, so
the copy takes seconds.
- Why: cluster nodes egress through the VPC NAT gateway at about 1 MB/s per flow, so a 226 MB
  image took 3-4 min per node (iec-builder#173).
- **Non-blocking:** any failure is `[warn] Mirror push … failed … The deploy continues`, and nodes
  pull from upstream as before.
- A misconfiguration (a scheme or path in `MIRROR_REGISTRY`, or missing credentials) is a `[warn]`,
  and the mirror push is skipped.
- Credentials: `MIRROR_REGISTRY_USER` / `MIRROR_REGISTRY_TOKEN` (the mirror's `pusher` user, daryx
  vault `tawa-mirror-pusher`) go to `docker login --password-stdin` over stdin. They never appear
  in argv or a build log line.
- Until nodes carry a containerd `hosts.toml` entry for the mirror (planned after the DOKS 1.36
  upgrade), the copy is pushed but not yet pulled from.

**Fail behavior:** Hard fail on any non-zero exit.

---

### 9. Namespace + Koko Namespace Registration

**Namespace:** `kubectl create namespace {service}-{environment}` (idempotent).

**Koko registration:** Registers namespace with a gas multiplier:
- `sandbox` → `0.1` (90% discount on all gas charges)
- all others → `1.0`

---

### 10. Database Provisioning

**Two modes:**

**Phase 2 (Vault, preferred for MongoDB):**
- Vault feature-flagged via `VAULT_ENABLED=true` + `isVaultHealthy()`
- Creates Vault MongoDB dynamic credentials role per service
- Creates Vault policy + K8s auth binding + ServiceAccount
- Returns Vault Agent pod annotations for sidecar injection
- Pod reads credentials from `/vault/secrets/` at runtime (short-lived, auto-rotated)
- Falls back to Phase 1 on any Vault error

**Phase 1 (static K8s secrets, fallback):**
- Calls `provisionDatabases()` per database type
- MongoDB: creates `svc_{service}_{env}` user with `readWrite` on `{service}-{env}` DB
- Redis: creates ACL user with `svc_{service}_{env}` username
- Neo4j: creates user with `svc_{service}_{env}` username
- Postgres (iec-builder#231): creates login role `svc_{service}_{env}` owning database `{service}-{env}` (or `name:`), `CONNECT` revoked from `PUBLIC`. Always Phase 1 — no Vault path yet. `DB_POSTGRES_ADMIN_URI` unset → `[warn]` in the build log and nothing injected (it never falls back to an unauthenticated URL)
- Creates K8s Secret `{service}-db-{type}` with connection string
- **Requires:** `DB_MONGODB_ADMIN_URI`, `DB_REDIS_ADMIN_URL`, `DB_NEO4J_ADMIN_URI`, or `DB_POSTGRES_ADMIN_URI`

**Env vars injected:**
| Database | Env var |
|----------|---------|
| MongoDB | `MONGODB_URI` |
| Redis | `REDIS_URL` |
| Neo4j | `NEO4J_URI`, `NEO4J_USERNAME`, `NEO4J_PASSWORD` |
| Postgres | `DATABASE_URL`, `POSTGRES_URI` (same value) |

**Koko DB registration:** Stores the connection string in Koko so `koko-db` CLI can discover it.
Only registers when authenticated credentials exist.

**Database sharing:** If `spec.databases[].sharedWith` is declared, stores sharing config on the
service record. Consumer services can then access the DB via `spec.consumesDatabase`.

---

### 11. Consumed Database Provisioning

**When:** `catalog.consumesDatabase` is non-empty (cross-service DB access).

**Scope grant check:**
| Status | Behavior |
|--------|----------|
| `approved` | Proceed |
| `none` | Warn + proceed (fail-open during rollout) |
| `pending` | Warn + proceed (fail-open during rollout) |
| `denied` / `revoked` | Hard fail |
| Koko unreachable | Warn + proceed |

**What:** Creates a `svc_{consumer}_on_{owner}_{env}` MongoDB user with the declared access
level (`readOnly` or `readWrite`) on the owner's database.

---

### 12. Object Storage Provisioning (catalog 0.3.0+)

**Requires:** Vault enabled + healthy. MinIO running at `MINIO_HOST:MINIO_PORT`.

**Steps:**
1. Create bucket via MinIO SDK (idempotent)
2. Set quota via `mc quota set` CLI
3. `ensureVaultRole` — creates Vault MinIO role with an inline IAM policy scoped to the bucket
4. `ensureVaultPolicy` — adds MinIO creds path to the Vault policy
5. `ensureKubeAuthRole` — ensures K8s auth can issue tokens for the service's ServiceAccount
6. Returns Vault Agent annotations for sidecar injection

**Tiers and gas:**
| Tier | Quota | Gas/month |
|------|-------|-----------|
| `s3-sm` | 1 GB | 200 |
| `s3-md` | 5 GB | 800 |
| `s3-lg` | 25 GB | 3,000 |
| `s3-xl` | 100 GB | 10,000 |

**Env vars (injected via Vault Agent to `/vault/secrets/storage`):**
`S3_BUCKET`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_ENDPOINT`, `S3_REGION`

---

### 13. Dependency Resolution (catalog 0.4.0+)

**Two formats in catalog-info.yaml:**

**Legacy `spec.internalDependencies`** (no scopes):
- No Bio-ID scope check
- Always resolves via Koko to K8s or Janus URL
- Injects `{SERVICE}_URL` + `{SERVICE}_CHARGE_MODE`

**Unified `spec.dependencies`** (with scopes):
1. If scopes non-empty: check Bio-ID scope grant status
2. `denied`/`revoked` → block (hard fail after OAuth provisioning pass), no URL injected
3. `pending` → block (awaiting approval), no URL injected
4. `none` → record as a potential blocker **but still resolve + inject the URL**, then let the
   grant-creation pass decide: target has no Bio-ID module → fail-open (proceed); same-org → auto-approved
   (proceed); cross-org with a module → creates a `pending` request that re-blocks. Injecting the URL here
   is what makes fail-open / auto-approved deps actually functional.
5. `unreachable` → warn + proceed (URL injected)
6. `approved` → proceed (URL injected)
7. `transport: gateway` → no URL injection (consumer calls Janus public gateway)
8. `transport: direct` → inject K8s internal URL (same-org only; cross-org silently routes via Janus)
9. `transport: janus` → inject Janus proxy URL (`{JANUS_URL}/i/{service}`)

> URL injection also requires the dependency to be **registered in Koko** — a dep that isn't in the
> registry resolves to no URL (warn), independent of grant status.

**URL formats:**
| Transport | Injected URL |
|-----------|-------------|
| `direct` | `http://{svc}.{svc}-{env}.svc.cluster.local:3000` |
| `janus` | `http://janus.janus-{env}.svc.cluster.local:3000/i/{svc}` |

**Charge mode injected alongside URL:** `{SERVICE}_CHARGE_MODE = callerOrg | service`

---

### 13b. Provider Webhook Secrets

**When:** a `spec.dependencies` entry declares `webhooks.path`.

**What:** reads the target's webhook contract from Koko (`metadata.webhookProvider`), computes the
consumer's receiver URL (verified custom domain for the environment, else the platform hostname,
plus the path), mints the builder's Bio-ID `client_credentials` token (`BIO_CLIENT_ID` /
`BIO_CLIENT_SECRET` in `/opt/iec-builder/.env`), and POSTs
`{ orgSlug, url, consumerService, environment }` to the provider's `provisionEndpoint` on its
public platform hostname. The returned secret is merged into the provisioned env vars under the
contract's `secretEnv` and reaches the pod through `{service}-managed-secrets`.

A value the developer set with `tawa config set` (plain or `--secret`) wins and is not overwritten;
a mismatch is a `[warn]`. The secret is never written to the build log, Koko, or the deploy snapshot.

**Fail behavior:** Fail-open — every failure is a `[warn]` linking
https://tawa.insureco.io/reference/provider-webhook-secrets.

**Provider side:** a provider's own deploy syncs `spec.webhookProvider` to Koko as
`metadata.webhookProvider` (null when absent) during service registration (step 22).

---

### 14. OAuth Client Provisioning

**When:** `catalog.authMode == 'sso'` OR `catalog.authMode == 'service-only'` OR service has
any scoped dependencies.

**What:** Calls Bio-ID `POST /api/admin/oauth-clients` (idempotent — upserts).

**Redirect URI registered:**
- Sandbox: `https://{service}.sandbox.tawa.pro/api/auth/callback`
- Prod: `https://{service}.tawa.pro/api/auth/callback`
- Custom domain (if `dnsVerified`): `https://{customDomain}/api/auth/callback`

**What is merged vs overwritten on an existing client:** the builder reads the client first, then
sends the **union** of what it derives and what Bio-ID already holds for `redirectUris`,
`allowedScopes` and `allowedGrantTypes` — so a deploy only ever **adds** to those three, and a scope
an admin granted with `tawa oauth update --scopes` survives. Preserved entries are named in the
build log (`OAuth scopes preserved from Bio-ID (not derived from the catalog): …`). If the client
cannot be read, the scope and grant lists are omitted from the PATCH entirely rather than narrowed —
fail-open, with a warning. Catalog-declared fields (`spec.auth.passport` → `passportConfig`,
`headlessEnabled`, `allowedEmbedOrigins`) are written from the catalog every deploy. See
[auth](auth.md) and iec-builder#159.

**Env vars injected:**
- `BIO_CLIENT_ID` — OAuth client ID (`{service}-{environment}`)
- `BIO_CLIENT_SECRET` — OAuth client secret (rotated on each deploy). A credential, so it rides
  the managed secret, not the Helm command — see Step 18
- `BIO_ID_URL` — Bio-ID public URL
- `BIO_ID_CALLBACK_URL` — Full callback URL for this service

---

### 14b. Claude Access

**When:** an OAuth client was provisioned in step 14, Anthropic federation is configured on the
builder, and that client has **Claude access** (`claudeAccessEnabled`) switched on in Bio-ID.

**What:** the builder reads the client back from Bio-ID. If the flag is on it finds, or creates,
the client's own Anthropic service account and federation rule — the same idempotent call the
console switch makes, so a deploy also repairs either one if it has gone missing — and injects:

- `ANTHROPIC_FEDERATION_RULE_ID`, `ANTHROPIC_ORGANIZATION_ID`, `ANTHROPIC_SERVICE_ACCOUNT_ID`,
  `ANTHROPIC_WORKSPACE_ID`

These are identifiers, not credentials, so they travel as plain `--set env.*` values, not through
the managed secret. All four are reserved against `tawa config set`.

**Build log:** `Claude access provisioned for {clientId}: rule fdrl_…, service account svac_…. Injected …`

**Fail behavior:** Fail-open. Bio-ID unreadable, Anthropic unreachable, or an existing rule that
is not what the client should have: one `[warn]` line, the variables are not injected, the deploy
continues. A client with the flag off, or a builder with no federation configured, logs nothing.

Full reference: [Claude Access](claude-access.md).

---

### 15. Scope Grant Creation

**When:** Service has scoped dependencies (`hasScopedDeps = true`).

**For each dep with scopes that doesn't have an approved grant:**
1. `POST /api/admin/scope-grants` in Bio-ID
2. Same-org → auto-approved immediately. "Same org" compares the **target's** Bio-ID module
   owner with the **requesting** service's org: its own module owner if it declares
   `spec.modules`, otherwise the org of its OAuth client (`{service}-{env}`, set from
   `spec.owner` on deploy). A consumer does not need modules of its own.
3. Cross-org → status `pending` — target org must approve in Console → Permissions → API Access
4. Bio-ID unreachable → fail-open (auto-add to approved set, proceed)
5. Target module not registered → fail-open

**For each dep whose approved grant lacks a declared scope (`approved_missing`):** the same
`POST /api/admin/scope-grants` with the **full** declared scope list, which widens the existing grant
for same-org — see [Adding a scope to an existing dependency](#adding-a-scope-to-an-existing-dependency).

**Never for a revoked grant** — see [Revoked grants](#revoked-grants).

**Block enforcement:** After all scope grant creation attempts, any deps still in the `blocked`
list (`pending`/`denied`/`revoked`/`none`/`approved_missing` that weren't auto-approved, widened
or fail-open) cause a hard fail.

> **Important:** A `pending` grant does NOT inject the `{SERVICE}_URL` — the resolver `continue`s
> past URL injection when a dep is blocked. So a stuck-pending grant looks like *"the URL never got
> injected"* at runtime, but the root cause is the unapproved grant. Approve it, then redeploy.

**Surfacing (so you know exactly what to approve):** Each blocking dep is emitted as its own build
log line and persisted to a structured `build.blockedGrants` array on the build record:

```
[blocked] relay [relay:send] — grant pending (grantId: a7d21c07-…). Approve at https://tawa.insureco.io/console/permissions/api-access
```

```jsonc
// build.blockedGrants[]
{ "service": "relay", "scopes": ["relay:send"], "status": "pending",
  "grantId": "a7d21c07-…", "approveUrl": ".../console/permissions/api-access",
  "reason": "Scope grant pending for relay [relay:send] (grantId: …) — approve at …" }
```

When the grant was opened by this very deploy and Bio-ID did not auto-approve it, the reason says
why, in Bio-ID's words:

```
Scope grant pending for x [x:read] (grantId: …) — not auto-approved as same-org:
cross-org: 'my-svc' is owned by 'acme', the target by 'insureco'. Fix: …
```

Other reasons: `no org is recorded for '<svc>'` (neither a module owner nor an org on its OAuth
client) and `the target module has no owner org recorded in Bio-ID`. Either one on two services
you know share an org is a platform bug, not a grant to approve by habit.

> **Correction (2026-09-30):** until iec-bio#160, same-org was decided from the requesting
> service's **module** only. A consumer without `spec.modules` — portal, and most relay callers —
> therefore never matched, and every same-org scoped dependency blocked its first deploy on a
> pending grant someone had to approve by hand (iec-builder#204). Grants already approved by hand
> stay approved; nothing needs redoing.

The `Deploy blocked: …` error reply also carries the grantId inline. Approve via the console, or
`PATCH /api/admin/scope-grants/{grantId}` with `{ "status": "approved" }` (super_admin or owner of
the target service).

#### Adding a scope to an existing dependency

Adding a scope to a dependency that already has an **approved** grant
(`[webmaster:read]` → `[webmaster:read, webmaster:dns]`) does not open a second grant. Bio-ID
keeps one grant per service pair + credential, and since iec-bio#161 `POST
/api/admin/scope-grants` against an approved grant behaves like this:

| Situation | Bio-ID response |
|---|---|
| Same-org, request adds scopes | **200**, `data.outcome: "expanded"` — the missing scopes are added to the existing grant (`requestedScopes` and `grantedScopes`) and to the OAuth client, in place. No revoke, no gap in access. |
| Request already covered by the grant | **200**, `data.outcome: "unchanged"` |
| Cross-org, request adds scopes | **409 CONFLICT**, naming the missing scopes. Widening a cross-org grant needs a new review. |
| Grant still pending | **409 CONFLICT** — approve or deny the pending one first |
| No grant | **201**, `data.outcome: "created"` (approved at once if same-org, else pending) |

**The builder does this for you (iec-builder#221).** When the resolver finds an approved grant
that does not cover every declared scope, it records status **`approved_missing`** — never
`pending`; nothing is pending and nobody has been asked — and the grant-creation pass POSTs the
full scope list with the same `requestingServiceId`, `targetServiceId`, `credentialType:
"oauth_client"` and `credentialId` (`{service}-{env}`):

| Bio-ID answer | Deploy | Build log |
|---|---|---|
| `outcome: "expanded"` | proceeds, `{SERVICE}_URL` injected | `Scope grant for iec-webmaster [webmaster:read, webmaster:sync, webmaster:dns]: approved, added scope webmaster:dns (grantId: …)` |
| `outcome: "unchanged"` | proceeds | `…: approved, already covers webmaster:dns (grantId: …)` |
| `outcome: "created"`, approved (the old grant was on another credential) | proceeds | `…: approved, new grant for this client (grantId: …)` |
| **409** (cross-org) | **blocked** | `[blocked] iec-webmaster [webmaster:read, webmaster:sync, webmaster:dns] — grant approved, missing webmaster:dns — needs approval by insureco (grantId: …). Fix: …` |
| Bio-ID unreachable / other error | **blocked** (an existing grant is not a fail-open case) | `… could not ask Bio-ID to add it: <error>. Fix: redeploy to retry.` |

On a block, `build.blockedGrants[]` carries the detail:

```jsonc
{ "service": "iec-webmaster", "scopes": ["webmaster:read", "webmaster:sync", "webmaster:dns"],
  "status": "approved_missing", "grantId": "…", "missingScopes": ["webmaster:dns"],
  "needsApprovalBy": "insureco",
  "reason": "Scope grant for iec-webmaster [...] is approved, missing webmaster:dns — needs approval by insureco (grantId: …). Bio-ID: Approved scope grant … does not include: webmaster:dns. … Fix: …",
  "fix": "Fix: an admin of 'insureco' revokes the approved grant (`tawa scopes revoke …`), you run `tawa scopes request iec-webmaster webmaster:read,webmaster:sync,webmaster:dns` …" }
```

`needsApprovalBy` is the target service's owning org from Koko; when Koko did not say (a `gateway`
dependency), it reads `the org that owns '<service>'`.

So, **same-org**: just redeploy — the grant is widened in place, with no revoke and no gap in
access. Do **not** revoke the grant to make room for a new one: that cuts access until the new
grant exists, and the builder will not re-request a revoked grant.

**Cross-org**: the target org has to review the wider set. Bio-ID will not widen a cross-org grant,
so the fix the block hands you is: an admin of the target org revokes the grant
(`tawa scopes revoke <grantId>`), you request the full list (`tawa scopes request <service>
<all,scopes>`), and they approve the new grant (`tawa scopes grant <grantId>`); then redeploy.
Access to the dependency is off in between.

> **Correction (2026-10-02):** before iec-builder#221 the builder reported an approved grant missing
> a scope as `grant pending (grantId: …)` and never asked Bio-ID to widen it, so every deploy
> blocked and the documented workaround was a manual `POST /api/admin/scope-grants`. That is no
> longer needed.

#### Revoked grants

When a dependency's only grant is `revoked`, the deploy blocks and the builder **does not
re-request it**. Bio-ID treats only a pending or approved grant as existing, so a re-request
creates a fresh grant — and for a same-org pair approves it at once. The builder would silently
undo a revocation someone made on purpose. The block says exactly that:

```
Access to relay [relay:send] has been revoked (grantId: …). The builder does not re-request a
revoked grant: a same-org request is approved at once, which would silently undo the revocation.
Fix: ask the owner of 'relay' why it was revoked. If access should be restored, run
`tawa scopes request relay relay:send` (same-org is approved at once; cross-org waits for an
admin of 'relay' to run `tawa scopes grant <grantId>`), then redeploy. Otherwise remove 'relay'
from spec.dependencies. See https://tawa.insureco.io/reference/build-pipeline
```

Once a new grant is approved, the gate finds it ahead of the revoked one and proceeds.

---

### 16. Module Registration (Bio-ID)

**When:** `catalog.modules` is non-empty.

**What:** Registers each module with Bio-ID. Returns a per-service API key injected as
`BIO_INTERNAL_KEY` into the pod (scoped to this service, not the builder's master key).

**Fail behavior:** Non-blocking — logs warning if registration fails.

---

### 17. userAccess Provisioning (catalog 0.5.0+)

**When:** `catalog.userAccess` is declared.

**What:**
- Injects `BIO_USERS_URL = https://bio.tawa.pro/api/v2/users`
- Provisions a `sak_` service API key in Bio-ID and injects it as `BIO_SERVICE_KEY`
- Registers the user access scope in Koko (for tawa-web Console display)
- `scope: cross-org` → lists the service as an allowed consumer in the Koko record

The key is keyed `{service}-{environment}` and provisioning is **idempotent** — Bio-ID returns
the same key on every subsequent deploy. Like every provisioned credential it reaches the pod
through the managed-secrets K8s secret (Step 18), never as a Helm `--set env.*` argument, and it
is **not** written into the service's own config: `tawa config list` does not show it.

Two conditions gate it, and a failure of either is **non-fatal** — the build continues and the
warning is the only signal:

| Condition | If it fails |
|-----------|-------------|
| `BIO_INTERNAL_KEY` is set on the builder | key provisioning is skipped silently; `BIO_USERS_URL` is still injected |
| Bio-ID returns a key | `[warn] Failed to provision BIO_SERVICE_KEY for {service}-{env} — user access may not work` |

So a service can deploy green, hold `BIO_USERS_URL`, and still have no key — check the build log
for the `User access service key provisioned` line rather than assuming.

> **Correction (2026-09-14):** this section previously said `BIO_SERVICE_KEY` is **NOT** injected
> and told you to provision one separately. That was wrong, and wrong in the costly direction: a
> reader concluded the platform had no service-level path to the user directory and went looking
> for a workaround. The builder has provisioned this key since `spec.userAccess` shipped
> ([`builder.ts`](https://git.insureco.io/insureco/iec-builder/src/branch/main/src/services/builder.ts),
> the `catalog.userAccess` block). [user-access.md](user-access.md) always documented it correctly
> — that page is the one to trust on this.

**A client-credentials token is not a substitute.** A service's own OAuth token
(`BIO_CLIENT_ID`/`BIO_CLIENT_SECRET`, `grant_type=client_credentials`) carries `client_id`,
`orgId`, `orgSlug` and `scope` — but no `sub`, no `roles`, and no `userType`. Bio-ID's directory
routes gate on caller identity, so verified against prod on 2026-09-14 it is refused everywhere:

| Route | With a client-credentials token |
|-------|--------------------------------|
| `GET /api/orgs/{slug}/members` | `403 Insufficient permissions` |
| `GET /api/orgs/{slug}` | `403 Insufficient permissions` |
| `GET /api/v2/users` | `401 Authentication required` |
| `GET /api/admin/users` | `401 Authentication required` |

`spec.userAccess` + `BIO_SERVICE_KEY` is the supported way for a service to read the user
directory with no user in the request. See [user-access.md](user-access.md).

---

### 18. Helm Deploy

**Command pattern:**
```bash
helm upgrade --install {service} {chartPath} \
  --namespace {namespace} \
  --create-namespace \
  --set image.tag={imageTag} \
  --set image.repository={registry}/{service} \
  --set service.port={port} \
  --set health.endpoint={healthEndpoint} \
  --set secretRef={service}-managed-secrets \
  --set env.KEY=VALUE  ... (service.config + the NON-credential provisioned vars)
  --set resources.requests.cpu={req} --set resources.requests.memory={req} \
  --set resources.limits.cpu={tier} --set resources.limits.memory={tier} \
  --set strategy.type=RollingUpdate \
  --set-string strategy.rollingUpdate.maxSurge=100% \
  --set strategy.rollingUpdate.maxUnavailable=0 \
  --wait --timeout ${HELM_TIMEOUT:-15m} --history-max 5
  # + Vault Agent annotations if Vault active
  # + NetworkPolicy labels for direct-transport deps
```

**Resource requests and limits.** Limits always come from `insureco.io/pod-tier`, and so does
hosting gas. Requests default to the tier's values (see the table in [Gas & Wallet](gas-wallet.md)),
and a service may move them with `insureco.io/memory-request` and `insureco.io/cpu-request`
(iec-builder#193). Formats, floors, the tier-limit ceiling and the warn-and-fall-back rule are in
[catalog-info → `memory-request` / `cpu-request`](catalog-info.md#memory-request--cpu-request--resource-requests-below-the-tier-default).
Every deploy writes one line naming the source of each request:

```
Resource requests: cpu 400m (xlarge tier default), memory 512Mi (insureco.io/memory-request); limits: cpu 4, memory 4Gi (xlarge tier)
```

These apply to the service container only. When Vault is active, the Vault Agent sidecar gets its
own requests through its injector annotations (iec-builder#192), logged as
`Vault Agent sidecar requests: cpu 10m, memory 48Mi (limits unchanged)`. A custom chart receives
the same `--set resources.*`; if an override is declared and its templates never read
`.Values.resources`, the build warns that the requests are passed but not applied.

**Every replica rolls at once, with zero downtime (iec-builder#171).** Before this change the
strategy was `maxSurge: 1, maxUnavailable: 0`, so a 2-replica service replaced its pods one after
the other. On bio, which pulls a 216 MB image at about 1 MB/s, that was two 3–4 minute pulls back to back:
about 7 minutes of rollout. With `maxSurge: 100%` a whole new ReplicaSet starts beside the old
one, every new pod pulls and starts in parallel, and old pods are removed only as new ones become
Ready (`maxUnavailable: 0`). A rollout now costs one pod's startup, not one per replica.

| | Before | Now |
|--|--------|-----|
| Pods started at a time | 1 | all replicas |
| Pods unavailable during rollout | 0 | 0 |
| Resources held during rollout | replicas + 1 | up to 2 × replicas |

The cost is transient. For the length of the rollout the service reserves up to twice its
replicas' resource requests. If no node has room, the surge pods sit Pending until the autoscaler
adds capacity: slower, never unsafe. The same default is in `helm/default-service/values.yaml`,
and the `--set` flags also apply to custom charts that read `.Values.strategy`. `--timeout` stays
at 15 minutes as headroom for that Pending case and for a cold node pulling a large image.

**One Helm run per release at a time.** The Helm step (status check, failed-release cleanup,
upgrade) runs under a per-`{namespace}/{service}` lock in Redis. A second build for the same
release — a different commit, since same-commit races are caught at trigger time — **waits**,
logging `[helm] another deploy of X is in progress — build <id> — waiting…` once a minute,
then runs in order. It does not fail with `release: already exists`. Waiting is bounded at 30
minutes; past that the waiter fails naming the holder, because a lock held that long is a
stuck deploy, not a slow one. A crashed holder's lock expires on its own (iec-builder#110).

**Helm chart priority:**
1. `.iec.yaml` `helmChart` field
2. `service.helmChart` stored in builder DB (set via `tawa services update --helm-chart`)
3. Auto-discovered: `helm/Chart.yaml`, `helm/{service}/Chart.yaml`, `helm/*/Chart.yaml`, `chart/Chart.yaml`
4. Built-in `helm/default-service` chart (fallback)

**Secrets:** the K8s Secret `{service}-managed-secrets`, mounted via `envFrom.secretRef`, is
**rebuilt from scratch on every deploy** from two inputs:

1. the developer's secrets — `tawa config set --secret`, stored encrypted in `service.secrets`;
2. **this deploy's provisioned credentials** — the OAuth client secret, `BIO_INTERNAL_KEY`,
   and any provisioned connection string carrying a password (`scheme://user:pass@host`).
   Classified by key shape (`SECRET`, `PASSWORD`, `TOKEN`, `CREDENTIAL`, `_KEY`) or value
   shape; `BIO_CLIENT_ID` and credential-free URLs stay on the Helm command as `--set env.*`.

Provisioned credentials **never enter `service.secrets`**. The builder does not create entries
in the developer's store; `tawa config list` shows only what the developer set. The Secret is
applied from a `0600` temp-file manifest (nothing sensitive in `kubectl`'s argv) and annotated
`tawa.pro/provisioned-keys` naming the keys the builder put there. The pod checksum covers the
merged data, so a rotated provisioned credential still rolls the pods.

What the build log says, every deploy:

| Line | Meaning |
|------|---------|
| `Provisioned credentials injected through the managed secret, not the deploy command: …` | The builder-managed keys for this deploy. Not shown by `tawa config`; re-provisioned every deploy. |
| `[notice] Newly provisioned since the last deploy: …` | A declaration was added (a database, `spec.auth`). |
| `[notice] No longer provisioned, removed from the managed secret: …` | A declaration was removed. The key is really gone from the pod from this deploy on — nothing to clean up by hand. |
| `[notice] Removed builder-written entries from your managed secrets: …` | One-time cleanup of entries an earlier builder version wrote into your store (iec-builder#114, and `BIO_SERVICE_KEY` before #283). An entry counts as builder-written when its key is reserved, or its value is exactly the one provisioned this deploy. The live values are injected directly. |
| `[warn] Your managed secret X is shadowed by the builder-provisioned value…` | You set a secret with a provisioned key's name; the builder's value wins. Remove yours with `tawa config unset X`. |

> **Correction (2026-09-10):** this page previously said every provisioned var, credentials
> included, went to Helm as `--set env.KEY=VALUE`. That was true until iec-builder#118 and put
> the values in the builder host's process table for the length of each deploy. Do not
> reintroduce that path. An earlier fix (#114) was reverted for writing provisioned values into
> the developer's `service.secrets`; the current design is what replaced it. `BIO_SERVICE_KEY`
> kept persisting itself into `service.secrets` until iec-builder#283, which made every
> `spec.userAccess` deploy warn that the developer's copy was shadowed and made
> `tawa config unset` useless. It no longer does, and the next deploy removes the old copy.

**NetworkPolicy:** Pods default to receiving traffic only from `tawa.pro/role: janus` namespace
(Janus sidecar). Direct-transport dependencies label the namespace so NetworkPolicy allows
direct pod-to-pod traffic.

**Egress NetworkPolicy (`spec.egress.denyPrivate`):** egress is unrestricted unless the catalog
declares `spec.egress.denyPrivate: true`. Then, just before this step, the builder computes the
allowed destinations (`src/services/egress-policy.ts`) and passes them as
`--set-json 'networkPolicy.egress={"enabled":true,"rules":[...]}'`; the default chart adds
`Egress` to `policyTypes` and renders the rules. The pod may reach the internet minus
`10/8`, `172.16/12`, `192.168/16`, `169.254/16` and `100.64/10`, plus cluster DNS, its own
namespace, the platform services, its declared dependencies, `ingress-nginx`, Vault (when it
runs an agent), the builder's backing-store hosts for what it declares (`DB_MONGODB_HOST`,
`DB_REDIS_HOST`, `DB_NEO4J_HOST`, `MINIO_INTERNAL_HOST` — which must be **IP addresses** for a
rule to be written), the egress-group proxy, and `spec.egress.allow`. The build log lists each
destination; a custom chart gets a `[warn]` because only the default chart renders the value.
Undeclared, nothing is passed and the chart renders byte-identically — enforced by golden
fixtures in iec-builder. Full reference:
[catalog-info → `spec.egress.denyPrivate`](catalog-info.md#specegressdenyprivate--keep-a-pod-off-the-private-network).

**Deploy snapshot:** immediately after `helm upgrade` succeeds, the builder POSTs one deploy
manifest to Koko (`POST /api/services/:id/manifests`). This is the **only** manifest write in the
pipeline, so a stored manifest always describes a build that shipped. It carries:

| Field | Value |
|-------|-------|
| `buildId`, `environment`, `catalogVersion` | from the build |
| `commit` | the resolved commit SHA the image was built from |
| `sdkVersions` | flat, normalized dependency map (deps + devDeps) — what Step 7's SDK upgrade gate diffs against next time |
| `deployedBy` | the triggering user, or `deploy` for platform re-rolls |
| `envVarManifest`, `decisions`, `resolved` | env-var classification (secrets redacted), build decisions, resolved runtime config |

The call is fire-and-forget — it never blocks or fails a deploy — but it is not silent: if Koko
rejects the manifest, the build log gets
`[warn] Deploy snapshot was not stored in Koko: <Koko's error>`. Treat that line as a bug report
against the builder↔Koko contract; deploy history and the SDK upgrade gate both depend on this write.

---

### 19. DNS Configuration

**When:** `ENABLE_DNS_MANAGEMENT=true`.

**What:** Creates or updates a Cloudflare CNAME record:
- `{service}.{env}.tawa.pro` (sandbox/uat) or `{service}.tawa.pro` (prod) → `INGRESS_TARGET`

**Fail behavior:** DNS errors are non-fatal — logged as `[warn]`, deploy continues.

After DNS: registers domain binding in Koko.

---

### 20. Post-Deploy Pod Health Check

**What:** `kubectl get pods -n {namespace}` + recent events. Logged to build output. Stored
as `build.diagnostics` if issues detected.

**Health reports the running commit.** Alongside the pod check, the builder makes one HTTPS GET, with a
10s timeout, to the service's public platform hostname (`https://{service}.tawa.pro{healthEndpoint}`,
or `.sandbox.` / `.uat.`) after the rollout, and parses the JSON. If the body has no
`commit` equal to the deployed SHA (the `GIT_COMMIT` it injected), the build log gets one line:

```
[warn] /health does not report the running commit. Return { status, commit: process.env.GIT_COMMIT } — see https://tawa.insureco.io/reference/catalog-info
```

The result is stored as `build.healthReportsCommit` (`true`, `false`, or `null` when it could not
be checked). `worker` and `internal: true` services are skipped, recording `null` with no warning. An
unreachable endpoint, a timeout or a non-2xx answer also records `null` with no warning. A
2xx body that is not JSON warns once. Handlers:
[catalog-info → health-endpoint](catalog-info.md#health-endpoint--one-value-three-consumers).

**Fail behavior:** Non-fatal — diagnostics only. Does not block subsequent steps, and the commit
check never fails a deploy.

---

### 20b. Vault Secret Readback

**When:** `VAULT_ENABLED=true`. Runs after the pod health check, before deploy-gated tests.

**What:** Confirms the Vault Agent actually **rendered** the declared secrets into the pod —
not just that the vault-init container came up. For each declared database + object storage,
the builder `kubectl exec`s into a Running pod and `cat`s the rendered file, verifying the
required keys are present with non-empty values:

| Declared | File | Required keys |
|----------|------|--------------|
| `mongodb` | `/vault/secrets/mongodb` | `MONGODB_URI` |
| `redis` | `/vault/secrets/redis` | `REDIS_URL` |
| `neo4j` | `/vault/secrets/neo4j` | `NEO4J_URI`, `NEO4J_USERNAME`, `NEO4J_PASSWORD` |
| `spec.storage` | `/vault/secrets/storage` | `S3_ENDPOINT`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY` |

**Result** (`healthy` | `degraded` | `missing` | `skipped` | `error`) is stored as
`build.vaultVerification` and cached on `service.vaultVerification`, and surfaced via
`GET /services/:name/vault-status` (powers the console vault health badge).

**Fail behavior:**
- **Re-rolls only** (auto-rotation and manual `rotation/refresh` deploys): a `missing` or
  `degraded` readback **hard-fails the build**, so the credential rotation clock does NOT
  advance on empty secrets. This is the post-reroll safety gate.
- **Normal deploys:** non-fatal — result is recorded and badged but never blocks the build.
- `error` (no ready pod / kubectl unavailable) and `skipped` (nothing declared) never block.

---

### 21. Deploy-Gated Tests (catalog 0.5.0+)

**When:** `IEC_TEST_URL` is set AND (`catalog.tests` is non-empty OR `catalog.walkthrough` is declared).

**Target URL:** Internal cluster URL (`http://{service}.{namespace}.svc.cluster.local:{port}`)
— avoids external DNS propagation latency.

**Flow:**
1. `POST {IEC_TEST_URL}/runs` with test specs
2. Poll `GET /runs/{id}` every `TEST_RUNNER_POLL_INTERVAL_MS` (default 5s)
3. Timeout after `TEST_RUNNER_MAX_POLL_MS` (default 10 minutes)

**Suites** (each requested only when it has specs):
| Suite | Environments | Gating |
|-------|-------------|--------|
| `smoke` | All | blocking |
| `e2e` | **All, including prod** | blocking in sandbox/uat; **non-blocking in prod** (`[warn]` in the build log) |
| `compliance` | All | blocking |
| `walkthrough` (catalog 0.7.0+) | **All, including prod** | blocking in sandbox/uat; **non-blocking in prod** (`[warn]` in the build log) |

The two browser suites share one posture, and the prod exemption is narrow for both: it applies
only when they are the **only** failures. A run where smoke also failed fails the build as usual.

> **e2e used to skip prod, which meant it never ran at all** — nothing on this platform deploys
> to sandbox or UAT. Fixed in iec-builder#161, together with the shape: the builder parsed an
> entry as `{ name, spec: <string> }` while iec-test required `{ name, steps: [...] }`, so a
> catalog that declared e2e got `Failed to create test run: 400`. Steps are now validated at
> catalog-parse time, before the image is built. See
> [Tawa Test](tawa-test.md#e2e) for the contract.

A walkthrough run also sends the resolved `commitSha`, and on completion the build log gets
`Walkthrough: <url>` pointing at the published bundle page. An e2e run logs its flow count and
which posture applies before the build starts.

**Fail behavior:** Hard fail if tests fail. Deployment NOT rolled back.

---

### 22. Koko Service Registration

**What:** `registerOrUpdateServiceInKoko` — registers/updates the service in the service
registry with:
- Routes (path, methods, auth, gas cost)
- Owner (org slug)
- Upstream — `http://{service}.{namespace}.svc.cluster.local:{port}`, where `port` is
  `insureco.io/port` from catalog-info.yaml (its framework default when omitted — `static` is
  8080), falling back to the cached service record, then 3000. This is the **same** port Helm
  gives the Service (Step 18), and it is what Janus `/i/` dials — Janus proxies internal calls to
  the Koko upstream, not to a URL of its own. The builder writes the resolved catalog port back
  to the service record after the deploy so the fallback stays current.
- Health endpoint — `insureco.io/health-endpoint` from catalog-info.yaml (falling back to the
  cached service record, then `/health`). This is the path iec-pulse probes, so a stale value
  here shows as 0% uptime on the status board.
- Pod tier + storage tiers
- Direct-transport dependency labels (for NetworkPolicy)

---

### 23. Queue + Schedule Bindings

**What:** `registerQueueBindings` / `registerScheduleBindings` — tells Koko the upstream K8s
URL for iec-queue and iec-cron to call when delivering jobs or firing schedules. It is the same
URL, and the same declared port, as the Step 22 registration.

> **Correction (2026-09-27):** before iec-builder#187 / #188, Steps 22 and 23 registered the
> upstream on `service.port || 3000` and ignored `insureco.io/port`, while Helm (Step 18) used the
> declared port. A service on any other port — refund-travel-rater on 3215, binddesk-api on 4000,
> static sites on 8080 — was registered at `:3000`, so Janus `/i/` and iec-cron callbacks to it
> hung. Such services must be **redeployed once** after the builder fix; Koko is not backfilled.

**Then it reconciles.** After registering, the builder lists Koko's bindings for
`{service}-{environment}` and **deletes any whose name the catalog no longer declares**. Remove a
schedule from `catalog-info.yaml` and the next deploy removes it from Koko too — it stops firing.

The prune runs even when the catalog declares **no** queues and **no** schedules, because a catalog
that dropped its last schedule is exactly the case that used to orphan one. It is scoped strictly to
this service's own namespace: Koko's list routes have no server-side namespace filter, so the list is
filtered on each record's `namespace` field, never by splitting the composite `{namespace}-{name}`
id. Another service's bindings are never touched.

`DELETE /services/:serviceId` does the same with an empty keep-list, across `prod`, `sandbox`, `uat`
and `dev`, before deleting the builder's own record.

> **Why this exists (iec-builder#165).** The builder only ever *added* bindings. `isc-split-pay` was
> renamed to `iec-split-pay`; seventeen days later iec-cron was still firing seven schedules at
> `isc-split-pay-prod`, each one debiting **5 gas before** the callback and getting a 404 back —
> about 150 gas a day for nothing. **Rename still has no builder event**: it is a new service plus an
> old one nobody deleted. Rename = deploy the new name, then `tawa services delete` the old one.

**Fail behavior:** Fail-open. Partial registration is logged. If Koko cannot be listed, **nothing is
deleted** and the build log gets a `[warn]` naming what may still be registered. Neither step ever
blocks a deploy.

---

### 24. Uptime Monitor (iec-pulse)

**When:** `IEC_PULSE_URL` is set.

**What:** Registers a monitor for `https://{service}.{env}.tawa.pro{healthEndpoint}`.
- prod: 30s poll interval
- sandbox/uat: 60s poll interval

**Fail behavior:** Fire-and-forget — never blocks.

---

### spec.seo: what the builder keeps and who can read it (iec-builder#194, #206)

Every deploy that parses a catalog writes `spec.seo` to the service record as `seoSpec`
(`enabled`, `entity`, `topic`, `pages`), or removes `seoSpec` if the catalog no longer
declares it. `seoSpec` is separate from `catalogSpec` because the builder only writes
`catalogSpec` for services that declare databases, and most frontends declare none.

`GET /seo/services?env=prod` (env defaults to `prod`) lists every service with
`seoSpec.enabled`:

```jsonc
{ "success": true, "data": [
  { "service": "raterspot-web", "env": "prod", "seo": { "enabled": true, "entity": "raterspot" },
    "domains": ["raterspot.com"],                        // custom domains dnsVerified for this env
    "lastDeploy": { "commit": "…", "at": "…" } } ] }     // the latest completed build for this env
```

iec-webmaster's hourly `service-sync` reads this route to catch a deploy whose
`POST /hooks/deploy` it missed.

This is the first builder route that accepts a platform service's **own Bio-ID login**:

| Credential | Result |
|---|---|
| `client_credentials` token whose `org_slug` (from Bio-ID introspection) is `insureco` | accepted |
| `client_credentials` token from any other org | `403` |
| No token, or an inactive one | `401` |
| A user token, the internal key, or an API key | handled by the normal `authenticate` path, then only super admins and members of the platform org may read |

The route is read-only and returns no credentials. The middleware is
`authenticateServiceOrUser()` (`src/middleware/service-token.ts`). Use it only on routes
whose answer is safe for any insureco service.

## Fail-Open vs Fail-Hard Quick Reference

| System | Scenario | Behavior |
|--------|----------|----------|
| Wallet service | Unreachable | Fail-open (deploy proceeds) |
| Wallet service | Insufficient gas | Hard fail |
| Bio-ID | Unreachable (scope grant check) | Fail-open (warn) |
| Bio-ID | Grant denied/revoked | Hard fail |
| Bio-ID | Grant pending | Hard fail (no URL injected until approved) |
| Bio-ID | Grant none (no module / same-org) | Fail-open / auto-approve — proceed **with** URL injected |
| Bio-ID | Grant none (cross-org, module exists) | Creates pending request → hard fail |
| Bio-ID | Grant approved, missing a scope (same-org) | Widened in place — proceed with URL injected |
| Bio-ID | Grant approved, missing a scope (cross-org, 409) or Bio-ID error | Hard fail (`approved_missing`) |
| Bio-ID | Only grant revoked | Hard fail; never re-requested |
| Koko | Unreachable (dep resolution) | Fail-open (skip dep) |
| Webhook provider | No contract / unreachable / error / no builder token | Fail-open (warn, no secret injected) |
| Koko | Changelog missing for SDK bump | Hard fail |
| Koko | Unreachable (changelog check) | Fail-open (warn) |
| Vault | Unhealthy | Fall back to static credentials |
| Claude access (Bio-ID read / Anthropic) | Error, or an existing rule that does not match | Fail-open (warn, ids not injected) |
| DNS (Cloudflare) | Error | Fail-open (warn) |
| Post-deploy diagnostics | Error | Fail-open (warn) |
| Deploy-gated tests | Fail | Hard fail (pod stays) |
| Docs publish | Error | Fail-open |
| Deploy snapshot (Koko) | Error / rejected | Fail-open — rejection surfaced as a `[warn]` build-log line |
| Uptime monitor | Error | Fail-open |
| Convention check | Blocking violation | Hard fail |
| Non-root Dockerfile | Root user detected | Hard fail |
| Config preflight | Missing required vars | Hard fail |
| Owner tamper guard | Owner mismatch | Hard fail |

---

## Environment Variables

All builder env vars with their defaults and purposes:

| Variable | Default | Purpose |
|----------|---------|---------|
| `PORT` | `3002` | Builder API port |
| `MONGODB_URI` | `mongodb://localhost:27017/builder` | Builder's own DB |
| `REDIS_URL` | `redis://localhost:6379` | Build queue |
| `WORKSPACE_DIR` | `/tmp/iec-builds` | Clone workspace |
| `DOCKER_REGISTRY` | `registry.insureco.io/insureco` | Image push target |
| `DOCKER_REGISTRY_USER` | — | Registry auth username |
| `DOCKER_REGISTRY_TOKEN` | — | Registry auth token |
| `MIRROR_REGISTRY` | — | In-cluster mirror host (`10.124.0.14`). Empty turns the mirror push off |
| `MIRROR_REGISTRY_USER` | — | Mirror push user (`pusher`) |
| `MIRROR_REGISTRY_TOKEN` | — | Mirror push password (stdin only) |
| `ENABLE_AUTO_DOCKERFILE` | `true` | Auto-generate Dockerfiles |
| `ENABLE_K8S_DEPLOY` | `false` | Enable Kubernetes deployment |
| `KUBECONFIG` | — | K8s cluster credentials |
| `PLATFORM_DOMAIN` | `tawa.pro` | External URL base domain |
| `CLOUDFLARE_API_TOKEN` | — | DNS management |
| `CLOUDFLARE_ZONE_ID` | — | Cloudflare zone |
| `INGRESS_TARGET` | — | K8s ingress IP/hostname for DNS CNAME |
| `ENABLE_DNS_MANAGEMENT` | `false` | Enable Cloudflare DNS updates |
| `FORGEJO_URL` | `https://git.insureco.io` | Git host |
| `FORGEJO_TOKEN` | — | Forgejo PAT for private repos |
| `GITHUB_TOKEN` | — | GitHub PAT for private repos |
| `KOKO_URL` | `https://koko.tawa.pro` | Service registry |
| `BIO_ID_URL` | `https://bio.tawa.pro` | OAuth provider |
| `BIO_INTERNAL_KEY` | — | Builder's Bio-ID internal API key |
| `WALLET_URL` | — | Gas wallet service URL |
| `INTERNAL_SERVICE_KEY` | — | Service-to-service auth key |
| `DB_MONGODB_ADMIN_URI` | — | Admin MongoDB URI for credential provisioning |
| `DB_MONGODB_HOST` | `localhost` | MongoDB host (VPC IP for pods) |
| `DB_MONGODB_PUBLIC_HOST` | — | MongoDB public IP (for whitelist connections) |
| `DB_REDIS_ADMIN_URL` | — | Admin Redis URL for ACL provisioning |
| `DB_REDIS_HOST` | `localhost` | Redis host |
| `DB_NEO4J_ADMIN_URI/USER/PASSWORD` | — | Neo4j admin credentials |
| `DB_NEO4J_HOST` | `localhost` | Neo4j host |
| `DB_POSTGRES_ADMIN_URI` | — | Postgres admin connection for role/database provisioning (unset = postgres skipped with a warning) |
| `DB_POSTGRES_HOST` / `DB_POSTGRES_PORT` | `localhost` / `5432` | Postgres host pods connect to (VPC IP) |
| `DB_POSTGRES_SSLMODE` | — | Optional `sslmode` appended to provisioned URLs (`require` for a managed server) |
| `MINIO_HOST` | `64.23.181.20` | MinIO host (public IP, builder→MinIO) |
| `MINIO_INTERNAL_HOST` | — | MinIO VPC IP (pods→MinIO, falls back to MINIO_HOST) |
| `MINIO_PORT` | `9000` | MinIO S3 API port |
| `VAULT_ADDR` | — | HashiCorp Vault address |
| `VAULT_TOKEN` | — | Builder's Vault token |
| `VAULT_ENABLED` | `false` | Enable Vault dynamic credentials |
| `ROTATION_TTL_DAYS` | `30` | Days between credential rotations |
| `IEC_TEST_URL` | — | iec-test service URL (empty = skip tests) |
| `IEC_PULSE_URL` | — | iec-pulse URL (empty = skip monitor registration) |
| `BUILD_CONCURRENCY` | `3` | Max parallel builds |
| `ENFORCE_MINIMUM_CATALOG_VERSION` | `false` | Reject catalogs below 0.2.0 |
| `DEPLOY_GATE_SKIP_SERVICES` | (platform services) | Comma-separated services that skip gas gate |
| `ALLOWED_DEPLOY_ORGS` | `insureco` | Fallback org allowlist |
| `CONFIG_ENCRYPTION_KEY` | — | AES-256-GCM key for managed secrets |
| `ANTHROPIC_API_KEY` | — | AI troubleshooter |
| `JCI_BRAIN_URL` | — | Knowledge base for troubleshooter advice |

---

## Auto-Injected Platform Env Vars (Always Available in Pods)

These are injected by the builder for every deploy — services do NOT need to declare them in
`internalDependencies`:

| Var | Value |
|-----|-------|
| `KOKO_URL` | Internal K8s URL for Koko |
| `JANUS_URL` | Internal K8s URL for Janus |
| `SEPTOR_URL` | Internal K8s URL for Septor |
| `IEC_WALLET_URL` | Internal K8s URL for iec-wallet |
| `BIO_ID_URL` | Bio-ID public URL |
| `NODE_ENV` | Always `production` for all deployed environments |
| `ENVIRONMENT` | Deployment environment: `prod`, `sandbox`, or `uat` |
| `GIT_COMMIT` | Full SHA of the commit this build was made from. Plain runtime env, not a build arg; skipped when the build SHA is a placeholder such as `push` or `latest`. Reserved (`tawa config set` refuses it). Return it as `commit` from your health endpoint |

---

## What Gets Written Where

| System | Written | When |
|--------|---------|------|
| **Builder MongoDB** | Build record (status, logs, imageTag, commitSha, diagnostics) | Throughout |
| **Builder MongoDB** | Service record (lastBuildId, credentialRotation, databaseCredentials, catalogSpec, seoSpec) | On success |
| **K8s** | Namespace | Step 18 |
| **K8s** | `{service}-db-{type}` secret (DB connection strings) | Step 10 |
| **K8s** | `{service}-oauth` secret | Step 14 |
| **K8s** | `{service}-managed-secrets` secret (developer secrets + this deploy's provisioned credentials, rebuilt every deploy) | Step 18 (Helm) |
| **K8s** | Deployment, Service, Ingress, ConfigMap, NetworkPolicy | Step 18 (Helm) |
| **Vault** | MongoDB dynamic credential role | Step 10 |
| **Vault** | Policy + K8s auth binding | Step 10 |
| **Vault** | MinIO credential role | Step 12 |
| **MinIO** | Bucket + quota | Step 12 |
| **Koko** | Namespace record (gas multiplier) | Step 9 |
| **Koko** | Database record (connection string) | Step 10 |
| **Koko** | Service record + routes | Step 22 |
| **Koko** | Domain binding | Step 19 |
| **Koko** | Queue + schedule bindings | Step 23 |
| **Koko** | Deploy snapshot manifest | Step 18 (fire-and-forget) |
| **Koko** | Available scopes (sharedWith) | Step 22 |
| **Koko** | Scope grants (auto-seed) | Step 22 |
| **Koko** | Onboarding spec | Step 22 |
| **Koko** | Role records (0.6.0+) | Step 23 |
| **Koko** | Registration metadata (0.6.0+) | Step 23 |
| **Bio-ID** | OAuth client (client ID + secret + redirect URI) | Step 14 |
| **Anthropic** | Service account + federation rule for the OAuth client (only when Claude access is on; normally already present) | Step 14b |
| **Bio-ID** | Scope grant request | Step 15 |
| **Bio-ID** | Module registration | Step 16 |
| **Cloudflare** | CNAME DNS record | Step 19 |
| **Septor** | `service.deployed` audit event | Step 18 |
| **Septor** | `namespace.created` audit event | Step 9 |
| **Septor** | `build.failed` audit event | On error |
| **iec-pulse** | Uptime monitor | Step 24 (fire-and-forget) |

---

## Common Failure Patterns

### "[warn] /health does not report the running commit"
The deploy succeeded; your health response has no `commit` equal to the deployed SHA
(`build.healthReportsCommit: false`). Return `commit: process.env.GIT_COMMIT ?? null` from the
health handler and redeploy. Check that a Next.js handler is `force-dynamic` (otherwise the
value is read at build, when it is unset) and that the declared `insureco.io/health-endpoint`
is the path you edited. Handlers: [catalog-info](catalog-info.md#health-endpoint--one-value-three-consumers).

### "Deploy gate: insufficient gas reserve"
Org wallet balance is below the 3-month reserve. Run `tawa wallet buy N` to top up, then redeploy.

### "Deploy gate: catalog spec.owner X does not match registered service owner Y"
Someone changed `spec.owner` in catalog-info.yaml. Either revert it, or use
`tawa services update <id> --org <new-org>` if the transfer is intentional.

### "Deploy blocked: Access to X has been denied/revoked"
The scope grant for a `spec.dependencies` entry was explicitly denied or revoked in Bio-ID.
Check Console → Permissions → API Access. The builder never re-requests a revoked grant — see
[Revoked grants](#revoked-grants) for why and what to run.

### "grant approved, missing X — needs approval by <org>"
A dependency gained a scope its approved grant does not cover, and the grant is cross-org, so
Bio-ID refused to widen it (409). Same-org grants are widened automatically and never show this.
Follow the `Fix:` in the block — revoke, request the full list, approve — see
[Adding a scope to an existing dependency](#adding-a-scope-to-an-existing-dependency).

### "Deploy blocked: Scope grant pending for X [scopes] (grantId: …)"
A cross-org `spec.dependencies` entry created a grant request that nobody approved yet. The build
log line `[blocked] X [scopes] — grant pending (grantId: …)` and `build.blockedGrants[]` name the
exact grant. Approve it in Console → Permissions → API Access, or
`PATCH /api/admin/scope-grants/{grantId}` `{ "status": "approved" }` (super_admin / target-service
owner), then redeploy. Note: while pending, `{X}_URL` is **not** injected — fix the grant, don't
hardcode the URL.

### "Convention check failed: N blocking violation(s)"
Source code violates a Koko gate rule (e.g., deprecated SDK usage). Fix the violations or
use `tawa deploy --skip-convention-check --reason "..."`.

### "Config preflight failed: missing required config vars: KEY1, KEY2"
`catalog.configDeclarations` lists required vars that aren't set. Run `tawa config set KEY1=... KEY2=...`.
On a **first** deploy this needs the service record to exist first — see the next entry.

### "Service 'X' not found" from `tawa config set` / `tawa config list`
Not a typo and not a permissions problem: the service has never been registered. The record is
created by `tawa link`, or automatically by the first `tawa deploy` — config routes have nothing
to attach values to until then. Fix: `tawa link` (or deploy once), `tawa config set …`, then
deploy again to inject the values. Full ordering:
[First Deploy: the deploy → config → deploy ordering](#first-deploy-the-deploy--config--deploy-ordering).

If the service *is* already deployed, check the name matches `metadata.name` in
`catalog-info.yaml` and that you are in the owning (or an authorized) org.

### "Undocumented upgrade: X Y → Z has no changelog entry"
An SDK was bumped without a Koko changelog entry. Run `tawa changelog --sdk X` to document it.

### "Deploy-gated tests failed"
Post-deploy smoke tests failed. The pod is still running — use `kubectl logs` to debug.
Re-deploy after fixing will re-run tests.

### Pod stuck in CrashLoopBackOff after deploy
Check `build.diagnostics` in the builder DB or run `kubectl describe pod` in the namespace.
Common causes: missing env vars, incorrect health endpoint, DB connection failure on startup.

## Related: Builder Logs

The builder's own PM2 logs are redacted (write-time + read-time), rotated, and exposed through a
platform-admin API. See [Builder Logs — Redaction, Rotation & Admin API](./builder-logs.md) for
the `GET /platform/logs` and `GET /platform/logs/tail` contracts.