Object Storage on Tawa

About

Declare a bucket in `catalog-info.yaml` (catalog `0.3.0`+). The builder provisions a MinIO bucket with a hard quota, and credentials are injected into your pod by Vault — dynamic, short-lived, scoped to your bucket. Use the `@insureco/storage` SDK at **`^0.2.0`**.

Skill Content

This is the raw markdown that gets installed as a Claude Code rule.

# Object Storage on Tawa

## The Short Version

Declare a bucket in `catalog-info.yaml` (catalog `0.3.0`+). The builder provisions a MinIO
bucket with a hard quota, and credentials are injected into your pod by Vault — dynamic,
short-lived, scoped to your bucket. Use the `@insureco/storage` SDK at **`^0.2.0`**.

```yaml
metadata:
  annotations:
    insureco.io/catalog-version: "0.3.0"
spec:
  storage:
    - name: uploads      # → S3_UPLOADS_BUCKET (+ S3_BUCKET alias if single-bucket)
      tier: s3-md        # s3-sm | s3-md | s3-lg | s3-xl
```

```typescript
import { createStorage } from '@insureco/storage'   // ^0.2.0 — see credentials section
const storage = createStorage()                     // single bucket
await storage.upload('report.pdf', buffer)
```

> The package is **`@insureco/storage`**. A `@tawa/storage` exists in the registry but is
> abandoned at `0.1.0` and reads credentials from `process.env`, which is broken in a
> deployed pod (see below). Do not use it.

## Tiers

| Tier | Capacity | Gas / month |
|------|----------|-------------|
| `s3-sm` | 1 GB | 200 |
| `s3-md` | 5 GB | 800 |
| `s3-lg` | 25 GB | 3000 |
| `s3-xl` | 100 GB | 10000 |

## How credentials work (and why that matters)

Credentials are **dynamic MinIO users** minted by a custom Vault secrets engine. Pods never
hold a static key:

1. The Vault Agent sidecar reads `minio/creds/{role}` → the plugin creates an ephemeral MinIO
   user scoped to your bucket, with a lease (`default_ttl: 1h`, `max_ttl: 24h`).
2. The creds are written to **`/vault/secrets/storage`**, which the sidecar keeps current.
3. `@insureco/storage` **≥ 0.2.0** reads that file — not `process.env` — re-reading it before
   each operation, so a rolled lease is picked up with no restart.

### NEVER read `S3_*` out of `process.env`

> **Requires `"@insureco/storage": "^0.2.0"`.** A `^0.1.x` pin cannot resolve it and will read
> `process.env`, which is broken. Check yours before assuming storage works.

The pod's `S3_*` environment variables are a **snapshot taken at boot** and they go stale
within about an hour. There are two Vault Agent renders, not one:

- the **init container** renders the file, the entrypoint sources it into the environment and
  `exec()`s your process — so your env holds *that* credential, forever;
- the **sidecar** then renders the file again, getting a *different* credential, and renews
  only its own.

Nothing renews the init container's credential. When its Vault token expires (~1h) the
credential it created is revoked, and every `S3_*` value in your environment is dead — while
the file on disk is still valid. Measured across production: every Vault-using pod is in this
state, and pods older than about an hour fail `InvalidAccessKeyId` on the env credential.

So:

```typescript
import { createStorage, hasVaultFile } from '@insureco/storage'

const storage = createStorage()          // ✅ reads the Vault file when deployed
const storage2 = createStorage('uploads') // ✅ named bucket, same behaviour

// ❌ never do this — these values are a dead boot snapshot in a deployed pod
new S3Client({ credentials: { accessKeyId: process.env.S3_ACCESS_KEY_ID, ... } })

// ❌ and never gate on them either — they are present whether or not they work
if (process.env.S3_ACCESS_KEY_ID) { /* proves nothing */ }
if (hasVaultFile()) { /* ✅ deployed with Vault-managed creds */ }
```

`hasVaultFile()` is the correct "am I deployed?" discriminator: file when deployed, env when
local. Tracked in iec-builder #31; see also iec-docman #7 for what it looks like when it bites.

### Rotation handling (fixed — iec-builder #31)

**Historically broken, now fixed.** The builder used to ask the Vault Agent to
`rm -f /vault/signals/alive` on rotation and probe `cat` on that file. It never worked for any
service: the command runs in the `vault-agent` **sidecar**, which never had the `vault-signals`
volume mounted, so it removed nothing — and `rm -f` on a missing path exits 0, so it reported
success while doing nothing.

There is also no longer a Vault **init container** (`agent-pre-populate: false`). Two agent
processes meant two independent reads, hence two credentials — the app captured one nobody
renewed. One render now means the credential in your environment is the one being kept alive.

Rotation is detected inside your own container instead: the entrypoint fingerprints the
**database** credentials it sourced, and the liveness probe re-fingerprints the live files.
A mismatch means Vault issued credentials the running process cannot pick up (its environment
is fixed after `exec`), so Kubernetes restarts it. The probe **fails open** — missing
fingerprint, missing files, or no `md5sum` all pass, so it can never cause a restart loop.

Storage is deliberately excluded from that fingerprint: `@insureco/storage` re-reads the file
per operation, so a storage roll needs no restart and must not cause one.

## MinIO IAM reset recovery (self-heal)

### The failure it guards against

The roll above only fires on Vault's lease clock. An **out-of-band MinIO IAM reset** — a
reinstall, a `.minio.sys` wipe, or a volume loss — is invisible to Vault: MinIO loses every
Vault-created user, but Vault still holds the leases and believes they're valid, so **no roll
fires**. Every storage pod keeps serving cached creds that now throw `InvalidAccessKeyId`,
silently, until `max_ttl`.

You cannot fix this inside the roll: the roll is time-driven and the reset has no signal into
Vault. The builder runs a **storage reconciler** that supplies that signal from outside.

### How the reconciler works

Every `STORAGE_RECONCILE_INTERVAL_MS` (default 2 min) the builder:

1. **Vault gate** — skips the pass entirely if Vault is unhealthy (an untrustworthy probe must
   never trigger a heal).
2. **Admin probe** — checks MinIO is reachable with root creds. If MinIO is *down*, that's an
   outage, **not** a reset → alert only, never heal.
3. **Canary probe** — holds a dedicated dynamic-cred "canary" identity and exercises the exact
   path pods use. If the canary's creds are rejected as a non-existent user
   (`InvalidAccessKeyId` / `SignatureDoesNotMatch`) **while admin still works**, MinIO has
   forgotten an account Vault issued — an IAM reset.
4. **Debounce** — requires `STORAGE_CANARY_FAILURE_THRESHOLD` (default 3) consecutive failures
   before acting, so a transient blip never restarts the fleet.

On a confirmed reset (and outside `STORAGE_HEAL_COOLDOWN_MS`), it heals every storage service:
re-asserts the Vault role, recreates buckets if also wiped, revokes the stale Vault leases, and
`kubectl rollout restart`s the deployment. On restart the Vault Agent re-reads creds, the plugin
re-creates the MinIO user, and the pod recovers. A Septor `storage.iam_reset_healed` event and an
alert email are emitted. The number of services healed per run is capped by
`STORAGE_HEAL_MAX_SERVICES` (default 50).

> **Non-destructive:** the heal only touches the identity layer (Vault leases, MinIO users, pod
> lifecycle). Object data is never deleted. If a reset *also* wiped object data, buckets are
> recreated **empty** — the reconciler restores *access*, not lost objects. Object durability is
> MinIO's job (distributed/erasure-coded mode + backups).

### Manual / on-demand heal

When ops *knowingly* reset MinIO, freshly-minted canary creds would mask the drift, so detection
is bypassed — call the platform-admin endpoint to re-provision the whole fleet immediately:

```
POST /storage/reconcile   { "reason": "minio reinstall 2026-06-24" }   # platform-admin
GET  /storage/health                                                   # last reconciler result
```

### Configuration

| Env var | Default | Purpose |
|---------|---------|---------|
| `STORAGE_SELF_HEAL_ENABLED` | `true` | Master switch for the reconciler loop |
| `STORAGE_RECONCILE_INTERVAL_MS` | `120000` | Time between detection passes |
| `STORAGE_CANARY_FAILURE_THRESHOLD` | `3` | Consecutive canary failures before healing |
| `STORAGE_HEAL_COOLDOWN_MS` | `900000` | Minimum gap between auto-heals (anti-thrash) |
| `STORAGE_HEAL_MAX_SERVICES` | `50` | Per-run restart cap |
| `STORAGE_CANARY_BUCKET` | `iec-storage-canary` | Canary probe bucket |
| `PLATFORM_ALERT_EMAIL` | (unset) | Recipient for heal alert emails (no email if unset) |

### Roadmap

The durable long-term fix is to federate storage identity (MinIO OIDC backed by Bio-ID) so there
is no stored MinIO user to lose at all — the same model AWS IRSA / GCP Workload Identity use. The
reconciler remains the right safety net even after that lands.

## Common Mistakes

- Expecting a stale storage pod to recover on its own after a MinIO reset — it won't until the
  lease rolls; the reconciler is what forces it (or `POST /storage/reconcile`).
- Assuming the heal can recover lost objects — it restores access, not data.
- Declaring `spec.storage` below catalog `0.3.0` — the field is ignored.

Install

Copy the skill content and save it to:

~/.claude/rules/tawa-object-storage.md

Details

Format

Rule

Category

provision

Version

1.0.88400

Tokens

~2,236

Updated

2026-08-28