Object Storage on Tawa

About

Declare a bucket in `catalog-info.yaml` (catalog `0.3.0`+). The builder provisions a MinIO bucket with a hard quota, and credentials are injected into your pod by Vault — dynamic, short-lived, scoped to your bucket. Use the `@tawa/storage` SDK.

Skill Content

This is the raw markdown that gets installed as a Claude Code rule.

# Object Storage on Tawa

## The Short Version

Declare a bucket in `catalog-info.yaml` (catalog `0.3.0`+). The builder provisions a MinIO
bucket with a hard quota, and credentials are injected into your pod by Vault — dynamic,
short-lived, scoped to your bucket. Use the `@tawa/storage` SDK.

```yaml
metadata:
  annotations:
    insureco.io/catalog-version: "0.3.0"
spec:
  storage:
    - name: uploads      # → S3_UPLOADS_BUCKET (+ S3_BUCKET alias if single-bucket)
      tier: s3-md        # s3-sm | s3-md | s3-lg | s3-xl
```

```typescript
import { createStorage } from '@tawa/storage'
const storage = createStorage()                 // reads S3_* env vars (single bucket)
await storage.upload('report.pdf', buffer)
```

## Tiers

| Tier | Capacity | Gas / month |
|------|----------|-------------|
| `s3-sm` | 1 GB | 200 |
| `s3-md` | 5 GB | 800 |
| `s3-lg` | 25 GB | 3000 |
| `s3-xl` | 100 GB | 10000 |

## How credentials work (and why that matters)

Credentials are **dynamic MinIO users** minted by a custom Vault secrets engine. Pods never
hold a static key:

1. The Vault Agent sidecar reads `minio/creds/{role}` → the plugin creates an ephemeral MinIO
   user scoped to your bucket, with a lease (`default_ttl: 1h`, `max_ttl: 24h`).
2. The creds are written to `/vault/secrets/storage` and sourced into `process.env`.
3. When the lease **rolls**, the sidecar re-renders and restarts the container with fresh creds.

This is self-consistent: Vault always creates the MinIO user at the same moment it issues the
credential, so MinIO, Vault, and the pod stay in sync.

## MinIO IAM reset recovery (self-heal)

### The failure it guards against

The roll above only fires on Vault's lease clock. An **out-of-band MinIO IAM reset** — a
reinstall, a `.minio.sys` wipe, or a volume loss — is invisible to Vault: MinIO loses every
Vault-created user, but Vault still holds the leases and believes they're valid, so **no roll
fires**. Every storage pod keeps serving cached creds that now throw `InvalidAccessKeyId`,
silently, until `max_ttl`.

You cannot fix this inside the roll: the roll is time-driven and the reset has no signal into
Vault. The builder runs a **storage reconciler** that supplies that signal from outside.

### How the reconciler works

Every `STORAGE_RECONCILE_INTERVAL_MS` (default 2 min) the builder:

1. **Vault gate** — skips the pass entirely if Vault is unhealthy (an untrustworthy probe must
   never trigger a heal).
2. **Admin probe** — checks MinIO is reachable with root creds. If MinIO is *down*, that's an
   outage, **not** a reset → alert only, never heal.
3. **Canary probe** — holds a dedicated dynamic-cred "canary" identity and exercises the exact
   path pods use. If the canary's creds are rejected as a non-existent user
   (`InvalidAccessKeyId` / `SignatureDoesNotMatch`) **while admin still works**, MinIO has
   forgotten an account Vault issued — an IAM reset.
4. **Debounce** — requires `STORAGE_CANARY_FAILURE_THRESHOLD` (default 3) consecutive failures
   before acting, so a transient blip never restarts the fleet.

On a confirmed reset (and outside `STORAGE_HEAL_COOLDOWN_MS`), it heals every storage service:
re-asserts the Vault role, recreates buckets if also wiped, revokes the stale Vault leases, and
`kubectl rollout restart`s the deployment. On restart the Vault Agent re-reads creds, the plugin
re-creates the MinIO user, and the pod recovers. A Septor `storage.iam_reset_healed` event and an
alert email are emitted. The number of services healed per run is capped by
`STORAGE_HEAL_MAX_SERVICES` (default 50).

> **Non-destructive:** the heal only touches the identity layer (Vault leases, MinIO users, pod
> lifecycle). Object data is never deleted. If a reset *also* wiped object data, buckets are
> recreated **empty** — the reconciler restores *access*, not lost objects. Object durability is
> MinIO's job (distributed/erasure-coded mode + backups).

### Manual / on-demand heal

When ops *knowingly* reset MinIO, freshly-minted canary creds would mask the drift, so detection
is bypassed — call the platform-admin endpoint to re-provision the whole fleet immediately:

```
POST /storage/reconcile   { "reason": "minio reinstall 2026-06-24" }   # platform-admin
GET  /storage/health                                                   # last reconciler result
```

### Configuration

| Env var | Default | Purpose |
|---------|---------|---------|
| `STORAGE_SELF_HEAL_ENABLED` | `true` | Master switch for the reconciler loop |
| `STORAGE_RECONCILE_INTERVAL_MS` | `120000` | Time between detection passes |
| `STORAGE_CANARY_FAILURE_THRESHOLD` | `3` | Consecutive canary failures before healing |
| `STORAGE_HEAL_COOLDOWN_MS` | `900000` | Minimum gap between auto-heals (anti-thrash) |
| `STORAGE_HEAL_MAX_SERVICES` | `50` | Per-run restart cap |
| `STORAGE_CANARY_BUCKET` | `iec-storage-canary` | Canary probe bucket |
| `PLATFORM_ALERT_EMAIL` | (unset) | Recipient for heal alert emails (no email if unset) |

### Roadmap

The durable long-term fix is to federate storage identity (MinIO OIDC backed by Bio-ID) so there
is no stored MinIO user to lose at all — the same model AWS IRSA / GCP Workload Identity use. The
reconciler remains the right safety net even after that lands.

## Common Mistakes

- Expecting a stale storage pod to recover on its own after a MinIO reset — it won't until the
  lease rolls; the reconciler is what forces it (or `POST /storage/reconcile`).
- Assuming the heal can recover lost objects — it restores access, not data.
- Declaring `spec.storage` below catalog `0.3.0` — the field is ignored.

Install

Copy the skill content and save it to:

~/.claude/rules/tawa-object-storage.md

Details

Format

Rule

Category

provision

Version

1.0.84133

Tokens

~1,400

Updated

2026-06-24