Databases on Tawa

About

Declare databases in `catalog-info.yaml`. The builder provisions connection strings and injects them as environment variables. Your code reads from `process.env`. Identical across all environments.

Skill Content

This is the raw markdown that gets installed as a Claude Code rule.

# Databases on Tawa

## The Short Version

Declare databases in `catalog-info.yaml`. The builder provisions connection strings and injects them as environment variables. Your code reads from `process.env`. Identical across all environments.

```yaml
spec:
  databases:
    - type: mongodb   # → MONGODB_URI
    - type: redis     # → REDIS_URL
    - type: neo4j     # → NEO4J_URI
    - type: postgres  # → DATABASE_URL (and POSTGRES_URI)
```

> **Koko first.** Use `koko` for every read and write on a service database, in every environment. See [Koko Data Access](koko-data-access.md).

## Supported Types

| Type | Env Variable | Notes |
|------|-------------|-------|
| `mongodb` | `MONGODB_URI` | Recommended for most services |
| `redis` | `REDIS_URL` | Use for caching and queues |
| `neo4j` | `NEO4J_URI` (plus `NEO4J_USERNAME` and `NEO4J_PASSWORD`) | Use for graph data |
| `postgres` | `DATABASE_URL` and `POSTGRES_URI` (same value) | For software that needs Postgres (e.g. SpiceDB). See [PostgreSQL](#postgresql) — **needs a platform Postgres server first** |

**Only these four types are provisioned.** Any other type (such as `mariadb`, `mysql`, or `sqlite`) is logged as a warning and skipped — the build still completes successfully. That database simply isn't provisioned, so its env var is never injected.

## Connecting

```typescript
// MongoDB
import mongoose from 'mongoose'
const uri = process.env.MONGODB_URI
if (!uri) throw new Error('MONGODB_URI not configured')
await mongoose.connect(uri)

// Redis
import Redis from 'ioredis'
const url = process.env.REDIS_URL
if (!url) throw new Error('REDIS_URL not configured')
const redis = new Redis(url)

// Neo4j — NEO4J_URI, NEO4J_USERNAME, and NEO4J_PASSWORD are all injected
import neo4j from 'neo4j-driver'
const uri = process.env.NEO4J_URI
if (!uri) throw new Error('NEO4J_URI not configured')
const driver = neo4j.driver(
  uri,
  neo4j.auth.basic(process.env.NEO4J_USERNAME!, process.env.NEO4J_PASSWORD!),
)

// Postgres — DATABASE_URL carries the credentials
import pg from 'pg'
const url = process.env.DATABASE_URL
if (!url) throw new Error('DATABASE_URL not configured')
const pool = new pg.Pool({ connectionString: url })
```

## PostgreSQL

Added in iec-builder#231. First consumer: SpiceDB (`--datastore-engine=postgres --datastore-conn-uri=$DATABASE_URL`).

```yaml
spec:
  databases:
    - type: postgres
      name: authz        # optional — default is {service}-{env}
```

On every deploy the builder, using its admin connection:

1. Creates (or, on redeploy, re-applies the stored password to) a login role `svc_{service}_{env}` — lowercased, `-` becomes `_`, `NOSUPERUSER NOCREATEDB NOCREATEROLE`.
2. Creates the database `{service}-{env}` (or `name:`) **owned by that role**, and revokes `CONNECT` from `PUBLIC` so no other service's role can open it. An existing database is never recreated.
3. Stores the password encrypted on the service record (`databaseCredentials`, type `postgres`) and reuses it on every later deploy.
4. Writes the K8s secret `{service}-db-postgres` and injects `DATABASE_URL` and `POSTGRES_URI`, both `postgresql://svc_…:…@{host}:{port}/{db}`. They carry a password, so they reach the pod through the managed secret, never the Helm command line.

| Detail | Value |
|--------|-------|
| Database name rule | letters, digits, `_`, `-`; at most 63 bytes — anything else fails the deploy with a message |
| Credentials | static (Phase 1). Vault dynamic credentials are a follow-up |
| `spec.egress.denyPrivate` | the Postgres host:port is allowed when `postgres` is declared |
| `tawa config set DATABASE_URL=…` | not reserved; if you set one *and* declare postgres, the provisioned value wins and the build log warns |

**Requires a platform Postgres server.** The platform server is **PostgreSQL 16 on the tawa builder host**, reachable from pods at `10.124.0.3:5432` (VPC only — it listens on localhost and the VPC address, and 5432 is firewalled to `10.0.0.0/8`; it is not reachable from the internet). Data lives on the attached volume (`/mnt/volume_sfo3_01/postgresql/16/main`), not the root disk. Connections are plaintext inside the VPC, like MongoDB and Redis. The builder provisions as the non-superuser admin role `tawa_admin` (vault alias `tawa-postgres-admin`). The builder needs `DB_POSTGRES_ADMIN_URI` (plus `DB_POSTGRES_HOST`, `DB_POSTGRES_PORT`, optional `DB_POSTGRES_SSLMODE`). When `DB_POSTGRES_ADMIN_URI` is unset the deploy still succeeds, nothing is injected, and the build log says:

```
[warn] spec.databases declares postgres, but DB_POSTGRES_ADMIN_URI is not configured on the builder — no Postgres database was provisioned and DATABASE_URL was not injected. See https://tawa.insureco.io/reference/databases
```

**Backups:** nightly and on-demand `pg_dump -Fc` backups are supported (iec-builder#233) — see [database-backups](database-backups.md).

**Not yet supported for postgres** (each tracked as a follow-up): `koko backup` / `koko backups` CLI support, Koko registration and `koko query` (Koko does not accept the type yet), `consumesDatabase` (refused with an error), credential rotation via `/databases/:type/rotate`, and `tawa preflight` (its schema still lists three types — it fails preflight, not the deploy).

## Database Naming

MongoDB and Postgres databases are named `{service}-{environment}` by default:

| Service | Env | Database |
|---------|-----|---------|
| my-api | sandbox | my-api-sandbox |
| my-api | prod | my-api-prod |

Override with a custom name:
```yaml
spec:
  databases:
    - type: mongodb
      name: shared-data   # uses "shared-data" instead of "{service}-{env}"
```

## Local Development

```
# .env.local
MONGODB_URI=mongodb://localhost:27017/my-api-dev
REDIS_URL=redis://localhost:6379/0
NEO4J_URI=bolt://localhost:7687
```

Or use `tawa config pull` to get the sandbox connection strings:
```bash
tawa config pull  # writes to .env.local
```

## Vault (Dynamic Credentials)

MongoDB credentials are rotated automatically via HashiCorp Vault. The Vault Agent sidecar handles credential injection — your code reads `MONGODB_URI` as always, but the underlying credentials rotate without a redeploy.

Redis and Neo4j Vault rotation is pending (Phase 2c/2d).

## Shared Databases (catalog 0.5.0+)

Services can share databases. The owning service declares `sharedWith`, the consumer declares `consumesDatabase`:

```yaml
# Owning service (analytics-api):
spec:
  databases:
    - type: mongodb
      sharedWith: [reporting-service]

# Consuming service (reporting-service):
spec:
  consumesDatabase:
    - from: analytics-api
      type: mongodb   # → MONGODB_URI_ANALYTICS_API env var
```

## Accessing Data in Running Services

Platform databases are not reachable from localhost. Use the `koko` CLI — it proxies all operations through the Koko API with authentication and safety controls.

```bash
# Install once
npm install -g koko-db

# Authenticate (shares tawa login session)
tawa login
```

### Read Data

```bash
koko query products -s my-api -e prod --filter '{"active":true}' --limit 20
koko sample users   -s my-api -e prod --limit 5
koko stats          -s my-api -e prod
koko collections    -s my-api -e prod
```

### Export / Import (EJSON — type-safe)

`koko export` outputs EJSON format: ObjectIds are preserved as `{"$oid":"..."}` and Dates as `{"$date":"..."}`. This means exports can be re-imported without ObjectId type corruption.

```bash
# Export prod → local file
koko export products -s my-api -e prod -o /tmp/products.ndjson

# Import into sandbox (koko import handles EJSON natively)
koko import products -s my-api -e sandbox -i /tmp/products.ndjson --upsert

# Import into local MongoDB (requires --legacy flag for $oid markers)
mongoimport --db my-api-dev --collection products --file /tmp/products.ndjson --legacy
```

> Always use `koko import` over raw `mongoimport` — no flags to remember, types are handled automatically.

### Copy Between Environments

```bash
# Direct server-side copy — fastest, no download
koko copy products --service my-api --from prod --to sandbox
```

See `tawa-docs/conventions/koko-data-access.md` for the full quickstart.

## What NOT to Do

```yaml
# ❌ WRONG: unsupported types — silently skipped (warning, non-fatal)
spec:
  databases:
    - type: mariadb    # not provisioned — logged as a warning and skipped
    - type: mysql      # not provisioned — logged as a warning and skipped

# ❌ WRONG: hardcoding connection strings
MONGODB_URI=mongodb://prod-host:27017/mydb  # in tawa config — let builder provision it

# ✅ CORRECT: declare in catalog, let builder provision
spec:
  databases:
    - type: mongodb
```

Install

Copy the skill content and save it to:

~/.claude/rules/tawa-database-provisioning.md

Details

Format

Rule

Category

provision

Version

1.0.43707

Tokens

~2,164

Updated

2026-10-13