Database Backups on Tawa

About

Every MongoDB and PostgreSQL database on the platform is backed up automatically every night, and you can take an on-demand backup of your own service's database any time. Backups are stored in platform object storage (MinIO) with grandfather-father-son retention. Nothing to declare in `catalog-info.yaml` — backups are on by default.

Skill Content

This is the raw markdown that gets installed as a Claude Code rule.

# Database Backups on Tawa

> **Koko first.** Use `koko` for every read and write on a service database. Back up before any prod write. See [Koko Data Access](koko-data-access.md).

## The Short Version

Every MongoDB and PostgreSQL database on the platform is backed up automatically every night, and you can take an on-demand backup of your own service's database any time. Backups are stored in platform object storage (MinIO) with grandfather-father-son retention. Nothing to declare in `catalog-info.yaml` — backups are on by default.

```bash
koko backup -s my-api -e prod        # take a backup now
koko backups -s my-api -e prod       # list backups
koko backup-download <id> -s my-api # get a temporary download link
koko restore <id> -s my-api --confirm   # restore (admin, destructive)
```

A service with a Postgres database (`spec.databases: [{ type: postgres }]`) is backed up the
same way. On-demand, pass `"type": "postgres"` in the request body (see
[Postgres databases](#postgres-databases)); a service that declares only Postgres gets it by
default.

`koko backup` **starts** a backup and returns straight away (`202`) with its id —
the dump itself runs for minutes on a large database. Poll `koko backups` until
that id shows `completed` before you rely on it.

## Automatic Nightly Backups

The builder runs a platform-wide backup every night (02:00 UTC by default). For each MongoDB database it runs `mongodump --archive --gzip`, and for each PostgreSQL database `pg_dump -Fc`, and uploads the archive to the `tawa-db-backups` bucket. An engine that is not configured on the builder (no admin URI) is skipped; a failure to list one engine's databases counts as one failed backup and never stops the other engine.

Each nightly run is tagged with a retention tier (grandfather-father-son):

| Day | Tier | Kept |
|-----|------|------|
| 1st of month | `monthly` | 3 |
| Sunday | `weekly` | 4 |
| Any other day | `daily` | 7 |

Older automated backups beyond the keep count are pruned automatically. **Manual backups are never auto-pruned** — you delete those yourself.

## On-Demand (Self-Service) Backups

Take a backup of your own service's database whenever you want — before a risky migration, a data fix, or a release.

```bash
koko backup -s my-api -e prod
```

This dumps your service's MongoDB database, uploads it, and records it. Requires the `member` org role on the service's org.

### It returns immediately — the backup runs in the background

A `mongodump` of a few hundred MB takes **minutes**. The request does not wait
for it. The builder writes the backup record, replies **`202 Accepted`** with
the id, and runs the dump detached:

```jsonc
// POST /services/my-api/databases/backup  ->  202
{ "success": true,
  "data": { "backupId": "350924a4-…", "status": "running",
            "databaseName": "my-api-prod", "environment": "prod",
            "startedAt": "2026-09-16T01:51:16.845Z" } }
```

**A 202 does not mean the backup is done.** Poll the id until it leaves
`running`:

```bash
koko backups -s my-api -e prod          # the running row is listed, with its status
```

```
GET /services/my-api/databases/backups/350924a4-…
  -> { "status": "running"   }   # still dumping
  -> { "status": "completed", "sizeBytes": 782417699, "completedAt": "…" }
  -> { "status": "failed",    "error": "mongodump failed for …" }
```

Only a `completed` backup can be downloaded or restored — the download and
restore endpoints reject anything else with `409 NOT_RESTORABLE`.

> **Why not just wait?** The builder sits behind a reverse proxy that severs an
> idle upstream connection at 60 seconds. Holding the request open for the whole
> dump meant every database over roughly a few hundred MB reported
> `504 Gateway Timeout` for a backup that in fact completed normally — right at
> the moment an operator was about to do something risky. See
> [iec-builder#124](https://git.insureco.io/insureco/iec-builder/issues/124).

### One backup per database at a time

Asking for a second backup of a database that already has one `running` is
refused with **`409 BACKUP_IN_PROGRESS`**, and the error carries the in-flight
id so you can poll that one instead:

```jsonc
{ "success": false,
  "error": { "code": "BACKUP_IN_PROGRESS",
             "message": "A backup of \"my-api-prod\" is already running (id 350924a4-…, started …)",
             "details": { "backupId": "350924a4-…", "startedAt": "…" } } }
```

Treat a 409 as "it is already happening", not as a failure. Two concurrent
dumps of one database double the load for no benefit.

### Waiting synchronously (`wait=true`)

`?wait=true` — or `{"wait": true}` in the body — restores the old behaviour and
returns the finished record with `200`:

```
POST /services/my-api/databases/backup?wait=true
```

**This is subject to the proxy's read timeout (60s by default).** It is fine for
a small sandbox database and a bad idea for anything large — that timeout is
exactly the problem `202` exists to avoid, and no client-side flag can raise it.
Prefer the 202 + poll path.

### Listing

```bash
koko backups -s my-api -e prod
```

Shows both your manual backups and the nightly automated ones for your database,
newest first, with size, tier, who took it, and status. **In-flight backups
appear here too**, as `running`, from the moment they start — both on-demand and
nightly ones. A dump that dies leaves a durable `failed` row with the error, so
"nothing in the list" now genuinely means "no backup was ever attempted".

### Downloading

```bash
koko backup-download <backup-id> -s my-api -e prod
```

Returns a short-lived (15-minute) presigned URL to the gzipped archive. The archive is a standard `mongodump --archive --gzip` file — restore it anywhere with `mongorestore --archive=<file> --gzip`.

## Postgres databases

Postgres backups (iec-builder#233) use the same endpoints, records, bucket, key scheme and
retention as MongoDB. What differs:

| | MongoDB | PostgreSQL |
|---|---|---|
| Dump | `mongodump --archive --gzip` | `pg_dump -Fc` (custom format, already compressed) |
| Object key suffix | `.archive.gz` | `.pgdump` |
| Restore | `mongorestore --drop --nsInclude=<db>.*` | `pg_restore --clean --if-exists --single-transaction --exit-on-error` into that one database |
| Record field | `engine: "mongodb"` (or absent, on older records) | `engine: "postgres"` |

Choosing the database for an on-demand backup:

```jsonc
// POST /services/my-api/databases/backup
{ "environment": "prod", "type": "postgres" }   // type: "mongodb" | "postgres"
```

Omit `type` and you get the MongoDB database, unless the service declares only Postgres. A
service with both has two databases of the **same default name** (`my-api-prod`), so the engine
is part of everything that identifies a backup: the in-flight check (`409 BACKUP_IN_PROGRESS`
is per engine), retention groups, and the object key suffix.

A Postgres restore runs in one transaction, so a restore that fails part-way rolls back and
leaves the database as it was. Download gives a standard custom-format archive — restore it
anywhere with `pg_restore --clean --if-exists -d <db> <file>`.

The dump runs as the builder's admin role (`DB_POSTGRES_ADMIN_URI`). Its password is passed in
`PGPASSWORD`, never on the command line.

> `koko backup` does not have a `--type` flag yet. Until it does, call the endpoint directly for
> a Postgres on-demand backup; the nightly run covers Postgres either way.

## Restore

Restore is **destructive**: for MongoDB it runs `mongorestore --drop`, which drops the target collections and replaces them from the backup, scoped to the original database only (`--nsInclude`). For Postgres it runs `pg_restore --clean` against the original database only. No other database is ever touched.

```bash
koko restore <backup-id> -s my-api --confirm
```

Requires the `admin` org role **and** the explicit `--confirm` flag. Without `--confirm` the request is rejected.

## API Endpoints

Self-service backups are exposed by the builder, mirroring the `tawa db connect` whitelist model:

| Method | Endpoint | Role | Purpose |
|--------|----------|------|---------|
| `POST` | `/services/:name/databases/backup` | member | Start an on-demand backup (body `type: mongodb \| postgres`, optional) — **`202`** + `{ backupId, status: "running", engine }`; `409 BACKUP_IN_PROGRESS` if one is already running; `?wait=true` blocks for the finished record (proxy-timeout bound) |
| `GET` | `/services/:name/databases/backups` | viewer | List backups (manual + auto, including `running` and `failed`) |
| `GET` | `/services/:name/databases/backups/:id` | viewer | One backup record — poll this until `status` leaves `running` |
| `GET` | `/services/:name/databases/backups/:id/download` | member | Presigned download URL (`completed` only) |
| `POST` | `/services/:name/databases/restore` | admin | Restore (`{ "backupId": "...", "confirm": true }`) |
| `GET` | `/databases/backups` | platform admin | List all backups across orgs |

## Where Backups Live

Backups are stored in the platform MinIO object store, bucket `tawa-db-backups`, keyed:

```
{env}/{service}/{tier}/{db}-{timestamp}.archive.gz   # MongoDB
{env}/{service}/{tier}/{db}-{timestamp}.pgdump       # PostgreSQL
```

### On disk

MinIO's data is on the attached DO volume (`volume-sfo3-01`, ext4, `/mnt/volume_sfo3_01`)
since 2026-10-03, at `/mnt/volume_sfo3_01/minio`, **bind-mounted at `/data/minio`**. Before
that it was a plain directory on the builder host's root disk, where `tawa-db-backups` alone
(~79 GB) had pushed the disk to 87-88% full (iec-builder#231).

**The path MinIO is given must stay `/data/minio`.** `/etc/default/minio` keeps
`MINIO_VOLUMES="/data/minio"` and the unit's `ExecStart` is untouched: `.minio.sys` records
the drive endpoint it was formatted with, so pointing MinIO at `/mnt/volume_sfo3_01/minio`
directly (a `ExecStart` or `MINIO_VOLUMES` override) fails at startup with
`file access denied endpoint="/data/minio"` / `0 drives provided`, and
`/minio/health/cluster/read` returns 503. A bind mount changes where the bytes live without
changing the path MinIO sees.

Three pieces make it boot-safe:

| Piece | Where | Does |
|-------|-------|------|
| bind mount | `/etc/fstab`: `/mnt/volume_sfo3_01/minio /data/minio none bind,nofail,x-systemd.requires-mounts-for=/mnt/volume_sfo3_01 0 0` | the bind waits for the volume to mount |
| `RequiresMountsFor=/data/minio` | `/etc/systemd/system/minio.service.d/10-requires-bind.conf` | MinIO is ordered after, and requires, the bind mount |
| `ExecStartPre=/usr/bin/mountpoint -q /data/minio` | same drop-in | MinIO refuses to start unless `/data/minio` is a mountpoint, so it can never start (and format) on an empty root-disk directory |

The drop-in carries **no** `ExecStart` override. An earlier version of this page described an
`override.conf` that repointed MinIO at the volume; that approach does not work (above).

#### Verifying it

```bash
findmnt /data/minio                                   # SOURCE /dev/sda[/minio]
for e in live cluster cluster/read; do
  curl -s -o /dev/null -w "$e %{http_code}\n" http://127.0.0.1:9000/minio/health/$e; done   # all 200
journalctl -u minio | grep -iE 'denied|0 drives'      # nothing
systemd-analyze verify /etc/systemd/system/minio.service ; findmnt --verify
```

`/minio/health/cluster/read` is the one that caught the failed path change: `live` returns
200 even when no drive is usable. Also do a real put and get (make a throwaway bucket, upload
a file, download it, compare checksums, delete both).

#### Migrating or repeating the move

1. Pre-sync while MinIO runs: `rsync -aHAX --delete /data/minio/ <volume>/minio/`.
2. `systemctl stop minio`; final `rsync -aHAX --delete` and compare file counts and total
   bytes exactly. A `--checksum` pass over ~86 GB took about 10 minutes with MinIO down; a
   size+mtime pass after the pre-sync is much shorter.
3. `mv /data/minio /data/minio.old-<YYYYMMDD-HHMM>`, `mkdir /data/minio`, `chown tawa:tawa`.
4. Back up `/etc/fstab`, add the line above, `mount /data/minio`, add the drop-in,
   `systemctl daemon-reload`, `systemctl start minio`, then run the checks above.

**Rollback:** `systemctl stop minio`, `umount /data/minio`, `rmdir /data/minio`, move
`/data/minio.old-*` back to `/data/minio`, delete the drop-in and the fstab line,
`daemon-reload`, start, re-check health.

#### Deleting the old copy

Keep `/data/minio.old-*` (it is the rollback and ~86 GB of root disk) until **both** the next
nightly backup (02:00 UTC) **and** the off-site mirror (03:30 UTC) have completed against the
new location. Only then delete it; root-disk usage drops to about 13%. The off-site copy is
`mc mirror`ed from MinIO, so it is only a valid check once it has run after the move.

## Backup Record Status

Every backup — on-demand **and** nightly — is written the moment its dump starts
and updated when it finishes:

| `status` | Meaning | Fields |
|----------|---------|--------|
| `running` | The dump is in progress | `startedAt` |
| `completed` | Archive uploaded, restorable | `startedAt`, `completedAt`, `sizeBytes`, `objectKey` |
| `failed` | The dump or upload failed | `startedAt`, `completedAt`, `error` |

A `running` row is not a stuck row — a multi-GB database legitimately takes many
minutes. `BACKUP_DUMP_TIMEOUT_MS` (default 120 minutes) is the ceiling after
which the dump is killed and the record moves to `failed`.

## Off-Site Replication (Disaster Recovery)

The platform MinIO runs on the **same host** as MongoDB, so same-box backups protect against bad migrations and accidental data loss — **not** against total host loss. To cover that, the backup bucket is replicated nightly to an **off-site MinIO** on a separate DigitalOcean droplet (`tawa-db-backups`, region **sfo2** — a different region from the builder host in sfo3).

```
02:00 UTC  builder dumps all DBs        -> tawa MinIO  (tawa-db-backups, on the builder host)
03:30 UTC  systemd timer: mc mirror     -> off-site MinIO droplet (sfo2, 200GB volume)
```

- Replication is an incremental `mc mirror --overwrite --remove`, so the off-site copy tracks retention pruning.
- Script: `iec-builder/scripts/backup-offsite-sync.sh` (reads `/etc/tawa-backup-offsite.env`); units: `tawa-backup-offsite.{service,timer}`.
- The off-site droplet's MinIO port is firewalled to the builder host's IP only.

If the builder host is lost entirely, the latest backups still exist on the off-site droplet and can be restored from there.

## Configuration (Platform Operators)

Controlled by builder environment variables (defaults shown):

| Var | Default | Purpose |
|-----|---------|---------|
| `BACKUP_ENABLED` | `true` | Master switch for the nightly scheduler |
| `BACKUP_BUCKET` | `tawa-db-backups` | MinIO bucket |
| `BACKUP_CRON_HOUR_UTC` | `2` | Nightly run hour (UTC) |
| `BACKUP_RETENTION_DAILY` | `7` | Daily backups kept |
| `BACKUP_RETENTION_WEEKLY` | `4` | Weekly backups kept |
| `BACKUP_RETENTION_MONTHLY` | `3` | Monthly backups kept |
| `BACKUP_DOWNLOAD_TTL_SECONDS` | `900` | Presigned download URL lifetime |
| `BACKUP_TMP_DIR` | `/tmp/iec-backup` | Scratch dir for archives (dumps of every engine). **The tawa host sets it to `/mnt/volume_sfo3_01/iec-backup-tmp`** so multi-hundred-MB dumps never land on the near-full root disk — keep it on the volume |
| `BACKUP_DUMP_TIMEOUT_MS` | `7200000` | Per-DB dump/restore timeout (120 min, mongodump and pg_dump alike) — raise for very large databases |
| `BACKUP_EXCLUDE_PATTERNS` | (empty) | Comma-separated globs of DB names to skip in the nightly run (e.g. `migration_*`). Skipped DBs are logged each run. |

Backups require MinIO admin credentials in Vault KV (`secret/minio/admin`) plus an admin URI per engine: `DB_MONGODB_ADMIN_URI` for MongoDB, `DB_POSTGRES_ADMIN_URI` for PostgreSQL (`pg_dump` / `pg_restore` 16 on the builder host's PATH). An engine without its URI is skipped; if neither is set, the scheduler logs a warning and stays idle.

## What NOT to Do

```bash
# ❌ WRONG: treating the 202 as "the backup is done"
#    202 means "started". Poll the id until status leaves "running" before you
#    migrate — a backup you have not confirmed is not a safety net.

# ❌ WRONG: re-running koko backup because the first one "didn't come back"
#    It did come back, with 202. A second request is refused with 409
#    BACKUP_IN_PROGRESS anyway; poll the id the 409 hands you.

# ❌ WRONG: reaching for ?wait=true on a large database
#    It re-creates the 60s proxy timeout the 202 path exists to avoid.

# ❌ WRONG: restore without --confirm — rejected (destructive guard)
koko restore <id> -s my-api

# ❌ WRONG: assuming same-box MinIO is full disaster recovery
#    It protects against bad data, not host loss. Off-box replication is separate.

# ✅ CORRECT: snapshot before a risky migration, then restore if it goes wrong
koko backup -s my-api -e prod           # 202 — note the backupId
koko backups -s my-api -e prod          # poll until that id shows "completed"
# ... run migration ...
koko restore <id> -s my-api --confirm   # only if needed
```

Install

Copy the skill content and save it to:

~/.claude/rules/tawa-database-backups.md

Details

Format

Rule

Category

provision

Version

1.0.5063

Tokens

~4,279

Updated

2026-10-13