# Database Backups on Tawa

## The Short Version

Every MongoDB database on the platform is backed up automatically every night, and you can take an on-demand backup of your own service's database any time. Backups are stored in platform object storage (MinIO) with grandfather-father-son retention. Nothing to declare in `catalog-info.yaml` — backups are on by default.

```bash
koko backup -s my-api -e prod        # take a backup now
koko backups -s my-api -e prod       # list backups
koko backup-download <id> -s my-api # get a temporary download link
koko restore <id> -s my-api --confirm   # restore (admin, destructive)
```

## Automatic Nightly Backups

The builder runs a platform-wide backup every night (02:00 UTC by default). For each database it runs `mongodump --archive --gzip` and uploads the archive to the `tawa-db-backups` bucket.

Each nightly run is tagged with a retention tier (grandfather-father-son):

| Day | Tier | Kept |
|-----|------|------|
| 1st of month | `monthly` | 3 |
| Sunday | `weekly` | 4 |
| Any other day | `daily` | 7 |

Older automated backups beyond the keep count are pruned automatically. **Manual backups are never auto-pruned** — you delete those yourself.

## On-Demand (Self-Service) Backups

Take a backup of your own service's database whenever you want — before a risky migration, a data fix, or a release.

```bash
koko backup -s my-api -e prod
```

This dumps your service's MongoDB database, uploads it, and records it. Requires the `member` org role on the service's org.

### Listing

```bash
koko backups -s my-api -e prod
```

Shows both your manual backups and the nightly automated ones for your database, newest first, with size, tier, who took it, and status.

### Downloading

```bash
koko backup-download <backup-id> -s my-api -e prod
```

Returns a short-lived (15-minute) presigned URL to the gzipped archive. The archive is a standard `mongodump --archive --gzip` file — restore it anywhere with `mongorestore --archive=<file> --gzip`.

## Restore

Restore is **destructive**: it runs `mongorestore --drop`, which drops the target collections and replaces them from the backup. It is scoped to the original database only (`--nsInclude`), so no other database is ever touched.

```bash
koko restore <backup-id> -s my-api --confirm
```

Requires the `admin` org role **and** the explicit `--confirm` flag. Without `--confirm` the request is rejected.

## API Endpoints

Self-service backups are exposed by the builder, mirroring the `tawa db connect` whitelist model:

| Method | Endpoint | Role | Purpose |
|--------|----------|------|---------|
| `POST` | `/services/:name/databases/backup` | member | Take an on-demand backup |
| `GET` | `/services/:name/databases/backups` | viewer | List backups (manual + auto) |
| `GET` | `/services/:name/databases/backups/:id/download` | member | Presigned download URL |
| `POST` | `/services/:name/databases/restore` | admin | Restore (`{ "backupId": "...", "confirm": true }`) |
| `GET` | `/databases/backups` | platform admin | List all backups across orgs |

## Where Backups Live

Backups are stored in the platform MinIO object store, bucket `tawa-db-backups`, keyed:

```
{env}/{service}/{tier}/{db}-{timestamp}.archive.gz
```

## Off-Site Replication (Disaster Recovery)

The platform MinIO runs on the **same host** as MongoDB, so same-box backups protect against bad migrations and accidental data loss — **not** against total host loss. To cover that, the backup bucket is replicated nightly to an **off-site MinIO** on a separate DigitalOcean droplet (`tawa-db-backups`, region **sfo2** — a different region from the builder host in sfo3).

```
02:00 UTC  builder dumps all DBs        -> tawa MinIO  (tawa-db-backups, on the builder host)
03:30 UTC  systemd timer: mc mirror     -> off-site MinIO droplet (sfo2, 200GB volume)
```

- Replication is an incremental `mc mirror --overwrite --remove`, so the off-site copy tracks retention pruning.
- Script: `iec-builder/scripts/backup-offsite-sync.sh` (reads `/etc/tawa-backup-offsite.env`); units: `tawa-backup-offsite.{service,timer}`.
- The off-site droplet's MinIO port is firewalled to the builder host's IP only.

If the builder host is lost entirely, the latest backups still exist on the off-site droplet and can be restored from there.

## Configuration (Platform Operators)

Controlled by builder environment variables (defaults shown):

| Var | Default | Purpose |
|-----|---------|---------|
| `BACKUP_ENABLED` | `true` | Master switch for the nightly scheduler |
| `BACKUP_BUCKET` | `tawa-db-backups` | MinIO bucket |
| `BACKUP_CRON_HOUR_UTC` | `2` | Nightly run hour (UTC) |
| `BACKUP_RETENTION_DAILY` | `7` | Daily backups kept |
| `BACKUP_RETENTION_WEEKLY` | `4` | Weekly backups kept |
| `BACKUP_RETENTION_MONTHLY` | `3` | Monthly backups kept |
| `BACKUP_DOWNLOAD_TTL_SECONDS` | `900` | Presigned download URL lifetime |
| `BACKUP_TMP_DIR` | `/tmp/iec-backup` | Scratch dir for archives |
| `BACKUP_DUMP_TIMEOUT_MS` | `7200000` | Per-DB mongodump/restore timeout (120 min) — raise for very large databases |
| `BACKUP_EXCLUDE_PATTERNS` | (empty) | Comma-separated globs of DB names to skip in the nightly run (e.g. `migration_*`). Skipped DBs are logged each run. |

Backups require `DB_MONGODB_ADMIN_URI` (admin Mongo access) and MinIO admin credentials in Vault KV (`secret/minio/admin`). If `DB_MONGODB_ADMIN_URI` is unset, the scheduler logs a warning and stays idle.

## What NOT to Do

```bash
# ❌ WRONG: restore without --confirm — rejected (destructive guard)
koko restore <id> -s my-api

# ❌ WRONG: assuming same-box MinIO is full disaster recovery
#    It protects against bad data, not host loss. Off-box replication is separate.

# ✅ CORRECT: snapshot before a risky migration, then restore if it goes wrong
koko backup -s my-api -e prod
# ... run migration ...
koko restore <id> -s my-api --confirm   # only if needed
```