Tawa Test
`https://iec-test.tawa.pro` — runs every check the platform makes against a service *after* it is deployed and *before* the build is called green. You declare…
Ruledeploy
About
`https://iec-test.tawa.pro` — runs every check the platform makes against a service *after* it is deployed and *before* the build is called green. You declare what it should check in `catalog-info.yaml`; the builder relays the specs after `helm upgrade` and waits for the answer.
Skill Content
This is the raw markdown that gets installed as a Claude Code rule.
# Tawa Test
## The Short Version
**Tawa Test** is the platform's test runner. One service — `iec-test`, at
`https://iec-test.tawa.pro` — runs every check the platform makes against a service *after* it
is deployed and *before* the build is called green. You declare what it should check in
`catalog-info.yaml`; the builder relays the specs after `helm upgrade` and waits for the answer.
```yaml
metadata:
annotations:
insureco.io/catalog-version: "0.7.0" # 0.5.0 for spec.tests alone
spec:
tests:
smoke:
- path: /health
expect: 200
walkthrough: # the standing screenshot tour
title: "Collections"
steps:
- path: /collections
expect: { h1: "Collections", "table tbody tr": "ACME" }
```
```bash
tawa walkthrough --prod # run the tour on demand, no deploy
tawa walkthrough tour.yaml --prod --label pr-42
```
## The feature set
| Feature | What it proves | Declared as | Runs in |
|---------|----------------|-------------|---------|
| **Walkthroughs** | The screens exist, show real data, and here is the picture | `spec.walkthrough` (catalog 0.7.0+) | every environment, including prod |
| **Smoke** | The service answers HTTP the way it says it will | `spec.tests.smoke` (0.5.0+) | every environment |
| **E2E** | A browser flow still completes end to end | `spec.tests.e2e` (0.5.0+) | every environment, including prod |
| **Compliance** | The audit and gas events the service claims to emit are really in Septor and the wallet | `spec.tests.compliance` (0.5.0+) | every environment |
| **Integration** | A declared dependency answers on its cluster-internal URL | **nothing** — CLI / HTTP API only | on demand |
Each suite is requested only when it has specs. A service that declares none is skipped entirely
and the build log says `Testing skipped`.
---
## Walkthroughs
A walkthrough is the feature the rest of Tawa Test exists to support: it drives a UI flow,
**asserts on every page**, screenshots each step, and publishes an immutable page showing what
shipped. A smoke check tells you a route returned 200. A walkthrough tells you what a person
saw — and refuses to publish a picture of a screen whose assertion failed.
```yaml
spec:
walkthrough:
title: "Collections & agency assignment"
auth: portal-editor # a profile NAME, never a token
viewport: { width: 1440, height: 900, scale: 2 } # the default
steps:
- path: /collections/assign
expect:
h1: "Assign Collections" # structural: the right page
"table tbody tr": "ACME" # substantive: the page has data
note: "Assignment now writes the collector's agency (#46)."
act: [{ click: "button:has-text('Scheduled reports')" }]
shot: { selector: "[data-testid='panel']" }
annotate: [{ at: "table thead th:last-child", text: "New agency column" }]
redact: ["[data-pii]"]
```
### What makes it evidence rather than decoration
1. **Every step needs `expect`.** A step without one is rejected by the catalog parser (the
build fails before an image is built, naming `steps[N]`), by the builder's on-demand route,
and by iec-test's own schema. There is no opt-out.
2. **A failed step is never published as a clean screenshot.** It renders as a failure card with
the assertion table, expected vs. actual, and **no image at all**. The page generator ignores
an `image` field on a failed record.
3. **A failure says where the page actually was.** `element not found` cannot distinguish a
signed-out render from a wrong route from a 404, so every failure also records the step's
`url`, the page's own `<title>` (`failure.pageTitle`) and the navigation's HTTP status
(`failure.httpStatus`, null for a step that did not navigate). A `200` on the right URL under
the title *Sign in* is a session problem; a `404` is a route problem. It is deliberately not
a body excerpt — `redact` blurs pixels and cannot redact text, and a bundle is readable by
anyone with its URL.
4. **Assert on data, not chrome.** `h1` proves you are on the right page. It does not prove the
page has content: a session can authenticate and still render an empty table.
### The published bundle
Every run writes a directory, and **the bundle is the page** — CSS inlined, relative image
references, no JavaScript, nothing fetched from the network. Copy it anywhere and it still opens.
```
walkthroughs/{service}/{environment}/{buildId}/
├── index.html # the page: provenance header, per-step assertion table, overlays
├── manifest.json # the machine-readable record — uploaded LAST, its presence means complete
└── 01-collections-assign.png …
```
`manifest.json` is a versioned contract (`version: 1`). The fields worth knowing:
| Field | |
|-------|--|
| `steps[].status` | `passed` \| `failed` |
| `steps[].failure` | `selector`, `expected`, `actual`, `pageTitle`, `httpStatus`, `message` |
| `steps[].warnings` | soft failures — a dropped annotation anchor, a `redact` that matched nothing |
| `steps[].redactions` | each selector and how many elements it actually blurred |
| `auth` | the profile **name** used, or null. Never a token |
| `authPersona` | `{ org, slug }` when the run was driven as a persona — names only |
| `authWarnings` | a guessed audience or cookie name, recorded here and on the page, not only in a pod log |
| `previousBuildId` / `previousBuildUrl` | the bundle captured before this one, for an eyeball diff |
Bundles are **immutable** — a re-capture is a new `buildId`, and an upload onto an existing
manifest is refused. A lost bundle is never a green gate: any upload failure turns the suite
result into `error`.
Retention, per service and environment:
| Class | Kept |
|-------|------|
| Failed capture | full for 7 days, then deleted |
| Last 10 successful builds | full bundle |
| 10–90 days old | manifest, page, and the first and last frame |
| Older than 90 days | manifest only |
### Running one on demand
A walkthrough normally runs as part of the deploy. To run one against the service **as it is
live right now** — no build, no deploy, no gate — use `tawa walkthrough`:
```bash
tawa walkthrough --prod # the catalog's spec.walkthrough
tawa walkthrough tour.yaml --sandbox # a spec from a YAML/JSON file
tawa walkthrough tour.yaml --prod --label pr-42 --open
```
| Flag | |
|------|--|
| `-s, --service <name>` | default: from `catalog-info.yaml` or `.tawa.yaml` |
| `-e, --env <env>` / `--prod` / `--sandbox` / `--uat` | default `prod` |
| `--label <text>` | letters, digits, `.`, `_`, `-`, up to 80 characters — makes the bundle findable later |
| `--host <hostname>` | drive this host instead of the default: the platform host or a verified custom domain for the environment (`tawa` 0.30.0+) |
| `--no-wait` | submit and exit without polling |
| `--json`, `--open` | print the run payload; open the report |
The builder relays a one-off run with the **walkthrough suite alone** — smoke, e2e and
compliance are not touched — under a synthetic build id, `manual-{label-}{time}-{random}`, so it
never collides with a deploy's. Requires the `member` org role (`viewer` can read the result).
Exit code is 1 when a step failed; an on-demand run is reported as failed in **every**
environment, because "non-blocking in prod" is a property of the deploy gate, not of a run
somebody asked for on purpose.
### Which host a run drives
| Where it ran | Target |
|--------------|--------|
| `tawa walkthrough` (on demand) | the service's **public** URL — its verified custom domain for the environment if it has one, else `{service}.{env-prefix}tawa.pro`. `--host` picks another of the service's own hosts; see [Walkthroughs](walkthroughs.md#choosing-the-host---host) |
| Deploy-time | the **cluster-internal** URL, `http://{service}.{namespace}.svc.cluster.local:{port}` |
Both hostnames serve the same deployment, so every status check stays green either way — but an
authenticated app's cookie host and login redirect are pinned to **one** of them. The session
cookie is scoped to the host being driven, so driving the other renders signed out and fails
every assertion with nothing saying why. The build log names both URLs.
### Signed-in runs
`auth:` names a **profile** configured on iec-test — never a token, in a catalog or anywhere
else. Two profiles matter on the platform:
| Profile kind | Session it produces |
|--------------|---------------------|
| `persona` | A real Bio-ID user in an org, **with roles**. Role-gated screens render. |
| `client-credentials` (`staff`) | iec-test's own service identity — an org, no user, no roles. |
A persona is the current answer and a `staff`-style profile is the role-less fallback. Full
mechanics — how the token is minted, what the builder supplies, and why a persona can write —
are in [Walkthroughs → Auth](walkthroughs.md#auth) and [Test Personas](personas.md).
### Authoring captions locally
Captions live in the service's own `catalog-info.yaml`, which means writing prose about a
screenshot you cannot see. The local loop fixes that — it runs `spec.walkthrough` against your
dev server through the same runner and page generator the deploy gate uses, and re-runs on every
save:
```bash
# in the iec-test repo
npx tsx src/cli.ts capture http://localhost:3000/login --out ./data/profiles/staff.json
npx tsx src/cli.ts walkthrough --catalog ../my-svc/catalog-info.yaml \
--target http://localhost:3000 --local --watch --storage-state ./data/profiles/staff.json
```
Only the touched step and the rest of its page group re-run; untouched steps keep their PNG. A
save that leaves the catalog invalid prints the exact path and keeps serving the last good bundle.
---
## Smoke
HTTP probes. The one suite almost every service should declare, and the one whose failure stops
every other suite in the run.
```yaml
spec:
tests:
smoke:
- path: /health
expect: 200
timeout: 5000 # optional
```
Entries carry `path`, `expect` and `timeout` — **nothing else**. An `expectBody` key on a
catalog smoke entry is warned about and dropped: the status check runs, the body assertion never
did. (The richer shape — `method`, `headers`, `body`, `expectBody`, `expectHeaders` — exists in
iec-test's own spec format, reachable from the CLI and the HTTP API, not from a catalog.)
If smoke fails or errors, the orchestrator **skips every remaining suite** in that run.
## E2E
Playwright browser flows — navigate, fill, click, assert, screenshot. Where a walkthrough asks
"does this screen still show the right thing, and here is the picture", e2e asks "does this
*sequence* still complete": log in, submit the form, land on the right page.
Runs in **every** environment. Blocking in sandbox and UAT; **non-blocking in prod**, the same
posture walkthroughs have.
```yaml
metadata:
annotations:
insureco.io/catalog-version: "0.5.0"
spec:
tests:
e2e:
- name: "Login flow"
auth: portal-editor # a profile NAME, never a token
timeout: 30000 # optional, ms
steps:
- { action: navigate, url: /login }
- { action: fill, selector: "#email", value: [email protected] }
- { action: click, selector: "button[type='submit']" }
- { action: wait, selector: "[data-testid='dashboard']", state: visible }
- { action: assert_url, pattern: /dashboard }
- { action: assert_text, selector: h1, text: Dashboard }
```
### The step vocabulary
| Action | Fields | |
|--------|--------|--|
| `navigate` | `url` | A rooted path (`/login`) resolves against the URL the run is driving — write one, not a hostname |
| `fill` | `selector`, `value` | |
| `click` | `selector` | |
| `wait` | `selector`, `state?` | `visible` (default), `hidden`, `attached` |
| `screenshot` | `name`, `fullPage?`, `selector?` | |
| `scroll` | `selector` | Scrolls the element into view |
| `assert_text` | `selector`, `text` | Substring match on the element's text. Several matches: any one may contain it. Visible elements only |
| `assert_url` | `pattern` | |
| `assert_visible` | `selector` | |
| `key` | `key` | e.g. `Escape` |
`{{SERVICE_URL}}` and `{{RUN_ID}}` interpolate in `url` and `value`.
**There is no `evaluate`.** Running arbitrary JS in a browser on iec-test's pod network is
refused from any remote caller; it exists only in the local CLI. A catalog that declares it
fails the build at parse time, by name.
### Steps in a file instead
A long flow does not have to live in the catalog. Point at a repo file holding `{ steps: [...] }`:
```yaml
spec:
tests:
e2e:
- name: "Collections"
spec: e2e/collections.yaml # relative to catalog-info.yaml
```
```yaml
# e2e/collections.yaml
steps:
- { action: navigate, url: /collections }
- { action: assert_text, selector: "table tbody tr", text: ACME }
```
`.yaml`, `.yml` or `.json`. The path is relative to `catalog-info.yaml` and may not be absolute
or contain `..` — the builder loads it out of the clone and will not read outside it. An entry
declares **either** `steps` or `spec`, never both and never neither.
### Everything is checked before the image is built
The builder validates every step against the vocabulary above at catalog-parse time, and names
what it found:
```
Catalog "my-api": spec.tests.e2e[2].steps[1] (action "click") requires "selector" (a non-empty string).
Catalog "my-api": spec.tests.e2e[0].steps[0] has unknown action "type". Valid actions: assert_text, assert_url, …
```
An unknown key on a step is an error rather than a warning: a misspelt `selector` silently
dropped leaves an assertion that cannot fail. `tawa preflight` (0.27.0+) runs the same checks
without spending a build.
### Signed-in flows
`auth:` names the same profiles walkthroughs use, resolved by iec-test from its own
configuration — see [Signed-in runs](#signed-in-runs) above and
[Test Personas](personas.md). Without it the flow runs signed out, which for a logged-in app
means asserting on a login page. A profile that cannot be established **fails the flow** rather
than quietly running it anonymously.
### Which host it drives — the same caveat as a walkthrough
A deploy-time e2e run drives the **cluster-internal** URL. An app whose session is bound to its
public host renders signed out there through no fault of the deploy
([iec-test#39](https://git.insureco.io/insureco/iec-test/issues/39)) — which is exactly why a
prod failure is non-blocking rather than a gate. Relative `navigate` urls resolve against
whichever host the run is driving, so the flow itself needs no change between the two.
## Compliance
Asks Septor and the wallet whether the events a service claims to emit are really there, with
retries because those writes are asynchronous.
```yaml
spec:
tests:
compliance:
septor:
- eventType: payment.created
entityId: "{{orgSlug}}"
wallet:
- type: transaction
walletId: "wallet-{{orgSlug}}"
```
A check that cannot be *performed* — missing config, a transport failure — is reported as
`error`, never as `fail`, so the gate neither passes nor fails on an infrastructure problem.
> **Moving a `spec.compliance` block under `spec.tests` turns it on.** In its correct position
> it becomes a deploy gate that queries Septor on every build, and an environment that has never
> emitted those events fails every deploy until it has real history. Move it once the service
> has produced the events you are asserting on.
## Integration
Builds each declared dependency's cluster-internal URL and probes it. It is a real suite in
iec-test and it is accepted by `POST /runs` — but there is no `spec.tests.integration`, so a
catalog cannot ask for it. Drive it from the CLI:
```bash
npx tsx src/cli.ts run specs/deps.json --target https://my-api.tawa.pro \
--suite integration --environment prod
```
---
## Gating, per environment
| Suite | sandbox / UAT | prod |
|-------|---------------|------|
| `smoke` | blocking | blocking |
| `e2e` | blocking | **non-blocking** — `[warn]` in the build log, deploy proceeds |
| `compliance` | blocking | blocking |
| `walkthrough` | blocking | **non-blocking** — `[warn]` in the build log, deploy proceeds, bundle still publishes with the failure card |
The prod exemption covers the two **browser** suites, and it is narrow: it applies only when
they are the **only** failing suites. A run where smoke also failed fails the build as usual.
Both are driven at the cluster-internal URL, where an app whose session is bound to its public
host renders signed out through no fault of the deploy — failing a prod deploy on that would
teach people to delete the suite rather than fix anything.
> **e2e used to be skipped in prod entirely**, which meant it never ran: nothing on this
> platform deploys to sandbox or UAT. Fixed in
> [iec-builder#161](https://git.insureco.io/insureco/iec-builder/issues/161) along with the
> payload shape (see [E2E](#e2e)).
A failed gate **does not roll back** the deployment. The pod stays up; the build is marked
failed. Fix and redeploy.
## Reading the result
- **The build log** — `Test result: walkthrough: passed (5/5)`, then `Walkthrough: <url>`.
- **The page** — `https://iec-test.tawa.pro/walkthroughs/{service}/{env}/{buildId}/index.html`.
Provenance header (service, environment, build id, commit, capture time, viewport, auth
profile, `N of M steps passed`), each step with its assertion rendered verbatim, positioned
annotation overlays, redaction notes, and a `← Previous build` link.
- **`manifest.json`** beside it — the fields in the table above.
- **The index.** `https://iec-test.tawa.pro/ui/walkthroughs` lists every walkthrough still in
storage, by service and then newest first, with search by label or title. It needs a Bio-ID
sign-in (org `insureco` or `super_admin`); the bundle URLs themselves stay open by URL. See
[Walkthroughs → Finding a walkthrough later](walkthroughs.md#finding-a-walkthrough-later).
- **The status page.** When the deploy-gated tests pass, the builder attaches the bundle URL to
the service's changelog in Koko, so the latest walkthrough is linked from the **What's new**
panel on the service's status page. A redeploy of the same commit that produces no new notes
still gets its walkthrough link attached to the changelog Koko already holds. See
[Uptime Monitoring](pulse.md).
Bundle pages are served by iec-test itself as a deliberately dumb read-through: the filename
must be one the bundle's **own manifest** lists, content types are fixed by filename, there is
no listing, and nothing from the request becomes part of an object key before it has matched.
## Where the code and these docs disagree
Recorded here because the code is what runs.
**There are five suites, not four.** `integration` is registered in iec-test alongside smoke,
e2e, compliance and walkthrough, and `POST /runs` accepts it — the builder simply never asks
for it, because `spec.tests` has no key for it.
**Execution order is fixed by the runner registry**, not by the order you declare: smoke →
integration → e2e → walkthrough → compliance. (This page previously listed compliance before
walkthrough; `RUNNERS` in `src/runners/registry.ts` has the walkthrough fourth.) If smoke fails
or errors, every remaining suite is skipped.
## What NOT to do
```yaml
# ❌ a walkthrough step with no assertion — rejected everywhere, at parse time
- path: /reports
note: "The new reports page"
# ❌ chrome only, on a data page — goes green on an empty table
- path: /collections
expect: { h1: "Collections" }
# ❌ a token where a profile name belongs
auth: eyJhbGciOi...
# ❌ expectBody on a catalog smoke entry — warned and dropped, the assertion never runs
spec:
tests:
smoke:
- path: /api/health
expect: 200
expectBody: { success: true }
# ❌ spec.compliance at the top level — parses, deploys, does nothing
spec:
compliance:
septor: [...]
# ❌ an e2e step with no assertion — a flow that only navigates proves nothing
spec:
tests:
e2e:
- name: "Dashboard"
steps:
- { action: navigate, url: /dashboard }
# ❌ a hostname in an e2e navigate — a deploy-time run drives the cluster-internal
# URL and an on-demand one drives the public URL, so one of them is always wrong
- { action: navigate, url: "https://my-api.tawa.pro/login" }
# ❌ evaluate in a catalog — refused at parse time; it is CLI-only
- { action: evaluate, script: "window.localStorage.setItem('x','1')" }
# ✅
spec:
tests:
smoke:
- path: /health
expect: 200
e2e:
- name: "Login flow"
auth: portal-editor
steps:
- { action: navigate, url: /login }
- { action: fill, selector: "#email", value: [email protected] }
- { action: click, selector: "button[type='submit']" }
- { action: assert_text, selector: h1, text: Dashboard }
compliance:
septor:
- eventType: payment.created
entityId: "{{orgSlug}}"
walkthrough:
title: "Collections"
auth: portal-editor
steps:
- path: /collections
expect: { h1: "Collections", "table tbody tr": "ACME" }
redact: ["td.insured-name", "td.email"]
```
- Don't put a one-off feature walkthrough in the catalog — it runs on every deploy from then on.
Use `tawa walkthrough <spec>` for that.
- Don't treat a deploy-time walkthrough of an app whose session is bound to its public host as
reliable — it drives the cluster-internal URL. Run that one on demand.
- Don't expect a failed gate to roll anything back. It doesn't.
- Don't publish bundle ids where they invite enumeration. Anyone with a URL can read that bundle;
`redact` is the primary control and the id namespace is the second. There is no third.
## See also
- [Walkthroughs](walkthroughs.md) — the spec in full, the authoring rules, and auth.
- [Test Personas](personas.md) — a logged-in session without a password.
- [`spec.walkthrough` in the catalog reference](catalog-info.md#specwalkthrough--asserted-screenshot-walkthroughs)
- [Build & Deploy Pipeline](build-pipeline.md) — step 21, where the gate runs.
- [Uptime Monitoring](pulse.md) — the changelog panel the bundle link lands in.
Install
Copy the skill content and save it to:
~/.claude/rules/tawa-tawa-test.mdComing soon via CLI:
tawa chaac install tawa-tawa-testDetails
- Format
- Rule
- Category
- deploy
- Version
- 1.0.71375
- Tokens
- ~5,603
- Updated
- 2026-10-13
platformtawa-testiec-testtestingwalkthroughsmokee2ecompliancedeploypersonas