How FrontierJS is tested

CI is one script, and it runs on your laptop.

bun run ci is the whole of CI. GitHub Actions calls that one file and does nothing else, so a green run on a laptop and a green run on GitHub are the same run. It has sixteen phases, from lint and typecheck to a real deploy.

No quiet skips. Every exception is a named entry with a reason in scripts/ci-allowances.json.
Known failures are named, so a new one is loud instead of lost in the noise.
Every run writes its summary to a file, so a failure can be read back instead of reproduced.

$ bun run ci -- --help
…
Phases, in the order a full run does them
  hygiene     fast
  lint        fast
  structure   fast
  registers   fast
  snapshots   fast
  access      fast
  coverage    fast
  registry    fast
  advisories  fast
  scaffold    fast
  typecheck   fast
  deploy      full
  mutate      full
  tutor       full
  perf        full
  tests       full

The phase list, printed by the script itself.

We test what you install, not just our repo.

A monorepo can pass every test and still ship a broken package. So CI packs every publishable package into a tarball, creates a new app from those tarballs, installs it and builds it. Then it deploys that app to a real machine, crashes it, resumes it and rolls it back.

The tutorial runs in CI. All thirteen fli tutor lessons execute end to end on every full run.
The docs run too. Every code block in Litestone's hand-written pages is built or executed, and a broken example is filed as a bug.

scaffold  ✓ a scaffolded app installs from the working tree and builds
deploy    ✓ a scaffolded app containerizes and answers health, from npm and from the tree
deploy    ✓ deploy → deploy → crash → resume → revert, against a real machine
deploy    ✓ a paused app refuses at the edge and serves again, against a real Caddy
deploy    ✓ a pause drains every queue in a running container and the unpause runs what it held

The scaffold and deploy checks, from a full CI run's log.

26,000 tests, and 97 drives against real browsers and servers.

Every package has its own suite, and every suite runs on every full CI run. The house rule is that behavior crossing packages is tested against the real dependency, not a fake client. Compiler tests parse their own output, because a clean compile is not proof of valid JavaScript.

Package	Realm	Tests passed
litestone	Data	8,092
cli	Tooling	4,326
junction	API	3,483
mesa	UI	2,588
sierra	UI	2,480
basecamp	Deployment	1,147
auth	API	685
everything else	—	3,342
Total		26,143

Unit tests are not enough for a framework that spans database to browser. A drive starts the real app and proves a flow end to end, in a real browser when the flow is a page: clicking, typing, and reading the console for errors. fli proves reads your diff and names the drives your change needs, and fli done will not call it finished until they pass.

$ fli proves --from 778c2dce4
· 86 file(s) changed
…
  sierra's shell facts — DEFAULT_LANG, SKIP_LINK, applyDocument() on every target, wrapDocument() — and PAGE_AUDIT in mesa/src/drive.js
  names the file — mesa/src/drive.js
    read  packages/sierra/test/document-shell.test.js
    run   cd example && bun run verify
    run   cd example && bun run verify:site
    run   cd packages/basecamp && bun run verify

One change's answer: the file it touched, and what it owes.

We test the tests.

A large test count can still miss the one rule that matters. In FrontierJS, who may read and change what is declared in the schema and enforced at the data layer. So litestone mutate breaks the schema one rule at a time — drops a deny, stores an @encrypted field in plaintext, loosens a relation's access — and checks that something catches each break.

A break nothing catches is a rule nothing tests, and it is reported by name.
The floor is 100%. CI fails if the share of breaks caught drops below it, and that floor can only go up.

$ bunx litestone mutate --schema db/schema.lite \
    --kinds deny-drop,encrypted-drop,guarded-drop,relator-loosen,relator-tighten

  litestone mutate

  9 mutants · schema.lite

  9/9 · 50s · encrypted-drop killed: PushSubscription: an @encrypted field stored in plaintext
  ✓  100% killed  9/9 graded · 50s

  Every mutant was noticed. Nothing to do.

Mutation testing on the example app's schema.

Quality only moves one way, and the gaps are written down.

Ratchets. Every package typechecks with zero errors except one, whose ceiling is written in a file. That number can only go down; raising it shows up in a diff and needs a reason.

Nineteen rules, each with an enforcer. The architecture rests on nineteen invariants, and a generated file names what enforces each one. Where coverage is partial it says so in plain words, instead of looking the same as a rule that is fully enforced.

Defects and decisions are public. Every known defect has an id in ISSUES.md and every ruling one in DECISIONS.md, both in the repo.

{
  "//": "Invariant 14 — typecheck ceilings ratchet DOWN only. One number per package (directory name). Absent = 0 = must be clean. …",
  "litestone": 47
}

scripts/typecheck-baselines.json: one package with a ceiling, every other at zero.

Covered: **19 of 19**.

##^{# 1. Dependency direction}

| Kind | What | Covers |
| --- | --- | --- |
| `bun run ci` | `hygiene` | the substrate package only — that `@frontierjs/toolbelt` declares no dependency. Nothing grades `Litestone ← Junction ← Sierra` itself |

A gap, written down: from invariants.snapshot.md.

The run this page was written from was not green.

The numbers above come from a full CI run on the release commit 3e4f6c4b2. Every phase passed except the test suites, where 26,143 tests passed and 3 failed: two of Litestone's CLI smoke tests and one Sierra test of search timing. CI named all three by test name, and this page will say so until a green run replaces it.

Run it yourself

Clone the repo and every claim on this page is one command away.

$ bun run ci          ^{# all sixteen phases, the same script GitHub runs}
$ fli proves --from main   ^{# which drives your change owes}
$ fli done            ^{# whether it is finished}

It is new. Here is what runs before it ships.