Skip to main content
Orbi gates deliveries with the full test suite, tiered coverage requirements, and the repository’s remote CI checks. The repository contract (see AGENTS.md) is: the full pytest suite with the tiered coverage gate (Issue #234) for the Python code. The same contract runs locally on the Runner machine and remotely on GitHub.

​
The tiered coverage policy

Code below 95% is only allowed for non-core, clearly explainable legacy/defensive branches; every such exception must stay locatable in the coverage report. Unjustified # pragma: no cover is a contract violation (a static test pins the few justified sites: the __main__ entry guards and one defensive except guard in a test).

​
Local contract commands

Run them from the repository root (production interpreter /usr/bin/python3, Python 3.14). COVERAGE_FILE (the official coverage.py environment variable) points the coverage data into the excluded run dir .orbi/ (Issue #302): the worktree root never receives a coverage artifact, so the dirty-worktree gate never trips over one:
How a contributor runs the tests locally (a virtual environment, a wrapper script, whatever) is the contributor’s private business — it is NOT a repository contract, and the repository maintains no local test chain (Issue #569). The report shows both real numbers (line and branch) per file. The two gate scripts are the gate, not a report: tools/coverage_gate.py exits non-zero when the line tier OR the branch tier is below 95% (it reads the coverage JSON totals percent_statements_covered and percent_branches_covered separately — the JSON field percent_covered is the merged line+branch percentage and is never the gate), and tools/diff_coverage_gate.py exits non-zero when a Python line or branch changed since the base ref is not 100% covered.

​
Remote CI (GitHub Actions)

.github/workflows/ci.yml runs the same contract on every pull_request and every push to main: one job, Python 3.14 pinned via actions/setup-python (GitHub-hosted runners do not have the production interpreter at the production path, so the workflow pins the same minor version and runs the identical commands through python3 on PATH), requirements.txt installed, then the contract commands above — the report, tools/coverage_gate.py, and tools/diff_coverage_gate.py origin/main (the changed-code tier against the PR base; a doc-only PR has no changed Python and passes). No lint, no matrix, no cache. The workflow declares its own environment — the CI toolchain belongs to ci.yml, not to the repository (Issue #569). The checkout keeps the default shallow clone and adds fetch-tags: true, then explicitly fetches the base ref and +refs/tags/*:refs/tags/* with a bounded command. It also unshallows the history for the diff-coverage base comparison; the deleted one-time reconciliation tests do not require that checkout setting. The release docs tests (tests/test_docs_releases.py) verify every released tag’s docs page against the real tag objects with git rev-parse / git cat-file, which the default --no-tags fetch would hide (Issue #126). The full-history checkout (fetch-depth: 0) that the removed release reconciliation tests once needed is gone (Issue #910). The existing tags are only made visible in CI — never moved, overwritten, or rewritten. The same single job also runs the Mintlify docs build smoke (Issue #116): the official mint CLI’s mint validate (strict build validation of the docs/ site — config, pages, links — non-zero exit on any warning or error).

​
macOS compatibility workflow (hosted compatibility boundary)

.github/workflows/ci-macos.yml adds a separate, serial macOS job on every pull_request, every push to main, and manual dispatch: one job on the pinned macos-15 runner image (never the moving macos-latest label), Python 3.14 via actions/setup-python, the workflow’s own test requirements, and a real runner probe (sw_vers, the Python version, launchctl, and launchctl print gui/$(id -u) — the user’s launchd GUI domain is the runner-level prerequisite every launchctl operation depends on). The job runs the full pytest suite as one bounded command with the real exit code, builds the package and smoke-installs the wheel from a neutral directory (orbi --version, orbi --help, the site-packages import source), and exercises the installer’s Darwin branch without a real installation: bash -n install.sh, plutil -lint on the launchd plist template, and a PATH-stubbed entry run in a temp ORBI_HOME where the real macOS uname/launchctl pass the scheduler gate and a stubbed git stops the script at the clone step. The hosted macOS image ships no GNU coreutils timeout (the same limitation install.sh handles, Issue #868), so the workflow provides a one-line timeout shim backed by perl’s alarm, keeping every command in the documented timeout <seconds> <cmd> form. The job never mutates the runner’s own launchd deployment (probe reads only, no enable/bootstrap). The boundary is deliberate and stays honest: hosted macOS CI verifies the macOS code paths (tests, packaging, installer logic) and the runner-level launchd prerequisite — it cannot prove a real user’s logged-in GUI session, credentials, model provider, or the full GitHub Issue-to-merge journey, and it does not replace a real macOS launchd smoke run on real Mac hardware. The same honesty applies inside the suite: the deployment, setup, doctor and health-check contracts are pinned to the systemd implementation through the systemd_scheduler test fixture (the launchd implementation keeps its own fakes-based tests), and the real-subprocess idle-recovery scenes (/proc-based descendant discovery) plus the runner-subprocess e2e scenes skip on Darwin with named reasons — the Ubuntu workflow above keeps every one of them authoritative. The coverage and changed-code gates stay authoritative in the Ubuntu workflow above; the macOS job still fails on any test, package, or installer compatibility failure — nothing is swallowed or converted into a pass. A PR is not mergeable while CI is red — the Runner’s merge gate also requires a mergeable PR.

​
What the tests cover

  • Behavioral tests for the runner, the CLI, slots, progress publishing, activity streaming and the resume/review/merge logic (the majority of the suite, including end-to-end tests against real git and gh fixtures).
  • Static contract tests that pin the repository to its own docs and files: AGENTS.md contract items, the label set, the systemd units (5-minute schedule, preflight), the CI workflow, the LICENSE, and the documentation site under docs/ (this page’s claims are enforced by tests/test_docs_site.py; the Chinese pages and the EN/ZH parity by tests/test_docs_i18n.py; the Mermaid diagrams by tests/test_docs_mermaid.py).
When you change code, update or add tests in the same PR and keep the coverage gate green.