Skip to main content
Each row names one failure, the layer that catches it, what the Runner does, and the test that proves it. This page is an index, not a second explanation. The mechanisms are described where they belong: Security, Workflow and Operations. What this page adds is the last column. tests/test_docs_safety_matrix.py collects every test id named there, so a renamed or deleted test fails the suite instead of leaving a claim nobody can check.

How to read it

  • Test that proves it is a pytest id. Run one with python3 -m pytest "<id>".
  • A proof is what the named test asserts, nothing wider. Most tests drive the Runner against fakes or a local Git repository; they do not prove a live deployment, real GitHub credentials or your repository’s branch rules.
  • (no test) means the behaviour has no test that proves it. Every such row is listed again under Known gaps.

The matrix

Known gaps

These are limits the maintainers live with today. Listing them here is not a promise to close them.
  • An agent session pushing the protected branch is not stopped by the Runner — (no test). The implementer and reviewer run with the machine’s Git credentials. The instruction not to merge or push the base branch is a prompt contract; the enforcing layer is GitHub branch protection or a ruleset on your repository. Without one, nothing in Orbi prevents it.
  • A release does not go through the review gate. A release task pushes its version commit directly to the base branch under its own gates (see Workflow). The rows above describe development deliveries only.
  • A human-decision verdict needs a maintainer. When the reviewer returns blocked_on_human_decision, or the round budget is exhausted, the Runner stops and does not merge. Someone has to decide, then merge by hand, relabel the Issue ai-fix-needed to hand the same PR back with a fresh round budget, or dispatch it again. The stop is proven by tests/test_review_merge.py::test_review_and_merge_human_decision_stops_without_fix_round; nothing automates the decision (the case behind Issue #1099 and PR #1100).
  • The /slots probe is opt-in. With model_wait_probe_url unset, a swallowed request is only caught by the longer model_wait_dead threshold.
  • allow_stale_runner turns the freshness gate into a warning. It is a deliberate host-side override, and a deployment that sets it has given up the first row.
  • Local tests are not a live boundary. No row proves real GitHub permissions, a real systemd or launchd host, or a real model endpoint.
Two gaps named when this page was requested have since been closed, and are kept here so the history stays readable:
  • The reviewer used to read the Issue body once, at claim time (Issue #1094). Each review round now re-reads it: tests/test_bootstrap_runner_prompt.py::test_run_review_refreshes_issue_body_each_round.
  • A resume comment used to promise that a missing worktree would be rebuilt while the code never rebuilt it (Issue #807). The worktree is now recreated from the remote branch: tests/test_resume_pr.py::test_verify_resumed_pr_recreates_missing_worktree_from_remote_branch. The general point stands: a comment that reports a recovery is not proof the recovery happened.