> ## Documentation Index > Fetch the complete documentation index at: https://docs.orbi.build/llms.txt > Use this file to discover all available pages before exploring further. # Operations # Operations Normal operation is fully automatic: the timer triggers a tick, the tick does at most one thing, and progress is published to the journal and GitHub by itself. You never need a status command, polling, or supervision in the normal path — the commands below are for first verification, troubleshooting and recovery. ## The timers The timer template `systemd/muyan-pilot@.timer` is deployed as TWO instances, `muyan-pilot@1.timer` and `muyan-pilot@2.timer`. Each fires every 5 minutes, 24 hours a day (`OnCalendar=*-*-* *:00/5`, `AccuracySec=30s`, `Persistent=false` — a missed tick is dropped, never queued) and starts its OWN service instance (`muyan-pilot@1.timer` → `muyan-pilot@1.service`, `muyan-pilot@2.timer` → `muyan-pilot@2.service`), so two independent Runner instances can run concurrently. Each tick's service instance: 1. **Fast-forwards the code first** (`ExecStartPre`, outside the Python process): `git fetch origin main && git merge --ff-only origin/main`. A dirty checkout, a failed fetch or a non-fast-forwardable state fails the preflight: the service does not start and the reason lands in the systemd journal (fail fast). A currently running long task is never hot-updated or killed — while one service instance is active, systemd ignores further starts of THAT instance, and the next real start picks up the latest code. Because two instances may run `ExecStartPre` in the same tick, the fetch + fast-forward is wrapped in a short-lived `flock` on the shared state-dir lock file (`base-sync.lock`, the Python-side sync takes the SAME lock): the main worktree is never written concurrently. Since Issue #171 EVERY fetch that updates the shared remote-tracking ref runs under that same lock: the Runner-side `freeze_base` / `verify_pr` / `merge_gate` / `confirm_merged` fetches go through `fetch_base_ref` (acquire lock → fetch → release, fail fast on lock or fetch errors), and the implement/review prompts instruct Pi to run the base-freshness fetch as `flock git fetch origin ` (the lock path is rendered into both prompts from the configured `repo_dir`). Two concurrent Runners — or a Runner and a Pi session — therefore never race the shared ref (no `cannot lock ref ... is at but expected ` failures). 2. **Runs one tick**: resume an opened PR (review/fix/merge) or claim one `ai-ready` Issue, then exit. Before any slot or claim the tick also runs the pre-start **git transport check** (Issue #114): the deployment checkout's configured `origin` remote must be SSH for the first configured source repo and `git ls-remote ` must exit 0 (SSH reachable and authenticated). A broken transport logs the structured `transport_check_failed ... reason=...` line and fails the start — no slot, no claim, no label change, **no HTTPS fallback** (git data operations, including `.github/workflows/*.yml` pushes, always go over SSH; GitHub API operations stay on the `gh` token). ```bash theme={null} systemctl --user list-timers 'muyan-pilot@*.timer' systemctl --user status 'muyan-pilot@*.service' ``` ## After a unit template changes (Issue #131, #142) The unit templates (`systemd/muyan-pilot@.service` and `systemd/muyan-pilot@.timer`) are deployment config, not ordinary code — but a template change needs NO human step after the merge. The next timer trigger's `ExecStartPre` fast-forwards the checkout, and the pre-start `unit_drift` check self-heals: it runs the SAME idempotent install (copy both templates into the user unit directory, daemon-reload, enable the two timer instances — it never starts, stops or restarts a running Runner), re-verifies with the SAME hash check, and the tick continues with one structured `unit_drift auto_synced` line per unit (before and after sha256, deployed commit). There is no per-tick drift loop until human intervention (the #131/#140 scene: every start failed with `unit_drift` until a human ran the sync command). The same install also migrates the pre-#149 non-templated units (`muyan-pilot.service` / `muyan-pilot.timer`) away once: it stops the legacy timer (a timer stop — never the service) and removes the legacy files, so the old single-instance schedule cannot keep firing the old service. `muyan-pilot install-units --config muyan-pilot.toml` stays the manual entry (first setup, or an immediate sync). A drift the self-heal cannot resolve — a failing install step, a missing template, a unit that drifts again before the re-verify — still fails the start (structured `unit_drift` line, non-zero exit, no slot, no claim): the hash check and the fail-fast canary are unchanged. ## Logs (journal) The journal is the local record. Every line of a run starts with the run id prefix `[]`, so one grep reconstructs the full timeline: ```bash theme={null} journalctl --user -u muyan-pilot@1.service -u muyan-pilot@2.service -f journalctl --user -u muyan-pilot@1.service -u muyan-pilot@2.service | grep e07383c2 ``` Stable `key=value` lines you will see: * `run_start` / `run_end` — the full scene (branch, worktree, session file) at start, and the result (PR URL, commit) at the end; * `activity` / `heartbeat` / `model_wait` / `resumed` — live Pi activity while a session runs (phase, last action, elapsed, idle); * `pi_idle` — one WARNING when there is no model/session activity for more than 5 minutes (`PI_IDLE_WARN_SECONDS=300`) and the model is not expected to reply; a slow active model (`model_wait`) never warns; * `pi_idle_term` / `pi_idle_kill` — the idle-stall recovery (Issue \#94): while the session stays stalled the Runner SIGTERMs the Pi descendants that already existed before the idle window (the hung tools — ppid chain + start time from `/proc//stat`, never a name guess; only Pi descendants are ever signaled), then SIGKILLs a target that survived; each line carries run id, pid, cmdline and result (`sent` / `already_dead` / `no_target` / `failed: ...`); * `run_failed` — the full scene plus the reason (`pi_exit_N`, `timeout_...s`, `upstream_dead_stale_...s` when a frozen `model_wait` past `PI_MODEL_WAIT_DEAD_SECONDS` (default 600 s) declares the upstream model dead, or `idle_recovery_stale_...s` when the session stayed idle for `PI_IDLE_RECOVERY_CYCLES` (default 3) consecutive idle windows and the Runner kills the Pi session itself). The idle warning, the idle recovery and the upstream-dead kill are log/health thresholds — they are not the 5-minute schedule and not a business task timeout. The idle recovery never holds the slot forever: a stalled session either resumes (the failure signal reached the model and the first new event resets the recovery state) or the run fails fast through the normal `ai-blocked` path. ## The CLI (`muyan-pilot`) The official entry is the installed `muyan-pilot` console script, as an **editable** `uv tool` install (the official local deployment — the tool env imports `muyan_pilot` from the clone dir, so the `ExecStartPre` checkout sync is picked up by the next CLI process automatically and ordinary source/template changes need no reinstall or upgrade; `uv tool install --force --reinstall --editable --python /usr/bin/python3 ` — see [Getting started](/getting-started)). `muyan-pilot doctor` reports the consistency (`cli_source: clean` or `cli_source: DRIFT` with the structured `cli_source_drift` line: actual import path, expected repo\_dir, the exact editable reinstall command), and `muyan-pilot setup` verifies or reinstalls it. The direct-execution entry of `muyan_pilot.py` stays a development/ compatibility path. ```bash theme={null} # Create an Issue in a configured source repo and label it ai-ready muyan-pilot add "task title" --body "task body" --config muyan-pilot.toml muyan-pilot add "task title" --repo OWNER/BACKLOG-REPO --config muyan-pilot.toml # Read-only queue view: current (ai-in-progress) task with live Pi # activity, the next ready Issue, and the most recent result per source # repo (ai-pr-opened / ai-fix-needed / ai-merged / ai-blocked) muyan-pilot status --config muyan-pilot.toml # Read-only deployment/health report: repo commit, unit drift, CLI # source (cli_source: clean / DRIFT + the structured cli_source_drift # line), git transport (configured origin URL, protocol, expected SSH # URL, SSH probe — a failed transport is reported as `transport: # FAILED ...`, not raised), timer/service instance state, slots, Pi # session, current Issue, recent journal muyan-pilot doctor --config muyan-pilot.toml # One-time, idempotent initialization (new machine / new repo): gh # auth + repo permissions, platform labels, systemd user units, and # the checkout check INCLUDING the git transport — an existing HTTPS # `origin` is migrated to `git@github.com:owner/repo.git` (the # human-authorized migration path; the Runner itself never rewrites a # remote) and the SSH connectivity is probed (fail fast, no HTTPS # fallback) muyan-pilot setup --config muyan-pilot.toml # Print the live Pi session JSONL path (fail fast when no session exists) muyan-pilot session --config muyan-pilot.toml muyan-pilot session --follow --config muyan-pilot.toml # tail -f muyan-pilot session --pretty --config muyan-pilot.toml # one-line summaries ``` All commands accept the config via `--config` or the `MUYAN_PILOT_CONFIG` environment variable (default `muyan-pilot.toml`). `status` and `session` are debug attachments — the journal and GitHub remain the normal observability path. ## Worktrees and base freshness Each claim freezes `origin/` (fetched first) and creates the task worktree and feature branch from that exact SHA — never from the main worktree's current HEAD. Branch and worktree names carry the run id (e.g. `.worktrees/<...>-issue-14-e07383c2`), so a retried Issue gets a new independent run and the old scene is preserved. `.worktrees/` is gitignored. The task worktree shares the deployment checkout's single `origin` remote (a `git worktree add` worktree inherits the main repository's remote configuration), so the git transport is configured once on the checkout and every worktree inherits it: new bootstrap worktrees have an SSH `git remote -v` by construction, and their fetch/push — including `.github/workflows/*.yml` — go over SSH (Issue #114). Before creating the PR, the implementer re-fetches the base: if `origin/` advanced, it merges the latest base into the task branch, resolves conflicts manually, reruns the full test suite, and only then pushes. The Runner verifies with `git merge-base --is-ancestor origin/ HEAD` and rejects a delivery whose head does not contain the latest remote base. ## Failure recovery | State | What happened | What to do | | -------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `ai-blocked` | The Runner failed fast (command error, unrecoverable scene, review exhausted) | Read the Issue comment (scene + reason) and the journal. Fix the environment or the task, then either relabel the same PR path as `ai-fix-needed` (same run continues) or remove the labels and re-dispatch as a fresh `ai-ready` (new run). It is never auto-recovered. | | `ai-fix-needed` | The PR head is not mergeable yet (review finding or base conflict) | Nothing — the next tick starts the next review session on the same PR, which absorbs the latest base in-session. | | Leftover `ai-in-progress` after a kill | The Runner was SIGKILLed mid-task | The next tick's resume scan picks it up (only when no other Runner is alive): same run id, same worktree, same progress comment — no new run. | The run artifacts (plan, test log, session JSONL) stay in the task worktree as the local record; GitHub carries the delivery record. ## Concurrency `max_concurrency` (default 1) bounds the number of concurrent deliveries on the machine. A slot is an exclusive `flock(2)` lock on `/.muyan-pilot/slots/slot-N`, taken before any claim and held for the whole delivery lifecycle (implement → review → merge); the kernel releases it when the process exits, however it exits. A Runner that cannot take a slot logs `capacity_full` and exits without claiming an Issue. The two timer instances may start two Runner instances in the same tick; they compete for the SAME slots — the capacity is `max_concurrency` (with the default 1 the behavior is exactly the pre-#149 one), never the instance count. ## Related topics - [Getting started](/getting-started.md) - [Index](/index.md) - [Security](/security.md) - [Workflow](/workflow.md) - [Setup](/setup.md)