Operations

Normal operation is fully automatic: the timer triggers a tick, the tick does at most one thing, and progress is published to the journal and GitHub by itself. You never need a status command, polling, or supervision in the normal path — the commands below are for first verification, troubleshooting and recovery.

The timers

The timer template systemd/[email protected] is deployed as TWO instances, [email protected] and [email protected]. Each fires every 5 minutes, 24 hours a day (OnCalendar=*-*-* *:00/5, AccuracySec=30s, Persistent=false — a missed tick is dropped, never queued) and starts its OWN service instance ([email protected] → [email protected], [email protected] → [email protected]), so two independent Runner instances can run concurrently. Each tick’s service instance:

Fast-forwards the code first (ExecStartPre, outside the Python process): git fetch origin main && git merge --ff-only origin/main. A dirty checkout, a failed fetch or a non-fast-forwardable state fails the preflight: the service does not start and the reason lands in the systemd journal (fail fast). A currently running long task is never hot-updated or killed — while one service instance is active, systemd ignores further starts of THAT instance, and the next real start picks up the latest code. Because two instances may run ExecStartPre in the same tick, the fetch + fast-forward is wrapped in a short-lived flock on the shared state-dir lock file (base-sync.lock, the Python-side sync takes the SAME lock): the main worktree is never written concurrently. Since Issue #171 EVERY fetch that updates the shared remote-tracking ref runs under that same lock: the Runner-side freeze_base / verify_pr / merge_gate / confirm_merged fetches go through fetch_base_ref (acquire lock → fetch → release, fail fast on lock or fetch errors), and the implement/review prompts instruct Pi to run the base-freshness fetch as flock <BASE_SYNC_LOCK> git fetch origin <base_branch> (the lock path is rendered into both prompts from the configured repo_dir). Two concurrent Runners — or a Runner and a Pi session — therefore never race the shared ref (no cannot lock ref ... is at <X> but expected <Y> failures).
Runs one tick: resume an opened PR (review/fix/merge) or claim one ai-ready Issue, then exit. Before any slot or claim the tick also runs the pre-start git transport check (Issue #114): the deployment checkout’s configured origin remote must be SSH for the first configured source repo and git ls-remote <ssh-url> must exit 0 (SSH reachable and authenticated). A broken transport logs the structured transport_check_failed ... reason=... line and fails the start — no slot, no claim, no label change, no HTTPS fallback (git data operations, including .github/workflows/*.yml pushes, always go over SSH; GitHub API operations stay on the gh token).

systemctl --user list-timers 'muyan-pilot@*.timer'
systemctl --user status 'muyan-pilot@*.service'

After a unit template changes (Issue #131, #142)

The unit templates (systemd/[email protected] and systemd/[email protected]) are deployment config, not ordinary code — but a template change needs NO human step after the merge. The next timer trigger’s ExecStartPre fast-forwards the checkout, and the pre-start unit_drift check self-heals: it runs the SAME idempotent install (copy both templates into the user unit directory, daemon-reload, enable the two timer instances — it never starts, stops or restarts a running Runner), re-verifies with the SAME hash check, and the tick continues with one structured unit_drift auto_synced line per unit (before and after sha256, deployed commit). There is no per-tick drift loop until human intervention (the #131/#140 scene: every start failed with unit_drift until a human ran the sync command). The same install also migrates the pre-#149 non-templated units (muyan-pilot.service / muyan-pilot.timer) away once: it stops the legacy timer (a timer stop — never the service) and removes the legacy files, so the old single-instance schedule cannot keep firing the old service. muyan-pilot install-units --config muyan-pilot.toml stays the manual entry (first setup, or an immediate sync). A drift the self-heal cannot resolve — a failing install step, a missing template, a unit that drifts again before the re-verify — still fails the start (structured unit_drift line, non-zero exit, no slot, no claim): the hash check and the fail-fast canary are unchanged.

Logs (journal)

The journal is the local record. Every line of a run starts with the run id prefix [<run_id>], so one grep reconstructs the full timeline:

journalctl --user -u [email protected] -u [email protected] -f
journalctl --user -u [email protected] -u [email protected] | grep e07383c2

Stable key=value lines you will see:

run_start / run_end — the full scene (branch, worktree, session file) at start, and the result (PR URL, commit) at the end;
activity / heartbeat / model_wait / resumed — live Pi activity while a session runs (phase, last action, elapsed, idle);
pi_idle — one WARNING when there is no model/session activity for more than 5 minutes (PI_IDLE_WARN_SECONDS=300) and the model is not expected to reply; a slow active model (model_wait) never warns;
pi_idle_term / pi_idle_kill — the idle-stall recovery (Issue #94): while the session stays stalled the Runner SIGTERMs the Pi descendants that already existed before the idle window (the hung tools — ppid chain + start time from /proc/<pid>/stat, never a name guess; only Pi descendants are ever signaled), then SIGKILLs a target that survived; each line carries run id, pid, cmdline and result (sent / already_dead / no_target / failed: ...);
run_failed — the full scene plus the reason (pi_exit_N, timeout_...s, upstream_dead_stale_...s when a frozen model_wait past PI_MODEL_WAIT_DEAD_SECONDS (default 600 s) declares the upstream model dead, or idle_recovery_stale_...s when the session stayed idle for PI_IDLE_RECOVERY_CYCLES (default 3) consecutive idle windows and the Runner kills the Pi session itself).

The idle warning, the idle recovery and the upstream-dead kill are log/health thresholds — they are not the 5-minute schedule and not a business task timeout. The idle recovery never holds the slot forever: a stalled session either resumes (the failure signal reached the model and the first new event resets the recovery state) or the run fails fast through the normal ai-blocked path.

The CLI (`muyan-pilot`)

The official entry is the installed muyan-pilot console script, as an editable uv tool install (the official local deployment — the tool env imports muyan_pilot from the clone dir, so the ExecStartPre checkout sync is picked up by the next CLI process automatically and ordinary source/template changes need no reinstall or upgrade; uv tool install --force --reinstall --editable --python /usr/bin/python3 <clone dir> — see Getting started). muyan-pilot doctor reports the consistency (cli_source: clean or cli_source: DRIFT with the structured cli_source_drift line: actual import path, expected repo_dir, the exact editable reinstall command), and muyan-pilot setup verifies or reinstalls it. The direct-execution entry of muyan_pilot.py stays a development/ compatibility path.

# Create an Issue in a configured source repo and label it ai-ready
muyan-pilot add "task title" --body "task body" --config muyan-pilot.toml
muyan-pilot add "task title" --repo OWNER/BACKLOG-REPO --config muyan-pilot.toml

# Read-only queue view: current (ai-in-progress) task with live Pi
# activity, the next ready Issue, and the most recent result per source
# repo (ai-pr-opened / ai-fix-needed / ai-merged / ai-blocked)
muyan-pilot status --config muyan-pilot.toml

# Read-only deployment/health report: repo commit, unit drift, CLI
# source (cli_source: clean / DRIFT + the structured cli_source_drift
# line), git transport (configured origin URL, protocol, expected SSH
# URL, SSH probe — a failed transport is reported as `transport:
# FAILED ...`, not raised), timer/service instance state, slots, Pi
# session, current Issue, recent journal
muyan-pilot doctor --config muyan-pilot.toml

# One-time, idempotent initialization (new machine / new repo): gh
# auth + repo permissions, platform labels, systemd user units, and
# the checkout check INCLUDING the git transport — an existing HTTPS
# `origin` is migrated to `[email protected]:owner/repo.git` (the
# human-authorized migration path; the Runner itself never rewrites a
# remote) and the SSH connectivity is probed (fail fast, no HTTPS
# fallback)
muyan-pilot setup --config muyan-pilot.toml

# Print the live Pi session JSONL path (fail fast when no session exists)
muyan-pilot session --config muyan-pilot.toml
muyan-pilot session --follow --config muyan-pilot.toml   # tail -f
muyan-pilot session --pretty --config muyan-pilot.toml   # one-line summaries

All commands accept the config via --config or the MUYAN_PILOT_CONFIG environment variable (default muyan-pilot.toml). status and session are debug attachments — the journal and GitHub remain the normal observability path.

Worktrees and base freshness

Each claim freezes origin/<base_branch> (fetched first) and creates the task worktree and feature branch from that exact SHA — never from the main worktree’s current HEAD. Branch and worktree names carry the run id (e.g. .worktrees/<...>-issue-14-e07383c2), so a retried Issue gets a new independent run and the old scene is preserved. .worktrees/ is gitignored. The task worktree shares the deployment checkout’s single origin remote (a git worktree add worktree inherits the main repository’s remote configuration), so the git transport is configured once on the checkout and every worktree inherits it: new bootstrap worktrees have an SSH git remote -v by construction, and their fetch/push — including .github/workflows/*.yml — go over SSH (Issue #114). Before creating the PR, the implementer re-fetches the base: if origin/<base_branch> advanced, it merges the latest base into the task branch, resolves conflicts manually, reruns the full test suite, and only then pushes. The Runner verifies with git merge-base --is-ancestor origin/<base_branch> HEAD and rejects a delivery whose head does not contain the latest remote base.

Failure recovery

State	What happened	What to do
`ai-blocked`	The Runner failed fast (command error, unrecoverable scene, review exhausted)	Read the Issue comment (scene + reason) and the journal. Fix the environment or the task, then either relabel the same PR path as `ai-fix-needed` (same run continues) or remove the labels and re-dispatch as a fresh `ai-ready` (new run). It is never auto-recovered.
`ai-fix-needed`	The PR head is not mergeable yet (review finding or base conflict)	Nothing — the next tick starts the next review session on the same PR, which absorbs the latest base in-session.
Leftover `ai-in-progress` after a kill	The Runner was SIGKILLed mid-task	The next tick’s resume scan picks it up (only when no other Runner is alive): same run id, same worktree, same progress comment — no new run.

The run artifacts (plan, test log, session JSONL) stay in the task worktree as the local record; GitHub carries the delivery record.

Concurrency

max_concurrency (default 1) bounds the number of concurrent deliveries on the machine. A slot is an exclusive flock(2) lock on <repo_dir>/.muyan-pilot/slots/slot-N, taken before any claim and held for the whole delivery lifecycle (implement → review → merge); the kernel releases it when the process exits, however it exits. A Runner that cannot take a slot logs capacity_full and exits without claiming an Issue. The two timer instances may start two Runner instances in the same tick; they compete for the SAME slots — the capacity is max_concurrency (with the default 1 the behavior is exactly the pre-#149 one), never the instance count.

Getting Started

Core Concepts

Guides

Optional Components

Releases

Operations

Operations

The timers

After a unit template changes (Issue #131, #142)

Logs (journal)

The CLI (`muyan-pilot`)

Worktrees and base freshness

Failure recovery

Concurrency

​Operations

​The timers

​After a unit template changes (Issue #131, #142)

​Logs (journal)

​The CLI (muyan-pilot)

​Worktrees and base freshness

​Failure recovery

​Concurrency

Operations

The timers

After a unit template changes (Issue #131, #142)

Logs (journal)

The CLI (`muyan-pilot`)

Worktrees and base freshness

Failure recovery

Concurrency