Graphs
All graphs run with "engine": "graph". Steps act once; waits hold until polled ready. Each node declares the queue label an issue resting there carries and whether it holds running. Edges are listed from → to [condition]; an edge without a condition is the fallback.
The tick
Section titled “The tick”| Step | Graph engine |
|---|---|
| Read | One GraphQL query per repo: queued issues, agent PRs with checks, merge state, review decision, newest 20 comments per issue. |
| Reconcile | Polls worker-running / triaging / diagnosing for each tracked worker; kills workers over their cap; tears down worktrees. |
| Watch | One PR graph turn per eligible PR; then awaiting-promotion tracking. |
| Sweep | A running lease with this host (or none) and no tracked worker goes to its node’s recovery. |
| Schedule | Starts Task workers by priority then age up to phases.task.cap; wakes replied-to issues; starts triage up to phases.dispatch.cap. |
| Release | Release graphs per repo. |
| Upkeep | Reaper, re-triage, then the rescue sweep. |
| Heartbeat | Writes the state file. |
Turns a q:task issue into a published PR and ends at submit. Entry: q:task → start-worker.
| Node | Kind | Queue | running |
|---|---|---|---|
start-worker |
step | q:task |
no |
worker-running |
wait | q:task |
yes |
submit |
step | none | no |
requeue |
step | q:task |
no |
to-human |
step | q:human |
no |
start-worker → worker-running [started] | to-human [budgetSpent] | requeueworker-running → end [issueGone] | submit [submitted] | requeuesubmit → endrequeue → start-worker [attemptsLeft] | to-humanto-human → endsubmit: historysubmitted,submittedAt, the PR on the marker, the issue out of every queue,Closes #nadded to a PR found only by its branch.- A worker exiting without a PR, or killed at
wallClockMinutes, goes torequeue. Spent attempts:q:human+why:budget. - Turns stop at a wait that is not ready, on entering a wait, on returning to
start-worker, or at the end.
One turn per tick for each agent PR (head agent/*) closing a cambium-owned issue, when the PR is merged and the issue open, or the PR is open, CI has settled and the issue has no Task queued or running. No waits.
| Node | Action |
|---|---|
assess |
Picks the next node (order below). |
merge |
Behind base: graft pr update (branch-updated). Else graft pr merge --auto (merge-armed). A refusal is recorded (merge-refused) and retried on a new head or after 10 min; a human-only refusal goes to to-human (why:approve). |
rework |
Back to the task graph at start-worker in q:task: attempt 1 of a fresh maxAttempts, reworks + 1, reason as lastFailure, head as failedSha. Past maxReworks: to-human (why:budget). |
close-pr |
Closes the PR with the reason and Closed by cambium. |
close-out |
Marker done, out of every queue, issue closed (Merged in #n. Closed by cambium.), branch deleted. |
to-human |
q:human with the why: label and the reason. |
assess order:
- Merged →
close-out. - Issue closed →
close-pr. - A newer open agent PR closes the same issue →
close-pr(superseded). - Red CI,
mergeable: CONFLICTINGorCHANGES_REQUESTED→rework. - Green, clean, repo
mergeis"green", nothing armed, no refusal on this head within 10 min →merge. - Not merged
mergeQueueMinutesafter submission →to-human(why:approve). - Otherwise nothing this tick.
assess → close-out [merged] | merge [toMerge] | rework [toRework] | close-pr [toClose] | to-human [toHumanInbox] | endmerge → to-human [needsApproval] | endrework → to-human [reworksSpent] | endFor an issue in q:human, the PR graph only merges a green, clean PR (not on a head a human-only refusal sent there). It never reworks or closes one.
dispatch
Section titled “dispatch”Triages a q:intake issue. Requires phases.dispatch.cap > 0. Entry: q:intake → start-triage.
| Node | Kind | Queue | running |
|---|---|---|---|
start-triage |
step | q:intake |
no |
triaging |
wait | q:intake |
yes |
retry-triage |
step | q:intake |
no |
to-task |
step | q:task |
no |
to-human |
step | q:human |
no |
close-duplicate |
step | none | no |
deferred |
step | q:intake |
no |
blocked |
step | q:intake |
no |
close-obsolete |
step | none | no |
start-triage → triaging [started] | to-human [budgetSpent] | retry-triagetriaging → end [issueGone] | retry-triage [leaseLost] | to-task [actionable] | close-duplicate [duplicate] | deferred [defer] | blocked [blocked] | close-obsolete [obsolete] | to-human [needsHuman] | to-humanretry-triage → start-triage [attemptsLeft] | to-humandeferred → to-human [budgetSpent] | endblocked → deferred [noneOpen] | endothers → endTriage verdict
Section titled “Triage verdict”The worker’s final message ends with a JSON block (the last fenced block of the last result event, else bare JSON):
{ verdict: 'actionable' | 'needs-human' | 'duplicate' | 'obsolete' | 'defer' | 'blocked', duplicateOf?: number, // required for duplicate; not the issue itself blockedBy?: number[], // required for blocked recheckHours?: number, reason: string, title?: string, acceptance?: string[], ask?: { // for needs-human kind: 'decide' | 'verify' | 'chore' | 'approve', question: string, options?: { label: string, detail: string }[], // two or more; required for decide recommendation?: string, interactive: boolean // a live session with the code would help }}| Verdict | Outcome |
|---|---|
actionable |
Comment with reason, title and acceptance checks; q:task at start-worker, attempt 1. The issue body is never rewritten. |
needs-human |
q:human + why:triage. With an ask: its ask:* labels too, and the ask rendered in the state comment. An invalid ask is dropped (the reason says why); the issue still lands in q:human. |
duplicate |
Closed as duplicate with Duplicate of #n (<link>). <reason>. |
obsolete |
Closed as not_planned with Obsolete: <reason>. |
defer |
Stays in q:intake at deferred with wait:deferred, until deferUntil (recheckHours, else deferHours). After maxDefers: q:human + why:triage. |
blocked |
Adds missing GitHub “blocked by” links; stays in q:intake at blocked with wait:blocked until every blocker closes. None open: treated as defer. |
| invalid/none | q:human + why:triage with the error. Also a worker killed at wallClockMinutes. |
- A deferred issue whose blockers all closed (one after deferral) is triaged at the next tick.
- A blocked wait does not count toward
maxDefers. - A human reply wakes a parked issue, and a
q:humanissue with anywhy:*exceptwhy:approve, back tostart-triageat attempt 1. The newest 5 replies (2000 characters each) are quoted to triage. - A human reply: a comment after the issue entered its state, whose author is not
graft.botLogin, not the state comment’s author, and not aBotaccount. - Closed or relabelled during triage: verdict dropped.
askis read only onneeds-human. An issue waiting on another open issue that is itself a human chore or check isblockedby it, notneeds-human.
release-prepare
Section titled “release-prepare”Subject: the promotion PR (release/<date>-N → release.to). Clock entry: daily at release.time, or cambium run release-prepare.
start → cleanup → prepare → end| Node | Action |
|---|---|
cleanup |
Closes, with a comment, open promotion PRs older than 24 h; removes q:promote. |
prepare |
from ahead of to: cuts release/<date>-N at the newest from commit with a green release.check, opens its PR listing the PRs it carries, labels it q:promote, writes the marker, then closes the older open promotion with “Superseded by”. Same cut already open: refreshes it. No green commit, or to holds content the cut lacks: a log line, nothing opened. |
release-promote
Section titled “release-promote”Entry: q:promote → merge, once the promotion PR is approved.
| Node | Kind | Action |
|---|---|---|
merge |
step | graft pr merge <n> --auto. A refusal is recorded and not retried on the same head. |
merging |
wait | Holds while the PR is open. |
promoted |
step | Merged: marker done, q:promote off. |
abandoned |
step | Closed unmerged: marker done, q:promote off. |
merge → merging [armed] | endmerging → promoted [merged] | abandonedNeither release graph has a timeout.
Awaiting promotion
Section titled “Awaiting promotion”For repos where release.from ≠ release.to (enabled or not):
- A merged agent PR not yet in
toputsq:promoteon each closed, cambium-owned issue it closes. - A closed
q:promoteissue whose PR is no longer ahead oftoloses the label. Skipped whiletois missing orfromis more than 100 commits ahead.
rescue
Section titled “rescue”Requires phases.rescue.cap > 0. Sweep every rescue.everyMinutes.
| Node | Kind | Action |
|---|---|---|
start-rescue |
step | Starts a read-only rescue worker. |
diagnosing |
wait | Holds while the worker runs. |
unstick |
step | Carries out the verdict’s actions. |
file-causes |
step | Files or updates root-cause tickets. |
report |
step | Comments on the item. |
to-human |
step | q:human + why:rescue. |
start-rescue → diagnosing [started] | to-human [budgetSpent] | enddiagnosing → end [resolved] | unstick [diagnosed] | to-humanunstick → file-causes → report → endStuck symptoms
Section titled “Stuck symptoms”| Symptom | Condition |
|---|---|
many-queues |
More than one q:* label. |
running-unqueued |
running outside q:task / q:intake (not another host’s lease). |
why-outside-human |
A why:* label without q:human. |
wait-outside-intake |
A wait:* label without q:intake. |
closed-labelled |
Closed issue still carrying cambium labels (q:promote aside). |
merged-open |
Agent PR merged, issue open in no queue. |
stranded |
Open in no queue, no open agent PR; its PR closed unmerged. |
stale-task |
In q:task without a lease for rescue.staleHours while a slot was free and the base not red. |
stale-intake |
Same in q:intake, triage slot free, not parked. |
pr-stale |
Open agent PR not updated for rescue.staleHours, its issue open in no queue. |
- A label change counts after
rescue.graceMinutes. - Items held by a live worker or moved this tick are skipped.
- Rescued within
cooldownHours: skipped. RescuedmaxRescuestimes in a week: straight toq:human+why:rescue.
Rescue verdict
Section titled “Rescue verdict”{ "stuck": true, "how": "the sequence of events, with times and evidence", "actions": [{ "do": "clear-labels", "labels": ["why:budget"] }, { "do": "requeue", "queue": "task", "reason": "…" }], "rootCauses": [{ "title": "…", "body": "…", "fingerprint": "kebab-case-cause", "repo": "owner/name" }]}| Action | Effect |
|---|---|
clear-labels |
Removes cambium labels. |
close-pr |
Closes an open agent PR. |
requeue |
To q:task or q:intake at the graph’s entry, attempt 1, outcome rescued. |
adopt-pr |
The task graph’s submit. |
close-issue |
completed or not_planned. |
to-human |
q:human + why:rescue. An optional ask (same shape as triage’s) adds its ask:* labels and is rendered in the state comment. |
clear-labels and close-pr run first; at most one move (requeue, adopt-pr, close-issue, to-human) runs last. "stuck": false takes no actions. An invalid verdict or a killed worker sends the item to q:human + why:rescue. If no starting symptom still holds when the worker exits, the verdict is dropped.
Root-cause tickets are labelled q:intake and rescue, filed in repo or rescue.causeRepo, and carry <!-- cambium:rescue-cause <fingerprint> -->. An open ticket with the same fingerprint gets a comment instead.
Upkeep
Section titled “Upkeep”Reaper
Section titled “Reaper”Every reaper.everyMinutes, for each open untracked agent/* PR idle reaper.staleMinutes. Its issue is the one it closes, else the one its branch names.
| Found | Action |
|---|---|
Issue open, no live worker, marker’s pr is not this PR |
Adopt: Closes #n added if missing, the task graph’s submit, a comment. |
| Issue closed | Close the PR with a comment. |
| No issue | Close after reaper.orphanDays untouched. The branch stays. |
| Live worker on the issue, or the issue tracks this PR | Leave it. |
Re-triage
Section titled “Re-triage”Daily at retriage.time when retriage.enabled. Every q:human issue returns to q:intake without its why:*, except why:approve, issues holding running, and issues whose triage inputs are unchanged.
When triage answers needs-human, the marker keeps a fingerprint of what triage saw:
- title and body
- labels, except
q:*,why:*,wait:*andrunning - human comments (not
graft.botLogin, not a bot) - GitHub “blocked by” issues and their states
- agent PRs naming the issue and their states
- the triage rubric version
| Fingerprint | Re-triage |
|---|---|
Matches, taken less than retriage.maxSkipDays ago |
Skipped: no worker, no label moves. |
| Differs (an input changed, or a new rubric) | Yes. |
Matches, taken retriage.maxSkipDays or more ago |
Yes. |
None (another route to q:human, or written before fingerprints) |
Yes. |
The log reports re-triage: k of n changed, with the moved and skipped counts.
Deferred rechecks are not gated: a defer waits on something outside the issue (a CD run, a release, a deploy), which the fingerprint cannot see.
Red base
Section titled “Red base”Per repo with a requiredCheck. The newest of the base’s last 20 commits with a verdict decides:
| State | Effect |
|---|---|
green |
None. |
red |
No Task workers start except the repair. Failures requeue at the same attempt (base-red). A repair issue is opened (red-base, q:task, first priority label). |
unknown |
No completed run in the window. Gates nothing. |
Green again: a repair issue with no open PR and no worker is closed.