End-to-end MC stability refactor

Problem statement#

MC has been in repeated restart loops despite multiple "fixes". Root cause analysis (2026-05-10):

1. start.sh's stale-build watchdog auto-restarts on BUILD_ID change (every concurrent rebuild = forced restart)
2. start.sh has a 5-attempt retry loop that kills server on verify-build failure → loops with concurrent rebuilds
3. start.sh auto-rebuilds when .next/BUILD_ID is missing → can be triggered by partial wipes
4. Multiple cron jobs and agents trigger rebuilds while server is alive
5. Memory peak hits 3.8G during concurrent build inside service cgroup → systemd kills

The "self-healing" design has become "self-destroying" — every recovery attempt creates more chaos.

Solution: shift from auto-recovery to manual control#

Maxime's directive: MC restarts ONLY when triggered by him via UI.

Architectural changes#

1. Strip auto-restart triggers from start.sh

   • Remove the BUILD_ID watchdog entirely
   • Remove the 5-attempt verify-build retry loop
   • Single-shot startup: verify once, succeed or exit 78 (no systemd auto-restart)
   • Keep only: stale-process cleanup + first-time build (if .next/ missing entirely)

2. Reboot queue system

   • Storage: /home/maxime/.openclaw/workspace/state/reboot-queue.json
   • API: POST /api/reboot-queue (agent submits), GET /api/reboot-queue (list), POST /api/reboot-queue/trigger (Maxime clicks)
   • UI: banner on dashboard showing pending count; dedicated /reboots page with details + trigger button
   • Agent helper: scripts/queue-reboot.sh --task <id> --reason <text> [--rebuild]

3. Restart log tracking

   • record-startup.js writes JSONL to /tmp/mc-restart-log.jsonl
   • /api/restart-log serves history with stats
   • Display on hardware page with crash-loop alert

4. Team protocol update

   • Agents NEVER restart MC themselves (no systemctl restart)
   • Agents ALWAYS queue a reboot request when their work needs server reload
   • Tasks pending reboot are marked pending_reboot (or stay in_progress with a flag) — agent can still pick up next task
   • Reboot queue lives in workspace state, persists across MC restarts

5. Tests

   • Unit tests for record-startup, restart-log API, reboot-queue API, queue helper script
   • Integration test for start.sh single-shot behavior
   • Regression test for "concurrent build doesn't crash server"

Files to change/create#

New files:

• app/api/reboot-queue/route.ts
• app/api/reboot-queue/trigger/route.ts
• app/api/reboot-queue/[id]/route.ts
• lib/reboot-queue.ts
• app/reboots/page.tsx
• components/RebootQueueBanner.tsx
• components/RestartLogDisplay.tsx
• scripts/queue-reboot.sh
• __tests__/reboot-queue.test.ts
• __tests__/restart-log.test.ts
• __tests__/record-startup.test.ts

Modified files:

• start.sh — major refactor (simplify, remove watchdog, remove retry loop)
• app/page.tsx — add reboot queue banner
• app/hardware/page.tsx or components/HardwareClient.tsx — show restart history
• /home/maxime/.openclaw/workspace/AGENTS.md — agent reboot rule
• /home/maxime/.openclaw/workspace/team/CONVENTIONS.md — reboot protocol

Files to consider removing/deprecating:

• The build watchdog logic (currently inside start.sh)
• The auto-rebuild retry tracking (currently BUILD_ATTEMPT_FILE)

Acceptance criteria#

• MC starts cleanly and stays up (no spontaneous restarts for 30+ min)
• Concurrent next build does NOT crash the running server (build lock prevents)
• Reboot queue UI shows pending requests
• Manual reboot via UI works
• Tests pass (existing 696 + new ones)
• start.sh is < 100 lines (currently 250+)
• AGENTS.md and CONVENTIONS.md reflect the new protocol

Notes#

• Restart log already partially built (record-startup.js exists, /api/restart-log/route.ts exists)
• record-startup.js should be enhanced to record reboot trigger source (agent ID, task ID, manual)
• Build lock semantics: any rebuild MUST go through the queue (or via manual user action), never directly while server alive

Completion Note (2026-05-10 13:00 SGT)#

Done end-to-end:

• start.sh refactored (113 lines, no watchdog, no retry loop, exit 78 on hard fail) — committed 48977e9
• lib/reboot-queue.ts — persistent queue with state machine + janitor — 48977e9
• 3 API routes: GET/POST /api/reboot-queue, GET/DELETE /api/reboot-queue/[id], POST /api/reboot-queue/trigger — 48977e9
• scripts/execute-reboot.sh — runs in transient systemd unit OUTSIDE MC cgroup so rebuild survives SIGTERM — 48977e9
• scripts/queue-reboot.sh — convenience CLI wrapper (already in repo)
• 17 unit tests, all passing — __tests__/reboot-queue.test.ts
• RebootQueuePanel UI on /restarts page — 927c4ab
• AGENTS.md / SOUL.md / team/CONVENTIONS.md §7 — full protocol documented
• Banner injected into all 5 worker AGENTS.md (~/.openclaw/workspaces/<id>/) and 3 of them in this repo
• 4 follow-up tasks created: t-3391 (Kelly coherence), t-3392 (Chris CI guard), t-3393 (Debbie janitor cron), t-3394 (Vera review)

Verified:

• Build passed (8.73 kB on /restarts page)
• MC running on new start.sh + new build (no watchdog, no auto-restart loop)
• E2E test: created request → listed → cancelled successfully via API
• 45 unit tests passing (reboot-queue + hardware)

Known issue surfaced during work: Another openclaw agent was running git stash && npx next build concurrently at 12:36, which has been the smoking gun cause of multiple restart loops. Killed that process. The new banner + CI guard task (t-3392) prevent this going forward.