System Hardening — 2026-05-09

Source review: projects/team-management/reviews/2026-05-09-system-review.md

Mission#

Eliminate silent failures and stuck processes. Make the crew self-healing. Lessons must be captured and the system must globally improve. This project is the execution arm for the 2026-05-09 review.

Core Principles (non-negotiable)#

1. Never fail silently — every error must produce a signal (alert, log entry, notification, blocker)
2. Stuck processes auto-recover — no task sits in in_progress longer than its expected runtime without intervention
3. Lessons are captured globally — single agent's lesson becomes crew-wide pattern through the reflection cycle
4. Process adoption is verified, not assumed — every new process gets an adoption-verification check
5. No single point of orchestration failure — health checks run independently of any one agent's session

Tracks#

Track 1: Critical Fixes (P0)#

• C1 — Task heartbeat + auto-recovery for stuck runs
• C2 — Verify Anthropic auth works
• C3 — Migrate memory embeddings to local nomic-embed

Track 2: Self-Healing Infrastructure (P1)#

• W2 — Task notification (event-driven or polling)
• W3 — Decouple health checks from heartbeat
• W4 — Verify reflection cycle is actually running
• Q1 — Wire silent-exit-0 detector into auto-recovery
• New: Cron job health monitor
• New: Process adoption verifier
• New: Run alive-pings (CLI heartbeat)

Track 3: Documentation & Cleanup (P2)#

• W1 — Create wiki/
• W5 — Cron job registry
• Q2 — Archive ghost projects

Track 4: Architecture (Roadmap)#

• R1 — Agent Supervisor (medium-effort path)

Success Criteria#

• No task can sit in in_progress >1h with a dead process undetected
• Anthropic models confirmed working in real API calls
• Memory search functional again (without Gemini)
• Reflection cycle confirmed running nightly with verifiable output
• All cron jobs registered and health-checked
• Wiki exists and serves as the primary query entry point
• At least one new self-healing pattern in production beyond what's listed