# MEMORY.md - Long-Term Memory Curated, durable facts. Daily logs live in `memory/YYYY-MM-DD.md`. ## 🔴 HARD RULE: Never hardcode 127.0.0.1 / localhost in ANY code the browser sees **Server-side backend fetches** to internal services (`127.0.0.1:4747`, `localhost:3000`) are fine — the server resolves them. **NEVER** return a hardcoded `127.0.0.1` or `localhost` URL in an API response that the browser will then fetch. When the user accesses the app from another machine on the LAN (e.g., `192.168.4.169:4760`), the browser resolves `127.0.0.1` as **itself**, not the server, causing `ERR_CONNECTION_REFUSED`. **Correct pattern for proxied media (images, files):** - Backend fetches from the internal service (server-side: `fetch('http://127.0.0.1:4747/...')` — OK) - Return a URL on the **same origin** to the frontend: `/api/image/proxy?path=/api/media/images/...` - Add a proxy route that streams the response from the internal service **Incorrect pattern (DO NOT DO THIS):** ```ts // ❌ Browser on another machine can't reach 127.0.0.1 return { url: 'http://127.0.0.1:4747/api/media/images/...' } ``` ## 🔴 HARD RULE: Never hardcode 127.0.0.1 / localhost in project code (backend config) All projects must be network-accessible. Never use `127.0.0.1`, `localhost`, or any local-only address in frontend code, API calls, or configuration. Use: - **Relative URLs** in frontend (`/api/...`) + Next.js rewrites to proxy to backend services - **`0.0.0.0`** for server listen addresses - **Proxied paths** in `next.config.ts` rewrites to route to internal services Reason: these projects are accessed from other computers on the LAN (e.g., `192.168.4.169`). A hardcoded `127.0.0.1` resolves to the client machine, not the server. ## 🔴 HARD RULE: Z-index hierarchy in every app — enforced proactively Sticky/fixed page headers MUST always be the top visible layer except for full-screen modals/lightboxes. Never make Maxime report this bug again. **Mandatory z-index scale (every app, every component):** | Layer | z-index | Examples | |---|---|---| | Full-screen modals/lightboxes | `z-50` | Delete confirm, image lightbox, AI generate modal | | Sticky page header | `z-50` | Top nav bar (highest standard Tailwind z-index that compiles) | | Dropdown menus / toasts | `z-10` max | Action menus, notification toasts | | Card-internal overlays | **NO z-index** | Delete buttons, hover gradients, drag handles inside cards | **Why card overlays get no z-index:** they're absolutely positioned inside `relative` cards — they stay contained automatically. Adding z-index pushes them into the global stacking context where they race against the sticky header. **🔴 Tailwind v4 silent failure warning:** Arbitrary z-index values like `z-[100]` or `z-[999]` are often NOT generated by Tailwind v4's JIT compiler. Always verify with `grep "z-index:" .next/static/css/*.css` after setting a z-index. If the class isn't in the compiled CSS, it silently does nothing. **When building ANY new UI with a sticky/fixed header, check this FIRST.** Grep for `z-` in the app and verify: 1. Header uses `z-50` (highest standard value guaranteed to compile) 2. No content element exceeds `z-10` unless it's a fixed-position modal (z-50, separate stacking context) 3. Card overlays have NO z-index at all **Lesson from 2026-06-04:** Three rounds of debugging — z-10 vs z-10 (cards won), z-40 vs z-50 (dropdown won), z-[100] silently not generated by Tailwind v4. Final fix: z-50 header + z-10 dropdowns + no z-index on card overlays. ## 🔴 HARD RULE: Cron jobs default to `thinking: "off"` (2026-06-08) **Root cause (the morning-whatsapp-summary stall):** when an isolated cron agentTurn runs without an explicit `payload.thinking`, the gateway forces `thinkingLevel=high`. Cheap reasoning models like `deepseek-v4-flash` burn their entire token budget on `reasoning_content` before emitting any visible `content`, then stall mid-stream. The stalled-session detector aborts after ~6 minutes with `EmbeddedAttemptSessionTakeoverError: This operation was aborted`. Two consecutive `morning-whatsapp-summary` failures and the timeouts on `persona-engine-nightly-maintenance` / `validate-model-registry` were all this same bug. **Convention:** - **Every new cron** gets `thinking: "off"` by default (Mission Control's New Job form does this automatically via `createCronJob` in `lib/actions.ts`). - **Existing crons** that did procedural tool work (read file → summarise; POST + parse response; run script + report exit code) were patched to `thinking: "off"`. - **Crons that genuinely benefit from reasoning** (e.g. `nightly-sleep-dream` — the nightly reflection cycle) are set to `thinking: "medium"` explicitly so the choice is visible. - **UI surface:** Calendar page (`/calendar`) shows a `no-think` badge on every row with `thinking=off` and an amber `think:` badge when reasoning is enabled, so any non-default is obvious at a glance. The EditPanel and NewJobForm both have a 6-button thinking selector (off / minimal / low / medium / high / xhigh) with explanatory help text. **Touched files:** - `mission-control/lib/workspace.ts` — extended `CronJob.payload` type with `thinking`, `model`, `timeoutSeconds`. - `mission-control/lib/actions.ts` — `createCronJob` defaults `--thinking off`; `updateCronJob` accepts a `thinking` patch field. - `mission-control/components/CalendarEditPanel.tsx` — thinking selector in edit panel, always sends explicit value on save. - `mission-control/components/CalendarNewJobForm.tsx` — thinking selector in new-job form (defaults to off). - `mission-control/components/CalendarJobList.tsx` — `ThinkingBadge` component on row header + reasoning line in expanded details. **Why this matters beyond the immediate bug:** Reasoning-by-default is the wrong policy for cron-style tool execution. A cron firing `curl POST` doesn't need to plan a 5-step decomposition first. Forcing thinking burns tokens, slows runs, masks errors as timeouts, and — with cheap models that have aggressive `reasoning_content` truncation — stalls completely. The default should be off and reasoning should be an explicit opt-in per job. ## Channels ### Discord (set up 2026-05-07) - **Bot username:** `@openclaw-bot` - **Bot client ID:** `1501734531374387290` - **Server:** `OpenClaw-private` (ID `1501736149562167316`) - **Owner User ID:** `772287003945205820` (max7064) - **Token:** stored in `~/.openclaw/.env` as `DISCORD_BOT_TOKEN`, referenced via `{source: env, provider: default, id: DISCORD_BOT_TOKEN}` in config - **Plugin:** `@openclaw/discord` (installed in `~/.openclaw/npm/node_modules/`) - **Config:** `channels.discord` in `~/.openclaw/openclaw.json` - `groupPolicy: "allowlist"`, single guild allowlisted - `requireMention: false` for the guild → bot responds to every message in any channel of OpenClaw-private without @mention - **Session model:** each guild channel = isolated session (`agent:main:discord:channel:`); DMs share `agent:main:main` (default `dmScope=main`, but Maxime has `dmScope=per-channel-peer` set globally — verify behavior if relevant) - **MEMORY.md does NOT auto-load in guild channels** (only DMs). Use `memory_search`/`memory_get` if context needed in channels. ### WhatsApp - Already configured. Allowlisted: `6580860085`. Self-chat mode on. - **`+6580860085` is Maxime's actual phone number** (current, post-port). The old number `+6582594910` is retired — do not use it. Self-chat mode is on because Maxime's WhatsApp account uses the same number that the bot is paired against. - Canonical session key for direct delivery: `agent:main:whatsapp:direct:+6580860085`. ## Setup gotchas (lessons learned) 1. **Discord plugin is a separate npm package.** Not bundled. Must run `openclaw plugins install @openclaw/discord` BEFORE the channel config will work. Symptom: gateway boots with "7 plugins" and no Discord; warning: "plugin not installed: discord". 2. **Schema validator on `openclaw config patch` is strict** — it requires explicit `groupPolicy`, `debounceMs`, `mediaMaxMb` even if defaults exist at runtime. Easier path for editing channels: edit `~/.openclaw/openclaw.json` directly with Python and run `openclaw config validate`. 3. **`openclaw gateway restart` kills the current session if it's running through the gateway.** Recovery: `openclaw gateway start` from a fresh exec, then poll status. The gateway service auto-recovers in ~5–10s. 4. **Config hot-reload works for some paths, requires restart for others.** `plugins.installs.*` and most `channels.discord.*` changes need a full restart. ## Workspace - Root: `/home/maxime/.openclaw/workspace` - Active config: `~/.openclaw/openclaw.json` - Logs: `/tmp/openclaw/openclaw-YYYY-MM-DD.log` - Service units (systemd user, both auto-start on boot): - `openclaw-gateway.service` — OpenClaw Gateway on `127.0.0.1:18789` - `mission-control.service` — Mission Control Next.js app on `:4747` - WorkingDir: `/home/maxime/projects/mission-control` - ExecStart: `/usr/bin/npm run start` - Restart=always, RestartPreventExitStatus=78, StartLimitBurst=5/60s - Logs: `/tmp/mission-control.log` (append, both stdout+stderr) - **Never `npm start` manually** — always `systemctl --user (start|stop|restart) mission-control.service`. Manual starts steal port 4747 from systemd and the service ends up `failed`. If burst-limited ("Start request repeated too quickly"), run `systemctl --user reset-failed mission-control.service` first. - After rebuilds (`npm run build`), `systemctl --user restart mission-control.service` to pick up new code. ## Inter-session relay (set up 2026-05-07) Full protocol: `team/RELAY-PROTOCOL.md`. Read it on session start. **One agent, many channels:** Robbie-WhatsApp = Robbie-Discord = Robbie-Mission-Control. Same person. Same applies to any other or future agents. ## GitHub (set up 2026-05-07) - **Username:** `Max-src` - **PAT:** stored in `~/.openclaw/.env` as `GITHUB_TOKEN` (scope: `repo`) - **Repos (all private):** - `Max-src/mission-control` — Next.js dashboard app - `Max-src/openclaw-workspace` — agent workspace (memory, tasks, projects, personalities) - **Auto-sync:** t-024 (workspace, every 6h) and t-025 (MC app, daily 2am SGT) pending setup - **Rule:** never request `delete_repo` scope. Never delete any repo. ## Robbie does it all No crew. No delegation. Robbie owns every task. ## Logging MC logs to journald (`journalctl --user -u mission-control.service`). Don't worry about log rotation beyond that. ## User preferences (Maxime) - **WhatsApp formatting:** no markdown. Plain text only. Use dashes or numbers for lists, CAPS or plain labels for emphasis. No bold, no headers, no tables. This applies to Robbie directly AND to anything agents push via cron/relay to WhatsApp. - TZ: Asia/Singapore (GMT+8) - Style: efficient, direct, no fluff. Master's degree, 20yr in IT/software. - Don't over-explain basics. Move fast. Push back when wrong. - Bias toward action; ask only for the one missing decision. ## ⚡ Task-First rule Create a task before starting work. Reference task ID in git commits: `[t-085] feat: ...`. That's it. ## Heartbeat governor (deprecated 2026-05-13) Old dispatch-cron tuning script (`scripts/heartbeat-governor.py`). Crew is mostly archived now; not used. ## Provider/model compat — ONE central layer (HARD RULE, 2026-05-14) Maxime called this out after the Cat 1 benchmark fix: when the Benchmark / Arena / Model Test surfaces a provider quirk (param mapping, header, auth shape, etc.), the fix MUST live in **one centralised compat module per request path**, not inside the single tool that surfaced it. **Two paths exist, each owns its own compat layer:** 1. **Path A** — Agent traffic through the OpenClaw gateway (`127.0.0.1:18789`). Compat lives in OpenClaw upstream (`shared---*.js` → `buildFoundryModelCompat`, `requiresFoundryMaxCompletionTokens`, …). To extend this path, file a PR or issue upstream. Don't try to monkey-patch the gateway locally. 2. **Path B** — Direct provider calls from Arena / Benchmark / Model Test via `lib/model-arena-gateway.ts` → `queryModelDirect`. Compat lives in **`lib/model-compat.ts`** in Mission Control. It exports `buildCompatProfile`, `applyRequestCompat`, `shouldRetryWithDifferentCompat`. **Why Path B exists**: the gateway forces every request through an agent persona (system-prompt taint), which contaminates raw model evaluation. So Arena/Benchmark bypass the gateway and call providers directly — but they MUST honour the same quirks as Path A. **Future workflow when a benchmark uncovers a new quirk**: 1. Reproduce with a single-model Arena call. 2. Open `lib/model-compat.ts`, extend `CompatProfile` (new flag) + `buildCompatProfile` (detection) + `applyRequestCompat` (transform) + the `KNOWN QUIRKS` block at the top. 3. Optionally add a forgiving fallback to `shouldRetryWithDifferentCompat` so the system self-heals before we codify the rule. 4. Build MC, restart the service, re-run the original benchmark. 5. Append a one-line entry to this section of `MEMORY.md`. **What NEVER to do**: - Per-call `if (modelId === "o3") …` patches inside Arena routes, Benchmark runner, or `callOpenAICompatible`. Any such patch will be invisible to the other call sites and break consistency. - Duplicating compat logic is not an issue — everything lives in MC's `lib/model-compat.ts`. **Reference doc**: `mission-control/docs/PROVIDER-INTEGRATION-PLAYBOOK.md` §9 ("Step 7: Provider/Model Compatibility Quirks") has the full workflow. **Codified quirks so far**: - 2026-05-14: OpenAI reasoning models on Azure / GitHub Models (`gpt-5`, `o1`, `o3`, `o4-mini` family except `gpt-5-chat`) require `max_completion_tokens` instead of `max_tokens`. Mirrors OpenClaw's upstream `requiresFoundryMaxCompletionTokens`. ## Building MC — simple rule (2026-05-13) If you need to rebuild any of our apps: stop it first, build, start it. Or just `npm run build` and accept a brief cached-chunk flicker on next page load. No reboot queue, no stability gates, no atomic-swap scripts. Use common sense. If TS check only: `npx tsc --noEmit` (no chunk emission). ## MC watchdog / auth-lockout / 2026-05-11 incidents (archived) See `memory/archive/MEMORY-pre-2026-05-12-archived-sections.md` for full details. The watchdog + reboot-queue + stability-gates stack from those incidents is deprecated; we don't use those mechanisms anymore (see "Stability gates / reboot queue (deprecated)" above). ## Promoted From Short-Term Memory (2026-05-13) - - Auto-refresh every 10s via /api/cron-jobs route - Run history expandable per row - New job form preserved ## Mission Control as systemd service (12:10 SGT) Problem: Robbie's code edits caused "Internal Server Error" — dev server would crash and Maxime had to SSH in to restart manually. Fix: created ~/.config/systemd/user/mission-control.service - Runs `npm run start` (production build, not dev) - Restart=always, RestartSec=5 - Enabled on boot via `loginctl enable-linger maxime` - Port: 4747 - Logs: /tmp/mission-control.log deploy.sh added to /home/maxime/projects/mission-control/ — agents run this after code changes (build + restart service). **Key distinction:** dev server (npm run dev) crashes on errors and has no auto-restart. Production service (npm run start) is stable, auto-restarts within 5s on any crash. ## Pending tasks as of session end - t-003: Chris onboarding (IDENTITY.md + SOUL.md) — backlog - t-014: MC UI real-time updates + run failure badges + dropdown overflow — backlog, Chris - t-026: Git status widget in MC dashboard — backlog, Chris - t-029: Scheduled Jobs run history detail — backlog, Chris - t-030: Blocked/stuck task UX in MC — backlog, Chris (HIGH — prevents silent stuck loops) - t-005: MC runner don't fire on done tasks — backlog (may already be fixed by Chris in earlier run, verify) ## Key file locations - Mission Control app: /home/maxime/projects/mission-control - Mission Control service: ~/.config/systemd/user/mission-control.service - Mission Control logs: /tmp/mission-control.log [score=0.858 recalls=12 avg=0.564 source=memory/2026-05-07.md:153-185] - - [21:45] Task t-034 "Mission Control — authentication (username/password + Google OAuth)" run FAILED (exit 1) by main (run r-t-034-1778161439751) - [21:47] Task t-039 "Register Ollama as an OpenClaw provider, smoke-test from CLI" run completed by main (run r-t-039-1778161215902) - [21:51] Task t-040 "Route one cheap recurring task to local Qwen — measure success rate" run FAILED (exit 1) by debbie (run r-t-040-1778161890827) - [21:54] Task t-040 "Route one cheap recurring task to local Qwen — measure success rate" run completed by debbie (run r-t-040-1778161955442) - [21:57] Task t-047 "Kelly — Discord channel + binding" run completed by main (run r-t-047-1778162104627) - [21:59] Task t-048 "Kelly — daily review skill (diff watcher + doc updater)" run completed by chris (run r-t-048-1778162093581) - [22:01] Task t-047 "Kelly — Discord channel + binding" run completed by main (run r-t-047-1778162331669) - [22:07] Task t-041 "Document local LLM setup in MEMORY.md and a runbook" run completed by main (run r-t-041-1778162611630) - [22:10] Task t-049 "Kelly — schedule daily 03:00 SGT review job" run completed by main (run r-t-049-1778162922562) - [22:28] Task t-051 "Smoke test - Debbie tells Maxime a joke on WhatsApp (via local Qwen)" run completed by debbie (run r-t-051-1778164027527) - [22:36] Task t-051 "Smoke test - Debbie tells Maxime a joke on WhatsApp (via local Qwen)" run completed by debbie (run r-t-051-1778164340267) [score=0.848 recalls=9 avg=0.606 source=memory/2026-05-07.md:222-232] - - t-030: Blocked/stuck task UX in MC — backlog, Chris (HIGH — prevents silent stuck loops) - t-005: MC runner don't fire on done tasks — backlog (may already be fixed by Chris in earlier run, verify) ## Key file locations - Mission Control app: /home/maxime/projects/mission-control - Mission Control service: ~/.config/systemd/user/mission-control.service - Mission Control logs: /tmp/mission-control.log - Agent workspace: /home/maxime/.openclaw/workspace - Relay protocol: team/RELAY-PROTOCOL.md - GitHub repos: Max-src/mission-control, Max-src/openclaw-workspace (both private) - Credentials: ~/.openclaw/.env (GITHUB_TOKEN, GITHUB_USERNAME, DISCORD_BOT_TOKEN) ## Agent crew - Robbie (main) — this agent. WhatsApp + Discord + Mission Control = same person. - Debbie (agent:debbie) — research/writing worker. IDENTITY.md + SOUL.md done. - Chris (agent:chris) — build worker, Mission Control dev. IDENTITY.md + SOUL.md done (t-003 still listed as backlog but likely done given Chris's active work today). - [12:20] Task t-030 "Mission Control — blocked/stuck task UX and agent input workflow" reassigned from chris → main - [12:20] Task t-030 "Mission Control — blocked/stuck task UX and agent input workflow" run completed by main (run r-t-030-1778127604835) - [12:22] Task t-029 "Scheduled Jobs — show payload, delivery, and last run info" run FAILED (exit 1) by chris (run r-t-029-1778127609954) - [12:42] Task t-031 "Mission Control — duplicate task guard and detector" run FAILED (exit 1) by chris (run r-t-031-1778128765861) [score=0.848 recalls=15 avg=0.540 source=memory/2026-05-07.md:178-199] ## 🔴 HARD RULE: Forward ALL fields in API proxy/relay endpoints — never silently drop **Context (2026-06-05):** Storyboards character-locked image generation stopped producing consistent faces. The logs showed references were "attached" but the results looked like different people. Root cause: `mc-router/src/index.ts` (the Express image generation service) only destructured the **singular** `faceReference` / `bodyReference` fields from the request body, but **dropped the plural arrays** `faceReferences` / `bodyReferences`. Storyboards sends ensembles (3 face angles + 2 body refs); the Express endpoint silently discarded them before calling `generateImage()`. The provider only received a single reference (or none), breaking identity consistency. **Rule:** When an API endpoint receives a request and forwards it to another layer (another HTTP service, a provider adapter, a queue), it MUST forward **every field** defined in the upstream contract. If the upstream supports both singular (`faceReference`) and plural (`faceReferences`) forms, forward BOTH. Never destructure only the fields you "think" are used — the caller may depend on the ones you drop. **Prevention:** When adding a new field to an API contract: 1. Update the type/interface at the source 2. Update EVERY relay/proxy/adapter endpoint that passes the request through 3. Add a comment like `// NEVER silently drop fields here — forward all reference roles` 4. Write a test that sends the plural form and asserts it reaches the provider ## 🔴 HARD RULE: No runtime JS errors — smoke-test every build endpoint **Context (2026-06-05):** The nightly MoodBoard build shipped with `crypto.randomUUID is not a function` because `crypto.randomUUID()` only works in secure contexts (HTTPS). On localhost HTTP it crashes silently. The first user interaction hit the crash. **Fix for `crypto.randomUUID`:** Always use a fallback generator for browser environments: ```ts function generateId(): string { if (typeof crypto !== 'undefined' && typeof crypto.randomUUID === 'function') { return crypto.randomUUID(); } return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, (c) => { const r = (Math.random() * 16) | 0; return (c === 'x' ? r : (r & 0x3) | 0x8).toString(16); }); } ``` **Nightly build (t-161) body updated with post-build validation requirements.** The build MUST: 1. Run `npm run build` — confirm exit code 0 2. Start the server, curl all routes, confirm HTTP 200 3. **Smoke-test the HTML/JS bundle** for runtime error patterns: `is not a function`, `is not defined`, `undefined is not`, `Cannot read`, `Cannot access` 4. **Smoke-test for hardcoded localhost/127.0.0.1** in API responses the browser sees 5. Fail if any pattern is found — diagnose and fix before writing summary **Nightly cron job (`nightly-10-dollar-project`) now includes a post-completion smoke-test step** that curls the app port, checks HTTP 200, scans the HTML for JS error patterns, then grep-scans new API routes for hardcoded localhost. ## 🔴 HARD RULE: Derive live data from store, not stale modal state **Context (2026-06-05):** After generating an image in Storyboards, the modal's "Current image" preview still showed the old image. Root cause: the modal stored a `scene` reference from `modalData` when it opened, and never updated it even after `fetchStoryboards()` refreshed the store. The user had to close and reopen the modal to see the new image. **Fix:** In modals/overlays that display mutable data, derive the live object from the global store (e.g., `const liveScene = storyboards.find(...).scenes.find(...)`) rather than relying on the snapshot captured when the modal opened. Use the live reference for display; keep the original only for identifying the target of mutations. **Also:** For long-running batch operations (regenerate-all, bulk import), poll `fetchStoryboards()` every few seconds while the operation is active so the UI shows incremental progress instead of freezing until the entire batch completes. ## 🔴 HARD RULE: Show per-item progress in batch operations **Context (2026-06-05):** Storyboards regenerate-all showed only a global "8 generating" badge. Users couldn't tell which scenes were done, which were queued, or which failed without manually inspecting each card. **Fix:** Added per-scene state tracking: - `generatingScenes: Set` — scenes currently being processed (spinner overlay) - `pendingScenes: Set` — scenes queued waiting their turn (clock overlay) - `sceneErrors: Record` — failed scenes with error message (red overlay) Both list view (SceneCard) and gallery view (GalleryView) show these states as overlays on the image preview. **Rule:** For any batch operation that affects multiple visible items: 1. Track per-item state: generating / pending / failed / done 2. Show the state directly on each item's UI element (overlay, badge, progress bar) 3. Failed items should display the error message inline — don't hide it in a toast 4. A global indicator is necessary but NOT sufficient — users need per-item visibility ## 🔴 HARD RULE: Never process long-running batches in a single HTTP request **Context (2026-06-05):** Storyboards "Regenerate All" for 8 scenes stopped after 1-2 pictures. Root cause: the API route processed all 8 scenes sequentially in ONE HTTP request. Each scene takes ~4 minutes → 32+ minutes total. Browser/proxy timeouts killed the connection after ~1-2 scenes. **Fix:** Changed `processRegenQueue` to call `/api/images` (single scene) per scene from the client side. Each request is independent and short-lived. Added `generatingCount` state + a global "X generating" indicator in the header. **Rule:** When a batch operation exceeds ~30 seconds total: 1. **NEVER** process it in a single HTTP request 2. Break it into individual per-item requests from the client 3. Show a global progress indicator (count badge, spinner, progress bar) 4. Queue items and process sequentially with small delays between them 5. Update the UI incrementally as each item completes ## 🔴 HARD RULE: Classify provider-specific error codes correctly **Context (2026-06-05):** Storyboards "Ashley's Lingerie Transformation" failed all 7 scenes with "Generation failed". Root cause: Google's Gemini returns `finishReason: "IMAGE_OTHER"` for lingerie/adult prompts that don't trigger `IMAGE_SAFETY` explicitly. The mc-router's `classifyLlmError()` only checked for `IMAGE_SAFETY`, so `IMAGE_OTHER` was classified as `unknown` — recorded in the failure registry, blacklisting Gemini for 5 minutes. All subsequent scenes failed because the model was unavailable. **Fix:** Added `IMAGE_OTHER` to the content_policy regex in `mc-router/src/llm-errors.ts`: ```ts /IMAGE_SAFETY|IMAGE_OTHER|content.*filter|.../i ``` **Rule:** When handling provider-specific error codes: 1. Research ALL possible error codes from the provider's API docs 2. Classify prompt-specific rejections (safety, content policy, IMAGE_OTHER) as `content_policy` — NEVER blacklist the model 3. Classify transient errors (rate limit, timeout, 5xx) as retryable 4. Classify permanent errors (auth, billing) as model failures 5. Update the classifier when new error codes are discovered ## 🔴 HARD RULE: Retry transient provider errors in batch operations **Context (2026-06-05):** Storyboards regenerate-all failed 6/8 scenes with "MC error 200" (unhelpful). Root cause: Scene 3 hit a transient Google Gemini `IMAGE_OTHER` error. The mc-router blacklisted Gemini for 5 minutes. Scenes 4-8 then all failed because Gemini was skipped. The same prompt worked when retried manually 30 seconds later. **Fix:** Added retry logic with exponential backoff (3s, 6s) for transient errors (`IMAGE_OTHER`, rate-limit, `RESOURCE_EXHAUSTED`, `exhausted`). Also added a 1.5s delay between batch items to avoid hammering the provider. Error messages now extract the specific provider reason from the MC response `attempts` array instead of showing generic "MC error 200". **Rule:** When calling an external API in a batch loop: 1. Add a small delay (1-2s) between items 2. Retry 2-3 times with exponential backoff on transient errors 3. Surface the actual provider error message, not just HTTP status 4. Distinguish retryable (rate limit, timeout, IMAGE_OTHER) vs permanent (content policy, auth) failures ## 🔴 HARD RULE: Surface per-item error details in batch operations **Context (2026-06-05):** Storyboards regenerate-all showed only "0 generated, 8 failed" with no indication of WHY each scene failed. The user had to guess whether it was a network issue, model failure, or content policy rejection. **Fix:** The API already returned per-scene `{sceneId, ok, error}` details, but the store only kept `succeeded`/`failed` counts. Updated the store to preserve the full `results` array and the UI to render a detailed error panel showing each failed scene's title + provider error message. **Rule:** For any batch operation (bulk generate, import, sync), always: 1. Return per-item success/failure + error message from the API 2. Store the full details in the client state (not just counts) 3. Render per-item errors in the UI — titles, IDs, and the actual error text 4. Keep error toasts visible longer (30s+) when failures exist **Data migration rule:** When fixing a URL format bug that affects persisted state (localStorage, indexedDB, JSON files), add an `onRehydrateStorage` or startup migration that rewrites old data to the new format. Otherwise existing users still see the old broken data even after the code fix ships. ## Image gen stability fix — 2026-05-17 18:46 SGT (HARD RULE) **The native `image_generate` tool is blocked at the gateway** (`tools.deny: ["image_generate", "video_generate", "music_generate", "tts"]` in openclaw.json). **Why:** the gateway's image-gen provider registry only ships an OpenAI adapter. DeepInfra, Google, fal, xai, etc. image providers are NOT registered. When agents called `image_generate` with a non-OpenAI model, every candidate failed with `No image-generation provider registered for `. Past agent behavior: "fix" the failure by editing openclaw.json (adding invented `providers` key, switching primary model to other unregistered ones). Each invalid edit triggered `[reload] config reload skipped (invalid config)`; one bad edit caused the validator to hard-fail the whole gateway on the next restart cycle. Doctor `--fix` was needed to recover. **Correct path for all image generation:** ``` POST http://127.0.0.1:4747/api/llm/image {prompt, agentId?, size?, count?} ``` Direct provider HTTP calls. No gateway involvement. Same architecture as benchmarks (Path B from compat memory). Stress-tested at 18:46 SGT: 8 parallel calls completed in 27s, gateway PID unchanged, zero restarts. **Codified hard rules** (in TOOLS.md): 1. Never call `image_generate` tool — it's denied. 2. Never edit `~/.openclaw/openclaw.json` from inside an agent turn to fix tool failures. Schema-invalid edits crash the gateway entirely (and with it WhatsApp/Discord/all sessions). 3. If a tool path is broken, report it — don't try to "auto-heal" by editing infra config. 4. For gateway config changes that genuinely need to happen: ask Maxime, then use `gateway` tool (`config.schema.lookup` → `config.patch`). Never raw `write` on openclaw.json. ## Gateway tool denied to agents — 2026-05-17 18:57 SGT **Also added to `tools.deny`:** `gateway`. After the image_generate doom-loop, an agent (WhatsApp Robbie) again called the `gateway` tool with `action: restart` at 18:47:11 and again at 18:52:13 with no good reason. Each restart drops all WhatsApp/Discord connections and adds ~2 min of latency. Now blocked at the gateway level. Full deny list as of 18:57 SGT: `["image_generate", "video_generate", "music_generate", "tts", "gateway"]`. The control-ui session (where Maxime orchestrates from) still has the `gateway` tool available — the deny applies to regular agent runtime calls, not the orchestrator/admin path. So legitimate `config.schema.lookup` / `config.patch` operations still work when initiated through the control UI. If this ever changes (e.g., deny becomes truly global), the workaround is to edit `~/.openclaw/openclaw.json` directly (with the strict atomic-write Python pattern in the gateway-fix memory section), then let the hot-reload pick it up. **Lesson:** The `gateway` tool is a power tool. Agents do not need it for normal work. When in doubt, deny. ## Gallery now logs every router image — 2026-05-17 19:05 SGT Maxime asked: "make sure every generated image appears in the gallery page as it was supposed to be. Since all the calls are centralised it should be easy." Yes — done. **Two fixes:** 1. **Gallery scanner now scans both image dirs.** `mission-control/lib/image-gen-state.ts` was only scanning the legacy `~/.openclaw/media/tool-image-generation/` directory (where the native `image_generate` tool used to drop files). Our router writes to `~/.openclaw/workspace/.media/images/`. Added that as a second `MEDIA_DIRS` entry, taught `parseFilename` to handle the router's `-.` naming, and now the gallery jumped from ~17 records to **80 records** showing every image both paths ever produced. 2. **Router endpoint now logs to the gallery JSONL on every success.** `mission-control/app/api/llm/image/route.ts` appends an `ImageGenRecord` to `~/.openclaw/workspace/.image-gen-log.jsonl` after a successful gen. Captures: full prompt (post-tag-strip), model, provider, duration, file path/size, content policy as `nsfw` flag, and provenance tags (`chain:image:default`, `matched:tag|keyword|default`, `policy:uncensored|strict|...`). Best-effort — a logging failure never blocks the response. **Centralised: yes.** One endpoint (`POST /api/llm/image`), one logger. Every image — from any agent on any channel, via tag or keyword or catch-all — lands in the gallery with full metadata. Verified via live test: zen garden prompt at 19:04:13 SGT logged with id `1779015853930-99e29b1d0205d109`, FLUX-1-schnell, default chain, full prompt preserved. **Key files touched:** - `mission-control/lib/image-gen-state.ts` - `mission-control/app/api/llm/image/route.ts` **Related stability rules already in MEMORY.md:** - Provider/model compat lives in `lib/model-compat.ts` per Path B — same principle: don't patch per-call sites, fix the central layer. - Path A (gateway-routed agent traffic) compat lives in OpenClaw upstream. Don't try to local-patch the gateway. ## Image routing rules (visible/editable in MC /models/strategy → Routing Rules → Image Rules tab) | Predicate | Chain | |---|---| | tag `[uncensored]` | image:uncensored (FLUX-1-dev → FLUX-1-schnell) | | tag `[strict]` | image:strict (Gemini 3.1 flash → gpt-image-2) | | keywords: portrait, headshot, photo, selfie, person, face, model | image:uncensored | | keywords: logo, icon, banner, ad, advertisement, brand, commercial, marketing | image:strict | | (catch-all) | image:default (FLUX-1-schnell → FLUX-1-dev) | Hot-reload: edits take effect on next call, no restart. ## Code Modification & Build Verification (HARD RULE, 2026-06-02) **Context:** During the implementation of the Reference Composer (Image-to-Image), blind regex substitutions caused duplicate code blocks and syntax errors. The build failed, but the service kept running old code, leading to confusing debugging loops. Furthermore, APIs (`/compare` vs `/similarity`) and schemas (`FINGERPRINT_SCHEMA`) were incompletely ported. **Rules:** 1. **No blind regex/string substitutions for complex logic:** When modifying large blocks of code, avoid loose regex. Use precise, well-anchored replacements, and explicitly check for duplicate code blocks or dangling brackets left behind. 2. **Always verify builds:** After editing TS/JS files in Next.js apps, ALWAYS run `npm run build` and confirm a clean exit code (`0`) before restarting the service. A failed build means the service restart will just serve the old broken code, invalidating any UI testing. 3. **Port features completely:** When porting a feature (e.g., Vision Judge evaluation or ArcFace similarity), ensure the *exact* payload shapes, API endpoints, and data schemas are identical to the proven working version. Do not write "simplified" versions if the UI expects the full schema. 4. **Sidecar API formats:** The local face similarity API (`http://127.0.0.1:4754`) uses the `/compare` endpoint with `{"a_path": "...", "b_path": "..."}`. It does not use `/similarity` with base64. ## 🔴 HARD RULE: All periodic work must be systemd timers, never OpenClaw cron agentTurns (2026-06-17) **Background (2026-06-13 → 2026-06-17):** The original "script-based" pattern still required one agent LLM call to parse "run this script". That one call was the single point of failure — the nightly $10 build at 1 AM failed 7 consecutive nights because DeepSeek V4 Flash stalled during the bootstrap LLM call while the other 4 crons were just lucky (not robust). On 2026-06-17 all 5 were migrated to pure systemd user timers — zero LLM calls, zero agent session overhead. **The definitive fix pattern:** - Write a standalone Python script in `scripts/` that does the real work - Create a systemd `oneshot` service that runs it directly - Create a systemd timer with `OnCalendar=... Asia/Singapore` for scheduling - Enable + start the timer (`systemctl --user enable --now .timer`) - Disable the old OpenClaw cron job - Scripts that need to deliver to WhatsApp (morning summary) use `openclaw message send --channel whatsapp --target +6580860085 --message "..."` internally **Files (all in `~/.config/systemd/user/`):** | Timer | Service | Schedule | Script | |---|---|---|---| | `morning-summary.timer` | `morning-summary.service` | 09:00 SGT | `scripts/morning-summary.py` | | `nightly-build.timer` | `nightly-build.service` | 01:00 SGT | `scripts/nightly-build-trigger.py` | | `nightly-sleep-dream.timer` | `nightly-sleep-dream.service` | 04:00 SGT | `scripts/nightly-sleep-dream.py` | | `persona-engine-maintenance.timer` | `persona-engine-maintenance.service` | 03:15 SGT | `curl -X POST :4753/api/maintenance/run` | | `validate-model-registry.timer` | `validate-model-registry.service` | 05:30 SGT | `scripts/validate-model-registry.py` | **To check logs:** `journalctl --user -u .service --since today` **To list timers:** `systemctl --user list-timers` ### 🔴 FOLLOW-UP FIX (2026-06-18): systemd PATH + central openclaw resolver **Symptom Maxime reported:** no morning WhatsApp summary for days; assumed the $10 build failed too. **Actual root cause:** systemd user services run with a minimal PATH (`/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`) that does NOT include `~/.local/bin` or `~/.npm-global/bin` where the `openclaw` binary lives. Every script shelling out to a bare `openclaw` died with `[Errno 2] No such file or directory: 'openclaw'`. - `morning-summary.py`: generated the summary text fine but the WhatsApp `openclaw message send` died → no message delivered. - `nightly-build-trigger.py`: the HTTP trigger to t-161 actually SUCCEEDED every night (verified: runs r-t-161-* all status=success exitCode 0). Only the *retry-scheduling* step (`openclaw cron add`) crashed with Errno 2 — harmless dead code, but it also left the state file stuck at `running` and was itself an agentTurn-cron anti-pattern. **The fix (committed 5cf5af3):** 1. `scripts/lib_openclaw.py` — central resolver. Checks `~/.local/bin/openclaw`, `~/.npm-global/bin/openclaw`, `/usr/local/bin`, `/usr/bin`, then `shutil.which`. Exposes `openclaw_bin()` and `run_openclaw(args)`. **Any script needing the openclaw CLI must import this, never call bare `openclaw`.** 2. `morning-summary.py` — delivers via `run_openclaw([...])`, logs non-zero exits. 3. `nightly-build-trigger.py` — rewritten to pure HTTP + in-process retry/backoff (MAX_ATTEMPTS=5). Removed all `openclaw cron` self-scheduling. Checks live task runs for today before triggering (double idempotency). Sends a WhatsApp alert only if all attempts fail. 4. Hardened `Environment=PATH=...` (with `~/.local/bin:~/.npm-global/bin` prepended) on `morning-summary.service` and `nightly-build.service` as belt-and-suspenders. **Verified live (2026-06-18 12:47 SGT):** ran both via `systemctl --user start` — morning-summary delivered to WhatsApp with zero errors; nightly-build idempotently detected today's success and exited 0. **RULE going forward:** never call a bare `openclaw` (or any user-installed binary) from a systemd-run script. Either import `lib_openclaw` or set an explicit absolute path / hardened `Environment=PATH`. ### 🔔 HARDENING (2026-06-18): generic systemd failure alerting + validate-registry fix (commit 21a6266) **Why:** the morning-summary outage was invisible for days because nothing alerts on a silent job failure. Audit also found `validate-model-registry.service` had been failing every 05:30 (404 on a removed endpoint) and nobody knew. **Failure alerting (the key systemic fix):** - `scripts/systemd-failure-alert.py` + `~/.config/systemd/user/failure-alert@.service` (template unit). On any failure it reads the unit's last journal lines and sends Maxime a WhatsApp ALERT (unit name + time + log tail) via `run_openclaw`. - Wired `OnFailure=failure-alert@%n.service` into: `morning-summary`, `nightly-build`, `nightly-sleep-dream`, `persona-engine-maintenance`, `validate-model-registry`. - **Tested live end-to-end** with a throwaway `/bin/false` unit — alert fired and delivered. So no scheduled job can fail silently again. - When adding any NEW periodic systemd job, also add `OnFailure=failure-alert@%n.service` to its `[Unit]` section. **validate-model-registry fix:** `/api/models/registry` was removed from MC; repointed the script to `/api/models` (returns `{models:[...]}`), fixed `priorityChains` parsing (now a dict of chainName -> [modelId strings], not a list of `{steps}`), bumped fetch timeout to 30s. Now exits 0 (passes with informational warnings) instead of crashing. **Other audit findings (healthy):** all 5 timers already have `Persistent=true` (catch up after sleep/downtime); morning-summary already has a robust LLM-down fallback that always produces a message; MC service is active and `/api/models/strategy/config` returns 200. **Note:** systemd unit files in `~/.config/systemd/user/` are NOT in the workspace git repo — only the scripts they call are. If migrating machines, the `.service`/`.timer` units must be recreated. ### 🌙 ROOT-CAUSE of the middle-of-the-night WhatsApp message (2026-06-18) **Maxime asked about a message received during the night.** Traced it: it was a **failure-alert from the OLD `nightly-10-dollar-project` OpenClaw cron**, which fired 01:00 SGT Wed 2026-06-17, timed out (`model-call-started` stall — the classic isolated-agentTurn-cron failure), and had `failureAlert` configured to WhatsApp after 2 errors. It had racked up `consecutiveErrors: 7`, so it was pinging the phone at night. The underlying script failure was the same bare-`openclaw` Errno 2 / timeout bug already fixed. **The deeper problem:** the 2026-06-17 systemd migration **disabled** the 5 old OpenClaw crons (`enabled:false`) but left them in place — each still carried a `failureAlert` config and stale error counters. A disabled cron doesn't fire, but if anything ever re-enabled them (sync/restore), they'd resume night-spamming. Belt-and-suspenders is to not leave zombie crons around. **Fix (2026-06-18):** since systemd timers now fully own all 5 jobs (verified active + scheduled in `list-timers`), **deleted all 5 OpenClaw crons outright** (`morning-whatsapp-summary`, `nightly-10-dollar-project`, `nightly-sleep-dream`, `persona-engine-nightly-maintenance`, `validate-model-registry`). `openclaw cron list` is now empty. There is exactly ONE mechanism per job (systemd), no duplicates, no zombie failure-alerts. **RULE:** when migrating a periodic job from OpenClaw-cron to systemd, DELETE the old cron, don't just disable it. A disabled cron with a `failureAlert` is a latent night-spam source. **Key principle:** A periodic task should never require an LLM call to bootstrap. If the actual business logic needs an LLM (summarisation, analysis, generation), the script calls MC's text router (`/api/llm/text`) which has proper chain cascade/failover across providers. The delivery mechanism (agent session) is replaced by direct systemd execution + optional `openclaw message send`.