Local LLM on the 4060
The home server has an RTX 4060 (8GB VRAM) sitting idle. We use Anthropic Opus/Sonnet for everything — even trivial classification, heartbeat checks, summarization, and routing. Most of those tasks don't need a frontier model. Running them locally:
- Cuts API spend on routine work
- No rate-limit cooldowns for noisy paths (heartbeats, monitors)
- Works offline / when network flakes
- Faster latency for simple short calls
- 8GB VRAM → 7B models at q4_K_M quant fit comfortably (4–5GB), leaves headroom for KV cache
- 13B at q4 just fits but degrades quality with long context
- No 70B / no FP16 of anything serious
- Runtime: Ollama (simplest, OpenAI-compatible API on
:11434, easy systemd unit, model swap on demand)
- Provider integration: Register Ollama as an OpenAI-compatible provider in
~/.openclaw/openclaw.json
- Routing: Per-task / per-cron model overrides — cheap stuff goes local, strategic work stays on Opus
| Model | Size | Use case |
|---|
qwen2.5:7b-instruct-q4_K_M | 4.7GB | Default all-rounder. Strong on chat + tool-calling. |
qwen2.5-coder:7b-q4_K_M | 4.5GB | Cheap code generation tasks for Chris (small refactors, lint fixes, comments). |
llama3.1:8b-instruct-q4_K_M | 4.9GB | Backup all-rounder if Qwen disappoints on something. |
nomic-embed-text | 270MB | Local embeddings for semantic search of memory/docs. |
- Ollama installed + running as a systemd user unit, autostarts on boot
- Models above pulled and tested
- Configured as an OpenClaw provider with auth-less local URL
- At least one cron job (e.g. heartbeat email check) routed to local model and proven to work end-to-end
- Cost dashboard in Mission Control shows local runs as $0
- Documented in
MEMORY.md so future-me knows how to swap models, restart Ollama, and where to look when things break
- t-038: Install + configure Ollama, pull starter models
- t-039: Register Ollama as OpenClaw provider, smoke-test from CLI
- t-040: Pick one cheap recurring task and route it to local Qwen — measure success rate
- t-041: Document the setup in workspace MEMORY.md + a
local-llm.md runbook
- Driver state on the 4060? (check
nvidia-smi) Need CUDA 12.x + recent NVIDIA driver for Ollama GPU.
- Which agent should be the first guinea pig — Debbie (ops/research) is the cheap-tasks workhorse, makes sense.
- Should we keep Anthropic as fallback for local-routed tasks, or fail loud when Ollama is down?