Vision & Image Generation

Enable the crew to analyze images (screenshots, diagrams, mockups) and generate images on demand — cost-effectively, across multiple providers.

Status

active

Notes

Vision & Image Generation for the Crew

Owner: Robbie 🧐

Goals#

Vision (phase 1): Enable the crew to analyze screenshots, mockups, diagrams, and any image shared via WhatsApp or Discord. Cost-sustainable — avoid expensive Claude Sonnet vision calls.
Generation (phase 2): Enable the crew to generate images on demand via multiple providers.

Approach#

Phase 1 — Vision#

Wire up Google Gemini 2.0 Flash as a vision model provider in OpenClaw.

Why Gemini Flash over alternatives
Cost	$0.10/MTok input — ~150x cheaper than Claude Sonnet
Free tier	1,500 requests/day — enough for daily use at zero cost
Quality	Solid for screenshots, code, diagrams, whiteboard drawings
Integration	Google provider is already built into OpenClaw's engine

How it works: When an image comes in, I call the image() tool with model: "google/gemini-2.0-flash" to route it through Gemini instead of the expensive Claude Sonnet session.

Phase 2 — Image Generation#

Multi-provider strategy for image generation:

Provider	Status	NSFW	Cost/image	Notes
DeepInfra (FLUX)	✅ Active	❌ Blocked	$0.003-0.025	Fast (1.5s), default gen model
Civitai	🔜 t-3454/3455	✅ Permissive	~$0.05-0.10	Bridge script needed
RunPod	🔜 t-3456/3457	✅ Full control	~$0.0003-0.0006	Docker/ComfyUI endpoint
Gemini Image Gen	❌ Blocked	N/A	$0.067	Credits depleted (HTTP 429)

Primary gen model: DeepInfra FLUX Schnell — fast, cheap, good quality. Fallback chain: DeepInfra FLUX → Gemini Flash Image Preview → Gemini Pro Image Preview.

Architecture#

Detailed architecture in ARCHITECTURE.md.

Status — Current#

What's already done#

Google provider registered in models.providers with api: google-generative-ai
gemini-2.0-flash model exists with input: ['text', 'image'], cost: $0.10/$0.40/MTok
Cost entries already correct
DeepInfra provider active for FLUX generation
Architecture doc written (ARCHITECTURE.md)

What needs fixing#

Validation errors — All 11 Google model entries have extra keys (freeTier, dailyFreeQuota, strengths, weaknesses, recommended) that the schema rejects. Need to strip these.
Missing alias — Need google/gemini-2.0-flash → alias gemini-flash in agents.defaults.models
API key — Maxime has the key; needs to add GEMINI_API_KEY to .env

After fix#

openclaw config validate returns zero issues
openclaw models list shows google/gemini-2.0-flash with input: text image
I can call image(model="google/gemini-2.0-flash", ...) to analyze screenshots

Task Dependency Graph#

t-135 [Maxime]  Get Gemini API key  ✅ DONE
   └── t-136 [Chris]  Fix Google models & alias  ✅ DONE
        └── t-3364 [Robbie]  Smoke-test vision  🔄 backlog
             └── t-3365 [Kelly]  Document vision setup  🔄 backlog

t-139 / t-1000 [Debbie]  Research image gen options  ✅ DONE
   └── DeepInfra FLUX  ✅ Active
   ├── t-3454 [Maxime/Robbie]  Civitai Phase 1: API key + config + bridge  ✅ DONE
   │    └── t-3455 [Chris]  Civitai Phase 2: MC integration + Buzz tracker  🔄 backlog
   └── t-3456 [Maxime]  RunPod Phase 1: Sign up + deploy endpoint  🔄 backlog
        └── t-3457 [Chris]  RunPod Phase 2: Bridge script  🔄 backlog

Crew Assignment#

Task	Who	Task ID	Status
Get API key	Maxime	(no task)	✅ Key obtained
Fix Google models & alias	Chris	(no task)	✅ Models registered
Smoke test vision	Robbie	t-3364	backlog
Document vision	Kelly	t-3365	backlog
Research gen options	Debbie	(no task)	✅ Done — DeepInfra/FLUX active
Merge image-gen project	Vera	(this)	✅ Done — merged into vision-integration
Civitai Phase 1 (sign up + config + bridge)	Maxime/Robbie	t-3454	✅ DONE
Civitai Phase 2 (bridge)	Chris	t-3455	backlog
RunPod Phase 1 (deploy)	Maxime	t-3456	backlog
RunPod Phase 2 (bridge)	Chris	t-3457	backlog

Key Decisions#

Decision	Status	Notes
Vision model	Gemini 2.0 Flash	Chosen. Provider already registered.
Primary gen model	DeepInfra FLUX Schnell	Active — fast, cheap, good quality
Gen fallback	Gemini Flash Image Preview	Vision-aware gen, fallback if DeepInfra down
High-quality gen	Gemini Pro Image Preview	For production-grade images
Civitai orchestration	t-3454/3455	✅ Phase 1 done, bridge script ready
RunPod serverless	t-3456/3457	Full control, lowest cost/image ($0.0003-0.0006)
Gemini image gen	❌ Blocked	Credits depleted, HTTP 429
Local vs API for gen	API recommended	DeepInfra active, Civitai/RunPod coming

ARCHITECTURE.md — architecture, pipeline design, conventions
CHRIS-SPEC.md — Chris spec for Google model fixes
image-gen-setup.md — legacy image gen setup reference (Gemini gen approach, preserved)
civitai-setup-spec.md — Civitai orchestration API setup
runpod-setup-spec.md — RunPod serverless setup

7/7

tasks done

Tasks (7)

View in board →