See parent task t-3594 for full context.

What#

Upgrade lib/benchmark-runner.ts to support:

1. Vision inputs: Accept image URLs/files as part of prompt. Use gateway vision API (image content blocks) for vision-capable models. Fall back gracefully for non-vision models.

2. Image generation: For image-gen models, run the prompt through the image generation API, compare outputs side-by-side (store image URLs, create comparison view).

3. Audio: Support audio file inputs for transcription/analysis models.

4. TTFT tracking: Use streaming API to measure time-to-first-token. Gateway supports SSE streaming via /v1/chat/completions?stream=true.

5. Auto-fetch pricing: Instead of hardcoded MODEL_META, query the gateway model registry for pricing. The gateway returns model info with _openclaw.cost. Falls back to hardcoded defaults.

6. Better error classification: Classify errors as: rate-limited, timeout, auth-failed, model-unavailable, content-filtered, unknown.

7. Cancel support: Allow canceling a running benchmark mid-execution.

Deliverables#

• Updated lib/benchmark-runner.ts
• New lib/benchmark-pricing.ts (extracted/expanded pricing logic)
• Tests for new functionality
• Existing tests still pass

Tags#

[benchmarks, runner, multimodal, streaming, pricing]