%3Aquality(90)%2F&w=3840&q=75)
%3Aquality(90)%2F&w=3840&q=75)
Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1
When evaluating AI models for observability, generating a plausible query or explanation isn't enough. Teams need to know whether an agent's conclusions are supported by telemetry, whether its changes to dashboards actually work, and whether it can deliver reliable results consistently.
This is why we built o11y-bench, Grafana Labs' open benchmark for evaluating AI models on observability and SRE tasks. And today, we're excited to share two updates: o11y-bench is now part of the Specialized Intelligence Index from Fireworks, the AI training and inference platform, and we're releasing o11y-bench 1.1, with updates that make grading more grounded, consistent, and reproducible.
Bringing o11y-bench to the Specialized Intelligence Index
The Specialized Intelligence Index brings together practitioner-built benchmarks for comparing how open, closed, and specialized AI models perform on domain-specific work, from healthcare and finance to cybersecurity and software engineering. It helps teams identify the best models for their workloads, understand where specialization adds value, and evaluate tradeoffs between quality, cost, and speed.
With o11y-bench now part of the index, SRE, platform engineering, and observability teams can explore how AI models perform on real observability workflows—from querying metrics, logs, and traces to investigating incidents and managing Grafana dashboards—alongside benchmarks for other specialized domains and use cases.
Run the benchmark with Fireworks inference
o11y-bench also now supports Fireworks as an inference provider in its default LiteLLM-backed agent. This means teams can evaluate both serverless models and models deployed within their own Fireworks accounts, using the same reproducible workflow available for other providers.
The integration handles Fireworks credentials, deployment names, and reasoning settings so benchmark runs are configured consistently. This makes it easier to reproduce results from the index or evaluate Fireworks-hosted models against your own requirements.
How o11y-bench evaluates models on real observability tasks
Observability deserves a place in model evaluation. During an incident, the hard part is rarely generating valid PromQL, LogQL, or TraceQL code. An engineer, or an agent, has to identify the relevant signal, use the right time range, connect evidence across telemetry types, distinguish a symptom from a cause, and sometimes modify persistent state without breaking something another engineer depends on.
o11y-bench evaluates those capabilities and more in a running Grafana stack. Agents use the Grafana MCP server to work with synthetic but realistic telemetry and complete 64 tasks across six areas:
- Prometheus and PromQL
- Loki and LogQL
- Tempo and TraceQL
- Multi-step incident investigations
- Grafana API workflows
- Dashboard creation, editing, and repair
The tasks are focused enough to grade, but still reflect real challenges engineers face in day-to-day observability workflows. For example, a query can be syntactically correct while selecting the wrong series, and a dashboard can still render even though its variables, data source, or saved query are wrong.
For this reason, o11y-bench starts with outcomes. It compares an agent’s claims with facts read from the same environment, and inspects the state the agent leaves behind. In other words, a convincing answer isn't enough; the results need to back it up.
Why consistency matters
Each o11y-bench task is run three times. The benchmark measures not just whether a model can solve a task once, but how consistently it performs that task across repeated attempts.
That consistency is especially important for observability. A model that reaches the right answer one time might follow a false lead or report an unsupported value the next. During an incident, that inconsistency can cost time and make it harder for teams to trust the results.
o11y-bench prioritizes Pass^3, the average benchmark score across three runs, while also reporting Pass@3, which captures whether the model succeeded at least once. Looking at both helps separate occasional wins from repeatable performance.
Strengthening the benchmark: what’s new in o11y-bench 1.1
Alongside o11y-bench's inclusion in the Specialized Intelligence Index, we’re releasing o11y-bench 1.1, which makes grading more outcome-based, reproducible, and resistant to answers that sound right without being grounded in the environment.
The improvements fall into three areas:
- Grounded outcomes: Rather than rewarding a particular approach or phrasing, o11y-bench checks the outcome: whether the model’s claims match the facts, its saved changes are correct, and its behavior meets the task’s requirements. Reported numbers are checked against reference values, and cited traces are read back from Tempo to verify that they support the agent’s claims.
- Deterministic grading: Dashboard edits are evaluated using before-and-after diffs of saved state. If a ground-truth query is empty or fails, the result is treated as a grading error rather than being allowed to influence pass rates.
- Reproducible, calibrated evaluation: Saved facts let results be regraded without rerunning the agent, and scenarios remain consistent over time. A no-tools control failed all 64 tasks, showing that plausible answers alone cannot pass.
Because o11y-bench 1.1 uses updated grading and task calibration, its results are not directly comparable with 1.0. Version 1.1 establishes a new baseline for future comparisons.
Open benchmarks make better benchmarks
o11y-bench’s tasks, environments, grading logic, and results are all open source. That means anyone can examine how the benchmark works, reproduce its results, and help identify areas for improvement.
Adding o11y-bench to the Specialized Intelligence Index extends that work to a broader model-evaluation effort. We’re excited to see observability represented alongside other specialized domains, and to give engineering teams more evidence for evaluating the models they use in their workflows.
We also expect the benchmark to keep evolving. New models, agent harnesses, observability tools, and failure modes will expose gaps and create opportunities to make the evaluation stronger. As always, we welcome feedback and contributions from the community as part of that process.
To get started:
