What Prompt & model evals measure

Prompt & model evals are workspace-authored suites that check whether a prompt and/or model still behaves the way you expect. You define golden cases with expected signals, run the suite, and review pass/fail history plus insight cards for regressions and efficiency from those runs.

Open them from the app sidebar under Insights → Prompt & model evals, or go to /insights/evals. The hub lists suites and insight cards. Open a suite to author cases, trigger a run, and read results. Any workspace member can open the pages. Creating suites and starting runs needs an editor or admin role.

Evals answer: “Did this prompt or model still pass our checks?” They do not replace spend charts or Agentic Foundation readiness.

How this differs from AI Usage and Token Efficiency

Three surfaces sound related. They answer different questions.

Surface	Question it answers	Where it lives
Prompt & model evals	Does this prompt/model still meet our golden cases?	Insights → Prompt & model evals (/insights/evals)
AI Usage & Spend	How much are we spending, and who/what is using Cursor or Claude?	Insights → AI Usage & Spend
Token Efficiency (Agentic Foundation)	Is repo context hygiene good enough that agents do not burn window on noise?	Agentic Foundation → The 9 Dimensions (dimension I)

• Evals score runtime quality of a prompt body and/or model against cases you wrote (contains, not-contains, regex, JSON path equals). Scores come from those suites, not from billing ingest or readiness rubrics.
• AI Usage & Spend is ingest spend and usage from connected AI tools. It has no eval suites, golden cases, or suite versions. See AI Usage & Spend.
• Token Efficiency is repo context hygiene in the readiness rubric (ignore files, doc layering, bounded always-on rules). It is not a prompt suite and does not use eval run scores. See The 9 Dimensions.

The evals hub can show average tokens or cost from successful eval cases when a run recorded them. That rollup is about those suite runs. It is not the Token Efficiency dimension, and it is not the AI Usage spend charts.

Starter packs vs custom suites

The hub lists platform starter packs (PR review, unit test generation, incident summary, support reply). Each pack is a ready-made suite template with a few golden cases and fixture completions, so the first run can succeed without an AI connector.

Editors and admins can Install a pack. That creates a normal editable suite in your workspace (same as creating one from scratch). You can rename it, change the prompt or model target, and edit cases afterward. Viewers can see the pack list but cannot install.

Use a blank Create eval suite when none of the packs match your workflow.

Suites, cases, and runs

A suite targets at least one of: a prompt (text snapshot, optional key) or a model (cursor or claude plus a model name). You add cases with an input and expected signals. Changing the target, cases, or prompt body bumps the suite version so older runs keep the snapshot they used.

When you start a run, the platform queues it and scores in the background. You can have one active run per suite at a time. Cases can use a fixture completion (works without AI connectors) or, when credentials exist for the suite’s provider, a live completion. Scoring is deterministic from your signals. There is no LLM-as-judge in this release.

After runs finish, the hub surfaces regressions versus the previous successful run, prompt tips from failed signals, and (when available) token or cost averages from those runs. Insight cards on the hub are the v1 share surface in the workspace. You can copy a markdown summary. There is no public internet share and no Report Center PDF for evals yet.

Compare runs and decisions

On a suite detail page, pick two succeeded runs (baseline and candidate) and open Compare. The panel shows suite-level score, token, and cost deltas, plus per-case pass/fail when both runs include the case. If the case set changed between versions, unpaired cases are listed separately.

Editors and admins can record a decision after compare:

Decision	What it does
Promote candidate	Saves the decision and applies the candidate run’s prompt/model target to the live suite (version bumps when the target changes). Cases on the suite stay as they are today.
Keep baseline	Saves the decision only. No suite change.
Reject candidate	Saves the decision only. No suite change.

Viewers can open compare and read past decisions. They cannot post a decision. A promotion gate that could block promote on CI or policy is not shipping in this release; decisions still record that the gate was not applied.

Compare is about eval configuration quality (prompt/model versions on the same suite). It is not AI Usage spend and not Agentic Foundation Token Efficiency.

What you need

Goal	What you need
Open the hub and read results	Active workspace membership (viewer is enough)
Install a starter pack, create or edit suites, start runs, record promote/keep/reject	Workspace editor or admin
Compare two succeeded runs	Active workspace membership (viewer is enough)
Run with fixture completions only	No AI connector required (starter packs include fixtures)
Live completion for a model target	Credentials for that provider in the workspace

Empty hub: install a starter pack or use Create eval suite. If a live run needs credentials and the suite has no fixtures, the live-run control stays disabled with a short explanation.

Limits in this release

• Not a replacement for AI Usage & Spend or Agentic Foundation Token Efficiency.
• No LLM judge, no pack marketplace or CMS, and no promotion gate that blocks promote on regression (gate attach is a later release).
• Eval insights are not exported through Report Center.

Next

• AI Usage & Spend
• The 9 Dimensions (Token Efficiency)
• Insights overview
• Insights & Recommendations