Set up human review

Human review lets people answer a few questions about a sample of conversations. Their answers appear next to judge scores, so you can check whether your evaluators agree with people. You can export every answer as CSV.

Human review is in public preview.

Before you begin

To create questions and review queues, you need the Agent Observability Admin role.
To answer questions, you need the Agent Observability Feedback Writer or Admin role.
Every role with data access can view queue progress and download answers.

For more information about roles, refer to Configure RBAC.

Understand questions and review queues

A question is an evaluator of kind Human review. Agent Observability never runs it. People answer it about the whole conversation.

A review queue is a collection of conversations plus a rubric: the questions reviewers answer for each conversation. A rubric holds up to 50 questions. Each question gets one answer per conversation in a queue, so two reviewers can’t answer the same conversation. You can use human review questions only in review queues, not in online evaluations, guards, experiments, or test suites.

Each question has one of three answer types:

Number: reviewers pick a value on an integer scale. Scales of up to 11 values show buttons; larger scales show a number field.
Boolean: reviewers pick Yes or No.
String: reviewers pick one value from the allowed values you define.

Answers are values, not free text. Reviewers can add one optional note each time they submit.

A question can have a pass condition, the same as other evaluators. Without one, Agent Observability keeps the answers but doesn’t count them toward the pass rate.

Create a question

Create one question for each thing you want reviewers to judge.

In Evaluations, click More and select View evaluators.
Click Create Evaluator and select Create manually.
In Evaluator ID, enter a unique ID, for example, support.helpfulness.
In Evaluator kind, select Human review.
In Question, enter what reviewers answer about the conversation.
Optional: In Instructions, add guidance that reviewers see next to the question.
Under Output type, select Number, Boolean, or String, and configure the pass condition. For details, refer to Configure outputs and pass verdicts.
Click Create.

Create a review queue

A review queue starts from a collection. Add conversations to a collection when you save conversations, or route them with evaluation actions.

Go to Evaluations and click Reviews.
Click New review queue.
In Collection, pick or create the collection to review.
Under Rubric, pick the questions reviewers answer.
Click Start reviewing this collection.

To change the questions later, open the queue, click the menu next to CSV, and select Edit rubric.

To stop reviewing a collection, select Stop reviewing in the same menu. The queue disappears from Reviews. Agent Observability keeps its conversations and every answer already given.

Review conversations

Reviewers answer every question in the rubric for one conversation at a time.

Go to Evaluations and click Reviews.
Click a queue to open it.
Click Start reviewing, or Continue reviewing if the queue already has reviewed conversations. The next conversation to review opens with the review panel.
Answer every question. You can submit once every question has an answer.
Optional: Add a note that explains your answers.
Click Submit and next, or press Cmd+Enter (Mac) or Ctrl+Enter (Linux/Windows).

You can also open the review panel from any conversation in a queue: click Review in the conversation header.

Remove a conversation without generations

A conversation that a guard blocked before the model call has no generations, so it can’t be reviewed. The review panel shows Remove from queue for it. Removing the conversation takes it out of the queue’s collection. Agent Observability keeps the conversation.

You need the Agent Observability Admin role to remove a conversation.

Monitor results and export answers

The queue page shows the progress of the queue, the pass rate of each question, and the result of each reviewed conversation. Reviewer answers also appear next to judge scores on the conversation page. A review covers the whole conversation, so Agent Observability attributes it to the agent that gives the final answer.

To export the answers, open the queue and click CSV. The file has one row per answer, with these columns: conversation_id, generation_id, agent_name, agent_version, evaluator_id, evaluator_version, score_key, value, passed, reviewer, note, and created_at.