Output Quality Evaluation¶

Output Quality Evaluation is used to check whether the output of a large language model meets expectations. By configuring evaluation rules, you can run evaluations on online model calls or data within a specified scope, and view scores and rule hit results.

Currently supported evaluation types are Prompt Eval and Schema Eval.

Create an Evaluator¶

Go to Agent Monitoring > Evaluation and click Create Evaluator.

Configure the following information:

1. Select an evaluator Skill template;
2. Select the evaluation target type;
3. Configure the evaluation input and judgment rules;
4. Run a test with sample data;
5. After confirming the results meet expectations, publish the evaluator.

Prompt Eval¶

Prompt Eval uses prompt words to judge model inputs or outputs. It is suitable for evaluation scenarios such as content quality, relevance, completeness, and security.

When configuring, you need to set:

• Evaluation prompt words;
• Evaluation result and scoring rules.

It is recommended to clearly specify the evaluation objective, judgment criteria, and return format in the prompt words to ensure stable evaluation results.

Schema Eval¶

Schema Eval is used to check whether the model output conforms to a specified data structure. It is suitable for scenarios that require the model to return JSON or other structured content.

When configuring, you need to set the target Schema. After the evaluation is executed, the system checks whether the fields, types, and structure of the output content meet the requirements.

Test an Evaluator¶

Before publishing an evaluator, you can input test data to run a trial.

The test results will show the evaluation output, scores, or structure validation results. You can adjust the prompt words, Schema, or judgment rules based on the results, and then re-test.

Once the test passes, click Publish to make the evaluator available for formal evaluation tasks.

Run an Evaluation¶

Online Evaluation¶

Online evaluation is used for continuous checking of model call data generated online.

Select the target application and evaluator, and configure the data range to be evaluated. After the task is enabled, the system will evaluate the matching data according to the configuration.

Batch Evaluation¶

Batch evaluation is used to evaluate historical model call data within a specified time range or query conditions.

When creating a batch evaluation task:

1. Select the target application;
2. Set the time range and filter conditions;
3. Select the evaluator(s) to be used;
4. Confirm and start the evaluation task.

View Evaluation Results¶

Go to the evaluation results page to view:

• Evaluator and evaluation method;
• Evaluation status;
• Score or rule validation results;
• Model input and output;
• Evaluation time;
• Failure reason.

You can filter evaluation results by application, evaluator, evaluation status, and time range.

Manual Annotation¶

Through manual annotation, you can review evaluation results by combining model input, output, and the call chain, and add samples to the dataset.

Create an Annotation Queue¶

Create an annotation queue based on Trace query conditions. Supports marking data such as failures and abnormal Agent judgments.

The queue shows the count of items pending annotation, completed, and skipped, as well as the current processing progress. The queue uses a shared list of items pending annotation; it does not support assignment to designated persons, claiming, or arbitration workflows.

Submit Annotations¶

In the annotation workbench, view the input, output, call chain, tool calls, user feedback, and existing evaluation results, then fill in:

After filling in, you can choose "Save and Next" to continue annotating, or "Save and Add to Dataset" to persist the sample.

Re-annotating will preserve the history. Manually annotated results will be marked with their source and displayed separately from platform evaluation results and external Scores.

Dataset¶

Datasets are formed from production Trace conditions and manual annotation conditions. They are used to centrally manage samples and execute batch evaluations.

Initiate a Batch Evaluation¶

Select samples within a dataset and initiate a batch evaluation. When the task starts, the sample scope used for this task is fixed; subsequent changes to dataset members will not affect the running task.

Permission Description¶

Creating an annotation queue and submitting manual annotations require annotation permissions; executing batch evaluations requires evaluation execution permissions. Access to datasets, annotations, and evaluation results is governed by the access permissions of the current workspace and the original Trace