Skip to main content

2.14.4. AI Evaluation

AI Evaluation is a tool for verifying the quality and stability of AI scenario execution. It allows you to compare the actual scenario execution result with the expected result based on predefined datasets and criteria.


AI Evaluation is useful for:

  • verifying changes in scenarios after updates or modifications,
  • comparing response quality across different models or configurations,
  • automated quality control of AI results before use in business processes.

2.14.4.1. Create an AI Evaluation​

  1. In the navigation panel, select the Studio 1 workspace.

  2. Select the AI Center 2 shortcut group, then select the AI Evaluation 3 shortcut.

  3. In the toolbar, select Create 4.

  4. Fill in the fields using the hints in the table below.

    FieldDescription
    Code*
    • The code must be unique.
    • The code must be short (typically up to 10 characters).
    • Use only Latin letters and digits.
    Name*Enter a short, clear name for the evaluation.
    DescriptionOptionally, enter a short description of the evaluation's purpose.
    Note:

    Fields marked with "*" are required.

  5. Go to the Dataset Schema 1 tab. In this tab, you must add the same variables as in the scenario you want to evaluate.

  6. Fill in the Input Data Schema section by adding input variables, that is, those that transfer information from the system to the scenario. For example, for the system scenario sys_doc_recognize, this could be a documentTypes variable that passes the context of system document types to the scenario. To add input data:

    1. Select Add Field 2.

    1. In the Name 1 field, enter the same name as the variable in the scenario you want to evaluate.

    2. In the Data Type 2 field, select the same data type as the variable in the scenario you want to evaluate.

    3. In the Description 3 field, enter a short description of the variable's content. For example, "data about all document types".

    4. Enable the Is Global 4 checkbox if you want to pass this variable's data for all records in the dataset (which we will add in the next tab).

    5. Select Save 5.

  7. Fill in the Output Data Schema section by adding output variables, that is, those that transfer information from the scenario to the system. For example, for the system scenario sys_doc_recognize, this could be a result variable that passes the recognition result. To add output data:

    1. Select Add Field.

    2. In the Name 1 field, enter the same name as the variable in the scenario you want to evaluate.

    3. In the Data Type 2 field, select the same data type as the variable in the scenario you want to evaluate.

    4. In the Description 3 field, enter a short description of the variable's content. For example, "recognition result".

    5. Select Save 4.

  8. Go to the Dataset 1 tab to add your expected result for each variable.

  9. Select + 2.

  10. In the Description 1 field, enter a name for the dataset.

  11. Next to each variable, select the 2 icon and upload the values you expect from the AI model.

Tip:

You can use the results of already executed AI scenarios as a reference for evaluation. If a certain execution produces a correct and desired result, its value can be copied directly to the evaluation dataset.

To do this, follow these steps: Studio 1 workspace > AI Center 2 shortcut group > AI Execution 3 shortcut > select an execution > in the top-right corner, select 4 > Copy to AI Evaluation Dataset 5.

In the AI Evaluation 1 field, select the evaluation to which you want to copy the results as a dataset, then select Copy 2.

Note:

You can add several such datasets. When running an evaluation, you will be able to select which dataset to use for the evaluation.

  1. Go to the Evaluation Criteria 1 tab to configure how the system will compare the scenario execution result with your expected result.

  2. Select Add 2.

  3. Add an evaluation criterion 1 using the hints in the table below, then select Save 2.

FieldDescription
Name*Enter a short, clear name for the evaluation criterion.
Rule*Select from the list how you want to compare the actual and expected results:
  • Custom Script — comparison is performed using a script you provide, which allows you to implement custom evaluation rules and logic.
  • Strict Equality — actual and expected results are compared for exact match of values. Any difference is considered an error.
  • LLM as Judge — an AI scenario of type AI Evaluation will be used for comparison.
Threshold*
field available only for Custom Script and LLM as Judge rules		Select a value between 0 and 1. Threshold is the minimum level of result compliance with expectations. For example, a value of 0.7 means the result must match the expected result by at least 70% to be considered successful.
Script*
field available only for Custom Script rule		Insert a script that will compare the actual and expected results. By default, this field will display an example of such a script. You can edit it to meet your needs.
AI Scenario*
field available only for LLM as Judge rule		Select from the list an AI scenario to be used for comparing results. You can choose one of the system evaluation scenarios:
  • sys_doc_recognize_judge — evaluation of document recognition scenario.
  • sys_doc_splitter_judge — evaluation of document splitting scenario.
  • sys_doc_summarize_in_doc_lang_judge — evaluation of document summarization scenario.
PathFill in this field only if you need to access a specific element or nested value in a dataset item. Enter the path to the value in the dataset item that will be used for evaluation. For example: result.answer or items[0].value. Leave empty to use the entire item.
Note:

Fields marked with "*" are required.

  1. In the toolbar, select Save.

2.14.4.2. Run AI Evaluation​

Method 1: from the AI Evaluation form

  1. Open the AI Evaluation you want to run:

    1. In the navigation panel, select the Studio 1 workspace.

    2. Select the AI Center 2 shortcut group, then select the AI Evaluation 3 shortcut.

    3. Open the required evaluation 4.

  2. In the toolbar, select Evaluate.

  3. In the AI Scenario 1 field, select the scenario you want to evaluate.

    Note:

    Make sure the variables of the selected scenario match the dataset schema you configured when creating the AI Evaluation.

  4. In the Number of Repetitions 2 field, specify the number of times to repeat the evaluation for each dataset record.

    Note:

    The maximum possible value is 100 repetitions.

  5. In the Evaluation Dataset Records 3 section, check the boxes next to the records you want to use in the current evaluation.

  6. Select Create 4.

After that, a list of dataset records A and their execution statuses B will appear.

You can view AI Execution for each record by following the link in the AI Execution (run) A row. After the AI execution is completed, the evaluation will be run, and you can view its details by following the link in the AI Execution (evaluation) B row.

Method 2: from the AI Scenario form

  1. Open the AI Scenario you want to evaluate:

    1. In the navigation panel, select the Studio 1 workspace.

    2. Select the AI Center 2 shortcut group, then select the AI Scenario 3 shortcut.

    3. Open the required scenario 4.

  2. In the toolbar, select Evaluate.

  3. In the AI Evaluation 1 field, select from the list the AI Evaluation you want to use to evaluate the scenario.

    Note:

    Make sure the variables of the selected scenario match the dataset schema you configured when creating the AI Evaluation.

  4. In the Number of Repetitions 2 field, specify the number of times to repeat the evaluation for each dataset record.

    Note:

    The maximum possible value is 100 repetitions.

  5. In the Evaluation Dataset Records section, check the boxes next to the records you want to use in the current evaluation 3.

  6. Select Create 4.

After that, a list of dataset records A and their execution statuses B will appear.

You can view AI Execution for each record by following the link in the AI Execution (run) A row. After the AI execution is completed, the evaluation will be run, and you can view its details by following the link in the AI Execution (evaluation) B row.

2.14.4.3. View AI Evaluation Execution Log​

Method 1: from the AI Scenarios form

  1. Open the scenario whose evaluation you want to view:

    1. In the navigation panel, select the Studio 1 workspace.

    2. Select the AI Center 2 shortcut group, then select the AI Scenarios 3 shortcut.

    3. Open the required scenario 4.

  2. Go to the AI Evaluation Log tab.

    Note:

    If the selected scenario has no records in the AI Evaluation Log tab, it means that this scenario has never been evaluated.

  3. Select the evaluation session whose details you want to view.

Method 2: in the Evaluation Log shortcut

  1. In the navigation panel, select the Studio 1 workspace.

  2. Select the AI Center 2 shortcut group, then select the AI Evaluation Log 3 shortcut.

  3. Select the evaluation session whose details you want to view 4.