Simulation Testing

Automatically validate agent quality with predefined test cases

Simulation testing allows you to automatically validate agent quality by running predefined test cases against your agents. This verifies consistent performance and helps catch regressions before they impact users.

Simulation Testing

Overview

Simulation testing in AGO provides:

• Automated Quality Assurance: Test agents against expected behaviors
• Regression Detection: Catch issues when changing prompts or models
• Tool Validation: Verify agents call the right tools with correct parameters
• Performance Tracking: Monitor success rates over time

Prerequisites

System Requirements

• AGO platform deployed and accessible
• Admin access to create datasets and run simulations

Knowledge Required

• Understanding of your agent’s expected behavior
• Familiarity with common user questions

---

How It Works

1. Create a Dataset with related test cases
2. Define Test Cases with questions and expected outcomes
3. Run Simulation against a specific agent
4. System uses LLM Evaluation to score responses
5. Review Results and track success rates

---

Creating Test Datasets

1. Navigate to AI Studio → Agent Lab
2. Select the Simulations tab, then click Create Dataset
3. Configure dataset:
   • Name: Descriptive identifier (e.g., “Support Agent Q4 Tests”)
   • Description: Purpose and scope
   • Agent: (Optional) Pre-bind to specific agent
4. Click Save

Dataset Configuration

Field	Type	Required	Description
name	string	Yes	Dataset identifier
description	text	No	Purpose description
agent	UUID	No	Default agent (can override at runtime)

---

Defining Test Cases

Test Case Types

Type	Use Case	Evaluation
answer	Text response validation	LLM compares to expected answer
tool_call	Tool usage validation	Checks tool name and arguments

Single-Turn Test Cases

Single-turn tests validate a single question-and-answer exchange.

Answer Type

For validating text responses:

1. In the dataset, click Add Test Case
2. Set Expected Type to “Answer”
3. Configure the test:
   • Question: The user message to test (e.g., “What are your business hours?”)
   • Expected Answer: The expected response content
4. Click Save

Note: The expected answer doesn’t need to be exact. The LLM evaluator checks semantic similarity.

Tool Call Type

For validating tool usage:

1. In the dataset, click Add Test Case
2. Set Expected Type to “Tool Call”
3. Configure the test:
   • Question: The user message that should trigger the tool
   • Expected Tool Name: The tool that should be called (e.g., “ago_ticketing”)
   • Expected Tool Arguments: (Optional) Expected argument values
4. Click Save

Multi-Turn Test Cases

Multi-turn tests validate a full conversation flow across multiple exchanges. This is useful for testing workflows that require back-and-forth interactions, such as collecting information or guiding users through a process.

1. In the dataset, click Add Test Case
2. Toggle from Single Turn to Multi-Turn
3. For each conversation step:
   • User Message: What the user says at this step
   • Expected Type: “Answer” or “Tool Call”
   • Expected Answer / Tool: The expected outcome for this step
4. Click Add Step to add more conversation turns
5. Click Save

All steps in a multi-turn test run in sequence within the same conversation thread, so the agent retains context from previous messages.

Generating Test Cases with AI

Instead of writing every test case by hand, you can generate them from the agent’s knowledge base:

1. Open the dataset and go to the Test Cases tab
2. Click Generate with AI
3. Pick the agent, the generation mode, the number of test cases (1–50), and the maximum number of user messages per conversation (1–10)
4. Click Generate with AI in the dialog

Two generation modes are available:

• From knowledge documents (default) — questions cover the topics found in the agent’s knowledge documents, phrased the way real users write.
• From a user profile — you describe the type of user and how they behave (for example, “a new payroll manager, in a hurry, who asks short and sometimes vague questions about leave management”). Questions are written the way that user would ask them, and expected answers stay grounded in the knowledge documents.

Generated test cases are multi-turn conversations by default: each one opens with a question and follows up the way a real user would (clarifications, edge cases, related details), up to the message limit you set. Set the limit to 1 to generate single-question test cases instead. Generation also reads the agent’s prompt, so the conversations exercise the rules and scenarios the agent is instructed to handle.

Generation runs in the background and works in small batches: each batch avoids repeating the questions generated before it (and those already in the dataset), so the test cases stay varied. They appear in the list after a few seconds. Each generated test case is an “answer” type, so you can run it right away or edit it first.

Generation reads the selected agent’s knowledge documents (or the whole knowledge base when the agent has no sources assigned). If no documents with readable content are found, no test cases are created.

Test Case Fields

Field	Type	Required	Description
dataset_id	UUID	Yes	Parent dataset
question	text	Yes	User message to test (single-turn)
expected_type	string	Yes	”answer”, “tool_call”, or “agent_routing”
expected_answer	text	For answer type	Expected response content
expected_tool_name	string	For tool_call type	Tool that should be called
expected_tool_args	JSON	No	Expected tool arguments
metadata	JSON	No	Additional test metadata
order	integer	No	Execution order (default: 0)
persona	Persona	No	Override persona for this test case

---

Personas

Personas let you customize test behavior by simulating different user profiles. Each persona can override the agent, adjust LLM temperature, and include behavior notes that influence how tests are evaluated.

Creating a Persona

1. In the dataset builder, navigate to the Personas section
2. Click Create Persona
3. Configure:
   • Name: Descriptive label (e.g., “Premium Customer”, “New User”)
   • Role: Short role description
   • Behavior Notes: Free-text instructions describing how this persona behaves
   • Override Agent: (Optional) Use a different agent for tests with this persona
   • LLM Temperature: (Optional) Override the LLM temperature
   • API Data Overrides: (Optional) Mock API data source responses during simulation
4. Click Save

API Data Overrides

When a persona has API data overrides configured, simulation tests using that persona will return the mocked data instead of making real API calls. This lets you test how your agent behaves with specific API responses without modifying your actual data sources.

To add API overrides to a persona:

1. In the persona creation dialog, click Add API Override
2. Select the API data source to mock from the dropdown
3. Enter the JSON response the source should return during simulation
4. Add multiple overrides for different data sources as needed

The mock data still respects the data source’s response data path and excluded paths settings, so you can provide the full API response structure and let the existing filters apply.

Using Personas

• Set a Default Persona on the dataset to apply it to all test cases automatically
• Override the persona on individual test cases when needed
• Test cases without a persona use the dataset’s default persona (if set)

---

Running Simulations

1. Navigate to AI Studio → Agent Lab
2. Select the Simulations tab, then select the dataset to run
3. Click Run Simulation
4. Select target agent (if not pre-bound)
5. Optionally check Detect knowledge gaps (see below)
6. Monitor progress in real-time

Detecting Knowledge Gaps from a Run

Check Detect knowledge gaps when starting a run to turn failed tests into actionable reports. When the run completes, AGO analyzes the conversations behind failed answer tests and files a Knowledge Update Request for each gap it finds, with the source “Simulation lab”. Each request includes the question, what was missing, and suggested edits to your knowledge base.

Combined with AI-generated test cases, this gives you a full audit loop: generate questions from your knowledge base, run them against the agent, and review the gap reports for everything the agent could not answer correctly.

Tool call and agent routing failures are not analyzed — only answer-type tests can reveal knowledge gaps.

Monitoring Progress

The simulation dashboard shows:

• Total Tests: Number of test cases in the dataset
• Passed Tests: Tests that met expectations
• Failed Tests: Tests that didn’t meet expectations
• Success Rate: Percentage of passed tests
• Duration: Time since simulation started

Run Statuses

Status	Description
pending	Run created, not started
running	Tests in progress
completed	All tests finished
failed	Run encountered error

---

Understanding Results

Viewing Result Details

Click on any test result to see:

Field	Description
Question	The test case question
Expected Answer	What the agent should respond
Actual Response	What the agent actually responded
Passed	Whether the test passed
Score	Overall evaluation score (0-1)
Reasoning	LLM explanation of the evaluation
Execution Time	How long the test took

Evaluation Criteria

The LLM evaluator scores responses on:

Criterion	Description
Accuracy	Correctness of information
Completeness	Covers all expected points
Relevance	Directly addresses the question

Success Rate Calculation

success_rate = passed_tests / total_tests × 100

---

Best Practices

Creating Effective Test Cases

1. Cover Core Functionality

   • Include the most common user questions
   • Test key workflows from start to finish

2. Test Edge Cases

   • Ambiguous questions
   • Out-of-scope requests
   • Multiple intents in one message

3. Validate Tool Usage

   • Test escalation scenarios
   • Verify correct tool selection
   • Check argument accuracy

4. Use Realistic Questions

   • Base tests on actual user queries
   • Include typos and variations

Organizing Datasets

Dataset Pattern	Use Case
By Feature	”Billing Tests”, “Technical Tests”
By Criticality	”Core Tests”, “Edge Cases”
By Release	”v2.1 Regression Tests”

Running Simulations

• Before Deployment: Run full suite before any changes
• After Prompt Changes: Validate no regressions
• After Model Updates: Test new model configurations
• Weekly: Scheduled regression testing

Interpreting Results

Success Rate	Action
95%+	Good to deploy
80-95%	Review failures, fix critical issues
Below 80%	Investigate root cause before deploying

---

API Reference

For programmatic simulation management, see the Public API v1 Reference.

---

Troubleshooting

Issue: Tests failing unexpectedly

Problem: Tests that should pass are marked as failed.

Solutions:

1. Review expected answer - may be too specific
2. Check LLM evaluation reasoning for insights
3. Adjust expected answer to be more semantic
4. Verify agent has access to required knowledge

Issue: Tool call not detected

Problem: Agent uses correct tool but test fails.

Solutions:

1. Check exact tool name spelling
2. Verify expected_tool_args are not too strict
3. Review actual_tool_calls in results
4. Check if tool was called with different argument names

Issue: Simulation runs too slowly

Problem: Large test suites take very long.

Solutions:

1. Run tests in parallel (configured automatically)
2. Break large datasets into smaller focused sets
3. Use faster LLM for non-critical tests
4. Run full test suites only before releases

Issue: Inconsistent results

Problem: Same test passes sometimes and fails others.

Solutions:

1. Lower agent temperature for consistency
2. Make expected answers more flexible
3. Add multiple acceptable variations
4. Check for time-sensitive content in responses

---

Related

• AI Debugger — Debug agent responses
• Agents — Agent configuration reference
• Monitor and Improve Performance — Analytics workflow