Simulation API - AGO Documentation

Run agent quality simulations, manage test datasets, and validate agent behavior through the API. Also available as MCP tools for AI agent workflows.

Authentication

All endpoints require an API key with agents_config:read or agents_config:write scope.

curl -H "X-API-Key: ago_v1_sk_your_key_here" \
  https://your-instance.ago.ai/api/v1/simulation/datasets

Quick Test

Run a single ad-hoc test against an agent. Returns the response synchronously (may take 10-30 seconds).

POST /api/v1/simulation/quick-test

{
  "agent_id": "agent-uuid",
  "question": "What are your business hours?"
}

Response:

{
  "object": "simulation_quick_test",
  "response": "We're open Monday through Friday, 9 AM to 5 PM EST.",
  "tool_calls": null,
  "execution_time_ms": 2450,
  "error": null,
  "routed_agent_id": null,
  "routed_agent_name": null
}

Scope: agents_config:write

Datasets

Datasets group related test cases for batch simulation runs.

List Datasets

GET /api/v1/simulation/datasets?page=1&page_size=50

Response:

{
  "object": "list",
  "data": [
    {
      "object": "simulation_dataset",
      "id": "dataset-uuid",
      "name": "Support Agent Tests",
      "description": "Core functionality tests",
      "agent_id": "agent-uuid",
      "agent_name": "Support Agent",
      "default_persona_id": null,
      "test_case_count": 25,
      "created_at": "2026-01-15T10:30:00+00:00",
      "updated_at": "2026-01-15T10:30:00+00:00"
    }
  ],
  "has_more": false,
  "total": 1
}

Scope: agents_config:read

Create Dataset

POST /api/v1/simulation/datasets

{
  "name": "Support Agent Tests",
  "description": "Core functionality tests for the support agent",
  "agent_id": "agent-uuid",
  "default_persona_id": "persona-uuid"
}

Field	Type	Required	Description
name	string	Yes	Dataset name
description	string	No	Purpose description
agent_id	string	No	Default agent for test runs
default_persona_id	string	No	Default persona for test cases

Scope: agents_config:write

Get, Update, Delete Dataset

GET    /api/v1/simulation/datasets/{dataset_id}
PUT    /api/v1/simulation/datasets/{dataset_id}
DELETE /api/v1/simulation/datasets/{dataset_id}

Deleting a dataset removes all its test cases and runs.

Test Cases

Each test case defines a question and the expected agent behavior.

Expected Types

Type	What It Tests	Required Fields
`answer`	Agent gives correct text response	`expected_answer`
`tool_call`	Agent calls the right tool	`expected_tool_name`, optionally `expected_tool_args`
`agent_routing`	Router selects the right agent	`expected_agent_id`

Create Test Case

POST /api/v1/simulation/datasets/{dataset_id}/test-cases

Answer test:

{
  "question": "What are your business hours?",
  "expected_type": "answer",
  "expected_answer": "Monday to Friday, 9 AM to 5 PM EST."
}

Tool call test:

{
  "question": "I need to speak with a human",
  "expected_type": "tool_call",
  "expected_tool_name": "ago_ticketing",
  "expected_tool_args": { "priority": "high" }
}

Multi-turn test (conversation with multiple exchanges):

{
  "question": "Multi-turn: collect address",
  "expected_type": "answer",
  "steps": [
    {
      "step_order": 1,
      "user_message": "I want to update my address",
      "expected_type": "answer",
      "expected_answer": "What is your new address?"
    },
    {
      "step_order": 2,
      "user_message": "123 Main St, Springfield",
      "expected_type": "tool_call",
      "expected_tool_name": "update_address"
    }
  ]
}

Steps run in sequence within the same conversation thread, so the agent retains context. Scope: agents_config:write

List Test Cases

GET /api/v1/simulation/datasets/{dataset_id}/test-cases?page=1&page_size=50

Scope: agents_config:read

Bulk Create

POST /api/v1/simulation/datasets/{dataset_id}/test-cases/bulk

{
  "test_cases": [
    { "question": "Question 1", "expected_type": "answer", "expected_answer": "Answer 1" },
    { "question": "Question 2", "expected_type": "answer", "expected_answer": "Answer 2" }
  ]
}

Import from Conversation

Extract a test case from an existing conversation thread:

POST /api/v1/simulation/datasets/{dataset_id}/import/conversation/{thread_id}

Uses the last user message and assistant response from the thread. Scope: agents_config:write

Personas

Personas simulate different user profiles during testing. They can override the agent, LLM temperature, available tools, and user attributes.

Create Persona

POST /api/v1/simulation/personas

{
  "name": "Premium Customer",
  "description": "Long-time customer with premium subscription",
  "role": "Premium",
  "behavior_notes": "Expects fast, personalized responses",
  "agent_id": "agent-uuid",
  "custom_llm_temperature": 0.3,
  "enabled_tool_ids": ["tool-uuid-1", "tool-uuid-2"],
  "user_overrides": {
    "email": "premium@example.com",
    "additional_info": { "plan": "premium" }
  }
}

Field	Type	Required	Description
name	string	Yes	Persona name
description	string	No	Description
role	string	No	Short role label
behavior_notes	string	No	Instructions for evaluation
agent_id	string	No	Override agent for this persona
custom_llm_temperature	number	No	Override LLM temperature
enabled_tool_ids	array	No	Restrict available tools
user_overrides	object	No	Mock user properties (email, additional_info, business_context, jwt_token)

Scope: agents_config:write

List, Get, Update, Delete Personas

GET    /api/v1/simulation/personas
GET    /api/v1/simulation/personas/{persona_id}
PUT    /api/v1/simulation/personas/{persona_id}
DELETE /api/v1/simulation/personas/{persona_id}

Simulation Runs

Start a Run

POST /api/v1/simulation/datasets/{dataset_id}/runs

{
  "agent_id": "agent-uuid"
}

Runs execute asynchronously. The response returns immediately with status: "pending". Scope: agents_config:write

Poll Run Status

GET /api/v1/simulation/runs/{run_id}/status

{
  "object": "simulation_run_status",
  "status": "running",
  "total_tests": 25,
  "completed_tests": 12,
  "progress_percentage": 48.0
}

Poll this endpoint until status is "completed" or "failed".

Status	Description
pending	Created, waiting to start
running	Tests in progress
completed	All tests finished
failed	Run encountered an error

Scope: agents_config:read

Get Run Details

GET /api/v1/simulation/runs/{run_id}

{
  "object": "simulation_run",
  "id": "run-uuid",
  "dataset_id": "dataset-uuid",
  "dataset_name": "Support Agent Tests",
  "agent_id": "agent-uuid",
  "agent_name": "Support Agent",
  "status": "completed",
  "total_tests": 25,
  "passed_tests": 23,
  "failed_tests": 2,
  "success_rate": 92.0,
  "error_message": null,
  "started_at": "2026-01-15T10:30:00+00:00",
  "completed_at": "2026-01-15T10:45:00+00:00"
}

Get Run Results

GET /api/v1/simulation/runs/{run_id}/results?page=1&page_size=50

{
  "object": "list",
  "data": [
    {
      "object": "simulation_result",
      "id": "result-uuid",
      "test_case_id": "test-case-uuid",
      "conversation_step_id": null,
      "thread_id": "thread-uuid",
      "actual_response": "We're open Monday through Friday, 9 AM to 5 PM EST.",
      "actual_tool_calls": null,
      "actual_routed_agent_id": null,
      "actual_routed_agent_name": null,
      "passed": true,
      "llm_evaluation": {
        "passed": true,
        "score": 0.95,
        "reasoning": "Response accurately conveys business hours.",
        "criteria_scores": { "accuracy": 1.0, "completeness": 0.9, "relevance": 1.0 }
      },
      "evaluation_score": 0.95,
      "executed_at": "2026-01-15T10:32:00+00:00",
      "execution_time_ms": 1250
    }
  ],
  "has_more": true,
  "total": 25
}

List Runs, Delete Run

GET    /api/v1/simulation/datasets/{dataset_id}/runs
DELETE /api/v1/simulation/runs/{run_id}

MCP Tools

All simulation endpoints are available as MCP tools. Connect to the MCP server at /api/v1/mcp with an API key that has agents_config scopes.

Workflow: Run a Simulation via MCP

Start the run: Call start_simulation_run with dataset_id and agent_id
Poll for completion: Call get_simulation_run_status with run_id until status is "completed" or "failed"
Get results: Call get_simulation_run_results with run_id to see individual test results

Available MCP Tools

Tool	Description	Scope
`list_simulation_datasets`	List all datasets	read
`get_simulation_dataset`	Get dataset by ID	read
`create_simulation_dataset`	Create a dataset	write
`update_simulation_dataset`	Update a dataset	write
`delete_simulation_dataset`	Delete a dataset	write
`list_simulation_test_cases`	List test cases in a dataset	read
`get_simulation_test_case`	Get test case by ID	read
`create_simulation_test_case`	Create a test case	write
`update_simulation_test_case`	Update a test case	write
`delete_simulation_test_case`	Delete a test case	write
`bulk_create_simulation_test_cases`	Create multiple test cases	write
`import_simulation_from_conversation`	Create test case from conversation	write
`list_simulation_personas`	List personas	read
`get_simulation_persona`	Get persona by ID	read
`create_simulation_persona`	Create a persona	write
`update_simulation_persona`	Update a persona	write
`delete_simulation_persona`	Delete a persona	write
`start_simulation_run`	Start a simulation run	write
`list_simulation_runs`	List runs for a dataset	read
`get_simulation_run`	Get run details	read
`get_simulation_run_status`	Poll run progress	read
`get_simulation_run_results`	Get run results	read
`delete_simulation_run`	Delete a run	write
`run_simulation_quick_test`	Run a single test	write

Error Responses

Errors follow the standard API v1 format:

{
  "error": {
    "type": "invalid_request_error",
    "code": "not_found",
    "message": "Dataset not found with ID 'abc'.",
    "param": "dataset_id",
    "doc_url": "https://ago.mintlify.app/api/errors#not_found"
  }
}

Simulation Testing Overview - Testing concepts and UI guide
MCP Integration - How to set up MCP clients
Agents API - Agent management endpoints

​Authentication

​Quick Test

​Datasets

​List Datasets

​Create Dataset

​Get, Update, Delete Dataset

​Test Cases

​Expected Types

​Create Test Case

​List Test Cases

​Bulk Create

​Import from Conversation

​Personas

​Create Persona

​List, Get, Update, Delete Personas

​Simulation Runs

​Start a Run

​Poll Run Status

​Get Run Details

​Get Run Results

​List Runs, Delete Run

​MCP Tools

​Workflow: Run a Simulation via MCP

​Available MCP Tools

​Error Responses

​Related

Authentication

Quick Test

Datasets

List Datasets

Create Dataset

Get, Update, Delete Dataset

Test Cases

Expected Types

Create Test Case

List Test Cases

Bulk Create

Import from Conversation

Personas

Create Persona

List, Get, Update, Delete Personas

Simulation Runs

Start a Run

Poll Run Status

Get Run Details

Get Run Results

List Runs, Delete Run

MCP Tools

Workflow: Run a Simulation via MCP

Available MCP Tools

Error Responses

Related