AI Debugger

Troubleshoot agent behavior with visibility into retrieval and reasoning

The AI Debugger helps you understand and troubleshoot agent behavior by providing visibility into the retrieval and reasoning process. Use it to diagnose why agents give certain responses and improve knowledge retrieval.

Overview

The AI Debugger provides:

• Similarity Search Debugging: See which documents are retrieved for queries
• Relevance Scoring: Understand why certain documents rank higher
• Conversation Analysis: Investigate responses using conversation and tool-call context
• Query Analysis: Test retrieval without running full agent responses

Prerequisites

System Requirements

• AGO platform deployed and accessible
• Admin access to debugging tools

Knowledge Required

• Understanding of vector similarity search
• Familiarity with your knowledge base structure

---

Accessing the Debugger

Via Admin Interface

1. Navigate to AI Studio → Search Lab for similarity search debugging
2. Navigate to AI Studio → Agent Lab for conversation debugging

Debug Modes

Mode	Purpose
Similarity Search	Test how queries match documents
Conversation Debug	Analyze why an agent responded a certain way

---

Similarity Search Debugging

Test how queries match against your knowledge base without involving the full agent pipeline.

Search Lab - Test document retrieval

Running a Search Debug

1. Enter your test query (e.g., “How do I reset my password?”)
2. Select the agent to test against
3. Set Results to the number of documents to retrieve
4. Click Search

Understanding Results

Results display documents ranked by relevance:

Field	Description
Title	Document title
Source	Knowledge source name
Similarity	Cosine similarity score (-1 to 1; higher means a closer match)
Document Content	Document text returned with the search result

---

Conversation Debug

Analyze why an agent gave a specific response in an existing conversation.

Automatic analysis includes the tool calls associated with messages in the conversation excerpt: the tool or delegated agent, supplied arguments, execution status, and available result. This helps distinguish a failed action from an incorrect answer based on a successful tool result. Router clarification questions and reply options are included too.

Long tool inputs and results may be shortened, and stored result summaries are labelled as compressed. The analysis uses the first ten messages of the conversation. Superadmins can use Full Thread Debug View for calls outside that excerpt or for full payloads.

Editing a Suggested Prompt Update

When the automatic analysis suggests updating an agent prompt, click Apply to review the proposed changes. Use Enter fullscreen in the top-right corner of the dialog to expand the editor. You can also expand Current Prompt to consult the existing instructions.

Click Exit fullscreen or press Escape to return to the smaller dialog without losing your edits. Changes are only applied when you click Confirm & Apply.

Accessing Conversation Debug

1. Open a conversation in the administration interface
2. Add an evaluation comment describing the issue in the conversation sidebar, then click Self-improve
3. Choose Auto Analysis

For the detailed diagnostic panel, superadmins can open Full Thread Debug View from the conversation sidebar.

Suggested document changes

Open Self-improve from a conversation and choose Auto Analysis to investigate an incorrect response. When the analysis recommends updating a document, its suggested action shows the target document as a link and the proposed changes as text. Follow the link to review the document before choosing Apply.

For retrieval issues, the suggested action identifies the document containing the answer and shows the related question to add. When documentation is missing, it shows the proposed new document title and content.

Choose Improve Document to focus on one document. After analysis, review the improvement explanation, document link, and proposed changes before applying an action.

Debug Information

Section	Contents
Retrieved Documents	Documents used for context
Tool Calls	Tools invoked during response
LLM Request	Prompt sent to the model
Token Usage	Input/output token counts

Debug Panel Structure

Agent Lab - Debug conversation messages

The debug panel displays expandable sections:

• Retrieved Documents: List of documents with relevance scores
• Tool Calls: Any tools invoked during response generation
• System Prompt: The full prompt sent to the LLM (expandable)
• Token Usage: Input, output, and total token counts

---

Common Debugging Scenarios

Scenario 1: Agent Can’t Find Relevant Content

Symptom: Agent says “I don’t know” for documented topics.

Debug Steps:

1. Open AI Debugger → Similarity Search
2. Enter the user’s exact question
3. Review retrieved documents:
   • Are relevant docs in results?
   • What are their similarity scores?

Common Causes:

• Document not synced (check source status)
• Query wording doesn’t match content
• Chunks are too large (relevant content diluted)
• Wrong knowledge source assigned to agent

Scenario 2: Wrong Document Ranked Higher

Symptom: Irrelevant content appears before relevant content.

Debug Steps:

1. Run similarity search with the query
2. Review the similarity scores for each document

Solutions:

• Improve document titles/headers
• Adjust chunk size for better granularity
• Add more specific content

Scenario 3: Agent Uses Wrong Tool

Symptom: Agent creates ticket when it should answer.

Debug Steps:

1. Open conversation debug for the message
2. Check retrieved documents (empty = no context)
3. Review tool call details
4. Check tool descriptions in configuration

Solutions:

• Improve tool prompt/description
• Verify knowledge covers the topic
• Adjust tool trigger conditions in prompt

Scenario 4: Response Quality Varies

Symptom: Same question gets different quality answers.

Debug Steps:

1. Compare debug info across multiple instances
2. Check if different documents are retrieved
3. Review LLM temperature settings

Solutions:

• Lower temperature for consistency
• Pin critical documents to agent
• Improve document quality

---

API Reference

For programmatic debugging access, see the Public API v1 Reference.

---

Best Practices

Regular Debugging

• Weekly Reviews: Sample random conversations to check quality
• After Sync: Debug after major knowledge updates
• User Complaints: Debug reported issues immediately

Improving Retrieval

1. Test variations: Try different phrasings of the same question
2. Check coverage: Verify all topics have matching content
3. Monitor scores: Low similarity scores indicate gaps

Documentation

• Keep notes on common issues
• Document patterns that cause problems
• Share findings with content team

---

Troubleshooting

Issue: Debug panel loads slowly

Solutions:

1. Reduce top_k for faster results
2. Check system performance
3. Verify embedding service is responsive

Issue: No documents returned

Solutions:

1. Verify knowledge source is synced
2. Check agent has source assigned
3. Try broader query terms
4. Check embedding model is configured

Issue: Similarity scores all low

Solutions:

1. Review document quality
2. Check if content matches expected queries
3. Verify embedding model is appropriate
4. Consider different chunking strategy

---

Related

• Simulation Testing — Automated testing
• Agents — Agent configuration reference