> ## Documentation Index
> Fetch the complete documentation index at: https://ago.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Thread Evaluator

> Automatically evaluate conversation quality using LLM-based analysis

Automatically evaluate conversation quality using LLM-based analysis. The Thread Evaluator provides objective quality assessments across multiple dimensions with actionable improvement suggestions.

<Frame caption="Thread Evaluator - Custom Instructions">
  <img src="https://mintcdn.com/ago/nVwBOovwlRSHxKDS/features/images/thread-evaluator-instructions.webp?fit=max&auto=format&n=nVwBOovwlRSHxKDS&q=85&s=a324021dbd08350183be94dcd6e6ff34" alt="Thread Evaluator settings page showing custom evaluation guidelines text editor" width="1280" height="800" data-path="features/images/thread-evaluator-instructions.webp" />
</Frame>

## Overview

The Thread Evaluator uses AI to analyze completed conversations and generate:

* **CX Score** - Overall quality score (0-100)
* **Dimension scores** - Detailed scoring across 5 quality dimensions
* **Review flags** - Identifies conversations needing human review
* **Improvement suggestions** - Actionable feedback for agent improvement

## Prerequisites

### System Requirements

* AGO platform deployed and accessible
* Staff-level access to configure evaluator

### Knowledge Required

* Understanding of customer experience metrics
* Familiarity with quality assessment concepts

### Configuration Required

* Thread Evaluator enabled in tenant settings
* LLM model configured (uses default or dedicated evaluation LLM)

***

## Enabling Thread Evaluator

### Via Admin Interface

1. Navigate to **Settings** → **Thread Evaluation** → **Thread Evaluator**
2. Toggle **Enable**
3. Configure CX Score weights (optional)

<Note>
  Thread Evaluator can also be configured via API. See the [Public API v1 Reference](/api/public-api-v1) for endpoint details.
</Note>

## How It Works

1. **Thread completion** - Conversation ends or times out
2. **Evaluation trigger** - Daily batch job or manual evaluation
3. **Prompt building** - Combines generic + custom + criteria instructions
4. **LLM analysis** - Structured evaluation with reasoning
5. **Score storage** - Results saved to thread record

<Note>
  The evaluator runs as a daily scheduled task, analyzing conversations from the previous day.
</Note>

***

## Quality Dimensions

The CX Score is calculated from five weighted dimensions:

| Dimension              | Default Weight | What It Measures                                                 |
| ---------------------- | -------------- | ---------------------------------------------------------------- |
| **Resolution Quality** | 25%            | Did the agent fully address the request within its capabilities? |
| **Response Accuracy**  | 20%            | Were responses consistent and free of contradictions?            |
| **Tone & Empathy**     | 20%            | Did the tone match the agent's configured guidelines?            |
| **Efficiency**         | 20%            | Was the conversation handled without unnecessary back-and-forth? |
| **Customer Effort**    | 15%            | How easy was it for the customer to get their answer?            |

### Score Ranges

| Score  | Rating        | Interpretation                                      |
| ------ | ------------- | --------------------------------------------------- |
| 90-100 | Exceptional   | Exceeds expectations in all areas                   |
| 70-89  | Good          | Meets expectations with minor improvements possible |
| 50-69  | Acceptable    | Functional but notable room for improvement         |
| 30-49  | Below Average | Significant issues requiring attention              |
| 0-29   | Poor          | Critical problems, needs immediate review           |

***

## Configuration

### Access Settings

Navigate to **Settings** → **Thread Evaluation** → **Thread Evaluator**

### Custom Instructions

Add organization-specific evaluation guidelines:

```markdown theme={null}
## Our Evaluation Standards

- Prioritize product knowledge accuracy
- Value empathetic responses to frustrated customers
- Consider regional language variations acceptable
- Flag any pricing or policy discussions for review
```

<Tip>
  Use custom instructions to align evaluations with your organization's specific quality standards and brand voice.
</Tip>

### Adjust Dimension Weights

<Frame caption="Thread Evaluator - CX Score Dimension Weights">
  <img src="https://mintcdn.com/ago/nVwBOovwlRSHxKDS/features/images/thread-evaluator-weights.webp?fit=max&auto=format&n=nVwBOovwlRSHxKDS&q=85&s=98effaf61d9af51d6732318761b8c627" alt="Thread Evaluator dimension weight sliders showing Resolution Quality, Response Accuracy, Tone & Empathy, Efficiency, and Customer Effort percentages" width="1280" height="800" data-path="features/images/thread-evaluator-weights.webp" />
</Frame>

Customize how much each dimension contributes to the overall score:

1. Use sliders to adjust each dimension (0-50%)
2. **Weights must sum to 100%**
3. Click **Save** to apply changes

**Example customization:**

* Increase Resolution Quality to 35% for support-focused agents
* Increase Tone to 30% for customer service agents
* Reduce Efficiency weight for complex technical support

### Minimum Score Thresholds

Configure minimum score thresholds to automatically flag conversations for review when scores fall below acceptable levels:

1. Navigate to **Settings** → **Thread Evaluation** → **Thread Evaluator**
2. Scroll to the **Minimum Score Thresholds** section
3. Adjust the **Overall CX Score Minimum** (default: 70)
4. Adjust per-dimension minimums (default: 50 each)
5. Click **Save Thresholds**

| Threshold                | Default | Description                                        |
| ------------------------ | ------- | -------------------------------------------------- |
| Overall CX Score Minimum | 70      | Conversations below this overall score are flagged |
| Resolution Quality       | 50      | Minimum score for this dimension                   |
| Response Accuracy        | 50      | Minimum score for this dimension                   |
| Tone & Empathy           | 50      | Minimum score for this dimension                   |
| Efficiency               | 50      | Minimum score for this dimension                   |
| Customer Effort          | 50      | Minimum score for this dimension                   |

<Tip>
  Start with the defaults and adjust based on your quality goals. Setting thresholds too high may flag too many conversations, while setting them too low may miss quality issues.
</Tip>

***

## Evaluation Prompt Structure

The evaluator uses a three-part prompt:

### 1. Generic Instructions (System)

Core evaluation principles - not editable:

* Expert evaluator perspective
* Independent dimension scoring
* Granular score differentiation

### 2. Tenant Instructions (Custom)

Your organization-specific guidelines - fully editable in admin

### 3. Scoring Criteria (Generated)

Detailed rubrics for each dimension based on current weights

Use **Preview Prompt** to see the complete evaluation prompt.

***

## Review Flags

Threads are flagged for human review when:

* The overall CX Score is below the minimum validation score (default: 70)
* Any individual dimension score is below its minimum threshold (default: 50 each)
* The LLM evaluator detects issues such as contradictory information or user frustration
* User explicitly requested human assistance

<Note>
  Score-based review flags are enforced at the system level and cannot be overridden by the LLM evaluator. If the LLM marks a conversation as "Validated" but the scores fall below the configured thresholds, the conversation is still flagged for review.
</Note>

### Managing Reviews

1. Go to **Chat History** in the sidebar
2. Filter by "Needs Review"
3. Review flagged conversations
4. Take action (training, knowledge gaps, escalation)

***

## Manual Evaluation

To evaluate a specific thread immediately:

1. Navigate to the thread in admin
2. Click **Evaluate** button
3. Wait for LLM analysis
4. View results in thread details

<Warning>
  Manual evaluation uses LLM credits. Use sparingly for specific investigations.
</Warning>

***

## Analytics Integration

Thread Evaluator data powers several analytics views:

### CX Score Distribution

* Breakdown of scores across ranges
* Identify quality trends

### CX Score by Agent

* Compare agent performance
* Identify training needs

### CX Score Trends

* Daily/weekly averages over time
* Track improvement initiatives

### Review Queue

* Count of threads needing review
* Prioritize quality assurance efforts

***

## Best Practices

### Custom Instructions

* **Be specific** - Vague guidelines produce inconsistent evaluations
* **Prioritize** - List most important criteria first
* **Provide examples** - Show what "good" and "bad" look like
* **Update regularly** - Evolve with your quality standards

### Weight Configuration

* **Start with defaults** - Observe results before customizing
* **Small adjustments** - Change weights by 5% increments
* **Align with goals** - Weights should reflect business priorities
* **Document changes** - Track why weights were modified

### Review Process

* **Sample reviews** - Manually verify LLM evaluation accuracy
* **Calibrate regularly** - Adjust custom instructions based on findings
* **Act on insights** - Use data to improve agent training and knowledge

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Threads not being evaluated">
    **Problem**: Conversations complete but no CX score appears.

    **Solutions**:

    1. Verify Thread Evaluator is enabled in settings
    2. Check if thread is marked as simulation (simulations are skipped)
    3. Verify the scheduled evaluation task is active
    4. Manually trigger evaluation to test
  </Accordion>

  <Accordion title="Scores seem inconsistent">
    **Problem**: Similar conversations receive very different scores.

    **Solutions**:

    1. Review custom instructions for clarity
    2. Check if weights are appropriately configured
    3. Examine reasoning in dimension scores for insights
    4. Consider adjusting evaluation LLM temperature (lower = more consistent)
  </Accordion>

  <Accordion title="Too many threads flagged for review">
    **Problem**: Excessive review flags overwhelming the team.

    **Solutions**:

    1. Adjust custom instructions to be more specific about review criteria
    2. Review flag triggers in the scoring criteria
    3. Consider which issues truly require human review
    4. Implement automated handling for common flag reasons
  </Accordion>

  <Accordion title="Evaluation takes too long">
    **Problem**: Large backlog of unevaluated threads.

    **Solutions**:

    1. Verify background processing has sufficient capacity
    2. Check evaluation LLM response times
    3. Consider using a faster LLM for evaluations
    4. Review if all threads need evaluation (filter criteria)
  </Accordion>
</AccordionGroup>

***

## Related Features

* [CX Score](/features/cx-score) - How the score is calculated and what each dimension measures
* [Evaluation Dashboard](/features/evaluation-dashboard) - View CX Score charts and trends
* [Conversations](/features/conversations-list) - Filter conversations by evaluation status and CX Score
* [Diagnose Quality Issues](/guides/diagnose-quality-issues) - Investigate low scores step by step
* [Monitor and Improve Performance](/guides/analytics-tutorial) - Full analytics tutorial