Thread Evaluator

Automatically evaluate conversation quality using LLM-based analysis

Automatically evaluate conversation quality using LLM-based analysis. The Thread Evaluator provides objective quality assessments across multiple dimensions with actionable improvement suggestions.

Thread Evaluator - Custom Instructions

Overview

The Thread Evaluator uses AI to analyze completed conversations and generate:

• CX Score - Overall quality score (0-100)
• Dimension scores - Detailed scoring across 5 quality dimensions
• Review flags - Identifies conversations needing human review
• Improvement suggestions - Actionable feedback for agent improvement

Prerequisites

System Requirements

• AGO platform deployed and accessible
• Staff-level access to configure evaluator

Knowledge Required

• Understanding of customer experience metrics
• Familiarity with quality assessment concepts

Configuration Required

• Thread Evaluator enabled in tenant settings
• LLM model configured (uses default or dedicated evaluation LLM)

---

Enabling Thread Evaluator

Via Admin Interface

1. Navigate to Settings → Thread Evaluation → Thread Evaluator
2. Toggle Enable
3. Configure CX Score weights (optional)

Thread Evaluator can also be configured via API. See the Public API v1 Reference for endpoint details.

How It Works

1. Thread completion - Conversation ends or times out
2. Evaluation trigger - Daily batch job or manual evaluation
3. Prompt building - Combines generic + custom + criteria instructions
4. LLM analysis - Structured evaluation with reasoning
5. Score storage - Results saved to thread record

The evaluator runs as a daily scheduled task, analyzing conversations from the previous day.

---

Quality Dimensions

The CX Score is calculated from five weighted dimensions:

Dimension	Default Weight	What It Measures
Resolution Quality	25%	Did the agent fully address the request within its capabilities?
Response Accuracy	20%	Were responses consistent and free of contradictions?
Tone & Empathy	20%	Did the tone match the agent’s configured guidelines?
Efficiency	20%	Was the conversation handled without unnecessary back-and-forth?
Customer Effort	15%	How easy was it for the customer to get their answer?

Score Ranges

Score	Rating	Interpretation
90-100	Exceptional	Exceeds expectations in all areas
70-89	Good	Meets expectations with minor improvements possible
50-69	Acceptable	Functional but notable room for improvement
30-49	Below Average	Significant issues requiring attention
0-29	Poor	Critical problems, needs immediate review

---

Configuration

Access Settings

Navigate to Settings → Thread Evaluation → Thread Evaluator

Custom Instructions

Add organization-specific evaluation guidelines:

## Our Evaluation Standards

- Prioritize product knowledge accuracy
- Value empathetic responses to frustrated customers
- Consider regional language variations acceptable
- Flag any pricing or policy discussions for review

Use custom instructions to align evaluations with your organization’s specific quality standards and brand voice.

Adjust Dimension Weights

Thread Evaluator - CX Score Dimension Weights

Customize how much each dimension contributes to the overall score:

1. Use sliders to adjust each dimension (0-50%)
2. Weights must sum to 100%
3. Click Save to apply changes

Example customization:

• Increase Resolution Quality to 35% for support-focused agents
• Increase Tone to 30% for customer service agents
• Reduce Efficiency weight for complex technical support

Minimum Score Thresholds

Configure minimum score thresholds to automatically flag conversations for review when scores fall below acceptable levels:

1. Navigate to Settings → Thread Evaluation → Thread Evaluator
2. Scroll to the Minimum Score Thresholds section
3. Adjust the Overall CX Score Minimum (default: 70)
4. Adjust per-dimension minimums (default: 50 each)
5. Click Save Thresholds

Threshold	Default	Description
Overall CX Score Minimum	70	Conversations below this overall score are flagged
Resolution Quality	50	Minimum score for this dimension
Response Accuracy	50	Minimum score for this dimension
Tone & Empathy	50	Minimum score for this dimension
Efficiency	50	Minimum score for this dimension
Customer Effort	50	Minimum score for this dimension

Start with the defaults and adjust based on your quality goals. Setting thresholds too high may flag too many conversations, while setting them too low may miss quality issues.

---

Evaluation Prompt Structure

The evaluator uses a three-part prompt:

1. Generic Instructions (System)

Core evaluation principles - not editable:

• Expert evaluator perspective
• Independent dimension scoring
• Granular score differentiation

2. Tenant Instructions (Custom)

Your organization-specific guidelines - fully editable in admin

3. Scoring Criteria (Generated)

Detailed rubrics for each dimension based on current weights

Use Preview Prompt to see the complete evaluation prompt.

---

Review Flags

Threads are flagged for human review when:

• The overall CX Score is below the minimum validation score (default: 70)
• Any individual dimension score is below its minimum threshold (default: 50 each)
• The LLM evaluator detects issues such as contradictory information or user frustration
• User explicitly requested human assistance

Score-based review flags are enforced at the system level and cannot be overridden by the LLM evaluator. If the LLM marks a conversation as “Validated” but the scores fall below the configured thresholds, the conversation is still flagged for review.

Managing Reviews

1. Go to Chat History in the sidebar
2. Filter by “Needs Review”
3. Review flagged conversations
4. Take action (training, knowledge gaps, escalation)

---

Manual Evaluation

To evaluate a specific thread immediately:

1. Navigate to the thread in admin
2. Click Evaluate button
3. Wait for LLM analysis
4. View results in thread details

Manual evaluation uses LLM credits. Use sparingly for specific investigations.

---

Analytics Integration

Thread Evaluator data powers several analytics views:

CX Score Distribution

• Breakdown of scores across ranges
• Identify quality trends

CX Score by Agent

• Compare agent performance
• Identify training needs

CX Score Trends

• Daily/weekly averages over time
• Track improvement initiatives

Review Queue

• Count of threads needing review
• Prioritize quality assurance efforts

---

Best Practices

Custom Instructions

• Be specific - Vague guidelines produce inconsistent evaluations
• Prioritize - List most important criteria first
• Provide examples - Show what “good” and “bad” look like
• Update regularly - Evolve with your quality standards

Weight Configuration

• Start with defaults - Observe results before customizing
• Small adjustments - Change weights by 5% increments
• Align with goals - Weights should reflect business priorities
• Document changes - Track why weights were modified

Review Process

• Sample reviews - Manually verify LLM evaluation accuracy
• Calibrate regularly - Adjust custom instructions based on findings
• Act on insights - Use data to improve agent training and knowledge

---

Troubleshooting

Threads not being evaluated

Problem: Conversations complete but no CX score appears.

Solutions:

1. Verify Thread Evaluator is enabled in settings
2. Check if thread is marked as simulation (simulations are skipped)
3. Verify the scheduled evaluation task is active
4. Manually trigger evaluation to test

Scores seem inconsistent

Problem: Similar conversations receive very different scores.

Solutions:

1. Review custom instructions for clarity
2. Check if weights are appropriately configured
3. Examine reasoning in dimension scores for insights
4. Consider adjusting evaluation LLM temperature (lower = more consistent)

Too many threads flagged for review

Problem: Excessive review flags overwhelming the team.

Solutions:

1. Adjust custom instructions to be more specific about review criteria
2. Review flag triggers in the scoring criteria
3. Consider which issues truly require human review
4. Implement automated handling for common flag reasons

Evaluation takes too long

Problem: Large backlog of unevaluated threads.

Solutions:

1. Verify background processing has sufficient capacity
2. Check evaluation LLM response times
3. Consider using a faster LLM for evaluations
4. Review if all threads need evaluation (filter criteria)

---

Related Features

• CX Score - How the score is calculated and what each dimension measures
• Evaluation Dashboard - View CX Score charts and trends
• Conversations - Filter conversations by evaluation status and CX Score
• Diagnose Quality Issues - Investigate low scores step by step
• Monitor and Improve Performance - Full analytics tutorial