Skip to main content

​
Overview

Action Advancement addresses the common pain points of unclear agent performance by measuring whether AI agents are actually helping users achieve their objectives rather than just providing responses. An assistant successfully advances a user’s goal when it:
  1. Provides a complete or partial answer to the user’s question
  2. Requests clarification or additional information to better understand the user’s needs
  3. Confirms that a requested action has been successfully completed
For an interaction to count as advancing the user’s goal, the assistant’s response must be:
  • Factually accurate
  • Directly addressing the user’s request
  • Consistent with any tool outputs used

​
Action Advancement at a glance

​
When to Use This Metric

​
When to Use This Metric

This metric shines when simple response quality metrics fall short, particularly for complex, multi-step interactions where progress toward goals matters more than individual response quality.
Agentic Workflows: When an AI agent must decide on actions and select appropriate tools.
Multi-step Tasks: When completing a user’s request requires multiple steps or decisions.
Tool-using Assistants: When evaluating if the assistant used available tools effectively.
Customer Service Agents: Resolving user issues through multi-step problem-solving.
Task-Oriented Assistants: Completing specific actions like booking flights or processing orders.
Research Assistants: Gathering and synthesizing information across multiple sources.
Creative Assistants: Understanding and building upon user requests iteratively.

​
Calculation method

If Action Advancement returns false, the assistant did not make progress on any user goal. Action Advancement is calculated by:
1

Model Request

Multiple evaluation requests are sent to an LLM evaluator to analyze the assistant’s progress toward user goals.
2

Prompt Engineering

A specialized chain-of-thought prompt guides the model to evaluate whether the assistant made progress on user goals based on the metric’s definition.
3

Evaluation Process

Each evaluation analyzes the interaction and produces both a detailed explanation and a binary judgment (yes/no) on goal advancement.
4

Result

The evaluator returns true when the assistant advances at least one user goal and false otherwise.
We display one of the generated explanations alongside the result, choosing one that aligns with the majority judgment.
This metric requires multiple LLM calls to compute, which may impact usage and billing.

​
How to improve Action Advancement scores

To improve Action Advancement scores, focus on ensuring your AI agents make meaningful progress toward user goals in every interaction.

​
Common issues and solutions

​
Best practices for optimization

  • Clear goal identification: Ensure your assistant can identify and prioritize user goals
  • Progressive disclosure: Break complex tasks into manageable steps
  • Tool integration: Make sure the assistant effectively uses available tools and APIs
  • Context awareness: Maintain conversation context to build on previous interactions

​
Comparison to other metrics

​
Best practices

To effectively implement and optimize Action Advancement in your AI systems, consider these key practices:

​
Track progress over time

Monitor Action Advancement scores across different versions of your agent to ensure improvements in task completion capabilities. This helps you identify whether your optimizations are actually improving goal advancement.

​
Analyze failure patterns

When Action Advancement scores are low, examine the specific steps where agents fail to make progress to identify systematic issues. Look for patterns in where agents get stuck or fail to advance user goals.

​
Combine with other metrics

Use Action Advancement alongside other agentic metrics to get a comprehensive view of your assistant’s effectiveness. This provides a more complete picture of your agent’s performance beyond just goal advancement.

​
Test edge cases

Create evaluation datasets that include complex, multi-step tasks to thoroughly assess your agent’s ability to advance user goals. This ensures your agent can handle challenging scenarios that require multiple steps.
When optimizing for Action Advancement, ensure you’re not sacrificing other important aspects like safety, factual accuracy, or user experience in pursuit of task completion.

​
Performance Benchmarks

We evaluated Action Advancement against human expert labels on an internal dataset of agentic conversation samples using top frontier models.

​
GPT-4.1 Classification Report

Benchmarks based on internal evaluation dataset. Performance may vary by use case.
If you would like to dive deeper or start implementing Action Advancement, check out the following resources:

​
Examples

  • Action Advancement Examples - Log in and explore the “Action Advancement” Log Stream in the “Preset Metric Examples” Project to see this metric in action.

​
How-to guides