MCPToolBench++¶

Overview¶

mcpbr run -c config.yaml --benchmark mcptoolbench -n 20

Overview¶

MCPToolBench++ is a benchmark specifically designed to evaluate how well AI agents use MCP (Model Context Protocol) tools. It tests the complete tool-use lifecycle across four key dimensions:

1. Tool Discovery: Understanding what MCP tools are available and their capabilities
2. Tool Selection: Choosing the appropriate tool(s) for a given task
3. Tool Invocation: Calling tools with correct parameters matching their schemas
4. Result Interpretation: Understanding and correctly using tool outputs

The benchmark covers 45+ categories spanning diverse tool-use scenarios:

• Browser: Web browsing and page interaction
• Finance: Financial data retrieval and calculations
• Code Analysis: Source code inspection and manipulation
• Database: Query and data management operations
• File Management: File system operations
• Communication: Email, messaging, and notification tools
• Weather: Weather data retrieval
• Search: Information retrieval and search operations
• And many more

Each task provides a query, a set of available MCP tools with their schemas, and a ground truth sequence of tool calls that correctly completes the task.

What It Measures¶

MCPToolBench++ evaluates MCP-specific tool use capabilities:

• Tool schema comprehension: Understanding tool definitions, parameter types, required vs. optional fields, and return value formats
• Tool selection accuracy: Identifying the correct tool(s) from a set of available options based on the task description
• Parameter precision: Providing the exact parameter names and values expected by the tool schema
• Sequential reasoning: For multi-step tasks, determining the correct order of tool calls and passing results between them
• Efficiency: Completing tasks without excessive exploratory or redundant tool calls

MCPToolBench++ does not test:

• The actual execution of tool calls (evaluation is based on the call structure, not results)
• Code generation or debugging
• Free-form reasoning or knowledge retrieval
• Tasks that require tools not listed in the task's available tool set

Task Structure¶

Each MCPToolBench++ task contains the following fields:

The agent receives the query, task metadata, and available tools, then must select and invoke the correct tools with proper parameters.

Example Task (Single-Step)¶

Category: finance
Task Type: single-step tool call
Available Tools: get_stock_price, get_exchange_rate, calculate_tax

Task:
What is the current stock price of Apple Inc. (AAPL)?

Expected Tool Call:
  - name: get_stock_price
    parameters:
      symbol: "AAPL"

Example Task (Multi-Step)¶

Category: finance
Task Type: multi-step tool call
Available Tools: get_stock_price, get_exchange_rate, calculate_tax

Task:
Get the current stock price of AAPL in USD, then convert it to EUR.

Expected Tool Calls:
  1. name: get_stock_price
     parameters: { symbol: "AAPL" }
  2. name: get_exchange_rate
     parameters: { from: "USD", to: "EUR" }

Configuration¶

Basic Configuration¶

# Run MCPToolBench++ with default settings
mcpbr run -c config.yaml --benchmark mcptoolbench

# Run a small sample
mcpbr run -c config.yaml --benchmark mcptoolbench -n 20

# Filter by difficulty (single vs multi-step)
mcpbr run -c config.yaml --benchmark mcptoolbench --filter-difficulty easy
mcpbr run -c config.yaml --benchmark mcptoolbench --filter-difficulty hard

# Filter by category
mcpbr run -c config.yaml --benchmark mcptoolbench --filter-category browser
mcpbr run -c config.yaml --benchmark mcptoolbench --filter-category finance

# Combine filters
mcpbr run -c config.yaml --benchmark mcptoolbench \
  --filter-difficulty easy --filter-category browser

# Run with verbose output and save results
mcpbr run -c config.yaml --benchmark mcptoolbench -n 50 -v -o results.json

benchmark: "mcptoolbench"
sample_size: 10
timeout_seconds: 300

mcp_server:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]

model: "sonnet"

# Optional: filter by difficulty and category
filter_difficulty:
  - "easy"    # single-step tasks
filter_category:
  - "browser"
  - "finance"

Advanced Options¶

Difficulty Filtering¶

MCPToolBench++ maps difficulty labels to the call_type field:

Category Filtering¶

Filter by any of the 45+ task categories. Category matching is case-insensitive. Common categories include:

Configuration for Multi-Step Evaluation¶

benchmark: "mcptoolbench"
sample_size: 20
timeout_seconds: 300
max_iterations: 20

filter_difficulty:
  - "hard"     # Multi-step tasks only

model: "sonnet"

agent_prompt: |
  {problem_statement}

  Output your tool calls as a JSON array. Each element should have:
  - "name": the tool name
  - "parameters": an object with parameter key-value pairs

  Example:
  [{"name": "tool_name", "parameters": {"param1": "value1"}}]

Configuration for Category-Specific Testing¶

benchmark: "mcptoolbench"
sample_size: 30
timeout_seconds: 180

filter_category:
  - "finance"
  - "search"
  - "weather"

model: "sonnet"

Evaluation Methodology¶

MCPToolBench++ evaluation compares the agent's tool calls against the ground truth using three metrics:

1. Tool Selection Accuracy¶

Measures how many of the expected tools the agent correctly selected:

tool_selection_accuracy = correct_tools / total_expected_tools

A tool is considered correctly selected if the agent made at least one call to a tool with the same name as an expected tool. Each expected tool is counted at most once.

2. Parameter Accuracy¶

Measures how correctly the agent filled in tool parameters:

parameter_accuracy = correct_parameters / total_expected_parameters

For each correctly selected tool, the agent's parameters are compared against the ground truth. A parameter is correct if both the name and value match exactly.

3. Sequence Match¶

A boolean indicating whether the agent called the exact same tools in the exact same order as the ground truth.

Overall Resolution¶

A task is resolved when all three conditions are met:

resolved = (tool_selection_accuracy >= 0.8)
       AND (parameter_accuracy >= 0.7)
       AND (agent_call_count <= expected_call_count * 1.5)

The thresholds allow for minor variations:

• 0.8 tool selection: Allows missing up to 20% of expected tools (e.g., 4 out of 5 correct)
• 0.7 parameter accuracy: Allows up to 30% parameter errors (e.g., minor formatting differences)
• 1.5x call limit: Allows some extra exploratory tool calls but prevents excessive flailing

Tool Call Extraction¶

The evaluation attempts to extract tool calls from the agent's response in two ways:

1. JSON parsing: If the response is valid JSON (a list of tool calls or an object with a tool_calls key), the calls are extracted directly.
2. Text parsing: If JSON parsing fails, the evaluation falls back to pattern matching in the response text.

Interpreting Results¶

Key Metrics¶

What Good Results Look Like¶

Common Failure Patterns¶

Example Output¶

Successful Evaluation¶

{
  "resolved": true,
  "tool_selection_accuracy": 1.0,
  "parameter_accuracy": 0.85,
  "sequence_match": true,
  "details": "Tool selection: 100.0%, Parameter accuracy: 85.0%, Sequence match: True"
}

Failed Evaluation (Low Tool Selection)¶

{
  "resolved": false,
  "tool_selection_accuracy": 0.5,
  "parameter_accuracy": 0.9,
  "sequence_match": false,
  "details": "Tool selection: 50.0%, Parameter accuracy: 90.0%, Sequence match: False"
}

Failed Evaluation (No Tool Calls Extracted)¶

{
  "resolved": false,
  "tool_selection_accuracy": 0.0,
  "parameter_accuracy": 0.0,
  "sequence_match": false,
  "details": "Agent made no tool calls"
}

Failed Evaluation (No Ground Truth)¶

{
  "resolved": false,
  "error": "No ground truth function calls provided for evaluation"
}

Best Practices¶

Recommended Workflow¶

1. Start with single-step tasks (--filter-difficulty easy) to establish baseline tool selection and invocation capability
2. Test category-by-category to identify which tool types your MCP server handles well
3. Progress to multi-step tasks once single-step accuracy exceeds 70%
4. Track all three metrics (tool selection, parameter accuracy, sequence match) separately for targeted optimization
5. Test with your actual MCP server -- MCPToolBench++ is most valuable when evaluating your real tool configuration

Performance Tips¶

• Use structured output prompts: MCPToolBench++ evaluation depends on extracting tool calls from the agent's response. JSON-formatted output is most reliable.
• Provide tool schemas: Ensure your agent has access to the full MCP tool definitions. The mcp_tools_dict field in each task contains the complete schemas.
• Increase timeout for multi-step tasks: Multi-step tasks require sequential tool calls and result interpretation. Use at least 300 seconds.
• Monitor all three metrics: Track tool selection accuracy, parameter accuracy, and sequence match separately. Each reveals different aspects of tool-use capability.

Cost Optimization¶

• MCPToolBench++ is moderately priced: Tasks are shorter than code generation benchmarks but involve tool schema processing
• Single-step tasks are cheapest: One tool call per task means fewer tokens and faster completion
• Use sonnet for all difficulty levels: Tool use tasks depend more on structured output capability than deep reasoning
• Filter by category for focused evaluation rather than running all 45+ categories
• Start with 20 tasks per category for statistically meaningful results without excessive cost
• JSON output prompts reduce cost: Structured prompts lead to more concise, parseable responses

Common Issues & Solutions¶

To inspect available categories:

uv run python -c "
from datasets import load_dataset
ds = load_dataset('MCPToolBench/MCPToolBenchPP', split='train')
cats = sorted(set(item['category'] for item in ds))
for cat in cats:
    print(cat)
"

Comparison with Similar Benchmarks¶

References¶

• MCPToolBench++ Dataset on HuggingFace
• Model Context Protocol Specification
• ToolBench -- general API tool use benchmark
• GAIA -- general AI assistant benchmark with tool use
• AgentBench -- multi-environment agent benchmark
• Benchmarks Overview
• MCP Integration Guide
• Configuration Reference
• CLI Reference