HumanEval¶

Overview¶

mcpbr run -c config.yaml --benchmark humaneval -n 20

Overview¶

HumanEval is a code generation benchmark created by OpenAI consisting of 164 hand-written Python programming problems. Each task provides a function signature with a descriptive docstring, and the agent must produce a correct implementation that passes all accompanying unit tests.

HumanEval is widely used as a baseline measure of code generation capability. Problems range from simple string manipulation and list operations to moderate algorithmic challenges. Because tasks are self-contained single functions with no external dependencies, HumanEval is an excellent choice for quick evaluations and smoke testing before running more expensive benchmarks like SWE-bench.

In mcpbr, HumanEval evaluates how effectively an MCP server assists the language model in generating correct, test-passing Python code within lightweight Docker containers.

What It Measures¶

HumanEval tests a focused set of code generation capabilities:

• Function-level code synthesis: The ability to translate a natural language specification (docstring) into correct Python code
• Type-aware programming: Functions include type hints that the agent must respect and leverage
• Edge case handling: Unit tests cover boundary conditions, empty inputs, and special cases that require careful implementation
• Standard library fluency: All problems use only the Python standard library, testing the agent's knowledge of built-in data structures, string operations, and mathematical functions
• Docstring comprehension: The agent must correctly interpret examples, constraints, and expected behavior described in the docstring

HumanEval does not test:

• Multi-file or multi-class code generation
• External library usage (numpy, pandas, etc.)
• Bug fixing or debugging existing code
• Long-context repository understanding

Task Structure¶

Each HumanEval task contains the following fields:

Example task prompt:

def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """

The agent receives the function signature and docstring, implements the function body, and saves the result to solution.py. Task IDs are converted to Docker-safe names by replacing slashes with underscores (e.g., HumanEval/0 becomes HumanEval_0).

Configuration¶

Basic Configuration¶

# Run HumanEval with default settings
mcpbr run -c config.yaml --benchmark humaneval

# Run a small sample for quick testing
mcpbr run -c config.yaml --benchmark humaneval -n 20

# Run specific tasks by ID
mcpbr run -c config.yaml --benchmark humaneval -t HumanEval/0 -t HumanEval/1

# Run with verbose output
mcpbr run -c config.yaml --benchmark humaneval -n 10 -v

# Save results to JSON
mcpbr run -c config.yaml --benchmark humaneval -n 20 -o results.json

benchmark: "humaneval"
sample_size: 10
timeout_seconds: 180
max_iterations: 15

mcp_server:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]

model: "sonnet"

Advanced Options¶

HumanEval does not support difficulty or category filtering since all 164 tasks are uniform Python function completions. However, you can select specific tasks by ID:

# Run specific tasks for targeted testing
mcpbr run -c config.yaml --benchmark humaneval \
  -t HumanEval/0 -t HumanEval/10 -t HumanEval/50

# Run all tasks with extended timeout and results
mcpbr run -c config.yaml --benchmark humaneval \
  -o results.json -r report.md -v

Configuration optimized for throughput:

benchmark: "humaneval"
sample_size: 164       # All tasks
timeout_seconds: 120
max_iterations: 10
max_concurrent: 8      # High concurrency -- lightweight containers

mcp_server:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]

model: "sonnet"

Evaluation Methodology¶

HumanEval evaluation follows a straightforward test-execution pipeline:

1. Solution discovery: The evaluator looks for the agent's solution file. It checks several candidate filenames in order: solution.py, answer.py, code.py, implementation.py, main.py. If no file is found, it attempts to extract code from the agent's raw response (markdown code blocks or inline function definitions).

2. Test assembly: The solution code is combined with the task's unit test code and a check() call that invokes the tests against the implemented entry point function.

3. Execution: The combined file is executed with python3 inside the Docker container with a 30-second timeout.

4. Verdict: The task is marked as resolved if the Python process exits with code 0 and no AssertionError appears in stderr.

The evaluation does not require any external packages. All HumanEval tasks use only the Python standard library.

Interpreting Results¶

Key Metrics¶

What Good Results Look Like¶

Common Failure Patterns¶

Example Output¶

Successful resolution:

{
  "resolved": true,
  "exit_code": 0,
  "stdout": "",
  "stderr": ""
}

Failed resolution (assertion error):

{
  "resolved": false,
  "exit_code": 1,
  "stdout": "",
  "stderr": "Traceback (most recent call last):\n  File \"test_solution.py\", line 12, in <module>\n    check(has_close_elements)\n  File \"test_solution.py\", line 8, in check\n    assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3) == True\nAssertionError",
  "error": "Test assertions failed"
}

Failed resolution (no solution found):

{
  "resolved": false,
  "error": "No solution file found and could not extract code from solution"
}

Best Practices¶

Recommended Workflow¶

1. Start with a small sample (10-20 tasks) to verify your setup before running all 164 tasks
2. Review the first few results to confirm the agent saves solutions to solution.py
3. Run the full benchmark once configuration is validated
4. Compare across models by running with different model configurations

Performance Tips¶

• Use shorter timeouts (60-180s) since HumanEval tasks are quick and well-defined
• Run high concurrency (max_concurrent: 8 or higher) since each task uses a lightweight Python container with minimal resource requirements
• Set max_iterations lower (10-15) since tasks are simpler than SWE-bench and require fewer agent turns
• Run HumanEval first before expensive benchmarks like SWE-bench as a smoke test for your MCP server integration

Cost Optimization¶

• 164 tasks is manageable: Running all tasks is feasible and recommended for consistent benchmarking
• Low token usage: Function completion typically requires fewer tokens than bug-fixing or multi-step reasoning tasks
• Quick turnaround: Most tasks complete in under 2 minutes, making full-suite runs practical even for iteration
• Use sonnet for cost efficiency: HumanEval tasks rarely benefit from the additional reasoning capability of more expensive models

Common Issues & Solutions¶

Comparison with Similar Benchmarks¶

References¶

• HumanEval Repository
• HumanEval Paper (Evaluating Large Language Models Trained on Code)
• HumanEval Dataset on HuggingFace
• MBPP Benchmark -- related code generation benchmark with more tasks
• SWE-bench -- real-world bug fixing benchmark
• Benchmarks Overview
• Configuration Reference
• CLI Reference