Skip to content

Installation

pip install mcpbr && mcpbr init && mcpbr run -c mcpbr.yaml -n 1 -v

That's it. For the full setup guide, read on.

Prerequisites

Before installing mcpbr, ensure you have the following:

Requirement Version Notes
Python 3.11+ Required for mcpbr
Docker Latest Must be running
Claude Code CLI Latest The claude command
Network access - For Docker images and API calls

API Key

You'll need an Anthropic API key:

export ANTHROPIC_API_KEY="sk-ant-..."

Get an API key

Sign up at console.anthropic.com to get your API key.

Claude Code CLI

Install the Claude Code CLI globally:

npm install -g @anthropic-ai/claude-code

Verify the installation:

which claude  # Should return the path to the CLI

Docker

Ensure Docker is installed and running:

docker info

If Docker isn't running, start it from your system's application launcher or:

open -a Docker

sudo systemctl start docker

Start Docker Desktop from the Start menu.

Installation Methods

pip install mcpbr

From npm

npm package

mcpbr is also available as an npm package for easy integration with Node.js workflows:

# Run with npx (no installation)
npx mcpbr-cli run -c config.yaml

# Or install globally
npm install -g mcpbr-cli
mcpbr run -c config.yaml

Package Details

Package name: mcpbr-cli

The npm package is a wrapper that requires Python 3.11+ and the mcpbr Python package to be installed separately: 

pip install mcpbr
npm install -g mcpbr-cli

From Source

git clone https://github.com/greynewell/mcpbr.git
cd mcpbr
pip install -e .

With uv

uv pip install mcpbr

Or from source:

git clone https://github.com/greynewell/mcpbr.git
cd mcpbr
uv pip install -e .

Verify Installation

After installation, verify everything is working:

# Check mcpbr is installed
mcpbr --version

# List supported models
mcpbr models

# Generate a test config
mcpbr init -o test-config.yaml

Supported Models

mcpbr supports the following Claude models:

Model ID Context Window
Claude Opus 4.5 opus or claude-opus-4-5-20251101 200,000
Claude Sonnet 4.5 sonnet or claude-sonnet-4-5-20250929 200,000
Claude Haiku 4.5 haiku or claude-haiku-4-5-20251001 200,000
Claude Opus 4 claude-opus-4-20250514 200,000
Claude Sonnet 4 claude-sonnet-4-20250514 200,000
Claude Haiku 4 claude-haiku-4-20250514 200,000
Claude 3.5 Sonnet claude-3-5-sonnet-20241022 200,000

Run mcpbr models to see the full list.

Apple Silicon Notes

On ARM64 Macs (M1/M2/M3), x86_64 Docker images run via emulation. This is:

  • Slower than native ARM64 images
  • Required for compatibility with all SWE-bench tasks
  • Automatic - no configuration needed

If you experience issues, ensure Rosetta 2 is installed:

softwareupdate --install-rosetta

Development Installation

For contributing to mcpbr:

git clone https://github.com/greynewell/mcpbr.git
cd mcpbr
pip install -e ".[dev]"

This installs additional development dependencies:

  • pytest - Testing framework
  • pytest-asyncio - Async test support
  • ruff - Linting and formatting

See Contributing for more details.

Next Steps

After installation, you have two options to get started:

Option 1: Use Example Configurations (Fastest)

Jump straight in with our ready-to-use examples:

# Set your API key
export ANTHROPIC_API_KEY="your-api-key"

# Run your first evaluation
mcpbr run -c examples/quick-start/getting-started.yaml -v

Explore 25+ example configurations in the examples/ directory covering benchmarks, MCP servers, and common scenarios. See the Examples README for the complete guide.

Option 2: Generate Custom Configuration

Create your own configuration:

mcpbr init
# Edit mcpbr.yaml
mcpbr run -c mcpbr.yaml -v

Continue Learning: