Skip to content
Strands Harness
PYPython SDKTSTypeScript SDK
Strands
EVEvalsSHShell
Organizations
strands-agentsstrands-labs

Benchmark Harnesses

Strands Benchmark Harnesses is a repository of Strands-based agents and harnesses for agentic benchmarks. It is a uv workspace: the repository root coordinates one or more member packages, each with its own setup and usage.

Simple Strands Agent (SSA)

Section titled “Simple Strands Agent (SSA)”

The flagship member is Simple Strands Agent - a lean, hackable harness for autonomous software engineering that achieves state-of-the-art performance across software-engineering benchmarks. It pairs frontier LLMs (Claude, GPT, Gemini, and open-weight models via Bedrock / LiteLLM / vLLM) with bash and file-editing tools inside isolated Docker environments to analyze codebases, diagnose bugs, write patches, and verify solutions.

Despite its simplicity, SSA delivers SOTA-level results on SWE-Bench Verified, SWE-Bench Pro, and Terminal Bench 2. See the technical report and the Amazon Science post.

Highlights

Section titled “Highlights”
  • Model-agnostic - first-class adapters for Anthropic, OpenAI, Google, xAI, Bedrock, and any OpenAI-compatible endpoint (vLLM, LiteLLM, Together, Vertex, Z.AI).
  • Composable tools - bash, str_replace_editor, think, and submit primitives with per-tool output clipping and timeout controls.
  • Isolated environments - Docker-backed sandboxes with streaming exec, automatic image resolution, and ECR support.
  • Hydra-powered configs - every knob is overridable from the command line; experiments are reproducible from a single YAML.
  • Built-in benchmarking - turnkey scripts for SWE-Bench Verified, SWE-Bench Pro, and Terminal Bench 2, including S3 result upload.

Getting started

Section titled “Getting started”
Terminal window
git clone https://github.com/strands-labs/benchmark-harnesses.git
cd benchmark-harnesses


# Recommended: sync the workspace (creates .venv with the package + deps)
uv sync
source .venv/bin/activate

Run a single instance:

Terminal window
uv run python -m ssa.run \
    --config-name=default.yaml \
    dataset.name=sbv \
    dataset.identifier=django__django-15987 \
    env.env_type=docker \
    env.docker.workdir="/testbed"

Running agents safely

Section titled “Running agents safely”

Agents in this repository are given shell access, which means the model can run commands in the environment where the agent starts. Run agents inside Docker or another sandbox, mount only what the agent needs, and avoid exposing credentials, SSH keys, or production data.

Section titled “Links”