# Evals v0.1.5

Released 2026-02-05
Release: https://github.com/strands-agents/evals/releases/tag/v0.1.5 · Package: https://pypi.org/project/strands-agents-evals/0.1.5/

## Features
- added ResponseRelevanceEvaluator (https://github.com/strands-agents/evals/pull/112)
- added ConcisenessEvaluator (https://github.com/strands-agents/evals/pull/115)

## Fixes
- replace deprecated structured\_output methods with new API (https://github.com/strands-agents/evals/pull/67)

## Other
- Feat/retry throttled tenacity (https://github.com/strands-agents/evals/pull/107)
- bump aws-actions/configure-aws-credentials from 5 to 6 (https://github.com/strands-agents/evals/pull/118)
- workflow: add strands-command for PR and issue (https://github.com/strands-agents/evals/pull/122)

## Major Features

### Response Relevance Evaluator - [PR#112](https://github.com/strands-agents/evals/pull/112)

The new `ResponseRelevanceEvaluator` measures how well an agent's response addresses the user's question. It uses a 5-level LLM-as-judge scoring system — *Not At All* (0.0), *Not Generally* (0.25), *Neutral/Mixed* (0.5), *Generally Yes* (0.75), and *Completely Yes* (1.0) — with a pass threshold at ≥0.5. Like other trace-level evaluators, it requires an `actual_trajectory` session and supports both sync and async evaluation.

```python
from strands_evals.evaluators import ResponseRelevanceEvaluator

evaluator = ResponseRelevanceEvaluator()
results = evaluator.evaluate(evaluation_data)

# results[0].score    -> 1.0  (for COMPLETELY_YES)
# results[0].test_pass -> True (score >= 0.5)
# results[0].reason   -> "The response directly answers the question."
# results[0].label    -> ResponseRelevanceScore.COMPLETELY_YES
```

### Conciseness Evaluator - [PR#115](https://github.com/strands-agents/evals/pull/115)

The new `ConcisenessEvaluator` assesses whether an agent's response is appropriately concise. It uses a 3-level scoring system: *Perfectly Concise* (1.0) for responses that deliver exactly what was asked, *Partially Concise* (0.5) for minor extra wording, and *Not Concise* (0.0) for verbose or repetitive content. The pass threshold is ≥0.5. Both evaluators accept an optional custom `model` and `system_prompt` for the LLM judge.

```python
from strands_evals.evaluators import ConcisenessEvaluator

evaluator = ConcisenessEvaluator()
results = evaluator.evaluate(evaluation_data)

# results[0].score    -> 0.0  (for NOT_CONCISE)
# results[0].test_pass -> False (score < 0.5)
# results[0].label    -> ConcisenessScore.NOT_CONCISE
```

### Automatic Retry with Exponential Backoff for Throttled Evaluations - [PR#107](https://github.com/strands-agents/evals/pull/107)

`Experiment.run_evaluations()` and `run_evaluations_async()` now automatically retry on throttling errors using [tenacity](https://github.com/jd/tenacity). Both task execution and evaluator execution are wrapped with exponential backoff (up to 6 attempts, 4s → 240s). Throttling is detected for `ModelThrottledException`, `EventLoopException`, and botocore `ClientError` with codes like `ThrottlingException` and `TooManyRequestsException`. Non-throttling errors are raised immediately without retrying. Additionally, one evaluator failing no longer prevents other evaluators from running on the same case.

```python
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluator, ConcisenessEvaluator

experiment = Experiment(
    cases=[Case(name="test", input="What is 2+2?", expected_output="4")],
    evaluators=[
        OutputEvaluator(rubric="Is the answer correct?"),
        ConcisenessEvaluator(),
    ],
)

# Throttling errors are retried automatically with exponential backoff.
# If one evaluator fails, the other still runs.
reports = experiment.run_evaluations(my_agent)
```

---

## Bug Fixes

* **Replace Deprecated Structured Output Methods** - [PR#67](https://github.com/strands-agents/evals/pull/67)
  Updated `OutputEvaluator` to call the agent with the `structured_output_model` parameter instead of the removed `structured_output()` and `structured_output_async()` methods. Without this fix, `OutputEvaluator` would fail on recent Strands SDK versions.

---