The `StructuredOutputSimilarity` evaluator scores an agent’s structured output against ground truth **field by field**, rather than comparing whole objects. Where [`Equals`](/docs/user-guide/evals-sdk/evaluators/deterministic_evaluators/index.md) returns only `0.0` or `1.0`, `StructuredOutputSimilarity` tells you *which field to fix*. An extraction that gets nine of ten fields right no longer looks the same as one that gets none right.

It is deterministic and offline: no LLM judge, no credentials, no per-call cost. Configuration comes from the Pydantic model your agent already supplies as `structured_output_model`, so no schema or annotation work is required. `explain()` shows every inferred decision. A complete example can be found [here](https://github.com/strands-agents/harness-sdk/blob/main/site/docs/examples/evals-sdk/structured_output_evaluator.py).

This evaluator requires the optional `stickler` extra, which installs [stickler](https://github.com/awslabs/stickler), the scoring engine behind it:

```bash
pip install "strands-agents-evals[stickler]"
```

Without it, constructing the evaluator raises an `ImportError` naming the command above.

## Key features

-   **Field-Level Scoring**: Per-field scores and per-field metrics, not a single pass/fail
-   **Order-Independent List Matching**: A reordered but identical list scores `1.0`, not `0.0`
-   **Zero Configuration**: Comparators and thresholds are inferred from the Pydantic model
-   **Deterministic and Offline**: Pure computation, with no LLM judge, no credentials, and no per-call cost
-   **Inspectable**: `explain()` reports the comparator chosen for each field and why
-   **Dataset Metrics**: Confusion-matrix counts per dotted field path across the whole run

## When to use

Use the `StructuredOutputSimilarity` evaluator when you need to:

-   Evaluate extraction or structured-output tasks where partial correctness matters
-   Identify *which* field regressed, not just that a case failed
-   Score outputs containing lists of nested objects, where ordering is not meaningful
-   Run evaluations in CI/CD without model credentials or inference cost
-   Track per-field accuracy across a dataset to target the worst-performing field

Prefer [`Equals`](/docs/user-guide/evals-sdk/evaluators/deterministic_evaluators/index.md) when the output is a scalar and only exact equality matters.

## Parameters

### `model_cls` (required)

-   **Type**: `type[BaseModel] | str`
-   **Description**: The Pydantic model to score against: the same model the agent emits as `structured_output_model`. Every case is coerced to it. Output that does not validate returns a score-`0.0` row naming why, rather than raising. Also accepts a dotted path to the class, so a saved experiment reloads. For a suite that mixes output types, run a separate `Experiment` per schema. Within one `Experiment`, every evaluator scores every case, so cross-schema pairs become score-`0.0` rows.

### `match_threshold` (optional)

-   **Type**: `float`
-   **Default**: `0.7`
-   **Description**: Similarity at or above which an object counts as a match. It sets the bar for matching `list[Model]` elements, single nested `BaseModel` fields, and `dict` fields. The weighted score is also compared against it, so it drives both halves of the `test_pass` rule (see [Scoring system](#scoring-system)). Scalar field thresholds are inferred from type and field name and are unaffected.

### `weight_hints` (optional)

-   **Type**: `bool`
-   **Default**: `False`
-   **Description**: Whether to infer per-field weights from the schema. Off by default so weights stay uniform and metrics are not skewed by guessed business criticality.

### `name` (optional)

-   **Type**: `str | None`
-   **Default**: `None`
-   **Description**: Evaluator name, forwarded to the base `Evaluator`. On a mixed-schema run, it is also the key you pass as `report.metrics(evaluator="invoice")` to read each schema’s rollup separately. It keeps rollups legible, not isolated: scoring still runs every evaluator over every case, so use one `Experiment` per schema.

## Basic usage

```python
from pydantic import BaseModel
from strands import Agent
from strands_evals import Case, Experiment, StructuredOutputReport
from strands_evals.evaluators import StructuredOutputSimilarity


class LineItem(BaseModel):
    sku: str | None = None
    quantity: int | None = None


class Invoice(BaseModel):
    invoice_id: str
    vendor_name: str
    total_amount: float | None = None
    line_items: list[LineItem] = []


def extract(case: Case) -> Invoice:
    agent = Agent(callback_handler=None)
    result = agent(case.input, structured_output_model=Invoice)
    return result.structured_output


cases = [
    Case(
        name="invoice-0042",
        input="INVOICE #INV-2024-0042, Acme Corporation: 2x WM-100, 5x UC-050. Total: $124.93",
        expected_output=Invoice(
            invoice_id="INV-2024-0042",
            vendor_name="Acme Corporation",
            total_amount=124.93,
            line_items=[
                LineItem(sku="WM-100", quantity=2),
                LineItem(sku="UC-050", quantity=5),
            ],
        ),
    ),
]

# The same model the agent already emits as structured_output_model.
evaluator = StructuredOutputSimilarity(Invoice)

# report_cls gives the report the rollups shown under Evaluation output.
experiment = Experiment(cases=cases, evaluators=[evaluator], report_cls=StructuredOutputReport)
report = experiment.run_evaluations(extract)
report.run_display()
```

## Evaluation output

The evaluator returns one `EvaluationOutput` per case with:

-   **score**: Weighted mean of the per-field scores, between `0.0` and `1.0`
-   **test\_pass**: Boolean. Requires the weighted score to clear `match_threshold` and, when there was anything to find, `recall` to clear it too. Per-field thresholds shape each field’s score but are not themselves gates
-   **reason**: The weakest imperfect fields, e.g. `"weakest fields: line_items=0.75"`, or `"all fields matched"`
-   **label**: The model class name
-   **metadata**: The field-level detail

The detail rides on `metadata` rather than on the evaluator, so it stays with the report. It survives `report.model_dump_json()`, and the rollups below work on a saved report reloaded through `StructuredOutputReport`. `report.per_case()` reads it back as one entry per scored case:

| Key | Value |
| --- | --- |
| `case`, `evaluator`, `model` | Case name, evaluator name, and model class name |
| `overall_score`, `test_pass` | The row’s score and verdict |
| `field_scores` | Score per top-level field, e.g. `{"invoice_id": 1.0, ..., "line_items": 0.75}` |
| `precision`, `recall`, `f1` | These ignore correct absence, so they stay meaningful on a sparse schema |
| `matched` | stickler’s score-only verdict: `score >= match_threshold` |

```python
for row in report.per_case():
    print(row["case"], row["overall_score"], row["recall"], row["field_scores"])
```

Across a whole run, read the rollups:

```python
report.metrics()       # per-field confusion matrix
evaluator.explain()    # the comparator chosen per field, and why

StructuredOutputReport.from_file("report.json").metrics()   # a saved report
```

`metrics` and `per_case` are methods of `StructuredOutputReport`. `run_evaluations` returns one only when the experiment passes `report_cls=StructuredOutputReport`; without it you get a plain `EvaluationReport`, which has neither. The CLI and an experiment reloaded with `Experiment.from_file` don’t carry `report_cls`, so their reports are the base class. To roll one up, reload the saved report through `StructuredOutputReport.from_file`. Both methods skip rows whose output failed to validate. Those cases stay visible in the report but drop out of the rollups. A run of two cases where one failed to validate reports a `document_count` of `1`.

`report.metrics()` returns a single `ProcessEvaluation`. Read its `field_metrics` attribute. The rollup is a Pydantic model, not a mapping, so `report.metrics()["field_metrics"]` raises `TypeError`. `field_metrics` itself is a dict keyed by dotted path, such as `line_items.sku` and `line_items.quantity`, so a suite pinpoints the failing field across the dataset. Each entry carries the counts (`tp`/`tn`/`fn`/`fa`/`fd`, plus `fp` = `fa` + `fd`) and the derived metrics under `cm_`\-prefixed keys: `cm_precision`, `cm_recall`, `cm_f1`, `cm_accuracy`. Most paths also carry a `mean_score`, but leaves under a `list[Model]` parent do not: `total_amount` has one, `line_items.sku` does not. Leaves under a single nested `BaseModel` do.

```python
metrics = report.metrics()
metrics.field_metrics["total_amount"]["cm_precision"]
```

Which child paths are counted depends on how the field nests, and children can hide what their parent shows:

| Field type | Child paths counted | Watch for |
| --- | --- | --- |
| `list[Model]` | Only for element pairs scoring at or above `match_threshold`, per element rather than per document | A badly wrong element drops out of its children’s counts, which can then read perfect |
| Nested `BaseModel` | Whenever the object is present on both sides, whatever it scored | Documents missing the object count only against the parent, so a child can read `cm_recall` `1.0` while the parent reads `0.667` |
| `dict[str, Model]` | Never, even on an exact match | Only the parent path exists |

Read a parent path such as `line_items` alongside its children.

`per_case()` omits rows, so do not positionally zip it

`report.per_case()` is in case order, but it omits rows whose output failed to validate. Those cases still appear in `report.scores` as zero-score rows, so the two sequences differ in length. Join on `case` rather than by position.

## Scoring system

The score is a **weighted mean of per-field scores**, ranging from `0.0` to `1.0`, where higher is better. Because it is computed rather than judged, the same inputs always produce the same score.

Ordering is the clearest difference from `Equals`. Given the invoice above:

| Scenario | `Equals` | `StructuredOutputSimilarity` |
| --- | --- | --- |
| List reordered, content identical | `0.0` | `1.0` |
| One quantity wrong, list reordered | `0.0` | `0.9375` |

In the second row, `reason` reads `"weakest fields: line_items=0.75"`. `field_scores` gives `1.0` for `invoice_id`, `vendor_name`, and `total_amount` against `0.75` for `line_items`, so the failing field is named, not inferred.

### Sparse schemas: read f1, not the score alone

A field absent on both sides is credited `1.0`: a value the model correctly left blank is a value it got right. On a mostly-empty schema, those credits dominate. A ten-field model whose ground truth fills two fields still scores `0.80` against a prediction that returns **nothing**.

Raising `match_threshold` cannot fix this, because the score needed to pass tends toward `1.0` as the optional tail grows. This is why `test_pass` checks `recall` as well as the score. Here the score of `0.80` clears the default `0.7`, but `recall` is `0.0`, so the case fails on recall alone. Each row’s `metadata` surfaces recall, precision, and f1 to show why.

Recall is not a complete safety net, though. It is `tp / (tp + fn)`, which ignores values that were found *wrong*: those land in `fd`, in neither term. On the ten-field example above, a prediction returning one correct and one incorrect value scores `0.90` with **`recall` of `1.0`**, so it **passes**. Its `precision` is `0.5` and its `f1` is `0.667`.

Two consequences follow:

-   On sparse schemas, report `f1` rather than the score or `recall` alone. `recall` ignores values returned **wrong**, so it can read `1.0` on a document containing them. But once *every* value is wrong, `tp` reaches zero and `recall` collapses to `0.0`. It reads `1.0` at best, `1.0` in the middle, and `0.0` at worst, so it does not degrade with quality.
-   No scalar tells the degenerate cases apart from *one another*. Only the counts in `report.metrics()` do.

Six documents on the ten-field model show the difference:

| prediction | score | `test_pass` | precision / recall / f1 | counts |
| --- | --- | --- | --- | --- |
| every value right | `1.0` | pass | `1.0` / `1.0` / `1.0` | `tp: 2, tn: 8` |
| nothing returned | `0.80` | **fail** | `0.0` / `0.0` / `0.0` | `tn: 8, fn: 2` |
| every value wrong | `0.80` | **fail** | `0.0` / `0.0` / `0.0` | `tn: 8, fd: 2` |
| values invented, ground truth blank | `0.80` | **fail** | `0.0` / `0.0` / `0.0` | `tn: 8, fa: 2` |
| genuinely blank on both sides | `1.0` | pass | `0.0` / `0.0` / `0.0` | `tn: 10` |
| every value found, plus two invented | `0.80` | pass | `0.5` / `1.0` / `0.667` | `tp: 2, tn: 6, fa: 2` |

Four of the six read `0.0` / `0.0` / `0.0` on all three scalars. One of those four is a perfectly correct document: the one whose ground truth was legitimately blank. The gate still separates them. It counts what there *was* to find as `tp + fn + fd`, so a document whose values are all wrong is not mistaken for one with nothing to find. When there was nothing to find, it requires `fa` to be zero, which catches an invention.

No scalar shows *why* a document failed, so read the counts. A miss is `fn`, a wrong value is `fd`, and an invention is `fa`. The genuinely blank document carries *nothing but* `tn`.

The last row shows a second limit, and this one is deliberate. With something to find, the gate reads recall alone, so a prediction that finds everything and also invents extra fields passes. Ground truth is often annotated sparsely, and a precision gate would fail an agent for extracting a field the annotation merely omitted. Read `precision` from the row’s `metadata` when you want the stricter check.

## Best practices

1.  **One `Experiment` per schema**: every evaluator in an `Experiment` scores every case. A single experiment mixing output types therefore produces score-`0.0` cross-pairs that contaminate `overall_score` and `test_passes`. Run a separate `Experiment` per schema and read each report on its own. `name=` keeps rollups legible but does not isolate scoring
2.  **Read `report.metrics()` to choose what to fix**: the per-field confusion matrix shows how often each path is missed across the dataset. Sort it yourself by `cm_recall`, or by `mean_score` if you skip the `list[Model]` children that lack one. Read parent paths beside nested ones
3.  **Audit inferences before trusting them**: call `explain()` to confirm the comparator and threshold chosen for each field
4.  **Report `f1` alongside the score on sparse schemas**: the score alone credits correct absence, and `recall` alone ignores values returned wrong
5.  **Combine with LLM evaluators**: use `StructuredOutputSimilarity` as a fast deterministic gate, then LLM-based evaluators for qualities that require judgment

## Related evaluators

-   [**Deterministic Evaluators**](/docs/user-guide/evals-sdk/evaluators/deterministic_evaluators/index.md): `Equals`, `Contains`, `StartsWith`, and other code-based checks
-   [**OutputEvaluator**](/docs/user-guide/evals-sdk/evaluators/output_evaluator/index.md): LLM-based output evaluation against a custom rubric
-   [**CorrectnessEvaluator**](/docs/user-guide/evals-sdk/evaluators/correctness_evaluator/index.md): LLM-based factual correctness, with or without a reference
-   [**CustomEvaluator**](/docs/user-guide/evals-sdk/evaluators/custom_evaluator/index.md): Build your own evaluation logic

## Related pages

- [LiteLLM](/docs/user-guide/sdk/model-providers/litellm/index.md) (1 shared tag)
- [Return structured output](/docs/user-guide/sdk/agents/structured-output/index.md) (1 shared tag)
- [Structured output](/docs/user-guide/sdk/agents/structured-output-behavior/index.md) (1 shared tag)
- [OpenAI](/docs/user-guide/sdk/model-providers/openai/index.md) (1 shared tag)
- [Writer](/docs/user-guide/sdk/model-providers/writer/index.md) (1 shared tag)
- [Anthropic](/docs/user-guide/sdk/model-providers/anthropic/index.md) (1 shared tag)