Structured output similarity evaluator

The StructuredOutputSimilarity evaluator scores an agent’s structured output against ground truth field by field, rather than comparing whole objects. Where Equals returns only 0.0 or 1.0, StructuredOutputSimilarity tells you which field to fix. An extraction that gets nine of ten fields right no longer looks the same as one that gets none right.

It is deterministic and offline: no LLM judge, no credentials, no per-call cost. Configuration comes from the Pydantic model your agent already supplies as structured_output_model, so no schema or annotation work is required. explain() shows every inferred decision. A complete example can be found here.

This evaluator requires the optional stickler extra, which installs stickler, the scoring engine behind it:

pip install "strands-agents-evals[stickler]"

Without it, constructing the evaluator raises an ImportError naming the command above.

Key features

• Field-Level Scoring: Per-field scores and per-field metrics, not a single pass/fail
• Order-Independent List Matching: A reordered but identical list scores 1.0, not 0.0
• Zero Configuration: Comparators and thresholds are inferred from the Pydantic model
• Deterministic and Offline: Pure computation, with no LLM judge, no credentials, and no per-call cost
• Inspectable: explain() reports the comparator chosen for each field and why
• Dataset Metrics: Confusion-matrix counts per dotted field path across the whole run

When to use

Use the StructuredOutputSimilarity evaluator when you need to:

• Evaluate extraction or structured-output tasks where partial correctness matters
• Identify which field regressed, not just that a case failed
• Score outputs containing lists of nested objects, where ordering is not meaningful
• Run evaluations in CI/CD without model credentials or inference cost
• Track per-field accuracy across a dataset to target the worst-performing field

Prefer Equals when the output is a scalar and only exact equality matters.

Parameters

model_cls (required)

• Type: type[BaseModel] | str
• Description: The Pydantic model to score against: the same model the agent emits as structured_output_model. Every case is coerced to it. Output that does not validate returns a score-0.0 row naming why, rather than raising. Also accepts a dotted path to the class, so a saved experiment reloads. For a suite that mixes output types, run a separate Experiment per schema. Within one Experiment, every evaluator scores every case, so cross-schema pairs become score-0.0 rows.

match_threshold (optional)

• Type: float
• Default: 0.7
• Description: Similarity at or above which an object counts as a match. It sets the bar for matching list[Model] elements, single nested BaseModel fields, and dict fields. The weighted score is also compared against it, so it drives both halves of the test_pass rule (see Scoring system). Scalar field thresholds are inferred from type and field name and are unaffected.

weight_hints (optional)

• Type: bool
• Default: False
• Description: Whether to infer per-field weights from the schema. Off by default so weights stay uniform and metrics are not skewed by guessed business criticality.

name (optional)

• Type: str | None
• Default: None
• Description: Evaluator name, forwarded to the base Evaluator. On a mixed-schema run, it is also the key you pass as report.metrics(evaluator="invoice") to read each schema’s rollup separately. It keeps rollups legible, not isolated: scoring still runs every evaluator over every case, so use one Experiment per schema.

Basic usage

from pydantic import BaseModel
from strands import Agent
from strands_evals import Case, Experiment, StructuredOutputReport
from strands_evals.evaluators import StructuredOutputSimilarity


class LineItem(BaseModel):
    sku: str | None = None
    quantity: int | None = None


class Invoice(BaseModel):
    invoice_id: str
    vendor_name: str
    total_amount: float | None = None
    line_items: list[LineItem] = []


def extract(case: Case) -> Invoice:
    agent = Agent(callback_handler=None)
    result = agent(case.input, structured_output_model=Invoice)
    return result.structured_output


cases = [
    Case(
        name="invoice-0042",
        input="INVOICE #INV-2024-0042, Acme Corporation: 2x WM-100, 5x UC-050. Total: $124.93",
        expected_output=Invoice(
            invoice_id="INV-2024-0042",
            vendor_name="Acme Corporation",
            total_amount=124.93,
            line_items=[
                LineItem(sku="WM-100", quantity=2),
                LineItem(sku="UC-050", quantity=5),
            ],
        ),
    ),
]

# The same model the agent already emits as structured_output_model.
evaluator = StructuredOutputSimilarity(Invoice)

# report_cls gives the report the rollups shown under Evaluation output.
experiment = Experiment(cases=cases, evaluators=[evaluator], report_cls=StructuredOutputReport)
report = experiment.run_evaluations(extract)
report.run_display()

Evaluation output

The evaluator returns one EvaluationOutput per case with:

• score: Weighted mean of the per-field scores, between 0.0 and 1.0
• test_pass: Boolean. Requires the weighted score to clear match_threshold and, when there was anything to find, recall to clear it too. Per-field thresholds shape each field’s score but are not themselves gates
• reason: The weakest imperfect fields, e.g. "weakest fields: line_items=0.75", or "all fields matched"
• label: The model class name
• metadata: The field-level detail

The detail rides on metadata rather than on the evaluator, so it stays with the report. It survives report.model_dump_json(), and the rollups below work on a saved report reloaded through StructuredOutputReport. report.per_case() reads it back as one entry per scored case:

Key	Value
case, evaluator, model	Case name, evaluator name, and model class name
overall_score, test_pass	The row’s score and verdict
field_scores	Score per top-level field, e.g. {"invoice_id": 1.0, ..., "line_items": 0.75}
precision, recall, f1	These ignore correct absence, so they stay meaningful on a sparse schema
matched	stickler’s score-only verdict: score >= match_threshold

for row in report.per_case():
    print(row["case"], row["overall_score"], row["recall"], row["field_scores"])

Across a whole run, read the rollups:

report.metrics()       # per-field confusion matrix
evaluator.explain()    # the comparator chosen per field, and why

StructuredOutputReport.from_file("report.json").metrics()   # a saved report

metrics and per_case are methods of StructuredOutputReport. run_evaluations returns one only when the experiment passes report_cls=StructuredOutputReport; without it you get a plain EvaluationReport, which has neither. The CLI and an experiment reloaded with Experiment.from_file don’t carry report_cls, so their reports are the base class. To roll one up, reload the saved report through StructuredOutputReport.from_file. Both methods skip rows whose output failed to validate. Those cases stay visible in the report but drop out of the rollups. A run of two cases where one failed to validate reports a document_count of 1.

report.metrics() returns a single ProcessEvaluation. Read its field_metrics attribute. The rollup is a Pydantic model, not a mapping, so report.metrics()["field_metrics"] raises TypeError. field_metrics itself is a dict keyed by dotted path, such as line_items.sku and line_items.quantity, so a suite pinpoints the failing field across the dataset. Each entry carries the counts (tp/tn/fn/fa/fd, plus fp = fa + fd) and the derived metrics under cm_-prefixed keys: cm_precision, cm_recall, cm_f1, cm_accuracy. Most paths also carry a mean_score, but leaves under a list[Model] parent do not: total_amount has one, line_items.sku does not. Leaves under a single nested BaseModel do.

metrics = report.metrics()
metrics.field_metrics["total_amount"]["cm_precision"]

Which child paths are counted depends on how the field nests, and children can hide what their parent shows:

Field type	Child paths counted	Watch for
list[Model]	Only for element pairs scoring at or above match_threshold, per element rather than per document	A badly wrong element drops out of its children’s counts, which can then read perfect
Nested BaseModel	Whenever the object is present on both sides, whatever it scored	Documents missing the object count only against the parent, so a child can read cm_recall 1.0 while the parent reads 0.667
dict[str, Model]	Never, even on an exact match	Only the parent path exists

Read a parent path such as line_items alongside its children.

per_case() omits rows, so do not positionally zip it

report.per_case() is in case order, but it omits rows whose output failed to validate. Those cases still appear in report.scores as zero-score rows, so the two sequences differ in length. Join on case rather than by position.

Scoring system

The score is a weighted mean of per-field scores, ranging from 0.0 to 1.0, where higher is better. Because it is computed rather than judged, the same inputs always produce the same score.

Ordering is the clearest difference from Equals. Given the invoice above:

Scenario	Equals	StructuredOutputSimilarity
List reordered, content identical	0.0	1.0
One quantity wrong, list reordered	0.0	0.9375

In the second row, reason reads "weakest fields: line_items=0.75". field_scores gives 1.0 for invoice_id, vendor_name, and total_amount against 0.75 for line_items, so the failing field is named, not inferred.

Sparse schemas: read f1, not the score alone

A field absent on both sides is credited 1.0: a value the model correctly left blank is a value it got right. On a mostly-empty schema, those credits dominate. A ten-field model whose ground truth fills two fields still scores 0.80 against a prediction that returns nothing.

Raising match_threshold cannot fix this, because the score needed to pass tends toward 1.0 as the optional tail grows. This is why test_pass checks recall as well as the score. Here the score of 0.80 clears the default 0.7, but recall is 0.0, so the case fails on recall alone. Each row’s metadata surfaces recall, precision, and f1 to show why.

Recall is not a complete safety net, though. It is tp / (tp + fn), which ignores values that were found wrong: those land in fd, in neither term. On the ten-field example above, a prediction returning one correct and one incorrect value scores 0.90 with recall of 1.0, so it passes. Its precision is 0.5 and its f1 is 0.667.

Two consequences follow:

• On sparse schemas, report f1 rather than the score or recall alone. recall ignores values returned wrong, so it can read 1.0 on a document containing them. But once every value is wrong, tp reaches zero and recall collapses to 0.0. It reads 1.0 at best, 1.0 in the middle, and 0.0 at worst, so it does not degrade with quality.
• No scalar tells the degenerate cases apart from one another. Only the counts in report.metrics() do.

Six documents on the ten-field model show the difference:

prediction	score	test_pass	precision / recall / f1	counts
every value right	1.0	pass	1.0 / 1.0 / 1.0	tp: 2, tn: 8
nothing returned	0.80	fail	0.0 / 0.0 / 0.0	tn: 8, fn: 2
every value wrong	0.80	fail	0.0 / 0.0 / 0.0	tn: 8, fd: 2
values invented, ground truth blank	0.80	fail	0.0 / 0.0 / 0.0	tn: 8, fa: 2
genuinely blank on both sides	1.0	pass	0.0 / 0.0 / 0.0	tn: 10
every value found, plus two invented	0.80	pass	0.5 / 1.0 / 0.667	tp: 2, tn: 6, fa: 2

Four of the six read 0.0 / 0.0 / 0.0 on all three scalars. One of those four is a perfectly correct document: the one whose ground truth was legitimately blank. The gate still separates them. It counts what there was to find as tp + fn + fd, so a document whose values are all wrong is not mistaken for one with nothing to find. When there was nothing to find, it requires fa to be zero, which catches an invention.

No scalar shows why a document failed, so read the counts. A miss is fn, a wrong value is fd, and an invention is fa. The genuinely blank document carries nothing but tn.

The last row shows a second limit, and this one is deliberate. With something to find, the gate reads recall alone, so a prediction that finds everything and also invents extra fields passes. Ground truth is often annotated sparsely, and a precision gate would fail an agent for extracting a field the annotation merely omitted. Read precision from the row’s metadata when you want the stricter check.

Best practices

1. One Experiment per schema: every evaluator in an Experiment scores every case. A single experiment mixing output types therefore produces score-0.0 cross-pairs that contaminate overall_score and test_passes. Run a separate Experiment per schema and read each report on its own. name= keeps rollups legible but does not isolate scoring
2. Read report.metrics() to choose what to fix: the per-field confusion matrix shows how often each path is missed across the dataset. Sort it yourself by cm_recall, or by mean_score if you skip the list[Model] children that lack one. Read parent paths beside nested ones
3. Audit inferences before trusting them: call explain() to confirm the comparator and threshold chosen for each field
4. Report f1 alongside the score on sparse schemas: the score alone credits correct absence, and recall alone ignores values returned wrong
5. Combine with LLM evaluators: use StructuredOutputSimilarity as a fast deterministic gate, then LLM-based evaluators for qualities that require judgment

Related evaluators

• Deterministic Evaluators: Equals, Contains, StartsWith, and other code-based checks
• OutputEvaluator: LLM-based output evaluation against a custom rubric
• CorrectnessEvaluator: LLM-based factual correctness, with or without a reference
• CustomEvaluator: Build your own evaluation logic

Tags

structured-output6