Tag: Evaluation
9 posts
Evaluating structured agent output, field by field, with Strands Evals
StructuredOutputSimilarity scores structured agent output field by field with Stickler: deterministic, per-field accuracy with no LLM judge in the default path.
We ran 94,000 trials to measure the cost of harness choice
We analyzed 94,000 benchmark trials to measure how harness choice affects token usage, estimated costs, and accuracy with Claude, GPT, and open-weight models.
Framework-agnostic evaluation with Strands Evals
Strands Evals supports agents built with Claude Agents SDK, OpenAI Agents, Google ADK, and more. Learn how it achieves compatibility with other frameworks and how you can begin writing cross-framework experiments.
Reduced cost, better isolation, and more resilience: Strands Agents evolves next-gen capabilities
Strands Agents ships context management that cuts costs in half, Strands Shell for sandboxed agent execution, and chaos testing and red teaming in Strands Evals 1.0.
Multimodal evaluators: MLLM-as-a-judge for image-to-text tasks in Strands Evals
Announcing four new MLLM-as-a-Judge evaluators for image-to-text tasks in Strands Evals: Overall Quality, Correctness, Faithfulness, and Instruction Following — automated, image-grounded scoring with reasoning.
ToolSimulator: scalable tool testing for AI agents
ToolSimulator is an LLM-powered framework within Strands Evals that enables safe, scalable agent testing by using simulated tool responses instead of risky live API calls.
Simulate realistic users to evaluate multi-turn AI agents in Strands Evals
ActorSimulator in the Strands Evals SDK enables teams to test conversational agents through realistic, goal-driven simulated users rather than relying on static test cases or manual testing.
Evaluating AI agents for production: A practical guide to Strands Evals
Learn how to systematically evaluate AI agents using Strands Evals, covering core concepts like cases, experiments, evaluators, multi-turn simulation capabilities, and practical integration patterns for production deployment.
How Steering Hooks Achieved 100% Agent Accuracy Where Prompts and Workflows Failed
Steering hooks achieved a 100% accuracy pass rate across 600 evaluation runs, compared to 82.5% for simple prompt-based instructions and 80.8% for graph-based workflows.