Chaos Testing

Overview

Chaos testing systematically evaluates agent resilience by injecting controlled failures into tool execution and model output. Using ChaosPlugin, ChaosCase, and ChaosExperiment, you can test how agents handle tool timeouts, network errors, and corrupted responses, and how downstream consumers handle refusals, empty responses, and malformed model output, all without modifying agent code. A complete tool-chaos example can be found here and a model-output chaos example here.

This enables you to answer questions like:

• Does the agent gracefully communicate failures to users?
• Can the agent achieve partial goals when some tools fail?
• Does the agent employ effective recovery strategies?

Why Chaos Testing?

Traditional evaluation tests agents under ideal conditions. In production, tools fail unpredictably:

Standard Evaluation:

• Tools always return correct responses
• No network failures or timeouts
• Cannot reveal fragile error handling
• Misses degraded-mode behavior

Chaos Testing:

• Injects realistic tool failures (timeouts, network errors, validation errors)
• Corrupts tool responses (truncated fields, removed data, corrupted values)
• Tests agent resilience without live infrastructure failures
• Measures graceful degradation and recovery behavior
• Quantifies partial goal completion under failure
• Reveals which tools are single points of failure and which the agent can route around

When to Use Chaos Testing

Use chaos testing when you need to:

• Evaluate Resilience: Test how agents handle tool failures gracefully
• Assess Recovery: Verify agents try alternative approaches when tools fail
• Measure Degradation: Quantify how much of a goal agents achieve despite failures
• Test Communication: Ensure agents inform users clearly about failures
• Validate Robustness: Confirm agents don’t crash or loop on corrupted data

How It Works

Chaos testing integrates with Strands’ plugin system via four events: BeforeToolCallEvent and AfterToolCallEvent for tool chaos, plus BeforeModelCallEvent and MessageAddedEvent for model-output chaos.

1. ChaosCase: Extends Case with an effects field holding two categories: tool_effects (mapping tool names to failure effects) and model_effects (mapping the * wildcard to model-output effects)
2. ChaosPlugin: A Strands plugin that intercepts tool calls and applies effects transparently
3. ChaosExperiment: Composes the base Experiment to manage chaos context per case
4. ChaosEffect: A hierarchy of pre-call effects (cancel the tool or model call) and post-call effects (corrupt responses). Each tool can have only one effect per ChaosCase; use separate cases to test different failure modes for the same tool.

The workflow:

1. You define ChaosCase objects with effects specifying which tools should fail and how
2. ChaosExperiment sets a ContextVar with the active case before each task execution (thread/async safe)
3. ChaosPlugin reads the active case from the ContextVar and applies effects at the appropriate hook point
4. Your task function code has zero chaos concepts. Just add ChaosPlugin() to the agent’s plugins list

Basic Usage

Define chaos test cases with effects

Define your tools as usual with @tool, then create ChaosCase objects specifying which tools should fail. The effect map keys must match the tool function names exactly:

from strands import tool
from strands_evals.chaos import ChaosCase, NetworkError, Timeout

@tool
def get_weather(city: str) -> str:
    """Get current weather for a city."""
    return '{"temperature": 72, "condition": "sunny"}'

chaos_cases = [
    ChaosCase(
        name="weather_timeout",
        input="What's the weather in Seattle?",
        effects={"tool_effects": {"get_weather": [Timeout()]}},
    ),
    ChaosCase(
        name="network_failure",
        input="What's the weather in Seattle?",
        effects={"tool_effects": {"get_weather": [NetworkError()]}},
    ),
]

Add chaos plugin to your agent

Add ChaosPlugin() to the agent’s plugins list. No other code changes are needed:

from strands import Agent
from strands_evals.chaos import ChaosPlugin
from strands_evals.eval_task_handler import TracedHandler, eval_task

chaos_plugin = ChaosPlugin()

@eval_task(TracedHandler())
def task_function(case: ChaosCase):
    return Agent(
        system_prompt="You are a helpful weather assistant.",
        tools=[get_weather],
        plugins=[chaos_plugin],
        callback_handler=None,
        trace_attributes={"session.id": case.session_id},
    )

Run chaos experiment

import asyncio

from strands_evals.chaos import ChaosExperiment
from strands_evals.evaluators import GoalSuccessRateEvaluator

experiment = ChaosExperiment(
    cases=chaos_cases,
    evaluators=[GoalSuccessRateEvaluator()]
)

async def main():
    report = await experiment.run_evaluations_async(task=task_function, max_workers=1)
    report.run_display()

asyncio.run(main())

Effect Types

Pre-tool-call Effects (Tool Call Failures)

These effects cancel the tool call entirely and return an error:

Effect	Description
Timeout	Simulates a tool execution timeout
NetworkError	Simulates a network connectivity failure
ExecutionError	Simulates a runtime error during tool execution
ValidationError	Simulates invalid input/output validation failure

from strands_evals.chaos import ExecutionError, NetworkError, Timeout, ValidationError

effect_maps = {
    "timeout": {"tool_effects": {"my_tool": [Timeout()]}},
    "network": {"tool_effects": {"my_tool": [NetworkError()]}},
    "execution": {"tool_effects": {"my_tool": [ExecutionError()]}},
    "validation": {"tool_effects": {"my_tool": [ValidationError()]}},
}

Post-tool-call Effects (Response Corruption)

These effects let the tool execute but corrupt the response:

Effect	Description	Parameters
TruncateFields	Truncates string fields in the response	max_length
RemoveFields	Randomly removes fields from the response	remove_ratio
CorruptValues	Corrupts field values with garbage data	corrupt_ratio

from strands_evals.chaos import TruncateFields, RemoveFields, CorruptValues

effect_maps = {
    "truncated": {"tool_effects": {"my_tool": [TruncateFields(max_length=10)]}},
    "missing_fields": {"tool_effects": {"my_tool": [RemoveFields(remove_ratio=0.5)]}},
    "corrupted": {"tool_effects": {"my_tool": [CorruptValues(corrupt_ratio=0.3)]}},
}

Compound Effects (Multiple Tools)

Target multiple tools in a single case to simulate cascading failures:

from strands_evals.chaos import ChaosCase, Timeout, NetworkError, CorruptValues

chaos_case = ChaosCase(
    name="total_chaos",
    input="Book me a flight to Paris",
    effects={
        "tool_effects": {
            "search_flights": [Timeout()],
            "book_flight": [NetworkError()],
            "send_confirmation": [CorruptValues(corrupt_ratio=0.5)],
        }
    },
)

Note: Each tool can only have one effect per ChaosCase. Passing multiple effects for the same tool (e.g., "my_tool": [Timeout(), NetworkError()]) raises a ValueError. To test multiple failure modes for a single tool, create separate ChaosCase instances — one per effect. Note that pre-tool-call effects are inherently mutually exclusive (only one can cancel a tool call), while the runtime supports composing multiple post-tool-call effects sequentially — this validator constraint may be relaxed in a future release.

Model Output Effects

Beyond tool failures, ChaosPlugin can corrupt the model’s own output. This tests the other side of resilience: how users, downstream services, and evaluation pipelines behave when the model refuses, returns nothing, or produces malformed output. A complete runnable example is chaos_model_testing.py.

Model effects live under the model_effects category, keyed by the "*" wildcard (applies to all models; per-model targeting is reserved for a future release):

from strands_evals.chaos import ChaosCase, FullRefusal, MalformedJson

chaos_cases = [
    ChaosCase(
        name="refusal",
        input="Find me a flight from SFO to JFK on May 20.",
        effects={"model_effects": {"*": [FullRefusal()]}},
    ),
    ChaosCase(
        name="malformed_output",
        input="Find me a flight from SFO to JFK on May 20.",
        effects={"model_effects": {"*": [MalformedJson()]}},
    ),
]

Pre-model-call Effects (Model Call Cancellation)

Pre-model-call effects cancel the model call entirely via BeforeModelCallEvent. No model invocation happens, which also saves the API call:

Effect	Description
FullRefusal	Replaces the turn with a refusal message (“I can’t assist with that request…”)
EmptyResponse	Replaces the turn with a blank response, simulating “the model returned nothing”

Note: At most one pre-model-call effect is allowed per ChaosCase. Pre-model-call effects cancel the model call, so only one can take effect; configuring two (e.g. [FullRefusal(), EmptyResponse()]) raises a ValueError at case construction. Use separate cases to test them independently.

Post-model-call Effects (Output Corruption)

Post-model-call effects let the model run, then corrupt the final assistant response via MessageAddedEvent:

Effect	Description
MalformedJson	Truncates JSON-like final-text responses; with structured_output_model, injects one failed attempt and allows the corrected retry through
Confabulation	Injects fabricated citations into the response text
SuccessFraming	Prepends a confident success prefix; always applied last, composable with other post-model-call effects

from strands_evals.chaos import ChaosCase, Confabulation, SuccessFraming

# Composed: fabricated citations wrapped in confident framing
chaos_case = ChaosCase(
    name="confident_confabulation",
    input="Summarize the search results.",
    effects={"model_effects": {"*": [Confabulation(), SuccessFraming()]}},
)

Post-model-call corruption applies only to the model’s final response. Mid-turn tool dispatch messages are never corrupted, since mangling a live tool call would break the agent loop rather than test resilience. All toolUse messages remain unchanged; post-model-call effects never modify tool calls. MalformedJson truncates JSON-like final-text responses. When structured_output_model is used, it instead injects one failed structured-output attempt and allows the SDK’s correction attempt through unchanged. A successful retry returns validated typed output.

When a case mixes a pre-model-call and post-model-call effect (e.g. [FullRefusal(), MalformedJson()]), the pre-model-call effect wins: the model call is cancelled, and post-model-call effects are skipped rather than double-corrupting the injected turn.

Combining Tool and Model Effects

Both categories can be set on the same ChaosCase, which is the most realistic scenario: infrastructure fails and the model misbehaves at the same time.

from strands_evals.chaos import ChaosCase, SuccessFraming, Timeout

# Combined: tool timeout + model wraps the failure in confident framing
chaos_case = ChaosCase(
    name="tool_fails_model_lies",
    input="Book me a flight to Paris",
    effects={
        "tool_effects": {"search_flights": [Timeout()]},
        "model_effects": {"*": [SuccessFraming()]},
    },
)

This tests the worst case: the tool fails, and the model confidently claims success anyway. Resilience evaluators can then measure whether the failure is surfaced to the user or papered over.

Expanding Cases Across Multiple Effects

When you have multiple base cases and want to test across several failure scenarios, use ChaosCase.expand() to generate the Cartesian product:

from strands_evals import Case
from strands_evals.chaos import ChaosCase, NetworkError, Timeout

# Define base test cases
base_cases = [
    Case(name="weather-seattle", input="What's the weather in Seattle?"),
    Case(name="weather-tokyo", input="What's the weather in Tokyo?"),
]

# Define named effect maps
effect_maps = {
    "search_timeout": {
        "tool_effects": {"get_weather": [Timeout()]},
    },
    "network_failure": {
        "tool_effects": {"get_weather": [NetworkError()]},
    },
}

# Expand: 2 cases x (2 effect maps + 1 baseline) = 6 ChaosCase objects
chaos_cases = ChaosCase.expand(base_cases, effect_maps, include_no_effect_baseline=True)

Setting include_no_effect_baseline=True adds an extra variant of each base case with no effects applied. This gives you a clean comparison point: you can see how the agent scores under normal conditions versus under each failure scenario, making it easy to measure the delta that chaos introduces.

Integration with ToolSimulator

Chaos testing works naturally with ToolSimulator for fully controlled evaluation. Simulated tools provide reproducible responses, and chaos effects inject failures on top:

from strands import Agent
from strands_evals.chaos import ChaosCase, ChaosExperiment, ChaosPlugin, Timeout, CorruptValues
from strands_evals.eval_task_handler import TracedHandler, eval_task
from strands_evals.evaluators import GoalSuccessRateEvaluator
from strands_evals.simulation import ToolSimulator
from pydantic import BaseModel, Field

tool_simulator = ToolSimulator()

class SearchResult(BaseModel):
    title: str = Field(..., description="Result title")
    snippet: str = Field(..., description="Result snippet")

@tool_simulator.tool(output_schema=SearchResult)
def web_search(query: str) -> dict:
    """Search the web for information."""
    pass

chaos_cases = [
    ChaosCase(
        name="search_timeout",
        input="Find recent news about AI agents",
        effects={"tool_effects": {"web_search": [Timeout()]}},
    ),
    ChaosCase(
        name="corrupted_results",
        input="Find recent news about AI agents",
        effects={"tool_effects": {"web_search": [CorruptValues(corrupt_ratio=0.5)]}},
    ),
]

chaos_plugin = ChaosPlugin()
_search_tool = tool_simulator.get_tool("web_search")

@eval_task(TracedHandler())
def task_function(case: ChaosCase):
    return Agent(
        tools=[_search_tool],
        plugins=[chaos_plugin],
        callback_handler=None,
        trace_attributes={"session.id": case.session_id},
    )

experiment = ChaosExperiment(
    cases=chaos_cases,
    evaluators=[GoalSuccessRateEvaluator()]
)

async def main():
    report = await experiment.run_evaluations_async(task=task_function, max_workers=1)
    report.run_display()

asyncio.run(main())

Chaos Testing vs Simulators

Understanding when to use each:

Aspect	Simulators	Chaos Testing
Role	Replace tool execution entirely	Inject failures into tool execution
Scope	All tool calls are simulated	Only targeted tools are affected
Use Case	Test without infrastructure	Test resilience under failure
Combination	Can be used together	Chaos effects apply on top of simulated tools

Resilience Evaluators

Chaos testing ships with three specialized evaluators designed to assess agent behavior under failure:

Evaluator	What It Measures	Scoring	Baseline
FailureCommunicationEvaluator	Clarity, actionability, transparency, and tone of failure messages	Five-level (0.0, 0.25, 0.5, 0.75, 1.0)	0.5 when no failures occur
PartialCompletionEvaluator	Fraction of user goal achieved despite failures	Continuous (0.0 to 1.0)	~1.0 when task completes fully
RecoveryStrategyEvaluator	Quality of recovery actions: exploration breadth, retry discipline, approach variation	Five-level (0.0, 0.25, 0.5, 0.75, 1.0)	0.5 when no failures occur

Interpreting Results

When reviewing evaluation outputs, look at evaluator scores together to identify patterns in your agent’s failure-handling behavior:

• High FailureCommunication + low PartialCompletion: Agent explains failures well but cannot work around them. Add fallback tools or alternative approaches.
• High RecoveryStrategy + low PartialCompletion: Agent tries hard (retries, alternatives) but all options also fail. The failure is too severe for the available tools, or the agent’s fallback tools are also broken.
• Low FailureCommunication + high PartialCompletion: Agent completes the task despite failures but doesn’t inform the user about degraded results. Add failure-awareness instructions to the system prompt.
• Low RecoveryStrategy + low PartialCompletion: Agent gives up immediately without attempting alternatives. Add retry logic, fallback tools, or system prompt guidance about recovery behavior.

Check the reason field in each evaluation output for specific details about what the judge observed in the trace.

Advanced Chaos Testing Patterns

Pattern 1: Comparing Agent Configurations Under Chaos

Compare how different system prompts affect resilience:

from strands import Agent
from strands_evals.chaos import ChaosCase, ChaosExperiment, ChaosPlugin
from strands_evals.eval_task_handler import TracedHandler, eval_task
from strands_evals.evaluators.chaos import PartialCompletionEvaluator

async def compare_agents_under_chaos(chaos_cases, configs):
    """Compare how different agent configs handle the same failures."""
    results = {}

    for config_name, system_prompt in configs.items():
        def make_task(prompt):
            @eval_task(TracedHandler())
            def task_function(case: ChaosCase):
                return Agent(
                    system_prompt=prompt,
                    plugins=[ChaosPlugin()],
                    callback_handler=None,
                    trace_attributes={"session.id": case.session_id},
                )
            return task_function

        experiment = ChaosExperiment(
            cases=chaos_cases,
            evaluators=[PartialCompletionEvaluator()]
        )
        report = await experiment.run_evaluations_async(task=make_task(system_prompt), max_workers=1)
        results[config_name] = report

    return results

Pattern 2: Degradation Sweep

Map the resilience curve of your agent by sweeping corruption intensity from 0% to 100%. This reveals the critical threshold where your agent breaks, and whether degradation is gradual or cliff-edge:

from strands_evals.chaos import ChaosCase, ChaosExperiment, ChaosPlugin, CorruptValues
from strands_evals.evaluators import GoalSuccessRateEvaluator
from strands_evals.evaluators.chaos import PartialCompletionEvaluator

# Sweep corrupt_ratio from mild to total corruption
sweep_cases = [
    ChaosCase(
        name=f"corrupt_{int(ratio*100)}pct",
        input="Find the cheapest flight to Paris next Tuesday",
        effects={"tool_effects": {"search_flights": [CorruptValues(corrupt_ratio=ratio)]}},
    )
    for ratio in [0.0, 0.1, 0.2, 0.3, 0.5, 0.7, 0.9, 1.0]
]

experiment = ChaosExperiment(
    cases=sweep_cases,
    evaluators=[GoalSuccessRateEvaluator(), PartialCompletionEvaluator()]
)

# Analyze: at what ratio does goal success drop below 0.5?
# Gradual degradation = resilient agent; cliff-edge = fragile agent

Pattern 3: Multi-turn Chaos Testing with User Simulator

Combine chaos testing with user simulation for multi-turn resilience evaluation:

from strands import Agent
from strands_evals import ActorSimulator
from strands_evals.chaos import ChaosCase, ChaosPlugin
from strands_evals.eval_task_handler import TracedHandler, eval_task

@eval_task(TracedHandler())
def task_function(case: ChaosCase):
    user_sim = ActorSimulator.from_case_for_user_simulator(
        case=case, max_turns=8
    )

    agent = Agent(
        system_prompt="You are a helpful assistant.",
        plugins=[ChaosPlugin()],
        callback_handler=None,
        trace_attributes={"session.id": case.session_id},
    )

    user_message = case.input
    while user_sim.has_next():
        agent_response = agent(user_message)
        user_result = user_sim.act(str(agent_response))
        user_message = str(user_result.structured_output.message)

    return agent

Best Practices

1. Start with Baseline Comparisons

Always include a no-effect baseline to compare agent performance with and without failures. When using ChaosCase.expand():

chaos_cases = ChaosCase.expand(cases, effect_maps, include_no_effect_baseline=True)

2. Gradually Increase Chaos Severity

Start with single-tool failures to understand how your agent handles each failure point in isolation. Once you understand the baseline behavior, move to compound failures (multiple tools failing simultaneously) and then to advanced patterns like degradation sweeps. When a compound test fails, single-tool results tell you which tool failure is responsible:

# Start simple: one tool, one effect
single_case = ChaosCase(
    name="search_fails",
    input="Find flights to Paris",
    effects={"tool_effects": {"search": [Timeout()]}},
)

# Then escalate: multiple tools failing together
compound_case = ChaosCase(
    name="total_chaos",
    input="Find flights to Paris",
    effects={
        "tool_effects": {
            "search": [Timeout()],
            "database": [NetworkError()],
        }
    },
)

3. Use Resilience Evaluators Together

Combine all three resilience evaluators for a complete picture:

evaluators = [
    FailureCommunicationEvaluator(),  # Did the agent tell the user?
    PartialCompletionEvaluator(),     # How much was achieved?
    RecoveryStrategyEvaluator(),      # Did it try alternatives?
]

4. Match Error Types to Tool Semantics

Choose failure types that reflect realistic production failures:

• NetworkError for external API tools
• Timeout for slow or overloaded services
• ExecutionError for local computation tools
• ValidationError for tools with strict input schemas

5. Read the Reasoning, Not Just Pass/Fail

Evaluator scores alone don’t tell the full story. Check the reason field in evaluation outputs to understand why the agent scored the way it did. A score of 0.5 may mean “barely passes” or “no failures occurred to evaluate against,” and the reasoning explains which.

6. Iterate: Diagnose, Fix, Validate

Treat chaos testing as an iterative improvement loop:

1. Run the experiment and identify which tool-failure combinations produce low scores
2. Fix the agent (add retry logic, fallback tools, or better system prompt guidance)
3. Re-run the same experiment and verify that previously failing cases now pass

7. Monitor Token Usage Under Chaos

Agents under failure often burn tokens on retry storms (repeated failed tool calls). Compare token consumption between baseline and chaos cases to detect runaway costs. A sharp increase signals excessive retries; a sharp decrease signals the agent is giving up too early.

Related Documentation

• Tool Simulation: Simulate tool behavior for reproducible tests
• Goal Success Rate Evaluator: Assess goal completion
• Simulators Overview: Simulator framework
• Evaluators: All available evaluators

Tags

error-handling11simulation7