Using Decision Models with Amazon Bedrock AgentCore Evaluations

Amazon Bedrock AgentCore Evaluations allows development and production scoring of AI agents through built-in evaluators, LLM-as-a-judge evaluators, and code-based evaluators running on AWS Lambda. AWS outlined an architecture connecting decision models, such as TypeSafe's hosted Jev model and the open-source Strands Decider 2B, to AgentCore via Lambda functions. Unlike general LLM evaluators that produce free-form text and explanations, decision models are specialized models designed to classify structured inputs into fixed answer sets with calibrated probability distributions. By processing OpenTelemetry spans through Lambda adapters, developers can execute yes/no, multi-choice, and scored evaluation questions to establish numerical thresholds for metrics such as groundedness, helpfulness, and goal completion.

Key takeaways

• Amazon Bedrock AgentCore Evaluations supports code-based evaluators implemented through AWS Lambda, enabling integration with external APIs and decision models alongside 15 built-in evaluators.
• Decision models output calibrated probability distributions over predefined options rather than free-form text, supporting yes/no queries, multi-option selection up to 255 choices, and multi-level scoring rubrics.
• AWS provided reference implementations connecting TypeSafe Jev and the open-source Strands Decider 2B model to evaluate an HR Assistant agent across trace and session levels.