Motorway, a UK-based online car marketplace handling up to 8,000 dealer bids on 2,500 vehicles daily, partnered with AWS Prototyping and AI Customer Engineering (PACE) to build an AI-powered dealer stock search agent that replaces hours of manual filtering with natural language queries (AWS Machine Learning Blog). The agent faced critical challenges including tool selection errors, semantic search misinterpretations, context drift in multi-turn conversations, and non-deterministic outputs that made single-trial testing unreliable (AWS Machine Learning Blog).
Together, Motorway and AWS built an evaluation pipeline that reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection time from few hours to few minutes (AWS Machine Learning Blog). The solution combines a two-phase evaluation strategy—build-time testing with strands-agents-evals and production monitoring with Amazon Bedrock AgentCore Evaluations—with a three-layer framework assessing tool usage (greater than 95 percent threshold), reasoning (greater than 85 percent threshold), and output quality (greater than 90 percent threshold) (AWS Machine Learning Blog). For commerce practitioners managing high-concurrency customer-facing agents, this blueprint demonstrates how to gate deployments on reliability metrics like pass^k—the probability of succeeding in k consecutive trials—ensuring users experience consistent quality on every interaction rather than sporadic failures.
AWS provides a deployable companion repository with a five-stage deployment pipeline and least-privilege security practices, with estimated evaluation costs of $5–10 in Amazon Bedrock inference charges for the sample suite (AWS Machine Learning Blog). The core principles are system-agnostic and applicable to any production-ready AI agent, making this framework essential guidance for e-commerce teams deploying autonomous agents that handle real transactions and dealer trust.