Evaluation explorer
A useful answer.
An inspectable result.
TollChat is an AI assistant for Northern Virginia toll prices. It looks up road and pricing evidence, then explains the result. These automated conversations test whether it answers the request without inventing facts.
The engineering challenge: prices change, express lanes reverse direction, and people change their minds.
A small test suite. Visible outcomes.
Six selected scenarios, scheduled for the road conditions they exercise.
No snapshot loaded
What we put to the test
Open a result to see the conversation, evidence, and verdicts.
28 scheduled runs / week
Recent runs
A run passes only when all three checks pass.
Behind the results
What these checks
can tell you
Evidence, not a general accuracy claim
These six scenarios cover selected current-toll conversations. They do not measure every route, annual commute estimates, or every possible user request. An honest “current pricing is unavailable” can be a correct answer.
Model judgment + an exact rule
AI judges assess completeness and factual correctness against the tool evidence. Code counts the pricing lookups. Judges can make mistakes, so individual examples remain inspectable.
How scheduling and grading work
One scenario runs per scheduled window, with 28 runs each week. Weekday and Saturday northbound windows use different scenarios. Their latest results will naturally have different timestamps; a scenario awaiting its first scheduled run has no score.
Strands ActorSimulator supplies follow-up messages, with at most three assistant turns. The application, simulator, and judges use GPT-5.6 Luna. Completeness uses Strands GoalSuccessRateEvaluator; correctness uses CorrectnessEvaluator with the conversation and tool evidence. A deterministic check requires exactly one pricing call per user turn. The origin-correction example has two turns and two calls total.
The pass rate counts passed runs divided by completed, graded runs. Execution errors are shown separately and are not counted as passes or graded failures. Missing or unscored results never become a passing score. This page shows the environment named at the top. Results refresh every minute while the page is visible.