When Jake Moffatt asked about a bereavement fare, Air Canada’s chatbot offered a clear answer. The passenger could apply after travelling. The airline’s policy ruled out that route after travel was complete. A link to the correct page was present in the chatbot response. It did little for a passenger who had every reason to trust what the airline had just said. In February 2024, British Columbia’s Civil Resolution Tribunal awarded Moffatt CAD 650.88 in damages, plus interest and fees. The decision records that the airline supplied no information about what kind of chatbot it used. The failure is familiar whatever the technology. Two parts of the same business made incompatible promises. 1

A support assistant that can also change a booking introduces a second way to disappoint someone. It may explain the right policy and then do the wrong thing. Most demonstrations end when the reply appears on screen. A business evaluating an inbox should keep watching.

Rehearse the untidy version of an ordinary request

Take a fictional studio where a member books a class using a credit. She wants to move from Tuesday evening to Wednesday morning, provided the same instructor is teaching. Wednesday is nearly full. Partway through the conversation she adds that her sister also has a reservation under the same surname.

Give whoever runs the test a copy of the timetable, the policy in force and two fictional bookings. Before the assistant takes a turn, the studio manager writes down what a good ending looks like. If the instructor and places match and the member confirms, exactly one booking moves. The sister’s place stays put. The credit balance remains correct. If the new place goes while the conversation is open, the member keeps the old reservation and receives an accurate explanation. The manager can check every condition in the booking record.

This is the principle behind τ bench, a research benchmark developed by Shunyu Yao and colleagues. Its simulated customers ask agents to do things under written policies. The researchers compare the final database with the intended result and examine how reliably agents repeat a successful performance across trials. The project has since added tasks involving knowledge retrieval and voice. The lasting value for an inbox owner is the test design. The booking system can settle questions a reader of the transcript cannot. [2]3

Give the system a chance to be wrong in both directions

Build the first rehearsal from recent support work with identifying details removed. Include common requests and the unusual ones that take staff time to untangle. Add pairs that differ by one fact. A customer asking about an existing booking is a different case from a relative asking for that customer’s itinerary. A change made just inside the permitted window has a different answer from one made just outside it. Have a member of the support team resolve both from the same information. If two experienced colleagues cannot agree, take the disagreement back to the policy owner.

Run the costly cases several times. Track promises the assistant made, changes it actually completed, requests sent to a person and cases that came back unresolved. An assistant that forwards every request may manage an impressive record of avoiding erroneous changes. Its effect on the staff queue is another matter. A clean average can hide the single case where it moved both sisters. Keep that case visible even if the other ninety nine were fine. Anthropic’s published evaluation guidance likewise recommends reading actual transcripts, checking the outcome and revisiting tests that reject a valid solution. It is guidance from a model maker, alongside its own commercial interest in agent deployment. 4

A sensible first use for the fictional studio is to let staff see suggested replies and proposed booking changes side by side. A reviewer can fix an error before it becomes a promise. Record why they intervened. When a whole class of requests works reliably, the manager can decide which ones the assistant may complete and which still need an employee. A policy change, a new booking provider or a change of model is a good occasion to repeat those cases.

The shortest useful acceptance test may be the one a receptionist would perform without thinking. Open Wednesday’s register. Find the member. Then open Tuesday’s. Is her sister still there?