When an AI Bot Rewrote Airline Policy — And a Tribunal Agreed
In 2024, an Air Canada customer turned to the airline's chatbot for help with bereavement fares. The bot replied with confidence, outlining a refund policy that, as it turned out, didn't actually exist. The airline refused to honor it. A tribunal ultimately ruled in the customer's favor. The chatbot hadn't made a decision — it had predicted an answer based on patterns in its training data. The company, however, had treated that prediction as established policy.
This incident isn't just a cautionary tale about chatbot guardrails. It exposes a fundamental tension at the heart of modern product design: probabilistic systems wrapped in deterministic interfaces. The AI makes a statistically informed guess, the interface presents it as settled fact, and users — or entire organizations — act on it as though it were gospel.
For designers, product managers, and anyone building with AI today, this gap is one of the most important problems to solve.
The Human Brain Is Wired for Certainty
Understanding why this gap is so dangerous requires a brief detour into how humans think. By default, most of us reason deterministically. We assume that past actions reliably produce predictable future outcomes, that causes lead to effects in neat, traceable chains. Flip a coin 999 times, get heads every time, and the deterministic mind concludes the coin must be rigged. The probabilistic mind, on the other hand, accepts that the 1000th flip is still a 50/50 proposition regardless of what came before.
That second mindset is harder to sustain. It requires holding ambiguity without resolving it prematurely. But it is precisely the mindset that designers need to cultivate in an era defined by AI-assisted decision-making. Products operate in complex, nonlinear environments. AI is accelerating that complexity rather than simplifying it. When teams treat AI outputs as the answer rather than one of many possible answers, they build brittle experiences — and in high-stakes domains like medical diagnostics, financial forecasting, or legal guidance, they can build genuinely harmful ones.
What Probabilistic Thinking Actually Means in Practice
Probabilistic thinking doesn't mean being indecisive or drowning users in caveats. It means designing systems that accurately represent the confidence level behind any given output, and it means using AI as a thinking partner rather than an oracle.
Most questions posed to AI do not produce binary answers. They produce probability distributions shaped by patterns in training data. Ask an AI whether extraterrestrial life exists, and you won't get a yes or no — you'll get a framing of the question, one that acknowledges the scientific plausibility of life elsewhere while noting the absence of confirmed evidence. The answer doesn't close the question; it contextualizes it. That is how designers should think about every AI-assisted output in a product: as a frame, not a conclusion.
This reframing has practical implications for how AI-powered features should be built, communicated, and tested.
Three Core Principles for Designing Probabilistically With AI
1. Surface Uncertainty Honestly in the Interface
If an AI model is 70% confident in a recommendation, the user experience should reflect that. This doesn't mean cluttering every screen with percentage scores or technical jargon. It means using language, visual hierarchy, and interaction patterns that communicate the degree of confidence without overstating it. Phrases like "Based on available data, this looks likely" communicate uncertainty without alarming the user. Presenting the same output as a definitive fact, however, creates a false sense of reliability that erodes trust the moment the system is wrong — and all systems are wrong sometimes.
2. Account for Model Bias and the Limits of Training Data
AI models are not neutral. They reflect the data they were trained on, including its gaps, historical biases, and blind spots. A model trained primarily on data from one demographic group will produce outputs that are less reliable for others. Designers who ignore this reality build products that perform unevenly and unfairly. Incorporating bias audits into the design and testing process, and being transparent with users about the scope and limitations of a model's knowledge, is not optional — it is a core design responsibility.
3. Design for Human Override at Every Critical Juncture
Probabilistic outputs should inform human judgment, not replace it. In any context where the consequences of a wrong answer are significant — a medical second opinion, a financial recommendation, a legal interpretation — the interface should actively create space for human review and override. This means designing workflows where AI surfaces options rather than dictates outcomes, where decision checkpoints are visible rather than buried, and where users are never left feeling that they had no agency in a consequential choice.
AI as a Sharpening Tool, Not a Shortcut
The most valuable way to use AI in a design process is as a mechanism for stress-testing assumptions. Feed a design brief into an AI model and ask it to identify edge cases, failure modes, or underrepresented user scenarios. Use AI to generate a range of possible outcomes for a given feature, then evaluate those outcomes through the lens of perceived risk and human sentiment. The goal is to expand the possibility space before committing to a direction, not to narrow it prematurely by anchoring to the AI's first suggestion.
This is where the Air Canada case becomes instructive in a second, less obvious way. The failure wasn't purely technical. It was organizational. Somewhere in the chain between model output and customer-facing product, a team treated a prediction as a policy. Building probabilistic thinking into design culture means training every stakeholder — not just engineers and UX designers, but product owners, legal teams, and executives — to ask the same question: what is the confidence level behind this output, and what are the consequences if it's wrong?
Building Products That Are Honest About What They Don't Know
There is a competitive advantage waiting for the teams that get this right. Users have grown more sophisticated about AI. They know these systems make mistakes. What they want — and what builds lasting trust — is honesty about the nature of those mistakes and genuine investment in designing around them.
Products that surface uncertainty clearly, account for model limitations transparently, and keep humans meaningfully in the loop are not weaker than products that project false confidence. They are more resilient, more trustworthy, and ultimately more useful. In a landscape where AI capabilities are advancing faster than most teams' ability to evaluate them, probabilistic thinking isn't a design philosophy for specialists. It is the baseline competency every product team needs to develop.
The future of designing with AI is not about finding the model with the highest accuracy score and trusting it completely. It is about building the organizational habits, the interface patterns, and the team culture to work thoughtfully within the bounds of what any model — however powerful — genuinely cannot know for certain.