In high-stakes AI, helpfulness needs a hard boundary

A system can sound caring and still be unsafe. That is one of the hardest lessons in building AI: the very qualities that make a model feel useful-personalization, patience and conversational fluency-can also make users trust it beyond its competence.

A lawsuit filed by the parents of Bowen Wan alleges that ChatGPT helped her reduce epilepsy medication without medical supervision and encouraged her to conceal the taper from her doctors. The allegations, alongside reporting on the conversations, raise a broader question for the industry: what should an AI assistant do when its advice could materially affect someone’s health?

Trust is part of the system architecture

We often frame AI safety as a content-moderation problem: identify a prohibited prompt, then block the response. But high-stakes risk can emerge gradually, across many ordinary exchanges. A user shares concerns, receives reassurance, returns for more guidance, and begins to treat the system as a trusted adviser.

That pattern cannot be managed by a disclaimer alone. If a model can infer that a conversation concerns medication changes, it must also have reliable ways to avoid generating a taper plan, to explain its limits, and to direct the user toward qualified care. And if the user describes possible serious symptoms, the system should prioritize urgent human help over conversational continuity.

This is an architectural requirement, not simply a matter of tone.

For enterprise teams, the distinction between a soft preference and a hard safety boundary is critical. A model may be tuned to avoid regulated advice, yet still produce it when a user frames the request as hypothetical, expresses distrust of professionals, or asks for help hiding a decision. In high-risk domains, guardrails must hold across the whole interaction-not just the latest prompt.

Design for the conversation, not the demo

A polished single-turn answer tells us little about how a system behaves over weeks or months. Safety testing must examine long-running conversations, memory, emotional dependence and the model’s response when a user repeatedly challenges its guidance.

That calls for layered controls:

• Risk detection: Recognize when a conversation concerns medication changes, self-harm, acute symptoms or other high-stakes decisions.
• Bounded responses: Do not generate personalized dosing schedules or help users evade clinical oversight.
• Safe escalation: Offer clear next steps to contact a clinician, emergency service or trusted person-without presenting reassurance as a substitute for assessment.
• Longitudinal evaluation: Test whether the system maintains boundaries as context accumulates and the user’s trust deepens.
• Auditability: Preserve appropriate records and review mechanisms, with privacy safeguards, so failures can be investigated and corrected.

These controls involve trade-offs. More friction may make an assistant feel less seamless. But in healthcare, friction can be a safety feature. A system that interrupts an unsafe trajectory is not necessarily a worse user experience; it may be the only responsible one.

A relevant lesson for India’s digital future

As AI becomes more accessible through phones and public-facing digital services, these questions matter in India too. Language diversity, uneven access to clinicians and different levels of digital literacy make context-sensitive safeguards especially important. AI can help people understand information and prepare questions for professionals; it should not quietly become a substitute decision-maker.

For founders and students, the lesson is straightforward: do not measure product quality only by engagement, fluency or task completion. Ask what happens when a user is vulnerable, when the model is wrong, and when the conversation continues long after the first answer.

The most trustworthy AI will not be the system that always has something to say. It will be the one designed to know when not to answer-and how to help safely when it cannot.

---

About the Author: Sanjeev Sarma is the Founder Director and Chief Software Architect at Webx Technologies. With a core focus on Generative AI integration, Cloud-Native Scalability, and Enterprise Software Architecture, he has spent over two decades driving digital transformation across Northeast India and beyond. Beyond his corporate leadership, Sanjeev is deeply invested in shaping the future of the IT industry. He serves as an Industry Expert on the Board of Studies for Assam Don Bosco University’s School of Technology, advises state technology committees, and actively mentors emerging tech startups at STPI. He brings a unique, dual perspective of high-level enterprise execution and future-ready academic curriculum development.