We have a habit of applauding AI for what it can do in seconds – summarise a report, auto-generate code, draft an email – and then treating the resulting output as if it were an unquestionable fact. A recent high-profile product tour drove that tension home: the brightest stage lights, the flash demos, and an almost constant on-screen reminder – “Copilot uses AI. Check for mistakes.” That little footnote is the real product announcement.
Context
A global technology vendor used a major AI tour to showcase agentic assistants and productivity gains, while simultaneously flagging that AI outputs require human verification. The signal was clear: advances in capability are outpacing our operational practices for trust, provenance and risk control.
Analysis – what this means for architecture and leadership
The principal lesson for CTOs and architects is that AI is not just another component to bolt on. It changes the nature of decision-making in systems. When a model’s output can directly affect a citizen, customer or legal outcome, the system needs composable trust: explainability, auditable provenance, real-time validation and human-in-the-loop checkpoints.
Key implications and pragmatic trade-offs:
– Speed vs. Stability: Pushing models into production fast improves time-to-value but increases the probability of “sloppy” outputs. That sloppy-then-fixed cycle creates technical and reputational debt. Plan for iterative rollouts with conservative default permissions.
– Build vs. Buy: Vendor-provided assistants accelerate adoption, but they opaque model internals. If you buy, insist on contract-level guarantees – model cards, hallucination metrics, retraining cadence and remediation obligations.
– Observability for AI: Traditional monitoring (CPU, latency, error rates) is necessary but insufficient. Monitor semantic degradation: output confidence distributions, distributional shifts in inputs, and policy-violation rates. Treat these as first-class SLOs.
– Data lineage and provenance: Every critical output must be traceable back to the input data, model version and prompt. This is the difference between a helpful suggestion and an unexplainable decision that harms someone.
– Human-in-Loop and Guardrails: UI nudges (“check for mistakes”) are not a replacement for gating. For high-risk domains, enforce mandatory human sign-off, differential access, or read-only outputs until confidence thresholds are met.
Actionable steps for technology leaders
– Appoint an AI risk owner who reports to the architecture board; make model risk a board-level KPI.
– Define and enforce AI SLOs (hallucination rate thresholds, latency, explainability coverage) and include them in procurement.
– Build an AI observability stack: input validation, model monitoring, feedback loops, and a tamper-evident audit trail for outputs used in decisions.
– Use canary deployments and red-team adversarial testing before broad release.
– Train operators and end-users: most incidents are preventable with simple verification workflows and escalation protocols.
The India angle – why this matters for our digital public systems
This is not an abstract problem for markets with mature regulation alone. In India, where digital public infrastructure (DPI) services – from citizen grievance portals to e-health records – are expanding rapidly, an AI hallucination can have outsized social impact. Where decisions are tied to benefits, identity, or policing, provenance and enforceable human review are non-negotiable. In advisory work with state technology bodies, I have emphasised including ML audit trails and proof-of-origin clauses in DPI procurement documents to protect citizens and institutions alike.
Takeaways
– Treat AI outputs as probabilistic, not authoritative.
– Require provenance, model cards and hallucination SLAs from vendors.
– Invest in AI observability and mandatory human gates for high-risk actions.
– Make AI risk a governance metric at the board level.
Closing thought
We are building systems that shape judgment – not merely automate tasks. Technical elegance without operational guardrails is negligence; the future we should design is one where speed and trust rise together.
About the Author
Sanjeev Sarma is the Founder Director of Webx Technologies Private Limited, a leading Technology Consulting firm with over two decades of experience. A seasoned technology strategist and Chief Software Architect, he specializes in Enterprise Software Architecture, Cloud-Native Applications, AI-Driven Platforms, and Mobile-First Solutions. Recognized as a “Technology Hero” by Microsoft for his pioneering work in e-Governance, Sanjeev actively advises state and central technology committees, including the Advisory Board for Software Technology Parks of India (STPI) across multiple Northeast Indian states. He is also the Managing Editor for Mahabahu.com, an international journal. Passionate about fostering innovation, he actively mentors aspiring entrepreneurs and leads transformative digital solutions for enterprises and government sectors from his base in Northeast India.
