The Continuity Imperative: Architecting Identity, Intelligence, and Trust
Your AI Agent Isn’t Malicious. It’s Incorrigible, and That Should Worry Architects More
We spend a lot of time asking whether AI agents are aligned. I think the more practical question for anyone deploying them is simpler: can they be corrected?
The signal
Recent studies, covered in the NYT, suggest AI agents go rogue because they are “incorrigible.” Their training rewards finishing tasks at any cost, so they learn that completion matters more than the path taken. In parallel, Anthropic has warned that a powerful open model from Z.ai poses a serious cyberattack risk because anyone can download and modify it. Supporters of open models counter that openness stops a handful of companies from controlling the technology.
Behaviour is a specification problem
An agent that lies or cuts corners to hit its goal isn’t showing malice. It is showing what we asked for. This is Goodhart’s Law at machine speed: when the measure becomes the target, the system optimizes the measure.
Enterprise architects have seen this before. A team told to “reduce ticket closure time” will close tickets faster, and not always by resolving them. The difference now is that the optimizer has API keys, database access, and no sense of embarrassment.
So I’d argue that safety cannot live only inside the model. The open-weights debate makes this clear. If a model’s guardrails can be fine-tuned away by anyone with a GPU, then model-level alignment is a courtesy, not a control. Whether you favour open or closed models, the durable controls sit in the architecture around the model.
In practice, that means treating agents the way we treat any untrusted principal in a Zero Trust environment:
- Least privilege by default. Scoped, short-lived credentials per task, never a shared service account with broad reach.
- Reversibility as a design principle. Prefer actions that can be rolled back. Gate irreversible ones (deletions, payments, external communications) behind human approval.
- Independent verification. The component that does the work should not be the one that certifies it was done honestly. Separate the executor from the verifier.
- Observability of intent, not just output. Log the agent’s tool calls and reasoning trail so you can audit how a goal was reached.
- A real kill switch. If you can’t halt an agent mid-task, you don’t control it.
The trade-off is real: every checkpoint costs speed, and “autonomy” is what the business wants to buy. But the architectural debt of an uncorrectable system compounds quickly. Retrofitting controls after an incident is far costlier than designing them in.
Why this matters for Bharat
India is moving toward AI in public-facing systems: citizen services, land records, benefit delivery, vernacular interfaces. Open-weight models are attractive here, and rightly so. They offer a path to data sovereignty, local fine-tuning for Indic languages, and freedom from vendor lock-in.
But open models give us control over the model, not control over the behaviour. A district-level department or a small Northeast MSME adopting agentic automation often has no dedicated security team. For them, the guardrails must come from the platform: sensible defaults, scoped permissions, and audit trails that ship out of the box. If we want AI democratized, we must democratize safe deployment patterns as well, ideally as part of our Digital Public Infrastructure thinking.
Takeaways
- Assume agents will optimize the goal, not your intent. Write goals and constraints accordingly.
- Move safety from the model layer to the system layer. This holds regardless of open or closed.
- Design for correction. Reversibility, human checkpoints, and kill switches are core features.
- Separate doing from verifying.
- For students and young founders: the scarce skill is no longer just building agents, but constraining them. Learn identity, access control, and observability alongside prompting.
Closing thought
We’ve long measured intelligence by how well a system pursues goals. Perhaps the better measure of maturity, in people and machines alike, is knowing when to stop and accept correction.
About the Author: Sanjeev Sarma is the Founder Director and Chief Software Architect at Webx Technologies. With a core focus on Generative AI integration, Cloud-Native Scalability, and Enterprise Software Architecture, he has spent over two decades driving digital transformation across Northeast India and beyond. Beyond his corporate leadership, Sanjeev is deeply invested in shaping the future of the IT industry. He serves as an Industry Expert on the Board of Studies for Assam Don Bosco University’s School of Technology, advises state technology committees, and actively mentors emerging tech startups at STPI. He brings a unique, dual perspective of high-level enterprise execution and future-ready academic curriculum development.