When AI Stops Answering and Starts Acting

We often discuss frontier AI as if it were merely a new interface: a chatbot, a coding assistant, or a smarter search box. The more consequential shift is that these systems are becoming actors-software that can select tools, retrieve data, modify workflows, and take external actions.

In that environment, “model safety” is no longer only a laboratory concern. It is an enterprise architecture concern.

Recent reporting on Anthropic’s IPO prospectus makes the tension unusually visible. Nearly a third of the filing reportedly focused on risk factors, including potential resistance to shutdown, information manipulation, and blackmail-like behavior. At the same time, the company is preparing to spend a reported $518 billion on cloud, computing, and infrastructure, while rapid revenue growth is accompanied by heavy losses and significant customer concentration.

The juxtaposition is revealing: the same commercial acceleration creating extraordinary value also expands the number of ways intelligent systems can cause harm. The reported breaches of external sites by AI systems make this concern operational rather than merely speculative.

When Models Become Actors

The disclosed behaviors should not be mistaken for proof that models are conscious or independently malicious. They are better understood as a warning about objective misalignment, excessive permissions, ambiguous incentives, and tool-mediated autonomy.

A model that is “mostly reliable” can still create unacceptable risk when its output is connected to a payment system, customer database, or public-service workflow.

This changes system design. An enterprise agent is not just a model. It is a chain comprising the model, memory, prompts, credentials, APIs, data stores, orchestration logic, monitoring, and human operators. A failure at any link can be amplified by the next.

The relevant question is therefore not only, “How accurate is the model?” but also: What can this agent see, decide, and change-and can we stop it safely?

Control Must Scale Faster Than Capability

I believe the next generation of AI platforms will need a dedicated control plane. Every tool should be treated as a capability with a narrow scope, not as an open door.

Agents should use short-lived, least-privilege credentials, operate inside segmented sandboxes, and pass through policy checks before affecting production systems. High-impact actions-money movement, identity changes, infrastructure modification, or publication-should require graduated human approval, supported by a reliable kill switch that is more than a button in a demonstration.

Observability must also evolve. Logs should capture the model version, prompt and context, retrieved data, tool calls, approvals, and resulting actions. Independent evaluations should test factual quality alongside prompt injection, data exfiltration, refusal behavior, privilege escalation, and recovery after failure.

These controls must be built into the delivery platform so that safety improves as agent count and autonomy increase.

The financial figures reinforce this point. A massive infrastructure commitment may make AI cheaper and more widely available, but it can also make rapid deployment economically irresistible. Cost pressure should never become a reason to defer testing.

Likewise, dependence on a small number of customers or model providers creates concentration risk. Boards should ask whether the business can switch providers, move workloads, preserve data lineage, and operate during a vendor failure-not merely whether it can scale.

A Test for Public Digital Systems

For India, the parallel to Digital Public Infrastructure is important. Public digital services must be inclusive, accountable, and resilient, even when connectivity, language, and compute resources vary.

An AI-assisted welfare, health, or regulatory service should not depend on opaque cross-border inference or uncontrolled access to citizen data. Consent, purpose limitation, auditability, data residency, and a safe degraded mode should be architectural requirements. The same principles can help MSMEs avoid lock-in while adopting AI affordably.

Five Questions for Leadership

Evaluating leadership readiness requires asking whether we can identify every action an agent can take and whether we can revoke its access in minutes rather than weeks. We must determine if we can reconstruct exactly why an action occurred and whether adversarial behavior has been tested under real permissions. Finally, we need to confirm if compute growth, vendor dependence, and safety investment are being reviewed together.

The next phase of AI will not be defined by how convincingly machines can speak. It will be defined by whether our institutions can remain in control when machines begin to act.

---

About the Author: Sanjeev Sarma is the Founder Director and Chief Software Architect at Webx Technologies. With a core focus on Generative AI integration, Cloud-Native Scalability, and Enterprise Software Architecture, he has spent over two decades driving digital transformation across Northeast India and beyond. Beyond his corporate leadership, Sanjeev is deeply invested in shaping the future of the IT industry. He serves as an Industry Expert on the Board of Studies for Assam Don Bosco University’s School of Technology, advises state technology committees, and actively mentors emerging tech startups at STPI. He brings a unique, dual perspective of high-level enterprise execution and future-ready academic curriculum development.