Conviction Collapse: Strategy for Human-Centered AI Software
We used to sell software as a thing you could ship and defend. Today the better metaphor is a studio: a shifting process that assembles skills, agents and experiments – and sometimes discovers the product only after a string of complete rewrites. That shift matters for how architects, CTOs and policy makers design, fund and govern software.
Context
I recently read a piece that coined the term “conviction collapse” to describe how product conviction erodes when iterations move at the pace of large language models and agentic systems: teams raise money and, within days, produce what used to take months. The result is not one monolithic product but a sequence of complete product experiments – each one feeding learnings into the next.
Why this matters for enterprise architecture and strategy
1. Product becomes process. When the delivered artifact is an evolving set of skills and agent behaviors rather than a single packaged app, architecture must prioritise composition, versioned capabilities, and clear contracts. Monoliths die; composition and capability registries live.
2. Speed vs. stability trade-off gets structural. Rapid iteration is powerful for discovery, but it amplifies operational risk: drift, brittle integrations, and emergent behaviour from agent ensembles. Enterprises must choose where they will tolerate volatility (R&D sandboxes) and where they need immutability (finance, identity, public services).
3. Observability, lineage and auditability are non-negotiable. Skills and agents must carry provenance: who trained them, what data fed them, which prompts or orchestration logic produced outputs. Without that telemetry and traceability, debugging and compliance are impossible.
4. Security and boundary discipline changes. Agents internal to a team may behave like collaborators; once they touch PII, payments, or core workflows, they must be treated as supply-chain components with own threat models, access controls and lifecycle policies.
5. Funding and organisational design must follow the tempo of discovery. Traditional nine-month product milestones no longer map to value creation. Expect investors and leaders to need new KPIs that reward rapid hypothesis testing and measurable gated learnings, not only feature velocity.
Actionable guidance for CTOs and founders
– Treat “skills” as first-class artifacts: maintain a skill registry with semantic versioning, test suites, and clear input/output contracts.
– Create a two-speed environment: isolated agent sandboxes for discovery; hardened pipelines with stricter governance for production capabilities.
– Build provenance and lineage into every pipeline: prompt versions, training data hashes, evaluation reports and human review logs.
– Define agent boundaries and least-privilege policies up front. If an agent can access payments or identity, require multi-step approvals and runtime attestation.
– Invest in scenario-based testing (adversarial prompts, bias probes, safety fences) rather than only unit tests.
– Rework funding/milestones to reward validated learning: short experimental budgets with defined success criteria that translate into engineering allocation.
– Preserve reproducibility: containerise runtimes, pin model versions and keep offline fallbacks when service interruption would cause harm.
– Measure success beyond usage metrics: include safety incidents avoided, false-positive/negative rates, and audit completion time.
A note for public systems and India
This model of “product as process” is powerful, but in contexts like Digital Public Infrastructure the balance shifts: citizens expect stability, auditability and long-lived compatibility. In regions with intermittent connectivity (many parts of Northeast India), designs must favour offline-first, deterministic behaviour and explainability. Experimentation belongs in well-isolated labs – not behind public-facing endpoints that citizens rely on daily.
Closing thought
We are moving from producing software artifacts to curating living systems: compositions of skills, social practices and experiments. The new role of the architect is less about lockstep delivery and more about designing safe playgrounds for discovery – with clear fences around the parts we can’t afford to let wander.
About the Author
Sanjeev Sarma is the Founder Director of Webx Technologies Private Limited, a leading Technology Consulting firm with over two decades of experience. A seasoned technology strategist and Chief Software Architect, he specializes in Enterprise Software Architecture, Cloud-Native Applications, AI-Driven Platforms, and Mobile-First Solutions. Recognized as a “Technology Hero” by Microsoft for his pioneering work in e-Governance, Sanjeev actively advises state and central technology committees, including the Advisory Board for Software Technology Parks of India (STPI) across multiple Northeast Indian states. He is also the Managing Editor for Mahabahu.com, an international journal. Passionate about fostering innovation, he actively mentors aspiring entrepreneurs and leads transformative digital solutions for enterprises and government sectors from his base in Northeast India.