2Why AI governance, why now

Three forces have converged to make AI governance an immediate obligation rather than a future ambition: the nature of the technology has changed, the Australian mandate has hardened, and the cost of getting it wrong is now well understood. This section sets out each, and then frames the three tests that the rest of the framework is built to satisfy.

2.1From automation to agency

The move from analogue to digital was forgiving. A broken process stopped, a failed input halted the flow, and an organisation could dial automation up or down, choosing how much to hand to the machine. There were always shades of grey. Agentic AI removes them. Autonomy is closer to binary: once an agent is activated, it acts. Bad data does not stop it, it distorts its decisions, and in a network of interacting systems that distortion compounds before any human notices. The decisive design question is no longer what the person will experience, but how the system will behave when no one is watching.

This is why governance cannot remain what it has always been. Traditional governance is retrospective: logs are reviewed, policies audited and compliance verified after the fact. By the time a log is reviewed, an autonomous system has already acted, perhaps thousands of times. Governing AI means moving controls from after the event to the moment of action, from documents to executable guardrails, from oversight to participation. The figure below captures the shift.

Digital era

People decide; systems automate tasks. Governance looks back: logs reviewed, policies audited, compliance checked after the fact.

AI era

Systems decide, at machine speed. Governance acts at the moment of action: executable guardrails, runtime gates, people as participants.

Importantly, this does not mean every agency is deploying autonomous agents tomorrow. Most government AI today sits at the assistive end of the spectrum: tools and assistants that suggest, draft and analyse while a person decides. But the direction of travel is clear, and the discipline required to govern the autonomous end is the same discipline that makes the assistive end safe. Building it now, while the stakes are lower, is how an agency earns the right to use AI more ambitiously later.

2.2The Australian moment

Australia has chosen a distinctive path. Rather than a single AI Act on the European model, the National AI Plan of December 2025 confirmed a technology-neutral approach: existing laws and sector regulators carry the load, supported by voluntary guidance and a new Australian AI Safety Institute. For government agencies, however, the settings are not voluntary. The Policy for the responsible use of AI in government binds non-corporate Commonwealth entities, and across 2025 it grew teeth. Agencies must now have an AI strategy, named accountability, a use-case register, impact assessments and public transparency statements, and must align their AI to a technical standard and an assurance framework. The era of experimentation without governance has closed.

This places a particular onus on agencies that adopt AI through partners and platforms rather than building it themselves. Accountability cannot be outsourced. When a vendor brings proprietary AI tools into an agency’s environment, the agency remains accountable for the outcome, for the impact assessment, for the transparency statement, and for the security and sovereignty of its data. The framework is built to make that accountability real and dischargeable, not nominal.

2.3The cost of getting it wrong

Australia already has a defining lesson in what happens when automated decision-making is deployed without adequate governance. The Royal Commission into the Robodebt Scheme found that an automated approach to debt-raising, built on a flawed assumption encoded in software, produced unlawful decisions at scale, caused profound harm, and corroded public trust. Robodebt was not, strictly, an AI system. But its lessons are the AI governance agenda in miniature: a decision rule embedded in code must be lawful; the people affected must be able to understand and contest it; a human must be accountable; and the system must be tested against reality before it is trusted with people’s lives. An AI system that rewrites the application of policy into new code, at speed and at scale, raises every one of these risks in sharper form.

Four systemic risks recur whenever the move to AI runs ahead of its governance, and they map cleanly onto public-sector failure modes:

The purpose of a self-balancing framework is to counter exactly these. It keeps strategy connected to delivery, delivery grounded in architecture, architecture expressed in safe design, and design held to account by governance that learns and feeds back. The point is not to eliminate risk, which is impossible, but to make the system observable, correctable and accountable: to replace blind automation with governable autonomy, what a public servant would call earned confidence.

2.4Safe, secure and effective: the three tests

Throughout this framework, every recommendation is held to three tests, in the agency’s own words. They are not abstractions; they are the questions a Secretary, an audit committee and a citizen are each entitled to ask.

These three tests are the reason governance exists, and the standard against which this framework should be judged. Sections 5 to 9 build the capability to meet them; Sections 10 to 13 turn that capability into an operating rhythm, an executive agenda and day-to-day practice.