8Design (precision and safety)

Design is the domain that determines how an individual AI system actually behaves: what it perceives, what bounds it, what it optimises for, and how it is corrected. Where architecture builds the foundation, design shapes the system. Done well, it produces systems that are purposeful, safe and contestable; done badly, it produces automated hope, software with the surface of intelligence and none of the assurance. This section sets out a design discipline built for AI rather than borrowed from human-centred software, and connects it to the AI technical standard, Secure by Design and the contestability and privacy obligations.

8.1Designing from the machine’s perspective

Human-centred design methods, user journeys, empathy maps, process swimlanes, all assume a person is at the centre of control. An autonomous AI is not. To design a system that behaves safely under autonomy, you have to start from how the machine experiences the world, specify the system from that point of view, and only then re-introduce people, as governed participants with clear and bounded roles. The sequence matters: design first for the machine that will act, then design the controlled world it acts within, and place humans back in at the points where their judgment counts.

In practice this means specifying, for every AI component, four things in a fixed order: the signals it perceives and how they are validated; the constraints that bound it and when it must escalate; the goals it optimises and how conflicts are resolved; and the feedback that corrects it over time. An AI component whose signals, constraints, goals and feedback are not written down is not ready to be built, let alone deployed. Appendix G provides an agent specification template that captures exactly these.

8.2Agent roles: the work plane and the control plane

Complex AI behaviour is safest when it is composed from simple, single-purpose parts, so that the agency can place its controls at the right joints rather than trusting one opaque, do-everything model. The popular agentic literature offers a taxonomy for this, typically a set of nine agent archetypes, but it is precisely where that literature is weakest. The nine names overlap (it is rarely clear where perception ends and analysis begins, or how a verifier differs from a decision-maker), they mix grammatical forms, and, most importantly, they put the things that perform the work and the things that govern it into a single undifferentiated list. For a governance framework, that conflation is the whole problem. We therefore use a cleaner cut: two planes.

Work plane

1. Sensorperceives and validates inputs
2. Reasonerinterprets, plans and decides under policy
3. Actorchanges systems of record, reversibly

Perceive, reason, act, observe: the observe step feeds the next perception.

Control plane

• GateGo, Ask or Stop
• Recorderwrites the decision passport
• Overseerwhere reserved human judgment enters
• Improverupdates models and rules off the live path
• Coordinatorarbitrates between agents

The work plane has three roles, and only three, mapped to the perceive-reason-act-observe loop the engineering field has converged on. A sensor perceives and validates inputs. A reasoner interprets, plans and decides under policy. An actor effects change in systems of record, and does so reversibly. Behaviour flows around the loop, and the observe step feeds the next perception. Keeping the work plane this small is deliberate: three clearly distinct roles are easier to specify, test and assure than nine that blur into one another.

The control plane holds the governance functions, and they are controls, not peer agents. A gate is the autonomy gate of the next section: Go, Ask or Stop. A recorder writes the decision passport, the provenance that makes a decision auditable. An overseer is the human participation point, where reserved judgment enters. An improver updates models and rules, strictly off the live path, so learning never changes a decision while it is being made. And a coordinator arbitrates between agents, but only where there is genuinely more than one. These controls map directly onto the four functions of the NIST AI Risk Management Framework, govern, map, measure and manage, which is what makes them defensible to an auditor. The cardinal safety rule falls straight out of the separation: no irreversible action occurs until the work has passed the gate and been recorded, and every action an actor takes is reversible and logged.

8.3Autonomy gates: deciding when an agent may act

The single most important design control for government is the autonomy gate: the point at which the system decides whether an AI may act on its own, must ask a human first, or must stop. The control reduces to one scale and three states, Go, Ask and Stop, governed by trust and evidence, in a gate that always fails closed.

GoTrust and evidence above the threshold for this decision class. The agent acts within its granted rights, and logs.

AskInside a band where human judgment is required. A person checks and decides before the action proceeds.

StopBelow the floor, or required evidence is missing. The agent does not act; it escalates and fails safe.

1. Before activationMay this agent act, here and now?
2. In flightDo the constraints still hold?
3. After the actionDid the outcome stay within policy?

Every gate fails closed: missing evidence means Stop.

A gate sits at three points in every action: before activation (may this agent wake, here and now?), in flight (do the constraints still hold as it acts?), and after the action (did the outcome stay within policy?). Each gate yields one of three outcomes: allow, deny, or allow with obligations. The design discipline is to build the gate once per class of decision and reuse it everywhere, so that the agency’s risk appetite is enforced consistently rather than reinvented per project. In practice the gates are concrete: no AI-generated code change reaches production without passing provenance, security and human-review gates, and no automated decision affecting a person operates in Go without an explicit, assured basis.

8.4Human participation by design

Designing from the machine’s perspective does not remove people; it places them where they matter. The better frame is human-as-participant rather than human-in-the-loop, with five distinct roles, each with clear rights and accountabilities: designers who shape the system, governance stewards who tend the guardrails, operators who run it, decision-makers who exercise reserved judgment, and ethical custodians who safeguard legitimacy. The design question is not whether a human is involved, but which human, with what information, at which gate, and with what authority. Done well, this is also how an agency satisfies the contestability principle and prepares for the automated decision-making transparency obligations that commence in December 2026: the points of human judgment and the basis for decisions are designed in, not discovered later.

8.5Safe by design, secure by design

Two further design disciplines complete the picture. Secure by design, as the ACSC defines it, means security is a property of the system from the first line, not a later addition; for AI this extends to the supply chain, the pipelines and the third-party tools. Safe by design means treating five hard truths about AI as a pre-deployment checklist: that the system perceives only what data encodes, that meaning can drift, that it acts at machine speed, that boundaries must be designed rather than assumed, and that governance must be enforced at runtime. Before any AI system ships, the design team should be able to answer, for each, that it has been addressed. If any answer is no, the system is not safe to field; it is automated hope, and it should not be trusted with public outcomes.

8.6Design outputs