AI Capability Maturity Assessment · Part II. The eight domains

Domain 6 · Practice layerDesign and engineering practice

18 minute read · Version 1.0.0 · five criteria

In this chapter

Covers how AI systems are designed, tested and released: specification before build, deliberate autonomy and human oversight, evaluation and simulation, assurance of AI-generated code and content, and secure and safe design.

What this domain covers
The domain at each level
Risks when it is weak
The five criteria
6.1 Specification before build
6.2 Autonomy and human oversight design
6.3 Testing, evaluation and simulation
6.4 AI-generated code and content assurance
6.5 Secure and safe by design
In Governing Intelligence

What this domain covers

Most AI failures are design failures: a system given more autonomy than its controls can hold, a reviewer with no time or information to make a real decision, a model update accepted without testing, a fallback nobody designed. Good engineering practice makes AI bounded, testable and reversible, which is what allows autonomy to be extended safely.

Initiatives for this domain are run through the Engineering practice playbook.

The domain at each level

A summary of what the domain looks like at each level. Each criterion below has its own descriptors and pass tests: score the criteria, not this summary. The domain score is calculated from them (see domain aggregation).

Level	What it looks like
1Initial	AI is built or configured from demos and prompts with no specification. Autonomy is whatever the tool does by default. Testing stops at a demonstration. AI-generated code and content get no extra review, and safety is considered after problems appear.
2Repeatable	Some requirements exist but describe features, not behaviour. A person is expected to check outputs, without criteria or records. Builders write a few tests and run them manually. Developers are told to review AI output, but nothing enforces it.
3Defined	Every in-scope component has a signed-off specification including constraints, failure modes and fallback. Autonomy and approval rules are documented per system and approvals are recorded. Evaluation suites with agreed pass criteria cover typical, edge and adversarial cases, with shadow runs for higher tiers. Merge gates and scanning apply to AI-generated code, and secure-and-safe design reviews are signed off before release.
4Managed	Specifications are versioned and traced to tests. The platform enforces autonomy settings and binds approvals to what was reviewed, and approval and override rates are monitored. Evaluations run automatically on every change and block failing releases. Provenance of AI-assisted work is recorded and its quality measured.
5Optimising	Autonomy is earned and withdrawn on measured performance within limits the system cannot exceed. Specifications are machine-readable and enforced at runtime. Evaluation suites grow from production incidents, and safe defaults are inherited from shared patterns.

Risks when it is weak

Systems with more autonomy than their controls can hold
Rubber-stamp approvals that provide no real oversight
Silent regressions when models, prompts or data change
Vulnerable or unlicensed AI-generated code in production
No fallback when the AI is unsure or unavailable

The five criteria

No.	Criterion	Importance	Scope
6.1	Specification before build	Enabling	Local practice
6.2	Autonomy and human oversight design	Core	Local practice
6.3	Testing, evaluation and simulation	Core	Local practice
6.4	AI-generated code and content assurance	Enabling	Local practice
6.5	Secure and safe by design	Enabling	Local practice

Criterion 6.1

Specification before build

Each AI component is specified before it is built or bought: purpose, inputs and signals, constraints, decision rights, success measures, failure modes and fallback.

Why it matters

A system with no written specification cannot be tested, governed or held to account, because nobody agreed what it should and should not do. Specifying behaviour, not just features, is what makes autonomy boundable.

The five levels

Level	What it looks like	To score this level
1Initial	AI is built or configured from a prompt or a demonstration. There is no written specification.	The starting point: no test.
2Repeatable	Requirements exist for some projects but describe features rather than behaviour: no constraints, failure modes or fallbacks.	Written requirements exist for at least one AI component.
3Defined	A standard specification template (purpose, scope, inputs, constraints, autonomy and decision rights, success and failure measures, fallback, human roles) is completed for every in-scope component and signed off at the design gate.	The template, and signed-off specifications for every in-scope component designed in the last 12 months.
4Managed	Specifications are versioned with the system. Test cases and monitoring thresholds are derived from them, and any change in behaviour requires a specification change.	Specification versions linked to system releases, and traceability from specification items to test cases.
5Optimising	Specifications are machine-readable and enforced at runtime (policies, permissions and tests generated from them), and specification patterns are reused across the portfolio.	Runtime policies or permissions generated from a machine-readable specification.

Evidence to request

Documents

Specification template
Completed specifications

Records of it operating

Design gate sign-offs
Specification version history
Traceability from specification to tests

In systems

Machine-readable specifications or agent definitions (optional)

In the evidence request: ER-6-01.

Interview probes

Engineering or architecture leadShow me the specification for this agent. What is it not allowed to do?
Use-case owner or business leadWhat did you agree the system should do when it is unsure or cannot complete a task?
Risk, compliance and internal auditHow do you check that the built system matches its specification?

Red flags

Specifications written after the build
No failure modes or fallback described
A vendor brochure used as the specification

Standards crosswalk

ISO/IEC 42001: Annex A.6
EU AI Act: Article 11
NIST AI RMF: MAP 1MAP 3

Government and organisation editions

Australian Government edition

The AI technical standard sets design-stage practices, and its agentic AI addendum (June 2026) adds bounded autonomy and human override for higher-autonomy systems. Governing Intelligence Appendix G provides an agent specification template.

References

Technical standard for government’s use of artificial intelligence (AI technical standard), with the Agentic AI addendumDesign statements; agentic AI addendum

Organisation edition

ISO/IEC 42001 requires AI system requirements and design to be documented. Providers of high-risk systems in the EU must prepare technical documentation before placing them on the market.

References

ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management systemAnnex A.6
Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1MAP function
Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI)Article 11, providers of high-risk systems

In Governing Intelligence

In the AI governance assessment: Every AI component has a written specification: signals, constraints, goals and feedback.

Initiatives that lift it

I-601 Adopt an AI specification template and design gate (level 1 to 3)
I-602 Version specifications and trace them to tests (level 3 to 4)

Rate 6.1 in the assessment Back to top

Criterion 6.2

Autonomy and human oversight design

ImportanceCore ScopeLocal practice Responsible AIDevelopment and operation AI Ethics PrinciplesHuman-centred values; Accountability; Contestability

For each class of decision, a system’s autonomy (act, ask a person first, or stop) is set deliberately against risk, and the people who take part have defined roles, the information they need and real authority to change the outcome.

Why it matters

Human oversight that consists of a person clicking ‘approve’ in four seconds is not oversight. Autonomy must be granted deliberately, enforced by the platform, and matched by reviewers who have the time, information and authority to say no.

The five levels

Level	What it looks like	To score this level
1Initial	Autonomy is whatever the tool does by default, and human oversight means someone may look at the outputs.	The starting point: no test.
2Repeatable	Human review is expected informally (‘a person checks it’), but reviewers lack criteria, time or information, and reviews are not recorded.	A documented expectation that a person reviews outputs for at least one system.
3Defined	Each in-scope system documents its autonomy level and which actions are allowed, need approval or are prohibited. Human participation points name the role, the information shown and the authority held, and approvals are recorded.	An autonomy and approval matrix for each in-scope system, and approval records for actions that need approval.
4Managed	Autonomy settings are enforced by the platform rather than by convention. Approvals are bound to exactly what was reviewed, approval and override rates are monitored to detect rubber-stamping or over-reliance, and changes to autonomy pass a gate.	Platform configuration that enforces the matrix, a report of approval and override rates, and a gate record for an autonomy change.
5Optimising	Autonomy is earned on evidence: settings are raised or lowered on measured performance and override data, within policy limits the system cannot exceed, and every change is reversible.	An autonomy change justified by measured performance data, made within documented policy limits.

Evidence to request

Documents

Autonomy and approval matrix per system
Human participation design (roles, information, authority)

Records of it operating

Approval records
Approval and override rate reports
Autonomy change gate records

In systems

Platform configuration enforcing allowed, ask-first and prohibited actions

In the evidence request: ER-6-02.

Interview probes

Use-case owner or business leadWhich actions can this system take without asking anyone?
Frontline staffWhen you approve an AI action, what information do you see, and how long do you usually spend?
Engineering or architecture leadWhere is the approval requirement enforced, and could an agent bypass it?

Staff survey items: S14 I sometimes rely on AI outputs without checking them because there is not enough time.; S18 When I am asked to approve or review an AI output or action, I have enough information and time to make a real decision.

Red flags

Approval rates near 100 percent with a few seconds per approval
Autonomy raised without a recorded decision
Prohibited actions listed only in a document, not blocked in the system

Standards crosswalk

ISO/IEC 42001: Annex A.9
EU AI Act: Article 14Article 26
NIST AI RMF: MAP 3MANAGE 2

Government and organisation editions

Australian Government edition

The agentic AI addendum to the AI technical standard addresses bounded autonomy, runtime control and human override. The ISM requires risky actions to be flagged for human approval (June 2026), amended in September 2026 to require approval before sensitive or high-impact actions. For decisions about people, the Automated Decision Making Better Practice Guide and the Privacy Act’s automated-decision transparency obligations apply.

References

Technical standard for government’s use of artificial intelligence (AI technical standard), with the Agentic AI addendumAgentic AI addendum
Information Security ManualHuman approval before sensitive or high-impact agent actions
Careful adoption of agentic AI services
Automated Decision-Making Better Practice Guide
Privacy and Other Legislation Amendment Act 2024

Organisation edition

High-risk systems in the EU must be designed for effective human oversight, and their deployers must assign oversight to competent people with authority. The Guidance for AI Adoption makes maintaining human control one of its six practices.

References

Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI)Articles 14 and 26, high-risk systems
Guidance for AI AdoptionMaintain human control
Voluntary AI Safety StandardGuardrail 5
ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management systemAnnex A.9
Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1MAP and MANAGE functions

In Governing Intelligence

In the AI governance assessment: Autonomy gates are defined for each class of decision, and they fail closed. Human participation points are designed in, with their authority stated.

Initiatives that lift it

I-605 Set autonomy and approval rules for each system (level 1 to 3)
I-606 Enforce autonomy in the platform and watch approval rates (level 3 to 4)

Rate 6.2 in the assessment Back to top

Criterion 6.3

Testing, evaluation and simulation

ImportanceCore ScopeLocal practice Responsible AIDevelopment and operation AI Ethics PrinciplesReliability and safety; Fairness

AI systems are evaluated before release and after every change against realistic, edge and adversarial cases with defined pass criteria, and higher-tier systems run in shadow or simulation before they are allowed to act.

Why it matters

AI behaviour changes when models, prompts, tools or data change, often silently. Without repeatable evaluation, a provider’s model update can degrade a system that worked last week, and nobody notices until the outputs cause harm.

The five levels

Level	What it looks like	To score this level
1Initial	Testing stops at a demonstration that the system works on a few hand-picked examples, and nothing is repeated after changes.	The starting point: no test.
2Repeatable	Some test cases exist, written by the builder, run manually and not repeated after changes. There is no adversarial testing.	A set of test cases exists for at least one AI system.
3Defined	Each in-scope system has an evaluation suite with pass criteria agreed by the owner, covering typical, edge and adversarial cases (and fairness where people are affected). Results are recorded at the testing gate, and higher-tier systems run in shadow before they act.	An evaluation suite with owner-agreed pass criteria for each in-scope system, recorded results at the testing gate, and a shadow report for each high-tier system.
4Managed	Evaluations run automatically on every model, prompt, tool or data change and block release on failure. Shadow comparisons measure agreement with current practice, and results are reviewed by someone independent of the builder.	Pipeline evidence that evaluations run on change and block failing releases, shadow agreement measures, and an independent review record.
5Optimising	Evaluation suites grow from production incidents and overrides, simulation tests multi-agent and system-level behaviour, and evaluation methods are benchmarked externally.	Evaluation cases added from production incidents, and a system-level simulation report.

Evidence to request

Documents

Evaluation suites and pass criteria
Shadow and simulation method

Records of it operating

Evaluation results at gates
Shadow comparison reports
Independent review records

In systems

Pipeline configuration that runs evaluations and blocks failing releases

In the evidence request: ER-6-03.

Interview probes

Engineering or architecture leadWhen the model provider last updated the model, what tests ran before you accepted it?
Use-case owner or business leadWhat pass rate did you agree for this system, and who agreed it?
Risk, compliance and internal auditWho reviews evaluation results other than the team that built the system?

Red flags

Provider model updates accepted without any testing
No adversarial or edge cases in the test set
The builder sets and marks their own tests

Standards crosswalk

ISO/IEC 42001: Annex A.6
EU AI Act: Article 9Article 15
NIST AI RMF: MEASURE 1MEASURE 2

Government and organisation editions

Australian Government edition

The AI technical standard includes testing and evaluation practices across Discover and Operate. Governing Intelligence makes a passed simulation (gate G2) and a shadow period (gate G4) preconditions for higher-risk use cases.

References

Organisation edition

The NIST AI RMF MEASURE function covers evaluation of trustworthiness characteristics, and the Generative AI Profile adds evaluation of generative risks such as confabulation. Providers of high-risk systems in the EU must test them against defined metrics before placing them on the market.

References

Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1MEASURE function
Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1
ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management systemAnnex A.6
Guidance for AI AdoptionTest and monitor
Voluntary AI Safety StandardGuardrail 4
Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI)Articles 9 and 15, high-risk systems

In Governing Intelligence

10.3Rehearsing before reality: simulation

Initiatives that lift it

I-603 Build evaluation suites with agreed pass criteria (level 1 to 3)
I-604 Run evaluations on every change and review them independently (level 3 to 4)

Rate 6.3 in the assessment Back to top

Criterion 6.4

AI-generated code and content assurance

ImportanceEnabling ScopeLocal practice Responsible AIDevelopment and operation AI Ethics PrinciplesReliability and safety; Transparency and explainability

Code, configuration and content produced with AI assistance pass the same or stronger review as human work, with provenance recorded and dedicated gates for security and licensing.

Why it matters

AI assistants write code and content faster than people can review it. Without gates, vulnerable code, licensing exposure and unchecked claims reach production and the public at a pace no manual process will catch.

The five levels

Level	What it looks like	To score this level
1Initial	AI-generated code and content are used with no distinction from other work and no added review.	The starting point: no test.
2Repeatable	Developers and authors are told to review AI output, but there is no gate, no record of which work was AI-assisted, and reviews are inconsistent.	Guidance tells developers or authors to review AI output.
3Defined	Policy defines review requirements for AI-generated code and content. Merge gates require human review plus security scanning and licence checks, and material AI-generated content for external audiences is reviewed and labelled where required.	The policy, merge gate configuration requiring review, and security scanning and licence check evidence on repositories with AI-assisted work.
4Managed	Provenance of AI-assisted changes is recorded automatically. Defect and vulnerability rates in AI-assisted work are measured against a baseline, and review effort is targeted where defects concentrate.	Automatically recorded provenance of AI-assisted changes, and a defect or vulnerability comparison against baseline.
5Optimising	Coding agents operate under scoped permissions with mandatory checks, and measured quality decides how much autonomy they receive.	Scoped permission configuration for coding agents, and an autonomy decision based on measured quality.

Evidence to request

Documents

Policy for AI-generated code and content
Content labelling rules

Records of it operating

Review records
Defect and vulnerability metrics for AI-assisted work

In systems

Branch protection and merge gate configuration
Security scanning and licence check results
Provenance metadata on commits or content

In the evidence request: ER-6-04.

Interview probes

Engineering or architecture leadHow do you know which changes in production were written with AI assistance?
CISO or security leadWhat scanning runs on AI-generated code before it merges?
Legal, privacy and records leadHow do you check licensing and copyright exposure in AI-generated content?

Red flags

Coding agents with production credentials
No security scanning on repositories where AI writes code
AI-generated content published without review

Standards crosswalk

ISO/IEC 42001: Annex A.6
EU AI Act: Article 50
NIST AI RMF: MEASURE 2

Government and organisation editions

Australian Government edition

Governing Intelligence Appendix E provides an AI-generated code assurance checklist. The ISM’s secure software development controls apply to AI-assisted development as to any other.

References

Information Security ManualSoftware development controls
Technical standard for government’s use of artificial intelligence (AI technical standard), with the Agentic AI addendum

Organisation edition

The NIST Generative AI Profile identifies information integrity and intellectual property risks in generated content. Organisations operating in the EU must label certain AI-generated or manipulated content.

References

Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1Information integrity; intellectual property
ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management systemAnnex A.6
Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI)Article 50 transparency for generated content

In Governing Intelligence

Appendix EAI-generated code assurance checklist

Initiatives that lift it

I-607 Put review and scanning gates on AI-generated code and content (level 1 to 3)
I-608 Record provenance of AI-assisted work and measure its quality (level 3 to 4)

Rate 6.4 in the assessment Back to top

Criterion 6.5

Secure and safe by design

ImportanceEnabling ScopeLocal practice Responsible AIDevelopment and operation AI Ethics PrinciplesReliability and safety; Human, societal and environmental wellbeing

Every in-scope AI system receives a recorded secure-by-design and safe-by-design review before release, covering misuse, failure behaviour, fallbacks and effects on people, signed off by accountable roles.

Why it matters

Safety is cheapest in design. A system that fails closed, falls back to a manual process and was examined for misuse before release avoids the incidents that a later fix cannot undo.

The five levels

Level	What it looks like	To score this level
1Initial	Security and safety are considered, if at all, after the system is built and only when a problem appears in use.	The starting point: no test.
2Repeatable	Security reviews happen for major releases using general checklists. Safety (harm to people, misuse, failure behaviour) is not reviewed.	A general security review record exists for at least one AI release.
3Defined	An AI-specific secure-and-safe design checklist (misuse cases, fail-closed behaviour, fallback to manual, data minimisation, harm analysis) is completed before release, with sign-off by security and the owner recorded.	The AI-specific checklist and signed-off reviews for every in-scope release in the last 12 months.
4Managed	Design reviews draw on threat models, previous incidents and evaluation results. Issues found are tracked to closure before release, and findings are analysed across systems.	Review issues tracked to closure before release, and a cross-system analysis of findings.
5Optimising	Safe defaults are built into shared platforms and patterns so new systems inherit them, design reviews concentrate on novel risks, and patterns are shared beyond the organisation.	A shared pattern or platform default that encodes a safety control, adopted by several systems.

Evidence to request

Documents

Secure-and-safe design checklist for AI

Records of it operating

Completed reviews and sign-offs
Design issue tracker
Cross-system findings analysis

In systems

Platform defaults that encode safety controls (optional)

In the evidence request: ER-6-05.

Interview probes

Engineering or architecture leadWhat does this system do when it fails or is unsure, and how was that decided?
CISO or security leadShow me the last design review you signed off for an AI system.
Use-case owner or business leadWhat misuse of this system did the design consider?

Red flags

Systems fail open when a dependency is unavailable
No fallback to a manual process
Sign-off by the builder only

Standards crosswalk

ISO/IEC 42001: Annex A.6
EU AI Act: Article 15
NIST AI RMF: MAP 5MANAGE 2

Government and organisation editions

Australian Government edition

ASD’s secure-by-design guidance and its AI deployment guidance set the security baseline. Governing Intelligence Section 8.5 sets out the safe-by-design checks for government AI.

References

Organisation edition

ISO/IEC 42001 life cycle controls and the NIST AI RMF’s MAP and MANAGE functions cover anticipating and treating misuse and failure. High-risk systems in the EU must achieve appropriate accuracy, robustness and cybersecurity.

References

ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management systemAnnex A.6
Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1MAP and MANAGE functions
Deploying AI systems securely
Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI)Article 15, high-risk systems

In Governing Intelligence

8.5Safe by design, secure by design

In the AI governance assessment: Secure-by-design and safe-by-design sign-off is recorded before deployment.

Initiatives that lift it

I-609 Adopt a secure-and-safe design review for AI (level 1 to 3)
I-610 Feed incidents and threat models into design reviews (level 3 to 4)

Rate 6.5 in the assessment Back to top

In Governing Intelligence

The framework chapters that describe the practice this domain measures, and the governance assessment questions that test it at a higher level.

In the AI governance assessment: Every AI component has a written specification: signals, constraints, goals and feedback. Autonomy gates are defined for each class of decision, and they fail closed. Human participation points are designed in, with their authority stated. Secure-by-design and safe-by-design sign-off is recorded before deployment.

This assessment is not legal advice and is not a certification. Obligations and dates were checked against official sources on 15 October 2026; confirm them against the source instruments, which can change.