AI Capability Maturity Assessment · Part II. The eight domains
Domain 4 · Practice layerData, knowledge and semantics
In this chapter
Covers the data and knowledge AI depends on: quality, shared meaning, classification and handling, records of what AI did and why, and retention of AI content as records.
What this domain covers
Covers the data and knowledge AI depends on: quality, shared meaning, classification and handling, records of what AI did and why, and retention of AI content as records.
AI amplifies the data it is given. Poor or unrepresentative data produces confident, wrong outputs at scale; uncurated knowledge sources produce answers based on superseded policy; and without decision records an organisation cannot explain, defend or contest what its AI did. Classification rules decide whether sensitive information reaches services that should never see it.
Initiatives for this domain are run through the Data and knowledge playbook.
The domain at each level
A summary of what the domain looks like at each level. Each criterion below has its own descriptors and pass tests: score the criteria, not this summary. The domain score is calculated from them (see domain aggregation).
| Level | What it looks like |
|---|---|
| 1Initial | The quality of data feeding AI is unknown. There are no shared definitions, no rules on what information may enter AI tools, no record of what AI did, and AI content is not treated as a record. |
| 2Repeatable | Quality and definitions are managed in some domains but not linked to AI use. Policy forbids some information in AI tools but nothing enforces it. Logs exist but are technical and short-lived. Retention depends on each tool’s defaults. |
| 3Defined | Priority data domains feeding AI have owners, quality thresholds and a measured baseline. Knowledge sources are curated with owners. Each AI tool has an approved sensitivity ceiling. A decision-record standard is applied to in-scope uses, and AI content is mapped to retention rules. |
| 4Managed | Quality is monitored for production AI with defined responses to breaches. Ceilings are enforced technically. Decision records are tamper-evident, complete and used to answer reviews and audits. Legal holds and disposal apply to AI content and have been tested. |
| 5Optimising | A shared semantic layer is used across systems and agents. Data issues found by AI monitoring are fixed at source. Decision records drive explanations at the point of decision and are shared across the supply chain where needed. |
Risks when it is weak
- Confident, wrong outputs from poor or unrepresentative data
- Sensitive or personal information exposed through AI tools
- Decisions that cannot be reconstructed for a review, complaint, audit or court
- Records obligations breached because AI content was deleted or never captured
- Assistants answering from superseded policies or conflicting definitions
The five criteria
| No. | Criterion | Importance | Scope |
|---|---|---|---|
| 4.1 | Data quality for AI | Enabling | Local practice |
| 4.2 | Shared vocabulary and curated knowledge | Enabling | Local practice |
| 4.3 | Classification and handling | Foundation | Enterprise arrangement, applied locally |
| 4.4 | Provenance and decision records | Core | Local practice |
| 4.5 | Records and retention of AI content | Enabling | Enterprise arrangement, applied locally |
Criterion 4.1
Data quality for AI
The quality of data feeding AI (completeness, accuracy, timeliness and representativeness) is measured for priority domains, with named owners and thresholds that decide whether the data is fit for AI use.
Why it matters
AI amplifies the data it is given. Unmeasured data quality produces confident, wrong outputs at scale, and unrepresentative data produces unfair outcomes that nobody notices until people are harmed.
The five levels
| Level | What it looks like | To score this level |
|---|---|---|
| 1Initial | The quality of data feeding AI is unknown. AI systems are connected to whatever data is available. | The starting point: no test. |
| 2Repeatable | Quality issues are known anecdotally and fixed reactively. Some domains have quality reports, but they are not tied to AI use. | A data quality report exists for at least one domain that AI uses. |
| 3Defined | Priority data domains feeding AI have named owners, defined quality dimensions and thresholds, and a measured baseline. Use-case assessments check data fitness before build. | Quality scorecards with a baseline and thresholds for each priority domain, named owners, and a data fitness check in at least three use-case assessments. |
| 4Managed | Quality is monitored continuously for data feeding production AI, with alerts when thresholds are breached and a defined response (degrade, pause or human review). Representativeness is tested for uses that affect people. | Monitoring alert configuration for production AI data, at least one alert-response record, and a representativeness test for a use that affects people. |
| 5Optimising | Quality improvement is prioritised by AI value and risk. Data issues found by AI monitoring go back to source-system owners and are fixed at source, with the effect measured. | A source-system fix triggered by AI monitoring, with the measured effect. |
Evidence to request
Documents
- Quality dimensions and thresholds per priority domain
- Data owner assignments
Records of it operating
- Data quality scorecards and baselines
- Fitness checks in assessments
- Alert-response records
- Representativeness test results
In systems
- Data quality monitoring and alert configuration
In the evidence request: ER-4-01.
Interview probes
- Chief Data Officer or data ownerFor the data feeding your highest-value AI use, what is its measured quality and who owns it?
- Engineering or architecture leadWhat happens to the AI system when input data quality drops below the threshold?
- Use-case owner or business leadHow did you confirm that the data was fit for this purpose?
Staff survey items: S20 The information our AI tools draw on is current and correct.
Red flags
- Nobody owns the data feeding AI
- Representativeness never tested for decisions about people
- ‘The model will clean it up’
Standards crosswalk
- ISO/IEC 42001
- Annex A.7
- EU AI Act
- Article 10
- NIST AI RMF
- MAP 2MEASURE 2
Government and organisation editions
Australian Government edition
Data governance is one of the five cornerstones of the National framework for the assurance of AI in government. ASD’s AI Data Security guidance covers protecting the data used to train and operate AI.
References
Organisation edition
ISO/IEC 42001 sets controls for data used in AI systems, including quality and provenance. Providers of high-risk systems in the EU must meet data governance requirements for training, validation and testing data.
References
- ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management systemAnnex A.7
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1MAP and MEASURE functions
- Voluntary AI Safety StandardGuardrail 3
- AI data security
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI)Article 10, providers of high-risk systems
In Governing Intelligence
In the AI governance assessment: An ontology and data-quality baseline exists for our priority domains.
Initiatives that lift it
Criterion 4.2
Shared vocabulary and curated knowledge
Priority domains have an agreed vocabulary (glossary, taxonomy or ontology), and the knowledge sources AI systems draw on are curated, owned and kept current, so that key terms mean the same thing to every system and person.
Why it matters
An assistant pointed at every document the organisation holds will answer from superseded policies and conflicting definitions with equal confidence. Shared meaning and curated sources are what make answers correct and explainable.
The five levels
| Level | What it looks like | To score this level |
|---|---|---|
| 1Initial | There are no shared definitions. AI systems are pointed at document stores containing conflicting and outdated content. | The starting point: no test. |
| 2Repeatable | Glossaries exist in some areas but are not maintained or used by AI systems. Knowledge sources for assistants are uncurated (whole sites or shared drives). | A glossary or curated content set exists for at least one domain. |
| 3Defined | Priority domains have a maintained glossary with owners. Knowledge sources used by AI are curated, with owners, currency rules and removal of superseded content, and AI specifications reference the glossary. | A glossary with owners for each priority domain, and an inventory of AI knowledge sources showing owner and currency rule for each. |
| 4Managed | Retrieval sources have measured currency and coverage. Answer quality is evaluated against the curated sources, and conflicts between sources are detected and resolved through a defined process. | Answer-quality evaluation results against curated sources, and records of source conflicts resolved. |
| 5Optimising | A shared semantic layer (ontology or knowledge graph) is used across systems and agents. Changes to definitions propagate through versioned contracts, and meaning is captured in decision records. | A versioned ontology or semantic layer consumed by more than one AI system. |
Evidence to request
Documents
- Glossary, taxonomy or ontology for priority domains
- Content curation and currency rules
Records of it operating
- Inventory of AI knowledge sources with owners
- Answer-quality evaluation results
- Source conflict log
In systems
- Retrieval configuration showing which sources each assistant or agent uses
In the evidence request: ER-4-02.
Interview probes
- Chief Data Officer or data ownerWhich definitions do your AI systems depend on, and who owns them?
- Engineering or architecture leadWhat content does the assistant retrieve from, and how is outdated content removed?
- Frontline staffHas an AI tool given you an answer based on an old policy or document?
Staff survey items: S20 The information our AI tools draw on is current and correct.
Red flags
- Assistants indexed on everything with no owner for the content
- The same term defined differently in two systems
- Superseded policies still retrievable by AI
Standards crosswalk
- ISO/IEC 42001
- Annex A.7
- EU AI Act
- No direct mapping
- NIST AI RMF
- MAP 2
Government and organisation editions
Australian Government edition
The AI technical standard and the assurance framework both depend on knowing what data and knowledge an AI system draws on. Curated sources also reduce the risk of advice to ministers or the public resting on superseded policy.
References
Organisation edition
ISO/IEC 42001 controls for data in AI systems cover the data’s provenance and suitability, which includes the knowledge sources assistants retrieve from.
References
In Governing Intelligence
In the AI governance assessment: An ontology and data-quality baseline exists for our priority domains.
Initiatives that lift it
Criterion 4.3
Classification and handling
Information classification and handling rules apply to AI data flows: each AI tool and platform has an approved ceiling for information sensitivity, and the ceiling is enforced technically as well as by policy.
Why it matters
Most AI data leaks are not attacks: they are staff entering information into a tool that should never see it. A known ceiling per tool, enforced by technology, is what prevents sensitive and personal information reaching services that have not been assessed for it.
The five levels
| Level | What it looks like | To score this level |
|---|---|---|
| 1Initial | There are no rules about what information may be entered into AI tools. Staff decide for themselves. | The starting point: no test. |
| 2Repeatable | Policy states what must not be entered (for example personal or sensitive information), but tools have no assessed ceiling and nothing technical prevents breaches. | A written rule about what information must not be entered into AI tools. |
| 3Defined | Each approved AI tool has a documented ceiling (sensitivity or classification level, and whether personal information is permitted) based on assessment. Ceilings are published to staff, and data flows for in-scope use cases are mapped. | An approved tool list stating the ceiling for each tool, and data flow maps for in-scope use cases. |
| 4Managed | Ceilings are enforced technically (sensitivity labels, data loss prevention, masking of identifiers, routing), breaches are detected and reported, and exceptions need approval. | Technical enforcement configuration (labels, data loss prevention or masking) and a detection report from the last quarter. |
| 5Optimising | Classification-aware routing selects the model or service by the sensitivity of each request automatically, and enforcement effectiveness is tested and reported. | Routing configuration that selects services by sensitivity, and an effectiveness test report. |
Evidence to request
Documents
- Approved tool list with ceilings
- Handling rules for AI
- Data flow maps for in-scope use cases
Records of it operating
- Breach detection reports
- Exception approvals
In systems
- Sensitivity label, data loss prevention and masking configuration
- Routing rules by sensitivity
In the evidence request: ER-4-03.
Interview probes
- CISO or security leadHow would you know if someone pasted sensitive information into an AI tool?
- Frontline staffWhat information are you allowed to put into the AI tools you use?
- Chief Data Officer or data ownerWhich data flows into AI services leave your environment, and where are they processed?
Staff survey items: S09 I know what information I must not enter into AI tools.
Red flags
- Staff do not know the ceiling of the tools they use
- Personal information entered into tools not assessed for it
- Reliance on training alone, with no technical control
Standards crosswalk
- ISO/IEC 42001
- Annex A.7
- EU AI Act
- No direct mapping
- NIST AI RMF
- MAP 4MANAGE 2
Government and organisation editions
Australian Government edition
PSPF classification and handling rules apply to AI systems and their data. The ISM and IRAP assessment establish what each service may handle, and the Hosting Certification Framework constrains where data and AI are hosted. The PSPF direction on DeepSeek products (February 2025) shows that specific AI products can be prohibited outright.
References
Organisation edition
The Australian Privacy Principles govern use, disclosure, cross-border disclosure and security of personal information entered into AI tools. The OAIC advises against entering personal information, especially sensitive information, into publicly available AI tools.
References
In Governing Intelligence
Initiatives that lift it
Criterion 4.4
Provenance and decision records
AI-assisted decisions and actions carry a record of what happened: the person or matter concerned, the policy or rule applied, the model and version, the sources used, confidence where relevant, the human approvals, and a reference to the rationale, retained so decisions can be explained and contested.
Why it matters
A decision that cannot be reconstructed cannot be explained to the person affected, defended to a regulator or corrected when wrong. Provenance is what turns AI from an opaque assistant into something an organisation can answer for.
The five levels
| Level | What it looks like | To score this level |
|---|---|---|
| 1Initial | There is no record of what an AI system did or why beyond what users remember or happen to keep. | The starting point: no test. |
| 2Repeatable | Application logs exist but are technical, incomplete or short-lived, and they do not link outputs to model versions, sources or human approvals. | Logs for at least one AI system can be produced. |
| 3Defined | A decision-record standard defines minimum fields for in-scope uses (subject, policy, model and version, sources, confidence, human approvals, rationale reference). Records are produced for every in-scope decision and retained. | A decision-record standard with minimum fields, and a sample of five records from each in-scope system containing those fields. |
| 4Managed | Records are tamper-evident, queryable and used: reviews, complaints and audits are answered from them, and completeness is monitored with gaps alerted. | A tamper-evidence mechanism, a complaint, review or audit answered from records, and completeness monitoring. |
| 5Optimising | Decision records drive explanations at the point of decision and feed monitoring, learning and contestability at scale. Record formats are shared with partners across the supply chain where needed. | Explanations generated from records at the point of decision, and a shared record format used with a partner. |
Evidence to request
Documents
- Decision-record standard (minimum fields)
- Retention rule for decision records
Records of it operating
- Sample decision records per in-scope system
- Complaint or audit responses drawn from records
- Completeness reports
In systems
- Logging and evidence configuration
- Tamper-evidence mechanism (for example hash chaining or write-once storage)
In the evidence request: ER-4-04.
Interview probes
- Legal, privacy and records leadPick a decision from last month. Can we reconstruct which model, which version and which sources were involved?
- Engineering or architecture leadHow are decision records protected from alteration?
- Risk, compliance and internal auditHow complete are decision records, and how do you know?
Red flags
- Logs rotate after 30 days
- The model version is not recorded
- Human approvals are not linked to the action approved
Standards crosswalk
- ISO/IEC 42001
- Annex A.6
- EU AI Act
- Article 12Article 26
- NIST AI RMF
- GOVERN 1MEASURE 2
Government and organisation editions
Australian Government edition
Under the Archives Act, the records an AI system creates, including its decisions and their basis, are Commonwealth records. The Automated Decision Making Better Practice Guide expects reasons to be available for automated decisions.
References
Organisation edition
High-risk AI systems in the EU must support automatic logging, and their deployers must keep the logs. ISO/IEC 42001 sets life cycle controls that include recording events. The Voluntary AI Safety Standard asks organisations to keep records that let third parties assess compliance.
References
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI)Articles 12 and 26, high-risk systems
- ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management systemAnnex A.6
- Voluntary AI Safety StandardGuardrail 9
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1MEASURE function
In Governing Intelligence
In the AI governance assessment: Decisions in scope carry the minimum decision-passport fields.
Initiatives that lift it
Criterion 4.5
Records and retention of AI content
AI inputs, outputs and decision records that are business records are captured, classified against retention rules and disposed of lawfully, and legal holds can be applied to AI content.
Why it matters
AI content is business content. If chat histories, generated drafts and agent action logs sit in tools with vendor-default retention, records are destroyed that should be kept and kept that should be destroyed, and discovery or freedom-of-information requests cannot be answered.
The five levels
| Level | What it looks like | To score this level |
|---|---|---|
| 1Initial | AI outputs are not considered records. Chat histories and generated content sit in tools with vendor-default retention, or none. | The starting point: no test. |
| 2Repeatable | The records policy mentions AI in general terms, but AI content types are not mapped to retention classes, and retention depends on each tool’s settings. | The records policy refers to AI content. |
| 3Defined | AI content types (business prompts and outputs, decision records, agent action logs) are mapped to retention classes. Tool retention settings are configured to match, and staff guidance says when AI-assisted work must be saved as a record. | A mapping of AI content types to retention classes, retention configuration evidence for at least two AI tools, and staff guidance on saving AI-assisted work. |
| 4Managed | Retention and disposal are applied and evidenced in the systems where AI content lives. Legal holds can be applied to AI content and have been tested, and a discovery or freedom-of-information request has been answered including AI content. | A legal hold test on AI content, disposal evidence, and a discovery or freedom-of-information response that included AI content. |
| 5Optimising | Records capture happens automatically at creation with metadata, disposal runs on schedule with an audit trail, and retention rules are reviewed as new AI content types appear. | Automatic capture with metadata, and a scheduled disposal run with its audit trail. |
Evidence to request
Documents
- Records policy
- Mapping of AI content types to retention classes
- Staff guidance on AI-assisted records
Records of it operating
- Legal hold test
- Disposal records
- Discovery or freedom-of-information response including AI content
In systems
- Retention configuration in each AI tool
In the evidence request: ER-4-05.
Interview probes
- Legal, privacy and records leadWhere are chat histories with AI tools kept, for how long, and who decided?
- CIO or CTOCould you place a legal hold on AI content today?
- Frontline staffWhen do you save an AI-assisted draft as a record?
Staff survey items: S10 I know when work I have done with AI needs to be saved as a record.
Red flags
- Vendor-default retention on every AI tool
- Users can delete business records held in AI tools
- No way to place a legal hold on AI content
Standards crosswalk
- ISO/IEC 42001
- Clause 7.5
- EU AI Act
- Article 12Article 26
- NIST AI RMF
- No direct mapping
Government and organisation editions
Australian Government edition
AI outputs and decisions are Commonwealth records under the Archives Act and must be kept and disposed of under records authorities issued by the National Archives of Australia. The National Archives’ position statement on records created using AI technologies covers automated decision-making records, including inputs, outputs, logs and design and testing records. AI content may also be requested under the Freedom of Information Act.
References
Organisation edition
APP 11 requires personal information to be destroyed or de-identified when no longer needed, which applies to AI chat histories and logs. Deployers of high-risk systems in the EU must keep the logs the system generates for at least six months unless other law says otherwise.
References
- Privacy Act 1988 and the Australian Privacy PrinciplesAPP 11
- ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management systemClause 7.5 documented information
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI)Article 26, deployers of high-risk systems
In Governing Intelligence
Initiatives that lift it
In Governing Intelligence
The framework chapters that describe the practice this domain measures, and the governance assessment questions that test it at a higher level.
- 7.2The AI capability stack: a foundation for intelligence
- 7.3The decision passport: a backbone for trust
- 3.7The security, privacy and records spine
In the AI governance assessment: An ontology and data-quality baseline exists for our priority domains.
Decisions in scope carry the minimum decision-passport fields.
This assessment is not legal advice and is not a certification. Obligations and dates were checked against official sources on 15 October 2026; confirm them against the source instruments, which can change.