Written by Jill Flyn, Lead Consultant, at Info.Blueprint.
South Africa’s Draft National Artificial Intelligence Policy has transitioned from a regulatory roadmap into a sobering case study.
On 26 April 2026, Minister Solly Malatsi withdrew the draft after it was discovered that the reference list contained fictitious sources. The official explanation pointed to the most plausible culprit: AI-generated citations included without proper human verification.
The situation is notable because this was not a minor administrative memo; it was the foundation intended to guide the country’s digital future. When the very document designed to regulate AI produces hallucinations, it highlights a significant gap in institutional trust.
The problem here is not just that an AI model generated false sources.
It is well understood that large language models can produce such errors. The real failure lies in the institutional controls that were meant to catch them.
The withdrawal of this policy exposes a disconnect between high-level policy language and operational maturity. There were clear gaps in the process: basic fact-checking was bypassed, the quality assurance process failed to identify non-existent citations, and human oversight became a symbolic gesture rather than a functional reality.
This is a central governance lesson for both corporate and public sectors in South Africa. AI does not fail in a vacuum; it fails when the people, processes, and structures around it are not mature enough to manage its outputs.
In many organisations, data governance and review discipline are treated as technical background tasks. This incident proves they are actually the core of a trust architecture. A reference list is an evidence chain, and if the citations are fake, that chain is broken. Without a reliable evidence base, a policy’s authority is inevitably damaged.
This is a warning to every South African board, legal firm, and consultancy using generative AI.
The phrase ‘human in the loop’ is often used as a safety blanket, but it is frequently a myth in practice. A person can be present in a process without exercising actual judgment. If a reviewer signs off on a document without checking the sources, the loop is effectively broken.
Real oversight requires competence, time, and clear lines of responsibility. Institutions need enforceable rules for AI-assisted drafting that track when AI was used, what it produced, and who verified the final output.
The lesson is not to abandon AI policy, but to approach it with greater discipline. The irony remains that the AI policy itself required stronger AI governance.
South Africa still has the opportunity to build a credible framework, but the next version must show that these lessons have been integrated. Responsible AI cannot simply be declared; it must be evidenced in the way the work is conducted. Policy is a statement of intent, but governance is the process that ensures that intent is met. In the digital age, that process is the only reliable path to trust.
When AI policy fails its own test: A South African governance lesson
Written by Jill Flyn, Lead Consultant, at Info.Blueprint.
South Africa’s Draft National Artificial Intelligence Policy has transitioned from a regulatory roadmap into a sobering case study.
On 26 April 2026, Minister Solly Malatsi withdrew the draft after it was discovered that the reference list contained fictitious sources. The official explanation pointed to the most plausible culprit: AI-generated citations included without proper human verification.
The situation is notable because this was not a minor administrative memo; it was the foundation intended to guide the country’s digital future. When the very document designed to regulate AI produces hallucinations, it highlights a significant gap in institutional trust.
The problem here is not just that an AI model generated false sources.
It is well understood that large language models can produce such errors. The real failure lies in the institutional controls that were meant to catch them.
The withdrawal of this policy exposes a disconnect between high-level policy language and operational maturity. There were clear gaps in the process: basic fact-checking was bypassed, the quality assurance process failed to identify non-existent citations, and human oversight became a symbolic gesture rather than a functional reality.
This is a central governance lesson for both corporate and public sectors in South Africa. AI does not fail in a vacuum; it fails when the people, processes, and structures around it are not mature enough to manage its outputs.
In many organisations, data governance and review discipline are treated as technical background tasks. This incident proves they are actually the core of a trust architecture. A reference list is an evidence chain, and if the citations are fake, that chain is broken. Without a reliable evidence base, a policy’s authority is inevitably damaged.
This is a warning to every South African board, legal firm, and consultancy using generative AI.
The phrase ‘human in the loop’ is often used as a safety blanket, but it is frequently a myth in practice. A person can be present in a process without exercising actual judgment. If a reviewer signs off on a document without checking the sources, the loop is effectively broken.
Real oversight requires competence, time, and clear lines of responsibility. Institutions need enforceable rules for AI-assisted drafting that track when AI was used, what it produced, and who verified the final output.
The lesson is not to abandon AI policy, but to approach it with greater discipline. The irony remains that the AI policy itself required stronger AI governance.
South Africa still has the opportunity to build a credible framework, but the next version must show that these lessons have been integrated. Responsible AI cannot simply be declared; it must be evidenced in the way the work is conducted. Policy is a statement of intent, but governance is the process that ensures that intent is met. In the digital age, that process is the only reliable path to trust.
Categories