Nightmare on LLM Street: Preventing Poor Data from Haunting AI
Wed 7 May 2025
As AI tools sweep across industries, the public sector faces unique challenges in ensuring these technologies are deployed responsibly and effectively. In this feature, Matt Flenley, Head of Strategy at Datactics, explores how the UK’s government agencies can transform citizen services with AI—provided they first address legacy systems, consent management, and the data foundations on which these innovations rely. With billions in investment and growing public scrutiny, Flenley argues that success hinges not just on innovation, but on trust, standardisation, and ethical stewardship of data.
– – – – – –
As the pace of technological advancement accelerates, the need for robust management practices has never been more critical. With significant investments being made in AI, public sector leaders must ensure their digital transformation efforts are underpinned by reliable and standardised practices.
The UK Government’s substantial £800 million ($1 billion) budget in October aimed at advancing digital and AI reflects the sector’s interest in leveraging AI and automation to improve services. This momentum continues into 2025, with the Labour government emphasising technological progress, including a ‘single unique identifier’ to support children and families.
To drive public sector efficiency, new AI tools like ‘Connect’ and ‘Scout’ are being introduced by the Government to streamline processes and improve service delivery, ultimately contributing to the broader goal of transforming public services through technology.
However, challenges like outdated systems, disconnected data and a lack of standardisation threaten to disrupt AI initiatives before they deliver results.
The Perils of Acting Without Planning
The rapid rise of generative AI tools, like ChatGPT, has heightened public expectations about what AI can achieve. However, this enthusiasm often leads to an overestimation of AI’s capabilities and a rush to implement solutions without adequate planning.
A recent example is LinkedIn’s decision to include user data in its AI models without seeking explicit consent, a move that drew backlash and required intervention from the UK’s Information Commissioner’s Office (ICO). While LinkedIn’s existing consent frameworks enabled a swift course correction, the public sector, often plagued by fragmented data systems, may not have the same agility.
To avoid similar problems, public sector organisations must adopt a reality-based approach. Many government systems lack integrated data consent frameworks and, in a sector already constrained by tight budgets, care must be taken to build AI systems that inspire trust and deliver accurate results rather than eroding public confidence through errors or misuse of data.
Streamlining Data and Consent Management
A unified consent management framework is critical for trustworthy AI. This framework must resemble GDPR mandates, ensuring consistent and transparent citizen data handling.
Integrating consent management with an interoperable single-citizen view can provide a solid foundation for AI development. This approach enables efficient consent management while ensuring responsible data use and aligns consent with demographic information to help model creators identify and address dataset biases, ensuring models are fair and effective.
The COVID-era exam grades fiasco highlights the consequences of poor data governance. Flawed algorithms and unstandardised data usage can affect real lives and erode public trust. Solid, transparent foundations are essential for public sector AI initiatives.
Overcoming Legacy Challenges
Legacy systems often store personal information in different formats, which can complicate data standardisation. Traditional solutions range from direct matching algorithms to labour-intensive manual reviews, a scenario no organisation wants to ensure.
Modern approaches offer the solution to this. Using data management technologies, advanced algorithms and machine learning to reconcile data across legacy systems can streamline processes and enable reliable data use for AI models.
Effective data matching also facilitates analysing opt-ins and opt-outs with demographic data. This helps identify biases and assess whether AI models are representative and safe for deployment.
Addressing these issues proactively enables equitable and effective AI systems and builds public trust in AI.
Public sector organisations must prioritise transparency to comply with regulations and encourage individual participation in data-sharing initiatives.
When individuals understand how their data is used and see the evidence of responsible governance, they are more likely to support AI-driven projects. This trust is vital for data-driven transformations to demonstrate ethical data use to reduce fears and build confidence in the technology.
Embracing AI’s Responsibility
Despite challenges, AI holds immense potential to improve public services. NHS England’s AI tool, for example, is used for detecting heart disease by analysing images 30 times faster than human clinicians, which enhances diagnostic efficiency. Similarly, the Mid and South Essex NHS Foundation’s predictive analytics model reduced missed appointments by identifying the patients at risk and implementing personalised reminders.
These examples illustrate AI’s transformative power when applied responsibly, and by prioritising data practices, public sector organisations can unlock similar benefits while maintaining public trust.
The public sector’s pursuit of AI excellence need not be daunting. By aligning citizen data with clear consent frameworks and leveraging intelligent data-matching technologies, government departments can build reliable, trustworthy AI systems. These efforts will prevent pitfalls and pave the way for innovative solutions that improve individuals’ lives.
Public sector leaders can inspire confidence in AI’s potential by prioritising transparency, standardisation, and ethical data use. With these measures in place, the future of AI in the public sector can be both promising and secure.
