UK government drowning in legacy documents it dare not erase
Tue 7 Feb 2017
A new report outlines the extent to which the British government’s civil service is crippled by a document mountain it can neither legally erase nor keep.
The British government and civil service, a report reveals, is overwhelmed by unstructured legacy data – such as documents, emails and directives – which it cannot destroy for legislative and security reasons, but which it neither can usefully access any longer. It’s a problem said to be costing half a billion pounds a year.
The Cabinet Office Digital Records and Information Management Team, working in collaboration with The National Archives and Government Digital Service, have outlined the extent of the problem in an ironically lengthy PDF document released recently.
The problem emerged from government’s uncertain response to the advent of the digital age, and from an apparently endless series of ‘new broom’ initiatives that attempted to stem the tide of emails and documentation, only to be abandoned for the next as the ePaper mountain continued to pile up.
‘It is clear,’ the report admits, ‘that government faces significant risks from departments’ accumulated legacy digital information and is unable to get any benefit from it.’
Can’t get the staff
The report has no one single solution to offer, and notes the need to ensure that documents containing confidential citizen data are either purposely thrown away or purposely retained, and to what extent recourse to Big Data document mapping technologies limits the possibilities for automation.
“Human beings cannot be removed from the process: they need to operate software, set search parameters and (in the vast majority of cases) manually review potentially sensitive documents. The technical limitations of the tools are significant. An inability to identify sensitivity from context means that analytics and eDiscovery software do not currently offer a complete solution to Inquiry search requests or sensitivity review, though they can support the process.”
But the conundrum, as the paper explains, is that the civil service has lost a fifth of its staff since 2010.
Because so much of the administrative and regulatory paper trail has been likewise lost to legacy formats and abandoned document management schemes over the last 17 years, the report notes that there is a risk that the high turnover of staff and short collective memory of the civil service will lead to the invention of coping initiatives which were already tried (and which failed) long ago. The report states: ‘Estimates suggest that wasted effort re-creating old work might cost government nearly £500m per year.’
‘The way government currently does information management was designed for the paper era so it is not surprising that specific elements do not sit well in the digital world.’
Though the report is relatively optimistic about Big Data approaches to the document avalanche, it observes that departments are restrained by their current IT facilities, ‘especially when IT is outsourced’ and adds that certain departments have secondary security concerns which makes an ad hoc approach impossible.
This ‘semi-lost’ information is spread among an ungovernable diversity of locations, and within often-defunct, outmoded or otherwise inaccessible formats.
‘Emails pose a particular challenge to departments that is not easily solved by any single tool. For example, emails might be held in formats that are difficult to read (e.g. PST files) or sit within active email accounts (e.g. of long-serving civil servants).’
The requirements of The Public Records Act (PRA) of 1958 means that pruning the blizzard is fraught with risk, yet keeping all of it, even without doing any further work on the problem, threatens to break numerous other regulations concerning the limits on the legal retention of data. Further constrictions are laid down by the Data Protection Act 1998 (DPA), the Freedom of Information Act 2000 (FOI Act) and the Re-use of Public Sector Information Regulations of 2015, all of which constitute a labyrinthine regulatory framework which is expecting order and accountability in document trails.
Damned if you do…
So the civil service neither has the money to throw the necessary manpower at the problem, nor possesses an adequately mature and context-sensitive AI solution which could be applied across the board – the more so, in the latter case, because security requirements vary so much from department to department, and because the departments are traditionally circumspect about cooperating among themselves:
‘The ability of civil servants to find previous work also depends on their department’s attitude to sharing work across their platform. Some departments have, for information security or other cultural reasons, an ingrained reticence to share material except in very limited circumstances. This is an obvious barrier to that information being found and used in other policy work.’
The report presents a raft of possible approaches to the issue, including an office ‘ranking scheme’, where workers could see which among their colleagues are being praised for good information management. Another approach is encouraging workers to set up online profiles so that they can be found in the years after they may have moved on from the department or the service, in the form of a ‘people finder’.
Here we go again
In a section headed ‘Removing the impulse to re-invent the wheel’, the report returns to the problem of repeating old mistakes, suggesting that ‘A long-term goal might therefore be to create the expectation that civil servants will consider previous literature and the expertise of their peers within and outside government before developing new policy ideas.’
However, since there is no universally usable matrix or referable archive (which is what the rest of the report indicates), this approach seems problematic.
The report concludes with the notion that Knowledge & Information Management (KID) operatives in the government could become a professional intra-departmental service across branches. However, the report concedes that such a generic approach risks ‘the potential loss of bespoke approaches (e.g. to sensitivity review) for each department and unclear lines of accountability.’
