An open, digital Iraq inquiry

Lord Hutton's 2003/4 inquiry in the death of David Kelly was a model of open-ness and transparency, unusual in Britain at that time. Freedom of Information is still in its early days in Britain and daily transcripts live video, documents made avialble online were a novelty in 2003. The 2004 Butler Review into the intelligence failures around Iraqi WMD was a secretive affair, in part because it dealt with intelligence. Sir John Chilcot, head of the new Iraq inquiry served on the Butler review.

The Iraq inquiry has already got into a dispute over secrecy and open-ness. It seems to his credit that Sir John is pro-openness where possible. How could his new Iraq inquiry use modern online methods to better Lord Hutton and become a model for digital transparency?

COI and the Government's Director of Digital Engagement are already no doubt advising the new Inquiry Secretary, Margaret Aldred on an approach. HMSO who handled the MPs expenses publication and redaction will also be in the loop I guess. But I thought it would be interesting to try a Blackhall approach and bring many external minds to bear. Blackhall is a metonym for a Whitehall that embraces modern collaborative working tools and greater transparency see this post, slides and a paper on Blackhall.

This post is inspired by Sir Tim Berners Lee's recent note on publishing government data and my late night experience of marking up my MPs expenses on the Guardian's superb crowdsourced expenses website.

Some of the following might sound a bit 'techy' or complicated to the uninitiated. It is just good practice and low cost to implement. In many cases small up front investments bring large downstream benefits and cost savings.

Documents

The Ministry of Defence and Foreign Office are good at paper trails in making decisions. There will no doubt be very large quantities of documents for the inquiry. The Scott inquiry into arms to Iraq received 130,000 documents. There will, quite rightly, be some redaction of names and posts to protect people. Redaction is usually done manually with a marker pen and the document rescanned and pdf'd or in som cases TIFFed. So you can't tell by looking at the electronic file what is in it without reading it. A search engine can't penetrate the innards of the scanned pdf. The same applies to any document that has been scanned to a pdf. For people going through huge numbers of pdfs this electronic opacity slows things down enormously and adds to expense, cost and frustration for all concerned.

A tiny investment as the documents are sent by departments to the inquiry would work wonders. It is possible to put electronic labels or 'tags' on the documents so that computers and search engines can tell what is in them without having a human to read. This information about what is in a document is known as 'metadata'. Books have metadata - the index, ISBN number, title etc that make them easier to use - electronic documents have something similar that you can't see but the computer can when say you are searching for it.

Metadata, like making dictionaries is dull work. But the inquiry can insist on the format of information sent to it electronically. If it were to ask for basic metadata in each document it would greatly speed their prioritisation and later re examination. What metadata should be in each file?

• Date of origination
• for a communication, name from
• name to
• title of document as written
• place of origination
• keywords that describe content, including names of units, people mentioned etc
• if map or image description of what it is representation of
• any unique reference number in the original document
• reference number for department sending to inquiry
• inquiry reference number

In the Hutton inquiry the secretariat used the time honoured civil service technique of giving each pdf document a file name composed of the inquiry's reference number. This method is primitve and only helps the inquiry, not others.

Metadata tagging by the originator transfers cost and man hours from the inquiry team and the general public back onto the Departmental originators. Which is only right. It is an investment in transparency, democracy and openness.

The inquiry's log of documents and reference numbers should be live online

Website principles

The inquiry will have a website. What are some basic tenets it could follow to be fully open and electronically transparent?

The inquiry should procure the servcies of a competent technical advisor with proven track record in reusing information online or publishing it for reuse (not me - i am not technical enough). And perhaps assemble an informal bunch of online publishing advisors.

The inquiry should assume that interesting things will be done with the information they publish off their website by independents. And design the site accordingly following the principles of the Power of Information Taskforce report. See for instance They Work for You or the Guardian crowd sourcing of MPs expenses.

All documents, maps, images, transcripts, audio and video should be on the inquiry website as soon as they are received or created. This would be different from the Hutton inquiry where only documents used by Counsel were published. There would be some exemptions for legal correspondence.

Documents should be put online with a unique static URI/URL (or the site should in general follow RESTful principles). This means people can easily link to and reuse information.

Documents should record how many times that have been viewed and in particular be marked as not viewed if no one has looked at them.

The site should be archived to the highest standards by National Archives so that it does not disappear and its collection is held together in perpetuity. The Hutton inquiry site is archived in full.

It should not be acceptable for departments to publish evidence documents on their websites alone where there is a greater risk of links disappearing, especially with an election due in the near future which always causes disruption to departmental sites.

Google site search should run on the site (pay Google $2000 to remove adverts) not a bespoke search engine nor google enterprise box.

Documents being discussed by the inquiry on a particular day in open session should be brought to the front of the website and associated with the witnesses that day.

The site should either be designed for low bandwidth connections or have a 'low graphics' option so that it can easily be seen by people in Iraq who may have slow internet connections or older computers and browsers.

The inquiry should consider funding an Iraqi internet service provider to cache or mirror all its material locally in Iraq.

Some sort of simple API should be provided to let third parties reuse the information on the site

All material published should be free of copyright or under a permissive creative commons licence.

Video/Audio

The video stream from the inquiry should be declared to be free of copyright and for anyone to reuse. The video stream should be stored online for people to search through. Each change of speaker should be tagged in the video stream.

The time clock on the video stream should be synchronised carefully with the time stamp in the written record so the two can be matched up. (Hansard time stamps for instance are out of synch with the parliamentary video causing problems for reusers).

Audio should be treated in the same way as video. This is especially for countries such as Iraq where the bandwidth may not be available to view video easily.

Conclusion

This post has probably only scratched the surface and i may well have got some things wrong. Let me know in the comments how you could improve this.

Comments are moderated to be on topic, polite, free of conspiracy theories, sane, in english and playing the ball, not the man etc.

Declaration - I used to work for the UK government, but not on Iraq issues. This post is in an entirely personal capacity and does not represent the view of my former employers. My career details can be found on Linked In. I am currently on sabbatical running a hyperlocal startup Talk About Local.