Government Data Warehouse
Submitted by: Mark Mayes
Describe your idea. How does it work and who does it help?
Unlike many of the excellent suggestions on this web site, which are aimed at people viewing or interacting with the government data through a web site or desktop, my idea is based on making the storage, retrieval and display of that data far easier and accessible for other programmers.
Currently, there are numerous, disparate datasets, which all have their own programmable interfaces (API's). This means that each time a programmer needs to access data from a different dataset, they need to program a new way of retrieving that data. This is akin to re-inventing the wheel over and over again.
My idea is to create a data warehouse, which is effectively a dynamic (i.e. updated on a regular basis) store of data, which consists of the data from ALL available datasets.
In addition to simply storing this data in a uniform fashion, the real advantage of this approach is that a common interface can be developed for this data, which allows permitted programmers to access ALL of this data in a common way.
Paying for programmers is an expensive business and developing systems to hook into disparate government datasets would be a VERY expensive business. This data warehouse would save a massive amount of time and money for the government when developing systems, which access their data.
What information or services do you need?
As many datasets as possible!
Posted by John Maslen on 22 July 2008 at 06:43 PM
Better still, don't set up another centralised data warehouse - mandate that all Government Depts disseminating 'indicator style data' (ie. basically univariate statistics with a time and space dimension) should establish data harvesting and discovery services (using standard web service protocols) that conform to the new CLG LGDx XML schema.
Mandating is probably highly unlikely so the next best thing would be 'incentives' eg. to ensure no new project gets funded unless it conforms to this standard. But then this needs someone to police this - which has been the problem in the past with the 'policing' unit being under-resourced to do the job.
Not sure what the best solution is but it needs to be federated not centralised as that's the only way it will work successfully across government.