HIC: Dataset silo layer

[jamoore]

So far we have focused on the project silo model. The data for which was manually anonymised, extracted and transferred from the mssql servers on the NHS network at HIC to the HIC/OMERO server on the UNI network.

It is going to be useful for other stories that we develop the dataset silo model as a parent layer of the project silo within this pilot.

The dataset silo is a complete anonymised mirror of the nhs datasets held on the mssql servers at HIC. The project silos are then prepared on a project-by-project basis from the dataset silo all within the HIC/OMERO architecture. This will include anonymisation, data cleaning and modelling steps.

Josh, has developed a diagram (​pdf) outlining the proposed structure and the relationship with existing tickets.

Each of the clinical data files we've provided from the GoDARTS project represents a dataset. For instance the separate files for SMR, RX (prescribing), BIOCHEM, etc are all separate datasets. I think the only way we can work this in terms of the pilot project governance is to use the GoDARTS data as the real data and create a mass of fake data based on the schemas. These would then form the dataset silo, from which we can rebuild the project silo and test that users only get to work with the project data they are supposed to.

There are various reasons why this is useful - simplify/automate data flows, governance and audit, end user aggregate queries (e.g. study estimates). But of particular interest is the governance and the following 2 use cases:

1. The audit trail should allow dataset inspection e.g. which project have used smr... In theory a data owner (the custodians, patients or caldicotts) may want to see how safe our model is or who is using 'their' data.

2. The embedding of risk prediction models, privacy impact assessments and disclosure controls. These are mechanisms that are being discussed in the SHIP blueprint as ways to facilitate risk based & proportionate data governance.

The following points are from the Jan2011 draft of the blueprint, once the final version is available I'll update this story - although this may need pulling out into a separate story.

• Assessing privacy risks is an integral component of a data controller’s responsibilities and should form a central part of their privacy policy. This process should include the identification of confidentiality, security and privacy risks of any data handling including linkages, storage and access considerations. The Information Commissioner's Office have developed a privacy impact assessment handbook ​http://bit.ly/A2cga containing guidance for carrying out risk assessments.

• Appropriate disclosure control should be applied to all outputs; this should be carried out under the authority and oversight of the designated privacy officer. The Information Services Divison (ISD) of NHS Scotland have developed various documents on data protection and confidentiality ​http://bit.ly/nPOMZY and of particular importance is their protocol for disclosure control.

Implementation

A dataset silo in OMERO should be exportable to a re-anonymized project silo. If the work for #4652 (server-side API), then this work can be implemented as sub-directories in the silo fs repository. Along with the links between input tables and output tables, metadata about any operations (see #6321) that were performed on the data should also be recorded.

[adjudson]

This is related (possible duplicate of) user story #6330.

[jmoore]

Merging description from #6330

[jmoore]

Moving to "Unscheduled" as 4.4.0 release approaches.

[jamoore]

Closing all specific HIC tasks.