[Go-essp-tech] What is the risk that science is done using 'deprecated' data?

Fri Mar 9 00:26:50 MST 2012

Hi Gavin,

That would definitely help but I don't think it's sufficient.  How many of us would notice if a centre republished the same dataset (same dataset_id and facet metadata) with different checksums?  Estani would I expect :-) but the system itself wouldn't.

I would like to see a hash of invariants of each dataset used as identifiers.  For that we'd need to strip-out all the information from a THREDDS catalog which might legitimately change without changing the data: URL paths, service endpoints, last-modified, etc., but keeping filenames, checksums and some properties.  Canonicalise a serialisiation then generate a hash.

We'd also need to really keep track of these hashes.  We have checksums and tracking_ids right now and are under-utilising them.

Cheers,
Stephen.

On 9 Mar 2012, at 05:05, Gavin M. Bell wrote:

Hello,

If we enforced checksums to be done as a part of publication, then this would address this issue, right?

On 3/8/12 8:39 AM, stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk> wrote:

Tobias, sorry I miss-typed your name :-)
S.

On 8 Mar 2012, at 16:00, <stephen.pascoe at stfc.ac.uk><mailto:stephen.pascoe at stfc.ac.uk>
 wrote:

Hi Thomas,

As you say, it's too late to do much re-engineering of the system now -- we've attempted to put in place various identifier systems and none of them are working particularly well -- however I think there is another perspective to your proposal:

1. ESG/CMIP5 is deployed globally across multiple administrative domains and each domain has the ability to cut corners to get things done, e.g. replacing files silently without changing identifiers.

2. ESG/CMIP5 system is so complex that who'd blame a sys-admin for doing #1 to get the data to scientists when they need it.  Any system that makes it impossible, or even only difficult, to change the underlying data is going to be more complex and difficult to administer than a system that doesn't, unless that system was very rigorously designed, implemented and tested.

Because of #1 I'm convinced that a fit-for-purpose identifier system wouldn't use randomly generated UUIDs but would take the GIT approach of hashing invariants of the dataset so that any changes behind the scenes can be detected.

Because of #2 I'm convinced that now is not the time to start building more software to do this.  We have to stabilise the system and learn the lessons of CMIP5 first.

Cheers,
Stephen.

On 8 Mar 2012, at 15:32, Tobias Weigel wrote:

Jamie/All,

these are important questions I have been wondering about as well; we just had a small internal meeting yesterday with Estani and Martina, so I'll try to sum some points up here. I am not too familiar with the ESG publishing process, so I can only guess that Stephen's #1 has something to do with the bending of policies that are for pragmatic reasons not enforced in the CMIP5 process. (My intuition is that *ideally* it should be impossible to make data available without going through the whole publication process. Please correct me if I am misunderstanding this.)

Most of what I have been thinking about however concerns point #2. I'd claim that the risk here should not be underestimated; data consumers being unable to find the data they need is bad ("the advanced search issue"), but users relying on deprecated data - most likely without being aware of it - is certainly dangerous for scientific credibility.
My suggestion to address this problem is to use globally persistent identifiers (PIDs) that are automatically assigned to data objects (and metadata etc.) on ESG-publication; data should ideally not be known by its file name or system-internal ID, but via a global identifier that never changes after it has been published. Of course, this sounds like the DOIs, but these are extremely coarse grained and very static. The idea is to attach identifiers to the low-level entities and provide solutions to build up a hierarchical ID system (virtual collections) to account for the various layers used in our data. Such persistent identifiers should then be placed prominently in any user interface dealing with managed data. The important thing is: If data is updated, we don't update the data behind identifier x, but assign a new identifier y and create a typed link between these two (which may be the most challenging part) and perhaps put a small annotation on x that this data is depreca
ted. A clever user interface should then redirect a user consistently to the latest version of a dataset if a user accesses the old identifier.
This does not make it impossible to use deprecated data, but at least it raises the consumer's awareness of the issue and lowers the barrier to re-retrieve valid data.

As for the point in time; I'd be certain that it is too late now, but it is always a good idea to have plans for future improvement.. :)

Best, Tobias

Am 08.03.2012 13:06, schrieb Kettleborough, Jamie:

Thanks for the replies on this - any other replies are still very welcome.

Stephen - being selfish - we aren't too worried about 2 as its less of an issue for us (we do a daily trawl of thredds catalogues for new datasets), but I agree it is a problem more generally.  I don't have a feel for which of the problems 1-3 would minimise the risk most if you solved it.  I think making sure new data has a new version is a foundation though.

Part of me wonders though whether its already too late to really do anything with versioning in its current form.  *But* I may be overestimating the size of the problem of new datasets appearing without versions being updated.

Jamie

-----Original Message-----
From: go-essp-tech-bounces at ucar.edu<mailto:go-essp-tech-bounces at ucar.edu>
[mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Sébastien Denvil
Sent: 08 March 2012 10:41
To: go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>
Subject: Re: [Go-essp-tech] What is the risk that science is
done using 'deprecated' data?

Hi Stephen, let me add a third point:

3. Users are aware of a new versions but can't download files
so as to have a coherent set of files.

With respect to that point the p2p transition (especially the
attribut caching on the node) will be a major step forward.
GFDL just upgrad and we have an amazing success rate of 98%.

And I agree with Ashish.

Regards.
Sébastien

Le 08/03/2012 11:34, stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk> a écrit :

Hi Jamie,

I can imagine there is a risk of papers being written on

deprecated data in two scenarios:

 1. Data is being updated at datanodes without creating a

new version

 2. Users are unaware of new versions available and

therefore using

deprecated data

Are you concerned about both of these scenarios?  Your

email seems to mainly address #1.

Thanks,
Stephen.

On 8 Mar 2012, at 10:21, Kettleborough, Jamie wrote:

Hello,

Does anyone have a feel for the current level of risk that

analysists

are doing work (with the intention to publish) on data

that has been

found to be wrong by the data providers and so deprecated (in some
sense)?

My feeling is that versioning isn't working (that may be

putting it a

bit strongly.  It is too easy for data providers - in their
understandable drive to get their data out - to have

updated files on

disk without publishing a new version.   How big a deal does anyone
think this is?

If the risk that papers are being written based on

deprecated data is

sufficiently large then is there an agreed strategy for

coping with

this?  Does it have implications for the requirements of the data
publishing/delivery system?

Thanks,

Jamie
_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5

Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02

_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

--
Tobias Weigel

Department of Data Management
Deutsches Klimarechenzentrum GmbH (German Climate Computing Center)
Bundesstr. 45a
20146 Hamburg
Germany

Tel.: +49 40 460094 104
E-Mail: weigel at dkrz.de<mailto:weigel at dkrz.de>
Website: www.dkrz.de<http://www.dkrz.de/>

Managing Director: Prof. Dr. Thomas Ludwig

Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784

_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

--
Scanned by iCritical.
_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

--
Gavin M. Bell
--

 "Never mistake a clear view for a short distance."
               -Paul Saffo

-- 
Scanned by iCritical.