<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffcc">
Stephen, <br>
<br>
I wholly agree with you. Let's get as close to the data as
possible. The hash not just is a unique value / identifier but is
also has a deep semantic meaning directly relating to the data. The
solution here is what Stephen (and I) have been kicking around. I
suggest a clean envelope type of structure capturing this notion of
mutable and immutable data. It is a simpler solution and no service
is required to generate it or verify it... you get all of those
things for free, especially in the context of ESGF where these
values are a part of the index itself. I am with Stephen on this
Occam's razor cuts the best. <br>
<br>
On 3/9/12 1:55 AM, <a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a> wrote:
<blockquote
cite="mid:D0962E9A-7074-4108-8007-FE439E0D6783@stfc.ac.uk"
type="cite">
<pre wrap="">Hi Tobias,
I'm not familiar with EPIC. Does anyone have a reference? I assume it's based on registration. In my view hashing provides a more powerful mechanism for global identifiers if you can create a canonical representation of the entity. Then it's impossible for the entity to change without the identifier changing instead of relying on some trusted authority to enforce the correspondence.
Last time I talked to DOI people it still wasn't completely sorted out whether a dataset can change without changing the DOI. Sure it shouldn't but there were examples of landing pages evolving over time. I can foresee a similar problem with any registration system.
Cheers,
Stephen.
On 9 Mar 2012, at 09:11, Tobias Weigel wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I am prety much thinking about the EPIC infrastructure, and as long as that is not fully ready yet, at least the basic Handle System. ExArch is a very good candidate to explore some ideas and limitations, as well as the German c3grid project (trace provenance information across workflows).
On 09.03.2012 10:08:04, Bryan Lawrence wrote:
</pre>
<blockquote type="cite">
<pre wrap="">there are othre setting up digital identificaiton services, we should not. e.g. EPIC ... could we use them?
Cheers
Bryan
</pre>
<blockquote type="cite">
<pre wrap="">Martin
This problem space seems suitable for EXARCH. I.E. setting up a digital identication service. This service would be very long term infrastructure and thus would need to scale to several billions of identifiers plus associated metadata references.
Mark
On 8 Mar 2012, at 17:18,<a class="moz-txt-link-rfc2396E" href="mailto:martin.juckes@stfc.ac.uk"><martin.juckes@stfc.ac.uk></a> <a class="moz-txt-link-rfc2396E" href="mailto:martin.juckes@stfc.ac.uk"><martin.juckes@stfc.ac.uk></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I agree, particularly on the last point.
There are a lot of things which could be improved. From a software developers point of view, getting the data providers and data users to agree a set of requirements before starting development would be a good idea -- but we obviously missed the chance to do that, if it ever existed, by several years,
Cheers,
Martin
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">-----Original Message-----
From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-freetext" href="mailto:go-essp-tech">mailto:go-essp-tech</a>-
<a class="moz-txt-link-abbreviated" href="mailto:bounces@ucar.edu">bounces@ucar.edu</a>] On Behalf Of <a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>
Sent: 08 March 2012 16:01
To: <a class="moz-txt-link-abbreviated" href="mailto:weigel@dkrz.de">weigel@dkrz.de</a>
Cc: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>
Subject: Re: [Go-essp-tech] What is the risk that science is done
using 'deprecated' data?
Hi Thomas,
As you say, it's too late to do much re-engineering of the system now
-- we've attempted to put in place various identifier systems and none
of them are working particularly well -- however I think there is
another perspective to your proposal:
1. ESG/CMIP5 is deployed globally across multiple administrative
domains and each domain has the ability to cut corners to get things
done, e.g. replacing files silently without changing identifiers.
2. ESG/CMIP5 system is so complex that who'd blame a sys-admin for
doing #1 to get the data to scientists when they need it. Any system
that makes it impossible, or even only difficult, to change the
underlying data is going to be more complex and difficult to
administer than a system that doesn't, unless that system was very
rigorously designed, implemented and tested.
Because of #1 I'm convinced that a fit-for-purpose identifier system
wouldn't use randomly generated UUIDs but would take the GIT approach
of hashing invariants of the dataset so that any changes behind the
scenes can be detected.
Because of #2 I'm convinced that now is not the time to start building
more software to do this. We have to stabilise the system and learn
the lessons of CMIP5 first.
Cheers,
Stephen.
On 8 Mar 2012, at 15:32, Tobias Weigel wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Jamie/All,
these are important questions I have been wondering about as well;
</pre>
</blockquote>
<pre wrap="">we just had a small internal meeting yesterday with Estani and
Martina, so I'll try to sum some points up here. I am not too familiar
with the ESG publishing process, so I can only guess that Stephen's #1
has something to do with the bending of policies that are for
pragmatic reasons not enforced in the CMIP5 process. (My intuition is
that *ideally* it should be impossible to make data available without
going through the whole publication process. Please correct me if I am
misunderstanding this.)
</pre>
<blockquote type="cite">
<pre wrap="">Most of what I have been thinking about however concerns point #2.
</pre>
</blockquote>
<pre wrap="">I'd claim that the risk here should not be underestimated; data
consumers being unable to find the data they need is bad ("the
advanced search issue"), but users relying on deprecated data - most
likely without being aware of it - is certainly dangerous for
scientific credibility.
</pre>
<blockquote type="cite">
<pre wrap="">My suggestion to address this problem is to use globally persistent
</pre>
</blockquote>
<pre wrap="">identifiers (PIDs) that are automatically assigned to data objects
(and metadata etc.) on ESG-publication; data should ideally not be
known by its file name or system-internal ID, but via a global
identifier that never changes after it has been published. Of course,
this sounds like the DOIs, but these are extremely coarse grained and
very static. The idea is to attach identifiers to the low-level
entities and provide solutions to build up a hierarchical ID system
(virtual collections) to account for the various layers used in our
data. Such persistent identifiers should then be placed prominently in
any user interface dealing with managed data. The important thing is:
If data is updated, we don't update the data behind identifier x, but
assign a new identifier y and create a typed link between these two
(which may be the most challenging part) and perhaps put a small
annotation on x that this data is deprecated. A clever user interface
should then redirect a user consistently to the latest version of a
dataset if a user accesses the old identifier.
</pre>
<blockquote type="cite">
<pre wrap="">This does not make it impossible to use deprecated data, but at
</pre>
</blockquote>
<pre wrap="">least it raises the consumer's awareness of the issue and lowers the
barrier to re-retrieve valid data.
</pre>
<blockquote type="cite">
<pre wrap="">As for the point in time; I'd be certain that it is too late now,
</pre>
</blockquote>
<pre wrap="">but it is always a good idea to have plans for future improvement.. :)
</pre>
<blockquote type="cite">
<pre wrap="">Best, Tobias
Am 08.03.2012 13:06, schrieb Kettleborough, Jamie:
</pre>
<blockquote type="cite">
<pre wrap="">Thanks for the replies on this - any other replies are still very
</pre>
</blockquote>
</blockquote>
<pre wrap="">welcome.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">Stephen - being selfish - we aren't too worried about 2 as its less
</pre>
</blockquote>
</blockquote>
<pre wrap="">of an issue for us (we do a daily trawl of thredds catalogues for new
datasets), but I agree it is a problem more generally. I don't have a
feel for which of the problems 1-3 would minimise the risk most if you
solved it. I think making sure new data has a new version is a
foundation though.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">Part of me wonders though whether its already too late to really do
</pre>
</blockquote>
</blockquote>
<pre wrap="">anything with versioning in its current form. *But* I may be
overestimating the size of the problem of new datasets appearing
without versions being updated.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">Jamie
</pre>
<blockquote type="cite">
<pre wrap="">-----Original Message-----
From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a>
[<a class="moz-txt-link-freetext" href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Sébastien
</pre>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">Denvil
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">Sent: 08 March 2012 10:41
To: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>
Subject: Re: [Go-essp-tech] What is the risk that science is
done using 'deprecated' data?
Hi Stephen, let me add a third point:
3. Users are aware of a new versions but can't download files
so as to have a coherent set of files.
With respect to that point the p2p transition (especially the
attribut caching on the node) will be a major step forward.
GFDL just upgrad and we have an amazing success rate of 98%.
And I agree with Ashish.
Regards.
Sébastien
Le 08/03/2012 11:34, <a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a> a écrit :
</pre>
<blockquote type="cite">
<pre wrap="">Hi Jamie,
I can imagine there is a risk of papers being written on
</pre>
</blockquote>
<pre wrap="">deprecated data in two scenarios:
</pre>
<blockquote type="cite">
<pre wrap=""> 1. Data is being updated at datanodes without creating a
</pre>
</blockquote>
<pre wrap="">new version
</pre>
<blockquote type="cite">
<pre wrap=""> 2. Users are unaware of new versions available and
</pre>
</blockquote>
<pre wrap="">therefore using
</pre>
<blockquote type="cite">
<pre wrap="">deprecated data
Are you concerned about both of these scenarios? Your
</pre>
</blockquote>
<pre wrap="">email seems to mainly address #1.
</pre>
<blockquote type="cite">
<pre wrap="">Thanks,
Stephen.
On 8 Mar 2012, at 10:21, Kettleborough, Jamie wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hello,
Does anyone have a feel for the current level of risk that
</pre>
</blockquote>
</blockquote>
<pre wrap="">analysists
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">are doing work (with the intention to publish) on data
</pre>
</blockquote>
</blockquote>
<pre wrap="">that has been
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">found to be wrong by the data providers and so deprecated (in
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">some
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">sense)?
My feeling is that versioning isn't working (that may be
</pre>
</blockquote>
</blockquote>
<pre wrap="">putting it a
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">bit strongly. It is too easy for data providers - in their
understandable drive to get their data out - to have
</pre>
</blockquote>
</blockquote>
<pre wrap="">updated files on
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">disk without publishing a new version. How big a deal does
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">anyone
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">think this is?
If the risk that papers are being written based on
</pre>
</blockquote>
</blockquote>
<pre wrap="">deprecated data is
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">sufficiently large then is there an agreed strategy for
</pre>
</blockquote>
</blockquote>
<pre wrap="">coping with
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">this? Does it have implications for the requirements of the
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">data
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">publishing/delivery system?
Thanks,
Jamie
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
</blockquote>
<pre wrap="">--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
</pre>
</blockquote>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<pre wrap="">
--
Tobias Weigel
Department of Data Management
Deutsches Klimarechenzentrum GmbH (German Climate Computing Center)
Bundesstr. 45a
20146 Hamburg
Germany
Tel.: +49 40 460094 104
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:weigel@dkrz.de">weigel@dkrz.de</a>
Website: <a class="moz-txt-link-abbreviated" href="http://www.dkrz.de">www.dkrz.de</a>
Managing Director: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<pre wrap="">--
Scanned by iCritical.
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">---------------------------------------------------
Mark Morgan
Software Architect / Engineer
Institut Pierre Simon Laplace (IPSL),
Université Pierre Marie Curie,
4 Place Jussieu,
Tour 45-55, Salle #207,
Paris 75005
France.
Tel : +33 (0) 1 44 27 49 10
Email: <a class="moz-txt-link-abbreviated" href="mailto:momipsl@ipsl.jussieu.fr">momipsl@ipsl.jussieu.fr</a>
---------------------------------------------------
</pre>
</blockquote>
<pre wrap="">--
Bryan Lawrence
University of Reading: Professor of Weather and Climate Computing.
National Centre for Atmospheric Science: Director of Models and Data.
STFC: Director of the Centre for Environmental Data Archival.
Ph: +44 118 3786507 or 1235 445012; Web:home.badc.rl.ac.uk/lawrence
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<pre wrap="">
--
Tobias Weigel
Department of Data Management
Deutsches Klimarechenzentrum GmbH (German Climate Computing Center)
Bundesstr. 45a
20146 Hamburg
Germany
Tel.: +49 40 460094 104
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:weigel@dkrz.de">weigel@dkrz.de</a>
Website: <a class="moz-txt-link-abbreviated" href="http://www.dkrz.de">www.dkrz.de</a>
Managing Director: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<pre wrap="">
--
Scanned by iCritical.
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Gavin M. Bell
--
"Never mistake a clear view for a short distance."
-Paul Saffo
</pre>
</body>
</html>