<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div id="IDstID">Oh, and I am also talking about CMIP6+ here - no
use in targeting CMIP5 except for hypothetical 'what-ifs' on
lessens to learn.<br>
<br>
On 09.03.2012 10:53:51, Tobias Weigel wrote:</div>
<blockquote cite="mid:4F59D29D.2030706@dkrz.de" type="cite">On
09.03.2012 10:31:28, Estanislao Gonzalez wrote:
<br>
<blockquote type="cite">Making a hash that uniquely identifies all
the information, like the one
<br>
you've proposed Stephen, is certainly appealing. Though we will
have a
<br>
lot of hashes, most of them pointing to the same data from the
user
<br>
perspective. For instance, the user download a variable and from
that
<br>
point onwards, other variables got add and changed, catalogs get
<br>
republished unintentionally or moved to other machines, errors
at all
<br>
stages get corrected, new access to the data get inserted to the
<br>
catalogs, etc (They do happen, I've done them all). I'd expect
about 20
<br>
different hashes from it, none of them would be interesting to
the user.
<br>
IMHO we need to find proper versioning units; the publication
unit
<br>
(realm dataset) as we now use might not be the best option.
<br>
AFAICT everything moves in variable units (atomic datasets). My
<br>
publication tasks are very different, but it never matches the
<br>
publication unit we use.... hardly ever.
<br>
</blockquote>
<br>
From the user's perspective, meaningful IDs (like the ones
currently visible in the gateway, "cmip5.output1.MPI-M...") are
preferable to hashes. However, from what you are writing here I'd
think that such IDs can only be applied to very high-level
entities and are useless for the actual data management. This
could be addressed through collections/aggregations perhaps. In
general, I'd be comfortable with hashes as Stephen originally
proposed. I've never seen anyone complaining about git in this
respect.
<br>
<br>
<blockquote type="cite">For example this is how publication looks
from my perspective:
<br>
Normally I get information about a complete ensemble that was
created
<br>
anew. No information on what was changed or not. Just the data
(and the
<br>
computed checksums). I have to find out which datasets are there
and how
<br>
they relate to the ones I've already published (i.e. I have to
<br>
distinguish between new, changed and deleted).
<br>
Then the other common tasks are, e.g., when a variable was
wrongly
<br>
computed. So I get something like, "umo,vmo from 1pctCO2 are
wrong and
<br>
will be recalculated". This requires me to extract the variables
from
<br>
the datasets (finding out which), publish a new version without
them and
<br>
when they are corrected, generated yet another new version to
include
<br>
the corrected variables.
<br>
This generates 2 Versions without any meaning for those users
interested
<br>
in other variables.
<br>
</blockquote>
<br>
So, to just make a quick shot in terms of PIDs/EPIC:
<br>
You'd rather assign identifiers to all the low-level entities and
build up a hierarchy through aggregations. If the variables are
corrected, you'd publish a new identifier for an extended version
of the old collection (some cloning involved here, but still only
on the identifier side), and where possible still reference the
old variables as their data has not changed. If consequently done,
this decouples the identification/publication of identifiers issue
from the data layer, and that's one of the strong advantages I can
see in a global PID infrastructure.
<br>
<br>
Best, Tobias
<br>
<br>
<blockquote type="cite">All the data I get is not related at all
with the "realm" datasets (or
<br>
publication units). And this makes the data management more
difficult.
<br>
<br>
I just think we might want to review what we need and
distinguish the
<br>
bes units from three different perspectives: the producer
(cmor), the
<br>
data manager (esg) and the user.
<br>
Once we know that for sure (and I doubt it will be the same unit
for
<br>
all), then we can think about unique ids and a hashing
procedure, which
<br>
I strongly support.
<br>
<br>
My 2c,
<br>
Estani
<br>
<br>
Am 09.03.2012 08:26, schrieb <a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk:">stephen.pascoe@stfc.ac.uk:</a>
<br>
<blockquote type="cite">Hi Gavin,
<br>
<br>
That would definitely help but I don't think it's sufficient.
How many of us would notice if a centre republished the same
dataset (same dataset_id and facet metadata) with different
checksums? Estani would I expect :-) but the system itself
wouldn't.
<br>
<br>
I would like to see a hash of invariants of each dataset used
as identifiers. For that we'd need to strip-out all the
information from a THREDDS catalog which might legitimately
change without changing the data: URL paths, service
endpoints, last-modified, etc., but keeping filenames,
checksums and some properties. Canonicalise a serialisiation
then generate a hash.
<br>
<br>
We'd also need to really keep track of these hashes. We have
checksums and tracking_ids right now and are under-utilising
them.
<br>
<br>
Cheers,
<br>
Stephen.
<br>
<br>
On 9 Mar 2012, at 05:05, Gavin M. Bell wrote:
<br>
<br>
Hello,
<br>
<br>
If we enforced checksums to be done as a part of publication,
then this would address this issue, right?
<br>
<br>
<br>
On 3/8/12 8:39 AM,
<a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a><a class="moz-txt-link-rfc2396E" href="mailto:stephen.pascoe@stfc.ac.uk"><mailto:stephen.pascoe@stfc.ac.uk></a>
wrote:
<br>
<br>
Tobias, sorry I miss-typed your name :-)
<br>
S.
<br>
<br>
On 8 Mar 2012, at
16:00,<a class="moz-txt-link-rfc2396E" href="mailto:stephen.pascoe@stfc.ac.uk"><stephen.pascoe@stfc.ac.uk></a><a class="moz-txt-link-rfc2396E" href="mailto:stephen.pascoe@stfc.ac.uk"><mailto:stephen.pascoe@stfc.ac.uk></a><br>
wrote:
<br>
<br>
<br>
<br>
Hi Thomas,
<br>
<br>
As you say, it's too late to do much re-engineering of the
system now -- we've attempted to put in place various
identifier systems and none of them are working particularly
well -- however I think there is another perspective to your
proposal:
<br>
<br>
1. ESG/CMIP5 is deployed globally across multiple
administrative domains and each domain has the ability to cut
corners to get things done, e.g. replacing files silently
without changing identifiers.
<br>
<br>
2. ESG/CMIP5 system is so complex that who'd blame a sys-admin
for doing #1 to get the data to scientists when they need it.
Any system that makes it impossible, or even only difficult,
to change the underlying data is going to be more complex and
difficult to administer than a system that doesn't, unless
that system was very rigorously designed, implemented and
tested.
<br>
<br>
Because of #1 I'm convinced that a fit-for-purpose identifier
system wouldn't use randomly generated UUIDs but would take
the GIT approach of hashing invariants of the dataset so that
any changes behind the scenes can be detected.
<br>
<br>
Because of #2 I'm convinced that now is not the time to start
building more software to do this. We have to stabilise the
system and learn the lessons of CMIP5 first.
<br>
<br>
Cheers,
<br>
Stephen.
<br>
<br>
<br>
On 8 Mar 2012, at 15:32, Tobias Weigel wrote:
<br>
<br>
<br>
<br>
Jamie/All,
<br>
<br>
these are important questions I have been wondering about as
well; we just had a small internal meeting yesterday with
Estani and Martina, so I'll try to sum some points up here. I
am not too familiar with the ESG publishing process, so I can
only guess that Stephen's #1 has something to do with the
bending of policies that are for pragmatic reasons not
enforced in the CMIP5 process. (My intuition is that *ideally*
it should be impossible to make data available without going
through the whole publication process. Please correct me if I
am misunderstanding this.)
<br>
<br>
Most of what I have been thinking about however concerns point
#2. I'd claim that the risk here should not be underestimated;
data consumers being unable to find the data they need is bad
("the advanced search issue"), but users relying on deprecated
data - most likely without being aware of it - is certainly
dangerous for scientific credibility.
<br>
My suggestion to address this problem is to use globally
persistent identifiers (PIDs) that are automatically assigned
to data objects (and metadata etc.) on ESG-publication; data
should ideally not be known by its file name or
system-internal ID, but via a global identifier that never
changes after it has been published. Of course, this sounds
like the DOIs, but these are extremely coarse grained and very
static. The idea is to attach identifiers to the low-level
entities and provide solutions to build up a hierarchical ID
system (virtual collections) to account for the various layers
used in our data. Such persistent identifiers should then be
placed prominently in any user interface dealing with managed
data. The important thing is: If data is updated, we don't
update the data behind identifier x, but assign a new
identifier y and create a typed link between these two (which
may be the most challenging part) and perhaps put a small
annotation on x that this data is depreca
<br>
ted. A clever user interface should then redirect a user
consistently to the latest version of a dataset if a user
accesses the old identifier.
<br>
This does not make it impossible to use deprecated data, but
at least it raises the consumer's awareness of the issue and
lowers the barrier to re-retrieve valid data.
<br>
<br>
As for the point in time; I'd be certain that it is too late
now, but it is always a good idea to have plans for future
improvement.. :)
<br>
<br>
Best, Tobias
<br>
<br>
Am 08.03.2012 13:06, schrieb Kettleborough, Jamie:
<br>
<br>
<br>
Thanks for the replies on this - any other replies are still
very welcome.
<br>
<br>
Stephen - being selfish - we aren't too worried about 2 as its
less of an issue for us (we do a daily trawl of thredds
catalogues for new datasets), but I agree it is a problem more
generally. I don't have a feel for which of the problems 1-3
would minimise the risk most if you solved it. I think making
sure new data has a new version is a foundation though.
<br>
<br>
Part of me wonders though whether its already too late to
really do anything with versioning in its current form. *But*
I may be overestimating the size of the problem of new
datasets appearing without versions being updated.
<br>
<br>
Jamie
<br>
<br>
<br>
<br>
<br>
-----Original Message-----
<br>
From:
<a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a><a class="moz-txt-link-rfc2396E" href="mailto:go-essp-tech-bounces@ucar.edu"><mailto:go-essp-tech-bounces@ucar.edu></a><br>
[<a class="moz-txt-link-freetext" href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Sébastien
Denvil
<br>
Sent: 08 March 2012 10:41
<br>
To: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><a class="moz-txt-link-rfc2396E" href="mailto:go-essp-tech@ucar.edu"><mailto:go-essp-tech@ucar.edu></a>
<br>
Subject: Re: [Go-essp-tech] What is the risk that science is
<br>
done using 'deprecated' data?
<br>
<br>
Hi Stephen, let me add a third point:
<br>
<br>
3. Users are aware of a new versions but can't download files
<br>
so as to have a coherent set of files.
<br>
<br>
With respect to that point the p2p transition (especially the
<br>
attribut caching on the node) will be a major step forward.
<br>
GFDL just upgrad and we have an amazing success rate of 98%.
<br>
<br>
And I agree with Ashish.
<br>
<br>
Regards.
<br>
Sébastien
<br>
<br>
Le 08/03/2012 11:34,
<a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a><a class="moz-txt-link-rfc2396E" href="mailto:stephen.pascoe@stfc.ac.uk"><mailto:stephen.pascoe@stfc.ac.uk></a>
a écrit :
<br>
<br>
<br>
Hi Jamie,
<br>
<br>
I can imagine there is a risk of papers being written on
<br>
<br>
<br>
deprecated data in two scenarios:
<br>
<br>
<br>
1. Data is being updated at datanodes without creating a
<br>
<br>
<br>
new version
<br>
<br>
<br>
2. Users are unaware of new versions available and
<br>
<br>
<br>
therefore using
<br>
<br>
<br>
deprecated data
<br>
<br>
Are you concerned about both of these scenarios? Your
<br>
<br>
<br>
email seems to mainly address #1.
<br>
<br>
<br>
Thanks,
<br>
Stephen.
<br>
<br>
On 8 Mar 2012, at 10:21, Kettleborough, Jamie wrote:
<br>
<br>
<br>
<br>
Hello,
<br>
<br>
Does anyone have a feel for the current level of risk that
<br>
<br>
<br>
analysists
<br>
<br>
<br>
are doing work (with the intention to publish) on data
<br>
<br>
<br>
that has been
<br>
<br>
<br>
found to be wrong by the data providers and so deprecated (in
some
<br>
sense)?
<br>
<br>
My feeling is that versioning isn't working (that may be
<br>
<br>
<br>
putting it a
<br>
<br>
<br>
bit strongly. It is too easy for data providers - in their
<br>
understandable drive to get their data out - to have
<br>
<br>
<br>
updated files on
<br>
<br>
<br>
disk without publishing a new version. How big a deal does
anyone
<br>
think this is?
<br>
<br>
If the risk that papers are being written based on
<br>
<br>
<br>
deprecated data is
<br>
<br>
<br>
sufficiently large then is there an agreed strategy for
<br>
<br>
<br>
coping with
<br>
<br>
<br>
this? Does it have implications for the requirements of the
data
<br>
publishing/delivery system?
<br>
<br>
Thanks,
<br>
<br>
Jamie
<br>
_______________________________________________
<br>
GO-ESSP-TECH mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><a class="moz-txt-link-rfc2396E" href="mailto:GO-ESSP-TECH@ucar.edu"><mailto:GO-ESSP-TECH@ucar.edu></a>
<br>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
<br>
<br>
<br>
--
<br>
Sébastien Denvil
<br>
IPSL, Pôle de modélisation du climat
<br>
UPMC, Case 101, 4 place Jussieu,
<br>
75252 Paris Cedex 5
<br>
<br>
Tour 45-55 2ème étage Bureau 209
<br>
Tel: 33 1 44 27 21 10
<br>
Fax: 33 1 44 27 39 02
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________
<br>
GO-ESSP-TECH mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><a class="moz-txt-link-rfc2396E" href="mailto:GO-ESSP-TECH@ucar.edu"><mailto:GO-ESSP-TECH@ucar.edu></a>
<br>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
<br>
<br>
<br>
<br>
<br>
--
<br>
Tobias Weigel
<br>
<br>
Department of Data Management
<br>
Deutsches Klimarechenzentrum GmbH (German Climate Computing
Center)
<br>
Bundesstr. 45a
<br>
20146 Hamburg
<br>
Germany
<br>
<br>
Tel.: +49 40 460094 104
<br>
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:weigel@dkrz.de">weigel@dkrz.de</a><a class="moz-txt-link-rfc2396E" href="mailto:weigel@dkrz.de"><mailto:weigel@dkrz.de></a>
<br>
Website: <a class="moz-txt-link-abbreviated" href="http://www.dkrz.de">www.dkrz.de</a><a class="moz-txt-link-rfc2396E" href="http://www.dkrz.de/"><http://www.dkrz.de/></a>
<br>
<br>
Managing Director: Prof. Dr. Thomas Ludwig
<br>
<br>
Sitz der Gesellschaft: Hamburg
<br>
Amtsgericht Hamburg HRB 39784
<br>
<br>
<br>
_______________________________________________
<br>
GO-ESSP-TECH mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><a class="moz-txt-link-rfc2396E" href="mailto:GO-ESSP-TECH@ucar.edu"><mailto:GO-ESSP-TECH@ucar.edu></a>
<br>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
<br>
<br>
<br>
--
<br>
Scanned by iCritical.
<br>
_______________________________________________
<br>
GO-ESSP-TECH mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><a class="moz-txt-link-rfc2396E" href="mailto:GO-ESSP-TECH@ucar.edu"><mailto:GO-ESSP-TECH@ucar.edu></a>
<br>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
<br>
<br>
<br>
<br>
--
<br>
Gavin M. Bell
<br>
--
<br>
<br>
"Never mistake a clear view for a short distance."
<br>
-Paul Saffo
<br>
<br>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Tobias Weigel
Department of Data Management
Deutsches Klimarechenzentrum GmbH (German Climate Computing Center)
Bundesstr. 45a
20146 Hamburg
Germany
Tel.: +49 40 460094 104
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:weigel@dkrz.de">weigel@dkrz.de</a>
Website: <a class="moz-txt-link-abbreviated" href="http://www.dkrz.de">www.dkrz.de</a>
Managing Director: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784</pre>
</body>
</html>