<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
Dear all,<br>
<br>
so as to share the way we manage versions and downloads I have
written down a few key points.<br>
<br>
The main one could be : at the scale of CMIP5 "the way to easiest
science" will be achieved with distributed archive but with
multi-centralized/multi-polar analysis.... Just a thought.<br>
<br>
<b>- Versioning policy.</b><br>
<br>
We used drslib since our first publication and follow exactly the
DRS<br>
We decided that any change in a dataset (new file(s), removed
file(s), updated file(s)) will trigger a new dataset version, even
before QC take place<br>
Successive dataset versions are managed by drslib (obvious but
still...)<br>
We are in the process of publishing new dataset version due to new,
removed & updated file(s). So we will let you know our findings<br>
The document describing "Versioning policy" that I was looking after
could be this one.<br>
<b>CMIP5 Dataset Version Directory Structure (Stephen Pascoe,
version 0.5, 2010-06-04)</b> I have a printed version of it but
cannot find it in my mail box.<br>
<br>
<b>- Download strategy</b>.<br>
<br>
We download a predefined subset of the CMIP5 archive to sustain
analysis activity of scientists (100+ from WG1 family) on a
dedicated cluster (<span id="ctl00_cC_res1_rE_ctl00_lblWordType"
title="Nom - Féminin" direction=""></span><span
id="ctl00_cC_res1_rE_ctl00_lblTarget" class="ellipsis_text"
direction="target">economy of scale</span>)<br>
We download from PKI enabled data node only<br>
We rely on thredds catalogue information to make decision for
downloading (dataset_version, checksums, tracking_id)<br>
We rely on thredds catalogue information and on drslib to locally
reproduce DRS structure alongside IPSL data (scientists love
homogeneous directory structure)<br>
We maintain manually a datanodes list we try to download from<br>
- first pass --> download subset (store all info as you can
in a db at the file level : transfert_rate, date, versions,
tracking_id, data size per time step ....)<br>
check checksums when available otherwise
check size<br>
-<span id="result_box" class="short_text" lang="en"><span
title="Cliquer ici pour voir d'autres traductions" class="hps">
periodically</span></span><br>
- update the subset definition<br>
- from db compute new subset size ---> yes/no it can fit
on our storage<br>
- from db compute time it should take to download<br>
- next pass --> if dataset_version changed<br>
download file if checksum changed
(if no checksum : download file if tracking_id changed) and update
db.<br>
check checksums when available
otherwise check size<br>
<br>
That's how it is, imperfect but it fulfils main use cases.<br>
Regards.<br>
Sébastien<br>
<br>
On 08/07/2011 12:48, Estanislao Gonzalez wrote:
<blockquote cite="mid:4E16E07F.7070305@dkrz.de" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
Hi Sebastien,<br>
<br>
The only part which was not resolved (or agreed upon) was the
first level after /fileServer/ for the file access capabilities of
the TDS. so every node has something different (we will be
publishing to /fileServer/cmip/out... though)<br>
<br>
what the drslib do is threefold:<br>
1) It separates output into output1 and output2 since only output2
is interesting for replication (that's how I got IPSL aqua4K
experiment, just skipped everything that was output2..)<br>
2) It version the files (and thus inserts the version into the DRS
structure). This helps finding the version and is the only way I
know of, that it can be gathered from the Gateway.<br>
3) it recreates the DRS structure assuring is a valid one. For
reasons I'm not aware of, it misses the activity part so you can
still end up with a non valid DRS... in CNRM case it means it will
not validate the CMIP5 which should have been cmip5 (sadly
computers are worse than the worst bureaucrats :-)<br>
<br>
The drslib it's quite well described (In my opinion) and it's
here: <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://esgf.org/esgf-drslib-site/">http://esgf.org/esgf-drslib-site/</a><br>
All documentation regarding the datanode and all tools around it
can be found here: <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://esgf.org/wiki/ESGF_Node">http://esgf.org/wiki/ESGF_Node</a><br>
<br>
Hope this helps,<br>
Estani<br>
<br>
Am 08.07.2011 12:13, schrieb Stéphane Senesi:
<blockquote cite="mid:4E16D858.5010802@meteo.fr" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<link href="chrome://translator/skin/floatingPanel.css"
type="text/css" rel="stylesheet">
Hi all,<br>
<br>
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:martin.juckes@stfc.ac.uk">martin.juckes@stfc.ac.uk</a>
wrote, On 08/07/2011 11:09:
<blockquote
cite="mid:E21FBC3F00D7304687CB46529F9676D712D5CB@EXCHMBX01.fed.cclrc.ac.uk"
type="cite">
<pre wrap="">Hi Estani,
I agree with you that this is an important issue and that we want to have a clean implementation.
Unfortunately, given where we are now, I don't think there is going to be any support for withdrawing data nodes which don't meet this implementation standard -- so enforcement by the gateway won't work. So I think the only way forward is to work on simplifying the installation and then persuade the node managers to adopt the standard. Making it the default would, as you suggest, be a huge help.
I keep telling our users in the UK that the archive is currently in a very early stage, with a significant chance that data will be replaced. The same applies to the level of service. I think we need to work on demonstrating best practise as far as data node deployment goes.
At the moment I see the PKI security as a higher priority, since most of our users want scripting access rather than clicking through the gateways, and this only works when the PKI security is enabled.
For the versioning implementation, it would help to have a step by step guide on esgf.org (or if it is already there, it would help me to understand the issues if I knew where it is) -- but I guess this will have to wait until Stephen has worked through some other priorities.
</pre>
</blockquote>
<br>
Regarding CNRM data node, what prevented us to turn to the
"recommended" directory structure (it is not coined as
"standard" in CMIP5 documents), was the lack of such a guide<br>
<br>
I agree with Martin that it is important to ease an
OpenDAP-enabled scripted access for data users; if it appears
that this version issue is the only obstacle for computing
datafiles addresses (not quoting the issue of data node name),
then we can consider changing the directory structure (assuming
we have the guide). <br>
<br>
Alss, I note that the first part of HTTPServer URL's also show a
part which may vary on a datanode basis,and even on an
experiment or realm basis (such as the boldface part in <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://esg.cnrm-game-meteo.fr/thredds/fileServer/">http://esg.cnrm-game-meteo.fr/thredds/fileServer/</a><b>esg_dataroot1</b>/CMIP5/output/CNRM-CERFACS/CNRM-CM5/historicalGHG/mon/land/evspsblsoi/r1i1p1/evspsblsoi_Lmon_CNRM-CM5_historicalGHG_r1i1p1_190001-194912.nc)
.
Would this be cured by applying drslib ?<br>
<br>
On a very close subject, may I quote Sébastien Denvil ( 9 june
2011), with whom I agree :<br>
<br>
<blockquote type="cite">I would like to remind us all that
having a clear add/remove/update procedure is a requirement
together with add/remove/update impact on versions (dataset
version, file version). <br>
<br>
It's clear we do our best to publish the right dataset. It's
clear too that QC process, and scientific process will spot
issues (acceptable or not) and will trigger add/remove/update
actions. <br>
<br>
I can't remember if a clear document describing
publish/unpublish procedure exist. That should describe from
both perspective (data provider/those in charge of
replication) how to: <br>
<br>
- add file(s) within existing datasets <br>
- remove file(s) from existing datasets <br>
- update files(s) from existing datasets (is that just
add/remove? not if we want easy life for replication. Yes if
we want easy life for data provider) <br>
<br>
If such document doesn't exist yet I think it is a priority
(given where we are) to produce one. <br>
<br>
Can someone points me that document? <br>
</blockquote>
<br>
Regards<br>
<br>
Stéphane<br>
<br>
<blockquote
cite="mid:E21FBC3F00D7304687CB46529F9676D712D5CB@EXCHMBX01.fed.cclrc.ac.uk"
type="cite">
<pre wrap="">It should be possible to get all this fixed in time, but I think people are working through a large number of issues in parallel at present.
Cheers,
Martin
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">-----Original Message-----
From: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a moz-do-not-send="true" class="moz-txt-link-freetext" href="mailto:go-essp-tech">mailto:go-essp-tech</a>-
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:bounces@ucar.edu">bounces@ucar.edu</a>] On Behalf Of Estanislao Gonzalez
Sent: 07 July 2011 16:46
To: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>
Subject: [Go-essp-tech] DRS, version number & Co
Hi,
What's the current stand regarding DRS and dataset version number?
I've seen too many data nodes with too many different configurations.
>From invalid datasets name to invalid DRS structure, names, missing
version numbers, etc.
The version number is a particular interesting one, since in some
cases
the only way to find it is by parsing the TDS Catalogs themselves,
since
the Gateway is not providing this info (AFAICT) and if the DRS is not
followed can neither be implied from the directory structure of its
files.
Obviously neither the publisher nor the Gateway is enforcing those
constraints. I think this should be changed ASAP.
Both Node and Gateway publishing steps should enforce this when
publishing for cmip5. I think is the most direct way to get to the
publisher at the right time.
If we keep drifting away from what we already agreed on, we won't be
able to do anything useful with the data at all, since we won't be
able
to handle it properly.
I'll urge the data node managers to check DRS compliance.
I've only seen BADC publishing according to the DRS structure. I know
PCMDI, BCC, CNRM and NCCS are not. I haven't checked others.
Thanks,
Estani
--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a>
_______________________________________________
GO-ESSP-TECH mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
</blockquote>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Stéphane Sénési
Ingénieur - équipe Assemblage du Système Terre
Centre National de Recherches Météorologiques
Groupe de Météorologie à Grande Echelle et Climat
CNRM/GMGEC/ASTER
42 Av Coriolis
F-31057 Toulouse Cedex 1
+33.5.61.07.99.31 (Fax :....9610)</pre>
<div style="display: none; bottom: auto; left: 145px; right:
auto; top: 283px;" class="translator-theme-default"
id="translator-floating-panel"> </div>
<pre wrap=""><fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> </pre>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
</pre>
</body>
</html>