<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title></title>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<meta name="GENERATOR" content="MSHTML 8.00.6001.19088">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi everyone,<br>
<br>
we promised to describe the problems regarding the non-DRS file
structures at the data nodes.<br>
Estani has already started the discussion on the replication/user
download problems (see attached email and document).<br>
<br>
Implications for the QC:<br>
- In the QCDB we need DRS syntax. The DOI process, creation of CIM
documents, and identification of the data the QC results are
connected to rely on that.<br>
- The QC needs to know the version of the data checked. The DOI at
the end of the QC process is assigned to a specific not-changable
data version. At least at DKRZ we have to guarantee that the data is
not changed after assignment of the DOI, therefore we store a data
copy in our archive.<br>
- The QC checker tool runs on files in a given directory structure
and creates results in a copy of this structure. The QC wrapper can
deal with recombinations of path parts. So, if the directory
structure includes all parts of the DRS syntax, the wrapper can
create the DRS syntax before insert in the QCDB. But we deal with
structures at the data nodes, where some information is missing in
the directory path, i.e. version and MIP table. Therefore an
additional information would be needed for that mapping.<br>
<br>
Possible solutions to map the given file structure to the DRS
directory structure before insert in the QCDB:<br>
<br>
1. The publication on the data nodes of the three gateways who store
replicas (PCMDI, BADC, DKRZ) publish data in the DRS directory
structure. Then the QC run is possible without mapping. Replication
problems? <br>
<br>
2. The directory structures of the data nodes are replicated as they
are. We store the data under a certain version. How? Are there
implications for the replication from the data nodes? The individual
file structures down to the chunk level are stored together with its
DRS identification in a repository and a service is created to
access the DRS id for the given file in the given file structure.
The QC and maybe other user data services use this service for
mapping. That will slow down the QC insert process. Before each
insert of a chunk name, a qc result for a specific variable, and the
qc result on the experiment level that service has to be called. And
who can set-up and maintain such a repository? DKRZ has not the man
power to do that in the next months.<br>
<br>
Cheers,<br>
Martina<br>
<br>
<br>
<br>
-------- Original-Nachricht --------
<table class="moz-email-headers-table" border="0" cellpadding="0"
cellspacing="0">
<tbody>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Betreff: </th>
<td>RE: ESG discussion</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Datum: </th>
<td>Wed, 10 Aug 2011 15:35:04 +0100</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Von: </th>
<td>Kettleborough, Jamie <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:jamie.kettleborough@metoffice.gov.uk"><jamie.kettleborough@metoffice.gov.uk></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">An: </th>
<td>Karl Taylor <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:taylor13@llnl.gov"><taylor13@llnl.gov></a>,
Wood, Richard <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:richard.wood@metoffice.gov.uk"><richard.wood@metoffice.gov.uk></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">CC: </th>
<td>Carter, Mick <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:mick.carter@metoffice.gov.uk"><mick.carter@metoffice.gov.uk></a>,
Elkington, Mark <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:mark.elkington@metoffice.gov.uk"><mark.elkington@metoffice.gov.uk></a>,
Bentley, Philip <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:philip.bentley@metoffice.gov.uk"><philip.bentley@metoffice.gov.uk></a>,
Senior, Cath <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:cath.senior@metoffice.gov.uk"><cath.senior@metoffice.gov.uk></a>,
Hines, Adrian <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:adrian.hines@metoffice.gov.uk"><adrian.hines@metoffice.gov.uk></a>,
Dean N. Williams <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:williams13@llnl.gov"><williams13@llnl.gov></a>,
Estanislao Gonzalez <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:gonzalez@dkrz.de"><gonzalez@dkrz.de></a>,
<a moz-do-not-send="true" class="moz-txt-link-rfc2396E"
href="mailto:martin.juckes@stfc.ac.uk"><martin.juckes@stfc.ac.uk></a>,
Kettleborough, Jamie <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:jamie.kettleborough@metoffice.gov.uk"><jamie.kettleborough@metoffice.gov.uk></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<title></title>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<meta name="GENERATOR" content="MSHTML 8.00.6001.19088">
<div dir="ltr" align="left">
<p style="margin: 6pt 0cm 0pt;" class="MsoNormal"><span style=""><font
face="Arial">Hello Karl,<span class="581071214-10082011">
Dean,</span></font></span></p>
<p style="margin: 0cm 0cm 0pt;" class="MsoNormal"><span
style="font-family: 'Times New Roman'; color: black;"> <!--?xml:namespace
prefix = o ns = "urn:schemas-microsoft-com:office:office"
/--><o:p></o:p></span></p>
<p style="margin: 0cm 0cm 0pt;" class="MsoNormal"><span style=""><font
face="Arial">Thanks for you reply on this, and the fact you
are taking our concerns seriously. You are right to
challenge us for the specific issues, rather than <span
class="581071214-10082011">us </span>just highlight<span
class="581071214-10082011">ing </span>the things that
don't meet our (possibly idealised) expectations of how the
system should look. As a result, we have had a thorough
review of our key issues. I think some of them are issues
that make if harder for us to do things now; other issues
are maybe more concerns of problems being stored up. </font></span></p>
<span style=""><o:p>
<p style="margin: 6pt 0cm 0pt;" class="MsoNormal"><span
style=""><font face="Arial">This <span
class="581071214-10082011">document </span>has been
prepared with<span class="581071214-10082011"> the help</span> Estani
Gonzalez. We would like to have Martin Juckes input on
this too - but he is currently away on holiday. I hope
he can add to this when he returns - he has spen<span
class="581071214-10082011">t a lot of time thinking
about the implications of data node directory
structure on versioning.</span></font></span></p>
<p style="margin: 6pt 0cm 0pt;" class="MsoNormal"><span
style="font-family: 'Times New Roman'; color: black;"><o:p><span
class="581071214-10082011"><font face="Arial">I hope
this helps clarify issues, if not please let use
know,</font></span></o:p></span></p>
<p style="margin: 6pt 0cm 0pt;" class="MsoNormal"><span
style="font-family: 'Times New Roman'; color: black;"><o:p><span
class="581071214-10082011"><font face="Arial">Thanks,</font></span></o:p></span></p>
<p style="margin: 6pt 0cm 0pt;" class="MsoNormal"><span
style="font-family: 'Times New Roman'; color: black;"><o:p><span
class="581071214-10082011"><font face="Arial">Jamie</font></span></o:p></span></p>
</o:p></span></div>
<br>
<blockquote style="border-left: 2px solid rgb(0, 0, 255);
padding-left: 5px; margin-left: 5px; margin-right: 0px;" dir="ltr">
<div dir="ltr" class="OutlookMessageHeader" lang="en-us"
align="left">
<hr tabindex="-1"> <font face="Tahoma" size="2"><b>From:</b>
Karl Taylor [<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="mailto:taylor13@llnl.gov">mailto:taylor13@llnl.gov</a>]
<br>
<b>Sent:</b> 09 August 2011 01:48<br>
<b>To:</b> Wood, Richard<br>
<b>Cc:</b> Carter, Mick; Kettleborough, Jamie; Elkington,
Mark; Bentley, Philip; Senior, Cath; Hines, Adrian; Dean N.
Williams<br>
<b>Subject:</b> Re: ESG discussion<br>
</font><br>
</div>
<font face="Times New Roman">Dear all,<br>
<br>
Thanks for taking the time to bring to my attention the ESG
issues that I hope can be addressed reasonably soon. I think
we're in general agreement that the user's experience should be
improved. <br>
<br>
I've discussed this briefly with Dean. I plan to meet with him
and others here, and, drawing on your suggestions, we'll attempt
to find solutions and methods of communication that might
improve matters. Before doing this, it would help if you could
briefly answer the following questions:<br>
<br>
1. Is the main issue that it is currently difficult to script
downloads from all the nodes because only some support PKI?
What other uniformity among nodes is required for you to be able
to do what you want to do (i.e., what do you specifically want
to do that is difficult to do now)? [nb. all data nodes are
scheduled to be operating with PKI authentication by September
1.]<br>
<br>
2. Is there anything from the perspective of a data *provider*
that needs to be done (other than make things easier for data
users)?<br>
<br>
3. Currently ESG and CMIP5 do not dictate the directory
structure found at each data node (although most nodes are
adhering to the recommendations of the DRS). The gateway
software and catalog make it possible to get to the data
regardless of directory structure. It is possible that
"versioning" might impose additional constraints on the
directory structure, but I'm not sure about this. </font><font
face="Times New Roman">(By the way, I'm not sure what the
"versioning" issue is since currently I think it's impossible
for users to know about more than one version; is that the
issue?) </font><font face="Times New Roman">From a user's or
provider's perspective, is there any essential reason that the
directory structure should be the same at each node? <br>
<br>
4. ESG allows considerable flexibility in publishing data, and
CMIP5 has suggested "best practices" to reduce differences.
Only some of the "best practices" are currently requirements. A
certain amount of flexibility is essential since different data
providers have resources to support the potential capabilities
of ESG (e.g., not all can support server-side calculations,
which will be put in place at some nodes). Likewise a provider
can currently turn off the "checksum", if this is deemed to slow
publication too much (although we could insist that checksums be
stored in the thredds catalogue). Nevertheless, it is unlikely
that every data node will be identically configured for all
options. What are the *essential* ways that the data nodes
should respond identically (we may not be able to insist on
uniformity that isn't essential for serving our users)?<br>
<br>
Thanks again for your input, and I look forward to your further
help with this.<br>
<br>
Best regards,<br>
Karl<br>
<br>
</font><br>
On 8/5/11 10:43 AM, Wood, Richard wrote:
<blockquote
cite="mid:E51EDFEBF10BE44BB4BDAF5FC2F024B90322854A@EXXMAIL02.desktop.frd.metoffice.com"
type="cite">
<pre wrap="">Dear Karl,
Following on from our phone call I had a discussion with technical
colleagues here (Mick Carter, Jamie Kettleborough, Mark Elkington, also
earlier with Phil Bentley), and with Adrian Hines who is coordinating
our CMIP5 analysis work, about ideas for future development of the ESG.
Our observations are from the user perspective, and based on what we can
gather from mailing lists and our own experience. Coming out of our
discussion we have a couple of suggestions that could help with
visibility for data providers and users:
- Some areas need agreement among the data nodes as to the technical
solution, and then implementation across all the nodes, while others
need a specific solution to be developed in one place and rolled out.
The group teleconferences that Dean organises appear to be a good forum
for airing specific technical ideas and solutions. However, in our
experience it can be difficult in that kind of forum to discuss
planning and prioritisation questions. From our perspective we don't
have visibility of the more project-related issues such as key technical
decisions, prioritisation and timelines, or of whether issues that have
arisen in the mailing list discussions are being followed up. We guess
these may be discussed in separate project teleconferences involving the
technical leads from the data nodes. As users we would not necessarily
expect to be involved in those discussions, but as data providers and
dowloaders it would be very helpful for our planning to see the outcomes
of the discussions. The sort of thing we had in mind would be a simple
web page showing the priority development areas, agreed solutions and
estimated dates for completion/release. Some solutions will need to be
implemented separately across all the participating data nodes, and in
these cases it would be useful to see the estimated timeframe for
implementation at each node.
This would not be intended as a 'big stick' to the partners, but simply
as a planning aid so that everyone can see what's available when and the
project can identify any potential bottlenecks or issues in advance.
Also the intention is not to generate a lot of extra work. Hopefully
providing this information would be pretty light on people's time.
- From where we sit it appears that some nodes are quite successful in
following best practice and implementing the federation policies as far
as they are aware of them. Could what these nodes do be made helpful to
all the data nodes (e.g. by using identical software)? We realise there
may be real differences between some data nodes - but where possible we
think that what is similar could be enforced or made explicitly the same
through sharing the software components and tools.
To set the discussion on priorities rolling, Jamie has prepared, in
consultation with others here, a short document showing the Met Office
view of current priority issues (attached). If you could update us on
the status of work on these issues, that would be very useful (ideally
via the web pages proposed above, which we think would be of interest to
many users, or via email in the interim). Many thanks for the update on
tokenless authentication, which is very good news.
Once again, our thanks to you, Dean and the team for all the hard work
we know is going into this. Please let us know what you think of the
above ideas and the attachment, and if there is anything we can do to
help.
Best wishes,
Richard
--------------
Richard Wood
Met Office Fellow and Head (Oceans, Cryosphere and Dangerous Climate
Change)
Met Office Hadley Centre
FitzRoy Road, Exeter EX1 3PB, UK
Phone +44 (0)1392 886641 Fax +44 (0)1392 885681
Email <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:richard.wood@metoffice.gov.uk">richard.wood@metoffice.gov.uk</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.metoffice.gov.uk">http://www.metoffice.gov.uk</a>
Personal web page
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.metoffice.gov.uk/research/scientists/cryosphere-oceans/richar">http://www.metoffice.gov.uk/research/scientists/cryosphere-oceans/richar</a>
d-wood
*** Please note I also work as Theme Leader (Climate System) for the
Natural Environment Research Council ***
*** Where possible please send emails on NERC matters to <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:rwtl@nerc.ac.uk">rwtl@nerc.ac.uk</a>
***
</pre>
</blockquote>
</blockquote>
</body>
</html>