<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hello All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The careful users can keep track of the versions downloaded by keeping their wget scripts – but I agree that this is not a very good system. If every data node
was using the agreed DRS directory structure, it would be easy to modify the wget scripts to preserve the version information in the directory structure on the user side. At present, however, it appears that most many modelling groups are not using the DRS
directory structure or any form of version control. So the archive managers can’t keep track of version changes, let alone the users.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I believe it was intentional to make access much less restricted than at the equivalent stage of CMIP3 in order to get as many groups as possible involved in
the initial evaluation of the data. As far as I can see, the only way of warning users about the volatility at present would be a prominent message on the gateway home page (and we should review what we are saying on the BADC ESGF gateway).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">On the issue of where to put QC information, we have a clear plan to put it in the CIM metadata – and I think that work to link this up to the listing of datasets
in the gateway is progressing. But, of course, the plan is to run the QC after the data has been moved to PCMDI, BADC or DKRZ and the software to achieve that move in a reliable way is not ready – so getting QC information is not easy. Possibly this does not
matter, as a message of the form “no quality assurance information available” should give most users the information they need,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Martin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> go-essp-tech-bounces@ucar.edu
[mailto:go-essp-tech-bounces@ucar.edu] <b>On Behalf Of </b>Estanislao Gonzalez<br>
<b>Sent:</b> 26 May 2011 08:22<br>
<b>To:</b> Sébastien Denvil<br>
<b>Cc:</b> go-essp-tech@ucar.edu<br>
<b>Subject:</b> Re: [Go-essp-tech] resolution on securing opendap aggregations via ESGF<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Sébastien,<br>
<br>
I'm aware this is how it was intended to be. But among the increasing number of problems submitted to the esg-support, there are a few regarding the retraction of datasets. This tells me that either some modelling groups are not aware of this possibility or
that there are non-modelers accessing the data at this stage. Adding the fact that you cannot tell which version was downloaded (AFAIK it's only encoded in the DRS structure and lost while downloaded) I expect it to cause problems even for modelers.<br>
<br>
I'm just thinking how we can minimize the number of complaints we get.<br>
<br>
Thanks,<br>
Estani<br>
Am 25.05.2011 12:57, schrieb Sébastien Denvil: <o:p></o:p></p>
<p class="MsoNormal">Hi all,<br>
<br>
On 24/05/2011 09:59, Estanislao Gonzalez wrote: <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi,<br>
<br>
just to be more precise: indeed I think security and CIM data (meta-data from experiments and so on) should be kept away from the catalogs.<o:p></o:p></p>
<p class="MsoNormal"><br>
I also agree that access policy should be decoupled from the publication step and that CIM metadata should be kept away from the catalogs. CIM instances are exposed through atom feed and services ; that's enough. I agree it's not fairly easy to match files
and associated CIM instances but everything is available to do this mapping.<br>
<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">Still, Luca's idea of merging QC level info with the files itself might be a valid idea. The difference is that QC is pretty much like a "semantic" checksum for a file. AFAIK you cannot "downgrade" the QC without altering the file, i.e.
if the file is QC L2 and while performing QC L3 checks an error is found, the file will either be QC L3 approved (the modeler defines the "oddity" as "expected") or it doesn't, which implies a "QC L2 passed; QC L3 failed" flag or the retraction (and maybe
re-publication) of it altogether.<br>
<br>
Well, that's at least why I think the QC flag is a little different and it's *closely* related to the file. The only difference with the checksum is IMHO that it takes more time to be determined (as well as require other files for it's computation) and thus
it's performed in an out-of-band fashion.<br>
<br>
We need that QC flag somewhere... and it's far more important than the rest of the CIM meta-data (getting back to Gavin's point about CIM issues and differentiating it to this QC flag: yes, you can still get to the file and download it without CIM data... but
without the QC flag you'll have no clue if you *really* want to rely on this data).<o:p></o:p></p>
<p class="MsoNormal"><br>
Generally speaking I don't think QC flag is far more important than the rest of the CIM meta-data. At least climate modellers and climate scientists will be able to perform their own QC. If something is wrong with a file (bad units, bad variable (example :
precipitation claiming it's a temperature),... , or others discrepancy far more difficult to detect) they will most likely detect it and gave feedback to the appropriate modelling groups. This direct feedback is very important to the whole process. CIM metadata
will put some more light over those data and will help scientists to decide up to which point they can rely on a dataset for a particular purpose.<br>
<br>
Regarding WG2, WG3, and commercial use of this data the QC flag will be something important. But one must keep in mind that producing information from a multi model - multi experiment project like CMIP5 is a challenging and extremely difficult task. One need
an incredible amount of information to perform the right decision. The QC flag won't be able to summarise that with a "use this data : yes or no"<br>
<br>
<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">To be honest I can't understand why people download this data if they *know* it might get corrected. Would you start writing a paper on something that might be altogether wrong?... I suspect they don't realize this.<o:p></o:p></p>
<p class="MsoNormal"><br>
The group CMIP5 research know that ; it's 100% part of the job. The process they will follow:<br>
<br>
- download variables & experiments they are interested in<br>
- perform a first home made analysis on those files<br>
- it's very likely they will catch a lot of things the QC tools won't<br>
- give feedbacks to the modelling groups that produced a dataset they found "strange"<br>
- modelling groups will analyse the situation and will decide to update or delete or keep the files unchanged<br>
- perform a more detailled analysis (catching may be a few more errors)<br>
- give feedbacks to the modelling groups that produced a dataset they found "strange"<br>
- modelling groups will analyse the situation and will decide to update or delete or keep the files unchanged<br>
- start to write their paper<br>
- will do a second round to check if data has been updated (new version, erased version)<br>
- download files that has been updated<br>
- discard files that has been deleted on esg<br>
- rerun their analysis procedure<br>
- update figures and conclusion of the analysis<br>
- will publish a paper that will include proper datasets<br>
- this paper will clearly mention a strange dataset. <br>
<br>
This process has already started.<br>
<br>
So as a modelling group you will pay extra attention to the feedbacks your receive (you don't want thousands of paper saying your data are strange). And you want to be sure that analyst will use the latest version of your files, or won't use it if you decided
to erase it from ESG.<br>
<br>
That worked like a charm for CMIP3. We want it better for CMIP5.<br>
<br>
Regards.<br>
Sébastien<br>
<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">Anyway, my 2c...<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 24.05.2011 03:06, schrieb Gavin M. Bell: <o:p></o:p></p>
<p class="MsoNormal">Hi Luca, <br>
<br>
I think that the separation of concerns trumps the apparent "simplicity". Though it is apparently easy to republish (I am not sure I fully agree with that, at least not from the anecdotal information I hear from folks)... it is unnecessary to publish if we
keep concerns separated.<br>
<br>
As Estani said, the publisher publishes and does basic mechanical sanity checks on data. That should be the full extent of its operation. As far as what is easy... one could 'easily' set up an index over the CIM info and "join" on datasetid. This also provides
loose coupling. If the CIM system has issues, that just means that when you look at your search results you won't see CIM info, but you will still see the dataset and be able to fetch and manipulate it and everything else. Also if the CIM changes it doesn't
affect the pubblisher or publishing in any way. Catalogs should be viewed as "files" in the system... they essentially are logical files (containing pointers to physical files).<br>
<br>
I am still not convinced by your arguments that fusing and coupling these two semantically different aspects of the system so tightly is the right long term architectural solution. It may be good now, but it not as flexible later. We should leave open the
avenue for other meta-metadata to be imbued onto our system ex-post-facto without much ado.<br>
<br>
my $0.02<br>
<br>
On 5/23/11 2:08 AM, <a href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a> wrote:
<o:p></o:p></p>
<pre>I'm with Estani on this. Authorisation decisions are best decoupled from the application where possible. Phil is on leave today but I'm sure he'd say the same thing and give much more detailed reasoning. <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I think the catalogue already mixes slightly too much information together: location-independent file metadata and location-specific service information. If we add access control it becomes too tightly coupled.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Stephen.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>---<o:p></o:p></pre>
<pre>Stephen Pascoe +44 (0)1235 445980<o:p></o:p></pre>
<pre>Centre of Environmental Data Archival<o:p></o:p></pre>
<pre>STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>-----Original Message-----<o:p></o:p></pre>
<pre>From: <a href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Estanislao Gonzalez<o:p></o:p></pre>
<pre>Sent: 21 May 2011 09:30<o:p></o:p></pre>
<pre>To: Cinquini, Luca (3880)<o:p></o:p></pre>
<pre>Cc: <a href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><o:p></o:p></pre>
<pre>Subject: Re: [Go-essp-tech] resolution on securing opendap aggregations via ESGF<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Hi,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>In my opinion we shouldn't encode the access restriction in the catalog <o:p></o:p></pre>
<pre>for these reasons:<o:p></o:p></pre>
<pre>1) Changing the access would involved re-publishing the files. (this <o:p></o:p></pre>
<pre>will be done for instance when QC L2 is reached CMIP5 Research -> CMIP5 <o:p></o:p></pre>
<pre>Commercial). And think about what would happen if we want to change the <o:p></o:p></pre>
<pre>access restriction in a couple of years... we should publish everything <o:p></o:p></pre>
<pre>again, and that would involve quite some effort to understand the <o:p></o:p></pre>
<pre>procedure again...<o:p></o:p></pre>
<pre>2) I'm not sure of this, but I fear TDS security cannot handle multiple <o:p></o:p></pre>
<pre>roles. Right now you can publish to as many roles as required, and read <o:p></o:p></pre>
<pre>and write access is kept separately. This would involve extending the <o:p></o:p></pre>
<pre>TDS capabilities.<o:p></o:p></pre>
<pre>3) There could be potential inconsistencies if the authorization service <o:p></o:p></pre>
<pre>is detached from datanode (like with the gateway right now) and the <o:p></o:p></pre>
<pre>publisher alters the role but forgets to cascade the changes to the <o:p></o:p></pre>
<pre>authorizing service (which would proceed according to the last harvested <o:p></o:p></pre>
<pre>info)<o:p></o:p></pre>
<pre>4) And last but not least, I'm not sure we want to mix administration <o:p></o:p></pre>
<pre>with publication. The publisher should only care about making data <o:p></o:p></pre>
<pre>available, the administrator should organize this and be responsible for <o:p></o:p></pre>
<pre>the security.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>So basically I don't agree :-) Although I do think, if required, we <o:p></o:p></pre>
<pre>could change "esg-user" for "esgf-controlled" if it's more intuitive.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>My 2c anyways,<o:p></o:p></pre>
<pre>Estani<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Am 20.05.2011 19:17, schrieb Cinquini, Luca (3880):<o:p></o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>Hi,<o:p></o:p></pre>
<pre> a few points again on the issue of securing opendap aggregations served by the TDS with ESGF filters:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>o There's a new release of the ESGF security filters (esg-orp 1.1.2) that maps the TDS request URI to the dataset ID, and should solve this problem. You can experiment with the JPL test TDS server:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><a href="http://test-datanode.jpl.nasa.gov/thredds/catalog.html">http://test-datanode.jpl.nasa.gov/thredds/catalog.html</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>where the AIRS dataset (and aggregations) is secured, the MLS is not.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>o Now the data node authorization filter will correctly identify the aggregation as secured, and call the configured authorization service. Currently, the p2p Node authorization service can be configured to allow authorization based on URL matching, so it will work. The gateway authorization service will have to implement its own logic to establish authorization.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>o Finally, I am wondering if we shouldn't change the way we encode authorization in thredds catalogs. Right now, we use restrictAccess="esg-user" for ALL collections, but should we consider about encoding the proper required access control attribute instead, for example restrictAccess="CMIP5 Research" ? Something to think about - there are prons and cons about this - it's all a question on wether the access control belongs in the catalog (and can be harvested for searching...) or not.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>thanks, Luca<o:p></o:p></pre>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>GO-ESSP-TECH mailing list<o:p></o:p></pre>
<pre><a href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><o:p></o:p></pre>
<pre><a href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Gavin M. Bell<o:p></o:p></pre>
<pre>--<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> "Never mistake a clear view for a short distance."<o:p></o:p></pre>
<pre> -Paul Saffo<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Estanislao Gonzalez<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Max-Planck-Institut für Meteorologie (MPI-M)<o:p></o:p></pre>
<pre>Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre<o:p></o:p></pre>
<pre>Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Phone: +49 (40) 46 00 94-126<o:p></o:p></pre>
<pre>E-Mail: <a href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>GO-ESSP-TECH mailing list<o:p></o:p></pre>
<pre><a href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><o:p></o:p></pre>
<pre><a href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a><o:p></o:p></pre>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Sébastien Denvil<o:p></o:p></pre>
<pre>IPSL, Pôle de modélisation du climat<o:p></o:p></pre>
<pre>UPMC, Case 101, 4 place Jussieu,<o:p></o:p></pre>
<pre>75252 Paris Cedex 5<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Tour 45-55 2ème étage Bureau 209<o:p></o:p></pre>
<pre>Tel: 33 1 44 27 21 10<o:p></o:p></pre>
<pre>Fax: 33 1 44 27 39 02<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>GO-ESSP-TECH mailing list<o:p></o:p></pre>
<pre><a href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><o:p></o:p></pre>
<pre><a href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a><o:p></o:p></pre>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Estanislao Gonzalez<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Max-Planck-Institut für Meteorologie (MPI-M)<o:p></o:p></pre>
<pre>Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre<o:p></o:p></pre>
<pre>Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Phone: +49 (40) 46 00 94-126<o:p></o:p></pre>
<pre>E-Mail: <a href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> <o:p></o:p></pre>
</div>
<br><p>--
<BR>Scanned by iCritical.
</p>
<br></body>
</html>