<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
I've wrote a long email about this and then realized I've
misunderstood you Stephen. You weren't talking about replication at
all...<br>
<br>
But you have a nice point there. As QC checking entities we have no
access to the publishing data nodes (at least to some) so if any
file fails the check, the only thing we are able to do is to retract
them from the Gateway (We will contact the publishers and beg them
to republish the data with a "failed" flag which they will be
reluctant to do, as it will be better for them just to wait and
publish the "new" version).<br>
<br>
But this procedure is not standardized and I'm not sure it's easy to
accomplish (The normal case it's that the "publisher" retract the
dataset, from outside it would be really hard to achieve and
probably something we would like to avoid anyways)<br>
<br>
So we might run into problems... it is indeed simpler to keep things
completely separated. I retract what I said about QC flags, don't
put them into the catalogs :-)<br>
<br>
Thanks,<br>
Estani<br>
<br>
<br>
Am 24.05.2011 12:01, schrieb <a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk:">stephen.pascoe@stfc.ac.uk:</a>
<blockquote
cite="mid:4C353E6E4A08AE4792B350DAA392B521198B72@EXCHMBX01.fed.cclrc.ac.uk"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 12 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);">IMHO CIM is a red herring. Why would we want to
put a CIM simulation record (a substantial chunk of XML)
into every single THREDDS catalog? That would be extreme
database denormalisation.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);">QC data could be more feasible. However, what if
data from datanode A is replicated to DKRZ, they do QC L2
and find it doesn't pass. How is this communicated? Do we
ask datanode A to republish saying their data is bad? Will
they do it? Even without this human problem keeping QC
information consistent across replicas will become a
headache.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);">S.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt;
font-family: Consolas; color: rgb(31, 73, 125);">---<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt;
font-family: Consolas; color: rgb(31, 73, 125);">Stephen
Pascoe +44 (0)1235 445980<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt;
font-family: Consolas; color: rgb(31, 73, 125);">Centre of
Environmental Data Archival<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt;
font-family: Consolas; color: rgb(31, 73, 125);">STFC
Rutherford Appleton Laboratory, Harwell Oxford, Didcot
OX11 0QX, UK<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);"><o:p> </o:p></span></p>
<div>
<div style="border-right: medium none; border-width: 1pt
medium medium; border-style: solid none none; border-color:
rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color;
padding: 3pt 0cm 0cm;">
<p class="MsoNormal"><b><span style="font-size: 10pt;
font-family:
"Tahoma","sans-serif"; color:
windowtext;" lang="EN-US">From:</span></b><span
style="font-size: 10pt; font-family:
"Tahoma","sans-serif"; color:
windowtext;" lang="EN-US"> Estanislao Gonzalez
[<a class="moz-txt-link-freetext" href="mailto:gonzalez@dkrz.de">mailto:gonzalez@dkrz.de</a>]
<br>
<b>Sent:</b> 24 May 2011 08:59<br>
<b>To:</b> Gavin M. Bell<br>
<b>Cc:</b> Pascoe, Stephen (STFC,RAL,RALSP);
<a class="moz-txt-link-abbreviated" href="mailto:Luca.Cinquini@jpl.nasa.gov">Luca.Cinquini@jpl.nasa.gov</a>; <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><br>
<b>Subject:</b> Re: [Go-essp-tech] resolution on
securing opendap aggregations via ESGF<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi,<br>
<br>
just to be more precise: indeed I think security and CIM data
(meta-data from experiments and so on) should be kept away
from the catalogs.<br>
<br>
Still, Luca's idea of merging QC level info with the files
itself might be a valid idea. The difference is that QC is
pretty much like a "semantic" checksum for a file. AFAIK you
cannot "downgrade" the QC without altering the file, i.e. if
the file is QC L2 and while performing QC L3 checks an error
is found, the file will either be QC L3 approved (the modeler
defines the "oddity" as "expected") or it doesn't, which
implies a "QC L2 passed; QC L3 failed" flag or the retraction
(and maybe re-publication) of it altogether.<br>
<br>
Well, that's at least why I think the QC flag is a little
different and it's *closely* related to the file. The only
difference with the checksum is IMHO that it takes more time
to be determined (as well as require other files for it's
computation) and thus it's performed in an out-of-band
fashion.<br>
<br>
We need that QC flag somewhere... and it's far more important
than the rest of the CIM meta-data (getting back to Gavin's
point about CIM issues and differentiating it to this QC flag:
yes, you can still get to the file and download it without CIM
data... but without the QC flag you'll have no clue if you
*really* want to rely on this data).<br>
<br>
To be honest I can't understand why people download this data
if they *know* it might get corrected. Would you start writing
a paper on something that might be altogether wrong?... I
suspect they don't realize this.<br>
<br>
Anyway, my 2c...<br>
<br>
Thanks,<br>
Estani<br>
<br>
Am 24.05.2011 03:06, schrieb Gavin M. Bell: <o:p></o:p></p>
<p class="MsoNormal">Hi Luca, <br>
<br>
I think that the separation of concerns trumps the apparent
"simplicity". Though it is apparently easy to republish (I am
not sure I fully agree with that, at least not from the
anecdotal information I hear from folks)... it is unnecessary
to publish if we keep concerns separated.<br>
<br>
As Estani said, the publisher publishes and does basic
mechanical sanity checks on data. That should be the full
extent of its operation. As far as what is easy... one could
'easily' set up an index over the CIM info and "join" on
datasetid. This also provides loose coupling. If the CIM
system has issues, that just means that when you look at your
search results you won't see CIM info, but you will still see
the dataset and be able to fetch and manipulate it and
everything else. Also if the CIM changes it doesn't affect
the pubblisher or publishing in any way. Catalogs should be
viewed as "files" in the system... they essentially are
logical files (containing pointers to physical files).<br>
<br>
I am still not convinced by your arguments that fusing and
coupling these two semantically different aspects of the
system so tightly is the right long term architectural
solution. It may be good now, but it not as flexible later.
We should leave open the avenue for other meta-metadata to be
imbued onto our system ex-post-facto without much ado.<br>
<br>
my $0.02<br>
<br>
On 5/23/11 2:08 AM, <a moz-do-not-send="true"
href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>
wrote:
<o:p></o:p></p>
<pre>I'm with Estani on this. Authorisation decisions are best decoupled from the application where possible. Phil is on leave today but I'm sure he'd say the same thing and give much more detailed reasoning. <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I think the catalogue already mixes slightly too much information together: location-independent file metadata and location-specific service information. If we add access control it becomes too tightly coupled.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Stephen.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>---<o:p></o:p></pre>
<pre>Stephen Pascoe +44 (0)1235 445980<o:p></o:p></pre>
<pre>Centre of Environmental Data Archival<o:p></o:p></pre>
<pre>STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>-----Original Message-----<o:p></o:p></pre>
<pre>From: <a moz-do-not-send="true" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a moz-do-not-send="true" href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Estanislao Gonzalez<o:p></o:p></pre>
<pre>Sent: 21 May 2011 09:30<o:p></o:p></pre>
<pre>To: Cinquini, Luca (3880)<o:p></o:p></pre>
<pre>Cc: <a moz-do-not-send="true" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><o:p></o:p></pre>
<pre>Subject: Re: [Go-essp-tech] resolution on securing opendap aggregations via ESGF<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Hi,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>In my opinion we shouldn't encode the access restriction in the catalog <o:p></o:p></pre>
<pre>for these reasons:<o:p></o:p></pre>
<pre>1) Changing the access would involved re-publishing the files. (this <o:p></o:p></pre>
<pre>will be done for instance when QC L2 is reached CMIP5 Research -> CMIP5 <o:p></o:p></pre>
<pre>Commercial). And think about what would happen if we want to change the <o:p></o:p></pre>
<pre>access restriction in a couple of years... we should publish everything <o:p></o:p></pre>
<pre>again, and that would involve quite some effort to understand the <o:p></o:p></pre>
<pre>procedure again...<o:p></o:p></pre>
<pre>2) I'm not sure of this, but I fear TDS security cannot handle multiple <o:p></o:p></pre>
<pre>roles. Right now you can publish to as many roles as required, and read <o:p></o:p></pre>
<pre>and write access is kept separately. This would involve extending the <o:p></o:p></pre>
<pre>TDS capabilities.<o:p></o:p></pre>
<pre>3) There could be potential inconsistencies if the authorization service <o:p></o:p></pre>
<pre>is detached from datanode (like with the gateway right now) and the <o:p></o:p></pre>
<pre>publisher alters the role but forgets to cascade the changes to the <o:p></o:p></pre>
<pre>authorizing service (which would proceed according to the last harvested <o:p></o:p></pre>
<pre>info)<o:p></o:p></pre>
<pre>4) And last but not least, I'm not sure we want to mix administration <o:p></o:p></pre>
<pre>with publication. The publisher should only care about making data <o:p></o:p></pre>
<pre>available, the administrator should organize this and be responsible for <o:p></o:p></pre>
<pre>the security.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>So basically I don't agree :-) Although I do think, if required, we <o:p></o:p></pre>
<pre>could change "esg-user" for "esgf-controlled" if it's more intuitive.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>My 2c anyways,<o:p></o:p></pre>
<pre>Estani<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Am 20.05.2011 19:17, schrieb Cinquini, Luca (3880):<o:p></o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>Hi,<o:p></o:p></pre>
<pre> a few points again on the issue of securing opendap aggregations served by the TDS with ESGF filters:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>o There's a new release of the ESGF security filters (esg-orp 1.1.2) that maps the TDS request URI to the dataset ID, and should solve this problem. You can experiment with the JPL test TDS server:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><a moz-do-not-send="true" href="http://test-datanode.jpl.nasa.gov/thredds/catalog.html">http://test-datanode.jpl.nasa.gov/thredds/catalog.html</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>where the AIRS dataset (and aggregations) is secured, the MLS is not.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>o Now the data node authorization filter will correctly identify the aggregation as secured, and call the configured authorization service. Currently, the p2p Node authorization service can be configured to allow authorization based on URL matching, so it will work. The gateway authorization service will have to implement its own logic to establish authorization.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>o Finally, I am wondering if we shouldn't change the way we encode authorization in thredds catalogs. Right now, we use restrictAccess="esg-user" for ALL collections, but should we consider about encoding the proper required access control attribute instead, for example restrictAccess="CMIP5 Research" ? Something to think about - there are prons and cons about this - it's all a question on wether the access control belongs in the catalog (and can be harvested for searching...) or not.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>thanks, Luca<o:p></o:p></pre>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>GO-ESSP-TECH mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a><o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Gavin M. Bell<o:p></o:p></pre>
<pre>--<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> "Never mistake a clear view for a short distance."<o:p></o:p></pre>
<pre> -Paul Saffo<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Estanislao Gonzalez<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Max-Planck-Institut für Meteorologie (MPI-M)<o:p></o:p></pre>
<pre>Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre<o:p></o:p></pre>
<pre>Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Phone: +49 (40) 46 00 94-126<o:p></o:p></pre>
<pre>E-Mail: <a moz-do-not-send="true" href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> <o:p></o:p></pre>
</div>
<br>
<p>-- <br>
Scanned by iCritical.
</p>
<br>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> </pre>
</body>
</html>