<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffcc">
Hi Craig, <br>
<br>
Indeed, as long as said catfish gets skinned in the end :-).<br>
I am going to investigate another way. If my investigation bears
fruit we can trade notes and put this thing to bed.<br>
<br>
<br>
On 5/24/11 8:56 AM, Craig E. Ward wrote:
<blockquote cite="mid:4DDBD536.1050701@ISI.EDU" type="cite">
<pre wrap="">As the saying goes, there's more than one way to skin a catfish. From my
perspective, there are issues with the way replication is integrated into ESG,
but I don't think we can solve all of them all at once. The proposed
gateway-centric solution solves the immediate problem without limiting changes
at a future stage of the project.
Craig
On 5/23/11 6:09 PM, Gavin M. Bell wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Craig,
I am not thinking about the gateway at all. I am thinking about
limiting the number of actors in this transaction. By all means press
on. I think there is another way to accomplish this task.
On 5/23/11 11:50 AM, Craig E. Ward wrote:
</pre>
<blockquote type="cite">
<pre wrap="">This doesn't actually address the issue at hand. The problem is distinguishing
the different flavors of GridFTP service in the gateway meta data. As the meta
data from the gateway is already being retrieved and processed, it is simpler
to modify how it is processed than to add yet another retrieval and additional
processing.
As I understand the meta data, it contains two elements that allow clients to
create URLs for files. One element is called "data_access_capability" and
contains attributes naming the service, identifying the type of service, and
specifying a base URI. The other element is "file_access_point," which contains
an attribute "data_access_capability" with the name of the service the URI
string is applicable to. The "file_access_point" element is a child of the
"file" element and "data_access_capability" is a child of "dataset."
Ideally, the "type" attribute of the "data_access_capability" element could be
set to indicate the type of GridFTP service, i.e. "normal" or "ESG/BDM." In
another thread, Bob Drach may have written that the values for "type" are
restricted. We need this clarified.
The "name" attribute for the element may be more free form and we could use
that. This name may come from the value assigned to "thredds_file_services" in
the esg.ini file. If this is correct, the naming convention could be
standardized administratively and not require code changes outside of the
replication client itself.
One way or another, a choice needs to be made.
The most expeditious route to a solution is to make use of either the type
attribute or name attribute of the data_access_capability element. Trying to
take other routes will require significant redesign and reimplementation and
take more time than I think this project has for this stage.
Craig
p.s. I removed the list <a class="moz-txt-link-rfc2396E" href="mailto:esgf-mechanic@lists.llnl.gov">"esgf-mechanic@lists.llnl.gov"</a> because I am not a
subscriber to it. I assume that everyone that needs to know about this is on
the "go-essp-tech" list.
On 5/17/11 7:39 PM, Gavin M. Bell wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Craig,
Perhaps I can help elucidate some things. This is at least how I see
things moving forward.
The key thing is that you need information about nodes and what they
have running on them, specifically GridFTP and what configuration is
supported.
In the current ESGF P2P Node there is a registry service that comes with
every node and is present in all configurations. The registry's
external view is an XML file that adheres to an xsd that Rachana, Neill,
Philip, Eric and myself put together. This xml file will contain
essentially a manifest of all nodes participating in the P2P federation
including what services are running on them. For services like the
gateway, that currently doesn't run directly in the P2P architecture,
any peer can be contacted for this information. For example PCMDI3 will
be a peer that can be contacted for inspection of the registry, but
PCMDI3 is no more special that any other node that is present in the P2P
federation - so by convention instead of by construction PCMDI3 is a
name we all know and love and will be albe to go to. So, you may
inspect the registration information on the node you are running the
replication client on for the information you need i.e. gridftp nodes,
their configuration and ports and such.
I have attached a quick registration.xml document from a test node, it
only contains two node entries at the moment, but that should be enough
to illustrate that there may be many more nodes represented in the same
fashion. Note the context that you should read this in is that each
node is advertising the services it has running. It is up the the
"reader" to glean from this collection other derivative information,
like, 'which nodes are running gridftp services in a particular
configuration', 'which what nodes are pointing to a particular IDP', etc.
I think this solves your questions, essentially get the info straight
from the horses mouth (aka the registry running on the nodes where the
data lives). Does this help? Let me know if there are bits of
information that you don't find accommodated so we can have them present
if needed. :-) Between the registry and the search api you should be
golden. I know you have recently installed the P2P Node, so you may
already have the manager in place. The version is v0.5.0 (check
/etc/esg.install_log and look for the esgf-node-manager entry).
P.S.
I would suggest that folks start upgrading their node installations. We
are doing a bit of cleanup and spit and polish on our respective ESGF
P2P Node components, but what is posted right now, is ready - modulo
spit and/or polish. :-). I am the last man standing with some
installation cleanup, however, by end of next week we will "release the
hounds"! The intrepid may go at it now. ...(Hmmm.. that reminds me... we
have a lot of documentation to post as well.)
On 5/17/11 3:23 PM, Craig E. Ward wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On the GO-ESSP call today, we bounced around some ideas about how replication
could distinguish between the different types of GridFTP servers that the ESG
Federation will have. This is what I heard. I'd like to make sure we're all on
the same page.
A naming convention will be applied to the a relevant property (Which
property?) that will mark a particular service as either "normal" GridFTP or
the ESG-security-specific (i.e. BDM) GridFTP. The replication client will use
an appropriate name when selecting services for the user's preferred data
movement agent.
This also requires a new configuration option for the replication client that
allows the user to control what the which service to use in the transfer
control file.
In the meta data XML, the replication client is looking at the name stored for
"data_access_capability." The default value to match is "GridFTP." It isn't
clear to me where this value is coming from, but the "serviceType" attribute of
the "service" element in the TDS catalog is set to "GridFTP" for that type of
protocol. Is the gateway placing the "serviceType" value into the
"data_access_capability" attribute?
In another thread, Bob wrote about restrictions on what names were allowed at
certain points. If so, this could complicate the issue, but not prevent this
solution from working.
Who remembers things differently?
Thanks,
Craig
</pre>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Gavin M. Bell
--
"Never mistake a clear view for a short distance."
-Paul Saffo
</pre>
</body>
</html>