[Go-essp-tech] Upcoming gateway/datanode coordination points
Eric Nienhouse
ejn at ucar.edu
Tue Mar 29 14:09:37 MDT 2011
Hi Gavin and the Data Node Team,
Good to catch up on the GO-ESSP call today and thanks for the discussion
on the Federation Registry. I feel we need to further clarify the "who,
what and when" of the centralized federation registry service. Based on
the email discussions and today's call I understand the following:
* A centralized federation registry service is critical to
simplifying ESG federation management (fed gw, nodes, truststore,
etc.)
* Work is underway to develop an XML schema to describe the
federation gateways and data nodes.
* There is desire to keep the registry service simple and to support
production use as soon as possible (ideally for v. 1.3.0)
* PCMDI has proposed a federation registry solution based on the
data node manager and work is underway to develop a test service.
* The production federation registry service is planned to reside at
PCMDI.
* A temporary federation registry XML document is available for
1.3.0 federation testing at NCAR.
Please let me know if I've misrepresented any of the above. I realize
we're all busy and moving quickly on this.
As we've all discussed in email and on the call this morning, we as a
collaboration need more information about the federation registry
service under development.
* What is the scope of the registry service under development?
* When will it be available for testing?
* What is the general architecture of the federation registry system?
* The registry service will be Hessian initially (REST proposed
for future.) Please share the interface specification of the test
service.
* You noted tweaks to the federation registry XML schema - has it
changed? Please share the current form of the registry schema.
* What are the plans for deploying a production federation
registry service on pcmdi3 (time-line, basic service level)?
This information is particularly useful for coordinating ESG federation
software releases which have been difficult to align in the past. As
Phil, Rachana and others have noted, getting a simple solution in place
first, then later refining it, has benefited us in the past. We're
simply looking to learn more about the work underway, see how we can
help out and best collaborate on this federation piece. We greatly
appreciate you taking on this integration effort in support of the ESG
federation.
Regards,
-Eric
Bryan Lawrence wrote:
> Hi Gavin
>
> I'll provide a proper reply next week ...
>
> Meanwhile, the thing about parterships, especially distributed
> partnerships, is that even when things are "in progress" you have to
> tell folks what you are doing and why. Sometimes knowing that there
> *might be* something to report on is just as important as the eventual
> report itself & you have to remember that not everyone is on the call
> (any call).
>
> I know all this communication is an overhead, and a pain, but it's one
> of the necessary pieces of collaboration :-)
>
> Have a good weeknd!
>
> Cheers
> Bryan
>
>
>> Hi Bryan,
>>
>> Pardon my misunderstanding... I am a bit daft when it comes to
>> nuance, at least that is what my wife tells me :-).
>>
>> On 3/18/11 12:45 PM, Bryan Lawrence wrote:
>>
>>> Hi Gavin
>>>
>>> I think you are misunderstanding the thrust of my question, which
>>> is possibily fair enough, since I was being a wee bit provocative.
>>>
>> :
>> :-| (see above) :-)
>> :
>>
>>> I don't much care what the SLA level is, I just care that it is
>>> jointly understood, documented, and everyone knows what it is. It
>>> clearly isn't. At this stage I cam concerend that most of us have
>>> no visibility of what you are doing, what the dependencies are for
>>> the rest of us on what you are doing, what the interfaces are,
>>> and what the timetables are.
>>>
>> What I am doing is putting in code to generate the XML "registration"
>> document that describes a node and the services that it is running
>> and additional details that may be needed (this is the iterative
>> part that requires the initial coordination with those more
>> intimately involved with the actual component data needs) so that we
>> can get a clean XSD. I have a tweaked the initial version that
>> Neil, Rachana, Nate and myself came up with, hence we should touch
>> base again. Each node will stamp out this registration document and
>> put it in an agreed upon
>> web-reachable location. These files will be able to be fetched and
>> aggregated and thus also made available to all who ask - i.e. a
>> registry service. This is what we are very close to having. The
>> interface? - Simple the XSD and wget/curl... et.al. This is what
>> was discussed at the last meetings and in a few meetings prior with
>> a smaller group going through iterations on the xsd. The question
>> of ingestion of this data is something that is up to the actor
>> pulling this data.
>>
>>
>>> Given your response: I am particularly interested in where you have
>>> written detials of how the new architecture of the node will
>>> "ameliorate" if not eliminate the issue of availabity of
>>> federation metadata for the security.
>>>
>> The current plan is to have a central registry located at PCMDI...
>> essentially, the gateway functionality that Nate mentioned, on the
>> Tuesday call. The gateway now knows how to ingest and aggregate this
>> information. The last part is for the gateway to know how to fetch
>> this information, something that was not detailed in Nate's
>> description of the gateway but be imagined without much effort. The
>> gateway in question that all will look to for this information is
>> PCMDI3.
>>
>> Iterating on this... it is clear, as you alluded to in your original
>> query (SLAs and such), that having a central anything in a system is
>> a focal point and a single point of failure. As the "node people" I
>> am interested in addressing the issue of single point of failure by
>> distributing this registry responsibility.
>>
>> To the finer point you make about visibility. The question is not so
>> much of visibility but of things being in a state of flux. Nothing
>> is in secret, they are just *in progress* :-). So until there is
>> something to report, there's nothing to really report. It would be
>> great if everything were just done, but life and other things get
>> 'in the way'... so modulo that, we are all making progress everyday
>> and pushing to the goal of having a debut of the new and improved
>> node. However, this is not in the critical path.
>>
>> I hope you have a clearer idea of what is going on, and what is not
>> going on, and the collective path forward. Please let us know if
>> there is anything that was unclear.
>>
>> Thanks again. I really enjoy this collaboration and am hoping to
>> cousin some help from Phil, Stephen and your team on some of this
>> next iteration work.
>>
>> Oh, timelines...! This is always sticky... Let's just say the goal is
>> to have all things in play in the next couple weeks or so, modulo
>> Murphy's Law.... the nature of the beast.
>>
>>
>>> Thanks
>>> Bryan
>>>
>>>
>>>> Hi Bryan,
>>>>
>>>> As the other team members mentioned in this thread, availability
>>>> is not under such a stringent 'SLA' such that it will impede the
>>>> functioning of the federation. In addition the architecture of
>>>> the node will greatly ameliorate, if not eliminate the issue of
>>>> availability.
>>>>
>>>> Indeed Phil brought up a good point with the Attribute Service. I
>>>> would like to fix a time for us to all meet and discuss the issues
>>>> surrounding that service and availability.
>>>>
>>>> On 3/18/11 2:15 AM, Bryan Lawrence wrote:
>>>>
>>>>> Hi Gavin
>>>>>
>>>>> What will the API be to this registry?
>>>>> What service level with PCMDI provide for it (since it's going to
>>>>> have to work 24x7 globally)?
>>>>>
>>>>> Thanks
>>>>> Bryan
>>>>>
>>>>>
>>>>>> We will have a single registry, it should be served from PCMDI
>>>>>> on pcmdi3. The esgf nodes will all be coded to set pcmdi3 as
>>>>>> the default location for this registry. Indeed coordination
>>>>>> with Nate and Neill and Rachana will have to take place. I
>>>>>> suspect that early next week will be that time. Stay tuned,
>>>>>> we'll make this happen. ;-)
>>>>>>
>>>>>> On 3/17/11 8:32 AM, philip.kershaw at stfc.ac.uk wrote:
>>>>>>
>>>>>>> So long as we are co-ordinated on this. Will we have a single
>>>>>>> registry for the federation or many? I'm opening that question
>>>>>>> more widely to all …
>>>>>>>
>>>>> --
>>>>> Bryan Lawrence
>>>>> Director of Environmental Archival and Associated Research
>>>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>>>> STFC, Rutherford Appleton Laboratory
>>>>> Phone +44 1235 445012; Fax ... 5848;
>>>>> Web: home.badc.rl.ac.uk/lawrence
>>>>>
>>> --
>>> Bryan Lawrence
>>> Director of Environmental Archival and Associated Research
>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>> STFC, Rutherford Appleton Laboratory
>>> Phone +44 1235 445012; Fax ... 5848;
>>> Web: home.badc.rl.ac.uk/lawrence
>>>
>
> --
> Bryan Lawrence
> Director of Environmental Archival and Associated Research
> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
> STFC, Rutherford Appleton Laboratory
> Phone +44 1235 445012; Fax ... 5848;
> Web: home.badc.rl.ac.uk/lawrence
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
More information about the GO-ESSP-TECH
mailing list