[Go-essp-tech] Upcoming gateway/datanode coordination points
Gavin M. Bell
gavin at llnl.gov
Fri Apr 1 09:38:58 MDT 2011
Almost done with it.
On 4/1/11 7:02 AM, philip.kershaw at stfc.ac.uk wrote:
> Hi Gavin,
>
> Any news on a finalised schema?
>
> Cheers,
> Phil
>
> On 29/03/2011 21:09, "Eric Nienhouse" <ejn at ucar.edu> wrote:
>
>> Hi Gavin and the Data Node Team,
>>
>> Good to catch up on the GO-ESSP call today and thanks for the discussion
>> on the Federation Registry. I feel we need to further clarify the "who,
>> what and when" of the centralized federation registry service. Based on
>> the email discussions and today's call I understand the following:
>>
>> * A centralized federation registry service is critical to
>> simplifying ESG federation management (fed gw, nodes, truststore,
>> etc.)
>> * Work is underway to develop an XML schema to describe the
>> federation gateways and data nodes.
>> * There is desire to keep the registry service simple and to support
>> production use as soon as possible (ideally for v. 1.3.0)
>> * PCMDI has proposed a federation registry solution based on the
>> data node manager and work is underway to develop a test service.
>> * The production federation registry service is planned to reside at
>> PCMDI.
>> * A temporary federation registry XML document is available for
>> 1.3.0 federation testing at NCAR.
>>
>> Please let me know if I've misrepresented any of the above. I realize
>> we're all busy and moving quickly on this.
>>
>> As we've all discussed in email and on the call this morning, we as a
>> collaboration need more information about the federation registry
>> service under development.
>>
>> * What is the scope of the registry service under development?
>> * When will it be available for testing?
>> * What is the general architecture of the federation registry
>> system?
>> * The registry service will be Hessian initially (REST proposed
>> for future.) Please share the interface specification of the test
>> service.
>> * You noted tweaks to the federation registry XML schema - has it
>> changed? Please share the current form of the registry schema.
>> * What are the plans for deploying a production federation
>> registry service on pcmdi3 (time-line, basic service level)?
>>
>> This information is particularly useful for coordinating ESG federation
>> software releases which have been difficult to align in the past. As
>> Phil, Rachana and others have noted, getting a simple solution in place
>> first, then later refining it, has benefited us in the past. We're
>> simply looking to learn more about the work underway, see how we can
>> help out and best collaborate on this federation piece. We greatly
>> appreciate you taking on this integration effort in support of the ESG
>> federation.
>>
>> Regards,
>>
>> -Eric
>>
>> Bryan Lawrence wrote:
>>> Hi Gavin
>>>
>>> I'll provide a proper reply next week ...
>>>
>>> Meanwhile, the thing about parterships, especially distributed
>>> partnerships, is that even when things are "in progress" you have to
>>> tell folks what you are doing and why. Sometimes knowing that there
>>> *might be* something to report on is just as important as the eventual
>>> report itself & you have to remember that not everyone is on the call
>>> (any call).
>>>
>>> I know all this communication is an overhead, and a pain, but it's one
>>> of the necessary pieces of collaboration :-)
>>>
>>> Have a good weeknd!
>>>
>>> Cheers
>>> Bryan
>>>
>>>
>>>> Hi Bryan,
>>>>
>>>> Pardon my misunderstanding... I am a bit daft when it comes to
>>>> nuance, at least that is what my wife tells me :-).
>>>>
>>>> On 3/18/11 12:45 PM, Bryan Lawrence wrote:
>>>>
>>>>> Hi Gavin
>>>>>
>>>>> I think you are misunderstanding the thrust of my question, which
>>>>> is possibily fair enough, since I was being a wee bit provocative.
>>>>>
>>>> :
>>>> :-| (see above) :-)
>>>> :
>>>>
>>>>> I don't much care what the SLA level is, I just care that it is
>>>>> jointly understood, documented, and everyone knows what it is. It
>>>>> clearly isn't. At this stage I cam concerend that most of us have
>>>>> no visibility of what you are doing, what the dependencies are for
>>>>> the rest of us on what you are doing, what the interfaces are,
>>>>> and what the timetables are.
>>>>>
>>>> What I am doing is putting in code to generate the XML "registration"
>>>> document that describes a node and the services that it is running
>>>> and additional details that may be needed (this is the iterative
>>>> part that requires the initial coordination with those more
>>>> intimately involved with the actual component data needs) so that we
>>>> can get a clean XSD. I have a tweaked the initial version that
>>>> Neil, Rachana, Nate and myself came up with, hence we should touch
>>>> base again. Each node will stamp out this registration document and
>>>> put it in an agreed upon
>>>> web-reachable location. These files will be able to be fetched and
>>>> aggregated and thus also made available to all who ask - i.e. a
>>>> registry service. This is what we are very close to having. The
>>>> interface? - Simple the XSD and wget/curl... et.al. This is what
>>>> was discussed at the last meetings and in a few meetings prior with
>>>> a smaller group going through iterations on the xsd. The question
>>>> of ingestion of this data is something that is up to the actor
>>>> pulling this data.
>>>>
>>>>
>>>>> Given your response: I am particularly interested in where you have
>>>>> written detials of how the new architecture of the node will
>>>>> "ameliorate" if not eliminate the issue of availabity of
>>>>> federation metadata for the security.
>>>>>
>>>> The current plan is to have a central registry located at PCMDI...
>>>> essentially, the gateway functionality that Nate mentioned, on the
>>>> Tuesday call. The gateway now knows how to ingest and aggregate this
>>>> information. The last part is for the gateway to know how to fetch
>>>> this information, something that was not detailed in Nate's
>>>> description of the gateway but be imagined without much effort. The
>>>> gateway in question that all will look to for this information is
>>>> PCMDI3.
>>>>
>>>> Iterating on this... it is clear, as you alluded to in your original
>>>> query (SLAs and such), that having a central anything in a system is
>>>> a focal point and a single point of failure. As the "node people" I
>>>> am interested in addressing the issue of single point of failure by
>>>> distributing this registry responsibility.
>>>>
>>>> To the finer point you make about visibility. The question is not so
>>>> much of visibility but of things being in a state of flux. Nothing
>>>> is in secret, they are just *in progress* :-). So until there is
>>>> something to report, there's nothing to really report. It would be
>>>> great if everything were just done, but life and other things get
>>>> 'in the way'... so modulo that, we are all making progress everyday
>>>> and pushing to the goal of having a debut of the new and improved
>>>> node. However, this is not in the critical path.
>>>>
>>>> I hope you have a clearer idea of what is going on, and what is not
>>>> going on, and the collective path forward. Please let us know if
>>>> there is anything that was unclear.
>>>>
>>>> Thanks again. I really enjoy this collaboration and am hoping to
>>>> cousin some help from Phil, Stephen and your team on some of this
>>>> next iteration work.
>>>>
>>>> Oh, timelines...! This is always sticky... Let's just say the goal is
>>>> to have all things in play in the next couple weeks or so, modulo
>>>> Murphy's Law.... the nature of the beast.
>>>>
>>>>
>>>>> Thanks
>>>>> Bryan
>>>>>
>>>>>
>>>>>> Hi Bryan,
>>>>>>
>>>>>> As the other team members mentioned in this thread, availability
>>>>>> is not under such a stringent 'SLA' such that it will impede the
>>>>>> functioning of the federation. In addition the architecture of
>>>>>> the node will greatly ameliorate, if not eliminate the issue of
>>>>>> availability.
>>>>>>
>>>>>> Indeed Phil brought up a good point with the Attribute Service. I
>>>>>> would like to fix a time for us to all meet and discuss the issues
>>>>>> surrounding that service and availability.
>>>>>>
>>>>>> On 3/18/11 2:15 AM, Bryan Lawrence wrote:
>>>>>>
>>>>>>> Hi Gavin
>>>>>>>
>>>>>>> What will the API be to this registry?
>>>>>>> What service level with PCMDI provide for it (since it's going to
>>>>>>> have to work 24x7 globally)?
>>>>>>>
>>>>>>> Thanks
>>>>>>> Bryan
>>>>>>>
>>>>>>>
>>>>>>>> We will have a single registry, it should be served from PCMDI
>>>>>>>> on pcmdi3. The esgf nodes will all be coded to set pcmdi3 as
>>>>>>>> the default location for this registry. Indeed coordination
>>>>>>>> with Nate and Neill and Rachana will have to take place. I
>>>>>>>> suspect that early next week will be that time. Stay tuned,
>>>>>>>> we'll make this happen. ;-)
>>>>>>>>
>>>>>>>> On 3/17/11 8:32 AM, philip.kershaw at stfc.ac.uk wrote:
>>>>>>>>
>>>>>>>>> So long as we are co-ordinated on this. Will we have a single
>>>>>>>>> registry for the federation or many? I'm opening that question
>>>>>>>>> more widely to all Š
>>>>>>>>>
>>>>>>> --
>>>>>>> Bryan Lawrence
>>>>>>> Director of Environmental Archival and Associated Research
>>>>>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>>>>>> STFC, Rutherford Appleton Laboratory
>>>>>>> Phone +44 1235 445012; Fax ... 5848;
>>>>>>> Web: home.badc.rl.ac.uk/lawrence
>>>>>>>
>>>>> --
>>>>> Bryan Lawrence
>>>>> Director of Environmental Archival and Associated Research
>>>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>>>> STFC, Rutherford Appleton Laboratory
>>>>> Phone +44 1235 445012; Fax ... 5848;
>>>>> Web: home.badc.rl.ac.uk/lawrence
>>>>>
>>> --
>>> Bryan Lawrence
>>> Director of Environmental Archival and Associated Research
>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>> STFC, Rutherford Appleton Laboratory
>>> Phone +44 1235 445012; Fax ... 5848;
>>> Web: home.badc.rl.ac.uk/lawrence
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
--
Gavin M. Bell
--
"Never mistake a clear view for a short distance."
-Paul Saffo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110401/7debbf30/attachment-0001.html
More information about the GO-ESSP-TECH
mailing list