[Go-essp-tech] [esg-gateway-dev] [esg-node-dev] Re: Upcoming gateway/datanode coordination points
Nathan Wilhelmi
wilhelmi at ucar.edu
Thu Mar 24 12:39:52 MDT 2011
Hi,
One the biggest issues we have right now is the requirement that the
datanode be on a dedicated machine. Right now we don't have a machine
that meets that criteria. In the future a virtualization solution may
be an option, but that is a ways off.
A related issue we have is TDS performance. Our current TDS is not
holding up well with our CMIP3 data, we don't think it will handle any
significant additions. One plan that was previously discussed was to
standup additional TDS instances to load balance, while keeping a single
instance of the publisher and the related user accounts and
configuration. This would allow us to scale with out current
infrastructure configuration. However is doesn't appear the data node is
headed in a direction to support this approach?
Treating the datanode as a black box also raises some configuration
questions as well.
* What happens to any custom configurations. Say we plug in something
other than cdat or ferret behind LAS, or change the LAS configuration.
Will that be preserved between updates or will we have to reconfigure
for each upgrade?
* We will need external database support as well as noted on earlier
e-mails.
* Short of reverse engineering the script how do we find out what it
does, installs, etc. The use case here is our system folks come to us
and ask what is this data node running. We aren't sure isn't a
reassuring answer that may raise a lot of questions. One sticking point
we may have here is manually installing packages and by-passing the
package management system (where available).
* Is the datanode process going to be more transparent? There doesn't
seem to be much information on what is being worked on, what version
changes are, what versions are meant for production vs. testing etc.
Thanks!
-Nate
Don Middleton wrote:
> I didn't see any more follow up on this topic - was there any? I couldn't find an initial response, either... As Nate indicated, we still don't have the ability to stand up a metrics service which means we're pretty far away from being able to complete development and work through testing. It doesn't look like there's any way to get this into the 1.3.0 release, and will have to get shelved for a later release. Nate, re your Friday email, it would be good to get more details back on this list on why the node installation script approach doesn't help us here.
>
> don
>
>
> On Mar 18, 2011, at 1:38 PM, Nathan Wilhelmi wrote:
>
>
>> Hi Gavin,
>>
>> Right now using the script isn't an option for us. The question is how can we stand up this service without the script?
>>
>> Thanks!
>>
>> -Nate
>>
>> Gavin M. Bell wrote:
>>
>>> Hi Don,
>>>
>>> I responded that they would need to install the node using the script to have a working model of the node. The installation would install will install and setup the key pieces on the node side of the equation, namely the access logging table and the filter in front of the TDS. As for the ability to fetch the access log data, I have provided the complete client side tool for making this happen. Please let me know specifically what further assistance is needed, as far as I know everything is a go.
>>>
>>> On 3/18/11 11:25 AM, Don Middleton wrote:
>>>
>>>> Thanks for the response on the registry topic, Gavin. Can you also respond to Nate's metrics question at the beginning of the thread? There has been a desire and intent to get metrics support into 1.3, but we're just about out of time on it. There needs to be a viable service up such that development and integration testing can be done. Otherwise, this may have to be pushed out to a later release.
>>>>
>>>> thanks - don
>>>>
>>>>
>>>> On Mar 18, 2011, at 11:53 AM, Gavin M. Bell wrote:
>>>>
>>>>
>>>>> Hello Team,
>>>>>
>>>>> So indeed you guys are all correct, thanks for responding to Bryan giving him a good context in which to think about the registry service. Nathan, indeed as you indicated there are points of integration that we need to touch base on. I think next week would be a good time to do so. It would be good to get on the same page with the xsd. I am not sure which version you are using and I discovered a few more modifications that had to be made, which also simplifies things as well. We have to make sure we capture all the information and agree on the xsd organization.
>>>>>
>>>>> Indeed the registration information will be able to be easily aggregated and disseminated and as you and Estani mentioned, cached.
>>>>>
>>>>> On 3/18/11 7:13 AM, Nathan Wilhelmi wrote:
>>>>>
>>>>>> Hi Estani,
>>>>>>
>>>>>> The caching you describe is implemented in the gateway now. All the registry information is persisted in the database and the code that synchronizes with the registry is decoupled from the parts that use the registry information. As such if worst came to worst one could directly modify the database if a change was needed and the service is down.
>>>>>>
>>>>>> -Nate
>>>>>>
>>>>>> On 03/18/2011 04:21 AM, Estanislao Gonzalez wrote:
>>>>>>
>>>>>>
>>>>>>> Hi Bryan,
>>>>>>>
>>>>>>> I don't think it is important to be available 24x7. I hope (probably is
>>>>>>> not developed, but will be pretty simple to do) that the Gateway caches
>>>>>>> the last registry document found, so that it can be still restarted
>>>>>>> without problem if precisely at that moment PCMDI's central registry is
>>>>>>> not available.
>>>>>>>
>>>>>>> The worst case scenario would be if someone wants to install a Gateway
>>>>>>> anew and at precisely that moment the registry is down. this could be
>>>>>>> solved by packing a "pre-cached" version of the registry with it (not
>>>>>>> that I think this will be that important, as adding this new Gateway to
>>>>>>> the federation will take time).
>>>>>>>
>>>>>>> So while the server is down, no new Gateway can be added to the
>>>>>>> federation, nor any other changes be done. But the federation itself
>>>>>>> should work without a hitch.
>>>>>>>
>>>>>>> But perhaps I'm missing something here...
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Estani
>>>>>>>
>>>>>>>
>>>>>>> Am 18.03.2011 10:15, schrieb Bryan Lawrence:
>>>>>>>
>>>>>>>
>>>>>>>> Hi Gavin
>>>>>>>>
>>>>>>>> What will the API be to this registry?
>>>>>>>> What service level with PCMDI provide for it (since it's going to have
>>>>>>>> to work 24x7 globally)?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Bryan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> We will have a single registry, it should be served from PCMDI on
>>>>>>>>> pcmdi3. The esgf nodes will all be coded to set pcmdi3 as the
>>>>>>>>> default location for this registry. Indeed coordination with Nate
>>>>>>>>> and Neill and Rachana will have to take place. I suspect that early
>>>>>>>>> next week will be that time. Stay tuned, we'll make this happen.
>>>>>>>>> ;-)
>>>>>>>>>
>>>>>>>>> On 3/17/11 8:32 AM, philip.kershaw at stfc.ac.uk wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> So long as we are co-ordinated on this. Will we have a single
>>>>>>>>>> registry for the federation or many? I'm opening that question
>>>>>>>>>> more widely to all …
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> --
>>>>>>>> Bryan Lawrence
>>>>>>>> Director of Environmental Archival and Associated Research
>>>>>>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>>>>>>> STFC, Rutherford Appleton Laboratory
>>>>>>>> Phone +44 1235 445012; Fax ... 5848;
>>>>>>>> Web: home.badc.rl.ac.uk/lawrence
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> esg-gateway-dev mailing list
>>>>>> esg-gateway-dev at mailman.earthsystemgrid.org
>>>>>> http://mailman.earthsystemgrid.org/mailman/listinfo/esg-gateway-dev
>>>>>>
>>>>>>
>>>>> --
>>>>> Gavin M. Bell
>>>>> --
>>>>>
>>>>> "Never mistake a clear view for a short distance."
>>>>> -Paul Saffo
>>>>>
>>>>> _______________________________________________
>>>>> GO-ESSP-TECH mailing list
>>>>> GO-ESSP-TECH at ucar.edu <mailto:GO-ESSP-TECH at ucar.edu>
>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>
>>> --
>>> Gavin M. Bell
>>> --
>>>
>>> "Never mistake a clear view for a short distance."
>>> -Paul Saffo
>>>
>>>
>>>
>
>
>
More information about the GO-ESSP-TECH
mailing list