[Go-essp-tech] verification of datanode status

Tue Dec 20 01:36:47 MST 2011

Hello Karl,

Some test output from CORDEX was published to demonstrate the functionality and the way in which the additional attributes needed to describe CORDEX data (region, RCM, etc) are (or are not) propagated through the system - removing it from the BADC gateway is probably a task for Stephen, who is now on leave for Xmas. The data appears under "project=CORDEX" in our gateway. Real data from CORDEX has been imminent for some time. March/April now looks like a plausible time frame for first deliveries,

Cheers,
Martin

From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Karl Taylor
Sent: 20 December 2011 00:16
To: go-essp-tech at ucar.edu
Subject: Re: [Go-essp-tech] verification of datanode status

Hi All,

No matter what, it seems that published datasets that are not available should be "unpublished" (unless this is a temporary situation).  In the case under discussion, could someone tell me whom to contact at DMI to ask them to do this (or better yet, write them yourself, if that will work).

I'm curious ...  what model output looks like it is currently available from DMI but really isn't?

thanks,
Karl

On 12/19/11 2:02 PM, Eric Nienhouse wrote:

Hi All,

A long standing Gateway requirement is to provide search and discovery

of datasets (and other metadata) regardless of the state of remote

services.  In the event that a data node service is unavailable, users

should still be able to identify datasets, determine what has been

published and generate download scripts.

This guiding requirement was discussed at great length by the ESG-CET

project group and was accepted as a key element in support of the

community's best interest for data discovery.  Identifying "what has

been published" was a key use case driving this need.  This advantage of

this approach is that it allows users to find out "what exists" during

periods of unexpected downtime or other service unavailability.

Henrik noted that the DMI data node is no longer serving datasets

publishing into ESG.  In this case these datasets can be discovered at

the Gateway, however, they are inaccessible and out of sync.  If these

data are no longer meant to be accessed, I'd suggest they be "retracted"

from the gateway and they will no longer appear in the search results.

Thanks,

-Eric

Estanislao Gonzalez wrote:

T(sorry the message got cut)

...it's up to the publisher to define when data shouldn't be accessible

anymore.

There are some improvements that can be doen, but most I can think of

will make the understanding of the system more complex to the end user.

Datanode admin should rely on tools that help them get their nodes up

for as long as possible (nagios & Co).

My 2c,

Estani

On 19.12.2011 08:04, Estanislao Gonzalez wrote:

Hi Luca,

That's not what Henrik meant. Neither the architecture retains a

living

link to a data nose (not an index one, as you've pointed out)

This is a feature IMO as the search engine is detached from the

dataone. The index might indeed "prune" the data nodes down, but

unless

this is done synchroneusly it would difficult the federation

interaction.

Or to say it differently: Is up to the data node to assure data is

available, and if that's not desired anymore, On 19.12.2011 07:15,

Cinquini, Luca (3880) wrote:

Hi Henirik,

    not sure about the gateway, but this is a feature the P2P system

does have: has soon as a datanode is inaccessible, the search

automatically prunes that node away, so the search results never

contain dead links.

thanks, Luca

On Dec 19, 2011, at 3:58 AM, Henrik Wiberg wrote:

Does the gateway somehow 'ping' its registered datanodes to verify

that

they are accessible? The datanode at dmi has not been running for 5

mounts still the datanodes published datasets are searchable and

displayed at the gateway cmip-gw-badc. Should not inaccessible

datasets

be removed from the search result?

_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

-- 
Scanned by iCritical.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111220/77c76971/attachment-0001.html 