[Go-essp-tech] Status of the CMIP5 Archive

[Mark Morgan]

Bob / Bryan

Ontop of the possibilty of the DataNode exposing ESG-Publisher AtomPub HTTP endpoints, may I also add to the mix the possibility of ESG-Publisher search web services.  Such web services would in turn permit the development of a ESG distributed search broker, i.e. an aggregator of search results pulled from multiple data nodes.  

At IPSL we have developed a portal that launches an overnight batch job to harvest & aggregate meta-data derived from the THREDDS catalogs published at each of our data nodes.  We could certainly use the ESG-Publisher AtomPub feeds to optimise the synchronisation of the aggregated meta-data.

However ultimately all paths lead to distributed search as in a few years we will have several millions of files/variables to search against and there are limits to what can be acheived with a solution based upon aggregation.  Hence the ESG-Publisher search web services and associated distributed search broker are of real interest.

Regards

Mark


On 27 Apr 2011, at 23:54, Drach, Bob wrote:

> Hi Brian,
> 
> Excellent suggestion. Anyone familiar with setting up an RSS feed?
> 
> --Bob
> 
> 
> On 4/27/11 12:58 PM, "Bryan Lawrence" <bryan.lawrence at stfc.ac.uk> wrote:
> 
>> hi Bob
>> 
>> I wonder how hard it would be to produce a data node feed (or a TDS
>> feed) of datasets published/revised as part of the publication step?
>> 
>> It'd then be relatively easy to parse that for a "new items" page ...
>> 
>> Cheers
>> Bryan
>> 
>> 
>>> I'm happy to post a list of publication events if that would be
>>> useful. But like you I would see this as a temporary solution until
>>> some sort of registry solution could be devised (famous last words
>>> ...). Also I can't really commit to keeping such a list up-to-date
>>> when vacation etc. intervenes.
>>> 
>>> I'd be curious to hear if others think this is a good idea as well.
>>> 
>>> --Bob
>>> 
>>> 
>>> On 4/27/11 3:30 AM, "Estanislao Gonzalez"
>>> <estanislao.gonzalez at zmaw.de>
>>> 
>>> wrote:
>>>> Hi Sébastien,
>>>> 
>>>> indeed this is a great idea, but changing this manually on every
>>>> gateway is not practical at all.
>>>> 
>>>> I've already proposed moving this to a central registry of some
>>>> kind, but considering that the current registry, which is
>>>> essential, is not ready I'd suggest a quick and dirty procedure:
>>>> 
>>>> * Bob (I know you love this :-), could you put this info in a file
>>>> publicly accessible at pcmdi3? (plain txt file, no headers,
>>>> nothing) * Gateway team: could you give us a one line ajax command
>>>> (or anything similar, preferably from the client side) depending
>>>> on the current js libraries to insert this text where it should?
>>>> 
>>>> I think that'll do.
>>>> 
>>>> Thanks,
>>>> Estani
>>>> 
>>>> Am 27.04.2011 11:53, schrieb Sébastien Denvil:
>>>>> Hi Bob, Stephen, Estanislao
>>>>> 
>>>>> I noticed that pcmdi gateway have a notice on the homepage listing
>>>>> new available datasets.
>>>>> 
>>>>> Latest notice being : "BCC datasets will be available at the end
>>>>> of April."
>>>>> 
>>>>> Bob, could add this to your list:
>>>>> "IPSL-CM5A-LR piControl and historical datasets available"
>>>>> They are open to CMIP5-research role since 20th of April.
>>>>> 
>>>>> Stephen, Estanislao I think it could be a good idea to duplicate
>>>>> this notice on the other gateway to help people identifying which
>>>>> datasets are accessible.
>>>>> 
>>>>> Cheers.
>>>>> Sébastien
>>> 
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>> 
>> --
>> Bryan Lawrence
>> Director of Environmental Archival and Associated Research
>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>> STFC, Rutherford Appleton Laboratory
>> Phone +44 1235 445012; Fax ... 5848;
>> Web: home.badc.rl.ac.uk/lawrence
> 
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> 

---------------------------------------------------
Mark Morgan
Software Architect / Engineer
Institut Pierre Simon Laplace (IPSL),
Université Pierre Marie Curie,
4 Place Jussieu,
Tour 45-55, Salle #207,
Paris 75005
France.
Tel : +33 (0) 1 44 27 49 10
Email: momipsl at ipsl.jussieu.fr
---------------------------------------------------



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110428/79ff939f/attachment-0001.html