[Go-essp-tech] [esg-gateway-dev] Status of the CMIP5 Archive

Thu Apr 28 06:52:28 MDT 2011

Maybe, but I'd be slightly more reluctant to do this for a high-bandwidth server like the datanode than for the gateway.  I don't have any numbers but I would expect it to add some overhead.

Stephen.

---
Stephen Pascoe  +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK

From: Kershaw, Philip (STFC,RAL,RALSP)
Sent: 28 April 2011 13:50
To: Pascoe, Stephen (STFC,RAL,RALSP); momipsl at ipsl.jussieu.fr; drach1 at llnl.gov
Cc: esg-gateway-dev at earthsystemgrid.org; go-essp-tech at ucar.edu; sebastien.denvil at ipsl.jussieu.fr
Subject: Re: [esg-gateway-dev] [Go-essp-tech] Status of the CMIP5 Archive

Another option would be to proxy tomcat through Apache as JPL have done then the Pylons app could run via mod_wsgi.

Cheers,
Phil

From: <stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk>>
Date: Thu, 28 Apr 2011 08:49:43 +0000
To: <momipsl at ipsl.jussieu.fr<mailto:momipsl at ipsl.jussieu.fr>>, <drach1 at llnl.gov<mailto:drach1 at llnl.gov>>
Cc: <esg-gateway-dev at earthsystemgrid.org<mailto:esg-gateway-dev at earthsystemgrid.org>>, <go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>>, <sebastien.denvil at ipsl.jussieu.fr<mailto:sebastien.denvil at ipsl.jussieu.fr>>
Subject: Re: [esg-gateway-dev] [Go-essp-tech] Status of the CMIP5 Archive

Hi All,

Similarly we have a web app which exposes publishing information in the datanode database for use between us and the MetOffice.  It uses the same SQLAlchemy model as esgcet with a few extra tables.  I don't think it would take me long to create a view that displays recently published datasets as an atom feed.

However, there are complications.  This is a Pylons app which we deploy on a separate machine to the datanode or gateway.  If we were to put it on the datanode it would have to listen on a separate port to tomcat.  Or we could implement something similar in Java and put it in the datanode's tomcat.

Also, Luca and Mark are right.  To be useful we would need to aggregate the information from each datanode and provide less fine-grained info like "Which experiments have just been published".  It begins to look a lot more like a query service with output as Atom/RSS.

S.

---
Stephen Pascoe  +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK

From: go-essp-tech-bounces at ucar.edu<mailto:go-essp-tech-bounces at ucar.edu> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Mark Morgan
Sent: 28 April 2011 11:11
To: Drach, Bob
Cc: Sébastien Denvil; go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>; esg-gateway-dev at earthsystemgrid.org<mailto:esg-gateway-dev at earthsystemgrid.org>
Subject: Re: [Go-essp-tech] Status of the CMIP5 Archive

Bob / Bryan

Ontop of the possibilty of the DataNode exposing ESG-Publisher AtomPub HTTP endpoints, may I also add to the mix the possibility of ESG-Publisher search web services.  Such web services would in turn permit the development of a ESG distributed search broker, i.e. an aggregator of search results pulled from multiple data nodes.

At IPSL we have developed a portal that launches an overnight batch job to harvest & aggregate meta-data derived from the THREDDS catalogs published at each of our data nodes.  We could certainly use the ESG-Publisher AtomPub feeds to optimise the synchronisation of the aggregated meta-data.

However ultimately all paths lead to distributed search as in a few years we will have several millions of files/variables to search against and there are limits to what can be acheived with a solution based upon aggregation.  Hence the ESG-Publisher search web services and associated distributed search broker are of real interest.

Regards

Mark

On 27 Apr 2011, at 23:54, Drach, Bob wrote:

Hi Brian,

Excellent suggestion. Anyone familiar with setting up an RSS feed?

--Bob

On 4/27/11 12:58 PM, "Bryan Lawrence" <bryan.lawrence at stfc.ac.uk<mailto:bryan.lawrence at stfc.ac.uk>> wrote:

hi Bob

I wonder how hard it would be to produce a data node feed (or a TDS
feed) of datasets published/revised as part of the publication step?

It'd then be relatively easy to parse that for a "new items" page ...

Cheers
Bryan

I'm happy to post a list of publication events if that would be
useful. But like you I would see this as a temporary solution until
some sort of registry solution could be devised (famous last words
...). Also I can't really commit to keeping such a list up-to-date
when vacation etc. intervenes.

I'd be curious to hear if others think this is a good idea as well.

--Bob

On 4/27/11 3:30 AM, "Estanislao Gonzalez"
<estanislao.gonzalez at zmaw.de<mailto:estanislao.gonzalez at zmaw.de>>

wrote:
Hi Sébastien,

indeed this is a great idea, but changing this manually on every
gateway is not practical at all.

I've already proposed moving this to a central registry of some
kind, but considering that the current registry, which is
essential, is not ready I'd suggest a quick and dirty procedure:

* Bob (I know you love this :-), could you put this info in a file
publicly accessible at pcmdi3? (plain txt file, no headers,
nothing) * Gateway team: could you give us a one line ajax command
(or anything similar, preferably from the client side) depending
on the current js libraries to insert this text where it should?

I think that'll do.

Thanks,
Estani

Am 27.04.2011 11:53, schrieb Sébastien Denvil:
Hi Bob, Stephen, Estanislao

I noticed that pcmdi gateway have a notice on the homepage listing
new available datasets.

Latest notice being : "BCC datasets will be available at the end
of April."

Bob, could add this to your list:
"IPSL-CM5A-LR piControl and historical datasets available"
They are open to CMIP5-research role since 20th of April.

Stephen, Estanislao I think it could be a good idea to duplicate
this notice on the other gateway to help people identifying which
datasets are accessible.

Cheers.
Sébastien

_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

--
Bryan Lawrence
Director of Environmental Archival and Associated Research
(NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
STFC, Rutherford Appleton Laboratory
Phone +44 1235 445012; Fax ... 5848;
Web: home.badc.rl.ac.uk/lawrence

_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

---------------------------------------------------
Mark Morgan
Software Architect / Engineer
Institut Pierre Simon Laplace (IPSL),
Université Pierre Marie Curie,
4 Place Jussieu,
Tour 45-55, Salle #207,
Paris 75005
France.
Tel : +33 (0) 1 44 27 49 10
Email: momipsl at ipsl.jussieu.fr<mailto:momipsl at ipsl.jussieu.fr>
---------------------------------------------------

--
Scanned by iCritical.

_______________________________________________ esg-gateway-dev mailing list esg-gateway-dev at mailman.earthsystemgrid.org<mailto:esg-gateway-dev at mailman.earthsystemgrid.org> http://mailman.earthsystemgrid.org/mailman/listinfo/esg-gateway-dev

-- 
Scanned by iCritical.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110428/3e89786c/attachment-0001.html 