[Go-essp-tech] [is-enes-sa2-jra4] Example of configuring a datanode to serve CMIP3-DRS

[stephen.pascoe at stfc.ac.uk]

I think what Bryan is saying is that we *may* need to deploy 2 ESG
datanodes.  I'm less confident we've worked out a solution yet.
However, we do have the ability to run 2 it's just we may make them
independent or we may load-balance them.

As I understand it thredds_dataset_roots maps each project to a
filesystem root.  Therefore we could use a separate project to publish
the replicated and local portions of CMIP5.  I'd like to see someone try
this and see what it looks like.

esgpublisher has a concept of an offline dataset that is supposed to
manage datasets held on tape.  However, I don't know how it works so
you'll need to ask Bob Drach to explain that.

Then there's the "product" portion of the DRS.  This was introduced
because we expect the datasets we replicate to be different from the
ones in the full archive -- replicated datasets may contain some but not
all of the files in the equivalent full archive dataset.  Therefore we
could maintain symbolic links on the filesystem that put /cmip5/output
and /cmip5/requested on separate disks.  This is sort of similar to what
Estani suggests.

S.

---
Stephen Pascoe  +44 (0)1235 445980
British Atmospheric Data Centre
Rutherford Appleton Laboratory

-----Original Message-----
From: Bryan Lawrence [mailto:bryan.lawrence at stfc.ac.uk] 
Sent: 01 July 2010 11:26
To: is-enes-sa2-jra4 at lists.enes.org
Cc: Estanislao Gonzalez; Pascoe, Stephen (STFC,RAL,SSTD);
go-essp-tech at ucar.edu
Subject: Re: [is-enes-sa2-jra4] [Go-essp-tech] Example of configuring a
datanode to serve CMIP3-DRS

Hi Estani

Just to note that your approach to the local versus replication will be
to run two different ESG nodes ... which is in fact the desired outcome
so as to get the right things in the catalogues at the right time (vis-
a-viz qc etc).

The issue with respect to cache, I'm not so sure about, in what way do
you want to expose that into ESG?

Bryan

On Wednesday 30 Jun 2010 17:05:57 Estanislao Gonzalez wrote:
> Hi Stephen,
> 
> the page contains really helpful information, thanks a lot!
> 
> I'm also interested in some variables of the DEFAULT section from the 
> esg.ini configuration file. More specifically: thredds_dataset_roots 
> (and maybe thredds_aggregation_services or any other which was  
> changed or you think it might be important)
> 
> The main question here is: how can different local directory  
> structures be published to the same DRS structure?
> The example scenario in our case will be:
> /replicated/<DRS structure> - for replicated data /local/<DRS 
> structure> - for non replicated data hold on disk /cache/<DRS 
> structure> - for data staged from a HPSS system
> 
> The only solution I can think of is to extend the URL before the DRS 
> structure starts (the URL won't be 100% DRS conform anyway).
> So
>     http://server/thredds/fileserver/<DRS structure> will turn into
>     http://server/thredds/fileserver/replicated/<DRS structure>
>     http://server/thredds/fileserver/local/<DRS structure>
>     http://server/thredds/fileserver/cache/<DRS structure>
> 
> Is that viable? Are there any other options?
> 
> Thanks,
> Estani
> 
> stephen.pascoe at stfc.ac.uk wrote:
> > To illustrate how the ESG datanode can be configured to serve data 
> > for CMIP5 we have deployed a datanode containing a subset of CMIP3 
> > in the Data Reference Syntax. Some key features of this deployment
> > are:
> >
> >     * The underlying directory structure is based on the Data
> >       Reference Syntax.
> >     * Datasets published at the realm level.
> >     * The token-based security filter is replaced by the
> >       OpenidRelyingParty security filter.
> >
> > Further notes can be found at
> > http://proj.badc.rl.ac.uk/go-essp/wiki/CMIP3_Datanode
> >
> > This test deployment should be of interest to anyone wanting to know

> > how DRS identifiers could be exposed in THREDDS catalogues and the 
> > TDS HTML interface.  You can also try downloading files with OpenID 
> > authentication or via wget with SSL-client certificate 
> > authentication.  See the link above for details.
> >
> > Cheers,
> > Stephen.
> >
> >
> > ---
> > Stephen Pascoe  +44 (0)1235 445980
> > British Atmospheric Data Centre
> > Rutherford Appleton Laboratory
> >
> >
> >
> >
> > -------------------------------------------------------------------
> >-----
> >
> > _______________________________________________
> > GO-ESSP-TECH mailing list
> > GO-ESSP-TECH at ucar.edu
> > http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> 

--
Bryan Lawrence
Director of Environmental Archival and Associated Research (NCAS/British
Atmospheric Data Centre and NCEO/NERC NEODC) STFC, Rutherford Appleton
Laboratory Phone +44 1235 445012; Fax ... 5848;
Web: home.badc.rl.ac.uk/lawrence
--
Scanned by iCritical.
-- 
Scanned by iCritical.