[Go-essp-tech] [is-enes-sa2-jra4] Example of configuring a datanode to serve CMIP3-DRS

[Bob Drach]

'Offline' datasets are those which cannot be opened directly for  
scanning, and therefore will be published with minimal metadata,  
typically just filename, size, and modification date. The presumtion  
is that a lister application such as srm-ls or hsils is accessible to  
provide this information.

Bob

On Jul 1, 2010, at 5:43 AM, <stephen.pascoe at stfc.ac.uk> <stephen.pascoe at stfc.ac.uk 
 > wrote:

>
>
> I think what Bryan is saying is that we *may* need to deploy 2 ESG
> datanodes.  I'm less confident we've worked out a solution yet.
> However, we do have the ability to run 2 it's just we may make them
> independent or we may load-balance them.
>
> As I understand it thredds_dataset_roots maps each project to a
> filesystem root.  Therefore we could use a separate project to publish
> the replicated and local portions of CMIP5.  I'd like to see someone  
> try
> this and see what it looks like.
>
> esgpublisher has a concept of an offline dataset that is supposed to
> manage datasets held on tape.  However, I don't know how it works so
> you'll need to ask Bob Drach to explain that.
>
> Then there's the "product" portion of the DRS.  This was introduced
> because we expect the datasets we replicate to be different from the
> ones in the full archive -- replicated datasets may contain some but  
> not
> all of the files in the equivalent full archive dataset.  Therefore we
> could maintain symbolic links on the filesystem that put /cmip5/output
> and /cmip5/requested on separate disks.  This is sort of similar to  
> what
> Estani suggests.
>
> S.
>
> ---
> Stephen Pascoe  +44 (0)1235 445980
> British Atmospheric Data Centre
> Rutherford Appleton Laboratory
>
> -----Original Message-----
> From: Bryan Lawrence [mailto:bryan.lawrence at stfc.ac.uk]
> Sent: 01 July 2010 11:26
> To: is-enes-sa2-jra4 at lists.enes.org
> Cc: Estanislao Gonzalez; Pascoe, Stephen (STFC,RAL,SSTD);
> go-essp-tech at ucar.edu
> Subject: Re: [is-enes-sa2-jra4] [Go-essp-tech] Example of  
> configuring a
> datanode to serve CMIP3-DRS
>
> Hi Estani
>
> Just to note that your approach to the local versus replication will  
> be
> to run two different ESG nodes ... which is in fact the desired  
> outcome
> so as to get the right things in the catalogues at the right time  
> (vis-
> a-viz qc etc).
>
> The issue with respect to cache, I'm not so sure about, in what way do
> you want to expose that into ESG?
>
> Bryan
>
> On Wednesday 30 Jun 2010 17:05:57 Estanislao Gonzalez wrote:
>> Hi Stephen,
>>
>> the page contains really helpful information, thanks a lot!
>>
>> I'm also interested in some variables of the DEFAULT section from the
>> esg.ini configuration file. More specifically: thredds_dataset_roots
>> (and maybe thredds_aggregation_services or any other which was
>> changed or you think it might be important)
>>
>> The main question here is: how can different local directory
>> structures be published to the same DRS structure?
>> The example scenario in our case will be:
>> /replicated/<DRS structure> - for replicated data /local/<DRS
>> structure> - for non replicated data hold on disk /cache/<DRS
>> structure> - for data staged from a HPSS system
>>
>> The only solution I can think of is to extend the URL before the DRS
>> structure starts (the URL won't be 100% DRS conform anyway).
>> So
>>    http://*server/thredds/fileserver/<DRS structure> will turn into
>>    http://*server/thredds/fileserver/replicated/<DRS structure>
>>    http://*server/thredds/fileserver/local/<DRS structure>
>>    http://*server/thredds/fileserver/cache/<DRS structure>
>>
>> Is that viable? Are there any other options?
>>
>> Thanks,
>> Estani
>>
>> stephen.pascoe at stfc.ac.uk wrote:
>>> To illustrate how the ESG datanode can be configured to serve data
>>> for CMIP5 we have deployed a datanode containing a subset of CMIP3
>>> in the Data Reference Syntax. Some key features of this deployment
>>> are:
>>>
>>>    * The underlying directory structure is based on the Data
>>>      Reference Syntax.
>>>    * Datasets published at the realm level.
>>>    * The token-based security filter is replaced by the
>>>      OpenidRelyingParty security filter.
>>>
>>> Further notes can be found at
>>> http://*proj.badc.rl.ac.uk/go-essp/wiki/CMIP3_Datanode
>>>
>>> This test deployment should be of interest to anyone wanting to know
>
>>> how DRS identifiers could be exposed in THREDDS catalogues and the
>>> TDS HTML interface.  You can also try downloading files with OpenID
>>> authentication or via wget with SSL-client certificate
>>> authentication.  See the link above for details.
>>>
>>> Cheers,
>>> Stephen.
>>>
>>>
>>> ---
>>> Stephen Pascoe  +44 (0)1235 445980
>>> British Atmospheric Data Centre
>>> Rutherford Appleton Laboratory
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------------------
>>> -----
>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>
> --
> Bryan Lawrence
> Director of Environmental Archival and Associated Research (NCAS/ 
> British
> Atmospheric Data Centre and NCEO/NERC NEODC) STFC, Rutherford Appleton
> Laboratory Phone +44 1235 445012; Fax ... 5848;
> Web: home.badc.rl.ac.uk/lawrence
> --
> Scanned by iCritical.
> -- 
> Scanned by iCritical.
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
>