[Go-essp-tech] [is-enes-sa2-jra4] Example of configuring a datanode to serve CMIP3-DRS
Alex Sim
asim at lbl.gov
Fri Jul 2 12:43:46 MDT 2010
Stephen,
Is your offline datasets managed by CASTOR?
Then, it should work with what Bob described as is.
-- Alex
On 7/2/10 11:39 AM, Bob Drach wrote:
> 'Offline' datasets are those which cannot be opened directly for
> scanning, and therefore will be published with minimal metadata,
> typically just filename, size, and modification date. The presumtion
> is that a lister application such as srm-ls or hsils is accessible to
> provide this information.
>
> Bob
>
> On Jul 1, 2010, at 5:43 AM, <stephen.pascoe at stfc.ac.uk> <stephen.pascoe at stfc.ac.uk
> > wrote:
>
>
>>
>> I think what Bryan is saying is that we *may* need to deploy 2 ESG
>> datanodes. I'm less confident we've worked out a solution yet.
>> However, we do have the ability to run 2 it's just we may make them
>> independent or we may load-balance them.
>>
>> As I understand it thredds_dataset_roots maps each project to a
>> filesystem root. Therefore we could use a separate project to publish
>> the replicated and local portions of CMIP5. I'd like to see someone
>> try
>> this and see what it looks like.
>>
>> esgpublisher has a concept of an offline dataset that is supposed to
>> manage datasets held on tape. However, I don't know how it works so
>> you'll need to ask Bob Drach to explain that.
>>
>> Then there's the "product" portion of the DRS. This was introduced
>> because we expect the datasets we replicate to be different from the
>> ones in the full archive -- replicated datasets may contain some but
>> not
>> all of the files in the equivalent full archive dataset. Therefore we
>> could maintain symbolic links on the filesystem that put /cmip5/output
>> and /cmip5/requested on separate disks. This is sort of similar to
>> what
>> Estani suggests.
>>
>> S.
>>
>> ---
>> Stephen Pascoe +44 (0)1235 445980
>> British Atmospheric Data Centre
>> Rutherford Appleton Laboratory
>>
>> -----Original Message-----
>> From: Bryan Lawrence [mailto:bryan.lawrence at stfc.ac.uk]
>> Sent: 01 July 2010 11:26
>> To: is-enes-sa2-jra4 at lists.enes.org
>> Cc: Estanislao Gonzalez; Pascoe, Stephen (STFC,RAL,SSTD);
>> go-essp-tech at ucar.edu
>> Subject: Re: [is-enes-sa2-jra4] [Go-essp-tech] Example of
>> configuring a
>> datanode to serve CMIP3-DRS
>>
>> Hi Estani
>>
>> Just to note that your approach to the local versus replication will
>> be
>> to run two different ESG nodes ... which is in fact the desired
>> outcome
>> so as to get the right things in the catalogues at the right time
>> (vis-
>> a-viz qc etc).
>>
>> The issue with respect to cache, I'm not so sure about, in what way do
>> you want to expose that into ESG?
>>
>> Bryan
>>
>> On Wednesday 30 Jun 2010 17:05:57 Estanislao Gonzalez wrote:
>>
>>> Hi Stephen,
>>>
>>> the page contains really helpful information, thanks a lot!
>>>
>>> I'm also interested in some variables of the DEFAULT section from the
>>> esg.ini configuration file. More specifically: thredds_dataset_roots
>>> (and maybe thredds_aggregation_services or any other which was
>>> changed or you think it might be important)
>>>
>>> The main question here is: how can different local directory
>>> structures be published to the same DRS structure?
>>> The example scenario in our case will be:
>>> /replicated/<DRS structure> - for replicated data /local/<DRS
>>> structure> - for non replicated data hold on disk /cache/<DRS
>>> structure> - for data staged from a HPSS system
>>>
>>> The only solution I can think of is to extend the URL before the DRS
>>> structure starts (the URL won't be 100% DRS conform anyway).
>>> So
>>> http://*server/thredds/fileserver/<DRS structure> will turn into
>>> http://*server/thredds/fileserver/replicated/<DRS structure>
>>> http://*server/thredds/fileserver/local/<DRS structure>
>>> http://*server/thredds/fileserver/cache/<DRS structure>
>>>
>>> Is that viable? Are there any other options?
>>>
>>> Thanks,
>>> Estani
>>>
>>> stephen.pascoe at stfc.ac.uk wrote:
>>>
>>>> To illustrate how the ESG datanode can be configured to serve data
>>>> for CMIP5 we have deployed a datanode containing a subset of CMIP3
>>>> in the Data Reference Syntax. Some key features of this deployment
>>>> are:
>>>>
>>>> * The underlying directory structure is based on the Data
>>>> Reference Syntax.
>>>> * Datasets published at the realm level.
>>>> * The token-based security filter is replaced by the
>>>> OpenidRelyingParty security filter.
>>>>
>>>> Further notes can be found at
>>>> http://*proj.badc.rl.ac.uk/go-essp/wiki/CMIP3_Datanode
>>>>
>>>> This test deployment should be of interest to anyone wanting to know
>>>>
>>
>>>> how DRS identifiers could be exposed in THREDDS catalogues and the
>>>> TDS HTML interface. You can also try downloading files with OpenID
>>>> authentication or via wget with SSL-client certificate
>>>> authentication. See the link above for details.
>>>>
>>>> Cheers,
>>>> Stephen.
>>>>
>>>>
>>>> ---
>>>> Stephen Pascoe +44 (0)1235 445980
>>>> British Atmospheric Data Centre
>>>> Rutherford Appleton Laboratory
>>>>
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> -----
>>>>
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>
>>>
>> --
>> Bryan Lawrence
>> Director of Environmental Archival and Associated Research (NCAS/
>> British
>> Atmospheric Data Centre and NCEO/NERC NEODC) STFC, Rutherford Appleton
>> Laboratory Phone +44 1235 445012; Fax ... 5848;
>> Web: home.badc.rl.ac.uk/lawrence
>> --
>> Scanned by iCritical.
>> --
>> Scanned by iCritical.
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
More information about the GO-ESSP-TECH
mailing list