[Go-essp-tech] Configuring esgcet for CMIP5 and the DRS structure

[Bob Drach]

Hi Stephen,

Funny you should mention ... I just published a portion of the CMIP3 
archive with DRS-style datasets. The project portion of the init file is 
attached.

For the attached init file to work you'll need the bleeding edge 
repository version of the publisher - there is a tweak to get the 
variable_standard_name and variable_long_name fields into the 
dataset_name_format options. [I believe if you remove any mention of 
variable_standard_name, that Version 2.1 should work, but I haven't 
tested it.]

You'll need to change a few things for this to work in your environment:

- parent_id = %(root_id)s.ipcc4.%(model)s assumes that there are 
existing intermediate datasets, one for each model.
- directory_format is probably unique to our environment.
- realm is deduced from the directory structure, in our case.

Part of the reason for testing this on our end is to evaluate 
publication performance with dataset granularity at the level of 
DRS-style datasets. One thing that became obvious is that there is a 
per-dataset overhead for each web-service call to the gateway - the 
final publish step. It's not too significant when all variables for a 
run are grouped together as has been the case up to now. But when each 
dataset is a single variable the number of datasets baloons to ~28,000, 
and a few seconds per call becomes very significant. The bleeding edge 
version has one change that reduces the overhead ~25%, and the gateway 
developers claim they have also sped up the publication processing. But 
the question still remains what the overhead of the web-service call 
itself is. I'll be interested to see what your experience is.

Bob

stephen.pascoe at stfc.ac.uk wrote:
> Hi Bob,
>  
> We now have the CMIP3 archive partially in DRS format -- 1pctto2x and 
> 1pctto4x experiments are done with the rest proceeding.  So I'm now 
> trying to configure esgcet to recognise this structure by defining a 
> new project in esg.ini "[project:cmip3_drs]". 
>  
> Do you have a sample [project:cmip5] section you are working on or 
> should I continue to follow my intuition on how one should map DRS 
> components to categories in esg.ini?
>  
> Cheers,
> Stephen.
>  
> ---
> Stephen Pascoe  +44 (0)1235 445980
> British Atmospheric Data Centre
> Rutherford Appleton Laboratory
>  
>
> -- 
> Scanned by iCritical.
>
>

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: drs_project.ini
Url: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20100201/4cc7f78e/attachment.pl