[Go-essp-tech] DRS v1.0 and CMOR2.x

Thu Nov 11 02:51:59 MST 2010

Karl, Sebastien 

My understanding was that ESG publisher would be told which version to assign when drslib is used to construct the directory structure.  Unless esgpublish is adapted to run drslib underneath I don't see how else it can be done.  Surely there is no need to have 2 versions for a dataset.

Cheers,

Stephen.

From: owner-cmor at lists.llnl.gov [mailto:owner-cmor at lists.llnl.gov] On Behalf Of Sébastien Denvil
Sent: 10 November 2010 23:25
To: cmor at lists.llnl.gov; go-essp-tech at ucar.edu
Subject: Re: DRS v1.0 and CMOR2.x

Thanks all for the prompt answer and clarification.

Le 10/11/2010 20:59, Karl Taylor a écrit : 

Dear all,

The filenames called for by the DRS and output by CMOR2 (recent versions) are the same and unchanged.  Right?

I'm not absolutely certain that the quoted section of the DRS is accurate.  My latest understanding is that the "ESG version will usually be assigned by the ESG publisher, and it cannot be guaranteed that it always will be consistent with version number appearing in the ESGF data node directory structure for the data being published.  In many cases the directory structure will be generated some days prior to ESG publication, so the dates may differ."

Does anyone disagree with this?  

That's clearer. So the DRS needs some adaptation concerning the version number "issuer" ; the publisher is the real driver. It implies that if we publish a simulation in several streams so at different dates (many reason to support that) they will have different version number. I think loudly just to put myself in situation.

In any case I think it's much too late to change the directory structure created by CMOR2 (which, by the way, is optional).

Ok. 

Note that drslib has functionality to transform from any directory structure whatsoever to the ESGF data node directory structure called for by the DRS.  Also note that the ESG publisher doesn't rely on the directory structure being consistent with DRS, although we think it is important that it be so.

Yes we think as well it's important that at the filesystem level we stick to the ESG-F datanode DRS. To organize multi model / multi experiment analysis by AR5 author and by institute researcher this is important for every group. We will use drslib in the production system then.

I still have questions for Martin about the different "derived product class" and their link with "raw" CMIP5 output. I will discuss that first with Martin "offline" and may get back "online" to enhance my understanding.

Regards.
Sébastien

regards,
Karl

On 11/10/10 6:51 AM, Sébastien Denvil wrote: 

Dear all,

I have seen that version v1.0 of the DRS has been released 
http://BLOCKEDBLOCKEDcmip-pcmdi.llnl.gov/cmip5/output_req.html?submenuheader=2#req_format 
which is very good.

Reading it, it's clear that CMOR2.x will output files to a directory 
structure (but not a filename encoding) mapping the old DRS version 
(v0.27).

My question is, what is the strong rational behind that fact? Can't 
CMOR2.x follow the version v1.0 DRS and be natively compatible with the 
ESGF data node directory structure?

The version number is apparently the reason.
<activity>/<product>/<institute>/<model>/<experiment>/<frequency>/<modeling 
realm>/<MIP table>/<ensemble member>/<version number>/<variable name>/ 
<CMOR filename>.nc

But quoting the DRS:
"Note that the version number assigned to the dataset by ESG is supposed 
to reflect the date of ESG publication, but the version will usually be 
assigned by the user so this cannot generally be guaranteed. The user 
will be instructed to provide ESG with the date that appears in the ESGF 
data node directory structure for the dataset being published. In many 
cases the directory structure will be generated some days prior to 
publication, so the date will not in fact reflect the date of 
publication, but the date that the directory structure was created."

In fine, the user will decide/provide the version number. So I think the 
user should do that via CMOR2.x.

CMOR2.x could use the current date as a version number. Or preferably, 
because cmor post-processing can last few days for a MIP_TABLE, be 
assigned to CMOR2.x via an option by the user. This would guarantee that 
all variables belonging to the same "stream" belongs to the same version 
number.

As we all hold the presses waiting for the CMOR2.5 release to continue 
the production, it could be a good timing to have CMOR2.x compatible 
with DRS v1.0. Depending on the CMIP5 data distribution system "big 
opening day" I'm willing to rewrite now what we have to be from the 
beginning in line with CMOR2.x, DRS v1.0 and publication aspect.

Regards.
Sébastien

-- 
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5

Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02

-- 
Scanned by iCritical.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20101111/18b41de0/attachment.html 