[Go-essp-tech] DRS v1.0 and CMOR2.x

[Sébastien Denvil]

Thanks all for the prompt answer and clarification.


Le 10/11/2010 20:59, Karl Taylor a écrit :
> Dear all,
>
> The filenames called for by the DRS and output by CMOR2 (recent 
> versions) are the same and unchanged.  Right?
>
> I'm not absolutely certain that the quoted section of the DRS is 
> accurate.  My latest understanding is that the "ESG version will 
> usually be assigned by the ESG publisher, and it cannot be guaranteed 
> that it always will be consistent with version number appearing in the 
> ESGF data node directory structure for the data being published.  In 
> many cases the directory structure will be generated some days prior 
> to ESG publication, so the dates may differ."
>
> Does anyone disagree with this?

That's clearer. So the DRS needs some adaptation concerning the version 
number "issuer" ; the publisher is the real driver. It implies that if 
we publish a simulation in several streams so at different dates (many 
reason to support that) they will have different version number. I think 
loudly just to put myself in situation.


> In any case I think it's much too late to change the directory 
> structure created by CMOR2 (which, by the way, is optional).

Ok.

> Note that drslib has functionality to transform from any directory 
> structure whatsoever to the ESGF data node directory structure called 
> for by the DRS.  Also note that the ESG publisher doesn't rely on the 
> directory structure being consistent with DRS, although we think it is 
> important that it be so.

Yes we think as well it's important that at the filesystem level we 
stick to the ESG-F datanode DRS. To organize multi model / multi 
experiment analysis by AR5 author and by institute researcher this is 
important for every group. We will use drslib in the production system then.

I still have questions for Martin about the different "derived product 
class" and their link with "raw" CMIP5 output. I will discuss that first 
with Martin "offline" and may get back "online" to enhance my understanding.

Regards.
Sébastien


> regards,
> Karl
>
> On 11/10/10 6:51 AM, Sébastien Denvil wrote:
>> Dear all,
>>
>> I have seen that version v1.0 of the DRS has been released
>> http://BLOCKEDcmip-pcmdi.llnl.gov/cmip5/output_req.html?submenuheader=2#req_format
>> which is very good.
>>
>> Reading it, it's clear that CMOR2.x will output files to a directory
>> structure (but not a filename encoding) mapping the old DRS version
>> (v0.27).
>>      
>
>> My question is, what is the strong rational behind that fact? Can't
>> CMOR2.x follow the version v1.0 DRS and be natively compatible with the
>> ESGF data node directory structure?
>>
>> The version number is apparently the reason.
>> <activity>/<product>/<institute>/<model>/<experiment>/<frequency>/<modeling
>> realm>/<MIP table>/<ensemble member>/<version number>/<variable name>/
>> <CMOR filename>.nc
>>
>> But quoting the DRS:
>> "Note that the version number assigned to the dataset by ESG is supposed
>> to reflect the date of ESG publication, but the version will usually be
>> assigned by the user so this cannot generally be guaranteed. The user
>> will be instructed to provide ESG with the date that appears in the ESGF
>> data node directory structure for the dataset being published. In many
>> cases the directory structure will be generated some days prior to
>> publication, so the date will not in fact reflect the date of
>> publication, but the date that the directory structure was created."
>>
>> In fine, the user will decide/provide the version number. So I think the
>> user should do that via CMOR2.x.
>>
>> CMOR2.x could use the current date as a version number. Or preferably,
>> because cmor post-processing can last few days for a MIP_TABLE, be
>> assigned to CMOR2.x via an option by the user. This would guarantee that
>> all variables belonging to the same "stream" belongs to the same version
>> number.
>>
>> As we all hold the presses waiting for the CMOR2.5 release to continue
>> the production, it could be a good timing to have CMOR2.x compatible
>> with DRS v1.0. Depending on the CMIP5 data distribution system "big
>> opening day" I'm willing to rewrite now what we have to be from the
>> beginning in line with CMOR2.x, DRS v1.0 and publication aspect.
>>
>> Regards.
>> Sébastien
>>
>>      


-- 
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5

Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20101111/4dcc2dbc/attachment-0001.html