[Go-essp-tech] ESGF Telco today 2010/09/07

Thu Sep 9 16:44:14 MDT 2010

  Hi Balaji and all,

We're *not* going to change the directory structure that CMOR generates 
and writes to so it's o.k. to write your output to the structure found 
at 
http://cmip-pcmdi.llnl.gov/cmip5/docs/CMIP5_output_metadata_requirements.pdf.  
The publisher, however, will modify (in effect) the structure:

1) to include a "version number" subdirectory, as already agreed (and 
which is *essential* because the filenames from different versions may 
be identical, and CMOR won't know what version is being written)

2) to move <ensemble member> from after to before <variable name> (as 
already agreed to make it possible for different ensemble members to be 
published as separate "ESG datasets"),

3)  and now also to add a <variable table>  subdirectory, so files in 
table 6hrPlev and 6hrLev won't appear in the same directory.

**** PLEASE NOTE**** :  If there are no very strong objections, the 
publisher (but not CMOR) should insert the <variable table> subdirectory 
immediately *before* <ensemble member>.  It can't appear after <variable 
name> because we want to be able to publish 6hrPlev and 6hrLev variables 
as different "ESG datasets".  I'm placing it just after <frequency> and 
<modeling realm> since in almost all cases "frequency" and "realm" imply 
a unique table.  (In all cases table implies a unique "frequency" and 
"realm".)

So the ESG directory structure will look like:

cmip5/<product>/<institution>/<model>/<experiment>/<frequency>/<modeling realm>/<variable table>/<ensemble member>/<version>/<variable name>/<files>

whereas the CMOR2 directory structure looks like:

cmip5/<product>/<institution>/<model>/<experiment>/<frequency>/<modeling realm>/<variable name>/<ensemble member>/<files>

So the two different directory structures are based on the same DRS, but CMOR omits more elements of the DRS than does the ESG publisher (see also Stephen's email listing different uses of the DRS)

I hope someone checks all of the above in case I've made any mistakes.

thanks,
Karl

On 9/9/10 1:50 PM, V. Balaji wrote:
> I don't have strong opinions on this, but I'd like someone to take the
> executive decision on this, post it to the cmip5 website, and thereafter
> freeze it (even if subsequent flaws are discovered: we can version
> datasets, but not the DRS itself:-). GFDL is on the verge of running its
> entire CMIP5 post-processing workflow in anger. At that point we are
> committed.
>
> Thanks,
>
> Drach, Bob writes:
>
>> All,
>>
>> I'm happy with adding cmor_table as another DRS field. This is a cleaner
>> solution, and easier to implement in the publisher than combining variable
>> and table.
>>
>> As far as ordering goes, I would suggest that cmor_table follow realm, since
>> in every case but one (6hrLev | 6hrPlev) the cmor_table can be inferred from
>> the combination of frequency and realm. This would make the ordering:
>>
>> cmip5/<product>/<institution>/<model>/<experiment>/<frequency>/<realm>/<cmor
>> _table>/<ensemble>/<version>/<variable>/<file>
>>
>> Regards,
>>
>> Bob
>>
>>
>> On 9/8/10 10:55 PM, "Martina Stockhause"<martina.stockhause at zmaw.de>  wrote:
>>
>>> Dear Karl, dear all,
>>>
>>> the last suggestion was Gavin's in the discussion. We will implement
>>> that in the QCWrapper unless anyone has objections against it.
>>>
>>> cmip5/<product>/<institution>/<model>/<experiment>/<frequency>/<realm>/<ensemb
>>> le>/<version>/<variable>/<cmor2_table>/<file>
>>>
>>> Thanks for the clarification !
>>>
>>> Best wishes,
>>> Martina
>>>
>>>
>>> Karl Taylor wrote:
>>>>   Dear all,
>>>>
>>>> We probably shouldn't spend any more time on this.   Either proposed
>>>> solution seems to solve the probelm, so maybe we go with "group
>>>> think", unless someone can point out some practical implication that
>>>> that choice will make more work for someone (especially the users of
>>>> the archive).
>>>>
>>>> best regards,
>>>> Karl
>>>>
>>>> On 9/8/10 10:34 AM, Gavin M. Bell wrote:
>>>>> Suggestion....
>>>>>
>>>>> Hi Stephen,
>>>>>
>>>>> Can you please make the second to last token, namely:
>>>>> <variable>_<table>, be like the other tokens in the taxonomy.
>>>>> Specifically make it<variable>/<table>.  When parsing this, it makes
>>>>> it a bit difficult to look for this new delimiter.  It complicates
>>>>> things a bit.  Does it also mean that "_" are not allowed as a
>>>>> character in any of the other bits of the taxonomy.  The regex
>>>>> becomes a real pain and "regular" parsing is even worse.  It would
>>>>> make everything quite a bit simpler if it was like the others. i.e.
>>>>>
>>>>> cmip5/<product>/<institution>/<model>/<experiment>/<frequency>/<realm>/<ense
>>>>> mble>/<version>/<variable>/<table>/<file>
>>>>>
>>>>> Furthermore, is this structure saying that we are always going to be
>>>>> using "cmip5" as the root of this taxonomy?
>>>>>
>>>>> Please let me know your thoughts.  If you can accommodate this it
>>>>> would be great!
>>>>>
>>>>> On 9/7/10 3:17 AM, stephen.pascoe at stfc.ac.uk wrote:
>>>>>> Today we plan to discuss the DRS directory structure and
>>>>>> prerequisites for starting replication tests between DKRZ and BADC.
>>>>>> I am putting together an agenda at the link below, it's still
>>>>>> evolving but comments welcome.  Telephone details are also below.
>>>>>> Thanks,
>>>>>> Stephen
>>>>>> *Agenda:
>>>>>> *http://****proj.badc.rl.ac.uk/go-essp/wiki/CMIP5/Meetings/telco100907
>>>>>> *Telco details:*
>>>>>> 16:00 BST, 17:00 CEST, 8:00 PDT, 9:00 MDT, 11:00 EDT.
>>>>>> +01 (925) 424-8105 access code 305757#
>>>>>> ---
>>>>>> Stephen Pascoe  +44 (0)1235 445980
>>>>>> British Atmospheric Data Centre
>>>>>> Rutherford Appleton Laboratory
>>>>>>
>>>>>> --
>>>>>> Scanned by iCritical.
>>>>>>
>>>>>>
>>>>> --
>>>>> Gavin M. Bell
>>>>> Lawrence Livermore National Labs
>>>>> --
>>>>>
>>>>>    "Never mistake a clear view for a short distance."
>>>>>                     -Paul Saffo
>>>>>
>>>>> (GPG Key -http://***rainbow.llnl.gov/dist/keys/gavin.asc)
>>>>>
>>>>>    A796 CE39 9C31 68A4 52A7  1F6B 66B7 B250 21D5 6D3E
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://**mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>