[Go-essp-tech] ESGF Telco today 2010/09/07
V. Balaji
V.Balaji at noaa.gov
Thu Sep 9 17:09:29 MDT 2010
Thanks for the clarification. I think Sergey tried to explain the
distinction to me yesterday but I missed it. Do we have to update
the ESG datanode software stack once these changes are committed?
Karl Taylor writes:
> Hi Balaji and all,
>
> We're *not* going to change the directory structure that CMOR generates and
> writes to so it's o.k. to write your output to the structure found at
> http://cmip-pcmdi.llnl.gov/cmip5/docs/CMIP5_output_metadata_requirements.pdf.
> The publisher, however, will modify (in effect) the structure:
>
> 1) to include a "version number" subdirectory, as already agreed (and which
> is *essential* because the filenames from different versions may be
> identical, and CMOR won't know what version is being written)
>
> 2) to move <ensemble member> from after to before <variable name> (as already
> agreed to make it possible for different ensemble members to be published as
> separate "ESG datasets"),
>
> 3) and now also to add a <variable table> subdirectory, so files in table
> 6hrPlev and 6hrLev won't appear in the same directory.
>
> **** PLEASE NOTE**** : If there are no very strong objections, the publisher
> (but not CMOR) should insert the <variable table> subdirectory immediately
> *before* <ensemble member>. It can't appear after <variable name> because we
> want to be able to publish 6hrPlev and 6hrLev variables as different "ESG
> datasets". I'm placing it just after <frequency> and <modeling realm> since
> in almost all cases "frequency" and "realm" imply a unique table. (In all
> cases table implies a unique "frequency" and "realm".)
>
> So the ESG directory structure will look like:
>
> cmip5/<product>/<institution>/<model>/<experiment>/<frequency>/<modeling
> realm>/<variable table>/<ensemble member>/<version>/<variable name>/<files>
>
> whereas the CMOR2 directory structure looks like:
>
> cmip5/<product>/<institution>/<model>/<experiment>/<frequency>/<modeling
> realm>/<variable name>/<ensemble member>/<files>
>
> So the two different directory structures are based on the same DRS, but CMOR
> omits more elements of the DRS than does the ESG publisher (see also
> Stephen's email listing different uses of the DRS)
>
> I hope someone checks all of the above in case I've made any mistakes.
>
> thanks,
> Karl
>
>
>
> On 9/9/10 1:50 PM, V. Balaji wrote:
>> I don't have strong opinions on this, but I'd like someone to take the
>> executive decision on this, post it to the cmip5 website, and thereafter
>> freeze it (even if subsequent flaws are discovered: we can version
>> datasets, but not the DRS itself:-). GFDL is on the verge of running its
>> entire CMIP5 post-processing workflow in anger. At that point we are
>> committed.
>>
>> Thanks,
>>
>> Drach, Bob writes:
>>
>>> All,
>>>
>>> I'm happy with adding cmor_table as another DRS field. This is a cleaner
>>> solution, and easier to implement in the publisher than combining variable
>>> and table.
>>>
>>> As far as ordering goes, I would suggest that cmor_table follow realm,
>>> since
>>> in every case but one (6hrLev | 6hrPlev) the cmor_table can be inferred
>>> from
>>> the combination of frequency and realm. This would make the ordering:
>>>
>>> cmip5/<product>/<institution>/<model>/<experiment>/<frequency>/<realm>/<cmor
>>> _table>/<ensemble>/<version>/<variable>/<file>
>>>
>>> Regards,
>>>
>>> Bob
>>>
>>>
>>> On 9/8/10 10:55 PM, "Martina Stockhause"<martina.stockhause at zmaw.de>
>>> wrote:
>>>
>>>> Dear Karl, dear all,
>>>>
>>>> the last suggestion was Gavin's in the discussion. We will implement
>>>> that in the QCWrapper unless anyone has objections against it.
>>>>
>>>> cmip5/<product>/<institution>/<model>/<experiment>/<frequency>/<realm>/<ensemb
>>>> le>/<version>/<variable>/<cmor2_table>/<file>
>>>>
>>>> Thanks for the clarification !
>>>>
>>>> Best wishes,
>>>> Martina
>>>>
>>>>
>>>> Karl Taylor wrote:
>>>>> Dear all,
>>>>>
>>>>> We probably shouldn't spend any more time on this. Either proposed
>>>>> solution seems to solve the probelm, so maybe we go with "group
>>>>> think", unless someone can point out some practical implication that
>>>>> that choice will make more work for someone (especially the users of
>>>>> the archive).
>>>>>
>>>>> best regards,
>>>>> Karl
>>>>>
>>>>> On 9/8/10 10:34 AM, Gavin M. Bell wrote:
>>>>>> Suggestion....
>>>>>>
>>>>>> Hi Stephen,
>>>>>>
>>>>>> Can you please make the second to last token, namely:
>>>>>> <variable>_<table>, be like the other tokens in the taxonomy.
>>>>>> Specifically make it<variable>/<table>. When parsing this, it makes
>>>>>> it a bit difficult to look for this new delimiter. It complicates
>>>>>> things a bit. Does it also mean that "_" are not allowed as a
>>>>>> character in any of the other bits of the taxonomy. The regex
>>>>>> becomes a real pain and "regular" parsing is even worse. It would
>>>>>> make everything quite a bit simpler if it was like the others. i.e.
>>>>>>
>>>>>> cmip5/<product>/<institution>/<model>/<experiment>/<frequency>/<realm>/<ense
>>>>>> mble>/<version>/<variable>/<table>/<file>
>>>>>>
>>>>>> Furthermore, is this structure saying that we are always going to be
>>>>>> using "cmip5" as the root of this taxonomy?
>>>>>>
>>>>>> Please let me know your thoughts. If you can accommodate this it
>>>>>> would be great!
>>>>>>
>>>>>> On 9/7/10 3:17 AM, stephen.pascoe at stfc.ac.uk wrote:
>>>>>>> Today we plan to discuss the DRS directory structure and
>>>>>>> prerequisites for starting replication tests between DKRZ and BADC.
>>>>>>> I am putting together an agenda at the link below, it's still
>>>>>>> evolving but comments welcome. Telephone details are also below.
>>>>>>> Thanks,
>>>>>>> Stephen
>>>>>>> *Agenda:
>>>>>>> *http://****proj.badc.rl.ac.uk/go-essp/wiki/CMIP5/Meetings/telco100907
>>>>>>> *Telco details:*
>>>>>>> 16:00 BST, 17:00 CEST, 8:00 PDT, 9:00 MDT, 11:00 EDT.
>>>>>>> +01 (925) 424-8105 access code 305757#
>>>>>>> ---
>>>>>>> Stephen Pascoe +44 (0)1235 445980
>>>>>>> British Atmospheric Data Centre
>>>>>>> Rutherford Appleton Laboratory
>>>>>>>
>>>>>>> --
>>>>>>> Scanned by iCritical.
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Gavin M. Bell
>>>>>> Lawrence Livermore National Labs
>>>>>> --
>>>>>>
>>>>>> "Never mistake a clear view for a short distance."
>>>>>> -Paul Saffo
>>>>>>
>>>>>> (GPG Key -http://***rainbow.llnl.gov/dist/keys/gavin.asc)
>>>>>>
>>>>>> A796 CE39 9C31 68A4 52A7 1F6B 66B7 B250 21D5 6D3E
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> GO-ESSP-TECH mailing list
>>>>> GO-ESSP-TECH at ucar.edu
>>>>> http://**mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>
>
--
V. Balaji Office: +1-609-452-6516
Head, Modeling Systems Group, GFDL Home: +1-212-253-6662
Princeton University Email: v.balaji at noaa.gov
More information about the GO-ESSP-TECH
mailing list