<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Sebastian,<div><span class="Apple-tab-span" style="white-space: pre; "> </span>thanks for your summary of the current status, it all seems correct to me. A few comments regarding the 3 points below:</div><div><br></div><div>1) Stephen Pascoe pointed out that the latest agreement is to derive the dataset version number from the date, for example v20100105 for data published on January 5th. I updated the wiki proposal to reflect this:</div><div><br></div><div><a href="http://oodt.jpl.nasa.gov/wiki/display/CLIMATE/Data+and+Metadata+Requirements+for+CMIP5+Observational+Datasets">http://oodt.jpl.nasa.gov/wiki/display/CLIMATE/Data+and+Metadata+Requirements+for+CMIP5+Observational+Datasets</a></div><div><br></div><div>2) I agree we should focus on formalizing the DRS Controlled Vocabulary for obs via document sharing tools... when we have a first version, maybe PCMDI can encode it in a CMOR table ?</div><div>3) Ok, let's use [project:cmip5] in esg.ini. We might be able to share the same snippet of configuration across all agencies involved.</div><div><br></div><div>thanks, Luca</div><div><div><br></div><div>On Jan 27, 2011, at 7:41 AM, Sébastien Denvil wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div text="#000000" bgcolor="#ffffff">
Hello Luca, all,<br>
<br>
within CFMIP-obs context IPSL plan to publish observational
datasets that will be made part of the CMIP5 database. We have up to
now identify:<br>
- CALIPSO-GOCCP (monthly & daily)<br>
- PARASOL (monthly & daily)<br>
- CloudSat diagnostics (consistent with COSP output), which are:<br>
- CFADs (radar reflectivity histograms function of altitude)
prepared by Roj Marchand (CSU)
<br>
- ISCCP diagnostics consistent with COSP outputs. Prepared by
Yuying Zhang (PCMDI) and Rob Pincus (CIRES)<br>
- SIRTA-Cabauw sites database developed within the EU FP7 EUCLIPSE
project, prepared by Chiriaco & Haeffelin (IPSL). Could be
included in the obsCfSites : in principle close to the ARM stations
case.<br>
<br>
To resume what I understood from this thread. There are 3 documents
to produce:<br>
1) agree on a DRS suitable for observational datasets (Luca's
proposal looks good to me)<br>
2) agree on a set of new cmor table (possibly to be included in the
cmip5 cmor table git repository)<br>
3) agree on a esg.ini set up able to publish observational datasets
the way we want to<br>
<br>
Regarding esg.ini what Renata suggested seems to me like a good
idea:<br>
"to make it a CMIP5 project then your definitions should be in the
section
<div>[project:cmip5]</div>
<div>which then means extending the existing cmip5 definitions and
options, that mostly seems to be not a problem."<br>
</div>
<br>
Setting up new table may prove to be the "hard point". Work on a
Google Docs page with the CMOR table spreadsheet could ease the
process.<br>
<br>
Regards.<br>
Sébastien<br>
<br>
On 27/01/2011 14:28, <a class="moz-txt-link-abbreviated" href="mailto:martin.juckes@stfc.ac.uk">martin.juckes@stfc.ac.uk</a> wrote:
<blockquote cite="mid:E21FBC3F00D7304687CB46529F9676D7039DD9@EXCHMBX01.fed.cclrc.ac.uk" type="cite">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Lucida Grande";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1"><p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);">Hello All,<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);"><o:p> </o:p></span></p><p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);">This is an interesting discussion – I’ve copied
Victoria in because she is involved in discussions with the
ESA Climate Change Initiative about data formats etc for the
climate datasets ESA is planning to produce. It would be
really good to try to coordinate efforts, at least up to the
level of agreeing a set of controlled vocabularies to
identify datasets, and agreeing what file attributes will be
used to hold that information in the file. Luca’s wiki page
lays out the issues clearly – so perhaps we should see what
the ESA CCI group think.
<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);"><o:p> </o:p></span></p><p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);">Cheers,<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);">Martin<o:p></o:p></span></p><p class="MsoNormal"><span style="font-size: 11pt; font-family:
"Calibri","sans-serif"; color: rgb(31,
73, 125);"><o:p> </o:p></span></p>
<div>
<div style="border-right: medium none; border-width: 1pt
medium medium; border-style: solid none none; border-color:
rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color;
padding: 3pt 0cm 0cm;"><p class="MsoNormal"><b><span style="font-size: 10pt;
font-family:
"Tahoma","sans-serif";" lang="EN-US">From:</span></b><span style="font-size:
10pt; font-family:
"Tahoma","sans-serif";" lang="EN-US">
<a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a>
[<a class="moz-txt-link-freetext" href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>]
<b>On Behalf Of </b>Renata McCoy<br>
<b>Sent:</b> 27 January 2011 00:42<br>
<b>To:</b> Cinquini, Luca (3880)<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>; climate-obs; Giri
Palanisamy; Renata McCoy<br>
<b>Subject:</b> Re: [Go-essp-tech] Extending the DRS
syntax to observations<o:p></o:p></span></p>
</div>
</div><p class="MsoNormal"><o:p> </o:p></p>
<div><p class="MsoNormal"><span class="apple-style-span"><span style="font-size: 13.5pt; font-family: "Lucida
Grande","serif"; color: black;">Hi Luca,</span></span><span style="font-size: 13.5pt; font-family:
"Arial","sans-serif"; color: black;"><br>
<br>
</span><o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">I see that you defined your project as an
AIRS project. If we want to make it a CMIP5 project then
your definitions should be in the section<o:p></o:p></p>
</div>
<div><p class="MsoNormal">[project:cmip5]<o:p></o:p></p>
</div>
<div><p class="MsoNormal">which then means extending the existing
cmip5 definitions and options, that mostly seems to be not a
problem. <o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">Giri set up a Google Docs page with the
CMOR table spreadsheet. If he doesn't mind, we could all
work on it. I will work on it tomorrow to clean it up.
Right now it is very ARM specific, but we discussed about
changing that and making it all observation inclusive. <o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">Renata<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div><p class="MsoNormal"><o:p> </o:p></p>
<div>
<div><p class="MsoNormal">On Jan 26, 2011, at 4:19 PM, Cinquini,
Luca (3880) wrote:<o:p></o:p></p>
</div><p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div><p class="MsoNormal">Hi Renata,<o:p></o:p></p>
<div><p class="MsoNormal"><span class="apple-tab-span">
</span>I'm glad you are experimenting with the same
issues...<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">I actually also experimented with
using the ESG publisher to publish AIRS data that was
organized in a DRS-like fashion like I mentioned:<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<div><p class="MsoNormal"><root
directory>/cmip5/observations/nasa/aqua/airs/mon/atmos/ta/l3/vYYYYMMDD/<files><o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">I was able to succeed by
using the following snippet of code in esg.ini:<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div><p class="MsoNormal">[project:AIRS]<o:p></o:p></p>
</div>
<div><p class="MsoNormal">category_defaults =<o:p></o:p></p>
</div>
<div><p class="MsoNormal"> experiment |
airs_exp<o:p></o:p></p>
</div>
<div><p class="MsoNormal"> model | airs_model<o:p></o:p></p>
</div>
<div><p class="MsoNormal">dataset_id =
%(activity)s.%(product)s.%(agency)s.%(mission)s.%(instrument)s.%(time_frequency)s.%(realm)s.%(variable)s.%(level)s<o:p></o:p></p>
</div>
<div><p class="MsoNormal">dataset_name_format = AIRS
Level 3 Monthly Data (NetCDF)<o:p></o:p></p>
</div>
<div><p class="MsoNormal">directory_format =
/esg/data/%(activity)s/%(product)s/%(agency)s/%(mission)s/%(instrument)s/%(time_frequency)s/%(realm)s/%(variable)s/%(level)s/%(version)s<o:p></o:p></p>
</div>
<div><p class="MsoNormal">handler =
esgcet.config.netcdf_handler:NetcdfHandler<o:p></o:p></p>
</div>
<div><p class="MsoNormal">parent_id = nasa.airs<o:p></o:p></p>
</div>
<div><p class="MsoNormal">variable_per_file = true<o:p></o:p></p>
</div>
<div><p class="MsoNormal">las_configure = true<o:p></o:p></p>
</div>
<div><p class="MsoNormal">maps = las_time_delta_map<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div><p class="MsoNormal">In other words, I instructed
the publisher to use fictitious values for
experiment and model, but I also instructed it
to read all the relevant DRS fields from the
directory structure itself. That seemed to work.<o:p></o:p></p>
</div>
<div><p class="MsoNormal">Maybe Bob can comment if this
seems reasonable to him.<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">And I agree that a CMOR table
for observations needs to be developed. Could we
open up that process so we can include all
relevant values not just from ORNL, but also
NASA, NOAA and any agency who will contribute
observations ?<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">thanks a lot,<o:p></o:p></p>
</div>
<div><p class="MsoNormal">Luca<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
<div>
<div><p class="MsoNormal">On Jan 26, 2011, at 4:50 PM,
Renata McCoy wrote:<o:p></o:p></p>
</div><p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<div><p class="MsoNormal">Hi Luca,<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">I was experimenting with
setting up a publishing script for the ARM cmbe
observational data following the proposal below. I
see few issues:<o:p></o:p></p>
</div>
<div><p class="MsoNormal">I understand (I am cc-ing Bob
Drach to check on that) the publisher needs at
least those 3 fields for the "CMIP5' project:<o:p></o:p></p>
</div>
<div><p class="MsoNormal">project (which is 'CMIP5'),
experiment (that could be specified as 'none' in
command options, but maybe should be
'observations' ?), and model <o:p></o:p></p>
</div>
<div><p class="MsoNormal">A 'model' is an important field
and I was trying to set it up so as 'cmbe' would
be my 'model', which seems to be equivalent to
'airs' for NASA AIRS data<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">The other problem is that we
need a specific table (CMOR table) to establish
the controlled variable list and data frequency
that is obs specific.<o:p></o:p></p>
</div>
<div><p class="MsoNormal">We (with ORNL team) are trying
to create an obsCfSites table, that would be
mostly a copy of cfSites table with obs specific
time definition (for cmbe it's 1 hour average, for
CDIAC data - 30 min average, and for 3D data with
vertical axis - we would like to specify that any
pressure level could be reported). I think it
would be good if we could encompass all the
observational data needs in one observational
table. <o:p></o:p></p>
</div>
<div><p class="MsoNormal"> <o:p></o:p></p>
</div>
<div><p class="MsoNormal">I did not try to test publish
the data yet, I am working on rewriting the data
with the appropriate metadata first, so I am not
sure what other problems may pop up.<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div><p class="MsoNormal">Greetings,<o:p></o:p></p>
</div>
<div><p class="MsoNormal">Renata<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div><p class="MsoNormal"><span style="font-size: 13.5pt;
font-family:
"Helvetica","sans-serif";">------------------------------------------------------------<br>
Renata B. McCoy, Ph.D<br>
Program for Climate Model
Diagnosis and
Intercomparison(L-103)<br>
Lawrence Livermore National
Laboratory<br>
P.O. Box 808<br>
Livermore, CA 94551<br>
<br>
(925) 424-5237 (voice)<br>
(925) 422-7675 (fax)<br>
<a moz-do-not-send="true" href="mailto:mccoy20@llnl.gov">mccoy20@llnl.gov</a><br>
------------------------------------------------------------<o:p></o:p></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size: 13.5pt;
font-family:
"Helvetica","sans-serif";"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div><p class="MsoNormal"><o:p> </o:p></p>
<div>
<div><p class="MsoNormal">On Jan 26, 2011, at 2:50 PM,
Cinquini, Luca (3880) wrote:<o:p></o:p></p>
</div><p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div><p class="MsoNormal">Hi all,<o:p></o:p></p>
<div><p class="MsoNormal"><span class="apple-tab-span"> </span>apologies
for cross-posting...<o:p></o:p></p>
</div>
<div><p class="MsoNormal"><br>
I'd like to start a discussion on how the DRS
specification for CMIP5 model output (<a moz-do-not-send="true" href="http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf">http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf</a>)<br>
can be applied to observational datasets that
will be made part of the same archive. A
proposal based on recent workshops with PCMDI
is detailed on this wiki page:<br>
<br>
<u><span style="color: rgb(20, 79, 174);"><a moz-do-not-send="true" href="http://oodt.jpl.nasa.gov/wiki/display/CLIMATE/Data+and+Metadata+Requirements+for+CMIP5+Observational+Datasets">http://oodt.jpl.nasa.gov/wiki/display/CLIMATE/Data+and+Metadata+Requirements+for+CMIP5+Observational+Datasets</a></span></u><o:p></o:p></p>
</div>
<div><p class="MsoNormal"><u><span style="color:
rgb(20, 79, 174);"><br>
</span></u>As an example, the directory
structure for the NASA AIRS dataset would
look like this:<br>
<br>
<root
directory>/cmip5/observations/nasa/aqua/airs/mon/atmos/ta/l3/vYYYYMMDD/<files><br>
<br>
where:<br>
<br>
"observations"=<product>, same as DRS
"output1" or "output2"<br>
"nasa"=<agency>, same as DRS "institute"<br>
"aqua"=<mission>, replaces DRS "model"<br>
"airs"=<instrument>, replaces DRS
"experiment"<br>
"l3"=<processing level>, replaces DRS
"ensemble member"<br>
vYYYYMMDD=the dated version, same as specified
by the DRS<br>
<br>
The values for the various fields
<agency>, <mission>,
<instrument> and <processing
level> would need to be selected from a
controlled vocabulary similar to the one
established for models.<br>
<br>
Any comment or insight on the matter is
appreciated.... The idea is to try to finalize
the specification relatively quickly, let's
say a couple of weeks, so that we can start
preparing<br>
and publishing these observations into the
CMIP5 archive.<br>
<br>
thanks in advance,<br>
<br>
Luca<o:p></o:p></p>
<div><p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div><p class="MsoNormal"><o:p> </o:p></p>
</div>
</div><p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div><p class="MsoNormal"><o:p> </o:p></p>
</div>
<br><p>-- <br>
Scanned by iCritical.
</p>
<br>
<pre wrap=""><fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
</pre>
</div>
</blockquote></div><br></body></html>