<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
A small editing suggestion: that this DRS extension should be
referred to a an extension for **gridded observations**. The term
"observations", standing alone, would be misunderstood by some
climate-relevant communities.<br>
<br>
I just read through the wiki proposal and could not find an explicit
discussion of spatial sampling concepts. Those seem to be implicit
in statements like "enforces a proper syntax for coordinate axes and
grids". (The statement "Datasets should be organized as
time-series" seems potentially misleading, no?) A few additional
words to help reach a wider audience might be worthwhile.<br>
<br>
- Steve<br>
<br>
=================================<br>
<br>
On 1/27/2011 7:03 AM, Cinquini, Luca (3880) wrote:
<blockquote
cite="mid:CAA6C926-FE34-421C-8CB7-48E27C13BDFC@jpl.nasa.gov"
type="cite">Hi Sebastian,
<div><span class="Apple-tab-span" style="white-space: pre;"> </span>thanks
for your summary of the current status, it all seems correct to
me. A few comments regarding the 3 points below:</div>
<div><br>
</div>
<div>1) Stephen Pascoe pointed out that the latest agreement is to
derive the dataset version number from the date, for example
v20100105 for data published on January 5th. I updated the wiki
proposal to reflect this:</div>
<div><br>
</div>
<div><a moz-do-not-send="true"
href="http://oodt.jpl.nasa.gov/wiki/display/CLIMATE/Data+and+Metadata+Requirements+for+CMIP5+Observational+Datasets">http://oodt.jpl.nasa.gov/wiki/display/CLIMATE/Data+and+Metadata+Requirements+for+CMIP5+Observational+Datasets</a></div>
<div><br>
</div>
<div>2) I agree we should focus on formalizing the DRS Controlled
Vocabulary for obs via document sharing tools... when we have a
first version, maybe PCMDI can encode it in a CMOR table ?</div>
<div>3) Ok, let's use [project:cmip5] in esg.ini. We might be able
to share the same snippet of configuration across all agencies
involved.</div>
<div><br>
</div>
<div>thanks, Luca</div>
<div>
<div><br>
</div>
<div>On Jan 27, 2011, at 7:41 AM, Sébastien Denvil wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div text="#000000" bgcolor="#ffffff"> Hello Luca, all,<br>
<br>
within CFMIP-obs context IPSL plan to publish observational
datasets that will be made part of the CMIP5 database. We
have up to now identify:<br>
- CALIPSO-GOCCP (monthly & daily)<br>
- PARASOL (monthly & daily)<br>
- CloudSat diagnostics (consistent with COSP output), which
are:<br>
- CFADs (radar reflectivity histograms function of
altitude) prepared by Roj Marchand (CSU) <br>
- ISCCP diagnostics consistent with COSP outputs.
Prepared by Yuying Zhang (PCMDI) and Rob Pincus (CIRES)<br>
- SIRTA-Cabauw sites database developed within the EU FP7
EUCLIPSE project, prepared by Chiriaco & Haeffelin
(IPSL). Could be included in the obsCfSites : in principle
close to the ARM stations case.<br>
<br>
To resume what I understood from this thread. There are 3
documents to produce:<br>
1) agree on a DRS suitable for observational datasets
(Luca's proposal looks good to me)<br>
2) agree on a set of new cmor table (possibly to be included
in the cmip5 cmor table git repository)<br>
3) agree on a esg.ini set up able to publish observational
datasets the way we want to<br>
<br>
Regarding esg.ini what Renata suggested seems to me like a
good idea:<br>
"to make it a CMIP5 project then your definitions should be
in the section
<div>[project:cmip5]</div>
<div>which then means extending the existing cmip5
definitions and options, that mostly seems to be not a
problem."<br>
</div>
<br>
Setting up new table may prove to be the "hard point". Work
on a Google Docs page with the CMOR table spreadsheet could
ease the process.<br>
<br>
Regards.<br>
Sébastien<br>
<br>
On 27/01/2011 14:28, <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:martin.juckes@stfc.ac.uk">martin.juckes@stfc.ac.uk</a>
wrote:
<blockquote
cite="mid:E21FBC3F00D7304687CB46529F9676D7039DD9@EXCHMBX01.fed.cclrc.ac.uk"
type="cite">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Lucida Grande";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size: 11pt;
font-family:
"Calibri","sans-serif"; color:
rgb(31, 73, 125);">Hello All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family:
"Calibri","sans-serif"; color:
rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family:
"Calibri","sans-serif"; color:
rgb(31, 73, 125);">This is an interesting discussion
– I’ve copied Victoria in because she is involved in
discussions with the ESA Climate Change Initiative
about data formats etc for the climate datasets ESA
is planning to produce. It would be really good to
try to coordinate efforts, at least up to the level
of agreeing a set of controlled vocabularies to
identify datasets, and agreeing what file attributes
will be used to hold that information in the file.
Luca’s wiki page lays out the issues clearly – so
perhaps we should see what the ESA CCI group think.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family:
"Calibri","sans-serif"; color:
rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family:
"Calibri","sans-serif"; color:
rgb(31, 73, 125);">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family:
"Calibri","sans-serif"; color:
rgb(31, 73, 125);">Martin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family:
"Calibri","sans-serif"; color:
rgb(31, 73, 125);"><o:p> </o:p></span></p>
<div>
<div style="border-width: 1pt medium medium;
border-style: solid none none; border-color:
rgb(181, 196, 223) -moz-use-text-color
-moz-use-text-color; padding: 3pt 0cm 0cm;">
<p class="MsoNormal"><b><span style="font-size:
10pt; font-family:
"Tahoma","sans-serif";"
lang="EN-US">From:</span></b><span
style="font-size: 10pt; font-family:
"Tahoma","sans-serif";"
lang="EN-US"> <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a>
[<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>]
<b>On Behalf Of </b>Renata McCoy<br>
<b>Sent:</b> 27 January 2011 00:42<br>
<b>To:</b> Cinquini, Luca (3880)<br>
<b>Cc:</b> <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>;
climate-obs; Giri Palanisamy; Renata McCoy<br>
<b>Subject:</b> Re: [Go-essp-tech] Extending the
DRS syntax to observations<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span class="apple-style-span"><span
style="font-size: 13.5pt;">Hi Luca,</span></span><span
style="font-size: 13.5pt; font-family:
"Arial","sans-serif"; color:
black;"><br>
<br>
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I see that you defined your
project as an AIRS project. If we want to make it a
CMIP5 project then your definitions should be in the
section<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[project:cmip5]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">which then means extending the
existing cmip5 definitions and options, that mostly
seems to be not a problem. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Giri set up a Google Docs page
with the CMOR table spreadsheet. If he doesn't mind,
we could all work on it. I will work on it tomorrow
to clean it up. Right now it is very ARM specific,
but we discussed about changing that and making it
all observation inclusive. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Renata<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Jan 26, 2011, at 4:19 PM,
Cinquini, Luca (3880) wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">Hi Renata,<o:p></o:p></p>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>I'm glad you are experimenting with the
same issues...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I actually also experimented
with using the ESG publisher to publish AIRS
data that was organized in a DRS-like fashion
like I mentioned:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><root
directory>/cmip5/observations/nasa/aqua/airs/mon/atmos/ta/l3/vYYYYMMDD/<files><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I was able to succeed
by using the following snippet of code
in esg.ini:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">[project:AIRS]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">category_defaults =<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> experiment
| airs_exp<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> model |
airs_model<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">dataset_id =
%(activity)s.%(product)s.%(agency)s.%(mission)s.%(instrument)s.%(time_frequency)s.%(realm)s.%(variable)s.%(level)s<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">dataset_name_format
= AIRS Level 3 Monthly Data (NetCDF)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">directory_format =
/esg/data/%(activity)s/%(product)s/%(agency)s/%(mission)s/%(instrument)s/%(time_frequency)s/%(realm)s/%(variable)s/%(level)s/%(version)s<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">handler =
esgcet.config.netcdf_handler:NetcdfHandler<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">parent_id =
nasa.airs<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">variable_per_file =
true<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">las_configure =
true<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">maps =
las_time_delta_map<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">In other words, I
instructed the publisher to use
fictitious values for experiment and
model, but I also instructed it to read
all the relevant DRS fields from the
directory structure itself. That seemed
to work.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Maybe Bob can comment
if this seems reasonable to him.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">And I agree that a
CMOR table for observations needs to be
developed. Could we open up that process
so we can include all relevant values
not just from ORNL, but also NASA, NOAA
and any agency who will contribute
observations ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">thanks a lot,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Luca<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Jan 26, 2011, at 4:50
PM, Renata McCoy wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Hi Luca,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I was experimenting
with setting up a publishing script for
the ARM cmbe observational data following
the proposal below. I see few issues:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I understand (I am
cc-ing Bob Drach to check on that) the
publisher needs at least those 3 fields
for the "CMIP5' project:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">project (which is
'CMIP5'), experiment (that could be
specified as 'none' in command options,
but maybe should be 'observations' ?),
and model <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">A 'model' is an
important field and I was trying to set it
up so as 'cmbe' would be my 'model', which
seems to be equivalent to 'airs' for NASA
AIRS data<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The other problem is
that we need a specific table (CMOR table)
to establish the controlled variable list
and data frequency that is obs specific.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">We (with ORNL team) are
trying to create an obsCfSites table, that
would be mostly a copy of cfSites table
with obs specific time definition (for
cmbe it's 1 hour average, for CDIAC data -
30 min average, and for 3D data with
vertical axis - we would like to specify
that any pressure level could be
reported). I think it would be good if we
could encompass all the observational data
needs in one observational table. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I did not try to test
publish the data yet, I am working on
rewriting the data with the appropriate
metadata first, so I am not sure what
other problems may pop up.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Greetings,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Renata<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:
13.5pt; font-family:
"Helvetica","sans-serif";">------------------------------------------------------------<br>
Renata B. McCoy, Ph.D<br>
Program for Climate
Model Diagnosis and
Intercomparison(L-103)<br>
Lawrence Livermore
National Laboratory<br>
P.O. Box 808<br>
Livermore, CA 94551<br>
<br>
(925) 424-5237 (voice)<br>
(925) 422-7675 (fax)<br>
<a
moz-do-not-send="true"
href="mailto:mccoy20@llnl.gov">mccoy20@llnl.gov</a><br>
------------------------------------------------------------<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:
13.5pt; font-family:
"Helvetica","sans-serif";"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Jan 26, 2011, at
2:50 PM, Cinquini, Luca (3880) wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<div>
<p class="MsoNormal"><span
class="apple-tab-span"> </span>apologies
for cross-posting...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
I'd like to start a discussion on how
the DRS specification for CMIP5
model output (<a
moz-do-not-send="true"
href="http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf">http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf</a>)<br>
can be applied to observational
datasets that will be made part of the
same archive. A proposal based on
recent workshops with PCMDI is
detailed on this wiki page:<br>
<br>
<u><span style="color: rgb(20, 79,
174);"><a moz-do-not-send="true"
href="http://oodt.jpl.nasa.gov/wiki/display/CLIMATE/Data+and+Metadata+Requirements+for+CMIP5+Observational+Datasets">http://oodt.jpl.nasa.gov/wiki/display/CLIMATE/Data+and+Metadata+Requirements+for+CMIP5+Observational+Datasets</a></span></u><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><u><span
style="color: rgb(20, 79, 174);"><br>
</span></u>As an example, the
directory structure for the NASA AIRS
dataset would look like this:<br>
<br>
<root
directory>/cmip5/observations/nasa/aqua/airs/mon/atmos/ta/l3/vYYYYMMDD/<files><br>
<br>
where:<br>
<br>
"observations"=<product>, same
as DRS "output1" or "output2"<br>
"nasa"=<agency>, same as DRS
"institute"<br>
"aqua"=<mission>, replaces DRS
"model"<br>
"airs"=<instrument>, replaces
DRS "experiment"<br>
"l3"=<processing level>,
replaces DRS "ensemble member"<br>
vYYYYMMDD=the dated version, same as
specified by the DRS<br>
<br>
The values for the various fields
<agency>, <mission>,
<instrument> and <processing
level> would need to be selected
from a controlled vocabulary similar
to the one established for models.<br>
<br>
Any comment or insight on the matter
is appreciated.... The idea is to try
to finalize the specification
relatively quickly, let's say a couple
of weeks, so that we can start
preparing<br>
and publishing these observations into
the CMIP5 archive.<br>
<br>
thanks in advance,<br>
<br>
Luca<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<p>-- <br>
Scanned by iCritical. </p>
<br>
<pre wrap=""><fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
</pre>
</div>
</blockquote>
</div>
<br>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
</body>
</html>