<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font face="Times New Roman">Hi Estani, Stephen, et al.<br>
<br>
Charles and I looked at the CMOR2 code and confirmed that all
files that were successfully written should have two times in
their file names marking the first and last time slices written.
If a file has only one time written, this indicates it was a
*temporary* file that was opened by CMOR (at the time it attempted
to write the first time-slice), but it was never closed
successfully by CMOR. This means the file probably is either
incomplete or has errors and has not passed QCL1. <br>
<br>
CMOR's algorithm for generating file names is copied below. In
short, when we get to sub-6-hourly we include minutes (because
time-mean 3-hourly data, for example, is reported for periods
centered at 1.5, 4.5, 7.5 .... 22.5 hours, which would be recorded
in the filename as 0130, 0430, 0730, ... 2230), and sub-hourly
data could possibly be reported at times centered at half-minutes
(e.g, if reported every 25 minutes, perhaps at 12 min, 30 sec, 37
min 30 sec, etc.) By allowing for these possibilities, we felt it
best that all models include the same precision, even if the
finest unit was unnecessary in a particular case.<br>
<br>
The bottom line is that the number of digits should depend only on
the cmor table, not the model. Also, the DRS should be revised
because it implies that depending on when sampling occurs, you
might require more or less precision, when in fact the precision
is determined by the CMOR table (i.e., based only on the sampling
interval).<br>
<br>
Best regards,<br>
Karl<br>
<br>
<br>
Note: below "interval" is the approximate interval of time
between successive time-slices.<br>
<br>
</font>
<pre wrap=""> /* first time point */
strncat(outname,"_",CMOR_MAX_STRING-strlen(outname));
snprintf(msg2,CMOR_MAX_STRING,"%.4ld",comptime.year);
strncat(outname,msg2,CMOR_MAX_STRING-strlen(outname));
if (interval<29.E6) { /* less than a year */
snprintf(msg2,CMOR_MAX_STRING,"%.2i",comptime.month);
strncat(outname,msg2,CMOR_MAX_STRING-strlen(outname));
}
if (interval<2.E6) { /* less than a month */
snprintf(msg2,CMOR_MAX_STRING,"%.2i",comptime.day);
strncat(outname,msg2,CMOR_MAX_STRING-strlen(outname));
}
if (interval<86000) { /* less than a day */
snprintf(msg2,CMOR_MAX_STRING,"%.2i",(int)comptime.hour);
strncat(outname,msg2,CMOR_MAX_STRING-strlen(outname));
}
if (interval<21000) { /* less than 6hr */
/* from now on add 1 more level of precision since that frequency */
ierr = (int)((comptime.hour-(int)(comptime.hour))*60.);
snprintf(msg2,CMOR_MAX_STRING,"%.2i",ierr);
strncat(outname,msg2,CMOR_MAX_STRING-strlen(outname));
}
if (interval<3000) { /* less than an hour */
snprintf(msg2,CMOR_MAX_STRING,"%.2i",(int)((comptime.hour-(int)(comptime.hour))*3600.)-ierr*60);
strncat(outname,msg2,CMOR_MAX_STRING-strlen(outname));
}
/* separator between first and last time */
strncat(outname,"-",CMOR_MAX_STRING-strlen(outname));</pre>
<br>
<br>
On 2/9/12 9:10 AM, Estanislao Gonzalez wrote:
<blockquote cite="mid:4F33FE21.5090904@dkrz.de" type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
Hi Karl, Charles,<br>
<br>
AFAIK these files were created by CMOR. If this naming schema is
not valid, CMOR should be changed to forbid this from happening.
drslib is a utility tool, but CMOR is part of the QCL1, so this
shouldn't be allowed at all IMHO.<br>
<br>
By chance I stumble upon a subhourly file, which I wanted to
replicate, that has a time-slice defined up to the seconds. You
mentioned, just as what the documentation says, that the finest
resolution is minutes. Is this acceptable? Does it make sense?
(I'm still waiting for an answer if this was intentional at all,
since the seconds are always '00')<br>
<br>
Thanks,<br>
Estani<br>
<br>
<br>
Am 09.02.2012 17:48, schrieb <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:stephen.pascoe@stfc.ac.uk:">stephen.pascoe@stfc.ac.uk:</a>
<blockquote
cite="mid:4C353E6E4A08AE4792B350DAA392B5212671C904@EXCHMBX01.fed.cclrc.ac.uk"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<meta name="Generator" content="Microsoft Word 12 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks
Karl,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The
code is fixed in the repository to do the right thing with
only one time-slice but I'm glad we shouldn't be seeing
these files in general as other bits of the pipeline could
be affected.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Stephen.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span
style="font-size:10.5pt;font-family:Consolas;color:#1F497D">---<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.5pt;font-family:Consolas;color:#1F497D">Stephen
Pascoe +44 (0)1235 445980<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.5pt;font-family:Consolas;color:#1F497D">Centre
of Environmental Data Archival<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.5pt;font-family:Consolas;color:#1F497D">STFC
Rutherford Appleton Laboratory, Harwell Oxford, Didcot
OX11 0QX, UK<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"
lang="EN-US">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"
lang="EN-US"> Karl Taylor [<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="mailto:taylor13@llnl.gov">mailto:taylor13@llnl.gov</a>]
<br>
<b>Sent:</b> 09 February 2012 16:40<br>
<b>To:</b> Estanislao Gonzalez<br>
<b>Cc:</b> Pascoe, Stephen (STFC,RAL,RALSP); <a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:legutke@dkrz.de">legutke@dkrz.de</a>; <a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>;
Charles Doutriaux<br>
<b>Subject:</b> Re: [Go-essp-tech] DRS question -
temporal subset<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Estani and Charles,<br>
<br>
Unless only a single time-slice is stored, both the
beginning and ending times should be included in the file
name, so the file you are considering has an incorrect
name. The bit about "sufficient suffixes" relates to the
precision with which the times are specified (stopping, for
example, at month for monthly means, but going all the way
to minutes for sub-hourly). <br>
<br>
I don't think drslib needs to handle the case of a single
time-slice for CMIP5 because I think CMOR invariably
includes both the beginning and ending times (even if they
are identical). Were the files written with CMOR? <br>
<br>
Charles can tell us if I've said anything incorrect.<br>
<br>
thanks,<br>
Karl <br>
<br>
<br>
<br>
<br>
On 2/9/12 4:30 AM, Estanislao Gonzalez wrote: <o:p></o:p></p>
<pre>Sure,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>trans.filename_to_drs('clmcalipso_cf3hr_MPI-ESM-LR_amip_r1i1p1_200810221030.nc')<o:p></o:p></pre>
<pre>Traceback (most recent call last):<o:p></o:p></pre>
<pre> File "<stdin>", line 1, in <module><o:p></o:p></pre>
<pre> File <o:p></o:p></pre>
<pre>"/usr/local/cdat/lib/python2.6/site-packages/drslib-0.3.0a3-py2.6.egg/drslib/translate.py", <o:p></o:p></pre>
<pre>line 413, in filename_to_drs<o:p></o:p></pre>
<pre> t.filename_to_drs(context)<o:p></o:p></pre>
<pre> File <o:p></o:p></pre>
<pre>"/usr/local/cdat/lib/python2.6/site-packages/drslib-0.3.0a3-py2.6.egg/drslib/translate.py", <o:p></o:p></pre>
<pre>line 355, in filename_to_drs<o:p></o:p></pre>
<pre> context.drs.subset = (_to_date(n1), _to_date(n2), clim)<o:p></o:p></pre>
<pre> File <o:p></o:p></pre>
<pre>"/usr/local/cdat/lib/python2.6/site-packages/drslib-0.3.0a3-py2.6.egg/drslib/translate.py", <o:p></o:p></pre>
<pre>line 503, in _to_date<o:p></o:p></pre>
<pre> mo = re.match(r'(\d{4})(\d{2})?(\d{2})?(\d{2})?(\d{2})?', date_str)<o:p></o:p></pre>
<pre> File "/usr/local/cdat/lib/python2.6/re.py", line 137, in match<o:p></o:p></pre>
<pre> return _compile(pattern, flags).match(string)<o:p></o:p></pre>
<pre>TypeError: expected string or buffer<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>But I'm more concern with the question of how should it be. Missing that <o:p></o:p></pre>
<pre>info will cause that in some particular cases you can't infer the end <o:p></o:p></pre>
<pre>date, and thus the start date of the next junk, without knowing the <o:p></o:p></pre>
<pre>calender type. If you need to get the file to read the calender type and <o:p></o:p></pre>
<pre>*then* infer the name, the usefulness of having a naming scheme would be <o:p></o:p></pre>
<pre>diminished.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>As always I think we should comply with whatever rules and guidelines we <o:p></o:p></pre>
<pre>have, as a computer scientist I can't help but seeing the benefits of <o:p></o:p></pre>
<pre>homogeneity and the burden caused by exceptions.<o:p></o:p></pre>
<pre>That's why my question is: how should it be? I'm not sure I'm <o:p></o:p></pre>
<pre>interpreting the documentation properly.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks,<o:p></o:p></pre>
<pre>Estani<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Am 09.02.2012 13:14, schrieb <a moz-do-not-send="true" href="mailto:stephen.pascoe@stfc.ac.uk:">stephen.pascoe@stfc.ac.uk:</a><o:p></o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>I think drslib will cope -- if not it's a bug -- but the product deduction code often needs both time bounds.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Can you send me the error message.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks,<o:p></o:p></pre>
<pre>Stephen.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>---<o:p></o:p></pre>
<pre>Stephen Pascoe +44 (0)1235 445980<o:p></o:p></pre>
<pre>Centre of Environmental Data Archival<o:p></o:p></pre>
<pre>STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>-----Original Message-----<o:p></o:p></pre>
<pre>From: <a moz-do-not-send="true" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a moz-do-not-send="true" href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>] On Behalf Of Estanislao Gonzalez<o:p></o:p></pre>
<pre>Sent: 09 February 2012 11:06<o:p></o:p></pre>
<pre>To: <a moz-do-not-send="true" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><o:p></o:p></pre>
<pre>Cc: Stephanie Legutke<o:p></o:p></pre>
<pre>Subject: [Go-essp-tech] DRS question - temporal subset<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Hi all,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>This is what the DRS reference says:<o:p></o:p></pre>
<pre>"Temporal Subsets: Time instants and periods (N1(-N2))<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Time instants and periods will be represented by<o:p></o:p></pre>
<pre>‘yyyy[mm[dd[hh][mm]]][-clim]’, where ‘yyyy’,<o:p></o:p></pre>
<pre>‘mm’, ‘dd’, ‘hh’ ‘mm’ are integer year, month, day, hour, and minute,<o:p></o:p></pre>
<pre>respectively, and enough<o:p></o:p></pre>
<pre>(and just enough) of the suffixes should be added to unambiguously<o:p></o:p></pre>
<pre>resolve the interval between<o:p></o:p></pre>
<pre>time-samples contained in the file or virtual file URL. (For example,<o:p></o:p></pre>
<pre>monthly mean data would<o:p></o:p></pre>
<pre>include “mm”, but not “dd”, “hh”, or “mm”; “subhr” data would include<o:p></o:p></pre>
<pre>all suffixes.) The<o:p></o:p></pre>
<pre>optional “-clim” is appended when the file contains a climatology. For<o:p></o:p></pre>
<pre>example, a file with<o:p></o:p></pre>
<pre>sampling frequency of “mo” and the time designation 196001-198912-clim<o:p></o:p></pre>
<pre>represents the<o:p></o:p></pre>
<pre>monthly mean climatology (12 time values) computed for the period<o:p></o:p></pre>
<pre>extending from 1/1960-<o:p></o:p></pre>
<pre>12/1989. Note that the DRS does not explicitly specify the calendar type<o:p></o:p></pre>
<pre>(e.g., Julian,<o:p></o:p></pre>
<pre>Gregorian), but the calendar will be indicated by one of the attributes<o:p></o:p></pre>
<pre>in each netCDF file."[1]<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> From the title I infer that only the start point is required as part of<o:p></o:p></pre>
<pre>the temporal subset, though the text speaks about "enough information to<o:p></o:p></pre>
<pre>unambiguously resolve the interval between<o:p></o:p></pre>
<pre>time-samples".<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>The problem is that I get 3hr and subhr data with only the starting<o:p></o:p></pre>
<pre>time-stamp, the drslib cannot cope with this.<o:p></o:p></pre>
<pre>Should the drslib be adapted to accept that case or all files renamed?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>What about the rest of the system? Have anyone already tried publishing<o:p></o:p></pre>
<pre>such files? Make sense?<o:p></o:p></pre>
<pre>And what was the intention with the parenthesis in the Temporal Subsets<o:p></o:p></pre>
<pre>title?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks,<o:p></o:p></pre>
<pre>Estani<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>[1] - <a moz-do-not-send="true" href="http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf">http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> </pre>
</blockquote>
</body>
</html>