[Met_help] [rt.rap.ucar.edu #45240] History for Testing ensemble_stat

RAL HelpDesk {for John Halley Gotway} met_help at ucar.edu
Fri Mar 11 11:32:10 MST 2011


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

I'm having trouble getting ensemble_stat to work.  I run it from a batch file and get the following error:

Machine::read() -> unable to open input file "MANPATH=/usr/share/man:/usr/lpp/ssp/man:/usr/lpp/ssp/perl5/man"

I've never seen this error in grid_stat or any other MET routine.  Here's the batch file contents:

/sphome/qcteam/METv3.0/bin/ensemble_stat 10 /h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt /h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default -point_obs /h/data/global/WXQC/data/obs_prep/2011030918 -outdir /h/data/global/WXQC/data/met/stats/ensemble -v 2

And ens_files.txt contains my file names, each of the 10 space delimited.  The observation file is the NetCDF version of the PrepBufr formatted observation data from NCEP, converted via pb2nc.

Any thoughts as to what portion of MET might be throwing out the PATH variable issue?  That doesn't sound like anything ensemble_stat would worry about.

Thanks,
Matt

// SIGNED //
Matthew C. Sittel
Associate Scientist
University Corporation for Atmospheric Research
16WS/WXN, Offutt AFB, NE
(402) 294-3473  DSN: 271-3473



----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #45240] Testing ensemble_stat
From: John Halley Gotway
Time: Thu Mar 10 13:00:20 2011

Matt,

Please try running the following command instead:
/sphome/qcteam/METv3.0/bin/ensemble_stat
/h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt
/h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default
-point_obs
/h/data/global/WXQC/data/obs_prep/2011030918 -outdir
/h/data/global/WXQC/data/met/stats/ensemble -v 2

Notice that I've removed the "10" from the command line.  You can
specify the ensemble files in one of two ways:
(1) As "n_ens ens_file_1 ... ens_file_n" which is the number of files
followed by the file names themselves
(2) Or as "ens_file_list" which is an ASCII file containing the list
of file names

You're mixing the two of those methods together.  And looking at the
usage statement for Ensemble-Stat, I see why.  Here's a subset of it:

Usage: ensemble_stat
        n_ens ens_file_1 ... ens_file_n | ens_file_list

        where   "n_ens" is the number of ensemble members (required).
                "ens_file_1 ... ens_file_n" is a list of ensemble
member file names (required).
                "ens_file_list" is an ASCII file containing a list of
ensemble member file names (required).

I see why this setting is confusing.  I could change it to look like
this:

Usage: ensemble_stat
        n_ens ens_file_1 ... ens_file_n | ens_file_list

        where   "n_ens ens_file_1 ... ens_file_n" is the number of
ensemble members followed by a list of ensemble member file names
(required).
                "ens_file_list" is an ASCII file containing a list of
ensemble member file names (required).

That would probably be more clear.

Please let me know if removing "10" does the trick for you.

Thanks,
John

On 03/10/2011 12:12 PM, RAL HelpDesk {for
matthew.sittel at offutt.af.mil} wrote:
>
> Thu Mar 10 12:12:17 2011: Request 45240 was acted upon.
> Transaction: Ticket created by matthew.sittel at offutt.af.mil
>        Queue: met_help
>      Subject: Testing ensemble_stat
>        Owner: Nobody
>   Requestors: matthew.sittel at offutt.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45240 >
>
>
> I'm having trouble getting ensemble_stat to work.  I run it from a
batch file and get the following error:
>
> Machine::read() -> unable to open input file
"MANPATH=/usr/share/man:/usr/lpp/ssp/man:/usr/lpp/ssp/perl5/man"
>
> I've never seen this error in grid_stat or any other MET routine.
Here's the batch file contents:
>
> /sphome/qcteam/METv3.0/bin/ensemble_stat 10
/h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt
/h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default
-point_obs /h/data/global/WXQC/data/obs_prep/2011030918 -outdir
/h/data/global/WXQC/data/met/stats/ensemble -v 2
>
> And ens_files.txt contains my file names, each of the 10 space
delimited.  The observation file is the NetCDF version of the PrepBufr
formatted observation data from NCEP, converted via pb2nc.
>
> Any thoughts as to what portion of MET might be throwing out the
PATH variable issue?  That doesn't sound like anything ensemble_stat
would worry about.
>
> Thanks,
> Matt
>
> // SIGNED //
> Matthew C. Sittel
> Associate Scientist
> University Corporation for Atmospheric Research
> 16WS/WXN, Offutt AFB, NE
> (402) 294-3473  DSN: 271-3473
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #45240] Testing ensemble_stat
From: matthew.sittel at offutt.af.mil
Time: Fri Mar 11 06:42:40 2011

That solved the problem-thanks John!  

Note that I actually got 2
e-mails from you, so the met_help address works... sometimes!?
You're right on the documentation; I read that n_ens was required, and
took the '|' to mean that it was either a list of file names OR a text
file, but either way I had to explicitly state the number of files to
be read in.  That section of 3.9.1 is a bit confusing since it says 4
arguments are required, but the nature of them depends on how the list
of files is passed.  Far be it for me to comment on a User's Guide...
:)   But I *do* read them and wonder if I find something confusing
that others may be confused too.

Matt

-----Original Message-----
From: RAL HelpDesk {for John Halley Gotway} [mailto:met_help at ucar.edu]
Sent: Thursday, March 10, 2011 2:00 PM
To: Sittel, Matthew C CTR USAF
AFWA 16 WS/WXN
Cc: Craig, Robert J Civ USAF AFWA 16 WS/WXN;
met_help at ucar.edu
Subject: Re: [rt.rap.ucar.edu #45240] Testing
ensemble_stat

Matt,

Please try running the following command
instead:
/sphome/qcteam/METv3.0/bin/ensemble_stat
/h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt
/h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default
-point_obs
/h/data/global/WXQC/data/obs_prep/2011030918 -outdir
/h/data/global/WXQC/data/met/stats/ensemble -v 2

Notice that I've
removed the "10" from the command line.  You can specify the ensemble
files in one of two ways:
(1) As "n_ens ens_file_1 ... ens_file_n"
which is the number of files followed by the file names themselves
(2) Or as "ens_file_list" which is an ASCII file containing the list
of file names

You're mixing the two of those methods together.  And
looking at the usage statement for Ensemble-Stat, I see why.  Here's a
subset of it:

Usage: ensemble_stat
        n_ens ens_file_1 ...
ens_file_n | ens_file_list

        where   "n_ens" is the number of
ensemble members (required).
                "ens_file_1 ...
ens_file_n" is a list of ensemble member file names (required).
"ens_file_list" is an ASCII file containing a list of ensemble member
file names (required).

I see why this setting is confusing.  I
could change it to look like this:

Usage: ensemble_stat
n_ens ens_file_1 ... ens_file_n | ens_file_list

        where
"n_ens ens_file_1 ... ens_file_n" is the number of ensemble members
followed by a list of ensemble member file names (required).
"ens_file_list" is an ASCII file containing a list of ensemble member
file names (required).

That would probably be more clear.

Please
let me know if removing "10" does the trick for you.

Thanks,
John
On 03/10/2011 12:12 PM, RAL HelpDesk {for
matthew.sittel at offutt.af.mil} wrote:
> 
> Thu Mar 10 12:12:17 2011:
Request 45240 was acted upon.
> Transaction: Ticket created by
matthew.sittel at offutt.af.mil
>        Queue: met_help
>
Subject: Testing ensemble_stat
>        Owner: Nobody
>
Requestors: matthew.sittel at offutt.af.mil
>       Status: new
>
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45240 >
> 
> 
> I'm having trouble getting ensemble_stat to work.  I run it
from a batch file and get the following error:
> 
> Machine::read()
-> unable to open input file
"MANPATH=/usr/share/man:/usr/lpp/ssp/man:/usr/lpp/ssp/perl5/man"
>
> I've never seen this error in grid_stat or any other MET routine.
Here's the batch file contents:
> 
>
/sphome/qcteam/METv3.0/bin/ensemble_stat 10
/h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt
/h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default
-point_obs /h/data/global/WXQC/data/obs_prep/2011030918 -outdir
/h/data/global/WXQC/data/met/stats/ensemble -v 2
> 
> And
ens_files.txt contains my file names, each of the 10 space delimited.
The observation file is the NetCDF version of the PrepBufr formatted
observation data from NCEP, converted via pb2nc.
> 
> Any thoughts
as to what portion of MET might be throwing out the PATH variable
issue?  That doesn't sound like anything ensemble_stat would worry
about.
> 
> Thanks,
> Matt
> 
> // SIGNED //
> Matthew C. Sittel
> Associate Scientist
> University Corporation for Atmospheric
Research
> 16WS/WXN, Offutt AFB, NE
> (402) 294-3473  DSN: 271-3473
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #45240] Testing ensemble_stat
From: John Halley Gotway
Time: Fri Mar 11 08:55:19 2011

Matt,

Great, glad that did the trick.  And really, thanks for the feedback
about the user's guide.  I've made notes about the 3 issues you
pointed out and will fix them shortly.  We appreciate it.

I'll go ahead and resolve this ticket.  But please let us know if more
issues/questions come up in your use of MET.

Thanks,
John



On 03/11/2011 06:42 AM, RAL HelpDesk {for
matthew.sittel at offutt.af.mil} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45240 >
>
> That solved the problem-thanks John!
>
> Note that I actually got 2 e-mails from you, so the met_help address
works... sometimes!?
>
> You're right on the documentation; I read that n_ens was required,
and took the '|' to mean that it was either a list of file names OR a
text file, but either way I had to explicitly state the number of
files to be read in.  That section of 3.9.1 is a bit confusing since
it says 4 arguments are required, but the nature of them depends on
how the list of files is passed.  Far be it for me to comment on a
User's Guide... :)   But I *do* read them and wonder if I find
something confusing that others may be confused too.
>
> Matt
>
> -----Original Message-----
> From: RAL HelpDesk {for John Halley Gotway}
[mailto:met_help at ucar.edu]
> Sent: Thursday, March 10, 2011 2:00 PM
> To: Sittel, Matthew C CTR USAF AFWA 16 WS/WXN
> Cc: Craig, Robert J Civ USAF AFWA 16 WS/WXN; met_help at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #45240] Testing ensemble_stat
>
> Matt,
>
> Please try running the following command instead:
> /sphome/qcteam/METv3.0/bin/ensemble_stat
/h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt
/h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default
-point_obs
> /h/data/global/WXQC/data/obs_prep/2011030918 -outdir
/h/data/global/WXQC/data/met/stats/ensemble -v 2
>
> Notice that I've removed the "10" from the command line.  You can
specify the ensemble files in one of two ways:
> (1) As "n_ens ens_file_1 ... ens_file_n" which is the number of
files followed by the file names themselves
> (2) Or as "ens_file_list" which is an ASCII file containing the list
of file names
>
> You're mixing the two of those methods together.  And looking at the
usage statement for Ensemble-Stat, I see why.  Here's a subset of it:
>
> Usage: ensemble_stat
>         n_ens ens_file_1 ... ens_file_n | ens_file_list
>
>         where   "n_ens" is the number of ensemble members
(required).
>                 "ens_file_1 ... ens_file_n" is a list of ensemble
member file names (required).
>                 "ens_file_list" is an ASCII file containing a list
of ensemble member file names (required).
>
> I see why this setting is confusing.  I could change it to look like
this:
>
> Usage: ensemble_stat
>         n_ens ens_file_1 ... ens_file_n | ens_file_list
>
>         where   "n_ens ens_file_1 ... ens_file_n" is the number of
ensemble members followed by a list of ensemble member file names
(required).
>                 "ens_file_list" is an ASCII file containing a list
of ensemble member file names (required).
>
> That would probably be more clear.
>
> Please let me know if removing "10" does the trick for you.
>
> Thanks,
> John
>
> On 03/10/2011 12:12 PM, RAL HelpDesk {for
matthew.sittel at offutt.af.mil} wrote:
>>
>> Thu Mar 10 12:12:17 2011: Request 45240 was acted upon.
>> Transaction: Ticket created by matthew.sittel at offutt.af.mil
>>        Queue: met_help
>>      Subject: Testing ensemble_stat
>>        Owner: Nobody
>>   Requestors: matthew.sittel at offutt.af.mil
>>       Status: new
>>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45240 >
>>
>>
>> I'm having trouble getting ensemble_stat to work.  I run it from a
batch file and get the following error:
>>
>> Machine::read() -> unable to open input file
"MANPATH=/usr/share/man:/usr/lpp/ssp/man:/usr/lpp/ssp/perl5/man"
>>
>> I've never seen this error in grid_stat or any other MET routine.
Here's the batch file contents:
>>
>> /sphome/qcteam/METv3.0/bin/ensemble_stat 10
/h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt
/h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default
-point_obs /h/data/global/WXQC/data/obs_prep/2011030918 -outdir
/h/data/global/WXQC/data/met/stats/ensemble -v 2
>>
>> And ens_files.txt contains my file names, each of the 10 space
delimited.  The observation file is the NetCDF version of the PrepBufr
formatted observation data from NCEP, converted via pb2nc.
>>
>> Any thoughts as to what portion of MET might be throwing out the
PATH variable issue?  That doesn't sound like anything ensemble_stat
would worry about.
>>
>> Thanks,
>> Matt
>>
>> // SIGNED //
>> Matthew C. Sittel
>> Associate Scientist
>> University Corporation for Atmospheric Research
>> 16WS/WXN, Offutt AFB, NE
>> (402) 294-3473  DSN: 271-3473
>>
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #45240] Testing ensemble_stat
From: John Halley Gotway
Time: Fri Mar 11 09:41:42 2011

Matt,

Please try changing "fcst_field[] = [ "TMP/L2" ];" to "fcst_field[] =
[ "TMP/Z2" ];".  I know this is a very minor change but changing the
'L' to a 'Z' triggers a different set of logic in the code.
Namely, when you put 'Z2', that tells the code that this is really a
surface variable and should be matched up with observations of type
ADPSFC.  Hopefully that'll get you matched pairs.  Generally,
you should use 'Z2' to specify 2-meter fields (like TMP or specific
humidity) and 'Z10' for 10-meter winds.  And use a 'P' to specify
pressure levels, like 'P500' for 500mb fields.  The 'L' is used as
a "generic" level type - so it doesn't kick off the special processing
for surface fields.

Clear as mud!?

We do have an anonymous FTP site to which you could post data, but
perhaps we've already tried this.  The instructions for posting to it
can be found here:
   http://www.dtcenter.org/met/users/support/met_help.php#ftp
But I'm guessing your security rules will prevent you from accessing
that site.

Thanks,
John

On 03/11/2011 09:07 AM, Sittel, Matthew C CTR USAF AFWA 16 WS/WXN
wrote:
> Hi John,
>
> Got the met_help e-mail again... maybe we're functional after all on
the e-mail front and you don't need to double send.
>
> I still have an issue to figure out with ensemble_stat.  I'm using
Bob Craig's NetCDF data, brought down from NCEP and converted via
pb2nc.  When I run ensemble_stat it throws all of my observations and
all I get is a header output file with no actual numbers. The poly
mask I am using is CONUS, which is also the domain of the ensemble so
there should be something available after the poly mask is applied.
Bob uses the same files in point_stat and it works just fine... so I
can't figure out where I've made a mistake.  Excerpts from the log
file:
>
> Processing point observation file:
/h/data/global/WXQC/data/obs_prep/2011030918
> Searching 276268 observations from 67781 header messages.
>
> And then after it reads in my 10 ensemble members:
>
> Processing point verification TMP/L2 versus TMP/L2, for observation
type ADPSFC, over region FULL, for interpolation method UW_MEAN(1),
using 0 pairs.
> Processing point verification TMP/L2 versus TMP/L2, for observation
type ADPSFC, over region CONUS, for interpolation method UW_MEAN(1),
using 0 pairs.
>
> I know from doing an 'ncdump' on the data file and examining its
contents that there are ADPSFC records in the file.  What I'm unclear
on is if I have not set the obs_field[] array in the config file
correctly.  The forecasts are GriB data, but the obs are NetCDF.  I
set it as you see below to follow what Bob said worked when he ran
point_stat with it.  I've been confused by the explanation of how to
set the items in parentheses for NetCDF cases ever since I first read
it, since I and j aren't intuitive at all.  What dimensions do they
refer to if the last two (*,*) are the grid dimensions?  Well, that is
*if* it is gridded, because this file is not, thus I use the
-point_obs option in the command line.
>
> There's something I'm missing.  The NetCDF file is nearly 10 MB so
while I could e-mail it for you to test, I have no idea if AFWA will
properly send it to you through e-mail.  This is where we really could
use a reliable FTP connection between AFWA and NCAR to transfer some
files, but as you know most of this is done via sending DVDs in the
mail... frustrating.  Because I fear without you running all of the
files I have, we're really not replicating the error with the full
suite of files, config, model and obs, that I'm requesting be used in
my command line.
>
> Matt
>
> // Specifying verification fields for NetCDF files:
> //    var_name(i,...,j,*,*) for a single field
> //    Where var_name is the name of the NetCDF variable,
> //    and i,...,j specifies fixed dimension values,
> //    and *,* specifies the two dimensions for the gridded field.
> //
> // e.g. fcst_field[] = [ "61/A3", "RH/L10" ]; for GRIB input
> // e.g. fcst_field[] = [ "RAINC(0,*,*)", "QVAPOR(0,5,*,*)" ]; for
NetCDF input
> //
> fcst_field[] = [ "TMP/L2" ];
> obs_field[]  = [ "TMP/L2" ];
>
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at ucar.edu]
> Sent: Friday, March 11, 2011 9:55 AM
> To: met_help at ucar.edu; Sittel, Matthew C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu #45240] Testing ensemble_stat
>
> Matt,
>
> Great, glad that did the trick.  And really, thanks for the feedback
about the user's guide.  I've made notes about the 3 issues you
pointed out and will fix them shortly.  We appreciate it.
>
> I'll go ahead and resolve this ticket.  But please let us know if
more issues/questions come up in your use of MET.
>
> Thanks,
> John
>
>
>
> On 03/11/2011 06:42 AM, RAL HelpDesk {for
matthew.sittel at offutt.af.mil} wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45240 >
>>
>> That solved the problem-thanks John!
>>
>> Note that I actually got 2 e-mails from you, so the met_help
address works... sometimes!?
>>
>> You're right on the documentation; I read that n_ens was required,
and took the '|' to mean that it was either a list of file names OR a
text file, but either way I had to explicitly state the number of
files to be read in.  That section of 3.9.1 is a bit confusing since
it says 4 arguments are required, but the nature of them depends on
how the list of files is passed.  Far be it for me to comment on a
User's Guide... :)   But I *do* read them and wonder if I find
something confusing that others may be confused too.
>>
>> Matt
>>
>> -----Original Message-----
>> From: RAL HelpDesk {for John Halley Gotway}
[mailto:met_help at ucar.edu]
>> Sent: Thursday, March 10, 2011 2:00 PM
>> To: Sittel, Matthew C CTR USAF AFWA 16 WS/WXN
>> Cc: Craig, Robert J Civ USAF AFWA 16 WS/WXN; met_help at ucar.edu
>> Subject: Re: [rt.rap.ucar.edu #45240] Testing ensemble_stat
>>
>> Matt,
>>
>> Please try running the following command instead:
>> /sphome/qcteam/METv3.0/bin/ensemble_stat
/h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt
/h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default
-point_obs
>> /h/data/global/WXQC/data/obs_prep/2011030918 -outdir
/h/data/global/WXQC/data/met/stats/ensemble -v 2
>>
>> Notice that I've removed the "10" from the command line.  You can
specify the ensemble files in one of two ways:
>> (1) As "n_ens ens_file_1 ... ens_file_n" which is the number of
files followed by the file names themselves
>> (2) Or as "ens_file_list" which is an ASCII file containing the
list of file names
>>
>> You're mixing the two of those methods together.  And looking at
the usage statement for Ensemble-Stat, I see why.  Here's a subset of
it:
>>
>> Usage: ensemble_stat
>>         n_ens ens_file_1 ... ens_file_n | ens_file_list
>>
>>         where   "n_ens" is the number of ensemble members
(required).
>>                 "ens_file_1 ... ens_file_n" is a list of ensemble
member file names (required).
>>                 "ens_file_list" is an ASCII file containing a list
of ensemble member file names (required).
>>
>> I see why this setting is confusing.  I could change it to look
like this:
>>
>> Usage: ensemble_stat
>>         n_ens ens_file_1 ... ens_file_n | ens_file_list
>>
>>         where   "n_ens ens_file_1 ... ens_file_n" is the number of
ensemble members followed by a list of ensemble member file names
(required).
>>                 "ens_file_list" is an ASCII file containing a list
of ensemble member file names (required).
>>
>> That would probably be more clear.
>>
>> Please let me know if removing "10" does the trick for you.
>>
>> Thanks,
>> John
>>
>> On 03/10/2011 12:12 PM, RAL HelpDesk {for
matthew.sittel at offutt.af.mil} wrote:
>>>
>>> Thu Mar 10 12:12:17 2011: Request 45240 was acted upon.
>>> Transaction: Ticket created by matthew.sittel at offutt.af.mil
>>>        Queue: met_help
>>>      Subject: Testing ensemble_stat
>>>        Owner: Nobody
>>>   Requestors: matthew.sittel at offutt.af.mil
>>>       Status: new
>>>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45240 >
>>>
>>>
>>> I'm having trouble getting ensemble_stat to work.  I run it from a
batch file and get the following error:
>>>
>>> Machine::read() -> unable to open input file
"MANPATH=/usr/share/man:/usr/lpp/ssp/man:/usr/lpp/ssp/perl5/man"
>>>
>>> I've never seen this error in grid_stat or any other MET routine.
Here's the batch file contents:
>>>
>>> /sphome/qcteam/METv3.0/bin/ensemble_stat 10
/h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt
/h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default
-point_obs /h/data/global/WXQC/data/obs_prep/2011030918 -outdir
/h/data/global/WXQC/data/met/stats/ensemble -v 2
>>>
>>> And ens_files.txt contains my file names, each of the 10 space
delimited.  The observation file is the NetCDF version of the PrepBufr
formatted observation data from NCEP, converted via pb2nc.
>>>
>>> Any thoughts as to what portion of MET might be throwing out the
PATH variable issue?  That doesn't sound like anything ensemble_stat
would worry about.
>>>
>>> Thanks,
>>> Matt
>>>
>>> // SIGNED //
>>> Matthew C. Sittel
>>> Associate Scientist
>>> University Corporation for Atmospheric Research
>>> 16WS/WXN, Offutt AFB, NE
>>> (402) 294-3473  DSN: 271-3473
>>>
>>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #45240] Testing ensemble_stat
From: matthew.sittel at offutt.af.mil
Time: Fri Mar 11 10:58:39 2011

Once again, that did the trick!  You're pretty good at this MET
stuff...

What we do here to extract GriB fields is simply key on
the kpds5, 6 and 7 values for each record and equate a variable and
pressure level to a set of kpds5/6/7 values, since sometimes they
differ from model to model (e.g., precip is 61 or 240)... I was hoping
I could explicitly state them in some way to distinguish from surface
temperature vs. the 2-meter value.  I hoped L2 was specifically
controlling kpds7 when I invoked it.  I guess not!

Hey, the FTP
seemed to work!  I put 4 simple files (config stuff) out there to see
if you can view them on your end in a 'sittel_data' directory.  Let me
know if you can see them, and we'll be able to use that should we need
to exchange files.

Matt

-----Original Message-----
From: RAL
HelpDesk {for John Halley Gotway} [mailto:met_help at ucar.edu] 
Sent:
Friday, March 11, 2011 10:42 AM
To: Sittel, Matthew C CTR USAF AFWA
16 WS/WXN
Cc: Craig, Robert J Civ USAF AFWA 16 WS/WXN;
met_help at ucar.edu
Subject: Re: [rt.rap.ucar.edu #45240] Testing
ensemble_stat

Matt,

Please try changing "fcst_field[] = [
"TMP/L2" ];" to "fcst_field[] = [ "TMP/Z2" ];".  I know this is a very
minor change but changing the 'L' to a 'Z' triggers a different set of
logic in the code.
Namely, when you put 'Z2', that tells the code
that this is really a surface variable and should be matched up with
observations of type ADPSFC.  Hopefully that'll get you matched pairs.
Generally,
you should use 'Z2' to specify 2-meter fields (like TMP or
specific humidity) and 'Z10' for 10-meter winds.  And use a 'P' to
specify pressure levels, like 'P500' for 500mb fields.  The 'L' is
used as
a "generic" level type - so it doesn't kick off the special
processing for surface fields.

Clear as mud!?

We do have an
anonymous FTP site to which you could post data, but perhaps we've
already tried this.  The instructions for posting to it can be found
here:
   http://www.dtcenter.org/met/users/support/met_help.php#ftp
But I'm guessing your security rules will prevent you from accessing
that site.

Thanks,
John

On 03/11/2011 09:07 AM, Sittel, Matthew
C CTR USAF AFWA 16 WS/WXN wrote:
> Hi John,
> 
> Got the met_help
e-mail again... maybe we're functional after all on the e-mail front
and you don't need to double send.
> 
> I still have an issue to
figure out with ensemble_stat.  I'm using Bob Craig's NetCDF data,
brought down from NCEP and converted via pb2nc.  When I run
ensemble_stat it throws all of my observations and all I get is a
header output file with no actual numbers. The poly mask I am using is
CONUS, which is also the domain of the ensemble so there should be
something available after the poly mask is applied.  Bob uses the same
files in point_stat and it works just fine... so I can't figure out
where I've made a mistake.  Excerpts from the log file:
> 
>
Processing point observation file:
/h/data/global/WXQC/data/obs_prep/2011030918
> Searching 276268
observations from 67781 header messages.
> 
> And then after it
reads in my 10 ensemble members:
> 
> Processing point verification
TMP/L2 versus TMP/L2, for observation type ADPSFC, over region FULL,
for interpolation method UW_MEAN(1), using 0 pairs.
> Processing
point verification TMP/L2 versus TMP/L2, for observation type ADPSFC,
over region CONUS, for interpolation method UW_MEAN(1), using 0 pairs.
> 
> I know from doing an 'ncdump' on the data file and examining its
contents that there are ADPSFC records in the file.  What I'm unclear
on is if I have not set the obs_field[] array in the config file
correctly.  The forecasts are GriB data, but the obs are NetCDF.  I
set it as you see below to follow what Bob said worked when he ran
point_stat with it.  I've been confused by the explanation of how to
set the items in parentheses for NetCDF cases ever since I first read
it, since I and j aren't intuitive at all.  What dimensions do they
refer to if the last two (*,*) are the grid dimensions?  Well, that is
*if* it is gridded, because this file is not, thus I use the
-point_obs option in the command line.
> 
> There's something I'm
missing.  The NetCDF file is nearly 10 MB so while I could e-mail it
for you to test, I have no idea if AFWA will properly send it to you
through e-mail.  This is where we really could use a reliable FTP
connection between AFWA and NCAR to transfer some files, but as you
know most of this is done via sending DVDs in the mail... frustrating.
Because I fear without you running all of the files I have, we're
really not replicating the error with the full suite of files, config,
model and obs, that I'm requesting be used in my command line.
> 
>
Matt
> 
> // Specifying verification fields for NetCDF files:
> //
var_name(i,...,j,*,*) for a single field
> //    Where var_name is
the name of the NetCDF variable,
> //    and i,...,j specifies fixed
dimension values,
> //    and *,* specifies the two dimensions for
the gridded field.
> //
> // e.g. fcst_field[] = [ "61/A3", "RH/L10"
]; for GRIB input
> // e.g. fcst_field[] = [ "RAINC(0,*,*)",
"QVAPOR(0,5,*,*)" ]; for NetCDF input
> //
> fcst_field[] = [
"TMP/L2" ];
> obs_field[]  = [ "TMP/L2" ];
> 
> -----Original
Message-----
> From: John Halley Gotway [mailto:johnhg at ucar.edu] 
>
Sent: Friday, March 11, 2011 9:55 AM
> To: met_help at ucar.edu; Sittel,
Matthew C CTR USAF AFWA 16 WS/WXN
> Subject: Re: [rt.rap.ucar.edu
#45240] Testing ensemble_stat
> 
> Matt,
> 
> Great, glad that did
the trick.  And really, thanks for the feedback about the user's
guide.  I've made notes about the 3 issues you pointed out and will
fix them shortly.  We appreciate it.
> 
> I'll go ahead and resolve
this ticket.  But please let us know if more issues/questions come up
in your use of MET.
> 
> Thanks,
> John
> 
> 
> 
> On
03/11/2011 06:42 AM, RAL HelpDesk {for matthew.sittel at offutt.af.mil}
wrote:
>>
>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45240 >
>>
>> That
solved the problem-thanks John!  
>>
>> Note that I actually got 2
e-mails from you, so the met_help address works... sometimes!?
>>
>>
You're right on the documentation; I read that n_ens was required, and
took the '|' to mean that it was either a list of file names OR a text
file, but either way I had to explicitly state the number of files to
be read in.  That section of 3.9.1 is a bit confusing since it says 4
arguments are required, but the nature of them depends on how the list
of files is passed.  Far be it for me to comment on a User's Guide...
:)   But I *do* read them and wonder if I find something confusing
that others may be confused too.
>>
>> Matt
>>
>> -----Original
Message-----
>> From: RAL HelpDesk {for John Halley Gotway}
[mailto:met_help at ucar.edu] 
>> Sent: Thursday, March 10, 2011 2:00 PM
>> To: Sittel, Matthew C CTR USAF AFWA 16 WS/WXN
>> Cc: Craig, Robert
J Civ USAF AFWA 16 WS/WXN; met_help at ucar.edu
>> Subject: Re:
[rt.rap.ucar.edu #45240] Testing ensemble_stat
>>
>> Matt,
>>
>>
Please try running the following command instead:
>>
/sphome/qcteam/METv3.0/bin/ensemble_stat
/h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt
/h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default
-point_obs
>> /h/data/global/WXQC/data/obs_prep/2011030918 -outdir
/h/data/global/WXQC/data/met/stats/ensemble -v 2
>>
>> Notice that
I've removed the "10" from the command line.  You can specify the
ensemble files in one of two ways:
>> (1) As "n_ens ens_file_1 ...
ens_file_n" which is the number of files followed by the file names
themselves
>> (2) Or as "ens_file_list" which is an ASCII file
containing the list of file names
>>
>> You're mixing the two of
those methods together.  And looking at the usage statement for
Ensemble-Stat, I see why.  Here's a subset of it:
>>
>> Usage:
ensemble_stat
>>         n_ens ens_file_1 ... ens_file_n |
ens_file_list
>>
>>         where   "n_ens" is the number of
ensemble members (required).
>>                 "ens_file_1 ...
ens_file_n" is a list of ensemble member file names (required).
>>
"ens_file_list" is an ASCII file containing a list of ensemble member
file names (required).
>>
>> I see why this setting is confusing.  I
could change it to look like this:
>>
>> Usage: ensemble_stat
>>
n_ens ens_file_1 ... ens_file_n | ens_file_list
>>
>>         where
"n_ens ens_file_1 ... ens_file_n" is the number of ensemble members
followed by a list of ensemble member file names (required).
>>
"ens_file_list" is an ASCII file containing a list of ensemble member
file names (required).
>>
>> That would probably be more clear.
>>
>> Please let me know if removing "10" does the trick for you.
>>
>>
Thanks,
>> John
>>
>> On 03/10/2011 12:12 PM, RAL HelpDesk {for
matthew.sittel at offutt.af.mil} wrote:
>>>
>>> Thu Mar 10 12:12:17
2011: Request 45240 was acted upon.
>>> Transaction: Ticket created
by matthew.sittel at offutt.af.mil
>>>        Queue: met_help
>>>
Subject: Testing ensemble_stat
>>>        Owner: Nobody
>>>
Requestors: matthew.sittel at offutt.af.mil
>>>       Status: new
>>>
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45240 >
>>>
>>>
>>> I'm having trouble getting ensemble_stat to work.  I run
it from a batch file and get the following error:
>>>
>>>
Machine::read() -> unable to open input file
"MANPATH=/usr/share/man:/usr/lpp/ssp/man:/usr/lpp/ssp/perl5/man"
>>>
>>> I've never seen this error in grid_stat or any other MET routine.
Here's the batch file contents:
>>>
>>>
/sphome/qcteam/METv3.0/bin/ensemble_stat 10
/h/data/global/WXQC/data/met/cfg/ensemble/ens_files.txt
/h/data/global/WXQC/data/met/cfg/ensemble/ensemble_stat_cfg.default
-point_obs /h/data/global/WXQC/data/obs_prep/2011030918 -outdir
/h/data/global/WXQC/data/met/stats/ensemble -v 2
>>>
>>> And
ens_files.txt contains my file names, each of the 10 space delimited.
The observation file is the NetCDF version of the PrepBufr formatted
observation data from NCEP, converted via pb2nc.
>>>
>>> Any
thoughts as to what portion of MET might be throwing out the PATH
variable issue?  That doesn't sound like anything ensemble_stat would
worry about.
>>>
>>> Thanks,
>>> Matt
>>>
>>> // SIGNED //
>>>
Matthew C. Sittel
>>> Associate Scientist
>>> University Corporation
for Atmospheric Research
>>> 16WS/WXN, Offutt AFB, NE
>>> (402) 294-
3473  DSN: 271-3473
>>>
>>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #45240] Testing ensemble_stat
From: John Halley Gotway
Time: Fri Mar 11 11:31:41 2011

Matt,

Yep, I see them in
ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data.  So we should be
able to use that for exchanging data.

As for distinguishing between 2-meter temperature and surface
temperature, try using "TMP/Z2" for the former and "TMP/Z0" for the
latter.  I think that'd do the trick for picking out the input fields.
 The problem though is that we really don't have a way of filtering
the point observations to distinguish between 2-meter temp and surface
temp.  My understanding is that the ADPSFC observations of
temperature are all really 2-meter temperature.

But the "TMP/Z2" and "TMP/Z0" thing should allow you to distinguish
the gridded fields in MET - perhaps when you're running Grid-Stat or
MODE.  FYI, when you're setting up these MET tasks to run
initially, try using "-v 3" or "-v 4" on the command lines.  That'll
increase the level of logging.  You'll be able to explicitly see which
GRIB records the MET tool is reading, and you can verify
that it's using the ones you intended.

Thanks,
John

------------------------------------------------


More information about the Met_help mailing list