[Met_help] [rt.rap.ucar.edu #89694] History for handling missing data

Thu Apr 25 14:41:34 MDT 2019

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

I'm working with grid_stat, and I'd like to set this up so that I have one
file (based on valid time) for multiple fields, levels and leads.  I'm using
python embedding, so in the config I have multiple fields like.

{

       name       = "/users/satterfield/scripts/python/read_NRL_binary.py
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
.0_0000.0_glob360x181_2016010400_00960000_fcstfld";

       level      = ["P850"];

       cat_thresh = [ NA ];

     },

     {

       name       = "/users/satterfield/scripts/python/read_NRL_binary.py
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
.0_0000.0_glob360x181_2016010500_01200000_fcstfld";

       level      = ["P850"];

       cat_thresh = [ NA ];

     }

The issue that I have is that if a single field corresponds to a file that
is missing, I get an error and the stat file is empty. Is there a way to
handle this so that a file is produced and missing data is omitted?

Thanks,

Liz 

Elizabeth Satterfield
Editor, Monthly Weather Review

Chair, AMS Committee on Probability and Statistics

Meteorologist, Code 7531
Marine Meteorology Division
U.S. Naval Research Laboratory
T 831.656.4056  F 831.656.4026
 <http://www.nrl.navy.mil/> www.nrl.navy.mil

__________________________________________________________________
FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
Any misuse or unauthorized disclosure may result in both civil and criminal
penalties.

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: handling missing data
From: John Halley Gotway
Time: Mon Apr 15 12:41:59 2019

Liz,

When you run Grid-Stat, you typically pass in a forecast file and an
observation file.  If either of those files are missing, then Grid-
Stat
errors out and does not produce an output file.  This is an error
condition.

If both of those files do exist, then Grid-Stat steps through the
verification tasks defined in the config file.  If one of the fields
can't
be found in an input file, it prints a warning message and continues
to the
next task.  So this is a warning condition.

In your case, you're using the python embedding.  And it's probably
the
case that if whenever one of the python scripts fail, then Grid-Stat
errors
out entirely.  Here's what I see when the python script has problems:

*ERROR  : ERROR  : python_dataplane() -> an error occurred importing
module
"/d1/johnhg/MET/MET_development/svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
<http://svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py>"ERROR
: *

Is this the error message you're talking about?

Also note that with python embedding, setting the level value has no
impact:
*level = ["P850"];*

The metadata, including the level, is set by the Python script, not
the
config file.

Are you saying that you're prefer the error referenced above to be a
warning instead?

Thanks,
John

On Thu, Apr 11, 2019 at 3:19 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:

>
> Thu Apr 11 15:18:50 2019: Request 89694 was acted upon.
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
>        Queue: met_help
>      Subject: handling missing data
>        Owner: Nobody
>   Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
>
>
> I'm working with grid_stat, and I'd like to set this up so that I
have one
> file (based on valid time) for multiple fields, levels and leads.
I'm
> using
> python embedding, so in the config I have multiple fields like.
>
>
>
> {
>
>        name       =
"/users/satterfield/scripts/python/read_NRL_binary.py
>
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> .0_0000.0_glob360x181_2016010400_00960000_fcstfld";
>
>        level      = ["P850"];
>
>        cat_thresh = [ NA ];
>
>      },
>
>      {
>
>        name       =
"/users/satterfield/scripts/python/read_NRL_binary.py
>
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> .0_0000.0_glob360x181_2016010500_01200000_fcstfld";
>
>        level      = ["P850"];
>
>        cat_thresh = [ NA ];
>
>      }
>
>
>
> The issue that I have is that if a single field corresponds to a
file that
> is missing, I get an error and the stat file is empty. Is there a
way to
> handle this so that a file is produced and missing data is omitted?
>
>
>
> Thanks,
>
> Liz
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
> Editor, Monthly Weather Review
>
> Chair, AMS Committee on Probability and Statistics
>
> Meteorologist, Code 7531
> Marine Meteorology Division
> U.S. Naval Research Laboratory
> T 831.656.4056  F 831.656.4026
>  <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
> __________________________________________________________________
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> Any misuse or unauthorized disclosure may result in both civil and
criminal
> penalties.
>
>
>
>
>
>
>

------------------------------------------------
Subject: handling missing data
From: Satterfield, Dr. Elizabeth
Time: Mon Apr 15 13:35:58 2019

Hi John,

Yes, it is the case that whenever one of the python scripts fail, then
Grid-Stat errors
out entirely.

My preference would be a warning condition, so that I can list
separate files
for "fields" and run like the second case you described below.

Thanks,
Liz

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, April 15, 2019 11:42 AM
To: Satterfield, Dr. Elizabeth
Subject: Re: [rt.rap.ucar.edu #89694] handling missing data

Liz,