[Met_help] [rt.rap.ucar.edu #89694] History for handling missing data
John Halley Gotway via RT
met_help at ucar.edu
Mon May 6 12:00:05 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I'm working with grid_stat, and I'd like to set this up so that I have one
file (based on valid time) for multiple fields, levels and leads. I'm using
python embedding, so in the config I have multiple fields like.
{
name = "/users/satterfield/scripts/python/read_NRL_binary.py
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
.0_0000.0_glob360x181_2016010400_00960000_fcstfld";
level = ["P850"];
cat_thresh = [ NA ];
},
{
name = "/users/satterfield/scripts/python/read_NRL_binary.py
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
.0_0000.0_glob360x181_2016010500_01200000_fcstfld";
level = ["P850"];
cat_thresh = [ NA ];
}
The issue that I have is that if a single field corresponds to a file that
is missing, I get an error and the stat file is empty. Is there a way to
handle this so that a file is produced and missing data is omitted?
Thanks,
Liz
Elizabeth Satterfield
Editor, Monthly Weather Review
Chair, AMS Committee on Probability and Statistics
Meteorologist, Code 7531
Marine Meteorology Division
U.S. Naval Research Laboratory
T 831.656.4056 F 831.656.4026
<http://www.nrl.navy.mil/> www.nrl.navy.mil
__________________________________________________________________
FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
Any misuse or unauthorized disclosure may result in both civil and criminal
penalties.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: handling missing data
From: John Halley Gotway
Time: Mon Apr 15 12:41:59 2019
Liz,
When you run Grid-Stat, you typically pass in a forecast file and an
observation file. If either of those files are missing, then Grid-
Stat
errors out and does not produce an output file. This is an error
condition.
If both of those files do exist, then Grid-Stat steps through the
verification tasks defined in the config file. If one of the fields
can't
be found in an input file, it prints a warning message and continues
to the
next task. So this is a warning condition.
In your case, you're using the python embedding. And it's probably
the
case that if whenever one of the python scripts fail, then Grid-Stat
errors
out entirely. Here's what I see when the python script has problems:
*ERROR : ERROR : python_dataplane() -> an error occurred importing
module
"/d1/johnhg/MET/MET_development/svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
<http://svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py>"ERROR
: *
Is this the error message you're talking about?
Also note that with python embedding, setting the level value has no
impact:
*level = ["P850"];*
The metadata, including the level, is set by the Python script, not
the
config file.
Are you saying that you're prefer the error referenced above to be a
warning instead?
Thanks,
John
On Thu, Apr 11, 2019 at 3:19 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> Thu Apr 11 15:18:50 2019: Request 89694 was acted upon.
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
> Queue: met_help
> Subject: handling missing data
> Owner: Nobody
> Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
>
>
> I'm working with grid_stat, and I'd like to set this up so that I
have one
> file (based on valid time) for multiple fields, levels and leads.
I'm
> using
> python embedding, so in the config I have multiple fields like.
>
>
>
> {
>
> name =
"/users/satterfield/scripts/python/read_NRL_binary.py
>
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> .0_0000.0_glob360x181_2016010400_00960000_fcstfld";
>
> level = ["P850"];
>
> cat_thresh = [ NA ];
>
> },
>
> {
>
> name =
"/users/satterfield/scripts/python/read_NRL_binary.py
>
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> .0_0000.0_glob360x181_2016010500_01200000_fcstfld";
>
> level = ["P850"];
>
> cat_thresh = [ NA ];
>
> }
>
>
>
> The issue that I have is that if a single field corresponds to a
file that
> is missing, I get an error and the stat file is empty. Is there a
way to
> handle this so that a file is produced and missing data is omitted?
>
>
>
> Thanks,
>
> Liz
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
> Editor, Monthly Weather Review
>
> Chair, AMS Committee on Probability and Statistics
>
> Meteorologist, Code 7531
> Marine Meteorology Division
> U.S. Naval Research Laboratory
> T 831.656.4056 F 831.656.4026
> <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
> __________________________________________________________________
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> Any misuse or unauthorized disclosure may result in both civil and
criminal
> penalties.
>
>
>
>
>
>
>
------------------------------------------------
Subject: handling missing data
From: Satterfield, Dr. Elizabeth
Time: Mon Apr 15 13:35:58 2019
Hi John,
Yes, it is the case that whenever one of the python scripts fail, then
Grid-Stat errors
out entirely.
My preference would be a warning condition, so that I can list
separate files
for "fields" and run like the second case you described below.
Thanks,
Liz
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, April 15, 2019 11:42 AM
To: Satterfield, Dr. Elizabeth
Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
Liz,
When you run Grid-Stat, you typically pass in a forecast file and an
observation file. If either of those files are missing, then Grid-
Stat
errors out and does not produce an output file. This is an error
condition.
If both of those files do exist, then Grid-Stat steps through the
verification tasks defined in the config file. If one of the fields
can't
be found in an input file, it prints a warning message and continues
to the
next task. So this is a warning condition.
In your case, you're using the python embedding. And it's probably
the
case that if whenever one of the python scripts fail, then Grid-Stat
errors
out entirely. Here's what I see when the python script has problems:
*ERROR : ERROR : python_dataplane() -> an error occurred importing
module
"/d1/johnhg/MET/MET_development/svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
<http://svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py>"ERROR
: *
Is this the error message you're talking about?
Also note that with python embedding, setting the level value has no
impact:
*level = ["P850"];*
The metadata, including the level, is set by the Python script, not
the
config file.
Are you saying that you're prefer the error referenced above to be a
warning instead?
Thanks,
John
On Thu, Apr 11, 2019 at 3:19 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> Thu Apr 11 15:18:50 2019: Request 89694 was acted upon.
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
> Queue: met_help
> Subject: handling missing data
> Owner: Nobody
> Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
>
>
> I'm working with grid_stat, and I'd like to set this up so that I
have one
> file (based on valid time) for multiple fields, levels and leads.
I'm
> using
> python embedding, so in the config I have multiple fields like.
>
>
>
> {
>
> name =
"/users/satterfield/scripts/python/read_NRL_binary.py
>
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> .0_0000.0_glob360x181_2016010400_00960000_fcstfld";
>
> level = ["P850"];
>
> cat_thresh = [ NA ];
>
> },
>
> {
>
> name =
"/users/satterfield/scripts/python/read_NRL_binary.py
>
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> .0_0000.0_glob360x181_2016010500_01200000_fcstfld";
>
> level = ["P850"];
>
> cat_thresh = [ NA ];
>
> }
>
>
>
> The issue that I have is that if a single field corresponds to a
file that
> is missing, I get an error and the stat file is empty. Is there a
way to
> handle this so that a file is produced and missing data is omitted?
>
>
>
> Thanks,
>
> Liz
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
> Editor, Monthly Weather Review
>
> Chair, AMS Committee on Probability and Statistics
>
> Meteorologist, Code 7531
> Marine Meteorology Division
> U.S. Naval Research Laboratory
> T 831.656.4056 F 831.656.4026
> <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
> __________________________________________________________________
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> Any misuse or unauthorized disclosure may result in both civil and
criminal
> penalties.
>
>
>
>
>
>
>
------------------------------------------------
Subject: handling missing data
From: John Halley Gotway
Time: Mon Apr 15 16:49:19 2019
Liz,
OK, I worked on the code changes to support this change in logic.
I'll run
it by the other developers in our group to make sure this change makes
sense. I do agree that it'll make the logic of grid-stat more
consistent
across the supported input file types.
Thanks,
John
On Mon, Apr 15, 2019 at 1:36 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
>
> Hi John,
>
> Yes, it is the case that whenever one of the python scripts fail,
then
> Grid-Stat errors
> out entirely.
>
> My preference would be a warning condition, so that I can list
separate
> files
> for "fields" and run like the second case you described below.
>
> Thanks,
> Liz
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, April 15, 2019 11:42 AM
> To: Satterfield, Dr. Elizabeth
> Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
>
> Liz,
>
> When you run Grid-Stat, you typically pass in a forecast file and an
> observation file. If either of those files are missing, then Grid-
Stat
> errors out and does not produce an output file. This is an error
> condition.
>
> If both of those files do exist, then Grid-Stat steps through the
> verification tasks defined in the config file. If one of the fields
can't
> be found in an input file, it prints a warning message and continues
to the
> next task. So this is a warning condition.
>
> In your case, you're using the python embedding. And it's probably
the
> case that if whenever one of the python scripts fail, then Grid-Stat
errors
> out entirely. Here's what I see when the python script has
problems:
>
>
>
> *ERROR : ERROR : python_dataplane() -> an error occurred importing
module
> "/d1/johnhg/MET/MET_development/
> svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
> <
> http://svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
> >"ERROR
> : *
>
> Is this the error message you're talking about?
>
> Also note that with python embedding, setting the level value has no
> impact:
> *level = ["P850"];*
>
> The metadata, including the level, is set by the Python script, not
the
> config file.
>
> Are you saying that you're prefer the error referenced above to be a
> warning instead?
>
> Thanks,
> John
>
> On Thu, Apr 11, 2019 at 3:19 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Apr 11 15:18:50 2019: Request 89694 was acted upon.
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > Queue: met_help
> > Subject: handling missing data
> > Owner: Nobody
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
> >
> >
> > I'm working with grid_stat, and I'd like to set this up so that I
have
> one
> > file (based on valid time) for multiple fields, levels and leads.
I'm
> > using
> > python embedding, so in the config I have multiple fields like.
> >
> >
> >
> > {
> >
> > name =
"/users/satterfield/scripts/python/read_NRL_binary.py
> >
> >
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> > .0_0000.0_glob360x181_2016010400_00960000_fcstfld";
> >
> > level = ["P850"];
> >
> > cat_thresh = [ NA ];
> >
> > },
> >
> > {
> >
> > name =
"/users/satterfield/scripts/python/read_NRL_binary.py
> >
> >
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> > .0_0000.0_glob360x181_2016010500_01200000_fcstfld";
> >
> > level = ["P850"];
> >
> > cat_thresh = [ NA ];
> >
> > }
> >
> >
> >
> > The issue that I have is that if a single field corresponds to a
file
> that
> > is missing, I get an error and the stat file is empty. Is there a
way to
> > handle this so that a file is produced and missing data is
omitted?
> >
> >
> >
> > Thanks,
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> > Editor, Monthly Weather Review
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> > Meteorologist, Code 7531
> > Marine Meteorology Division
> > U.S. Naval Research Laboratory
> > T 831.656.4056 F 831.656.4026
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> > __________________________________________________________________
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > Any misuse or unauthorized disclosure may result in both civil and
> criminal
> > penalties.
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: handling missing data
From: Satterfield, Dr. Elizabeth
Time: Mon Apr 15 16:57:04 2019
Many thanks, John!
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, April 15, 2019 3:49 PM
To: Satterfield, Dr. Elizabeth
Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
Liz,
OK, I worked on the code changes to support this change in logic.
I'll run
it by the other developers in our group to make sure this change makes
sense. I do agree that it'll make the logic of grid-stat more
consistent
across the supported input file types.
Thanks,
John
On Mon, Apr 15, 2019 at 1:36 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
>
> Hi John,
>
> Yes, it is the case that whenever one of the python scripts fail,
then
> Grid-Stat errors
> out entirely.
>
> My preference would be a warning condition, so that I can list
separate
> files
> for "fields" and run like the second case you described below.
>
> Thanks,
> Liz
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, April 15, 2019 11:42 AM
> To: Satterfield, Dr. Elizabeth
> Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
>
> Liz,
>
> When you run Grid-Stat, you typically pass in a forecast file and an
> observation file. If either of those files are missing, then Grid-
Stat
> errors out and does not produce an output file. This is an error
> condition.
>
> If both of those files do exist, then Grid-Stat steps through the
> verification tasks defined in the config file. If one of the fields
can't
> be found in an input file, it prints a warning message and continues
to the
> next task. So this is a warning condition.
>
> In your case, you're using the python embedding. And it's probably
the
> case that if whenever one of the python scripts fail, then Grid-Stat
errors
> out entirely. Here's what I see when the python script has
problems:
>
>
>
> *ERROR : ERROR : python_dataplane() -> an error occurred importing
module
> "/d1/johnhg/MET/MET_development/
> svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
> <
> http://svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
> >"ERROR
> : *
>
> Is this the error message you're talking about?
>
> Also note that with python embedding, setting the level value has no
> impact:
> *level = ["P850"];*
>
> The metadata, including the level, is set by the Python script, not
the
> config file.
>
> Are you saying that you're prefer the error referenced above to be a
> warning instead?
>
> Thanks,
> John
>
> On Thu, Apr 11, 2019 at 3:19 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Apr 11 15:18:50 2019: Request 89694 was acted upon.
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > Queue: met_help
> > Subject: handling missing data
> > Owner: Nobody
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
> >
> >
> > I'm working with grid_stat, and I'd like to set this up so that I
have
> one
> > file (based on valid time) for multiple fields, levels and leads.
I'm
> > using
> > python embedding, so in the config I have multiple fields like.
> >
> >
> >
> > {
> >
> > name =
"/users/satterfield/scripts/python/read_NRL_binary.py
> >
> >
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> > .0_0000.0_glob360x181_2016010400_00960000_fcstfld";
> >
> > level = ["P850"];
> >
> > cat_thresh = [ NA ];
> >
> > },
> >
> > {
> >
> > name =
"/users/satterfield/scripts/python/read_NRL_binary.py
> >
> >
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> > .0_0000.0_glob360x181_2016010500_01200000_fcstfld";
> >
> > level = ["P850"];
> >
> > cat_thresh = [ NA ];
> >
> > }
> >
> >
> >
> > The issue that I have is that if a single field corresponds to a
file
> that
> > is missing, I get an error and the stat file is empty. Is there a
way to
> > handle this so that a file is produced and missing data is
omitted?
> >
> >
> >
> > Thanks,
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> > Editor, Monthly Weather Review
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> > Meteorologist, Code 7531
> > Marine Meteorology Division
> > U.S. Naval Research Laboratory
> > T 831.656.4056 F 831.656.4026
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> > __________________________________________________________________
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > Any misuse or unauthorized disclosure may result in both civil and
> criminal
> > penalties.
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: handling missing data
From: Satterfield, Dr. Elizabeth
Time: Fri May 03 17:34:24 2019
Hi John,
I'm wondering if this change has propagated up the chain? It would
really
help our development if we had a version with the code change
implemented.
Thanks,
Liz
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, April 15, 2019 3:49 PM
To: Satterfield, Dr. Elizabeth
Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
Liz,
OK, I worked on the code changes to support this change in logic.
I'll run
it by the other developers in our group to make sure this change makes
sense. I do agree that it'll make the logic of grid-stat more
consistent
across the supported input file types.
Thanks,
John
On Mon, Apr 15, 2019 at 1:36 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
>
> Hi John,
>
> Yes, it is the case that whenever one of the python scripts fail,
then
> Grid-Stat errors
> out entirely.
>
> My preference would be a warning condition, so that I can list
separate
> files
> for "fields" and run like the second case you described below.
>
> Thanks,
> Liz
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, April 15, 2019 11:42 AM
> To: Satterfield, Dr. Elizabeth
> Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
>
> Liz,
>
> When you run Grid-Stat, you typically pass in a forecast file and an
> observation file. If either of those files are missing, then Grid-
Stat
> errors out and does not produce an output file. This is an error
> condition.
>
> If both of those files do exist, then Grid-Stat steps through the
> verification tasks defined in the config file. If one of the fields
can't
> be found in an input file, it prints a warning message and continues
to the
> next task. So this is a warning condition.
>
> In your case, you're using the python embedding. And it's probably
the
> case that if whenever one of the python scripts fail, then Grid-Stat
errors
> out entirely. Here's what I see when the python script has
problems:
>
>
>
> *ERROR : ERROR : python_dataplane() -> an error occurred importing
module
> "/d1/johnhg/MET/MET_development/
> svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
> <
> http://svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
> >"ERROR
> : *
>
> Is this the error message you're talking about?
>
> Also note that with python embedding, setting the level value has no
> impact:
> *level = ["P850"];*
>
> The metadata, including the level, is set by the Python script, not
the
> config file.
>
> Are you saying that you're prefer the error referenced above to be a
> warning instead?
>
> Thanks,
> John
>
> On Thu, Apr 11, 2019 at 3:19 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Apr 11 15:18:50 2019: Request 89694 was acted upon.
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > Queue: met_help
> > Subject: handling missing data
> > Owner: Nobody
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
> >
> >
> > I'm working with grid_stat, and I'd like to set this up so that I
have
> one
> > file (based on valid time) for multiple fields, levels and leads.
I'm
> > using
> > python embedding, so in the config I have multiple fields like.
> >
> >
> >
> > {
> >
> > name =
"/users/satterfield/scripts/python/read_NRL_binary.py
> >
> >
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> > .0_0000.0_glob360x181_2016010400_00960000_fcstfld";
> >
> > level = ["P850"];
> >
> > cat_thresh = [ NA ];
> >
> > },
> >
> > {
> >
> > name =
"/users/satterfield/scripts/python/read_NRL_binary.py
> >
> >
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> > .0_0000.0_glob360x181_2016010500_01200000_fcstfld";
> >
> > level = ["P850"];
> >
> > cat_thresh = [ NA ];
> >
> > }
> >
> >
> >
> > The issue that I have is that if a single field corresponds to a
file
> that
> > is missing, I get an error and the stat file is empty. Is there a
way to
> > handle this so that a file is produced and missing data is
omitted?
> >
> >
> >
> > Thanks,
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> > Editor, Monthly Weather Review
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> > Meteorologist, Code 7531
> > Marine Meteorology Division
> > U.S. Naval Research Laboratory
> > T 831.656.4056 F 831.656.4026
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> > __________________________________________________________________
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > Any misuse or unauthorized disclosure may result in both civil and
> criminal
> > penalties.
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: handling missing data
From: John Halley Gotway
Time: Sat May 04 13:04:01 2019
Liz,
We literally just posted met-8.1 yesterday (Friday) and it does
include
this change. We're also switching over to a new website layout and
will be
sending out the announcement for the MET/METplus/METviewer releases
through
that on Tuesday.
But the met-8.1 is already posted and available for download:
https://dtcenter.org/community-code/model-evaluation-tools-
met/download
If you look through the met-8.1 release notes, you'll see this change
mentioned here:
- Updated the embedded python logic so that failures result in a
warning
instead of an error.
Thanks,
John
On Fri, May 3, 2019 at 5:34 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
>
> Hi John,
>
> I'm wondering if this change has propagated up the chain? It would
really
> help our development if we had a version with the code change
implemented.
>
> Thanks,
> Liz
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, April 15, 2019 3:49 PM
> To: Satterfield, Dr. Elizabeth
> Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
>
> Liz,
>
> OK, I worked on the code changes to support this change in logic.
I'll run
> it by the other developers in our group to make sure this change
makes
> sense. I do agree that it'll make the logic of grid-stat more
consistent
> across the supported input file types.
>
> Thanks,
> John
>
> On Mon, Apr 15, 2019 at 1:36 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
> >
> > Hi John,
> >
> > Yes, it is the case that whenever one of the python scripts fail,
then
> > Grid-Stat errors
> > out entirely.
> >
> > My preference would be a warning condition, so that I can list
separate
> > files
> > for "fields" and run like the second case you described below.
> >
> > Thanks,
> > Liz
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Monday, April 15, 2019 11:42 AM
> > To: Satterfield, Dr. Elizabeth
> > Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
> >
> > Liz,
> >
> > When you run Grid-Stat, you typically pass in a forecast file and
an
> > observation file. If either of those files are missing, then
Grid-Stat
> > errors out and does not produce an output file. This is an error
> > condition.
> >
> > If both of those files do exist, then Grid-Stat steps through the
> > verification tasks defined in the config file. If one of the
fields
> can't
> > be found in an input file, it prints a warning message and
continues to
> the
> > next task. So this is a warning condition.
> >
> > In your case, you're using the python embedding. And it's
probably the
> > case that if whenever one of the python scripts fail, then Grid-
Stat
> errors
> > out entirely. Here's what I see when the python script has
problems:
> >
> >
> >
> > *ERROR : ERROR : python_dataplane() -> an error occurred
importing
> module
> > "/d1/johnhg/MET/MET_development/
> > svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
> > <
> >
> http://svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
> > >"ERROR
> > : *
> >
> > Is this the error message you're talking about?
> >
> > Also note that with python embedding, setting the level value has
no
> > impact:
> > *level = ["P850"];*
> >
> > The metadata, including the level, is set by the Python script,
not the
> > config file.
> >
> > Are you saying that you're prefer the error referenced above to be
a
> > warning instead?
> >
> > Thanks,
> > John
> >
> > On Thu, Apr 11, 2019 at 3:19 PM Satterfield, Dr. Elizabeth via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Apr 11 15:18:50 2019: Request 89694 was acted upon.
> > > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > > Queue: met_help
> > > Subject: handling missing data
> > > Owner: Nobody
> > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694
> >
> > >
> > >
> > > I'm working with grid_stat, and I'd like to set this up so that
I have
> > one
> > > file (based on valid time) for multiple fields, levels and
leads. I'm
> > > using
> > > python embedding, so in the config I have multiple fields like.
> > >
> > >
> > >
> > > {
> > >
> > > name =
> "/users/satterfield/scripts/python/read_NRL_binary.py
> > >
> > >
> >
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> > > .0_0000.0_glob360x181_2016010400_00960000_fcstfld";
> > >
> > > level = ["P850"];
> > >
> > > cat_thresh = [ NA ];
> > >
> > > },
> > >
> > > {
> > >
> > > name =
> "/users/satterfield/scripts/python/read_NRL_binary.py
> > >
> > >
> >
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> > > .0_0000.0_glob360x181_2016010500_01200000_fcstfld";
> > >
> > > level = ["P850"];
> > >
> > > cat_thresh = [ NA ];
> > >
> > > }
> > >
> > >
> > >
> > > The issue that I have is that if a single field corresponds to a
file
> > that
> > > is missing, I get an error and the stat file is empty. Is there
a way
> to
> > > handle this so that a file is produced and missing data is
omitted?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Liz
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Elizabeth Satterfield
> > > Editor, Monthly Weather Review
> > >
> > > Chair, AMS Committee on Probability and Statistics
> > >
> > > Meteorologist, Code 7531
> > > Marine Meteorology Division
> > > U.S. Naval Research Laboratory
> > > T 831.656.4056 F 831.656.4026
> > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > >
> > >
__________________________________________________________________
> > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > > Any misuse or unauthorized disclosure may result in both civil
and
> > criminal
> > > penalties.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: handling missing data
From: Satterfield, Dr. Elizabeth
Time: Mon May 06 09:58:30 2019
John,
That's great! Thank you!!
Liz
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Saturday, May 4, 2019 12:04 PM
To: Satterfield, Dr. Elizabeth
Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
Liz,
We literally just posted met-8.1 yesterday (Friday) and it does
include
this change. We're also switching over to a new website layout and
will be
sending out the announcement for the MET/METplus/METviewer releases
through
that on Tuesday.
But the met-8.1 is already posted and available for download:
https://dtcenter.org/community-code/model-evaluation-tools-
met/download
If you look through the met-8.1 release notes, you'll see this change
mentioned here:
- Updated the embedded python logic so that failures result in a
warning
instead of an error.
Thanks,
John
On Fri, May 3, 2019 at 5:34 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
>
> Hi John,
>
> I'm wondering if this change has propagated up the chain? It would
really
> help our development if we had a version with the code change
implemented.
>
> Thanks,
> Liz
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, April 15, 2019 3:49 PM
> To: Satterfield, Dr. Elizabeth
> Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
>
> Liz,
>
> OK, I worked on the code changes to support this change in logic.
I'll run
> it by the other developers in our group to make sure this change
makes
> sense. I do agree that it'll make the logic of grid-stat more
consistent
> across the supported input file types.
>
> Thanks,
> John
>
> On Mon, Apr 15, 2019 at 1:36 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694 >
> >
> > Hi John,
> >
> > Yes, it is the case that whenever one of the python scripts fail,
then
> > Grid-Stat errors
> > out entirely.
> >
> > My preference would be a warning condition, so that I can list
separate
> > files
> > for "fields" and run like the second case you described below.
> >
> > Thanks,
> > Liz
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Monday, April 15, 2019 11:42 AM
> > To: Satterfield, Dr. Elizabeth
> > Subject: Re: [rt.rap.ucar.edu #89694] handling missing data
> >
> > Liz,
> >
> > When you run Grid-Stat, you typically pass in a forecast file and
an
> > observation file. If either of those files are missing, then
Grid-Stat
> > errors out and does not produce an output file. This is an error
> > condition.
> >
> > If both of those files do exist, then Grid-Stat steps through the
> > verification tasks defined in the config file. If one of the
fields
> can't
> > be found in an input file, it prints a warning message and
continues to
> the
> > next task. So this is a warning condition.
> >
> > In your case, you're using the python embedding. And it's
probably the
> > case that if whenever one of the python scripts fail, then Grid-
Stat
> errors
> > out entirely. Here's what I see when the python script has
problems:
> >
> >
> >
> > *ERROR : ERROR : python_dataplane() -> an error occurred
importing
> module
> > "/d1/johnhg/MET/MET_development/
> > svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
> > <
> >
> http://svn-met-
dev.cgd.ucar.edu/trunk/met/share/met/python/read_ascii_numpy.py
> > >"ERROR
> > : *
> >
> > Is this the error message you're talking about?
> >
> > Also note that with python embedding, setting the level value has
no
> > impact:
> > *level = ["P850"];*
> >
> > The metadata, including the level, is set by the Python script,
not the
> > config file.
> >
> > Are you saying that you're prefer the error referenced above to be
a
> > warning instead?
> >
> > Thanks,
> > John
> >
> > On Thu, Apr 11, 2019 at 3:19 PM Satterfield, Dr. Elizabeth via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Apr 11 15:18:50 2019: Request 89694 was acted upon.
> > > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > > Queue: met_help
> > > Subject: handling missing data
> > > Owner: Nobody
> > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89694
> >
> > >
> > >
> > > I'm working with grid_stat, and I'd like to set this up so that
I have
> > one
> > > file (based on valid time) for multiple fields, levels and
leads. I'm
> > > using
> > > python embedding, so in the config I have multiple fields like.
> > >
> > >
> > >
> > > {
> > >
> > > name =
> "/users/satterfield/scripts/python/read_NRL_binary.py
> > >
> > >
> >
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> > > .0_0000.0_glob360x181_2016010400_00960000_fcstfld";
> > >
> > > level = ["P850"];
> > >
> > > cat_thresh = [ NA ];
> > >
> > > },
> > >
> > > {
> > >
> > > name =
> "/users/satterfield/scripts/python/read_NRL_binary.py
> > >
> > >
> >
>
/data2/satterfield/GEFS_member0/winter/flatfiles/g2016010400/geopht_pre_0850
> > > .0_0000.0_glob360x181_2016010500_01200000_fcstfld";
> > >
> > > level = ["P850"];
> > >
> > > cat_thresh = [ NA ];
> > >
> > > }
> > >
> > >
> > >
> > > The issue that I have is that if a single field corresponds to a
file
> > that
> > > is missing, I get an error and the stat file is empty. Is there
a way
> to
> > > handle this so that a file is produced and missing data is
omitted?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Liz
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Elizabeth Satterfield
> > > Editor, Monthly Weather Review
> > >
> > > Chair, AMS Committee on Probability and Statistics
> > >
> > > Meteorologist, Code 7531
> > > Marine Meteorology Division
> > > U.S. Naval Research Laboratory
> > > T 831.656.4056 F 831.656.4026
> > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > >
> > >
__________________________________________________________________
> > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > > Any misuse or unauthorized disclosure may result in both civil
and
> > criminal
> > > penalties.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list