[Met_help] [rt.rap.ucar.edu #98153] History for python embedding for ensemble_stat
John Halley Gotway via RT
met_help at ucar.edu
Thu Jan 14 12:13:21 MST 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John G. and others.
At one point you helped us with a python reader for our binary data files,
'read_NRL_binary.py'. Now that MET supports python embedding for
ensemble_stat, what we'd like is to use that script, or similar, to read in
the ensemble files.
Currently, we have a separate file for each ensemble member. My question is
- what is the best way to do this? Use read_nrl_binary 20 times for each
ensemble member or create a different script to pass all 20 members
simultaneously?
Thanks,
Liz
Elizabeth Satterfield
Editor, Monthly Weather Review
Chair, AMS Committee on Probability and Statistics
Meteorologist, Code 7531
Marine Meteorology Division
U.S. Naval Research Laboratory
T 831.656.4056 F 831.656.4026
<http://www.nrl.navy.mil/> www.nrl.navy.mil
__________________________________________________________________
FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
Any misuse or unauthorized disclosure may result in both civil and criminal
penalties.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: python embedding for ensemble_stat
From: John Halley Gotway
Time: Thu Jan 07 12:55:44 2021
Hi Liz,
The python embedding functionality in Ensemble-Stat still only
supports the
reading of a single field at a time.
So Ensemble-Stat will call that script 20 times to read data from
those 20
files... in the same way that it'd read data from 20 different GRIB
input
files.
We have a toy example of running ensemble-stat with python embedding
in our
unit tests:
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/test/xml/unit_python.xml#L239
But it's a little difficult to read through the XML. Here's the sort
of
command you'd run using read_NRL_binary.py
ensemble_stat ens_file_list EnsembleStatConfig_NRL -point_obs
point_obs_data.nc
Where...
'ens_file_list' is an ASCII file containing the list of ensemble
member
binary data files.
'point_obs_data.nc' contains point obs generated by pb2nc or ascii2nc.
You
could also verify against gridded data using the "-grid_obs" command
line
option.
'EnsembleStatConfig_NRL' is an Ensemble-Stat config file which
contains
settings like:
ens = {
file_type = PYTHON_NUMPY;
field = [ { name = "/path/to/read_NRL_binary.py
MET_PYTHON_INPUT_ARG"; }
];
}
So the logic is... when Ensemble-Stat sees that the file_type is set
to
PYTHON_NUMPY, it loops over the ensemble member input files you
specified
on the command line. For each, it replaces the special keyword
"MET_PYTHON_INPUT_ARG" with the current file being processed. And then
it
executes that command to retrieve the gridded data for that ensemble
member.
Make sense?
Please give it a shot and let me know if you run into issues or have
other
questions.
Thanks,
John Halley Gotway
On Thu, Jan 7, 2021 at 11:54 AM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> Thu Jan 07 11:53:43 2021: Request 98153 was acted upon.
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
> Queue: met_help
> Subject: python embedding for ensemble_stat
> Owner: Nobody
> Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153 >
>
>
> John G. and others.
>
>
>
> At one point you helped us with a python reader for our binary data
files,
> 'read_NRL_binary.py'. Now that MET supports python embedding for
> ensemble_stat, what we'd like is to use that script, or similar, to
read in
> the ensemble files.
>
>
>
> Currently, we have a separate file for each ensemble member. My
question
> is
> - what is the best way to do this? Use read_nrl_binary 20 times for
each
> ensemble member or create a different script to pass all 20 members
> simultaneously?
>
>
>
> Thanks,
>
> Liz
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
> Editor, Monthly Weather Review
>
> Chair, AMS Committee on Probability and Statistics
>
> Meteorologist, Code 7531
> Marine Meteorology Division
> U.S. Naval Research Laboratory
> T 831.656.4056 F 831.656.4026
> <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
> __________________________________________________________________
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> Any misuse or unauthorized disclosure may result in both civil and
criminal
> penalties.
>
>
>
>
>
>
>
>
>
------------------------------------------------
Subject: python embedding for ensemble_stat
From: Satterfield, Dr. Elizabeth
Time: Thu Jan 07 12:57:00 2021
Thanks! This makes sense.
Liz
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, January 7, 2021 11:56 AM
To: Satterfield, Dr. Elizabeth <Elizabeth.Satterfield at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
ensemble_stat
Hi Liz,
The python embedding functionality in Ensemble-Stat still only
supports the
reading of a single field at a time.
So Ensemble-Stat will call that script 20 times to read data from
those 20
files... in the same way that it'd read data from 20 different GRIB
input
files.
We have a toy example of running ensemble-stat with python embedding
in our
unit tests:
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/test/xml/unit_python.xml#L239
But it's a little difficult to read through the XML. Here's the sort
of
command you'd run using read_NRL_binary.py
ensemble_stat ens_file_list EnsembleStatConfig_NRL -point_obs
point_obs_data.nc
Where...
'ens_file_list' is an ASCII file containing the list of ensemble
member binary
data files.
'point_obs_data.nc' contains point obs generated by pb2nc or ascii2nc.
You
could also verify against gridded data using the "-grid_obs" command
line
option.
'EnsembleStatConfig_NRL' is an Ensemble-Stat config file which
contains
settings like:
ens = {
file_type = PYTHON_NUMPY;
field = [ { name = "/path/to/read_NRL_binary.py
MET_PYTHON_INPUT_ARG"; } ]; }
So the logic is... when Ensemble-Stat sees that the file_type is set
to
PYTHON_NUMPY, it loops over the ensemble member input files you
specified on
the command line. For each, it replaces the special keyword
"MET_PYTHON_INPUT_ARG" with the current file being processed. And then
it
executes that command to retrieve the gridded data for that ensemble
member.
Make sense?
Please give it a shot and let me know if you run into issues or have
other
questions.
Thanks,
John Halley Gotway
On Thu, Jan 7, 2021 at 11:54 AM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> Thu Jan 07 11:53:43 2021: Request 98153 was acted upon.
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
> Queue: met_help
> Subject: python embedding for ensemble_stat
> Owner: Nobody
> Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153
> >
>
>
> John G. and others.
>
>
>
> At one point you helped us with a python reader for our binary data
> files, 'read_NRL_binary.py'. Now that MET supports python embedding
> for ensemble_stat, what we'd like is to use that script, or similar,
> to read in the ensemble files.
>
>
>
> Currently, we have a separate file for each ensemble member. My
> question is
> - what is the best way to do this? Use read_nrl_binary 20 times for
> each ensemble member or create a different script to pass all 20
> members simultaneously?
>
>
>
> Thanks,
>
> Liz
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
> Editor, Monthly Weather Review
>
> Chair, AMS Committee on Probability and Statistics
>
> Meteorologist, Code 7531
> Marine Meteorology Division
> U.S. Naval Research Laboratory
> T 831.656.4056 F 831.656.4026
> <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
> __________________________________________________________________
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE Any misuse or unauthorized
> disclosure may result in both civil and criminal penalties.
>
>
>
>
>
>
>
>
>
------------------------------------------------
Subject: python embedding for ensemble_stat
From: Satterfield, Dr. Elizabeth
Time: Tue Jan 12 17:58:43 2021
Hi John,
I'm having trouble getting this to work.
In my config, I have the following:
ens = {
file_type = PYTHON_NUMPY;
field = [ { name = "./read_NRL_binary.py MET_PYTHON_INPUT_ARG"; }
]; }
I'm trying to run ensemble_stat as follows:
ensemble_stat ens_file_list
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/config/EnsembleStatConfigACOR_2016072200
-grid_obs
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/test_data_nc/verif/test_data_geopht_pre_0500_2016072200_000_000.nc
-outdir OUT_ENS_test_data/geopht_pre_0500_120_2016072200
I'm wondering if I need an additional argument, since I'm getting the
following errors:
WARNING:
WARNING: python_dataplane() -> an error occurred importing module
"geopht_pre_0500.0"
WARNING:
WARNING:
WARNING: process_n_vld() -> no data found for forecast field "PYTHON"
in file
"test_data/g2016072200/mbr001/outp/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld"
Thanks,
Liz
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, January 7, 2021 11:56 AM
To: Satterfield, Dr. Elizabeth <Elizabeth.Satterfield at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
ensemble_stat
Hi Liz,
The python embedding functionality in Ensemble-Stat still only
supports the
reading of a single field at a time.
So Ensemble-Stat will call that script 20 times to read data from
those 20
files... in the same way that it'd read data from 20 different GRIB
input
files.
We have a toy example of running ensemble-stat with python embedding
in our
unit tests:
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/test/xml/unit_python.xml#L239
But it's a little difficult to read through the XML. Here's the sort
of
command you'd run using read_NRL_binary.py
ensemble_stat ens_file_list EnsembleStatConfig_NRL -point_obs
point_obs_data.nc
Where...
'ens_file_list' is an ASCII file containing the list of ensemble
member binary
data files.
'point_obs_data.nc' contains point obs generated by pb2nc or ascii2nc.
You
could also verify against gridded data using the "-grid_obs" command
line
option.
'EnsembleStatConfig_NRL' is an Ensemble-Stat config file which
contains
settings like:
ens = {
file_type = PYTHON_NUMPY;
field = [ { name = "/path/to/read_NRL_binary.py
MET_PYTHON_INPUT_ARG"; } ]; }
So the logic is... when Ensemble-Stat sees that the file_type is set
to
PYTHON_NUMPY, it loops over the ensemble member input files you
specified on
the command line. For each, it replaces the special keyword
"MET_PYTHON_INPUT_ARG" with the current file being processed. And then
it
executes that command to retrieve the gridded data for that ensemble
member.
Make sense?
Please give it a shot and let me know if you run into issues or have
other
questions.
Thanks,
John Halley Gotway
On Thu, Jan 7, 2021 at 11:54 AM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> Thu Jan 07 11:53:43 2021: Request 98153 was acted upon.
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
> Queue: met_help
> Subject: python embedding for ensemble_stat
> Owner: Nobody
> Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153
> >
>
>
> John G. and others.
>
>
>
> At one point you helped us with a python reader for our binary data
> files, 'read_NRL_binary.py'. Now that MET supports python embedding
> for ensemble_stat, what we'd like is to use that script, or similar,
> to read in the ensemble files.
>
>
>
> Currently, we have a separate file for each ensemble member. My
> question is
> - what is the best way to do this? Use read_nrl_binary 20 times for
> each ensemble member or create a different script to pass all 20
> members simultaneously?
>
>
>
> Thanks,
>
> Liz
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
> Editor, Monthly Weather Review
>
> Chair, AMS Committee on Probability and Statistics
>
> Meteorologist, Code 7531
> Marine Meteorology Division
> U.S. Naval Research Laboratory
> T 831.656.4056 F 831.656.4026
> <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
> __________________________________________________________________
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE Any misuse or unauthorized
> disclosure may result in both civil and criminal penalties.
>
>
>
>
>
>
>
>
>
------------------------------------------------
Subject: python embedding for ensemble_stat
From: John Halley Gotway
Time: Wed Jan 13 09:44:13 2021
Liz,
I don't see any obvious problems in what you sent to me.
To avoid potential path problems, I'd recommend replacing
"./read_NRL_binary.py" with the full path to that script instead. But
I'm
doubtful that'll fix everything.
Next question, what version of MET are you running. Checking the
release
notes, I see that MET_PYTHON_INPUT_ARG was added in met-9.0.
If you think it'd be useful to send me sample data for testing, I'd be
happy to do that.
Thanks,
John
On Tue, Jan 12, 2021 at 5:59 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153 >
>
> Hi John,
>
> I'm having trouble getting this to work.
>
> In my config, I have the following:
> ens = {
> file_type = PYTHON_NUMPY;
> field = [ { name = "./read_NRL_binary.py MET_PYTHON_INPUT_ARG"; }
]; }
>
> I'm trying to run ensemble_stat as follows:
> ensemble_stat ens_file_list
>
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/config/EnsembleStatConfigACOR_2016072200
>
> -grid_obs
>
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/test_data_nc/verif/
> test_data_geopht_pre_0500_2016072200_000_000.nc
> -outdir OUT_ENS_test_data/geopht_pre_0500_120_2016072200
>
>
> I'm wondering if I need an additional argument, since I'm getting
the
> following errors:
> WARNING:
> WARNING: python_dataplane() -> an error occurred importing module
> "geopht_pre_0500.0"
> WARNING:
> WARNING:
> WARNING: process_n_vld() -> no data found for forecast field
"PYTHON" in
> file
>
>
"test_data/g2016072200/mbr001/outp/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld"
>
> Thanks,
> Liz
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 7, 2021 11:56 AM
> To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
ensemble_stat
>
> Hi Liz,
>
> The python embedding functionality in Ensemble-Stat still only
supports
> the
> reading of a single field at a time.
>
> So Ensemble-Stat will call that script 20 times to read data from
those 20
> files... in the same way that it'd read data from 20 different GRIB
input
> files.
>
> We have a toy example of running ensemble-stat with python embedding
in
> our
> unit tests:
>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/test/xml/unit_python.xml#L239
>
> But it's a little difficult to read through the XML. Here's the sort
of
> command you'd run using read_NRL_binary.py
>
> ensemble_stat ens_file_list EnsembleStatConfig_NRL -point_obs
> point_obs_data.nc
>
> Where...
> 'ens_file_list' is an ASCII file containing the list of ensemble
member
> binary
> data files.
> 'point_obs_data.nc' contains point obs generated by pb2nc or
ascii2nc.
> You
> could also verify against gridded data using the "-grid_obs" command
line
> option.
> 'EnsembleStatConfig_NRL' is an Ensemble-Stat config file which
contains
> settings like:
>
> ens = {
> file_type = PYTHON_NUMPY;
> field = [ { name = "/path/to/read_NRL_binary.py
> MET_PYTHON_INPUT_ARG"; } ]; }
>
> So the logic is... when Ensemble-Stat sees that the file_type is set
to
> PYTHON_NUMPY, it loops over the ensemble member input files you
specified
> on
> the command line. For each, it replaces the special keyword
> "MET_PYTHON_INPUT_ARG" with the current file being processed. And
then it
> executes that command to retrieve the gridded data for that ensemble
> member.
>
> Make sense?
>
> Please give it a shot and let me know if you run into issues or have
other
> questions.
>
> Thanks,
> John Halley Gotway
>
> On Thu, Jan 7, 2021 at 11:54 AM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jan 07 11:53:43 2021: Request 98153 was acted upon.
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > Queue: met_help
> > Subject: python embedding for ensemble_stat
> > Owner: Nobody
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153
> > >
> >
> >
> > John G. and others.
> >
> >
> >
> > At one point you helped us with a python reader for our binary
data
> > files, 'read_NRL_binary.py'. Now that MET supports python
embedding
> > for ensemble_stat, what we'd like is to use that script, or
similar,
> > to read in the ensemble files.
> >
> >
> >
> > Currently, we have a separate file for each ensemble member. My
> > question is
> > - what is the best way to do this? Use read_nrl_binary 20 times
for
> > each ensemble member or create a different script to pass all 20
> > members simultaneously?
> >
> >
> >
> > Thanks,
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> > Editor, Monthly Weather Review
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> > Meteorologist, Code 7531
> > Marine Meteorology Division
> > U.S. Naval Research Laboratory
> > T 831.656.4056 F 831.656.4026
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> > __________________________________________________________________
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE Any misuse or
unauthorized
> > disclosure may result in both civil and criminal penalties.
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: python embedding for ensemble_stat
From: Satterfield, Dr. Elizabeth
Time: Wed Jan 13 10:41:50 2021
John,
The full path isn't a fix. And, I'm running with 9.1.
I'll try to bundle the data in a bit and send you something for
testing.
Thanks,
Liz
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, January 13, 2021 8:44 AM
To: Satterfield, Dr. Elizabeth <Elizabeth.Satterfield at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
ensemble_stat
Liz,
I don't see any obvious problems in what you sent to me.
To avoid potential path problems, I'd recommend replacing
"./read_NRL_binary.py" with the full path to that script instead. But
I'm
doubtful that'll fix everything.
Next question, what version of MET are you running. Checking the
release
notes, I see that MET_PYTHON_INPUT_ARG was added in met-9.0.
If you think it'd be useful to send me sample data for testing, I'd be
happy
to do that.
Thanks,
John
On Tue, Jan 12, 2021 at 5:59 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153 >
>
> Hi John,
>
> I'm having trouble getting this to work.
>
> In my config, I have the following:
> ens = {
> file_type = PYTHON_NUMPY;
> field = [ { name = "./read_NRL_binary.py MET_PYTHON_INPUT_ARG"; }
> ]; }
>
> I'm trying to run ensemble_stat as follows:
> ensemble_stat ens_file_list
>
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/config/Ensemble
> StatConfigACOR_2016072200
>
> -grid_obs
>
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/test_data_nc/ve
> rif/ test_data_geopht_pre_0500_2016072200_000_000.nc
> -outdir OUT_ENS_test_data/geopht_pre_0500_120_2016072200
>
>
> I'm wondering if I need an additional argument, since I'm getting
the
> following errors:
> WARNING:
> WARNING: python_dataplane() -> an error occurred importing module
> "geopht_pre_0500.0"
> WARNING:
> WARNING:
> WARNING: process_n_vld() -> no data found for forecast field
"PYTHON"
> in file
>
>
"test_data/g2016072200/mbr001/outp/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld"
>
> Thanks,
> Liz
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 7, 2021 11:56 AM
> To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
> ensemble_stat
>
> Hi Liz,
>
> The python embedding functionality in Ensemble-Stat still only
> supports the reading of a single field at a time.
>
> So Ensemble-Stat will call that script 20 times to read data from
> those 20 files... in the same way that it'd read data from 20
> different GRIB input files.
>
> We have a toy example of running ensemble-stat with python embedding
> in our unit tests:
>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
> 5981bb8/test/xml/unit_python.xml#L239
>
> But it's a little difficult to read through the XML. Here's the sort
> of command you'd run using read_NRL_binary.py
>
> ensemble_stat ens_file_list EnsembleStatConfig_NRL -point_obs
> point_obs_data.nc
>
> Where...
> 'ens_file_list' is an ASCII file containing the list of ensemble
> member binary data files.
> 'point_obs_data.nc' contains point obs generated by pb2nc or
ascii2nc.
> You
> could also verify against gridded data using the "-grid_obs" command
> line option.
> 'EnsembleStatConfig_NRL' is an Ensemble-Stat config file which
> contains settings like:
>
> ens = {
> file_type = PYTHON_NUMPY;
> field = [ { name = "/path/to/read_NRL_binary.py
> MET_PYTHON_INPUT_ARG"; } ]; }
>
> So the logic is... when Ensemble-Stat sees that the file_type is set
> to PYTHON_NUMPY, it loops over the ensemble member input files you
> specified on the command line. For each, it replaces the special
> keyword "MET_PYTHON_INPUT_ARG" with the current file being
processed.
> And then it executes that command to retrieve the gridded data for
> that ensemble member.
>
> Make sense?
>
> Please give it a shot and let me know if you run into issues or have
> other questions.
>
> Thanks,
> John Halley Gotway
>
> On Thu, Jan 7, 2021 at 11:54 AM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jan 07 11:53:43 2021: Request 98153 was acted upon.
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > Queue: met_help
> > Subject: python embedding for ensemble_stat
> > Owner: Nobody
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153
> > >
> >
> >
> > John G. and others.
> >
> >
> >
> > At one point you helped us with a python reader for our binary
data
> > files, 'read_NRL_binary.py'. Now that MET supports python
embedding
> > for ensemble_stat, what we'd like is to use that script, or
similar,
> > to read in the ensemble files.
> >
> >
> >
> > Currently, we have a separate file for each ensemble member. My
> > question is
> > - what is the best way to do this? Use read_nrl_binary 20 times
for
> > each ensemble member or create a different script to pass all 20
> > members simultaneously?
> >
> >
> >
> > Thanks,
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> > Editor, Monthly Weather Review
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> > Meteorologist, Code 7531
> > Marine Meteorology Division
> > U.S. Naval Research Laboratory
> > T 831.656.4056 F 831.656.4026
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> > __________________________________________________________________
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE Any misuse or
unauthorized
> > disclosure may result in both civil and criminal penalties.
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: python embedding for ensemble_stat
From: Satterfield, Dr. Elizabeth
Time: Wed Jan 13 15:01:18 2021
John,
One more quick question. How does the fcst dictionary need to be set
for this
case. I have it set with the name/level of the ensemble fields.
//
// Forecast and observation fields to be verified
//
fcst = {
field = [
{
name = "geopht";
level = [ "pre_0500.0" ];
}
];
}
obs = {
field = [
{
name = "./read_NRL_binary.py
verif/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld";
}
];
}
Thanks,
Liz
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, January 13, 2021 8:44 AM
To: Satterfield, Dr. Elizabeth <Elizabeth.Satterfield at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
ensemble_stat
Liz,
I don't see any obvious problems in what you sent to me.
To avoid potential path problems, I'd recommend replacing
"./read_NRL_binary.py" with the full path to that script instead. But
I'm
doubtful that'll fix everything.
Next question, what version of MET are you running. Checking the
release
notes, I see that MET_PYTHON_INPUT_ARG was added in met-9.0.
If you think it'd be useful to send me sample data for testing, I'd be
happy
to do that.
Thanks,
John
On Tue, Jan 12, 2021 at 5:59 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153 >
>
> Hi John,
>
> I'm having trouble getting this to work.
>
> In my config, I have the following:
> ens = {
> file_type = PYTHON_NUMPY;
> field = [ { name = "./read_NRL_binary.py MET_PYTHON_INPUT_ARG"; }
> ]; }
>
> I'm trying to run ensemble_stat as follows:
> ensemble_stat ens_file_list
>
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/config/Ensemble
> StatConfigACOR_2016072200
>
> -grid_obs
>
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/test_data_nc/ve
> rif/ test_data_geopht_pre_0500_2016072200_000_000.nc
> -outdir OUT_ENS_test_data/geopht_pre_0500_120_2016072200
>
>
> I'm wondering if I need an additional argument, since I'm getting
the
> following errors:
> WARNING:
> WARNING: python_dataplane() -> an error occurred importing module
> "geopht_pre_0500.0"
> WARNING:
> WARNING:
> WARNING: process_n_vld() -> no data found for forecast field
"PYTHON"
> in file
>
>
"test_data/g2016072200/mbr001/outp/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld"
>
> Thanks,
> Liz
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 7, 2021 11:56 AM
> To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
> ensemble_stat
>
> Hi Liz,
>
> The python embedding functionality in Ensemble-Stat still only
> supports the reading of a single field at a time.
>
> So Ensemble-Stat will call that script 20 times to read data from
> those 20 files... in the same way that it'd read data from 20
> different GRIB input files.
>
> We have a toy example of running ensemble-stat with python embedding
> in our unit tests:
>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
> 5981bb8/test/xml/unit_python.xml#L239
>
> But it's a little difficult to read through the XML. Here's the sort
> of command you'd run using read_NRL_binary.py
>
> ensemble_stat ens_file_list EnsembleStatConfig_NRL -point_obs
> point_obs_data.nc
>
> Where...
> 'ens_file_list' is an ASCII file containing the list of ensemble
> member binary data files.
> 'point_obs_data.nc' contains point obs generated by pb2nc or
ascii2nc.
> You
> could also verify against gridded data using the "-grid_obs" command
> line option.
> 'EnsembleStatConfig_NRL' is an Ensemble-Stat config file which
> contains settings like:
>
> ens = {
> file_type = PYTHON_NUMPY;
> field = [ { name = "/path/to/read_NRL_binary.py
> MET_PYTHON_INPUT_ARG"; } ]; }
>
> So the logic is... when Ensemble-Stat sees that the file_type is set
> to PYTHON_NUMPY, it loops over the ensemble member input files you
> specified on the command line. For each, it replaces the special
> keyword "MET_PYTHON_INPUT_ARG" with the current file being
processed.
> And then it executes that command to retrieve the gridded data for
> that ensemble member.
>
> Make sense?
>
> Please give it a shot and let me know if you run into issues or have
> other questions.
>
> Thanks,
> John Halley Gotway
>
> On Thu, Jan 7, 2021 at 11:54 AM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jan 07 11:53:43 2021: Request 98153 was acted upon.
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > Queue: met_help
> > Subject: python embedding for ensemble_stat
> > Owner: Nobody
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153
> > >
> >
> >
> > John G. and others.
> >
> >
> >
> > At one point you helped us with a python reader for our binary
data
> > files, 'read_NRL_binary.py'. Now that MET supports python
embedding
> > for ensemble_stat, what we'd like is to use that script, or
similar,
> > to read in the ensemble files.
> >
> >
> >
> > Currently, we have a separate file for each ensemble member. My
> > question is
> > - what is the best way to do this? Use read_nrl_binary 20 times
for
> > each ensemble member or create a different script to pass all 20
> > members simultaneously?
> >
> >
> >
> > Thanks,
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> > Editor, Monthly Weather Review
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> > Meteorologist, Code 7531
> > Marine Meteorology Division
> > U.S. Naval Research Laboratory
> > T 831.656.4056 F 831.656.4026
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> > __________________________________________________________________
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE Any misuse or
unauthorized
> > disclosure may result in both civil and criminal penalties.
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: python embedding for ensemble_stat
From: Satterfield, Dr. Elizabeth
Time: Wed Jan 13 16:25:22 2021
Ah...I see my mistake. Seems to be working now!
fcst = {
file_type = PYTHON_NUMPY;
field = [ { name = "./read_NRL_binary.py MET_PYTHON_INPUT_ARG"; }
]; }
-----Original Message-----
From: Satterfield, Dr. Elizabeth
Sent: Wednesday, January 13, 2021 2:01 PM
To: 'met_help at ucar.edu' <met_help at ucar.edu>
Subject: RE: [rt.rap.ucar.edu #98153] python embedding for
ensemble_stat
John,
One more quick question. How does the fcst dictionary need to be set
for this
case. I have it set with the name/level of the ensemble fields.
//
// Forecast and observation fields to be verified
//
fcst = {
field = [
{
name = "geopht";
level = [ "pre_0500.0" ];
}
];
}
obs = {
field = [
{
name = "./read_NRL_binary.py
verif/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld";
}
];
}
Thanks,
Liz
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, January 13, 2021 8:44 AM
To: Satterfield, Dr. Elizabeth <Elizabeth.Satterfield at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
ensemble_stat
Liz,
I don't see any obvious problems in what you sent to me.
To avoid potential path problems, I'd recommend replacing
"./read_NRL_binary.py" with the full path to that script instead. But
I'm
doubtful that'll fix everything.
Next question, what version of MET are you running. Checking the
release
notes, I see that MET_PYTHON_INPUT_ARG was added in met-9.0.
If you think it'd be useful to send me sample data for testing, I'd be
happy
to do that.
Thanks,
John
On Tue, Jan 12, 2021 at 5:59 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153 >
>
> Hi John,
>
> I'm having trouble getting this to work.
>
> In my config, I have the following:
> ens = {
> file_type = PYTHON_NUMPY;
> field = [ { name = "./read_NRL_binary.py MET_PYTHON_INPUT_ARG"; }
> ]; }
>
> I'm trying to run ensemble_stat as follows:
> ensemble_stat ens_file_list
>
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/config/Ensemble
> StatConfigACOR_2016072200
>
> -grid_obs
>
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/test_data_nc/ve
> rif/ test_data_geopht_pre_0500_2016072200_000_000.nc
> -outdir OUT_ENS_test_data/geopht_pre_0500_120_2016072200
>
>
> I'm wondering if I need an additional argument, since I'm getting
the
> following errors:
> WARNING:
> WARNING: python_dataplane() -> an error occurred importing module
> "geopht_pre_0500.0"
> WARNING:
> WARNING:
> WARNING: process_n_vld() -> no data found for forecast field
"PYTHON"
> in file
>
>
"test_data/g2016072200/mbr001/outp/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld"
>
> Thanks,
> Liz
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 7, 2021 11:56 AM
> To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
> ensemble_stat
>
> Hi Liz,
>
> The python embedding functionality in Ensemble-Stat still only
> supports the reading of a single field at a time.
>
> So Ensemble-Stat will call that script 20 times to read data from
> those 20 files... in the same way that it'd read data from 20
> different GRIB input files.
>
> We have a toy example of running ensemble-stat with python embedding
> in our unit tests:
>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
> 5981bb8/test/xml/unit_python.xml#L239
>
> But it's a little difficult to read through the XML. Here's the sort
> of command you'd run using read_NRL_binary.py
>
> ensemble_stat ens_file_list EnsembleStatConfig_NRL -point_obs
> point_obs_data.nc
>
> Where...
> 'ens_file_list' is an ASCII file containing the list of ensemble
> member binary data files.
> 'point_obs_data.nc' contains point obs generated by pb2nc or
ascii2nc.
> You
> could also verify against gridded data using the "-grid_obs" command
> line option.
> 'EnsembleStatConfig_NRL' is an Ensemble-Stat config file which
> contains settings like:
>
> ens = {
> file_type = PYTHON_NUMPY;
> field = [ { name = "/path/to/read_NRL_binary.py
> MET_PYTHON_INPUT_ARG"; } ]; }
>
> So the logic is... when Ensemble-Stat sees that the file_type is set
> to PYTHON_NUMPY, it loops over the ensemble member input files you
> specified on the command line. For each, it replaces the special
> keyword "MET_PYTHON_INPUT_ARG" with the current file being
processed.
> And then it executes that command to retrieve the gridded data for
> that ensemble member.
>
> Make sense?
>
> Please give it a shot and let me know if you run into issues or have
> other questions.
>
> Thanks,
> John Halley Gotway
>
> On Thu, Jan 7, 2021 at 11:54 AM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jan 07 11:53:43 2021: Request 98153 was acted upon.
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > Queue: met_help
> > Subject: python embedding for ensemble_stat
> > Owner: Nobody
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153
> > >
> >
> >
> > John G. and others.
> >
> >
> >
> > At one point you helped us with a python reader for our binary
data
> > files, 'read_NRL_binary.py'. Now that MET supports python
embedding
> > for ensemble_stat, what we'd like is to use that script, or
similar,
> > to read in the ensemble files.
> >
> >
> >
> > Currently, we have a separate file for each ensemble member. My
> > question is
> > - what is the best way to do this? Use read_nrl_binary 20 times
for
> > each ensemble member or create a different script to pass all 20
> > members simultaneously?
> >
> >
> >
> > Thanks,
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> > Editor, Monthly Weather Review
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> > Meteorologist, Code 7531
> > Marine Meteorology Division
> > U.S. Naval Research Laboratory
> > T 831.656.4056 F 831.656.4026
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> > __________________________________________________________________
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE Any misuse or
unauthorized
> > disclosure may result in both civil and criminal penalties.
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: python embedding for ensemble_stat
From: John Halley Gotway
Time: Wed Jan 13 16:36:28 2021
Liz,
Great! Glad you figured it out. Just so I understand, was the problem
that
you sent the "ens" dictionary but did not set the "fcst" dictionary?
John
On Wed, Jan 13, 2021 at 4:25 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153 >
>
> Ah...I see my mistake. Seems to be working now!
>
> fcst = {
> file_type = PYTHON_NUMPY;
> field = [ { name = "./read_NRL_binary.py MET_PYTHON_INPUT_ARG"; }
]; }
>
>
> -----Original Message-----
> From: Satterfield, Dr. Elizabeth
> Sent: Wednesday, January 13, 2021 2:01 PM
> To: 'met_help at ucar.edu' <met_help at ucar.edu>
> Subject: RE: [rt.rap.ucar.edu #98153] python embedding for
ensemble_stat
>
> John,
>
> One more quick question. How does the fcst dictionary need to be
set for
> this
> case. I have it set with the name/level of the ensemble fields.
>
>
> //
> // Forecast and observation fields to be verified
> //
> fcst = {
> field = [
> {
> name = "geopht";
> level = [ "pre_0500.0" ];
> }
> ];
> }
>
> obs = {
> field = [
> {
> name = "./read_NRL_binary.py
>
verif/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld";
> }
> ];
> }
>
> Thanks,
> Liz
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Wednesday, January 13, 2021 8:44 AM
> To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
ensemble_stat
>
> Liz,
>
> I don't see any obvious problems in what you sent to me.
>
> To avoid potential path problems, I'd recommend replacing
> "./read_NRL_binary.py" with the full path to that script instead.
But I'm
> doubtful that'll fix everything.
>
> Next question, what version of MET are you running. Checking the
release
> notes, I see that MET_PYTHON_INPUT_ARG was added in met-9.0.
>
> If you think it'd be useful to send me sample data for testing, I'd
be
> happy
> to do that.
>
> Thanks,
> John
>
> On Tue, Jan 12, 2021 at 5:59 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153 >
> >
> > Hi John,
> >
> > I'm having trouble getting this to work.
> >
> > In my config, I have the following:
> > ens = {
> > file_type = PYTHON_NUMPY;
> > field = [ { name = "./read_NRL_binary.py MET_PYTHON_INPUT_ARG";
}
> > ]; }
> >
> > I'm trying to run ensemble_stat as follows:
> > ensemble_stat ens_file_list
> >
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/config/Ensemble
> > StatConfigACOR_2016072200
> >
> > -grid_obs
> >
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/test_data_nc/ve
> > rif/ test_data_geopht_pre_0500_2016072200_000_000.nc
> > -outdir OUT_ENS_test_data/geopht_pre_0500_120_2016072200
> >
> >
> > I'm wondering if I need an additional argument, since I'm getting
the
> > following errors:
> > WARNING:
> > WARNING: python_dataplane() -> an error occurred importing module
> > "geopht_pre_0500.0"
> > WARNING:
> > WARNING:
> > WARNING: process_n_vld() -> no data found for forecast field
"PYTHON"
> > in file
> >
> >
>
"test_data/g2016072200/mbr001/outp/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld"
> >
> > Thanks,
> > Liz
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 7, 2021 11:56 AM
> > To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> > Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
> > ensemble_stat
> >
> > Hi Liz,
> >
> > The python embedding functionality in Ensemble-Stat still only
> > supports the reading of a single field at a time.
> >
> > So Ensemble-Stat will call that script 20 times to read data from
> > those 20 files... in the same way that it'd read data from 20
> > different GRIB input files.
> >
> > We have a toy example of running ensemble-stat with python
embedding
> > in our unit tests:
> >
> >
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
> > 5981bb8/test/xml/unit_python.xml#L239
> >
> > But it's a little difficult to read through the XML. Here's the
sort
> > of command you'd run using read_NRL_binary.py
> >
> > ensemble_stat ens_file_list EnsembleStatConfig_NRL -point_obs
> > point_obs_data.nc
> >
> > Where...
> > 'ens_file_list' is an ASCII file containing the list of ensemble
> > member binary data files.
> > 'point_obs_data.nc' contains point obs generated by pb2nc or
ascii2nc.
> > You
> > could also verify against gridded data using the "-grid_obs"
command
> > line option.
> > 'EnsembleStatConfig_NRL' is an Ensemble-Stat config file which
> > contains settings like:
> >
> > ens = {
> > file_type = PYTHON_NUMPY;
> > field = [ { name = "/path/to/read_NRL_binary.py
> > MET_PYTHON_INPUT_ARG"; } ]; }
> >
> > So the logic is... when Ensemble-Stat sees that the file_type is
set
> > to PYTHON_NUMPY, it loops over the ensemble member input files you
> > specified on the command line. For each, it replaces the special
> > keyword "MET_PYTHON_INPUT_ARG" with the current file being
processed.
> > And then it executes that command to retrieve the gridded data for
> > that ensemble member.
> >
> > Make sense?
> >
> > Please give it a shot and let me know if you run into issues or
have
> > other questions.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Thu, Jan 7, 2021 at 11:54 AM Satterfield, Dr. Elizabeth via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jan 07 11:53:43 2021: Request 98153 was acted upon.
> > > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > > Queue: met_help
> > > Subject: python embedding for ensemble_stat
> > > Owner: Nobody
> > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153
> > > >
> > >
> > >
> > > John G. and others.
> > >
> > >
> > >
> > > At one point you helped us with a python reader for our binary
data
> > > files, 'read_NRL_binary.py'. Now that MET supports python
embedding
> > > for ensemble_stat, what we'd like is to use that script, or
similar,
> > > to read in the ensemble files.
> > >
> > >
> > >
> > > Currently, we have a separate file for each ensemble member. My
> > > question is
> > > - what is the best way to do this? Use read_nrl_binary 20 times
for
> > > each ensemble member or create a different script to pass all 20
> > > members simultaneously?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Liz
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Elizabeth Satterfield
> > > Editor, Monthly Weather Review
> > >
> > > Chair, AMS Committee on Probability and Statistics
> > >
> > > Meteorologist, Code 7531
> > > Marine Meteorology Division
> > > U.S. Naval Research Laboratory
> > > T 831.656.4056 F 831.656.4026
> > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > >
> > >
__________________________________________________________________
> > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE Any misuse or
unauthorized
> > > disclosure may result in both civil and criminal penalties.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: python embedding for ensemble_stat
From: Satterfield, Dr. Elizabeth
Time: Wed Jan 13 16:43:48 2021
Yes -- I wasn't clear on that.
I had the fcst dictionary set with the field name, but not the python
embedding keywords. Once I changed that everything worked.
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, January 13, 2021 3:36 PM
To: Satterfield, Dr. Elizabeth <Elizabeth.Satterfield at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
ensemble_stat
Liz,
Great! Glad you figured it out. Just so I understand, was the problem
that you
sent the "ens" dictionary but did not set the "fcst" dictionary?
John
On Wed, Jan 13, 2021 at 4:25 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153 >
>
> Ah...I see my mistake. Seems to be working now!
>
> fcst = {
> file_type = PYTHON_NUMPY;
> field = [ { name = "./read_NRL_binary.py MET_PYTHON_INPUT_ARG"; }
> ]; }
>
>
> -----Original Message-----
> From: Satterfield, Dr. Elizabeth
> Sent: Wednesday, January 13, 2021 2:01 PM
> To: 'met_help at ucar.edu' <met_help at ucar.edu>
> Subject: RE: [rt.rap.ucar.edu #98153] python embedding for
> ensemble_stat
>
> John,
>
> One more quick question. How does the fcst dictionary need to be
set
> for this case. I have it set with the name/level of the ensemble
> fields.
>
>
> //
> // Forecast and observation fields to be verified // fcst = {
> field = [
> {
> name = "geopht";
> level = [ "pre_0500.0" ];
> }
> ];
> }
>
> obs = {
> field = [
> {
> name = "./read_NRL_binary.py
>
verif/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld";
> }
> ];
> }
>
> Thanks,
> Liz
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Wednesday, January 13, 2021 8:44 AM
> To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
> ensemble_stat
>
> Liz,
>
> I don't see any obvious problems in what you sent to me.
>
> To avoid potential path problems, I'd recommend replacing
> "./read_NRL_binary.py" with the full path to that script instead.
But
> I'm doubtful that'll fix everything.
>
> Next question, what version of MET are you running. Checking the
> release notes, I see that MET_PYTHON_INPUT_ARG was added in met-9.0.
>
> If you think it'd be useful to send me sample data for testing, I'd
be
> happy to do that.
>
> Thanks,
> John
>
> On Tue, Jan 12, 2021 at 5:59 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153 >
> >
> > Hi John,
> >
> > I'm having trouble getting this to work.
> >
> > In my config, I have the following:
> > ens = {
> > file_type = PYTHON_NUMPY;
> > field = [ { name = "./read_NRL_binary.py MET_PYTHON_INPUT_ARG";
}
> > ]; }
> >
> > I'm trying to run ensemble_stat as follows:
> > ensemble_stat ens_file_list
> >
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/config/Ensemb
> > le
> > StatConfigACOR_2016072200
> >
> > -grid_obs
> >
/unified_verif/work/workliz/uv/scripts/NAVGEM_ensemble/test_data_nc/
> > ve rif/ test_data_geopht_pre_0500_2016072200_000_000.nc
> > -outdir OUT_ENS_test_data/geopht_pre_0500_120_2016072200
> >
> >
> > I'm wondering if I need an additional argument, since I'm getting
> > the following errors:
> > WARNING:
> > WARNING: python_dataplane() -> an error occurred importing module
> > "geopht_pre_0500.0"
> > WARNING:
> > WARNING:
> > WARNING: process_n_vld() -> no data found for forecast field
"PYTHON"
> > in file
> >
> >
>
"test_data/g2016072200/mbr001/outp/geopht_pre_0500.0_0000.0_glob360x181_2016072200_00000000_fcstfld"
> >
> > Thanks,
> > Liz
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 7, 2021 11:56 AM
> > To: Satterfield, Dr. Elizabeth
> > <Elizabeth.Satterfield at nrlmry.navy.mil>
> > Subject: Re: [rt.rap.ucar.edu #98153] python embedding for
> > ensemble_stat
> >
> > Hi Liz,
> >
> > The python embedding functionality in Ensemble-Stat still only
> > supports the reading of a single field at a time.
> >
> > So Ensemble-Stat will call that script 20 times to read data from
> > those 20 files... in the same way that it'd read data from 20
> > different GRIB input files.
> >
> > We have a toy example of running ensemble-stat with python
embedding
> > in our unit tests:
> >
> >
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254c
> > cc
> > 5981bb8/test/xml/unit_python.xml#L239
> >
> > But it's a little difficult to read through the XML. Here's the
sort
> > of command you'd run using read_NRL_binary.py
> >
> > ensemble_stat ens_file_list EnsembleStatConfig_NRL -point_obs
> > point_obs_data.nc
> >
> > Where...
> > 'ens_file_list' is an ASCII file containing the list of ensemble
> > member binary data files.
> > 'point_obs_data.nc' contains point obs generated by pb2nc or
ascii2nc.
> > You
> > could also verify against gridded data using the "-grid_obs"
command
> > line option.
> > 'EnsembleStatConfig_NRL' is an Ensemble-Stat config file which
> > contains settings like:
> >
> > ens = {
> > file_type = PYTHON_NUMPY;
> > field = [ { name = "/path/to/read_NRL_binary.py
> > MET_PYTHON_INPUT_ARG"; } ]; }
> >
> > So the logic is... when Ensemble-Stat sees that the file_type is
set
> > to PYTHON_NUMPY, it loops over the ensemble member input files you
> > specified on the command line. For each, it replaces the special
> > keyword "MET_PYTHON_INPUT_ARG" with the current file being
processed.
> > And then it executes that command to retrieve the gridded data for
> > that ensemble member.
> >
> > Make sense?
> >
> > Please give it a shot and let me know if you run into issues or
have
> > other questions.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Thu, Jan 7, 2021 at 11:54 AM Satterfield, Dr. Elizabeth via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jan 07 11:53:43 2021: Request 98153 was acted upon.
> > > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > > Queue: met_help
> > > Subject: python embedding for ensemble_stat
> > > Owner: Nobody
> > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98153
> > > >
> > >
> > >
> > > John G. and others.
> > >
> > >
> > >
> > > At one point you helped us with a python reader for our binary
> > > data files, 'read_NRL_binary.py'. Now that MET supports python
> > > embedding for ensemble_stat, what we'd like is to use that
script,
> > > or similar, to read in the ensemble files.
> > >
> > >
> > >
> > > Currently, we have a separate file for each ensemble member. My
> > > question is
> > > - what is the best way to do this? Use read_nrl_binary 20 times
> > > for each ensemble member or create a different script to pass
all
> > > 20 members simultaneously?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Liz
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Elizabeth Satterfield
> > > Editor, Monthly Weather Review
> > >
> > > Chair, AMS Committee on Probability and Statistics
> > >
> > > Meteorologist, Code 7531
> > > Marine Meteorology Division
> > > U.S. Naval Research Laboratory
> > > T 831.656.4056 F 831.656.4026
> > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > >
> > >
__________________________________________________________________
> > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE Any misuse or
> > > unauthorized disclosure may result in both civil and criminal
penalties.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list