[Met_help] [rt.rap.ucar.edu #95583] History for Series Analysis Config File Issues
John Halley Gotway via RT
met_help at ucar.edu
Fri Jun 19 12:39:07 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning MET help
I'm having an issue with series_analysis that I could use your assistance
with. I am currently testing series_analysis for QPF06 (config file
attached). I am unable to get valid output values for anything in the CTS
and MCTS categories. The TOTAL grid for each score group looks good (I'm
currently testing with 7 days of data). The CSI values are all constant at
1. I am unable to get any values for HSS, either from CTS or MCTS. All
output (MAE, ME, RMSE) from the CNT score group seem to work fine.
I'm assuming the issue is setting the thresholds in my config file.
The scores that I see in my netCDF file for the CTS scores all end with
"_NA", where I'm assuming a category should be set. I am using the same
thresholds that I use in grid_stat. Should I be setting them up
differently for series_analysis? Is there a sample config file that I
could look at for reference?
If it helps, my test output file can be found on WCOSS
here: /gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
Thanks
John
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Series Analysis Config File Issues
From: Minna Win
Time: Thu Jun 11 08:58:30 2020
Hello John,
It looks like you need some assistance setting up your series analysis
in
MET. I'm assigning this ticket to John Halley Gotway. Please allow a
few
business days for a response.
Regards,
Minna
---------------
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax: 303-497-8401
On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> Transaction: Ticket created by john.l.wagner at noaa.gov
> Queue: met_help
> Subject: Series Analysis Config File Issues
> Owner: Nobody
> Requestors: john.l.wagner at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
>
> Good morning MET help
> I'm having an issue with series_analysis that I could use your
assistance
> with. I am currently testing series_analysis for QPF06 (config file
> attached). I am unable to get valid output values for anything in
the CTS
> and MCTS categories. The TOTAL grid for each score group looks good
(I'm
> currently testing with 7 days of data). The CSI values are all
constant at
> 1. I am unable to get any values for HSS, either from CTS or MCTS.
All
> output (MAE, ME, RMSE) from the CNT score group seem to work fine.
> I'm assuming the issue is setting the thresholds in my config file.
> The scores that I see in my netCDF file for the CTS scores all end
with
> "_NA", where I'm assuming a category should be set. I am using the
same
> thresholds that I use in grid_stat. Should I be setting them up
> differently for series_analysis? Is there a sample config file that
I
> could look at for reference?
> If it helps, my test output file can be found on WCOSS
> here:
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
>
> Thanks
> John
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Thu Jun 11 09:59:17 2020
Hello John,
I see you're having difficulty configuring Series-Analysis to apply
the
continuous and categorical statistics you defined.
It's funny, this exact same issue arose via met-help last week, and it
makes me think we should modify the code in some way to make it more
intuitive. Please try editing your config file by moving the
cnt_thresh
and cat_thresh settings up outside of the field array, as shown below:
*fcst = {*
* cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
>=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4, >=25.4&&<76.2, >=76.2
];*
* cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05, >=25.4,
>=50.8, >=76.2 ];*
* field = [ { name = "${FIELD}"; level =
"${LEVEL}";
} ];}obs = fcst;*
When you rerun after this change, does it produce the expected output?
I realize that this is confusing, especially if you're used to setting
up
config files for Point-Stat and Grid-Stat. Those tools are set up to
process multiple fields in a single run. Therefore you can specify
cat_thresh and cnt_thresh separately for each field array entry.
Unlike
Point-Stat and Grid-Stat, Series-Analysis can only process a single
series
of data in each run. There are multiple main ways of defining a
series.
(1) By far, the most common way is specifying a single field that is
extracted from a time-series of input data files... as you're doing.
(2) You can also define a series of field array entries that can be
extracted from the same input file. For example, compute stats over
multiple vertical levels from the same input file.
(3) You define N input fields to be extracted from N input files.
Because of options (2) and (3), the field entry is defined as an array
rather than a single dictionary. In practice though, the field array
typically has length 1.
But the cat_thresh and cnt_thresh settings must remain constant over
the
entire series, and that's why the code is setup to parse them from the
"fcst" and "obs" dictionaries, and not separately from each field
array
entry.
Seems like it'd be a good idea to check for this situation and print a
warning message for the user.
Thanks,
John
On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
> Hello John,
>
> It looks like you need some assistance setting up your series
analysis in
> MET. I'm assigning this ticket to John Halley Gotway. Please allow
a few
> business days for a response.
>
> Regards,
> Minna
> ---------------
> Minna Win
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax: 303-497-8401
>
>
>
> On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > Transaction: Ticket created by john.l.wagner at noaa.gov
> > Queue: met_help
> > Subject: Series Analysis Config File Issues
> > Owner: Nobody
> > Requestors: john.l.wagner at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >
> >
> > Good morning MET help
> > I'm having an issue with series_analysis that I could use your
assistance
> > with. I am currently testing series_analysis for QPF06 (config
file
> > attached). I am unable to get valid output values for anything in
the
> CTS
> > and MCTS categories. The TOTAL grid for each score group looks
good (I'm
> > currently testing with 7 days of data). The CSI values are all
constant
> at
> > 1. I am unable to get any values for HSS, either from CTS or
MCTS. All
> > output (MAE, ME, RMSE) from the CNT score group seem to work fine.
> > I'm assuming the issue is setting the thresholds in my config
file.
> > The scores that I see in my netCDF file for the CTS scores all end
with
> > "_NA", where I'm assuming a category should be set. I am using
the same
> > thresholds that I use in grid_stat. Should I be setting them up
> > differently for series_analysis? Is there a sample config file
that I
> > could look at for reference?
> > If it helps, my test output file can be found on WCOSS
> > here:
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> >
> > Thanks
> > John
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John L Wagner - NOAA Federal
Time: Thu Jun 11 13:39:51 2020
Thanks John. I'm testing the change now.
I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just 7 days
of
data. Its been running for over 2 hours now. Is this the expected
behavior? I do have -v set to 3. Its writing steadily to the log.
Is
that slowing things down?
On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hello John,
>
> I see you're having difficulty configuring Series-Analysis to apply
the
> continuous and categorical statistics you defined.
>
> It's funny, this exact same issue arose via met-help last week, and
it
> makes me think we should modify the code in some way to make it more
> intuitive. Please try editing your config file by moving the
cnt_thresh
> and cat_thresh settings up outside of the field array, as shown
below:
>
> *fcst = {*
>
> * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
> >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4, >=25.4&&<76.2, >=76.2
];*
> * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05, >=25.4,
> >=50.8, >=76.2 ];*
>
>
>
>
>
>
>
>
> * field = [ { name = "${FIELD}"; level =
"${LEVEL}";
> } ];}obs = fcst;*
>
> When you rerun after this change, does it produce the expected
output?
>
> I realize that this is confusing, especially if you're used to
setting up
> config files for Point-Stat and Grid-Stat. Those tools are set up to
> process multiple fields in a single run. Therefore you can specify
> cat_thresh and cnt_thresh separately for each field array entry.
Unlike
> Point-Stat and Grid-Stat, Series-Analysis can only process a single
series
> of data in each run. There are multiple main ways of defining a
series.
> (1) By far, the most common way is specifying a single field that is
> extracted from a time-series of input data files... as you're doing.
> (2) You can also define a series of field array entries that can be
> extracted from the same input file. For example, compute stats over
> multiple vertical levels from the same input file.
> (3) You define N input fields to be extracted from N input files.
>
> Because of options (2) and (3), the field entry is defined as an
array
> rather than a single dictionary. In practice though, the field array
> typically has length 1.
>
> But the cat_thresh and cnt_thresh settings must remain constant over
the
> entire series, and that's why the code is setup to parse them from
the
> "fcst" and "obs" dictionaries, and not separately from each field
array
> entry.
>
> Seems like it'd be a good idea to check for this situation and print
a
> warning message for the user.
>
> Thanks,
> John
>
> On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >
> > Hello John,
> >
> > It looks like you need some assistance setting up your series
analysis in
> > MET. I'm assigning this ticket to John Halley Gotway. Please
allow a
> few
> > business days for a response.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax: 303-497-8401
> >
> >
> >
> > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > Queue: met_help
> > > Subject: Series Analysis Config File Issues
> > > Owner: Nobody
> > > Requestors: john.l.wagner at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> >
> > >
> > >
> > > Good morning MET help
> > > I'm having an issue with series_analysis that I could use your
> assistance
> > > with. I am currently testing series_analysis for QPF06 (config
file
> > > attached). I am unable to get valid output values for anything
in the
> > CTS
> > > and MCTS categories. The TOTAL grid for each score group looks
good
> (I'm
> > > currently testing with 7 days of data). The CSI values are all
> constant
> > at
> > > 1. I am unable to get any values for HSS, either from CTS or
MCTS.
> All
> > > output (MAE, ME, RMSE) from the CNT score group seem to work
fine.
> > > I'm assuming the issue is setting the thresholds in my config
file.
> > > The scores that I see in my netCDF file for the CTS scores all
end with
> > > "_NA", where I'm assuming a category should be set. I am using
the
> same
> > > thresholds that I use in grid_stat. Should I be setting them up
> > > differently for series_analysis? Is there a sample config file
that I
> > > could look at for reference?
> > > If it helps, my test output file can be found on WCOSS
> > > here:
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > >
> > > Thanks
> > > John
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Division
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> > >
> >
> >
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Thu Jun 11 15:24:06 2020
John,
I'd recommend modifying the "block_size" config file option and
rerunning.
The default value is set very low, ensuring that you don't run out of
memory but requiring many passes through the data... which is slow.
Here's an excerpt from this README file which describes how that
works.
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
////////////////////////////////////////////////////////////////////////////////
//
// SeriesAnalysisConfig_default
//
////////////////////////////////////////////////////////////////////////////////
//
// Computation may be memory intensive, especially for large grids.
// The "block_size" entry sets the number of grid points to be
processed
// concurrently (i.e. in one pass through a time series). Smaller
values
// require less memory but increase the number of passes through the
data.
//
block_size = 1024;
Look near the top of your log file for a message that looks something
like
this:
*DEBUG 2: Computing statistics using a block size of 10000, requiring
3
pass(es) through the 169 x 154 grid.DEBUG 2: Processing data pass
number 1
of 3 for grid points 1 to 10000.*
This will tell you how many times it's reading through the input data
files. Ideally you'd set the block size high enough so that it only
needs
to read them once. But that all depends on the number of grid points
and
length of your series. If you set the block_size too high, it may
consume
all the memory.
Hope that helps clarify.
Thanks,
John
On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
> Thanks John. I'm testing the change now.
> I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just 7 days
of
> data. Its been running for over 2 hours now. Is this the expected
> behavior? I do have -v set to 3. Its writing steadily to the log.
Is
> that slowing things down?
>
> On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Hello John,
> >
> > I see you're having difficulty configuring Series-Analysis to
apply the
> > continuous and categorical statistics you defined.
> >
> > It's funny, this exact same issue arose via met-help last week,
and it
> > makes me think we should modify the code in some way to make it
more
> > intuitive. Please try editing your config file by moving the
cnt_thresh
> > and cat_thresh settings up outside of the field array, as shown
below:
> >
> > *fcst = {*
> >
> > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
> > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4, >=25.4&&<76.2,
>=76.2 ];*
> > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05,
>=25.4,
> > >=50.8, >=76.2 ];*
> >
> >
> >
> >
> >
> >
> >
> >
> > * field = [ { name = "${FIELD}"; level =
> "${LEVEL}";
> > } ];}obs = fcst;*
> >
> > When you rerun after this change, does it produce the expected
output?
> >
> > I realize that this is confusing, especially if you're used to
setting up
> > config files for Point-Stat and Grid-Stat. Those tools are set up
to
> > process multiple fields in a single run. Therefore you can specify
> > cat_thresh and cnt_thresh separately for each field array entry.
Unlike
> > Point-Stat and Grid-Stat, Series-Analysis can only process a
single
> series
> > of data in each run. There are multiple main ways of defining a
series.
> > (1) By far, the most common way is specifying a single field that
is
> > extracted from a time-series of input data files... as you're
doing.
> > (2) You can also define a series of field array entries that can
be
> > extracted from the same input file. For example, compute stats
over
> > multiple vertical levels from the same input file.
> > (3) You define N input fields to be extracted from N input files.
> >
> > Because of options (2) and (3), the field entry is defined as an
array
> > rather than a single dictionary. In practice though, the field
array
> > typically has length 1.
> >
> > But the cat_thresh and cnt_thresh settings must remain constant
over the
> > entire series, and that's why the code is setup to parse them from
the
> > "fcst" and "obs" dictionaries, and not separately from each field
array
> > entry.
> >
> > Seems like it'd be a good idea to check for this situation and
print a
> > warning message for the user.
> >
> > Thanks,
> > John
> >
> > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > >
> > > Hello John,
> > >
> > > It looks like you need some assistance setting up your series
analysis
> in
> > > MET. I'm assigning this ticket to John Halley Gotway. Please
allow a
> > few
> > > business days for a response.
> > >
> > > Regards,
> > > Minna
> > > ---------------
> > > Minna Win
> > > National Center for Atmospheric Research
> > > Developmental Testbed Center
> > > Phone: 303-497-8423
> > > Fax: 303-497-8401
> > >
> > >
> > >
> > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > Queue: met_help
> > > > Subject: Series Analysis Config File Issues
> > > > Owner: Nobody
> > > > Requestors: john.l.wagner at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > >
> > > >
> > > >
> > > > Good morning MET help
> > > > I'm having an issue with series_analysis that I could use your
> > assistance
> > > > with. I am currently testing series_analysis for QPF06
(config file
> > > > attached). I am unable to get valid output values for
anything in
> the
> > > CTS
> > > > and MCTS categories. The TOTAL grid for each score group looks
good
> > (I'm
> > > > currently testing with 7 days of data). The CSI values are
all
> > constant
> > > at
> > > > 1. I am unable to get any values for HSS, either from CTS or
MCTS.
> > All
> > > > output (MAE, ME, RMSE) from the CNT score group seem to work
fine.
> > > > I'm assuming the issue is setting the thresholds in my config
file.
> > > > The scores that I see in my netCDF file for the CTS scores all
end
> with
> > > > "_NA", where I'm assuming a category should be set. I am
using the
> > same
> > > > thresholds that I use in grid_stat. Should I be setting them
up
> > > > differently for series_analysis? Is there a sample config
file that
> I
> > > > could look at for reference?
> > > > If it helps, my test output file can be found on WCOSS
> > > > here:
> > > >
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > > >
> > > > Thanks
> > > > John
> > > > --
> > > > John Wagner
> > > > Verification Task Lead
> > > > NOAA/National Weather Service
> > > > Meteorological Development Laboratory
> > > > Digital Forecast Services Division
> > > > SSMC2 Room 10106
> > > > Silver Spring, MD 20910
> > > > (301) 427-9471 (office)
> > > > (908) 902-4155 (cell/text)
> > > >
> > > >
> > >
> > >
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John L Wagner - NOAA Federal
Time: Thu Jun 11 16:46:54 2020
I set the block size to my grid size (4194304) previously. Its still
running 5+ hours later. It will time out after 6 hours.
I have been testing on WCOSS with Julie's MET V9.0. Tomorrow I'll run
a
test with V8.1. Otherwise, we'll need to rethink what we want to show
in
the viewer.
Thanks
John
On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> John,
>
> I'd recommend modifying the "block_size" config file option and
rerunning.
> The default value is set very low, ensuring that you don't run out
of
> memory but requiring many passes through the data... which is slow.
>
> Here's an excerpt from this README file which describes how that
works.
>
> https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
>
>
>
////////////////////////////////////////////////////////////////////////////////
> //
> // SeriesAnalysisConfig_default
> //
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Computation may be memory intensive, especially for large grids.
> // The "block_size" entry sets the number of grid points to be
processed
> // concurrently (i.e. in one pass through a time series). Smaller
values
> // require less memory but increase the number of passes through the
data.
> //
> block_size = 1024;
>
> Look near the top of your log file for a message that looks
something like
> this:
>
>
> *DEBUG 2: Computing statistics using a block size of 10000,
requiring 3
> pass(es) through the 169 x 154 grid.DEBUG 2: Processing data pass
number 1
> of 3 for grid points 1 to 10000.*
>
> This will tell you how many times it's reading through the input
data
> files. Ideally you'd set the block size high enough so that it only
needs
> to read them once. But that all depends on the number of grid points
and
> length of your series. If you set the block_size too high, it may
consume
> all the memory.
>
> Hope that helps clarify.
>
> Thanks,
> John
>
> On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >
> > Thanks John. I'm testing the change now.
> > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just 7
days of
> > data. Its been running for over 2 hours now. Is this the
expected
> > behavior? I do have -v set to 3. Its writing steadily to the
log. Is
> > that slowing things down?
> >
> > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Hello John,
> > >
> > > I see you're having difficulty configuring Series-Analysis to
apply the
> > > continuous and categorical statistics you defined.
> > >
> > > It's funny, this exact same issue arose via met-help last week,
and it
> > > makes me think we should modify the code in some way to make it
more
> > > intuitive. Please try editing your config file by moving the
> cnt_thresh
> > > and cat_thresh settings up outside of the field array, as shown
below:
> > >
> > > *fcst = {*
> > >
> > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
> > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4, >=25.4&&<76.2,
>=76.2
> ];*
> > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05,
>=25.4,
> > > >=50.8, >=76.2 ];*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > * field = [ { name = "${FIELD}"; level =
> > "${LEVEL}";
> > > } ];}obs = fcst;*
> > >
> > > When you rerun after this change, does it produce the expected
output?
> > >
> > > I realize that this is confusing, especially if you're used to
setting
> up
> > > config files for Point-Stat and Grid-Stat. Those tools are set
up to
> > > process multiple fields in a single run. Therefore you can
specify
> > > cat_thresh and cnt_thresh separately for each field array entry.
Unlike
> > > Point-Stat and Grid-Stat, Series-Analysis can only process a
single
> > series
> > > of data in each run. There are multiple main ways of defining a
series.
> > > (1) By far, the most common way is specifying a single field
that is
> > > extracted from a time-series of input data files... as you're
doing.
> > > (2) You can also define a series of field array entries that can
be
> > > extracted from the same input file. For example, compute stats
over
> > > multiple vertical levels from the same input file.
> > > (3) You define N input fields to be extracted from N input
files.
> > >
> > > Because of options (2) and (3), the field entry is defined as an
array
> > > rather than a single dictionary. In practice though, the field
array
> > > typically has length 1.
> > >
> > > But the cat_thresh and cnt_thresh settings must remain constant
over
> the
> > > entire series, and that's why the code is setup to parse them
from the
> > > "fcst" and "obs" dictionaries, and not separately from each
field array
> > > entry.
> > >
> > > Seems like it'd be a good idea to check for this situation and
print a
> > > warning message for the user.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>
> > > >
> > > > Hello John,
> > > >
> > > > It looks like you need some assistance setting up your series
> analysis
> > in
> > > > MET. I'm assigning this ticket to John Halley Gotway. Please
allow
> a
> > > few
> > > > business days for a response.
> > > >
> > > > Regards,
> > > > Minna
> > > > ---------------
> > > > Minna Win
> > > > National Center for Atmospheric Research
> > > > Developmental Testbed Center
> > > > Phone: 303-497-8423
> > > > Fax: 303-497-8401
> > > >
> > > >
> > > >
> > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal
via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: Series Analysis Config File Issues
> > > > > Owner: Nobody
> > > > > Requestors: john.l.wagner at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > > >
> > > > >
> > > > >
> > > > > Good morning MET help
> > > > > I'm having an issue with series_analysis that I could use
your
> > > assistance
> > > > > with. I am currently testing series_analysis for QPF06
(config
> file
> > > > > attached). I am unable to get valid output values for
anything in
> > the
> > > > CTS
> > > > > and MCTS categories. The TOTAL grid for each score group
looks good
> > > (I'm
> > > > > currently testing with 7 days of data). The CSI values are
all
> > > constant
> > > > at
> > > > > 1. I am unable to get any values for HSS, either from CTS
or MCTS.
> > > All
> > > > > output (MAE, ME, RMSE) from the CNT score group seem to work
fine.
> > > > > I'm assuming the issue is setting the thresholds in my
config file.
> > > > > The scores that I see in my netCDF file for the CTS scores
all end
> > with
> > > > > "_NA", where I'm assuming a category should be set. I am
using the
> > > same
> > > > > thresholds that I use in grid_stat. Should I be setting
them up
> > > > > differently for series_analysis? Is there a sample config
file
> that
> > I
> > > > > could look at for reference?
> > > > > If it helps, my test output file can be found on WCOSS
> > > > > here:
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > > > >
> > > > > Thanks
> > > > > John
> > > > > --
> > > > > John Wagner
> > > > > Verification Task Lead
> > > > > NOAA/National Weather Service
> > > > > Meteorological Development Laboratory
> > > > > Digital Forecast Services Division
> > > > > SSMC2 Room 10106
> > > > > Silver Spring, MD 20910
> > > > > (301) 427-9471 (office)
> > > > > (908) 902-4155 (cell/text)
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Thu Jun 11 17:07:58 2020
Agreed, that's much too long! I suppose another possibility is that
it's
consuming too much memory and is using swap space, which is slow.
If possible, please check what percent of memory the process is using.
If
it's up around 100%, then let's try cutting the block size in half and
rerunning.
Honestly, the real fix for this is enhancing Series-Analysis with MPI
calls
to subset the 4+ million grid point into tiles, and then do the
processing
for each tile on a separate node.
However, none of the MET tools are currently parallelized. So that'd
obviously be a significant development effort.
Another option, just to get something to actually complete in the time
window, would be applying a masking region to subset the data being
processed.
Thanks,
John
On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
> I set the block size to my grid size (4194304) previously. Its
still
> running 5+ hours later. It will time out after 6 hours.
> I have been testing on WCOSS with Julie's MET V9.0. Tomorrow I'll
run a
> test with V8.1. Otherwise, we'll need to rethink what we want to
show in
> the viewer.
> Thanks
> John
>
> On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > John,
> >
> > I'd recommend modifying the "block_size" config file option and
> rerunning.
> > The default value is set very low, ensuring that you don't run out
of
> > memory but requiring many passes through the data... which is
slow.
> >
> > Here's an excerpt from this README file which describes how that
works.
> >
> >
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > //
> > // SeriesAnalysisConfig_default
> > //
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Computation may be memory intensive, especially for large
grids.
> > // The "block_size" entry sets the number of grid points to be
processed
> > // concurrently (i.e. in one pass through a time series). Smaller
values
> > // require less memory but increase the number of passes through
the
> data.
> > //
> > block_size = 1024;
> >
> > Look near the top of your log file for a message that looks
something
> like
> > this:
> >
> >
> > *DEBUG 2: Computing statistics using a block size of 10000,
requiring 3
> > pass(es) through the 169 x 154 grid.DEBUG 2: Processing data pass
number
> 1
> > of 3 for grid points 1 to 10000.*
> >
> > This will tell you how many times it's reading through the input
data
> > files. Ideally you'd set the block size high enough so that it
only needs
> > to read them once. But that all depends on the number of grid
points and
> > length of your series. If you set the block_size too high, it may
consume
> > all the memory.
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John
> >
> > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > >
> > > Thanks John. I'm testing the change now.
> > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just 7
days of
> > > data. Its been running for over 2 hours now. Is this the
expected
> > > behavior? I do have -v set to 3. Its writing steadily to the
log. Is
> > > that slowing things down?
> > >
> > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Hello John,
> > > >
> > > > I see you're having difficulty configuring Series-Analysis to
apply
> the
> > > > continuous and categorical statistics you defined.
> > > >
> > > > It's funny, this exact same issue arose via met-help last
week, and
> it
> > > > makes me think we should modify the code in some way to make
it more
> > > > intuitive. Please try editing your config file by moving the
> > cnt_thresh
> > > > and cat_thresh settings up outside of the field array, as
shown
> below:
> > > >
> > > > *fcst = {*
> > > >
> > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
> > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4, >=25.4&&<76.2,
>=76.2
> > ];*
> > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05,
>=25.4,
> > > > >=50.8, >=76.2 ];*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > * field = [ { name = "${FIELD}"; level =
> > > "${LEVEL}";
> > > > } ];}obs = fcst;*
> > > >
> > > > When you rerun after this change, does it produce the expected
> output?
> > > >
> > > > I realize that this is confusing, especially if you're used to
> setting
> > up
> > > > config files for Point-Stat and Grid-Stat. Those tools are set
up to
> > > > process multiple fields in a single run. Therefore you can
specify
> > > > cat_thresh and cnt_thresh separately for each field array
entry.
> Unlike
> > > > Point-Stat and Grid-Stat, Series-Analysis can only process a
single
> > > series
> > > > of data in each run. There are multiple main ways of defining
a
> series.
> > > > (1) By far, the most common way is specifying a single field
that is
> > > > extracted from a time-series of input data files... as you're
doing.
> > > > (2) You can also define a series of field array entries that
can be
> > > > extracted from the same input file. For example, compute stats
over
> > > > multiple vertical levels from the same input file.
> > > > (3) You define N input fields to be extracted from N input
files.
> > > >
> > > > Because of options (2) and (3), the field entry is defined as
an
> array
> > > > rather than a single dictionary. In practice though, the field
array
> > > > typically has length 1.
> > > >
> > > > But the cat_thresh and cnt_thresh settings must remain
constant over
> > the
> > > > entire series, and that's why the code is setup to parse them
from
> the
> > > > "fcst" and "obs" dictionaries, and not separately from each
field
> array
> > > > entry.
> > > >
> > > > Seems like it'd be a good idea to check for this situation and
print
> a
> > > > warning message for the user.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT
<met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > > > >
> > > > > Hello John,
> > > > >
> > > > > It looks like you need some assistance setting up your
series
> > analysis
> > > in
> > > > > MET. I'm assigning this ticket to John Halley Gotway.
Please
> allow
> > a
> > > > few
> > > > > business days for a response.
> > > > >
> > > > > Regards,
> > > > > Minna
> > > > > ---------------
> > > > > Minna Win
> > > > > National Center for Atmospheric Research
> > > > > Developmental Testbed Center
> > > > > Phone: 303-497-8423
> > > > > Fax: 303-497-8401
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal
via
> RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: Series Analysis Config File Issues
> > > > > > Owner: Nobody
> > > > > > Requestors: john.l.wagner at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > > > >
> > > > > >
> > > > > >
> > > > > > Good morning MET help
> > > > > > I'm having an issue with series_analysis that I could use
your
> > > > assistance
> > > > > > with. I am currently testing series_analysis for QPF06
(config
> > file
> > > > > > attached). I am unable to get valid output values for
anything
> in
> > > the
> > > > > CTS
> > > > > > and MCTS categories. The TOTAL grid for each score group
looks
> good
> > > > (I'm
> > > > > > currently testing with 7 days of data). The CSI values
are all
> > > > constant
> > > > > at
> > > > > > 1. I am unable to get any values for HSS, either from CTS
or
> MCTS.
> > > > All
> > > > > > output (MAE, ME, RMSE) from the CNT score group seem to
work
> fine.
> > > > > > I'm assuming the issue is setting the thresholds in my
config
> file.
> > > > > > The scores that I see in my netCDF file for the CTS scores
all
> end
> > > with
> > > > > > "_NA", where I'm assuming a category should be set. I am
using
> the
> > > > same
> > > > > > thresholds that I use in grid_stat. Should I be setting
them up
> > > > > > differently for series_analysis? Is there a sample config
file
> > that
> > > I
> > > > > > could look at for reference?
> > > > > > If it helps, my test output file can be found on WCOSS
> > > > > > here:
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > > > > >
> > > > > > Thanks
> > > > > > John
> > > > > > --
> > > > > > John Wagner
> > > > > > Verification Task Lead
> > > > > > NOAA/National Weather Service
> > > > > > Meteorological Development Laboratory
> > > > > > Digital Forecast Services Division
> > > > > > SSMC2 Room 10106
> > > > > > Silver Spring, MD 20910
> > > > > > (301) 427-9471 (office)
> > > > > > (908) 902-4155 (cell/text)
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Division
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> > >
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John L Wagner - NOAA Federal
Time: Fri Jun 12 05:39:39 2020
Here is the usage summary from the test I ran yesterday:
[image: SerA Test1 Log.PNG]
The job did not time out, and finished in 5 hr 43 min. It did not use
all
of the memory, which surprised me. When I was testing incorrectly
with the
thresholds defined within the field array, I had to keep increasing
the
memory setting in my bsub command to prevent OOM kills.
I'm running a test now with just the lowest CNT threshold. This may
be
what we end up running anyway and I'm hoping the run time will come
down so
I can run more tests. I'll try to get through tests today with
cutting the
block_size in half, using a mask, and different memory settings with
bsub.
I'll let you know what I find.
As for MPI, I agree with that solution. I'm not going to pretend to
know
what needs to be changed with C code and compilers. When we updated
the
MOS-2000 Fortran code, we managed to get our ensemble code run time
down
from 3 hours to 12 minutes. I realize that not all MET users are
running
on supercomputers that can take advantage of MPI, but those of us that
are
would certainly appreciate it.
Thanks
John
On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Agreed, that's much too long! I suppose another possibility is that
it's
> consuming too much memory and is using swap space, which is slow.
> If possible, please check what percent of memory the process is
using. If
> it's up around 100%, then let's try cutting the block size in half
and
> rerunning.
>
> Honestly, the real fix for this is enhancing Series-Analysis with
MPI calls
> to subset the 4+ million grid point into tiles, and then do the
processing
> for each tile on a separate node.
> However, none of the MET tools are currently parallelized. So that'd
> obviously be a significant development effort.
>
> Another option, just to get something to actually complete in the
time
> window, would be applying a masking region to subset the data being
> processed.
>
> Thanks,
> John
>
> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >
> > I set the block size to my grid size (4194304) previously. Its
still
> > running 5+ hours later. It will time out after 6 hours.
> > I have been testing on WCOSS with Julie's MET V9.0. Tomorrow I'll
run a
> > test with V8.1. Otherwise, we'll need to rethink what we want to
show in
> > the viewer.
> > Thanks
> > John
> >
> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > John,
> > >
> > > I'd recommend modifying the "block_size" config file option and
> > rerunning.
> > > The default value is set very low, ensuring that you don't run
out of
> > > memory but requiring many passes through the data... which is
slow.
> > >
> > > Here's an excerpt from this README file which describes how that
works.
> > >
> > >
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > //
> > > // SeriesAnalysisConfig_default
> > > //
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Computation may be memory intensive, especially for large
grids.
> > > // The "block_size" entry sets the number of grid points to be
> processed
> > > // concurrently (i.e. in one pass through a time series).
Smaller
> values
> > > // require less memory but increase the number of passes through
the
> > data.
> > > //
> > > block_size = 1024;
> > >
> > > Look near the top of your log file for a message that looks
something
> > like
> > > this:
> > >
> > >
> > > *DEBUG 2: Computing statistics using a block size of 10000,
requiring 3
> > > pass(es) through the 169 x 154 grid.DEBUG 2: Processing data
pass
> number
> > 1
> > > of 3 for grid points 1 to 10000.*
> > >
> > > This will tell you how many times it's reading through the input
data
> > > files. Ideally you'd set the block size high enough so that it
only
> needs
> > > to read them once. But that all depends on the number of grid
points
> and
> > > length of your series. If you set the block_size too high, it
may
> consume
> > > all the memory.
> > >
> > > Hope that helps clarify.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>
> > > >
> > > > Thanks John. I'm testing the change now.
> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just
7 days
> of
> > > > data. Its been running for over 2 hours now. Is this the
expected
> > > > behavior? I do have -v set to 3. Its writing steadily to the
log.
> Is
> > > > that slowing things down?
> > > >
> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Hello John,
> > > > >
> > > > > I see you're having difficulty configuring Series-Analysis
to apply
> > the
> > > > > continuous and categorical statistics you defined.
> > > > >
> > > > > It's funny, this exact same issue arose via met-help last
week, and
> > it
> > > > > makes me think we should modify the code in some way to make
it
> more
> > > > > intuitive. Please try editing your config file by moving
the
> > > cnt_thresh
> > > > > and cat_thresh settings up outside of the field array, as
shown
> > below:
> > > > >
> > > > > *fcst = {*
> > > > >
> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
>=2.54&&<6.35,
> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
>=25.4&&<76.2,
> >=76.2
> > > ];*
> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05,
> >=25.4,
> > > > > >=50.8, >=76.2 ];*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > * field = [ { name = "${FIELD}"; level
=
> > > > "${LEVEL}";
> > > > > } ];}obs = fcst;*
> > > > >
> > > > > When you rerun after this change, does it produce the
expected
> > output?
> > > > >
> > > > > I realize that this is confusing, especially if you're used
to
> > setting
> > > up
> > > > > config files for Point-Stat and Grid-Stat. Those tools are
set up
> to
> > > > > process multiple fields in a single run. Therefore you can
specify
> > > > > cat_thresh and cnt_thresh separately for each field array
entry.
> > Unlike
> > > > > Point-Stat and Grid-Stat, Series-Analysis can only process a
single
> > > > series
> > > > > of data in each run. There are multiple main ways of
defining a
> > series.
> > > > > (1) By far, the most common way is specifying a single field
that
> is
> > > > > extracted from a time-series of input data files... as
you're
> doing.
> > > > > (2) You can also define a series of field array entries that
can be
> > > > > extracted from the same input file. For example, compute
stats over
> > > > > multiple vertical levels from the same input file.
> > > > > (3) You define N input fields to be extracted from N input
files.
> > > > >
> > > > > Because of options (2) and (3), the field entry is defined
as an
> > array
> > > > > rather than a single dictionary. In practice though, the
field
> array
> > > > > typically has length 1.
> > > > >
> > > > > But the cat_thresh and cnt_thresh settings must remain
constant
> over
> > > the
> > > > > entire series, and that's why the code is setup to parse
them from
> > the
> > > > > "fcst" and "obs" dictionaries, and not separately from each
field
> > array
> > > > > entry.
> > > > >
> > > > > Seems like it'd be a good idea to check for this situation
and
> print
> > a
> > > > > warning message for the user.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
> met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > > > > >
> > > > > > Hello John,
> > > > > >
> > > > > > It looks like you need some assistance setting up your
series
> > > analysis
> > > > in
> > > > > > MET. I'm assigning this ticket to John Halley Gotway.
Please
> > allow
> > > a
> > > > > few
> > > > > > business days for a response.
> > > > > >
> > > > > > Regards,
> > > > > > Minna
> > > > > > ---------------
> > > > > > Minna Win
> > > > > > National Center for Atmospheric Research
> > > > > > Developmental Testbed Center
> > > > > > Phone: 303-497-8423
> > > > > > Fax: 303-497-8401
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA
Federal via
> > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > > > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > > > > Queue: met_help
> > > > > > > Subject: Series Analysis Config File Issues
> > > > > > > Owner: Nobody
> > > > > > > Requestors: john.l.wagner at noaa.gov
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Good morning MET help
> > > > > > > I'm having an issue with series_analysis that I could
use your
> > > > > assistance
> > > > > > > with. I am currently testing series_analysis for QPF06
(config
> > > file
> > > > > > > attached). I am unable to get valid output values for
anything
> > in
> > > > the
> > > > > > CTS
> > > > > > > and MCTS categories. The TOTAL grid for each score group
looks
> > good
> > > > > (I'm
> > > > > > > currently testing with 7 days of data). The CSI values
are all
> > > > > constant
> > > > > > at
> > > > > > > 1. I am unable to get any values for HSS, either from
CTS or
> > MCTS.
> > > > > All
> > > > > > > output (MAE, ME, RMSE) from the CNT score group seem to
work
> > fine.
> > > > > > > I'm assuming the issue is setting the thresholds in my
config
> > file.
> > > > > > > The scores that I see in my netCDF file for the CTS
scores all
> > end
> > > > with
> > > > > > > "_NA", where I'm assuming a category should be set. I
am using
> > the
> > > > > same
> > > > > > > thresholds that I use in grid_stat. Should I be setting
them
> up
> > > > > > > differently for series_analysis? Is there a sample
config file
> > > that
> > > > I
> > > > > > > could look at for reference?
> > > > > > > If it helps, my test output file can be found on WCOSS
> > > > > > > here:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > > > > > >
> > > > > > > Thanks
> > > > > > > John
> > > > > > > --
> > > > > > > John Wagner
> > > > > > > Verification Task Lead
> > > > > > > NOAA/National Weather Service
> > > > > > > Meteorological Development Laboratory
> > > > > > > Digital Forecast Services Division
> > > > > > > SSMC2 Room 10106
> > > > > > > Silver Spring, MD 20910
> > > > > > > (301) 427-9471 (office)
> > > > > > > (908) 902-4155 (cell/text)
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > John Wagner
> > > > Verification Task Lead
> > > > NOAA/National Weather Service
> > > > Meteorological Development Laboratory
> > > > Digital Forecast Services Division
> > > > SSMC2 Room 10106
> > > > Silver Spring, MD 20910
> > > > (301) 427-9471 (office)
> > > > (908) 902-4155 (cell/text)
> > > >
> > > >
> > >
> > >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John L Wagner - NOAA Federal
Time: Fri Jun 12 15:56:38 2020
John
I believe I've confirmed that the slow run times were from setting -v
3
when calling series_analysis. My last test ran in about 25 minutes.
I
have a few more tests to run, but I believe all of my issues are
solved.
Thanks
John
On Fri, Jun 12, 2020 at 7:39 AM John L Wagner - NOAA Federal <
john.l.wagner at noaa.gov> wrote:
> Here is the usage summary from the test I ran yesterday:
>
> [image: SerA Test1 Log.PNG]
> The job did not time out, and finished in 5 hr 43 min. It did not
use all
> of the memory, which surprised me. When I was testing incorrectly
with the
> thresholds defined within the field array, I had to keep increasing
the
> memory setting in my bsub command to prevent OOM kills.
>
> I'm running a test now with just the lowest CNT threshold. This may
be
> what we end up running anyway and I'm hoping the run time will come
down so
> I can run more tests. I'll try to get through tests today with
cutting the
> block_size in half, using a mask, and different memory settings with
bsub.
> I'll let you know what I find.
>
> As for MPI, I agree with that solution. I'm not going to pretend to
know
> what needs to be changed with C code and compilers. When we updated
the
> MOS-2000 Fortran code, we managed to get our ensemble code run time
down
> from 3 hours to 12 minutes. I realize that not all MET users are
running
> on supercomputers that can take advantage of MPI, but those of us
that are
> would certainly appreciate it.
>
> Thanks
> John
>
> On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Agreed, that's much too long! I suppose another possibility is that
it's
>> consuming too much memory and is using swap space, which is slow.
>> If possible, please check what percent of memory the process is
using. If
>> it's up around 100%, then let's try cutting the block size in half
and
>> rerunning.
>>
>> Honestly, the real fix for this is enhancing Series-Analysis with
MPI
>> calls
>> to subset the 4+ million grid point into tiles, and then do the
processing
>> for each tile on a separate node.
>> However, none of the MET tools are currently parallelized. So
that'd
>> obviously be a significant development effort.
>>
>> Another option, just to get something to actually complete in the
time
>> window, would be applying a masking region to subset the data being
>> processed.
>>
>> Thanks,
>> John
>>
>> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal via RT
<
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>> >
>> > I set the block size to my grid size (4194304) previously. Its
still
>> > running 5+ hours later. It will time out after 6 hours.
>> > I have been testing on WCOSS with Julie's MET V9.0. Tomorrow
I'll run a
>> > test with V8.1. Otherwise, we'll need to rethink what we want to
show
>> in
>> > the viewer.
>> > Thanks
>> > John
>> >
>> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > John,
>> > >
>> > > I'd recommend modifying the "block_size" config file option and
>> > rerunning.
>> > > The default value is set very low, ensuring that you don't run
out of
>> > > memory but requiring many passes through the data... which is
slow.
>> > >
>> > > Here's an excerpt from this README file which describes how
that
>> works.
>> > >
>> > >
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
>> > >
>> > >
>> > >
>> >
>>
////////////////////////////////////////////////////////////////////////////////
>> > > //
>> > > // SeriesAnalysisConfig_default
>> > > //
>> > >
>> > >
>> >
>>
////////////////////////////////////////////////////////////////////////////////
>> > >
>> > > //
>> > > // Computation may be memory intensive, especially for large
grids.
>> > > // The "block_size" entry sets the number of grid points to be
>> processed
>> > > // concurrently (i.e. in one pass through a time series).
Smaller
>> values
>> > > // require less memory but increase the number of passes
through the
>> > data.
>> > > //
>> > > block_size = 1024;
>> > >
>> > > Look near the top of your log file for a message that looks
something
>> > like
>> > > this:
>> > >
>> > >
>> > > *DEBUG 2: Computing statistics using a block size of 10000,
requiring
>> 3
>> > > pass(es) through the 169 x 154 grid.DEBUG 2: Processing data
pass
>> number
>> > 1
>> > > of 3 for grid points 1 to 10000.*
>> > >
>> > > This will tell you how many times it's reading through the
input data
>> > > files. Ideally you'd set the block size high enough so that it
only
>> needs
>> > > to read them once. But that all depends on the number of grid
points
>> and
>> > > length of your series. If you set the block_size too high, it
may
>> consume
>> > > all the memory.
>> > >
>> > > Hope that helps clarify.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal
via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>
>> > > >
>> > > > Thanks John. I'm testing the change now.
>> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just
7
>> days of
>> > > > data. Its been running for over 2 hours now. Is this the
expected
>> > > > behavior? I do have -v set to 3. Its writing steadily to
the
>> log. Is
>> > > > that slowing things down?
>> > > >
>> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > > Hello John,
>> > > > >
>> > > > > I see you're having difficulty configuring Series-Analysis
to
>> apply
>> > the
>> > > > > continuous and categorical statistics you defined.
>> > > > >
>> > > > > It's funny, this exact same issue arose via met-help last
week,
>> and
>> > it
>> > > > > makes me think we should modify the code in some way to
make it
>> more
>> > > > > intuitive. Please try editing your config file by moving
the
>> > > cnt_thresh
>> > > > > and cat_thresh settings up outside of the field array, as
shown
>> > below:
>> > > > >
>> > > > > *fcst = {*
>> > > > >
>> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
>=2.54&&<6.35,
>> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
>=25.4&&<76.2,
>> >=76.2
>> > > ];*
>> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7,
>=19.05,
>> >=25.4,
>> > > > > >=50.8, >=76.2 ];*
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > * field = [ { name = "${FIELD}";
level =
>> > > > "${LEVEL}";
>> > > > > } ];}obs = fcst;*
>> > > > >
>> > > > > When you rerun after this change, does it produce the
expected
>> > output?
>> > > > >
>> > > > > I realize that this is confusing, especially if you're used
to
>> > setting
>> > > up
>> > > > > config files for Point-Stat and Grid-Stat. Those tools are
set up
>> to
>> > > > > process multiple fields in a single run. Therefore you can
specify
>> > > > > cat_thresh and cnt_thresh separately for each field array
entry.
>> > Unlike
>> > > > > Point-Stat and Grid-Stat, Series-Analysis can only process
a
>> single
>> > > > series
>> > > > > of data in each run. There are multiple main ways of
defining a
>> > series.
>> > > > > (1) By far, the most common way is specifying a single
field that
>> is
>> > > > > extracted from a time-series of input data files... as
you're
>> doing.
>> > > > > (2) You can also define a series of field array entries
that can
>> be
>> > > > > extracted from the same input file. For example, compute
stats
>> over
>> > > > > multiple vertical levels from the same input file.
>> > > > > (3) You define N input fields to be extracted from N input
files.
>> > > > >
>> > > > > Because of options (2) and (3), the field entry is defined
as an
>> > array
>> > > > > rather than a single dictionary. In practice though, the
field
>> array
>> > > > > typically has length 1.
>> > > > >
>> > > > > But the cat_thresh and cnt_thresh settings must remain
constant
>> over
>> > > the
>> > > > > entire series, and that's why the code is setup to parse
them from
>> > the
>> > > > > "fcst" and "obs" dictionaries, and not separately from each
field
>> > array
>> > > > > entry.
>> > > > >
>> > > > > Seems like it'd be a good idea to check for this situation
and
>> print
>> > a
>> > > > > warning message for the user.
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
>> met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>> > > > > >
>> > > > > > Hello John,
>> > > > > >
>> > > > > > It looks like you need some assistance setting up your
series
>> > > analysis
>> > > > in
>> > > > > > MET. I'm assigning this ticket to John Halley Gotway.
Please
>> > allow
>> > > a
>> > > > > few
>> > > > > > business days for a response.
>> > > > > >
>> > > > > > Regards,
>> > > > > > Minna
>> > > > > > ---------------
>> > > > > > Minna Win
>> > > > > > National Center for Atmospheric Research
>> > > > > > Developmental Testbed Center
>> > > > > > Phone: 303-497-8423
>> > > > > > Fax: 303-497-8401
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA
Federal via
>> > RT <
>> > > > > > met_help at ucar.edu> wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
>> > > > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
>> > > > > > > Queue: met_help
>> > > > > > > Subject: Series Analysis Config File Issues
>> > > > > > > Owner: Nobody
>> > > > > > > Requestors: john.l.wagner at noaa.gov
>> > > > > > > Status: new
>> > > > > > > Ticket <URL:
>> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>> > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > Good morning MET help
>> > > > > > > I'm having an issue with series_analysis that I could
use your
>> > > > > assistance
>> > > > > > > with. I am currently testing series_analysis for QPF06
>> (config
>> > > file
>> > > > > > > attached). I am unable to get valid output values for
>> anything
>> > in
>> > > > the
>> > > > > > CTS
>> > > > > > > and MCTS categories. The TOTAL grid for each score
group looks
>> > good
>> > > > > (I'm
>> > > > > > > currently testing with 7 days of data). The CSI values
are
>> all
>> > > > > constant
>> > > > > > at
>> > > > > > > 1. I am unable to get any values for HSS, either from
CTS or
>> > MCTS.
>> > > > > All
>> > > > > > > output (MAE, ME, RMSE) from the CNT score group seem to
work
>> > fine.
>> > > > > > > I'm assuming the issue is setting the thresholds in my
config
>> > file.
>> > > > > > > The scores that I see in my netCDF file for the CTS
scores all
>> > end
>> > > > with
>> > > > > > > "_NA", where I'm assuming a category should be set. I
am
>> using
>> > the
>> > > > > same
>> > > > > > > thresholds that I use in grid_stat. Should I be
setting them
>> up
>> > > > > > > differently for series_analysis? Is there a sample
config
>> file
>> > > that
>> > > > I
>> > > > > > > could look at for reference?
>> > > > > > > If it helps, my test output file can be found on WCOSS
>> > > > > > > here:
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
>> > > > > > >
>> > > > > > > Thanks
>> > > > > > > John
>> > > > > > > --
>> > > > > > > John Wagner
>> > > > > > > Verification Task Lead
>> > > > > > > NOAA/National Weather Service
>> > > > > > > Meteorological Development Laboratory
>> > > > > > > Digital Forecast Services Division
>> > > > > > > SSMC2 Room 10106
>> > > > > > > Silver Spring, MD 20910
>> > > > > > > (301) 427-9471 (office)
>> > > > > > > (908) 902-4155 (cell/text)
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > John Wagner
>> > > > Verification Task Lead
>> > > > NOAA/National Weather Service
>> > > > Meteorological Development Laboratory
>> > > > Digital Forecast Services Division
>> > > > SSMC2 Room 10106
>> > > > Silver Spring, MD 20910
>> > > > (301) 427-9471 (office)
>> > > > (908) 902-4155 (cell/text)
>> > > >
>> > > >
>> > >
>> > >
>> >
>> > --
>> > John Wagner
>> > Verification Task Lead
>> > NOAA/National Weather Service
>> > Meteorological Development Laboratory
>> > Digital Forecast Services Division
>> > SSMC2 Room 10106
>> > Silver Spring, MD 20910
>> > (301) 427-9471 (office)
>> > (908) 902-4155 (cell/text)
>> >
>> >
>>
>>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Fri Jun 12 17:29:59 2020
John,
Oh wow, OK. Would you recommend that I change some of those log
messages to
a higher verbosity level... like 4 or 5... to avoid slow runtimes?
John
On Fri, Jun 12, 2020 at 3:57 PM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
> John
> I believe I've confirmed that the slow run times were from setting
-v 3
> when calling series_analysis. My last test ran in about 25 minutes.
I
> have a few more tests to run, but I believe all of my issues are
solved.
> Thanks
> John
>
> On Fri, Jun 12, 2020 at 7:39 AM John L Wagner - NOAA Federal <
> john.l.wagner at noaa.gov> wrote:
>
> > Here is the usage summary from the test I ran yesterday:
> >
> > [image: SerA Test1 Log.PNG]
> > The job did not time out, and finished in 5 hr 43 min. It did not
use
> all
> > of the memory, which surprised me. When I was testing incorrectly
with
> the
> > thresholds defined within the field array, I had to keep
increasing the
> > memory setting in my bsub command to prevent OOM kills.
> >
> > I'm running a test now with just the lowest CNT threshold. This
may be
> > what we end up running anyway and I'm hoping the run time will
come down
> so
> > I can run more tests. I'll try to get through tests today with
cutting
> the
> > block_size in half, using a mask, and different memory settings
with
> bsub.
> > I'll let you know what I find.
> >
> > As for MPI, I agree with that solution. I'm not going to pretend
to know
> > what needs to be changed with C code and compilers. When we
updated the
> > MOS-2000 Fortran code, we managed to get our ensemble code run
time down
> > from 3 hours to 12 minutes. I realize that not all MET users are
running
> > on supercomputers that can take advantage of MPI, but those of us
that
> are
> > would certainly appreciate it.
> >
> > Thanks
> > John
> >
> > On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Agreed, that's much too long! I suppose another possibility is
that it's
> >> consuming too much memory and is using swap space, which is slow.
> >> If possible, please check what percent of memory the process is
using.
> If
> >> it's up around 100%, then let's try cutting the block size in
half and
> >> rerunning.
> >>
> >> Honestly, the real fix for this is enhancing Series-Analysis with
MPI
> >> calls
> >> to subset the 4+ million grid point into tiles, and then do the
> processing
> >> for each tile on a separate node.
> >> However, none of the MET tools are currently parallelized. So
that'd
> >> obviously be a significant development effort.
> >>
> >> Another option, just to get something to actually complete in the
time
> >> window, would be applying a masking region to subset the data
being
> >> processed.
> >>
> >> Thanks,
> >> John
> >>
> >> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal via
RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >> >
> >> > I set the block size to my grid size (4194304) previously. Its
still
> >> > running 5+ hours later. It will time out after 6 hours.
> >> > I have been testing on WCOSS with Julie's MET V9.0. Tomorrow
I'll
> run a
> >> > test with V8.1. Otherwise, we'll need to rethink what we want
to show
> >> in
> >> > the viewer.
> >> > Thanks
> >> > John
> >> >
> >> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
> >> > met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > John,
> >> > >
> >> > > I'd recommend modifying the "block_size" config file option
and
> >> > rerunning.
> >> > > The default value is set very low, ensuring that you don't
run out
> of
> >> > > memory but requiring many passes through the data... which is
slow.
> >> > >
> >> > > Here's an excerpt from this README file which describes how
that
> >> works.
> >> > >
> >> > >
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
> >> > >
> >> > >
> >> > >
> >> >
> >>
>
////////////////////////////////////////////////////////////////////////////////
> >> > > //
> >> > > // SeriesAnalysisConfig_default
> >> > > //
> >> > >
> >> > >
> >> >
> >>
>
////////////////////////////////////////////////////////////////////////////////
> >> > >
> >> > > //
> >> > > // Computation may be memory intensive, especially for large
grids.
> >> > > // The "block_size" entry sets the number of grid points to
be
> >> processed
> >> > > // concurrently (i.e. in one pass through a time series).
Smaller
> >> values
> >> > > // require less memory but increase the number of passes
through the
> >> > data.
> >> > > //
> >> > > block_size = 1024;
> >> > >
> >> > > Look near the top of your log file for a message that looks
> something
> >> > like
> >> > > this:
> >> > >
> >> > >
> >> > > *DEBUG 2: Computing statistics using a block size of 10000,
> requiring
> >> 3
> >> > > pass(es) through the 169 x 154 grid.DEBUG 2: Processing data
pass
> >> number
> >> > 1
> >> > > of 3 for grid points 1 to 10000.*
> >> > >
> >> > > This will tell you how many times it's reading through the
input
> data
> >> > > files. Ideally you'd set the block size high enough so that
it only
> >> needs
> >> > > to read them once. But that all depends on the number of grid
points
> >> and
> >> > > length of your series. If you set the block_size too high, it
may
> >> consume
> >> > > all the memory.
> >> > >
> >> > > Hope that helps clarify.
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal
via RT
> <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >> > > >
> >> > > > Thanks John. I'm testing the change now.
> >> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for
just 7
> >> days of
> >> > > > data. Its been running for over 2 hours now. Is this the
> expected
> >> > > > behavior? I do have -v set to 3. Its writing steadily to
the
> >> log. Is
> >> > > > that slowing things down?
> >> > > >
> >> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT
<
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > > Hello John,
> >> > > > >
> >> > > > > I see you're having difficulty configuring Series-
Analysis to
> >> apply
> >> > the
> >> > > > > continuous and categorical statistics you defined.
> >> > > > >
> >> > > > > It's funny, this exact same issue arose via met-help last
week,
> >> and
> >> > it
> >> > > > > makes me think we should modify the code in some way to
make it
> >> more
> >> > > > > intuitive. Please try editing your config file by moving
the
> >> > > cnt_thresh
> >> > > > > and cat_thresh settings up outside of the field array, as
shown
> >> > below:
> >> > > > >
> >> > > > > *fcst = {*
> >> > > > >
> >> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
>=2.54&&<6.35,
> >> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
>=25.4&&<76.2,
> >> >=76.2
> >> > > ];*
> >> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7,
>=19.05,
> >> >=25.4,
> >> > > > > >=50.8, >=76.2 ];*
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > * field = [ { name = "${FIELD}";
level =
> >> > > > "${LEVEL}";
> >> > > > > } ];}obs = fcst;*
> >> > > > >
> >> > > > > When you rerun after this change, does it produce the
expected
> >> > output?
> >> > > > >
> >> > > > > I realize that this is confusing, especially if you're
used to
> >> > setting
> >> > > up
> >> > > > > config files for Point-Stat and Grid-Stat. Those tools
are set
> up
> >> to
> >> > > > > process multiple fields in a single run. Therefore you
can
> specify
> >> > > > > cat_thresh and cnt_thresh separately for each field array
entry.
> >> > Unlike
> >> > > > > Point-Stat and Grid-Stat, Series-Analysis can only
process a
> >> single
> >> > > > series
> >> > > > > of data in each run. There are multiple main ways of
defining a
> >> > series.
> >> > > > > (1) By far, the most common way is specifying a single
field
> that
> >> is
> >> > > > > extracted from a time-series of input data files... as
you're
> >> doing.
> >> > > > > (2) You can also define a series of field array entries
that can
> >> be
> >> > > > > extracted from the same input file. For example, compute
stats
> >> over
> >> > > > > multiple vertical levels from the same input file.
> >> > > > > (3) You define N input fields to be extracted from N
input
> files.
> >> > > > >
> >> > > > > Because of options (2) and (3), the field entry is
defined as an
> >> > array
> >> > > > > rather than a single dictionary. In practice though, the
field
> >> array
> >> > > > > typically has length 1.
> >> > > > >
> >> > > > > But the cat_thresh and cnt_thresh settings must remain
constant
> >> over
> >> > > the
> >> > > > > entire series, and that's why the code is setup to parse
them
> from
> >> > the
> >> > > > > "fcst" and "obs" dictionaries, and not separately from
each
> field
> >> > array
> >> > > > > entry.
> >> > > > >
> >> > > > > Seems like it'd be a good idea to check for this
situation and
> >> print
> >> > a
> >> > > > > warning message for the user.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > John
> >> > > > >
> >> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
> >> met_help at ucar.edu>
> >> > > > > wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> >
> >> > > > > >
> >> > > > > > Hello John,
> >> > > > > >
> >> > > > > > It looks like you need some assistance setting up your
series
> >> > > analysis
> >> > > > in
> >> > > > > > MET. I'm assigning this ticket to John Halley Gotway.
Please
> >> > allow
> >> > > a
> >> > > > > few
> >> > > > > > business days for a response.
> >> > > > > >
> >> > > > > > Regards,
> >> > > > > > Minna
> >> > > > > > ---------------
> >> > > > > > Minna Win
> >> > > > > > National Center for Atmospheric Research
> >> > > > > > Developmental Testbed Center
> >> > > > > > Phone: 303-497-8423
> >> > > > > > Fax: 303-497-8401
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA
Federal
> via
> >> > RT <
> >> > > > > > met_help at ucar.edu> wrote:
> >> > > > > >
> >> > > > > > >
> >> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted
upon.
> >> > > > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> >> > > > > > > Queue: met_help
> >> > > > > > > Subject: Series Analysis Config File Issues
> >> > > > > > > Owner: Nobody
> >> > > > > > > Requestors: john.l.wagner at noaa.gov
> >> > > > > > > Status: new
> >> > > > > > > Ticket <URL:
> >> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> >> > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Good morning MET help
> >> > > > > > > I'm having an issue with series_analysis that I could
use
> your
> >> > > > > assistance
> >> > > > > > > with. I am currently testing series_analysis for
QPF06
> >> (config
> >> > > file
> >> > > > > > > attached). I am unable to get valid output values
for
> >> anything
> >> > in
> >> > > > the
> >> > > > > > CTS
> >> > > > > > > and MCTS categories. The TOTAL grid for each score
group
> looks
> >> > good
> >> > > > > (I'm
> >> > > > > > > currently testing with 7 days of data). The CSI
values are
> >> all
> >> > > > > constant
> >> > > > > > at
> >> > > > > > > 1. I am unable to get any values for HSS, either
from CTS
> or
> >> > MCTS.
> >> > > > > All
> >> > > > > > > output (MAE, ME, RMSE) from the CNT score group seem
to work
> >> > fine.
> >> > > > > > > I'm assuming the issue is setting the thresholds in
my
> config
> >> > file.
> >> > > > > > > The scores that I see in my netCDF file for the CTS
scores
> all
> >> > end
> >> > > > with
> >> > > > > > > "_NA", where I'm assuming a category should be set.
I am
> >> using
> >> > the
> >> > > > > same
> >> > > > > > > thresholds that I use in grid_stat. Should I be
setting
> them
> >> up
> >> > > > > > > differently for series_analysis? Is there a sample
config
> >> file
> >> > > that
> >> > > > I
> >> > > > > > > could look at for reference?
> >> > > > > > > If it helps, my test output file can be found on
WCOSS
> >> > > > > > > here:
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> >> > > > > > >
> >> > > > > > > Thanks
> >> > > > > > > John
> >> > > > > > > --
> >> > > > > > > John Wagner
> >> > > > > > > Verification Task Lead
> >> > > > > > > NOAA/National Weather Service
> >> > > > > > > Meteorological Development Laboratory
> >> > > > > > > Digital Forecast Services Division
> >> > > > > > > SSMC2 Room 10106
> >> > > > > > > Silver Spring, MD 20910
> >> > > > > > > (301) 427-9471 (office)
> >> > > > > > > (908) 902-4155 (cell/text)
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > > --
> >> > > > John Wagner
> >> > > > Verification Task Lead
> >> > > > NOAA/National Weather Service
> >> > > > Meteorological Development Laboratory
> >> > > > Digital Forecast Services Division
> >> > > > SSMC2 Room 10106
> >> > > > Silver Spring, MD 20910
> >> > > > (301) 427-9471 (office)
> >> > > > (908) 902-4155 (cell/text)
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > John Wagner
> >> > Verification Task Lead
> >> > NOAA/National Weather Service
> >> > Meteorological Development Laboratory
> >> > Digital Forecast Services Division
> >> > SSMC2 Room 10106
> >> > Silver Spring, MD 20910
> >> > (301) 427-9471 (office)
> >> > (908) 902-4155 (cell/text)
> >> >
> >> >
> >>
> >>
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Fri Jun 12 17:42:16 2020
John,
You know, I think I should leave well-enough alone. But I'm very
surprised
about this change in runtime. I ran a short test case at verbosity
level 0
and it took 20 seconds. At verbosity level 4, it took 22 seconds.
More, but
not a crazy amount more.
But I'm glad it's running in a more reasonable amount of time, and
hope it
continues to do so. I'll go ahead and resolve this ticket.
Thanks,
John
On Fri, Jun 12, 2020 at 5:29 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Oh wow, OK. Would you recommend that I change some of those log
messages
> to a higher verbosity level... like 4 or 5... to avoid slow
runtimes?
>
> John
>
> On Fri, Jun 12, 2020 at 3:57 PM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>>
>> John
>> I believe I've confirmed that the slow run times were from setting
-v 3
>> when calling series_analysis. My last test ran in about 25
minutes. I
>> have a few more tests to run, but I believe all of my issues are
solved.
>> Thanks
>> John
>>
>> On Fri, Jun 12, 2020 at 7:39 AM John L Wagner - NOAA Federal <
>> john.l.wagner at noaa.gov> wrote:
>>
>> > Here is the usage summary from the test I ran yesterday:
>> >
>> > [image: SerA Test1 Log.PNG]
>> > The job did not time out, and finished in 5 hr 43 min. It did
not use
>> all
>> > of the memory, which surprised me. When I was testing
incorrectly with
>> the
>> > thresholds defined within the field array, I had to keep
increasing the
>> > memory setting in my bsub command to prevent OOM kills.
>> >
>> > I'm running a test now with just the lowest CNT threshold. This
may be
>> > what we end up running anyway and I'm hoping the run time will
come
>> down so
>> > I can run more tests. I'll try to get through tests today with
cutting
>> the
>> > block_size in half, using a mask, and different memory settings
with
>> bsub.
>> > I'll let you know what I find.
>> >
>> > As for MPI, I agree with that solution. I'm not going to pretend
to
>> know
>> > what needs to be changed with C code and compilers. When we
updated the
>> > MOS-2000 Fortran code, we managed to get our ensemble code run
time down
>> > from 3 hours to 12 minutes. I realize that not all MET users are
>> running
>> > on supercomputers that can take advantage of MPI, but those of us
that
>> are
>> > would certainly appreciate it.
>> >
>> > Thanks
>> > John
>> >
>> > On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >> Agreed, that's much too long! I suppose another possibility is
that
>> it's
>> >> consuming too much memory and is using swap space, which is
slow.
>> >> If possible, please check what percent of memory the process is
using.
>> If
>> >> it's up around 100%, then let's try cutting the block size in
half and
>> >> rerunning.
>> >>
>> >> Honestly, the real fix for this is enhancing Series-Analysis
with MPI
>> >> calls
>> >> to subset the 4+ million grid point into tiles, and then do the
>> processing
>> >> for each tile on a separate node.
>> >> However, none of the MET tools are currently parallelized. So
that'd
>> >> obviously be a significant development effort.
>> >>
>> >> Another option, just to get something to actually complete in
the time
>> >> window, would be applying a masking region to subset the data
being
>> >> processed.
>> >>
>> >> Thanks,
>> >> John
>> >>
>> >> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal via
RT <
>> >> met_help at ucar.edu> wrote:
>> >>
>> >> >
>> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>
>> >> >
>> >> > I set the block size to my grid size (4194304) previously.
Its still
>> >> > running 5+ hours later. It will time out after 6 hours.
>> >> > I have been testing on WCOSS with Julie's MET V9.0. Tomorrow
I'll
>> run a
>> >> > test with V8.1. Otherwise, we'll need to rethink what we want
to
>> show
>> >> in
>> >> > the viewer.
>> >> > Thanks
>> >> > John
>> >> >
>> >> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
>> >> > met_help at ucar.edu>
>> >> > wrote:
>> >> >
>> >> > > John,
>> >> > >
>> >> > > I'd recommend modifying the "block_size" config file option
and
>> >> > rerunning.
>> >> > > The default value is set very low, ensuring that you don't
run out
>> of
>> >> > > memory but requiring many passes through the data... which
is slow.
>> >> > >
>> >> > > Here's an excerpt from this README file which describes how
that
>> >> works.
>> >> > >
>> >> > >
>> https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
>> >> > >
>> >> > >
>> >> > >
>> >> >
>> >>
>>
////////////////////////////////////////////////////////////////////////////////
>> >> > > //
>> >> > > // SeriesAnalysisConfig_default
>> >> > > //
>> >> > >
>> >> > >
>> >> >
>> >>
>>
////////////////////////////////////////////////////////////////////////////////
>> >> > >
>> >> > > //
>> >> > > // Computation may be memory intensive, especially for large
grids.
>> >> > > // The "block_size" entry sets the number of grid points to
be
>> >> processed
>> >> > > // concurrently (i.e. in one pass through a time series).
Smaller
>> >> values
>> >> > > // require less memory but increase the number of passes
through
>> the
>> >> > data.
>> >> > > //
>> >> > > block_size = 1024;
>> >> > >
>> >> > > Look near the top of your log file for a message that looks
>> something
>> >> > like
>> >> > > this:
>> >> > >
>> >> > >
>> >> > > *DEBUG 2: Computing statistics using a block size of 10000,
>> requiring
>> >> 3
>> >> > > pass(es) through the 169 x 154 grid.DEBUG 2: Processing data
pass
>> >> number
>> >> > 1
>> >> > > of 3 for grid points 1 to 10000.*
>> >> > >
>> >> > > This will tell you how many times it's reading through the
input
>> data
>> >> > > files. Ideally you'd set the block size high enough so that
it only
>> >> needs
>> >> > > to read them once. But that all depends on the number of
grid
>> points
>> >> and
>> >> > > length of your series. If you set the block_size too high,
it may
>> >> consume
>> >> > > all the memory.
>> >> > >
>> >> > > Hope that helps clarify.
>> >> > >
>> >> > > Thanks,
>> >> > > John
>> >> > >
>> >> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal
via
>> RT <
>> >> > > met_help at ucar.edu> wrote:
>> >> > >
>> >> > > >
>> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>> >> > > >
>> >> > > > Thanks John. I'm testing the change now.
>> >> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for
just 7
>> >> days of
>> >> > > > data. Its been running for over 2 hours now. Is this the
>> expected
>> >> > > > behavior? I do have -v set to 3. Its writing steadily to
the
>> >> log. Is
>> >> > > > that slowing things down?
>> >> > > >
>> >> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT
<
>> >> > > > met_help at ucar.edu> wrote:
>> >> > > >
>> >> > > > > Hello John,
>> >> > > > >
>> >> > > > > I see you're having difficulty configuring Series-
Analysis to
>> >> apply
>> >> > the
>> >> > > > > continuous and categorical statistics you defined.
>> >> > > > >
>> >> > > > > It's funny, this exact same issue arose via met-help
last week,
>> >> and
>> >> > it
>> >> > > > > makes me think we should modify the code in some way to
make it
>> >> more
>> >> > > > > intuitive. Please try editing your config file by
moving the
>> >> > > cnt_thresh
>> >> > > > > and cat_thresh settings up outside of the field array,
as shown
>> >> > below:
>> >> > > > >
>> >> > > > > *fcst = {*
>> >> > > > >
>> >> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
>=2.54&&<6.35,
>> >> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
>=25.4&&<76.2,
>> >> >=76.2
>> >> > > ];*
>> >> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7,
>=19.05,
>> >> >=25.4,
>> >> > > > > >=50.8, >=76.2 ];*
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > * field = [ { name = "${FIELD}";
level =
>> >> > > > "${LEVEL}";
>> >> > > > > } ];}obs = fcst;*
>> >> > > > >
>> >> > > > > When you rerun after this change, does it produce the
expected
>> >> > output?
>> >> > > > >
>> >> > > > > I realize that this is confusing, especially if you're
used to
>> >> > setting
>> >> > > up
>> >> > > > > config files for Point-Stat and Grid-Stat. Those tools
are set
>> up
>> >> to
>> >> > > > > process multiple fields in a single run. Therefore you
can
>> specify
>> >> > > > > cat_thresh and cnt_thresh separately for each field
array
>> entry.
>> >> > Unlike
>> >> > > > > Point-Stat and Grid-Stat, Series-Analysis can only
process a
>> >> single
>> >> > > > series
>> >> > > > > of data in each run. There are multiple main ways of
defining a
>> >> > series.
>> >> > > > > (1) By far, the most common way is specifying a single
field
>> that
>> >> is
>> >> > > > > extracted from a time-series of input data files... as
you're
>> >> doing.
>> >> > > > > (2) You can also define a series of field array entries
that
>> can
>> >> be
>> >> > > > > extracted from the same input file. For example, compute
stats
>> >> over
>> >> > > > > multiple vertical levels from the same input file.
>> >> > > > > (3) You define N input fields to be extracted from N
input
>> files.
>> >> > > > >
>> >> > > > > Because of options (2) and (3), the field entry is
defined as
>> an
>> >> > array
>> >> > > > > rather than a single dictionary. In practice though, the
field
>> >> array
>> >> > > > > typically has length 1.
>> >> > > > >
>> >> > > > > But the cat_thresh and cnt_thresh settings must remain
constant
>> >> over
>> >> > > the
>> >> > > > > entire series, and that's why the code is setup to parse
them
>> from
>> >> > the
>> >> > > > > "fcst" and "obs" dictionaries, and not separately from
each
>> field
>> >> > array
>> >> > > > > entry.
>> >> > > > >
>> >> > > > > Seems like it'd be a good idea to check for this
situation and
>> >> print
>> >> > a
>> >> > > > > warning message for the user.
>> >> > > > >
>> >> > > > > Thanks,
>> >> > > > > John
>> >> > > > >
>> >> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
>> >> met_help at ucar.edu>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > >
>> >> > > > > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>> >> > > > > >
>> >> > > > > > Hello John,
>> >> > > > > >
>> >> > > > > > It looks like you need some assistance setting up your
series
>> >> > > analysis
>> >> > > > in
>> >> > > > > > MET. I'm assigning this ticket to John Halley Gotway.
>> Please
>> >> > allow
>> >> > > a
>> >> > > > > few
>> >> > > > > > business days for a response.
>> >> > > > > >
>> >> > > > > > Regards,
>> >> > > > > > Minna
>> >> > > > > > ---------------
>> >> > > > > > Minna Win
>> >> > > > > > National Center for Atmospheric Research
>> >> > > > > > Developmental Testbed Center
>> >> > > > > > Phone: 303-497-8423
>> >> > > > > > Fax: 303-497-8401
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA
Federal
>> via
>> >> > RT <
>> >> > > > > > met_help at ucar.edu> wrote:
>> >> > > > > >
>> >> > > > > > >
>> >> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted
upon.
>> >> > > > > > > Transaction: Ticket created by
john.l.wagner at noaa.gov
>> >> > > > > > > Queue: met_help
>> >> > > > > > > Subject: Series Analysis Config File Issues
>> >> > > > > > > Owner: Nobody
>> >> > > > > > > Requestors: john.l.wagner at noaa.gov
>> >> > > > > > > Status: new
>> >> > > > > > > Ticket <URL:
>> >> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>> >> > > > > >
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > Good morning MET help
>> >> > > > > > > I'm having an issue with series_analysis that I
could use
>> your
>> >> > > > > assistance
>> >> > > > > > > with. I am currently testing series_analysis for
QPF06
>> >> (config
>> >> > > file
>> >> > > > > > > attached). I am unable to get valid output values
for
>> >> anything
>> >> > in
>> >> > > > the
>> >> > > > > > CTS
>> >> > > > > > > and MCTS categories. The TOTAL grid for each score
group
>> looks
>> >> > good
>> >> > > > > (I'm
>> >> > > > > > > currently testing with 7 days of data). The CSI
values are
>> >> all
>> >> > > > > constant
>> >> > > > > > at
>> >> > > > > > > 1. I am unable to get any values for HSS, either
from CTS
>> or
>> >> > MCTS.
>> >> > > > > All
>> >> > > > > > > output (MAE, ME, RMSE) from the CNT score group seem
to
>> work
>> >> > fine.
>> >> > > > > > > I'm assuming the issue is setting the thresholds in
my
>> config
>> >> > file.
>> >> > > > > > > The scores that I see in my netCDF file for the CTS
scores
>> all
>> >> > end
>> >> > > > with
>> >> > > > > > > "_NA", where I'm assuming a category should be set.
I am
>> >> using
>> >> > the
>> >> > > > > same
>> >> > > > > > > thresholds that I use in grid_stat. Should I be
setting
>> them
>> >> up
>> >> > > > > > > differently for series_analysis? Is there a sample
config
>> >> file
>> >> > > that
>> >> > > > I
>> >> > > > > > > could look at for reference?
>> >> > > > > > > If it helps, my test output file can be found on
WCOSS
>> >> > > > > > > here:
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
>> >> > > > > > >
>> >> > > > > > > Thanks
>> >> > > > > > > John
>> >> > > > > > > --
>> >> > > > > > > John Wagner
>> >> > > > > > > Verification Task Lead
>> >> > > > > > > NOAA/National Weather Service
>> >> > > > > > > Meteorological Development Laboratory
>> >> > > > > > > Digital Forecast Services Division
>> >> > > > > > > SSMC2 Room 10106
>> >> > > > > > > Silver Spring, MD 20910
>> >> > > > > > > (301) 427-9471 (office)
>> >> > > > > > > (908) 902-4155 (cell/text)
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > > > --
>> >> > > > John Wagner
>> >> > > > Verification Task Lead
>> >> > > > NOAA/National Weather Service
>> >> > > > Meteorological Development Laboratory
>> >> > > > Digital Forecast Services Division
>> >> > > > SSMC2 Room 10106
>> >> > > > Silver Spring, MD 20910
>> >> > > > (301) 427-9471 (office)
>> >> > > > (908) 902-4155 (cell/text)
>> >> > > >
>> >> > > >
>> >> > >
>> >> > >
>> >> >
>> >> > --
>> >> > John Wagner
>> >> > Verification Task Lead
>> >> > NOAA/National Weather Service
>> >> > Meteorological Development Laboratory
>> >> > Digital Forecast Services Division
>> >> > SSMC2 Room 10106
>> >> > Silver Spring, MD 20910
>> >> > (301) 427-9471 (office)
>> >> > (908) 902-4155 (cell/text)
>> >> >
>> >> >
>> >>
>> >>
>> >
>> > --
>> > John Wagner
>> > Verification Task Lead
>> > NOAA/National Weather Service
>> > Meteorological Development Laboratory
>> > Digital Forecast Services Division
>> > SSMC2 Room 10106
>> > Silver Spring, MD 20910
>> > (301) 427-9471 (office)
>> > (908) 902-4155 (cell/text)
>> >
>>
>>
>> --
>> John Wagner
>> Verification Task Lead
>> NOAA/National Weather Service
>> Meteorological Development Laboratory
>> Digital Forecast Services Division
>> SSMC2 Room 10106
>> Silver Spring, MD 20910
>> (301) 427-9471 (office)
>> (908) 902-4155 (cell/text)
>>
>>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John L Wagner - NOAA Federal
Time: Sat Jun 13 08:46:20 2020
That's fine. With -v 0, my log file is 512K. With -v 3, my log file
is
170M. Messages similar to "DEBUG 3: Using 7 of 7 pairs for forecast
filtering threshold >=0.254, observation filtering threshold >=0.254,
and
field logic Union." are printed 1,326,213 times. I'm not sure where
you're
testing, but I wouldn't be surprised if this was a WCOSS issue. I
haven't
exactly been blown away by the write speeds on phase 3.
I'm going to run today for more projections and elements. I'll let
you
know if there are any more issues.
Thanks
John
On Fri, Jun 12, 2020 at 7:42 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> John,
>
> You know, I think I should leave well-enough alone. But I'm very
surprised
> about this change in runtime. I ran a short test case at verbosity
level 0
> and it took 20 seconds. At verbosity level 4, it took 22 seconds.
More, but
> not a crazy amount more.
>
> But I'm glad it's running in a more reasonable amount of time, and
hope it
> continues to do so. I'll go ahead and resolve this ticket.
>
> Thanks,
> John
>
> On Fri, Jun 12, 2020 at 5:29 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > John,
> >
> > Oh wow, OK. Would you recommend that I change some of those log
messages
> > to a higher verbosity level... like 4 or 5... to avoid slow
runtimes?
> >
> > John
> >
> > On Fri, Jun 12, 2020 at 3:57 PM John L Wagner - NOAA Federal via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >>
> >> John
> >> I believe I've confirmed that the slow run times were from
setting -v 3
> >> when calling series_analysis. My last test ran in about 25
minutes. I
> >> have a few more tests to run, but I believe all of my issues are
solved.
> >> Thanks
> >> John
> >>
> >> On Fri, Jun 12, 2020 at 7:39 AM John L Wagner - NOAA Federal <
> >> john.l.wagner at noaa.gov> wrote:
> >>
> >> > Here is the usage summary from the test I ran yesterday:
> >> >
> >> > [image: SerA Test1 Log.PNG]
> >> > The job did not time out, and finished in 5 hr 43 min. It did
not use
> >> all
> >> > of the memory, which surprised me. When I was testing
incorrectly
> with
> >> the
> >> > thresholds defined within the field array, I had to keep
increasing
> the
> >> > memory setting in my bsub command to prevent OOM kills.
> >> >
> >> > I'm running a test now with just the lowest CNT threshold.
This may
> be
> >> > what we end up running anyway and I'm hoping the run time will
come
> >> down so
> >> > I can run more tests. I'll try to get through tests today with
> cutting
> >> the
> >> > block_size in half, using a mask, and different memory settings
with
> >> bsub.
> >> > I'll let you know what I find.
> >> >
> >> > As for MPI, I agree with that solution. I'm not going to
pretend to
> >> know
> >> > what needs to be changed with C code and compilers. When we
updated
> the
> >> > MOS-2000 Fortran code, we managed to get our ensemble code run
time
> down
> >> > from 3 hours to 12 minutes. I realize that not all MET users
are
> >> running
> >> > on supercomputers that can take advantage of MPI, but those of
us that
> >> are
> >> > would certainly appreciate it.
> >> >
> >> > Thanks
> >> > John
> >> >
> >> > On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> >> Agreed, that's much too long! I suppose another possibility is
that
> >> it's
> >> >> consuming too much memory and is using swap space, which is
slow.
> >> >> If possible, please check what percent of memory the process
is
> using.
> >> If
> >> >> it's up around 100%, then let's try cutting the block size in
half
> and
> >> >> rerunning.
> >> >>
> >> >> Honestly, the real fix for this is enhancing Series-Analysis
with MPI
> >> >> calls
> >> >> to subset the 4+ million grid point into tiles, and then do
the
> >> processing
> >> >> for each tile on a separate node.
> >> >> However, none of the MET tools are currently parallelized. So
that'd
> >> >> obviously be a significant development effort.
> >> >>
> >> >> Another option, just to get something to actually complete in
the
> time
> >> >> window, would be applying a masking region to subset the data
being
> >> >> processed.
> >> >>
> >> >> Thanks,
> >> >> John
> >> >>
> >> >> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal
via RT <
> >> >> met_help at ucar.edu> wrote:
> >> >>
> >> >> >
> >> >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >> >> >
> >> >> > I set the block size to my grid size (4194304) previously.
Its
> still
> >> >> > running 5+ hours later. It will time out after 6 hours.
> >> >> > I have been testing on WCOSS with Julie's MET V9.0.
Tomorrow I'll
> >> run a
> >> >> > test with V8.1. Otherwise, we'll need to rethink what we
want to
> >> show
> >> >> in
> >> >> > the viewer.
> >> >> > Thanks
> >> >> > John
> >> >> >
> >> >> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
> >> >> > met_help at ucar.edu>
> >> >> > wrote:
> >> >> >
> >> >> > > John,
> >> >> > >
> >> >> > > I'd recommend modifying the "block_size" config file
option and
> >> >> > rerunning.
> >> >> > > The default value is set very low, ensuring that you don't
run
> out
> >> of
> >> >> > > memory but requiring many passes through the data... which
is
> slow.
> >> >> > >
> >> >> > > Here's an excerpt from this README file which describes
how that
> >> >> works.
> >> >> > >
> >> >> > >
> >>
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >>
>
////////////////////////////////////////////////////////////////////////////////
> >> >> > > //
> >> >> > > // SeriesAnalysisConfig_default
> >> >> > > //
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >>
>
////////////////////////////////////////////////////////////////////////////////
> >> >> > >
> >> >> > > //
> >> >> > > // Computation may be memory intensive, especially for
large
> grids.
> >> >> > > // The "block_size" entry sets the number of grid points
to be
> >> >> processed
> >> >> > > // concurrently (i.e. in one pass through a time series).
Smaller
> >> >> values
> >> >> > > // require less memory but increase the number of passes
through
> >> the
> >> >> > data.
> >> >> > > //
> >> >> > > block_size = 1024;
> >> >> > >
> >> >> > > Look near the top of your log file for a message that
looks
> >> something
> >> >> > like
> >> >> > > this:
> >> >> > >
> >> >> > >
> >> >> > > *DEBUG 2: Computing statistics using a block size of
10000,
> >> requiring
> >> >> 3
> >> >> > > pass(es) through the 169 x 154 grid.DEBUG 2: Processing
data pass
> >> >> number
> >> >> > 1
> >> >> > > of 3 for grid points 1 to 10000.*
> >> >> > >
> >> >> > > This will tell you how many times it's reading through the
input
> >> data
> >> >> > > files. Ideally you'd set the block size high enough so
that it
> only
> >> >> needs
> >> >> > > to read them once. But that all depends on the number of
grid
> >> points
> >> >> and
> >> >> > > length of your series. If you set the block_size too high,
it may
> >> >> consume
> >> >> > > all the memory.
> >> >> > >
> >> >> > > Hope that helps clarify.
> >> >> > >
> >> >> > > Thanks,
> >> >> > > John
> >> >> > >
> >> >> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA
Federal via
> >> RT <
> >> >> > > met_help at ucar.edu> wrote:
> >> >> > >
> >> >> > > >
> >> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> >
> >> >> > > >
> >> >> > > > Thanks John. I'm testing the change now.
> >> >> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for
just 7
> >> >> days of
> >> >> > > > data. Its been running for over 2 hours now. Is this
the
> >> expected
> >> >> > > > behavior? I do have -v set to 3. Its writing steadily
to the
> >> >> log. Is
> >> >> > > > that slowing things down?
> >> >> > > >
> >> >> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via
RT <
> >> >> > > > met_help at ucar.edu> wrote:
> >> >> > > >
> >> >> > > > > Hello John,
> >> >> > > > >
> >> >> > > > > I see you're having difficulty configuring Series-
Analysis to
> >> >> apply
> >> >> > the
> >> >> > > > > continuous and categorical statistics you defined.
> >> >> > > > >
> >> >> > > > > It's funny, this exact same issue arose via met-help
last
> week,
> >> >> and
> >> >> > it
> >> >> > > > > makes me think we should modify the code in some way
to make
> it
> >> >> more
> >> >> > > > > intuitive. Please try editing your config file by
moving the
> >> >> > > cnt_thresh
> >> >> > > > > and cat_thresh settings up outside of the field array,
as
> shown
> >> >> > below:
> >> >> > > > >
> >> >> > > > > *fcst = {*
> >> >> > > > >
> >> >> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
> >=2.54&&<6.35,
> >> >> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
>=25.4&&<76.2,
> >> >> >=76.2
> >> >> > > ];*
> >> >> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7,
>=19.05,
> >> >> >=25.4,
> >> >> > > > > >=50.8, >=76.2 ];*
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > * field = [ { name = "${FIELD}";
level
> =
> >> >> > > > "${LEVEL}";
> >> >> > > > > } ];}obs = fcst;*
> >> >> > > > >
> >> >> > > > > When you rerun after this change, does it produce the
> expected
> >> >> > output?
> >> >> > > > >
> >> >> > > > > I realize that this is confusing, especially if you're
used
> to
> >> >> > setting
> >> >> > > up
> >> >> > > > > config files for Point-Stat and Grid-Stat. Those tools
are
> set
> >> up
> >> >> to
> >> >> > > > > process multiple fields in a single run. Therefore you
can
> >> specify
> >> >> > > > > cat_thresh and cnt_thresh separately for each field
array
> >> entry.
> >> >> > Unlike
> >> >> > > > > Point-Stat and Grid-Stat, Series-Analysis can only
process a
> >> >> single
> >> >> > > > series
> >> >> > > > > of data in each run. There are multiple main ways of
> defining a
> >> >> > series.
> >> >> > > > > (1) By far, the most common way is specifying a single
field
> >> that
> >> >> is
> >> >> > > > > extracted from a time-series of input data files... as
you're
> >> >> doing.
> >> >> > > > > (2) You can also define a series of field array
entries that
> >> can
> >> >> be
> >> >> > > > > extracted from the same input file. For example,
compute
> stats
> >> >> over
> >> >> > > > > multiple vertical levels from the same input file.
> >> >> > > > > (3) You define N input fields to be extracted from N
input
> >> files.
> >> >> > > > >
> >> >> > > > > Because of options (2) and (3), the field entry is
defined as
> >> an
> >> >> > array
> >> >> > > > > rather than a single dictionary. In practice though,
the
> field
> >> >> array
> >> >> > > > > typically has length 1.
> >> >> > > > >
> >> >> > > > > But the cat_thresh and cnt_thresh settings must remain
> constant
> >> >> over
> >> >> > > the
> >> >> > > > > entire series, and that's why the code is setup to
parse them
> >> from
> >> >> > the
> >> >> > > > > "fcst" and "obs" dictionaries, and not separately from
each
> >> field
> >> >> > array
> >> >> > > > > entry.
> >> >> > > > >
> >> >> > > > > Seems like it'd be a good idea to check for this
situation
> and
> >> >> print
> >> >> > a
> >> >> > > > > warning message for the user.
> >> >> > > > >
> >> >> > > > > Thanks,
> >> >> > > > > John
> >> >> > > > >
> >> >> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
> >> >> met_help at ucar.edu>
> >> >> > > > > wrote:
> >> >> > > > >
> >> >> > > > > >
> >> >> > > > > > <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >> >> > > > > >
> >> >> > > > > > Hello John,
> >> >> > > > > >
> >> >> > > > > > It looks like you need some assistance setting up
your
> series
> >> >> > > analysis
> >> >> > > > in
> >> >> > > > > > MET. I'm assigning this ticket to John Halley
Gotway.
> >> Please
> >> >> > allow
> >> >> > > a
> >> >> > > > > few
> >> >> > > > > > business days for a response.
> >> >> > > > > >
> >> >> > > > > > Regards,
> >> >> > > > > > Minna
> >> >> > > > > > ---------------
> >> >> > > > > > Minna Win
> >> >> > > > > > National Center for Atmospheric Research
> >> >> > > > > > Developmental Testbed Center
> >> >> > > > > > Phone: 303-497-8423
> >> >> > > > > > Fax: 303-497-8401
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA
> Federal
> >> via
> >> >> > RT <
> >> >> > > > > > met_help at ucar.edu> wrote:
> >> >> > > > > >
> >> >> > > > > > >
> >> >> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted
upon.
> >> >> > > > > > > Transaction: Ticket created by
john.l.wagner at noaa.gov
> >> >> > > > > > > Queue: met_help
> >> >> > > > > > > Subject: Series Analysis Config File Issues
> >> >> > > > > > > Owner: Nobody
> >> >> > > > > > > Requestors: john.l.wagner at noaa.gov
> >> >> > > > > > > Status: new
> >> >> > > > > > > Ticket <URL:
> >> >> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> >> >> > > > > >
> >> >> > > > > > >
> >> >> > > > > > >
> >> >> > > > > > > Good morning MET help
> >> >> > > > > > > I'm having an issue with series_analysis that I
could use
> >> your
> >> >> > > > > assistance
> >> >> > > > > > > with. I am currently testing series_analysis for
QPF06
> >> >> (config
> >> >> > > file
> >> >> > > > > > > attached). I am unable to get valid output values
for
> >> >> anything
> >> >> > in
> >> >> > > > the
> >> >> > > > > > CTS
> >> >> > > > > > > and MCTS categories. The TOTAL grid for each score
group
> >> looks
> >> >> > good
> >> >> > > > > (I'm
> >> >> > > > > > > currently testing with 7 days of data). The CSI
values
> are
> >> >> all
> >> >> > > > > constant
> >> >> > > > > > at
> >> >> > > > > > > 1. I am unable to get any values for HSS, either
from
> CTS
> >> or
> >> >> > MCTS.
> >> >> > > > > All
> >> >> > > > > > > output (MAE, ME, RMSE) from the CNT score group
seem to
> >> work
> >> >> > fine.
> >> >> > > > > > > I'm assuming the issue is setting the thresholds
in my
> >> config
> >> >> > file.
> >> >> > > > > > > The scores that I see in my netCDF file for the
CTS
> scores
> >> all
> >> >> > end
> >> >> > > > with
> >> >> > > > > > > "_NA", where I'm assuming a category should be
set. I am
> >> >> using
> >> >> > the
> >> >> > > > > same
> >> >> > > > > > > thresholds that I use in grid_stat. Should I be
setting
> >> them
> >> >> up
> >> >> > > > > > > differently for series_analysis? Is there a
sample
> config
> >> >> file
> >> >> > > that
> >> >> > > > I
> >> >> > > > > > > could look at for reference?
> >> >> > > > > > > If it helps, my test output file can be found on
WCOSS
> >> >> > > > > > > here:
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> >> >> > > > > > >
> >> >> > > > > > > Thanks
> >> >> > > > > > > John
> >> >> > > > > > > --
> >> >> > > > > > > John Wagner
> >> >> > > > > > > Verification Task Lead
> >> >> > > > > > > NOAA/National Weather Service
> >> >> > > > > > > Meteorological Development Laboratory
> >> >> > > > > > > Digital Forecast Services Division
> >> >> > > > > > > SSMC2 Room 10106
> >> >> > > > > > > Silver Spring, MD 20910
> >> >> > > > > > > (301) 427-9471 (office)
> >> >> > > > > > > (908) 902-4155 (cell/text)
> >> >> > > > > > >
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > > > --
> >> >> > > > John Wagner
> >> >> > > > Verification Task Lead
> >> >> > > > NOAA/National Weather Service
> >> >> > > > Meteorological Development Laboratory
> >> >> > > > Digital Forecast Services Division
> >> >> > > > SSMC2 Room 10106
> >> >> > > > Silver Spring, MD 20910
> >> >> > > > (301) 427-9471 (office)
> >> >> > > > (908) 902-4155 (cell/text)
> >> >> > > >
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> >
> >> >> > --
> >> >> > John Wagner
> >> >> > Verification Task Lead
> >> >> > NOAA/National Weather Service
> >> >> > Meteorological Development Laboratory
> >> >> > Digital Forecast Services Division
> >> >> > SSMC2 Room 10106
> >> >> > Silver Spring, MD 20910
> >> >> > (301) 427-9471 (office)
> >> >> > (908) 902-4155 (cell/text)
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >
> >> > --
> >> > John Wagner
> >> > Verification Task Lead
> >> > NOAA/National Weather Service
> >> > Meteorological Development Laboratory
> >> > Digital Forecast Services Division
> >> > SSMC2 Room 10106
> >> > Silver Spring, MD 20910
> >> > (301) 427-9471 (office)
> >> > (908) 902-4155 (cell/text)
> >> >
> >>
> >>
> >> --
> >> John Wagner
> >> Verification Task Lead
> >> NOAA/National Weather Service
> >> Meteorological Development Laboratory
> >> Digital Forecast Services Division
> >> SSMC2 Room 10106
> >> Silver Spring, MD 20910
> >> (301) 427-9471 (office)
> >> (908) 902-4155 (cell/text)
> >>
> >>
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Sat Jun 13 09:15:55 2020
Ah yes, that's a very nice comparison. The small test I ran did NOT
include
any cnt_thresh settings, so I wouldn't have seen those log messages.
And I
do see what that'd have a big impact.
And yes, that log message is written by the library code, which is
called
by many tools. When running Grid-Stat and Point-Stat, it seems
appropriate
to print that a level 3. For Series-Analysis, it seems like way too
much
info!
I could modify Series-Analysis to reset the logging level to 0 before
looping through the statistics computations... and then reinstate it
back
to the user's setting afterwards. Or perhaps only do that if the user
chose
5 or less. I worry that this could lead to confusing log messages...
but on
the other hand, I don't want users to go through the frustration and
debugging that you just did. Any opinions on that?
Also, since you're running Series-Analysis, I wanted to mention that
we had
a development request from the UK MetOffice this week. I wrote it up
in a
GitHub issue (https://github.com/NCAR/MET/issues/1371). The idea is to
enhance Series-Analysis to be able to read its own output and
aggregate
counts and partial sums across multiple runs. For example, perhaps
you've
run Series-Analysis to process 1 week at a time. And then at the end
of the
year, pass 52 weeks of runs into it to compute annual stats. Do you
think
you'd find that functionality useful?
Thanks,
John
On Sat, Jun 13, 2020 at 8:46 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
> That's fine. With -v 0, my log file is 512K. With -v 3, my log
file is
> 170M. Messages similar to "DEBUG 3: Using 7 of 7 pairs for forecast
> filtering threshold >=0.254, observation filtering threshold
>=0.254, and
> field logic Union." are printed 1,326,213 times. I'm not sure where
you're
> testing, but I wouldn't be surprised if this was a WCOSS issue. I
haven't
> exactly been blown away by the write speeds on phase 3.
> I'm going to run today for more projections and elements. I'll let
you
> know if there are any more issues.
> Thanks
> John
>
> On Fri, Jun 12, 2020 at 7:42 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > John,
> >
> > You know, I think I should leave well-enough alone. But I'm very
> surprised
> > about this change in runtime. I ran a short test case at verbosity
level
> 0
> > and it took 20 seconds. At verbosity level 4, it took 22 seconds.
More,
> but
> > not a crazy amount more.
> >
> > But I'm glad it's running in a more reasonable amount of time, and
hope
> it
> > continues to do so. I'll go ahead and resolve this ticket.
> >
> > Thanks,
> > John
> >
> > On Fri, Jun 12, 2020 at 5:29 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > John,
> > >
> > > Oh wow, OK. Would you recommend that I change some of those log
> messages
> > > to a higher verbosity level... like 4 or 5... to avoid slow
runtimes?
> > >
> > > John
> > >
> > > On Fri, Jun 12, 2020 at 3:57 PM John L Wagner - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > >>
> > >> John
> > >> I believe I've confirmed that the slow run times were from
setting -v
> 3
> > >> when calling series_analysis. My last test ran in about 25
minutes.
> I
> > >> have a few more tests to run, but I believe all of my issues
are
> solved.
> > >> Thanks
> > >> John
> > >>
> > >> On Fri, Jun 12, 2020 at 7:39 AM John L Wagner - NOAA Federal <
> > >> john.l.wagner at noaa.gov> wrote:
> > >>
> > >> > Here is the usage summary from the test I ran yesterday:
> > >> >
> > >> > [image: SerA Test1 Log.PNG]
> > >> > The job did not time out, and finished in 5 hr 43 min. It
did not
> use
> > >> all
> > >> > of the memory, which surprised me. When I was testing
incorrectly
> > with
> > >> the
> > >> > thresholds defined within the field array, I had to keep
increasing
> > the
> > >> > memory setting in my bsub command to prevent OOM kills.
> > >> >
> > >> > I'm running a test now with just the lowest CNT threshold.
This may
> > be
> > >> > what we end up running anyway and I'm hoping the run time
will come
> > >> down so
> > >> > I can run more tests. I'll try to get through tests today
with
> > cutting
> > >> the
> > >> > block_size in half, using a mask, and different memory
settings with
> > >> bsub.
> > >> > I'll let you know what I find.
> > >> >
> > >> > As for MPI, I agree with that solution. I'm not going to
pretend to
> > >> know
> > >> > what needs to be changed with C code and compilers. When we
updated
> > the
> > >> > MOS-2000 Fortran code, we managed to get our ensemble code
run time
> > down
> > >> > from 3 hours to 12 minutes. I realize that not all MET users
are
> > >> running
> > >> > on supercomputers that can take advantage of MPI, but those
of us
> that
> > >> are
> > >> > would certainly appreciate it.
> > >> >
> > >> > Thanks
> > >> > John
> > >> >
> > >> > On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> >> Agreed, that's much too long! I suppose another possibility
is that
> > >> it's
> > >> >> consuming too much memory and is using swap space, which is
slow.
> > >> >> If possible, please check what percent of memory the process
is
> > using.
> > >> If
> > >> >> it's up around 100%, then let's try cutting the block size
in half
> > and
> > >> >> rerunning.
> > >> >>
> > >> >> Honestly, the real fix for this is enhancing Series-Analysis
with
> MPI
> > >> >> calls
> > >> >> to subset the 4+ million grid point into tiles, and then do
the
> > >> processing
> > >> >> for each tile on a separate node.
> > >> >> However, none of the MET tools are currently parallelized.
So
> that'd
> > >> >> obviously be a significant development effort.
> > >> >>
> > >> >> Another option, just to get something to actually complete
in the
> > time
> > >> >> window, would be applying a masking region to subset the
data being
> > >> >> processed.
> > >> >>
> > >> >> Thanks,
> > >> >> John
> > >> >>
> > >> >> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal
via
> RT <
> > >> >> met_help at ucar.edu> wrote:
> > >> >>
> > >> >> >
> > >> >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > >> >> >
> > >> >> > I set the block size to my grid size (4194304) previously.
Its
> > still
> > >> >> > running 5+ hours later. It will time out after 6 hours.
> > >> >> > I have been testing on WCOSS with Julie's MET V9.0.
Tomorrow
> I'll
> > >> run a
> > >> >> > test with V8.1. Otherwise, we'll need to rethink what we
want to
> > >> show
> > >> >> in
> > >> >> > the viewer.
> > >> >> > Thanks
> > >> >> > John
> > >> >> >
> > >> >> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT
<
> > >> >> > met_help at ucar.edu>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > John,
> > >> >> > >
> > >> >> > > I'd recommend modifying the "block_size" config file
option and
> > >> >> > rerunning.
> > >> >> > > The default value is set very low, ensuring that you
don't run
> > out
> > >> of
> > >> >> > > memory but requiring many passes through the data...
which is
> > slow.
> > >> >> > >
> > >> >> > > Here's an excerpt from this README file which describes
how
> that
> > >> >> works.
> > >> >> > >
> > >> >> > >
> > >>
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >> >> > > //
> > >> >> > > // SeriesAnalysisConfig_default
> > >> >> > > //
> > >> >> > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >> >> > >
> > >> >> > > //
> > >> >> > > // Computation may be memory intensive, especially for
large
> > grids.
> > >> >> > > // The "block_size" entry sets the number of grid points
to be
> > >> >> processed
> > >> >> > > // concurrently (i.e. in one pass through a time
series).
> Smaller
> > >> >> values
> > >> >> > > // require less memory but increase the number of passes
> through
> > >> the
> > >> >> > data.
> > >> >> > > //
> > >> >> > > block_size = 1024;
> > >> >> > >
> > >> >> > > Look near the top of your log file for a message that
looks
> > >> something
> > >> >> > like
> > >> >> > > this:
> > >> >> > >
> > >> >> > >
> > >> >> > > *DEBUG 2: Computing statistics using a block size of
10000,
> > >> requiring
> > >> >> 3
> > >> >> > > pass(es) through the 169 x 154 grid.DEBUG 2: Processing
data
> pass
> > >> >> number
> > >> >> > 1
> > >> >> > > of 3 for grid points 1 to 10000.*
> > >> >> > >
> > >> >> > > This will tell you how many times it's reading through
the
> input
> > >> data
> > >> >> > > files. Ideally you'd set the block size high enough so
that it
> > only
> > >> >> needs
> > >> >> > > to read them once. But that all depends on the number of
grid
> > >> points
> > >> >> and
> > >> >> > > length of your series. If you set the block_size too
high, it
> may
> > >> >> consume
> > >> >> > > all the memory.
> > >> >> > >
> > >> >> > > Hope that helps clarify.
> > >> >> > >
> > >> >> > > Thanks,
> > >> >> > > John
> > >> >> > >
> > >> >> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA
Federal
> via
> > >> RT <
> > >> >> > > met_help at ucar.edu> wrote:
> > >> >> > >
> > >> >> > > >
> > >> >> > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > >
> > >> >> > > >
> > >> >> > > > Thanks John. I'm testing the change now.
> > >> >> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS
for
> just 7
> > >> >> days of
> > >> >> > > > data. Its been running for over 2 hours now. Is this
the
> > >> expected
> > >> >> > > > behavior? I do have -v set to 3. Its writing
steadily to
> the
> > >> >> log. Is
> > >> >> > > > that slowing things down?
> > >> >> > > >
> > >> >> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway
via RT <
> > >> >> > > > met_help at ucar.edu> wrote:
> > >> >> > > >
> > >> >> > > > > Hello John,
> > >> >> > > > >
> > >> >> > > > > I see you're having difficulty configuring Series-
Analysis
> to
> > >> >> apply
> > >> >> > the
> > >> >> > > > > continuous and categorical statistics you defined.
> > >> >> > > > >
> > >> >> > > > > It's funny, this exact same issue arose via met-help
last
> > week,
> > >> >> and
> > >> >> > it
> > >> >> > > > > makes me think we should modify the code in some way
to
> make
> > it
> > >> >> more
> > >> >> > > > > intuitive. Please try editing your config file by
moving
> the
> > >> >> > > cnt_thresh
> > >> >> > > > > and cat_thresh settings up outside of the field
array, as
> > shown
> > >> >> > below:
> > >> >> > > > >
> > >> >> > > > > *fcst = {*
> > >> >> > > > >
> > >> >> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
> > >=2.54&&<6.35,
> > >> >> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
> >=25.4&&<76.2,
> > >> >> >=76.2
> > >> >> > > ];*
> > >> >> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7,
> >=19.05,
> > >> >> >=25.4,
> > >> >> > > > > >=50.8, >=76.2 ];*
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > > * field = [ { name = "${FIELD}";
> level
> > =
> > >> >> > > > "${LEVEL}";
> > >> >> > > > > } ];}obs = fcst;*
> > >> >> > > > >
> > >> >> > > > > When you rerun after this change, does it produce
the
> > expected
> > >> >> > output?
> > >> >> > > > >
> > >> >> > > > > I realize that this is confusing, especially if
you're used
> > to
> > >> >> > setting
> > >> >> > > up
> > >> >> > > > > config files for Point-Stat and Grid-Stat. Those
tools are
> > set
> > >> up
> > >> >> to
> > >> >> > > > > process multiple fields in a single run. Therefore
you can
> > >> specify
> > >> >> > > > > cat_thresh and cnt_thresh separately for each field
array
> > >> entry.
> > >> >> > Unlike
> > >> >> > > > > Point-Stat and Grid-Stat, Series-Analysis can only
process
> a
> > >> >> single
> > >> >> > > > series
> > >> >> > > > > of data in each run. There are multiple main ways of
> > defining a
> > >> >> > series.
> > >> >> > > > > (1) By far, the most common way is specifying a
single
> field
> > >> that
> > >> >> is
> > >> >> > > > > extracted from a time-series of input data files...
as
> you're
> > >> >> doing.
> > >> >> > > > > (2) You can also define a series of field array
entries
> that
> > >> can
> > >> >> be
> > >> >> > > > > extracted from the same input file. For example,
compute
> > stats
> > >> >> over
> > >> >> > > > > multiple vertical levels from the same input file.
> > >> >> > > > > (3) You define N input fields to be extracted from N
input
> > >> files.
> > >> >> > > > >
> > >> >> > > > > Because of options (2) and (3), the field entry is
defined
> as
> > >> an
> > >> >> > array
> > >> >> > > > > rather than a single dictionary. In practice though,
the
> > field
> > >> >> array
> > >> >> > > > > typically has length 1.
> > >> >> > > > >
> > >> >> > > > > But the cat_thresh and cnt_thresh settings must
remain
> > constant
> > >> >> over
> > >> >> > > the
> > >> >> > > > > entire series, and that's why the code is setup to
parse
> them
> > >> from
> > >> >> > the
> > >> >> > > > > "fcst" and "obs" dictionaries, and not separately
from each
> > >> field
> > >> >> > array
> > >> >> > > > > entry.
> > >> >> > > > >
> > >> >> > > > > Seems like it'd be a good idea to check for this
situation
> > and
> > >> >> print
> > >> >> > a
> > >> >> > > > > warning message for the user.
> > >> >> > > > >
> > >> >> > > > > Thanks,
> > >> >> > > > > John
> > >> >> > > > >
> > >> >> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
> > >> >> met_help at ucar.edu>
> > >> >> > > > > wrote:
> > >> >> > > > >
> > >> >> > > > > >
> > >> >> > > > > > <URL:
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > >> >> > > > > >
> > >> >> > > > > > Hello John,
> > >> >> > > > > >
> > >> >> > > > > > It looks like you need some assistance setting up
your
> > series
> > >> >> > > analysis
> > >> >> > > > in
> > >> >> > > > > > MET. I'm assigning this ticket to John Halley
Gotway.
> > >> Please
> > >> >> > allow
> > >> >> > > a
> > >> >> > > > > few
> > >> >> > > > > > business days for a response.
> > >> >> > > > > >
> > >> >> > > > > > Regards,
> > >> >> > > > > > Minna
> > >> >> > > > > > ---------------
> > >> >> > > > > > Minna Win
> > >> >> > > > > > National Center for Atmospheric Research
> > >> >> > > > > > Developmental Testbed Center
> > >> >> > > > > > Phone: 303-497-8423
> > >> >> > > > > > Fax: 303-497-8401
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner -
NOAA
> > Federal
> > >> via
> > >> >> > RT <
> > >> >> > > > > > met_help at ucar.edu> wrote:
> > >> >> > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was
acted upon.
> > >> >> > > > > > > Transaction: Ticket created by
john.l.wagner at noaa.gov
> > >> >> > > > > > > Queue: met_help
> > >> >> > > > > > > Subject: Series Analysis Config File Issues
> > >> >> > > > > > > Owner: Nobody
> > >> >> > > > > > > Requestors: john.l.wagner at noaa.gov
> > >> >> > > > > > > Status: new
> > >> >> > > > > > > Ticket <URL:
> > >> >> > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > >> >> > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > > Good morning MET help
> > >> >> > > > > > > I'm having an issue with series_analysis that I
could
> use
> > >> your
> > >> >> > > > > assistance
> > >> >> > > > > > > with. I am currently testing series_analysis
for QPF06
> > >> >> (config
> > >> >> > > file
> > >> >> > > > > > > attached). I am unable to get valid output
values for
> > >> >> anything
> > >> >> > in
> > >> >> > > > the
> > >> >> > > > > > CTS
> > >> >> > > > > > > and MCTS categories. The TOTAL grid for each
score
> group
> > >> looks
> > >> >> > good
> > >> >> > > > > (I'm
> > >> >> > > > > > > currently testing with 7 days of data). The CSI
values
> > are
> > >> >> all
> > >> >> > > > > constant
> > >> >> > > > > > at
> > >> >> > > > > > > 1. I am unable to get any values for HSS,
either from
> > CTS
> > >> or
> > >> >> > MCTS.
> > >> >> > > > > All
> > >> >> > > > > > > output (MAE, ME, RMSE) from the CNT score group
seem to
> > >> work
> > >> >> > fine.
> > >> >> > > > > > > I'm assuming the issue is setting the thresholds
in my
> > >> config
> > >> >> > file.
> > >> >> > > > > > > The scores that I see in my netCDF file for the
CTS
> > scores
> > >> all
> > >> >> > end
> > >> >> > > > with
> > >> >> > > > > > > "_NA", where I'm assuming a category should be
set. I
> am
> > >> >> using
> > >> >> > the
> > >> >> > > > > same
> > >> >> > > > > > > thresholds that I use in grid_stat. Should I be
> setting
> > >> them
> > >> >> up
> > >> >> > > > > > > differently for series_analysis? Is there a
sample
> > config
> > >> >> file
> > >> >> > > that
> > >> >> > > > I
> > >> >> > > > > > > could look at for reference?
> > >> >> > > > > > > If it helps, my test output file can be found on
WCOSS
> > >> >> > > > > > > here:
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > >> >> > > > > > >
> > >> >> > > > > > > Thanks
> > >> >> > > > > > > John
> > >> >> > > > > > > --
> > >> >> > > > > > > John Wagner
> > >> >> > > > > > > Verification Task Lead
> > >> >> > > > > > > NOAA/National Weather Service
> > >> >> > > > > > > Meteorological Development Laboratory
> > >> >> > > > > > > Digital Forecast Services Division
> > >> >> > > > > > > SSMC2 Room 10106
> > >> >> > > > > > > Silver Spring, MD 20910
> > >> >> > > > > > > (301) 427-9471 (office)
> > >> >> > > > > > > (908) 902-4155 (cell/text)
> > >> >> > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > > > --
> > >> >> > > > John Wagner
> > >> >> > > > Verification Task Lead
> > >> >> > > > NOAA/National Weather Service
> > >> >> > > > Meteorological Development Laboratory
> > >> >> > > > Digital Forecast Services Division
> > >> >> > > > SSMC2 Room 10106
> > >> >> > > > Silver Spring, MD 20910
> > >> >> > > > (301) 427-9471 (office)
> > >> >> > > > (908) 902-4155 (cell/text)
> > >> >> > > >
> > >> >> > > >
> > >> >> > >
> > >> >> > >
> > >> >> >
> > >> >> > --
> > >> >> > John Wagner
> > >> >> > Verification Task Lead
> > >> >> > NOAA/National Weather Service
> > >> >> > Meteorological Development Laboratory
> > >> >> > Digital Forecast Services Division
> > >> >> > SSMC2 Room 10106
> > >> >> > Silver Spring, MD 20910
> > >> >> > (301) 427-9471 (office)
> > >> >> > (908) 902-4155 (cell/text)
> > >> >> >
> > >> >> >
> > >> >>
> > >> >>
> > >> >
> > >> > --
> > >> > John Wagner
> > >> > Verification Task Lead
> > >> > NOAA/National Weather Service
> > >> > Meteorological Development Laboratory
> > >> > Digital Forecast Services Division
> > >> > SSMC2 Room 10106
> > >> > Silver Spring, MD 20910
> > >> > (301) 427-9471 (office)
> > >> > (908) 902-4155 (cell/text)
> > >> >
> > >>
> > >>
> > >> --
> > >> John Wagner
> > >> Verification Task Lead
> > >> NOAA/National Weather Service
> > >> Meteorological Development Laboratory
> > >> Digital Forecast Services Division
> > >> SSMC2 Room 10106
> > >> Silver Spring, MD 20910
> > >> (301) 427-9471 (office)
> > >> (908) 902-4155 (cell/text)
> > >>
> > >>
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John L Wagner - NOAA Federal
Time: Sat Jun 13 09:34:53 2020
On Sat, Jun 13, 2020 at 11:16 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Ah yes, that's a very nice comparison. The small test I ran did NOT
include
> any cnt_thresh settings, so I wouldn't have seen those log messages.
And I
> do see what that'd have a big impact.
>
> And yes, that log message is written by the library code, which is
called
> by many tools. When running Grid-Stat and Point-Stat, it seems
appropriate
> to print that a level 3. For Series-Analysis, it seems like way too
much
> info!
>
> I could modify Series-Analysis to reset the logging level to 0
before
> looping through the statistics computations... and then reinstate it
back
> to the user's setting afterwards. Or perhaps only do that if the
user chose
> 5 or less. I worry that this could lead to confusing log messages...
but on
> the other hand, I don't want users to go through the frustration and
> debugging that you just did. Any opinions on that?
>
If you didn't want to change how the library works between programs,
could
you just put a message when Series-Analysis starts that says that
running
with -v = 3 or higher can cause significantly longer run times?
>
> Also, since you're running Series-Analysis, I wanted to mention that
we had
> a development request from the UK MetOffice this week. I wrote it up
in a
> GitHub issue (https://github.com/NCAR/MET/issues/1371). The idea is
to
> enhance Series-Analysis to be able to read its own output and
aggregate
> counts and partial sums across multiple runs. For example, perhaps
you've
> run Series-Analysis to process 1 week at a time. And then at the end
of the
> year, pass 52 weeks of runs into it to compute annual stats. Do you
think
> you'd find that functionality useful?
>
I can see where combining months into seasons would be useful for us.
Combining multiple seasons could be useful soon. Yes, I think this is
a
good idea!
>
> Thanks,
> John
>
> On Sat, Jun 13, 2020 at 8:46 AM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >
> > That's fine. With -v 0, my log file is 512K. With -v 3, my log
file is
> > 170M. Messages similar to "DEBUG 3: Using 7 of 7 pairs for
forecast
> > filtering threshold >=0.254, observation filtering threshold
>=0.254, and
> > field logic Union." are printed 1,326,213 times. I'm not sure
where
> you're
> > testing, but I wouldn't be surprised if this was a WCOSS issue. I
> haven't
> > exactly been blown away by the write speeds on phase 3.
> > I'm going to run today for more projections and elements. I'll
let you
> > know if there are any more issues.
> > Thanks
> > John
> >
> > On Fri, Jun 12, 2020 at 7:42 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > John,
> > >
> > > You know, I think I should leave well-enough alone. But I'm very
> > surprised
> > > about this change in runtime. I ran a short test case at
verbosity
> level
> > 0
> > > and it took 20 seconds. At verbosity level 4, it took 22
seconds. More,
> > but
> > > not a crazy amount more.
> > >
> > > But I'm glad it's running in a more reasonable amount of time,
and hope
> > it
> > > continues to do so. I'll go ahead and resolve this ticket.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Jun 12, 2020 at 5:29 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > John,
> > > >
> > > > Oh wow, OK. Would you recommend that I change some of those
log
> > messages
> > > > to a higher verbosity level... like 4 or 5... to avoid slow
runtimes?
> > > >
> > > > John
> > > >
> > > > On Fri, Jun 12, 2020 at 3:57 PM John L Wagner - NOAA Federal
via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>
> > > >>
> > > >> John
> > > >> I believe I've confirmed that the slow run times were from
setting
> -v
> > 3
> > > >> when calling series_analysis. My last test ran in about 25
minutes.
> > I
> > > >> have a few more tests to run, but I believe all of my issues
are
> > solved.
> > > >> Thanks
> > > >> John
> > > >>
> > > >> On Fri, Jun 12, 2020 at 7:39 AM John L Wagner - NOAA Federal
<
> > > >> john.l.wagner at noaa.gov> wrote:
> > > >>
> > > >> > Here is the usage summary from the test I ran yesterday:
> > > >> >
> > > >> > [image: SerA Test1 Log.PNG]
> > > >> > The job did not time out, and finished in 5 hr 43 min. It
did not
> > use
> > > >> all
> > > >> > of the memory, which surprised me. When I was testing
incorrectly
> > > with
> > > >> the
> > > >> > thresholds defined within the field array, I had to keep
> increasing
> > > the
> > > >> > memory setting in my bsub command to prevent OOM kills.
> > > >> >
> > > >> > I'm running a test now with just the lowest CNT threshold.
This
> may
> > > be
> > > >> > what we end up running anyway and I'm hoping the run time
will
> come
> > > >> down so
> > > >> > I can run more tests. I'll try to get through tests today
with
> > > cutting
> > > >> the
> > > >> > block_size in half, using a mask, and different memory
settings
> with
> > > >> bsub.
> > > >> > I'll let you know what I find.
> > > >> >
> > > >> > As for MPI, I agree with that solution. I'm not going to
pretend
> to
> > > >> know
> > > >> > what needs to be changed with C code and compilers. When
we
> updated
> > > the
> > > >> > MOS-2000 Fortran code, we managed to get our ensemble code
run
> time
> > > down
> > > >> > from 3 hours to 12 minutes. I realize that not all MET
users are
> > > >> running
> > > >> > on supercomputers that can take advantage of MPI, but those
of us
> > that
> > > >> are
> > > >> > would certainly appreciate it.
> > > >> >
> > > >> > Thanks
> > > >> > John
> > > >> >
> > > >> > On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> >> Agreed, that's much too long! I suppose another
possibility is
> that
> > > >> it's
> > > >> >> consuming too much memory and is using swap space, which
is slow.
> > > >> >> If possible, please check what percent of memory the
process is
> > > using.
> > > >> If
> > > >> >> it's up around 100%, then let's try cutting the block size
in
> half
> > > and
> > > >> >> rerunning.
> > > >> >>
> > > >> >> Honestly, the real fix for this is enhancing Series-
Analysis with
> > MPI
> > > >> >> calls
> > > >> >> to subset the 4+ million grid point into tiles, and then
do the
> > > >> processing
> > > >> >> for each tile on a separate node.
> > > >> >> However, none of the MET tools are currently parallelized.
So
> > that'd
> > > >> >> obviously be a significant development effort.
> > > >> >>
> > > >> >> Another option, just to get something to actually complete
in the
> > > time
> > > >> >> window, would be applying a masking region to subset the
data
> being
> > > >> >> processed.
> > > >> >>
> > > >> >> Thanks,
> > > >> >> John
> > > >> >>
> > > >> >> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA
Federal via
> > RT <
> > > >> >> met_help at ucar.edu> wrote:
> > > >> >>
> > > >> >> >
> > > >> >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> >
> > > >> >> >
> > > >> >> > I set the block size to my grid size (4194304)
previously. Its
> > > still
> > > >> >> > running 5+ hours later. It will time out after 6 hours.
> > > >> >> > I have been testing on WCOSS with Julie's MET V9.0.
Tomorrow
> > I'll
> > > >> run a
> > > >> >> > test with V8.1. Otherwise, we'll need to rethink what
we want
> to
> > > >> show
> > > >> >> in
> > > >> >> > the viewer.
> > > >> >> > Thanks
> > > >> >> > John
> > > >> >> >
> > > >> >> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via
RT <
> > > >> >> > met_help at ucar.edu>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> > > John,
> > > >> >> > >
> > > >> >> > > I'd recommend modifying the "block_size" config file
option
> and
> > > >> >> > rerunning.
> > > >> >> > > The default value is set very low, ensuring that you
don't
> run
> > > out
> > > >> of
> > > >> >> > > memory but requiring many passes through the data...
which is
> > > slow.
> > > >> >> > >
> > > >> >> > > Here's an excerpt from this README file which
describes how
> > that
> > > >> >> works.
> > > >> >> > >
> > > >> >> > >
> > > >>
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
> > > >> >> > >
> > > >> >> > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >> >> > > //
> > > >> >> > > // SeriesAnalysisConfig_default
> > > >> >> > > //
> > > >> >> > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >> >> > >
> > > >> >> > > //
> > > >> >> > > // Computation may be memory intensive, especially for
large
> > > grids.
> > > >> >> > > // The "block_size" entry sets the number of grid
points to
> be
> > > >> >> processed
> > > >> >> > > // concurrently (i.e. in one pass through a time
series).
> > Smaller
> > > >> >> values
> > > >> >> > > // require less memory but increase the number of
passes
> > through
> > > >> the
> > > >> >> > data.
> > > >> >> > > //
> > > >> >> > > block_size = 1024;
> > > >> >> > >
> > > >> >> > > Look near the top of your log file for a message that
looks
> > > >> something
> > > >> >> > like
> > > >> >> > > this:
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > *DEBUG 2: Computing statistics using a block size of
10000,
> > > >> requiring
> > > >> >> 3
> > > >> >> > > pass(es) through the 169 x 154 grid.DEBUG 2:
Processing data
> > pass
> > > >> >> number
> > > >> >> > 1
> > > >> >> > > of 3 for grid points 1 to 10000.*
> > > >> >> > >
> > > >> >> > > This will tell you how many times it's reading through
the
> > input
> > > >> data
> > > >> >> > > files. Ideally you'd set the block size high enough so
that
> it
> > > only
> > > >> >> needs
> > > >> >> > > to read them once. But that all depends on the number
of grid
> > > >> points
> > > >> >> and
> > > >> >> > > length of your series. If you set the block_size too
high, it
> > may
> > > >> >> consume
> > > >> >> > > all the memory.
> > > >> >> > >
> > > >> >> > > Hope that helps clarify.
> > > >> >> > >
> > > >> >> > > Thanks,
> > > >> >> > > John
> > > >> >> > >
> > > >> >> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA
Federal
> > via
> > > >> RT <
> > > >> >> > > met_help at ucar.edu> wrote:
> > > >> >> > >
> > > >> >> > > >
> > > >> >> > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > > >
> > > >> >> > > >
> > > >> >> > > > Thanks John. I'm testing the change now.
> > > >> >> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS
for
> > just 7
> > > >> >> days of
> > > >> >> > > > data. Its been running for over 2 hours now. Is
this the
> > > >> expected
> > > >> >> > > > behavior? I do have -v set to 3. Its writing
steadily to
> > the
> > > >> >> log. Is
> > > >> >> > > > that slowing things down?
> > > >> >> > > >
> > > >> >> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway
via RT
> <
> > > >> >> > > > met_help at ucar.edu> wrote:
> > > >> >> > > >
> > > >> >> > > > > Hello John,
> > > >> >> > > > >
> > > >> >> > > > > I see you're having difficulty configuring
> Series-Analysis
> > to
> > > >> >> apply
> > > >> >> > the
> > > >> >> > > > > continuous and categorical statistics you defined.
> > > >> >> > > > >
> > > >> >> > > > > It's funny, this exact same issue arose via met-
help last
> > > week,
> > > >> >> and
> > > >> >> > it
> > > >> >> > > > > makes me think we should modify the code in some
way to
> > make
> > > it
> > > >> >> more
> > > >> >> > > > > intuitive. Please try editing your config file by
moving
> > the
> > > >> >> > > cnt_thresh
> > > >> >> > > > > and cat_thresh settings up outside of the field
array, as
> > > shown
> > > >> >> > below:
> > > >> >> > > > >
> > > >> >> > > > > *fcst = {*
> > > >> >> > > > >
> > > >> >> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
> > > >=2.54&&<6.35,
> > > >> >> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
> > >=25.4&&<76.2,
> > > >> >> >=76.2
> > > >> >> > > ];*
> > > >> >> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35,
>=12.7,
> > >=19.05,
> > > >> >> >=25.4,
> > > >> >> > > > > >=50.8, >=76.2 ];*
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > > * field = [ { name = "${FIELD}";
> > level
> > > =
> > > >> >> > > > "${LEVEL}";
> > > >> >> > > > > } ];}obs = fcst;*
> > > >> >> > > > >
> > > >> >> > > > > When you rerun after this change, does it produce
the
> > > expected
> > > >> >> > output?
> > > >> >> > > > >
> > > >> >> > > > > I realize that this is confusing, especially if
you're
> used
> > > to
> > > >> >> > setting
> > > >> >> > > up
> > > >> >> > > > > config files for Point-Stat and Grid-Stat. Those
tools
> are
> > > set
> > > >> up
> > > >> >> to
> > > >> >> > > > > process multiple fields in a single run. Therefore
you
> can
> > > >> specify
> > > >> >> > > > > cat_thresh and cnt_thresh separately for each
field array
> > > >> entry.
> > > >> >> > Unlike
> > > >> >> > > > > Point-Stat and Grid-Stat, Series-Analysis can only
> process
> > a
> > > >> >> single
> > > >> >> > > > series
> > > >> >> > > > > of data in each run. There are multiple main ways
of
> > > defining a
> > > >> >> > series.
> > > >> >> > > > > (1) By far, the most common way is specifying a
single
> > field
> > > >> that
> > > >> >> is
> > > >> >> > > > > extracted from a time-series of input data
files... as
> > you're
> > > >> >> doing.
> > > >> >> > > > > (2) You can also define a series of field array
entries
> > that
> > > >> can
> > > >> >> be
> > > >> >> > > > > extracted from the same input file. For example,
compute
> > > stats
> > > >> >> over
> > > >> >> > > > > multiple vertical levels from the same input file.
> > > >> >> > > > > (3) You define N input fields to be extracted from
N
> input
> > > >> files.
> > > >> >> > > > >
> > > >> >> > > > > Because of options (2) and (3), the field entry is
> defined
> > as
> > > >> an
> > > >> >> > array
> > > >> >> > > > > rather than a single dictionary. In practice
though, the
> > > field
> > > >> >> array
> > > >> >> > > > > typically has length 1.
> > > >> >> > > > >
> > > >> >> > > > > But the cat_thresh and cnt_thresh settings must
remain
> > > constant
> > > >> >> over
> > > >> >> > > the
> > > >> >> > > > > entire series, and that's why the code is setup to
parse
> > them
> > > >> from
> > > >> >> > the
> > > >> >> > > > > "fcst" and "obs" dictionaries, and not separately
from
> each
> > > >> field
> > > >> >> > array
> > > >> >> > > > > entry.
> > > >> >> > > > >
> > > >> >> > > > > Seems like it'd be a good idea to check for this
> situation
> > > and
> > > >> >> print
> > > >> >> > a
> > > >> >> > > > > warning message for the user.
> > > >> >> > > > >
> > > >> >> > > > > Thanks,
> > > >> >> > > > > John
> > > >> >> > > > >
> > > >> >> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
> > > >> >> met_help at ucar.edu>
> > > >> >> > > > > wrote:
> > > >> >> > > > >
> > > >> >> > > > > >
> > > >> >> > > > > > <URL:
> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > > >> >> > > > > >
> > > >> >> > > > > > Hello John,
> > > >> >> > > > > >
> > > >> >> > > > > > It looks like you need some assistance setting
up your
> > > series
> > > >> >> > > analysis
> > > >> >> > > > in
> > > >> >> > > > > > MET. I'm assigning this ticket to John Halley
Gotway.
> > > >> Please
> > > >> >> > allow
> > > >> >> > > a
> > > >> >> > > > > few
> > > >> >> > > > > > business days for a response.
> > > >> >> > > > > >
> > > >> >> > > > > > Regards,
> > > >> >> > > > > > Minna
> > > >> >> > > > > > ---------------
> > > >> >> > > > > > Minna Win
> > > >> >> > > > > > National Center for Atmospheric Research
> > > >> >> > > > > > Developmental Testbed Center
> > > >> >> > > > > > Phone: 303-497-8423
> > > >> >> > > > > > Fax: 303-497-8401
> > > >> >> > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner -
NOAA
> > > Federal
> > > >> via
> > > >> >> > RT <
> > > >> >> > > > > > met_help at ucar.edu> wrote:
> > > >> >> > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was
acted
> upon.
> > > >> >> > > > > > > Transaction: Ticket created by
> john.l.wagner at noaa.gov
> > > >> >> > > > > > > Queue: met_help
> > > >> >> > > > > > > Subject: Series Analysis Config File
Issues
> > > >> >> > > > > > > Owner: Nobody
> > > >> >> > > > > > > Requestors: john.l.wagner at noaa.gov
> > > >> >> > > > > > > Status: new
> > > >> >> > > > > > > Ticket <URL:
> > > >> >> > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > > >> >> > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > > Good morning MET help
> > > >> >> > > > > > > I'm having an issue with series_analysis that
I could
> > use
> > > >> your
> > > >> >> > > > > assistance
> > > >> >> > > > > > > with. I am currently testing series_analysis
for
> QPF06
> > > >> >> (config
> > > >> >> > > file
> > > >> >> > > > > > > attached). I am unable to get valid output
values
> for
> > > >> >> anything
> > > >> >> > in
> > > >> >> > > > the
> > > >> >> > > > > > CTS
> > > >> >> > > > > > > and MCTS categories. The TOTAL grid for each
score
> > group
> > > >> looks
> > > >> >> > good
> > > >> >> > > > > (I'm
> > > >> >> > > > > > > currently testing with 7 days of data). The
CSI
> values
> > > are
> > > >> >> all
> > > >> >> > > > > constant
> > > >> >> > > > > > at
> > > >> >> > > > > > > 1. I am unable to get any values for HSS,
either
> from
> > > CTS
> > > >> or
> > > >> >> > MCTS.
> > > >> >> > > > > All
> > > >> >> > > > > > > output (MAE, ME, RMSE) from the CNT score
group seem
> to
> > > >> work
> > > >> >> > fine.
> > > >> >> > > > > > > I'm assuming the issue is setting the
thresholds in
> my
> > > >> config
> > > >> >> > file.
> > > >> >> > > > > > > The scores that I see in my netCDF file for
the CTS
> > > scores
> > > >> all
> > > >> >> > end
> > > >> >> > > > with
> > > >> >> > > > > > > "_NA", where I'm assuming a category should be
set.
> I
> > am
> > > >> >> using
> > > >> >> > the
> > > >> >> > > > > same
> > > >> >> > > > > > > thresholds that I use in grid_stat. Should I
be
> > setting
> > > >> them
> > > >> >> up
> > > >> >> > > > > > > differently for series_analysis? Is there a
sample
> > > config
> > > >> >> file
> > > >> >> > > that
> > > >> >> > > > I
> > > >> >> > > > > > > could look at for reference?
> > > >> >> > > > > > > If it helps, my test output file can be found
on
> WCOSS
> > > >> >> > > > > > > here:
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > > >> >> > > > > > >
> > > >> >> > > > > > > Thanks
> > > >> >> > > > > > > John
> > > >> >> > > > > > > --
> > > >> >> > > > > > > John Wagner
> > > >> >> > > > > > > Verification Task Lead
> > > >> >> > > > > > > NOAA/National Weather Service
> > > >> >> > > > > > > Meteorological Development Laboratory
> > > >> >> > > > > > > Digital Forecast Services Division
> > > >> >> > > > > > > SSMC2 Room 10106
> > > >> >> > > > > > > Silver Spring, MD 20910
> > > >> >> > > > > > > (301) 427-9471 (office)
> > > >> >> > > > > > > (908) 902-4155 (cell/text)
> > > >> >> > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > > > --
> > > >> >> > > > John Wagner
> > > >> >> > > > Verification Task Lead
> > > >> >> > > > NOAA/National Weather Service
> > > >> >> > > > Meteorological Development Laboratory
> > > >> >> > > > Digital Forecast Services Division
> > > >> >> > > > SSMC2 Room 10106
> > > >> >> > > > Silver Spring, MD 20910
> > > >> >> > > > (301) 427-9471 (office)
> > > >> >> > > > (908) 902-4155 (cell/text)
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> > >
> > > >> >> >
> > > >> >> > --
> > > >> >> > John Wagner
> > > >> >> > Verification Task Lead
> > > >> >> > NOAA/National Weather Service
> > > >> >> > Meteorological Development Laboratory
> > > >> >> > Digital Forecast Services Division
> > > >> >> > SSMC2 Room 10106
> > > >> >> > Silver Spring, MD 20910
> > > >> >> > (301) 427-9471 (office)
> > > >> >> > (908) 902-4155 (cell/text)
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >
> > > >> > --
> > > >> > John Wagner
> > > >> > Verification Task Lead
> > > >> > NOAA/National Weather Service
> > > >> > Meteorological Development Laboratory
> > > >> > Digital Forecast Services Division
> > > >> > SSMC2 Room 10106
> > > >> > Silver Spring, MD 20910
> > > >> > (301) 427-9471 (office)
> > > >> > (908) 902-4155 (cell/text)
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> John Wagner
> > > >> Verification Task Lead
> > > >> NOAA/National Weather Service
> > > >> Meteorological Development Laboratory
> > > >> Digital Forecast Services Division
> > > >> SSMC2 Room 10106
> > > >> Silver Spring, MD 20910
> > > >> (301) 427-9471 (office)
> > > >> (908) 902-4155 (cell/text)
> > > >>
> > > >>
> > >
> > >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Fri Jun 19 12:36:54 2020
John,
FYI, I'm adding this warning about -v 3 for Series-Analysis.
Here's the issue: https://github.com/NCAR/MET/issues/1382
And here's the change: https://github.com/NCAR/MET/pull/1383
I'll go ahead and resolve this ticket.
Thanks,
John
On Sat, Jun 13, 2020 at 9:35 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
> On Sat, Jun 13, 2020 at 11:16 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Ah yes, that's a very nice comparison. The small test I ran did
NOT
> include
> > any cnt_thresh settings, so I wouldn't have seen those log
messages. And
> I
> > do see what that'd have a big impact.
> >
> > And yes, that log message is written by the library code, which is
called
> > by many tools. When running Grid-Stat and Point-Stat, it seems
> appropriate
> > to print that a level 3. For Series-Analysis, it seems like way
too much
> > info!
> >
> > I could modify Series-Analysis to reset the logging level to 0
before
> > looping through the statistics computations... and then reinstate
it back
> > to the user's setting afterwards. Or perhaps only do that if the
user
> chose
> > 5 or less. I worry that this could lead to confusing log
messages... but
> on
> > the other hand, I don't want users to go through the frustration
and
> > debugging that you just did. Any opinions on that?
> >
>
> If you didn't want to change how the library works between programs,
could
> you just put a message when Series-Analysis starts that says that
running
> with -v = 3 or higher can cause significantly longer run times?
>
>
> >
> > Also, since you're running Series-Analysis, I wanted to mention
that we
> had
> > a development request from the UK MetOffice this week. I wrote it
up in a
> > GitHub issue (https://github.com/NCAR/MET/issues/1371). The idea
is to
> > enhance Series-Analysis to be able to read its own output and
aggregate
> > counts and partial sums across multiple runs. For example, perhaps
you've
> > run Series-Analysis to process 1 week at a time. And then at the
end of
> the
> > year, pass 52 weeks of runs into it to compute annual stats. Do
you think
> > you'd find that functionality useful?
> >
>
> I can see where combining months into seasons would be useful for
us.
> Combining multiple seasons could be useful soon. Yes, I think this
is a
> good idea!
>
>
> >
> > Thanks,
> > John
> >
> > On Sat, Jun 13, 2020 at 8:46 AM John L Wagner - NOAA Federal via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > >
> > > That's fine. With -v 0, my log file is 512K. With -v 3, my log
file
> is
> > > 170M. Messages similar to "DEBUG 3: Using 7 of 7 pairs for
forecast
> > > filtering threshold >=0.254, observation filtering threshold
>=0.254,
> and
> > > field logic Union." are printed 1,326,213 times. I'm not sure
where
> > you're
> > > testing, but I wouldn't be surprised if this was a WCOSS issue.
I
> > haven't
> > > exactly been blown away by the write speeds on phase 3.
> > > I'm going to run today for more projections and elements. I'll
let you
> > > know if there are any more issues.
> > > Thanks
> > > John
> > >
> > > On Fri, Jun 12, 2020 at 7:42 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > John,
> > > >
> > > > You know, I think I should leave well-enough alone. But I'm
very
> > > surprised
> > > > about this change in runtime. I ran a short test case at
verbosity
> > level
> > > 0
> > > > and it took 20 seconds. At verbosity level 4, it took 22
seconds.
> More,
> > > but
> > > > not a crazy amount more.
> > > >
> > > > But I'm glad it's running in a more reasonable amount of time,
and
> hope
> > > it
> > > > continues to do so. I'll go ahead and resolve this ticket.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Jun 12, 2020 at 5:29 PM John Halley Gotway
<johnhg at ucar.edu>
> > > > wrote:
> > > >
> > > > > John,
> > > > >
> > > > > Oh wow, OK. Would you recommend that I change some of those
log
> > > messages
> > > > > to a higher verbosity level... like 4 or 5... to avoid slow
> runtimes?
> > > > >
> > > > > John
> > > > >
> > > > > On Fri, Jun 12, 2020 at 3:57 PM John L Wagner - NOAA Federal
via
> RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > > > >>
> > > > >> John
> > > > >> I believe I've confirmed that the slow run times were from
setting
> > -v
> > > 3
> > > > >> when calling series_analysis. My last test ran in about 25
> minutes.
> > > I
> > > > >> have a few more tests to run, but I believe all of my
issues are
> > > solved.
> > > > >> Thanks
> > > > >> John
> > > > >>
> > > > >> On Fri, Jun 12, 2020 at 7:39 AM John L Wagner - NOAA
Federal <
> > > > >> john.l.wagner at noaa.gov> wrote:
> > > > >>
> > > > >> > Here is the usage summary from the test I ran yesterday:
> > > > >> >
> > > > >> > [image: SerA Test1 Log.PNG]
> > > > >> > The job did not time out, and finished in 5 hr 43 min.
It did
> not
> > > use
> > > > >> all
> > > > >> > of the memory, which surprised me. When I was testing
> incorrectly
> > > > with
> > > > >> the
> > > > >> > thresholds defined within the field array, I had to keep
> > increasing
> > > > the
> > > > >> > memory setting in my bsub command to prevent OOM kills.
> > > > >> >
> > > > >> > I'm running a test now with just the lowest CNT
threshold. This
> > may
> > > > be
> > > > >> > what we end up running anyway and I'm hoping the run time
will
> > come
> > > > >> down so
> > > > >> > I can run more tests. I'll try to get through tests
today with
> > > > cutting
> > > > >> the
> > > > >> > block_size in half, using a mask, and different memory
settings
> > with
> > > > >> bsub.
> > > > >> > I'll let you know what I find.
> > > > >> >
> > > > >> > As for MPI, I agree with that solution. I'm not going to
> pretend
> > to
> > > > >> know
> > > > >> > what needs to be changed with C code and compilers. When
we
> > updated
> > > > the
> > > > >> > MOS-2000 Fortran code, we managed to get our ensemble
code run
> > time
> > > > down
> > > > >> > from 3 hours to 12 minutes. I realize that not all MET
users
> are
> > > > >> running
> > > > >> > on supercomputers that can take advantage of MPI, but
those of
> us
> > > that
> > > > >> are
> > > > >> > would certainly appreciate it.
> > > > >> >
> > > > >> > Thanks
> > > > >> > John
> > > > >> >
> > > > >> > On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT
<
> > > > >> > met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> >> Agreed, that's much too long! I suppose another
possibility is
> > that
> > > > >> it's
> > > > >> >> consuming too much memory and is using swap space, which
is
> slow.
> > > > >> >> If possible, please check what percent of memory the
process is
> > > > using.
> > > > >> If
> > > > >> >> it's up around 100%, then let's try cutting the block
size in
> > half
> > > > and
> > > > >> >> rerunning.
> > > > >> >>
> > > > >> >> Honestly, the real fix for this is enhancing Series-
Analysis
> with
> > > MPI
> > > > >> >> calls
> > > > >> >> to subset the 4+ million grid point into tiles, and then
do the
> > > > >> processing
> > > > >> >> for each tile on a separate node.
> > > > >> >> However, none of the MET tools are currently
parallelized. So
> > > that'd
> > > > >> >> obviously be a significant development effort.
> > > > >> >>
> > > > >> >> Another option, just to get something to actually
complete in
> the
> > > > time
> > > > >> >> window, would be applying a masking region to subset the
data
> > being
> > > > >> >> processed.
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >> John
> > > > >> >>
> > > > >> >> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA
Federal
> via
> > > RT <
> > > > >> >> met_help at ucar.edu> wrote:
> > > > >> >>
> > > > >> >> >
> > > > >> >> > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > >
> > > > >> >> >
> > > > >> >> > I set the block size to my grid size (4194304)
previously.
> Its
> > > > still
> > > > >> >> > running 5+ hours later. It will time out after 6
hours.
> > > > >> >> > I have been testing on WCOSS with Julie's MET V9.0.
Tomorrow
> > > I'll
> > > > >> run a
> > > > >> >> > test with V8.1. Otherwise, we'll need to rethink what
we
> want
> > to
> > > > >> show
> > > > >> >> in
> > > > >> >> > the viewer.
> > > > >> >> > Thanks
> > > > >> >> > John
> > > > >> >> >
> > > > >> >> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via
RT <
> > > > >> >> > met_help at ucar.edu>
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> > > John,
> > > > >> >> > >
> > > > >> >> > > I'd recommend modifying the "block_size" config file
option
> > and
> > > > >> >> > rerunning.
> > > > >> >> > > The default value is set very low, ensuring that you
don't
> > run
> > > > out
> > > > >> of
> > > > >> >> > > memory but requiring many passes through the data...
which
> is
> > > > slow.
> > > > >> >> > >
> > > > >> >> > > Here's an excerpt from this README file which
describes how
> > > that
> > > > >> >> works.
> > > > >> >> > >
> > > > >> >> > >
> > > > >>
> https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >> >> > > //
> > > > >> >> > > // SeriesAnalysisConfig_default
> > > > >> >> > > //
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >> >> > >
> > > > >> >> > > //
> > > > >> >> > > // Computation may be memory intensive, especially
for
> large
> > > > grids.
> > > > >> >> > > // The "block_size" entry sets the number of grid
points to
> > be
> > > > >> >> processed
> > > > >> >> > > // concurrently (i.e. in one pass through a time
series).
> > > Smaller
> > > > >> >> values
> > > > >> >> > > // require less memory but increase the number of
passes
> > > through
> > > > >> the
> > > > >> >> > data.
> > > > >> >> > > //
> > > > >> >> > > block_size = 1024;
> > > > >> >> > >
> > > > >> >> > > Look near the top of your log file for a message
that looks
> > > > >> something
> > > > >> >> > like
> > > > >> >> > > this:
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > *DEBUG 2: Computing statistics using a block size of
10000,
> > > > >> requiring
> > > > >> >> 3
> > > > >> >> > > pass(es) through the 169 x 154 grid.DEBUG 2:
Processing
> data
> > > pass
> > > > >> >> number
> > > > >> >> > 1
> > > > >> >> > > of 3 for grid points 1 to 10000.*
> > > > >> >> > >
> > > > >> >> > > This will tell you how many times it's reading
through the
> > > input
> > > > >> data
> > > > >> >> > > files. Ideally you'd set the block size high enough
so that
> > it
> > > > only
> > > > >> >> needs
> > > > >> >> > > to read them once. But that all depends on the
number of
> grid
> > > > >> points
> > > > >> >> and
> > > > >> >> > > length of your series. If you set the block_size too
high,
> it
> > > may
> > > > >> >> consume
> > > > >> >> > > all the memory.
> > > > >> >> > >
> > > > >> >> > > Hope that helps clarify.
> > > > >> >> > >
> > > > >> >> > > Thanks,
> > > > >> >> > > John
> > > > >> >> > >
> > > > >> >> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA
> Federal
> > > via
> > > > >> RT <
> > > > >> >> > > met_help at ucar.edu> wrote:
> > > > >> >> > >
> > > > >> >> > > >
> > > > >> >> > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > > > >
> > > > >> >> > > >
> > > > >> >> > > > Thanks John. I'm testing the change now.
> > > > >> >> > > > I'm testing 1 QPF06 projection for MAE, CSI, and
HSS for
> > > just 7
> > > > >> >> days of
> > > > >> >> > > > data. Its been running for over 2 hours now. Is
this
> the
> > > > >> expected
> > > > >> >> > > > behavior? I do have -v set to 3. Its writing
steadily
> to
> > > the
> > > > >> >> log. Is
> > > > >> >> > > > that slowing things down?
> > > > >> >> > > >
> > > > >> >> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley
Gotway via
> RT
> > <
> > > > >> >> > > > met_help at ucar.edu> wrote:
> > > > >> >> > > >
> > > > >> >> > > > > Hello John,
> > > > >> >> > > > >
> > > > >> >> > > > > I see you're having difficulty configuring
> > Series-Analysis
> > > to
> > > > >> >> apply
> > > > >> >> > the
> > > > >> >> > > > > continuous and categorical statistics you
defined.
> > > > >> >> > > > >
> > > > >> >> > > > > It's funny, this exact same issue arose via met-
help
> last
> > > > week,
> > > > >> >> and
> > > > >> >> > it
> > > > >> >> > > > > makes me think we should modify the code in some
way to
> > > make
> > > > it
> > > > >> >> more
> > > > >> >> > > > > intuitive. Please try editing your config file
by
> moving
> > > the
> > > > >> >> > > cnt_thresh
> > > > >> >> > > > > and cat_thresh settings up outside of the field
array,
> as
> > > > shown
> > > > >> >> > below:
> > > > >> >> > > > >
> > > > >> >> > > > > *fcst = {*
> > > > >> >> > > > >
> > > > >> >> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
> > > > >=2.54&&<6.35,
> > > > >> >> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
> > > >=25.4&&<76.2,
> > > > >> >> >=76.2
> > > > >> >> > > ];*
> > > > >> >> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35,
>=12.7,
> > > >=19.05,
> > > > >> >> >=25.4,
> > > > >> >> > > > > >=50.8, >=76.2 ];*
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > > > * field = [ { name = "${FIELD}";
> > > level
> > > > =
> > > > >> >> > > > "${LEVEL}";
> > > > >> >> > > > > } ];}obs = fcst;*
> > > > >> >> > > > >
> > > > >> >> > > > > When you rerun after this change, does it
produce the
> > > > expected
> > > > >> >> > output?
> > > > >> >> > > > >
> > > > >> >> > > > > I realize that this is confusing, especially if
you're
> > used
> > > > to
> > > > >> >> > setting
> > > > >> >> > > up
> > > > >> >> > > > > config files for Point-Stat and Grid-Stat. Those
tools
> > are
> > > > set
> > > > >> up
> > > > >> >> to
> > > > >> >> > > > > process multiple fields in a single run.
Therefore you
> > can
> > > > >> specify
> > > > >> >> > > > > cat_thresh and cnt_thresh separately for each
field
> array
> > > > >> entry.
> > > > >> >> > Unlike
> > > > >> >> > > > > Point-Stat and Grid-Stat, Series-Analysis can
only
> > process
> > > a
> > > > >> >> single
> > > > >> >> > > > series
> > > > >> >> > > > > of data in each run. There are multiple main
ways of
> > > > defining a
> > > > >> >> > series.
> > > > >> >> > > > > (1) By far, the most common way is specifying a
single
> > > field
> > > > >> that
> > > > >> >> is
> > > > >> >> > > > > extracted from a time-series of input data
files... as
> > > you're
> > > > >> >> doing.
> > > > >> >> > > > > (2) You can also define a series of field array
entries
> > > that
> > > > >> can
> > > > >> >> be
> > > > >> >> > > > > extracted from the same input file. For example,
> compute
> > > > stats
> > > > >> >> over
> > > > >> >> > > > > multiple vertical levels from the same input
file.
> > > > >> >> > > > > (3) You define N input fields to be extracted
from N
> > input
> > > > >> files.
> > > > >> >> > > > >
> > > > >> >> > > > > Because of options (2) and (3), the field entry
is
> > defined
> > > as
> > > > >> an
> > > > >> >> > array
> > > > >> >> > > > > rather than a single dictionary. In practice
though,
> the
> > > > field
> > > > >> >> array
> > > > >> >> > > > > typically has length 1.
> > > > >> >> > > > >
> > > > >> >> > > > > But the cat_thresh and cnt_thresh settings must
remain
> > > > constant
> > > > >> >> over
> > > > >> >> > > the
> > > > >> >> > > > > entire series, and that's why the code is setup
to
> parse
> > > them
> > > > >> from
> > > > >> >> > the
> > > > >> >> > > > > "fcst" and "obs" dictionaries, and not
separately from
> > each
> > > > >> field
> > > > >> >> > array
> > > > >> >> > > > > entry.
> > > > >> >> > > > >
> > > > >> >> > > > > Seems like it'd be a good idea to check for this
> > situation
> > > > and
> > > > >> >> print
> > > > >> >> > a
> > > > >> >> > > > > warning message for the user.
> > > > >> >> > > > >
> > > > >> >> > > > > Thanks,
> > > > >> >> > > > > John
> > > > >> >> > > > >
> > > > >> >> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT
<
> > > > >> >> met_help at ucar.edu>
> > > > >> >> > > > > wrote:
> > > > >> >> > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > > > <URL:
> > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > > > >> >> > > > > >
> > > > >> >> > > > > > Hello John,
> > > > >> >> > > > > >
> > > > >> >> > > > > > It looks like you need some assistance setting
up
> your
> > > > series
> > > > >> >> > > analysis
> > > > >> >> > > > in
> > > > >> >> > > > > > MET. I'm assigning this ticket to John Halley
> Gotway.
> > > > >> Please
> > > > >> >> > allow
> > > > >> >> > > a
> > > > >> >> > > > > few
> > > > >> >> > > > > > business days for a response.
> > > > >> >> > > > > >
> > > > >> >> > > > > > Regards,
> > > > >> >> > > > > > Minna
> > > > >> >> > > > > > ---------------
> > > > >> >> > > > > > Minna Win
> > > > >> >> > > > > > National Center for Atmospheric Research
> > > > >> >> > > > > > Developmental Testbed Center
> > > > >> >> > > > > > Phone: 303-497-8423
> > > > >> >> > > > > > Fax: 303-497-8401
> > > > >> >> > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner
- NOAA
> > > > Federal
> > > > >> via
> > > > >> >> > RT <
> > > > >> >> > > > > > met_help at ucar.edu> wrote:
> > > > >> >> > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was
acted
> > upon.
> > > > >> >> > > > > > > Transaction: Ticket created by
> > john.l.wagner at noaa.gov
> > > > >> >> > > > > > > Queue: met_help
> > > > >> >> > > > > > > Subject: Series Analysis Config File
Issues
> > > > >> >> > > > > > > Owner: Nobody
> > > > >> >> > > > > > > Requestors: john.l.wagner at noaa.gov
> > > > >> >> > > > > > > Status: new
> > > > >> >> > > > > > > Ticket <URL:
> > > > >> >> > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > > > >> >> > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > Good morning MET help
> > > > >> >> > > > > > > I'm having an issue with series_analysis
that I
> could
> > > use
> > > > >> your
> > > > >> >> > > > > assistance
> > > > >> >> > > > > > > with. I am currently testing
series_analysis for
> > QPF06
> > > > >> >> (config
> > > > >> >> > > file
> > > > >> >> > > > > > > attached). I am unable to get valid output
values
> > for
> > > > >> >> anything
> > > > >> >> > in
> > > > >> >> > > > the
> > > > >> >> > > > > > CTS
> > > > >> >> > > > > > > and MCTS categories. The TOTAL grid for each
score
> > > group
> > > > >> looks
> > > > >> >> > good
> > > > >> >> > > > > (I'm
> > > > >> >> > > > > > > currently testing with 7 days of data). The
CSI
> > values
> > > > are
> > > > >> >> all
> > > > >> >> > > > > constant
> > > > >> >> > > > > > at
> > > > >> >> > > > > > > 1. I am unable to get any values for HSS,
either
> > from
> > > > CTS
> > > > >> or
> > > > >> >> > MCTS.
> > > > >> >> > > > > All
> > > > >> >> > > > > > > output (MAE, ME, RMSE) from the CNT score
group
> seem
> > to
> > > > >> work
> > > > >> >> > fine.
> > > > >> >> > > > > > > I'm assuming the issue is setting the
thresholds in
> > my
> > > > >> config
> > > > >> >> > file.
> > > > >> >> > > > > > > The scores that I see in my netCDF file for
the CTS
> > > > scores
> > > > >> all
> > > > >> >> > end
> > > > >> >> > > > with
> > > > >> >> > > > > > > "_NA", where I'm assuming a category should
be set.
> > I
> > > am
> > > > >> >> using
> > > > >> >> > the
> > > > >> >> > > > > same
> > > > >> >> > > > > > > thresholds that I use in grid_stat. Should
I be
> > > setting
> > > > >> them
> > > > >> >> up
> > > > >> >> > > > > > > differently for series_analysis? Is there a
sample
> > > > config
> > > > >> >> file
> > > > >> >> > > that
> > > > >> >> > > > I
> > > > >> >> > > > > > > could look at for reference?
> > > > >> >> > > > > > > If it helps, my test output file can be
found on
> > WCOSS
> > > > >> >> > > > > > > here:
> > > > >> >> > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > Thanks
> > > > >> >> > > > > > > John
> > > > >> >> > > > > > > --
> > > > >> >> > > > > > > John Wagner
> > > > >> >> > > > > > > Verification Task Lead
> > > > >> >> > > > > > > NOAA/National Weather Service
> > > > >> >> > > > > > > Meteorological Development Laboratory
> > > > >> >> > > > > > > Digital Forecast Services Division
> > > > >> >> > > > > > > SSMC2 Room 10106
> > > > >> >> > > > > > > Silver Spring, MD 20910
> > > > >> >> > > > > > > (301) 427-9471 (office)
> > > > >> >> > > > > > > (908) 902-4155 (cell/text)
> > > > >> >> > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > > > --
> > > > >> >> > > > John Wagner
> > > > >> >> > > > Verification Task Lead
> > > > >> >> > > > NOAA/National Weather Service
> > > > >> >> > > > Meteorological Development Laboratory
> > > > >> >> > > > Digital Forecast Services Division
> > > > >> >> > > > SSMC2 Room 10106
> > > > >> >> > > > Silver Spring, MD 20910
> > > > >> >> > > > (301) 427-9471 (office)
> > > > >> >> > > > (908) 902-4155 (cell/text)
> > > > >> >> > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >> > --
> > > > >> >> > John Wagner
> > > > >> >> > Verification Task Lead
> > > > >> >> > NOAA/National Weather Service
> > > > >> >> > Meteorological Development Laboratory
> > > > >> >> > Digital Forecast Services Division
> > > > >> >> > SSMC2 Room 10106
> > > > >> >> > Silver Spring, MD 20910
> > > > >> >> > (301) 427-9471 (office)
> > > > >> >> > (908) 902-4155 (cell/text)
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >> >>
> > > > >> >
> > > > >> > --
> > > > >> > John Wagner
> > > > >> > Verification Task Lead
> > > > >> > NOAA/National Weather Service
> > > > >> > Meteorological Development Laboratory
> > > > >> > Digital Forecast Services Division
> > > > >> > SSMC2 Room 10106
> > > > >> > Silver Spring, MD 20910
> > > > >> > (301) 427-9471 (office)
> > > > >> > (908) 902-4155 (cell/text)
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >> John Wagner
> > > > >> Verification Task Lead
> > > > >> NOAA/National Weather Service
> > > > >> Meteorological Development Laboratory
> > > > >> Digital Forecast Services Division
> > > > >> SSMC2 Room 10106
> > > > >> Silver Spring, MD 20910
> > > > >> (301) 427-9471 (office)
> > > > >> (908) 902-4155 (cell/text)
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Division
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> > >
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
More information about the Met_help
mailing list