[Met_help] [rt.rap.ucar.edu #95583] History for Series Analysis Config File Issues
John Halley Gotway via RT
met_help at ucar.edu
Fri Jun 12 17:43:29 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning MET help
I'm having an issue with series_analysis that I could use your assistance
with. I am currently testing series_analysis for QPF06 (config file
attached). I am unable to get valid output values for anything in the CTS
and MCTS categories. The TOTAL grid for each score group looks good (I'm
currently testing with 7 days of data). The CSI values are all constant at
1. I am unable to get any values for HSS, either from CTS or MCTS. All
output (MAE, ME, RMSE) from the CNT score group seem to work fine.
I'm assuming the issue is setting the thresholds in my config file.
The scores that I see in my netCDF file for the CTS scores all end with
"_NA", where I'm assuming a category should be set. I am using the same
thresholds that I use in grid_stat. Should I be setting them up
differently for series_analysis? Is there a sample config file that I
could look at for reference?
If it helps, my test output file can be found on WCOSS
here: /gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
Thanks
John
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Series Analysis Config File Issues
From: Minna Win
Time: Thu Jun 11 08:58:30 2020
Hello John,
It looks like you need some assistance setting up your series analysis
in
MET. I'm assigning this ticket to John Halley Gotway. Please allow a
few
business days for a response.
Regards,
Minna
---------------
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax: 303-497-8401
On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> Transaction: Ticket created by john.l.wagner at noaa.gov
> Queue: met_help
> Subject: Series Analysis Config File Issues
> Owner: Nobody
> Requestors: john.l.wagner at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
>
> Good morning MET help
> I'm having an issue with series_analysis that I could use your
assistance
> with. I am currently testing series_analysis for QPF06 (config file
> attached). I am unable to get valid output values for anything in
the CTS
> and MCTS categories. The TOTAL grid for each score group looks good
(I'm
> currently testing with 7 days of data). The CSI values are all
constant at
> 1. I am unable to get any values for HSS, either from CTS or MCTS.
All
> output (MAE, ME, RMSE) from the CNT score group seem to work fine.
> I'm assuming the issue is setting the thresholds in my config file.
> The scores that I see in my netCDF file for the CTS scores all end
with
> "_NA", where I'm assuming a category should be set. I am using the
same
> thresholds that I use in grid_stat. Should I be setting them up
> differently for series_analysis? Is there a sample config file that
I
> could look at for reference?
> If it helps, my test output file can be found on WCOSS
> here:
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
>
> Thanks
> John
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Thu Jun 11 09:59:17 2020
Hello John,
I see you're having difficulty configuring Series-Analysis to apply
the
continuous and categorical statistics you defined.
It's funny, this exact same issue arose via met-help last week, and it
makes me think we should modify the code in some way to make it more
intuitive. Please try editing your config file by moving the
cnt_thresh
and cat_thresh settings up outside of the field array, as shown below:
*fcst = {*
* cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
>=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4, >=25.4&&<76.2, >=76.2
];*
* cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05, >=25.4,
>=50.8, >=76.2 ];*
* field = [ { name = "${FIELD}"; level =
"${LEVEL}";
} ];}obs = fcst;*
When you rerun after this change, does it produce the expected output?
I realize that this is confusing, especially if you're used to setting
up
config files for Point-Stat and Grid-Stat. Those tools are set up to
process multiple fields in a single run. Therefore you can specify
cat_thresh and cnt_thresh separately for each field array entry.
Unlike
Point-Stat and Grid-Stat, Series-Analysis can only process a single
series
of data in each run. There are multiple main ways of defining a
series.
(1) By far, the most common way is specifying a single field that is
extracted from a time-series of input data files... as you're doing.
(2) You can also define a series of field array entries that can be
extracted from the same input file. For example, compute stats over
multiple vertical levels from the same input file.
(3) You define N input fields to be extracted from N input files.
Because of options (2) and (3), the field entry is defined as an array
rather than a single dictionary. In practice though, the field array
typically has length 1.
But the cat_thresh and cnt_thresh settings must remain constant over
the
entire series, and that's why the code is setup to parse them from the
"fcst" and "obs" dictionaries, and not separately from each field
array
entry.
Seems like it'd be a good idea to check for this situation and print a
warning message for the user.
Thanks,
John
On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
> Hello John,
>
> It looks like you need some assistance setting up your series
analysis in
> MET. I'm assigning this ticket to John Halley Gotway. Please allow
a few
> business days for a response.
>
> Regards,
> Minna
> ---------------
> Minna Win
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax: 303-497-8401
>
>
>
> On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > Transaction: Ticket created by john.l.wagner at noaa.gov
> > Queue: met_help
> > Subject: Series Analysis Config File Issues
> > Owner: Nobody
> > Requestors: john.l.wagner at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >
> >
> > Good morning MET help
> > I'm having an issue with series_analysis that I could use your
assistance
> > with. I am currently testing series_analysis for QPF06 (config
file
> > attached). I am unable to get valid output values for anything in
the
> CTS
> > and MCTS categories. The TOTAL grid for each score group looks
good (I'm
> > currently testing with 7 days of data). The CSI values are all
constant
> at
> > 1. I am unable to get any values for HSS, either from CTS or
MCTS. All
> > output (MAE, ME, RMSE) from the CNT score group seem to work fine.
> > I'm assuming the issue is setting the thresholds in my config
file.
> > The scores that I see in my netCDF file for the CTS scores all end
with
> > "_NA", where I'm assuming a category should be set. I am using
the same
> > thresholds that I use in grid_stat. Should I be setting them up
> > differently for series_analysis? Is there a sample config file
that I
> > could look at for reference?
> > If it helps, my test output file can be found on WCOSS
> > here:
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> >
> > Thanks
> > John
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John L Wagner - NOAA Federal
Time: Thu Jun 11 13:39:51 2020
Thanks John. I'm testing the change now.
I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just 7 days
of
data. Its been running for over 2 hours now. Is this the expected
behavior? I do have -v set to 3. Its writing steadily to the log.
Is
that slowing things down?
On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hello John,
>
> I see you're having difficulty configuring Series-Analysis to apply
the
> continuous and categorical statistics you defined.
>
> It's funny, this exact same issue arose via met-help last week, and
it
> makes me think we should modify the code in some way to make it more
> intuitive. Please try editing your config file by moving the
cnt_thresh
> and cat_thresh settings up outside of the field array, as shown
below:
>
> *fcst = {*
>
> * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
> >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4, >=25.4&&<76.2, >=76.2
];*
> * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05, >=25.4,
> >=50.8, >=76.2 ];*
>
>
>
>
>
>
>
>
> * field = [ { name = "${FIELD}"; level =
"${LEVEL}";
> } ];}obs = fcst;*
>
> When you rerun after this change, does it produce the expected
output?
>
> I realize that this is confusing, especially if you're used to
setting up
> config files for Point-Stat and Grid-Stat. Those tools are set up to
> process multiple fields in a single run. Therefore you can specify
> cat_thresh and cnt_thresh separately for each field array entry.
Unlike
> Point-Stat and Grid-Stat, Series-Analysis can only process a single
series
> of data in each run. There are multiple main ways of defining a
series.
> (1) By far, the most common way is specifying a single field that is
> extracted from a time-series of input data files... as you're doing.
> (2) You can also define a series of field array entries that can be
> extracted from the same input file. For example, compute stats over
> multiple vertical levels from the same input file.
> (3) You define N input fields to be extracted from N input files.
>
> Because of options (2) and (3), the field entry is defined as an
array
> rather than a single dictionary. In practice though, the field array
> typically has length 1.
>
> But the cat_thresh and cnt_thresh settings must remain constant over
the
> entire series, and that's why the code is setup to parse them from
the
> "fcst" and "obs" dictionaries, and not separately from each field
array
> entry.
>
> Seems like it'd be a good idea to check for this situation and print
a
> warning message for the user.
>
> Thanks,
> John
>
> On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >
> > Hello John,
> >
> > It looks like you need some assistance setting up your series
analysis in
> > MET. I'm assigning this ticket to John Halley Gotway. Please
allow a
> few
> > business days for a response.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax: 303-497-8401
> >
> >
> >
> > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > Queue: met_help
> > > Subject: Series Analysis Config File Issues
> > > Owner: Nobody
> > > Requestors: john.l.wagner at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> >
> > >
> > >
> > > Good morning MET help
> > > I'm having an issue with series_analysis that I could use your
> assistance
> > > with. I am currently testing series_analysis for QPF06 (config
file
> > > attached). I am unable to get valid output values for anything
in the
> > CTS
> > > and MCTS categories. The TOTAL grid for each score group looks
good
> (I'm
> > > currently testing with 7 days of data). The CSI values are all
> constant
> > at
> > > 1. I am unable to get any values for HSS, either from CTS or
MCTS.
> All
> > > output (MAE, ME, RMSE) from the CNT score group seem to work
fine.
> > > I'm assuming the issue is setting the thresholds in my config
file.
> > > The scores that I see in my netCDF file for the CTS scores all
end with
> > > "_NA", where I'm assuming a category should be set. I am using
the
> same
> > > thresholds that I use in grid_stat. Should I be setting them up
> > > differently for series_analysis? Is there a sample config file
that I
> > > could look at for reference?
> > > If it helps, my test output file can be found on WCOSS
> > > here:
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > >
> > > Thanks
> > > John
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Division
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> > >
> >
> >
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Thu Jun 11 15:24:06 2020
John,
I'd recommend modifying the "block_size" config file option and
rerunning.
The default value is set very low, ensuring that you don't run out of
memory but requiring many passes through the data... which is slow.
Here's an excerpt from this README file which describes how that
works.
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
////////////////////////////////////////////////////////////////////////////////
//
// SeriesAnalysisConfig_default
//
////////////////////////////////////////////////////////////////////////////////
//
// Computation may be memory intensive, especially for large grids.
// The "block_size" entry sets the number of grid points to be
processed
// concurrently (i.e. in one pass through a time series). Smaller
values
// require less memory but increase the number of passes through the
data.
//
block_size = 1024;
Look near the top of your log file for a message that looks something
like
this:
*DEBUG 2: Computing statistics using a block size of 10000, requiring
3
pass(es) through the 169 x 154 grid.DEBUG 2: Processing data pass
number 1
of 3 for grid points 1 to 10000.*
This will tell you how many times it's reading through the input data
files. Ideally you'd set the block size high enough so that it only
needs
to read them once. But that all depends on the number of grid points
and
length of your series. If you set the block_size too high, it may
consume
all the memory.
Hope that helps clarify.
Thanks,
John
On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
> Thanks John. I'm testing the change now.
> I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just 7 days
of
> data. Its been running for over 2 hours now. Is this the expected
> behavior? I do have -v set to 3. Its writing steadily to the log.
Is
> that slowing things down?
>
> On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Hello John,
> >
> > I see you're having difficulty configuring Series-Analysis to
apply the
> > continuous and categorical statistics you defined.
> >
> > It's funny, this exact same issue arose via met-help last week,
and it
> > makes me think we should modify the code in some way to make it
more
> > intuitive. Please try editing your config file by moving the
cnt_thresh
> > and cat_thresh settings up outside of the field array, as shown
below:
> >
> > *fcst = {*
> >
> > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
> > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4, >=25.4&&<76.2,
>=76.2 ];*
> > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05,
>=25.4,
> > >=50.8, >=76.2 ];*
> >
> >
> >
> >
> >
> >
> >
> >
> > * field = [ { name = "${FIELD}"; level =
> "${LEVEL}";
> > } ];}obs = fcst;*
> >
> > When you rerun after this change, does it produce the expected
output?
> >
> > I realize that this is confusing, especially if you're used to
setting up
> > config files for Point-Stat and Grid-Stat. Those tools are set up
to
> > process multiple fields in a single run. Therefore you can specify
> > cat_thresh and cnt_thresh separately for each field array entry.
Unlike
> > Point-Stat and Grid-Stat, Series-Analysis can only process a
single
> series
> > of data in each run. There are multiple main ways of defining a
series.
> > (1) By far, the most common way is specifying a single field that
is
> > extracted from a time-series of input data files... as you're
doing.
> > (2) You can also define a series of field array entries that can
be
> > extracted from the same input file. For example, compute stats
over
> > multiple vertical levels from the same input file.
> > (3) You define N input fields to be extracted from N input files.
> >
> > Because of options (2) and (3), the field entry is defined as an
array
> > rather than a single dictionary. In practice though, the field
array
> > typically has length 1.
> >
> > But the cat_thresh and cnt_thresh settings must remain constant
over the
> > entire series, and that's why the code is setup to parse them from
the
> > "fcst" and "obs" dictionaries, and not separately from each field
array
> > entry.
> >
> > Seems like it'd be a good idea to check for this situation and
print a
> > warning message for the user.
> >
> > Thanks,
> > John
> >
> > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > >
> > > Hello John,
> > >
> > > It looks like you need some assistance setting up your series
analysis
> in
> > > MET. I'm assigning this ticket to John Halley Gotway. Please
allow a
> > few
> > > business days for a response.
> > >
> > > Regards,
> > > Minna
> > > ---------------
> > > Minna Win
> > > National Center for Atmospheric Research
> > > Developmental Testbed Center
> > > Phone: 303-497-8423
> > > Fax: 303-497-8401
> > >
> > >
> > >
> > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > Queue: met_help
> > > > Subject: Series Analysis Config File Issues
> > > > Owner: Nobody
> > > > Requestors: john.l.wagner at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > >
> > > >
> > > >
> > > > Good morning MET help
> > > > I'm having an issue with series_analysis that I could use your
> > assistance
> > > > with. I am currently testing series_analysis for QPF06
(config file
> > > > attached). I am unable to get valid output values for
anything in
> the
> > > CTS
> > > > and MCTS categories. The TOTAL grid for each score group looks
good
> > (I'm
> > > > currently testing with 7 days of data). The CSI values are
all
> > constant
> > > at
> > > > 1. I am unable to get any values for HSS, either from CTS or
MCTS.
> > All
> > > > output (MAE, ME, RMSE) from the CNT score group seem to work
fine.
> > > > I'm assuming the issue is setting the thresholds in my config
file.
> > > > The scores that I see in my netCDF file for the CTS scores all
end
> with
> > > > "_NA", where I'm assuming a category should be set. I am
using the
> > same
> > > > thresholds that I use in grid_stat. Should I be setting them
up
> > > > differently for series_analysis? Is there a sample config
file that
> I
> > > > could look at for reference?
> > > > If it helps, my test output file can be found on WCOSS
> > > > here:
> > > >
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > > >
> > > > Thanks
> > > > John
> > > > --
> > > > John Wagner
> > > > Verification Task Lead
> > > > NOAA/National Weather Service
> > > > Meteorological Development Laboratory
> > > > Digital Forecast Services Division
> > > > SSMC2 Room 10106
> > > > Silver Spring, MD 20910
> > > > (301) 427-9471 (office)
> > > > (908) 902-4155 (cell/text)
> > > >
> > > >
> > >
> > >
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John L Wagner - NOAA Federal
Time: Thu Jun 11 16:46:54 2020
I set the block size to my grid size (4194304) previously. Its still
running 5+ hours later. It will time out after 6 hours.
I have been testing on WCOSS with Julie's MET V9.0. Tomorrow I'll run
a
test with V8.1. Otherwise, we'll need to rethink what we want to show
in
the viewer.
Thanks
John
On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> John,
>
> I'd recommend modifying the "block_size" config file option and
rerunning.
> The default value is set very low, ensuring that you don't run out
of
> memory but requiring many passes through the data... which is slow.
>
> Here's an excerpt from this README file which describes how that
works.
>
> https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
>
>
>
////////////////////////////////////////////////////////////////////////////////
> //
> // SeriesAnalysisConfig_default
> //
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Computation may be memory intensive, especially for large grids.
> // The "block_size" entry sets the number of grid points to be
processed
> // concurrently (i.e. in one pass through a time series). Smaller
values
> // require less memory but increase the number of passes through the
data.
> //
> block_size = 1024;
>
> Look near the top of your log file for a message that looks
something like
> this:
>
>
> *DEBUG 2: Computing statistics using a block size of 10000,
requiring 3
> pass(es) through the 169 x 154 grid.DEBUG 2: Processing data pass
number 1
> of 3 for grid points 1 to 10000.*
>
> This will tell you how many times it's reading through the input
data
> files. Ideally you'd set the block size high enough so that it only
needs
> to read them once. But that all depends on the number of grid points
and
> length of your series. If you set the block_size too high, it may
consume
> all the memory.
>
> Hope that helps clarify.
>
> Thanks,
> John
>
> On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >
> > Thanks John. I'm testing the change now.
> > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just 7
days of
> > data. Its been running for over 2 hours now. Is this the
expected
> > behavior? I do have -v set to 3. Its writing steadily to the
log. Is
> > that slowing things down?
> >
> > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Hello John,
> > >
> > > I see you're having difficulty configuring Series-Analysis to
apply the
> > > continuous and categorical statistics you defined.
> > >
> > > It's funny, this exact same issue arose via met-help last week,
and it
> > > makes me think we should modify the code in some way to make it
more
> > > intuitive. Please try editing your config file by moving the
> cnt_thresh
> > > and cat_thresh settings up outside of the field array, as shown
below:
> > >
> > > *fcst = {*
> > >
> > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
> > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4, >=25.4&&<76.2,
>=76.2
> ];*
> > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05,
>=25.4,
> > > >=50.8, >=76.2 ];*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > * field = [ { name = "${FIELD}"; level =
> > "${LEVEL}";
> > > } ];}obs = fcst;*
> > >
> > > When you rerun after this change, does it produce the expected
output?
> > >
> > > I realize that this is confusing, especially if you're used to
setting
> up
> > > config files for Point-Stat and Grid-Stat. Those tools are set
up to
> > > process multiple fields in a single run. Therefore you can
specify
> > > cat_thresh and cnt_thresh separately for each field array entry.
Unlike
> > > Point-Stat and Grid-Stat, Series-Analysis can only process a
single
> > series
> > > of data in each run. There are multiple main ways of defining a
series.
> > > (1) By far, the most common way is specifying a single field
that is
> > > extracted from a time-series of input data files... as you're
doing.
> > > (2) You can also define a series of field array entries that can
be
> > > extracted from the same input file. For example, compute stats
over
> > > multiple vertical levels from the same input file.
> > > (3) You define N input fields to be extracted from N input
files.
> > >
> > > Because of options (2) and (3), the field entry is defined as an
array
> > > rather than a single dictionary. In practice though, the field
array
> > > typically has length 1.
> > >
> > > But the cat_thresh and cnt_thresh settings must remain constant
over
> the
> > > entire series, and that's why the code is setup to parse them
from the
> > > "fcst" and "obs" dictionaries, and not separately from each
field array
> > > entry.
> > >
> > > Seems like it'd be a good idea to check for this situation and
print a
> > > warning message for the user.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>
> > > >
> > > > Hello John,
> > > >
> > > > It looks like you need some assistance setting up your series
> analysis
> > in
> > > > MET. I'm assigning this ticket to John Halley Gotway. Please
allow
> a
> > > few
> > > > business days for a response.
> > > >
> > > > Regards,
> > > > Minna
> > > > ---------------
> > > > Minna Win
> > > > National Center for Atmospheric Research
> > > > Developmental Testbed Center
> > > > Phone: 303-497-8423
> > > > Fax: 303-497-8401
> > > >
> > > >
> > > >
> > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal
via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: Series Analysis Config File Issues
> > > > > Owner: Nobody
> > > > > Requestors: john.l.wagner at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > > >
> > > > >
> > > > >
> > > > > Good morning MET help
> > > > > I'm having an issue with series_analysis that I could use
your
> > > assistance
> > > > > with. I am currently testing series_analysis for QPF06
(config
> file
> > > > > attached). I am unable to get valid output values for
anything in
> > the
> > > > CTS
> > > > > and MCTS categories. The TOTAL grid for each score group
looks good
> > > (I'm
> > > > > currently testing with 7 days of data). The CSI values are
all
> > > constant
> > > > at
> > > > > 1. I am unable to get any values for HSS, either from CTS
or MCTS.
> > > All
> > > > > output (MAE, ME, RMSE) from the CNT score group seem to work
fine.
> > > > > I'm assuming the issue is setting the thresholds in my
config file.
> > > > > The scores that I see in my netCDF file for the CTS scores
all end
> > with
> > > > > "_NA", where I'm assuming a category should be set. I am
using the
> > > same
> > > > > thresholds that I use in grid_stat. Should I be setting
them up
> > > > > differently for series_analysis? Is there a sample config
file
> that
> > I
> > > > > could look at for reference?
> > > > > If it helps, my test output file can be found on WCOSS
> > > > > here:
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > > > >
> > > > > Thanks
> > > > > John
> > > > > --
> > > > > John Wagner
> > > > > Verification Task Lead
> > > > > NOAA/National Weather Service
> > > > > Meteorological Development Laboratory
> > > > > Digital Forecast Services Division
> > > > > SSMC2 Room 10106
> > > > > Silver Spring, MD 20910
> > > > > (301) 427-9471 (office)
> > > > > (908) 902-4155 (cell/text)
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Thu Jun 11 17:07:58 2020
Agreed, that's much too long! I suppose another possibility is that
it's
consuming too much memory and is using swap space, which is slow.
If possible, please check what percent of memory the process is using.
If
it's up around 100%, then let's try cutting the block size in half and
rerunning.
Honestly, the real fix for this is enhancing Series-Analysis with MPI
calls
to subset the 4+ million grid point into tiles, and then do the
processing
for each tile on a separate node.
However, none of the MET tools are currently parallelized. So that'd
obviously be a significant development effort.
Another option, just to get something to actually complete in the time
window, would be applying a masking region to subset the data being
processed.
Thanks,
John
On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
> I set the block size to my grid size (4194304) previously. Its
still
> running 5+ hours later. It will time out after 6 hours.
> I have been testing on WCOSS with Julie's MET V9.0. Tomorrow I'll
run a
> test with V8.1. Otherwise, we'll need to rethink what we want to
show in
> the viewer.
> Thanks
> John
>
> On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > John,
> >
> > I'd recommend modifying the "block_size" config file option and
> rerunning.
> > The default value is set very low, ensuring that you don't run out
of
> > memory but requiring many passes through the data... which is
slow.
> >
> > Here's an excerpt from this README file which describes how that
works.
> >
> >
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > //
> > // SeriesAnalysisConfig_default
> > //
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Computation may be memory intensive, especially for large
grids.
> > // The "block_size" entry sets the number of grid points to be
processed
> > // concurrently (i.e. in one pass through a time series). Smaller
values
> > // require less memory but increase the number of passes through
the
> data.
> > //
> > block_size = 1024;
> >
> > Look near the top of your log file for a message that looks
something
> like
> > this:
> >
> >
> > *DEBUG 2: Computing statistics using a block size of 10000,
requiring 3
> > pass(es) through the 169 x 154 grid.DEBUG 2: Processing data pass
number
> 1
> > of 3 for grid points 1 to 10000.*
> >
> > This will tell you how many times it's reading through the input
data
> > files. Ideally you'd set the block size high enough so that it
only needs
> > to read them once. But that all depends on the number of grid
points and
> > length of your series. If you set the block_size too high, it may
consume
> > all the memory.
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John
> >
> > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > >
> > > Thanks John. I'm testing the change now.
> > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just 7
days of
> > > data. Its been running for over 2 hours now. Is this the
expected
> > > behavior? I do have -v set to 3. Its writing steadily to the
log. Is
> > > that slowing things down?
> > >
> > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Hello John,
> > > >
> > > > I see you're having difficulty configuring Series-Analysis to
apply
> the
> > > > continuous and categorical statistics you defined.
> > > >
> > > > It's funny, this exact same issue arose via met-help last
week, and
> it
> > > > makes me think we should modify the code in some way to make
it more
> > > > intuitive. Please try editing your config file by moving the
> > cnt_thresh
> > > > and cat_thresh settings up outside of the field array, as
shown
> below:
> > > >
> > > > *fcst = {*
> > > >
> > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54, >=2.54&&<6.35,
> > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4, >=25.4&&<76.2,
>=76.2
> > ];*
> > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05,
>=25.4,
> > > > >=50.8, >=76.2 ];*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > * field = [ { name = "${FIELD}"; level =
> > > "${LEVEL}";
> > > > } ];}obs = fcst;*
> > > >
> > > > When you rerun after this change, does it produce the expected
> output?
> > > >
> > > > I realize that this is confusing, especially if you're used to
> setting
> > up
> > > > config files for Point-Stat and Grid-Stat. Those tools are set
up to
> > > > process multiple fields in a single run. Therefore you can
specify
> > > > cat_thresh and cnt_thresh separately for each field array
entry.
> Unlike
> > > > Point-Stat and Grid-Stat, Series-Analysis can only process a
single
> > > series
> > > > of data in each run. There are multiple main ways of defining
a
> series.
> > > > (1) By far, the most common way is specifying a single field
that is
> > > > extracted from a time-series of input data files... as you're
doing.
> > > > (2) You can also define a series of field array entries that
can be
> > > > extracted from the same input file. For example, compute stats
over
> > > > multiple vertical levels from the same input file.
> > > > (3) You define N input fields to be extracted from N input
files.
> > > >
> > > > Because of options (2) and (3), the field entry is defined as
an
> array
> > > > rather than a single dictionary. In practice though, the field
array
> > > > typically has length 1.
> > > >
> > > > But the cat_thresh and cnt_thresh settings must remain
constant over
> > the
> > > > entire series, and that's why the code is setup to parse them
from
> the
> > > > "fcst" and "obs" dictionaries, and not separately from each
field
> array
> > > > entry.
> > > >
> > > > Seems like it'd be a good idea to check for this situation and
print
> a
> > > > warning message for the user.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT
<met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > > > >
> > > > > Hello John,
> > > > >
> > > > > It looks like you need some assistance setting up your
series
> > analysis
> > > in
> > > > > MET. I'm assigning this ticket to John Halley Gotway.
Please
> allow
> > a
> > > > few
> > > > > business days for a response.
> > > > >
> > > > > Regards,
> > > > > Minna
> > > > > ---------------
> > > > > Minna Win
> > > > > National Center for Atmospheric Research
> > > > > Developmental Testbed Center
> > > > > Phone: 303-497-8423
> > > > > Fax: 303-497-8401
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA Federal
via
> RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: Series Analysis Config File Issues
> > > > > > Owner: Nobody
> > > > > > Requestors: john.l.wagner at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > > > >
> > > > > >
> > > > > >
> > > > > > Good morning MET help
> > > > > > I'm having an issue with series_analysis that I could use
your
> > > > assistance
> > > > > > with. I am currently testing series_analysis for QPF06
(config
> > file
> > > > > > attached). I am unable to get valid output values for
anything
> in
> > > the
> > > > > CTS
> > > > > > and MCTS categories. The TOTAL grid for each score group
looks
> good
> > > > (I'm
> > > > > > currently testing with 7 days of data). The CSI values
are all
> > > > constant
> > > > > at
> > > > > > 1. I am unable to get any values for HSS, either from CTS
or
> MCTS.
> > > > All
> > > > > > output (MAE, ME, RMSE) from the CNT score group seem to
work
> fine.
> > > > > > I'm assuming the issue is setting the thresholds in my
config
> file.
> > > > > > The scores that I see in my netCDF file for the CTS scores
all
> end
> > > with
> > > > > > "_NA", where I'm assuming a category should be set. I am
using
> the
> > > > same
> > > > > > thresholds that I use in grid_stat. Should I be setting
them up
> > > > > > differently for series_analysis? Is there a sample config
file
> > that
> > > I
> > > > > > could look at for reference?
> > > > > > If it helps, my test output file can be found on WCOSS
> > > > > > here:
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > > > > >
> > > > > > Thanks
> > > > > > John
> > > > > > --
> > > > > > John Wagner
> > > > > > Verification Task Lead
> > > > > > NOAA/National Weather Service
> > > > > > Meteorological Development Laboratory
> > > > > > Digital Forecast Services Division
> > > > > > SSMC2 Room 10106
> > > > > > Silver Spring, MD 20910
> > > > > > (301) 427-9471 (office)
> > > > > > (908) 902-4155 (cell/text)
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Division
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> > >
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John L Wagner - NOAA Federal
Time: Fri Jun 12 05:39:39 2020
Here is the usage summary from the test I ran yesterday:
[image: SerA Test1 Log.PNG]
The job did not time out, and finished in 5 hr 43 min. It did not use
all
of the memory, which surprised me. When I was testing incorrectly
with the
thresholds defined within the field array, I had to keep increasing
the
memory setting in my bsub command to prevent OOM kills.
I'm running a test now with just the lowest CNT threshold. This may
be
what we end up running anyway and I'm hoping the run time will come
down so
I can run more tests. I'll try to get through tests today with
cutting the
block_size in half, using a mask, and different memory settings with
bsub.
I'll let you know what I find.
As for MPI, I agree with that solution. I'm not going to pretend to
know
what needs to be changed with C code and compilers. When we updated
the
MOS-2000 Fortran code, we managed to get our ensemble code run time
down
from 3 hours to 12 minutes. I realize that not all MET users are
running
on supercomputers that can take advantage of MPI, but those of us that
are
would certainly appreciate it.
Thanks
John
On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Agreed, that's much too long! I suppose another possibility is that
it's
> consuming too much memory and is using swap space, which is slow.
> If possible, please check what percent of memory the process is
using. If
> it's up around 100%, then let's try cutting the block size in half
and
> rerunning.
>
> Honestly, the real fix for this is enhancing Series-Analysis with
MPI calls
> to subset the 4+ million grid point into tiles, and then do the
processing
> for each tile on a separate node.
> However, none of the MET tools are currently parallelized. So that'd
> obviously be a significant development effort.
>
> Another option, just to get something to actually complete in the
time
> window, would be applying a masking region to subset the data being
> processed.
>
> Thanks,
> John
>
> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >
> > I set the block size to my grid size (4194304) previously. Its
still
> > running 5+ hours later. It will time out after 6 hours.
> > I have been testing on WCOSS with Julie's MET V9.0. Tomorrow I'll
run a
> > test with V8.1. Otherwise, we'll need to rethink what we want to
show in
> > the viewer.
> > Thanks
> > John
> >
> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > John,
> > >
> > > I'd recommend modifying the "block_size" config file option and
> > rerunning.
> > > The default value is set very low, ensuring that you don't run
out of
> > > memory but requiring many passes through the data... which is
slow.
> > >
> > > Here's an excerpt from this README file which describes how that
works.
> > >
> > >
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > //
> > > // SeriesAnalysisConfig_default
> > > //
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Computation may be memory intensive, especially for large
grids.
> > > // The "block_size" entry sets the number of grid points to be
> processed
> > > // concurrently (i.e. in one pass through a time series).
Smaller
> values
> > > // require less memory but increase the number of passes through
the
> > data.
> > > //
> > > block_size = 1024;
> > >
> > > Look near the top of your log file for a message that looks
something
> > like
> > > this:
> > >
> > >
> > > *DEBUG 2: Computing statistics using a block size of 10000,
requiring 3
> > > pass(es) through the 169 x 154 grid.DEBUG 2: Processing data
pass
> number
> > 1
> > > of 3 for grid points 1 to 10000.*
> > >
> > > This will tell you how many times it's reading through the input
data
> > > files. Ideally you'd set the block size high enough so that it
only
> needs
> > > to read them once. But that all depends on the number of grid
points
> and
> > > length of your series. If you set the block_size too high, it
may
> consume
> > > all the memory.
> > >
> > > Hope that helps clarify.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>
> > > >
> > > > Thanks John. I'm testing the change now.
> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just
7 days
> of
> > > > data. Its been running for over 2 hours now. Is this the
expected
> > > > behavior? I do have -v set to 3. Its writing steadily to the
log.
> Is
> > > > that slowing things down?
> > > >
> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Hello John,
> > > > >
> > > > > I see you're having difficulty configuring Series-Analysis
to apply
> > the
> > > > > continuous and categorical statistics you defined.
> > > > >
> > > > > It's funny, this exact same issue arose via met-help last
week, and
> > it
> > > > > makes me think we should modify the code in some way to make
it
> more
> > > > > intuitive. Please try editing your config file by moving
the
> > > cnt_thresh
> > > > > and cat_thresh settings up outside of the field array, as
shown
> > below:
> > > > >
> > > > > *fcst = {*
> > > > >
> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
>=2.54&&<6.35,
> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
>=25.4&&<76.2,
> >=76.2
> > > ];*
> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7, >=19.05,
> >=25.4,
> > > > > >=50.8, >=76.2 ];*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > * field = [ { name = "${FIELD}"; level
=
> > > > "${LEVEL}";
> > > > > } ];}obs = fcst;*
> > > > >
> > > > > When you rerun after this change, does it produce the
expected
> > output?
> > > > >
> > > > > I realize that this is confusing, especially if you're used
to
> > setting
> > > up
> > > > > config files for Point-Stat and Grid-Stat. Those tools are
set up
> to
> > > > > process multiple fields in a single run. Therefore you can
specify
> > > > > cat_thresh and cnt_thresh separately for each field array
entry.
> > Unlike
> > > > > Point-Stat and Grid-Stat, Series-Analysis can only process a
single
> > > > series
> > > > > of data in each run. There are multiple main ways of
defining a
> > series.
> > > > > (1) By far, the most common way is specifying a single field
that
> is
> > > > > extracted from a time-series of input data files... as
you're
> doing.
> > > > > (2) You can also define a series of field array entries that
can be
> > > > > extracted from the same input file. For example, compute
stats over
> > > > > multiple vertical levels from the same input file.
> > > > > (3) You define N input fields to be extracted from N input
files.
> > > > >
> > > > > Because of options (2) and (3), the field entry is defined
as an
> > array
> > > > > rather than a single dictionary. In practice though, the
field
> array
> > > > > typically has length 1.
> > > > >
> > > > > But the cat_thresh and cnt_thresh settings must remain
constant
> over
> > > the
> > > > > entire series, and that's why the code is setup to parse
them from
> > the
> > > > > "fcst" and "obs" dictionaries, and not separately from each
field
> > array
> > > > > entry.
> > > > >
> > > > > Seems like it'd be a good idea to check for this situation
and
> print
> > a
> > > > > warning message for the user.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
> met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> > > > > >
> > > > > > Hello John,
> > > > > >
> > > > > > It looks like you need some assistance setting up your
series
> > > analysis
> > > > in
> > > > > > MET. I'm assigning this ticket to John Halley Gotway.
Please
> > allow
> > > a
> > > > > few
> > > > > > business days for a response.
> > > > > >
> > > > > > Regards,
> > > > > > Minna
> > > > > > ---------------
> > > > > > Minna Win
> > > > > > National Center for Atmospheric Research
> > > > > > Developmental Testbed Center
> > > > > > Phone: 303-497-8423
> > > > > > Fax: 303-497-8401
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA
Federal via
> > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
> > > > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > > > > Queue: met_help
> > > > > > > Subject: Series Analysis Config File Issues
> > > > > > > Owner: Nobody
> > > > > > > Requestors: john.l.wagner at noaa.gov
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Good morning MET help
> > > > > > > I'm having an issue with series_analysis that I could
use your
> > > > > assistance
> > > > > > > with. I am currently testing series_analysis for QPF06
(config
> > > file
> > > > > > > attached). I am unable to get valid output values for
anything
> > in
> > > > the
> > > > > > CTS
> > > > > > > and MCTS categories. The TOTAL grid for each score group
looks
> > good
> > > > > (I'm
> > > > > > > currently testing with 7 days of data). The CSI values
are all
> > > > > constant
> > > > > > at
> > > > > > > 1. I am unable to get any values for HSS, either from
CTS or
> > MCTS.
> > > > > All
> > > > > > > output (MAE, ME, RMSE) from the CNT score group seem to
work
> > fine.
> > > > > > > I'm assuming the issue is setting the thresholds in my
config
> > file.
> > > > > > > The scores that I see in my netCDF file for the CTS
scores all
> > end
> > > > with
> > > > > > > "_NA", where I'm assuming a category should be set. I
am using
> > the
> > > > > same
> > > > > > > thresholds that I use in grid_stat. Should I be setting
them
> up
> > > > > > > differently for series_analysis? Is there a sample
config file
> > > that
> > > > I
> > > > > > > could look at for reference?
> > > > > > > If it helps, my test output file can be found on WCOSS
> > > > > > > here:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> > > > > > >
> > > > > > > Thanks
> > > > > > > John
> > > > > > > --
> > > > > > > John Wagner
> > > > > > > Verification Task Lead
> > > > > > > NOAA/National Weather Service
> > > > > > > Meteorological Development Laboratory
> > > > > > > Digital Forecast Services Division
> > > > > > > SSMC2 Room 10106
> > > > > > > Silver Spring, MD 20910
> > > > > > > (301) 427-9471 (office)
> > > > > > > (908) 902-4155 (cell/text)
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > John Wagner
> > > > Verification Task Lead
> > > > NOAA/National Weather Service
> > > > Meteorological Development Laboratory
> > > > Digital Forecast Services Division
> > > > SSMC2 Room 10106
> > > > Silver Spring, MD 20910
> > > > (301) 427-9471 (office)
> > > > (908) 902-4155 (cell/text)
> > > >
> > > >
> > >
> > >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John L Wagner - NOAA Federal
Time: Fri Jun 12 15:56:38 2020
John
I believe I've confirmed that the slow run times were from setting -v
3
when calling series_analysis. My last test ran in about 25 minutes.
I
have a few more tests to run, but I believe all of my issues are
solved.
Thanks
John
On Fri, Jun 12, 2020 at 7:39 AM John L Wagner - NOAA Federal <
john.l.wagner at noaa.gov> wrote:
> Here is the usage summary from the test I ran yesterday:
>
> [image: SerA Test1 Log.PNG]
> The job did not time out, and finished in 5 hr 43 min. It did not
use all
> of the memory, which surprised me. When I was testing incorrectly
with the
> thresholds defined within the field array, I had to keep increasing
the
> memory setting in my bsub command to prevent OOM kills.
>
> I'm running a test now with just the lowest CNT threshold. This may
be
> what we end up running anyway and I'm hoping the run time will come
down so
> I can run more tests. I'll try to get through tests today with
cutting the
> block_size in half, using a mask, and different memory settings with
bsub.
> I'll let you know what I find.
>
> As for MPI, I agree with that solution. I'm not going to pretend to
know
> what needs to be changed with C code and compilers. When we updated
the
> MOS-2000 Fortran code, we managed to get our ensemble code run time
down
> from 3 hours to 12 minutes. I realize that not all MET users are
running
> on supercomputers that can take advantage of MPI, but those of us
that are
> would certainly appreciate it.
>
> Thanks
> John
>
> On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Agreed, that's much too long! I suppose another possibility is that
it's
>> consuming too much memory and is using swap space, which is slow.
>> If possible, please check what percent of memory the process is
using. If
>> it's up around 100%, then let's try cutting the block size in half
and
>> rerunning.
>>
>> Honestly, the real fix for this is enhancing Series-Analysis with
MPI
>> calls
>> to subset the 4+ million grid point into tiles, and then do the
processing
>> for each tile on a separate node.
>> However, none of the MET tools are currently parallelized. So
that'd
>> obviously be a significant development effort.
>>
>> Another option, just to get something to actually complete in the
time
>> window, would be applying a masking region to subset the data being
>> processed.
>>
>> Thanks,
>> John
>>
>> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal via RT
<
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>> >
>> > I set the block size to my grid size (4194304) previously. Its
still
>> > running 5+ hours later. It will time out after 6 hours.
>> > I have been testing on WCOSS with Julie's MET V9.0. Tomorrow
I'll run a
>> > test with V8.1. Otherwise, we'll need to rethink what we want to
show
>> in
>> > the viewer.
>> > Thanks
>> > John
>> >
>> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > John,
>> > >
>> > > I'd recommend modifying the "block_size" config file option and
>> > rerunning.
>> > > The default value is set very low, ensuring that you don't run
out of
>> > > memory but requiring many passes through the data... which is
slow.
>> > >
>> > > Here's an excerpt from this README file which describes how
that
>> works.
>> > >
>> > >
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
>> > >
>> > >
>> > >
>> >
>>
////////////////////////////////////////////////////////////////////////////////
>> > > //
>> > > // SeriesAnalysisConfig_default
>> > > //
>> > >
>> > >
>> >
>>
////////////////////////////////////////////////////////////////////////////////
>> > >
>> > > //
>> > > // Computation may be memory intensive, especially for large
grids.
>> > > // The "block_size" entry sets the number of grid points to be
>> processed
>> > > // concurrently (i.e. in one pass through a time series).
Smaller
>> values
>> > > // require less memory but increase the number of passes
through the
>> > data.
>> > > //
>> > > block_size = 1024;
>> > >
>> > > Look near the top of your log file for a message that looks
something
>> > like
>> > > this:
>> > >
>> > >
>> > > *DEBUG 2: Computing statistics using a block size of 10000,
requiring
>> 3
>> > > pass(es) through the 169 x 154 grid.DEBUG 2: Processing data
pass
>> number
>> > 1
>> > > of 3 for grid points 1 to 10000.*
>> > >
>> > > This will tell you how many times it's reading through the
input data
>> > > files. Ideally you'd set the block size high enough so that it
only
>> needs
>> > > to read them once. But that all depends on the number of grid
points
>> and
>> > > length of your series. If you set the block_size too high, it
may
>> consume
>> > > all the memory.
>> > >
>> > > Hope that helps clarify.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal
via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>
>> > > >
>> > > > Thanks John. I'm testing the change now.
>> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for just
7
>> days of
>> > > > data. Its been running for over 2 hours now. Is this the
expected
>> > > > behavior? I do have -v set to 3. Its writing steadily to
the
>> log. Is
>> > > > that slowing things down?
>> > > >
>> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > > Hello John,
>> > > > >
>> > > > > I see you're having difficulty configuring Series-Analysis
to
>> apply
>> > the
>> > > > > continuous and categorical statistics you defined.
>> > > > >
>> > > > > It's funny, this exact same issue arose via met-help last
week,
>> and
>> > it
>> > > > > makes me think we should modify the code in some way to
make it
>> more
>> > > > > intuitive. Please try editing your config file by moving
the
>> > > cnt_thresh
>> > > > > and cat_thresh settings up outside of the field array, as
shown
>> > below:
>> > > > >
>> > > > > *fcst = {*
>> > > > >
>> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
>=2.54&&<6.35,
>> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
>=25.4&&<76.2,
>> >=76.2
>> > > ];*
>> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7,
>=19.05,
>> >=25.4,
>> > > > > >=50.8, >=76.2 ];*
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > * field = [ { name = "${FIELD}";
level =
>> > > > "${LEVEL}";
>> > > > > } ];}obs = fcst;*
>> > > > >
>> > > > > When you rerun after this change, does it produce the
expected
>> > output?
>> > > > >
>> > > > > I realize that this is confusing, especially if you're used
to
>> > setting
>> > > up
>> > > > > config files for Point-Stat and Grid-Stat. Those tools are
set up
>> to
>> > > > > process multiple fields in a single run. Therefore you can
specify
>> > > > > cat_thresh and cnt_thresh separately for each field array
entry.
>> > Unlike
>> > > > > Point-Stat and Grid-Stat, Series-Analysis can only process
a
>> single
>> > > > series
>> > > > > of data in each run. There are multiple main ways of
defining a
>> > series.
>> > > > > (1) By far, the most common way is specifying a single
field that
>> is
>> > > > > extracted from a time-series of input data files... as
you're
>> doing.
>> > > > > (2) You can also define a series of field array entries
that can
>> be
>> > > > > extracted from the same input file. For example, compute
stats
>> over
>> > > > > multiple vertical levels from the same input file.
>> > > > > (3) You define N input fields to be extracted from N input
files.
>> > > > >
>> > > > > Because of options (2) and (3), the field entry is defined
as an
>> > array
>> > > > > rather than a single dictionary. In practice though, the
field
>> array
>> > > > > typically has length 1.
>> > > > >
>> > > > > But the cat_thresh and cnt_thresh settings must remain
constant
>> over
>> > > the
>> > > > > entire series, and that's why the code is setup to parse
them from
>> > the
>> > > > > "fcst" and "obs" dictionaries, and not separately from each
field
>> > array
>> > > > > entry.
>> > > > >
>> > > > > Seems like it'd be a good idea to check for this situation
and
>> print
>> > a
>> > > > > warning message for the user.
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
>> met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>> > > > > >
>> > > > > > Hello John,
>> > > > > >
>> > > > > > It looks like you need some assistance setting up your
series
>> > > analysis
>> > > > in
>> > > > > > MET. I'm assigning this ticket to John Halley Gotway.
Please
>> > allow
>> > > a
>> > > > > few
>> > > > > > business days for a response.
>> > > > > >
>> > > > > > Regards,
>> > > > > > Minna
>> > > > > > ---------------
>> > > > > > Minna Win
>> > > > > > National Center for Atmospheric Research
>> > > > > > Developmental Testbed Center
>> > > > > > Phone: 303-497-8423
>> > > > > > Fax: 303-497-8401
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA
Federal via
>> > RT <
>> > > > > > met_help at ucar.edu> wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted upon.
>> > > > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
>> > > > > > > Queue: met_help
>> > > > > > > Subject: Series Analysis Config File Issues
>> > > > > > > Owner: Nobody
>> > > > > > > Requestors: john.l.wagner at noaa.gov
>> > > > > > > Status: new
>> > > > > > > Ticket <URL:
>> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>> > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > Good morning MET help
>> > > > > > > I'm having an issue with series_analysis that I could
use your
>> > > > > assistance
>> > > > > > > with. I am currently testing series_analysis for QPF06
>> (config
>> > > file
>> > > > > > > attached). I am unable to get valid output values for
>> anything
>> > in
>> > > > the
>> > > > > > CTS
>> > > > > > > and MCTS categories. The TOTAL grid for each score
group looks
>> > good
>> > > > > (I'm
>> > > > > > > currently testing with 7 days of data). The CSI values
are
>> all
>> > > > > constant
>> > > > > > at
>> > > > > > > 1. I am unable to get any values for HSS, either from
CTS or
>> > MCTS.
>> > > > > All
>> > > > > > > output (MAE, ME, RMSE) from the CNT score group seem to
work
>> > fine.
>> > > > > > > I'm assuming the issue is setting the thresholds in my
config
>> > file.
>> > > > > > > The scores that I see in my netCDF file for the CTS
scores all
>> > end
>> > > > with
>> > > > > > > "_NA", where I'm assuming a category should be set. I
am
>> using
>> > the
>> > > > > same
>> > > > > > > thresholds that I use in grid_stat. Should I be
setting them
>> up
>> > > > > > > differently for series_analysis? Is there a sample
config
>> file
>> > > that
>> > > > I
>> > > > > > > could look at for reference?
>> > > > > > > If it helps, my test output file can be found on WCOSS
>> > > > > > > here:
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
>> > > > > > >
>> > > > > > > Thanks
>> > > > > > > John
>> > > > > > > --
>> > > > > > > John Wagner
>> > > > > > > Verification Task Lead
>> > > > > > > NOAA/National Weather Service
>> > > > > > > Meteorological Development Laboratory
>> > > > > > > Digital Forecast Services Division
>> > > > > > > SSMC2 Room 10106
>> > > > > > > Silver Spring, MD 20910
>> > > > > > > (301) 427-9471 (office)
>> > > > > > > (908) 902-4155 (cell/text)
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > John Wagner
>> > > > Verification Task Lead
>> > > > NOAA/National Weather Service
>> > > > Meteorological Development Laboratory
>> > > > Digital Forecast Services Division
>> > > > SSMC2 Room 10106
>> > > > Silver Spring, MD 20910
>> > > > (301) 427-9471 (office)
>> > > > (908) 902-4155 (cell/text)
>> > > >
>> > > >
>> > >
>> > >
>> >
>> > --
>> > John Wagner
>> > Verification Task Lead
>> > NOAA/National Weather Service
>> > Meteorological Development Laboratory
>> > Digital Forecast Services Division
>> > SSMC2 Room 10106
>> > Silver Spring, MD 20910
>> > (301) 427-9471 (office)
>> > (908) 902-4155 (cell/text)
>> >
>> >
>>
>>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Fri Jun 12 17:29:59 2020
John,
Oh wow, OK. Would you recommend that I change some of those log
messages to
a higher verbosity level... like 4 or 5... to avoid slow runtimes?
John
On Fri, Jun 12, 2020 at 3:57 PM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>
> John
> I believe I've confirmed that the slow run times were from setting
-v 3
> when calling series_analysis. My last test ran in about 25 minutes.
I
> have a few more tests to run, but I believe all of my issues are
solved.
> Thanks
> John
>
> On Fri, Jun 12, 2020 at 7:39 AM John L Wagner - NOAA Federal <
> john.l.wagner at noaa.gov> wrote:
>
> > Here is the usage summary from the test I ran yesterday:
> >
> > [image: SerA Test1 Log.PNG]
> > The job did not time out, and finished in 5 hr 43 min. It did not
use
> all
> > of the memory, which surprised me. When I was testing incorrectly
with
> the
> > thresholds defined within the field array, I had to keep
increasing the
> > memory setting in my bsub command to prevent OOM kills.
> >
> > I'm running a test now with just the lowest CNT threshold. This
may be
> > what we end up running anyway and I'm hoping the run time will
come down
> so
> > I can run more tests. I'll try to get through tests today with
cutting
> the
> > block_size in half, using a mask, and different memory settings
with
> bsub.
> > I'll let you know what I find.
> >
> > As for MPI, I agree with that solution. I'm not going to pretend
to know
> > what needs to be changed with C code and compilers. When we
updated the
> > MOS-2000 Fortran code, we managed to get our ensemble code run
time down
> > from 3 hours to 12 minutes. I realize that not all MET users are
running
> > on supercomputers that can take advantage of MPI, but those of us
that
> are
> > would certainly appreciate it.
> >
> > Thanks
> > John
> >
> > On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Agreed, that's much too long! I suppose another possibility is
that it's
> >> consuming too much memory and is using swap space, which is slow.
> >> If possible, please check what percent of memory the process is
using.
> If
> >> it's up around 100%, then let's try cutting the block size in
half and
> >> rerunning.
> >>
> >> Honestly, the real fix for this is enhancing Series-Analysis with
MPI
> >> calls
> >> to subset the 4+ million grid point into tiles, and then do the
> processing
> >> for each tile on a separate node.
> >> However, none of the MET tools are currently parallelized. So
that'd
> >> obviously be a significant development effort.
> >>
> >> Another option, just to get something to actually complete in the
time
> >> window, would be applying a masking region to subset the data
being
> >> processed.
> >>
> >> Thanks,
> >> John
> >>
> >> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal via
RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >> >
> >> > I set the block size to my grid size (4194304) previously. Its
still
> >> > running 5+ hours later. It will time out after 6 hours.
> >> > I have been testing on WCOSS with Julie's MET V9.0. Tomorrow
I'll
> run a
> >> > test with V8.1. Otherwise, we'll need to rethink what we want
to show
> >> in
> >> > the viewer.
> >> > Thanks
> >> > John
> >> >
> >> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
> >> > met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > John,
> >> > >
> >> > > I'd recommend modifying the "block_size" config file option
and
> >> > rerunning.
> >> > > The default value is set very low, ensuring that you don't
run out
> of
> >> > > memory but requiring many passes through the data... which is
slow.
> >> > >
> >> > > Here's an excerpt from this README file which describes how
that
> >> works.
> >> > >
> >> > >
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
> >> > >
> >> > >
> >> > >
> >> >
> >>
>
////////////////////////////////////////////////////////////////////////////////
> >> > > //
> >> > > // SeriesAnalysisConfig_default
> >> > > //
> >> > >
> >> > >
> >> >
> >>
>
////////////////////////////////////////////////////////////////////////////////
> >> > >
> >> > > //
> >> > > // Computation may be memory intensive, especially for large
grids.
> >> > > // The "block_size" entry sets the number of grid points to
be
> >> processed
> >> > > // concurrently (i.e. in one pass through a time series).
Smaller
> >> values
> >> > > // require less memory but increase the number of passes
through the
> >> > data.
> >> > > //
> >> > > block_size = 1024;
> >> > >
> >> > > Look near the top of your log file for a message that looks
> something
> >> > like
> >> > > this:
> >> > >
> >> > >
> >> > > *DEBUG 2: Computing statistics using a block size of 10000,
> requiring
> >> 3
> >> > > pass(es) through the 169 x 154 grid.DEBUG 2: Processing data
pass
> >> number
> >> > 1
> >> > > of 3 for grid points 1 to 10000.*
> >> > >
> >> > > This will tell you how many times it's reading through the
input
> data
> >> > > files. Ideally you'd set the block size high enough so that
it only
> >> needs
> >> > > to read them once. But that all depends on the number of grid
points
> >> and
> >> > > length of your series. If you set the block_size too high, it
may
> >> consume
> >> > > all the memory.
> >> > >
> >> > > Hope that helps clarify.
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal
via RT
> <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
> >> > > >
> >> > > > Thanks John. I'm testing the change now.
> >> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for
just 7
> >> days of
> >> > > > data. Its been running for over 2 hours now. Is this the
> expected
> >> > > > behavior? I do have -v set to 3. Its writing steadily to
the
> >> log. Is
> >> > > > that slowing things down?
> >> > > >
> >> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT
<
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > > Hello John,
> >> > > > >
> >> > > > > I see you're having difficulty configuring Series-
Analysis to
> >> apply
> >> > the
> >> > > > > continuous and categorical statistics you defined.
> >> > > > >
> >> > > > > It's funny, this exact same issue arose via met-help last
week,
> >> and
> >> > it
> >> > > > > makes me think we should modify the code in some way to
make it
> >> more
> >> > > > > intuitive. Please try editing your config file by moving
the
> >> > > cnt_thresh
> >> > > > > and cat_thresh settings up outside of the field array, as
shown
> >> > below:
> >> > > > >
> >> > > > > *fcst = {*
> >> > > > >
> >> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
>=2.54&&<6.35,
> >> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
>=25.4&&<76.2,
> >> >=76.2
> >> > > ];*
> >> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7,
>=19.05,
> >> >=25.4,
> >> > > > > >=50.8, >=76.2 ];*
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > * field = [ { name = "${FIELD}";
level =
> >> > > > "${LEVEL}";
> >> > > > > } ];}obs = fcst;*
> >> > > > >
> >> > > > > When you rerun after this change, does it produce the
expected
> >> > output?
> >> > > > >
> >> > > > > I realize that this is confusing, especially if you're
used to
> >> > setting
> >> > > up
> >> > > > > config files for Point-Stat and Grid-Stat. Those tools
are set
> up
> >> to
> >> > > > > process multiple fields in a single run. Therefore you
can
> specify
> >> > > > > cat_thresh and cnt_thresh separately for each field array
entry.
> >> > Unlike
> >> > > > > Point-Stat and Grid-Stat, Series-Analysis can only
process a
> >> single
> >> > > > series
> >> > > > > of data in each run. There are multiple main ways of
defining a
> >> > series.
> >> > > > > (1) By far, the most common way is specifying a single
field
> that
> >> is
> >> > > > > extracted from a time-series of input data files... as
you're
> >> doing.
> >> > > > > (2) You can also define a series of field array entries
that can
> >> be
> >> > > > > extracted from the same input file. For example, compute
stats
> >> over
> >> > > > > multiple vertical levels from the same input file.
> >> > > > > (3) You define N input fields to be extracted from N
input
> files.
> >> > > > >
> >> > > > > Because of options (2) and (3), the field entry is
defined as an
> >> > array
> >> > > > > rather than a single dictionary. In practice though, the
field
> >> array
> >> > > > > typically has length 1.
> >> > > > >
> >> > > > > But the cat_thresh and cnt_thresh settings must remain
constant
> >> over
> >> > > the
> >> > > > > entire series, and that's why the code is setup to parse
them
> from
> >> > the
> >> > > > > "fcst" and "obs" dictionaries, and not separately from
each
> field
> >> > array
> >> > > > > entry.
> >> > > > >
> >> > > > > Seems like it'd be a good idea to check for this
situation and
> >> print
> >> > a
> >> > > > > warning message for the user.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > John
> >> > > > >
> >> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
> >> met_help at ucar.edu>
> >> > > > > wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> >
> >> > > > > >
> >> > > > > > Hello John,
> >> > > > > >
> >> > > > > > It looks like you need some assistance setting up your
series
> >> > > analysis
> >> > > > in
> >> > > > > > MET. I'm assigning this ticket to John Halley Gotway.
Please
> >> > allow
> >> > > a
> >> > > > > few
> >> > > > > > business days for a response.
> >> > > > > >
> >> > > > > > Regards,
> >> > > > > > Minna
> >> > > > > > ---------------
> >> > > > > > Minna Win
> >> > > > > > National Center for Atmospheric Research
> >> > > > > > Developmental Testbed Center
> >> > > > > > Phone: 303-497-8423
> >> > > > > > Fax: 303-497-8401
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA
Federal
> via
> >> > RT <
> >> > > > > > met_help at ucar.edu> wrote:
> >> > > > > >
> >> > > > > > >
> >> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted
upon.
> >> > > > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> >> > > > > > > Queue: met_help
> >> > > > > > > Subject: Series Analysis Config File Issues
> >> > > > > > > Owner: Nobody
> >> > > > > > > Requestors: john.l.wagner at noaa.gov
> >> > > > > > > Status: new
> >> > > > > > > Ticket <URL:
> >> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
> >> > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Good morning MET help
> >> > > > > > > I'm having an issue with series_analysis that I could
use
> your
> >> > > > > assistance
> >> > > > > > > with. I am currently testing series_analysis for
QPF06
> >> (config
> >> > > file
> >> > > > > > > attached). I am unable to get valid output values
for
> >> anything
> >> > in
> >> > > > the
> >> > > > > > CTS
> >> > > > > > > and MCTS categories. The TOTAL grid for each score
group
> looks
> >> > good
> >> > > > > (I'm
> >> > > > > > > currently testing with 7 days of data). The CSI
values are
> >> all
> >> > > > > constant
> >> > > > > > at
> >> > > > > > > 1. I am unable to get any values for HSS, either
from CTS
> or
> >> > MCTS.
> >> > > > > All
> >> > > > > > > output (MAE, ME, RMSE) from the CNT score group seem
to work
> >> > fine.
> >> > > > > > > I'm assuming the issue is setting the thresholds in
my
> config
> >> > file.
> >> > > > > > > The scores that I see in my netCDF file for the CTS
scores
> all
> >> > end
> >> > > > with
> >> > > > > > > "_NA", where I'm assuming a category should be set.
I am
> >> using
> >> > the
> >> > > > > same
> >> > > > > > > thresholds that I use in grid_stat. Should I be
setting
> them
> >> up
> >> > > > > > > differently for series_analysis? Is there a sample
config
> >> file
> >> > > that
> >> > > > I
> >> > > > > > > could look at for reference?
> >> > > > > > > If it helps, my test output file can be found on
WCOSS
> >> > > > > > > here:
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
> >> > > > > > >
> >> > > > > > > Thanks
> >> > > > > > > John
> >> > > > > > > --
> >> > > > > > > John Wagner
> >> > > > > > > Verification Task Lead
> >> > > > > > > NOAA/National Weather Service
> >> > > > > > > Meteorological Development Laboratory
> >> > > > > > > Digital Forecast Services Division
> >> > > > > > > SSMC2 Room 10106
> >> > > > > > > Silver Spring, MD 20910
> >> > > > > > > (301) 427-9471 (office)
> >> > > > > > > (908) 902-4155 (cell/text)
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > > --
> >> > > > John Wagner
> >> > > > Verification Task Lead
> >> > > > NOAA/National Weather Service
> >> > > > Meteorological Development Laboratory
> >> > > > Digital Forecast Services Division
> >> > > > SSMC2 Room 10106
> >> > > > Silver Spring, MD 20910
> >> > > > (301) 427-9471 (office)
> >> > > > (908) 902-4155 (cell/text)
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > John Wagner
> >> > Verification Task Lead
> >> > NOAA/National Weather Service
> >> > Meteorological Development Laboratory
> >> > Digital Forecast Services Division
> >> > SSMC2 Room 10106
> >> > Silver Spring, MD 20910
> >> > (301) 427-9471 (office)
> >> > (908) 902-4155 (cell/text)
> >> >
> >> >
> >>
> >>
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Series Analysis Config File Issues
From: John Halley Gotway
Time: Fri Jun 12 17:42:16 2020
John,
You know, I think I should leave well-enough alone. But I'm very
surprised
about this change in runtime. I ran a short test case at verbosity
level 0
and it took 20 seconds. At verbosity level 4, it took 22 seconds.
More, but
not a crazy amount more.
But I'm glad it's running in a more reasonable amount of time, and
hope it
continues to do so. I'll go ahead and resolve this ticket.
Thanks,
John
On Fri, Jun 12, 2020 at 5:29 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Oh wow, OK. Would you recommend that I change some of those log
messages
> to a higher verbosity level... like 4 or 5... to avoid slow
runtimes?
>
> John
>
> On Fri, Jun 12, 2020 at 3:57 PM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>>
>> John
>> I believe I've confirmed that the slow run times were from setting
-v 3
>> when calling series_analysis. My last test ran in about 25
minutes. I
>> have a few more tests to run, but I believe all of my issues are
solved.
>> Thanks
>> John
>>
>> On Fri, Jun 12, 2020 at 7:39 AM John L Wagner - NOAA Federal <
>> john.l.wagner at noaa.gov> wrote:
>>
>> > Here is the usage summary from the test I ran yesterday:
>> >
>> > [image: SerA Test1 Log.PNG]
>> > The job did not time out, and finished in 5 hr 43 min. It did
not use
>> all
>> > of the memory, which surprised me. When I was testing
incorrectly with
>> the
>> > thresholds defined within the field array, I had to keep
increasing the
>> > memory setting in my bsub command to prevent OOM kills.
>> >
>> > I'm running a test now with just the lowest CNT threshold. This
may be
>> > what we end up running anyway and I'm hoping the run time will
come
>> down so
>> > I can run more tests. I'll try to get through tests today with
cutting
>> the
>> > block_size in half, using a mask, and different memory settings
with
>> bsub.
>> > I'll let you know what I find.
>> >
>> > As for MPI, I agree with that solution. I'm not going to pretend
to
>> know
>> > what needs to be changed with C code and compilers. When we
updated the
>> > MOS-2000 Fortran code, we managed to get our ensemble code run
time down
>> > from 3 hours to 12 minutes. I realize that not all MET users are
>> running
>> > on supercomputers that can take advantage of MPI, but those of us
that
>> are
>> > would certainly appreciate it.
>> >
>> > Thanks
>> > John
>> >
>> > On Thu, Jun 11, 2020 at 7:08 PM John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >> Agreed, that's much too long! I suppose another possibility is
that
>> it's
>> >> consuming too much memory and is using swap space, which is
slow.
>> >> If possible, please check what percent of memory the process is
using.
>> If
>> >> it's up around 100%, then let's try cutting the block size in
half and
>> >> rerunning.
>> >>
>> >> Honestly, the real fix for this is enhancing Series-Analysis
with MPI
>> >> calls
>> >> to subset the 4+ million grid point into tiles, and then do the
>> processing
>> >> for each tile on a separate node.
>> >> However, none of the MET tools are currently parallelized. So
that'd
>> >> obviously be a significant development effort.
>> >>
>> >> Another option, just to get something to actually complete in
the time
>> >> window, would be applying a masking region to subset the data
being
>> >> processed.
>> >>
>> >> Thanks,
>> >> John
>> >>
>> >> On Thu, Jun 11, 2020 at 4:47 PM John L Wagner - NOAA Federal via
RT <
>> >> met_help at ucar.edu> wrote:
>> >>
>> >> >
>> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>
>> >> >
>> >> > I set the block size to my grid size (4194304) previously.
Its still
>> >> > running 5+ hours later. It will time out after 6 hours.
>> >> > I have been testing on WCOSS with Julie's MET V9.0. Tomorrow
I'll
>> run a
>> >> > test with V8.1. Otherwise, we'll need to rethink what we want
to
>> show
>> >> in
>> >> > the viewer.
>> >> > Thanks
>> >> > John
>> >> >
>> >> > On Thu, Jun 11, 2020 at 5:24 PM John Halley Gotway via RT <
>> >> > met_help at ucar.edu>
>> >> > wrote:
>> >> >
>> >> > > John,
>> >> > >
>> >> > > I'd recommend modifying the "block_size" config file option
and
>> >> > rerunning.
>> >> > > The default value is set very low, ensuring that you don't
run out
>> of
>> >> > > memory but requiring many passes through the data... which
is slow.
>> >> > >
>> >> > > Here's an excerpt from this README file which describes how
that
>> >> works.
>> >> > >
>> >> > >
>> https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
>> >> > >
>> >> > >
>> >> > >
>> >> >
>> >>
>>
////////////////////////////////////////////////////////////////////////////////
>> >> > > //
>> >> > > // SeriesAnalysisConfig_default
>> >> > > //
>> >> > >
>> >> > >
>> >> >
>> >>
>>
////////////////////////////////////////////////////////////////////////////////
>> >> > >
>> >> > > //
>> >> > > // Computation may be memory intensive, especially for large
grids.
>> >> > > // The "block_size" entry sets the number of grid points to
be
>> >> processed
>> >> > > // concurrently (i.e. in one pass through a time series).
Smaller
>> >> values
>> >> > > // require less memory but increase the number of passes
through
>> the
>> >> > data.
>> >> > > //
>> >> > > block_size = 1024;
>> >> > >
>> >> > > Look near the top of your log file for a message that looks
>> something
>> >> > like
>> >> > > this:
>> >> > >
>> >> > >
>> >> > > *DEBUG 2: Computing statistics using a block size of 10000,
>> requiring
>> >> 3
>> >> > > pass(es) through the 169 x 154 grid.DEBUG 2: Processing data
pass
>> >> number
>> >> > 1
>> >> > > of 3 for grid points 1 to 10000.*
>> >> > >
>> >> > > This will tell you how many times it's reading through the
input
>> data
>> >> > > files. Ideally you'd set the block size high enough so that
it only
>> >> needs
>> >> > > to read them once. But that all depends on the number of
grid
>> points
>> >> and
>> >> > > length of your series. If you set the block_size too high,
it may
>> >> consume
>> >> > > all the memory.
>> >> > >
>> >> > > Hope that helps clarify.
>> >> > >
>> >> > > Thanks,
>> >> > > John
>> >> > >
>> >> > > On Thu, Jun 11, 2020 at 1:40 PM John L Wagner - NOAA Federal
via
>> RT <
>> >> > > met_help at ucar.edu> wrote:
>> >> > >
>> >> > > >
>> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>> >> > > >
>> >> > > > Thanks John. I'm testing the change now.
>> >> > > > I'm testing 1 QPF06 projection for MAE, CSI, and HSS for
just 7
>> >> days of
>> >> > > > data. Its been running for over 2 hours now. Is this the
>> expected
>> >> > > > behavior? I do have -v set to 3. Its writing steadily to
the
>> >> log. Is
>> >> > > > that slowing things down?
>> >> > > >
>> >> > > > On Thu, Jun 11, 2020 at 11:59 AM John Halley Gotway via RT
<
>> >> > > > met_help at ucar.edu> wrote:
>> >> > > >
>> >> > > > > Hello John,
>> >> > > > >
>> >> > > > > I see you're having difficulty configuring Series-
Analysis to
>> >> apply
>> >> > the
>> >> > > > > continuous and categorical statistics you defined.
>> >> > > > >
>> >> > > > > It's funny, this exact same issue arose via met-help
last week,
>> >> and
>> >> > it
>> >> > > > > makes me think we should modify the code in some way to
make it
>> >> more
>> >> > > > > intuitive. Please try editing your config file by
moving the
>> >> > > cnt_thresh
>> >> > > > > and cat_thresh settings up outside of the field array,
as shown
>> >> > below:
>> >> > > > >
>> >> > > > > *fcst = {*
>> >> > > > >
>> >> > > > > * cnt_thresh = [ >=0&&<0.254, >=0.254&&<2.54,
>=2.54&&<6.35,
>> >> > > > > >=6.35&&<12.7, >=12.7&&<19.05, >=19.05&&<25.4,
>=25.4&&<76.2,
>> >> >=76.2
>> >> > > ];*
>> >> > > > > * cat_thresh = [ >=0.254, >=2.54, >=6.35, >=12.7,
>=19.05,
>> >> >=25.4,
>> >> > > > > >=50.8, >=76.2 ];*
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > * field = [ { name = "${FIELD}";
level =
>> >> > > > "${LEVEL}";
>> >> > > > > } ];}obs = fcst;*
>> >> > > > >
>> >> > > > > When you rerun after this change, does it produce the
expected
>> >> > output?
>> >> > > > >
>> >> > > > > I realize that this is confusing, especially if you're
used to
>> >> > setting
>> >> > > up
>> >> > > > > config files for Point-Stat and Grid-Stat. Those tools
are set
>> up
>> >> to
>> >> > > > > process multiple fields in a single run. Therefore you
can
>> specify
>> >> > > > > cat_thresh and cnt_thresh separately for each field
array
>> entry.
>> >> > Unlike
>> >> > > > > Point-Stat and Grid-Stat, Series-Analysis can only
process a
>> >> single
>> >> > > > series
>> >> > > > > of data in each run. There are multiple main ways of
defining a
>> >> > series.
>> >> > > > > (1) By far, the most common way is specifying a single
field
>> that
>> >> is
>> >> > > > > extracted from a time-series of input data files... as
you're
>> >> doing.
>> >> > > > > (2) You can also define a series of field array entries
that
>> can
>> >> be
>> >> > > > > extracted from the same input file. For example, compute
stats
>> >> over
>> >> > > > > multiple vertical levels from the same input file.
>> >> > > > > (3) You define N input fields to be extracted from N
input
>> files.
>> >> > > > >
>> >> > > > > Because of options (2) and (3), the field entry is
defined as
>> an
>> >> > array
>> >> > > > > rather than a single dictionary. In practice though, the
field
>> >> array
>> >> > > > > typically has length 1.
>> >> > > > >
>> >> > > > > But the cat_thresh and cnt_thresh settings must remain
constant
>> >> over
>> >> > > the
>> >> > > > > entire series, and that's why the code is setup to parse
them
>> from
>> >> > the
>> >> > > > > "fcst" and "obs" dictionaries, and not separately from
each
>> field
>> >> > array
>> >> > > > > entry.
>> >> > > > >
>> >> > > > > Seems like it'd be a good idea to check for this
situation and
>> >> print
>> >> > a
>> >> > > > > warning message for the user.
>> >> > > > >
>> >> > > > > Thanks,
>> >> > > > > John
>> >> > > > >
>> >> > > > > On Thu, Jun 11, 2020 at 8:59 AM Minna Win via RT <
>> >> met_help at ucar.edu>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > >
>> >> > > > > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583 >
>> >> > > > > >
>> >> > > > > > Hello John,
>> >> > > > > >
>> >> > > > > > It looks like you need some assistance setting up your
series
>> >> > > analysis
>> >> > > > in
>> >> > > > > > MET. I'm assigning this ticket to John Halley Gotway.
>> Please
>> >> > allow
>> >> > > a
>> >> > > > > few
>> >> > > > > > business days for a response.
>> >> > > > > >
>> >> > > > > > Regards,
>> >> > > > > > Minna
>> >> > > > > > ---------------
>> >> > > > > > Minna Win
>> >> > > > > > National Center for Atmospheric Research
>> >> > > > > > Developmental Testbed Center
>> >> > > > > > Phone: 303-497-8423
>> >> > > > > > Fax: 303-497-8401
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > On Thu, Jun 11, 2020 at 8:53 AM John L Wagner - NOAA
Federal
>> via
>> >> > RT <
>> >> > > > > > met_help at ucar.edu> wrote:
>> >> > > > > >
>> >> > > > > > >
>> >> > > > > > > Thu Jun 11 08:53:50 2020: Request 95583 was acted
upon.
>> >> > > > > > > Transaction: Ticket created by
john.l.wagner at noaa.gov
>> >> > > > > > > Queue: met_help
>> >> > > > > > > Subject: Series Analysis Config File Issues
>> >> > > > > > > Owner: Nobody
>> >> > > > > > > Requestors: john.l.wagner at noaa.gov
>> >> > > > > > > Status: new
>> >> > > > > > > Ticket <URL:
>> >> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95583
>> >> > > > > >
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > Good morning MET help
>> >> > > > > > > I'm having an issue with series_analysis that I
could use
>> your
>> >> > > > > assistance
>> >> > > > > > > with. I am currently testing series_analysis for
QPF06
>> >> (config
>> >> > > file
>> >> > > > > > > attached). I am unable to get valid output values
for
>> >> anything
>> >> > in
>> >> > > > the
>> >> > > > > > CTS
>> >> > > > > > > and MCTS categories. The TOTAL grid for each score
group
>> looks
>> >> > good
>> >> > > > > (I'm
>> >> > > > > > > currently testing with 7 days of data). The CSI
values are
>> >> all
>> >> > > > > constant
>> >> > > > > > at
>> >> > > > > > > 1. I am unable to get any values for HSS, either
from CTS
>> or
>> >> > MCTS.
>> >> > > > > All
>> >> > > > > > > output (MAE, ME, RMSE) from the CNT score group seem
to
>> work
>> >> > fine.
>> >> > > > > > > I'm assuming the issue is setting the thresholds in
my
>> config
>> >> > file.
>> >> > > > > > > The scores that I see in my netCDF file for the CTS
scores
>> all
>> >> > end
>> >> > > > with
>> >> > > > > > > "_NA", where I'm assuming a category should be set.
I am
>> >> using
>> >> > the
>> >> > > > > same
>> >> > > > > > > thresholds that I use in grid_stat. Should I be
setting
>> them
>> >> up
>> >> > > > > > > differently for series_analysis? Is there a sample
config
>> >> file
>> >> > > that
>> >> > > > I
>> >> > > > > > > could look at for reference?
>> >> > > > > > > If it helps, my test output file can be found on
WCOSS
>> >> > > > > > > here:
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>>
/gpfs/dell2/mdl/mdlens/noscrub/John.L.Wagner/mdl.verif/blend/co/sera/1911/00/q6/blend1911q6_012
>> >> > > > > > >
>> >> > > > > > > Thanks
>> >> > > > > > > John
>> >> > > > > > > --
>> >> > > > > > > John Wagner
>> >> > > > > > > Verification Task Lead
>> >> > > > > > > NOAA/National Weather Service
>> >> > > > > > > Meteorological Development Laboratory
>> >> > > > > > > Digital Forecast Services Division
>> >> > > > > > > SSMC2 Room 10106
>> >> > > > > > > Silver Spring, MD 20910
>> >> > > > > > > (301) 427-9471 (office)
>> >> > > > > > > (908) 902-4155 (cell/text)
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > > > --
>> >> > > > John Wagner
>> >> > > > Verification Task Lead
>> >> > > > NOAA/National Weather Service
>> >> > > > Meteorological Development Laboratory
>> >> > > > Digital Forecast Services Division
>> >> > > > SSMC2 Room 10106
>> >> > > > Silver Spring, MD 20910
>> >> > > > (301) 427-9471 (office)
>> >> > > > (908) 902-4155 (cell/text)
>> >> > > >
>> >> > > >
>> >> > >
>> >> > >
>> >> >
>> >> > --
>> >> > John Wagner
>> >> > Verification Task Lead
>> >> > NOAA/National Weather Service
>> >> > Meteorological Development Laboratory
>> >> > Digital Forecast Services Division
>> >> > SSMC2 Room 10106
>> >> > Silver Spring, MD 20910
>> >> > (301) 427-9471 (office)
>> >> > (908) 902-4155 (cell/text)
>> >> >
>> >> >
>> >>
>> >>
>> >
>> > --
>> > John Wagner
>> > Verification Task Lead
>> > NOAA/National Weather Service
>> > Meteorological Development Laboratory
>> > Digital Forecast Services Division
>> > SSMC2 Room 10106
>> > Silver Spring, MD 20910
>> > (301) 427-9471 (office)
>> > (908) 902-4155 (cell/text)
>> >
>>
>>
>> --
>> John Wagner
>> Verification Task Lead
>> NOAA/National Weather Service
>> Meteorological Development Laboratory
>> Digital Forecast Services Division
>> SSMC2 Room 10106
>> Silver Spring, MD 20910
>> (301) 427-9471 (office)
>> (908) 902-4155 (cell/text)
>>
>>
------------------------------------------------
More information about the Met_help
mailing list