[Met_help] [rt.rap.ucar.edu #86792] History for FSS calculation with grid stat

John Halley Gotway via RT met_help at ucar.edu
Mon Sep 10 09:18:53 MDT 2018 

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi,

I recently converted all of my MET configuration files from using v6.0 to
v7.0.  One issue I'm now having is that my FSS calculations are taking much
longer than they did when I was using METv6.0.  Currently it takes ~15-20
minutes to generate one .stat file.

I've attached the grid stat configure file that I am now using with v7.0
(GridStatConfig_APCP03_FSS_href) and the configure file I was using with
v6.0 (GridStatConfig_03h_APCP_NBR).

I modified cat_thresh for APCP A03 and I also reverted back from SHAPE =
CIRCLE to SHAPE = SQUARE for the neighborhood methods section, but the grid
stat calculation still takes much longer than it used to.  I also tried
using the CONUS.poly file in shared/met/poly instead of my CONUS.nc to see
if that would make a difference but it did not.  The v7.0 configure file
has many more options in it so perhaps the slowdown is related to something
I added in.

Do you have any suggestions for why it might be running so slowly?

Thanks!
Ben


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: FSS calculation with grid stat
From: John Halley Gotway
Time: Wed Aug 29 09:51:13 2018

Hi Ben,

I see that you've found the met-7.0 runtime of Grid-Stat is much
slower
than the met-6.0 version.  We did find/fix a Grid-Stat memory
allocation
issue that's posted to the MET website:
   https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php

I have two questions...

(1) What's the patch level of the code you're currently running?
Please
run "grid_stat --version" for the particular executable you're running
and
send me the result.  Should be something like this:

MET Version: V7.0

Repository: https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix

Revision:         5607

Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)


(2) Does this bugfix actually fix the problem you're seeing?  I'd like
to
test this out myself using your config file.  I can run 3 versions:

   - met-6.0 with latest patches

   - met-7.0 with no patches

   - met-7.0 with latest patches


What model/analysis are you evaluating?  I might as well use the same
resolution for my testing.

Thanks,
John

On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:

>
> Wed Aug 29 09:01:08 2018: Request 86792 was acted upon.
> Transaction: Ticket created by benjamin.blake at noaa.gov
>        Queue: met_help
>      Subject: FSS calculation with grid stat
>        Owner: Nobody
>   Requestors: benjamin.blake at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>
>
> Hi,
>
> I recently converted all of my MET configuration files from using
v6.0 to
> v7.0.  One issue I'm now having is that my FSS calculations are
taking much
> longer than they did when I was using METv6.0.  Currently it takes
~15-20
> minutes to generate one .stat file.
>
> I've attached the grid stat configure file that I am now using with
v7.0
> (GridStatConfig_APCP03_FSS_href) and the configure file I was using
with
> v6.0 (GridStatConfig_03h_APCP_NBR).
>
> I modified cat_thresh for APCP A03 and I also reverted back from
SHAPE =
> CIRCLE to SHAPE = SQUARE for the neighborhood methods section, but
the grid
> stat calculation still takes much longer than it used to.  I also
tried
> using the CONUS.poly file in shared/met/poly instead of my CONUS.nc
to see
> if that would make a difference but it did not.  The v7.0 configure
file
> has many more options in it so perhaps the slowdown is related to
something
> I added in.
>
> Do you have any suggestions for why it might be running so slowly?
>
> Thanks!
> Ben
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Wed Aug 29 10:06:37 2018

Hi John,

Thanks for letting me know about the grid stat memory allocation
issue.  I
am currently pointing to Julie's met/7.0 installation on WCOSS (Tide),
and
here is the output of grid_stat --version:

MET Version:    V7.0
Repository:    https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
Revision:    5483
Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun 2018)

Since the grid stat fix was put in place on July 24th the version I'm
using
likely doesn't have that fix, so I'm not sure if the bug fix will
resolve
the issue or not.  I could try pointing to a newer version of MET.  I
should also let you know that grid stat for the other line types does
not
seem to be slower, I only see a noticeable slowdown with the NBRCNT
line
type.

I am evaluating the deterministic members of the HREFv2 and the NCAR
ensemble as well as their respective means.  I verify them on NCEP
grid 227
which is a ~5-km grid (5.079 km).  I can send you some sample files
for
your testing if that would be easiest.

Thanks!

Ben


On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Hi Ben,
>
> I see that you've found the met-7.0 runtime of Grid-Stat is much
slower
> than the met-6.0 version.  We did find/fix a Grid-Stat memory
allocation
> issue that's posted to the MET website:
>
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
>
> I have two questions...
>
> (1) What's the patch level of the code you're currently running?
Please
> run "grid_stat --version" for the particular executable you're
running and
> send me the result.  Should be something like this:
>
> MET Version: V7.0
>
> Repository: https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
>
> Revision:         5607
>
> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
>
>
> (2) Does this bugfix actually fix the problem you're seeing?  I'd
like to
> test this out myself using your config file.  I can run 3 versions:
>
>    - met-6.0 with latest patches
>
>    - met-7.0 with no patches
>
>    - met-7.0 with latest patches
>
>
> What model/analysis are you evaluating?  I might as well use the
same
> resolution for my testing.
>
> Thanks,
> John
>
> On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Aug 29 09:01:08 2018: Request 86792 was acted upon.
> > Transaction: Ticket created by benjamin.blake at noaa.gov
> >        Queue: met_help
> >      Subject: FSS calculation with grid stat
> >        Owner: Nobody
> >   Requestors: benjamin.blake at noaa.gov
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >
> >
> > Hi,
> >
> > I recently converted all of my MET configuration files from using
v6.0 to
> > v7.0.  One issue I'm now having is that my FSS calculations are
taking
> much
> > longer than they did when I was using METv6.0.  Currently it takes
~15-20
> > minutes to generate one .stat file.
> >
> > I've attached the grid stat configure file that I am now using
with v7.0
> > (GridStatConfig_APCP03_FSS_href) and the configure file I was
using with
> > v6.0 (GridStatConfig_03h_APCP_NBR).
> >
> > I modified cat_thresh for APCP A03 and I also reverted back from
SHAPE =
> > CIRCLE to SHAPE = SQUARE for the neighborhood methods section, but
the
> grid
> > stat calculation still takes much longer than it used to.  I also
tried
> > using the CONUS.poly file in shared/met/poly instead of my
CONUS.nc to
> see
> > if that would make a difference but it did not.  The v7.0
configure file
> > has many more options in it so perhaps the slowdown is related to
> something
> > I added in.
> >
> > Do you have any suggestions for why it might be running so slowly?
> >
> > Thanks!
> > Ben
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: John Halley Gotway
Time: Wed Aug 29 10:09:58 2018

Ben,

Yeah, that'd be great.  If you send me a single input forecast and
analysis
file for comparison, I'll run those 3 versions of grid-stat using your
configuration and quantify the timing differences.

You can either point me to them on theia or post them to our anonymous
ftp
site:
   https://dtcenter.org/met/users/support/met_help.php#ftp

Thanks,
John

On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>
> Hi John,
>
> Thanks for letting me know about the grid stat memory allocation
issue.  I
> am currently pointing to Julie's met/7.0 installation on WCOSS
(Tide), and
> here is the output of grid_stat --version:
>
> MET Version:    V7.0
> Repository:
> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> Revision:    5483
> Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun 2018)
>
> Since the grid stat fix was put in place on July 24th the version
I'm using
> likely doesn't have that fix, so I'm not sure if the bug fix will
resolve
> the issue or not.  I could try pointing to a newer version of MET.
I
> should also let you know that grid stat for the other line types
does not
> seem to be slower, I only see a noticeable slowdown with the NBRCNT
line
> type.
>
> I am evaluating the deterministic members of the HREFv2 and the NCAR
> ensemble as well as their respective means.  I verify them on NCEP
grid 227
> which is a ~5-km grid (5.079 km).  I can send you some sample files
for
> your testing if that would be easiest.
>
> Thanks!
>
> Ben
>
>
> On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Hi Ben,
> >
> > I see that you've found the met-7.0 runtime of Grid-Stat is much
slower
> > than the met-6.0 version.  We did find/fix a Grid-Stat memory
allocation
> > issue that's posted to the MET website:
> >
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
> >
> > I have two questions...
> >
> > (1) What's the patch level of the code you're currently running?
Please
> > run "grid_stat --version" for the particular executable you're
running
> and
> > send me the result.  Should be something like this:
> >
> > MET Version: V7.0
> >
> > Repository: https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> >
> > Revision:         5607
> >
> > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
> >
> >
> > (2) Does this bugfix actually fix the problem you're seeing?  I'd
like to
> > test this out myself using your config file.  I can run 3
versions:
> >
> >    - met-6.0 with latest patches
> >
> >    - met-7.0 with no patches
> >
> >    - met-7.0 with latest patches
> >
> >
> > What model/analysis are you evaluating?  I might as well use the
same
> > resolution for my testing.
> >
> > Thanks,
> > John
> >
> > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Aug 29 09:01:08 2018: Request 86792 was acted upon.
> > > Transaction: Ticket created by benjamin.blake at noaa.gov
> > >        Queue: met_help
> > >      Subject: FSS calculation with grid stat
> > >        Owner: Nobody
> > >   Requestors: benjamin.blake at noaa.gov
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >
> > >
> > >
> > > Hi,
> > >
> > > I recently converted all of my MET configuration files from
using v6.0
> to
> > > v7.0.  One issue I'm now having is that my FSS calculations are
taking
> > much
> > > longer than they did when I was using METv6.0.  Currently it
takes
> ~15-20
> > > minutes to generate one .stat file.
> > >
> > > I've attached the grid stat configure file that I am now using
with
> v7.0
> > > (GridStatConfig_APCP03_FSS_href) and the configure file I was
using
> with
> > > v6.0 (GridStatConfig_03h_APCP_NBR).
> > >
> > > I modified cat_thresh for APCP A03 and I also reverted back from
SHAPE
> =
> > > CIRCLE to SHAPE = SQUARE for the neighborhood methods section,
but the
> > grid
> > > stat calculation still takes much longer than it used to.  I
also tried
> > > using the CONUS.poly file in shared/met/poly instead of my
CONUS.nc to
> > see
> > > if that would make a difference but it did not.  The v7.0
configure
> file
> > > has many more options in it so perhaps the slowdown is related
to
> > something
> > > I added in.
> > >
> > > Do you have any suggestions for why it might be running so
slowly?
> > >
> > > Thanks!
> > > Ben
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Julie Prestopnik
Time: Wed Aug 29 10:12:56 2018

I'll get met-7.0 updated on tide right now.  I'll follow up once it
update
is complete.

Julie

On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>
> Ben,
>
> Yeah, that'd be great.  If you send me a single input forecast and
analysis
> file for comparison, I'll run those 3 versions of grid-stat using
your
> configuration and quantify the timing differences.
>
> You can either point me to them on theia or post them to our
anonymous ftp
> site:
>    https://dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
> On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >
> > Hi John,
> >
> > Thanks for letting me know about the grid stat memory allocation
issue.
> I
> > am currently pointing to Julie's met/7.0 installation on WCOSS
(Tide),
> and
> > here is the output of grid_stat --version:
> >
> > MET Version:    V7.0
> > Repository:
> > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > Revision:    5483
> > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun 2018)
> >
> > Since the grid stat fix was put in place on July 24th the version
I'm
> using
> > likely doesn't have that fix, so I'm not sure if the bug fix will
resolve
> > the issue or not.  I could try pointing to a newer version of MET.
I
> > should also let you know that grid stat for the other line types
does not
> > seem to be slower, I only see a noticeable slowdown with the
NBRCNT line
> > type.
> >
> > I am evaluating the deterministic members of the HREFv2 and the
NCAR
> > ensemble as well as their respective means.  I verify them on NCEP
grid
> 227
> > which is a ~5-km grid (5.079 km).  I can send you some sample
files for
> > your testing if that would be easiest.
> >
> > Thanks!
> >
> > Ben
> >
> >
> > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Hi Ben,
> > >
> > > I see that you've found the met-7.0 runtime of Grid-Stat is much
slower
> > > than the met-6.0 version.  We did find/fix a Grid-Stat memory
> allocation
> > > issue that's posted to the MET website:
> > >
>
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
> > >
> > > I have two questions...
> > >
> > > (1) What's the patch level of the code you're currently running?
> Please
> > > run "grid_stat --version" for the particular executable you're
running
> > and
> > > send me the result.  Should be something like this:
> > >
> > > MET Version: V7.0
> > >
> > > Repository:
> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > >
> > > Revision:         5607
> > >
> > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
> > >
> > >
> > > (2) Does this bugfix actually fix the problem you're seeing?
I'd like
> to
> > > test this out myself using your config file.  I can run 3
versions:
> > >
> > >    - met-6.0 with latest patches
> > >
> > >    - met-7.0 with no patches
> > >
> > >    - met-7.0 with latest patches
> > >
> > >
> > > What model/analysis are you evaluating?  I might as well use the
same
> > > resolution for my testing.
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Aug 29 09:01:08 2018: Request 86792 was acted upon.
> > > > Transaction: Ticket created by benjamin.blake at noaa.gov
> > > >        Queue: met_help
> > > >      Subject: FSS calculation with grid stat
> > > >        Owner: Nobody
> > > >   Requestors: benjamin.blake at noaa.gov
> > > >       Status: new
> > > >  Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > > I recently converted all of my MET configuration files from
using
> v6.0
> > to
> > > > v7.0.  One issue I'm now having is that my FSS calculations
are
> taking
> > > much
> > > > longer than they did when I was using METv6.0.  Currently it
takes
> > ~15-20
> > > > minutes to generate one .stat file.
> > > >
> > > > I've attached the grid stat configure file that I am now using
with
> > v7.0
> > > > (GridStatConfig_APCP03_FSS_href) and the configure file I was
using
> > with
> > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> > > >
> > > > I modified cat_thresh for APCP A03 and I also reverted back
from
> SHAPE
> > =
> > > > CIRCLE to SHAPE = SQUARE for the neighborhood methods section,
but
> the
> > > grid
> > > > stat calculation still takes much longer than it used to.  I
also
> tried
> > > > using the CONUS.poly file in shared/met/poly instead of my
CONUS.nc
> to
> > > see
> > > > if that would make a difference but it did not.  The v7.0
configure
> > file
> > > > has many more options in it so perhaps the slowdown is related
to
> > > something
> > > > I added in.
> > > >
> > > > Do you have any suggestions for why it might be running so
slowly?
> > > >
> > > > Thanks!
> > > > Ben
> > > >
> > > >
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Wed Aug 29 10:21:51 2018

John,

Sure no problem, I copied a HiResW ARW forecast file and a CCPA file
from
WCOSS to Theia and they are located here:
/scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/testfiles

I also put the CONUS.nc mask file that I am using in that directory.
Using
that file or CONUS.poly didn't seem to make a difference though.

Here is the grid stat command I was using on WCOSS (after removing the
unrelevant WCOSS paths):
grid_stat hiresw.t00z.arw_5km.f03.conus.grib2 ccpa.2017052803.03h.gb2
GridStatConfig_APCP03_FSS_href -v 2

Julie - That'd be great, thanks!

Ben

On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT
<met_help at ucar.edu
> wrote:

> I'll get met-7.0 updated on tide right now.  I'll follow up once it
update
> is complete.
>
> Julie
>
> On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >
> > Ben,
> >
> > Yeah, that'd be great.  If you send me a single input forecast and
> analysis
> > file for comparison, I'll run those 3 versions of grid-stat using
your
> > configuration and quantify the timing differences.
> >
> > You can either point me to them on theia or post them to our
anonymous
> ftp
> > site:
> >    https://dtcenter.org/met/users/support/met_help.php#ftp
> >
> > Thanks,
> > John
> >
> > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > >
> > > Hi John,
> > >
> > > Thanks for letting me know about the grid stat memory allocation
issue.
> > I
> > > am currently pointing to Julie's met/7.0 installation on WCOSS
(Tide),
> > and
> > > here is the output of grid_stat --version:
> > >
> > > MET Version:    V7.0
> > > Repository:
> > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > > Revision:    5483
> > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun 2018)
> > >
> > > Since the grid stat fix was put in place on July 24th the
version I'm
> > using
> > > likely doesn't have that fix, so I'm not sure if the bug fix
will
> resolve
> > > the issue or not.  I could try pointing to a newer version of
MET.  I
> > > should also let you know that grid stat for the other line types
does
> not
> > > seem to be slower, I only see a noticeable slowdown with the
NBRCNT
> line
> > > type.
> > >
> > > I am evaluating the deterministic members of the HREFv2 and the
NCAR
> > > ensemble as well as their respective means.  I verify them on
NCEP grid
> > 227
> > > which is a ~5-km grid (5.079 km).  I can send you some sample
files for
> > > your testing if that would be easiest.
> > >
> > > Thanks!
> > >
> > > Ben
> > >
> > >
> > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Hi Ben,
> > > >
> > > > I see that you've found the met-7.0 runtime of Grid-Stat is
much
> slower
> > > > than the met-6.0 version.  We did find/fix a Grid-Stat memory
> > allocation
> > > > issue that's posted to the MET website:
> > > >
> >
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
> > > >
> > > > I have two questions...
> > > >
> > > > (1) What's the patch level of the code you're currently
running?
> > Please
> > > > run "grid_stat --version" for the particular executable you're
> running
> > > and
> > > > send me the result.  Should be something like this:
> > > >
> > > > MET Version: V7.0
> > > >
> > > > Repository:
> > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > > >
> > > > Revision:         5607
> > > >
> > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
> > > >
> > > >
> > > > (2) Does this bugfix actually fix the problem you're seeing?
I'd
> like
> > to
> > > > test this out myself using your config file.  I can run 3
versions:
> > > >
> > > >    - met-6.0 with latest patches
> > > >
> > > >    - met-7.0 with no patches
> > > >
> > > >    - met-7.0 with latest patches
> > > >
> > > >
> > > > What model/analysis are you evaluating?  I might as well use
the same
> > > > resolution for my testing.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA
Affiliate via
> RT
> > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Aug 29 09:01:08 2018: Request 86792 was acted upon.
> > > > > Transaction: Ticket created by benjamin.blake at noaa.gov
> > > > >        Queue: met_help
> > > > >      Subject: FSS calculation with grid stat
> > > > >        Owner: Nobody
> > > > >   Requestors: benjamin.blake at noaa.gov
> > > > >       Status: new
> > > > >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > I recently converted all of my MET configuration files from
using
> > v6.0
> > > to
> > > > > v7.0.  One issue I'm now having is that my FSS calculations
are
> > taking
> > > > much
> > > > > longer than they did when I was using METv6.0.  Currently it
takes
> > > ~15-20
> > > > > minutes to generate one .stat file.
> > > > >
> > > > > I've attached the grid stat configure file that I am now
using with
> > > v7.0
> > > > > (GridStatConfig_APCP03_FSS_href) and the configure file I
was
> using
> > > with
> > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> > > > >
> > > > > I modified cat_thresh for APCP A03 and I also reverted back
from
> > SHAPE
> > > =
> > > > > CIRCLE to SHAPE = SQUARE for the neighborhood methods
section, but
> > the
> > > > grid
> > > > > stat calculation still takes much longer than it used to.  I
also
> > tried
> > > > > using the CONUS.poly file in shared/met/poly instead of my
CONUS.nc
> > to
> > > > see
> > > > > if that would make a difference but it did not.  The v7.0
configure
> > > file
> > > > > has many more options in it so perhaps the slowdown is
related to
> > > > something
> > > > > I added in.
> > > > >
> > > > > Do you have any suggestions for why it might be running so
slowly?
> > > > >
> > > > > Thanks!
> > > > > Ben
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Julie Prestopnik
Time: Wed Aug 29 10:30:00 2018

Hi Ben.

The version of met-7.0 has now been updated on tide:

> MET Version: V7.0
>
> Repository: https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
>
> Revision: 5607
>
> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
>

I will update on surge once I devonprod access is returned.

Julie


On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>
> John,
>
> Sure no problem, I copied a HiResW ARW forecast file and a CCPA file
from
> WCOSS to Theia and they are located here:
> /scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/testfiles
>
> I also put the CONUS.nc mask file that I am using in that directory.
Using
> that file or CONUS.poly didn't seem to make a difference though.
>
> Here is the grid stat command I was using on WCOSS (after removing
the
> unrelevant WCOSS paths):
> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
ccpa.2017052803.03h.gb2
> GridStatConfig_APCP03_FSS_href -v 2
>
> Julie - That'd be great, thanks!
>
> Ben
>
> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT <
> met_help at ucar.edu
> > wrote:
>
> > I'll get met-7.0 updated on tide right now.  I'll follow up once
it
> update
> > is complete.
> >
> > Julie
> >
> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > >
> > > Ben,
> > >
> > > Yeah, that'd be great.  If you send me a single input forecast
and
> > analysis
> > > file for comparison, I'll run those 3 versions of grid-stat
using your
> > > configuration and quantify the timing differences.
> > >
> > > You can either point me to them on theia or post them to our
anonymous
> > ftp
> > > site:
> > >    https://dtcenter.org/met/users/support/met_help.php#ftp
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>
> > > >
> > > > Hi John,
> > > >
> > > > Thanks for letting me know about the grid stat memory
allocation
> issue.
> > > I
> > > > am currently pointing to Julie's met/7.0 installation on WCOSS
> (Tide),
> > > and
> > > > here is the output of grid_stat --version:
> > > >
> > > > MET Version:    V7.0
> > > > Repository:
> > > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > > > Revision:    5483
> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun 2018)
> > > >
> > > > Since the grid stat fix was put in place on July 24th the
version I'm
> > > using
> > > > likely doesn't have that fix, so I'm not sure if the bug fix
will
> > resolve
> > > > the issue or not.  I could try pointing to a newer version of
MET.  I
> > > > should also let you know that grid stat for the other line
types does
> > not
> > > > seem to be slower, I only see a noticeable slowdown with the
NBRCNT
> > line
> > > > type.
> > > >
> > > > I am evaluating the deterministic members of the HREFv2 and
the NCAR
> > > > ensemble as well as their respective means.  I verify them on
NCEP
> grid
> > > 227
> > > > which is a ~5-km grid (5.079 km).  I can send you some sample
files
> for
> > > > your testing if that would be easiest.
> > > >
> > > > Thanks!
> > > >
> > > > Ben
> > > >
> > > >
> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Hi Ben,
> > > > >
> > > > > I see that you've found the met-7.0 runtime of Grid-Stat is
much
> > slower
> > > > > than the met-6.0 version.  We did find/fix a Grid-Stat
memory
> > > allocation
> > > > > issue that's posted to the MET website:
> > > > >
> > >
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
> > > > >
> > > > > I have two questions...
> > > > >
> > > > > (1) What's the patch level of the code you're currently
running?
> > > Please
> > > > > run "grid_stat --version" for the particular executable
you're
> > running
> > > > and
> > > > > send me the result.  Should be something like this:
> > > > >
> > > > > MET Version: V7.0
> > > > >
> > > > > Repository:
> > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > > > >
> > > > > Revision:         5607
> > > > >
> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
> > > > >
> > > > >
> > > > > (2) Does this bugfix actually fix the problem you're seeing?
I'd
> > like
> > > to
> > > > > test this out myself using your config file.  I can run 3
versions:
> > > > >
> > > > >    - met-6.0 with latest patches
> > > > >
> > > > >    - met-7.0 with no patches
> > > > >
> > > > >    - met-7.0 with latest patches
> > > > >
> > > > >
> > > > > What model/analysis are you evaluating?  I might as well use
the
> same
> > > > > resolution for my testing.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA
Affiliate via
> > RT
> > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was acted upon.
> > > > > > Transaction: Ticket created by benjamin.blake at noaa.gov
> > > > > >        Queue: met_help
> > > > > >      Subject: FSS calculation with grid stat
> > > > > >        Owner: Nobody
> > > > > >   Requestors: benjamin.blake at noaa.gov
> > > > > >       Status: new
> > > > > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I recently converted all of my MET configuration files
from using
> > > v6.0
> > > > to
> > > > > > v7.0.  One issue I'm now having is that my FSS
calculations are
> > > taking
> > > > > much
> > > > > > longer than they did when I was using METv6.0.  Currently
it
> takes
> > > > ~15-20
> > > > > > minutes to generate one .stat file.
> > > > > >
> > > > > > I've attached the grid stat configure file that I am now
using
> with
> > > > v7.0
> > > > > > (GridStatConfig_APCP03_FSS_href) and the configure file I
was
> > using
> > > > with
> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> > > > > >
> > > > > > I modified cat_thresh for APCP A03 and I also reverted
back from
> > > SHAPE
> > > > =
> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood methods
section,
> but
> > > the
> > > > > grid
> > > > > > stat calculation still takes much longer than it used to.
I also
> > > tried
> > > > > > using the CONUS.poly file in shared/met/poly instead of my
> CONUS.nc
> > > to
> > > > > see
> > > > > > if that would make a difference but it did not.  The v7.0
> configure
> > > > file
> > > > > > has many more options in it so perhaps the slowdown is
related to
> > > > > something
> > > > > > I added in.
> > > > > >
> > > > > > Do you have any suggestions for why it might be running so
> slowly?
> > > > > >
> > > > > > Thanks!
> > > > > > Ben
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Julie Prestopnik
Time: Wed Aug 29 10:31:34 2018

Pardon, I meant "luna" not "surge".  :)

Julie

On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik <jpresto at ucar.edu>
wrote:

> Hi Ben.
>
> The version of met-7.0 has now been updated on tide:
>
>> MET Version: V7.0
>>
>> Repository: https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
>>
>> Revision: 5607
>>
>> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
>>
>
> I will update on surge once I devonprod access is returned.
>
> Julie
>
>
> On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>>
>> John,
>>
>> Sure no problem, I copied a HiResW ARW forecast file and a CCPA
file from
>> WCOSS to Theia and they are located here:
>> /scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/testfiles
>>
>> I also put the CONUS.nc mask file that I am using in that
directory.
>> Using
>> that file or CONUS.poly didn't seem to make a difference though.
>>
>> Here is the grid stat command I was using on WCOSS (after removing
the
>> unrelevant WCOSS paths):
>> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
ccpa.2017052803.03h.gb2
>> GridStatConfig_APCP03_FSS_href -v 2
>>
>> Julie - That'd be great, thanks!
>>
>> Ben
>>
>> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT <
>> met_help at ucar.edu
>> > wrote:
>>
>> > I'll get met-7.0 updated on tide right now.  I'll follow up once
it
>> update
>> > is complete.
>> >
>> > Julie
>> >
>> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>> > >
>> > > Ben,
>> > >
>> > > Yeah, that'd be great.  If you send me a single input forecast
and
>> > analysis
>> > > file for comparison, I'll run those 3 versions of grid-stat
using your
>> > > configuration and quantify the timing differences.
>> > >
>> > > You can either point me to them on theia or post them to our
anonymous
>> > ftp
>> > > site:
>> > >    https://dtcenter.org/met/users/support/met_help.php#ftp
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA
Affiliate via
>> RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>
>> > > >
>> > > > Hi John,
>> > > >
>> > > > Thanks for letting me know about the grid stat memory
allocation
>> issue.
>> > > I
>> > > > am currently pointing to Julie's met/7.0 installation on
WCOSS
>> (Tide),
>> > > and
>> > > > here is the output of grid_stat --version:
>> > > >
>> > > > MET Version:    V7.0
>> > > > Repository:
>> > > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
>> > > > Revision:    5483
>> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun 2018)
>> > > >
>> > > > Since the grid stat fix was put in place on July 24th the
version
>> I'm
>> > > using
>> > > > likely doesn't have that fix, so I'm not sure if the bug fix
will
>> > resolve
>> > > > the issue or not.  I could try pointing to a newer version of
MET.
>> I
>> > > > should also let you know that grid stat for the other line
types
>> does
>> > not
>> > > > seem to be slower, I only see a noticeable slowdown with the
NBRCNT
>> > line
>> > > > type.
>> > > >
>> > > > I am evaluating the deterministic members of the HREFv2 and
the NCAR
>> > > > ensemble as well as their respective means.  I verify them on
NCEP
>> grid
>> > > 227
>> > > > which is a ~5-km grid (5.079 km).  I can send you some sample
files
>> for
>> > > > your testing if that would be easiest.
>> > > >
>> > > > Thanks!
>> > > >
>> > > > Ben
>> > > >
>> > > >
>> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > > Hi Ben,
>> > > > >
>> > > > > I see that you've found the met-7.0 runtime of Grid-Stat is
much
>> > slower
>> > > > > than the met-6.0 version.  We did find/fix a Grid-Stat
memory
>> > > allocation
>> > > > > issue that's posted to the MET website:
>> > > > >
>> > >
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
>> > > > >
>> > > > > I have two questions...
>> > > > >
>> > > > > (1) What's the patch level of the code you're currently
running?
>> > > Please
>> > > > > run "grid_stat --version" for the particular executable
you're
>> > running
>> > > > and
>> > > > > send me the result.  Should be something like this:
>> > > > >
>> > > > > MET Version: V7.0
>> > > > >
>> > > > > Repository:
>> > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
>> > > > >
>> > > > > Revision:         5607
>> > > > >
>> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
>> > > > >
>> > > > >
>> > > > > (2) Does this bugfix actually fix the problem you're
seeing?  I'd
>> > like
>> > > to
>> > > > > test this out myself using your config file.  I can run 3
>> versions:
>> > > > >
>> > > > >    - met-6.0 with latest patches
>> > > > >
>> > > > >    - met-7.0 with no patches
>> > > > >
>> > > > >    - met-7.0 with latest patches
>> > > > >
>> > > > >
>> > > > > What model/analysis are you evaluating?  I might as well
use the
>> same
>> > > > > resolution for my testing.
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA
Affiliate
>> via
>> > RT
>> > > <
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > > >
>> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was acted upon.
>> > > > > > Transaction: Ticket created by benjamin.blake at noaa.gov
>> > > > > >        Queue: met_help
>> > > > > >      Subject: FSS calculation with grid stat
>> > > > > >        Owner: Nobody
>> > > > > >   Requestors: benjamin.blake at noaa.gov
>> > > > > >       Status: new
>> > > > > >  Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> > > > >
>> > > > > >
>> > > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > I recently converted all of my MET configuration files
from
>> using
>> > > v6.0
>> > > > to
>> > > > > > v7.0.  One issue I'm now having is that my FSS
calculations are
>> > > taking
>> > > > > much
>> > > > > > longer than they did when I was using METv6.0.  Currently
it
>> takes
>> > > > ~15-20
>> > > > > > minutes to generate one .stat file.
>> > > > > >
>> > > > > > I've attached the grid stat configure file that I am now
using
>> with
>> > > > v7.0
>> > > > > > (GridStatConfig_APCP03_FSS_href) and the configure file I
was
>> > using
>> > > > with
>> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
>> > > > > >
>> > > > > > I modified cat_thresh for APCP A03 and I also reverted
back from
>> > > SHAPE
>> > > > =
>> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood methods
section,
>> but
>> > > the
>> > > > > grid
>> > > > > > stat calculation still takes much longer than it used to.
I
>> also
>> > > tried
>> > > > > > using the CONUS.poly file in shared/met/poly instead of
my
>> CONUS.nc
>> > > to
>> > > > > see
>> > > > > > if that would make a difference but it did not.  The v7.0
>> configure
>> > > > file
>> > > > > > has many more options in it so perhaps the slowdown is
related
>> to
>> > > > > something
>> > > > > > I added in.
>> > > > > >
>> > > > > > Do you have any suggestions for why it might be running
so
>> slowly?
>> > > > > >
>> > > > > > Thanks!
>> > > > > > Ben
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Wed Aug 29 10:33:33 2018

Hi Julie,

Ok great, thanks.  I'll try running grid stat with the updated
version.
And yeah, I figured you meant Luna. :)

Ben

On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT
<met_help at ucar.edu
> wrote:

> Pardon, I meant "luna" not "surge".  :)
>
> Julie
>
> On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik <jpresto at ucar.edu>
> wrote:
>
> > Hi Ben.
> >
> > The version of met-7.0 has now been updated on tide:
> >
> >> MET Version: V7.0
> >>
> >> Repository: https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_
> bugfix
> >>
> >> Revision: 5607
> >>
> >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
> >>
> >
> > I will update on surge once I devonprod access is returned.
> >
> > Julie
> >
> >
> > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >>
> >> John,
> >>
> >> Sure no problem, I copied a HiResW ARW forecast file and a CCPA
file
> from
> >> WCOSS to Theia and they are located here:
> >> /scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/testfiles
> >>
> >> I also put the CONUS.nc mask file that I am using in that
directory.
> >> Using
> >> that file or CONUS.poly didn't seem to make a difference though.
> >>
> >> Here is the grid stat command I was using on WCOSS (after
removing the
> >> unrelevant WCOSS paths):
> >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
ccpa.2017052803.03h.gb2
> >> GridStatConfig_APCP03_FSS_href -v 2
> >>
> >> Julie - That'd be great, thanks!
> >>
> >> Ben
> >>
> >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT <
> >> met_help at ucar.edu
> >> > wrote:
> >>
> >> > I'll get met-7.0 updated on tide right now.  I'll follow up
once it
> >> update
> >> > is complete.
> >> >
> >> > Julie
> >> >
> >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>
> >> > >
> >> > > Ben,
> >> > >
> >> > > Yeah, that'd be great.  If you send me a single input
forecast and
> >> > analysis
> >> > > file for comparison, I'll run those 3 versions of grid-stat
using
> your
> >> > > configuration and quantify the timing differences.
> >> > >
> >> > > You can either point me to them on theia or post them to our
> anonymous
> >> > ftp
> >> > > site:
> >> > >    https://dtcenter.org/met/users/support/met_help.php#ftp
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA
Affiliate via
> >> RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >> > > >
> >> > > > Hi John,
> >> > > >
> >> > > > Thanks for letting me know about the grid stat memory
allocation
> >> issue.
> >> > > I
> >> > > > am currently pointing to Julie's met/7.0 installation on
WCOSS
> >> (Tide),
> >> > > and
> >> > > > here is the output of grid_stat --version:
> >> > > >
> >> > > > MET Version:    V7.0
> >> > > > Repository:
> >> > > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> >> > > > Revision:    5483
> >> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun
2018)
> >> > > >
> >> > > > Since the grid stat fix was put in place on July 24th the
version
> >> I'm
> >> > > using
> >> > > > likely doesn't have that fix, so I'm not sure if the bug
fix will
> >> > resolve
> >> > > > the issue or not.  I could try pointing to a newer version
of MET.
> >> I
> >> > > > should also let you know that grid stat for the other line
types
> >> does
> >> > not
> >> > > > seem to be slower, I only see a noticeable slowdown with
the
> NBRCNT
> >> > line
> >> > > > type.
> >> > > >
> >> > > > I am evaluating the deterministic members of the HREFv2 and
the
> NCAR
> >> > > > ensemble as well as their respective means.  I verify them
on NCEP
> >> grid
> >> > > 227
> >> > > > which is a ~5-km grid (5.079 km).  I can send you some
sample
> files
> >> for
> >> > > > your testing if that would be easiest.
> >> > > >
> >> > > > Thanks!
> >> > > >
> >> > > > Ben
> >> > > >
> >> > > >
> >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway via RT
<
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > > Hi Ben,
> >> > > > >
> >> > > > > I see that you've found the met-7.0 runtime of Grid-Stat
is much
> >> > slower
> >> > > > > than the met-6.0 version.  We did find/fix a Grid-Stat
memory
> >> > > allocation
> >> > > > > issue that's posted to the MET website:
> >> > > > >
> >> > > https://dtcenter.org/met/users/support/known_issues/
> METv7.0/index.php
> >> > > > >
> >> > > > > I have two questions...
> >> > > > >
> >> > > > > (1) What's the patch level of the code you're currently
running?
> >> > > Please
> >> > > > > run "grid_stat --version" for the particular executable
you're
> >> > running
> >> > > > and
> >> > > > > send me the result.  Should be something like this:
> >> > > > >
> >> > > > > MET Version: V7.0
> >> > > > >
> >> > > > > Repository:
> >> > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> >> > > > >
> >> > > > > Revision:         5607
> >> > > > >
> >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
> >> > > > >
> >> > > > >
> >> > > > > (2) Does this bugfix actually fix the problem you're
seeing?
> I'd
> >> > like
> >> > > to
> >> > > > > test this out myself using your config file.  I can run 3
> >> versions:
> >> > > > >
> >> > > > >    - met-6.0 with latest patches
> >> > > > >
> >> > > > >    - met-7.0 with no patches
> >> > > > >
> >> > > > >    - met-7.0 with latest patches
> >> > > > >
> >> > > > >
> >> > > > > What model/analysis are you evaluating?  I might as well
use the
> >> same
> >> > > > > resolution for my testing.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > John
> >> > > > >
> >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA
Affiliate
> >> via
> >> > RT
> >> > > <
> >> > > > > met_help at ucar.edu> wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was acted upon.
> >> > > > > > Transaction: Ticket created by benjamin.blake at noaa.gov
> >> > > > > >        Queue: met_help
> >> > > > > >      Subject: FSS calculation with grid stat
> >> > > > > >        Owner: Nobody
> >> > > > > >   Requestors: benjamin.blake at noaa.gov
> >> > > > > >       Status: new
> >> > > > > >  Ticket <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >> > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > I recently converted all of my MET configuration files
from
> >> using
> >> > > v6.0
> >> > > > to
> >> > > > > > v7.0.  One issue I'm now having is that my FSS
calculations
> are
> >> > > taking
> >> > > > > much
> >> > > > > > longer than they did when I was using METv6.0.
Currently it
> >> takes
> >> > > > ~15-20
> >> > > > > > minutes to generate one .stat file.
> >> > > > > >
> >> > > > > > I've attached the grid stat configure file that I am
now using
> >> with
> >> > > > v7.0
> >> > > > > > (GridStatConfig_APCP03_FSS_href) and the configure file
I was
> >> > using
> >> > > > with
> >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> >> > > > > >
> >> > > > > > I modified cat_thresh for APCP A03 and I also reverted
back
> from
> >> > > SHAPE
> >> > > > =
> >> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood methods
section,
> >> but
> >> > > the
> >> > > > > grid
> >> > > > > > stat calculation still takes much longer than it used
to.  I
> >> also
> >> > > tried
> >> > > > > > using the CONUS.poly file in shared/met/poly instead of
my
> >> CONUS.nc
> >> > > to
> >> > > > > see
> >> > > > > > if that would make a difference but it did not.  The
v7.0
> >> configure
> >> > > > file
> >> > > > > > has many more options in it so perhaps the slowdown is
related
> >> to
> >> > > > > something
> >> > > > > > I added in.
> >> > > > > >
> >> > > > > > Do you have any suggestions for why it might be running
so
> >> slowly?
> >> > > > > >
> >> > > > > > Thanks!
> >> > > > > > Ben
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Wed Aug 29 11:13:19 2018

Hi John,

I ran a test using Julie's updated MET v7.0 installation and my
runtime was
still ~15-20 minutes to generate one .stat file.

I also loaded MET v6.0 on Tide with
/usrx/local/dev/modulefiles/met/6.0 and
used my older configuration file which I sent you, and the .stat file
was
generated in less than a minute.  Let me know if you are able to
reproduce
this behavior.  Maybe I added something in the newer configuration
file
that needs to be removed?

Thanks,
Ben

On Wed, Aug 29, 2018 at 12:33 PM, Benjamin Blake - NOAA Affiliate <
benjamin.blake at noaa.gov> wrote:

> Hi Julie,
>
> Ok great, thanks.  I'll try running grid stat with the updated
version.
> And yeah, I figured you meant Luna. :)
>
> Ben
>
> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT <
> met_help at ucar.edu> wrote:
>
>> Pardon, I meant "luna" not "surge".  :)
>>
>> Julie
>>
>> On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik
<jpresto at ucar.edu>
>> wrote:
>>
>> > Hi Ben.
>> >
>> > The version of met-7.0 has now been updated on tide:
>> >
>> >> MET Version: V7.0
>> >>
>> >> Repository: https://svn-met-dev.cgd.ucar.e
>> du/branches/met/met-7.0_bugfix
>> >>
>> >> Revision: 5607
>> >>
>> >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
>> >>
>> >
>> > I will update on surge once I devonprod access is returned.
>> >
>> > Julie
>> >
>> >
>> > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA Affiliate
via RT
>> <
>> > met_help at ucar.edu> wrote:
>> >
>> >>
>> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>> >>
>> >> John,
>> >>
>> >> Sure no problem, I copied a HiResW ARW forecast file and a CCPA
file
>> from
>> >> WCOSS to Theia and they are located here:
>> >> /scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/testfiles
>> >>
>> >> I also put the CONUS.nc mask file that I am using in that
directory.
>> >> Using
>> >> that file or CONUS.poly didn't seem to make a difference though.
>> >>
>> >> Here is the grid stat command I was using on WCOSS (after
removing the
>> >> unrelevant WCOSS paths):
>> >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
ccpa.2017052803.03h.gb2
>> >> GridStatConfig_APCP03_FSS_href -v 2
>> >>
>> >> Julie - That'd be great, thanks!
>> >>
>> >> Ben
>> >>
>> >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT <
>> >> met_help at ucar.edu
>> >> > wrote:
>> >>
>> >> > I'll get met-7.0 updated on tide right now.  I'll follow up
once it
>> >> update
>> >> > is complete.
>> >> >
>> >> > Julie
>> >> >
>> >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via RT <
>> >> > met_help at ucar.edu> wrote:
>> >> >
>> >> > >
>> >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>> >> > >
>> >> > > Ben,
>> >> > >
>> >> > > Yeah, that'd be great.  If you send me a single input
forecast and
>> >> > analysis
>> >> > > file for comparison, I'll run those 3 versions of grid-stat
using
>> your
>> >> > > configuration and quantify the timing differences.
>> >> > >
>> >> > > You can either point me to them on theia or post them to our
>> anonymous
>> >> > ftp
>> >> > > site:
>> >> > >    https://dtcenter.org/met/users/support/met_help.php#ftp
>> >> > >
>> >> > > Thanks,
>> >> > > John
>> >> > >
>> >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA
Affiliate
>> via
>> >> RT <
>> >> > > met_help at ucar.edu> wrote:
>> >> > >
>> >> > > >
>> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>> >> > > >
>> >> > > > Hi John,
>> >> > > >
>> >> > > > Thanks for letting me know about the grid stat memory
allocation
>> >> issue.
>> >> > > I
>> >> > > > am currently pointing to Julie's met/7.0 installation on
WCOSS
>> >> (Tide),
>> >> > > and
>> >> > > > here is the output of grid_stat --version:
>> >> > > >
>> >> > > > MET Version:    V7.0
>> >> > > > Repository:
>> >> > > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
>> >> > > > Revision:    5483
>> >> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun
2018)
>> >> > > >
>> >> > > > Since the grid stat fix was put in place on July 24th the
version
>> >> I'm
>> >> > > using
>> >> > > > likely doesn't have that fix, so I'm not sure if the bug
fix will
>> >> > resolve
>> >> > > > the issue or not.  I could try pointing to a newer version
of
>> MET.
>> >> I
>> >> > > > should also let you know that grid stat for the other line
types
>> >> does
>> >> > not
>> >> > > > seem to be slower, I only see a noticeable slowdown with
the
>> NBRCNT
>> >> > line
>> >> > > > type.
>> >> > > >
>> >> > > > I am evaluating the deterministic members of the HREFv2
and the
>> NCAR
>> >> > > > ensemble as well as their respective means.  I verify them
on
>> NCEP
>> >> grid
>> >> > > 227
>> >> > > > which is a ~5-km grid (5.079 km).  I can send you some
sample
>> files
>> >> for
>> >> > > > your testing if that would be easiest.
>> >> > > >
>> >> > > > Thanks!
>> >> > > >
>> >> > > > Ben
>> >> > > >
>> >> > > >
>> >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway via
RT <
>> >> > > > met_help at ucar.edu> wrote:
>> >> > > >
>> >> > > > > Hi Ben,
>> >> > > > >
>> >> > > > > I see that you've found the met-7.0 runtime of Grid-Stat
is
>> much
>> >> > slower
>> >> > > > > than the met-6.0 version.  We did find/fix a Grid-Stat
memory
>> >> > > allocation
>> >> > > > > issue that's posted to the MET website:
>> >> > > > >
>> >> > > https://dtcenter.org/met/users/support/known_issues/METv7.0/
>> index.php
>> >> > > > >
>> >> > > > > I have two questions...
>> >> > > > >
>> >> > > > > (1) What's the patch level of the code you're currently
>> running?
>> >> > > Please
>> >> > > > > run "grid_stat --version" for the particular executable
you're
>> >> > running
>> >> > > > and
>> >> > > > > send me the result.  Should be something like this:
>> >> > > > >
>> >> > > > > MET Version: V7.0
>> >> > > > >
>> >> > > > > Repository:
>> >> > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
>> >> > > > >
>> >> > > > > Revision:         5607
>> >> > > > >
>> >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul
2018)
>> >> > > > >
>> >> > > > >
>> >> > > > > (2) Does this bugfix actually fix the problem you're
seeing?
>> I'd
>> >> > like
>> >> > > to
>> >> > > > > test this out myself using your config file.  I can run
3
>> >> versions:
>> >> > > > >
>> >> > > > >    - met-6.0 with latest patches
>> >> > > > >
>> >> > > > >    - met-7.0 with no patches
>> >> > > > >
>> >> > > > >    - met-7.0 with latest patches
>> >> > > > >
>> >> > > > >
>> >> > > > > What model/analysis are you evaluating?  I might as well
use
>> the
>> >> same
>> >> > > > > resolution for my testing.
>> >> > > > >
>> >> > > > > Thanks,
>> >> > > > > John
>> >> > > > >
>> >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA
Affiliate
>> >> via
>> >> > RT
>> >> > > <
>> >> > > > > met_help at ucar.edu> wrote:
>> >> > > > >
>> >> > > > > >
>> >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was acted
upon.
>> >> > > > > > Transaction: Ticket created by benjamin.blake at noaa.gov
>> >> > > > > >        Queue: met_help
>> >> > > > > >      Subject: FSS calculation with grid stat
>> >> > > > > >        Owner: Nobody
>> >> > > > > >   Requestors: benjamin.blake at noaa.gov
>> >> > > > > >       Status: new
>> >> > > > > >  Ticket <URL:
>> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> >> > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > Hi,
>> >> > > > > >
>> >> > > > > > I recently converted all of my MET configuration files
from
>> >> using
>> >> > > v6.0
>> >> > > > to
>> >> > > > > > v7.0.  One issue I'm now having is that my FSS
calculations
>> are
>> >> > > taking
>> >> > > > > much
>> >> > > > > > longer than they did when I was using METv6.0.
Currently it
>> >> takes
>> >> > > > ~15-20
>> >> > > > > > minutes to generate one .stat file.
>> >> > > > > >
>> >> > > > > > I've attached the grid stat configure file that I am
now
>> using
>> >> with
>> >> > > > v7.0
>> >> > > > > > (GridStatConfig_APCP03_FSS_href) and the configure
file I
>> was
>> >> > using
>> >> > > > with
>> >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
>> >> > > > > >
>> >> > > > > > I modified cat_thresh for APCP A03 and I also reverted
back
>> from
>> >> > > SHAPE
>> >> > > > =
>> >> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood methods
>> section,
>> >> but
>> >> > > the
>> >> > > > > grid
>> >> > > > > > stat calculation still takes much longer than it used
to.  I
>> >> also
>> >> > > tried
>> >> > > > > > using the CONUS.poly file in shared/met/poly instead
of my
>> >> CONUS.nc
>> >> > > to
>> >> > > > > see
>> >> > > > > > if that would make a difference but it did not.  The
v7.0
>> >> configure
>> >> > > > file
>> >> > > > > > has many more options in it so perhaps the slowdown is
>> related
>> >> to
>> >> > > > > something
>> >> > > > > > I added in.
>> >> > > > > >
>> >> > > > > > Do you have any suggestions for why it might be
running so
>> >> slowly?
>> >> > > > > >
>> >> > > > > > Thanks!
>> >> > > > > > Ben
>> >> > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > > >
>> >> > >
>> >> > >
>> >> >
>> >> >
>> >>
>> >>
>>
>>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: John Halley Gotway
Time: Wed Aug 29 11:17:39 2018

Ben,

Thanks for sending the sample data.  Here's how long this test case
took on
my machine using 2 versions of MET:
met-6.0 took 01:24.
met-7.0 with patches took 10:49.

So unfortunately (as you just emailed me), the patch won't fix this.

The difference in versions is this... In met-6.0, we only supported
"square" neighborhoods.  In met-7.0, we added support for "circular"
neighborhoods.  In doing so, we leveraged existing code from another
part
of RAL.  However, as highlighted by your configuration, it slower...
especially for large neighborhood sizes.

Let me look more closely at the code to see how we can address this
issue.

Thanks,
John

On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>
> Hi Julie,
>
> Ok great, thanks.  I'll try running grid stat with the updated
version.
> And yeah, I figured you meant Luna. :)
>
> Ben
>
> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT <
> met_help at ucar.edu
> > wrote:
>
> > Pardon, I meant "luna" not "surge".  :)
> >
> > Julie
> >
> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik
<jpresto at ucar.edu>
> > wrote:
> >
> > > Hi Ben.
> > >
> > > The version of met-7.0 has now been updated on tide:
> > >
> > >> MET Version: V7.0
> > >>
> > >> Repository: https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_
> > bugfix
> > >>
> > >> Revision: 5607
> > >>
> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
> > >>
> > >
> > > I will update on surge once I devonprod access is returned.
> > >
> > > Julie
> > >
> > >
> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > >>
> > >> John,
> > >>
> > >> Sure no problem, I copied a HiResW ARW forecast file and a CCPA
file
> > from
> > >> WCOSS to Theia and they are located here:
> > >> /scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/testfiles
> > >>
> > >> I also put the CONUS.nc mask file that I am using in that
directory.
> > >> Using
> > >> that file or CONUS.poly didn't seem to make a difference
though.
> > >>
> > >> Here is the grid stat command I was using on WCOSS (after
removing the
> > >> unrelevant WCOSS paths):
> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
ccpa.2017052803.03h.gb2
> > >> GridStatConfig_APCP03_FSS_href -v 2
> > >>
> > >> Julie - That'd be great, thanks!
> > >>
> > >> Ben
> > >>
> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT <
> > >> met_help at ucar.edu
> > >> > wrote:
> > >>
> > >> > I'll get met-7.0 updated on tide right now.  I'll follow up
once it
> > >> update
> > >> > is complete.
> > >> >
> > >> > Julie
> > >> >
> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > >> > >
> > >> > > Ben,
> > >> > >
> > >> > > Yeah, that'd be great.  If you send me a single input
forecast and
> > >> > analysis
> > >> > > file for comparison, I'll run those 3 versions of grid-stat
using
> > your
> > >> > > configuration and quantify the timing differences.
> > >> > >
> > >> > > You can either point me to them on theia or post them to
our
> > anonymous
> > >> > ftp
> > >> > > site:
> > >> > >    https://dtcenter.org/met/users/support/met_help.php#ftp
> > >> > >
> > >> > > Thanks,
> > >> > > John
> > >> > >
> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA
Affiliate
> via
> > >> RT <
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > > >
> > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > >> > > >
> > >> > > > Hi John,
> > >> > > >
> > >> > > > Thanks for letting me know about the grid stat memory
allocation
> > >> issue.
> > >> > > I
> > >> > > > am currently pointing to Julie's met/7.0 installation on
WCOSS
> > >> (Tide),
> > >> > > and
> > >> > > > here is the output of grid_stat --version:
> > >> > > >
> > >> > > > MET Version:    V7.0
> > >> > > > Repository:
> > >> > > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> > >> > > > Revision:    5483
> > >> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun
2018)
> > >> > > >
> > >> > > > Since the grid stat fix was put in place on July 24th the
> version
> > >> I'm
> > >> > > using
> > >> > > > likely doesn't have that fix, so I'm not sure if the bug
fix
> will
> > >> > resolve
> > >> > > > the issue or not.  I could try pointing to a newer
version of
> MET.
> > >> I
> > >> > > > should also let you know that grid stat for the other
line types
> > >> does
> > >> > not
> > >> > > > seem to be slower, I only see a noticeable slowdown with
the
> > NBRCNT
> > >> > line
> > >> > > > type.
> > >> > > >
> > >> > > > I am evaluating the deterministic members of the HREFv2
and the
> > NCAR
> > >> > > > ensemble as well as their respective means.  I verify
them on
> NCEP
> > >> grid
> > >> > > 227
> > >> > > > which is a ~5-km grid (5.079 km).  I can send you some
sample
> > files
> > >> for
> > >> > > > your testing if that would be easiest.
> > >> > > >
> > >> > > > Thanks!
> > >> > > >
> > >> > > > Ben
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway via
RT <
> > >> > > > met_help at ucar.edu> wrote:
> > >> > > >
> > >> > > > > Hi Ben,
> > >> > > > >
> > >> > > > > I see that you've found the met-7.0 runtime of Grid-
Stat is
> much
> > >> > slower
> > >> > > > > than the met-6.0 version.  We did find/fix a Grid-Stat
memory
> > >> > > allocation
> > >> > > > > issue that's posted to the MET website:
> > >> > > > >
> > >> > > https://dtcenter.org/met/users/support/known_issues/
> > METv7.0/index.php
> > >> > > > >
> > >> > > > > I have two questions...
> > >> > > > >
> > >> > > > > (1) What's the patch level of the code you're currently
> running?
> > >> > > Please
> > >> > > > > run "grid_stat --version" for the particular executable
you're
> > >> > running
> > >> > > > and
> > >> > > > > send me the result.  Should be something like this:
> > >> > > > >
> > >> > > > > MET Version: V7.0
> > >> > > > >
> > >> > > > > Repository:
> > >> > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> > >> > > > >
> > >> > > > > Revision:         5607
> > >> > > > >
> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul
2018)
> > >> > > > >
> > >> > > > >
> > >> > > > > (2) Does this bugfix actually fix the problem you're
seeing?
> > I'd
> > >> > like
> > >> > > to
> > >> > > > > test this out myself using your config file.  I can run
3
> > >> versions:
> > >> > > > >
> > >> > > > >    - met-6.0 with latest patches
> > >> > > > >
> > >> > > > >    - met-7.0 with no patches
> > >> > > > >
> > >> > > > >    - met-7.0 with latest patches
> > >> > > > >
> > >> > > > >
> > >> > > > > What model/analysis are you evaluating?  I might as
well use
> the
> > >> same
> > >> > > > > resolution for my testing.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > John
> > >> > > > >
> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA
> Affiliate
> > >> via
> > >> > RT
> > >> > > <
> > >> > > > > met_help at ucar.edu> wrote:
> > >> > > > >
> > >> > > > > >
> > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was acted
upon.
> > >> > > > > > Transaction: Ticket created by
benjamin.blake at noaa.gov
> > >> > > > > >        Queue: met_help
> > >> > > > > >      Subject: FSS calculation with grid stat
> > >> > > > > >        Owner: Nobody
> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
> > >> > > > > >       Status: new
> > >> > > > > >  Ticket <URL:
> > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > >> > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Hi,
> > >> > > > > >
> > >> > > > > > I recently converted all of my MET configuration
files from
> > >> using
> > >> > > v6.0
> > >> > > > to
> > >> > > > > > v7.0.  One issue I'm now having is that my FSS
calculations
> > are
> > >> > > taking
> > >> > > > > much
> > >> > > > > > longer than they did when I was using METv6.0.
Currently it
> > >> takes
> > >> > > > ~15-20
> > >> > > > > > minutes to generate one .stat file.
> > >> > > > > >
> > >> > > > > > I've attached the grid stat configure file that I am
now
> using
> > >> with
> > >> > > > v7.0
> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and the configure
file I
> was
> > >> > using
> > >> > > > with
> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> > >> > > > > >
> > >> > > > > > I modified cat_thresh for APCP A03 and I also
reverted back
> > from
> > >> > > SHAPE
> > >> > > > =
> > >> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood methods
> section,
> > >> but
> > >> > > the
> > >> > > > > grid
> > >> > > > > > stat calculation still takes much longer than it used
to.  I
> > >> also
> > >> > > tried
> > >> > > > > > using the CONUS.poly file in shared/met/poly instead
of my
> > >> CONUS.nc
> > >> > > to
> > >> > > > > see
> > >> > > > > > if that would make a difference but it did not.  The
v7.0
> > >> configure
> > >> > > > file
> > >> > > > > > has many more options in it so perhaps the slowdown
is
> related
> > >> to
> > >> > > > > something
> > >> > > > > > I added in.
> > >> > > > > >
> > >> > > > > > Do you have any suggestions for why it might be
running so
> > >> slowly?
> > >> > > > > >
> > >> > > > > > Thanks!
> > >> > > > > > Ben
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >>
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Wed Aug 29 11:47:23 2018

John,

Ok, thanks for letting me know.  In the short term maybe I can just
use
v6.0 for running FSS since it has everything I need.

One suggestion for speeding up the runtime could be to use an fft
convolve
function.  I used those in Python before and they can be pretty fast.

Thanks,
Ben

On Wed, Aug 29, 2018 at 1:17 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Ben,
>
> Thanks for sending the sample data.  Here's how long this test case
took on
> my machine using 2 versions of MET:
> met-6.0 took 01:24.
> met-7.0 with patches took 10:49.
>
> So unfortunately (as you just emailed me), the patch won't fix this.
>
> The difference in versions is this... In met-6.0, we only supported
> "square" neighborhoods.  In met-7.0, we added support for "circular"
> neighborhoods.  In doing so, we leveraged existing code from another
part
> of RAL.  However, as highlighted by your configuration, it slower...
> especially for large neighborhood sizes.
>
> Let me look more closely at the code to see how we can address this
issue.
>
> Thanks,
> John
>
> On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >
> > Hi Julie,
> >
> > Ok great, thanks.  I'll try running grid stat with the updated
version.
> > And yeah, I figured you meant Luna. :)
> >
> > Ben
> >
> > On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Pardon, I meant "luna" not "surge".  :)
> > >
> > > Julie
> > >
> > > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik
<jpresto at ucar.edu>
> > > wrote:
> > >
> > > > Hi Ben.
> > > >
> > > > The version of met-7.0 has now been updated on tide:
> > > >
> > > >> MET Version: V7.0
> > > >>
> > > >> Repository: https://svn-met-
dev.cgd.ucar.edu/branches/met/met-7.0_
> > > bugfix
> > > >>
> > > >> Revision: 5607
> > > >>
> > > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
> > > >>
> > > >
> > > > I will update on surge once I devonprod access is returned.
> > > >
> > > > Julie
> > > >
> > > >
> > > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA
Affiliate via
> > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>
> > > >>
> > > >> John,
> > > >>
> > > >> Sure no problem, I copied a HiResW ARW forecast file and a
CCPA file
> > > from
> > > >> WCOSS to Theia and they are located here:
> > > >> /scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/testfiles
> > > >>
> > > >> I also put the CONUS.nc mask file that I am using in that
directory.
> > > >> Using
> > > >> that file or CONUS.poly didn't seem to make a difference
though.
> > > >>
> > > >> Here is the grid stat command I was using on WCOSS (after
removing
> the
> > > >> unrelevant WCOSS paths):
> > > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
> ccpa.2017052803.03h.gb2
> > > >> GridStatConfig_APCP03_FSS_href -v 2
> > > >>
> > > >> Julie - That'd be great, thanks!
> > > >>
> > > >> Ben
> > > >>
> > > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT <
> > > >> met_help at ucar.edu
> > > >> > wrote:
> > > >>
> > > >> > I'll get met-7.0 updated on tide right now.  I'll follow up
once
> it
> > > >> update
> > > >> > is complete.
> > > >> >
> > > >> > Julie
> > > >> >
> > > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via RT
<
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > > >> > >
> > > >> > > Ben,
> > > >> > >
> > > >> > > Yeah, that'd be great.  If you send me a single input
forecast
> and
> > > >> > analysis
> > > >> > > file for comparison, I'll run those 3 versions of grid-
stat
> using
> > > your
> > > >> > > configuration and quantify the timing differences.
> > > >> > >
> > > >> > > You can either point me to them on theia or post them to
our
> > > anonymous
> > > >> > ftp
> > > >> > > site:
> > > >> > >
https://dtcenter.org/met/users/support/met_help.php#ftp
> > > >> > >
> > > >> > > Thanks,
> > > >> > > John
> > > >> > >
> > > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA
Affiliate
> > via
> > > >> RT <
> > > >> > > met_help at ucar.edu> wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >
> > > >> > > >
> > > >> > > > Hi John,
> > > >> > > >
> > > >> > > > Thanks for letting me know about the grid stat memory
> allocation
> > > >> issue.
> > > >> > > I
> > > >> > > > am currently pointing to Julie's met/7.0 installation
on WCOSS
> > > >> (Tide),
> > > >> > > and
> > > >> > > > here is the output of grid_stat --version:
> > > >> > > >
> > > >> > > > MET Version:    V7.0
> > > >> > > > Repository:
> > > >> > > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> > > >> > > > Revision:    5483
> > > >> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun
2018)
> > > >> > > >
> > > >> > > > Since the grid stat fix was put in place on July 24th
the
> > version
> > > >> I'm
> > > >> > > using
> > > >> > > > likely doesn't have that fix, so I'm not sure if the
bug fix
> > will
> > > >> > resolve
> > > >> > > > the issue or not.  I could try pointing to a newer
version of
> > MET.
> > > >> I
> > > >> > > > should also let you know that grid stat for the other
line
> types
> > > >> does
> > > >> > not
> > > >> > > > seem to be slower, I only see a noticeable slowdown
with the
> > > NBRCNT
> > > >> > line
> > > >> > > > type.
> > > >> > > >
> > > >> > > > I am evaluating the deterministic members of the HREFv2
and
> the
> > > NCAR
> > > >> > > > ensemble as well as their respective means.  I verify
them on
> > NCEP
> > > >> grid
> > > >> > > 227
> > > >> > > > which is a ~5-km grid (5.079 km).  I can send you some
sample
> > > files
> > > >> for
> > > >> > > > your testing if that would be easiest.
> > > >> > > >
> > > >> > > > Thanks!
> > > >> > > >
> > > >> > > > Ben
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway
via RT <
> > > >> > > > met_help at ucar.edu> wrote:
> > > >> > > >
> > > >> > > > > Hi Ben,
> > > >> > > > >
> > > >> > > > > I see that you've found the met-7.0 runtime of Grid-
Stat is
> > much
> > > >> > slower
> > > >> > > > > than the met-6.0 version.  We did find/fix a Grid-
Stat
> memory
> > > >> > > allocation
> > > >> > > > > issue that's posted to the MET website:
> > > >> > > > >
> > > >> > > https://dtcenter.org/met/users/support/known_issues/
> > > METv7.0/index.php
> > > >> > > > >
> > > >> > > > > I have two questions...
> > > >> > > > >
> > > >> > > > > (1) What's the patch level of the code you're
currently
> > running?
> > > >> > > Please
> > > >> > > > > run "grid_stat --version" for the particular
executable
> you're
> > > >> > running
> > > >> > > > and
> > > >> > > > > send me the result.  Should be something like this:
> > > >> > > > >
> > > >> > > > > MET Version: V7.0
> > > >> > > > >
> > > >> > > > > Repository:
> > > >> > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> > > >> > > > >
> > > >> > > > > Revision:         5607
> > > >> > > > >
> > > >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul
2018)
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > (2) Does this bugfix actually fix the problem you're
seeing?
> > > I'd
> > > >> > like
> > > >> > > to
> > > >> > > > > test this out myself using your config file.  I can
run 3
> > > >> versions:
> > > >> > > > >
> > > >> > > > >    - met-6.0 with latest patches
> > > >> > > > >
> > > >> > > > >    - met-7.0 with no patches
> > > >> > > > >
> > > >> > > > >    - met-7.0 with latest patches
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > What model/analysis are you evaluating?  I might as
well use
> > the
> > > >> same
> > > >> > > > > resolution for my testing.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > John
> > > >> > > > >
> > > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA
> > Affiliate
> > > >> via
> > > >> > RT
> > > >> > > <
> > > >> > > > > met_help at ucar.edu> wrote:
> > > >> > > > >
> > > >> > > > > >
> > > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was acted
upon.
> > > >> > > > > > Transaction: Ticket created by
benjamin.blake at noaa.gov
> > > >> > > > > >        Queue: met_help
> > > >> > > > > >      Subject: FSS calculation with grid stat
> > > >> > > > > >        Owner: Nobody
> > > >> > > > > >   Requestors: benjamin.blake at noaa.gov
> > > >> > > > > >       Status: new
> > > >> > > > > >  Ticket <URL:
> > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > > >> > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Hi,
> > > >> > > > > >
> > > >> > > > > > I recently converted all of my MET configuration
files
> from
> > > >> using
> > > >> > > v6.0
> > > >> > > > to
> > > >> > > > > > v7.0.  One issue I'm now having is that my FSS
> calculations
> > > are
> > > >> > > taking
> > > >> > > > > much
> > > >> > > > > > longer than they did when I was using METv6.0.
Currently
> it
> > > >> takes
> > > >> > > > ~15-20
> > > >> > > > > > minutes to generate one .stat file.
> > > >> > > > > >
> > > >> > > > > > I've attached the grid stat configure file that I
am now
> > using
> > > >> with
> > > >> > > > v7.0
> > > >> > > > > > (GridStatConfig_APCP03_FSS_href) and the configure
file I
> > was
> > > >> > using
> > > >> > > > with
> > > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> > > >> > > > > >
> > > >> > > > > > I modified cat_thresh for APCP A03 and I also
reverted
> back
> > > from
> > > >> > > SHAPE
> > > >> > > > =
> > > >> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood
methods
> > section,
> > > >> but
> > > >> > > the
> > > >> > > > > grid
> > > >> > > > > > stat calculation still takes much longer than it
used
> to.  I
> > > >> also
> > > >> > > tried
> > > >> > > > > > using the CONUS.poly file in shared/met/poly
instead of my
> > > >> CONUS.nc
> > > >> > > to
> > > >> > > > > see
> > > >> > > > > > if that would make a difference but it did not.
The v7.0
> > > >> configure
> > > >> > > > file
> > > >> > > > > > has many more options in it so perhaps the slowdown
is
> > related
> > > >> to
> > > >> > > > > something
> > > >> > > > > > I added in.
> > > >> > > > > >
> > > >> > > > > > Do you have any suggestions for why it might be
running so
> > > >> slowly?
> > > >> > > > > >
> > > >> > > > > > Thanks!
> > > >> > > > > > Ben
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: John Halley Gotway
Time: Wed Aug 29 11:50:36 2018

Ben,

Looking closely at the code, here's the difference...

In met-6.0, for square neighborhoods, every time we increment 1 step
in the
x-direction, we decrement counts from the column we're discarding and
increment counts for the column we're adding.

In met-7.0, for square and circular neighborhoods, we reprocess the
whole
neighborhood each time since for circles, it isn't as simple as
dropping/adding a new column.

I agree that using an FFT instead would be preferable.  The question
is
whether we get the same result?  And whether or not it handles missing
data
well.

John

On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway <johnhg at ucar.edu>
wrote:

> Ben,
>
> Thanks for sending the sample data.  Here's how long this test case
took
> on my machine using 2 versions of MET:
> met-6.0 took 01:24.
> met-7.0 with patches took 10:49.
>
> So unfortunately (as you just emailed me), the patch won't fix this.
>
> The difference in versions is this... In met-6.0, we only supported
> "square" neighborhoods.  In met-7.0, we added support for "circular"
> neighborhoods.  In doing so, we leveraged existing code from another
part
> of RAL.  However, as highlighted by your configuration, it slower...
> especially for large neighborhood sizes.
>
> Let me look more closely at the code to see how we can address this
issue.
>
> Thanks,
> John
>
> On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>>
>> Hi Julie,
>>
>> Ok great, thanks.  I'll try running grid stat with the updated
version.
>> And yeah, I figured you meant Luna. :)
>>
>> Ben
>>
>> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT <
>> met_help at ucar.edu
>> > wrote:
>>
>> > Pardon, I meant "luna" not "surge".  :)
>> >
>> > Julie
>> >
>> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik
<jpresto at ucar.edu>
>> > wrote:
>> >
>> > > Hi Ben.
>> > >
>> > > The version of met-7.0 has now been updated on tide:
>> > >
>> > >> MET Version: V7.0
>> > >>
>> > >> Repository: https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_
>> > bugfix
>> > >>
>> > >> Revision: 5607
>> > >>
>> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
>> > >>
>> > >
>> > > I will update on surge once I devonprod access is returned.
>> > >
>> > > Julie
>> > >
>> > >
>> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA
Affiliate via
>> RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > >>
>> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>
>> > >>
>> > >> John,
>> > >>
>> > >> Sure no problem, I copied a HiResW ARW forecast file and a
CCPA file
>> > from
>> > >> WCOSS to Theia and they are located here:
>> > >> /scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/testfiles
>> > >>
>> > >> I also put the CONUS.nc mask file that I am using in that
directory.
>> > >> Using
>> > >> that file or CONUS.poly didn't seem to make a difference
though.
>> > >>
>> > >> Here is the grid stat command I was using on WCOSS (after
removing
>> the
>> > >> unrelevant WCOSS paths):
>> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
ccpa.2017052803.03h.gb2
>> > >> GridStatConfig_APCP03_FSS_href -v 2
>> > >>
>> > >> Julie - That'd be great, thanks!
>> > >>
>> > >> Ben
>> > >>
>> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT <
>> > >> met_help at ucar.edu
>> > >> > wrote:
>> > >>
>> > >> > I'll get met-7.0 updated on tide right now.  I'll follow up
once it
>> > >> update
>> > >> > is complete.
>> > >> >
>> > >> > Julie
>> > >> >
>> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via RT <
>> > >> > met_help at ucar.edu> wrote:
>> > >> >
>> > >> > >
>> > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>> > >> > >
>> > >> > > Ben,
>> > >> > >
>> > >> > > Yeah, that'd be great.  If you send me a single input
forecast
>> and
>> > >> > analysis
>> > >> > > file for comparison, I'll run those 3 versions of grid-
stat using
>> > your
>> > >> > > configuration and quantify the timing differences.
>> > >> > >
>> > >> > > You can either point me to them on theia or post them to
our
>> > anonymous
>> > >> > ftp
>> > >> > > site:
>> > >> > >    https://dtcenter.org/met/users/support/met_help.php#ftp
>> > >> > >
>> > >> > > Thanks,
>> > >> > > John
>> > >> > >
>> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA
Affiliate
>> via
>> > >> RT <
>> > >> > > met_help at ucar.edu> wrote:
>> > >> > >
>> > >> > > >
>> > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> >
>> > >> > > >
>> > >> > > > Hi John,
>> > >> > > >
>> > >> > > > Thanks for letting me know about the grid stat memory
>> allocation
>> > >> issue.
>> > >> > > I
>> > >> > > > am currently pointing to Julie's met/7.0 installation on
WCOSS
>> > >> (Tide),
>> > >> > > and
>> > >> > > > here is the output of grid_stat --version:
>> > >> > > >
>> > >> > > > MET Version:    V7.0
>> > >> > > > Repository:
>> > >> > > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
>> > >> > > > Revision:    5483
>> > >> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun
2018)
>> > >> > > >
>> > >> > > > Since the grid stat fix was put in place on July 24th
the
>> version
>> > >> I'm
>> > >> > > using
>> > >> > > > likely doesn't have that fix, so I'm not sure if the bug
fix
>> will
>> > >> > resolve
>> > >> > > > the issue or not.  I could try pointing to a newer
version of
>> MET.
>> > >> I
>> > >> > > > should also let you know that grid stat for the other
line
>> types
>> > >> does
>> > >> > not
>> > >> > > > seem to be slower, I only see a noticeable slowdown with
the
>> > NBRCNT
>> > >> > line
>> > >> > > > type.
>> > >> > > >
>> > >> > > > I am evaluating the deterministic members of the HREFv2
and the
>> > NCAR
>> > >> > > > ensemble as well as their respective means.  I verify
them on
>> NCEP
>> > >> grid
>> > >> > > 227
>> > >> > > > which is a ~5-km grid (5.079 km).  I can send you some
sample
>> > files
>> > >> for
>> > >> > > > your testing if that would be easiest.
>> > >> > > >
>> > >> > > > Thanks!
>> > >> > > >
>> > >> > > > Ben
>> > >> > > >
>> > >> > > >
>> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway via
RT <
>> > >> > > > met_help at ucar.edu> wrote:
>> > >> > > >
>> > >> > > > > Hi Ben,
>> > >> > > > >
>> > >> > > > > I see that you've found the met-7.0 runtime of Grid-
Stat is
>> much
>> > >> > slower
>> > >> > > > > than the met-6.0 version.  We did find/fix a Grid-Stat
memory
>> > >> > > allocation
>> > >> > > > > issue that's posted to the MET website:
>> > >> > > > >
>> > >> > > https://dtcenter.org/met/users/support/known_issues/
>> > METv7.0/index.php
>> > >> > > > >
>> > >> > > > > I have two questions...
>> > >> > > > >
>> > >> > > > > (1) What's the patch level of the code you're
currently
>> running?
>> > >> > > Please
>> > >> > > > > run "grid_stat --version" for the particular
executable
>> you're
>> > >> > running
>> > >> > > > and
>> > >> > > > > send me the result.  Should be something like this:
>> > >> > > > >
>> > >> > > > > MET Version: V7.0
>> > >> > > > >
>> > >> > > > > Repository:
>> > >> > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
>> > >> > > > >
>> > >> > > > > Revision:         5607
>> > >> > > > >
>> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul
2018)
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > (2) Does this bugfix actually fix the problem you're
seeing?
>> > I'd
>> > >> > like
>> > >> > > to
>> > >> > > > > test this out myself using your config file.  I can
run 3
>> > >> versions:
>> > >> > > > >
>> > >> > > > >    - met-6.0 with latest patches
>> > >> > > > >
>> > >> > > > >    - met-7.0 with no patches
>> > >> > > > >
>> > >> > > > >    - met-7.0 with latest patches
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > What model/analysis are you evaluating?  I might as
well use
>> the
>> > >> same
>> > >> > > > > resolution for my testing.
>> > >> > > > >
>> > >> > > > > Thanks,
>> > >> > > > > John
>> > >> > > > >
>> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake - NOAA
>> Affiliate
>> > >> via
>> > >> > RT
>> > >> > > <
>> > >> > > > > met_help at ucar.edu> wrote:
>> > >> > > > >
>> > >> > > > > >
>> > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was acted
upon.
>> > >> > > > > > Transaction: Ticket created by
benjamin.blake at noaa.gov
>> > >> > > > > >        Queue: met_help
>> > >> > > > > >      Subject: FSS calculation with grid stat
>> > >> > > > > >        Owner: Nobody
>> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
>> > >> > > > > >       Status: new
>> > >> > > > > >  Ticket <URL:
>> > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> > >> > > > >
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > Hi,
>> > >> > > > > >
>> > >> > > > > > I recently converted all of my MET configuration
files from
>> > >> using
>> > >> > > v6.0
>> > >> > > > to
>> > >> > > > > > v7.0.  One issue I'm now having is that my FSS
calculations
>> > are
>> > >> > > taking
>> > >> > > > > much
>> > >> > > > > > longer than they did when I was using METv6.0.
Currently
>> it
>> > >> takes
>> > >> > > > ~15-20
>> > >> > > > > > minutes to generate one .stat file.
>> > >> > > > > >
>> > >> > > > > > I've attached the grid stat configure file that I am
now
>> using
>> > >> with
>> > >> > > > v7.0
>> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and the configure
file I
>> was
>> > >> > using
>> > >> > > > with
>> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
>> > >> > > > > >
>> > >> > > > > > I modified cat_thresh for APCP A03 and I also
reverted back
>> > from
>> > >> > > SHAPE
>> > >> > > > =
>> > >> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood
methods
>> section,
>> > >> but
>> > >> > > the
>> > >> > > > > grid
>> > >> > > > > > stat calculation still takes much longer than it
used to.
>> I
>> > >> also
>> > >> > > tried
>> > >> > > > > > using the CONUS.poly file in shared/met/poly instead
of my
>> > >> CONUS.nc
>> > >> > > to
>> > >> > > > > see
>> > >> > > > > > if that would make a difference but it did not.  The
v7.0
>> > >> configure
>> > >> > > > file
>> > >> > > > > > has many more options in it so perhaps the slowdown
is
>> related
>> > >> to
>> > >> > > > > something
>> > >> > > > > > I added in.
>> > >> > > > > >
>> > >> > > > > > Do you have any suggestions for why it might be
running so
>> > >> slowly?
>> > >> > > > > >
>> > >> > > > > > Thanks!
>> > >> > > > > > Ben
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> > >
>> > >> >
>> > >> >
>> > >>
>> > >>
>> >
>> >
>>
>>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Wed Aug 29 11:56:42 2018

Hi John,

Ahh ok, yeah I'm not sure if you would get the same result and how
well it
handles missing data.  I'd think the result may be slightly different.

Would it be possible to have different code for the square and
circular
neighborhood options?  That way if a user chose square then it would
do the
calculations with the method from v6.0 which is more efficient.  I
don't
know how the code is set up so maybe that's not a viable option.

Ben

On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Ben,
>
> Looking closely at the code, here's the difference...
>
> In met-6.0, for square neighborhoods, every time we increment 1 step
in the
> x-direction, we decrement counts from the column we're discarding
and
> increment counts for the column we're adding.
>
> In met-7.0, for square and circular neighborhoods, we reprocess the
whole
> neighborhood each time since for circles, it isn't as simple as
> dropping/adding a new column.
>
> I agree that using an FFT instead would be preferable.  The question
is
> whether we get the same result?  And whether or not it handles
missing data
> well.
>
> John
>
> On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Ben,
> >
> > Thanks for sending the sample data.  Here's how long this test
case took
> > on my machine using 2 versions of MET:
> > met-6.0 took 01:24.
> > met-7.0 with patches took 10:49.
> >
> > So unfortunately (as you just emailed me), the patch won't fix
this.
> >
> > The difference in versions is this... In met-6.0, we only
supported
> > "square" neighborhoods.  In met-7.0, we added support for
"circular"
> > neighborhoods.  In doing so, we leveraged existing code from
another part
> > of RAL.  However, as highlighted by your configuration, it
slower...
> > especially for large neighborhood sizes.
> >
> > Let me look more closely at the code to see how we can address
this
> issue.
> >
> > Thanks,
> > John
> >
> > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >>
> >> Hi Julie,
> >>
> >> Ok great, thanks.  I'll try running grid stat with the updated
version.
> >> And yeah, I figured you meant Luna. :)
> >>
> >> Ben
> >>
> >> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT <
> >> met_help at ucar.edu
> >> > wrote:
> >>
> >> > Pardon, I meant "luna" not "surge".  :)
> >> >
> >> > Julie
> >> >
> >> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik
<jpresto at ucar.edu>
> >> > wrote:
> >> >
> >> > > Hi Ben.
> >> > >
> >> > > The version of met-7.0 has now been updated on tide:
> >> > >
> >> > >> MET Version: V7.0
> >> > >>
> >> > >> Repository: https://svn-met-
dev.cgd.ucar.edu/branches/met/met-7.0_
> >> > bugfix
> >> > >>
> >> > >> Revision: 5607
> >> > >>
> >> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
> >> > >>
> >> > >
> >> > > I will update on surge once I devonprod access is returned.
> >> > >
> >> > > Julie
> >> > >
> >> > >
> >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA
Affiliate via
> >> RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > >>
> >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >> > >>
> >> > >> John,
> >> > >>
> >> > >> Sure no problem, I copied a HiResW ARW forecast file and a
CCPA
> file
> >> > from
> >> > >> WCOSS to Theia and they are located here:
> >> > >> /scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/testfiles
> >> > >>
> >> > >> I also put the CONUS.nc mask file that I am using in that
> directory.
> >> > >> Using
> >> > >> that file or CONUS.poly didn't seem to make a difference
though.
> >> > >>
> >> > >> Here is the grid stat command I was using on WCOSS (after
removing
> >> the
> >> > >> unrelevant WCOSS paths):
> >> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
> ccpa.2017052803.03h.gb2
> >> > >> GridStatConfig_APCP03_FSS_href -v 2
> >> > >>
> >> > >> Julie - That'd be great, thanks!
> >> > >>
> >> > >> Ben
> >> > >>
> >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT <
> >> > >> met_help at ucar.edu
> >> > >> > wrote:
> >> > >>
> >> > >> > I'll get met-7.0 updated on tide right now.  I'll follow
up once
> it
> >> > >> update
> >> > >> > is complete.
> >> > >> >
> >> > >> > Julie
> >> > >> >
> >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via RT
<
> >> > >> > met_help at ucar.edu> wrote:
> >> > >> >
> >> > >> > >
> >> > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >
> >> > >> > >
> >> > >> > > Ben,
> >> > >> > >
> >> > >> > > Yeah, that'd be great.  If you send me a single input
forecast
> >> and
> >> > >> > analysis
> >> > >> > > file for comparison, I'll run those 3 versions of grid-
stat
> using
> >> > your
> >> > >> > > configuration and quantify the timing differences.
> >> > >> > >
> >> > >> > > You can either point me to them on theia or post them to
our
> >> > anonymous
> >> > >> > ftp
> >> > >> > > site:
> >> > >> > >
https://dtcenter.org/met/users/support/met_help.php#ftp
> >> > >> > >
> >> > >> > > Thanks,
> >> > >> > > John
> >> > >> > >
> >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA
> Affiliate
> >> via
> >> > >> RT <
> >> > >> > > met_help at ucar.edu> wrote:
> >> > >> > >
> >> > >> > > >
> >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=86792
> >> >
> >> > >> > > >
> >> > >> > > > Hi John,
> >> > >> > > >
> >> > >> > > > Thanks for letting me know about the grid stat memory
> >> allocation
> >> > >> issue.
> >> > >> > > I
> >> > >> > > > am currently pointing to Julie's met/7.0 installation
on
> WCOSS
> >> > >> (Tide),
> >> > >> > > and
> >> > >> > > > here is the output of grid_stat --version:
> >> > >> > > >
> >> > >> > > > MET Version:    V7.0
> >> > >> > > > Repository:
> >> > >> > > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> >> > >> > > > Revision:    5483
> >> > >> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13 Jun
2018)
> >> > >> > > >
> >> > >> > > > Since the grid stat fix was put in place on July 24th
the
> >> version
> >> > >> I'm
> >> > >> > > using
> >> > >> > > > likely doesn't have that fix, so I'm not sure if the
bug fix
> >> will
> >> > >> > resolve
> >> > >> > > > the issue or not.  I could try pointing to a newer
version of
> >> MET.
> >> > >> I
> >> > >> > > > should also let you know that grid stat for the other
line
> >> types
> >> > >> does
> >> > >> > not
> >> > >> > > > seem to be slower, I only see a noticeable slowdown
with the
> >> > NBRCNT
> >> > >> > line
> >> > >> > > > type.
> >> > >> > > >
> >> > >> > > > I am evaluating the deterministic members of the
HREFv2 and
> the
> >> > NCAR
> >> > >> > > > ensemble as well as their respective means.  I verify
them on
> >> NCEP
> >> > >> grid
> >> > >> > > 227
> >> > >> > > > which is a ~5-km grid (5.079 km).  I can send you some
sample
> >> > files
> >> > >> for
> >> > >> > > > your testing if that would be easiest.
> >> > >> > > >
> >> > >> > > > Thanks!
> >> > >> > > >
> >> > >> > > > Ben
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway
via RT <
> >> > >> > > > met_help at ucar.edu> wrote:
> >> > >> > > >
> >> > >> > > > > Hi Ben,
> >> > >> > > > >
> >> > >> > > > > I see that you've found the met-7.0 runtime of Grid-
Stat is
> >> much
> >> > >> > slower
> >> > >> > > > > than the met-6.0 version.  We did find/fix a Grid-
Stat
> memory
> >> > >> > > allocation
> >> > >> > > > > issue that's posted to the MET website:
> >> > >> > > > >
> >> > >> > > https://dtcenter.org/met/users/support/known_issues/
> >> > METv7.0/index.php
> >> > >> > > > >
> >> > >> > > > > I have two questions...
> >> > >> > > > >
> >> > >> > > > > (1) What's the patch level of the code you're
currently
> >> running?
> >> > >> > > Please
> >> > >> > > > > run "grid_stat --version" for the particular
executable
> >> you're
> >> > >> > running
> >> > >> > > > and
> >> > >> > > > > send me the result.  Should be something like this:
> >> > >> > > > >
> >> > >> > > > > MET Version: V7.0
> >> > >> > > > >
> >> > >> > > > > Repository:
> >> > >> > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> >> > >> > > > >
> >> > >> > > > > Revision:         5607
> >> > >> > > > >
> >> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul
2018)
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > (2) Does this bugfix actually fix the problem you're
> seeing?
> >> > I'd
> >> > >> > like
> >> > >> > > to
> >> > >> > > > > test this out myself using your config file.  I can
run 3
> >> > >> versions:
> >> > >> > > > >
> >> > >> > > > >    - met-6.0 with latest patches
> >> > >> > > > >
> >> > >> > > > >    - met-7.0 with no patches
> >> > >> > > > >
> >> > >> > > > >    - met-7.0 with latest patches
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > What model/analysis are you evaluating?  I might as
well
> use
> >> the
> >> > >> same
> >> > >> > > > > resolution for my testing.
> >> > >> > > > >
> >> > >> > > > > Thanks,
> >> > >> > > > > John
> >> > >> > > > >
> >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake -
NOAA
> >> Affiliate
> >> > >> via
> >> > >> > RT
> >> > >> > > <
> >> > >> > > > > met_help at ucar.edu> wrote:
> >> > >> > > > >
> >> > >> > > > > >
> >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was acted
upon.
> >> > >> > > > > > Transaction: Ticket created by
benjamin.blake at noaa.gov
> >> > >> > > > > >        Queue: met_help
> >> > >> > > > > >      Subject: FSS calculation with grid stat
> >> > >> > > > > >        Owner: Nobody
> >> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
> >> > >> > > > > >       Status: new
> >> > >> > > > > >  Ticket <URL:
> >> > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >> > >> > > > >
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > > Hi,
> >> > >> > > > > >
> >> > >> > > > > > I recently converted all of my MET configuration
files
> from
> >> > >> using
> >> > >> > > v6.0
> >> > >> > > > to
> >> > >> > > > > > v7.0.  One issue I'm now having is that my FSS
> calculations
> >> > are
> >> > >> > > taking
> >> > >> > > > > much
> >> > >> > > > > > longer than they did when I was using METv6.0.
Currently
> >> it
> >> > >> takes
> >> > >> > > > ~15-20
> >> > >> > > > > > minutes to generate one .stat file.
> >> > >> > > > > >
> >> > >> > > > > > I've attached the grid stat configure file that I
am now
> >> using
> >> > >> with
> >> > >> > > > v7.0
> >> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and the configure
file
> I
> >> was
> >> > >> > using
> >> > >> > > > with
> >> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> >> > >> > > > > >
> >> > >> > > > > > I modified cat_thresh for APCP A03 and I also
reverted
> back
> >> > from
> >> > >> > > SHAPE
> >> > >> > > > =
> >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood
methods
> >> section,
> >> > >> but
> >> > >> > > the
> >> > >> > > > > grid
> >> > >> > > > > > stat calculation still takes much longer than it
used to.
> >> I
> >> > >> also
> >> > >> > > tried
> >> > >> > > > > > using the CONUS.poly file in shared/met/poly
instead of
> my
> >> > >> CONUS.nc
> >> > >> > > to
> >> > >> > > > > see
> >> > >> > > > > > if that would make a difference but it did not.
The v7.0
> >> > >> configure
> >> > >> > > > file
> >> > >> > > > > > has many more options in it so perhaps the
slowdown is
> >> related
> >> > >> to
> >> > >> > > > > something
> >> > >> > > > > > I added in.
> >> > >> > > > > >
> >> > >> > > > > > Do you have any suggestions for why it might be
running
> so
> >> > >> slowly?
> >> > >> > > > > >
> >> > >> > > > > > Thanks!
> >> > >> > > > > > Ben
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >> >
> >> > >>
> >> > >>
> >> >
> >> >
> >>
> >>
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: John Halley Gotway
Time: Wed Aug 29 13:13:31 2018

Ben,

Yep, that's what I'm thinking too.  For a square neighborhood, call
the
met-6.0 code.  For a circular neighborhood, call the newer (slower)
code.

John

On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>
> Hi John,
>
> Ahh ok, yeah I'm not sure if you would get the same result and how
well it
> handles missing data.  I'd think the result may be slightly
different.
>
> Would it be possible to have different code for the square and
circular
> neighborhood options?  That way if a user chose square then it would
do the
> calculations with the method from v6.0 which is more efficient.  I
don't
> know how the code is set up so maybe that's not a viable option.
>
> Ben
>
> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Ben,
> >
> > Looking closely at the code, here's the difference...
> >
> > In met-6.0, for square neighborhoods, every time we increment 1
step in
> the
> > x-direction, we decrement counts from the column we're discarding
and
> > increment counts for the column we're adding.
> >
> > In met-7.0, for square and circular neighborhoods, we reprocess
the whole
> > neighborhood each time since for circles, it isn't as simple as
> > dropping/adding a new column.
> >
> > I agree that using an FFT instead would be preferable.  The
question is
> > whether we get the same result?  And whether or not it handles
missing
> data
> > well.
> >
> > John
> >
> > On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Ben,
> > >
> > > Thanks for sending the sample data.  Here's how long this test
case
> took
> > > on my machine using 2 versions of MET:
> > > met-6.0 took 01:24.
> > > met-7.0 with patches took 10:49.
> > >
> > > So unfortunately (as you just emailed me), the patch won't fix
this.
> > >
> > > The difference in versions is this... In met-6.0, we only
supported
> > > "square" neighborhoods.  In met-7.0, we added support for
"circular"
> > > neighborhoods.  In doing so, we leveraged existing code from
another
> part
> > > of RAL.  However, as highlighted by your configuration, it
slower...
> > > especially for large neighborhood sizes.
> > >
> > > Let me look more closely at the code to see how we can address
this
> > issue.
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > >>
> > >> Hi Julie,
> > >>
> > >> Ok great, thanks.  I'll try running grid stat with the updated
> version.
> > >> And yeah, I figured you meant Luna. :)
> > >>
> > >> Ben
> > >>
> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT <
> > >> met_help at ucar.edu
> > >> > wrote:
> > >>
> > >> > Pardon, I meant "luna" not "surge".  :)
> > >> >
> > >> > Julie
> > >> >
> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik
<jpresto at ucar.edu
> >
> > >> > wrote:
> > >> >
> > >> > > Hi Ben.
> > >> > >
> > >> > > The version of met-7.0 has now been updated on tide:
> > >> > >
> > >> > >> MET Version: V7.0
> > >> > >>
> > >> > >> Repository:
> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_
> > >> > bugfix
> > >> > >>
> > >> > >> Revision: 5607
> > >> > >>
> > >> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
> > >> > >>
> > >> > >
> > >> > > I will update on surge once I devonprod access is returned.
> > >> > >
> > >> > > Julie
> > >> > >
> > >> > >
> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA
Affiliate
> via
> > >> RT <
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > >>
> > >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > >> > >>
> > >> > >> John,
> > >> > >>
> > >> > >> Sure no problem, I copied a HiResW ARW forecast file and a
CCPA
> > file
> > >> > from
> > >> > >> WCOSS to Theia and they are located here:
> > >> > >> /scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/testfiles
> > >> > >>
> > >> > >> I also put the CONUS.nc mask file that I am using in that
> > directory.
> > >> > >> Using
> > >> > >> that file or CONUS.poly didn't seem to make a difference
though.
> > >> > >>
> > >> > >> Here is the grid stat command I was using on WCOSS (after
> removing
> > >> the
> > >> > >> unrelevant WCOSS paths):
> > >> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
> > ccpa.2017052803.03h.gb2
> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
> > >> > >>
> > >> > >> Julie - That'd be great, thanks!
> > >> > >>
> > >> > >> Ben
> > >> > >>
> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT
<
> > >> > >> met_help at ucar.edu
> > >> > >> > wrote:
> > >> > >>
> > >> > >> > I'll get met-7.0 updated on tide right now.  I'll follow
up
> once
> > it
> > >> > >> update
> > >> > >> > is complete.
> > >> > >> >
> > >> > >> > Julie
> > >> > >> >
> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via
RT <
> > >> > >> > met_help at ucar.edu> wrote:
> > >> > >> >
> > >> > >> > >
> > >> > >> > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > >
> > >> > >> > >
> > >> > >> > > Ben,
> > >> > >> > >
> > >> > >> > > Yeah, that'd be great.  If you send me a single input
> forecast
> > >> and
> > >> > >> > analysis
> > >> > >> > > file for comparison, I'll run those 3 versions of
grid-stat
> > using
> > >> > your
> > >> > >> > > configuration and quantify the timing differences.
> > >> > >> > >
> > >> > >> > > You can either point me to them on theia or post them
to our
> > >> > anonymous
> > >> > >> > ftp
> > >> > >> > > site:
> > >> > >> > >
https://dtcenter.org/met/users/support/met_help.php#ftp
> > >> > >> > >
> > >> > >> > > Thanks,
> > >> > >> > > John
> > >> > >> > >
> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake - NOAA
> > Affiliate
> > >> via
> > >> > >> RT <
> > >> > >> > > met_help at ucar.edu> wrote:
> > >> > >> > >
> > >> > >> > > >
> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=86792
> > >> >
> > >> > >> > > >
> > >> > >> > > > Hi John,
> > >> > >> > > >
> > >> > >> > > > Thanks for letting me know about the grid stat
memory
> > >> allocation
> > >> > >> issue.
> > >> > >> > > I
> > >> > >> > > > am currently pointing to Julie's met/7.0
installation on
> > WCOSS
> > >> > >> (Tide),
> > >> > >> > > and
> > >> > >> > > > here is the output of grid_stat --version:
> > >> > >> > > >
> > >> > >> > > > MET Version:    V7.0
> > >> > >> > > > Repository:
> > >> > >> > > >
> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > >> > >> > > > Revision:    5483
> > >> > >> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13
Jun
> 2018)
> > >> > >> > > >
> > >> > >> > > > Since the grid stat fix was put in place on July
24th the
> > >> version
> > >> > >> I'm
> > >> > >> > > using
> > >> > >> > > > likely doesn't have that fix, so I'm not sure if the
bug
> fix
> > >> will
> > >> > >> > resolve
> > >> > >> > > > the issue or not.  I could try pointing to a newer
version
> of
> > >> MET.
> > >> > >> I
> > >> > >> > > > should also let you know that grid stat for the
other line
> > >> types
> > >> > >> does
> > >> > >> > not
> > >> > >> > > > seem to be slower, I only see a noticeable slowdown
with
> the
> > >> > NBRCNT
> > >> > >> > line
> > >> > >> > > > type.
> > >> > >> > > >
> > >> > >> > > > I am evaluating the deterministic members of the
HREFv2 and
> > the
> > >> > NCAR
> > >> > >> > > > ensemble as well as their respective means.  I
verify them
> on
> > >> NCEP
> > >> > >> grid
> > >> > >> > > 227
> > >> > >> > > > which is a ~5-km grid (5.079 km).  I can send you
some
> sample
> > >> > files
> > >> > >> for
> > >> > >> > > > your testing if that would be easiest.
> > >> > >> > > >
> > >> > >> > > > Thanks!
> > >> > >> > > >
> > >> > >> > > > Ben
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley Gotway
via
> RT <
> > >> > >> > > > met_help at ucar.edu> wrote:
> > >> > >> > > >
> > >> > >> > > > > Hi Ben,
> > >> > >> > > > >
> > >> > >> > > > > I see that you've found the met-7.0 runtime of
Grid-Stat
> is
> > >> much
> > >> > >> > slower
> > >> > >> > > > > than the met-6.0 version.  We did find/fix a Grid-
Stat
> > memory
> > >> > >> > > allocation
> > >> > >> > > > > issue that's posted to the MET website:
> > >> > >> > > > >
> > >> > >> > > https://dtcenter.org/met/users/support/known_issues/
> > >> > METv7.0/index.php
> > >> > >> > > > >
> > >> > >> > > > > I have two questions...
> > >> > >> > > > >
> > >> > >> > > > > (1) What's the patch level of the code you're
currently
> > >> running?
> > >> > >> > > Please
> > >> > >> > > > > run "grid_stat --version" for the particular
executable
> > >> you're
> > >> > >> > running
> > >> > >> > > > and
> > >> > >> > > > > send me the result.  Should be something like
this:
> > >> > >> > > > >
> > >> > >> > > > > MET Version: V7.0
> > >> > >> > > > >
> > >> > >> > > > > Repository:
> > >> > >> > > https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> > >> > >> > > > >
> > >> > >> > > > > Revision:         5607
> > >> > >> > > > >
> > >> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24
Jul 2018)
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > (2) Does this bugfix actually fix the problem
you're
> > seeing?
> > >> > I'd
> > >> > >> > like
> > >> > >> > > to
> > >> > >> > > > > test this out myself using your config file.  I
can run 3
> > >> > >> versions:
> > >> > >> > > > >
> > >> > >> > > > >    - met-6.0 with latest patches
> > >> > >> > > > >
> > >> > >> > > > >    - met-7.0 with no patches
> > >> > >> > > > >
> > >> > >> > > > >    - met-7.0 with latest patches
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > What model/analysis are you evaluating?  I might
as well
> > use
> > >> the
> > >> > >> same
> > >> > >> > > > > resolution for my testing.
> > >> > >> > > > >
> > >> > >> > > > > Thanks,
> > >> > >> > > > > John
> > >> > >> > > > >
> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake -
NOAA
> > >> Affiliate
> > >> > >> via
> > >> > >> > RT
> > >> > >> > > <
> > >> > >> > > > > met_help at ucar.edu> wrote:
> > >> > >> > > > >
> > >> > >> > > > > >
> > >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was
acted upon.
> > >> > >> > > > > > Transaction: Ticket created by
benjamin.blake at noaa.gov
> > >> > >> > > > > >        Queue: met_help
> > >> > >> > > > > >      Subject: FSS calculation with grid stat
> > >> > >> > > > > >        Owner: Nobody
> > >> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
> > >> > >> > > > > >       Status: new
> > >> > >> > > > > >  Ticket <URL:
> > >> > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > >> > >> > > > >
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > > Hi,
> > >> > >> > > > > >
> > >> > >> > > > > > I recently converted all of my MET configuration
files
> > from
> > >> > >> using
> > >> > >> > > v6.0
> > >> > >> > > > to
> > >> > >> > > > > > v7.0.  One issue I'm now having is that my FSS
> > calculations
> > >> > are
> > >> > >> > > taking
> > >> > >> > > > > much
> > >> > >> > > > > > longer than they did when I was using METv6.0.
> Currently
> > >> it
> > >> > >> takes
> > >> > >> > > > ~15-20
> > >> > >> > > > > > minutes to generate one .stat file.
> > >> > >> > > > > >
> > >> > >> > > > > > I've attached the grid stat configure file that
I am
> now
> > >> using
> > >> > >> with
> > >> > >> > > > v7.0
> > >> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and the
configure file
> > I
> > >> was
> > >> > >> > using
> > >> > >> > > > with
> > >> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> > >> > >> > > > > >
> > >> > >> > > > > > I modified cat_thresh for APCP A03 and I also
reverted
> > back
> > >> > from
> > >> > >> > > SHAPE
> > >> > >> > > > =
> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood
methods
> > >> section,
> > >> > >> but
> > >> > >> > > the
> > >> > >> > > > > grid
> > >> > >> > > > > > stat calculation still takes much longer than it
used
> to.
> > >> I
> > >> > >> also
> > >> > >> > > tried
> > >> > >> > > > > > using the CONUS.poly file in shared/met/poly
instead of
> > my
> > >> > >> CONUS.nc
> > >> > >> > > to
> > >> > >> > > > > see
> > >> > >> > > > > > if that would make a difference but it did not.
The
> v7.0
> > >> > >> configure
> > >> > >> > > > file
> > >> > >> > > > > > has many more options in it so perhaps the
slowdown is
> > >> related
> > >> > >> to
> > >> > >> > > > > something
> > >> > >> > > > > > I added in.
> > >> > >> > > > > >
> > >> > >> > > > > > Do you have any suggestions for why it might be
running
> > so
> > >> > >> slowly?
> > >> > >> > > > > >
> > >> > >> > > > > > Thanks!
> > >> > >> > > > > > Ben
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> >
> > >> > >> >
> > >> > >>
> > >> > >>
> > >> >
> > >> >
> > >>
> > >>
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: John Halley Gotway
Time: Wed Aug 29 15:02:12 2018

Ben,

OK, I modified the development version to call the old logic for
squares
and new (slower) logic for circles.

Running your test case on my machine, squares take 1:15 to run and
circles
take 7:27 to run.

I presume that you'd like to see this change in the next version of
MET...
at least unless we're able to make the new implementation much, much
faster.

Is that right?

Thanks,
John





On Wed, Aug 29, 2018 at 1:15 PM John Halley Gotway <johnhg at ucar.edu>
wrote:

> Ben,
>
> Yep, that's what I'm thinking too.  For a square neighborhood, call
the
> met-6.0 code.  For a circular neighborhood, call the newer (slower)
code.
>
> John
>
> On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>>
>> Hi John,
>>
>> Ahh ok, yeah I'm not sure if you would get the same result and how
well it
>> handles missing data.  I'd think the result may be slightly
different.
>>
>> Would it be possible to have different code for the square and
circular
>> neighborhood options?  That way if a user chose square then it
would do
>> the
>> calculations with the method from v6.0 which is more efficient.  I
don't
>> know how the code is set up so maybe that's not a viable option.
>>
>> Ben
>>
>> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>> > Ben,
>> >
>> > Looking closely at the code, here's the difference...
>> >
>> > In met-6.0, for square neighborhoods, every time we increment 1
step in
>> the
>> > x-direction, we decrement counts from the column we're discarding
and
>> > increment counts for the column we're adding.
>> >
>> > In met-7.0, for square and circular neighborhoods, we reprocess
the
>> whole
>> > neighborhood each time since for circles, it isn't as simple as
>> > dropping/adding a new column.
>> >
>> > I agree that using an FFT instead would be preferable.  The
question is
>> > whether we get the same result?  And whether or not it handles
missing
>> data
>> > well.
>> >
>> > John
>> >
>> > On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway
<johnhg at ucar.edu>
>> > wrote:
>> >
>> > > Ben,
>> > >
>> > > Thanks for sending the sample data.  Here's how long this test
case
>> took
>> > > on my machine using 2 versions of MET:
>> > > met-6.0 took 01:24.
>> > > met-7.0 with patches took 10:49.
>> > >
>> > > So unfortunately (as you just emailed me), the patch won't fix
this.
>> > >
>> > > The difference in versions is this... In met-6.0, we only
supported
>> > > "square" neighborhoods.  In met-7.0, we added support for
"circular"
>> > > neighborhoods.  In doing so, we leveraged existing code from
another
>> part
>> > > of RAL.  However, as highlighted by your configuration, it
slower...
>> > > especially for large neighborhood sizes.
>> > >
>> > > Let me look more closely at the code to see how we can address
this
>> > issue.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA
Affiliate via
>> RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > >>
>> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>
>> > >>
>> > >> Hi Julie,
>> > >>
>> > >> Ok great, thanks.  I'll try running grid stat with the updated
>> version.
>> > >> And yeah, I figured you meant Luna. :)
>> > >>
>> > >> Ben
>> > >>
>> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT <
>> > >> met_help at ucar.edu
>> > >> > wrote:
>> > >>
>> > >> > Pardon, I meant "luna" not "surge".  :)
>> > >> >
>> > >> > Julie
>> > >> >
>> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik <
>> jpresto at ucar.edu>
>> > >> > wrote:
>> > >> >
>> > >> > > Hi Ben.
>> > >> > >
>> > >> > > The version of met-7.0 has now been updated on tide:
>> > >> > >
>> > >> > >> MET Version: V7.0
>> > >> > >>
>> > >> > >> Repository:
>> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_
>> > >> > bugfix
>> > >> > >>
>> > >> > >> Revision: 5607
>> > >> > >>
>> > >> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul 2018)
>> > >> > >>
>> > >> > >
>> > >> > > I will update on surge once I devonprod access is
returned.
>> > >> > >
>> > >> > > Julie
>> > >> > >
>> > >> > >
>> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA
Affiliate
>> via
>> > >> RT <
>> > >> > > met_help at ucar.edu> wrote:
>> > >> > >
>> > >> > >>
>> > >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>> > >> > >>
>> > >> > >> John,
>> > >> > >>
>> > >> > >> Sure no problem, I copied a HiResW ARW forecast file and
a CCPA
>> > file
>> > >> > from
>> > >> > >> WCOSS to Theia and they are located here:
>> > >> > >> /scratch4/NCEPDEV/fv3-
cam/noscrub/Benjamin.Blake/testfiles
>> > >> > >>
>> > >> > >> I also put the CONUS.nc mask file that I am using in that
>> > directory.
>> > >> > >> Using
>> > >> > >> that file or CONUS.poly didn't seem to make a difference
though.
>> > >> > >>
>> > >> > >> Here is the grid stat command I was using on WCOSS (after
>> removing
>> > >> the
>> > >> > >> unrelevant WCOSS paths):
>> > >> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
>> > ccpa.2017052803.03h.gb2
>> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
>> > >> > >>
>> > >> > >> Julie - That'd be great, thanks!
>> > >> > >>
>> > >> > >> Ben
>> > >> > >>
>> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via RT
<
>> > >> > >> met_help at ucar.edu
>> > >> > >> > wrote:
>> > >> > >>
>> > >> > >> > I'll get met-7.0 updated on tide right now.  I'll
follow up
>> once
>> > it
>> > >> > >> update
>> > >> > >> > is complete.
>> > >> > >> >
>> > >> > >> > Julie
>> > >> > >> >
>> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway via
RT <
>> > >> > >> > met_help at ucar.edu> wrote:
>> > >> > >> >
>> > >> > >> > >
>> > >> > >> > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> > >
>> > >> > >> > >
>> > >> > >> > > Ben,
>> > >> > >> > >
>> > >> > >> > > Yeah, that'd be great.  If you send me a single input
>> forecast
>> > >> and
>> > >> > >> > analysis
>> > >> > >> > > file for comparison, I'll run those 3 versions of
grid-stat
>> > using
>> > >> > your
>> > >> > >> > > configuration and quantify the timing differences.
>> > >> > >> > >
>> > >> > >> > > You can either point me to them on theia or post them
to our
>> > >> > anonymous
>> > >> > >> > ftp
>> > >> > >> > > site:
>> > >> > >> > >
https://dtcenter.org/met/users/support/met_help.php#ftp
>> > >> > >> > >
>> > >> > >> > > Thanks,
>> > >> > >> > > John
>> > >> > >> > >
>> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake -
NOAA
>> > Affiliate
>> > >> via
>> > >> > >> RT <
>> > >> > >> > > met_help at ucar.edu> wrote:
>> > >> > >> > >
>> > >> > >> > > >
>> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
>> > Ticket/Display.html?id=86792
>> > >> >
>> > >> > >> > > >
>> > >> > >> > > > Hi John,
>> > >> > >> > > >
>> > >> > >> > > > Thanks for letting me know about the grid stat
memory
>> > >> allocation
>> > >> > >> issue.
>> > >> > >> > > I
>> > >> > >> > > > am currently pointing to Julie's met/7.0
installation on
>> > WCOSS
>> > >> > >> (Tide),
>> > >> > >> > > and
>> > >> > >> > > > here is the output of grid_stat --version:
>> > >> > >> > > >
>> > >> > >> > > > MET Version:    V7.0
>> > >> > >> > > > Repository:
>> > >> > >> > > >
>> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
>> > >> > >> > > > Revision:    5483
>> > >> > >> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed, 13
Jun
>> 2018)
>> > >> > >> > > >
>> > >> > >> > > > Since the grid stat fix was put in place on July
24th the
>> > >> version
>> > >> > >> I'm
>> > >> > >> > > using
>> > >> > >> > > > likely doesn't have that fix, so I'm not sure if
the bug
>> fix
>> > >> will
>> > >> > >> > resolve
>> > >> > >> > > > the issue or not.  I could try pointing to a newer
>> version of
>> > >> MET.
>> > >> > >> I
>> > >> > >> > > > should also let you know that grid stat for the
other line
>> > >> types
>> > >> > >> does
>> > >> > >> > not
>> > >> > >> > > > seem to be slower, I only see a noticeable slowdown
with
>> the
>> > >> > NBRCNT
>> > >> > >> > line
>> > >> > >> > > > type.
>> > >> > >> > > >
>> > >> > >> > > > I am evaluating the deterministic members of the
HREFv2
>> and
>> > the
>> > >> > NCAR
>> > >> > >> > > > ensemble as well as their respective means.  I
verify
>> them on
>> > >> NCEP
>> > >> > >> grid
>> > >> > >> > > 227
>> > >> > >> > > > which is a ~5-km grid (5.079 km).  I can send you
some
>> sample
>> > >> > files
>> > >> > >> for
>> > >> > >> > > > your testing if that would be easiest.
>> > >> > >> > > >
>> > >> > >> > > > Thanks!
>> > >> > >> > > >
>> > >> > >> > > > Ben
>> > >> > >> > > >
>> > >> > >> > > >
>> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley
Gotway via
>> RT <
>> > >> > >> > > > met_help at ucar.edu> wrote:
>> > >> > >> > > >
>> > >> > >> > > > > Hi Ben,
>> > >> > >> > > > >
>> > >> > >> > > > > I see that you've found the met-7.0 runtime of
>> Grid-Stat is
>> > >> much
>> > >> > >> > slower
>> > >> > >> > > > > than the met-6.0 version.  We did find/fix a
Grid-Stat
>> > memory
>> > >> > >> > > allocation
>> > >> > >> > > > > issue that's posted to the MET website:
>> > >> > >> > > > >
>> > >> > >> > > https://dtcenter.org/met/users/support/known_issues/
>> > >> > METv7.0/index.php
>> > >> > >> > > > >
>> > >> > >> > > > > I have two questions...
>> > >> > >> > > > >
>> > >> > >> > > > > (1) What's the patch level of the code you're
currently
>> > >> running?
>> > >> > >> > > Please
>> > >> > >> > > > > run "grid_stat --version" for the particular
executable
>> > >> you're
>> > >> > >> > running
>> > >> > >> > > > and
>> > >> > >> > > > > send me the result.  Should be something like
this:
>> > >> > >> > > > >
>> > >> > >> > > > > MET Version: V7.0
>> > >> > >> > > > >
>> > >> > >> > > > > Repository:
>> > >> > >> > >
>> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
>> > >> > >> > > > >
>> > >> > >> > > > > Revision:         5607
>> > >> > >> > > > >
>> > >> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24
Jul
>> 2018)
>> > >> > >> > > > >
>> > >> > >> > > > >
>> > >> > >> > > > > (2) Does this bugfix actually fix the problem
you're
>> > seeing?
>> > >> > I'd
>> > >> > >> > like
>> > >> > >> > > to
>> > >> > >> > > > > test this out myself using your config file.  I
can run
>> 3
>> > >> > >> versions:
>> > >> > >> > > > >
>> > >> > >> > > > >    - met-6.0 with latest patches
>> > >> > >> > > > >
>> > >> > >> > > > >    - met-7.0 with no patches
>> > >> > >> > > > >
>> > >> > >> > > > >    - met-7.0 with latest patches
>> > >> > >> > > > >
>> > >> > >> > > > >
>> > >> > >> > > > > What model/analysis are you evaluating?  I might
as well
>> > use
>> > >> the
>> > >> > >> same
>> > >> > >> > > > > resolution for my testing.
>> > >> > >> > > > >
>> > >> > >> > > > > Thanks,
>> > >> > >> > > > > John
>> > >> > >> > > > >
>> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake -
NOAA
>> > >> Affiliate
>> > >> > >> via
>> > >> > >> > RT
>> > >> > >> > > <
>> > >> > >> > > > > met_help at ucar.edu> wrote:
>> > >> > >> > > > >
>> > >> > >> > > > > >
>> > >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was
acted
>> upon.
>> > >> > >> > > > > > Transaction: Ticket created by
>> benjamin.blake at noaa.gov
>> > >> > >> > > > > >        Queue: met_help
>> > >> > >> > > > > >      Subject: FSS calculation with grid stat
>> > >> > >> > > > > >        Owner: Nobody
>> > >> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
>> > >> > >> > > > > >       Status: new
>> > >> > >> > > > > >  Ticket <URL:
>> > >> > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> > >> > >> > > > >
>> > >> > >> > > > > >
>> > >> > >> > > > > >
>> > >> > >> > > > > > Hi,
>> > >> > >> > > > > >
>> > >> > >> > > > > > I recently converted all of my MET
configuration files
>> > from
>> > >> > >> using
>> > >> > >> > > v6.0
>> > >> > >> > > > to
>> > >> > >> > > > > > v7.0.  One issue I'm now having is that my FSS
>> > calculations
>> > >> > are
>> > >> > >> > > taking
>> > >> > >> > > > > much
>> > >> > >> > > > > > longer than they did when I was using METv6.0.
>> Currently
>> > >> it
>> > >> > >> takes
>> > >> > >> > > > ~15-20
>> > >> > >> > > > > > minutes to generate one .stat file.
>> > >> > >> > > > > >
>> > >> > >> > > > > > I've attached the grid stat configure file that
I am
>> now
>> > >> using
>> > >> > >> with
>> > >> > >> > > > v7.0
>> > >> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and the
configure
>> file
>> > I
>> > >> was
>> > >> > >> > using
>> > >> > >> > > > with
>> > >> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
>> > >> > >> > > > > >
>> > >> > >> > > > > > I modified cat_thresh for APCP A03 and I also
reverted
>> > back
>> > >> > from
>> > >> > >> > > SHAPE
>> > >> > >> > > > =
>> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood
methods
>> > >> section,
>> > >> > >> but
>> > >> > >> > > the
>> > >> > >> > > > > grid
>> > >> > >> > > > > > stat calculation still takes much longer than
it used
>> to.
>> > >> I
>> > >> > >> also
>> > >> > >> > > tried
>> > >> > >> > > > > > using the CONUS.poly file in shared/met/poly
instead
>> of
>> > my
>> > >> > >> CONUS.nc
>> > >> > >> > > to
>> > >> > >> > > > > see
>> > >> > >> > > > > > if that would make a difference but it did not.
The
>> v7.0
>> > >> > >> configure
>> > >> > >> > > > file
>> > >> > >> > > > > > has many more options in it so perhaps the
slowdown is
>> > >> related
>> > >> > >> to
>> > >> > >> > > > > something
>> > >> > >> > > > > > I added in.
>> > >> > >> > > > > >
>> > >> > >> > > > > > Do you have any suggestions for why it might be
>> running
>> > so
>> > >> > >> slowly?
>> > >> > >> > > > > >
>> > >> > >> > > > > > Thanks!
>> > >> > >> > > > > > Ben
>> > >> > >> > > > > >
>> > >> > >> > > > > >
>> > >> > >> > > > >
>> > >> > >> > > > >
>> > >> > >> > > >
>> > >> > >> > > >
>> > >> > >> > >
>> > >> > >> > >
>> > >> > >> >
>> > >> > >> >
>> > >> > >>
>> > >> > >>
>> > >> >
>> > >> >
>> > >>
>> > >>
>> >
>> >
>>
>>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Thu Aug 30 05:33:42 2018

Hi John,

Ok great, and yeah the next version of MET would be fine.

Thanks!
Ben

On Wed, Aug 29, 2018 at 5:02 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Ben,
>
> OK, I modified the development version to call the old logic for
squares
> and new (slower) logic for circles.
>
> Running your test case on my machine, squares take 1:15 to run and
circles
> take 7:27 to run.
>
> I presume that you'd like to see this change in the next version of
MET...
> at least unless we're able to make the new implementation much, much
> faster.
>
> Is that right?
>
> Thanks,
> John
>
>
>
>
>
> On Wed, Aug 29, 2018 at 1:15 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Ben,
> >
> > Yep, that's what I'm thinking too.  For a square neighborhood,
call the
> > met-6.0 code.  For a circular neighborhood, call the newer
(slower) code.
> >
> > John
> >
> > On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >>
> >> Hi John,
> >>
> >> Ahh ok, yeah I'm not sure if you would get the same result and
how well
> it
> >> handles missing data.  I'd think the result may be slightly
different.
> >>
> >> Would it be possible to have different code for the square and
circular
> >> neighborhood options?  That way if a user chose square then it
would do
> >> the
> >> calculations with the method from v6.0 which is more efficient.
I don't
> >> know how the code is set up so maybe that's not a viable option.
> >>
> >> Ben
> >>
> >> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> > Ben,
> >> >
> >> > Looking closely at the code, here's the difference...
> >> >
> >> > In met-6.0, for square neighborhoods, every time we increment 1
step
> in
> >> the
> >> > x-direction, we decrement counts from the column we're
discarding and
> >> > increment counts for the column we're adding.
> >> >
> >> > In met-7.0, for square and circular neighborhoods, we reprocess
the
> >> whole
> >> > neighborhood each time since for circles, it isn't as simple as
> >> > dropping/adding a new column.
> >> >
> >> > I agree that using an FFT instead would be preferable.  The
question
> is
> >> > whether we get the same result?  And whether or not it handles
missing
> >> data
> >> > well.
> >> >
> >> > John
> >> >
> >> > On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway
<johnhg at ucar.edu>
> >> > wrote:
> >> >
> >> > > Ben,
> >> > >
> >> > > Thanks for sending the sample data.  Here's how long this
test case
> >> took
> >> > > on my machine using 2 versions of MET:
> >> > > met-6.0 took 01:24.
> >> > > met-7.0 with patches took 10:49.
> >> > >
> >> > > So unfortunately (as you just emailed me), the patch won't
fix this.
> >> > >
> >> > > The difference in versions is this... In met-6.0, we only
supported
> >> > > "square" neighborhoods.  In met-7.0, we added support for
"circular"
> >> > > neighborhoods.  In doing so, we leveraged existing code from
another
> >> part
> >> > > of RAL.  However, as highlighted by your configuration, it
slower...
> >> > > especially for large neighborhood sizes.
> >> > >
> >> > > Let me look more closely at the code to see how we can
address this
> >> > issue.
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA
Affiliate via
> >> RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > >>
> >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >> > >>
> >> > >> Hi Julie,
> >> > >>
> >> > >> Ok great, thanks.  I'll try running grid stat with the
updated
> >> version.
> >> > >> And yeah, I figured you meant Luna. :)
> >> > >>
> >> > >> Ben
> >> > >>
> >> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT <
> >> > >> met_help at ucar.edu
> >> > >> > wrote:
> >> > >>
> >> > >> > Pardon, I meant "luna" not "surge".  :)
> >> > >> >
> >> > >> > Julie
> >> > >> >
> >> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik <
> >> jpresto at ucar.edu>
> >> > >> > wrote:
> >> > >> >
> >> > >> > > Hi Ben.
> >> > >> > >
> >> > >> > > The version of met-7.0 has now been updated on tide:
> >> > >> > >
> >> > >> > >> MET Version: V7.0
> >> > >> > >>
> >> > >> > >> Repository:
> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_
> >> > >> > bugfix
> >> > >> > >>
> >> > >> > >> Revision: 5607
> >> > >> > >>
> >> > >> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul
2018)
> >> > >> > >>
> >> > >> > >
> >> > >> > > I will update on surge once I devonprod access is
returned.
> >> > >> > >
> >> > >> > > Julie
> >> > >> > >
> >> > >> > >
> >> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA
> Affiliate
> >> via
> >> > >> RT <
> >> > >> > > met_help at ucar.edu> wrote:
> >> > >> > >
> >> > >> > >>
> >> > >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >
> >> > >> > >>
> >> > >> > >> John,
> >> > >> > >>
> >> > >> > >> Sure no problem, I copied a HiResW ARW forecast file
and a
> CCPA
> >> > file
> >> > >> > from
> >> > >> > >> WCOSS to Theia and they are located here:
> >> > >> > >> /scratch4/NCEPDEV/fv3-
cam/noscrub/Benjamin.Blake/testfiles
> >> > >> > >>
> >> > >> > >> I also put the CONUS.nc mask file that I am using in
that
> >> > directory.
> >> > >> > >> Using
> >> > >> > >> that file or CONUS.poly didn't seem to make a
difference
> though.
> >> > >> > >>
> >> > >> > >> Here is the grid stat command I was using on WCOSS
(after
> >> removing
> >> > >> the
> >> > >> > >> unrelevant WCOSS paths):
> >> > >> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
> >> > ccpa.2017052803.03h.gb2
> >> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
> >> > >> > >>
> >> > >> > >> Julie - That'd be great, thanks!
> >> > >> > >>
> >> > >> > >> Ben
> >> > >> > >>
> >> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via
RT <
> >> > >> > >> met_help at ucar.edu
> >> > >> > >> > wrote:
> >> > >> > >>
> >> > >> > >> > I'll get met-7.0 updated on tide right now.  I'll
follow up
> >> once
> >> > it
> >> > >> > >> update
> >> > >> > >> > is complete.
> >> > >> > >> >
> >> > >> > >> > Julie
> >> > >> > >> >
> >> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway
via RT <
> >> > >> > >> > met_help at ucar.edu> wrote:
> >> > >> > >> >
> >> > >> > >> > >
> >> > >> > >> > > <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >> > >
> >> > >> > >> > >
> >> > >> > >> > > Ben,
> >> > >> > >> > >
> >> > >> > >> > > Yeah, that'd be great.  If you send me a single
input
> >> forecast
> >> > >> and
> >> > >> > >> > analysis
> >> > >> > >> > > file for comparison, I'll run those 3 versions of
> grid-stat
> >> > using
> >> > >> > your
> >> > >> > >> > > configuration and quantify the timing differences.
> >> > >> > >> > >
> >> > >> > >> > > You can either point me to them on theia or post
them to
> our
> >> > >> > anonymous
> >> > >> > >> > ftp
> >> > >> > >> > > site:
> >> > >> > >> > >    https://dtcenter.org/met/
> users/support/met_help.php#ftp
> >> > >> > >> > >
> >> > >> > >> > > Thanks,
> >> > >> > >> > > John
> >> > >> > >> > >
> >> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake -
NOAA
> >> > Affiliate
> >> > >> via
> >> > >> > >> RT <
> >> > >> > >> > > met_help at ucar.edu> wrote:
> >> > >> > >> > >
> >> > >> > >> > > >
> >> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
> >> > Ticket/Display.html?id=86792
> >> > >> >
> >> > >> > >> > > >
> >> > >> > >> > > > Hi John,
> >> > >> > >> > > >
> >> > >> > >> > > > Thanks for letting me know about the grid stat
memory
> >> > >> allocation
> >> > >> > >> issue.
> >> > >> > >> > > I
> >> > >> > >> > > > am currently pointing to Julie's met/7.0
installation on
> >> > WCOSS
> >> > >> > >> (Tide),
> >> > >> > >> > > and
> >> > >> > >> > > > here is the output of grid_stat --version:
> >> > >> > >> > > >
> >> > >> > >> > > > MET Version:    V7.0
> >> > >> > >> > > > Repository:
> >> > >> > >> > > >
> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> >> > >> > >> > > > Revision:    5483
> >> > >> > >> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed,
13 Jun
> >> 2018)
> >> > >> > >> > > >
> >> > >> > >> > > > Since the grid stat fix was put in place on July
24th
> the
> >> > >> version
> >> > >> > >> I'm
> >> > >> > >> > > using
> >> > >> > >> > > > likely doesn't have that fix, so I'm not sure if
the bug
> >> fix
> >> > >> will
> >> > >> > >> > resolve
> >> > >> > >> > > > the issue or not.  I could try pointing to a
newer
> >> version of
> >> > >> MET.
> >> > >> > >> I
> >> > >> > >> > > > should also let you know that grid stat for the
other
> line
> >> > >> types
> >> > >> > >> does
> >> > >> > >> > not
> >> > >> > >> > > > seem to be slower, I only see a noticeable
slowdown with
> >> the
> >> > >> > NBRCNT
> >> > >> > >> > line
> >> > >> > >> > > > type.
> >> > >> > >> > > >
> >> > >> > >> > > > I am evaluating the deterministic members of the
HREFv2
> >> and
> >> > the
> >> > >> > NCAR
> >> > >> > >> > > > ensemble as well as their respective means.  I
verify
> >> them on
> >> > >> NCEP
> >> > >> > >> grid
> >> > >> > >> > > 227
> >> > >> > >> > > > which is a ~5-km grid (5.079 km).  I can send you
some
> >> sample
> >> > >> > files
> >> > >> > >> for
> >> > >> > >> > > > your testing if that would be easiest.
> >> > >> > >> > > >
> >> > >> > >> > > > Thanks!
> >> > >> > >> > > >
> >> > >> > >> > > > Ben
> >> > >> > >> > > >
> >> > >> > >> > > >
> >> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley
Gotway via
> >> RT <
> >> > >> > >> > > > met_help at ucar.edu> wrote:
> >> > >> > >> > > >
> >> > >> > >> > > > > Hi Ben,
> >> > >> > >> > > > >
> >> > >> > >> > > > > I see that you've found the met-7.0 runtime of
> >> Grid-Stat is
> >> > >> much
> >> > >> > >> > slower
> >> > >> > >> > > > > than the met-6.0 version.  We did find/fix a
Grid-Stat
> >> > memory
> >> > >> > >> > > allocation
> >> > >> > >> > > > > issue that's posted to the MET website:
> >> > >> > >> > > > >
> >> > >> > >> > >
https://dtcenter.org/met/users/support/known_issues/
> >> > >> > METv7.0/index.php
> >> > >> > >> > > > >
> >> > >> > >> > > > > I have two questions...
> >> > >> > >> > > > >
> >> > >> > >> > > > > (1) What's the patch level of the code you're
> currently
> >> > >> running?
> >> > >> > >> > > Please
> >> > >> > >> > > > > run "grid_stat --version" for the particular
> executable
> >> > >> you're
> >> > >> > >> > running
> >> > >> > >> > > > and
> >> > >> > >> > > > > send me the result.  Should be something like
this:
> >> > >> > >> > > > >
> >> > >> > >> > > > > MET Version: V7.0
> >> > >> > >> > > > >
> >> > >> > >> > > > > Repository:
> >> > >> > >> > >
> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> >> > >> > >> > > > >
> >> > >> > >> > > > > Revision:         5607
> >> > >> > >> > > > >
> >> > >> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24
Jul
> >> 2018)
> >> > >> > >> > > > >
> >> > >> > >> > > > >
> >> > >> > >> > > > > (2) Does this bugfix actually fix the problem
you're
> >> > seeing?
> >> > >> > I'd
> >> > >> > >> > like
> >> > >> > >> > > to
> >> > >> > >> > > > > test this out myself using your config file.  I
can
> run
> >> 3
> >> > >> > >> versions:
> >> > >> > >> > > > >
> >> > >> > >> > > > >    - met-6.0 with latest patches
> >> > >> > >> > > > >
> >> > >> > >> > > > >    - met-7.0 with no patches
> >> > >> > >> > > > >
> >> > >> > >> > > > >    - met-7.0 with latest patches
> >> > >> > >> > > > >
> >> > >> > >> > > > >
> >> > >> > >> > > > > What model/analysis are you evaluating?  I
might as
> well
> >> > use
> >> > >> the
> >> > >> > >> same
> >> > >> > >> > > > > resolution for my testing.
> >> > >> > >> > > > >
> >> > >> > >> > > > > Thanks,
> >> > >> > >> > > > > John
> >> > >> > >> > > > >
> >> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake
- NOAA
> >> > >> Affiliate
> >> > >> > >> via
> >> > >> > >> > RT
> >> > >> > >> > > <
> >> > >> > >> > > > > met_help at ucar.edu> wrote:
> >> > >> > >> > > > >
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was
acted
> >> upon.
> >> > >> > >> > > > > > Transaction: Ticket created by
> >> benjamin.blake at noaa.gov
> >> > >> > >> > > > > >        Queue: met_help
> >> > >> > >> > > > > >      Subject: FSS calculation with grid stat
> >> > >> > >> > > > > >        Owner: Nobody
> >> > >> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
> >> > >> > >> > > > > >       Status: new
> >> > >> > >> > > > > >  Ticket <URL:
> >> > >> > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >> > >> > >> > > > >
> >> > >> > >> > > > > >
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > Hi,
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > I recently converted all of my MET
configuration
> files
> >> > from
> >> > >> > >> using
> >> > >> > >> > > v6.0
> >> > >> > >> > > > to
> >> > >> > >> > > > > > v7.0.  One issue I'm now having is that my
FSS
> >> > calculations
> >> > >> > are
> >> > >> > >> > > taking
> >> > >> > >> > > > > much
> >> > >> > >> > > > > > longer than they did when I was using
METv6.0.
> >> Currently
> >> > >> it
> >> > >> > >> takes
> >> > >> > >> > > > ~15-20
> >> > >> > >> > > > > > minutes to generate one .stat file.
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > I've attached the grid stat configure file
that I am
> >> now
> >> > >> using
> >> > >> > >> with
> >> > >> > >> > > > v7.0
> >> > >> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and the
configure
> >> file
> >> > I
> >> > >> was
> >> > >> > >> > using
> >> > >> > >> > > > with
> >> > >> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > I modified cat_thresh for APCP A03 and I also
> reverted
> >> > back
> >> > >> > from
> >> > >> > >> > > SHAPE
> >> > >> > >> > > > =
> >> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the neighborhood
> methods
> >> > >> section,
> >> > >> > >> but
> >> > >> > >> > > the
> >> > >> > >> > > > > grid
> >> > >> > >> > > > > > stat calculation still takes much longer than
it
> used
> >> to.
> >> > >> I
> >> > >> > >> also
> >> > >> > >> > > tried
> >> > >> > >> > > > > > using the CONUS.poly file in shared/met/poly
instead
> >> of
> >> > my
> >> > >> > >> CONUS.nc
> >> > >> > >> > > to
> >> > >> > >> > > > > see
> >> > >> > >> > > > > > if that would make a difference but it did
not.  The
> >> v7.0
> >> > >> > >> configure
> >> > >> > >> > > > file
> >> > >> > >> > > > > > has many more options in it so perhaps the
slowdown
> is
> >> > >> related
> >> > >> > >> to
> >> > >> > >> > > > > something
> >> > >> > >> > > > > > I added in.
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > Do you have any suggestions for why it might
be
> >> running
> >> > so
> >> > >> > >> slowly?
> >> > >> > >> > > > > >
> >> > >> > >> > > > > > Thanks!
> >> > >> > >> > > > > > Ben
> >> > >> > >> > > > > >
> >> > >> > >> > > > > >
> >> > >> > >> > > > >
> >> > >> > >> > > > >
> >> > >> > >> > > >
> >> > >> > >> > > >
> >> > >> > >> > >
> >> > >> > >> > >
> >> > >> > >> >
> >> > >> > >> >
> >> > >> > >>
> >> > >> > >>
> >> > >> >
> >> > >> >
> >> > >>
> >> > >>
> >> >
> >> >
> >>
> >>
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Thu Aug 30 08:30:26 2018

Hi John,

Do you know when the next beta version will be released?  Would it be
possible to have those code changes in that version?  That way I can
start
running with that version of the code sooner.

Thanks again,
Ben

On Thu, Aug 30, 2018 at 7:33 AM, Benjamin Blake - NOAA Affiliate <
benjamin.blake at noaa.gov> wrote:

> Hi John,
>
> Ok great, and yeah the next version of MET would be fine.
>
> Thanks!
> Ben
>
> On Wed, Aug 29, 2018 at 5:02 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Ben,
>>
>> OK, I modified the development version to call the old logic for
squares
>> and new (slower) logic for circles.
>>
>> Running your test case on my machine, squares take 1:15 to run and
circles
>> take 7:27 to run.
>>
>> I presume that you'd like to see this change in the next version of
MET...
>> at least unless we're able to make the new implementation much,
much
>> faster.
>>
>> Is that right?
>>
>> Thanks,
>> John
>>
>>
>>
>>
>>
>> On Wed, Aug 29, 2018 at 1:15 PM John Halley Gotway
<johnhg at ucar.edu>
>> wrote:
>>
>> > Ben,
>> >
>> > Yep, that's what I'm thinking too.  For a square neighborhood,
call the
>> > met-6.0 code.  For a circular neighborhood, call the newer
(slower)
>> code.
>> >
>> > John
>> >
>> > On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake - NOAA Affiliate
via RT
>> <
>> > met_help at ucar.edu> wrote:
>> >
>> >>
>> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>> >>
>> >> Hi John,
>> >>
>> >> Ahh ok, yeah I'm not sure if you would get the same result and
how
>> well it
>> >> handles missing data.  I'd think the result may be slightly
different.
>> >>
>> >> Would it be possible to have different code for the square and
circular
>> >> neighborhood options?  That way if a user chose square then it
would do
>> >> the
>> >> calculations with the method from v6.0 which is more efficient.
I
>> don't
>> >> know how the code is set up so maybe that's not a viable option.
>> >>
>> >> Ben
>> >>
>> >> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway via RT <
>> >> met_help at ucar.edu> wrote:
>> >>
>> >> > Ben,
>> >> >
>> >> > Looking closely at the code, here's the difference...
>> >> >
>> >> > In met-6.0, for square neighborhoods, every time we increment
1 step
>> in
>> >> the
>> >> > x-direction, we decrement counts from the column we're
discarding and
>> >> > increment counts for the column we're adding.
>> >> >
>> >> > In met-7.0, for square and circular neighborhoods, we
reprocess the
>> >> whole
>> >> > neighborhood each time since for circles, it isn't as simple
as
>> >> > dropping/adding a new column.
>> >> >
>> >> > I agree that using an FFT instead would be preferable.  The
question
>> is
>> >> > whether we get the same result?  And whether or not it handles
>> missing
>> >> data
>> >> > well.
>> >> >
>> >> > John
>> >> >
>> >> > On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway
<johnhg at ucar.edu
>> >
>> >> > wrote:
>> >> >
>> >> > > Ben,
>> >> > >
>> >> > > Thanks for sending the sample data.  Here's how long this
test case
>> >> took
>> >> > > on my machine using 2 versions of MET:
>> >> > > met-6.0 took 01:24.
>> >> > > met-7.0 with patches took 10:49.
>> >> > >
>> >> > > So unfortunately (as you just emailed me), the patch won't
fix
>> this.
>> >> > >
>> >> > > The difference in versions is this... In met-6.0, we only
supported
>> >> > > "square" neighborhoods.  In met-7.0, we added support for
>> "circular"
>> >> > > neighborhoods.  In doing so, we leveraged existing code from
>> another
>> >> part
>> >> > > of RAL.  However, as highlighted by your configuration, it
>> slower...
>> >> > > especially for large neighborhood sizes.
>> >> > >
>> >> > > Let me look more closely at the code to see how we can
address this
>> >> > issue.
>> >> > >
>> >> > > Thanks,
>> >> > > John
>> >> > >
>> >> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA
Affiliate
>> via
>> >> RT <
>> >> > > met_help at ucar.edu> wrote:
>> >> > >
>> >> > >>
>> >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>> >> > >>
>> >> > >> Hi Julie,
>> >> > >>
>> >> > >> Ok great, thanks.  I'll try running grid stat with the
updated
>> >> version.
>> >> > >> And yeah, I figured you meant Luna. :)
>> >> > >>
>> >> > >> Ben
>> >> > >>
>> >> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT <
>> >> > >> met_help at ucar.edu
>> >> > >> > wrote:
>> >> > >>
>> >> > >> > Pardon, I meant "luna" not "surge".  :)
>> >> > >> >
>> >> > >> > Julie
>> >> > >> >
>> >> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik <
>> >> jpresto at ucar.edu>
>> >> > >> > wrote:
>> >> > >> >
>> >> > >> > > Hi Ben.
>> >> > >> > >
>> >> > >> > > The version of met-7.0 has now been updated on tide:
>> >> > >> > >
>> >> > >> > >> MET Version: V7.0
>> >> > >> > >>
>> >> > >> > >> Repository:
>> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_
>> >> > >> > bugfix
>> >> > >> > >>
>> >> > >> > >> Revision: 5607
>> >> > >> > >>
>> >> > >> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul
2018)
>> >> > >> > >>
>> >> > >> > >
>> >> > >> > > I will update on surge once I devonprod access is
returned.
>> >> > >> > >
>> >> > >> > > Julie
>> >> > >> > >
>> >> > >> > >
>> >> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake - NOAA
>> Affiliate
>> >> via
>> >> > >> RT <
>> >> > >> > > met_help at ucar.edu> wrote:
>> >> > >> > >
>> >> > >> > >>
>> >> > >> > >> <URL: https://rt.rap.ucar.edu/rt/Tic
>> ket/Display.html?id=86792 >
>> >> > >> > >>
>> >> > >> > >> John,
>> >> > >> > >>
>> >> > >> > >> Sure no problem, I copied a HiResW ARW forecast file
and a
>> CCPA
>> >> > file
>> >> > >> > from
>> >> > >> > >> WCOSS to Theia and they are located here:
>> >> > >> > >> /scratch4/NCEPDEV/fv3-
cam/noscrub/Benjamin.Blake/testfiles
>> >> > >> > >>
>> >> > >> > >> I also put the CONUS.nc mask file that I am using in
that
>> >> > directory.
>> >> > >> > >> Using
>> >> > >> > >> that file or CONUS.poly didn't seem to make a
difference
>> though.
>> >> > >> > >>
>> >> > >> > >> Here is the grid stat command I was using on WCOSS
(after
>> >> removing
>> >> > >> the
>> >> > >> > >> unrelevant WCOSS paths):
>> >> > >> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
>> >> > ccpa.2017052803.03h.gb2
>> >> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
>> >> > >> > >>
>> >> > >> > >> Julie - That'd be great, thanks!
>> >> > >> > >>
>> >> > >> > >> Ben
>> >> > >> > >>
>> >> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik via
RT <
>> >> > >> > >> met_help at ucar.edu
>> >> > >> > >> > wrote:
>> >> > >> > >>
>> >> > >> > >> > I'll get met-7.0 updated on tide right now.  I'll
follow up
>> >> once
>> >> > it
>> >> > >> > >> update
>> >> > >> > >> > is complete.
>> >> > >> > >> >
>> >> > >> > >> > Julie
>> >> > >> > >> >
>> >> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley Gotway
via RT
>> <
>> >> > >> > >> > met_help at ucar.edu> wrote:
>> >> > >> > >> >
>> >> > >> > >> > >
>> >> > >> > >> > > <URL:
>> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> >> > >
>> >> > >> > >> > >
>> >> > >> > >> > > Ben,
>> >> > >> > >> > >
>> >> > >> > >> > > Yeah, that'd be great.  If you send me a single
input
>> >> forecast
>> >> > >> and
>> >> > >> > >> > analysis
>> >> > >> > >> > > file for comparison, I'll run those 3 versions of
>> grid-stat
>> >> > using
>> >> > >> > your
>> >> > >> > >> > > configuration and quantify the timing differences.
>> >> > >> > >> > >
>> >> > >> > >> > > You can either point me to them on theia or post
them to
>> our
>> >> > >> > anonymous
>> >> > >> > >> > ftp
>> >> > >> > >> > > site:
>> >> > >> > >> > >    https://dtcenter.org/met/users
>> /support/met_help.php#ftp
>> >> > >> > >> > >
>> >> > >> > >> > > Thanks,
>> >> > >> > >> > > John
>> >> > >> > >> > >
>> >> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake -
NOAA
>> >> > Affiliate
>> >> > >> via
>> >> > >> > >> RT <
>> >> > >> > >> > > met_help at ucar.edu> wrote:
>> >> > >> > >> > >
>> >> > >> > >> > > >
>> >> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
>> >> > Ticket/Display.html?id=86792
>> >> > >> >
>> >> > >> > >> > > >
>> >> > >> > >> > > > Hi John,
>> >> > >> > >> > > >
>> >> > >> > >> > > > Thanks for letting me know about the grid stat
memory
>> >> > >> allocation
>> >> > >> > >> issue.
>> >> > >> > >> > > I
>> >> > >> > >> > > > am currently pointing to Julie's met/7.0
installation
>> on
>> >> > WCOSS
>> >> > >> > >> (Tide),
>> >> > >> > >> > > and
>> >> > >> > >> > > > here is the output of grid_stat --version:
>> >> > >> > >> > > >
>> >> > >> > >> > > > MET Version:    V7.0
>> >> > >> > >> > > > Repository:
>> >> > >> > >> > > >
>> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
>> >> > >> > >> > > > Revision:    5483
>> >> > >> > >> > > > Change Date:    2018-06-13 09:57:36 -0600 (Wed,
13 Jun
>> >> 2018)
>> >> > >> > >> > > >
>> >> > >> > >> > > > Since the grid stat fix was put in place on July
24th
>> the
>> >> > >> version
>> >> > >> > >> I'm
>> >> > >> > >> > > using
>> >> > >> > >> > > > likely doesn't have that fix, so I'm not sure if
the
>> bug
>> >> fix
>> >> > >> will
>> >> > >> > >> > resolve
>> >> > >> > >> > > > the issue or not.  I could try pointing to a
newer
>> >> version of
>> >> > >> MET.
>> >> > >> > >> I
>> >> > >> > >> > > > should also let you know that grid stat for the
other
>> line
>> >> > >> types
>> >> > >> > >> does
>> >> > >> > >> > not
>> >> > >> > >> > > > seem to be slower, I only see a noticeable
slowdown
>> with
>> >> the
>> >> > >> > NBRCNT
>> >> > >> > >> > line
>> >> > >> > >> > > > type.
>> >> > >> > >> > > >
>> >> > >> > >> > > > I am evaluating the deterministic members of the
HREFv2
>> >> and
>> >> > the
>> >> > >> > NCAR
>> >> > >> > >> > > > ensemble as well as their respective means.  I
verify
>> >> them on
>> >> > >> NCEP
>> >> > >> > >> grid
>> >> > >> > >> > > 227
>> >> > >> > >> > > > which is a ~5-km grid (5.079 km).  I can send
you some
>> >> sample
>> >> > >> > files
>> >> > >> > >> for
>> >> > >> > >> > > > your testing if that would be easiest.
>> >> > >> > >> > > >
>> >> > >> > >> > > > Thanks!
>> >> > >> > >> > > >
>> >> > >> > >> > > > Ben
>> >> > >> > >> > > >
>> >> > >> > >> > > >
>> >> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley
Gotway
>> via
>> >> RT <
>> >> > >> > >> > > > met_help at ucar.edu> wrote:
>> >> > >> > >> > > >
>> >> > >> > >> > > > > Hi Ben,
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > I see that you've found the met-7.0 runtime of
>> >> Grid-Stat is
>> >> > >> much
>> >> > >> > >> > slower
>> >> > >> > >> > > > > than the met-6.0 version.  We did find/fix a
>> Grid-Stat
>> >> > memory
>> >> > >> > >> > > allocation
>> >> > >> > >> > > > > issue that's posted to the MET website:
>> >> > >> > >> > > > >
>> >> > >> > >> > >
https://dtcenter.org/met/users/support/known_issues/
>> >> > >> > METv7.0/index.php
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > I have two questions...
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > (1) What's the patch level of the code you're
>> currently
>> >> > >> running?
>> >> > >> > >> > > Please
>> >> > >> > >> > > > > run "grid_stat --version" for the particular
>> executable
>> >> > >> you're
>> >> > >> > >> > running
>> >> > >> > >> > > > and
>> >> > >> > >> > > > > send me the result.  Should be something like
this:
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > MET Version: V7.0
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > Repository:
>> >> > >> > >> > >
>> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > Revision:         5607
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue,
24 Jul
>> >> 2018)
>> >> > >> > >> > > > >
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > (2) Does this bugfix actually fix the problem
you're
>> >> > seeing?
>> >> > >> > I'd
>> >> > >> > >> > like
>> >> > >> > >> > > to
>> >> > >> > >> > > > > test this out myself using your config file.
I can
>> run
>> >> 3
>> >> > >> > >> versions:
>> >> > >> > >> > > > >
>> >> > >> > >> > > > >    - met-6.0 with latest patches
>> >> > >> > >> > > > >
>> >> > >> > >> > > > >    - met-7.0 with no patches
>> >> > >> > >> > > > >
>> >> > >> > >> > > > >    - met-7.0 with latest patches
>> >> > >> > >> > > > >
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > What model/analysis are you evaluating?  I
might as
>> well
>> >> > use
>> >> > >> the
>> >> > >> > >> same
>> >> > >> > >> > > > > resolution for my testing.
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > Thanks,
>> >> > >> > >> > > > > John
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin Blake
- NOAA
>> >> > >> Affiliate
>> >> > >> > >> via
>> >> > >> > >> > RT
>> >> > >> > >> > > <
>> >> > >> > >> > > > > met_help at ucar.edu> wrote:
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > >
>> >> > >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792 was
acted
>> >> upon.
>> >> > >> > >> > > > > > Transaction: Ticket created by
>> >> benjamin.blake at noaa.gov
>> >> > >> > >> > > > > >        Queue: met_help
>> >> > >> > >> > > > > >      Subject: FSS calculation with grid stat
>> >> > >> > >> > > > > >        Owner: Nobody
>> >> > >> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
>> >> > >> > >> > > > > >       Status: new
>> >> > >> > >> > > > > >  Ticket <URL:
>> >> > >> > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> >> > >> > >> > > > >
>> >> > >> > >> > > > > >
>> >> > >> > >> > > > > >
>> >> > >> > >> > > > > > Hi,
>> >> > >> > >> > > > > >
>> >> > >> > >> > > > > > I recently converted all of my MET
configuration
>> files
>> >> > from
>> >> > >> > >> using
>> >> > >> > >> > > v6.0
>> >> > >> > >> > > > to
>> >> > >> > >> > > > > > v7.0.  One issue I'm now having is that my
FSS
>> >> > calculations
>> >> > >> > are
>> >> > >> > >> > > taking
>> >> > >> > >> > > > > much
>> >> > >> > >> > > > > > longer than they did when I was using
METv6.0.
>> >> Currently
>> >> > >> it
>> >> > >> > >> takes
>> >> > >> > >> > > > ~15-20
>> >> > >> > >> > > > > > minutes to generate one .stat file.
>> >> > >> > >> > > > > >
>> >> > >> > >> > > > > > I've attached the grid stat configure file
that I
>> am
>> >> now
>> >> > >> using
>> >> > >> > >> with
>> >> > >> > >> > > > v7.0
>> >> > >> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and the
configure
>> >> file
>> >> > I
>> >> > >> was
>> >> > >> > >> > using
>> >> > >> > >> > > > with
>> >> > >> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
>> >> > >> > >> > > > > >
>> >> > >> > >> > > > > > I modified cat_thresh for APCP A03 and I
also
>> reverted
>> >> > back
>> >> > >> > from
>> >> > >> > >> > > SHAPE
>> >> > >> > >> > > > =
>> >> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the
neighborhood
>> methods
>> >> > >> section,
>> >> > >> > >> but
>> >> > >> > >> > > the
>> >> > >> > >> > > > > grid
>> >> > >> > >> > > > > > stat calculation still takes much longer
than it
>> used
>> >> to.
>> >> > >> I
>> >> > >> > >> also
>> >> > >> > >> > > tried
>> >> > >> > >> > > > > > using the CONUS.poly file in shared/met/poly
>> instead
>> >> of
>> >> > my
>> >> > >> > >> CONUS.nc
>> >> > >> > >> > > to
>> >> > >> > >> > > > > see
>> >> > >> > >> > > > > > if that would make a difference but it did
not.
>> The
>> >> v7.0
>> >> > >> > >> configure
>> >> > >> > >> > > > file
>> >> > >> > >> > > > > > has many more options in it so perhaps the
>> slowdown is
>> >> > >> related
>> >> > >> > >> to
>> >> > >> > >> > > > > something
>> >> > >> > >> > > > > > I added in.
>> >> > >> > >> > > > > >
>> >> > >> > >> > > > > > Do you have any suggestions for why it might
be
>> >> running
>> >> > so
>> >> > >> > >> slowly?
>> >> > >> > >> > > > > >
>> >> > >> > >> > > > > > Thanks!
>> >> > >> > >> > > > > > Ben
>> >> > >> > >> > > > > >
>> >> > >> > >> > > > > >
>> >> > >> > >> > > > >
>> >> > >> > >> > > > >
>> >> > >> > >> > > >
>> >> > >> > >> > > >
>> >> > >> > >> > >
>> >> > >> > >> > >
>> >> > >> > >> >
>> >> > >> > >> >
>> >> > >> > >>
>> >> > >> > >>
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> > >>
>> >> >
>> >> >
>> >>
>> >>
>>
>>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: John Halley Gotway
Time: Thu Aug 30 10:55:07 2018

Ben,

Yes, it'll be included in the next beta version... although we'll be
releasing met-8.0 in September.  And we don't currently have a planned
beta
release version before then.

However, rather than simply reverting to the old logic, I'd prefer to
"refine" the new logic.  That way, using the circles would be just as
fast
as the squares.  So I'm going to try to figure that out and will let
you
know how it goes.

Thanks,
John



On Thu, Aug 30, 2018 at 8:31 AM Benjamin Blake - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>
> Hi John,
>
> Do you know when the next beta version will be released?  Would it
be
> possible to have those code changes in that version?  That way I can
start
> running with that version of the code sooner.
>
> Thanks again,
> Ben
>
> On Thu, Aug 30, 2018 at 7:33 AM, Benjamin Blake - NOAA Affiliate <
> benjamin.blake at noaa.gov> wrote:
>
> > Hi John,
> >
> > Ok great, and yeah the next version of MET would be fine.
> >
> > Thanks!
> > Ben
> >
> > On Wed, Aug 29, 2018 at 5:02 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Ben,
> >>
> >> OK, I modified the development version to call the old logic for
squares
> >> and new (slower) logic for circles.
> >>
> >> Running your test case on my machine, squares take 1:15 to run
and
> circles
> >> take 7:27 to run.
> >>
> >> I presume that you'd like to see this change in the next version
of
> MET...
> >> at least unless we're able to make the new implementation much,
much
> >> faster.
> >>
> >> Is that right?
> >>
> >> Thanks,
> >> John
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Aug 29, 2018 at 1:15 PM John Halley Gotway
<johnhg at ucar.edu>
> >> wrote:
> >>
> >> > Ben,
> >> >
> >> > Yep, that's what I'm thinking too.  For a square neighborhood,
call
> the
> >> > met-6.0 code.  For a circular neighborhood, call the newer
(slower)
> >> code.
> >> >
> >> > John
> >> >
> >> > On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake - NOAA
Affiliate via
> RT
> >> <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> >>
> >> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>
> >> >>
> >> >> Hi John,
> >> >>
> >> >> Ahh ok, yeah I'm not sure if you would get the same result and
how
> >> well it
> >> >> handles missing data.  I'd think the result may be slightly
> different.
> >> >>
> >> >> Would it be possible to have different code for the square and
> circular
> >> >> neighborhood options?  That way if a user chose square then it
would
> do
> >> >> the
> >> >> calculations with the method from v6.0 which is more
efficient.  I
> >> don't
> >> >> know how the code is set up so maybe that's not a viable
option.
> >> >>
> >> >> Ben
> >> >>
> >> >> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway via RT <
> >> >> met_help at ucar.edu> wrote:
> >> >>
> >> >> > Ben,
> >> >> >
> >> >> > Looking closely at the code, here's the difference...
> >> >> >
> >> >> > In met-6.0, for square neighborhoods, every time we
increment 1
> step
> >> in
> >> >> the
> >> >> > x-direction, we decrement counts from the column we're
discarding
> and
> >> >> > increment counts for the column we're adding.
> >> >> >
> >> >> > In met-7.0, for square and circular neighborhoods, we
reprocess the
> >> >> whole
> >> >> > neighborhood each time since for circles, it isn't as simple
as
> >> >> > dropping/adding a new column.
> >> >> >
> >> >> > I agree that using an FFT instead would be preferable.  The
> question
> >> is
> >> >> > whether we get the same result?  And whether or not it
handles
> >> missing
> >> >> data
> >> >> > well.
> >> >> >
> >> >> > John
> >> >> >
> >> >> > On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway <
> johnhg at ucar.edu
> >> >
> >> >> > wrote:
> >> >> >
> >> >> > > Ben,
> >> >> > >
> >> >> > > Thanks for sending the sample data.  Here's how long this
test
> case
> >> >> took
> >> >> > > on my machine using 2 versions of MET:
> >> >> > > met-6.0 took 01:24.
> >> >> > > met-7.0 with patches took 10:49.
> >> >> > >
> >> >> > > So unfortunately (as you just emailed me), the patch won't
fix
> >> this.
> >> >> > >
> >> >> > > The difference in versions is this... In met-6.0, we only
> supported
> >> >> > > "square" neighborhoods.  In met-7.0, we added support for
> >> "circular"
> >> >> > > neighborhoods.  In doing so, we leveraged existing code
from
> >> another
> >> >> part
> >> >> > > of RAL.  However, as highlighted by your configuration, it
> >> slower...
> >> >> > > especially for large neighborhood sizes.
> >> >> > >
> >> >> > > Let me look more closely at the code to see how we can
address
> this
> >> >> > issue.
> >> >> > >
> >> >> > > Thanks,
> >> >> > > John
> >> >> > >
> >> >> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA
Affiliate
> >> via
> >> >> RT <
> >> >> > > met_help at ucar.edu> wrote:
> >> >> > >
> >> >> > >>
> >> >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >> >> > >>
> >> >> > >> Hi Julie,
> >> >> > >>
> >> >> > >> Ok great, thanks.  I'll try running grid stat with the
updated
> >> >> version.
> >> >> > >> And yeah, I figured you meant Luna. :)
> >> >> > >>
> >> >> > >> Ben
> >> >> > >>
> >> >> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via RT
<
> >> >> > >> met_help at ucar.edu
> >> >> > >> > wrote:
> >> >> > >>
> >> >> > >> > Pardon, I meant "luna" not "surge".  :)
> >> >> > >> >
> >> >> > >> > Julie
> >> >> > >> >
> >> >> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik <
> >> >> jpresto at ucar.edu>
> >> >> > >> > wrote:
> >> >> > >> >
> >> >> > >> > > Hi Ben.
> >> >> > >> > >
> >> >> > >> > > The version of met-7.0 has now been updated on tide:
> >> >> > >> > >
> >> >> > >> > >> MET Version: V7.0
> >> >> > >> > >>
> >> >> > >> > >> Repository:
> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_
> >> >> > >> > bugfix
> >> >> > >> > >>
> >> >> > >> > >> Revision: 5607
> >> >> > >> > >>
> >> >> > >> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul
2018)
> >> >> > >> > >>
> >> >> > >> > >
> >> >> > >> > > I will update on surge once I devonprod access is
returned.
> >> >> > >> > >
> >> >> > >> > > Julie
> >> >> > >> > >
> >> >> > >> > >
> >> >> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake -
NOAA
> >> Affiliate
> >> >> via
> >> >> > >> RT <
> >> >> > >> > > met_help at ucar.edu> wrote:
> >> >> > >> > >
> >> >> > >> > >>
> >> >> > >> > >> <URL: https://rt.rap.ucar.edu/rt/Tic
> >> ket/Display.html?id=86792 >
> >> >> > >> > >>
> >> >> > >> > >> John,
> >> >> > >> > >>
> >> >> > >> > >> Sure no problem, I copied a HiResW ARW forecast file
and a
> >> CCPA
> >> >> > file
> >> >> > >> > from
> >> >> > >> > >> WCOSS to Theia and they are located here:
> >> >> > >> > >> /scratch4/NCEPDEV/fv3-
cam/noscrub/Benjamin.Blake/testfiles
> >> >> > >> > >>
> >> >> > >> > >> I also put the CONUS.nc mask file that I am using in
that
> >> >> > directory.
> >> >> > >> > >> Using
> >> >> > >> > >> that file or CONUS.poly didn't seem to make a
difference
> >> though.
> >> >> > >> > >>
> >> >> > >> > >> Here is the grid stat command I was using on WCOSS
(after
> >> >> removing
> >> >> > >> the
> >> >> > >> > >> unrelevant WCOSS paths):
> >> >> > >> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
> >> >> > ccpa.2017052803.03h.gb2
> >> >> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
> >> >> > >> > >>
> >> >> > >> > >> Julie - That'd be great, thanks!
> >> >> > >> > >>
> >> >> > >> > >> Ben
> >> >> > >> > >>
> >> >> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik
via RT <
> >> >> > >> > >> met_help at ucar.edu
> >> >> > >> > >> > wrote:
> >> >> > >> > >>
> >> >> > >> > >> > I'll get met-7.0 updated on tide right now.  I'll
follow
> up
> >> >> once
> >> >> > it
> >> >> > >> > >> update
> >> >> > >> > >> > is complete.
> >> >> > >> > >> >
> >> >> > >> > >> > Julie
> >> >> > >> > >> >
> >> >> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley
Gotway via
> RT
> >> <
> >> >> > >> > >> > met_help at ucar.edu> wrote:
> >> >> > >> > >> >
> >> >> > >> > >> > >
> >> >> > >> > >> > > <URL:
> >> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >> >> > >
> >> >> > >> > >> > >
> >> >> > >> > >> > > Ben,
> >> >> > >> > >> > >
> >> >> > >> > >> > > Yeah, that'd be great.  If you send me a single
input
> >> >> forecast
> >> >> > >> and
> >> >> > >> > >> > analysis
> >> >> > >> > >> > > file for comparison, I'll run those 3 versions
of
> >> grid-stat
> >> >> > using
> >> >> > >> > your
> >> >> > >> > >> > > configuration and quantify the timing
differences.
> >> >> > >> > >> > >
> >> >> > >> > >> > > You can either point me to them on theia or post
them
> to
> >> our
> >> >> > >> > anonymous
> >> >> > >> > >> > ftp
> >> >> > >> > >> > > site:
> >> >> > >> > >> > >    https://dtcenter.org/met/users
> >> /support/met_help.php#ftp
> >> >> > >> > >> > >
> >> >> > >> > >> > > Thanks,
> >> >> > >> > >> > > John
> >> >> > >> > >> > >
> >> >> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake
- NOAA
> >> >> > Affiliate
> >> >> > >> via
> >> >> > >> > >> RT <
> >> >> > >> > >> > > met_help at ucar.edu> wrote:
> >> >> > >> > >> > >
> >> >> > >> > >> > > >
> >> >> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
> >> >> > Ticket/Display.html?id=86792
> >> >> > >> >
> >> >> > >> > >> > > >
> >> >> > >> > >> > > > Hi John,
> >> >> > >> > >> > > >
> >> >> > >> > >> > > > Thanks for letting me know about the grid stat
memory
> >> >> > >> allocation
> >> >> > >> > >> issue.
> >> >> > >> > >> > > I
> >> >> > >> > >> > > > am currently pointing to Julie's met/7.0
installation
> >> on
> >> >> > WCOSS
> >> >> > >> > >> (Tide),
> >> >> > >> > >> > > and
> >> >> > >> > >> > > > here is the output of grid_stat --version:
> >> >> > >> > >> > > >
> >> >> > >> > >> > > > MET Version:    V7.0
> >> >> > >> > >> > > > Repository:
> >> >> > >> > >> > > >
> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> >> >> > >> > >> > > > Revision:    5483
> >> >> > >> > >> > > > Change Date:    2018-06-13 09:57:36 -0600
(Wed, 13
> Jun
> >> >> 2018)
> >> >> > >> > >> > > >
> >> >> > >> > >> > > > Since the grid stat fix was put in place on
July 24th
> >> the
> >> >> > >> version
> >> >> > >> > >> I'm
> >> >> > >> > >> > > using
> >> >> > >> > >> > > > likely doesn't have that fix, so I'm not sure
if the
> >> bug
> >> >> fix
> >> >> > >> will
> >> >> > >> > >> > resolve
> >> >> > >> > >> > > > the issue or not.  I could try pointing to a
newer
> >> >> version of
> >> >> > >> MET.
> >> >> > >> > >> I
> >> >> > >> > >> > > > should also let you know that grid stat for
the other
> >> line
> >> >> > >> types
> >> >> > >> > >> does
> >> >> > >> > >> > not
> >> >> > >> > >> > > > seem to be slower, I only see a noticeable
slowdown
> >> with
> >> >> the
> >> >> > >> > NBRCNT
> >> >> > >> > >> > line
> >> >> > >> > >> > > > type.
> >> >> > >> > >> > > >
> >> >> > >> > >> > > > I am evaluating the deterministic members of
the
> HREFv2
> >> >> and
> >> >> > the
> >> >> > >> > NCAR
> >> >> > >> > >> > > > ensemble as well as their respective means.  I
verify
> >> >> them on
> >> >> > >> NCEP
> >> >> > >> > >> grid
> >> >> > >> > >> > > 227
> >> >> > >> > >> > > > which is a ~5-km grid (5.079 km).  I can send
you
> some
> >> >> sample
> >> >> > >> > files
> >> >> > >> > >> for
> >> >> > >> > >> > > > your testing if that would be easiest.
> >> >> > >> > >> > > >
> >> >> > >> > >> > > > Thanks!
> >> >> > >> > >> > > >
> >> >> > >> > >> > > > Ben
> >> >> > >> > >> > > >
> >> >> > >> > >> > > >
> >> >> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley
Gotway
> >> via
> >> >> RT <
> >> >> > >> > >> > > > met_help at ucar.edu> wrote:
> >> >> > >> > >> > > >
> >> >> > >> > >> > > > > Hi Ben,
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > I see that you've found the met-7.0 runtime
of
> >> >> Grid-Stat is
> >> >> > >> much
> >> >> > >> > >> > slower
> >> >> > >> > >> > > > > than the met-6.0 version.  We did find/fix a
> >> Grid-Stat
> >> >> > memory
> >> >> > >> > >> > > allocation
> >> >> > >> > >> > > > > issue that's posted to the MET website:
> >> >> > >> > >> > > > >
> >> >> > >> > >> > >
https://dtcenter.org/met/users/support/known_issues/
> >> >> > >> > METv7.0/index.php
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > I have two questions...
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > (1) What's the patch level of the code
you're
> >> currently
> >> >> > >> running?
> >> >> > >> > >> > > Please
> >> >> > >> > >> > > > > run "grid_stat --version" for the particular
> >> executable
> >> >> > >> you're
> >> >> > >> > >> > running
> >> >> > >> > >> > > > and
> >> >> > >> > >> > > > > send me the result.  Should be something
like this:
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > MET Version: V7.0
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > Repository:
> >> >> > >> > >> > >
> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > Revision:         5607
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600 (Tue,
24 Jul
> >> >> 2018)
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > (2) Does this bugfix actually fix the
problem
> you're
> >> >> > seeing?
> >> >> > >> > I'd
> >> >> > >> > >> > like
> >> >> > >> > >> > > to
> >> >> > >> > >> > > > > test this out myself using your config file.
I can
> >> run
> >> >> 3
> >> >> > >> > >> versions:
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > >    - met-6.0 with latest patches
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > >    - met-7.0 with no patches
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > >    - met-7.0 with latest patches
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > What model/analysis are you evaluating?  I
might as
> >> well
> >> >> > use
> >> >> > >> the
> >> >> > >> > >> same
> >> >> > >> > >> > > > > resolution for my testing.
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > Thanks,
> >> >> > >> > >> > > > > John
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin
Blake -
> NOAA
> >> >> > >> Affiliate
> >> >> > >> > >> via
> >> >> > >> > >> > RT
> >> >> > >> > >> > > <
> >> >> > >> > >> > > > > met_help at ucar.edu> wrote:
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > >
> >> >> > >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792
was acted
> >> >> upon.
> >> >> > >> > >> > > > > > Transaction: Ticket created by
> >> >> benjamin.blake at noaa.gov
> >> >> > >> > >> > > > > >        Queue: met_help
> >> >> > >> > >> > > > > >      Subject: FSS calculation with grid
stat
> >> >> > >> > >> > > > > >        Owner: Nobody
> >> >> > >> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
> >> >> > >> > >> > > > > >       Status: new
> >> >> > >> > >> > > > > >  Ticket <URL:
> >> >> > >> > >> > >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > > >
> >> >> > >> > >> > > > > >
> >> >> > >> > >> > > > > > Hi,
> >> >> > >> > >> > > > > >
> >> >> > >> > >> > > > > > I recently converted all of my MET
configuration
> >> files
> >> >> > from
> >> >> > >> > >> using
> >> >> > >> > >> > > v6.0
> >> >> > >> > >> > > > to
> >> >> > >> > >> > > > > > v7.0.  One issue I'm now having is that my
FSS
> >> >> > calculations
> >> >> > >> > are
> >> >> > >> > >> > > taking
> >> >> > >> > >> > > > > much
> >> >> > >> > >> > > > > > longer than they did when I was using
METv6.0.
> >> >> Currently
> >> >> > >> it
> >> >> > >> > >> takes
> >> >> > >> > >> > > > ~15-20
> >> >> > >> > >> > > > > > minutes to generate one .stat file.
> >> >> > >> > >> > > > > >
> >> >> > >> > >> > > > > > I've attached the grid stat configure file
that I
> >> am
> >> >> now
> >> >> > >> using
> >> >> > >> > >> with
> >> >> > >> > >> > > > v7.0
> >> >> > >> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and the
> configure
> >> >> file
> >> >> > I
> >> >> > >> was
> >> >> > >> > >> > using
> >> >> > >> > >> > > > with
> >> >> > >> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> >> >> > >> > >> > > > > >
> >> >> > >> > >> > > > > > I modified cat_thresh for APCP A03 and I
also
> >> reverted
> >> >> > back
> >> >> > >> > from
> >> >> > >> > >> > > SHAPE
> >> >> > >> > >> > > > =
> >> >> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the
neighborhood
> >> methods
> >> >> > >> section,
> >> >> > >> > >> but
> >> >> > >> > >> > > the
> >> >> > >> > >> > > > > grid
> >> >> > >> > >> > > > > > stat calculation still takes much longer
than it
> >> used
> >> >> to.
> >> >> > >> I
> >> >> > >> > >> also
> >> >> > >> > >> > > tried
> >> >> > >> > >> > > > > > using the CONUS.poly file in
shared/met/poly
> >> instead
> >> >> of
> >> >> > my
> >> >> > >> > >> CONUS.nc
> >> >> > >> > >> > > to
> >> >> > >> > >> > > > > see
> >> >> > >> > >> > > > > > if that would make a difference but it did
not.
> >> The
> >> >> v7.0
> >> >> > >> > >> configure
> >> >> > >> > >> > > > file
> >> >> > >> > >> > > > > > has many more options in it so perhaps the
> >> slowdown is
> >> >> > >> related
> >> >> > >> > >> to
> >> >> > >> > >> > > > > something
> >> >> > >> > >> > > > > > I added in.
> >> >> > >> > >> > > > > >
> >> >> > >> > >> > > > > > Do you have any suggestions for why it
might be
> >> >> running
> >> >> > so
> >> >> > >> > >> slowly?
> >> >> > >> > >> > > > > >
> >> >> > >> > >> > > > > > Thanks!
> >> >> > >> > >> > > > > > Ben
> >> >> > >> > >> > > > > >
> >> >> > >> > >> > > > > >
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > > >
> >> >> > >> > >> > > >
> >> >> > >> > >> > > >
> >> >> > >> > >> > >
> >> >> > >> > >> > >
> >> >> > >> > >> >
> >> >> > >> > >> >
> >> >> > >> > >>
> >> >> > >> > >>
> >> >> > >> >
> >> >> > >> >
> >> >> > >>
> >> >> > >>
> >> >> >
> >> >> >
> >> >>
> >> >>
> >>
> >>
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: John Halley Gotway
Time: Thu Aug 30 17:12:11 2018

Ben,

I made some good progress on this task today.

I implemented the old smarter logic using the new more flexible data
structures... making sure that the results are identical.
I tested 3 versions on your sample configuration (met-6.0, met-7.0,
and my
proposed changes for met-8.0) and found:

  met-6.0 for squares takes 1:30.
  met-7.0 for squares takes 10:58.
  met-8.0 for squares takes 1:50.

  met-7.0 for circles takes 7:39.
  met-8.0 for circles takes 1:50.

So met-8.0 is still a bit slower than met-6.0... but it's much closer.
The
big advantage is that this improvement also applies to the circle
shape.

I know how I could make the logic more slick to speed it up a bit
more, but
the trouble coding it up.  Rather than moving like a type-writer,
resetting
every line, we could move like a snake, moving back and forth across
the
grid.

Thanks,
John




On Thu, Aug 30, 2018 at 10:57 AM John Halley Gotway <johnhg at ucar.edu>
wrote:

> Ben,
>
> Yes, it'll be included in the next beta version... although we'll be
> releasing met-8.0 in September.  And we don't currently have a
planned beta
> release version before then.
>
> However, rather than simply reverting to the old logic, I'd prefer
to
> "refine" the new logic.  That way, using the circles would be just
as fast
> as the squares.  So I'm going to try to figure that out and will let
you
> know how it goes.
>
> Thanks,
> John
>
>
>
> On Thu, Aug 30, 2018 at 8:31 AM Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>>
>> Hi John,
>>
>> Do you know when the next beta version will be released?  Would it
be
>> possible to have those code changes in that version?  That way I
can start
>> running with that version of the code sooner.
>>
>> Thanks again,
>> Ben
>>
>> On Thu, Aug 30, 2018 at 7:33 AM, Benjamin Blake - NOAA Affiliate <
>> benjamin.blake at noaa.gov> wrote:
>>
>> > Hi John,
>> >
>> > Ok great, and yeah the next version of MET would be fine.
>> >
>> > Thanks!
>> > Ben
>> >
>> > On Wed, Aug 29, 2018 at 5:02 PM, John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >> Ben,
>> >>
>> >> OK, I modified the development version to call the old logic for
>> squares
>> >> and new (slower) logic for circles.
>> >>
>> >> Running your test case on my machine, squares take 1:15 to run
and
>> circles
>> >> take 7:27 to run.
>> >>
>> >> I presume that you'd like to see this change in the next version
of
>> MET...
>> >> at least unless we're able to make the new implementation much,
much
>> >> faster.
>> >>
>> >> Is that right?
>> >>
>> >> Thanks,
>> >> John
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Wed, Aug 29, 2018 at 1:15 PM John Halley Gotway
<johnhg at ucar.edu>
>> >> wrote:
>> >>
>> >> > Ben,
>> >> >
>> >> > Yep, that's what I'm thinking too.  For a square neighborhood,
call
>> the
>> >> > met-6.0 code.  For a circular neighborhood, call the newer
(slower)
>> >> code.
>> >> >
>> >> > John
>> >> >
>> >> > On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake - NOAA
Affiliate via
>> RT
>> >> <
>> >> > met_help at ucar.edu> wrote:
>> >> >
>> >> >>
>> >> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>
>> >> >>
>> >> >> Hi John,
>> >> >>
>> >> >> Ahh ok, yeah I'm not sure if you would get the same result
and how
>> >> well it
>> >> >> handles missing data.  I'd think the result may be slightly
>> different.
>> >> >>
>> >> >> Would it be possible to have different code for the square
and
>> circular
>> >> >> neighborhood options?  That way if a user chose square then
it
>> would do
>> >> >> the
>> >> >> calculations with the method from v6.0 which is more
efficient.  I
>> >> don't
>> >> >> know how the code is set up so maybe that's not a viable
option.
>> >> >>
>> >> >> Ben
>> >> >>
>> >> >> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway via RT <
>> >> >> met_help at ucar.edu> wrote:
>> >> >>
>> >> >> > Ben,
>> >> >> >
>> >> >> > Looking closely at the code, here's the difference...
>> >> >> >
>> >> >> > In met-6.0, for square neighborhoods, every time we
increment 1
>> step
>> >> in
>> >> >> the
>> >> >> > x-direction, we decrement counts from the column we're
discarding
>> and
>> >> >> > increment counts for the column we're adding.
>> >> >> >
>> >> >> > In met-7.0, for square and circular neighborhoods, we
reprocess
>> the
>> >> >> whole
>> >> >> > neighborhood each time since for circles, it isn't as
simple as
>> >> >> > dropping/adding a new column.
>> >> >> >
>> >> >> > I agree that using an FFT instead would be preferable.  The
>> question
>> >> is
>> >> >> > whether we get the same result?  And whether or not it
handles
>> >> missing
>> >> >> data
>> >> >> > well.
>> >> >> >
>> >> >> > John
>> >> >> >
>> >> >> > On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway <
>> johnhg at ucar.edu
>> >> >
>> >> >> > wrote:
>> >> >> >
>> >> >> > > Ben,
>> >> >> > >
>> >> >> > > Thanks for sending the sample data.  Here's how long this
test
>> case
>> >> >> took
>> >> >> > > on my machine using 2 versions of MET:
>> >> >> > > met-6.0 took 01:24.
>> >> >> > > met-7.0 with patches took 10:49.
>> >> >> > >
>> >> >> > > So unfortunately (as you just emailed me), the patch
won't fix
>> >> this.
>> >> >> > >
>> >> >> > > The difference in versions is this... In met-6.0, we only
>> supported
>> >> >> > > "square" neighborhoods.  In met-7.0, we added support for
>> >> "circular"
>> >> >> > > neighborhoods.  In doing so, we leveraged existing code
from
>> >> another
>> >> >> part
>> >> >> > > of RAL.  However, as highlighted by your configuration,
it
>> >> slower...
>> >> >> > > especially for large neighborhood sizes.
>> >> >> > >
>> >> >> > > Let me look more closely at the code to see how we can
address
>> this
>> >> >> > issue.
>> >> >> > >
>> >> >> > > Thanks,
>> >> >> > > John
>> >> >> > >
>> >> >> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA
Affiliate
>> >> via
>> >> >> RT <
>> >> >> > > met_help at ucar.edu> wrote:
>> >> >> > >
>> >> >> > >>
>> >> >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> >
>> >> >> > >>
>> >> >> > >> Hi Julie,
>> >> >> > >>
>> >> >> > >> Ok great, thanks.  I'll try running grid stat with the
updated
>> >> >> version.
>> >> >> > >> And yeah, I figured you meant Luna. :)
>> >> >> > >>
>> >> >> > >> Ben
>> >> >> > >>
>> >> >> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via
RT <
>> >> >> > >> met_help at ucar.edu
>> >> >> > >> > wrote:
>> >> >> > >>
>> >> >> > >> > Pardon, I meant "luna" not "surge".  :)
>> >> >> > >> >
>> >> >> > >> > Julie
>> >> >> > >> >
>> >> >> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik <
>> >> >> jpresto at ucar.edu>
>> >> >> > >> > wrote:
>> >> >> > >> >
>> >> >> > >> > > Hi Ben.
>> >> >> > >> > >
>> >> >> > >> > > The version of met-7.0 has now been updated on tide:
>> >> >> > >> > >
>> >> >> > >> > >> MET Version: V7.0
>> >> >> > >> > >>
>> >> >> > >> > >> Repository:
>> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_
>> >> >> > >> > bugfix
>> >> >> > >> > >>
>> >> >> > >> > >> Revision: 5607
>> >> >> > >> > >>
>> >> >> > >> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24 Jul
2018)
>> >> >> > >> > >>
>> >> >> > >> > >
>> >> >> > >> > > I will update on surge once I devonprod access is
returned.
>> >> >> > >> > >
>> >> >> > >> > > Julie
>> >> >> > >> > >
>> >> >> > >> > >
>> >> >> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake -
NOAA
>> >> Affiliate
>> >> >> via
>> >> >> > >> RT <
>> >> >> > >> > > met_help at ucar.edu> wrote:
>> >> >> > >> > >
>> >> >> > >> > >>
>> >> >> > >> > >> <URL: https://rt.rap.ucar.edu/rt/Tic
>> >> ket/Display.html?id=86792 >
>> >> >> > >> > >>
>> >> >> > >> > >> John,
>> >> >> > >> > >>
>> >> >> > >> > >> Sure no problem, I copied a HiResW ARW forecast
file and a
>> >> CCPA
>> >> >> > file
>> >> >> > >> > from
>> >> >> > >> > >> WCOSS to Theia and they are located here:
>> >> >> > >> > >> /scratch4/NCEPDEV/fv3-
cam/noscrub/Benjamin.Blake/testfiles
>> >> >> > >> > >>
>> >> >> > >> > >> I also put the CONUS.nc mask file that I am using
in that
>> >> >> > directory.
>> >> >> > >> > >> Using
>> >> >> > >> > >> that file or CONUS.poly didn't seem to make a
difference
>> >> though.
>> >> >> > >> > >>
>> >> >> > >> > >> Here is the grid stat command I was using on WCOSS
(after
>> >> >> removing
>> >> >> > >> the
>> >> >> > >> > >> unrelevant WCOSS paths):
>> >> >> > >> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
>> >> >> > ccpa.2017052803.03h.gb2
>> >> >> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
>> >> >> > >> > >>
>> >> >> > >> > >> Julie - That'd be great, thanks!
>> >> >> > >> > >>
>> >> >> > >> > >> Ben
>> >> >> > >> > >>
>> >> >> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie Prestopnik
via RT
>> <
>> >> >> > >> > >> met_help at ucar.edu
>> >> >> > >> > >> > wrote:
>> >> >> > >> > >>
>> >> >> > >> > >> > I'll get met-7.0 updated on tide right now.  I'll
>> follow up
>> >> >> once
>> >> >> > it
>> >> >> > >> > >> update
>> >> >> > >> > >> > is complete.
>> >> >> > >> > >> >
>> >> >> > >> > >> > Julie
>> >> >> > >> > >> >
>> >> >> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley
Gotway via
>> RT
>> >> <
>> >> >> > >> > >> > met_help at ucar.edu> wrote:
>> >> >> > >> > >> >
>> >> >> > >> > >> > >
>> >> >> > >> > >> > > <URL:
>> >> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> >> >> > >
>> >> >> > >> > >> > >
>> >> >> > >> > >> > > Ben,
>> >> >> > >> > >> > >
>> >> >> > >> > >> > > Yeah, that'd be great.  If you send me a single
input
>> >> >> forecast
>> >> >> > >> and
>> >> >> > >> > >> > analysis
>> >> >> > >> > >> > > file for comparison, I'll run those 3 versions
of
>> >> grid-stat
>> >> >> > using
>> >> >> > >> > your
>> >> >> > >> > >> > > configuration and quantify the timing
differences.
>> >> >> > >> > >> > >
>> >> >> > >> > >> > > You can either point me to them on theia or
post them
>> to
>> >> our
>> >> >> > >> > anonymous
>> >> >> > >> > >> > ftp
>> >> >> > >> > >> > > site:
>> >> >> > >> > >> > >    https://dtcenter.org/met/users
>> >> /support/met_help.php#ftp
>> >> >> > >> > >> > >
>> >> >> > >> > >> > > Thanks,
>> >> >> > >> > >> > > John
>> >> >> > >> > >> > >
>> >> >> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin Blake
- NOAA
>> >> >> > Affiliate
>> >> >> > >> via
>> >> >> > >> > >> RT <
>> >> >> > >> > >> > > met_help at ucar.edu> wrote:
>> >> >> > >> > >> > >
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
>> >> >> > Ticket/Display.html?id=86792
>> >> >> > >> >
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > > Hi John,
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > > Thanks for letting me know about the grid
stat
>> memory
>> >> >> > >> allocation
>> >> >> > >> > >> issue.
>> >> >> > >> > >> > > I
>> >> >> > >> > >> > > > am currently pointing to Julie's met/7.0
>> installation
>> >> on
>> >> >> > WCOSS
>> >> >> > >> > >> (Tide),
>> >> >> > >> > >> > > and
>> >> >> > >> > >> > > > here is the output of grid_stat --version:
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > > MET Version:    V7.0
>> >> >> > >> > >> > > > Repository:
>> >> >> > >> > >> > > >
>> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
>> >> >> > >> > >> > > > Revision:    5483
>> >> >> > >> > >> > > > Change Date:    2018-06-13 09:57:36 -0600
(Wed, 13
>> Jun
>> >> >> 2018)
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > > Since the grid stat fix was put in place on
July
>> 24th
>> >> the
>> >> >> > >> version
>> >> >> > >> > >> I'm
>> >> >> > >> > >> > > using
>> >> >> > >> > >> > > > likely doesn't have that fix, so I'm not sure
if the
>> >> bug
>> >> >> fix
>> >> >> > >> will
>> >> >> > >> > >> > resolve
>> >> >> > >> > >> > > > the issue or not.  I could try pointing to a
newer
>> >> >> version of
>> >> >> > >> MET.
>> >> >> > >> > >> I
>> >> >> > >> > >> > > > should also let you know that grid stat for
the
>> other
>> >> line
>> >> >> > >> types
>> >> >> > >> > >> does
>> >> >> > >> > >> > not
>> >> >> > >> > >> > > > seem to be slower, I only see a noticeable
slowdown
>> >> with
>> >> >> the
>> >> >> > >> > NBRCNT
>> >> >> > >> > >> > line
>> >> >> > >> > >> > > > type.
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > > I am evaluating the deterministic members of
the
>> HREFv2
>> >> >> and
>> >> >> > the
>> >> >> > >> > NCAR
>> >> >> > >> > >> > > > ensemble as well as their respective means.
I
>> verify
>> >> >> them on
>> >> >> > >> NCEP
>> >> >> > >> > >> grid
>> >> >> > >> > >> > > 227
>> >> >> > >> > >> > > > which is a ~5-km grid (5.079 km).  I can send
you
>> some
>> >> >> sample
>> >> >> > >> > files
>> >> >> > >> > >> for
>> >> >> > >> > >> > > > your testing if that would be easiest.
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > > Thanks!
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > > Ben
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John Halley
Gotway
>> >> via
>> >> >> RT <
>> >> >> > >> > >> > > > met_help at ucar.edu> wrote:
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > > > Hi Ben,
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > I see that you've found the met-7.0 runtime
of
>> >> >> Grid-Stat is
>> >> >> > >> much
>> >> >> > >> > >> > slower
>> >> >> > >> > >> > > > > than the met-6.0 version.  We did find/fix
a
>> >> Grid-Stat
>> >> >> > memory
>> >> >> > >> > >> > > allocation
>> >> >> > >> > >> > > > > issue that's posted to the MET website:
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > >
https://dtcenter.org/met/users/support/known_issues/
>> >> >> > >> > METv7.0/index.php
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > I have two questions...
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > (1) What's the patch level of the code
you're
>> >> currently
>> >> >> > >> running?
>> >> >> > >> > >> > > Please
>> >> >> > >> > >> > > > > run "grid_stat --version" for the
particular
>> >> executable
>> >> >> > >> you're
>> >> >> > >> > >> > running
>> >> >> > >> > >> > > > and
>> >> >> > >> > >> > > > > send me the result.  Should be something
like
>> this:
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > MET Version: V7.0
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > Repository:
>> >> >> > >> > >> > >
>> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > Revision:         5607
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600
(Tue, 24
>> Jul
>> >> >> 2018)
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > (2) Does this bugfix actually fix the
problem
>> you're
>> >> >> > seeing?
>> >> >> > >> > I'd
>> >> >> > >> > >> > like
>> >> >> > >> > >> > > to
>> >> >> > >> > >> > > > > test this out myself using your config
file.  I
>> can
>> >> run
>> >> >> 3
>> >> >> > >> > >> versions:
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > >    - met-6.0 with latest patches
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > >    - met-7.0 with no patches
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > >    - met-7.0 with latest patches
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > What model/analysis are you evaluating?  I
might
>> as
>> >> well
>> >> >> > use
>> >> >> > >> the
>> >> >> > >> > >> same
>> >> >> > >> > >> > > > > resolution for my testing.
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > Thanks,
>> >> >> > >> > >> > > > > John
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin
Blake -
>> NOAA
>> >> >> > >> Affiliate
>> >> >> > >> > >> via
>> >> >> > >> > >> > RT
>> >> >> > >> > >> > > <
>> >> >> > >> > >> > > > > met_help at ucar.edu> wrote:
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > >
>> >> >> > >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792
was
>> acted
>> >> >> upon.
>> >> >> > >> > >> > > > > > Transaction: Ticket created by
>> >> >> benjamin.blake at noaa.gov
>> >> >> > >> > >> > > > > >        Queue: met_help
>> >> >> > >> > >> > > > > >      Subject: FSS calculation with grid
stat
>> >> >> > >> > >> > > > > >        Owner: Nobody
>> >> >> > >> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
>> >> >> > >> > >> > > > > >       Status: new
>> >> >> > >> > >> > > > > >  Ticket <URL:
>> >> >> > >> > >> > >
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > > >
>> >> >> > >> > >> > > > > >
>> >> >> > >> > >> > > > > > Hi,
>> >> >> > >> > >> > > > > >
>> >> >> > >> > >> > > > > > I recently converted all of my MET
configuration
>> >> files
>> >> >> > from
>> >> >> > >> > >> using
>> >> >> > >> > >> > > v6.0
>> >> >> > >> > >> > > > to
>> >> >> > >> > >> > > > > > v7.0.  One issue I'm now having is that
my FSS
>> >> >> > calculations
>> >> >> > >> > are
>> >> >> > >> > >> > > taking
>> >> >> > >> > >> > > > > much
>> >> >> > >> > >> > > > > > longer than they did when I was using
METv6.0.
>> >> >> Currently
>> >> >> > >> it
>> >> >> > >> > >> takes
>> >> >> > >> > >> > > > ~15-20
>> >> >> > >> > >> > > > > > minutes to generate one .stat file.
>> >> >> > >> > >> > > > > >
>> >> >> > >> > >> > > > > > I've attached the grid stat configure
file that
>> I
>> >> am
>> >> >> now
>> >> >> > >> using
>> >> >> > >> > >> with
>> >> >> > >> > >> > > > v7.0
>> >> >> > >> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and the
>> configure
>> >> >> file
>> >> >> > I
>> >> >> > >> was
>> >> >> > >> > >> > using
>> >> >> > >> > >> > > > with
>> >> >> > >> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
>> >> >> > >> > >> > > > > >
>> >> >> > >> > >> > > > > > I modified cat_thresh for APCP A03 and I
also
>> >> reverted
>> >> >> > back
>> >> >> > >> > from
>> >> >> > >> > >> > > SHAPE
>> >> >> > >> > >> > > > =
>> >> >> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the
neighborhood
>> >> methods
>> >> >> > >> section,
>> >> >> > >> > >> but
>> >> >> > >> > >> > > the
>> >> >> > >> > >> > > > > grid
>> >> >> > >> > >> > > > > > stat calculation still takes much longer
than it
>> >> used
>> >> >> to.
>> >> >> > >> I
>> >> >> > >> > >> also
>> >> >> > >> > >> > > tried
>> >> >> > >> > >> > > > > > using the CONUS.poly file in
shared/met/poly
>> >> instead
>> >> >> of
>> >> >> > my
>> >> >> > >> > >> CONUS.nc
>> >> >> > >> > >> > > to
>> >> >> > >> > >> > > > > see
>> >> >> > >> > >> > > > > > if that would make a difference but it
did not.
>> >> The
>> >> >> v7.0
>> >> >> > >> > >> configure
>> >> >> > >> > >> > > > file
>> >> >> > >> > >> > > > > > has many more options in it so perhaps
the
>> >> slowdown is
>> >> >> > >> related
>> >> >> > >> > >> to
>> >> >> > >> > >> > > > > something
>> >> >> > >> > >> > > > > > I added in.
>> >> >> > >> > >> > > > > >
>> >> >> > >> > >> > > > > > Do you have any suggestions for why it
might be
>> >> >> running
>> >> >> > so
>> >> >> > >> > >> slowly?
>> >> >> > >> > >> > > > > >
>> >> >> > >> > >> > > > > > Thanks!
>> >> >> > >> > >> > > > > > Ben
>> >> >> > >> > >> > > > > >
>> >> >> > >> > >> > > > > >
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > > >
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > > >
>> >> >> > >> > >> > >
>> >> >> > >> > >> > >
>> >> >> > >> > >> >
>> >> >> > >> > >> >
>> >> >> > >> > >>
>> >> >> > >> > >>
>> >> >> > >> >
>> >> >> > >> >
>> >> >> > >>
>> >> >> > >>
>> >> >> >
>> >> >> >
>> >> >>
>> >> >>
>> >>
>> >>
>> >
>>
>>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Fri Aug 31 05:46:25 2018

Hi John,

That sounds great!  1:50 is much better than 10:58 or 7:39 so I think
your
proposed solution works well.

I think you or Tara may have mentioned this at one point but when do
you
think MET v8.0 will be released in September?

Thanks,
Ben

On Thu, Aug 30, 2018 at 7:12 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Ben,
>
> I made some good progress on this task today.
>
> I implemented the old smarter logic using the new more flexible data
> structures... making sure that the results are identical.
> I tested 3 versions on your sample configuration (met-6.0, met-7.0,
and my
> proposed changes for met-8.0) and found:
>
>   met-6.0 for squares takes 1:30.
>   met-7.0 for squares takes 10:58.
>   met-8.0 for squares takes 1:50.
>
>   met-7.0 for circles takes 7:39.
>   met-8.0 for circles takes 1:50.
>
> So met-8.0 is still a bit slower than met-6.0... but it's much
closer.  The
> big advantage is that this improvement also applies to the circle
shape.
>
> I know how I could make the logic more slick to speed it up a bit
more, but
> the trouble coding it up.  Rather than moving like a type-writer,
resetting
> every line, we could move like a snake, moving back and forth across
the
> grid.
>
> Thanks,
> John
>
>
>
>
> On Thu, Aug 30, 2018 at 10:57 AM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Ben,
> >
> > Yes, it'll be included in the next beta version... although we'll
be
> > releasing met-8.0 in September.  And we don't currently have a
planned
> beta
> > release version before then.
> >
> > However, rather than simply reverting to the old logic, I'd prefer
to
> > "refine" the new logic.  That way, using the circles would be just
as
> fast
> > as the squares.  So I'm going to try to figure that out and will
let you
> > know how it goes.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Thu, Aug 30, 2018 at 8:31 AM Benjamin Blake - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >>
> >> Hi John,
> >>
> >> Do you know when the next beta version will be released?  Would
it be
> >> possible to have those code changes in that version?  That way I
can
> start
> >> running with that version of the code sooner.
> >>
> >> Thanks again,
> >> Ben
> >>
> >> On Thu, Aug 30, 2018 at 7:33 AM, Benjamin Blake - NOAA Affiliate
<
> >> benjamin.blake at noaa.gov> wrote:
> >>
> >> > Hi John,
> >> >
> >> > Ok great, and yeah the next version of MET would be fine.
> >> >
> >> > Thanks!
> >> > Ben
> >> >
> >> > On Wed, Aug 29, 2018 at 5:02 PM, John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> >> Ben,
> >> >>
> >> >> OK, I modified the development version to call the old logic
for
> >> squares
> >> >> and new (slower) logic for circles.
> >> >>
> >> >> Running your test case on my machine, squares take 1:15 to run
and
> >> circles
> >> >> take 7:27 to run.
> >> >>
> >> >> I presume that you'd like to see this change in the next
version of
> >> MET...
> >> >> at least unless we're able to make the new implementation
much, much
> >> >> faster.
> >> >>
> >> >> Is that right?
> >> >>
> >> >> Thanks,
> >> >> John
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Wed, Aug 29, 2018 at 1:15 PM John Halley Gotway
<johnhg at ucar.edu>
> >> >> wrote:
> >> >>
> >> >> > Ben,
> >> >> >
> >> >> > Yep, that's what I'm thinking too.  For a square
neighborhood, call
> >> the
> >> >> > met-6.0 code.  For a circular neighborhood, call the newer
(slower)
> >> >> code.
> >> >> >
> >> >> > John
> >> >> >
> >> >> > On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake - NOAA
Affiliate
> via
> >> RT
> >> >> <
> >> >> > met_help at ucar.edu> wrote:
> >> >> >
> >> >> >>
> >> >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >> >> >>
> >> >> >> Hi John,
> >> >> >>
> >> >> >> Ahh ok, yeah I'm not sure if you would get the same result
and how
> >> >> well it
> >> >> >> handles missing data.  I'd think the result may be slightly
> >> different.
> >> >> >>
> >> >> >> Would it be possible to have different code for the square
and
> >> circular
> >> >> >> neighborhood options?  That way if a user chose square then
it
> >> would do
> >> >> >> the
> >> >> >> calculations with the method from v6.0 which is more
efficient.  I
> >> >> don't
> >> >> >> know how the code is set up so maybe that's not a viable
option.
> >> >> >>
> >> >> >> Ben
> >> >> >>
> >> >> >> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway via RT
<
> >> >> >> met_help at ucar.edu> wrote:
> >> >> >>
> >> >> >> > Ben,
> >> >> >> >
> >> >> >> > Looking closely at the code, here's the difference...
> >> >> >> >
> >> >> >> > In met-6.0, for square neighborhoods, every time we
increment 1
> >> step
> >> >> in
> >> >> >> the
> >> >> >> > x-direction, we decrement counts from the column we're
> discarding
> >> and
> >> >> >> > increment counts for the column we're adding.
> >> >> >> >
> >> >> >> > In met-7.0, for square and circular neighborhoods, we
reprocess
> >> the
> >> >> >> whole
> >> >> >> > neighborhood each time since for circles, it isn't as
simple as
> >> >> >> > dropping/adding a new column.
> >> >> >> >
> >> >> >> > I agree that using an FFT instead would be preferable.
The
> >> question
> >> >> is
> >> >> >> > whether we get the same result?  And whether or not it
handles
> >> >> missing
> >> >> >> data
> >> >> >> > well.
> >> >> >> >
> >> >> >> > John
> >> >> >> >
> >> >> >> > On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway <
> >> johnhg at ucar.edu
> >> >> >
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> > > Ben,
> >> >> >> > >
> >> >> >> > > Thanks for sending the sample data.  Here's how long
this test
> >> case
> >> >> >> took
> >> >> >> > > on my machine using 2 versions of MET:
> >> >> >> > > met-6.0 took 01:24.
> >> >> >> > > met-7.0 with patches took 10:49.
> >> >> >> > >
> >> >> >> > > So unfortunately (as you just emailed me), the patch
won't fix
> >> >> this.
> >> >> >> > >
> >> >> >> > > The difference in versions is this... In met-6.0, we
only
> >> supported
> >> >> >> > > "square" neighborhoods.  In met-7.0, we added support
for
> >> >> "circular"
> >> >> >> > > neighborhoods.  In doing so, we leveraged existing code
from
> >> >> another
> >> >> >> part
> >> >> >> > > of RAL.  However, as highlighted by your configuration,
it
> >> >> slower...
> >> >> >> > > especially for large neighborhood sizes.
> >> >> >> > >
> >> >> >> > > Let me look more closely at the code to see how we can
address
> >> this
> >> >> >> > issue.
> >> >> >> > >
> >> >> >> > > Thanks,
> >> >> >> > > John
> >> >> >> > >
> >> >> >> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake - NOAA
> Affiliate
> >> >> via
> >> >> >> RT <
> >> >> >> > > met_help at ucar.edu> wrote:
> >> >> >> > >
> >> >> >> > >>
> >> >> >> > >> <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=86792
> >> >
> >> >> >> > >>
> >> >> >> > >> Hi Julie,
> >> >> >> > >>
> >> >> >> > >> Ok great, thanks.  I'll try running grid stat with the
> updated
> >> >> >> version.
> >> >> >> > >> And yeah, I figured you meant Luna. :)
> >> >> >> > >>
> >> >> >> > >> Ben
> >> >> >> > >>
> >> >> >> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik via
RT <
> >> >> >> > >> met_help at ucar.edu
> >> >> >> > >> > wrote:
> >> >> >> > >>
> >> >> >> > >> > Pardon, I meant "luna" not "surge".  :)
> >> >> >> > >> >
> >> >> >> > >> > Julie
> >> >> >> > >> >
> >> >> >> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik <
> >> >> >> jpresto at ucar.edu>
> >> >> >> > >> > wrote:
> >> >> >> > >> >
> >> >> >> > >> > > Hi Ben.
> >> >> >> > >> > >
> >> >> >> > >> > > The version of met-7.0 has now been updated on
tide:
> >> >> >> > >> > >
> >> >> >> > >> > >> MET Version: V7.0
> >> >> >> > >> > >>
> >> >> >> > >> > >> Repository:
> >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_
> >> >> >> > >> > bugfix
> >> >> >> > >> > >>
> >> >> >> > >> > >> Revision: 5607
> >> >> >> > >> > >>
> >> >> >> > >> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24
Jul
> 2018)
> >> >> >> > >> > >>
> >> >> >> > >> > >
> >> >> >> > >> > > I will update on surge once I devonprod access is
> returned.
> >> >> >> > >> > >
> >> >> >> > >> > > Julie
> >> >> >> > >> > >
> >> >> >> > >> > >
> >> >> >> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake -
NOAA
> >> >> Affiliate
> >> >> >> via
> >> >> >> > >> RT <
> >> >> >> > >> > > met_help at ucar.edu> wrote:
> >> >> >> > >> > >
> >> >> >> > >> > >>
> >> >> >> > >> > >> <URL: https://rt.rap.ucar.edu/rt/Tic
> >> >> ket/Display.html?id=86792 >
> >> >> >> > >> > >>
> >> >> >> > >> > >> John,
> >> >> >> > >> > >>
> >> >> >> > >> > >> Sure no problem, I copied a HiResW ARW forecast
file
> and a
> >> >> CCPA
> >> >> >> > file
> >> >> >> > >> > from
> >> >> >> > >> > >> WCOSS to Theia and they are located here:
> >> >> >> > >> > >> /scratch4/NCEPDEV/fv3-cam/noscrub/Benjamin.Blake/
> testfiles
> >> >> >> > >> > >>
> >> >> >> > >> > >> I also put the CONUS.nc mask file that I am using
in
> that
> >> >> >> > directory.
> >> >> >> > >> > >> Using
> >> >> >> > >> > >> that file or CONUS.poly didn't seem to make a
difference
> >> >> though.
> >> >> >> > >> > >>
> >> >> >> > >> > >> Here is the grid stat command I was using on
WCOSS
> (after
> >> >> >> removing
> >> >> >> > >> the
> >> >> >> > >> > >> unrelevant WCOSS paths):
> >> >> >> > >> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
> >> >> >> > ccpa.2017052803.03h.gb2
> >> >> >> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
> >> >> >> > >> > >>
> >> >> >> > >> > >> Julie - That'd be great, thanks!
> >> >> >> > >> > >>
> >> >> >> > >> > >> Ben
> >> >> >> > >> > >>
> >> >> >> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie
Prestopnik via
> RT
> >> <
> >> >> >> > >> > >> met_help at ucar.edu
> >> >> >> > >> > >> > wrote:
> >> >> >> > >> > >>
> >> >> >> > >> > >> > I'll get met-7.0 updated on tide right now.
I'll
> >> follow up
> >> >> >> once
> >> >> >> > it
> >> >> >> > >> > >> update
> >> >> >> > >> > >> > is complete.
> >> >> >> > >> > >> >
> >> >> >> > >> > >> > Julie
> >> >> >> > >> > >> >
> >> >> >> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley
Gotway
> via
> >> RT
> >> >> <
> >> >> >> > >> > >> > met_help at ucar.edu> wrote:
> >> >> >> > >> > >> >
> >> >> >> > >> > >> > >
> >> >> >> > >> > >> > > <URL:
> >> >> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >> >> >> > >
> >> >> >> > >> > >> > >
> >> >> >> > >> > >> > > Ben,
> >> >> >> > >> > >> > >
> >> >> >> > >> > >> > > Yeah, that'd be great.  If you send me a
single
> input
> >> >> >> forecast
> >> >> >> > >> and
> >> >> >> > >> > >> > analysis
> >> >> >> > >> > >> > > file for comparison, I'll run those 3
versions of
> >> >> grid-stat
> >> >> >> > using
> >> >> >> > >> > your
> >> >> >> > >> > >> > > configuration and quantify the timing
differences.
> >> >> >> > >> > >> > >
> >> >> >> > >> > >> > > You can either point me to them on theia or
post
> them
> >> to
> >> >> our
> >> >> >> > >> > anonymous
> >> >> >> > >> > >> > ftp
> >> >> >> > >> > >> > > site:
> >> >> >> > >> > >> > >    https://dtcenter.org/met/users
> >> >> /support/met_help.php#ftp
> >> >> >> > >> > >> > >
> >> >> >> > >> > >> > > Thanks,
> >> >> >> > >> > >> > > John
> >> >> >> > >> > >> > >
> >> >> >> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin
Blake -
> NOAA
> >> >> >> > Affiliate
> >> >> >> > >> via
> >> >> >> > >> > >> RT <
> >> >> >> > >> > >> > > met_help at ucar.edu> wrote:
> >> >> >> > >> > >> > >
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
> >> >> >> > Ticket/Display.html?id=86792
> >> >> >> > >> >
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > > Hi John,
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > > Thanks for letting me know about the grid
stat
> >> memory
> >> >> >> > >> allocation
> >> >> >> > >> > >> issue.
> >> >> >> > >> > >> > > I
> >> >> >> > >> > >> > > > am currently pointing to Julie's met/7.0
> >> installation
> >> >> on
> >> >> >> > WCOSS
> >> >> >> > >> > >> (Tide),
> >> >> >> > >> > >> > > and
> >> >> >> > >> > >> > > > here is the output of grid_stat --version:
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > > MET Version:    V7.0
> >> >> >> > >> > >> > > > Repository:
> >> >> >> > >> > >> > > >
> >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> >> >> >> > >> > >> > > > Revision:    5483
> >> >> >> > >> > >> > > > Change Date:    2018-06-13 09:57:36 -0600
(Wed, 13
> >> Jun
> >> >> >> 2018)
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > > Since the grid stat fix was put in place on
July
> >> 24th
> >> >> the
> >> >> >> > >> version
> >> >> >> > >> > >> I'm
> >> >> >> > >> > >> > > using
> >> >> >> > >> > >> > > > likely doesn't have that fix, so I'm not
sure if
> the
> >> >> bug
> >> >> >> fix
> >> >> >> > >> will
> >> >> >> > >> > >> > resolve
> >> >> >> > >> > >> > > > the issue or not.  I could try pointing to
a newer
> >> >> >> version of
> >> >> >> > >> MET.
> >> >> >> > >> > >> I
> >> >> >> > >> > >> > > > should also let you know that grid stat for
the
> >> other
> >> >> line
> >> >> >> > >> types
> >> >> >> > >> > >> does
> >> >> >> > >> > >> > not
> >> >> >> > >> > >> > > > seem to be slower, I only see a noticeable
> slowdown
> >> >> with
> >> >> >> the
> >> >> >> > >> > NBRCNT
> >> >> >> > >> > >> > line
> >> >> >> > >> > >> > > > type.
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > > I am evaluating the deterministic members
of the
> >> HREFv2
> >> >> >> and
> >> >> >> > the
> >> >> >> > >> > NCAR
> >> >> >> > >> > >> > > > ensemble as well as their respective means.
I
> >> verify
> >> >> >> them on
> >> >> >> > >> NCEP
> >> >> >> > >> > >> grid
> >> >> >> > >> > >> > > 227
> >> >> >> > >> > >> > > > which is a ~5-km grid (5.079 km).  I can
send you
> >> some
> >> >> >> sample
> >> >> >> > >> > files
> >> >> >> > >> > >> for
> >> >> >> > >> > >> > > > your testing if that would be easiest.
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > > Thanks!
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > > Ben
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John
Halley
> Gotway
> >> >> via
> >> >> >> RT <
> >> >> >> > >> > >> > > > met_help at ucar.edu> wrote:
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > > > Hi Ben,
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > I see that you've found the met-7.0
runtime of
> >> >> >> Grid-Stat is
> >> >> >> > >> much
> >> >> >> > >> > >> > slower
> >> >> >> > >> > >> > > > > than the met-6.0 version.  We did
find/fix a
> >> >> Grid-Stat
> >> >> >> > memory
> >> >> >> > >> > >> > > allocation
> >> >> >> > >> > >> > > > > issue that's posted to the MET website:
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > https://dtcenter.org/met/
> users/support/known_issues/
> >> >> >> > >> > METv7.0/index.php
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > I have two questions...
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > (1) What's the patch level of the code
you're
> >> >> currently
> >> >> >> > >> running?
> >> >> >> > >> > >> > > Please
> >> >> >> > >> > >> > > > > run "grid_stat --version" for the
particular
> >> >> executable
> >> >> >> > >> you're
> >> >> >> > >> > >> > running
> >> >> >> > >> > >> > > > and
> >> >> >> > >> > >> > > > > send me the result.  Should be something
like
> >> this:
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > MET Version: V7.0
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > Repository:
> >> >> >> > >> > >> > >
> >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > Revision:         5607
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600
(Tue, 24
> >> Jul
> >> >> >> 2018)
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > (2) Does this bugfix actually fix the
problem
> >> you're
> >> >> >> > seeing?
> >> >> >> > >> > I'd
> >> >> >> > >> > >> > like
> >> >> >> > >> > >> > > to
> >> >> >> > >> > >> > > > > test this out myself using your config
file.  I
> >> can
> >> >> run
> >> >> >> 3
> >> >> >> > >> > >> versions:
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > >    - met-6.0 with latest patches
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > >    - met-7.0 with no patches
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > >    - met-7.0 with latest patches
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > What model/analysis are you evaluating?
I might
> >> as
> >> >> well
> >> >> >> > use
> >> >> >> > >> the
> >> >> >> > >> > >> same
> >> >> >> > >> > >> > > > > resolution for my testing.
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > Thanks,
> >> >> >> > >> > >> > > > > John
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM Benjamin
Blake -
> >> NOAA
> >> >> >> > >> Affiliate
> >> >> >> > >> > >> via
> >> >> >> > >> > >> > RT
> >> >> >> > >> > >> > > <
> >> >> >> > >> > >> > > > > met_help at ucar.edu> wrote:
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > >
> >> >> >> > >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request 86792
was
> >> acted
> >> >> >> upon.
> >> >> >> > >> > >> > > > > > Transaction: Ticket created by
> >> >> >> benjamin.blake at noaa.gov
> >> >> >> > >> > >> > > > > >        Queue: met_help
> >> >> >> > >> > >> > > > > >      Subject: FSS calculation with grid
stat
> >> >> >> > >> > >> > > > > >        Owner: Nobody
> >> >> >> > >> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
> >> >> >> > >> > >> > > > > >       Status: new
> >> >> >> > >> > >> > > > > >  Ticket <URL:
> >> >> >> > >> > >> > >
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > > >
> >> >> >> > >> > >> > > > > >
> >> >> >> > >> > >> > > > > > Hi,
> >> >> >> > >> > >> > > > > >
> >> >> >> > >> > >> > > > > > I recently converted all of my MET
> configuration
> >> >> files
> >> >> >> > from
> >> >> >> > >> > >> using
> >> >> >> > >> > >> > > v6.0
> >> >> >> > >> > >> > > > to
> >> >> >> > >> > >> > > > > > v7.0.  One issue I'm now having is that
my FSS
> >> >> >> > calculations
> >> >> >> > >> > are
> >> >> >> > >> > >> > > taking
> >> >> >> > >> > >> > > > > much
> >> >> >> > >> > >> > > > > > longer than they did when I was using
METv6.0.
> >> >> >> Currently
> >> >> >> > >> it
> >> >> >> > >> > >> takes
> >> >> >> > >> > >> > > > ~15-20
> >> >> >> > >> > >> > > > > > minutes to generate one .stat file.
> >> >> >> > >> > >> > > > > >
> >> >> >> > >> > >> > > > > > I've attached the grid stat configure
file
> that
> >> I
> >> >> am
> >> >> >> now
> >> >> >> > >> using
> >> >> >> > >> > >> with
> >> >> >> > >> > >> > > > v7.0
> >> >> >> > >> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and
the
> >> configure
> >> >> >> file
> >> >> >> > I
> >> >> >> > >> was
> >> >> >> > >> > >> > using
> >> >> >> > >> > >> > > > with
> >> >> >> > >> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> >> >> >> > >> > >> > > > > >
> >> >> >> > >> > >> > > > > > I modified cat_thresh for APCP A03 and
I also
> >> >> reverted
> >> >> >> > back
> >> >> >> > >> > from
> >> >> >> > >> > >> > > SHAPE
> >> >> >> > >> > >> > > > =
> >> >> >> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the
neighborhood
> >> >> methods
> >> >> >> > >> section,
> >> >> >> > >> > >> but
> >> >> >> > >> > >> > > the
> >> >> >> > >> > >> > > > > grid
> >> >> >> > >> > >> > > > > > stat calculation still takes much
longer than
> it
> >> >> used
> >> >> >> to.
> >> >> >> > >> I
> >> >> >> > >> > >> also
> >> >> >> > >> > >> > > tried
> >> >> >> > >> > >> > > > > > using the CONUS.poly file in
shared/met/poly
> >> >> instead
> >> >> >> of
> >> >> >> > my
> >> >> >> > >> > >> CONUS.nc
> >> >> >> > >> > >> > > to
> >> >> >> > >> > >> > > > > see
> >> >> >> > >> > >> > > > > > if that would make a difference but it
did
> not.
> >> >> The
> >> >> >> v7.0
> >> >> >> > >> > >> configure
> >> >> >> > >> > >> > > > file
> >> >> >> > >> > >> > > > > > has many more options in it so perhaps
the
> >> >> slowdown is
> >> >> >> > >> related
> >> >> >> > >> > >> to
> >> >> >> > >> > >> > > > > something
> >> >> >> > >> > >> > > > > > I added in.
> >> >> >> > >> > >> > > > > >
> >> >> >> > >> > >> > > > > > Do you have any suggestions for why it
might
> be
> >> >> >> running
> >> >> >> > so
> >> >> >> > >> > >> slowly?
> >> >> >> > >> > >> > > > > >
> >> >> >> > >> > >> > > > > > Thanks!
> >> >> >> > >> > >> > > > > > Ben
> >> >> >> > >> > >> > > > > >
> >> >> >> > >> > >> > > > > >
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > > >
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > > >
> >> >> >> > >> > >> > >
> >> >> >> > >> > >> > >
> >> >> >> > >> > >> >
> >> >> >> > >> > >> >
> >> >> >> > >> > >>
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> > >>
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >> >
> >>
> >>
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: John Halley Gotway
Time: Fri Aug 31 11:37:12 2018

Ben,

I tried out the updated logic for computing the fractional coverage
field -
moving like a snake instead of a typewriter.  I never actually got it
to
compute the correct values... very close, but not right.  And when
computing those wrong values, I found that it was only a few seconds
faster
on our test case: 1:46 versus 1:50.  Randy and I talked and decided
that
the relatively small gain in efficiency isn't worth the large increase
in
complexity in the code.

So I eventually gave up.  Hopefully the 1:50 time is acceptable.

As for release date, the goal is mid-September.  Do you have a
particular
deadline in September we should know about?

Thanks,
John

On Fri, Aug 31, 2018 at 5:46 AM Benjamin Blake - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>
> Hi John,
>
> That sounds great!  1:50 is much better than 10:58 or 7:39 so I
think your
> proposed solution works well.
>
> I think you or Tara may have mentioned this at one point but when do
you
> think MET v8.0 will be released in September?
>
> Thanks,
> Ben
>
> On Thu, Aug 30, 2018 at 7:12 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Ben,
> >
> > I made some good progress on this task today.
> >
> > I implemented the old smarter logic using the new more flexible
data
> > structures... making sure that the results are identical.
> > I tested 3 versions on your sample configuration (met-6.0, met-
7.0, and
> my
> > proposed changes for met-8.0) and found:
> >
> >   met-6.0 for squares takes 1:30.
> >   met-7.0 for squares takes 10:58.
> >   met-8.0 for squares takes 1:50.
> >
> >   met-7.0 for circles takes 7:39.
> >   met-8.0 for circles takes 1:50.
> >
> > So met-8.0 is still a bit slower than met-6.0... but it's much
closer.
> The
> > big advantage is that this improvement also applies to the circle
shape.
> >
> > I know how I could make the logic more slick to speed it up a bit
more,
> but
> > the trouble coding it up.  Rather than moving like a type-writer,
> resetting
> > every line, we could move like a snake, moving back and forth
across the
> > grid.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> > On Thu, Aug 30, 2018 at 10:57 AM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Ben,
> > >
> > > Yes, it'll be included in the next beta version... although
we'll be
> > > releasing met-8.0 in September.  And we don't currently have a
planned
> > beta
> > > release version before then.
> > >
> > > However, rather than simply reverting to the old logic, I'd
prefer to
> > > "refine" the new logic.  That way, using the circles would be
just as
> > fast
> > > as the squares.  So I'm going to try to figure that out and will
let
> you
> > > know how it goes.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Thu, Aug 30, 2018 at 8:31 AM Benjamin Blake - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > >>
> > >> Hi John,
> > >>
> > >> Do you know when the next beta version will be released?  Would
it be
> > >> possible to have those code changes in that version?  That way
I can
> > start
> > >> running with that version of the code sooner.
> > >>
> > >> Thanks again,
> > >> Ben
> > >>
> > >> On Thu, Aug 30, 2018 at 7:33 AM, Benjamin Blake - NOAA
Affiliate <
> > >> benjamin.blake at noaa.gov> wrote:
> > >>
> > >> > Hi John,
> > >> >
> > >> > Ok great, and yeah the next version of MET would be fine.
> > >> >
> > >> > Thanks!
> > >> > Ben
> > >> >
> > >> > On Wed, Aug 29, 2018 at 5:02 PM, John Halley Gotway via RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> >> Ben,
> > >> >>
> > >> >> OK, I modified the development version to call the old logic
for
> > >> squares
> > >> >> and new (slower) logic for circles.
> > >> >>
> > >> >> Running your test case on my machine, squares take 1:15 to
run and
> > >> circles
> > >> >> take 7:27 to run.
> > >> >>
> > >> >> I presume that you'd like to see this change in the next
version of
> > >> MET...
> > >> >> at least unless we're able to make the new implementation
much,
> much
> > >> >> faster.
> > >> >>
> > >> >> Is that right?
> > >> >>
> > >> >> Thanks,
> > >> >> John
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Wed, Aug 29, 2018 at 1:15 PM John Halley Gotway <
> johnhg at ucar.edu>
> > >> >> wrote:
> > >> >>
> > >> >> > Ben,
> > >> >> >
> > >> >> > Yep, that's what I'm thinking too.  For a square
neighborhood,
> call
> > >> the
> > >> >> > met-6.0 code.  For a circular neighborhood, call the newer
> (slower)
> > >> >> code.
> > >> >> >
> > >> >> > John
> > >> >> >
> > >> >> > On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake - NOAA
Affiliate
> > via
> > >> RT
> > >> >> <
> > >> >> > met_help at ucar.edu> wrote:
> > >> >> >
> > >> >> >>
> > >> >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > >> >> >>
> > >> >> >> Hi John,
> > >> >> >>
> > >> >> >> Ahh ok, yeah I'm not sure if you would get the same
result and
> how
> > >> >> well it
> > >> >> >> handles missing data.  I'd think the result may be
slightly
> > >> different.
> > >> >> >>
> > >> >> >> Would it be possible to have different code for the
square and
> > >> circular
> > >> >> >> neighborhood options?  That way if a user chose square
then it
> > >> would do
> > >> >> >> the
> > >> >> >> calculations with the method from v6.0 which is more
> efficient.  I
> > >> >> don't
> > >> >> >> know how the code is set up so maybe that's not a viable
option.
> > >> >> >>
> > >> >> >> Ben
> > >> >> >>
> > >> >> >> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway via
RT <
> > >> >> >> met_help at ucar.edu> wrote:
> > >> >> >>
> > >> >> >> > Ben,
> > >> >> >> >
> > >> >> >> > Looking closely at the code, here's the difference...
> > >> >> >> >
> > >> >> >> > In met-6.0, for square neighborhoods, every time we
increment
> 1
> > >> step
> > >> >> in
> > >> >> >> the
> > >> >> >> > x-direction, we decrement counts from the column we're
> > discarding
> > >> and
> > >> >> >> > increment counts for the column we're adding.
> > >> >> >> >
> > >> >> >> > In met-7.0, for square and circular neighborhoods, we
> reprocess
> > >> the
> > >> >> >> whole
> > >> >> >> > neighborhood each time since for circles, it isn't as
simple
> as
> > >> >> >> > dropping/adding a new column.
> > >> >> >> >
> > >> >> >> > I agree that using an FFT instead would be preferable.
The
> > >> question
> > >> >> is
> > >> >> >> > whether we get the same result?  And whether or not it
handles
> > >> >> missing
> > >> >> >> data
> > >> >> >> > well.
> > >> >> >> >
> > >> >> >> > John
> > >> >> >> >
> > >> >> >> > On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway <
> > >> johnhg at ucar.edu
> > >> >> >
> > >> >> >> > wrote:
> > >> >> >> >
> > >> >> >> > > Ben,
> > >> >> >> > >
> > >> >> >> > > Thanks for sending the sample data.  Here's how long
this
> test
> > >> case
> > >> >> >> took
> > >> >> >> > > on my machine using 2 versions of MET:
> > >> >> >> > > met-6.0 took 01:24.
> > >> >> >> > > met-7.0 with patches took 10:49.
> > >> >> >> > >
> > >> >> >> > > So unfortunately (as you just emailed me), the patch
won't
> fix
> > >> >> this.
> > >> >> >> > >
> > >> >> >> > > The difference in versions is this... In met-6.0, we
only
> > >> supported
> > >> >> >> > > "square" neighborhoods.  In met-7.0, we added support
for
> > >> >> "circular"
> > >> >> >> > > neighborhoods.  In doing so, we leveraged existing
code from
> > >> >> another
> > >> >> >> part
> > >> >> >> > > of RAL.  However, as highlighted by your
configuration, it
> > >> >> slower...
> > >> >> >> > > especially for large neighborhood sizes.
> > >> >> >> > >
> > >> >> >> > > Let me look more closely at the code to see how we
can
> address
> > >> this
> > >> >> >> > issue.
> > >> >> >> > >
> > >> >> >> > > Thanks,
> > >> >> >> > > John
> > >> >> >> > >
> > >> >> >> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake -
NOAA
> > Affiliate
> > >> >> via
> > >> >> >> RT <
> > >> >> >> > > met_help at ucar.edu> wrote:
> > >> >> >> > >
> > >> >> >> > >>
> > >> >> >> > >> <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=86792
> > >> >
> > >> >> >> > >>
> > >> >> >> > >> Hi Julie,
> > >> >> >> > >>
> > >> >> >> > >> Ok great, thanks.  I'll try running grid stat with
the
> > updated
> > >> >> >> version.
> > >> >> >> > >> And yeah, I figured you meant Luna. :)
> > >> >> >> > >>
> > >> >> >> > >> Ben
> > >> >> >> > >>
> > >> >> >> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik
via RT <
> > >> >> >> > >> met_help at ucar.edu
> > >> >> >> > >> > wrote:
> > >> >> >> > >>
> > >> >> >> > >> > Pardon, I meant "luna" not "surge".  :)
> > >> >> >> > >> >
> > >> >> >> > >> > Julie
> > >> >> >> > >> >
> > >> >> >> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie Prestopnik
<
> > >> >> >> jpresto at ucar.edu>
> > >> >> >> > >> > wrote:
> > >> >> >> > >> >
> > >> >> >> > >> > > Hi Ben.
> > >> >> >> > >> > >
> > >> >> >> > >> > > The version of met-7.0 has now been updated on
tide:
> > >> >> >> > >> > >
> > >> >> >> > >> > >> MET Version: V7.0
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> Repository:
> > >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_
> > >> >> >> > >> > bugfix
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> Revision: 5607
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue, 24
Jul
> > 2018)
> > >> >> >> > >> > >>
> > >> >> >> > >> > >
> > >> >> >> > >> > > I will update on surge once I devonprod access
is
> > returned.
> > >> >> >> > >> > >
> > >> >> >> > >> > > Julie
> > >> >> >> > >> > >
> > >> >> >> > >> > >
> > >> >> >> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin Blake
- NOAA
> > >> >> Affiliate
> > >> >> >> via
> > >> >> >> > >> RT <
> > >> >> >> > >> > > met_help at ucar.edu> wrote:
> > >> >> >> > >> > >
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> <URL: https://rt.rap.ucar.edu/rt/Tic
> > >> >> ket/Display.html?id=86792 >
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> John,
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> Sure no problem, I copied a HiResW ARW forecast
file
> > and a
> > >> >> CCPA
> > >> >> >> > file
> > >> >> >> > >> > from
> > >> >> >> > >> > >> WCOSS to Theia and they are located here:
> > >> >> >> > >> > >> /scratch4/NCEPDEV/fv3-
cam/noscrub/Benjamin.Blake/
> > testfiles
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> I also put the CONUS.nc mask file that I am
using in
> > that
> > >> >> >> > directory.
> > >> >> >> > >> > >> Using
> > >> >> >> > >> > >> that file or CONUS.poly didn't seem to make a
> difference
> > >> >> though.
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> Here is the grid stat command I was using on
WCOSS
> > (after
> > >> >> >> removing
> > >> >> >> > >> the
> > >> >> >> > >> > >> unrelevant WCOSS paths):
> > >> >> >> > >> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
> > >> >> >> > ccpa.2017052803.03h.gb2
> > >> >> >> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> Julie - That'd be great, thanks!
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> Ben
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie
Prestopnik via
> > RT
> > >> <
> > >> >> >> > >> > >> met_help at ucar.edu
> > >> >> >> > >> > >> > wrote:
> > >> >> >> > >> > >>
> > >> >> >> > >> > >> > I'll get met-7.0 updated on tide right now.
I'll
> > >> follow up
> > >> >> >> once
> > >> >> >> > it
> > >> >> >> > >> > >> update
> > >> >> >> > >> > >> > is complete.
> > >> >> >> > >> > >> >
> > >> >> >> > >> > >> > Julie
> > >> >> >> > >> > >> >
> > >> >> >> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John Halley
Gotway
> > via
> > >> RT
> > >> >> <
> > >> >> >> > >> > >> > met_help at ucar.edu> wrote:
> > >> >> >> > >> > >> >
> > >> >> >> > >> > >> > >
> > >> >> >> > >> > >> > > <URL:
> > >> >> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > >> >> >> > >
> > >> >> >> > >> > >> > >
> > >> >> >> > >> > >> > > Ben,
> > >> >> >> > >> > >> > >
> > >> >> >> > >> > >> > > Yeah, that'd be great.  If you send me a
single
> > input
> > >> >> >> forecast
> > >> >> >> > >> and
> > >> >> >> > >> > >> > analysis
> > >> >> >> > >> > >> > > file for comparison, I'll run those 3
versions of
> > >> >> grid-stat
> > >> >> >> > using
> > >> >> >> > >> > your
> > >> >> >> > >> > >> > > configuration and quantify the timing
differences.
> > >> >> >> > >> > >> > >
> > >> >> >> > >> > >> > > You can either point me to them on theia or
post
> > them
> > >> to
> > >> >> our
> > >> >> >> > >> > anonymous
> > >> >> >> > >> > >> > ftp
> > >> >> >> > >> > >> > > site:
> > >> >> >> > >> > >> > >    https://dtcenter.org/met/users
> > >> >> /support/met_help.php#ftp
> > >> >> >> > >> > >> > >
> > >> >> >> > >> > >> > > Thanks,
> > >> >> >> > >> > >> > > John
> > >> >> >> > >> > >> > >
> > >> >> >> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin
Blake -
> > NOAA
> > >> >> >> > Affiliate
> > >> >> >> > >> via
> > >> >> >> > >> > >> RT <
> > >> >> >> > >> > >> > > met_help at ucar.edu> wrote:
> > >> >> >> > >> > >> > >
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
> > >> >> >> > Ticket/Display.html?id=86792
> > >> >> >> > >> >
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > > Hi John,
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > > Thanks for letting me know about the grid
stat
> > >> memory
> > >> >> >> > >> allocation
> > >> >> >> > >> > >> issue.
> > >> >> >> > >> > >> > > I
> > >> >> >> > >> > >> > > > am currently pointing to Julie's met/7.0
> > >> installation
> > >> >> on
> > >> >> >> > WCOSS
> > >> >> >> > >> > >> (Tide),
> > >> >> >> > >> > >> > > and
> > >> >> >> > >> > >> > > > here is the output of grid_stat
--version:
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > > MET Version:    V7.0
> > >> >> >> > >> > >> > > > Repository:
> > >> >> >> > >> > >> > > >
> > >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> > >> >> >> > >> > >> > > > Revision:    5483
> > >> >> >> > >> > >> > > > Change Date:    2018-06-13 09:57:36 -0600
(Wed,
> 13
> > >> Jun
> > >> >> >> 2018)
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > > Since the grid stat fix was put in place
on July
> > >> 24th
> > >> >> the
> > >> >> >> > >> version
> > >> >> >> > >> > >> I'm
> > >> >> >> > >> > >> > > using
> > >> >> >> > >> > >> > > > likely doesn't have that fix, so I'm not
sure if
> > the
> > >> >> bug
> > >> >> >> fix
> > >> >> >> > >> will
> > >> >> >> > >> > >> > resolve
> > >> >> >> > >> > >> > > > the issue or not.  I could try pointing
to a
> newer
> > >> >> >> version of
> > >> >> >> > >> MET.
> > >> >> >> > >> > >> I
> > >> >> >> > >> > >> > > > should also let you know that grid stat
for the
> > >> other
> > >> >> line
> > >> >> >> > >> types
> > >> >> >> > >> > >> does
> > >> >> >> > >> > >> > not
> > >> >> >> > >> > >> > > > seem to be slower, I only see a
noticeable
> > slowdown
> > >> >> with
> > >> >> >> the
> > >> >> >> > >> > NBRCNT
> > >> >> >> > >> > >> > line
> > >> >> >> > >> > >> > > > type.
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > > I am evaluating the deterministic members
of the
> > >> HREFv2
> > >> >> >> and
> > >> >> >> > the
> > >> >> >> > >> > NCAR
> > >> >> >> > >> > >> > > > ensemble as well as their respective
means.  I
> > >> verify
> > >> >> >> them on
> > >> >> >> > >> NCEP
> > >> >> >> > >> > >> grid
> > >> >> >> > >> > >> > > 227
> > >> >> >> > >> > >> > > > which is a ~5-km grid (5.079 km).  I can
send
> you
> > >> some
> > >> >> >> sample
> > >> >> >> > >> > files
> > >> >> >> > >> > >> for
> > >> >> >> > >> > >> > > > your testing if that would be easiest.
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > > Thanks!
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > > Ben
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John
Halley
> > Gotway
> > >> >> via
> > >> >> >> RT <
> > >> >> >> > >> > >> > > > met_help at ucar.edu> wrote:
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > > > Hi Ben,
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > I see that you've found the met-7.0
runtime of
> > >> >> >> Grid-Stat is
> > >> >> >> > >> much
> > >> >> >> > >> > >> > slower
> > >> >> >> > >> > >> > > > > than the met-6.0 version.  We did
find/fix a
> > >> >> Grid-Stat
> > >> >> >> > memory
> > >> >> >> > >> > >> > > allocation
> > >> >> >> > >> > >> > > > > issue that's posted to the MET website:
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > https://dtcenter.org/met/
> > users/support/known_issues/
> > >> >> >> > >> > METv7.0/index.php
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > I have two questions...
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > (1) What's the patch level of the code
you're
> > >> >> currently
> > >> >> >> > >> running?
> > >> >> >> > >> > >> > > Please
> > >> >> >> > >> > >> > > > > run "grid_stat --version" for the
particular
> > >> >> executable
> > >> >> >> > >> you're
> > >> >> >> > >> > >> > running
> > >> >> >> > >> > >> > > > and
> > >> >> >> > >> > >> > > > > send me the result.  Should be
something like
> > >> this:
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > MET Version: V7.0
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > Repository:
> > >> >> >> > >> > >> > >
> > >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > Revision:         5607
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > Change Date: 2018-07-24 14:46:55 -0600
(Tue,
> 24
> > >> Jul
> > >> >> >> 2018)
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > (2) Does this bugfix actually fix the
problem
> > >> you're
> > >> >> >> > seeing?
> > >> >> >> > >> > I'd
> > >> >> >> > >> > >> > like
> > >> >> >> > >> > >> > > to
> > >> >> >> > >> > >> > > > > test this out myself using your config
file.
> I
> > >> can
> > >> >> run
> > >> >> >> 3
> > >> >> >> > >> > >> versions:
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > >    - met-6.0 with latest patches
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > >    - met-7.0 with no patches
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > >    - met-7.0 with latest patches
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > What model/analysis are you evaluating?
I
> might
> > >> as
> > >> >> well
> > >> >> >> > use
> > >> >> >> > >> the
> > >> >> >> > >> > >> same
> > >> >> >> > >> > >> > > > > resolution for my testing.
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > Thanks,
> > >> >> >> > >> > >> > > > > John
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM
Benjamin
> Blake -
> > >> NOAA
> > >> >> >> > >> Affiliate
> > >> >> >> > >> > >> via
> > >> >> >> > >> > >> > RT
> > >> >> >> > >> > >> > > <
> > >> >> >> > >> > >> > > > > met_help at ucar.edu> wrote:
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > >
> > >> >> >> > >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request
86792 was
> > >> acted
> > >> >> >> upon.
> > >> >> >> > >> > >> > > > > > Transaction: Ticket created by
> > >> >> >> benjamin.blake at noaa.gov
> > >> >> >> > >> > >> > > > > >        Queue: met_help
> > >> >> >> > >> > >> > > > > >      Subject: FSS calculation with
grid stat
> > >> >> >> > >> > >> > > > > >        Owner: Nobody
> > >> >> >> > >> > >> > > > > >   Requestors: benjamin.blake at noaa.gov
> > >> >> >> > >> > >> > > > > >       Status: new
> > >> >> >> > >> > >> > > > > >  Ticket <URL:
> > >> >> >> > >> > >> > >
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > > >
> > >> >> >> > >> > >> > > > > >
> > >> >> >> > >> > >> > > > > > Hi,
> > >> >> >> > >> > >> > > > > >
> > >> >> >> > >> > >> > > > > > I recently converted all of my MET
> > configuration
> > >> >> files
> > >> >> >> > from
> > >> >> >> > >> > >> using
> > >> >> >> > >> > >> > > v6.0
> > >> >> >> > >> > >> > > > to
> > >> >> >> > >> > >> > > > > > v7.0.  One issue I'm now having is
that my
> FSS
> > >> >> >> > calculations
> > >> >> >> > >> > are
> > >> >> >> > >> > >> > > taking
> > >> >> >> > >> > >> > > > > much
> > >> >> >> > >> > >> > > > > > longer than they did when I was using
> METv6.0.
> > >> >> >> Currently
> > >> >> >> > >> it
> > >> >> >> > >> > >> takes
> > >> >> >> > >> > >> > > > ~15-20
> > >> >> >> > >> > >> > > > > > minutes to generate one .stat file.
> > >> >> >> > >> > >> > > > > >
> > >> >> >> > >> > >> > > > > > I've attached the grid stat configure
file
> > that
> > >> I
> > >> >> am
> > >> >> >> now
> > >> >> >> > >> using
> > >> >> >> > >> > >> with
> > >> >> >> > >> > >> > > > v7.0
> > >> >> >> > >> > >> > > > > > (GridStatConfig_APCP03_FSS_href) and
the
> > >> configure
> > >> >> >> file
> > >> >> >> > I
> > >> >> >> > >> was
> > >> >> >> > >> > >> > using
> > >> >> >> > >> > >> > > > with
> > >> >> >> > >> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> > >> >> >> > >> > >> > > > > >
> > >> >> >> > >> > >> > > > > > I modified cat_thresh for APCP A03
and I
> also
> > >> >> reverted
> > >> >> >> > back
> > >> >> >> > >> > from
> > >> >> >> > >> > >> > > SHAPE
> > >> >> >> > >> > >> > > > =
> > >> >> >> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the
> neighborhood
> > >> >> methods
> > >> >> >> > >> section,
> > >> >> >> > >> > >> but
> > >> >> >> > >> > >> > > the
> > >> >> >> > >> > >> > > > > grid
> > >> >> >> > >> > >> > > > > > stat calculation still takes much
longer
> than
> > it
> > >> >> used
> > >> >> >> to.
> > >> >> >> > >> I
> > >> >> >> > >> > >> also
> > >> >> >> > >> > >> > > tried
> > >> >> >> > >> > >> > > > > > using the CONUS.poly file in
shared/met/poly
> > >> >> instead
> > >> >> >> of
> > >> >> >> > my
> > >> >> >> > >> > >> CONUS.nc
> > >> >> >> > >> > >> > > to
> > >> >> >> > >> > >> > > > > see
> > >> >> >> > >> > >> > > > > > if that would make a difference but
it did
> > not.
> > >> >> The
> > >> >> >> v7.0
> > >> >> >> > >> > >> configure
> > >> >> >> > >> > >> > > > file
> > >> >> >> > >> > >> > > > > > has many more options in it so
perhaps the
> > >> >> slowdown is
> > >> >> >> > >> related
> > >> >> >> > >> > >> to
> > >> >> >> > >> > >> > > > > something
> > >> >> >> > >> > >> > > > > > I added in.
> > >> >> >> > >> > >> > > > > >
> > >> >> >> > >> > >> > > > > > Do you have any suggestions for why
it might
> > be
> > >> >> >> running
> > >> >> >> > so
> > >> >> >> > >> > >> slowly?
> > >> >> >> > >> > >> > > > > >
> > >> >> >> > >> > >> > > > > > Thanks!
> > >> >> >> > >> > >> > > > > > Ben
> > >> >> >> > >> > >> > > > > >
> > >> >> >> > >> > >> > > > > >
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > > >
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > > >
> > >> >> >> > >> > >> > >
> > >> >> >> > >> > >> > >
> > >> >> >> > >> > >> >
> > >> >> >> > >> > >> >
> > >> >> >> > >> > >>
> > >> >> >> > >> > >>
> > >> >> >> > >> >
> > >> >> >> > >> >
> > >> >> >> > >>
> > >> >> >> > >>
> > >> >> >> >
> > >> >> >> >
> > >> >> >>
> > >> >> >>
> > >> >>
> > >> >>
> > >> >
> > >>
> > >>
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Fri Aug 31 11:46:18 2018

Hi John,

The 1:50 time is certainly acceptable and is a huge improvement from
what
is in v7.0.  Thanks again for working to fix that!

Mid-September would be fine.  The stats that I want to generate will
be for
the AMS annual meeting in January 2019.

Thanks,
Ben

On Fri, Aug 31, 2018 at 1:37 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Ben,
>
> I tried out the updated logic for computing the fractional coverage
field -
> moving like a snake instead of a typewriter.  I never actually got
it to
> compute the correct values... very close, but not right.  And when
> computing those wrong values, I found that it was only a few seconds
faster
> on our test case: 1:46 versus 1:50.  Randy and I talked and decided
that
> the relatively small gain in efficiency isn't worth the large
increase in
> complexity in the code.
>
> So I eventually gave up.  Hopefully the 1:50 time is acceptable.
>
> As for release date, the goal is mid-September.  Do you have a
particular
> deadline in September we should know about?
>
> Thanks,
> John
>
> On Fri, Aug 31, 2018 at 5:46 AM Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >
> > Hi John,
> >
> > That sounds great!  1:50 is much better than 10:58 or 7:39 so I
think
> your
> > proposed solution works well.
> >
> > I think you or Tara may have mentioned this at one point but when
do you
> > think MET v8.0 will be released in September?
> >
> > Thanks,
> > Ben
> >
> > On Thu, Aug 30, 2018 at 7:12 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Ben,
> > >
> > > I made some good progress on this task today.
> > >
> > > I implemented the old smarter logic using the new more flexible
data
> > > structures... making sure that the results are identical.
> > > I tested 3 versions on your sample configuration (met-6.0, met-
7.0, and
> > my
> > > proposed changes for met-8.0) and found:
> > >
> > >   met-6.0 for squares takes 1:30.
> > >   met-7.0 for squares takes 10:58.
> > >   met-8.0 for squares takes 1:50.
> > >
> > >   met-7.0 for circles takes 7:39.
> > >   met-8.0 for circles takes 1:50.
> > >
> > > So met-8.0 is still a bit slower than met-6.0... but it's much
closer.
> > The
> > > big advantage is that this improvement also applies to the
circle
> shape.
> > >
> > > I know how I could make the logic more slick to speed it up a
bit more,
> > but
> > > the trouble coding it up.  Rather than moving like a type-
writer,
> > resetting
> > > every line, we could move like a snake, moving back and forth
across
> the
> > > grid.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > > On Thu, Aug 30, 2018 at 10:57 AM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > Ben,
> > > >
> > > > Yes, it'll be included in the next beta version... although
we'll be
> > > > releasing met-8.0 in September.  And we don't currently have a
> planned
> > > beta
> > > > release version before then.
> > > >
> > > > However, rather than simply reverting to the old logic, I'd
prefer to
> > > > "refine" the new logic.  That way, using the circles would be
just as
> > > fast
> > > > as the squares.  So I'm going to try to figure that out and
will let
> > you
> > > > know how it goes.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > > On Thu, Aug 30, 2018 at 8:31 AM Benjamin Blake - NOAA
Affiliate via
> RT
> > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>
> > > >>
> > > >> Hi John,
> > > >>
> > > >> Do you know when the next beta version will be released?
Would it
> be
> > > >> possible to have those code changes in that version?  That
way I can
> > > start
> > > >> running with that version of the code sooner.
> > > >>
> > > >> Thanks again,
> > > >> Ben
> > > >>
> > > >> On Thu, Aug 30, 2018 at 7:33 AM, Benjamin Blake - NOAA
Affiliate <
> > > >> benjamin.blake at noaa.gov> wrote:
> > > >>
> > > >> > Hi John,
> > > >> >
> > > >> > Ok great, and yeah the next version of MET would be fine.
> > > >> >
> > > >> > Thanks!
> > > >> > Ben
> > > >> >
> > > >> > On Wed, Aug 29, 2018 at 5:02 PM, John Halley Gotway via RT
<
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> >> Ben,
> > > >> >>
> > > >> >> OK, I modified the development version to call the old
logic for
> > > >> squares
> > > >> >> and new (slower) logic for circles.
> > > >> >>
> > > >> >> Running your test case on my machine, squares take 1:15 to
run
> and
> > > >> circles
> > > >> >> take 7:27 to run.
> > > >> >>
> > > >> >> I presume that you'd like to see this change in the next
version
> of
> > > >> MET...
> > > >> >> at least unless we're able to make the new implementation
much,
> > much
> > > >> >> faster.
> > > >> >>
> > > >> >> Is that right?
> > > >> >>
> > > >> >> Thanks,
> > > >> >> John
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Wed, Aug 29, 2018 at 1:15 PM John Halley Gotway <
> > johnhg at ucar.edu>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Ben,
> > > >> >> >
> > > >> >> > Yep, that's what I'm thinking too.  For a square
neighborhood,
> > call
> > > >> the
> > > >> >> > met-6.0 code.  For a circular neighborhood, call the
newer
> > (slower)
> > > >> >> code.
> > > >> >> >
> > > >> >> > John
> > > >> >> >
> > > >> >> > On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake - NOAA
> Affiliate
> > > via
> > > >> RT
> > > >> >> <
> > > >> >> > met_help at ucar.edu> wrote:
> > > >> >> >
> > > >> >> >>
> > > >> >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> >
> > > >> >> >>
> > > >> >> >> Hi John,
> > > >> >> >>
> > > >> >> >> Ahh ok, yeah I'm not sure if you would get the same
result and
> > how
> > > >> >> well it
> > > >> >> >> handles missing data.  I'd think the result may be
slightly
> > > >> different.
> > > >> >> >>
> > > >> >> >> Would it be possible to have different code for the
square and
> > > >> circular
> > > >> >> >> neighborhood options?  That way if a user chose square
then it
> > > >> would do
> > > >> >> >> the
> > > >> >> >> calculations with the method from v6.0 which is more
> > efficient.  I
> > > >> >> don't
> > > >> >> >> know how the code is set up so maybe that's not a
viable
> option.
> > > >> >> >>
> > > >> >> >> Ben
> > > >> >> >>
> > > >> >> >> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway via
RT <
> > > >> >> >> met_help at ucar.edu> wrote:
> > > >> >> >>
> > > >> >> >> > Ben,
> > > >> >> >> >
> > > >> >> >> > Looking closely at the code, here's the difference...
> > > >> >> >> >
> > > >> >> >> > In met-6.0, for square neighborhoods, every time we
> increment
> > 1
> > > >> step
> > > >> >> in
> > > >> >> >> the
> > > >> >> >> > x-direction, we decrement counts from the column
we're
> > > discarding
> > > >> and
> > > >> >> >> > increment counts for the column we're adding.
> > > >> >> >> >
> > > >> >> >> > In met-7.0, for square and circular neighborhoods, we
> > reprocess
> > > >> the
> > > >> >> >> whole
> > > >> >> >> > neighborhood each time since for circles, it isn't as
simple
> > as
> > > >> >> >> > dropping/adding a new column.
> > > >> >> >> >
> > > >> >> >> > I agree that using an FFT instead would be
preferable.  The
> > > >> question
> > > >> >> is
> > > >> >> >> > whether we get the same result?  And whether or not
it
> handles
> > > >> >> missing
> > > >> >> >> data
> > > >> >> >> > well.
> > > >> >> >> >
> > > >> >> >> > John
> > > >> >> >> >
> > > >> >> >> > On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway <
> > > >> johnhg at ucar.edu
> > > >> >> >
> > > >> >> >> > wrote:
> > > >> >> >> >
> > > >> >> >> > > Ben,
> > > >> >> >> > >
> > > >> >> >> > > Thanks for sending the sample data.  Here's how
long this
> > test
> > > >> case
> > > >> >> >> took
> > > >> >> >> > > on my machine using 2 versions of MET:
> > > >> >> >> > > met-6.0 took 01:24.
> > > >> >> >> > > met-7.0 with patches took 10:49.
> > > >> >> >> > >
> > > >> >> >> > > So unfortunately (as you just emailed me), the
patch won't
> > fix
> > > >> >> this.
> > > >> >> >> > >
> > > >> >> >> > > The difference in versions is this... In met-6.0,
we only
> > > >> supported
> > > >> >> >> > > "square" neighborhoods.  In met-7.0, we added
support for
> > > >> >> "circular"
> > > >> >> >> > > neighborhoods.  In doing so, we leveraged existing
code
> from
> > > >> >> another
> > > >> >> >> part
> > > >> >> >> > > of RAL.  However, as highlighted by your
configuration, it
> > > >> >> slower...
> > > >> >> >> > > especially for large neighborhood sizes.
> > > >> >> >> > >
> > > >> >> >> > > Let me look more closely at the code to see how we
can
> > address
> > > >> this
> > > >> >> >> > issue.
> > > >> >> >> > >
> > > >> >> >> > > Thanks,
> > > >> >> >> > > John
> > > >> >> >> > >
> > > >> >> >> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake -
NOAA
> > > Affiliate
> > > >> >> via
> > > >> >> >> RT <
> > > >> >> >> > > met_help at ucar.edu> wrote:
> > > >> >> >> > >
> > > >> >> >> > >>
> > > >> >> >> > >> <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=86792
> > > >> >
> > > >> >> >> > >>
> > > >> >> >> > >> Hi Julie,
> > > >> >> >> > >>
> > > >> >> >> > >> Ok great, thanks.  I'll try running grid stat with
the
> > > updated
> > > >> >> >> version.
> > > >> >> >> > >> And yeah, I figured you meant Luna. :)
> > > >> >> >> > >>
> > > >> >> >> > >> Ben
> > > >> >> >> > >>
> > > >> >> >> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie Prestopnik
via
> RT <
> > > >> >> >> > >> met_help at ucar.edu
> > > >> >> >> > >> > wrote:
> > > >> >> >> > >>
> > > >> >> >> > >> > Pardon, I meant "luna" not "surge".  :)
> > > >> >> >> > >> >
> > > >> >> >> > >> > Julie
> > > >> >> >> > >> >
> > > >> >> >> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie
Prestopnik <
> > > >> >> >> jpresto at ucar.edu>
> > > >> >> >> > >> > wrote:
> > > >> >> >> > >> >
> > > >> >> >> > >> > > Hi Ben.
> > > >> >> >> > >> > >
> > > >> >> >> > >> > > The version of met-7.0 has now been updated on
tide:
> > > >> >> >> > >> > >
> > > >> >> >> > >> > >> MET Version: V7.0
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> Repository:
> > > >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_
> > > >> >> >> > >> > bugfix
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> Revision: 5607
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> Change Date: 2018-07-24 14:46:55 -0600 (Tue,
24 Jul
> > > 2018)
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >
> > > >> >> >> > >> > > I will update on surge once I devonprod access
is
> > > returned.
> > > >> >> >> > >> > >
> > > >> >> >> > >> > > Julie
> > > >> >> >> > >> > >
> > > >> >> >> > >> > >
> > > >> >> >> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin
Blake -
> NOAA
> > > >> >> Affiliate
> > > >> >> >> via
> > > >> >> >> > >> RT <
> > > >> >> >> > >> > > met_help at ucar.edu> wrote:
> > > >> >> >> > >> > >
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> <URL: https://rt.rap.ucar.edu/rt/Tic
> > > >> >> ket/Display.html?id=86792 >
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> John,
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> Sure no problem, I copied a HiResW ARW
forecast file
> > > and a
> > > >> >> CCPA
> > > >> >> >> > file
> > > >> >> >> > >> > from
> > > >> >> >> > >> > >> WCOSS to Theia and they are located here:
> > > >> >> >> > >> > >> /scratch4/NCEPDEV/fv3-
cam/noscrub/Benjamin.Blake/
> > > testfiles
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> I also put the CONUS.nc mask file that I am
using in
> > > that
> > > >> >> >> > directory.
> > > >> >> >> > >> > >> Using
> > > >> >> >> > >> > >> that file or CONUS.poly didn't seem to make a
> > difference
> > > >> >> though.
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> Here is the grid stat command I was using on
WCOSS
> > > (after
> > > >> >> >> removing
> > > >> >> >> > >> the
> > > >> >> >> > >> > >> unrelevant WCOSS paths):
> > > >> >> >> > >> > >> grid_stat hiresw.t00z.arw_5km.f03.conus.grib2
> > > >> >> >> > ccpa.2017052803.03h.gb2
> > > >> >> >> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> Julie - That'd be great, thanks!
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> Ben
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie
Prestopnik
> via
> > > RT
> > > >> <
> > > >> >> >> > >> > >> met_help at ucar.edu
> > > >> >> >> > >> > >> > wrote:
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >> > I'll get met-7.0 updated on tide right now.
I'll
> > > >> follow up
> > > >> >> >> once
> > > >> >> >> > it
> > > >> >> >> > >> > >> update
> > > >> >> >> > >> > >> > is complete.
> > > >> >> >> > >> > >> >
> > > >> >> >> > >> > >> > Julie
> > > >> >> >> > >> > >> >
> > > >> >> >> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John
Halley
> Gotway
> > > via
> > > >> RT
> > > >> >> <
> > > >> >> >> > >> > >> > met_help at ucar.edu> wrote:
> > > >> >> >> > >> > >> >
> > > >> >> >> > >> > >> > >
> > > >> >> >> > >> > >> > > <URL:
> > > >> >> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > > >> >> >> > >
> > > >> >> >> > >> > >> > >
> > > >> >> >> > >> > >> > > Ben,
> > > >> >> >> > >> > >> > >
> > > >> >> >> > >> > >> > > Yeah, that'd be great.  If you send me a
single
> > > input
> > > >> >> >> forecast
> > > >> >> >> > >> and
> > > >> >> >> > >> > >> > analysis
> > > >> >> >> > >> > >> > > file for comparison, I'll run those 3
versions
> of
> > > >> >> grid-stat
> > > >> >> >> > using
> > > >> >> >> > >> > your
> > > >> >> >> > >> > >> > > configuration and quantify the timing
> differences.
> > > >> >> >> > >> > >> > >
> > > >> >> >> > >> > >> > > You can either point me to them on theia
or post
> > > them
> > > >> to
> > > >> >> our
> > > >> >> >> > >> > anonymous
> > > >> >> >> > >> > >> > ftp
> > > >> >> >> > >> > >> > > site:
> > > >> >> >> > >> > >> > >    https://dtcenter.org/met/users
> > > >> >> /support/met_help.php#ftp
> > > >> >> >> > >> > >> > >
> > > >> >> >> > >> > >> > > Thanks,
> > > >> >> >> > >> > >> > > John
> > > >> >> >> > >> > >> > >
> > > >> >> >> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM Benjamin
Blake
> -
> > > NOAA
> > > >> >> >> > Affiliate
> > > >> >> >> > >> via
> > > >> >> >> > >> > >> RT <
> > > >> >> >> > >> > >> > > met_help at ucar.edu> wrote:
> > > >> >> >> > >> > >> > >
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
> > > >> >> >> > Ticket/Display.html?id=86792
> > > >> >> >> > >> >
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > > Hi John,
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > > Thanks for letting me know about the
grid stat
> > > >> memory
> > > >> >> >> > >> allocation
> > > >> >> >> > >> > >> issue.
> > > >> >> >> > >> > >> > > I
> > > >> >> >> > >> > >> > > > am currently pointing to Julie's
met/7.0
> > > >> installation
> > > >> >> on
> > > >> >> >> > WCOSS
> > > >> >> >> > >> > >> (Tide),
> > > >> >> >> > >> > >> > > and
> > > >> >> >> > >> > >> > > > here is the output of grid_stat
--version:
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > > MET Version:    V7.0
> > > >> >> >> > >> > >> > > > Repository:
> > > >> >> >> > >> > >> > > >
> > > >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> > > >> >> >> > >> > >> > > > Revision:    5483
> > > >> >> >> > >> > >> > > > Change Date:    2018-06-13 09:57:36
-0600
> (Wed,
> > 13
> > > >> Jun
> > > >> >> >> 2018)
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > > Since the grid stat fix was put in
place on
> July
> > > >> 24th
> > > >> >> the
> > > >> >> >> > >> version
> > > >> >> >> > >> > >> I'm
> > > >> >> >> > >> > >> > > using
> > > >> >> >> > >> > >> > > > likely doesn't have that fix, so I'm
not sure
> if
> > > the
> > > >> >> bug
> > > >> >> >> fix
> > > >> >> >> > >> will
> > > >> >> >> > >> > >> > resolve
> > > >> >> >> > >> > >> > > > the issue or not.  I could try pointing
to a
> > newer
> > > >> >> >> version of
> > > >> >> >> > >> MET.
> > > >> >> >> > >> > >> I
> > > >> >> >> > >> > >> > > > should also let you know that grid stat
for
> the
> > > >> other
> > > >> >> line
> > > >> >> >> > >> types
> > > >> >> >> > >> > >> does
> > > >> >> >> > >> > >> > not
> > > >> >> >> > >> > >> > > > seem to be slower, I only see a
noticeable
> > > slowdown
> > > >> >> with
> > > >> >> >> the
> > > >> >> >> > >> > NBRCNT
> > > >> >> >> > >> > >> > line
> > > >> >> >> > >> > >> > > > type.
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > > I am evaluating the deterministic
members of
> the
> > > >> HREFv2
> > > >> >> >> and
> > > >> >> >> > the
> > > >> >> >> > >> > NCAR
> > > >> >> >> > >> > >> > > > ensemble as well as their respective
means.  I
> > > >> verify
> > > >> >> >> them on
> > > >> >> >> > >> NCEP
> > > >> >> >> > >> > >> grid
> > > >> >> >> > >> > >> > > 227
> > > >> >> >> > >> > >> > > > which is a ~5-km grid (5.079 km).  I
can send
> > you
> > > >> some
> > > >> >> >> sample
> > > >> >> >> > >> > files
> > > >> >> >> > >> > >> for
> > > >> >> >> > >> > >> > > > your testing if that would be easiest.
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > > Thanks!
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > > Ben
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM, John
Halley
> > > Gotway
> > > >> >> via
> > > >> >> >> RT <
> > > >> >> >> > >> > >> > > > met_help at ucar.edu> wrote:
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > > > Hi Ben,
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > I see that you've found the met-7.0
runtime
> of
> > > >> >> >> Grid-Stat is
> > > >> >> >> > >> much
> > > >> >> >> > >> > >> > slower
> > > >> >> >> > >> > >> > > > > than the met-6.0 version.  We did
find/fix a
> > > >> >> Grid-Stat
> > > >> >> >> > memory
> > > >> >> >> > >> > >> > > allocation
> > > >> >> >> > >> > >> > > > > issue that's posted to the MET
website:
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > https://dtcenter.org/met/
> > > users/support/known_issues/
> > > >> >> >> > >> > METv7.0/index.php
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > I have two questions...
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > (1) What's the patch level of the
code
> you're
> > > >> >> currently
> > > >> >> >> > >> running?
> > > >> >> >> > >> > >> > > Please
> > > >> >> >> > >> > >> > > > > run "grid_stat --version" for the
particular
> > > >> >> executable
> > > >> >> >> > >> you're
> > > >> >> >> > >> > >> > running
> > > >> >> >> > >> > >> > > > and
> > > >> >> >> > >> > >> > > > > send me the result.  Should be
something
> like
> > > >> this:
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > MET Version: V7.0
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > Repository:
> > > >> >> >> > >> > >> > >
> > > >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_bugfix
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > Revision:         5607
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > Change Date: 2018-07-24 14:46:55
-0600 (Tue,
> > 24
> > > >> Jul
> > > >> >> >> 2018)
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > (2) Does this bugfix actually fix the
> problem
> > > >> you're
> > > >> >> >> > seeing?
> > > >> >> >> > >> > I'd
> > > >> >> >> > >> > >> > like
> > > >> >> >> > >> > >> > > to
> > > >> >> >> > >> > >> > > > > test this out myself using your
config file.
> > I
> > > >> can
> > > >> >> run
> > > >> >> >> 3
> > > >> >> >> > >> > >> versions:
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > >    - met-6.0 with latest patches
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > >    - met-7.0 with no patches
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > >    - met-7.0 with latest patches
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > What model/analysis are you
evaluating?  I
> > might
> > > >> as
> > > >> >> well
> > > >> >> >> > use
> > > >> >> >> > >> the
> > > >> >> >> > >> > >> same
> > > >> >> >> > >> > >> > > > > resolution for my testing.
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > Thanks,
> > > >> >> >> > >> > >> > > > > John
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM
Benjamin
> > Blake -
> > > >> NOAA
> > > >> >> >> > >> Affiliate
> > > >> >> >> > >> > >> via
> > > >> >> >> > >> > >> > RT
> > > >> >> >> > >> > >> > > <
> > > >> >> >> > >> > >> > > > > met_help at ucar.edu> wrote:
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > >
> > > >> >> >> > >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request
86792
> was
> > > >> acted
> > > >> >> >> upon.
> > > >> >> >> > >> > >> > > > > > Transaction: Ticket created by
> > > >> >> >> benjamin.blake at noaa.gov
> > > >> >> >> > >> > >> > > > > >        Queue: met_help
> > > >> >> >> > >> > >> > > > > >      Subject: FSS calculation with
grid
> stat
> > > >> >> >> > >> > >> > > > > >        Owner: Nobody
> > > >> >> >> > >> > >> > > > > >   Requestors:
benjamin.blake at noaa.gov
> > > >> >> >> > >> > >> > > > > >       Status: new
> > > >> >> >> > >> > >> > > > > >  Ticket <URL:
> > > >> >> >> > >> > >> > >
> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > > >
> > > >> >> >> > >> > >> > > > > >
> > > >> >> >> > >> > >> > > > > > Hi,
> > > >> >> >> > >> > >> > > > > >
> > > >> >> >> > >> > >> > > > > > I recently converted all of my MET
> > > configuration
> > > >> >> files
> > > >> >> >> > from
> > > >> >> >> > >> > >> using
> > > >> >> >> > >> > >> > > v6.0
> > > >> >> >> > >> > >> > > > to
> > > >> >> >> > >> > >> > > > > > v7.0.  One issue I'm now having is
that my
> > FSS
> > > >> >> >> > calculations
> > > >> >> >> > >> > are
> > > >> >> >> > >> > >> > > taking
> > > >> >> >> > >> > >> > > > > much
> > > >> >> >> > >> > >> > > > > > longer than they did when I was
using
> > METv6.0.
> > > >> >> >> Currently
> > > >> >> >> > >> it
> > > >> >> >> > >> > >> takes
> > > >> >> >> > >> > >> > > > ~15-20
> > > >> >> >> > >> > >> > > > > > minutes to generate one .stat file.
> > > >> >> >> > >> > >> > > > > >
> > > >> >> >> > >> > >> > > > > > I've attached the grid stat
configure file
> > > that
> > > >> I
> > > >> >> am
> > > >> >> >> now
> > > >> >> >> > >> using
> > > >> >> >> > >> > >> with
> > > >> >> >> > >> > >> > > > v7.0
> > > >> >> >> > >> > >> > > > > > (GridStatConfig_APCP03_FSS_href)
and the
> > > >> configure
> > > >> >> >> file
> > > >> >> >> > I
> > > >> >> >> > >> was
> > > >> >> >> > >> > >> > using
> > > >> >> >> > >> > >> > > > with
> > > >> >> >> > >> > >> > > > > > v6.0 (GridStatConfig_03h_APCP_NBR).
> > > >> >> >> > >> > >> > > > > >
> > > >> >> >> > >> > >> > > > > > I modified cat_thresh for APCP A03
and I
> > also
> > > >> >> reverted
> > > >> >> >> > back
> > > >> >> >> > >> > from
> > > >> >> >> > >> > >> > > SHAPE
> > > >> >> >> > >> > >> > > > =
> > > >> >> >> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the
> > neighborhood
> > > >> >> methods
> > > >> >> >> > >> section,
> > > >> >> >> > >> > >> but
> > > >> >> >> > >> > >> > > the
> > > >> >> >> > >> > >> > > > > grid
> > > >> >> >> > >> > >> > > > > > stat calculation still takes much
longer
> > than
> > > it
> > > >> >> used
> > > >> >> >> to.
> > > >> >> >> > >> I
> > > >> >> >> > >> > >> also
> > > >> >> >> > >> > >> > > tried
> > > >> >> >> > >> > >> > > > > > using the CONUS.poly file in
> shared/met/poly
> > > >> >> instead
> > > >> >> >> of
> > > >> >> >> > my
> > > >> >> >> > >> > >> CONUS.nc
> > > >> >> >> > >> > >> > > to
> > > >> >> >> > >> > >> > > > > see
> > > >> >> >> > >> > >> > > > > > if that would make a difference but
it did
> > > not.
> > > >> >> The
> > > >> >> >> v7.0
> > > >> >> >> > >> > >> configure
> > > >> >> >> > >> > >> > > > file
> > > >> >> >> > >> > >> > > > > > has many more options in it so
perhaps the
> > > >> >> slowdown is
> > > >> >> >> > >> related
> > > >> >> >> > >> > >> to
> > > >> >> >> > >> > >> > > > > something
> > > >> >> >> > >> > >> > > > > > I added in.
> > > >> >> >> > >> > >> > > > > >
> > > >> >> >> > >> > >> > > > > > Do you have any suggestions for why
it
> might
> > > be
> > > >> >> >> running
> > > >> >> >> > so
> > > >> >> >> > >> > >> slowly?
> > > >> >> >> > >> > >> > > > > >
> > > >> >> >> > >> > >> > > > > > Thanks!
> > > >> >> >> > >> > >> > > > > > Ben
> > > >> >> >> > >> > >> > > > > >
> > > >> >> >> > >> > >> > > > > >
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > > >
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > > >
> > > >> >> >> > >> > >> > >
> > > >> >> >> > >> > >> > >
> > > >> >> >> > >> > >> >
> > > >> >> >> > >> > >> >
> > > >> >> >> > >> > >>
> > > >> >> >> > >> > >>
> > > >> >> >> > >> >
> > > >> >> >> > >> >
> > > >> >> >> > >>
> > > >> >> >> > >>
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >>
> > > >> >> >>
> > > >> >>
> > > >> >>
> > > >> >
> > > >>
> > > >>
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: John Halley Gotway
Time: Fri Sep 07 16:48:48 2018

Ben,

We've heard this feedback about met-7.0 being very slow in computing
neighborhood methods from a few different users via met-help.  So we
decided to post a met-7.0 bugfix for this:
   https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php

When Julie Prestopnik is compiling the next met-8.0 beta releases,
I've
asked her to also recompile met-7.0 with this patch file.

Thanks,
John

On Fri, Aug 31, 2018 at 11:46 AM Benjamin Blake - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
>
> Hi John,
>
> The 1:50 time is certainly acceptable and is a huge improvement from
what
> is in v7.0.  Thanks again for working to fix that!
>
> Mid-September would be fine.  The stats that I want to generate will
be for
> the AMS annual meeting in January 2019.
>
> Thanks,
> Ben
>
> On Fri, Aug 31, 2018 at 1:37 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Ben,
> >
> > I tried out the updated logic for computing the fractional
coverage
> field -
> > moving like a snake instead of a typewriter.  I never actually got
it to
> > compute the correct values... very close, but not right.  And when
> > computing those wrong values, I found that it was only a few
seconds
> faster
> > on our test case: 1:46 versus 1:50.  Randy and I talked and
decided that
> > the relatively small gain in efficiency isn't worth the large
increase in
> > complexity in the code.
> >
> > So I eventually gave up.  Hopefully the 1:50 time is acceptable.
> >
> > As for release date, the goal is mid-September.  Do you have a
particular
> > deadline in September we should know about?
> >
> > Thanks,
> > John
> >
> > On Fri, Aug 31, 2018 at 5:46 AM Benjamin Blake - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > >
> > > Hi John,
> > >
> > > That sounds great!  1:50 is much better than 10:58 or 7:39 so I
think
> > your
> > > proposed solution works well.
> > >
> > > I think you or Tara may have mentioned this at one point but
when do
> you
> > > think MET v8.0 will be released in September?
> > >
> > > Thanks,
> > > Ben
> > >
> > > On Thu, Aug 30, 2018 at 7:12 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Ben,
> > > >
> > > > I made some good progress on this task today.
> > > >
> > > > I implemented the old smarter logic using the new more
flexible data
> > > > structures... making sure that the results are identical.
> > > > I tested 3 versions on your sample configuration (met-6.0,
met-7.0,
> and
> > > my
> > > > proposed changes for met-8.0) and found:
> > > >
> > > >   met-6.0 for squares takes 1:30.
> > > >   met-7.0 for squares takes 10:58.
> > > >   met-8.0 for squares takes 1:50.
> > > >
> > > >   met-7.0 for circles takes 7:39.
> > > >   met-8.0 for circles takes 1:50.
> > > >
> > > > So met-8.0 is still a bit slower than met-6.0... but it's much
> closer.
> > > The
> > > > big advantage is that this improvement also applies to the
circle
> > shape.
> > > >
> > > > I know how I could make the logic more slick to speed it up a
bit
> more,
> > > but
> > > > the trouble coding it up.  Rather than moving like a type-
writer,
> > > resetting
> > > > every line, we could move like a snake, moving back and forth
across
> > the
> > > > grid.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 30, 2018 at 10:57 AM John Halley Gotway
<johnhg at ucar.edu
> >
> > > > wrote:
> > > >
> > > > > Ben,
> > > > >
> > > > > Yes, it'll be included in the next beta version... although
we'll
> be
> > > > > releasing met-8.0 in September.  And we don't currently have
a
> > planned
> > > > beta
> > > > > release version before then.
> > > > >
> > > > > However, rather than simply reverting to the old logic, I'd
prefer
> to
> > > > > "refine" the new logic.  That way, using the circles would
be just
> as
> > > > fast
> > > > > as the squares.  So I'm going to try to figure that out and
will
> let
> > > you
> > > > > know how it goes.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 30, 2018 at 8:31 AM Benjamin Blake - NOAA
Affiliate via
> > RT
> > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > > > >>
> > > > >> Hi John,
> > > > >>
> > > > >> Do you know when the next beta version will be released?
Would it
> > be
> > > > >> possible to have those code changes in that version?  That
way I
> can
> > > > start
> > > > >> running with that version of the code sooner.
> > > > >>
> > > > >> Thanks again,
> > > > >> Ben
> > > > >>
> > > > >> On Thu, Aug 30, 2018 at 7:33 AM, Benjamin Blake - NOAA
Affiliate <
> > > > >> benjamin.blake at noaa.gov> wrote:
> > > > >>
> > > > >> > Hi John,
> > > > >> >
> > > > >> > Ok great, and yeah the next version of MET would be fine.
> > > > >> >
> > > > >> > Thanks!
> > > > >> > Ben
> > > > >> >
> > > > >> > On Wed, Aug 29, 2018 at 5:02 PM, John Halley Gotway via
RT <
> > > > >> > met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> >> Ben,
> > > > >> >>
> > > > >> >> OK, I modified the development version to call the old
logic
> for
> > > > >> squares
> > > > >> >> and new (slower) logic for circles.
> > > > >> >>
> > > > >> >> Running your test case on my machine, squares take 1:15
to run
> > and
> > > > >> circles
> > > > >> >> take 7:27 to run.
> > > > >> >>
> > > > >> >> I presume that you'd like to see this change in the next
> version
> > of
> > > > >> MET...
> > > > >> >> at least unless we're able to make the new
implementation much,
> > > much
> > > > >> >> faster.
> > > > >> >>
> > > > >> >> Is that right?
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >> John
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> On Wed, Aug 29, 2018 at 1:15 PM John Halley Gotway <
> > > johnhg at ucar.edu>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Ben,
> > > > >> >> >
> > > > >> >> > Yep, that's what I'm thinking too.  For a square
> neighborhood,
> > > call
> > > > >> the
> > > > >> >> > met-6.0 code.  For a circular neighborhood, call the
newer
> > > (slower)
> > > > >> >> code.
> > > > >> >> >
> > > > >> >> > John
> > > > >> >> >
> > > > >> >> > On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake - NOAA
> > Affiliate
> > > > via
> > > > >> RT
> > > > >> >> <
> > > > >> >> > met_help at ucar.edu> wrote:
> > > > >> >> >
> > > > >> >> >>
> > > > >> >> >> <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > >
> > > > >> >> >>
> > > > >> >> >> Hi John,
> > > > >> >> >>
> > > > >> >> >> Ahh ok, yeah I'm not sure if you would get the same
result
> and
> > > how
> > > > >> >> well it
> > > > >> >> >> handles missing data.  I'd think the result may be
slightly
> > > > >> different.
> > > > >> >> >>
> > > > >> >> >> Would it be possible to have different code for the
square
> and
> > > > >> circular
> > > > >> >> >> neighborhood options?  That way if a user chose
square then
> it
> > > > >> would do
> > > > >> >> >> the
> > > > >> >> >> calculations with the method from v6.0 which is more
> > > efficient.  I
> > > > >> >> don't
> > > > >> >> >> know how the code is set up so maybe that's not a
viable
> > option.
> > > > >> >> >>
> > > > >> >> >> Ben
> > > > >> >> >>
> > > > >> >> >> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway
via RT <
> > > > >> >> >> met_help at ucar.edu> wrote:
> > > > >> >> >>
> > > > >> >> >> > Ben,
> > > > >> >> >> >
> > > > >> >> >> > Looking closely at the code, here's the
difference...
> > > > >> >> >> >
> > > > >> >> >> > In met-6.0, for square neighborhoods, every time we
> > increment
> > > 1
> > > > >> step
> > > > >> >> in
> > > > >> >> >> the
> > > > >> >> >> > x-direction, we decrement counts from the column
we're
> > > > discarding
> > > > >> and
> > > > >> >> >> > increment counts for the column we're adding.
> > > > >> >> >> >
> > > > >> >> >> > In met-7.0, for square and circular neighborhoods,
we
> > > reprocess
> > > > >> the
> > > > >> >> >> whole
> > > > >> >> >> > neighborhood each time since for circles, it isn't
as
> simple
> > > as
> > > > >> >> >> > dropping/adding a new column.
> > > > >> >> >> >
> > > > >> >> >> > I agree that using an FFT instead would be
preferable.
> The
> > > > >> question
> > > > >> >> is
> > > > >> >> >> > whether we get the same result?  And whether or not
it
> > handles
> > > > >> >> missing
> > > > >> >> >> data
> > > > >> >> >> > well.
> > > > >> >> >> >
> > > > >> >> >> > John
> > > > >> >> >> >
> > > > >> >> >> > On Wed, Aug 29, 2018 at 11:20 AM John Halley Gotway
<
> > > > >> johnhg at ucar.edu
> > > > >> >> >
> > > > >> >> >> > wrote:
> > > > >> >> >> >
> > > > >> >> >> > > Ben,
> > > > >> >> >> > >
> > > > >> >> >> > > Thanks for sending the sample data.  Here's how
long
> this
> > > test
> > > > >> case
> > > > >> >> >> took
> > > > >> >> >> > > on my machine using 2 versions of MET:
> > > > >> >> >> > > met-6.0 took 01:24.
> > > > >> >> >> > > met-7.0 with patches took 10:49.
> > > > >> >> >> > >
> > > > >> >> >> > > So unfortunately (as you just emailed me), the
patch
> won't
> > > fix
> > > > >> >> this.
> > > > >> >> >> > >
> > > > >> >> >> > > The difference in versions is this... In met-6.0,
we
> only
> > > > >> supported
> > > > >> >> >> > > "square" neighborhoods.  In met-7.0, we added
support
> for
> > > > >> >> "circular"
> > > > >> >> >> > > neighborhoods.  In doing so, we leveraged
existing code
> > from
> > > > >> >> another
> > > > >> >> >> part
> > > > >> >> >> > > of RAL.  However, as highlighted by your
configuration,
> it
> > > > >> >> slower...
> > > > >> >> >> > > especially for large neighborhood sizes.
> > > > >> >> >> > >
> > > > >> >> >> > > Let me look more closely at the code to see how
we can
> > > address
> > > > >> this
> > > > >> >> >> > issue.
> > > > >> >> >> > >
> > > > >> >> >> > > Thanks,
> > > > >> >> >> > > John
> > > > >> >> >> > >
> > > > >> >> >> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake -
NOAA
> > > > Affiliate
> > > > >> >> via
> > > > >> >> >> RT <
> > > > >> >> >> > > met_help at ucar.edu> wrote:
> > > > >> >> >> > >
> > > > >> >> >> > >>
> > > > >> >> >> > >> <URL: https://rt.rap.ucar.edu/rt/
> > > > Ticket/Display.html?id=86792
> > > > >> >
> > > > >> >> >> > >>
> > > > >> >> >> > >> Hi Julie,
> > > > >> >> >> > >>
> > > > >> >> >> > >> Ok great, thanks.  I'll try running grid stat
with the
> > > > updated
> > > > >> >> >> version.
> > > > >> >> >> > >> And yeah, I figured you meant Luna. :)
> > > > >> >> >> > >>
> > > > >> >> >> > >> Ben
> > > > >> >> >> > >>
> > > > >> >> >> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie
Prestopnik via
> > RT <
> > > > >> >> >> > >> met_help at ucar.edu
> > > > >> >> >> > >> > wrote:
> > > > >> >> >> > >>
> > > > >> >> >> > >> > Pardon, I meant "luna" not "surge".  :)
> > > > >> >> >> > >> >
> > > > >> >> >> > >> > Julie
> > > > >> >> >> > >> >
> > > > >> >> >> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie
Prestopnik <
> > > > >> >> >> jpresto at ucar.edu>
> > > > >> >> >> > >> > wrote:
> > > > >> >> >> > >> >
> > > > >> >> >> > >> > > Hi Ben.
> > > > >> >> >> > >> > >
> > > > >> >> >> > >> > > The version of met-7.0 has now been updated
on
> tide:
> > > > >> >> >> > >> > >
> > > > >> >> >> > >> > >> MET Version: V7.0
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> Repository:
> > > > >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_
> > > > >> >> >> > >> > bugfix
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> Revision: 5607
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> Change Date: 2018-07-24 14:46:55 -0600
(Tue, 24
> Jul
> > > > 2018)
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >
> > > > >> >> >> > >> > > I will update on surge once I devonprod
access is
> > > > returned.
> > > > >> >> >> > >> > >
> > > > >> >> >> > >> > > Julie
> > > > >> >> >> > >> > >
> > > > >> >> >> > >> > >
> > > > >> >> >> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin
Blake -
> > NOAA
> > > > >> >> Affiliate
> > > > >> >> >> via
> > > > >> >> >> > >> RT <
> > > > >> >> >> > >> > > met_help at ucar.edu> wrote:
> > > > >> >> >> > >> > >
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> <URL: https://rt.rap.ucar.edu/rt/Tic
> > > > >> >> ket/Display.html?id=86792 >
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> John,
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> Sure no problem, I copied a HiResW ARW
forecast
> file
> > > > and a
> > > > >> >> CCPA
> > > > >> >> >> > file
> > > > >> >> >> > >> > from
> > > > >> >> >> > >> > >> WCOSS to Theia and they are located here:
> > > > >> >> >> > >> > >> /scratch4/NCEPDEV/fv3-
cam/noscrub/Benjamin.Blake/
> > > > testfiles
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> I also put the CONUS.nc mask file that I am
using
> in
> > > > that
> > > > >> >> >> > directory.
> > > > >> >> >> > >> > >> Using
> > > > >> >> >> > >> > >> that file or CONUS.poly didn't seem to make
a
> > > difference
> > > > >> >> though.
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> Here is the grid stat command I was using
on WCOSS
> > > > (after
> > > > >> >> >> removing
> > > > >> >> >> > >> the
> > > > >> >> >> > >> > >> unrelevant WCOSS paths):
> > > > >> >> >> > >> > >> grid_stat
hiresw.t00z.arw_5km.f03.conus.grib2
> > > > >> >> >> > ccpa.2017052803.03h.gb2
> > > > >> >> >> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> Julie - That'd be great, thanks!
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> Ben
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie
Prestopnik
> > via
> > > > RT
> > > > >> <
> > > > >> >> >> > >> > >> met_help at ucar.edu
> > > > >> >> >> > >> > >> > wrote:
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >> > I'll get met-7.0 updated on tide right
now.
> I'll
> > > > >> follow up
> > > > >> >> >> once
> > > > >> >> >> > it
> > > > >> >> >> > >> > >> update
> > > > >> >> >> > >> > >> > is complete.
> > > > >> >> >> > >> > >> >
> > > > >> >> >> > >> > >> > Julie
> > > > >> >> >> > >> > >> >
> > > > >> >> >> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John
Halley
> > Gotway
> > > > via
> > > > >> RT
> > > > >> >> <
> > > > >> >> >> > >> > >> > met_help at ucar.edu> wrote:
> > > > >> >> >> > >> > >> >
> > > > >> >> >> > >> > >> > >
> > > > >> >> >> > >> > >> > > <URL:
> > > > >> >> >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > > > >> >> >> > >
> > > > >> >> >> > >> > >> > >
> > > > >> >> >> > >> > >> > > Ben,
> > > > >> >> >> > >> > >> > >
> > > > >> >> >> > >> > >> > > Yeah, that'd be great.  If you send me
a
> single
> > > > input
> > > > >> >> >> forecast
> > > > >> >> >> > >> and
> > > > >> >> >> > >> > >> > analysis
> > > > >> >> >> > >> > >> > > file for comparison, I'll run those 3
versions
> > of
> > > > >> >> grid-stat
> > > > >> >> >> > using
> > > > >> >> >> > >> > your
> > > > >> >> >> > >> > >> > > configuration and quantify the timing
> > differences.
> > > > >> >> >> > >> > >> > >
> > > > >> >> >> > >> > >> > > You can either point me to them on
theia or
> post
> > > > them
> > > > >> to
> > > > >> >> our
> > > > >> >> >> > >> > anonymous
> > > > >> >> >> > >> > >> > ftp
> > > > >> >> >> > >> > >> > > site:
> > > > >> >> >> > >> > >> > >    https://dtcenter.org/met/users
> > > > >> >> /support/met_help.php#ftp
> > > > >> >> >> > >> > >> > >
> > > > >> >> >> > >> > >> > > Thanks,
> > > > >> >> >> > >> > >> > > John
> > > > >> >> >> > >> > >> > >
> > > > >> >> >> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM
Benjamin
> Blake
> > -
> > > > NOAA
> > > > >> >> >> > Affiliate
> > > > >> >> >> > >> via
> > > > >> >> >> > >> > >> RT <
> > > > >> >> >> > >> > >> > > met_help at ucar.edu> wrote:
> > > > >> >> >> > >> > >> > >
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > >> >> >> > Ticket/Display.html?id=86792
> > > > >> >> >> > >> >
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > > Hi John,
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > > Thanks for letting me know about the
grid
> stat
> > > > >> memory
> > > > >> >> >> > >> allocation
> > > > >> >> >> > >> > >> issue.
> > > > >> >> >> > >> > >> > > I
> > > > >> >> >> > >> > >> > > > am currently pointing to Julie's
met/7.0
> > > > >> installation
> > > > >> >> on
> > > > >> >> >> > WCOSS
> > > > >> >> >> > >> > >> (Tide),
> > > > >> >> >> > >> > >> > > and
> > > > >> >> >> > >> > >> > > > here is the output of grid_stat
--version:
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > > MET Version:    V7.0
> > > > >> >> >> > >> > >> > > > Repository:
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >>
> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > > > >> >> >> > >> > >> > > > Revision:    5483
> > > > >> >> >> > >> > >> > > > Change Date:    2018-06-13 09:57:36
-0600
> > (Wed,
> > > 13
> > > > >> Jun
> > > > >> >> >> 2018)
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > > Since the grid stat fix was put in
place on
> > July
> > > > >> 24th
> > > > >> >> the
> > > > >> >> >> > >> version
> > > > >> >> >> > >> > >> I'm
> > > > >> >> >> > >> > >> > > using
> > > > >> >> >> > >> > >> > > > likely doesn't have that fix, so I'm
not
> sure
> > if
> > > > the
> > > > >> >> bug
> > > > >> >> >> fix
> > > > >> >> >> > >> will
> > > > >> >> >> > >> > >> > resolve
> > > > >> >> >> > >> > >> > > > the issue or not.  I could try
pointing to a
> > > newer
> > > > >> >> >> version of
> > > > >> >> >> > >> MET.
> > > > >> >> >> > >> > >> I
> > > > >> >> >> > >> > >> > > > should also let you know that grid
stat for
> > the
> > > > >> other
> > > > >> >> line
> > > > >> >> >> > >> types
> > > > >> >> >> > >> > >> does
> > > > >> >> >> > >> > >> > not
> > > > >> >> >> > >> > >> > > > seem to be slower, I only see a
noticeable
> > > > slowdown
> > > > >> >> with
> > > > >> >> >> the
> > > > >> >> >> > >> > NBRCNT
> > > > >> >> >> > >> > >> > line
> > > > >> >> >> > >> > >> > > > type.
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > > I am evaluating the deterministic
members of
> > the
> > > > >> HREFv2
> > > > >> >> >> and
> > > > >> >> >> > the
> > > > >> >> >> > >> > NCAR
> > > > >> >> >> > >> > >> > > > ensemble as well as their respective
> means.  I
> > > > >> verify
> > > > >> >> >> them on
> > > > >> >> >> > >> NCEP
> > > > >> >> >> > >> > >> grid
> > > > >> >> >> > >> > >> > > 227
> > > > >> >> >> > >> > >> > > > which is a ~5-km grid (5.079 km).  I
can
> send
> > > you
> > > > >> some
> > > > >> >> >> sample
> > > > >> >> >> > >> > files
> > > > >> >> >> > >> > >> for
> > > > >> >> >> > >> > >> > > > your testing if that would be
easiest.
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > > Thanks!
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > > Ben
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM,
John
> Halley
> > > > Gotway
> > > > >> >> via
> > > > >> >> >> RT <
> > > > >> >> >> > >> > >> > > > met_help at ucar.edu> wrote:
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > > > Hi Ben,
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > I see that you've found the met-7.0
> runtime
> > of
> > > > >> >> >> Grid-Stat is
> > > > >> >> >> > >> much
> > > > >> >> >> > >> > >> > slower
> > > > >> >> >> > >> > >> > > > > than the met-6.0 version.  We did
> find/fix a
> > > > >> >> Grid-Stat
> > > > >> >> >> > memory
> > > > >> >> >> > >> > >> > > allocation
> > > > >> >> >> > >> > >> > > > > issue that's posted to the MET
website:
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > https://dtcenter.org/met/
> > > > users/support/known_issues/
> > > > >> >> >> > >> > METv7.0/index.php
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > I have two questions...
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > (1) What's the patch level of the
code
> > you're
> > > > >> >> currently
> > > > >> >> >> > >> running?
> > > > >> >> >> > >> > >> > > Please
> > > > >> >> >> > >> > >> > > > > run "grid_stat --version" for the
> particular
> > > > >> >> executable
> > > > >> >> >> > >> you're
> > > > >> >> >> > >> > >> > running
> > > > >> >> >> > >> > >> > > > and
> > > > >> >> >> > >> > >> > > > > send me the result.  Should be
something
> > like
> > > > >> this:
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > MET Version: V7.0
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > Repository:
> > > > >> >> >> > >> > >> > >
> > > > >> >> >>
> https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > Revision:         5607
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > Change Date: 2018-07-24 14:46:55
-0600
> (Tue,
> > > 24
> > > > >> Jul
> > > > >> >> >> 2018)
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > (2) Does this bugfix actually fix
the
> > problem
> > > > >> you're
> > > > >> >> >> > seeing?
> > > > >> >> >> > >> > I'd
> > > > >> >> >> > >> > >> > like
> > > > >> >> >> > >> > >> > > to
> > > > >> >> >> > >> > >> > > > > test this out myself using your
config
> file.
> > > I
> > > > >> can
> > > > >> >> run
> > > > >> >> >> 3
> > > > >> >> >> > >> > >> versions:
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > >    - met-6.0 with latest patches
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > >    - met-7.0 with no patches
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > >    - met-7.0 with latest patches
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > What model/analysis are you
evaluating?  I
> > > might
> > > > >> as
> > > > >> >> well
> > > > >> >> >> > use
> > > > >> >> >> > >> the
> > > > >> >> >> > >> > >> same
> > > > >> >> >> > >> > >> > > > > resolution for my testing.
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > Thanks,
> > > > >> >> >> > >> > >> > > > > John
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM
Benjamin
> > > Blake -
> > > > >> NOAA
> > > > >> >> >> > >> Affiliate
> > > > >> >> >> > >> > >> via
> > > > >> >> >> > >> > >> > RT
> > > > >> >> >> > >> > >> > > <
> > > > >> >> >> > >> > >> > > > > met_help at ucar.edu> wrote:
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > >
> > > > >> >> >> > >> > >> > > > > > Wed Aug 29 09:01:08 2018: Request
86792
> > was
> > > > >> acted
> > > > >> >> >> upon.
> > > > >> >> >> > >> > >> > > > > > Transaction: Ticket created by
> > > > >> >> >> benjamin.blake at noaa.gov
> > > > >> >> >> > >> > >> > > > > >        Queue: met_help
> > > > >> >> >> > >> > >> > > > > >      Subject: FSS calculation
with grid
> > stat
> > > > >> >> >> > >> > >> > > > > >        Owner: Nobody
> > > > >> >> >> > >> > >> > > > > >   Requestors:
benjamin.blake at noaa.gov
> > > > >> >> >> > >> > >> > > > > >       Status: new
> > > > >> >> >> > >> > >> > > > > >  Ticket <URL:
> > > > >> >> >> > >> > >> > >
> > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > > >
> > > > >> >> >> > >> > >> > > > > >
> > > > >> >> >> > >> > >> > > > > > Hi,
> > > > >> >> >> > >> > >> > > > > >
> > > > >> >> >> > >> > >> > > > > > I recently converted all of my
MET
> > > > configuration
> > > > >> >> files
> > > > >> >> >> > from
> > > > >> >> >> > >> > >> using
> > > > >> >> >> > >> > >> > > v6.0
> > > > >> >> >> > >> > >> > > > to
> > > > >> >> >> > >> > >> > > > > > v7.0.  One issue I'm now having
is that
> my
> > > FSS
> > > > >> >> >> > calculations
> > > > >> >> >> > >> > are
> > > > >> >> >> > >> > >> > > taking
> > > > >> >> >> > >> > >> > > > > much
> > > > >> >> >> > >> > >> > > > > > longer than they did when I was
using
> > > METv6.0.
> > > > >> >> >> Currently
> > > > >> >> >> > >> it
> > > > >> >> >> > >> > >> takes
> > > > >> >> >> > >> > >> > > > ~15-20
> > > > >> >> >> > >> > >> > > > > > minutes to generate one .stat
file.
> > > > >> >> >> > >> > >> > > > > >
> > > > >> >> >> > >> > >> > > > > > I've attached the grid stat
configure
> file
> > > > that
> > > > >> I
> > > > >> >> am
> > > > >> >> >> now
> > > > >> >> >> > >> using
> > > > >> >> >> > >> > >> with
> > > > >> >> >> > >> > >> > > > v7.0
> > > > >> >> >> > >> > >> > > > > > (GridStatConfig_APCP03_FSS_href)
and the
> > > > >> configure
> > > > >> >> >> file
> > > > >> >> >> > I
> > > > >> >> >> > >> was
> > > > >> >> >> > >> > >> > using
> > > > >> >> >> > >> > >> > > > with
> > > > >> >> >> > >> > >> > > > > > v6.0
(GridStatConfig_03h_APCP_NBR).
> > > > >> >> >> > >> > >> > > > > >
> > > > >> >> >> > >> > >> > > > > > I modified cat_thresh for APCP
A03 and I
> > > also
> > > > >> >> reverted
> > > > >> >> >> > back
> > > > >> >> >> > >> > from
> > > > >> >> >> > >> > >> > > SHAPE
> > > > >> >> >> > >> > >> > > > =
> > > > >> >> >> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for the
> > > neighborhood
> > > > >> >> methods
> > > > >> >> >> > >> section,
> > > > >> >> >> > >> > >> but
> > > > >> >> >> > >> > >> > > the
> > > > >> >> >> > >> > >> > > > > grid
> > > > >> >> >> > >> > >> > > > > > stat calculation still takes much
longer
> > > than
> > > > it
> > > > >> >> used
> > > > >> >> >> to.
> > > > >> >> >> > >> I
> > > > >> >> >> > >> > >> also
> > > > >> >> >> > >> > >> > > tried
> > > > >> >> >> > >> > >> > > > > > using the CONUS.poly file in
> > shared/met/poly
> > > > >> >> instead
> > > > >> >> >> of
> > > > >> >> >> > my
> > > > >> >> >> > >> > >> CONUS.nc
> > > > >> >> >> > >> > >> > > to
> > > > >> >> >> > >> > >> > > > > see
> > > > >> >> >> > >> > >> > > > > > if that would make a difference
but it
> did
> > > > not.
> > > > >> >> The
> > > > >> >> >> v7.0
> > > > >> >> >> > >> > >> configure
> > > > >> >> >> > >> > >> > > > file
> > > > >> >> >> > >> > >> > > > > > has many more options in it so
perhaps
> the
> > > > >> >> slowdown is
> > > > >> >> >> > >> related
> > > > >> >> >> > >> > >> to
> > > > >> >> >> > >> > >> > > > > something
> > > > >> >> >> > >> > >> > > > > > I added in.
> > > > >> >> >> > >> > >> > > > > >
> > > > >> >> >> > >> > >> > > > > > Do you have any suggestions for
why it
> > might
> > > > be
> > > > >> >> >> running
> > > > >> >> >> > so
> > > > >> >> >> > >> > >> slowly?
> > > > >> >> >> > >> > >> > > > > >
> > > > >> >> >> > >> > >> > > > > > Thanks!
> > > > >> >> >> > >> > >> > > > > > Ben
> > > > >> >> >> > >> > >> > > > > >
> > > > >> >> >> > >> > >> > > > > >
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > > >
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > > >
> > > > >> >> >> > >> > >> > >
> > > > >> >> >> > >> > >> > >
> > > > >> >> >> > >> > >> >
> > > > >> >> >> > >> > >> >
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> > >>
> > > > >> >> >> > >> >
> > > > >> >> >> > >> >
> > > > >> >> >> > >>
> > > > >> >> >> > >>
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > >
> >
> >
>
>

------------------------------------------------
Subject: FSS calculation with grid stat
From: Benjamin Blake - NOAA Affiliate
Time: Mon Sep 10 05:49:29 2018

John,

That sounds great, thanks!

Ben

On Fri, Sep 7, 2018 at 6:48 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:

> Ben,
>
> We've heard this feedback about met-7.0 being very slow in computing
> neighborhood methods from a few different users via met-help.  So we
> decided to post a met-7.0 bugfix for this:
>
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
>
> When Julie Prestopnik is compiling the next met-8.0 beta releases,
I've
> asked her to also recompile met-7.0 with this patch file.
>
> Thanks,
> John
>
> On Fri, Aug 31, 2018 at 11:46 AM Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> >
> > Hi John,
> >
> > The 1:50 time is certainly acceptable and is a huge improvement
from what
> > is in v7.0.  Thanks again for working to fix that!
> >
> > Mid-September would be fine.  The stats that I want to generate
will be
> for
> > the AMS annual meeting in January 2019.
> >
> > Thanks,
> > Ben
> >
> > On Fri, Aug 31, 2018 at 1:37 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Ben,
> > >
> > > I tried out the updated logic for computing the fractional
coverage
> > field -
> > > moving like a snake instead of a typewriter.  I never actually
got it
> to
> > > compute the correct values... very close, but not right.  And
when
> > > computing those wrong values, I found that it was only a few
seconds
> > faster
> > > on our test case: 1:46 versus 1:50.  Randy and I talked and
decided
> that
> > > the relatively small gain in efficiency isn't worth the large
increase
> in
> > > complexity in the code.
> > >
> > > So I eventually gave up.  Hopefully the 1:50 time is acceptable.
> > >
> > > As for release date, the goal is mid-September.  Do you have a
> particular
> > > deadline in September we should know about?
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Aug 31, 2018 at 5:46 AM Benjamin Blake - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
>
> > > >
> > > > Hi John,
> > > >
> > > > That sounds great!  1:50 is much better than 10:58 or 7:39 so
I think
> > > your
> > > > proposed solution works well.
> > > >
> > > > I think you or Tara may have mentioned this at one point but
when do
> > you
> > > > think MET v8.0 will be released in September?
> > > >
> > > > Thanks,
> > > > Ben
> > > >
> > > > On Thu, Aug 30, 2018 at 7:12 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Ben,
> > > > >
> > > > > I made some good progress on this task today.
> > > > >
> > > > > I implemented the old smarter logic using the new more
flexible
> data
> > > > > structures... making sure that the results are identical.
> > > > > I tested 3 versions on your sample configuration (met-6.0,
met-7.0,
> > and
> > > > my
> > > > > proposed changes for met-8.0) and found:
> > > > >
> > > > >   met-6.0 for squares takes 1:30.
> > > > >   met-7.0 for squares takes 10:58.
> > > > >   met-8.0 for squares takes 1:50.
> > > > >
> > > > >   met-7.0 for circles takes 7:39.
> > > > >   met-8.0 for circles takes 1:50.
> > > > >
> > > > > So met-8.0 is still a bit slower than met-6.0... but it's
much
> > closer.
> > > > The
> > > > > big advantage is that this improvement also applies to the
circle
> > > shape.
> > > > >
> > > > > I know how I could make the logic more slick to speed it up
a bit
> > more,
> > > > but
> > > > > the trouble coding it up.  Rather than moving like a type-
writer,
> > > > resetting
> > > > > every line, we could move like a snake, moving back and
forth
> across
> > > the
> > > > > grid.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 30, 2018 at 10:57 AM John Halley Gotway <
> johnhg at ucar.edu
> > >
> > > > > wrote:
> > > > >
> > > > > > Ben,
> > > > > >
> > > > > > Yes, it'll be included in the next beta version...
although we'll
> > be
> > > > > > releasing met-8.0 in September.  And we don't currently
have a
> > > planned
> > > > > beta
> > > > > > release version before then.
> > > > > >
> > > > > > However, rather than simply reverting to the old logic,
I'd
> prefer
> > to
> > > > > > "refine" the new logic.  That way, using the circles would
be
> just
> > as
> > > > > fast
> > > > > > as the squares.  So I'm going to try to figure that out
and will
> > let
> > > > you
> > > > > > know how it goes.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 30, 2018 at 8:31 AM Benjamin Blake - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >>
> > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792 >
> > > > > >>
> > > > > >> Hi John,
> > > > > >>
> > > > > >> Do you know when the next beta version will be released?
Would
> it
> > > be
> > > > > >> possible to have those code changes in that version?
That way I
> > can
> > > > > start
> > > > > >> running with that version of the code sooner.
> > > > > >>
> > > > > >> Thanks again,
> > > > > >> Ben
> > > > > >>
> > > > > >> On Thu, Aug 30, 2018 at 7:33 AM, Benjamin Blake - NOAA
> Affiliate <
> > > > > >> benjamin.blake at noaa.gov> wrote:
> > > > > >>
> > > > > >> > Hi John,
> > > > > >> >
> > > > > >> > Ok great, and yeah the next version of MET would be
fine.
> > > > > >> >
> > > > > >> > Thanks!
> > > > > >> > Ben
> > > > > >> >
> > > > > >> > On Wed, Aug 29, 2018 at 5:02 PM, John Halley Gotway via
RT <
> > > > > >> > met_help at ucar.edu> wrote:
> > > > > >> >
> > > > > >> >> Ben,
> > > > > >> >>
> > > > > >> >> OK, I modified the development version to call the old
logic
> > for
> > > > > >> squares
> > > > > >> >> and new (slower) logic for circles.
> > > > > >> >>
> > > > > >> >> Running your test case on my machine, squares take
1:15 to
> run
> > > and
> > > > > >> circles
> > > > > >> >> take 7:27 to run.
> > > > > >> >>
> > > > > >> >> I presume that you'd like to see this change in the
next
> > version
> > > of
> > > > > >> MET...
> > > > > >> >> at least unless we're able to make the new
implementation
> much,
> > > > much
> > > > > >> >> faster.
> > > > > >> >>
> > > > > >> >> Is that right?
> > > > > >> >>
> > > > > >> >> Thanks,
> > > > > >> >> John
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> On Wed, Aug 29, 2018 at 1:15 PM John Halley Gotway <
> > > > johnhg at ucar.edu>
> > > > > >> >> wrote:
> > > > > >> >>
> > > > > >> >> > Ben,
> > > > > >> >> >
> > > > > >> >> > Yep, that's what I'm thinking too.  For a square
> > neighborhood,
> > > > call
> > > > > >> the
> > > > > >> >> > met-6.0 code.  For a circular neighborhood, call the
newer
> > > > (slower)
> > > > > >> >> code.
> > > > > >> >> >
> > > > > >> >> > John
> > > > > >> >> >
> > > > > >> >> > On Wed, Aug 29, 2018 at 11:56 AM Benjamin Blake -
NOAA
> > > Affiliate
> > > > > via
> > > > > >> RT
> > > > > >> >> <
> > > > > >> >> > met_help at ucar.edu> wrote:
> > > > > >> >> >
> > > > > >> >> >>
> > > > > >> >> >> <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > > >
> > > > > >> >> >>
> > > > > >> >> >> Hi John,
> > > > > >> >> >>
> > > > > >> >> >> Ahh ok, yeah I'm not sure if you would get the same
result
> > and
> > > > how
> > > > > >> >> well it
> > > > > >> >> >> handles missing data.  I'd think the result may be
> slightly
> > > > > >> different.
> > > > > >> >> >>
> > > > > >> >> >> Would it be possible to have different code for the
square
> > and
> > > > > >> circular
> > > > > >> >> >> neighborhood options?  That way if a user chose
square
> then
> > it
> > > > > >> would do
> > > > > >> >> >> the
> > > > > >> >> >> calculations with the method from v6.0 which is
more
> > > > efficient.  I
> > > > > >> >> don't
> > > > > >> >> >> know how the code is set up so maybe that's not a
viable
> > > option.
> > > > > >> >> >>
> > > > > >> >> >> Ben
> > > > > >> >> >>
> > > > > >> >> >> On Wed, Aug 29, 2018 at 1:50 PM, John Halley Gotway
via
> RT <
> > > > > >> >> >> met_help at ucar.edu> wrote:
> > > > > >> >> >>
> > > > > >> >> >> > Ben,
> > > > > >> >> >> >
> > > > > >> >> >> > Looking closely at the code, here's the
difference...
> > > > > >> >> >> >
> > > > > >> >> >> > In met-6.0, for square neighborhoods, every time
we
> > > increment
> > > > 1
> > > > > >> step
> > > > > >> >> in
> > > > > >> >> >> the
> > > > > >> >> >> > x-direction, we decrement counts from the column
we're
> > > > > discarding
> > > > > >> and
> > > > > >> >> >> > increment counts for the column we're adding.
> > > > > >> >> >> >
> > > > > >> >> >> > In met-7.0, for square and circular
neighborhoods, we
> > > > reprocess
> > > > > >> the
> > > > > >> >> >> whole
> > > > > >> >> >> > neighborhood each time since for circles, it
isn't as
> > simple
> > > > as
> > > > > >> >> >> > dropping/adding a new column.
> > > > > >> >> >> >
> > > > > >> >> >> > I agree that using an FFT instead would be
preferable.
> > The
> > > > > >> question
> > > > > >> >> is
> > > > > >> >> >> > whether we get the same result?  And whether or
not it
> > > handles
> > > > > >> >> missing
> > > > > >> >> >> data
> > > > > >> >> >> > well.
> > > > > >> >> >> >
> > > > > >> >> >> > John
> > > > > >> >> >> >
> > > > > >> >> >> > On Wed, Aug 29, 2018 at 11:20 AM John Halley
Gotway <
> > > > > >> johnhg at ucar.edu
> > > > > >> >> >
> > > > > >> >> >> > wrote:
> > > > > >> >> >> >
> > > > > >> >> >> > > Ben,
> > > > > >> >> >> > >
> > > > > >> >> >> > > Thanks for sending the sample data.  Here's how
long
> > this
> > > > test
> > > > > >> case
> > > > > >> >> >> took
> > > > > >> >> >> > > on my machine using 2 versions of MET:
> > > > > >> >> >> > > met-6.0 took 01:24.
> > > > > >> >> >> > > met-7.0 with patches took 10:49.
> > > > > >> >> >> > >
> > > > > >> >> >> > > So unfortunately (as you just emailed me), the
patch
> > won't
> > > > fix
> > > > > >> >> this.
> > > > > >> >> >> > >
> > > > > >> >> >> > > The difference in versions is this... In met-
6.0, we
> > only
> > > > > >> supported
> > > > > >> >> >> > > "square" neighborhoods.  In met-7.0, we added
support
> > for
> > > > > >> >> "circular"
> > > > > >> >> >> > > neighborhoods.  In doing so, we leveraged
existing
> code
> > > from
> > > > > >> >> another
> > > > > >> >> >> part
> > > > > >> >> >> > > of RAL.  However, as highlighted by your
> configuration,
> > it
> > > > > >> >> slower...
> > > > > >> >> >> > > especially for large neighborhood sizes.
> > > > > >> >> >> > >
> > > > > >> >> >> > > Let me look more closely at the code to see how
we can
> > > > address
> > > > > >> this
> > > > > >> >> >> > issue.
> > > > > >> >> >> > >
> > > > > >> >> >> > > Thanks,
> > > > > >> >> >> > > John
> > > > > >> >> >> > >
> > > > > >> >> >> > > On Wed, Aug 29, 2018 at 10:34 AM Benjamin Blake
- NOAA
> > > > > Affiliate
> > > > > >> >> via
> > > > > >> >> >> RT <
> > > > > >> >> >> > > met_help at ucar.edu> wrote:
> > > > > >> >> >> > >
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> <URL: https://rt.rap.ucar.edu/rt/
> > > > > Ticket/Display.html?id=86792
> > > > > >> >
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> Hi Julie,
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> Ok great, thanks.  I'll try running grid stat
with
> the
> > > > > updated
> > > > > >> >> >> version.
> > > > > >> >> >> > >> And yeah, I figured you meant Luna. :)
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> Ben
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> On Wed, Aug 29, 2018 at 12:31 PM, Julie
Prestopnik
> via
> > > RT <
> > > > > >> >> >> > >> met_help at ucar.edu
> > > > > >> >> >> > >> > wrote:
> > > > > >> >> >> > >>
> > > > > >> >> >> > >> > Pardon, I meant "luna" not "surge".  :)
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >> > Julie
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >> > On Wed, Aug 29, 2018 at 10:29 AM Julie
Prestopnik <
> > > > > >> >> >> jpresto at ucar.edu>
> > > > > >> >> >> > >> > wrote:
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >> > > Hi Ben.
> > > > > >> >> >> > >> > >
> > > > > >> >> >> > >> > > The version of met-7.0 has now been
updated on
> > tide:
> > > > > >> >> >> > >> > >
> > > > > >> >> >> > >> > >> MET Version: V7.0
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> Repository:
> > > > > >> >> >> https://svn-met-dev.cgd.ucar.edu/branches/met/met-
7.0_
> > > > > >> >> >> > >> > bugfix
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> Revision: 5607
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> Change Date: 2018-07-24 14:46:55 -0600
(Tue, 24
> > Jul
> > > > > 2018)
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >
> > > > > >> >> >> > >> > > I will update on surge once I devonprod
access is
> > > > > returned.
> > > > > >> >> >> > >> > >
> > > > > >> >> >> > >> > > Julie
> > > > > >> >> >> > >> > >
> > > > > >> >> >> > >> > >
> > > > > >> >> >> > >> > > On Wed, Aug 29, 2018 at 10:22 AM Benjamin
Blake -
> > > NOAA
> > > > > >> >> Affiliate
> > > > > >> >> >> via
> > > > > >> >> >> > >> RT <
> > > > > >> >> >> > >> > > met_help at ucar.edu> wrote:
> > > > > >> >> >> > >> > >
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> <URL: https://rt.rap.ucar.edu/rt/Tic
> > > > > >> >> ket/Display.html?id=86792 >
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> John,
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> Sure no problem, I copied a HiResW ARW
forecast
> > file
> > > > > and a
> > > > > >> >> CCPA
> > > > > >> >> >> > file
> > > > > >> >> >> > >> > from
> > > > > >> >> >> > >> > >> WCOSS to Theia and they are located here:
> > > > > >> >> >> > >> > >> /scratch4/NCEPDEV/fv3-cam/
> noscrub/Benjamin.Blake/
> > > > > testfiles
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> I also put the CONUS.nc mask file that I
am
> using
> > in
> > > > > that
> > > > > >> >> >> > directory.
> > > > > >> >> >> > >> > >> Using
> > > > > >> >> >> > >> > >> that file or CONUS.poly didn't seem to
make a
> > > > difference
> > > > > >> >> though.
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> Here is the grid stat command I was using
on
> WCOSS
> > > > > (after
> > > > > >> >> >> removing
> > > > > >> >> >> > >> the
> > > > > >> >> >> > >> > >> unrelevant WCOSS paths):
> > > > > >> >> >> > >> > >> grid_stat
hiresw.t00z.arw_5km.f03.conus.grib2
> > > > > >> >> >> > ccpa.2017052803.03h.gb2
> > > > > >> >> >> > >> > >> GridStatConfig_APCP03_FSS_href -v 2
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> Julie - That'd be great, thanks!
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> Ben
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> On Wed, Aug 29, 2018 at 12:12 PM, Julie
> Prestopnik
> > > via
> > > > > RT
> > > > > >> <
> > > > > >> >> >> > >> > >> met_help at ucar.edu
> > > > > >> >> >> > >> > >> > wrote:
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >> > I'll get met-7.0 updated on tide right
now.
> > I'll
> > > > > >> follow up
> > > > > >> >> >> once
> > > > > >> >> >> > it
> > > > > >> >> >> > >> > >> update
> > > > > >> >> >> > >> > >> > is complete.
> > > > > >> >> >> > >> > >> >
> > > > > >> >> >> > >> > >> > Julie
> > > > > >> >> >> > >> > >> >
> > > > > >> >> >> > >> > >> > On Wed, Aug 29, 2018 at 10:10 AM John
Halley
> > > Gotway
> > > > > via
> > > > > >> RT
> > > > > >> >> <
> > > > > >> >> >> > >> > >> > met_help at ucar.edu> wrote:
> > > > > >> >> >> > >> > >> >
> > > > > >> >> >> > >> > >> > >
> > > > > >> >> >> > >> > >> > > <URL:
> > > > > >> >> >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > > > > >> >> >> > >
> > > > > >> >> >> > >> > >> > >
> > > > > >> >> >> > >> > >> > > Ben,
> > > > > >> >> >> > >> > >> > >
> > > > > >> >> >> > >> > >> > > Yeah, that'd be great.  If you send
me a
> > single
> > > > > input
> > > > > >> >> >> forecast
> > > > > >> >> >> > >> and
> > > > > >> >> >> > >> > >> > analysis
> > > > > >> >> >> > >> > >> > > file for comparison, I'll run those 3
> versions
> > > of
> > > > > >> >> grid-stat
> > > > > >> >> >> > using
> > > > > >> >> >> > >> > your
> > > > > >> >> >> > >> > >> > > configuration and quantify the timing
> > > differences.
> > > > > >> >> >> > >> > >> > >
> > > > > >> >> >> > >> > >> > > You can either point me to them on
theia or
> > post
> > > > > them
> > > > > >> to
> > > > > >> >> our
> > > > > >> >> >> > >> > anonymous
> > > > > >> >> >> > >> > >> > ftp
> > > > > >> >> >> > >> > >> > > site:
> > > > > >> >> >> > >> > >> > >    https://dtcenter.org/met/users
> > > > > >> >> /support/met_help.php#ftp
> > > > > >> >> >> > >> > >> > >
> > > > > >> >> >> > >> > >> > > Thanks,
> > > > > >> >> >> > >> > >> > > John
> > > > > >> >> >> > >> > >> > >
> > > > > >> >> >> > >> > >> > > On Wed, Aug 29, 2018 at 10:06 AM
Benjamin
> > Blake
> > > -
> > > > > NOAA
> > > > > >> >> >> > Affiliate
> > > > > >> >> >> > >> via
> > > > > >> >> >> > >> > >> RT <
> > > > > >> >> >> > >> > >> > > met_help at ucar.edu> wrote:
> > > > > >> >> >> > >> > >> > >
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > >> >> >> > Ticket/Display.html?id=86792
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > > Hi John,
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > > Thanks for letting me know about
the grid
> > stat
> > > > > >> memory
> > > > > >> >> >> > >> allocation
> > > > > >> >> >> > >> > >> issue.
> > > > > >> >> >> > >> > >> > > I
> > > > > >> >> >> > >> > >> > > > am currently pointing to Julie's
met/7.0
> > > > > >> installation
> > > > > >> >> on
> > > > > >> >> >> > WCOSS
> > > > > >> >> >> > >> > >> (Tide),
> > > > > >> >> >> > >> > >> > > and
> > > > > >> >> >> > >> > >> > > > here is the output of grid_stat
--version:
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > > MET Version:    V7.0
> > > > > >> >> >> > >> > >> > > > Repository:
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >>
> > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > > > > >> >> >> > >> > >> > > > Revision:    5483
> > > > > >> >> >> > >> > >> > > > Change Date:    2018-06-13 09:57:36
-0600
> > > (Wed,
> > > > 13
> > > > > >> Jun
> > > > > >> >> >> 2018)
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > > Since the grid stat fix was put in
place
> on
> > > July
> > > > > >> 24th
> > > > > >> >> the
> > > > > >> >> >> > >> version
> > > > > >> >> >> > >> > >> I'm
> > > > > >> >> >> > >> > >> > > using
> > > > > >> >> >> > >> > >> > > > likely doesn't have that fix, so
I'm not
> > sure
> > > if
> > > > > the
> > > > > >> >> bug
> > > > > >> >> >> fix
> > > > > >> >> >> > >> will
> > > > > >> >> >> > >> > >> > resolve
> > > > > >> >> >> > >> > >> > > > the issue or not.  I could try
pointing
> to a
> > > > newer
> > > > > >> >> >> version of
> > > > > >> >> >> > >> MET.
> > > > > >> >> >> > >> > >> I
> > > > > >> >> >> > >> > >> > > > should also let you know that grid
stat
> for
> > > the
> > > > > >> other
> > > > > >> >> line
> > > > > >> >> >> > >> types
> > > > > >> >> >> > >> > >> does
> > > > > >> >> >> > >> > >> > not
> > > > > >> >> >> > >> > >> > > > seem to be slower, I only see a
noticeable
> > > > > slowdown
> > > > > >> >> with
> > > > > >> >> >> the
> > > > > >> >> >> > >> > NBRCNT
> > > > > >> >> >> > >> > >> > line
> > > > > >> >> >> > >> > >> > > > type.
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > > I am evaluating the deterministic
members
> of
> > > the
> > > > > >> HREFv2
> > > > > >> >> >> and
> > > > > >> >> >> > the
> > > > > >> >> >> > >> > NCAR
> > > > > >> >> >> > >> > >> > > > ensemble as well as their
respective
> > means.  I
> > > > > >> verify
> > > > > >> >> >> them on
> > > > > >> >> >> > >> NCEP
> > > > > >> >> >> > >> > >> grid
> > > > > >> >> >> > >> > >> > > 227
> > > > > >> >> >> > >> > >> > > > which is a ~5-km grid (5.079 km).
I can
> > send
> > > > you
> > > > > >> some
> > > > > >> >> >> sample
> > > > > >> >> >> > >> > files
> > > > > >> >> >> > >> > >> for
> > > > > >> >> >> > >> > >> > > > your testing if that would be
easiest.
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > > Thanks!
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > > Ben
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > > On Wed, Aug 29, 2018 at 11:51 AM,
John
> > Halley
> > > > > Gotway
> > > > > >> >> via
> > > > > >> >> >> RT <
> > > > > >> >> >> > >> > >> > > > met_help at ucar.edu> wrote:
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > > > Hi Ben,
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > I see that you've found the met-
7.0
> > runtime
> > > of
> > > > > >> >> >> Grid-Stat is
> > > > > >> >> >> > >> much
> > > > > >> >> >> > >> > >> > slower
> > > > > >> >> >> > >> > >> > > > > than the met-6.0 version.  We did
> > find/fix a
> > > > > >> >> Grid-Stat
> > > > > >> >> >> > memory
> > > > > >> >> >> > >> > >> > > allocation
> > > > > >> >> >> > >> > >> > > > > issue that's posted to the MET
website:
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > https://dtcenter.org/met/
> > > > > users/support/known_issues/
> > > > > >> >> >> > >> > METv7.0/index.php
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > I have two questions...
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > (1) What's the patch level of the
code
> > > you're
> > > > > >> >> currently
> > > > > >> >> >> > >> running?
> > > > > >> >> >> > >> > >> > > Please
> > > > > >> >> >> > >> > >> > > > > run "grid_stat --version" for the
> > particular
> > > > > >> >> executable
> > > > > >> >> >> > >> you're
> > > > > >> >> >> > >> > >> > running
> > > > > >> >> >> > >> > >> > > > and
> > > > > >> >> >> > >> > >> > > > > send me the result.  Should be
something
> > > like
> > > > > >> this:
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > MET Version: V7.0
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > Repository:
> > > > > >> >> >> > >> > >> > >
> > > > > >> >> >>
> > https://svn-met-dev.cgd.ucar.edu/branches/met/met-7.0_bugfix
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > Revision:         5607
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > Change Date: 2018-07-24 14:46:55
-0600
> > (Tue,
> > > > 24
> > > > > >> Jul
> > > > > >> >> >> 2018)
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > (2) Does this bugfix actually fix
the
> > > problem
> > > > > >> you're
> > > > > >> >> >> > seeing?
> > > > > >> >> >> > >> > I'd
> > > > > >> >> >> > >> > >> > like
> > > > > >> >> >> > >> > >> > > to
> > > > > >> >> >> > >> > >> > > > > test this out myself using your
config
> > file.
> > > > I
> > > > > >> can
> > > > > >> >> run
> > > > > >> >> >> 3
> > > > > >> >> >> > >> > >> versions:
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > >    - met-6.0 with latest patches
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > >    - met-7.0 with no patches
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > >    - met-7.0 with latest patches
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > What model/analysis are you
> evaluating?  I
> > > > might
> > > > > >> as
> > > > > >> >> well
> > > > > >> >> >> > use
> > > > > >> >> >> > >> the
> > > > > >> >> >> > >> > >> same
> > > > > >> >> >> > >> > >> > > > > resolution for my testing.
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > Thanks,
> > > > > >> >> >> > >> > >> > > > > John
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > On Wed, Aug 29, 2018 at 9:01 AM
Benjamin
> > > > Blake -
> > > > > >> NOAA
> > > > > >> >> >> > >> Affiliate
> > > > > >> >> >> > >> > >> via
> > > > > >> >> >> > >> > >> > RT
> > > > > >> >> >> > >> > >> > > <
> > > > > >> >> >> > >> > >> > > > > met_help at ucar.edu> wrote:
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > >
> > > > > >> >> >> > >> > >> > > > > > Wed Aug 29 09:01:08 2018:
Request
> 86792
> > > was
> > > > > >> acted
> > > > > >> >> >> upon.
> > > > > >> >> >> > >> > >> > > > > > Transaction: Ticket created by
> > > > > >> >> >> benjamin.blake at noaa.gov
> > > > > >> >> >> > >> > >> > > > > >        Queue: met_help
> > > > > >> >> >> > >> > >> > > > > >      Subject: FSS calculation
with
> grid
> > > stat
> > > > > >> >> >> > >> > >> > > > > >        Owner: Nobody
> > > > > >> >> >> > >> > >> > > > > >   Requestors:
benjamin.blake at noaa.gov
> > > > > >> >> >> > >> > >> > > > > >       Status: new
> > > > > >> >> >> > >> > >> > > > > >  Ticket <URL:
> > > > > >> >> >> > >> > >> > >
> > > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86792
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > > >
> > > > > >> >> >> > >> > >> > > > > >
> > > > > >> >> >> > >> > >> > > > > > Hi,
> > > > > >> >> >> > >> > >> > > > > >
> > > > > >> >> >> > >> > >> > > > > > I recently converted all of my
MET
> > > > > configuration
> > > > > >> >> files
> > > > > >> >> >> > from
> > > > > >> >> >> > >> > >> using
> > > > > >> >> >> > >> > >> > > v6.0
> > > > > >> >> >> > >> > >> > > > to
> > > > > >> >> >> > >> > >> > > > > > v7.0.  One issue I'm now having
is
> that
> > my
> > > > FSS
> > > > > >> >> >> > calculations
> > > > > >> >> >> > >> > are
> > > > > >> >> >> > >> > >> > > taking
> > > > > >> >> >> > >> > >> > > > > much
> > > > > >> >> >> > >> > >> > > > > > longer than they did when I was
using
> > > > METv6.0.
> > > > > >> >> >> Currently
> > > > > >> >> >> > >> it
> > > > > >> >> >> > >> > >> takes
> > > > > >> >> >> > >> > >> > > > ~15-20
> > > > > >> >> >> > >> > >> > > > > > minutes to generate one .stat
file.
> > > > > >> >> >> > >> > >> > > > > >
> > > > > >> >> >> > >> > >> > > > > > I've attached the grid stat
configure
> > file
> > > > > that
> > > > > >> I
> > > > > >> >> am
> > > > > >> >> >> now
> > > > > >> >> >> > >> using
> > > > > >> >> >> > >> > >> with
> > > > > >> >> >> > >> > >> > > > v7.0
> > > > > >> >> >> > >> > >> > > > > >
(GridStatConfig_APCP03_FSS_href) and
> the
> > > > > >> configure
> > > > > >> >> >> file
> > > > > >> >> >> > I
> > > > > >> >> >> > >> was
> > > > > >> >> >> > >> > >> > using
> > > > > >> >> >> > >> > >> > > > with
> > > > > >> >> >> > >> > >> > > > > > v6.0
(GridStatConfig_03h_APCP_NBR).
> > > > > >> >> >> > >> > >> > > > > >
> > > > > >> >> >> > >> > >> > > > > > I modified cat_thresh for APCP
A03
> and I
> > > > also
> > > > > >> >> reverted
> > > > > >> >> >> > back
> > > > > >> >> >> > >> > from
> > > > > >> >> >> > >> > >> > > SHAPE
> > > > > >> >> >> > >> > >> > > > =
> > > > > >> >> >> > >> > >> > > > > > CIRCLE to SHAPE = SQUARE for
the
> > > > neighborhood
> > > > > >> >> methods
> > > > > >> >> >> > >> section,
> > > > > >> >> >> > >> > >> but
> > > > > >> >> >> > >> > >> > > the
> > > > > >> >> >> > >> > >> > > > > grid
> > > > > >> >> >> > >> > >> > > > > > stat calculation still takes
much
> longer
> > > > than
> > > > > it
> > > > > >> >> used
> > > > > >> >> >> to.
> > > > > >> >> >> > >> I
> > > > > >> >> >> > >> > >> also
> > > > > >> >> >> > >> > >> > > tried
> > > > > >> >> >> > >> > >> > > > > > using the CONUS.poly file in
> > > shared/met/poly
> > > > > >> >> instead
> > > > > >> >> >> of
> > > > > >> >> >> > my
> > > > > >> >> >> > >> > >> CONUS.nc
> > > > > >> >> >> > >> > >> > > to
> > > > > >> >> >> > >> > >> > > > > see
> > > > > >> >> >> > >> > >> > > > > > if that would make a difference
but it
> > did
> > > > > not.
> > > > > >> >> The
> > > > > >> >> >> v7.0
> > > > > >> >> >> > >> > >> configure
> > > > > >> >> >> > >> > >> > > > file
> > > > > >> >> >> > >> > >> > > > > > has many more options in it so
perhaps
> > the
> > > > > >> >> slowdown is
> > > > > >> >> >> > >> related
> > > > > >> >> >> > >> > >> to
> > > > > >> >> >> > >> > >> > > > > something
> > > > > >> >> >> > >> > >> > > > > > I added in.
> > > > > >> >> >> > >> > >> > > > > >
> > > > > >> >> >> > >> > >> > > > > > Do you have any suggestions for
why it
> > > might
> > > > > be
> > > > > >> >> >> running
> > > > > >> >> >> > so
> > > > > >> >> >> > >> > >> slowly?
> > > > > >> >> >> > >> > >> > > > > >
> > > > > >> >> >> > >> > >> > > > > > Thanks!
> > > > > >> >> >> > >> > >> > > > > > Ben
> > > > > >> >> >> > >> > >> > > > > >
> > > > > >> >> >> > >> > >> > > > > >
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > > >
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > > >
> > > > > >> >> >> > >> > >> > >
> > > > > >> >> >> > >> > >> > >
> > > > > >> >> >> > >> > >> >
> > > > > >> >> >> > >> > >> >
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> > >>
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >> >
> > > > > >> >> >> > >>
> > > > > >> >> >> > >>
> > > > > >> >> >> >
> > > > > >> >> >> >
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

------------------------------------------------

More information about the Met_help mailing list