[Met_help] [rt.rap.ucar.edu #96739] History for PoP12 smoothing in grid_stat
George McCabe via RT
met_help at ucar.edu
Wed Sep 16 12:02:44 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Greetings
I could use some help understanding what causes smoothing to occur inside
of grid_stat. Currently, when I run for PoP12, I see the following for
each mask when using -v 4:
DEBUG 2: Processing poA12 versus APCP_12A12, for smoothing method
NEAREST(1), over region conus, using 1326186 matched pairs.
DEBUG 2: Computing Probabilistic Statistics.
DEBUG 2: Processing poA12 versus APCP_12A12, for smoothing method
NEAREST(1), over region CR, using 431071 matched pairs.
DEBUG 2: Computing Probabilistic Statistics.
For PoP12, my inputs are a single model (NDFD, Blend, etc) and the URMA
QPF12. I use regrid_data_plane to create netCDF files for each model,
using nearest neighbor interpolation (planning on switching to using the
regrid within grid_stat soon). I use pcp_combine to add two URMA QPF06
grids together, then use regrid_data_plane to resize the grid. I have
tested with nearest neighbor and budget interpolation. Either way, the
smoothing above still takes place.
The memory usage for grid_stat is a bit higher when smoothing is involved.
For most elements, I can run about 40 jobs in parallel on a single node.
For PoP12, I can only run about 5 jobs in parallel (more testing is needed
here, after the current dev outage on venus). If possible, I would like to
avoid the smoothing step in grid_stat. I'm not sure if I'm missing a
setting somewhere in grid_stat, a setting in regrid_data_plane, or in
pcp_combine. If you have any information on what I might be missing, I
would appreciate it.
Probably not relevant, but PoP12 is the only element that I am running
through grid_stat currently for probabilistic scores.
If you need any more info from me, please let me know.
Thanks
John
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: PoP12 smoothing in grid_stat
From: George McCabe
Time: Wed Sep 16 09:52:05 2020
Hi John,
The interp dictionary in the Grid-Stat config file defines smoothing.
The
default values of the type array inside interp (method = NEAREST and
width
= 1) does not perform any smoothing. You can also force no smoothing
by
setting the interp field value to NONE. When smoothing is actually
taking
place, you will see a level 3 log message starting with "Smoothing
field
using the ..."
Based on your log output, it looks like no smoothing is being done, so
that
doesn't seem to be the source of the higher memory usage. Could you
send 2
Grid-Stat config files, one that is showing high memory usage and one
that
is not? This could give me a better idea of what you have set that is
causing your issue.
Thanks,
George
On Wed, Sep 16, 2020 at 8:15 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> Wed Sep 16 08:14:42 2020: Request 96739 was acted upon.
> Transaction: Ticket created by john.l.wagner at noaa.gov
> Queue: met_help
> Subject: PoP12 smoothing in grid_stat
> Owner: Nobody
> Requestors: john.l.wagner at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96739 >
>
>
> Greetings
> I could use some help understanding what causes smoothing to occur
inside
> of grid_stat. Currently, when I run for PoP12, I see the following
for
> each mask when using -v 4:
>
> DEBUG 2: Processing poA12 versus APCP_12A12, for smoothing method
> NEAREST(1), over region conus, using 1326186 matched pairs.
> DEBUG 2: Computing Probabilistic Statistics.
> DEBUG 2: Processing poA12 versus APCP_12A12, for smoothing method
> NEAREST(1), over region CR, using 431071 matched pairs.
> DEBUG 2: Computing Probabilistic Statistics.
>
> For PoP12, my inputs are a single model (NDFD, Blend, etc) and the
URMA
> QPF12. I use regrid_data_plane to create netCDF files for each
model,
> using nearest neighbor interpolation (planning on switching to using
the
> regrid within grid_stat soon). I use pcp_combine to add two URMA
QPF06
> grids together, then use regrid_data_plane to resize the grid. I
have
> tested with nearest neighbor and budget interpolation. Either way,
the
> smoothing above still takes place.
>
> The memory usage for grid_stat is a bit higher when smoothing is
involved.
> For most elements, I can run about 40 jobs in parallel on a single
node.
> For PoP12, I can only run about 5 jobs in parallel (more testing is
needed
> here, after the current dev outage on venus). If possible, I would
like to
> avoid the smoothing step in grid_stat. I'm not sure if I'm missing
a
> setting somewhere in grid_stat, a setting in regrid_data_plane, or
in
> pcp_combine. If you have any information on what I might be
missing, I
> would appreciate it.
>
> Probably not relevant, but PoP12 is the only element that I am
running
> through grid_stat currently for probabilistic scores.
>
> If you need any more info from me, please let me know.
> Thanks
> John
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PoP12 smoothing in grid_stat
From: John L Wagner - NOAA Federal
Time: Wed Sep 16 10:46:03 2020
Thanks George. grid_stat_config_brier_rel_po is what I'm using for
PoP12,
grid_stat_config_mae_bias is what I'm using for temperature, which
doesn't
have an issue.
On Wed, Sep 16, 2020 at 11:52 AM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi John,
>
> The interp dictionary in the Grid-Stat config file defines
smoothing. The
> default values of the type array inside interp (method = NEAREST and
width
> = 1) does not perform any smoothing. You can also force no smoothing
by
> setting the interp field value to NONE. When smoothing is actually
taking
> place, you will see a level 3 log message starting with "Smoothing
field
> using the ..."
>
> Based on your log output, it looks like no smoothing is being done,
so that
> doesn't seem to be the source of the higher memory usage. Could you
send 2
> Grid-Stat config files, one that is showing high memory usage and
one that
> is not? This could give me a better idea of what you have set that
is
> causing your issue.
>
> Thanks,
> George
>
> On Wed, Sep 16, 2020 at 8:15 AM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Wed Sep 16 08:14:42 2020: Request 96739 was acted upon.
> > Transaction: Ticket created by john.l.wagner at noaa.gov
> > Queue: met_help
> > Subject: PoP12 smoothing in grid_stat
> > Owner: Nobody
> > Requestors: john.l.wagner at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96739 >
> >
> >
> > Greetings
> > I could use some help understanding what causes smoothing to occur
inside
> > of grid_stat. Currently, when I run for PoP12, I see the
following for
> > each mask when using -v 4:
> >
> > DEBUG 2: Processing poA12 versus APCP_12A12, for smoothing method
> > NEAREST(1), over region conus, using 1326186 matched pairs.
> > DEBUG 2: Computing Probabilistic Statistics.
> > DEBUG 2: Processing poA12 versus APCP_12A12, for smoothing method
> > NEAREST(1), over region CR, using 431071 matched pairs.
> > DEBUG 2: Computing Probabilistic Statistics.
> >
> > For PoP12, my inputs are a single model (NDFD, Blend, etc) and the
URMA
> > QPF12. I use regrid_data_plane to create netCDF files for each
model,
> > using nearest neighbor interpolation (planning on switching to
using the
> > regrid within grid_stat soon). I use pcp_combine to add two URMA
QPF06
> > grids together, then use regrid_data_plane to resize the grid. I
have
> > tested with nearest neighbor and budget interpolation. Either
way, the
> > smoothing above still takes place.
> >
> > The memory usage for grid_stat is a bit higher when smoothing is
> involved.
> > For most elements, I can run about 40 jobs in parallel on a single
node.
> > For PoP12, I can only run about 5 jobs in parallel (more testing
is
> needed
> > here, after the current dev outage on venus). If possible, I
would like
> to
> > avoid the smoothing step in grid_stat. I'm not sure if I'm
missing a
> > setting somewhere in grid_stat, a setting in regrid_data_plane, or
in
> > pcp_combine. If you have any information on what I might be
missing, I
> > would appreciate it.
> >
> > Probably not relevant, but PoP12 is the only element that I am
running
> > through grid_stat currently for probabilistic scores.
> >
> > If you need any more info from me, please let me know.
> > Thanks
> > John
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: PoP12 smoothing in grid_stat
From: George McCabe
Time: Wed Sep 16 10:59:39 2020
Hi John,
It looks like the config file that is causing the issue is computing
probabilistic outputs while the other is not. The probabilistic
calculations call a completely different set of functions, so it is
not
unlikely that the memory usage differs between them. There are likely
improvements that could be made to optimize the code used for
probabilistic
verification. Let me know if you have any other questions.
Thanks,
George
On Wed, Sep 16, 2020 at 10:46 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96739 >
>
> Thanks George. grid_stat_config_brier_rel_po is what I'm using for
PoP12,
> grid_stat_config_mae_bias is what I'm using for temperature, which
doesn't
> have an issue.
>
> On Wed, Sep 16, 2020 at 11:52 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi John,
> >
> > The interp dictionary in the Grid-Stat config file defines
smoothing. The
> > default values of the type array inside interp (method = NEAREST
and
> width
> > = 1) does not perform any smoothing. You can also force no
smoothing by
> > setting the interp field value to NONE. When smoothing is actually
taking
> > place, you will see a level 3 log message starting with "Smoothing
field
> > using the ..."
> >
> > Based on your log output, it looks like no smoothing is being
done, so
> that
> > doesn't seem to be the source of the higher memory usage. Could
you send
> 2
> > Grid-Stat config files, one that is showing high memory usage and
one
> that
> > is not? This could give me a better idea of what you have set that
is
> > causing your issue.
> >
> > Thanks,
> > George
> >
> > On Wed, Sep 16, 2020 at 8:15 AM John L Wagner - NOAA Federal via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Sep 16 08:14:42 2020: Request 96739 was acted upon.
> > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > Queue: met_help
> > > Subject: PoP12 smoothing in grid_stat
> > > Owner: Nobody
> > > Requestors: john.l.wagner at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96739
> >
> > >
> > >
> > > Greetings
> > > I could use some help understanding what causes smoothing to
occur
> inside
> > > of grid_stat. Currently, when I run for PoP12, I see the
following for
> > > each mask when using -v 4:
> > >
> > > DEBUG 2: Processing poA12 versus APCP_12A12, for smoothing
method
> > > NEAREST(1), over region conus, using 1326186 matched pairs.
> > > DEBUG 2: Computing Probabilistic Statistics.
> > > DEBUG 2: Processing poA12 versus APCP_12A12, for smoothing
method
> > > NEAREST(1), over region CR, using 431071 matched pairs.
> > > DEBUG 2: Computing Probabilistic Statistics.
> > >
> > > For PoP12, my inputs are a single model (NDFD, Blend, etc) and
the URMA
> > > QPF12. I use regrid_data_plane to create netCDF files for each
model,
> > > using nearest neighbor interpolation (planning on switching to
using
> the
> > > regrid within grid_stat soon). I use pcp_combine to add two
URMA QPF06
> > > grids together, then use regrid_data_plane to resize the grid.
I have
> > > tested with nearest neighbor and budget interpolation. Either
way, the
> > > smoothing above still takes place.
> > >
> > > The memory usage for grid_stat is a bit higher when smoothing is
> > involved.
> > > For most elements, I can run about 40 jobs in parallel on a
single
> node.
> > > For PoP12, I can only run about 5 jobs in parallel (more testing
is
> > needed
> > > here, after the current dev outage on venus). If possible, I
would
> like
> > to
> > > avoid the smoothing step in grid_stat. I'm not sure if I'm
missing a
> > > setting somewhere in grid_stat, a setting in regrid_data_plane,
or in
> > > pcp_combine. If you have any information on what I might be
missing, I
> > > would appreciate it.
> > >
> > > Probably not relevant, but PoP12 is the only element that I am
running
> > > through grid_stat currently for probabilistic scores.
> > >
> > > If you need any more info from me, please let me know.
> > > Thanks
> > > John
> > >
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Division
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PoP12 smoothing in grid_stat
From: John L Wagner - NOAA Federal
Time: Wed Sep 16 11:13:54 2020
Thanks George. I'll focus then on optimizing my probabilistic jobs
differently from my other elements. And thanks for confirming that no
additional smoothing is taking place.
John
On Wed, Sep 16, 2020 at 12:59 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi John,
>
> It looks like the config file that is causing the issue is computing
> probabilistic outputs while the other is not. The probabilistic
> calculations call a completely different set of functions, so it is
not
> unlikely that the memory usage differs between them. There are
likely
> improvements that could be made to optimize the code used for
probabilistic
> verification. Let me know if you have any other questions.
>
> Thanks,
> George
>
> On Wed, Sep 16, 2020 at 10:46 AM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96739 >
> >
> > Thanks George. grid_stat_config_brier_rel_po is what I'm using
for
> PoP12,
> > grid_stat_config_mae_bias is what I'm using for temperature, which
> doesn't
> > have an issue.
> >
> > On Wed, Sep 16, 2020 at 11:52 AM George McCabe via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > > Hi John,
> > >
> > > The interp dictionary in the Grid-Stat config file defines
smoothing.
> The
> > > default values of the type array inside interp (method = NEAREST
and
> > width
> > > = 1) does not perform any smoothing. You can also force no
smoothing by
> > > setting the interp field value to NONE. When smoothing is
actually
> taking
> > > place, you will see a level 3 log message starting with
"Smoothing
> field
> > > using the ..."
> > >
> > > Based on your log output, it looks like no smoothing is being
done, so
> > that
> > > doesn't seem to be the source of the higher memory usage. Could
you
> send
> > 2
> > > Grid-Stat config files, one that is showing high memory usage
and one
> > that
> > > is not? This could give me a better idea of what you have set
that is
> > > causing your issue.
> > >
> > > Thanks,
> > > George
> > >
> > > On Wed, Sep 16, 2020 at 8:15 AM John L Wagner - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Sep 16 08:14:42 2020: Request 96739 was acted upon.
> > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > Queue: met_help
> > > > Subject: PoP12 smoothing in grid_stat
> > > > Owner: Nobody
> > > > Requestors: john.l.wagner at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96739
> > >
> > > >
> > > >
> > > > Greetings
> > > > I could use some help understanding what causes smoothing to
occur
> > inside
> > > > of grid_stat. Currently, when I run for PoP12, I see the
following
> for
> > > > each mask when using -v 4:
> > > >
> > > > DEBUG 2: Processing poA12 versus APCP_12A12, for smoothing
method
> > > > NEAREST(1), over region conus, using 1326186 matched pairs.
> > > > DEBUG 2: Computing Probabilistic Statistics.
> > > > DEBUG 2: Processing poA12 versus APCP_12A12, for smoothing
method
> > > > NEAREST(1), over region CR, using 431071 matched pairs.
> > > > DEBUG 2: Computing Probabilistic Statistics.
> > > >
> > > > For PoP12, my inputs are a single model (NDFD, Blend, etc) and
the
> URMA
> > > > QPF12. I use regrid_data_plane to create netCDF files for
each
> model,
> > > > using nearest neighbor interpolation (planning on switching to
using
> > the
> > > > regrid within grid_stat soon). I use pcp_combine to add two
URMA
> QPF06
> > > > grids together, then use regrid_data_plane to resize the grid.
I
> have
> > > > tested with nearest neighbor and budget interpolation. Either
way,
> the
> > > > smoothing above still takes place.
> > > >
> > > > The memory usage for grid_stat is a bit higher when smoothing
is
> > > involved.
> > > > For most elements, I can run about 40 jobs in parallel on a
single
> > node.
> > > > For PoP12, I can only run about 5 jobs in parallel (more
testing is
> > > needed
> > > > here, after the current dev outage on venus). If possible, I
would
> > like
> > > to
> > > > avoid the smoothing step in grid_stat. I'm not sure if I'm
missing a
> > > > setting somewhere in grid_stat, a setting in
regrid_data_plane, or in
> > > > pcp_combine. If you have any information on what I might be
> missing, I
> > > > would appreciate it.
> > > >
> > > > Probably not relevant, but PoP12 is the only element that I am
> running
> > > > through grid_stat currently for probabilistic scores.
> > > >
> > > > If you need any more info from me, please let me know.
> > > > Thanks
> > > > John
> > > >
> > > > --
> > > > John Wagner
> > > > Verification Task Lead
> > > > NOAA/National Weather Service
> > > > Meteorological Development Laboratory
> > > > Digital Forecast Services Division
> > > > SSMC2 Room 10106
> > > > Silver Spring, MD 20910
> > > > (301) 427-9471 (office)
> > > > (908) 902-4155 (cell/text)
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
More information about the Met_help
mailing list