[Met_help] [rt.rap.ucar.edu #80907] History for Grid_stat Issue When Applying Mask
John Halley Gotway via RT
met_help at ucar.edu
Tue Jun 20 17:22:54 MDT 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning!
I am attempting to run grid_stat on two different netCDF files: 1)
20170613_ERO.nc which is a binomial field of ones and zeros and 2)
20170613_ST4gFFG.nc, which is another binomial field. I created these
netCDF files from python, so some of the metadata description will be
strange (they were not created by regrid_data_plane). When I run grid_stat
with no masking field, everything runs fine. However, I want to mask the
data to only include verification statistics inside the region of where
20170613_ERO.nc is equal to one. When I try to include 20170613_ERO.nc as a
mask, I get an error:
*ERROR : mask_bad_data() -> grid dimensions do not match*
My error-ing code in the grid_stat config file is:
*mask = { grid = [ "FULL" ];// poly = []; poly = [
"/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc {name =
\"ERO_MRGLs\"; level = \"Surface\";} > 0"];}*
I find this error strange, since all of the data in an earlier step was
interpolated to a 0.1 X 0.1 degree lat/lon grid.
I have also attached my config file, 20170613_ERO.nc, and
20170613_ST4gFFG.nc.
Sorry for the multiple questions over the past few weeks and thank you for
all of your help!
Mike
--
Michael J. Erickson, Ph.D
Cooperative Institute for Research in Environmental Sciences (CIRES)
CIRES Contractor for the Weather Prediction Center (WPC)
Phone: 301-683-1546
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Grid_stat Issue When Applying Mask
From: John Halley Gotway
Time: Mon Jun 19 10:45:44 2017
Mike,
Thanks for sending your sample data files. The first thing I did with
your
data was run it through plot_data_plane. The resulting images are
attached:
met-6.0/bin/plot_data_plane 20170613_ERO.nc 20170613_ERO.ps
'name="ERO_MRGL"; level="(*,*)";'
met-6.0/bin/plot_data_plane 20170613_ST4gFFG.nc 20170613_ST4gFFG.ps
'name="ST4gFFG"; level="(*,*)";'
I mention this because when I compare these images to what ncview
shows,
they are flipped vertically. So the order of the latitudes in your
file
(which ncview uses) differ from the global attribute grid definition
(which
MET uses). I'm guessing the ncview orientation is the "correct" one.
Next, on to your Grid-Stat error. I think you've found a very
misleading
error message! The underlying problem is just a typo in the name of
the
field. Here's your setting:
poly = [ "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
{name =
\"ERO_MRGLs\"; level = \"Surface\";} > 0"];
But the NetCDF variable is named "ERO_MRGL" *without* the trailing
"s".
When I remove the "s", it works fine. Also the threshold should be
">0"
without a space... but I do see that having the space "> 0" also
works.
Please try this setting:
poly = [ "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
{name =
\"ERO_MRGL\"; level = \"Surface\";} >0"];
And I'll use your data to test the current development version. We
obviously want a more useful error message than what you saw.
Thanks,
John
On Mon, Jun 19, 2017 at 8:49 AM, Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Mon Jun 19 08:49:26 2017: Request 80907 was acted upon.
> Transaction: Ticket created by michael.j.erickson at noaa.gov
> Queue: met_help
> Subject: Grid_stat Issue When Applying Mask
> Owner: Nobody
> Requestors: michael.j.erickson at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80907 >
>
>
> Good morning!
>
> I am attempting to run grid_stat on two different netCDF files: 1)
> 20170613_ERO.nc which is a binomial field of ones and zeros and 2)
> 20170613_ST4gFFG.nc, which is another binomial field. I created
these
> netCDF files from python, so some of the metadata description will
be
> strange (they were not created by regrid_data_plane). When I run
grid_stat
> with no masking field, everything runs fine. However, I want to mask
the
> data to only include verification statistics inside the region of
where
> 20170613_ERO.nc is equal to one. When I try to include
20170613_ERO.nc as a
> mask, I get an error:
>
> *ERROR : mask_bad_data() -> grid dimensions do not match*
>
> My error-ing code in the grid_stat config file is:
>
>
>
>
>
>
> *mask = { grid = [ "FULL" ];// poly = []; poly = [
> "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc {name =
> \"ERO_MRGLs\"; level = \"Surface\";} > 0"];}*
> I find this error strange, since all of the data in an earlier step
was
> interpolated to a 0.1 X 0.1 degree lat/lon grid.
>
> I have also attached my config file, 20170613_ERO.nc, and
> 20170613_ST4gFFG.nc.
>
> Sorry for the multiple questions over the past few weeks and thank
you for
> all of your help!
>
> Mike
>
> --
>
> Michael J. Erickson, Ph.D
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> CIRES Contractor for the Weather Prediction Center (WPC)
>
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: Grid_stat Issue When Applying Mask
From: Michael Erickson - NOAA Affiliate
Time: Mon Jun 19 12:06:09 2017
Thank you John for your helpful response!
I've reversed the order of the latitude array and adjusted the gridded
data
accordingly before creating the netcdf file. Thereafter, I've adjusted
my
config file as you suggested. Attached is my "*pairs.nc" file from the
resulting output.
I believe it is correct, but I wanted to make sure. Thanks again!
Mike
On Mon, Jun 19, 2017 at 4:45 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Mike,
>
> Thanks for sending your sample data files. The first thing I did
with your
> data was run it through plot_data_plane. The resulting images are
> attached:
> met-6.0/bin/plot_data_plane 20170613_ERO.nc 20170613_ERO.ps
> 'name="ERO_MRGL"; level="(*,*)";'
> met-6.0/bin/plot_data_plane 20170613_ST4gFFG.nc
20170613_ST4gFFG.ps
> 'name="ST4gFFG"; level="(*,*)";'
>
> I mention this because when I compare these images to what ncview
shows,
> they are flipped vertically. So the order of the latitudes in your
file
> (which ncview uses) differ from the global attribute grid definition
(which
> MET uses). I'm guessing the ncview orientation is the "correct"
one.
>
> Next, on to your Grid-Stat error. I think you've found a very
misleading
> error message! The underlying problem is just a typo in the name of
the
> field. Here's your setting:
>
> poly = [ "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
{name =
> \"ERO_MRGLs\"; level = \"Surface\";} > 0"];
>
> But the NetCDF variable is named "ERO_MRGL" *without* the trailing
"s".
> When I remove the "s", it works fine. Also the threshold should be
">0"
> without a space... but I do see that having the space "> 0" also
works.
> Please try this setting:
>
> poly = [ "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
{name =
> \"ERO_MRGL\"; level = \"Surface\";} >0"];
>
> And I'll use your data to test the current development version. We
> obviously want a more useful error message than what you saw.
>
> Thanks,
> John
>
>
> On Mon, Jun 19, 2017 at 8:49 AM, Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Jun 19 08:49:26 2017: Request 80907 was acted upon.
> > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > Queue: met_help
> > Subject: Grid_stat Issue When Applying Mask
> > Owner: Nobody
> > Requestors: michael.j.erickson at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80907 >
> >
> >
> > Good morning!
> >
> > I am attempting to run grid_stat on two different netCDF files: 1)
> > 20170613_ERO.nc which is a binomial field of ones and zeros and 2)
> > 20170613_ST4gFFG.nc, which is another binomial field. I created
these
> > netCDF files from python, so some of the metadata description will
be
> > strange (they were not created by regrid_data_plane). When I run
> grid_stat
> > with no masking field, everything runs fine. However, I want to
mask the
> > data to only include verification statistics inside the region of
where
> > 20170613_ERO.nc is equal to one. When I try to include
20170613_ERO.nc
> as a
> > mask, I get an error:
> >
> > *ERROR : mask_bad_data() -> grid dimensions do not match*
> >
> > My error-ing code in the grid_stat config file is:
> >
> >
> >
> >
> >
> >
> > *mask = { grid = [ "FULL" ];// poly = []; poly = [
> > "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc {name =
> > \"ERO_MRGLs\"; level = \"Surface\";} > 0"];}*
> > I find this error strange, since all of the data in an earlier
step was
> > interpolated to a 0.1 X 0.1 degree lat/lon grid.
> >
> > I have also attached my config file, 20170613_ERO.nc, and
> > 20170613_ST4gFFG.nc.
> >
> > Sorry for the multiple questions over the past few weeks and thank
you
> for
> > all of your help!
> >
> > Mike
> >
> > --
> >
> > Michael J. Erickson, Ph.D
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > CIRES Contractor for the Weather Prediction Center (WPC)
> >
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson, Ph.D
Cooperative Institute for Research in Environmental Sciences (CIRES)
CIRES Contractor for the Weather Prediction Center (WPC)
Phone: 301-683-1546
------------------------------------------------
Subject: Grid_stat Issue When Applying Mask
From: John Halley Gotway
Time: Mon Jun 19 13:30:35 2017
Mike,
The pairs file looks *reasonable* to me, but you need to confirm that
the
data really is in the spot it is supposed to be. Based on the file
you
sent, it looks like the forecast data is a large swath of 1's across
the
northern United States. And the observations are very speckled across
the
southern US and up the east coast. I ran the following
plot_data_plane
command to plot the difference field from grid_stat (attached):
met-6.0/bin/plot_data_plane \
grid_stat_230000L_20170614_120000V_pairs.nc diff.ps \
'name="DIFF_ERO_MRGL_Surface_ST4gFFG_Surface_FULL"; level="(*,*)";' \
-color_table
/usr/local/met-
6.0/share/met/colortables/NCL_colortables/BlueRed.ctable
As long as you can confirm that the data is oriented correctly, I
don't see
any problems. I tried to grab a plot of the StageIV data for this
time
from this site:
ftp://ftp.emc.ncep.noaa.gov/mmb/precip/st2n4.arch/201706/
But for some reason, data for 12Z on 20170614 is not yet available.
Out of curiosity, why are you producing a NetCDF version of Stage4
instead
of just using the GRIB2 data available publicly? Or is this some
specialized variant of it?
Thanks,
John
On Mon, Jun 19, 2017 at 12:06 PM, Michael Erickson - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80907 >
>
> Thank you John for your helpful response!
>
> I've reversed the order of the latitude array and adjusted the
gridded data
> accordingly before creating the netcdf file. Thereafter, I've
adjusted my
> config file as you suggested. Attached is my "*pairs.nc" file from
the
> resulting output.
>
> I believe it is correct, but I wanted to make sure. Thanks again!
>
> Mike
>
> On Mon, Jun 19, 2017 at 4:45 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Mike,
> >
> > Thanks for sending your sample data files. The first thing I did
with
> your
> > data was run it through plot_data_plane. The resulting images are
> > attached:
> > met-6.0/bin/plot_data_plane 20170613_ERO.nc 20170613_ERO.ps
> > 'name="ERO_MRGL"; level="(*,*)";'
> > met-6.0/bin/plot_data_plane 20170613_ST4gFFG.nc
20170613_ST4gFFG.ps
> > 'name="ST4gFFG"; level="(*,*)";'
> >
> > I mention this because when I compare these images to what ncview
shows,
> > they are flipped vertically. So the order of the latitudes in
your file
> > (which ncview uses) differ from the global attribute grid
definition
> (which
> > MET uses). I'm guessing the ncview orientation is the "correct"
one.
> >
> > Next, on to your Grid-Stat error. I think you've found a very
misleading
> > error message! The underlying problem is just a typo in the name
of the
> > field. Here's your setting:
> >
> > poly = [ "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
> {name =
> > \"ERO_MRGLs\"; level = \"Surface\";} > 0"];
> >
> > But the NetCDF variable is named "ERO_MRGL" *without* the trailing
"s".
> > When I remove the "s", it works fine. Also the threshold should
be ">0"
> > without a space... but I do see that having the space "> 0" also
works.
> > Please try this setting:
> >
> > poly = [ "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
> {name =
> > \"ERO_MRGL\"; level = \"Surface\";} >0"];
> >
> > And I'll use your data to test the current development version.
We
> > obviously want a more useful error message than what you saw.
> >
> > Thanks,
> > John
> >
> >
> > On Mon, Jun 19, 2017 at 8:49 AM, Michael Erickson - NOAA Affiliate
via
> RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Jun 19 08:49:26 2017: Request 80907 was acted upon.
> > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > Queue: met_help
> > > Subject: Grid_stat Issue When Applying Mask
> > > Owner: Nobody
> > > Requestors: michael.j.erickson at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80907
> >
> > >
> > >
> > > Good morning!
> > >
> > > I am attempting to run grid_stat on two different netCDF files:
1)
> > > 20170613_ERO.nc which is a binomial field of ones and zeros and
2)
> > > 20170613_ST4gFFG.nc, which is another binomial field. I created
these
> > > netCDF files from python, so some of the metadata description
will be
> > > strange (they were not created by regrid_data_plane). When I run
> > grid_stat
> > > with no masking field, everything runs fine. However, I want to
mask
> the
> > > data to only include verification statistics inside the region
of where
> > > 20170613_ERO.nc is equal to one. When I try to include
20170613_ERO.nc
> > as a
> > > mask, I get an error:
> > >
> > > *ERROR : mask_bad_data() -> grid dimensions do not match*
> > >
> > > My error-ing code in the grid_stat config file is:
> > >
> > >
> > >
> > >
> > >
> > >
> > > *mask = { grid = [ "FULL" ];// poly = []; poly = [
> > > "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc {name
=
> > > \"ERO_MRGLs\"; level = \"Surface\";} > 0"];}*
> > > I find this error strange, since all of the data in an earlier
step was
> > > interpolated to a 0.1 X 0.1 degree lat/lon grid.
> > >
> > > I have also attached my config file, 20170613_ERO.nc, and
> > > 20170613_ST4gFFG.nc.
> > >
> > > Sorry for the multiple questions over the past few weeks and
thank you
> > for
> > > all of your help!
> > >
> > > Mike
> > >
> > > --
> > >
> > > Michael J. Erickson, Ph.D
> > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > CIRES Contractor for the Weather Prediction Center (WPC)
> > >
> > > Phone: 301-683-1546
> > >
> > >
> >
> >
>
>
> --
>
> Michael J. Erickson, Ph.D
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> CIRES Contractor for the Weather Prediction Center (WPC)
>
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: Grid_stat Issue When Applying Mask
From: Michael Erickson - NOAA Affiliate
Time: Mon Jun 19 13:49:49 2017
Hey John,
Thanks for your help on this.
Regarding the Stage4, it's strange that site hasn't updated for a
week. It
may wait until the final version, which takes about that long. I pull
from
our WPC internal site, which has Stage4 up to 18 UTC today.
It might seem strange what I'm doing given the fields I've given you.
The
first (the image you sent attached) is a binomial grid of the WPC
excessive
rainfall outlook. In your attached image, everywhere = 1 is under a
marginal risk. The second field compares flash flood guidance to
Stage4
(originally in grib2) for several 1, 3, and 6 hour increments
throughout a
24 hour period. If anywhere at any point had Stage4 exceed flash flood
guidance, the grid is = 1. My next step is to apply a 40 km filter to
the
Stage4 exceeding flash flood guidance grid. I also need to time match
these
two datasets.
Thanks for your help!
Mike
On Mon, Jun 19, 2017 at 7:30 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Mike,
>
> The pairs file looks *reasonable* to me, but you need to confirm
that the
> data really is in the spot it is supposed to be. Based on the file
you
> sent, it looks like the forecast data is a large swath of 1's across
the
> northern United States. And the observations are very speckled
across the
> southern US and up the east coast. I ran the following
plot_data_plane
> command to plot the difference field from grid_stat (attached):
>
> met-6.0/bin/plot_data_plane \
> grid_stat_230000L_20170614_120000V_pairs.nc diff.ps \
> 'name="DIFF_ERO_MRGL_Surface_ST4gFFG_Surface_FULL"; level="(*,*)";'
\
> -color_table
> /usr/local/met-
6.0/share/met/colortables/NCL_colortables/BlueRed.ctable
>
> As long as you can confirm that the data is oriented correctly, I
don't see
> any problems. I tried to grab a plot of the StageIV data for this
time
> from this site:
> ftp://ftp.emc.ncep.noaa.gov/mmb/precip/st2n4.arch/201706/
>
> But for some reason, data for 12Z on 20170614 is not yet available.
>
> Out of curiosity, why are you producing a NetCDF version of Stage4
instead
> of just using the GRIB2 data available publicly? Or is this some
> specialized variant of it?
>
> Thanks,
> John
>
>
> On Mon, Jun 19, 2017 at 12:06 PM, Michael Erickson - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80907 >
> >
> > Thank you John for your helpful response!
> >
> > I've reversed the order of the latitude array and adjusted the
gridded
> data
> > accordingly before creating the netcdf file. Thereafter, I've
adjusted my
> > config file as you suggested. Attached is my "*pairs.nc" file from
the
> > resulting output.
> >
> > I believe it is correct, but I wanted to make sure. Thanks again!
> >
> > Mike
> >
> > On Mon, Jun 19, 2017 at 4:45 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Mike,
> > >
> > > Thanks for sending your sample data files. The first thing I
did with
> > your
> > > data was run it through plot_data_plane. The resulting images
are
> > > attached:
> > > met-6.0/bin/plot_data_plane 20170613_ERO.nc 20170613_ERO.ps
> > > 'name="ERO_MRGL"; level="(*,*)";'
> > > met-6.0/bin/plot_data_plane 20170613_ST4gFFG.nc
20170613_ST4gFFG.ps
> > > 'name="ST4gFFG"; level="(*,*)";'
> > >
> > > I mention this because when I compare these images to what
ncview
> shows,
> > > they are flipped vertically. So the order of the latitudes in
your
> file
> > > (which ncview uses) differ from the global attribute grid
definition
> > (which
> > > MET uses). I'm guessing the ncview orientation is the "correct"
one.
> > >
> > > Next, on to your Grid-Stat error. I think you've found a very
> misleading
> > > error message! The underlying problem is just a typo in the
name of
> the
> > > field. Here's your setting:
> > >
> > > poly = [
"/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
> > {name =
> > > \"ERO_MRGLs\"; level = \"Surface\";} > 0"];
> > >
> > > But the NetCDF variable is named "ERO_MRGL" *without* the
trailing "s".
> > > When I remove the "s", it works fine. Also the threshold should
be
> ">0"
> > > without a space... but I do see that having the space "> 0" also
works.
> > > Please try this setting:
> > >
> > > poly = [
"/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
> > {name =
> > > \"ERO_MRGL\"; level = \"Surface\";} >0"];
> > >
> > > And I'll use your data to test the current development version.
We
> > > obviously want a more useful error message than what you saw.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Mon, Jun 19, 2017 at 8:49 AM, Michael Erickson - NOAA
Affiliate via
> > RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Jun 19 08:49:26 2017: Request 80907 was acted upon.
> > > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > > Queue: met_help
> > > > Subject: Grid_stat Issue When Applying Mask
> > > > Owner: Nobody
> > > > Requestors: michael.j.erickson at noaa.gov
> > > > Status: new
> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=80907
> > >
> > > >
> > > >
> > > > Good morning!
> > > >
> > > > I am attempting to run grid_stat on two different netCDF
files: 1)
> > > > 20170613_ERO.nc which is a binomial field of ones and zeros
and 2)
> > > > 20170613_ST4gFFG.nc, which is another binomial field. I
created these
> > > > netCDF files from python, so some of the metadata description
will be
> > > > strange (they were not created by regrid_data_plane). When I
run
> > > grid_stat
> > > > with no masking field, everything runs fine. However, I want
to mask
> > the
> > > > data to only include verification statistics inside the region
of
> where
> > > > 20170613_ERO.nc is equal to one. When I try to include
> 20170613_ERO.nc
> > > as a
> > > > mask, I get an error:
> > > >
> > > > *ERROR : mask_bad_data() -> grid dimensions do not match*
> > > >
> > > > My error-ing code in the grid_stat config file is:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *mask = { grid = [ "FULL" ];// poly = []; poly = [
> > > > "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
{name =
> > > > \"ERO_MRGLs\"; level = \"Surface\";} > 0"];}*
> > > > I find this error strange, since all of the data in an earlier
step
> was
> > > > interpolated to a 0.1 X 0.1 degree lat/lon grid.
> > > >
> > > > I have also attached my config file, 20170613_ERO.nc, and
> > > > 20170613_ST4gFFG.nc.
> > > >
> > > > Sorry for the multiple questions over the past few weeks and
thank
> you
> > > for
> > > > all of your help!
> > > >
> > > > Mike
> > > >
> > > > --
> > > >
> > > > Michael J. Erickson, Ph.D
> > > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > > CIRES Contractor for the Weather Prediction Center (WPC)
> > > >
> > > > Phone: 301-683-1546
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> >
> > Michael J. Erickson, Ph.D
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > CIRES Contractor for the Weather Prediction Center (WPC)
> >
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson, Ph.D
Cooperative Institute for Research in Environmental Sciences (CIRES)
CIRES Contractor for the Weather Prediction Center (WPC)
Phone: 301-683-1546
------------------------------------------------
Subject: Grid_stat Issue When Applying Mask
From: John Halley Gotway
Time: Tue Jun 20 11:35:13 2017
Mike,
Regarding the last point you made, applying a 40km filter to the
Stage4
exceedance data, you may be able to do this directly in Grid-Stat.
You
could try configuring:
//
// Data smoothing methods
//
interp = {
field = OBS;
vld_thresh = 1.0;
shape = CIRCLE;
type = [
{
method = MAX;
width = 11;
}
];
}
This tells Grid-Stat to process each observation grid point by drawing
a
circle of radius 11 around it, find the maximum value of all grid
points in
that circle, and write that value to the current grid point. I
*think*
this logic would essentially "fatten up" the StageIV data in the way
you
want.
Hope that helps.
Thanks,
John
On Mon, Jun 19, 2017 at 1:49 PM, Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80907 >
>
> Hey John,
>
> Thanks for your help on this.
>
> Regarding the Stage4, it's strange that site hasn't updated for a
week. It
> may wait until the final version, which takes about that long. I
pull from
> our WPC internal site, which has Stage4 up to 18 UTC today.
>
> It might seem strange what I'm doing given the fields I've given
you. The
> first (the image you sent attached) is a binomial grid of the WPC
excessive
> rainfall outlook. In your attached image, everywhere = 1 is under a
> marginal risk. The second field compares flash flood guidance to
Stage4
> (originally in grib2) for several 1, 3, and 6 hour increments
throughout a
> 24 hour period. If anywhere at any point had Stage4 exceed flash
flood
> guidance, the grid is = 1. My next step is to apply a 40 km filter
to the
> Stage4 exceeding flash flood guidance grid. I also need to time
match these
> two datasets.
>
> Thanks for your help!
>
> Mike
>
>
>
> On Mon, Jun 19, 2017 at 7:30 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Mike,
> >
> > The pairs file looks *reasonable* to me, but you need to confirm
that the
> > data really is in the spot it is supposed to be. Based on the
file you
> > sent, it looks like the forecast data is a large swath of 1's
across the
> > northern United States. And the observations are very speckled
across
> the
> > southern US and up the east coast. I ran the following
plot_data_plane
> > command to plot the difference field from grid_stat (attached):
> >
> > met-6.0/bin/plot_data_plane \
> > grid_stat_230000L_20170614_120000V_pairs.nc diff.ps \
> > 'name="DIFF_ERO_MRGL_Surface_ST4gFFG_Surface_FULL";
level="(*,*)";' \
> > -color_table
> > /usr/local/met-
6.0/share/met/colortables/NCL_colortables/BlueRed.ctable
> >
> > As long as you can confirm that the data is oriented correctly, I
don't
> see
> > any problems. I tried to grab a plot of the StageIV data for this
time
> > from this site:
> > ftp://ftp.emc.ncep.noaa.gov/mmb/precip/st2n4.arch/201706/
> >
> > But for some reason, data for 12Z on 20170614 is not yet
available.
> >
> > Out of curiosity, why are you producing a NetCDF version of Stage4
> instead
> > of just using the GRIB2 data available publicly? Or is this some
> > specialized variant of it?
> >
> > Thanks,
> > John
> >
> >
> > On Mon, Jun 19, 2017 at 12:06 PM, Michael Erickson - NOAA
Affiliate via
> RT
> > <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80907 >
> > >
> > > Thank you John for your helpful response!
> > >
> > > I've reversed the order of the latitude array and adjusted the
gridded
> > data
> > > accordingly before creating the netcdf file. Thereafter, I've
adjusted
> my
> > > config file as you suggested. Attached is my "*pairs.nc" file
from the
> > > resulting output.
> > >
> > > I believe it is correct, but I wanted to make sure. Thanks
again!
> > >
> > > Mike
> > >
> > > On Mon, Jun 19, 2017 at 4:45 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Mike,
> > > >
> > > > Thanks for sending your sample data files. The first thing I
did
> with
> > > your
> > > > data was run it through plot_data_plane. The resulting images
are
> > > > attached:
> > > > met-6.0/bin/plot_data_plane 20170613_ERO.nc 20170613_ERO.ps
> > > > 'name="ERO_MRGL"; level="(*,*)";'
> > > > met-6.0/bin/plot_data_plane 20170613_ST4gFFG.nc
> 20170613_ST4gFFG.ps
> > > > 'name="ST4gFFG"; level="(*,*)";'
> > > >
> > > > I mention this because when I compare these images to what
ncview
> > shows,
> > > > they are flipped vertically. So the order of the latitudes in
your
> > file
> > > > (which ncview uses) differ from the global attribute grid
definition
> > > (which
> > > > MET uses). I'm guessing the ncview orientation is the
"correct" one.
> > > >
> > > > Next, on to your Grid-Stat error. I think you've found a very
> > misleading
> > > > error message! The underlying problem is just a typo in the
name of
> > the
> > > > field. Here's your setting:
> > > >
> > > > poly = [
"/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
> > > {name =
> > > > \"ERO_MRGLs\"; level = \"Surface\";} > 0"];
> > > >
> > > > But the NetCDF variable is named "ERO_MRGL" *without* the
trailing
> "s".
> > > > When I remove the "s", it works fine. Also the threshold
should be
> > ">0"
> > > > without a space... but I do see that having the space "> 0"
also
> works.
> > > > Please try this setting:
> > > >
> > > > poly = [
"/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
> > > {name =
> > > > \"ERO_MRGL\"; level = \"Surface\";} >0"];
> > > >
> > > > And I'll use your data to test the current development
version. We
> > > > obviously want a more useful error message than what you saw.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Mon, Jun 19, 2017 at 8:49 AM, Michael Erickson - NOAA
Affiliate
> via
> > > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Mon Jun 19 08:49:26 2017: Request 80907 was acted upon.
> > > > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: Grid_stat Issue When Applying Mask
> > > > > Owner: Nobody
> > > > > Requestors: michael.j.erickson at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=80907
> > > >
> > > > >
> > > > >
> > > > > Good morning!
> > > > >
> > > > > I am attempting to run grid_stat on two different netCDF
files: 1)
> > > > > 20170613_ERO.nc which is a binomial field of ones and zeros
and 2)
> > > > > 20170613_ST4gFFG.nc, which is another binomial field. I
created
> these
> > > > > netCDF files from python, so some of the metadata
description will
> be
> > > > > strange (they were not created by regrid_data_plane). When I
run
> > > > grid_stat
> > > > > with no masking field, everything runs fine. However, I want
to
> mask
> > > the
> > > > > data to only include verification statistics inside the
region of
> > where
> > > > > 20170613_ERO.nc is equal to one. When I try to include
> > 20170613_ERO.nc
> > > > as a
> > > > > mask, I get an error:
> > > > >
> > > > > *ERROR : mask_bad_data() -> grid dimensions do not match*
> > > > >
> > > > > My error-ing code in the grid_stat config file is:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *mask = { grid = [ "FULL" ];// poly = []; poly = [
> > > > > "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
{name =
> > > > > \"ERO_MRGLs\"; level = \"Surface\";} > 0"];}*
> > > > > I find this error strange, since all of the data in an
earlier step
> > was
> > > > > interpolated to a 0.1 X 0.1 degree lat/lon grid.
> > > > >
> > > > > I have also attached my config file, 20170613_ERO.nc, and
> > > > > 20170613_ST4gFFG.nc.
> > > > >
> > > > > Sorry for the multiple questions over the past few weeks and
thank
> > you
> > > > for
> > > > > all of your help!
> > > > >
> > > > > Mike
> > > > >
> > > > > --
> > > > >
> > > > > Michael J. Erickson, Ph.D
> > > > > Cooperative Institute for Research in Environmental Sciences
> (CIRES)
> > > > > CIRES Contractor for the Weather Prediction Center (WPC)
> > > > >
> > > > > Phone: 301-683-1546
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Michael J. Erickson, Ph.D
> > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > CIRES Contractor for the Weather Prediction Center (WPC)
> > >
> > > Phone: 301-683-1546
> > >
> > >
> >
> >
>
>
> --
>
> Michael J. Erickson, Ph.D
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> CIRES Contractor for the Weather Prediction Center (WPC)
>
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: Grid_stat Issue When Applying Mask
From: Michael Erickson - NOAA Affiliate
Time: Tue Jun 20 12:22:18 2017
Hi John,
Your suggestion worked great! Thanks!
Mike
On Tue, Jun 20, 2017 at 5:35 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Mike,
>
> Regarding the last point you made, applying a 40km filter to the
Stage4
> exceedance data, you may be able to do this directly in Grid-Stat.
You
> could try configuring:
>
> //
> // Data smoothing methods
> //
> interp = {
> field = OBS;
> vld_thresh = 1.0;
> shape = CIRCLE;
>
> type = [
> {
> method = MAX;
> width = 11;
> }
> ];
> }
>
> This tells Grid-Stat to process each observation grid point by
drawing a
> circle of radius 11 around it, find the maximum value of all grid
points in
> that circle, and write that value to the current grid point. I
*think*
> this logic would essentially "fatten up" the StageIV data in the way
you
> want.
>
> Hope that helps.
>
> Thanks,
> John
>
> On Mon, Jun 19, 2017 at 1:49 PM, Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80907 >
> >
> > Hey John,
> >
> > Thanks for your help on this.
> >
> > Regarding the Stage4, it's strange that site hasn't updated for a
week.
> It
> > may wait until the final version, which takes about that long. I
pull
> from
> > our WPC internal site, which has Stage4 up to 18 UTC today.
> >
> > It might seem strange what I'm doing given the fields I've given
you. The
> > first (the image you sent attached) is a binomial grid of the WPC
> excessive
> > rainfall outlook. In your attached image, everywhere = 1 is under
a
> > marginal risk. The second field compares flash flood guidance to
Stage4
> > (originally in grib2) for several 1, 3, and 6 hour increments
throughout
> a
> > 24 hour period. If anywhere at any point had Stage4 exceed flash
flood
> > guidance, the grid is = 1. My next step is to apply a 40 km filter
to the
> > Stage4 exceeding flash flood guidance grid. I also need to time
match
> these
> > two datasets.
> >
> > Thanks for your help!
> >
> > Mike
> >
> >
> >
> > On Mon, Jun 19, 2017 at 7:30 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Mike,
> > >
> > > The pairs file looks *reasonable* to me, but you need to confirm
that
> the
> > > data really is in the spot it is supposed to be. Based on the
file you
> > > sent, it looks like the forecast data is a large swath of 1's
across
> the
> > > northern United States. And the observations are very speckled
across
> > the
> > > southern US and up the east coast. I ran the following
plot_data_plane
> > > command to plot the difference field from grid_stat (attached):
> > >
> > > met-6.0/bin/plot_data_plane \
> > > grid_stat_230000L_20170614_120000V_pairs.nc diff.ps \
> > > 'name="DIFF_ERO_MRGL_Surface_ST4gFFG_Surface_FULL";
level="(*,*)";' \
> > > -color_table
> > > /usr/local/met-6.0/share/met/colortables/NCL_colortables/
> BlueRed.ctable
> > >
> > > As long as you can confirm that the data is oriented correctly,
I don't
> > see
> > > any problems. I tried to grab a plot of the StageIV data for
this time
> > > from this site:
> > > ftp://ftp.emc.ncep.noaa.gov/mmb/precip/st2n4.arch/201706/
> > >
> > > But for some reason, data for 12Z on 20170614 is not yet
available.
> > >
> > > Out of curiosity, why are you producing a NetCDF version of
Stage4
> > instead
> > > of just using the GRIB2 data available publicly? Or is this
some
> > > specialized variant of it?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Mon, Jun 19, 2017 at 12:06 PM, Michael Erickson - NOAA
Affiliate via
> > RT
> > > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80907
>
> > > >
> > > > Thank you John for your helpful response!
> > > >
> > > > I've reversed the order of the latitude array and adjusted the
> gridded
> > > data
> > > > accordingly before creating the netcdf file. Thereafter, I've
> adjusted
> > my
> > > > config file as you suggested. Attached is my "*pairs.nc" file
from
> the
> > > > resulting output.
> > > >
> > > > I believe it is correct, but I wanted to make sure. Thanks
again!
> > > >
> > > > Mike
> > > >
> > > > On Mon, Jun 19, 2017 at 4:45 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Mike,
> > > > >
> > > > > Thanks for sending your sample data files. The first thing
I did
> > with
> > > > your
> > > > > data was run it through plot_data_plane. The resulting
images are
> > > > > attached:
> > > > > met-6.0/bin/plot_data_plane 20170613_ERO.nc
20170613_ERO.ps
> > > > > 'name="ERO_MRGL"; level="(*,*)";'
> > > > > met-6.0/bin/plot_data_plane 20170613_ST4gFFG.nc
> > 20170613_ST4gFFG.ps
> > > > > 'name="ST4gFFG"; level="(*,*)";'
> > > > >
> > > > > I mention this because when I compare these images to what
ncview
> > > shows,
> > > > > they are flipped vertically. So the order of the latitudes
in your
> > > file
> > > > > (which ncview uses) differ from the global attribute grid
> definition
> > > > (which
> > > > > MET uses). I'm guessing the ncview orientation is the
"correct"
> one.
> > > > >
> > > > > Next, on to your Grid-Stat error. I think you've found a
very
> > > misleading
> > > > > error message! The underlying problem is just a typo in the
name
> of
> > > the
> > > > > field. Here's your setting:
> > > > >
> > > > > poly = [
"/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
> > > > {name =
> > > > > \"ERO_MRGLs\"; level = \"Surface\";} > 0"];
> > > > >
> > > > > But the NetCDF variable is named "ERO_MRGL" *without* the
trailing
> > "s".
> > > > > When I remove the "s", it works fine. Also the threshold
should be
> > > ">0"
> > > > > without a space... but I do see that having the space "> 0"
also
> > works.
> > > > > Please try this setting:
> > > > >
> > > > > poly = [
"/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
> > > > {name =
> > > > > \"ERO_MRGL\"; level = \"Surface\";} >0"];
> > > > >
> > > > > And I'll use your data to test the current development
version. We
> > > > > obviously want a more useful error message than what you
saw.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Mon, Jun 19, 2017 at 8:49 AM, Michael Erickson - NOAA
Affiliate
> > via
> > > > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Mon Jun 19 08:49:26 2017: Request 80907 was acted upon.
> > > > > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: Grid_stat Issue When Applying Mask
> > > > > > Owner: Nobody
> > > > > > Requestors: michael.j.erickson at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=80907
> > > > >
> > > > > >
> > > > > >
> > > > > > Good morning!
> > > > > >
> > > > > > I am attempting to run grid_stat on two different netCDF
files:
> 1)
> > > > > > 20170613_ERO.nc which is a binomial field of ones and
zeros and
> 2)
> > > > > > 20170613_ST4gFFG.nc, which is another binomial field. I
created
> > these
> > > > > > netCDF files from python, so some of the metadata
description
> will
> > be
> > > > > > strange (they were not created by regrid_data_plane). When
I run
> > > > > grid_stat
> > > > > > with no masking field, everything runs fine. However, I
want to
> > mask
> > > > the
> > > > > > data to only include verification statistics inside the
region of
> > > where
> > > > > > 20170613_ERO.nc is equal to one. When I try to include
> > > 20170613_ERO.nc
> > > > > as a
> > > > > > mask, I get an error:
> > > > > >
> > > > > > *ERROR : mask_bad_data() -> grid dimensions do not match*
> > > > > >
> > > > > > My error-ing code in the grid_stat config file is:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *mask = { grid = [ "FULL" ];// poly = []; poly = [
> > > > > > "/usr1/wpc_cpgffh/gribs/temp/ERO_20170614/20170613_ERO.nc
{name
> =
> > > > > > \"ERO_MRGLs\"; level = \"Surface\";} > 0"];}*
> > > > > > I find this error strange, since all of the data in an
earlier
> step
> > > was
> > > > > > interpolated to a 0.1 X 0.1 degree lat/lon grid.
> > > > > >
> > > > > > I have also attached my config file, 20170613_ERO.nc, and
> > > > > > 20170613_ST4gFFG.nc.
> > > > > >
> > > > > > Sorry for the multiple questions over the past few weeks
and
> thank
> > > you
> > > > > for
> > > > > > all of your help!
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Michael J. Erickson, Ph.D
> > > > > > Cooperative Institute for Research in Environmental
Sciences
> > (CIRES)
> > > > > > CIRES Contractor for the Weather Prediction Center (WPC)
> > > > > >
> > > > > > Phone: 301-683-1546
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Michael J. Erickson, Ph.D
> > > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > > CIRES Contractor for the Weather Prediction Center (WPC)
> > > >
> > > > Phone: 301-683-1546
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> >
> > Michael J. Erickson, Ph.D
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > CIRES Contractor for the Weather Prediction Center (WPC)
> >
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson, Ph.D
Cooperative Institute for Research in Environmental Sciences (CIRES)
CIRES Contractor for the Weather Prediction Center (WPC)
Phone: 301-683-1546
------------------------------------------------
More information about the Met_help
mailing list