[Met_help] [rt.rap.ucar.edu #82911] History for Alternative way of Invoking 'mask_missing_flag' for MTD
John Halley Gotway via RT
met_help at ucar.edu
Wed Nov 8 14:19:06 MST 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning!
I am using MODE time-domain comparing Stave IV to model data.
Unfortunately, from what I can tell, MTD does not allow the
'mask_missing_flag=BOTH' flag like MODE. I would like to mask both model
and observation data if either one has missing data.
Is there a convenient way to invoke another MET function, or MTD, to just
mask the data with 'mask_missing_flag=BOTH?' If not, I would imagine that
gen_vx_mask could be used with some sort of union flag.
Thank you again!
Mike
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/Weather Prediction Center
Phone: 301-683-1546
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Alternative way of Invoking 'mask_missing_flag' for MTD
From: Randy Bullock
Time: Tue Nov 07 10:54:46 2017
Hello Mike -
The convolution algorithm that MTD uses assumes that there is no
missing or
"bad" data. This algorithm was designed this way entirely for the
sake of
speed. Earlier (research) versions of MTD took **FAR** too long to
run
(like, hours) because of the time taken for convolution.
Because of this, implementing the mask_missing_flag for MTD is not
something that will happen anytime soon. Sorry.
You could try fooling MTD by setting the appropriate data values to
zero,
if you'd like. That wouldn't be quite the same as having MTD handle
the
missing values directly in the convolution code, but it'd be a fairly
decent "Plan B".
Sorry I don't have a better answer for you right now.
Randy Bullock
On Tue, Nov 7, 2017 at 8:01 AM, Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Tue Nov 07 08:01:54 2017: Request 82911 was acted upon.
> Transaction: Ticket created by michael.j.erickson at noaa.gov
> Queue: met_help
> Subject: Alternative way of Invoking 'mask_missing_flag' for
MTD
> Owner: Nobody
> Requestors: michael.j.erickson at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=82911 >
>
>
> Good morning!
>
> I am using MODE time-domain comparing Stave IV to model data.
> Unfortunately, from what I can tell, MTD does not allow the
> 'mask_missing_flag=BOTH' flag like MODE. I would like to mask both
model
> and observation data if either one has missing data.
>
> Is there a convenient way to invoke another MET function, or MTD, to
just
> mask the data with 'mask_missing_flag=BOTH?' If not, I would imagine
that
> gen_vx_mask could be used with some sort of union flag.
>
> Thank you again!
>
> Mike
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: Alternative way of Invoking 'mask_missing_flag' for MTD
From: Michael Erickson - NOAA Affiliate
Time: Tue Nov 07 11:25:05 2017
Hi Randy,
Thank you for your quick response. Is it possible to use MTD to change
all
missing values to zero, or would you recommend another MET tool?
Thanks!
Mike
On Tue, Nov 7, 2017 at 5:54 PM, Randy Bullock via RT
<met_help at ucar.edu>
wrote:
> Hello Mike -
>
> The convolution algorithm that MTD uses assumes that there is no
missing or
> "bad" data. This algorithm was designed this way entirely for the
sake of
> speed. Earlier (research) versions of MTD took **FAR** too long to
run
> (like, hours) because of the time taken for convolution.
>
> Because of this, implementing the mask_missing_flag for MTD is not
> something that will happen anytime soon. Sorry.
>
> You could try fooling MTD by setting the appropriate data values to
zero,
> if you'd like. That wouldn't be quite the same as having MTD handle
the
> missing values directly in the convolution code, but it'd be a
fairly
> decent "Plan B".
>
> Sorry I don't have a better answer for you right now.
>
> Randy Bullock
>
> On Tue, Nov 7, 2017 at 8:01 AM, Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Nov 07 08:01:54 2017: Request 82911 was acted upon.
> > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > Queue: met_help
> > Subject: Alternative way of Invoking 'mask_missing_flag' for
MTD
> > Owner: Nobody
> > Requestors: michael.j.erickson at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=82911 >
> >
> >
> > Good morning!
> >
> > I am using MODE time-domain comparing Stave IV to model data.
> > Unfortunately, from what I can tell, MTD does not allow the
> > 'mask_missing_flag=BOTH' flag like MODE. I would like to mask both
model
> > and observation data if either one has missing data.
> >
> > Is there a convenient way to invoke another MET function, or MTD,
to just
> > mask the data with 'mask_missing_flag=BOTH?' If not, I would
imagine that
> > gen_vx_mask could be used with some sort of union flag.
> >
> > Thank you again!
> >
> > Mike
> >
> > --
> > Michael J. Erickson
> >
> > Research Scientist
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > NOAA/NWS/Weather Prediction Center
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
Subject: Alternative way of Invoking 'mask_missing_flag' for MTD
From: Randy Bullock
Time: Tue Nov 07 12:06:02 2017
Hi Mike -
We don't currently have a nice way to do that in MET. What you could
do,
if you wanted, is to read the data files directly, keep track of which
grid
(x, y) points always have data, and then zero out all the data in the
files
whose (x, y) coordinates are not on that list.
Randy
On Tue, Nov 7, 2017 at 11:25 AM, Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=82911 >
>
> Hi Randy,
>
> Thank you for your quick response. Is it possible to use MTD to
change all
> missing values to zero, or would you recommend another MET tool?
>
> Thanks!
>
> Mike
>
> On Tue, Nov 7, 2017 at 5:54 PM, Randy Bullock via RT
<met_help at ucar.edu>
> wrote:
>
> > Hello Mike -
> >
> > The convolution algorithm that MTD uses assumes that there is no
missing
> or
> > "bad" data. This algorithm was designed this way entirely for the
sake
> of
> > speed. Earlier (research) versions of MTD took **FAR** too long
to run
> > (like, hours) because of the time taken for convolution.
> >
> > Because of this, implementing the mask_missing_flag for MTD is not
> > something that will happen anytime soon. Sorry.
> >
> > You could try fooling MTD by setting the appropriate data values
to zero,
> > if you'd like. That wouldn't be quite the same as having MTD
handle the
> > missing values directly in the convolution code, but it'd be a
fairly
> > decent "Plan B".
> >
> > Sorry I don't have a better answer for you right now.
> >
> > Randy Bullock
> >
> > On Tue, Nov 7, 2017 at 8:01 AM, Michael Erickson - NOAA Affiliate
via RT
> <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Nov 07 08:01:54 2017: Request 82911 was acted upon.
> > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > Queue: met_help
> > > Subject: Alternative way of Invoking 'mask_missing_flag'
for MTD
> > > Owner: Nobody
> > > Requestors: michael.j.erickson at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=82911
> >
> > >
> > >
> > > Good morning!
> > >
> > > I am using MODE time-domain comparing Stave IV to model data.
> > > Unfortunately, from what I can tell, MTD does not allow the
> > > 'mask_missing_flag=BOTH' flag like MODE. I would like to mask
both
> model
> > > and observation data if either one has missing data.
> > >
> > > Is there a convenient way to invoke another MET function, or
MTD, to
> just
> > > mask the data with 'mask_missing_flag=BOTH?' If not, I would
imagine
> that
> > > gen_vx_mask could be used with some sort of union flag.
> > >
> > > Thank you again!
> > >
> > > Mike
> > >
> > > --
> > > Michael J. Erickson
> > >
> > > Research Scientist
> > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > NOAA/NWS/Weather Prediction Center
> > > Phone: 301-683-1546
> > >
> > >
> >
> >
>
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: Alternative way of Invoking 'mask_missing_flag' for MTD
From: Michael Erickson - NOAA Affiliate
Time: Tue Nov 07 13:06:20 2017
Hi Randy,
Thanks, I'll explore and may go that path.
Have a good day!
Mike
On Tue, Nov 7, 2017 at 7:06 PM, Randy Bullock via RT
<met_help at ucar.edu>
wrote:
> Hi Mike -
>
> We don't currently have a nice way to do that in MET. What you
could do,
> if you wanted, is to read the data files directly, keep track of
which grid
> (x, y) points always have data, and then zero out all the data in
the files
> whose (x, y) coordinates are not on that list.
>
> Randy
>
> On Tue, Nov 7, 2017 at 11:25 AM, Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=82911 >
> >
> > Hi Randy,
> >
> > Thank you for your quick response. Is it possible to use MTD to
change
> all
> > missing values to zero, or would you recommend another MET tool?
> >
> > Thanks!
> >
> > Mike
> >
> > On Tue, Nov 7, 2017 at 5:54 PM, Randy Bullock via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hello Mike -
> > >
> > > The convolution algorithm that MTD uses assumes that there is no
> missing
> > or
> > > "bad" data. This algorithm was designed this way entirely for
the sake
> > of
> > > speed. Earlier (research) versions of MTD took **FAR** too long
to run
> > > (like, hours) because of the time taken for convolution.
> > >
> > > Because of this, implementing the mask_missing_flag for MTD is
not
> > > something that will happen anytime soon. Sorry.
> > >
> > > You could try fooling MTD by setting the appropriate data values
to
> zero,
> > > if you'd like. That wouldn't be quite the same as having MTD
handle
> the
> > > missing values directly in the convolution code, but it'd be a
fairly
> > > decent "Plan B".
> > >
> > > Sorry I don't have a better answer for you right now.
> > >
> > > Randy Bullock
> > >
> > > On Tue, Nov 7, 2017 at 8:01 AM, Michael Erickson - NOAA
Affiliate via
> RT
> > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Nov 07 08:01:54 2017: Request 82911 was acted upon.
> > > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > > Queue: met_help
> > > > Subject: Alternative way of Invoking 'mask_missing_flag'
for MTD
> > > > Owner: Nobody
> > > > Requestors: michael.j.erickson at noaa.gov
> > > > Status: new
> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=82911
> > >
> > > >
> > > >
> > > > Good morning!
> > > >
> > > > I am using MODE time-domain comparing Stave IV to model data.
> > > > Unfortunately, from what I can tell, MTD does not allow the
> > > > 'mask_missing_flag=BOTH' flag like MODE. I would like to mask
both
> > model
> > > > and observation data if either one has missing data.
> > > >
> > > > Is there a convenient way to invoke another MET function, or
MTD, to
> > just
> > > > mask the data with 'mask_missing_flag=BOTH?' If not, I would
imagine
> > that
> > > > gen_vx_mask could be used with some sort of union flag.
> > > >
> > > > Thank you again!
> > > >
> > > > Mike
> > > >
> > > > --
> > > > Michael J. Erickson
> > > >
> > > > Research Scientist
> > > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > > NOAA/NWS/Weather Prediction Center
> > > > Phone: 301-683-1546
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Michael J. Erickson
> >
> > Research Scientist
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > NOAA/NWS/Weather Prediction Center
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
Subject: Alternative way of Invoking 'mask_missing_flag' for MTD
From: John Halley Gotway
Time: Tue Nov 07 13:17:33 2017
Randy and Mike,
Thought I'd jump in here on this ticket with some more info.
As of met-6.0, there's no *easy* way of doing this... but there is a
*difficult* way using the gen_vx_mask tool.
In the upcoming release of met-6.1, there will be an easy way of doing
this. We are adding support for some data processing logic called
"censoring". You define one or more thresholds to be applied to your
data
and one or more replacement values, like this:
censor_thresh = [ <0, >100 ];
censor_val = [ 0, 100 ];
In the example above, we're applying 2 censoring methods. Any data
values
less than 0 are replaced by a value of 0. And any data values greater
than
100 are replaced by a value of 100.
For you Mike, to change bad data to 0, you'd use:
censor_thresh = [ ==-9999 ];
censor_val = 0;
Any data values equal to -9999 (i.e. MET's internal value for bad
data) are
replaced by a value of 0.
And now, for the difficult way in met-6.0 using "data" masking. This
is
essentially the same logic as the censoring that's in met-6.1, but
you'd
run it separately for each input file to MTD. To demonstrate this, I
grabbed a recent URMA file with 6-hour APCP:
# Pull data file
wget
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/urma/prod/urma2p5.20171107/urma2p5.2017110712.pcp_06h.184.grb2
# Plot this data
met-6.0/bin/plot_data_plane urma2p5.2017110712.pcp_06h.184.grb2
urma.ps
'name="APCP"; level="A06";' -v 5
# Run gen_vx_mask
met-6.0/bin/gen_vx_mask \
urma2p5.2017110712.pcp_06h.184.grb2 \
urma2p5.2017110712.pcp_06h.184.grb2 \
urma_with_zeros.nc \
-type data \
-input_field 'name="APCP"; level="A6";' \
-mask_field 'name="APCP"; level="A6";' \
-thresh eq-9999 -value 0 -name APCP_06
# Plot the result
met-6.0/bin/plot_data_plane urma_with_zeros.nc urma_with_zeros.ps
'name="APCP_06"; level="(*,*)";'
I've attached the plots of the input/output from this process. As you
can
see, the gray bad data areas of the input are white values of 0 in the
output.
Lots of details to unpack in that call to gen_vx_mask! And that's
what
makes it difficult. If you'd like a more thorough explanation, just
let me
know.
Thanks,
John
------------------------------------------------
Subject: Alternative way of Invoking 'mask_missing_flag' for MTD
From: Michael Erickson - NOAA Affiliate
Time: Wed Nov 08 13:58:36 2017
Hi John,
Thanks for your help with this! This makes sense. I may consider a
variant
of your technique which considers the union of missing data between
the
model and observation fields. When tracking with MTD, I've noticed
that the
hourly Stage IV data has frequent missing data in the west. This can
be
problematic when considering aggregated statistics over a month, when
there
is no information regarding the total number of model runs considered
per
point on the spatial grid.
Thank you Randy and John for your help!
Mike
On Tue, Nov 7, 2017 at 8:17 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Randy and Mike,
>
> Thought I'd jump in here on this ticket with some more info.
>
> As of met-6.0, there's no *easy* way of doing this... but there is a
> *difficult* way using the gen_vx_mask tool.
>
> In the upcoming release of met-6.1, there will be an easy way of
doing
> this. We are adding support for some data processing logic called
> "censoring". You define one or more thresholds to be applied to
your data
> and one or more replacement values, like this:
>
> censor_thresh = [ <0, >100 ];
> censor_val = [ 0, 100 ];
>
> In the example above, we're applying 2 censoring methods. Any data
values
> less than 0 are replaced by a value of 0. And any data values
greater than
> 100 are replaced by a value of 100.
>
> For you Mike, to change bad data to 0, you'd use:
>
> censor_thresh = [ ==-9999 ];
> censor_val = 0;
>
> Any data values equal to -9999 (i.e. MET's internal value for bad
data) are
> replaced by a value of 0.
>
> And now, for the difficult way in met-6.0 using "data" masking.
This is
> essentially the same logic as the censoring that's in met-6.1, but
you'd
> run it separately for each input file to MTD. To demonstrate this,
I
> grabbed a recent URMA file with 6-hour APCP:
>
> # Pull data file
> wget
> ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/urma/prod/
> urma2p5.20171107/urma2p5.2017110712.pcp_06h.184.grb2
>
> # Plot this data
> met-6.0/bin/plot_data_plane urma2p5.2017110712.pcp_06h.184.grb2
urma.ps
> 'name="APCP"; level="A06";' -v 5
>
> # Run gen_vx_mask
> met-6.0/bin/gen_vx_mask \
> urma2p5.2017110712.pcp_06h.184.grb2 \
> urma2p5.2017110712.pcp_06h.184.grb2 \
> urma_with_zeros.nc \
> -type data \
> -input_field 'name="APCP"; level="A6";' \
> -mask_field 'name="APCP"; level="A6";' \
> -thresh eq-9999 -value 0 -name APCP_06
>
> # Plot the result
> met-6.0/bin/plot_data_plane urma_with_zeros.nc urma_with_zeros.ps
> 'name="APCP_06"; level="(*,*)";'
>
> I've attached the plots of the input/output from this process. As
you can
> see, the gray bad data areas of the input are white values of 0 in
the
> output.
>
> Lots of details to unpack in that call to gen_vx_mask! And that's
what
> makes it difficult. If you'd like a more thorough explanation, just
let me
> know.
>
> Thanks,
> John
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
More information about the Met_help
mailing list