[Met_help] [rt.rap.ucar.edu #88833] History for mask_missing_flag Option in MTD
John Halley Gotway via RT
met_help at ucar.edu
Wed Feb 13 10:27:25 MST 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello again,
I am running MTD for a model over a limited domain (NEWSe) and the Stage IV
analysis over CONUS. For other functions, I noticed this can be done by:
mask_missing_flag = BOTH;
I see this is possible for MODE after obs = *** is specified in the config
file, but this does not seem to work for MTD. I have attached an image of
the raw forecast and analysis fields.
Simple analysis objects are still identified outside of the model domain. I
was hoping to identify objects where both model and analysis data exists to
save time and efficiency. Is there a way to do this with MTD or with
another function?
Thanks,
Mike
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/NCEP/Weather Prediction Center
Phone: 301-683-1546
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: mask_missing_flag Option in MTD
From: John Halley Gotway
Time: Fri Feb 08 11:48:59 2019
Mike,
Yes, you're correct. The *mask_missing_flag* configuration option is
supported in the MODE config file, but not currently supported in the
MTD
config file. There's a handful of configuration options like that.
Thanks for sending the sample images. That really help illustrate the
point.
We actually do already have a MET development issue to add this option
in
for mtd:
* JIRA-645: Enhance MTD to support mask_missing_flag and vld_thresh*
I'll bump up the priority level on that issue and add a comment to
indicate
that you wrote asking about it. In the meantime, we'll need to use a
workaround. Fortunately, we have a couple of options.
If your forecast limited model data really lives on the domain defined
by
the tile in your picture, you could just use the "regrid" option in
the MTD
config to place all of your data on that domain:
*regrid = { to_grid = FCST; method = BUDGET; width = 2;
vld_thresh = 0.5; shape = SQUARE;*
*}*
And you can pick what regridding method you use... perhaps BUDGET
interpolation since it looks like precip?
If that doesn't work for some reason, I can explain how you could run
the
gen_vx_mask utility in a tricky way to mask out the bad data value in
observation dataset, but that solution would be more involved.
Thanks,
John
On Fri, Feb 8, 2019 at 8:58 AM Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Fri Feb 08 08:57:56 2019: Request 88833 was acted upon.
> Transaction: Ticket created by michael.j.erickson at noaa.gov
> Queue: met_help
> Subject: mask_missing_flag Option in MTD
> Owner: Nobody
> Requestors: michael.j.erickson at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833 >
>
>
> Hello again,
>
> I am running MTD for a model over a limited domain (NEWSe) and the
Stage IV
> analysis over CONUS. For other functions, I noticed this can be done
by:
>
> mask_missing_flag = BOTH;
>
> I see this is possible for MODE after obs = *** is specified in the
config
> file, but this does not seem to work for MTD. I have attached an
image of
> the raw forecast and analysis fields.
>
> Simple analysis objects are still identified outside of the model
domain. I
> was hoping to identify objects where both model and analysis data
exists to
> save time and efficiency. Is there a way to do this with MTD or with
> another function?
>
> Thanks,
>
> Mike
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/NCEP/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: mask_missing_flag Option in MTD
From: Michael Erickson - NOAA Affiliate
Time: Fri Feb 08 13:29:20 2019
Thanks John!
I appreciate your help. I had to step out of the office to renew my
CAC and
am teleworking for the remainder of today. I will work in your
solution to
my code on Monday.
Have a great weekend!
Mike
On Fri, Feb 8, 2019 at 1:49 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Mike,
>
> Yes, you're correct. The *mask_missing_flag* configuration option
is
> supported in the MODE config file, but not currently supported in
the MTD
> config file. There's a handful of configuration options like that.
>
> Thanks for sending the sample images. That really help illustrate
the
> point.
>
> We actually do already have a MET development issue to add this
option in
> for mtd:
> * JIRA-645: Enhance MTD to support mask_missing_flag and vld_thresh*
>
> I'll bump up the priority level on that issue and add a comment to
indicate
> that you wrote asking about it. In the meantime, we'll need to use
a
> workaround. Fortunately, we have a couple of options.
>
> If your forecast limited model data really lives on the domain
defined by
> the tile in your picture, you could just use the "regrid" option in
the MTD
> config to place all of your data on that domain:
>
>
>
>
>
>
>
> *regrid = { to_grid = FCST; method = BUDGET; width = 2;
> vld_thresh = 0.5; shape = SQUARE;*
> *}*
>
> And you can pick what regridding method you use... perhaps BUDGET
> interpolation since it looks like precip?
>
> If that doesn't work for some reason, I can explain how you could
run the
> gen_vx_mask utility in a tricky way to mask out the bad data value
in
> observation dataset, but that solution would be more involved.
>
> Thanks,
> John
>
>
> On Fri, Feb 8, 2019 at 8:58 AM Michael Erickson - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Feb 08 08:57:56 2019: Request 88833 was acted upon.
> > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > Queue: met_help
> > Subject: mask_missing_flag Option in MTD
> > Owner: Nobody
> > Requestors: michael.j.erickson at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833 >
> >
> >
> > Hello again,
> >
> > I am running MTD for a model over a limited domain (NEWSe) and the
Stage
> IV
> > analysis over CONUS. For other functions, I noticed this can be
done by:
> >
> > mask_missing_flag = BOTH;
> >
> > I see this is possible for MODE after obs = *** is specified in
the
> config
> > file, but this does not seem to work for MTD. I have attached an
image of
> > the raw forecast and analysis fields.
> >
> > Simple analysis objects are still identified outside of the model
> domain. I
> > was hoping to identify objects where both model and analysis data
exists
> to
> > save time and efficiency. Is there a way to do this with MTD or
with
> > another function?
> >
> > Thanks,
> >
> > Mike
> >
> > --
> > Michael J. Erickson
> >
> > Research Scientist
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > NOAA/NWS/NCEP/Weather Prediction Center
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/NCEP/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
Subject: mask_missing_flag Option in MTD
From: Michael Erickson - NOAA Affiliate
Time: Tue Feb 12 06:38:58 2019
Hi John,
Thanks for your help. Unfortunately, your suggestion didn't work. Just
to
confirm the syntax, I adjusted my code to:
regrid = {
to_grid = FCST;
vld_thresh = 0.5;
method = BUDGET;
width = 2;
shape = SQUARE;
}
I should point out that I use regrid_data_plane beforehand to
interpolate
both the limited model domain and analysis domain to the same grid.
During
the interpolation, missing data outside of the original domain looks
to be
set to fill_value = -9999.0. Perhaps there is something I can do back
at
the regrid_data_plane stage?
Thanks,
Mike
On Fri, Feb 8, 2019 at 8:29 PM Michael Erickson - NOAA Affiliate <
michael.j.erickson at noaa.gov> wrote:
> Thanks John!
>
> I appreciate your help. I had to step out of the office to renew my
CAC
> and am teleworking for the remainder of today. I will work in your
solution
> to my code on Monday.
>
> Have a great weekend!
>
> Mike
>
> On Fri, Feb 8, 2019 at 1:49 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Mike,
>>
>> Yes, you're correct. The *mask_missing_flag* configuration option
is
>> supported in the MODE config file, but not currently supported in
the MTD
>> config file. There's a handful of configuration options like that.
>>
>> Thanks for sending the sample images. That really help illustrate
the
>> point.
>>
>> We actually do already have a MET development issue to add this
option in
>> for mtd:
>> * JIRA-645: Enhance MTD to support mask_missing_flag and
vld_thresh*
>>
>> I'll bump up the priority level on that issue and add a comment to
>> indicate
>> that you wrote asking about it. In the meantime, we'll need to use
a
>> workaround. Fortunately, we have a couple of options.
>>
>> If your forecast limited model data really lives on the domain
defined by
>> the tile in your picture, you could just use the "regrid" option in
the
>> MTD
>> config to place all of your data on that domain:
>>
>>
>>
>>
>>
>>
>>
>> *regrid = { to_grid = FCST; method = BUDGET; width = 2;
>> vld_thresh = 0.5; shape = SQUARE;*
>> *}*
>>
>> And you can pick what regridding method you use... perhaps BUDGET
>> interpolation since it looks like precip?
>>
>> If that doesn't work for some reason, I can explain how you could
run the
>> gen_vx_mask utility in a tricky way to mask out the bad data value
in
>> observation dataset, but that solution would be more involved.
>>
>> Thanks,
>> John
>>
>>
>> On Fri, Feb 8, 2019 at 8:58 AM Michael Erickson - NOAA Affiliate
via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Fri Feb 08 08:57:56 2019: Request 88833 was acted upon.
>> > Transaction: Ticket created by michael.j.erickson at noaa.gov
>> > Queue: met_help
>> > Subject: mask_missing_flag Option in MTD
>> > Owner: Nobody
>> > Requestors: michael.j.erickson at noaa.gov
>> > Status: new
>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833 >
>> >
>> >
>> > Hello again,
>> >
>> > I am running MTD for a model over a limited domain (NEWSe) and
the
>> Stage IV
>> > analysis over CONUS. For other functions, I noticed this can be
done by:
>> >
>> > mask_missing_flag = BOTH;
>> >
>> > I see this is possible for MODE after obs = *** is specified in
the
>> config
>> > file, but this does not seem to work for MTD. I have attached an
image
>> of
>> > the raw forecast and analysis fields.
>> >
>> > Simple analysis objects are still identified outside of the model
>> domain. I
>> > was hoping to identify objects where both model and analysis data
>> exists to
>> > save time and efficiency. Is there a way to do this with MTD or
with
>> > another function?
>> >
>> > Thanks,
>> >
>> > Mike
>> >
>> > --
>> > Michael J. Erickson
>> >
>> > Research Scientist
>> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
>> > NOAA/NWS/NCEP/Weather Prediction Center
>> > Phone: 301-683-1546
>> >
>> >
>>
>>
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/NCEP/Weather Prediction Center
> Phone: 301-683-1546
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/NCEP/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
Subject: mask_missing_flag Option in MTD
From: John Halley Gotway
Time: Tue Feb 12 09:38:37 2019
Mike,
Ah OK, I understand the data flow better now. In the
regrid_data_plane
step, you could regrid the analysis to the grid used by the limited
model
domain. I assume that domain remains fixed and doesn't move through
time.
If you do the analysis on the limited model domain then you won't have
any
missing data values anymore.
Make sense?
FYI, if you prefer running regrid_data_plane it's fine to do so. But
I'm
just reminding you that the "regrid" section of the MET config files
does
the same thing. So you have the option of doing the regridding at
the
same time as you're running the verification tools.
Thanks,
John
On Tue, Feb 12, 2019 at 6:39 AM Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833 >
>
> Hi John,
>
> Thanks for your help. Unfortunately, your suggestion didn't work.
Just to
> confirm the syntax, I adjusted my code to:
>
> regrid = {
> to_grid = FCST;
> vld_thresh = 0.5;
> method = BUDGET;
> width = 2;
> shape = SQUARE;
> }
>
> I should point out that I use regrid_data_plane beforehand to
interpolate
> both the limited model domain and analysis domain to the same grid.
During
> the interpolation, missing data outside of the original domain looks
to be
> set to fill_value = -9999.0. Perhaps there is something I can do
back at
> the regrid_data_plane stage?
>
>
> Thanks,
>
>
> Mike
>
> On Fri, Feb 8, 2019 at 8:29 PM Michael Erickson - NOAA Affiliate <
> michael.j.erickson at noaa.gov> wrote:
>
> > Thanks John!
> >
> > I appreciate your help. I had to step out of the office to renew
my CAC
> > and am teleworking for the remainder of today. I will work in your
> solution
> > to my code on Monday.
> >
> > Have a great weekend!
> >
> > Mike
> >
> > On Fri, Feb 8, 2019 at 1:49 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Mike,
> >>
> >> Yes, you're correct. The *mask_missing_flag* configuration
option is
> >> supported in the MODE config file, but not currently supported in
the
> MTD
> >> config file. There's a handful of configuration options like
that.
> >>
> >> Thanks for sending the sample images. That really help
illustrate the
> >> point.
> >>
> >> We actually do already have a MET development issue to add this
option
> in
> >> for mtd:
> >> * JIRA-645: Enhance MTD to support mask_missing_flag and
vld_thresh*
> >>
> >> I'll bump up the priority level on that issue and add a comment
to
> >> indicate
> >> that you wrote asking about it. In the meantime, we'll need to
use a
> >> workaround. Fortunately, we have a couple of options.
> >>
> >> If your forecast limited model data really lives on the domain
defined
> by
> >> the tile in your picture, you could just use the "regrid" option
in the
> >> MTD
> >> config to place all of your data on that domain:
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> *regrid = { to_grid = FCST; method = BUDGET; width =
2;
> >> vld_thresh = 0.5; shape = SQUARE;*
> >> *}*
> >>
> >> And you can pick what regridding method you use... perhaps BUDGET
> >> interpolation since it looks like precip?
> >>
> >> If that doesn't work for some reason, I can explain how you could
run
> the
> >> gen_vx_mask utility in a tricky way to mask out the bad data
value in
> >> observation dataset, but that solution would be more involved.
> >>
> >> Thanks,
> >> John
> >>
> >>
> >> On Fri, Feb 8, 2019 at 8:58 AM Michael Erickson - NOAA Affiliate
via RT
> <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > Fri Feb 08 08:57:56 2019: Request 88833 was acted upon.
> >> > Transaction: Ticket created by michael.j.erickson at noaa.gov
> >> > Queue: met_help
> >> > Subject: mask_missing_flag Option in MTD
> >> > Owner: Nobody
> >> > Requestors: michael.j.erickson at noaa.gov
> >> > Status: new
> >> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833
> >
> >> >
> >> >
> >> > Hello again,
> >> >
> >> > I am running MTD for a model over a limited domain (NEWSe) and
the
> >> Stage IV
> >> > analysis over CONUS. For other functions, I noticed this can be
done
> by:
> >> >
> >> > mask_missing_flag = BOTH;
> >> >
> >> > I see this is possible for MODE after obs = *** is specified in
the
> >> config
> >> > file, but this does not seem to work for MTD. I have attached
an image
> >> of
> >> > the raw forecast and analysis fields.
> >> >
> >> > Simple analysis objects are still identified outside of the
model
> >> domain. I
> >> > was hoping to identify objects where both model and analysis
data
> >> exists to
> >> > save time and efficiency. Is there a way to do this with MTD or
with
> >> > another function?
> >> >
> >> > Thanks,
> >> >
> >> > Mike
> >> >
> >> > --
> >> > Michael J. Erickson
> >> >
> >> > Research Scientist
> >> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> >> > NOAA/NWS/NCEP/Weather Prediction Center
> >> > Phone: 301-683-1546
> >> >
> >> >
> >>
> >>
> >
> > --
> > Michael J. Erickson
> >
> > Research Scientist
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > NOAA/NWS/NCEP/Weather Prediction Center
> > Phone: 301-683-1546
> >
>
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/NCEP/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: mask_missing_flag Option in MTD
From: Michael Erickson - NOAA Affiliate
Time: Tue Feb 12 10:27:15 2019
Thanks John, that all makes sense. However, since I am using the
NEWSe, the
model domain will change each day the model is run, depending on where
the
weather of interest is occurring. I want to interpolate both grids to
a
CONUS domain, but also mask out all of the analysis data where there
is no
model data for that day. Is that possible?
Thanks,
Mike
On Tue, Feb 12, 2019 at 4:38 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Mike,
>
> Ah OK, I understand the data flow better now. In the
regrid_data_plane
> step, you could regrid the analysis to the grid used by the limited
model
> domain. I assume that domain remains fixed and doesn't move through
time.
>
> If you do the analysis on the limited model domain then you won't
have any
> missing data values anymore.
>
> Make sense?
>
> FYI, if you prefer running regrid_data_plane it's fine to do so.
But I'm
> just reminding you that the "regrid" section of the MET config files
does
> the same thing. So you have the option of doing the regridding at
the
> same time as you're running the verification tools.
>
> Thanks,
> John
>
> On Tue, Feb 12, 2019 at 6:39 AM Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833 >
> >
> > Hi John,
> >
> > Thanks for your help. Unfortunately, your suggestion didn't work.
Just to
> > confirm the syntax, I adjusted my code to:
> >
> > regrid = {
> > to_grid = FCST;
> > vld_thresh = 0.5;
> > method = BUDGET;
> > width = 2;
> > shape = SQUARE;
> > }
> >
> > I should point out that I use regrid_data_plane beforehand to
interpolate
> > both the limited model domain and analysis domain to the same
grid.
> During
> > the interpolation, missing data outside of the original domain
looks to
> be
> > set to fill_value = -9999.0. Perhaps there is something I can do
back at
> > the regrid_data_plane stage?
> >
> >
> > Thanks,
> >
> >
> > Mike
> >
> > On Fri, Feb 8, 2019 at 8:29 PM Michael Erickson - NOAA Affiliate <
> > michael.j.erickson at noaa.gov> wrote:
> >
> > > Thanks John!
> > >
> > > I appreciate your help. I had to step out of the office to renew
my CAC
> > > and am teleworking for the remainder of today. I will work in
your
> > solution
> > > to my code on Monday.
> > >
> > > Have a great weekend!
> > >
> > > Mike
> > >
> > > On Fri, Feb 8, 2019 at 1:49 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Mike,
> > >>
> > >> Yes, you're correct. The *mask_missing_flag* configuration
option is
> > >> supported in the MODE config file, but not currently supported
in the
> > MTD
> > >> config file. There's a handful of configuration options like
that.
> > >>
> > >> Thanks for sending the sample images. That really help
illustrate the
> > >> point.
> > >>
> > >> We actually do already have a MET development issue to add this
option
> > in
> > >> for mtd:
> > >> * JIRA-645: Enhance MTD to support mask_missing_flag and
vld_thresh*
> > >>
> > >> I'll bump up the priority level on that issue and add a comment
to
> > >> indicate
> > >> that you wrote asking about it. In the meantime, we'll need to
use a
> > >> workaround. Fortunately, we have a couple of options.
> > >>
> > >> If your forecast limited model data really lives on the domain
defined
> > by
> > >> the tile in your picture, you could just use the "regrid"
option in
> the
> > >> MTD
> > >> config to place all of your data on that domain:
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> *regrid = { to_grid = FCST; method = BUDGET; width
= 2;
> > >> vld_thresh = 0.5; shape = SQUARE;*
> > >> *}*
> > >>
> > >> And you can pick what regridding method you use... perhaps
BUDGET
> > >> interpolation since it looks like precip?
> > >>
> > >> If that doesn't work for some reason, I can explain how you
could run
> > the
> > >> gen_vx_mask utility in a tricky way to mask out the bad data
value in
> > >> observation dataset, but that solution would be more involved.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >>
> > >> On Fri, Feb 8, 2019 at 8:58 AM Michael Erickson - NOAA
Affiliate via
> RT
> > <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > Fri Feb 08 08:57:56 2019: Request 88833 was acted upon.
> > >> > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > >> > Queue: met_help
> > >> > Subject: mask_missing_flag Option in MTD
> > >> > Owner: Nobody
> > >> > Requestors: michael.j.erickson at noaa.gov
> > >> > Status: new
> > >> > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833
> > >
> > >> >
> > >> >
> > >> > Hello again,
> > >> >
> > >> > I am running MTD for a model over a limited domain (NEWSe)
and the
> > >> Stage IV
> > >> > analysis over CONUS. For other functions, I noticed this can
be done
> > by:
> > >> >
> > >> > mask_missing_flag = BOTH;
> > >> >
> > >> > I see this is possible for MODE after obs = *** is specified
in the
> > >> config
> > >> > file, but this does not seem to work for MTD. I have attached
an
> image
> > >> of
> > >> > the raw forecast and analysis fields.
> > >> >
> > >> > Simple analysis objects are still identified outside of the
model
> > >> domain. I
> > >> > was hoping to identify objects where both model and analysis
data
> > >> exists to
> > >> > save time and efficiency. Is there a way to do this with MTD
or with
> > >> > another function?
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Mike
> > >> >
> > >> > --
> > >> > Michael J. Erickson
> > >> >
> > >> > Research Scientist
> > >> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > >> > NOAA/NWS/NCEP/Weather Prediction Center
> > >> > Phone: 301-683-1546
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > Michael J. Erickson
> > >
> > > Research Scientist
> > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > NOAA/NWS/NCEP/Weather Prediction Center
> > > Phone: 301-683-1546
> > >
> >
> >
> > --
> > Michael J. Erickson
> >
> > Research Scientist
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > NOAA/NWS/NCEP/Weather Prediction Center
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/NCEP/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
Subject: mask_missing_flag Option in MTD
From: John Halley Gotway
Time: Tue Feb 12 13:09:31 2019
Mike,
Ah OK, that makes sense. I do still think you have 2 options... but
it
sounds like you'll prefer the second one.
(1) Your NEWSe model data lives on a small model domain and your
analysis
lives on CONUS.
Do not run regrid_data_plane at all. Instead, pass the raw model
output
and CONUS analysis directly to mtd.
In the MTD config file, use the "regrid" options, setting "to_grid =
FCST".
That'll regrid the analysis to your daily model domain --- whatever
the
NEWSe grid happens to be that day.
That has the advantage of being simple and the disadvantage of
potentially
making it more difficult to aggregate results across multiple days.
Since the grid changes every day, the X and Y locations on the grid
aren't
as comparable from run to run.
(2) The second option is continuing with your current workflow,
calling
regrid_data_plane to put the data on a common grid.
BUT add another processing step for the analysis data by calling
gen_vx_mask to reset any values of the analysis outside the NEWSe
domain to
missing data.
Let's say you have... newse.nc and analysis.nc are the outputs of
regrid_data_plane. And I'll just guess this is 6-hourly precip. Run
gen_vx_mask like this:
gen_vx_mask analysis.nc newse.nc analysis_mask_newse.nc \
-type data \
-input_field 'name="APCP_06"; level="(*,*)"; \
-mask_field 'name="APCP_06"; level="(*,*)'; \
-thresh eq-9999 \
-value -9999 \
-name APCP_06
This command does the following:
(1) Read grid from *analysis.nc <http://analysis.nc>* to define the
grid of
interest.
(2) Initialize the values of of the output data by setting *-
input_field
'name="APCP_06"; level="(*,*)";*
(3) From the mask file (*newse.nc <http://newse.nc>*) read the masking
data
by setting* -mask_field 'name="APCP_06"; level="(*,*)';*
(4) Do data masking (*-type data*) and anywhere the threshold (*-
thresh
eq-9999*) criteria is met, set the output value to bad data by setting
*-value
-9999*.
(5) Write the output variable name by setting *-name APCP_06*.
This should have the desired effect. Confused yet!?
Thanks,
John
On Tue, Feb 12, 2019 at 10:27 AM Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833 >
>
> Thanks John, that all makes sense. However, since I am using the
NEWSe, the
> model domain will change each day the model is run, depending on
where the
> weather of interest is occurring. I want to interpolate both grids
to a
> CONUS domain, but also mask out all of the analysis data where there
is no
> model data for that day. Is that possible?
>
> Thanks,
>
> Mike
>
> On Tue, Feb 12, 2019 at 4:38 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Mike,
> >
> > Ah OK, I understand the data flow better now. In the
regrid_data_plane
> > step, you could regrid the analysis to the grid used by the
limited model
> > domain. I assume that domain remains fixed and doesn't move
through
> time.
> >
> > If you do the analysis on the limited model domain then you won't
have
> any
> > missing data values anymore.
> >
> > Make sense?
> >
> > FYI, if you prefer running regrid_data_plane it's fine to do so.
But I'm
> > just reminding you that the "regrid" section of the MET config
files does
> > the same thing. So you have the option of doing the regridding
at the
> > same time as you're running the verification tools.
> >
> > Thanks,
> > John
> >
> > On Tue, Feb 12, 2019 at 6:39 AM Michael Erickson - NOAA Affiliate
via RT
> <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833 >
> > >
> > > Hi John,
> > >
> > > Thanks for your help. Unfortunately, your suggestion didn't
work. Just
> to
> > > confirm the syntax, I adjusted my code to:
> > >
> > > regrid = {
> > > to_grid = FCST;
> > > vld_thresh = 0.5;
> > > method = BUDGET;
> > > width = 2;
> > > shape = SQUARE;
> > > }
> > >
> > > I should point out that I use regrid_data_plane beforehand to
> interpolate
> > > both the limited model domain and analysis domain to the same
grid.
> > During
> > > the interpolation, missing data outside of the original domain
looks to
> > be
> > > set to fill_value = -9999.0. Perhaps there is something I can do
back
> at
> > > the regrid_data_plane stage?
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Mike
> > >
> > > On Fri, Feb 8, 2019 at 8:29 PM Michael Erickson - NOAA Affiliate
<
> > > michael.j.erickson at noaa.gov> wrote:
> > >
> > > > Thanks John!
> > > >
> > > > I appreciate your help. I had to step out of the office to
renew my
> CAC
> > > > and am teleworking for the remainder of today. I will work in
your
> > > solution
> > > > to my code on Monday.
> > > >
> > > > Have a great weekend!
> > > >
> > > > Mike
> > > >
> > > > On Fri, Feb 8, 2019 at 1:49 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >> Mike,
> > > >>
> > > >> Yes, you're correct. The *mask_missing_flag* configuration
option
> is
> > > >> supported in the MODE config file, but not currently
supported in
> the
> > > MTD
> > > >> config file. There's a handful of configuration options like
that.
> > > >>
> > > >> Thanks for sending the sample images. That really help
illustrate
> the
> > > >> point.
> > > >>
> > > >> We actually do already have a MET development issue to add
this
> option
> > > in
> > > >> for mtd:
> > > >> * JIRA-645: Enhance MTD to support mask_missing_flag and
vld_thresh*
> > > >>
> > > >> I'll bump up the priority level on that issue and add a
comment to
> > > >> indicate
> > > >> that you wrote asking about it. In the meantime, we'll need
to use
> a
> > > >> workaround. Fortunately, we have a couple of options.
> > > >>
> > > >> If your forecast limited model data really lives on the
domain
> defined
> > > by
> > > >> the tile in your picture, you could just use the "regrid"
option in
> > the
> > > >> MTD
> > > >> config to place all of your data on that domain:
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> *regrid = { to_grid = FCST; method = BUDGET; width
= 2;
> > > >> vld_thresh = 0.5; shape = SQUARE;*
> > > >> *}*
> > > >>
> > > >> And you can pick what regridding method you use... perhaps
BUDGET
> > > >> interpolation since it looks like precip?
> > > >>
> > > >> If that doesn't work for some reason, I can explain how you
could
> run
> > > the
> > > >> gen_vx_mask utility in a tricky way to mask out the bad data
value
> in
> > > >> observation dataset, but that solution would be more
involved.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >>
> > > >> On Fri, Feb 8, 2019 at 8:58 AM Michael Erickson - NOAA
Affiliate via
> > RT
> > > <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > Fri Feb 08 08:57:56 2019: Request 88833 was acted upon.
> > > >> > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > >> > Queue: met_help
> > > >> > Subject: mask_missing_flag Option in MTD
> > > >> > Owner: Nobody
> > > >> > Requestors: michael.j.erickson at noaa.gov
> > > >> > Status: new
> > > >> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833
> > > >
> > > >> >
> > > >> >
> > > >> > Hello again,
> > > >> >
> > > >> > I am running MTD for a model over a limited domain (NEWSe)
and the
> > > >> Stage IV
> > > >> > analysis over CONUS. For other functions, I noticed this
can be
> done
> > > by:
> > > >> >
> > > >> > mask_missing_flag = BOTH;
> > > >> >
> > > >> > I see this is possible for MODE after obs = *** is
specified in
> the
> > > >> config
> > > >> > file, but this does not seem to work for MTD. I have
attached an
> > image
> > > >> of
> > > >> > the raw forecast and analysis fields.
> > > >> >
> > > >> > Simple analysis objects are still identified outside of the
model
> > > >> domain. I
> > > >> > was hoping to identify objects where both model and
analysis data
> > > >> exists to
> > > >> > save time and efficiency. Is there a way to do this with
MTD or
> with
> > > >> > another function?
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Mike
> > > >> >
> > > >> > --
> > > >> > Michael J. Erickson
> > > >> >
> > > >> > Research Scientist
> > > >> > Cooperative Institute for Research in Environmental
Sciences
> (CIRES)
> > > >> > NOAA/NWS/NCEP/Weather Prediction Center
> > > >> > Phone: 301-683-1546
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > > --
> > > > Michael J. Erickson
> > > >
> > > > Research Scientist
> > > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > > NOAA/NWS/NCEP/Weather Prediction Center
> > > > Phone: 301-683-1546
> > > >
> > >
> > >
> > > --
> > > Michael J. Erickson
> > >
> > > Research Scientist
> > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > NOAA/NWS/NCEP/Weather Prediction Center
> > > Phone: 301-683-1546
> > >
> > >
> >
> >
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/NCEP/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: mask_missing_flag Option in MTD
From: Michael Erickson - NOAA Affiliate
Time: Wed Feb 13 08:48:33 2019
Thank you John!
That makes sense. Option 2 may be the best given my current workflow
design. My precipitation interval is either going to be 15 minutes or
1
hour with the NEWSe. I submitted another ticket on pcp_combine and
this
issue. Sorry for all of the tickets!
I will test this code out shortly.
Thanks,
Mike
On Tue, Feb 12, 2019 at 8:09 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Mike,
>
> Ah OK, that makes sense. I do still think you have 2 options... but
it
> sounds like you'll prefer the second one.
>
> (1) Your NEWSe model data lives on a small model domain and your
analysis
> lives on CONUS.
> Do not run regrid_data_plane at all. Instead, pass the raw model
output
> and CONUS analysis directly to mtd.
> In the MTD config file, use the "regrid" options, setting "to_grid =
FCST".
> That'll regrid the analysis to your daily model domain --- whatever
the
> NEWSe grid happens to be that day.
>
> That has the advantage of being simple and the disadvantage of
potentially
> making it more difficult to aggregate results across multiple days.
> Since the grid changes every day, the X and Y locations on the grid
aren't
> as comparable from run to run.
>
> (2) The second option is continuing with your current workflow,
calling
> regrid_data_plane to put the data on a common grid.
> BUT add another processing step for the analysis data by calling
> gen_vx_mask to reset any values of the analysis outside the NEWSe
domain to
> missing data.
> Let's say you have... newse.nc and analysis.nc are the outputs of
> regrid_data_plane. And I'll just guess this is 6-hourly precip.
Run
> gen_vx_mask like this:
>
> gen_vx_mask analysis.nc newse.nc analysis_mask_newse.nc \
> -type data \
> -input_field 'name="APCP_06"; level="(*,*)"; \
> -mask_field 'name="APCP_06"; level="(*,*)'; \
> -thresh eq-9999 \
> -value -9999 \
> -name APCP_06
>
> This command does the following:
>
> (1) Read grid from *analysis.nc <http://analysis.nc>* to define the
grid
> of
> interest.
> (2) Initialize the values of of the output data by setting *-
input_field
> 'name="APCP_06"; level="(*,*)";*
> (3) From the mask file (*newse.nc <http://newse.nc>*) read the
masking
> data
> by setting* -mask_field 'name="APCP_06"; level="(*,*)';*
> (4) Do data masking (*-type data*) and anywhere the threshold (*-
thresh
> eq-9999*) criteria is met, set the output value to bad data by
setting
> *-value
> -9999*.
> (5) Write the output variable name by setting *-name APCP_06*.
>
> This should have the desired effect. Confused yet!?
>
> Thanks,
> John
>
> On Tue, Feb 12, 2019 at 10:27 AM Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833 >
> >
> > Thanks John, that all makes sense. However, since I am using the
NEWSe,
> the
> > model domain will change each day the model is run, depending on
where
> the
> > weather of interest is occurring. I want to interpolate both grids
to a
> > CONUS domain, but also mask out all of the analysis data where
there is
> no
> > model data for that day. Is that possible?
> >
> > Thanks,
> >
> > Mike
> >
> > On Tue, Feb 12, 2019 at 4:38 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Mike,
> > >
> > > Ah OK, I understand the data flow better now. In the
regrid_data_plane
> > > step, you could regrid the analysis to the grid used by the
limited
> model
> > > domain. I assume that domain remains fixed and doesn't move
through
> > time.
> > >
> > > If you do the analysis on the limited model domain then you
won't have
> > any
> > > missing data values anymore.
> > >
> > > Make sense?
> > >
> > > FYI, if you prefer running regrid_data_plane it's fine to do so.
But
> I'm
> > > just reminding you that the "regrid" section of the MET config
files
> does
> > > the same thing. So you have the option of doing the regridding
at the
> > > same time as you're running the verification tools.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Feb 12, 2019 at 6:39 AM Michael Erickson - NOAA
Affiliate via
> RT
> > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833
>
> > > >
> > > > Hi John,
> > > >
> > > > Thanks for your help. Unfortunately, your suggestion didn't
work.
> Just
> > to
> > > > confirm the syntax, I adjusted my code to:
> > > >
> > > > regrid = {
> > > > to_grid = FCST;
> > > > vld_thresh = 0.5;
> > > > method = BUDGET;
> > > > width = 2;
> > > > shape = SQUARE;
> > > > }
> > > >
> > > > I should point out that I use regrid_data_plane beforehand to
> > interpolate
> > > > both the limited model domain and analysis domain to the same
grid.
> > > During
> > > > the interpolation, missing data outside of the original domain
looks
> to
> > > be
> > > > set to fill_value = -9999.0. Perhaps there is something I can
do back
> > at
> > > > the regrid_data_plane stage?
> > > >
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > Mike
> > > >
> > > > On Fri, Feb 8, 2019 at 8:29 PM Michael Erickson - NOAA
Affiliate <
> > > > michael.j.erickson at noaa.gov> wrote:
> > > >
> > > > > Thanks John!
> > > > >
> > > > > I appreciate your help. I had to step out of the office to
renew my
> > CAC
> > > > > and am teleworking for the remainder of today. I will work
in your
> > > > solution
> > > > > to my code on Monday.
> > > > >
> > > > > Have a great weekend!
> > > > >
> > > > > Mike
> > > > >
> > > > > On Fri, Feb 8, 2019 at 1:49 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >> Mike,
> > > > >>
> > > > >> Yes, you're correct. The *mask_missing_flag* configuration
option
> > is
> > > > >> supported in the MODE config file, but not currently
supported in
> > the
> > > > MTD
> > > > >> config file. There's a handful of configuration options
like
> that.
> > > > >>
> > > > >> Thanks for sending the sample images. That really help
illustrate
> > the
> > > > >> point.
> > > > >>
> > > > >> We actually do already have a MET development issue to add
this
> > option
> > > > in
> > > > >> for mtd:
> > > > >> * JIRA-645: Enhance MTD to support mask_missing_flag and
> vld_thresh*
> > > > >>
> > > > >> I'll bump up the priority level on that issue and add a
comment to
> > > > >> indicate
> > > > >> that you wrote asking about it. In the meantime, we'll
need to
> use
> > a
> > > > >> workaround. Fortunately, we have a couple of options.
> > > > >>
> > > > >> If your forecast limited model data really lives on the
domain
> > defined
> > > > by
> > > > >> the tile in your picture, you could just use the "regrid"
option
> in
> > > the
> > > > >> MTD
> > > > >> config to place all of your data on that domain:
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> *regrid = { to_grid = FCST; method = BUDGET; width
=
> 2;
> > > > >> vld_thresh = 0.5; shape = SQUARE;*
> > > > >> *}*
> > > > >>
> > > > >> And you can pick what regridding method you use... perhaps
BUDGET
> > > > >> interpolation since it looks like precip?
> > > > >>
> > > > >> If that doesn't work for some reason, I can explain how you
could
> > run
> > > > the
> > > > >> gen_vx_mask utility in a tricky way to mask out the bad
data value
> > in
> > > > >> observation dataset, but that solution would be more
involved.
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >>
> > > > >> On Fri, Feb 8, 2019 at 8:58 AM Michael Erickson - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >> >
> > > > >> > Fri Feb 08 08:57:56 2019: Request 88833 was acted upon.
> > > > >> > Transaction: Ticket created by
michael.j.erickson at noaa.gov
> > > > >> > Queue: met_help
> > > > >> > Subject: mask_missing_flag Option in MTD
> > > > >> > Owner: Nobody
> > > > >> > Requestors: michael.j.erickson at noaa.gov
> > > > >> > Status: new
> > > > >> > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88833
> > > > >
> > > > >> >
> > > > >> >
> > > > >> > Hello again,
> > > > >> >
> > > > >> > I am running MTD for a model over a limited domain
(NEWSe) and
> the
> > > > >> Stage IV
> > > > >> > analysis over CONUS. For other functions, I noticed this
can be
> > done
> > > > by:
> > > > >> >
> > > > >> > mask_missing_flag = BOTH;
> > > > >> >
> > > > >> > I see this is possible for MODE after obs = *** is
specified in
> > the
> > > > >> config
> > > > >> > file, but this does not seem to work for MTD. I have
attached an
> > > image
> > > > >> of
> > > > >> > the raw forecast and analysis fields.
> > > > >> >
> > > > >> > Simple analysis objects are still identified outside of
the
> model
> > > > >> domain. I
> > > > >> > was hoping to identify objects where both model and
analysis
> data
> > > > >> exists to
> > > > >> > save time and efficiency. Is there a way to do this with
MTD or
> > with
> > > > >> > another function?
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Mike
> > > > >> >
> > > > >> > --
> > > > >> > Michael J. Erickson
> > > > >> >
> > > > >> > Research Scientist
> > > > >> > Cooperative Institute for Research in Environmental
Sciences
> > (CIRES)
> > > > >> > NOAA/NWS/NCEP/Weather Prediction Center
> > > > >> > Phone: 301-683-1546
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > > Michael J. Erickson
> > > > >
> > > > > Research Scientist
> > > > > Cooperative Institute for Research in Environmental Sciences
> (CIRES)
> > > > > NOAA/NWS/NCEP/Weather Prediction Center
> > > > > Phone: 301-683-1546
> > > > >
> > > >
> > > >
> > > > --
> > > > Michael J. Erickson
> > > >
> > > > Research Scientist
> > > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > > NOAA/NWS/NCEP/Weather Prediction Center
> > > > Phone: 301-683-1546
> > > >
> > > >
> > >
> > >
> >
> > --
> > Michael J. Erickson
> >
> > Research Scientist
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > NOAA/NWS/NCEP/Weather Prediction Center
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/NCEP/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
More information about the Met_help
mailing list