[Met_help] [rt.rap.ucar.edu #96472] History for Request assistance with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Thu Aug 27 14:11:58 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
I am trying to use Gen_Vx_Mask to format radar observations output by the
RADX software to replace missing values (-9999) where there is no
reflectivity with zeros in order to ensure that the entire domain contains
valid data for use when running Grid-Stat to generate FSS for a range of
neighborhood sizes and threshold values. The attached text file shows the
command I am using to format the CM (Column Max Reflectivity) 2-D variable
and the resulting screen dialog. The attached NetCDF file is the output
file. The input NetCDF radar ob is too large (83mb) to send by email, but I
can send a link where you could download the file from a file exchange. I
have used a similar command to format radar obs containing the CZ variable
(3-D) with success. I suspect that there is something in the way I am
specifying the information for the CM variable is not working, but I can't
figure out what to try next.
R/
John
Mr. John W. Raby
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
Office:(575) 678-2004 DSN 258-2004
Teleworking: Contact Admin Specialist at (410) 278-6974
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Request assistance with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
From: Minna Win
Time: Thu Aug 27 11:36:06 2020
Hello John,
It looks like the Gen-Vx mask command you are using doesn't appear to
do
what you intended. I took a look at the command you used and it looks
like
it is correct. One thing I noticed from your netCDF file is that the
fill
value CM:_FillValue: -9999.f rather than the -9999 that I'm accustomed
to.
I'm not sure what to try next, I'm going to assign this ticket to John
Halley Gotway.
Regards,
Minna
--------------
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax: 303-497-8401
On Thu, Aug 27, 2020 at 9:26 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Thu Aug 27 09:25:37 2020: Request 96472 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Request assistance with MET V9.0 Gen_Vx_Mask
(UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I am trying to use Gen_Vx_Mask to format radar observations output
by the
> RADX software to replace missing values (-9999) where there is no
> reflectivity with zeros in order to ensure that the entire domain
contains
> valid data for use when running Grid-Stat to generate FSS for a
range of
> neighborhood sizes and threshold values. The attached text file
shows the
> command I am using to format the CM (Column Max Reflectivity) 2-D
variable
> and the resulting screen dialog. The attached NetCDF file is the
output
> file. The input NetCDF radar ob is too large (83mb) to send by
email, but I
> can send a link where you could download the file from a file
exchange. I
> have used a similar command to format radar obs containing the CZ
variable
> (3-D) with success. I suspect that there is something in the way I
am
> specifying the information for the CM variable is not working, but I
can't
> figure out what to try next.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> Office:(575) 678-2004 DSN 258-2004
> Teleworking: Contact Admin Specialist at (410) 278-6974
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: Request assistance with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Aug 27 12:18:48 2020
Hi John,
I see that you'd like to modify some data by replacing any bad data
values
of -9999 with a value of 0. This sort of data replacement can be done
in
MET using the censor_thresh and censor_val options.
By way of example, here are 2 plot_data_plane commands:
plot_data_plane ncf_20200315_055801.nc CM_with_bad_data.ps 'name="CM";
level="(*,*)";'
plot_data_plane ncf_20200315_055801.nc CM_censor_with_0.ps 'name="CM";
level="(*,*)"; censor_thresh=[eq-9999]; censor_val=[0];'
But I found that this doesn't actually work as expected! So why not...
Running ncdump, I see that the CM values are actually encoded as
'NaNf':
ncdump -v CM ncf_20200315_055801.nc
CM =
NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
NaNf...
And we don't really have a good way of matching the value NaN with
thresholds in MET.
So take a step back and look at how you generated that NetCDF file.
The NaN
values are causing problems.
John
On Thu, Aug 27, 2020 at 11:36 AM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472 >
>
> Hello John,
>
> It looks like the Gen-Vx mask command you are using doesn't appear
to do
> what you intended. I took a look at the command you used and it
looks like
> it is correct. One thing I noticed from your netCDF file is that
the fill
> value CM:_FillValue: -9999.f rather than the -9999 that I'm
accustomed to.
> I'm not sure what to try next, I'm going to assign this ticket to
John
> Halley Gotway.
>
> Regards,
> Minna
>
> --------------
> Minna Win
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax: 303-497-8401
>
>
>
> On Thu, Aug 27, 2020 at 9:26 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Aug 27 09:25:37 2020: Request 96472 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: Request assistance with MET V9.0 Gen_Vx_Mask
(UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I am trying to use Gen_Vx_Mask to format radar observations output
by the
> > RADX software to replace missing values (-9999) where there is no
> > reflectivity with zeros in order to ensure that the entire domain
> contains
> > valid data for use when running Grid-Stat to generate FSS for a
range of
> > neighborhood sizes and threshold values. The attached text file
shows the
> > command I am using to format the CM (Column Max Reflectivity) 2-D
> variable
> > and the resulting screen dialog. The attached NetCDF file is the
output
> > file. The input NetCDF radar ob is too large (83mb) to send by
email,
> but I
> > can send a link where you could download the file from a file
exchange. I
> > have used a similar command to format radar obs containing the CZ
> variable
> > (3-D) with success. I suspect that there is something in the way I
am
> > specifying the information for the CM variable is not working, but
I
> can't
> > figure out what to try next.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > Office:(575) 678-2004 DSN 258-2004
> > Teleworking: Contact Admin Specialist at (410) 278-6974
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
------------------------------------------------
Subject: Request assistance with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Aug 27 12:50:42 2020
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for your suggestion and diagnosis. Huaqing is using RADX to
generate
the NetCDF files and then uses Matlab to post-process the files to add
in the
attributes which you noted a week or so ago were missing. It appears
that
MatLab is putting the "NaNf" values in the file. He is working on
getting
MatLab to insert 0s instead of "NaNf". Judging from that NetCDF, if we
were to
figure out how to change the "NaNf" to 0s, would Grid-Stat be able to
run OK
or do you see any other obvious issues?
Thanks for the tip on using censor_thresh and censor_val. I can just
use that
as an argument when running Grid-Stat without having to use
Gen_Vx_Mask. I can
still use Gen_Vx_Mask the same way I've been using it for this
purpose,
correct?
If I use censor_thresh and censor_val as you specify, does that
actually
modify the data in the observation file or just change the data being
used
temporarily for computations by the MET tool?
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, August 27, 2020 12:19 PM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96472] Request
assistance with
MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
Hi John,
I see that you'd like to modify some data by replacing any bad data
values
of -9999 with a value of 0. This sort of data replacement can be done
in MET
using the censor_thresh and censor_val options.
By way of example, here are 2 plot_data_plane commands:
plot_data_plane ncf_20200315_055801.nc CM_with_bad_data.ps 'name="CM";
level="(*,*)";'
plot_data_plane ncf_20200315_055801.nc CM_censor_with_0.ps 'name="CM";
level="(*,*)"; censor_thresh=[eq-9999]; censor_val=[0];'
But I found that this doesn't actually work as expected! So why not...
Running ncdump, I see that the CM values are actually encoded as
'NaNf':
ncdump -v CM ncf_20200315_055801.nc
CM =
NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
NaNf...
And we don't really have a good way of matching the value NaN with
thresholds
in MET.
So take a step back and look at how you generated that NetCDF file.
The NaN
values are causing problems.
John
On Thu, Aug 27, 2020 at 11:36 AM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472 >
>
> Hello John,
>
> It looks like the Gen-Vx mask command you are using doesn't appear
to
> do what you intended. I took a look at the command you used and it
looks
> like
> it is correct. One thing I noticed from your netCDF file is that
the fill
> value CM:_FillValue: -9999.f rather than the -9999 that I'm
accustomed to.
> I'm not sure what to try next, I'm going to assign this ticket to
John
> Halley Gotway.
>
> Regards,
> Minna
>
> --------------
> Minna Win
> National Center for Atmospheric Research Developmental Testbed
Center
> Phone: 303-497-8423
> Fax: 303-497-8401
>
>
>
> On Thu, Aug 27, 2020 at 9:26 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Aug 27 09:25:37 2020: Request 96472 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: Request assistance with MET V9.0 Gen_Vx_Mask
(UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I am trying to use Gen_Vx_Mask to format radar observations output
by the
> > RADX software to replace missing values (-9999) where there is no
> > reflectivity with zeros in order to ensure that the entire domain
> contains
> > valid data for use when running Grid-Stat to generate FSS for a
range of
> > neighborhood sizes and threshold values. The attached text file
shows the
> > command I am using to format the CM (Column Max Reflectivity) 2-D
> variable
> > and the resulting screen dialog. The attached NetCDF file is the
output
> > file. The input NetCDF radar ob is too large (83mb) to send by
email,
> but I
> > can send a link where you could download the file from a file
exchange. I
> > have used a similar command to format radar obs containing the CZ
> variable
> > (3-D) with success. I suspect that there is something in the way I
am
> > specifying the information for the CM variable is not working, but
I
> can't
> > figure out what to try next.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > Office:(575) 678-2004 DSN 258-2004
> > Teleworking: Contact Admin Specialist at (410) 278-6974
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: Request assistance with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Aug 27 13:05:51 2020
John,
Below, I've listed an excerpt from this page of the user's guide:
https://dtcenter.github.io/MET/Users_Guide/data_io.html
The way we have the user's guide structured, I can't provide you a
direct
link to the think info. But we're working on fixing that up for the
next
release... moving the data/config/README file into the documentation
area.
Anyway, this is used as follows:
(1) The MET library code reads the input data.
(2) If a "convert(x)" function is defined, it is applied.
(3) If "censor_thresh" and "censor_val" options are defined, they are
applied.
(4) Then the application processes the converted and/or censored data
values.
It's up to you how you'd like to handle this.
You could run gen_vx_mask or pcp_combine up front to censor your data
in
this way. Then you'd do it once and be done with it.
But you're doubling your data files... one with uncensored data and
another
with censored data.
Or you could specify the censoring logic wherever you need it. In the
config files for Grid-Stat, Point-Stat, MODE... whatever.
So, up to you.
John
// - The "censor_thresh" entry is an array of thresholds to be
applied
// to the input data. The "censor_val" entry is an array of
numbers
// and must be the same length as "censor_thresh". These
arguments must
// appear together in the correct format (threshold and
number). For each
// censor threshold, any input values meeting the threshold
criteria will
// be reset to the corresponding censor value. An empty list
indicates
// that no censoring should be performed. The censoring logic
is applied
// prior to any regridding but after the convert function. All
statistics
// are computed on the censored data. These entries may be used
to apply
// quality control logic by resetting data outside of an
expected range
// to the bad data value of -9999. These entries are not
indicated in the
// metadata of any output files, but the user can set the
"desc" entry
// accordingly.
On Thu, Aug 27, 2020 at 12:51 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for your suggestion and diagnosis. Huaqing is using RADX to
> generate
> the NetCDF files and then uses Matlab to post-process the files to
add in
> the
> attributes which you noted a week or so ago were missing. It appears
that
> MatLab is putting the "NaNf" values in the file. He is working on
getting
> MatLab to insert 0s instead of "NaNf". Judging from that NetCDF, if
we
> were to
> figure out how to change the "NaNf" to 0s, would Grid-Stat be able
to run
> OK
> or do you see any other obvious issues?
>
> Thanks for the tip on using censor_thresh and censor_val. I can just
use
> that
> as an argument when running Grid-Stat without having to use
Gen_Vx_Mask. I
> can
> still use Gen_Vx_Mask the same way I've been using it for this
purpose,
> correct?
>
> If I use censor_thresh and censor_val as you specify, does that
actually
> modify the data in the observation file or just change the data
being used
> temporarily for computations by the MET tool?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, August 27, 2020 12:19 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96472] Request
assistance
> with
> MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Hi John,
>
> I see that you'd like to modify some data by replacing any bad data
values
> of -9999 with a value of 0. This sort of data replacement can be
done in
> MET
> using the censor_thresh and censor_val options.
>
> By way of example, here are 2 plot_data_plane commands:
>
> plot_data_plane ncf_20200315_055801.nc CM_with_bad_data.ps
'name="CM";
> level="(*,*)";'
>
> plot_data_plane ncf_20200315_055801.nc CM_censor_with_0.ps
'name="CM";
> level="(*,*)"; censor_thresh=[eq-9999]; censor_val=[0];'
>
>
> But I found that this doesn't actually work as expected! So why
not...
>
>
> Running ncdump, I see that the CM values are actually encoded as
'NaNf':
>
>
> ncdump -v CM ncf_20200315_055801.nc
>
>
> CM =
>
> NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
>
> NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf, NaNf,
>
> NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf, NaNf,
>
> NaNf...
>
>
> And we don't really have a good way of matching the value NaN with
> thresholds
> in MET.
>
>
> So take a step back and look at how you generated that NetCDF file.
The
> NaN
> values are causing problems.
>
>
> John
>
> On Thu, Aug 27, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472 >
> >
> > Hello John,
> >
> > It looks like the Gen-Vx mask command you are using doesn't appear
to
> > do what you intended. I took a look at the command you used and
it
> looks
> > like
> > it is correct. One thing I noticed from your netCDF file is that
the
> fill
> > value CM:_FillValue: -9999.f rather than the -9999 that I'm
accustomed
> to.
> > I'm not sure what to try next, I'm going to assign this ticket to
John
> > Halley Gotway.
> >
> > Regards,
> > Minna
> >
> > --------------
> > Minna Win
> > National Center for Atmospheric Research Developmental Testbed
Center
> > Phone: 303-497-8423
> > Fax: 303-497-8401
> >
> >
> >
> > On Thu, Aug 27, 2020 at 9:26 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Aug 27 09:25:37 2020: Request 96472 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: Request assistance with MET V9.0 Gen_Vx_Mask
> (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-url:
> > > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472
>
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I am trying to use Gen_Vx_Mask to format radar observations
output by
> the
> > > RADX software to replace missing values (-9999) where there is
no
> > > reflectivity with zeros in order to ensure that the entire
domain
> > contains
> > > valid data for use when running Grid-Stat to generate FSS for a
range
> of
> > > neighborhood sizes and threshold values. The attached text file
shows
> the
> > > command I am using to format the CM (Column Max Reflectivity) 2-
D
> > variable
> > > and the resulting screen dialog. The attached NetCDF file is the
output
> > > file. The input NetCDF radar ob is too large (83mb) to send by
email,
> > but I
> > > can send a link where you could download the file from a file
> exchange. I
> > > have used a similar command to format radar obs containing the
CZ
> > variable
> > > (3-D) with success. I suspect that there is something in the way
I am
> > > specifying the information for the CM variable is not working,
but I
> > can't
> > > figure out what to try next.
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > Office:(575) 678-2004 DSN 258-2004
> > > Teleworking: Contact Admin Specialist at (410) 278-6974
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: Request assistance with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Aug 27 13:36:43 2020
CLASSIFICATION: UNCLASSIFIED
Thanks for explaining all the options I have. Good info to know. I'm
used to
running Gen_Vx_Mask up front to create the input data which I know has
been
"formatted" so it will run in Grid-Stat, so I'm used to maintaining
double
sets of obs. My inclination would be to continue running Gen_Vx_Mask,
but
using the "censor_thresh" and "censor_val" options instead of the
option
(-mask_field) I was using before.
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, August 27, 2020 1:06 PM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96472] Request
assistance
with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Below, I've listed an excerpt from this page of the user's guide:
Caution-https://dtcenter.github.io/MET/Users_Guide/data_io.html
The way we have the user's guide structured, I can't provide you a
direct link
to the think info. But we're working on fixing that up for the next
release...
moving the data/config/README file into the documentation area.
Anyway, this is used as follows:
(1) The MET library code reads the input data.
(2) If a "convert(x)" function is defined, it is applied.
(3) If "censor_thresh" and "censor_val" options are defined, they are
applied.
(4) Then the application processes the converted and/or censored data
values.
It's up to you how you'd like to handle this.
You could run gen_vx_mask or pcp_combine up front to censor your data
in this
way. Then you'd do it once and be done with it.
But you're doubling your data files... one with uncensored data and
another
with censored data.
Or you could specify the censoring logic wherever you need it. In the
config
files for Grid-Stat, Point-Stat, MODE... whatever.
So, up to you.
John
// - The "censor_thresh" entry is an array of thresholds to be
applied
// to the input data. The "censor_val" entry is an array of
numbers
// and must be the same length as "censor_thresh". These
arguments must
// appear together in the correct format (threshold and
number). For
each
// censor threshold, any input values meeting the threshold
criteria
will
// be reset to the corresponding censor value. An empty list
indicates
// that no censoring should be performed. The censoring logic
is
applied
// prior to any regridding but after the convert function. All
statistics
// are computed on the censored data. These entries may be used
to
apply
// quality control logic by resetting data outside of an
expected range
// to the bad data value of -9999. These entries are not
indicated in
the
// metadata of any output files, but the user can set the
"desc" entry
// accordingly.
On Thu, Aug 27, 2020 at 12:51 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for your suggestion and diagnosis. Huaqing is using RADX to
> generate the NetCDF files and then uses Matlab to post-process the
> files to add in the attributes which you noted a week or so ago were
> missing. It appears that MatLab is putting the "NaNf" values in the
> file. He is working on getting MatLab to insert 0s instead of
"NaNf".
> Judging from that NetCDF, if we were to figure out how to change the
> "NaNf" to 0s, would Grid-Stat be able to run OK or do you see any
> other obvious issues?
>
> Thanks for the tip on using censor_thresh and censor_val. I can just
> use that as an argument when running Grid-Stat without having to use
> Gen_Vx_Mask. I can still use Gen_Vx_Mask the same way I've been
using
> it for this purpose, correct?
>
> If I use censor_thresh and censor_val as you specify, does that
> actually modify the data in the observation file or just change the
> data being used temporarily for computations by the MET tool?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, August 27, 2020 12:19 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96472] Request
> assistance with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> Hi John,
>
> I see that you'd like to modify some data by replacing any bad data
> values of -9999 with a value of 0. This sort of data replacement can
> be done in MET using the censor_thresh and censor_val options.
>
> By way of example, here are 2 plot_data_plane commands:
>
> plot_data_plane ncf_20200315_055801.nc CM_with_bad_data.ps
'name="CM";
> level="(*,*)";'
>
> plot_data_plane ncf_20200315_055801.nc CM_censor_with_0.ps
'name="CM";
> level="(*,*)"; censor_thresh=[eq-9999]; censor_val=[0];'
>
>
> But I found that this doesn't actually work as expected! So why
not...
>
>
> Running ncdump, I see that the CM values are actually encoded as
'NaNf':
>
>
> ncdump -v CM ncf_20200315_055801.nc
>
>
> CM =
>
> NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
> NaNf,
>
> NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
> NaNf,
>
> NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
> NaNf,
>
> NaNf...
>
>
> And we don't really have a good way of matching the value NaN with
> thresholds in MET.
>
>
> So take a step back and look at how you generated that NetCDF file.
> The NaN values are causing problems.
>
>
> John
>
> On Thu, Aug 27, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > 472 >
> >
> > Hello John,
> >
> > It looks like the Gen-Vx mask command you are using doesn't appear
> > to do what you intended. I took a look at the command you used
and
> > it
> looks
> > like
> > it is correct. One thing I noticed from your netCDF file is that
the
> fill
> > value CM:_FillValue: -9999.f rather than the -9999 that I'm
> > accustomed
> to.
> > I'm not sure what to try next, I'm going to assign this ticket to
> > John Halley Gotway.
> >
> > Regards,
> > Minna
> >
> > --------------
> > Minna Win
> > National Center for Atmospheric Research Developmental Testbed
> > Center
> > Phone: 303-497-8423
> > Fax: 303-497-8401
> >
> >
> >
> > On Thu, Aug 27, 2020 at 9:26 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Aug 27 09:25:37 2020: Request 96472 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: Request assistance with MET V9.0 Gen_Vx_Mask
> (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=
> > > 96472 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I am trying to use Gen_Vx_Mask to format radar observations
output
> > > by
> the
> > > RADX software to replace missing values (-9999) where there is
no
> > > reflectivity with zeros in order to ensure that the entire
domain
> > contains
> > > valid data for use when running Grid-Stat to generate FSS for a
> > > range
> of
> > > neighborhood sizes and threshold values. The attached text file
> > > shows
> the
> > > command I am using to format the CM (Column Max Reflectivity) 2-
D
> > variable
> > > and the resulting screen dialog. The attached NetCDF file is the
> > > output file. The input NetCDF radar ob is too large (83mb) to
send
> > > by email,
> > but I
> > > can send a link where you could download the file from a file
> exchange. I
> > > have used a similar command to format radar obs containing the
CZ
> > variable
> > > (3-D) with success. I suspect that there is something in the way
I
> > > am specifying the information for the CM variable is not
working,
> > > but I
> > can't
> > > figure out what to try next.
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > Office:(575) 678-2004 DSN 258-2004
> > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
(575)
> > > 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: Request assistance with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Aug 27 14:11:41 2020
John,
That works for me. There's always more than one way to solve it.
I will often run pcp_combine "-add" option as a pass-through like
this:
pcp_combine -add in.grib 'name="TMP"; level="Z2";
censor_thresh=[lt273];
censor_val=[-9999];' out.nc -name TMP_2M_ABOVE_FREEZING
That'll read 2-m temperature from the input file (in.grib), replace
any
values below 273 with bad data, and write the result to a variable
named
TMP_2M_ABOVE_FREEZING in the output file (out.nc).
So that's another way of doing it.
I'll go ahead and resolve this ticket. Hopefully Huaqing can get your
data
from Matlab formatted in the way you need.
Thanks,
John
On Thu, Aug 27, 2020 at 1:36 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Thanks for explaining all the options I have. Good info to know. I'm
used
> to
> running Gen_Vx_Mask up front to create the input data which I know
has
> been
> "formatted" so it will run in Grid-Stat, so I'm used to maintaining
double
> sets of obs. My inclination would be to continue running
Gen_Vx_Mask, but
> using the "censor_thresh" and "censor_val" options instead of the
option
> (-mask_field) I was using before.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, August 27, 2020 1:06 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96472] Request
> assistance
> with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Below, I've listed an excerpt from this page of the user's guide:
> Caution-https://dtcenter.github.io/MET/Users_Guide/data_io.html
>
> The way we have the user's guide structured, I can't provide you a
direct
> link
> to the think info. But we're working on fixing that up for the next
> release...
> moving the data/config/README file into the documentation area.
>
> Anyway, this is used as follows:
> (1) The MET library code reads the input data.
> (2) If a "convert(x)" function is defined, it is applied.
> (3) If "censor_thresh" and "censor_val" options are defined, they
are
> applied.
> (4) Then the application processes the converted and/or censored
data
> values.
>
> It's up to you how you'd like to handle this.
>
> You could run gen_vx_mask or pcp_combine up front to censor your
data in
> this
> way. Then you'd do it once and be done with it.
> But you're doubling your data files... one with uncensored data and
> another
> with censored data.
>
> Or you could specify the censoring logic wherever you need it. In
the
> config
> files for Grid-Stat, Point-Stat, MODE... whatever.
>
> So, up to you.
>
> John
>
> // - The "censor_thresh" entry is an array of thresholds to be
applied
> // to the input data. The "censor_val" entry is an array of
numbers
> // and must be the same length as "censor_thresh". These
arguments
> must
> // appear together in the correct format (threshold and
number).
> For
> each
> // censor threshold, any input values meeting the threshold
> criteria
> will
> // be reset to the corresponding censor value. An empty list
> indicates
> // that no censoring should be performed. The censoring logic
is
> applied
> // prior to any regridding but after the convert function.
All
> statistics
> // are computed on the censored data. These entries may be
used to
> apply
> // quality control logic by resetting data outside of an
expected
> range
> // to the bad data value of -9999. These entries are not
indicated
> in
> the
> // metadata of any output files, but the user can set the
"desc"
> entry
> // accordingly.
>
>
> On Thu, Aug 27, 2020 at 12:51 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96472 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for your suggestion and diagnosis. Huaqing is using RADX to
> > generate the NetCDF files and then uses Matlab to post-process the
> > files to add in the attributes which you noted a week or so ago
were
> > missing. It appears that MatLab is putting the "NaNf" values in
the
> > file. He is working on getting MatLab to insert 0s instead of
"NaNf".
> > Judging from that NetCDF, if we were to figure out how to change
the
> > "NaNf" to 0s, would Grid-Stat be able to run OK or do you see any
> > other obvious issues?
> >
> > Thanks for the tip on using censor_thresh and censor_val. I can
just
> > use that as an argument when running Grid-Stat without having to
use
> > Gen_Vx_Mask. I can still use Gen_Vx_Mask the same way I've been
using
> > it for this purpose, correct?
> >
> > If I use censor_thresh and censor_val as you specify, does that
> > actually modify the data in the observation file or just change
the
> > data being used temporarily for computations by the MET tool?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, August 27, 2020 12:19 PM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96472] Request
> > assistance with MET V9.0 Gen_Vx_Mask (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > Hi John,
> >
> > I see that you'd like to modify some data by replacing any bad
data
> > values of -9999 with a value of 0. This sort of data replacement
can
> > be done in MET using the censor_thresh and censor_val options.
> >
> > By way of example, here are 2 plot_data_plane commands:
> >
> > plot_data_plane ncf_20200315_055801.nc CM_with_bad_data.ps
'name="CM";
> > level="(*,*)";'
> >
> > plot_data_plane ncf_20200315_055801.nc CM_censor_with_0.ps
'name="CM";
> > level="(*,*)"; censor_thresh=[eq-9999]; censor_val=[0];'
> >
> >
> > But I found that this doesn't actually work as expected! So why
not...
> >
> >
> > Running ncdump, I see that the CM values are actually encoded as
'NaNf':
> >
> >
> > ncdump -v CM ncf_20200315_055801.nc
> >
> >
> > CM =
> >
> > NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
> > NaNf,
> >
> > NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
> > NaNf,
> >
> > NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf, NaNf,
NaNf,
> > NaNf,
> >
> > NaNf...
> >
> >
> > And we don't really have a good way of matching the value NaN with
> > thresholds in MET.
> >
> >
> > So take a step back and look at how you generated that NetCDF
file.
> > The NaN values are causing problems.
> >
> >
> > John
> >
> > On Thu, Aug 27, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > > 472 >
> > >
> > > Hello John,
> > >
> > > It looks like the Gen-Vx mask command you are using doesn't
appear
> > > to do what you intended. I took a look at the command you used
and
> > > it
> > looks
> > > like
> > > it is correct. One thing I noticed from your netCDF file is
that the
> > fill
> > > value CM:_FillValue: -9999.f rather than the -9999 that I'm
> > > accustomed
> > to.
> > > I'm not sure what to try next, I'm going to assign this ticket
to
> > > John Halley Gotway.
> > >
> > > Regards,
> > > Minna
> > >
> > > --------------
> > > Minna Win
> > > National Center for Atmospheric Research Developmental Testbed
> > > Center
> > > Phone: 303-497-8423
> > > Fax: 303-497-8401
> > >
> > >
> > >
> > > On Thu, Aug 27, 2020 at 9:26 AM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Aug 27 09:25:37 2020: Request 96472 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: Request assistance with MET V9.0 Gen_Vx_Mask
> > (UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-url:
> > > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=
> > > > 96472 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > I am trying to use Gen_Vx_Mask to format radar observations
output
> > > > by
> > the
> > > > RADX software to replace missing values (-9999) where there is
no
> > > > reflectivity with zeros in order to ensure that the entire
domain
> > > contains
> > > > valid data for use when running Grid-Stat to generate FSS for
a
> > > > range
> > of
> > > > neighborhood sizes and threshold values. The attached text
file
> > > > shows
> > the
> > > > command I am using to format the CM (Column Max Reflectivity)
2-D
> > > variable
> > > > and the resulting screen dialog. The attached NetCDF file is
the
> > > > output file. The input NetCDF radar ob is too large (83mb) to
send
> > > > by email,
> > > but I
> > > > can send a link where you could download the file from a file
> > exchange. I
> > > > have used a similar command to format radar obs containing the
CZ
> > > variable
> > > > (3-D) with success. I suspect that there is something in the
way I
> > > > am specifying the information for the CM variable is not
working,
> > > > but I
> > > can't
> > > > figure out what to try next.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > Office:(575) 678-2004 DSN 258-2004
> > > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
(575)
> > > > 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
More information about the Met_help
mailing list