[Met_help] [rt.rap.ucar.edu #96397] History for mask issues
John Halley Gotway via RT
met_help at ucar.edu
Tue Aug 25 12:36:44 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
GOod evening,
I thought I understood how to process the data into masked regions, but
apparently not. I'm using data that I processed into a netcdf file that is
now CF compatible. I'm processing the DLUSE field from this directory:
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>*
I then run this line, which is inside a loop within a script that I'm
using. The command looks like this:
*gen_vx_mask
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
-type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1 -name Urban*
I determined that something was wrong when much of my masked land plots -
all 24 of them - looked more or less the same. When inspecting Urban, I
found that there were 588 matched pairs! Since I realize that we don't
live on Coruscant, I decided to plot the data and this is what it looked
like (see attached: fig 1). The original DLUSE field looks like what you
see in the second attachment. I'm pretty sure that the gen_vx_mask was
utilized correctly here. I have the file that I'm using, the file that I'm
creating to store the result, the type (data), the mask field (DLUSE at the
surface), the threshold (set equal to 1 for urban), and the name of the
field in the file (Urban). One of the things I noticed was that the output
from the netcdf file that I generated is transposed from the original. The
matrix was originally 265 X 442. However, in the file that was created it
is 442X265. I'm not sure what's going on here at all to be honest. I
thought it would be as simple as running the command with the above
settings. I may try to create netcdf files with the matrix being 442X265.
Something clearly is off here.
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Fri Aug 21 07:28:27 2020
Transposing it didn't seem to do anything either. Here's the
gen_vx_mask
output. It looks okay at first glance:
[Edward.Strobach at m71a1 scripts]$ gen_vx_mask
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
aqm.t06z.metcro2d.00.nc
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
aqm.t06z.metcro2d.00.nc
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
-type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1 -name
Urban
DEBUG 1: Input Grid:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
aqm.t06z.metcro2d.00.nc
DEBUG 1: Mask File:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
aqm.t06z.metcro2d.00.nc
DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
DEBUG 1: Output File:
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
I'm also attaching the header content of the file:
>>> Dataset('aqm.t06z.metcro2d.00.nc')
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):
FileOrigins: NA
MET_version: V9.1
MET_tool: gen_vx_mask
Projection: Lambert Conformal
hemisphere: N
accum: 00
scale_lat_1: 33.000000
scale_lat_2: 45.000000
lat_pin: 21.821000
lon_pin: -120.628000
x_pin: 0.000000
y_pin: 0.000000
lon_orient: -97.000000
d_km: 12.000000
r_km: 6371.200000
nx: 265
ny: 442 grid_points
dimensions(sizes): lon(265), lat(442)
variables(dimensions): float32 LAT(lon,lat), float32 LON(lon,lat),
float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32 PBL(lon,lat),
float32
WSPD10(lon,lat), float32 WDIR10(lon,lat), float32 WSTAR(lon,lat),
float32
USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat), float32
GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat), float32
Q2(lon,lat)
groups:
Nothing here seems troubling. You can see that DLUSE(lon,lat).shape ==
(265,442)
The values in the matrix are of type float, not integer; but, I don't
think
that would be an issue.
Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
masked_array(
data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
...,
[16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
[16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
[16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
mask=[[False, False, False, ..., False, False, False],
[False, False, False, ..., False, False, False],
[False, False, False, ..., False, False, False],
...,
[False, False, False, ..., False, False, False],
[False, False, False, ..., False, False, False],
[False, False, False, ..., False, False, False]],
fill_value=9.96921e+36,
dtype=float32)
Do I need to change the dtype to integer?
On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> Greetings,
>
> This message has been automatically generated in response to the
creation
> of a trouble ticket regarding:
> "mask issues",
> a summary of which appears below.
>
> There is no need to reply to this message right now. Your ticket
has been
> assigned an ID of [rt.rap.ucar.edu #96397].
>
> Please include the string:
>
> [rt.rap.ucar.edu #96397]
>
> in the subject line of all future correspondence about this issue.
To do
> so, you may reply to this message.
>
> For more information, please see:
>
> MET Online Tutorial:
>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>
> MET Users Guide:
> https://www.dtcenter.org/met/users/docs/overview.php
>
> MET FAQs:
> https://www.dtcenter.org/met/users/support/faqs/index.php
>
> MET-Help Email Archive:
> http://mailman.ucar.edu/pipermail/met_help
>
> Thank you,
> met_help at ucar.edu
>
>
-------------------------------------------------------------------------
> GOod evening,
>
> I thought I understood how to process the data into masked regions,
but
> apparently not. I'm using data that I processed into a netcdf file
that is
> now CF compatible. I'm processing the DLUSE field from this
directory:
>
>
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
> <http://aqm.t06z.metcro2d.00.nc>*
>
> I then run this line, which is inside a loop within a script that
I'm
> using. The command looks like this:
> *gen_vx_mask
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
> <http://aqm.t06z.metcro2d.00.nc>
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
> <http://aqm.t06z.metcro2d.00.nc>
>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1 -name
Urban*
>
> I determined that something was wrong when much of my masked land
plots -
> all 24 of them - looked more or less the same. When inspecting
Urban, I
> found that there were 588 matched pairs! Since I realize that we
don't
> live on Coruscant, I decided to plot the data and this is what it
looked
> like (see attached: fig 1). The original DLUSE field looks like
what you
> see in the second attachment. I'm pretty sure that the gen_vx_mask
was
> utilized correctly here. I have the file that I'm using, the file
that I'm
> creating to store the result, the type (data), the mask field (DLUSE
at the
> surface), the threshold (set equal to 1 for urban), and the name of
the
> field in the file (Urban). One of the things I noticed was that the
output
> from the netcdf file that I generated is transposed from the
original. The
> matrix was originally 265 X 442. However, in the file that was
created it
> is 442X265. I'm not sure what's going on here at all to be honest.
I
> thought it would be as simple as running the command with the above
> settings. I may try to create netcdf files with the matrix being
442X265.
> Something clearly is off here.
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Fri Aug 21 08:05:41 2020
Originally, I had processed grib files with vegetation type. I used
this
record here: 880:Vegetation Type:Integer(0-13)
(instant):lambert:surface:level 0:fcst time 3 hrs:from 202008181800
You can see the shape of that is this: >>>
pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
*(288, 468)*
You can also see that upon processing this file through gen_vx_mask in
the
same way as done above produces the same sized matrix - not
transposed.
Dataset('land_mask_15.nc').variables
OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
float32 lat(lat, lon)
long_name: latitude
units: degrees_north
standard_name: latitude
unlimited dimensions:
current shape = (288, 468)
filling on, default _FillValue of 9.969209968386869e+36 used
), ('lon', <class 'netCDF4._netCDF4.Variable'>
float32 lon(lat, lon)
long_name: longitude
units: degrees_east
standard_name: longitude
unlimited dimensions:
current shape = (288, 468)
filling on, default _FillValue of 9.969209968386869e+36 used
), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
float32 data_mask(lat, lon)
long_name: data_mask masking region
_FillValue: -9999.0
mask_type: data==15
unlimited dimensions:
*current shape = (288, 468)*
filling on)])
Moreover, given the mask_type, it identified points having 15 as 1 and
others as 0. Therefore I have a 0/1 dataset.
This is different than when I process the newly created netcdf file.
The
netcdf file that was created has the Urban field with a shape of
(265,442)
When processing it through gen_vx_mask, it gets transposed. >>>
Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
*(442, 265)*.
So in the original case I have no change to the dimensionality of the
dataset, but in the latter case I do. This happens during the
gen_vx_mask
step, and I suspect that this may somehow be related to my issue. I
don't
know if there's anything else I can add that will help, but if I think
of
something I'll forward it along
On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Transposing it didn't seem to do anything either. Here's the
gen_vx_mask
> output. It looks okay at first glance:
>
> [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1 -name
Urban
> DEBUG 1: Input Grid:
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
> DEBUG 1: Mask File:
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
> DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> DEBUG 1: Output File:
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>
> I'm also attaching the header content of the file:
> >>> Dataset('aqm.t06z.metcro2d.00.nc')
> <class 'netCDF4._netCDF4.Dataset'>
> root group (NETCDF4 data model, file format HDF5):
> FileOrigins: NA
> MET_version: V9.1
> MET_tool: gen_vx_mask
> Projection: Lambert Conformal
> hemisphere: N
> accum: 00
> scale_lat_1: 33.000000
> scale_lat_2: 45.000000
> lat_pin: 21.821000
> lon_pin: -120.628000
> x_pin: 0.000000
> y_pin: 0.000000
> lon_orient: -97.000000
> d_km: 12.000000
> r_km: 6371.200000
> nx: 265
> ny: 442 grid_points
> dimensions(sizes): lon(265), lat(442)
> variables(dimensions): float32 LAT(lon,lat), float32
LON(lon,lat),
> float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32 PBL(lon,lat),
float32
> WSPD10(lon,lat), float32 WDIR10(lon,lat), float32 WSTAR(lon,lat),
float32
> USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat), float32
> GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat), float32
> Q2(lon,lat)
> groups:
>
> Nothing here seems troubling. You can see that DLUSE(lon,lat).shape
==
> (265,442)
>
> The values in the matrix are of type float, not integer; but, I
don't
> think that would be an issue.
> Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> masked_array(
> data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> ...,
> [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> mask=[[False, False, False, ..., False, False, False],
> [False, False, False, ..., False, False, False],
> [False, False, False, ..., False, False, False],
> ...,
> [False, False, False, ..., False, False, False],
> [False, False, False, ..., False, False, False],
> [False, False, False, ..., False, False, False]],
> fill_value=9.96921e+36,
> dtype=float32)
>
> Do I need to change the dtype to integer?
>
> On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Greetings,
>>
>> This message has been automatically generated in response to the
creation
>> of a trouble ticket regarding:
>> "mask issues",
>> a summary of which appears below.
>>
>> There is no need to reply to this message right now. Your ticket
has
>> been assigned an ID of [rt.rap.ucar.edu #96397].
>>
>> Please include the string:
>>
>> [rt.rap.ucar.edu #96397]
>>
>> in the subject line of all future correspondence about this issue.
To do
>> so, you may reply to this message.
>>
>> For more information, please see:
>>
>> MET Online Tutorial:
>>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>>
>> MET Users Guide:
>> https://www.dtcenter.org/met/users/docs/overview.php
>>
>> MET FAQs:
>> https://www.dtcenter.org/met/users/support/faqs/index.php
>>
>> MET-Help Email Archive:
>> http://mailman.ucar.edu/pipermail/met_help
>>
>> Thank you,
>> met_help at ucar.edu
>>
>>
-------------------------------------------------------------------------
>> GOod evening,
>>
>> I thought I understood how to process the data into masked regions,
but
>> apparently not. I'm using data that I processed into a netcdf file
that
>> is
>> now CF compatible. I'm processing the DLUSE field from this
directory:
>>
>>
>>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> aqm.t06z.metcro2d.00.nc
>> <http://aqm.t06z.metcro2d.00.nc>*
>>
>> I then run this line, which is inside a loop within a script that
I'm
>> using. The command looks like this:
>> *gen_vx_mask
>>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> aqm.t06z.metcro2d.00.nc
>> <http://aqm.t06z.metcro2d.00.nc>
>>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> aqm.t06z.metcro2d.00.nc
>> <http://aqm.t06z.metcro2d.00.nc>
>>
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
>> Urban*
>>
>> I determined that something was wrong when much of my masked land
plots -
>> all 24 of them - looked more or less the same. When inspecting
Urban, I
>> found that there were 588 matched pairs! Since I realize that we
don't
>> live on Coruscant, I decided to plot the data and this is what it
looked
>> like (see attached: fig 1). The original DLUSE field looks like
what you
>> see in the second attachment. I'm pretty sure that the gen_vx_mask
was
>> utilized correctly here. I have the file that I'm using, the file
that
>> I'm
>> creating to store the result, the type (data), the mask field
(DLUSE at
>> the
>> surface), the threshold (set equal to 1 for urban), and the name of
the
>> field in the file (Urban). One of the things I noticed was that
the
>> output
>> from the netcdf file that I generated is transposed from the
original.
>> The
>> matrix was originally 265 X 442. However, in the file that was
created it
>> is 442X265. I'm not sure what's going on here at all to be honest.
I
>> thought it would be as simple as running the command with the above
>> settings. I may try to create netcdf files with the matrix being
442X265.
>> Something clearly is off here.
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: mask issues
From: John Halley Gotway
Time: Fri Aug 21 10:08:40 2020
Ed,
Thanks for sending your sample commands and a description of the
behavior.
I see that you've prepared a NetCDF file that follows the CF-
convention,
but are getting some odd output from gen_vx_mask. I suspect that the
odd
output from gen_vx_mask is a red herring, and that the real issue is
in
reading the data from that input NetCDF file.
Whenever, getting started with a new dataset in MET, such as the
CF-compliant NetCDF file, it's always a good idea to plot the data
using
plot_data_plane. I suspect we'll find that plot_data_plane creates the
same
sort of odd plot when reading data directly from that input file. And
that's most likely due to either an issue in how the data is stored in
that
NetCDF file or perhaps some bug in the MET software.
I'll go log on to WCOSS, retrieve that file, and do some testing with
it.
Thanks,
John
On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>
> Originally, I had processed grib files with vegetation type. I used
this
> record here: 880:Vegetation Type:Integer(0-13)
> (instant):lambert:surface:level 0:fcst time 3 hrs:from 202008181800
>
> You can see the shape of that is this: >>>
> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> *(288, 468)*
>
> You can also see that upon processing this file through gen_vx_mask
in the
> same way as done above produces the same sized matrix - not
transposed.
>
> Dataset('land_mask_15.nc').variables
> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> float32 lat(lat, lon)
> long_name: latitude
> units: degrees_north
> standard_name: latitude
> unlimited dimensions:
> current shape = (288, 468)
> filling on, default _FillValue of 9.969209968386869e+36 used
> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> float32 lon(lat, lon)
> long_name: longitude
> units: degrees_east
> standard_name: longitude
> unlimited dimensions:
> current shape = (288, 468)
> filling on, default _FillValue of 9.969209968386869e+36 used
> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> float32 data_mask(lat, lon)
> long_name: data_mask masking region
> _FillValue: -9999.0
> mask_type: data==15
> unlimited dimensions:
> *current shape = (288, 468)*
> filling on)])
>
> Moreover, given the mask_type, it identified points having 15 as 1
and
> others as 0. Therefore I have a 0/1 dataset.
>
> This is different than when I process the newly created netcdf file.
The
> netcdf file that was created has the Urban field with a shape of
(265,442)
>
> When processing it through gen_vx_mask, it gets transposed. >>>
> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> *(442, 265)*.
>
> So in the original case I have no change to the dimensionality of
the
> dataset, but in the latter case I do. This happens during the
gen_vx_mask
> step, and I suspect that this may somehow be related to my issue. I
don't
> know if there's anything else I can add that will help, but if I
think of
> something I'll forward it along
>
> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Transposing it didn't seem to do anything either. Here's the
gen_vx_mask
> > output. It looks okay at first glance:
> >
> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > aqm.t06z.metcro2d.00.nc
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > aqm.t06z.metcro2d.00.nc
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> Urban
> > DEBUG 1: Input Grid:
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > aqm.t06z.metcro2d.00.nc
> > DEBUG 1: Mask File:
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > aqm.t06z.metcro2d.00.nc
> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> > DEBUG 1: Output File:
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >
> > I'm also attaching the header content of the file:
> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > <class 'netCDF4._netCDF4.Dataset'>
> > root group (NETCDF4 data model, file format HDF5):
> > FileOrigins: NA
> > MET_version: V9.1
> > MET_tool: gen_vx_mask
> > Projection: Lambert Conformal
> > hemisphere: N
> > accum: 00
> > scale_lat_1: 33.000000
> > scale_lat_2: 45.000000
> > lat_pin: 21.821000
> > lon_pin: -120.628000
> > x_pin: 0.000000
> > y_pin: 0.000000
> > lon_orient: -97.000000
> > d_km: 12.000000
> > r_km: 6371.200000
> > nx: 265
> > ny: 442 grid_points
> > dimensions(sizes): lon(265), lat(442)
> > variables(dimensions): float32 LAT(lon,lat), float32
LON(lon,lat),
> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
PBL(lon,lat),
> float32
> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32 WSTAR(lon,lat),
float32
> > USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat),
float32
> > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
float32
> > Q2(lon,lat)
> > groups:
> >
> > Nothing here seems troubling. You can see that
DLUSE(lon,lat).shape ==
> > (265,442)
> >
> > The values in the matrix are of type float, not integer; but, I
don't
> > think that would be an issue.
> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > masked_array(
> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > ...,
> > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > mask=[[False, False, False, ..., False, False, False],
> > [False, False, False, ..., False, False, False],
> > [False, False, False, ..., False, False, False],
> > ...,
> > [False, False, False, ..., False, False, False],
> > [False, False, False, ..., False, False, False],
> > [False, False, False, ..., False, False, False]],
> > fill_value=9.96921e+36,
> > dtype=float32)
> >
> > Do I need to change the dtype to integer?
> >
> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> Greetings,
> >>
> >> This message has been automatically generated in response to the
> creation
> >> of a trouble ticket regarding:
> >> "mask issues",
> >> a summary of which appears below.
> >>
> >> There is no need to reply to this message right now. Your ticket
has
> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> >>
> >> Please include the string:
> >>
> >> [rt.rap.ucar.edu #96397]
> >>
> >> in the subject line of all future correspondence about this
issue. To do
> >> so, you may reply to this message.
> >>
> >> For more information, please see:
> >>
> >> MET Online Tutorial:
> >>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> >>
> >> MET Users Guide:
> >> https://www.dtcenter.org/met/users/docs/overview.php
> >>
> >> MET FAQs:
> >> https://www.dtcenter.org/met/users/support/faqs/index.php
> >>
> >> MET-Help Email Archive:
> >> http://mailman.ucar.edu/pipermail/met_help
> >>
> >> Thank you,
> >> met_help at ucar.edu
> >>
> >>
>
-------------------------------------------------------------------------
> >> GOod evening,
> >>
> >> I thought I understood how to process the data into masked
regions, but
> >> apparently not. I'm using data that I processed into a netcdf
file that
> >> is
> >> now CF compatible. I'm processing the DLUSE field from this
directory:
> >>
> >>
> >>
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> aqm.t06z.metcro2d.00.nc
> >> <http://aqm.t06z.metcro2d.00.nc>*
> >>
> >> I then run this line, which is inside a loop within a script that
I'm
> >> using. The command looks like this:
> >> *gen_vx_mask
> >>
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> aqm.t06z.metcro2d.00.nc
> >> <http://aqm.t06z.metcro2d.00.nc>
> >>
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> aqm.t06z.metcro2d.00.nc
> >> <http://aqm.t06z.metcro2d.00.nc>
> >>
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> >> Urban*
> >>
> >> I determined that something was wrong when much of my masked land
plots
> -
> >> all 24 of them - looked more or less the same. When inspecting
Urban, I
> >> found that there were 588 matched pairs! Since I realize that we
don't
> >> live on Coruscant, I decided to plot the data and this is what it
looked
> >> like (see attached: fig 1). The original DLUSE field looks like
what
> you
> >> see in the second attachment. I'm pretty sure that the
gen_vx_mask was
> >> utilized correctly here. I have the file that I'm using, the
file that
> >> I'm
> >> creating to store the result, the type (data), the mask field
(DLUSE at
> >> the
> >> surface), the threshold (set equal to 1 for urban), and the name
of the
> >> field in the file (Urban). One of the things I noticed was that
the
> >> output
> >> from the netcdf file that I generated is transposed from the
original.
> >> The
> >> matrix was originally 265 X 442. However, in the file that was
created
> it
> >> is 442X265. I'm not sure what's going on here at all to be
honest. I
> >> thought it would be as simple as running the command with the
above
> >> settings. I may try to create netcdf files with the matrix being
> 442X265.
> >> Something clearly is off here.
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >>
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: mask issues
From: John Halley Gotway
Time: Fri Aug 21 10:41:41 2020
Ed,
I pulled that file and took a look. I've attached the image that
plot_data_plane generates with the following command:
*plot_data_plane aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>
aqm.t06z.metcro2d_DLUSE.00.ps <http://aqm.t06z.metcro2d_DLUSE.00.ps>
'name="DLUSE"; level="(*,*)";'*
So there's clearly a problem here! Looking at that file with ncdump, I
see
that this file is actually setup to follow MET's internal NetCDF
conventions rather the the NetCDF Climate-Forecast (CF) convention.
*ncdump -h aqm.t06z.metcro2d.00.nc <http://aqm.t06z.metcro2d.00.nc>*
Even though we recommend the use of the CF-convention, the MET tools
actually don't write to that convention themselves. But we should
probably
transition to it! For now, the tools write to a pretty rigid internal
format where the dimensions are always ordered (lat, lon). Your data
is
ordered (lon, lat) and I strongly suspect that's the reason for this
behavior. Please try rearranging to write it as (lat, lon) and then
run
plot_data_plane on the result to see if it's fixed.
Hope that helps.
Thanks,
John
On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Ed,
>
> Thanks for sending your sample commands and a description of the
behavior.
> I see that you've prepared a NetCDF file that follows the CF-
convention,
> but are getting some odd output from gen_vx_mask. I suspect that the
odd
> output from gen_vx_mask is a red herring, and that the real issue is
in
> reading the data from that input NetCDF file.
>
> Whenever, getting started with a new dataset in MET, such as the
> CF-compliant NetCDF file, it's always a good idea to plot the data
using
> plot_data_plane. I suspect we'll find that plot_data_plane creates
the same
> sort of odd plot when reading data directly from that input file.
And
> that's most likely due to either an issue in how the data is stored
in that
> NetCDF file or perhaps some bug in the MET software.
>
> I'll go log on to WCOSS, retrieve that file, and do some testing
with it.
>
> Thanks,
> John
>
> On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>>
>> Originally, I had processed grib files with vegetation type. I
used this
>> record here: 880:Vegetation Type:Integer(0-13)
>> (instant):lambert:surface:level 0:fcst time 3 hrs:from 202008181800
>>
>> You can see the shape of that is this: >>>
>> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
>> *(288, 468)*
>>
>> You can also see that upon processing this file through gen_vx_mask
in the
>> same way as done above produces the same sized matrix - not
transposed.
>>
>> Dataset('land_mask_15.nc').variables
>> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
>> float32 lat(lat, lon)
>> long_name: latitude
>> units: degrees_north
>> standard_name: latitude
>> unlimited dimensions:
>> current shape = (288, 468)
>> filling on, default _FillValue of 9.969209968386869e+36 used
>> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
>> float32 lon(lat, lon)
>> long_name: longitude
>> units: degrees_east
>> standard_name: longitude
>> unlimited dimensions:
>> current shape = (288, 468)
>> filling on, default _FillValue of 9.969209968386869e+36 used
>> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
>> float32 data_mask(lat, lon)
>> long_name: data_mask masking region
>> _FillValue: -9999.0
>> mask_type: data==15
>> unlimited dimensions:
>> *current shape = (288, 468)*
>> filling on)])
>>
>> Moreover, given the mask_type, it identified points having 15 as 1
and
>> others as 0. Therefore I have a 0/1 dataset.
>>
>> This is different than when I process the newly created netcdf
file. The
>> netcdf file that was created has the Urban field with a shape of
(265,442)
>>
>> When processing it through gen_vx_mask, it gets transposed. >>>
>> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
>> *(442, 265)*.
>>
>> So in the original case I have no change to the dimensionality of
the
>> dataset, but in the latter case I do. This happens during the
gen_vx_mask
>> step, and I suspect that this may somehow be related to my issue.
I don't
>> know if there's anything else I can add that will help, but if I
think of
>> something I'll forward it along
>>
>> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>> > Transposing it didn't seem to do anything either. Here's the
>> gen_vx_mask
>> > output. It looks okay at first glance:
>> >
>> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > aqm.t06z.metcro2d.00.nc
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > aqm.t06z.metcro2d.00.nc
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
>> Urban
>> > DEBUG 1: Input Grid:
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > aqm.t06z.metcro2d.00.nc
>> > DEBUG 1: Mask File:
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > aqm.t06z.metcro2d.00.nc
>> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
>> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
>> > DEBUG 1: Output File:
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> >
>> > I'm also attaching the header content of the file:
>> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
>> > <class 'netCDF4._netCDF4.Dataset'>
>> > root group (NETCDF4 data model, file format HDF5):
>> > FileOrigins: NA
>> > MET_version: V9.1
>> > MET_tool: gen_vx_mask
>> > Projection: Lambert Conformal
>> > hemisphere: N
>> > accum: 00
>> > scale_lat_1: 33.000000
>> > scale_lat_2: 45.000000
>> > lat_pin: 21.821000
>> > lon_pin: -120.628000
>> > x_pin: 0.000000
>> > y_pin: 0.000000
>> > lon_orient: -97.000000
>> > d_km: 12.000000
>> > r_km: 6371.200000
>> > nx: 265
>> > ny: 442 grid_points
>> > dimensions(sizes): lon(265), lat(442)
>> > variables(dimensions): float32 LAT(lon,lat), float32
LON(lon,lat),
>> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
PBL(lon,lat),
>> float32
>> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32 WSTAR(lon,lat),
>> float32
>> > USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat),
float32
>> > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
float32
>> > Q2(lon,lat)
>> > groups:
>> >
>> > Nothing here seems troubling. You can see that
DLUSE(lon,lat).shape ==
>> > (265,442)
>> >
>> > The values in the matrix are of type float, not integer; but, I
don't
>> > think that would be an issue.
>> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
>> > masked_array(
>> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>> > ...,
>> > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
>> > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
>> > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
>> > mask=[[False, False, False, ..., False, False, False],
>> > [False, False, False, ..., False, False, False],
>> > [False, False, False, ..., False, False, False],
>> > ...,
>> > [False, False, False, ..., False, False, False],
>> > [False, False, False, ..., False, False, False],
>> > [False, False, False, ..., False, False, False]],
>> > fill_value=9.96921e+36,
>> > dtype=float32)
>> >
>> > Do I need to change the dtype to integer?
>> >
>> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >>
>> >> Greetings,
>> >>
>> >> This message has been automatically generated in response to the
>> creation
>> >> of a trouble ticket regarding:
>> >> "mask issues",
>> >> a summary of which appears below.
>> >>
>> >> There is no need to reply to this message right now. Your
ticket has
>> >> been assigned an ID of [rt.rap.ucar.edu #96397].
>> >>
>> >> Please include the string:
>> >>
>> >> [rt.rap.ucar.edu #96397]
>> >>
>> >> in the subject line of all future correspondence about this
issue. To
>> do
>> >> so, you may reply to this message.
>> >>
>> >> For more information, please see:
>> >>
>> >> MET Online Tutorial:
>> >>
>>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>> >>
>> >> MET Users Guide:
>> >> https://www.dtcenter.org/met/users/docs/overview.php
>> >>
>> >> MET FAQs:
>> >> https://www.dtcenter.org/met/users/support/faqs/index.php
>> >>
>> >> MET-Help Email Archive:
>> >> http://mailman.ucar.edu/pipermail/met_help
>> >>
>> >> Thank you,
>> >> met_help at ucar.edu
>> >>
>> >>
>>
-------------------------------------------------------------------------
>> >> GOod evening,
>> >>
>> >> I thought I understood how to process the data into masked
regions, but
>> >> apparently not. I'm using data that I processed into a netcdf
file
>> that
>> >> is
>> >> now CF compatible. I'm processing the DLUSE field from this
directory:
>> >>
>> >>
>> >>
>>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> >> aqm.t06z.metcro2d.00.nc
>> >> <http://aqm.t06z.metcro2d.00.nc>*
>> >>
>> >> I then run this line, which is inside a loop within a script
that I'm
>> >> using. The command looks like this:
>> >> *gen_vx_mask
>> >>
>> >>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> >> aqm.t06z.metcro2d.00.nc
>> >> <http://aqm.t06z.metcro2d.00.nc>
>> >>
>> >>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> >> aqm.t06z.metcro2d.00.nc
>> >> <http://aqm.t06z.metcro2d.00.nc>
>> >>
>> >>
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
>> >> Urban*
>> >>
>> >> I determined that something was wrong when much of my masked
land
>> plots -
>> >> all 24 of them - looked more or less the same. When inspecting
Urban,
>> I
>> >> found that there were 588 matched pairs! Since I realize that
we don't
>> >> live on Coruscant, I decided to plot the data and this is what
it
>> looked
>> >> like (see attached: fig 1). The original DLUSE field looks like
what
>> you
>> >> see in the second attachment. I'm pretty sure that the
gen_vx_mask was
>> >> utilized correctly here. I have the file that I'm using, the
file that
>> >> I'm
>> >> creating to store the result, the type (data), the mask field
(DLUSE at
>> >> the
>> >> surface), the threshold (set equal to 1 for urban), and the name
of the
>> >> field in the file (Urban). One of the things I noticed was that
the
>> >> output
>> >> from the netcdf file that I generated is transposed from the
original.
>> >> The
>> >> matrix was originally 265 X 442. However, in the file that was
>> created it
>> >> is 442X265. I'm not sure what's going on here at all to be
honest. I
>> >> thought it would be as simple as running the command with the
above
>> >> settings. I may try to create netcdf files with the matrix
being
>> 442X265.
>> >> Something clearly is off here.
>> >>
>> >> --
>> >> Edward Strobach
>> >> EMC/NCEP/NWS/
>> >> IMSG Contractor
>> >> Cubicle#: 2029
>> >> 301-683-3717
>> >>
>> >>
>> >
>> > --
>> > Edward Strobach
>> > EMC/NCEP/NWS/
>> > IMSG Contractor
>> > Cubicle#: 2029
>> > 301-683-3717
>> >
>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>>
------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Fri Aug 21 10:42:14 2020
Hi John,
Hmm, I thought I sent the attachments in the original email. I don't
have
the colorbar attached in the screenshots, but the values are all
included
(1-24) for both plots. The first is the original dominant landuse
field,
which looks like what you would expect. The other is the result after
running gen_vx_mask. It seems that this matrix got reordered in the
process, which is why I originally surmised it was related to the
matrix
being transposed when running that step. I had expected only values
of 0's
and 1's, with 1's identifying urban; thus I thought that the map being
generated would be predominately 0's with isolated spots of 1's in the
major urban areas. Hopefully the attachments are viewable
On Fri, Aug 21, 2020 at 12:08 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Ed,
>
> Thanks for sending your sample commands and a description of the
behavior.
> I see that you've prepared a NetCDF file that follows the CF-
convention,
> but are getting some odd output from gen_vx_mask. I suspect that the
odd
> output from gen_vx_mask is a red herring, and that the real issue is
in
> reading the data from that input NetCDF file.
>
> Whenever, getting started with a new dataset in MET, such as the
> CF-compliant NetCDF file, it's always a good idea to plot the data
using
> plot_data_plane. I suspect we'll find that plot_data_plane creates
the same
> sort of odd plot when reading data directly from that input file.
And
> that's most likely due to either an issue in how the data is stored
in that
> NetCDF file or perhaps some bug in the MET software.
>
> I'll go log on to WCOSS, retrieve that file, and do some testing
with it.
>
> Thanks,
> John
>
> On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> >
> > Originally, I had processed grib files with vegetation type. I
used this
> > record here: 880:Vegetation Type:Integer(0-13)
> > (instant):lambert:surface:level 0:fcst time 3 hrs:from
202008181800
> >
> > You can see the shape of that is this: >>>
> > pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> > *(288, 468)*
> >
> > You can also see that upon processing this file through
gen_vx_mask in
> the
> > same way as done above produces the same sized matrix - not
transposed.
> >
> > Dataset('land_mask_15.nc').variables
> > OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> > float32 lat(lat, lon)
> > long_name: latitude
> > units: degrees_north
> > standard_name: latitude
> > unlimited dimensions:
> > current shape = (288, 468)
> > filling on, default _FillValue of 9.969209968386869e+36 used
> > ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> > float32 lon(lat, lon)
> > long_name: longitude
> > units: degrees_east
> > standard_name: longitude
> > unlimited dimensions:
> > current shape = (288, 468)
> > filling on, default _FillValue of 9.969209968386869e+36 used
> > ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> > float32 data_mask(lat, lon)
> > long_name: data_mask masking region
> > _FillValue: -9999.0
> > mask_type: data==15
> > unlimited dimensions:
> > *current shape = (288, 468)*
> > filling on)])
> >
> > Moreover, given the mask_type, it identified points having 15 as 1
and
> > others as 0. Therefore I have a 0/1 dataset.
> >
> > This is different than when I process the newly created netcdf
file. The
> > netcdf file that was created has the Urban field with a shape of
> (265,442)
> >
> > When processing it through gen_vx_mask, it gets transposed. >>>
> > Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> > *(442, 265)*.
> >
> > So in the original case I have no change to the dimensionality of
the
> > dataset, but in the latter case I do. This happens during the
> gen_vx_mask
> > step, and I suspect that this may somehow be related to my issue.
I
> don't
> > know if there's anything else I can add that will help, but if I
think of
> > something I'll forward it along
> >
> > On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > Transposing it didn't seem to do anything either. Here's the
> gen_vx_mask
> > > output. It looks okay at first glance:
> > >
> > > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > aqm.t06z.metcro2d.00.nc
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > aqm.t06z.metcro2d.00.nc
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> > Urban
> > > DEBUG 1: Input Grid:
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > aqm.t06z.metcro2d.00.nc
> > > DEBUG 1: Mask File:
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > aqm.t06z.metcro2d.00.nc
> > > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> > > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> > > DEBUG 1: Output File:
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > >
> > > I'm also attaching the header content of the file:
> > > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > > <class 'netCDF4._netCDF4.Dataset'>
> > > root group (NETCDF4 data model, file format HDF5):
> > > FileOrigins: NA
> > > MET_version: V9.1
> > > MET_tool: gen_vx_mask
> > > Projection: Lambert Conformal
> > > hemisphere: N
> > > accum: 00
> > > scale_lat_1: 33.000000
> > > scale_lat_2: 45.000000
> > > lat_pin: 21.821000
> > > lon_pin: -120.628000
> > > x_pin: 0.000000
> > > y_pin: 0.000000
> > > lon_orient: -97.000000
> > > d_km: 12.000000
> > > r_km: 6371.200000
> > > nx: 265
> > > ny: 442 grid_points
> > > dimensions(sizes): lon(265), lat(442)
> > > variables(dimensions): float32 LAT(lon,lat), float32
LON(lon,lat),
> > > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
PBL(lon,lat),
> > float32
> > > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
WSTAR(lon,lat),
> float32
> > > USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat),
float32
> > > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
float32
> > > Q2(lon,lat)
> > > groups:
> > >
> > > Nothing here seems troubling. You can see that
DLUSE(lon,lat).shape ==
> > > (265,442)
> > >
> > > The values in the matrix are of type float, not integer; but, I
don't
> > > think that would be an issue.
> > > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > > masked_array(
> > > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > ...,
> > > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > > mask=[[False, False, False, ..., False, False, False],
> > > [False, False, False, ..., False, False, False],
> > > [False, False, False, ..., False, False, False],
> > > ...,
> > > [False, False, False, ..., False, False, False],
> > > [False, False, False, ..., False, False, False],
> > > [False, False, False, ..., False, False, False]],
> > > fill_value=9.96921e+36,
> > > dtype=float32)
> > >
> > > Do I need to change the dtype to integer?
> > >
> > > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> Greetings,
> > >>
> > >> This message has been automatically generated in response to
the
> > creation
> > >> of a trouble ticket regarding:
> > >> "mask issues",
> > >> a summary of which appears below.
> > >>
> > >> There is no need to reply to this message right now. Your
ticket has
> > >> been assigned an ID of [rt.rap.ucar.edu #96397].
> > >>
> > >> Please include the string:
> > >>
> > >> [rt.rap.ucar.edu #96397]
> > >>
> > >> in the subject line of all future correspondence about this
issue. To
> do
> > >> so, you may reply to this message.
> > >>
> > >> For more information, please see:
> > >>
> > >> MET Online Tutorial:
> > >>
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > >>
> > >> MET Users Guide:
> > >> https://www.dtcenter.org/met/users/docs/overview.php
> > >>
> > >> MET FAQs:
> > >> https://www.dtcenter.org/met/users/support/faqs/index.php
> > >>
> > >> MET-Help Email Archive:
> > >> http://mailman.ucar.edu/pipermail/met_help
> > >>
> > >> Thank you,
> > >> met_help at ucar.edu
> > >>
> > >>
> >
-------------------------------------------------------------------------
> > >> GOod evening,
> > >>
> > >> I thought I understood how to process the data into masked
regions,
> but
> > >> apparently not. I'm using data that I processed into a netcdf
file
> that
> > >> is
> > >> now CF compatible. I'm processing the DLUSE field from this
> directory:
> > >>
> > >>
> > >>
> >
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> aqm.t06z.metcro2d.00.nc
> > >> <http://aqm.t06z.metcro2d.00.nc>*
> > >>
> > >> I then run this line, which is inside a loop within a script
that I'm
> > >> using. The command looks like this:
> > >> *gen_vx_mask
> > >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> aqm.t06z.metcro2d.00.nc
> > >> <http://aqm.t06z.metcro2d.00.nc>
> > >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> aqm.t06z.metcro2d.00.nc
> > >> <http://aqm.t06z.metcro2d.00.nc>
> > >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> > >> Urban*
> > >>
> > >> I determined that something was wrong when much of my masked
land
> plots
> > -
> > >> all 24 of them - looked more or less the same. When inspecting
> Urban, I
> > >> found that there were 588 matched pairs! Since I realize that
we
> don't
> > >> live on Coruscant, I decided to plot the data and this is what
it
> looked
> > >> like (see attached: fig 1). The original DLUSE field looks
like what
> > you
> > >> see in the second attachment. I'm pretty sure that the
gen_vx_mask
> was
> > >> utilized correctly here. I have the file that I'm using, the
file
> that
> > >> I'm
> > >> creating to store the result, the type (data), the mask field
(DLUSE
> at
> > >> the
> > >> surface), the threshold (set equal to 1 for urban), and the
name of
> the
> > >> field in the file (Urban). One of the things I noticed was
that the
> > >> output
> > >> from the netcdf file that I generated is transposed from the
original.
> > >> The
> > >> matrix was originally 265 X 442. However, in the file that was
> created
> > it
> > >> is 442X265. I'm not sure what's going on here at all to be
honest. I
> > >> thought it would be as simple as running the command with the
above
> > >> settings. I may try to create netcdf files with the matrix
being
> > 442X265.
> > >> Something clearly is off here.
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > >>
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Fri Aug 21 10:51:50 2020
Hi John,
Your plot looks just like mine. Thanks for confirming. I do believe
I
tried that, but I'll go ahead and try that suggestion again. I'll
attach
the result as well. When comparing with VGTYP - an example that
worked - I
noticed the "lat, lon" ordering as opposed to "lon, lat", which is
what
prompted me to go that route. I'll test this carefully. Thanks
again.
On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Ed,
>
> I pulled that file and took a look. I've attached the image that
> plot_data_plane generates with the following command:
>
> *plot_data_plane aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>
> aqm.t06z.metcro2d_DLUSE.00.ps <http://aqm.t06z.metcro2d_DLUSE.00.ps>
> 'name="DLUSE"; level="(*,*)";'*
>
>
>
> So there's clearly a problem here! Looking at that file with ncdump,
I see
> that this file is actually setup to follow MET's internal NetCDF
> conventions rather the the NetCDF Climate-Forecast (CF) convention.
>
>
> *ncdump -h aqm.t06z.metcro2d.00.nc <http://aqm.t06z.metcro2d.00.nc>*
>
>
>
> Even though we recommend the use of the CF-convention, the MET tools
> actually don't write to that convention themselves. But we should
probably
> transition to it! For now, the tools write to a pretty rigid
internal
> format where the dimensions are always ordered (lat, lon). Your data
is
> ordered (lon, lat) and I strongly suspect that's the reason for this
> behavior. Please try rearranging to write it as (lat, lon) and then
run
> plot_data_plane on the result to see if it's fixed.
>
>
> Hope that helps.
>
>
> Thanks,
> John
>
>
>
> On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Ed,
> >
> > Thanks for sending your sample commands and a description of the
> behavior.
> > I see that you've prepared a NetCDF file that follows the CF-
convention,
> > but are getting some odd output from gen_vx_mask. I suspect that
the odd
> > output from gen_vx_mask is a red herring, and that the real issue
is in
> > reading the data from that input NetCDF file.
> >
> > Whenever, getting started with a new dataset in MET, such as the
> > CF-compliant NetCDF file, it's always a good idea to plot the data
using
> > plot_data_plane. I suspect we'll find that plot_data_plane creates
the
> same
> > sort of odd plot when reading data directly from that input file.
And
> > that's most likely due to either an issue in how the data is
stored in
> that
> > NetCDF file or perhaps some bug in the MET software.
> >
> > I'll go log on to WCOSS, retrieve that file, and do some testing
with it.
> >
> > Thanks,
> > John
> >
> > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> >>
> >> Originally, I had processed grib files with vegetation type. I
used
> this
> >> record here: 880:Vegetation Type:Integer(0-13)
> >> (instant):lambert:surface:level 0:fcst time 3 hrs:from
202008181800
> >>
> >> You can see the shape of that is this: >>>
> >> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> >> *(288, 468)*
> >>
> >> You can also see that upon processing this file through
gen_vx_mask in
> the
> >> same way as done above produces the same sized matrix - not
transposed.
> >>
> >> Dataset('land_mask_15.nc').variables
> >> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> >> float32 lat(lat, lon)
> >> long_name: latitude
> >> units: degrees_north
> >> standard_name: latitude
> >> unlimited dimensions:
> >> current shape = (288, 468)
> >> filling on, default _FillValue of 9.969209968386869e+36 used
> >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> >> float32 lon(lat, lon)
> >> long_name: longitude
> >> units: degrees_east
> >> standard_name: longitude
> >> unlimited dimensions:
> >> current shape = (288, 468)
> >> filling on, default _FillValue of 9.969209968386869e+36 used
> >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> >> float32 data_mask(lat, lon)
> >> long_name: data_mask masking region
> >> _FillValue: -9999.0
> >> mask_type: data==15
> >> unlimited dimensions:
> >> *current shape = (288, 468)*
> >> filling on)])
> >>
> >> Moreover, given the mask_type, it identified points having 15 as
1 and
> >> others as 0. Therefore I have a 0/1 dataset.
> >>
> >> This is different than when I process the newly created netcdf
file.
> The
> >> netcdf file that was created has the Urban field with a shape of
> (265,442)
> >>
> >> When processing it through gen_vx_mask, it gets transposed. >>>
> >> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> >> *(442, 265)*.
> >>
> >> So in the original case I have no change to the dimensionality of
the
> >> dataset, but in the latter case I do. This happens during the
> gen_vx_mask
> >> step, and I suspect that this may somehow be related to my issue.
I
> don't
> >> know if there's anything else I can add that will help, but if I
think
> of
> >> something I'll forward it along
> >>
> >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA Affiliate
<
> >> edward.strobach at noaa.gov> wrote:
> >>
> >> > Transposing it didn't seem to do anything either. Here's the
> >> gen_vx_mask
> >> > output. It looks okay at first glance:
> >> >
> >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > aqm.t06z.metcro2d.00.nc
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > aqm.t06z.metcro2d.00.nc
> >> >
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> >> Urban
> >> > DEBUG 1: Input Grid:
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > aqm.t06z.metcro2d.00.nc
> >> > DEBUG 1: Mask File:
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > aqm.t06z.metcro2d.00.nc
> >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> >> > DEBUG 1: Output File:
> >> >
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> >
> >> > I'm also attaching the header content of the file:
> >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> >> > <class 'netCDF4._netCDF4.Dataset'>
> >> > root group (NETCDF4 data model, file format HDF5):
> >> > FileOrigins: NA
> >> > MET_version: V9.1
> >> > MET_tool: gen_vx_mask
> >> > Projection: Lambert Conformal
> >> > hemisphere: N
> >> > accum: 00
> >> > scale_lat_1: 33.000000
> >> > scale_lat_2: 45.000000
> >> > lat_pin: 21.821000
> >> > lon_pin: -120.628000
> >> > x_pin: 0.000000
> >> > y_pin: 0.000000
> >> > lon_orient: -97.000000
> >> > d_km: 12.000000
> >> > r_km: 6371.200000
> >> > nx: 265
> >> > ny: 442 grid_points
> >> > dimensions(sizes): lon(265), lat(442)
> >> > variables(dimensions): float32 LAT(lon,lat), float32
LON(lon,lat),
> >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
PBL(lon,lat),
> >> float32
> >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
WSTAR(lon,lat),
> >> float32
> >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat),
float32
> >> > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
float32
> >> > Q2(lon,lat)
> >> > groups:
> >> >
> >> > Nothing here seems troubling. You can see that
DLUSE(lon,lat).shape ==
> >> > (265,442)
> >> >
> >> > The values in the matrix are of type float, not integer; but, I
don't
> >> > think that would be an issue.
> >> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> >> > masked_array(
> >> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> >> > ...,
> >> > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> >> > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> >> > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> >> > mask=[[False, False, False, ..., False, False, False],
> >> > [False, False, False, ..., False, False, False],
> >> > [False, False, False, ..., False, False, False],
> >> > ...,
> >> > [False, False, False, ..., False, False, False],
> >> > [False, False, False, ..., False, False, False],
> >> > [False, False, False, ..., False, False, False]],
> >> > fill_value=9.96921e+36,
> >> > dtype=float32)
> >> >
> >> > Do I need to change the dtype to integer?
> >> >
> >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> >>
> >> >> Greetings,
> >> >>
> >> >> This message has been automatically generated in response to
the
> >> creation
> >> >> of a trouble ticket regarding:
> >> >> "mask issues",
> >> >> a summary of which appears below.
> >> >>
> >> >> There is no need to reply to this message right now. Your
ticket has
> >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> >> >>
> >> >> Please include the string:
> >> >>
> >> >> [rt.rap.ucar.edu #96397]
> >> >>
> >> >> in the subject line of all future correspondence about this
issue. To
> >> do
> >> >> so, you may reply to this message.
> >> >>
> >> >> For more information, please see:
> >> >>
> >> >> MET Online Tutorial:
> >> >>
> >>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> >> >>
> >> >> MET Users Guide:
> >> >> https://www.dtcenter.org/met/users/docs/overview.php
> >> >>
> >> >> MET FAQs:
> >> >> https://www.dtcenter.org/met/users/support/faqs/index.php
> >> >>
> >> >> MET-Help Email Archive:
> >> >> http://mailman.ucar.edu/pipermail/met_help
> >> >>
> >> >> Thank you,
> >> >> met_help at ucar.edu
> >> >>
> >> >>
> >>
>
-------------------------------------------------------------------------
> >> >> GOod evening,
> >> >>
> >> >> I thought I understood how to process the data into masked
regions,
> but
> >> >> apparently not. I'm using data that I processed into a netcdf
file
> >> that
> >> >> is
> >> >> now CF compatible. I'm processing the DLUSE field from this
> directory:
> >> >>
> >> >>
> >> >>
> >>
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> >> aqm.t06z.metcro2d.00.nc
> >> >> <http://aqm.t06z.metcro2d.00.nc>*
> >> >>
> >> >> I then run this line, which is inside a loop within a script
that I'm
> >> >> using. The command looks like this:
> >> >> *gen_vx_mask
> >> >>
> >> >>
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> >> aqm.t06z.metcro2d.00.nc
> >> >> <http://aqm.t06z.metcro2d.00.nc>
> >> >>
> >> >>
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> >> aqm.t06z.metcro2d.00.nc
> >> >> <http://aqm.t06z.metcro2d.00.nc>
> >> >>
> >> >>
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> >> >> Urban*
> >> >>
> >> >> I determined that something was wrong when much of my masked
land
> >> plots -
> >> >> all 24 of them - looked more or less the same. When
inspecting
> Urban,
> >> I
> >> >> found that there were 588 matched pairs! Since I realize that
we
> don't
> >> >> live on Coruscant, I decided to plot the data and this is what
it
> >> looked
> >> >> like (see attached: fig 1). The original DLUSE field looks
like what
> >> you
> >> >> see in the second attachment. I'm pretty sure that the
gen_vx_mask
> was
> >> >> utilized correctly here. I have the file that I'm using, the
file
> that
> >> >> I'm
> >> >> creating to store the result, the type (data), the mask field
(DLUSE
> at
> >> >> the
> >> >> surface), the threshold (set equal to 1 for urban), and the
name of
> the
> >> >> field in the file (Urban). One of the things I noticed was
that the
> >> >> output
> >> >> from the netcdf file that I generated is transposed from the
> original.
> >> >> The
> >> >> matrix was originally 265 X 442. However, in the file that
was
> >> created it
> >> >> is 442X265. I'm not sure what's going on here at all to be
honest.
> I
> >> >> thought it would be as simple as running the command with the
above
> >> >> settings. I may try to create netcdf files with the matrix
being
> >> 442X265.
> >> >> Something clearly is off here.
> >> >>
> >> >> --
> >> >> Edward Strobach
> >> >> EMC/NCEP/NWS/
> >> >> IMSG Contractor
> >> >> Cubicle#: 2029
> >> >> 301-683-3717
> >> >>
> >> >>
> >> >
> >> > --
> >> > Edward Strobach
> >> > EMC/NCEP/NWS/
> >> > IMSG Contractor
> >> > Cubicle#: 2029
> >> > 301-683-3717
> >> >
> >>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: mask issues
From: John Halley Gotway
Time: Fri Aug 21 14:46:09 2020
Ed,
Sounds good. Please let me know if reversing (lon,lat) to (lat,lon) in
the
NetCDF files solves the problem.
Have a good weekend.
Thanks,
John
On Fri, Aug 21, 2020 at 10:51 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>
> Hi John,
>
> Your plot looks just like mine. Thanks for confirming. I do
believe I
> tried that, but I'll go ahead and try that suggestion again. I'll
attach
> the result as well. When comparing with VGTYP - an example that
worked - I
> noticed the "lat, lon" ordering as opposed to "lon, lat", which is
what
> prompted me to go that route. I'll test this carefully. Thanks
again.
>
> On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Ed,
> >
> > I pulled that file and took a look. I've attached the image that
> > plot_data_plane generates with the following command:
> >
> > *plot_data_plane aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc
> >
> > aqm.t06z.metcro2d_DLUSE.00.ps
<http://aqm.t06z.metcro2d_DLUSE.00.ps>
> > 'name="DLUSE"; level="(*,*)";'*
> >
> >
> >
> > So there's clearly a problem here! Looking at that file with
ncdump, I
> see
> > that this file is actually setup to follow MET's internal NetCDF
> > conventions rather the the NetCDF Climate-Forecast (CF)
convention.
> >
> >
> > *ncdump -h aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>*
> >
> >
> >
> > Even though we recommend the use of the CF-convention, the MET
tools
> > actually don't write to that convention themselves. But we should
> probably
> > transition to it! For now, the tools write to a pretty rigid
internal
> > format where the dimensions are always ordered (lat, lon). Your
data is
> > ordered (lon, lat) and I strongly suspect that's the reason for
this
> > behavior. Please try rearranging to write it as (lat, lon) and
then run
> > plot_data_plane on the result to see if it's fixed.
> >
> >
> > Hope that helps.
> >
> >
> > Thanks,
> > John
> >
> >
> >
> > On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Ed,
> > >
> > > Thanks for sending your sample commands and a description of the
> > behavior.
> > > I see that you've prepared a NetCDF file that follows the
> CF-convention,
> > > but are getting some odd output from gen_vx_mask. I suspect that
the
> odd
> > > output from gen_vx_mask is a red herring, and that the real
issue is in
> > > reading the data from that input NetCDF file.
> > >
> > > Whenever, getting started with a new dataset in MET, such as the
> > > CF-compliant NetCDF file, it's always a good idea to plot the
data
> using
> > > plot_data_plane. I suspect we'll find that plot_data_plane
creates the
> > same
> > > sort of odd plot when reading data directly from that input
file. And
> > > that's most likely due to either an issue in how the data is
stored in
> > that
> > > NetCDF file or perhaps some bug in the MET software.
> > >
> > > I'll go log on to WCOSS, retrieve that file, and do some testing
with
> it.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> > >>
> > >> Originally, I had processed grib files with vegetation type. I
used
> > this
> > >> record here: 880:Vegetation Type:Integer(0-13)
> > >> (instant):lambert:surface:level 0:fcst time 3 hrs:from
202008181800
> > >>
> > >> You can see the shape of that is this: >>>
> > >> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> > >> *(288, 468)*
> > >>
> > >> You can also see that upon processing this file through
gen_vx_mask in
> > the
> > >> same way as done above produces the same sized matrix - not
> transposed.
> > >>
> > >> Dataset('land_mask_15.nc').variables
> > >> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> > >> float32 lat(lat, lon)
> > >> long_name: latitude
> > >> units: degrees_north
> > >> standard_name: latitude
> > >> unlimited dimensions:
> > >> current shape = (288, 468)
> > >> filling on, default _FillValue of 9.969209968386869e+36 used
> > >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> > >> float32 lon(lat, lon)
> > >> long_name: longitude
> > >> units: degrees_east
> > >> standard_name: longitude
> > >> unlimited dimensions:
> > >> current shape = (288, 468)
> > >> filling on, default _FillValue of 9.969209968386869e+36 used
> > >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> > >> float32 data_mask(lat, lon)
> > >> long_name: data_mask masking region
> > >> _FillValue: -9999.0
> > >> mask_type: data==15
> > >> unlimited dimensions:
> > >> *current shape = (288, 468)*
> > >> filling on)])
> > >>
> > >> Moreover, given the mask_type, it identified points having 15
as 1 and
> > >> others as 0. Therefore I have a 0/1 dataset.
> > >>
> > >> This is different than when I process the newly created netcdf
file.
> > The
> > >> netcdf file that was created has the Urban field with a shape
of
> > (265,442)
> > >>
> > >> When processing it through gen_vx_mask, it gets transposed.
>>>
> > >> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> > >> *(442, 265)*.
> > >>
> > >> So in the original case I have no change to the dimensionality
of the
> > >> dataset, but in the latter case I do. This happens during the
> > gen_vx_mask
> > >> step, and I suspect that this may somehow be related to my
issue. I
> > don't
> > >> know if there's anything else I can add that will help, but if
I think
> > of
> > >> something I'll forward it along
> > >>
> > >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA
Affiliate <
> > >> edward.strobach at noaa.gov> wrote:
> > >>
> > >> > Transposing it didn't seem to do anything either. Here's the
> > >> gen_vx_mask
> > >> > output. It looks okay at first glance:
> > >> >
> > >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> > aqm.t06z.metcro2d.00.nc
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> > aqm.t06z.metcro2d.00.nc
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > >> > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh
eq1 -name
> > >> Urban
> > >> > DEBUG 1: Input Grid:
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> > aqm.t06z.metcro2d.00.nc
> > >> > DEBUG 1: Mask File:
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> > aqm.t06z.metcro2d.00.nc
> > >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> > >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> > >> > DEBUG 1: Output File:
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > >> >
> > >> > I'm also attaching the header content of the file:
> > >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > >> > <class 'netCDF4._netCDF4.Dataset'>
> > >> > root group (NETCDF4 data model, file format HDF5):
> > >> > FileOrigins: NA
> > >> > MET_version: V9.1
> > >> > MET_tool: gen_vx_mask
> > >> > Projection: Lambert Conformal
> > >> > hemisphere: N
> > >> > accum: 00
> > >> > scale_lat_1: 33.000000
> > >> > scale_lat_2: 45.000000
> > >> > lat_pin: 21.821000
> > >> > lon_pin: -120.628000
> > >> > x_pin: 0.000000
> > >> > y_pin: 0.000000
> > >> > lon_orient: -97.000000
> > >> > d_km: 12.000000
> > >> > r_km: 6371.200000
> > >> > nx: 265
> > >> > ny: 442 grid_points
> > >> > dimensions(sizes): lon(265), lat(442)
> > >> > variables(dimensions): float32 LAT(lon,lat), float32
> LON(lon,lat),
> > >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
PBL(lon,lat),
> > >> float32
> > >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
WSTAR(lon,lat),
> > >> float32
> > >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat),
float32
> > >> > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
float32
> > >> > Q2(lon,lat)
> > >> > groups:
> > >> >
> > >> > Nothing here seems troubling. You can see that
DLUSE(lon,lat).shape
> ==
> > >> > (265,442)
> > >> >
> > >> > The values in the matrix are of type float, not integer; but,
I
> don't
> > >> > think that would be an issue.
> > >> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > >> > masked_array(
> > >> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > >> > ...,
> > >> > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > >> > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > >> > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > >> > mask=[[False, False, False, ..., False, False, False],
> > >> > [False, False, False, ..., False, False, False],
> > >> > [False, False, False, ..., False, False, False],
> > >> > ...,
> > >> > [False, False, False, ..., False, False, False],
> > >> > [False, False, False, ..., False, False, False],
> > >> > [False, False, False, ..., False, False, False]],
> > >> > fill_value=9.96921e+36,
> > >> > dtype=float32)
> > >> >
> > >> > Do I need to change the dtype to integer?
> > >> >
> > >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> >>
> > >> >> Greetings,
> > >> >>
> > >> >> This message has been automatically generated in response to
the
> > >> creation
> > >> >> of a trouble ticket regarding:
> > >> >> "mask issues",
> > >> >> a summary of which appears below.
> > >> >>
> > >> >> There is no need to reply to this message right now. Your
ticket
> has
> > >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> > >> >>
> > >> >> Please include the string:
> > >> >>
> > >> >> [rt.rap.ucar.edu #96397]
> > >> >>
> > >> >> in the subject line of all future correspondence about this
issue.
> To
> > >> do
> > >> >> so, you may reply to this message.
> > >> >>
> > >> >> For more information, please see:
> > >> >>
> > >> >> MET Online Tutorial:
> > >> >>
> > >>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > >> >>
> > >> >> MET Users Guide:
> > >> >> https://www.dtcenter.org/met/users/docs/overview.php
> > >> >>
> > >> >> MET FAQs:
> > >> >> https://www.dtcenter.org/met/users/support/faqs/index.php
> > >> >>
> > >> >> MET-Help Email Archive:
> > >> >> http://mailman.ucar.edu/pipermail/met_help
> > >> >>
> > >> >> Thank you,
> > >> >> met_help at ucar.edu
> > >> >>
> > >> >>
> > >>
> >
-------------------------------------------------------------------------
> > >> >> GOod evening,
> > >> >>
> > >> >> I thought I understood how to process the data into masked
regions,
> > but
> > >> >> apparently not. I'm using data that I processed into a
netcdf file
> > >> that
> > >> >> is
> > >> >> now CF compatible. I'm processing the DLUSE field from this
> > directory:
> > >> >>
> > >> >>
> > >> >>
> > >>
> >
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> >> aqm.t06z.metcro2d.00.nc
> > >> >> <http://aqm.t06z.metcro2d.00.nc>*
> > >> >>
> > >> >> I then run this line, which is inside a loop within a script
that
> I'm
> > >> >> using. The command looks like this:
> > >> >> *gen_vx_mask
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> >> aqm.t06z.metcro2d.00.nc
> > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> >> aqm.t06z.metcro2d.00.nc
> > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > >> >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh
eq1
> -name
> > >> >> Urban*
> > >> >>
> > >> >> I determined that something was wrong when much of my masked
land
> > >> plots -
> > >> >> all 24 of them - looked more or less the same. When
inspecting
> > Urban,
> > >> I
> > >> >> found that there were 588 matched pairs! Since I realize
that we
> > don't
> > >> >> live on Coruscant, I decided to plot the data and this is
what it
> > >> looked
> > >> >> like (see attached: fig 1). The original DLUSE field looks
like
> what
> > >> you
> > >> >> see in the second attachment. I'm pretty sure that the
gen_vx_mask
> > was
> > >> >> utilized correctly here. I have the file that I'm using,
the file
> > that
> > >> >> I'm
> > >> >> creating to store the result, the type (data), the mask
field
> (DLUSE
> > at
> > >> >> the
> > >> >> surface), the threshold (set equal to 1 for urban), and the
name of
> > the
> > >> >> field in the file (Urban). One of the things I noticed was
that
> the
> > >> >> output
> > >> >> from the netcdf file that I generated is transposed from the
> > original.
> > >> >> The
> > >> >> matrix was originally 265 X 442. However, in the file that
was
> > >> created it
> > >> >> is 442X265. I'm not sure what's going on here at all to be
honest.
> > I
> > >> >> thought it would be as simple as running the command with
the above
> > >> >> settings. I may try to create netcdf files with the matrix
being
> > >> 442X265.
> > >> >> Something clearly is off here.
> > >> >>
> > >> >> --
> > >> >> Edward Strobach
> > >> >> EMC/NCEP/NWS/
> > >> >> IMSG Contractor
> > >> >> Cubicle#: 2029
> > >> >> 301-683-3717
> > >> >>
> > >> >>
> > >> >
> > >> > --
> > >> > Edward Strobach
> > >> > EMC/NCEP/NWS/
> > >> > IMSG Contractor
> > >> > Cubicle#: 2029
> > >> > 301-683-3717
> > >> >
> > >>
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > >>
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Sat Aug 22 09:42:10 2020
Hi John,
Rewriting the files to follow MET's convention helped produce a normal
looking plot (see attached). That was made possible by writing my
files in
this way:
Dataset('land_mask_Urban.nc')
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):
FileOrigins: File
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc
generated 20200822_152206 UTC on host m72a1 by the MET gen_vx_mask
tool
MET_version: V9.1
MET_tool: gen_vx_mask
Projection: Lambert Conformal
hemisphere: N
scale_lat_1: 33.000000
scale_lat_2: 45.000000
lat_pin: 21.821000
lon_pin: -120.628000
x_pin: 0.000000
y_pin: 0.000000
lon_orient: -97.000000
d_km: 12.000000
r_km: 6371.200000
* nx: 442 ny: 265 grid_points dimensions(sizes): lat(265),
lon(442) variables(dimensions): float32 lat(lat,lon), float32
lon(lat,lon), float32 Urban(lat,lon)*
groups:
Following your earlier suggestion (e.g. gen_vx_mask
wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212 *land_mask.nc
<http://land_mask.nc>*
-type data -mask_field 'name="LAND"; level="Z0";' -thresh eq1), I went
ahead and executed something similar for Urban type:
gen_vx_mask
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
aqm.t06z.metcro2d.00.nc
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
aqm.t06z.metcro2d.00.nc
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
-type
data -name Urban -mask_field 'name="DLUSE"; level="Z0";' -thresh 'eq1'
However, as you can see from the map, I retain all values rather than
getting 0 for non-urban and 1 for urban. I don't see anything that's
really different between these two examples other than I specify the
written name of the mask field as Urban. I've included - and removing
-
the single quotes about eq1, but the result ended up being the same.
I also don't see anything alarmingly different when comparing the
header
content from this file with the examples that had worked with VGTYP.
I'll continue trying a couple of other things if some ideas come up.
If I
make traction, I'll let you know. Take care and have a good weekend.
On Fri, Aug 21, 2020 at 4:46 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Ed,
>
> Sounds good. Please let me know if reversing (lon,lat) to (lat,lon)
in the
> NetCDF files solves the problem.
>
> Have a good weekend.
>
> Thanks,
> John
>
> On Fri, Aug 21, 2020 at 10:51 AM Edward Strobach - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> >
> > Hi John,
> >
> > Your plot looks just like mine. Thanks for confirming. I do
believe I
> > tried that, but I'll go ahead and try that suggestion again. I'll
attach
> > the result as well. When comparing with VGTYP - an example that
worked
> - I
> > noticed the "lat, lon" ordering as opposed to "lon, lat", which is
what
> > prompted me to go that route. I'll test this carefully. Thanks
again.
> >
> > On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Ed,
> > >
> > > I pulled that file and took a look. I've attached the image that
> > > plot_data_plane generates with the following command:
> > >
> > > *plot_data_plane aqm.t06z.metcro2d.00.nc <
> http://aqm.t06z.metcro2d.00.nc
> > >
> > > aqm.t06z.metcro2d_DLUSE.00.ps
<http://aqm.t06z.metcro2d_DLUSE.00.ps>
> > > 'name="DLUSE"; level="(*,*)";'*
> > >
> > >
> > >
> > > So there's clearly a problem here! Looking at that file with
ncdump, I
> > see
> > > that this file is actually setup to follow MET's internal NetCDF
> > > conventions rather the the NetCDF Climate-Forecast (CF)
convention.
> > >
> > >
> > > *ncdump -h aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>*
> > >
> > >
> > >
> > > Even though we recommend the use of the CF-convention, the MET
tools
> > > actually don't write to that convention themselves. But we
should
> > probably
> > > transition to it! For now, the tools write to a pretty rigid
internal
> > > format where the dimensions are always ordered (lat, lon). Your
data is
> > > ordered (lon, lat) and I strongly suspect that's the reason for
this
> > > behavior. Please try rearranging to write it as (lat, lon) and
then run
> > > plot_data_plane on the result to see if it's fixed.
> > >
> > >
> > > Hope that helps.
> > >
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > Ed,
> > > >
> > > > Thanks for sending your sample commands and a description of
the
> > > behavior.
> > > > I see that you've prepared a NetCDF file that follows the
> > CF-convention,
> > > > but are getting some odd output from gen_vx_mask. I suspect
that the
> > odd
> > > > output from gen_vx_mask is a red herring, and that the real
issue is
> in
> > > > reading the data from that input NetCDF file.
> > > >
> > > > Whenever, getting started with a new dataset in MET, such as
the
> > > > CF-compliant NetCDF file, it's always a good idea to plot the
data
> > using
> > > > plot_data_plane. I suspect we'll find that plot_data_plane
creates
> the
> > > same
> > > > sort of odd plot when reading data directly from that input
file. And
> > > > that's most likely due to either an issue in how the data is
stored
> in
> > > that
> > > > NetCDF file or perhaps some bug in the MET software.
> > > >
> > > > I'll go log on to WCOSS, retrieve that file, and do some
testing with
> > it.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA
Affiliate via
> > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397
>
> > > >>
> > > >> Originally, I had processed grib files with vegetation type.
I used
> > > this
> > > >> record here: 880:Vegetation Type:Integer(0-13)
> > > >> (instant):lambert:surface:level 0:fcst time 3 hrs:from
202008181800
> > > >>
> > > >> You can see the shape of that is this: >>>
> > > >> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> > > >> *(288, 468)*
> > > >>
> > > >> You can also see that upon processing this file through
gen_vx_mask
> in
> > > the
> > > >> same way as done above produces the same sized matrix - not
> > transposed.
> > > >>
> > > >> Dataset('land_mask_15.nc').variables
> > > >> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> > > >> float32 lat(lat, lon)
> > > >> long_name: latitude
> > > >> units: degrees_north
> > > >> standard_name: latitude
> > > >> unlimited dimensions:
> > > >> current shape = (288, 468)
> > > >> filling on, default _FillValue of 9.969209968386869e+36 used
> > > >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> > > >> float32 lon(lat, lon)
> > > >> long_name: longitude
> > > >> units: degrees_east
> > > >> standard_name: longitude
> > > >> unlimited dimensions:
> > > >> current shape = (288, 468)
> > > >> filling on, default _FillValue of 9.969209968386869e+36 used
> > > >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> > > >> float32 data_mask(lat, lon)
> > > >> long_name: data_mask masking region
> > > >> _FillValue: -9999.0
> > > >> mask_type: data==15
> > > >> unlimited dimensions:
> > > >> *current shape = (288, 468)*
> > > >> filling on)])
> > > >>
> > > >> Moreover, given the mask_type, it identified points having 15
as 1
> and
> > > >> others as 0. Therefore I have a 0/1 dataset.
> > > >>
> > > >> This is different than when I process the newly created
netcdf file.
> > > The
> > > >> netcdf file that was created has the Urban field with a shape
of
> > > (265,442)
> > > >>
> > > >> When processing it through gen_vx_mask, it gets transposed.
>>>
> > > >> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> > > >> *(442, 265)*.
> > > >>
> > > >> So in the original case I have no change to the
dimensionality of
> the
> > > >> dataset, but in the latter case I do. This happens during
the
> > > gen_vx_mask
> > > >> step, and I suspect that this may somehow be related to my
issue. I
> > > don't
> > > >> know if there's anything else I can add that will help, but
if I
> think
> > > of
> > > >> something I'll forward it along
> > > >>
> > > >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA
Affiliate <
> > > >> edward.strobach at noaa.gov> wrote:
> > > >>
> > > >> > Transposing it didn't seem to do anything either. Here's
the
> > > >> gen_vx_mask
> > > >> > output. It looks okay at first glance:
> > > >> >
> > > >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> > aqm.t06z.metcro2d.00.nc
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> > aqm.t06z.metcro2d.00.nc
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > >> > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh
eq1
> -name
> > > >> Urban
> > > >> > DEBUG 1: Input Grid:
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> > aqm.t06z.metcro2d.00.nc
> > > >> > DEBUG 1: Mask File:
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> > aqm.t06z.metcro2d.00.nc
> > > >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> > > >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> > > >> > DEBUG 1: Output File:
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > >> >
> > > >> > I'm also attaching the header content of the file:
> > > >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > > >> > <class 'netCDF4._netCDF4.Dataset'>
> > > >> > root group (NETCDF4 data model, file format HDF5):
> > > >> > FileOrigins: NA
> > > >> > MET_version: V9.1
> > > >> > MET_tool: gen_vx_mask
> > > >> > Projection: Lambert Conformal
> > > >> > hemisphere: N
> > > >> > accum: 00
> > > >> > scale_lat_1: 33.000000
> > > >> > scale_lat_2: 45.000000
> > > >> > lat_pin: 21.821000
> > > >> > lon_pin: -120.628000
> > > >> > x_pin: 0.000000
> > > >> > y_pin: 0.000000
> > > >> > lon_orient: -97.000000
> > > >> > d_km: 12.000000
> > > >> > r_km: 6371.200000
> > > >> > nx: 265
> > > >> > ny: 442 grid_points
> > > >> > dimensions(sizes): lon(265), lat(442)
> > > >> > variables(dimensions): float32 LAT(lon,lat), float32
> > LON(lon,lat),
> > > >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
> PBL(lon,lat),
> > > >> float32
> > > >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
WSTAR(lon,lat),
> > > >> float32
> > > >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32
HFX(lon,lat),
> float32
> > > >> > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
> float32
> > > >> > Q2(lon,lat)
> > > >> > groups:
> > > >> >
> > > >> > Nothing here seems troubling. You can see that
> DLUSE(lon,lat).shape
> > ==
> > > >> > (265,442)
> > > >> >
> > > >> > The values in the matrix are of type float, not integer;
but, I
> > don't
> > > >> > think that would be an issue.
> > > >> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > > >> > masked_array(
> > > >> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > >> > ...,
> > > >> > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > > >> > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > > >> > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > > >> > mask=[[False, False, False, ..., False, False, False],
> > > >> > [False, False, False, ..., False, False, False],
> > > >> > [False, False, False, ..., False, False, False],
> > > >> > ...,
> > > >> > [False, False, False, ..., False, False, False],
> > > >> > [False, False, False, ..., False, False, False],
> > > >> > [False, False, False, ..., False, False, False]],
> > > >> > fill_value=9.96921e+36,
> > > >> > dtype=float32)
> > > >> >
> > > >> > Do I need to change the dtype to integer?
> > > >> >
> > > >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> >>
> > > >> >> Greetings,
> > > >> >>
> > > >> >> This message has been automatically generated in response
to the
> > > >> creation
> > > >> >> of a trouble ticket regarding:
> > > >> >> "mask issues",
> > > >> >> a summary of which appears below.
> > > >> >>
> > > >> >> There is no need to reply to this message right now. Your
ticket
> > has
> > > >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> > > >> >>
> > > >> >> Please include the string:
> > > >> >>
> > > >> >> [rt.rap.ucar.edu #96397]
> > > >> >>
> > > >> >> in the subject line of all future correspondence about
this
> issue.
> > To
> > > >> do
> > > >> >> so, you may reply to this message.
> > > >> >>
> > > >> >> For more information, please see:
> > > >> >>
> > > >> >> MET Online Tutorial:
> > > >> >>
> > > >>
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > >> >>
> > > >> >> MET Users Guide:
> > > >> >> https://www.dtcenter.org/met/users/docs/overview.php
> > > >> >>
> > > >> >> MET FAQs:
> > > >> >>
https://www.dtcenter.org/met/users/support/faqs/index.php
> > > >> >>
> > > >> >> MET-Help Email Archive:
> > > >> >> http://mailman.ucar.edu/pipermail/met_help
> > > >> >>
> > > >> >> Thank you,
> > > >> >> met_help at ucar.edu
> > > >> >>
> > > >> >>
> > > >>
> > >
>
-------------------------------------------------------------------------
> > > >> >> GOod evening,
> > > >> >>
> > > >> >> I thought I understood how to process the data into masked
> regions,
> > > but
> > > >> >> apparently not. I'm using data that I processed into a
netcdf
> file
> > > >> that
> > > >> >> is
> > > >> >> now CF compatible. I'm processing the DLUSE field from
this
> > > directory:
> > > >> >>
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> >> aqm.t06z.metcro2d.00.nc
> > > >> >> <http://aqm.t06z.metcro2d.00.nc>*
> > > >> >>
> > > >> >> I then run this line, which is inside a loop within a
script that
> > I'm
> > > >> >> using. The command looks like this:
> > > >> >> *gen_vx_mask
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> >> aqm.t06z.metcro2d.00.nc
> > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> >> aqm.t06z.metcro2d.00.nc
> > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > >> >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh
eq1
> > -name
> > > >> >> Urban*
> > > >> >>
> > > >> >> I determined that something was wrong when much of my
masked land
> > > >> plots -
> > > >> >> all 24 of them - looked more or less the same. When
inspecting
> > > Urban,
> > > >> I
> > > >> >> found that there were 588 matched pairs! Since I realize
that we
> > > don't
> > > >> >> live on Coruscant, I decided to plot the data and this is
what it
> > > >> looked
> > > >> >> like (see attached: fig 1). The original DLUSE field
looks like
> > what
> > > >> you
> > > >> >> see in the second attachment. I'm pretty sure that the
> gen_vx_mask
> > > was
> > > >> >> utilized correctly here. I have the file that I'm using,
the
> file
> > > that
> > > >> >> I'm
> > > >> >> creating to store the result, the type (data), the mask
field
> > (DLUSE
> > > at
> > > >> >> the
> > > >> >> surface), the threshold (set equal to 1 for urban), and
the name
> of
> > > the
> > > >> >> field in the file (Urban). One of the things I noticed
was that
> > the
> > > >> >> output
> > > >> >> from the netcdf file that I generated is transposed from
the
> > > original.
> > > >> >> The
> > > >> >> matrix was originally 265 X 442. However, in the file
that was
> > > >> created it
> > > >> >> is 442X265. I'm not sure what's going on here at all to
be
> honest.
> > > I
> > > >> >> thought it would be as simple as running the command with
the
> above
> > > >> >> settings. I may try to create netcdf files with the
matrix being
> > > >> 442X265.
> > > >> >> Something clearly is off here.
> > > >> >>
> > > >> >> --
> > > >> >> Edward Strobach
> > > >> >> EMC/NCEP/NWS/
> > > >> >> IMSG Contractor
> > > >> >> Cubicle#: 2029
> > > >> >> 301-683-3717
> > > >> >>
> > > >> >>
> > > >> >
> > > >> > --
> > > >> > Edward Strobach
> > > >> > EMC/NCEP/NWS/
> > > >> > IMSG Contractor
> > > >> > Cubicle#: 2029
> > > >> > 301-683-3717
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> Edward Strobach
> > > >> EMC/NCEP/NWS/
> > > >> IMSG Contractor
> > > >> Cubicle#: 2029
> > > >> 301-683-3717
> > > >>
> > > >>
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: mask issues
From: John Halley Gotway
Time: Mon Aug 24 10:59:00 2020
Ed,
Great, I'm glad you were able to get it working as expected. Thanks
for
confirming. I'll go ahead and resolve this ticket.
John
On Sat, Aug 22, 2020 at 9:42 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>
> Hi John,
>
> Rewriting the files to follow MET's convention helped produce a
normal
> looking plot (see attached). That was made possible by writing my
files in
> this way:
> Dataset('land_mask_Urban.nc')
> <class 'netCDF4._netCDF4.Dataset'>
> root group (NETCDF4 data model, file format HDF5):
> FileOrigins: File
>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc
> generated 20200822_152206 UTC on host m72a1 by the MET gen_vx_mask
tool
> MET_version: V9.1
> MET_tool: gen_vx_mask
> Projection: Lambert Conformal
> hemisphere: N
> scale_lat_1: 33.000000
> scale_lat_2: 45.000000
> lat_pin: 21.821000
> lon_pin: -120.628000
> x_pin: 0.000000
> y_pin: 0.000000
> lon_orient: -97.000000
> d_km: 12.000000
> r_km: 6371.200000
>
>
>
> * nx: 442 ny: 265 grid_points dimensions(sizes): lat(265),
> lon(442) variables(dimensions): float32 lat(lat,lon), float32
> lon(lat,lon), float32 Urban(lat,lon)*
> groups:
>
> Following your earlier suggestion (e.g. gen_vx_mask
> wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212 *land_mask.nc
> <http://land_mask.nc>*
> -type data -mask_field 'name="LAND"; level="Z0";' -thresh eq1), I
went
> ahead and executed something similar for Urban type:
>
> gen_vx_mask
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> aqm.t06z.metcro2d.00.nc
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> aqm.t06z.metcro2d.00.nc
>
>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
> -type
> data -name Urban -mask_field 'name="DLUSE"; level="Z0";' -thresh
'eq1'
>
> However, as you can see from the map, I retain all values rather
than
> getting 0 for non-urban and 1 for urban. I don't see anything
that's
> really different between these two examples other than I specify the
> written name of the mask field as Urban. I've included - and
removing -
> the single quotes about eq1, but the result ended up being the same.
>
>
>
> I also don't see anything alarmingly different when comparing the
header
> content from this file with the examples that had worked with VGTYP.
>
> I'll continue trying a couple of other things if some ideas come up.
If I
> make traction, I'll let you know. Take care and have a good
weekend.
>
>
>
> On Fri, Aug 21, 2020 at 4:46 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Ed,
> >
> > Sounds good. Please let me know if reversing (lon,lat) to
(lat,lon) in
> the
> > NetCDF files solves the problem.
> >
> > Have a good weekend.
> >
> > Thanks,
> > John
> >
> > On Fri, Aug 21, 2020 at 10:51 AM Edward Strobach - NOAA Affiliate
via RT
> <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> > >
> > > Hi John,
> > >
> > > Your plot looks just like mine. Thanks for confirming. I do
believe I
> > > tried that, but I'll go ahead and try that suggestion again.
I'll
> attach
> > > the result as well. When comparing with VGTYP - an example that
worked
> > - I
> > > noticed the "lat, lon" ordering as opposed to "lon, lat", which
is what
> > > prompted me to go that route. I'll test this carefully. Thanks
again.
> > >
> > > On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Ed,
> > > >
> > > > I pulled that file and took a look. I've attached the image
that
> > > > plot_data_plane generates with the following command:
> > > >
> > > > *plot_data_plane aqm.t06z.metcro2d.00.nc <
> > http://aqm.t06z.metcro2d.00.nc
> > > >
> > > > aqm.t06z.metcro2d_DLUSE.00.ps
<http://aqm.t06z.metcro2d_DLUSE.00.ps>
> > > > 'name="DLUSE"; level="(*,*)";'*
> > > >
> > > >
> > > >
> > > > So there's clearly a problem here! Looking at that file with
ncdump,
> I
> > > see
> > > > that this file is actually setup to follow MET's internal
NetCDF
> > > > conventions rather the the NetCDF Climate-Forecast (CF)
convention.
> > > >
> > > >
> > > > *ncdump -h aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>*
> > > >
> > > >
> > > >
> > > > Even though we recommend the use of the CF-convention, the MET
tools
> > > > actually don't write to that convention themselves. But we
should
> > > probably
> > > > transition to it! For now, the tools write to a pretty rigid
internal
> > > > format where the dimensions are always ordered (lat, lon).
Your data
> is
> > > > ordered (lon, lat) and I strongly suspect that's the reason
for this
> > > > behavior. Please try rearranging to write it as (lat, lon) and
then
> run
> > > > plot_data_plane on the result to see if it's fixed.
> > > >
> > > >
> > > > Hope that helps.
> > > >
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > > On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway
<johnhg at ucar.edu
> >
> > > > wrote:
> > > >
> > > > > Ed,
> > > > >
> > > > > Thanks for sending your sample commands and a description of
the
> > > > behavior.
> > > > > I see that you've prepared a NetCDF file that follows the
> > > CF-convention,
> > > > > but are getting some odd output from gen_vx_mask. I suspect
that
> the
> > > odd
> > > > > output from gen_vx_mask is a red herring, and that the real
issue
> is
> > in
> > > > > reading the data from that input NetCDF file.
> > > > >
> > > > > Whenever, getting started with a new dataset in MET, such as
the
> > > > > CF-compliant NetCDF file, it's always a good idea to plot
the data
> > > using
> > > > > plot_data_plane. I suspect we'll find that plot_data_plane
creates
> > the
> > > > same
> > > > > sort of odd plot when reading data directly from that input
file.
> And
> > > > > that's most likely due to either an issue in how the data is
stored
> > in
> > > > that
> > > > > NetCDF file or perhaps some bug in the MET software.
> > > > >
> > > > > I'll go log on to WCOSS, retrieve that file, and do some
testing
> with
> > > it.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA
Affiliate
> via
> > > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> > > > >>
> > > > >> Originally, I had processed grib files with vegetation
type. I
> used
> > > > this
> > > > >> record here: 880:Vegetation Type:Integer(0-13)
> > > > >> (instant):lambert:surface:level 0:fcst time 3 hrs:from
> 202008181800
> > > > >>
> > > > >> You can see the shape of that is this: >>>
> > > > >>
pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> > > > >> *(288, 468)*
> > > > >>
> > > > >> You can also see that upon processing this file through
> gen_vx_mask
> > in
> > > > the
> > > > >> same way as done above produces the same sized matrix - not
> > > transposed.
> > > > >>
> > > > >> Dataset('land_mask_15.nc').variables
> > > > >> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> > > > >> float32 lat(lat, lon)
> > > > >> long_name: latitude
> > > > >> units: degrees_north
> > > > >> standard_name: latitude
> > > > >> unlimited dimensions:
> > > > >> current shape = (288, 468)
> > > > >> filling on, default _FillValue of 9.969209968386869e+36
used
> > > > >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> > > > >> float32 lon(lat, lon)
> > > > >> long_name: longitude
> > > > >> units: degrees_east
> > > > >> standard_name: longitude
> > > > >> unlimited dimensions:
> > > > >> current shape = (288, 468)
> > > > >> filling on, default _FillValue of 9.969209968386869e+36
used
> > > > >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> > > > >> float32 data_mask(lat, lon)
> > > > >> long_name: data_mask masking region
> > > > >> _FillValue: -9999.0
> > > > >> mask_type: data==15
> > > > >> unlimited dimensions:
> > > > >> *current shape = (288, 468)*
> > > > >> filling on)])
> > > > >>
> > > > >> Moreover, given the mask_type, it identified points having
15 as 1
> > and
> > > > >> others as 0. Therefore I have a 0/1 dataset.
> > > > >>
> > > > >> This is different than when I process the newly created
netcdf
> file.
> > > > The
> > > > >> netcdf file that was created has the Urban field with a
shape of
> > > > (265,442)
> > > > >>
> > > > >> When processing it through gen_vx_mask, it gets transposed.
>>>
> > > > >> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> > > > >> *(442, 265)*.
> > > > >>
> > > > >> So in the original case I have no change to the
dimensionality of
> > the
> > > > >> dataset, but in the latter case I do. This happens during
the
> > > > gen_vx_mask
> > > > >> step, and I suspect that this may somehow be related to my
> issue. I
> > > > don't
> > > > >> know if there's anything else I can add that will help, but
if I
> > think
> > > > of
> > > > >> something I'll forward it along
> > > > >>
> > > > >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA
Affiliate <
> > > > >> edward.strobach at noaa.gov> wrote:
> > > > >>
> > > > >> > Transposing it didn't seem to do anything either. Here's
the
> > > > >> gen_vx_mask
> > > > >> > output. It looks okay at first glance:
> > > > >> >
> > > > >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > >> > aqm.t06z.metcro2d.00.nc
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > >> > aqm.t06z.metcro2d.00.nc
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > >> > -type data -mask_field 'name="DLUSE"; level="Z0";'
-thresh eq1
> > -name
> > > > >> Urban
> > > > >> > DEBUG 1: Input Grid:
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > >> > aqm.t06z.metcro2d.00.nc
> > > > >> > DEBUG 1: Mask File:
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > >> > aqm.t06z.metcro2d.00.nc
> > > > >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> > > > >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> > > > >> > DEBUG 1: Output File:
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > >> >
> > > > >> > I'm also attaching the header content of the file:
> > > > >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > > > >> > <class 'netCDF4._netCDF4.Dataset'>
> > > > >> > root group (NETCDF4 data model, file format HDF5):
> > > > >> > FileOrigins: NA
> > > > >> > MET_version: V9.1
> > > > >> > MET_tool: gen_vx_mask
> > > > >> > Projection: Lambert Conformal
> > > > >> > hemisphere: N
> > > > >> > accum: 00
> > > > >> > scale_lat_1: 33.000000
> > > > >> > scale_lat_2: 45.000000
> > > > >> > lat_pin: 21.821000
> > > > >> > lon_pin: -120.628000
> > > > >> > x_pin: 0.000000
> > > > >> > y_pin: 0.000000
> > > > >> > lon_orient: -97.000000
> > > > >> > d_km: 12.000000
> > > > >> > r_km: 6371.200000
> > > > >> > nx: 265
> > > > >> > ny: 442 grid_points
> > > > >> > dimensions(sizes): lon(265), lat(442)
> > > > >> > variables(dimensions): float32 LAT(lon,lat), float32
> > > LON(lon,lat),
> > > > >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
> > PBL(lon,lat),
> > > > >> float32
> > > > >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
> WSTAR(lon,lat),
> > > > >> float32
> > > > >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32
HFX(lon,lat),
> > float32
> > > > >> > GLW(lon,lat), float32 GSW(lon,lat), float32
RGRND(lon,lat),
> > float32
> > > > >> > Q2(lon,lat)
> > > > >> > groups:
> > > > >> >
> > > > >> > Nothing here seems troubling. You can see that
> > DLUSE(lon,lat).shape
> > > ==
> > > > >> > (265,442)
> > > > >> >
> > > > >> > The values in the matrix are of type float, not integer;
but, I
> > > don't
> > > > >> > think that would be an issue.
> > > > >> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > > > >> > masked_array(
> > > > >> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > >> > ...,
> > > > >> > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > > > >> > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > > > >> > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > > > >> > mask=[[False, False, False, ..., False, False, False],
> > > > >> > [False, False, False, ..., False, False, False],
> > > > >> > [False, False, False, ..., False, False, False],
> > > > >> > ...,
> > > > >> > [False, False, False, ..., False, False, False],
> > > > >> > [False, False, False, ..., False, False, False],
> > > > >> > [False, False, False, ..., False, False, False]],
> > > > >> > fill_value=9.96921e+36,
> > > > >> > dtype=float32)
> > > > >> >
> > > > >> > Do I need to change the dtype to integer?
> > > > >> >
> > > > >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT
<
> > > > >> > met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> >>
> > > > >> >> Greetings,
> > > > >> >>
> > > > >> >> This message has been automatically generated in
response to
> the
> > > > >> creation
> > > > >> >> of a trouble ticket regarding:
> > > > >> >> "mask issues",
> > > > >> >> a summary of which appears below.
> > > > >> >>
> > > > >> >> There is no need to reply to this message right now.
Your
> ticket
> > > has
> > > > >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> > > > >> >>
> > > > >> >> Please include the string:
> > > > >> >>
> > > > >> >> [rt.rap.ucar.edu #96397]
> > > > >> >>
> > > > >> >> in the subject line of all future correspondence about
this
> > issue.
> > > To
> > > > >> do
> > > > >> >> so, you may reply to this message.
> > > > >> >>
> > > > >> >> For more information, please see:
> > > > >> >>
> > > > >> >> MET Online Tutorial:
> > > > >> >>
> > > > >>
> >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > >> >>
> > > > >> >> MET Users Guide:
> > > > >> >> https://www.dtcenter.org/met/users/docs/overview.php
> > > > >> >>
> > > > >> >> MET FAQs:
> > > > >> >>
https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > >> >>
> > > > >> >> MET-Help Email Archive:
> > > > >> >> http://mailman.ucar.edu/pipermail/met_help
> > > > >> >>
> > > > >> >> Thank you,
> > > > >> >> met_help at ucar.edu
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> >
-------------------------------------------------------------------------
> > > > >> >> GOod evening,
> > > > >> >>
> > > > >> >> I thought I understood how to process the data into
masked
> > regions,
> > > > but
> > > > >> >> apparently not. I'm using data that I processed into a
netcdf
> > file
> > > > >> that
> > > > >> >> is
> > > > >> >> now CF compatible. I'm processing the DLUSE field from
this
> > > > directory:
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > >> >> <http://aqm.t06z.metcro2d.00.nc>*
> > > > >> >>
> > > > >> >> I then run this line, which is inside a loop within a
script
> that
> > > I'm
> > > > >> >> using. The command looks like this:
> > > > >> >> *gen_vx_mask
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > >> >> -type data -mask_field 'name="DLUSE"; level="Z0";'
-thresh eq1
> > > -name
> > > > >> >> Urban*
> > > > >> >>
> > > > >> >> I determined that something was wrong when much of my
masked
> land
> > > > >> plots -
> > > > >> >> all 24 of them - looked more or less the same. When
inspecting
> > > > Urban,
> > > > >> I
> > > > >> >> found that there were 588 matched pairs! Since I
realize that
> we
> > > > don't
> > > > >> >> live on Coruscant, I decided to plot the data and this
is what
> it
> > > > >> looked
> > > > >> >> like (see attached: fig 1). The original DLUSE field
looks
> like
> > > what
> > > > >> you
> > > > >> >> see in the second attachment. I'm pretty sure that the
> > gen_vx_mask
> > > > was
> > > > >> >> utilized correctly here. I have the file that I'm
using, the
> > file
> > > > that
> > > > >> >> I'm
> > > > >> >> creating to store the result, the type (data), the mask
field
> > > (DLUSE
> > > > at
> > > > >> >> the
> > > > >> >> surface), the threshold (set equal to 1 for urban), and
the
> name
> > of
> > > > the
> > > > >> >> field in the file (Urban). One of the things I noticed
was
> that
> > > the
> > > > >> >> output
> > > > >> >> from the netcdf file that I generated is transposed from
the
> > > > original.
> > > > >> >> The
> > > > >> >> matrix was originally 265 X 442. However, in the file
that was
> > > > >> created it
> > > > >> >> is 442X265. I'm not sure what's going on here at all to
be
> > honest.
> > > > I
> > > > >> >> thought it would be as simple as running the command
with the
> > above
> > > > >> >> settings. I may try to create netcdf files with the
matrix
> being
> > > > >> 442X265.
> > > > >> >> Something clearly is off here.
> > > > >> >>
> > > > >> >> --
> > > > >> >> Edward Strobach
> > > > >> >> EMC/NCEP/NWS/
> > > > >> >> IMSG Contractor
> > > > >> >> Cubicle#: 2029
> > > > >> >> 301-683-3717
> > > > >> >>
> > > > >> >>
> > > > >> >
> > > > >> > --
> > > > >> > Edward Strobach
> > > > >> > EMC/NCEP/NWS/
> > > > >> > IMSG Contractor
> > > > >> > Cubicle#: 2029
> > > > >> > 301-683-3717
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Edward Strobach
> > > > >> EMC/NCEP/NWS/
> > > > >> IMSG Contractor
> > > > >> Cubicle#: 2029
> > > > >> 301-683-3717
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Mon Aug 24 11:05:04 2020
Hi John,
I guess that part is resolved, but I'm not sure why it seems to
produce the
output from the field itself. Copying something from the previous
email..
Following your earlier suggestion (e.g. gen_vx_mask
wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212 *land_mask.nc
<http://land_mask.nc>*
-type data -mask_field 'name="LAND"; level="Z0";' -thresh eq1), I went
ahead and executed something similar for Urban type:
gen_vx_mask
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
aqm.t06z.metcro2d.00.nc
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
aqm.t06z.metcro2d.00.nc
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
-type
data -name Urban -mask_field 'name="DLUSE"; level="Z0";' -thresh 'eq1'
Here, I compared something that worked at assigning a 1 to vegetation
of
type 1 (-thresh eq1 for top command) with what I did to isolate urban
(also
-thresh -eq1 for bottom command). Instead of getting 0's and 1's for
regions coinciding with masked threshold values, I get 1 through 24,
with
the exact output as the input. I thought setting thresh to 1 would
have
worked. Both commands look pretty identical to me, so I'm not sure
why
things would have happened differently. I find that to be an
unexpected
result
On Mon, Aug 24, 2020 at 12:59 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Ed,
>
> Great, I'm glad you were able to get it working as expected. Thanks
for
> confirming. I'll go ahead and resolve this ticket.
>
> John
>
> On Sat, Aug 22, 2020 at 9:42 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> >
> > Hi John,
> >
> > Rewriting the files to follow MET's convention helped produce a
normal
> > looking plot (see attached). That was made possible by writing my
files
> in
> > this way:
> > Dataset('land_mask_Urban.nc')
> > <class 'netCDF4._netCDF4.Dataset'>
> > root group (NETCDF4 data model, file format HDF5):
> > FileOrigins: File
> >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc
> > generated 20200822_152206 UTC on host m72a1 by the MET gen_vx_mask
tool
> > MET_version: V9.1
> > MET_tool: gen_vx_mask
> > Projection: Lambert Conformal
> > hemisphere: N
> > scale_lat_1: 33.000000
> > scale_lat_2: 45.000000
> > lat_pin: 21.821000
> > lon_pin: -120.628000
> > x_pin: 0.000000
> > y_pin: 0.000000
> > lon_orient: -97.000000
> > d_km: 12.000000
> > r_km: 6371.200000
> >
> >
> >
> > * nx: 442 ny: 265 grid_points dimensions(sizes):
lat(265),
> > lon(442) variables(dimensions): float32 lat(lat,lon), float32
> > lon(lat,lon), float32 Urban(lat,lon)*
> > groups:
> >
> > Following your earlier suggestion (e.g. gen_vx_mask
> > wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212 *land_mask.nc
> > <http://land_mask.nc>*
> > -type data -mask_field 'name="LAND"; level="Z0";' -thresh eq1), I
went
> > ahead and executed something similar for Urban type:
> >
> > gen_vx_mask
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> > aqm.t06z.metcro2d.00.nc
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> > aqm.t06z.metcro2d.00.nc
> >
> >
>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
> > -type
> > data -name Urban -mask_field 'name="DLUSE"; level="Z0";' -thresh
'eq1'
> >
> > However, as you can see from the map, I retain all values rather
than
> > getting 0 for non-urban and 1 for urban. I don't see anything
that's
> > really different between these two examples other than I specify
the
> > written name of the mask field as Urban. I've included - and
removing -
> > the single quotes about eq1, but the result ended up being the
same.
> >
> >
> >
> > I also don't see anything alarmingly different when comparing the
header
> > content from this file with the examples that had worked with
VGTYP.
> >
> > I'll continue trying a couple of other things if some ideas come
up. If I
> > make traction, I'll let you know. Take care and have a good
weekend.
> >
> >
> >
> > On Fri, Aug 21, 2020 at 4:46 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Ed,
> > >
> > > Sounds good. Please let me know if reversing (lon,lat) to
(lat,lon) in
> > the
> > > NetCDF files solves the problem.
> > >
> > > Have a good weekend.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Aug 21, 2020 at 10:51 AM Edward Strobach - NOAA
Affiliate via
> RT
> > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397
>
> > > >
> > > > Hi John,
> > > >
> > > > Your plot looks just like mine. Thanks for confirming. I do
> believe I
> > > > tried that, but I'll go ahead and try that suggestion again.
I'll
> > attach
> > > > the result as well. When comparing with VGTYP - an example
that
> worked
> > > - I
> > > > noticed the "lat, lon" ordering as opposed to "lon, lat",
which is
> what
> > > > prompted me to go that route. I'll test this carefully.
Thanks
> again.
> > > >
> > > > On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Ed,
> > > > >
> > > > > I pulled that file and took a look. I've attached the image
that
> > > > > plot_data_plane generates with the following command:
> > > > >
> > > > > *plot_data_plane aqm.t06z.metcro2d.00.nc <
> > > http://aqm.t06z.metcro2d.00.nc
> > > > >
> > > > > aqm.t06z.metcro2d_DLUSE.00.ps <
> http://aqm.t06z.metcro2d_DLUSE.00.ps>
> > > > > 'name="DLUSE"; level="(*,*)";'*
> > > > >
> > > > >
> > > > >
> > > > > So there's clearly a problem here! Looking at that file with
> ncdump,
> > I
> > > > see
> > > > > that this file is actually setup to follow MET's internal
NetCDF
> > > > > conventions rather the the NetCDF Climate-Forecast (CF)
convention.
> > > > >
> > > > >
> > > > > *ncdump -h aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc
> >*
> > > > >
> > > > >
> > > > >
> > > > > Even though we recommend the use of the CF-convention, the
MET
> tools
> > > > > actually don't write to that convention themselves. But we
should
> > > > probably
> > > > > transition to it! For now, the tools write to a pretty rigid
> internal
> > > > > format where the dimensions are always ordered (lat, lon).
Your
> data
> > is
> > > > > ordered (lon, lat) and I strongly suspect that's the reason
for
> this
> > > > > behavior. Please try rearranging to write it as (lat, lon)
and then
> > run
> > > > > plot_data_plane on the result to see if it's fixed.
> > > > >
> > > > >
> > > > > Hope that helps.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway <
> johnhg at ucar.edu
> > >
> > > > > wrote:
> > > > >
> > > > > > Ed,
> > > > > >
> > > > > > Thanks for sending your sample commands and a description
of the
> > > > > behavior.
> > > > > > I see that you've prepared a NetCDF file that follows the
> > > > CF-convention,
> > > > > > but are getting some odd output from gen_vx_mask. I
suspect that
> > the
> > > > odd
> > > > > > output from gen_vx_mask is a red herring, and that the
real issue
> > is
> > > in
> > > > > > reading the data from that input NetCDF file.
> > > > > >
> > > > > > Whenever, getting started with a new dataset in MET, such
as the
> > > > > > CF-compliant NetCDF file, it's always a good idea to plot
the
> data
> > > > using
> > > > > > plot_data_plane. I suspect we'll find that plot_data_plane
> creates
> > > the
> > > > > same
> > > > > > sort of odd plot when reading data directly from that
input file.
> > And
> > > > > > that's most likely due to either an issue in how the data
is
> stored
> > > in
> > > > > that
> > > > > > NetCDF file or perhaps some bug in the MET software.
> > > > > >
> > > > > > I'll go log on to WCOSS, retrieve that file, and do some
testing
> > with
> > > > it.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA
Affiliate
> > via
> > > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >>
> > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> > > > > >>
> > > > > >> Originally, I had processed grib files with vegetation
type. I
> > used
> > > > > this
> > > > > >> record here: 880:Vegetation Type:Integer(0-13)
> > > > > >> (instant):lambert:surface:level 0:fcst time 3 hrs:from
> > 202008181800
> > > > > >>
> > > > > >> You can see the shape of that is this: >>>
> > > > > >>
pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> > > > > >> *(288, 468)*
> > > > > >>
> > > > > >> You can also see that upon processing this file through
> > gen_vx_mask
> > > in
> > > > > the
> > > > > >> same way as done above produces the same sized matrix -
not
> > > > transposed.
> > > > > >>
> > > > > >> Dataset('land_mask_15.nc').variables
> > > > > >> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> > > > > >> float32 lat(lat, lon)
> > > > > >> long_name: latitude
> > > > > >> units: degrees_north
> > > > > >> standard_name: latitude
> > > > > >> unlimited dimensions:
> > > > > >> current shape = (288, 468)
> > > > > >> filling on, default _FillValue of 9.969209968386869e+36
used
> > > > > >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> > > > > >> float32 lon(lat, lon)
> > > > > >> long_name: longitude
> > > > > >> units: degrees_east
> > > > > >> standard_name: longitude
> > > > > >> unlimited dimensions:
> > > > > >> current shape = (288, 468)
> > > > > >> filling on, default _FillValue of 9.969209968386869e+36
used
> > > > > >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> > > > > >> float32 data_mask(lat, lon)
> > > > > >> long_name: data_mask masking region
> > > > > >> _FillValue: -9999.0
> > > > > >> mask_type: data==15
> > > > > >> unlimited dimensions:
> > > > > >> *current shape = (288, 468)*
> > > > > >> filling on)])
> > > > > >>
> > > > > >> Moreover, given the mask_type, it identified points
having 15
> as 1
> > > and
> > > > > >> others as 0. Therefore I have a 0/1 dataset.
> > > > > >>
> > > > > >> This is different than when I process the newly created
netcdf
> > file.
> > > > > The
> > > > > >> netcdf file that was created has the Urban field with a
shape of
> > > > > (265,442)
> > > > > >>
> > > > > >> When processing it through gen_vx_mask, it gets
transposed. >>>
> > > > > >> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> > > > > >> *(442, 265)*.
> > > > > >>
> > > > > >> So in the original case I have no change to the
dimensionality
> of
> > > the
> > > > > >> dataset, but in the latter case I do. This happens
during the
> > > > > gen_vx_mask
> > > > > >> step, and I suspect that this may somehow be related to
my
> > issue. I
> > > > > don't
> > > > > >> know if there's anything else I can add that will help,
but if I
> > > think
> > > > > of
> > > > > >> something I'll forward it along
> > > > > >>
> > > > > >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA
> Affiliate <
> > > > > >> edward.strobach at noaa.gov> wrote:
> > > > > >>
> > > > > >> > Transposing it didn't seem to do anything either.
Here's the
> > > > > >> gen_vx_mask
> > > > > >> > output. It looks okay at first glance:
> > > > > >> >
> > > > > >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > > >> > -type data -mask_field 'name="DLUSE"; level="Z0";'
-thresh eq1
> > > -name
> > > > > >> Urban
> > > > > >> > DEBUG 1: Input Grid:
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > >> > DEBUG 1: Mask File:
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x
265)
> > > > > >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x
265)
> > > > > >> > DEBUG 1: Output File:
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > > >> >
> > > > > >> > I'm also attaching the header content of the file:
> > > > > >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > > > > >> > <class 'netCDF4._netCDF4.Dataset'>
> > > > > >> > root group (NETCDF4 data model, file format HDF5):
> > > > > >> > FileOrigins: NA
> > > > > >> > MET_version: V9.1
> > > > > >> > MET_tool: gen_vx_mask
> > > > > >> > Projection: Lambert Conformal
> > > > > >> > hemisphere: N
> > > > > >> > accum: 00
> > > > > >> > scale_lat_1: 33.000000
> > > > > >> > scale_lat_2: 45.000000
> > > > > >> > lat_pin: 21.821000
> > > > > >> > lon_pin: -120.628000
> > > > > >> > x_pin: 0.000000
> > > > > >> > y_pin: 0.000000
> > > > > >> > lon_orient: -97.000000
> > > > > >> > d_km: 12.000000
> > > > > >> > r_km: 6371.200000
> > > > > >> > nx: 265
> > > > > >> > ny: 442 grid_points
> > > > > >> > dimensions(sizes): lon(265), lat(442)
> > > > > >> > variables(dimensions): float32 LAT(lon,lat),
float32
> > > > LON(lon,lat),
> > > > > >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
> > > PBL(lon,lat),
> > > > > >> float32
> > > > > >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
> > WSTAR(lon,lat),
> > > > > >> float32
> > > > > >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32
HFX(lon,lat),
> > > float32
> > > > > >> > GLW(lon,lat), float32 GSW(lon,lat), float32
RGRND(lon,lat),
> > > float32
> > > > > >> > Q2(lon,lat)
> > > > > >> > groups:
> > > > > >> >
> > > > > >> > Nothing here seems troubling. You can see that
> > > DLUSE(lon,lat).shape
> > > > ==
> > > > > >> > (265,442)
> > > > > >> >
> > > > > >> > The values in the matrix are of type float, not
integer; but,
> I
> > > > don't
> > > > > >> > think that would be an issue.
> > > > > >> >
Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > > > > >> > masked_array(
> > > > > >> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > > >> > ...,
> > > > > >> > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > > > > >> > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > > > > >> > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > > > > >> > mask=[[False, False, False, ..., False, False,
False],
> > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > >> > ...,
> > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > >> > [False, False, False, ..., False, False,
False]],
> > > > > >> > fill_value=9.96921e+36,
> > > > > >> > dtype=float32)
> > > > > >> >
> > > > > >> > Do I need to change the dtype to integer?
> > > > > >> >
> > > > > >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via
RT <
> > > > > >> > met_help at ucar.edu> wrote:
> > > > > >> >
> > > > > >> >>
> > > > > >> >> Greetings,
> > > > > >> >>
> > > > > >> >> This message has been automatically generated in
response to
> > the
> > > > > >> creation
> > > > > >> >> of a trouble ticket regarding:
> > > > > >> >> "mask issues",
> > > > > >> >> a summary of which appears below.
> > > > > >> >>
> > > > > >> >> There is no need to reply to this message right now.
Your
> > ticket
> > > > has
> > > > > >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> > > > > >> >>
> > > > > >> >> Please include the string:
> > > > > >> >>
> > > > > >> >> [rt.rap.ucar.edu #96397]
> > > > > >> >>
> > > > > >> >> in the subject line of all future correspondence about
this
> > > issue.
> > > > To
> > > > > >> do
> > > > > >> >> so, you may reply to this message.
> > > > > >> >>
> > > > > >> >> For more information, please see:
> > > > > >> >>
> > > > > >> >> MET Online Tutorial:
> > > > > >> >>
> > > > > >>
> > >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > >> >>
> > > > > >> >> MET Users Guide:
> > > > > >> >>
https://www.dtcenter.org/met/users/docs/overview.php
> > > > > >> >>
> > > > > >> >> MET FAQs:
> > > > > >> >>
https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > >> >>
> > > > > >> >> MET-Help Email Archive:
> > > > > >> >> http://mailman.ucar.edu/pipermail/met_help
> > > > > >> >>
> > > > > >> >> Thank you,
> > > > > >> >> met_help at ucar.edu
> > > > > >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > >
>
-------------------------------------------------------------------------
> > > > > >> >> GOod evening,
> > > > > >> >>
> > > > > >> >> I thought I understood how to process the data into
masked
> > > regions,
> > > > > but
> > > > > >> >> apparently not. I'm using data that I processed into
a
> netcdf
> > > file
> > > > > >> that
> > > > > >> >> is
> > > > > >> >> now CF compatible. I'm processing the DLUSE field
from this
> > > > > directory:
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>*
> > > > > >> >>
> > > > > >> >> I then run this line, which is inside a loop within a
script
> > that
> > > > I'm
> > > > > >> >> using. The command looks like this:
> > > > > >> >> *gen_vx_mask
> > > > > >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > > > >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > > > >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > > >> >> -type data -mask_field 'name="DLUSE"; level="Z0";'
-thresh
> eq1
> > > > -name
> > > > > >> >> Urban*
> > > > > >> >>
> > > > > >> >> I determined that something was wrong when much of my
masked
> > land
> > > > > >> plots -
> > > > > >> >> all 24 of them - looked more or less the same. When
> inspecting
> > > > > Urban,
> > > > > >> I
> > > > > >> >> found that there were 588 matched pairs! Since I
realize
> that
> > we
> > > > > don't
> > > > > >> >> live on Coruscant, I decided to plot the data and this
is
> what
> > it
> > > > > >> looked
> > > > > >> >> like (see attached: fig 1). The original DLUSE field
looks
> > like
> > > > what
> > > > > >> you
> > > > > >> >> see in the second attachment. I'm pretty sure that
the
> > > gen_vx_mask
> > > > > was
> > > > > >> >> utilized correctly here. I have the file that I'm
using, the
> > > file
> > > > > that
> > > > > >> >> I'm
> > > > > >> >> creating to store the result, the type (data), the
mask field
> > > > (DLUSE
> > > > > at
> > > > > >> >> the
> > > > > >> >> surface), the threshold (set equal to 1 for urban),
and the
> > name
> > > of
> > > > > the
> > > > > >> >> field in the file (Urban). One of the things I
noticed was
> > that
> > > > the
> > > > > >> >> output
> > > > > >> >> from the netcdf file that I generated is transposed
from the
> > > > > original.
> > > > > >> >> The
> > > > > >> >> matrix was originally 265 X 442. However, in the file
that
> was
> > > > > >> created it
> > > > > >> >> is 442X265. I'm not sure what's going on here at all
to be
> > > honest.
> > > > > I
> > > > > >> >> thought it would be as simple as running the command
with the
> > > above
> > > > > >> >> settings. I may try to create netcdf files with the
matrix
> > being
> > > > > >> 442X265.
> > > > > >> >> Something clearly is off here.
> > > > > >> >>
> > > > > >> >> --
> > > > > >> >> Edward Strobach
> > > > > >> >> EMC/NCEP/NWS/
> > > > > >> >> IMSG Contractor
> > > > > >> >> Cubicle#: 2029
> > > > > >> >> 301-683-3717
> > > > > >> >>
> > > > > >> >>
> > > > > >> >
> > > > > >> > --
> > > > > >> > Edward Strobach
> > > > > >> > EMC/NCEP/NWS/
> > > > > >> > IMSG Contractor
> > > > > >> > Cubicle#: 2029
> > > > > >> > 301-683-3717
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Edward Strobach
> > > > > >> EMC/NCEP/NWS/
> > > > > >> IMSG Contractor
> > > > > >> Cubicle#: 2029
> > > > > >> 301-683-3717
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: mask issues
From: John Halley Gotway
Time: Mon Aug 24 15:23:25 2020
Ed,
Sorry I missed seeing this remaining issue in your response! OK, I was
able
to replicate the behavior you describe, and I understand what's going
on.
Here's what I'd recommend... just remove the "MET_tool" global
attribute
from your reformatted NetCDF file. For example,
:FileOrigins = "NA" ;
:MET_version = "V9.1" ;
* :MET_tool = "gen_vx_mask" ;*
:Projection = "Lambert Conformal" ;
:hemisphere = "N" ;
:accum = "00" ;
And then gen_vx_mask will run as you'd expect.
But why in the world would this matter? The gen_vx_mask tool is
designed to
be run iteratively, passing the output from one run as the input to
the
next. The way the tool determines whether it's reading its own output
is by
checking that "MET_tool" attribute. When it sees *MET_tool =
"gen_vx_mask";*,
it triggers some special logic. Rather than initializing the value at
each
grid point to 0, it instead initializes it to the values in the input
file.
When it applies the threshold you defined, it correctly sets those
output
grid point values to 1... and it leaves the other grid point values
unchanged. So the problem is in initializing the grid points to data
values
instead of 0's.
This really is the behavior we want in gen_vx_mask. It makes running
the
tool iteratively much more convenient without having the specify
"-input_field" on the command line each time. But this is a surprising
wrinkle I hadn't expected!
Removing MET_tool (or setting it to anything else) should make it work
fine.
Thanks,
John
On Mon, Aug 24, 2020 at 11:05 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>
> Hi John,
>
> I guess that part is resolved, but I'm not sure why it seems to
produce the
> output from the field itself. Copying something from the previous
email..
>
> Following your earlier suggestion (e.g. gen_vx_mask
> wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212 *land_mask.nc
> <http://land_mask.nc>*
> -type data -mask_field 'name="LAND"; level="Z0";' -thresh eq1), I
went
> ahead and executed something similar for Urban type:
>
> gen_vx_mask
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> aqm.t06z.metcro2d.00.nc
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> aqm.t06z.metcro2d.00.nc
>
>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
> -type
> data -name Urban -mask_field 'name="DLUSE"; level="Z0";' -thresh
'eq1'
>
> Here, I compared something that worked at assigning a 1 to
vegetation of
> type 1 (-thresh eq1 for top command) with what I did to isolate
urban (also
> -thresh -eq1 for bottom command). Instead of getting 0's and 1's
for
> regions coinciding with masked threshold values, I get 1 through 24,
with
> the exact output as the input. I thought setting thresh to 1 would
have
> worked. Both commands look pretty identical to me, so I'm not sure
why
> things would have happened differently. I find that to be an
unexpected
> result
>
> On Mon, Aug 24, 2020 at 12:59 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Ed,
> >
> > Great, I'm glad you were able to get it working as expected.
Thanks for
> > confirming. I'll go ahead and resolve this ticket.
> >
> > John
> >
> > On Sat, Aug 22, 2020 at 9:42 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> > >
> > > Hi John,
> > >
> > > Rewriting the files to follow MET's convention helped produce a
normal
> > > looking plot (see attached). That was made possible by writing
my
> files
> > in
> > > this way:
> > > Dataset('land_mask_Urban.nc')
> > > <class 'netCDF4._netCDF4.Dataset'>
> > > root group (NETCDF4 data model, file format HDF5):
> > > FileOrigins: File
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc
> > > generated 20200822_152206 UTC on host m72a1 by the MET
gen_vx_mask tool
> > > MET_version: V9.1
> > > MET_tool: gen_vx_mask
> > > Projection: Lambert Conformal
> > > hemisphere: N
> > > scale_lat_1: 33.000000
> > > scale_lat_2: 45.000000
> > > lat_pin: 21.821000
> > > lon_pin: -120.628000
> > > x_pin: 0.000000
> > > y_pin: 0.000000
> > > lon_orient: -97.000000
> > > d_km: 12.000000
> > > r_km: 6371.200000
> > >
> > >
> > >
> > > * nx: 442 ny: 265 grid_points dimensions(sizes):
lat(265),
> > > lon(442) variables(dimensions): float32 lat(lat,lon), float32
> > > lon(lat,lon), float32 Urban(lat,lon)*
> > > groups:
> > >
> > > Following your earlier suggestion (e.g. gen_vx_mask
> > > wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212
*land_mask.nc
> > > <http://land_mask.nc>*
> > > -type data -mask_field 'name="LAND"; level="Z0";' -thresh eq1),
I went
> > > ahead and executed something similar for Urban type:
> > >
> > > gen_vx_mask
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> > > aqm.t06z.metcro2d.00.nc
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> > > aqm.t06z.metcro2d.00.nc
> > >
> > >
> >
>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
> > > -type
> > > data -name Urban -mask_field 'name="DLUSE"; level="Z0";' -thresh
'eq1'
> > >
> > > However, as you can see from the map, I retain all values rather
than
> > > getting 0 for non-urban and 1 for urban. I don't see anything
that's
> > > really different between these two examples other than I specify
the
> > > written name of the mask field as Urban. I've included - and
removing
> -
> > > the single quotes about eq1, but the result ended up being the
same.
> > >
> > >
> > >
> > > I also don't see anything alarmingly different when comparing
the
> header
> > > content from this file with the examples that had worked with
VGTYP.
> > >
> > > I'll continue trying a couple of other things if some ideas come
up.
> If I
> > > make traction, I'll let you know. Take care and have a good
weekend.
> > >
> > >
> > >
> > > On Fri, Aug 21, 2020 at 4:46 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Ed,
> > > >
> > > > Sounds good. Please let me know if reversing (lon,lat) to
(lat,lon)
> in
> > > the
> > > > NetCDF files solves the problem.
> > > >
> > > > Have a good weekend.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Aug 21, 2020 at 10:51 AM Edward Strobach - NOAA
Affiliate via
> > RT
> > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Your plot looks just like mine. Thanks for confirming. I
do
> > believe I
> > > > > tried that, but I'll go ahead and try that suggestion again.
I'll
> > > attach
> > > > > the result as well. When comparing with VGTYP - an example
that
> > worked
> > > > - I
> > > > > noticed the "lat, lon" ordering as opposed to "lon, lat",
which is
> > what
> > > > > prompted me to go that route. I'll test this carefully.
Thanks
> > again.
> > > > >
> > > > > On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Ed,
> > > > > >
> > > > > > I pulled that file and took a look. I've attached the
image that
> > > > > > plot_data_plane generates with the following command:
> > > > > >
> > > > > > *plot_data_plane aqm.t06z.metcro2d.00.nc <
> > > > http://aqm.t06z.metcro2d.00.nc
> > > > > >
> > > > > > aqm.t06z.metcro2d_DLUSE.00.ps <
> > http://aqm.t06z.metcro2d_DLUSE.00.ps>
> > > > > > 'name="DLUSE"; level="(*,*)";'*
> > > > > >
> > > > > >
> > > > > >
> > > > > > So there's clearly a problem here! Looking at that file
with
> > ncdump,
> > > I
> > > > > see
> > > > > > that this file is actually setup to follow MET's internal
NetCDF
> > > > > > conventions rather the the NetCDF Climate-Forecast (CF)
> convention.
> > > > > >
> > > > > >
> > > > > > *ncdump -h aqm.t06z.metcro2d.00.nc <
> http://aqm.t06z.metcro2d.00.nc
> > >*
> > > > > >
> > > > > >
> > > > > >
> > > > > > Even though we recommend the use of the CF-convention, the
MET
> > tools
> > > > > > actually don't write to that convention themselves. But we
should
> > > > > probably
> > > > > > transition to it! For now, the tools write to a pretty
rigid
> > internal
> > > > > > format where the dimensions are always ordered (lat, lon).
Your
> > data
> > > is
> > > > > > ordered (lon, lat) and I strongly suspect that's the
reason for
> > this
> > > > > > behavior. Please try rearranging to write it as (lat, lon)
and
> then
> > > run
> > > > > > plot_data_plane on the result to see if it's fixed.
> > > > > >
> > > > > >
> > > > > > Hope that helps.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Ed,
> > > > > > >
> > > > > > > Thanks for sending your sample commands and a
description of
> the
> > > > > > behavior.
> > > > > > > I see that you've prepared a NetCDF file that follows
the
> > > > > CF-convention,
> > > > > > > but are getting some odd output from gen_vx_mask. I
suspect
> that
> > > the
> > > > > odd
> > > > > > > output from gen_vx_mask is a red herring, and that the
real
> issue
> > > is
> > > > in
> > > > > > > reading the data from that input NetCDF file.
> > > > > > >
> > > > > > > Whenever, getting started with a new dataset in MET,
such as
> the
> > > > > > > CF-compliant NetCDF file, it's always a good idea to
plot the
> > data
> > > > > using
> > > > > > > plot_data_plane. I suspect we'll find that
plot_data_plane
> > creates
> > > > the
> > > > > > same
> > > > > > > sort of odd plot when reading data directly from that
input
> file.
> > > And
> > > > > > > that's most likely due to either an issue in how the
data is
> > stored
> > > > in
> > > > > > that
> > > > > > > NetCDF file or perhaps some bug in the MET software.
> > > > > > >
> > > > > > > I'll go log on to WCOSS, retrieve that file, and do some
> testing
> > > with
> > > > > it.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA
> Affiliate
> > > via
> > > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > >>
> > > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397
> >
> > > > > > >>
> > > > > > >> Originally, I had processed grib files with vegetation
type.
> I
> > > used
> > > > > > this
> > > > > > >> record here: 880:Vegetation Type:Integer(0-13)
> > > > > > >> (instant):lambert:surface:level 0:fcst time 3 hrs:from
> > > 202008181800
> > > > > > >>
> > > > > > >> You can see the shape of that is this: >>>
> > > > > > >>
pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> > > > > > >> *(288, 468)*
> > > > > > >>
> > > > > > >> You can also see that upon processing this file through
> > > gen_vx_mask
> > > > in
> > > > > > the
> > > > > > >> same way as done above produces the same sized matrix -
not
> > > > > transposed.
> > > > > > >>
> > > > > > >> Dataset('land_mask_15.nc').variables
> > > > > > >> OrderedDict([('lat', <class
'netCDF4._netCDF4.Variable'>
> > > > > > >> float32 lat(lat, lon)
> > > > > > >> long_name: latitude
> > > > > > >> units: degrees_north
> > > > > > >> standard_name: latitude
> > > > > > >> unlimited dimensions:
> > > > > > >> current shape = (288, 468)
> > > > > > >> filling on, default _FillValue of 9.969209968386869e+36
used
> > > > > > >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> > > > > > >> float32 lon(lat, lon)
> > > > > > >> long_name: longitude
> > > > > > >> units: degrees_east
> > > > > > >> standard_name: longitude
> > > > > > >> unlimited dimensions:
> > > > > > >> current shape = (288, 468)
> > > > > > >> filling on, default _FillValue of 9.969209968386869e+36
used
> > > > > > >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> > > > > > >> float32 data_mask(lat, lon)
> > > > > > >> long_name: data_mask masking region
> > > > > > >> _FillValue: -9999.0
> > > > > > >> mask_type: data==15
> > > > > > >> unlimited dimensions:
> > > > > > >> *current shape = (288, 468)*
> > > > > > >> filling on)])
> > > > > > >>
> > > > > > >> Moreover, given the mask_type, it identified points
having 15
> > as 1
> > > > and
> > > > > > >> others as 0. Therefore I have a 0/1 dataset.
> > > > > > >>
> > > > > > >> This is different than when I process the newly created
netcdf
> > > file.
> > > > > > The
> > > > > > >> netcdf file that was created has the Urban field with a
shape
> of
> > > > > > (265,442)
> > > > > > >>
> > > > > > >> When processing it through gen_vx_mask, it gets
transposed.
> >>>
> > > > > > >>
Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> > > > > > >> *(442, 265)*.
> > > > > > >>
> > > > > > >> So in the original case I have no change to the
dimensionality
> > of
> > > > the
> > > > > > >> dataset, but in the latter case I do. This happens
during the
> > > > > > gen_vx_mask
> > > > > > >> step, and I suspect that this may somehow be related to
my
> > > issue. I
> > > > > > don't
> > > > > > >> know if there's anything else I can add that will help,
but
> if I
> > > > think
> > > > > > of
> > > > > > >> something I'll forward it along
> > > > > > >>
> > > > > > >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA
> > Affiliate <
> > > > > > >> edward.strobach at noaa.gov> wrote:
> > > > > > >>
> > > > > > >> > Transposing it didn't seem to do anything either.
Here's
> the
> > > > > > >> gen_vx_mask
> > > > > > >> > output. It looks okay at first glance:
> > > > > > >> >
> > > > > > >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > > > >> > -type data -mask_field 'name="DLUSE"; level="Z0";'
-thresh
> eq1
> > > > -name
> > > > > > >> Urban
> > > > > > >> > DEBUG 1: Input Grid:
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > > >> > DEBUG 1: Mask File:
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > > >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x
265)
> > > > > > >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x
265)
> > > > > > >> > DEBUG 1: Output File:
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > > > >> >
> > > > > > >> > I'm also attaching the header content of the file:
> > > > > > >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > > > > > >> > <class 'netCDF4._netCDF4.Dataset'>
> > > > > > >> > root group (NETCDF4 data model, file format HDF5):
> > > > > > >> > FileOrigins: NA
> > > > > > >> > MET_version: V9.1
> > > > > > >> > MET_tool: gen_vx_mask
> > > > > > >> > Projection: Lambert Conformal
> > > > > > >> > hemisphere: N
> > > > > > >> > accum: 00
> > > > > > >> > scale_lat_1: 33.000000
> > > > > > >> > scale_lat_2: 45.000000
> > > > > > >> > lat_pin: 21.821000
> > > > > > >> > lon_pin: -120.628000
> > > > > > >> > x_pin: 0.000000
> > > > > > >> > y_pin: 0.000000
> > > > > > >> > lon_orient: -97.000000
> > > > > > >> > d_km: 12.000000
> > > > > > >> > r_km: 6371.200000
> > > > > > >> > nx: 265
> > > > > > >> > ny: 442 grid_points
> > > > > > >> > dimensions(sizes): lon(265), lat(442)
> > > > > > >> > variables(dimensions): float32 LAT(lon,lat),
float32
> > > > > LON(lon,lat),
> > > > > > >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
> > > > PBL(lon,lat),
> > > > > > >> float32
> > > > > > >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
> > > WSTAR(lon,lat),
> > > > > > >> float32
> > > > > > >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32
HFX(lon,lat),
> > > > float32
> > > > > > >> > GLW(lon,lat), float32 GSW(lon,lat), float32
RGRND(lon,lat),
> > > > float32
> > > > > > >> > Q2(lon,lat)
> > > > > > >> > groups:
> > > > > > >> >
> > > > > > >> > Nothing here seems troubling. You can see that
> > > > DLUSE(lon,lat).shape
> > > > > ==
> > > > > > >> > (265,442)
> > > > > > >> >
> > > > > > >> > The values in the matrix are of type float, not
integer;
> but,
> > I
> > > > > don't
> > > > > > >> > think that would be an issue.
> > > > > > >> >
Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > > > > > >> > masked_array(
> > > > > > >> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > > > >> > ...,
> > > > > > >> > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > > > > > >> > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > > > > > >> > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > > > > > >> > mask=[[False, False, False, ..., False, False,
False],
> > > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > > >> > ...,
> > > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > > >> > [False, False, False, ..., False, False,
False]],
> > > > > > >> > fill_value=9.96921e+36,
> > > > > > >> > dtype=float32)
> > > > > > >> >
> > > > > > >> > Do I need to change the dtype to integer?
> > > > > > >> >
> > > > > > >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu
via RT <
> > > > > > >> > met_help at ucar.edu> wrote:
> > > > > > >> >
> > > > > > >> >>
> > > > > > >> >> Greetings,
> > > > > > >> >>
> > > > > > >> >> This message has been automatically generated in
response
> to
> > > the
> > > > > > >> creation
> > > > > > >> >> of a trouble ticket regarding:
> > > > > > >> >> "mask issues",
> > > > > > >> >> a summary of which appears below.
> > > > > > >> >>
> > > > > > >> >> There is no need to reply to this message right now.
Your
> > > ticket
> > > > > has
> > > > > > >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> > > > > > >> >>
> > > > > > >> >> Please include the string:
> > > > > > >> >>
> > > > > > >> >> [rt.rap.ucar.edu #96397]
> > > > > > >> >>
> > > > > > >> >> in the subject line of all future correspondence
about this
> > > > issue.
> > > > > To
> > > > > > >> do
> > > > > > >> >> so, you may reply to this message.
> > > > > > >> >>
> > > > > > >> >> For more information, please see:
> > > > > > >> >>
> > > > > > >> >> MET Online Tutorial:
> > > > > > >> >>
> > > > > > >>
> > > >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > >> >>
> > > > > > >> >> MET Users Guide:
> > > > > > >> >>
https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > >> >>
> > > > > > >> >> MET FAQs:
> > > > > > >> >>
> https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > >> >>
> > > > > > >> >> MET-Help Email Archive:
> > > > > > >> >> http://mailman.ucar.edu/pipermail/met_help
> > > > > > >> >>
> > > > > > >> >> Thank you,
> > > > > > >> >> met_help at ucar.edu
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > >
> >
-------------------------------------------------------------------------
> > > > > > >> >> GOod evening,
> > > > > > >> >>
> > > > > > >> >> I thought I understood how to process the data into
masked
> > > > regions,
> > > > > > but
> > > > > > >> >> apparently not. I'm using data that I processed
into a
> > netcdf
> > > > file
> > > > > > >> that
> > > > > > >> >> is
> > > > > > >> >> now CF compatible. I'm processing the DLUSE field
from
> this
> > > > > > directory:
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>*
> > > > > > >> >>
> > > > > > >> >> I then run this line, which is inside a loop within
a
> script
> > > that
> > > > > I'm
> > > > > > >> >> using. The command looks like this:
> > > > > > >> >> *gen_vx_mask
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > > > >> >> -type data -mask_field 'name="DLUSE"; level="Z0";'
-thresh
> > eq1
> > > > > -name
> > > > > > >> >> Urban*
> > > > > > >> >>
> > > > > > >> >> I determined that something was wrong when much of
my
> masked
> > > land
> > > > > > >> plots -
> > > > > > >> >> all 24 of them - looked more or less the same. When
> > inspecting
> > > > > > Urban,
> > > > > > >> I
> > > > > > >> >> found that there were 588 matched pairs! Since I
realize
> > that
> > > we
> > > > > > don't
> > > > > > >> >> live on Coruscant, I decided to plot the data and
this is
> > what
> > > it
> > > > > > >> looked
> > > > > > >> >> like (see attached: fig 1). The original DLUSE
field looks
> > > like
> > > > > what
> > > > > > >> you
> > > > > > >> >> see in the second attachment. I'm pretty sure that
the
> > > > gen_vx_mask
> > > > > > was
> > > > > > >> >> utilized correctly here. I have the file that I'm
using,
> the
> > > > file
> > > > > > that
> > > > > > >> >> I'm
> > > > > > >> >> creating to store the result, the type (data), the
mask
> field
> > > > > (DLUSE
> > > > > > at
> > > > > > >> >> the
> > > > > > >> >> surface), the threshold (set equal to 1 for urban),
and the
> > > name
> > > > of
> > > > > > the
> > > > > > >> >> field in the file (Urban). One of the things I
noticed was
> > > that
> > > > > the
> > > > > > >> >> output
> > > > > > >> >> from the netcdf file that I generated is transposed
from
> the
> > > > > > original.
> > > > > > >> >> The
> > > > > > >> >> matrix was originally 265 X 442. However, in the
file that
> > was
> > > > > > >> created it
> > > > > > >> >> is 442X265. I'm not sure what's going on here at
all to be
> > > > honest.
> > > > > > I
> > > > > > >> >> thought it would be as simple as running the command
with
> the
> > > > above
> > > > > > >> >> settings. I may try to create netcdf files with the
matrix
> > > being
> > > > > > >> 442X265.
> > > > > > >> >> Something clearly is off here.
> > > > > > >> >>
> > > > > > >> >> --
> > > > > > >> >> Edward Strobach
> > > > > > >> >> EMC/NCEP/NWS/
> > > > > > >> >> IMSG Contractor
> > > > > > >> >> Cubicle#: 2029
> > > > > > >> >> 301-683-3717
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > Edward Strobach
> > > > > > >> > EMC/NCEP/NWS/
> > > > > > >> > IMSG Contractor
> > > > > > >> > Cubicle#: 2029
> > > > > > >> > 301-683-3717
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Edward Strobach
> > > > > > >> EMC/NCEP/NWS/
> > > > > > >> IMSG Contractor
> > > > > > >> Cubicle#: 2029
> > > > > > >> 301-683-3717
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Aug 25 06:37:46 2020
Wow! Thanks John. I would have never guessed that to be the solution!
On Mon, Aug 24, 2020 at 5:23 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Ed,
>
> Sorry I missed seeing this remaining issue in your response! OK, I
was able
> to replicate the behavior you describe, and I understand what's
going on.
> Here's what I'd recommend... just remove the "MET_tool" global
attribute
> from your reformatted NetCDF file. For example,
>
> :FileOrigins = "NA" ;
>
> :MET_version = "V9.1" ;
>
> * :MET_tool = "gen_vx_mask" ;*
>
> :Projection = "Lambert Conformal" ;
>
> :hemisphere = "N" ;
>
> :accum = "00" ;
>
> And then gen_vx_mask will run as you'd expect.
>
> But why in the world would this matter? The gen_vx_mask tool is
designed to
> be run iteratively, passing the output from one run as the input to
the
> next. The way the tool determines whether it's reading its own
output is by
> checking that "MET_tool" attribute. When it sees *MET_tool =
> "gen_vx_mask";*,
> it triggers some special logic. Rather than initializing the value
at each
> grid point to 0, it instead initializes it to the values in the
input file.
> When it applies the threshold you defined, it correctly sets those
output
> grid point values to 1... and it leaves the other grid point values
> unchanged. So the problem is in initializing the grid points to data
values
> instead of 0's.
>
> This really is the behavior we want in gen_vx_mask. It makes running
the
> tool iteratively much more convenient without having the specify
> "-input_field" on the command line each time. But this is a
surprising
> wrinkle I hadn't expected!
>
> Removing MET_tool (or setting it to anything else) should make it
work
> fine.
>
> Thanks,
> John
>
> On Mon, Aug 24, 2020 at 11:05 AM Edward Strobach - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> >
> > Hi John,
> >
> > I guess that part is resolved, but I'm not sure why it seems to
produce
> the
> > output from the field itself. Copying something from the previous
> email..
> >
> > Following your earlier suggestion (e.g. gen_vx_mask
> > wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212 *land_mask.nc
> > <http://land_mask.nc>*
> > -type data -mask_field 'name="LAND"; level="Z0";' -thresh eq1), I
went
> > ahead and executed something similar for Urban type:
> >
> > gen_vx_mask
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> > aqm.t06z.metcro2d.00.nc
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> > aqm.t06z.metcro2d.00.nc
> >
> >
>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
> > -type
> > data -name Urban -mask_field 'name="DLUSE"; level="Z0";' -thresh
'eq1'
> >
> > Here, I compared something that worked at assigning a 1 to
vegetation of
> > type 1 (-thresh eq1 for top command) with what I did to isolate
urban
> (also
> > -thresh -eq1 for bottom command). Instead of getting 0's and 1's
for
> > regions coinciding with masked threshold values, I get 1 through
24, with
> > the exact output as the input. I thought setting thresh to 1
would have
> > worked. Both commands look pretty identical to me, so I'm not
sure why
> > things would have happened differently. I find that to be an
unexpected
> > result
> >
> > On Mon, Aug 24, 2020 at 12:59 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Ed,
> > >
> > > Great, I'm glad you were able to get it working as expected.
Thanks for
> > > confirming. I'll go ahead and resolve this ticket.
> > >
> > > John
> > >
> > > On Sat, Aug 22, 2020 at 9:42 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397
>
> > > >
> > > > Hi John,
> > > >
> > > > Rewriting the files to follow MET's convention helped produce
a
> normal
> > > > looking plot (see attached). That was made possible by
writing my
> > files
> > > in
> > > > this way:
> > > > Dataset('land_mask_Urban.nc')
> > > > <class 'netCDF4._netCDF4.Dataset'>
> > > > root group (NETCDF4 data model, file format HDF5):
> > > > FileOrigins: File
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc
> > > > generated 20200822_152206 UTC on host m72a1 by the MET
gen_vx_mask
> tool
> > > > MET_version: V9.1
> > > > MET_tool: gen_vx_mask
> > > > Projection: Lambert Conformal
> > > > hemisphere: N
> > > > scale_lat_1: 33.000000
> > > > scale_lat_2: 45.000000
> > > > lat_pin: 21.821000
> > > > lon_pin: -120.628000
> > > > x_pin: 0.000000
> > > > y_pin: 0.000000
> > > > lon_orient: -97.000000
> > > > d_km: 12.000000
> > > > r_km: 6371.200000
> > > >
> > > >
> > > >
> > > > * nx: 442 ny: 265 grid_points dimensions(sizes):
lat(265),
> > > > lon(442) variables(dimensions): float32 lat(lat,lon),
float32
> > > > lon(lat,lon), float32 Urban(lat,lon)*
> > > > groups:
> > > >
> > > > Following your earlier suggestion (e.g. gen_vx_mask
> > > > wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212
*land_mask.nc
> > > > <http://land_mask.nc>*
> > > > -type data -mask_field 'name="LAND"; level="Z0";' -thresh
eq1), I
> went
> > > > ahead and executed something similar for Urban type:
> > > >
> > > > gen_vx_mask
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> > > > aqm.t06z.metcro2d.00.nc
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> > > > aqm.t06z.metcro2d.00.nc
> > > >
> > > >
> > >
> >
>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
> > > > -type
> > > > data -name Urban -mask_field 'name="DLUSE"; level="Z0";'
-thresh
> 'eq1'
> > > >
> > > > However, as you can see from the map, I retain all values
rather than
> > > > getting 0 for non-urban and 1 for urban. I don't see anything
that's
> > > > really different between these two examples other than I
specify the
> > > > written name of the mask field as Urban. I've included - and
> removing
> > -
> > > > the single quotes about eq1, but the result ended up being the
same.
> > > >
> > > >
> > > >
> > > > I also don't see anything alarmingly different when comparing
the
> > header
> > > > content from this file with the examples that had worked with
VGTYP.
> > > >
> > > > I'll continue trying a couple of other things if some ideas
come up.
> > If I
> > > > make traction, I'll let you know. Take care and have a good
weekend.
> > > >
> > > >
> > > >
> > > > On Fri, Aug 21, 2020 at 4:46 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Ed,
> > > > >
> > > > > Sounds good. Please let me know if reversing (lon,lat) to
(lat,lon)
> > in
> > > > the
> > > > > NetCDF files solves the problem.
> > > > >
> > > > > Have a good weekend.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Fri, Aug 21, 2020 at 10:51 AM Edward Strobach - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Your plot looks just like mine. Thanks for confirming. I
do
> > > believe I
> > > > > > tried that, but I'll go ahead and try that suggestion
again.
> I'll
> > > > attach
> > > > > > the result as well. When comparing with VGTYP - an
example that
> > > worked
> > > > > - I
> > > > > > noticed the "lat, lon" ordering as opposed to "lon, lat",
which
> is
> > > what
> > > > > > prompted me to go that route. I'll test this carefully.
Thanks
> > > again.
> > > > > >
> > > > > > On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Ed,
> > > > > > >
> > > > > > > I pulled that file and took a look. I've attached the
image
> that
> > > > > > > plot_data_plane generates with the following command:
> > > > > > >
> > > > > > > *plot_data_plane aqm.t06z.metcro2d.00.nc <
> > > > > http://aqm.t06z.metcro2d.00.nc
> > > > > > >
> > > > > > > aqm.t06z.metcro2d_DLUSE.00.ps <
> > > http://aqm.t06z.metcro2d_DLUSE.00.ps>
> > > > > > > 'name="DLUSE"; level="(*,*)";'*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > So there's clearly a problem here! Looking at that file
with
> > > ncdump,
> > > > I
> > > > > > see
> > > > > > > that this file is actually setup to follow MET's
internal
> NetCDF
> > > > > > > conventions rather the the NetCDF Climate-Forecast (CF)
> > convention.
> > > > > > >
> > > > > > >
> > > > > > > *ncdump -h aqm.t06z.metcro2d.00.nc <
> > http://aqm.t06z.metcro2d.00.nc
> > > >*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Even though we recommend the use of the CF-convention,
the MET
> > > tools
> > > > > > > actually don't write to that convention themselves. But
we
> should
> > > > > > probably
> > > > > > > transition to it! For now, the tools write to a pretty
rigid
> > > internal
> > > > > > > format where the dimensions are always ordered (lat,
lon). Your
> > > data
> > > > is
> > > > > > > ordered (lon, lat) and I strongly suspect that's the
reason for
> > > this
> > > > > > > behavior. Please try rearranging to write it as (lat,
lon) and
> > then
> > > > run
> > > > > > > plot_data_plane on the result to see if it's fixed.
> > > > > > >
> > > > > > >
> > > > > > > Hope that helps.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway <
> > > johnhg at ucar.edu
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Ed,
> > > > > > > >
> > > > > > > > Thanks for sending your sample commands and a
description of
> > the
> > > > > > > behavior.
> > > > > > > > I see that you've prepared a NetCDF file that follows
the
> > > > > > CF-convention,
> > > > > > > > but are getting some odd output from gen_vx_mask. I
suspect
> > that
> > > > the
> > > > > > odd
> > > > > > > > output from gen_vx_mask is a red herring, and that the
real
> > issue
> > > > is
> > > > > in
> > > > > > > > reading the data from that input NetCDF file.
> > > > > > > >
> > > > > > > > Whenever, getting started with a new dataset in MET,
such as
> > the
> > > > > > > > CF-compliant NetCDF file, it's always a good idea to
plot the
> > > data
> > > > > > using
> > > > > > > > plot_data_plane. I suspect we'll find that
plot_data_plane
> > > creates
> > > > > the
> > > > > > > same
> > > > > > > > sort of odd plot when reading data directly from that
input
> > file.
> > > > And
> > > > > > > > that's most likely due to either an issue in how the
data is
> > > stored
> > > > > in
> > > > > > > that
> > > > > > > > NetCDF file or perhaps some bug in the MET software.
> > > > > > > >
> > > > > > > > I'll go log on to WCOSS, retrieve that file, and do
some
> > testing
> > > > with
> > > > > > it.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA
> > Affiliate
> > > > via
> > > > > > RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > >>
> > > > > > > >> <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397
> > >
> > > > > > > >>
> > > > > > > >> Originally, I had processed grib files with
vegetation type.
> > I
> > > > used
> > > > > > > this
> > > > > > > >> record here: 880:Vegetation Type:Integer(0-13)
> > > > > > > >> (instant):lambert:surface:level 0:fcst time 3
hrs:from
> > > > 202008181800
> > > > > > > >>
> > > > > > > >> You can see the shape of that is this: >>>
> > > > > > > >>
> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> > > > > > > >> *(288, 468)*
> > > > > > > >>
> > > > > > > >> You can also see that upon processing this file
through
> > > > gen_vx_mask
> > > > > in
> > > > > > > the
> > > > > > > >> same way as done above produces the same sized matrix
- not
> > > > > > transposed.
> > > > > > > >>
> > > > > > > >> Dataset('land_mask_15.nc').variables
> > > > > > > >> OrderedDict([('lat', <class
'netCDF4._netCDF4.Variable'>
> > > > > > > >> float32 lat(lat, lon)
> > > > > > > >> long_name: latitude
> > > > > > > >> units: degrees_north
> > > > > > > >> standard_name: latitude
> > > > > > > >> unlimited dimensions:
> > > > > > > >> current shape = (288, 468)
> > > > > > > >> filling on, default _FillValue of
9.969209968386869e+36 used
> > > > > > > >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> > > > > > > >> float32 lon(lat, lon)
> > > > > > > >> long_name: longitude
> > > > > > > >> units: degrees_east
> > > > > > > >> standard_name: longitude
> > > > > > > >> unlimited dimensions:
> > > > > > > >> current shape = (288, 468)
> > > > > > > >> filling on, default _FillValue of
9.969209968386869e+36 used
> > > > > > > >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> > > > > > > >> float32 data_mask(lat, lon)
> > > > > > > >> long_name: data_mask masking region
> > > > > > > >> _FillValue: -9999.0
> > > > > > > >> mask_type: data==15
> > > > > > > >> unlimited dimensions:
> > > > > > > >> *current shape = (288, 468)*
> > > > > > > >> filling on)])
> > > > > > > >>
> > > > > > > >> Moreover, given the mask_type, it identified points
having
> 15
> > > as 1
> > > > > and
> > > > > > > >> others as 0. Therefore I have a 0/1 dataset.
> > > > > > > >>
> > > > > > > >> This is different than when I process the newly
created
> netcdf
> > > > file.
> > > > > > > The
> > > > > > > >> netcdf file that was created has the Urban field with
a
> shape
> > of
> > > > > > > (265,442)
> > > > > > > >>
> > > > > > > >> When processing it through gen_vx_mask, it gets
transposed.
> > >>>
> > > > > > > >>
Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> > > > > > > >> *(442, 265)*.
> > > > > > > >>
> > > > > > > >> So in the original case I have no change to the
> dimensionality
> > > of
> > > > > the
> > > > > > > >> dataset, but in the latter case I do. This happens
during
> the
> > > > > > > gen_vx_mask
> > > > > > > >> step, and I suspect that this may somehow be related
to my
> > > > issue. I
> > > > > > > don't
> > > > > > > >> know if there's anything else I can add that will
help, but
> > if I
> > > > > think
> > > > > > > of
> > > > > > > >> something I'll forward it along
> > > > > > > >>
> > > > > > > >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach -
NOAA
> > > Affiliate <
> > > > > > > >> edward.strobach at noaa.gov> wrote:
> > > > > > > >>
> > > > > > > >> > Transposing it didn't seem to do anything either.
Here's
> > the
> > > > > > > >> gen_vx_mask
> > > > > > > >> > output. It looks okay at first glance:
> > > > > > > >> >
> > > > > > > >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > > > > >> > -type data -mask_field 'name="DLUSE"; level="Z0";'
-thresh
> > eq1
> > > > > -name
> > > > > > > >> Urban
> > > > > > > >> > DEBUG 1: Input Grid:
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > > > >> > DEBUG 1: Mask File:
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > > >> > aqm.t06z.metcro2d.00.nc
> > > > > > > >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442
x 265)
> > > > > > > >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x
265)
> > > > > > > >> > DEBUG 1: Output File:
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > > > > >> >
> > > > > > > >> > I'm also attaching the header content of the file:
> > > > > > > >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > > > > > > >> > <class 'netCDF4._netCDF4.Dataset'>
> > > > > > > >> > root group (NETCDF4 data model, file format HDF5):
> > > > > > > >> > FileOrigins: NA
> > > > > > > >> > MET_version: V9.1
> > > > > > > >> > MET_tool: gen_vx_mask
> > > > > > > >> > Projection: Lambert Conformal
> > > > > > > >> > hemisphere: N
> > > > > > > >> > accum: 00
> > > > > > > >> > scale_lat_1: 33.000000
> > > > > > > >> > scale_lat_2: 45.000000
> > > > > > > >> > lat_pin: 21.821000
> > > > > > > >> > lon_pin: -120.628000
> > > > > > > >> > x_pin: 0.000000
> > > > > > > >> > y_pin: 0.000000
> > > > > > > >> > lon_orient: -97.000000
> > > > > > > >> > d_km: 12.000000
> > > > > > > >> > r_km: 6371.200000
> > > > > > > >> > nx: 265
> > > > > > > >> > ny: 442 grid_points
> > > > > > > >> > dimensions(sizes): lon(265), lat(442)
> > > > > > > >> > variables(dimensions): float32 LAT(lon,lat),
float32
> > > > > > LON(lon,lat),
> > > > > > > >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat),
float32
> > > > > PBL(lon,lat),
> > > > > > > >> float32
> > > > > > > >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
> > > > WSTAR(lon,lat),
> > > > > > > >> float32
> > > > > > > >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32
> HFX(lon,lat),
> > > > > float32
> > > > > > > >> > GLW(lon,lat), float32 GSW(lon,lat), float32
> RGRND(lon,lat),
> > > > > float32
> > > > > > > >> > Q2(lon,lat)
> > > > > > > >> > groups:
> > > > > > > >> >
> > > > > > > >> > Nothing here seems troubling. You can see that
> > > > > DLUSE(lon,lat).shape
> > > > > > ==
> > > > > > > >> > (265,442)
> > > > > > > >> >
> > > > > > > >> > The values in the matrix are of type float, not
integer;
> > but,
> > > I
> > > > > > don't
> > > > > > > >> > think that would be an issue.
> > > > > > > >> >
Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > > > > > > >> > masked_array(
> > > > > > > >> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > > > > > >> > ...,
> > > > > > > >> > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > > > > > > >> > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > > > > > > >> > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > > > > > > >> > mask=[[False, False, False, ..., False, False,
False],
> > > > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > > > >> > ...,
> > > > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > > > >> > [False, False, False, ..., False, False,
False],
> > > > > > > >> > [False, False, False, ..., False, False,
False]],
> > > > > > > >> > fill_value=9.96921e+36,
> > > > > > > >> > dtype=float32)
> > > > > > > >> >
> > > > > > > >> > Do I need to change the dtype to integer?
> > > > > > > >> >
> > > > > > > >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu
via
> RT <
> > > > > > > >> > met_help at ucar.edu> wrote:
> > > > > > > >> >
> > > > > > > >> >>
> > > > > > > >> >> Greetings,
> > > > > > > >> >>
> > > > > > > >> >> This message has been automatically generated in
response
> > to
> > > > the
> > > > > > > >> creation
> > > > > > > >> >> of a trouble ticket regarding:
> > > > > > > >> >> "mask issues",
> > > > > > > >> >> a summary of which appears below.
> > > > > > > >> >>
> > > > > > > >> >> There is no need to reply to this message right
now.
> Your
> > > > ticket
> > > > > > has
> > > > > > > >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> > > > > > > >> >>
> > > > > > > >> >> Please include the string:
> > > > > > > >> >>
> > > > > > > >> >> [rt.rap.ucar.edu #96397]
> > > > > > > >> >>
> > > > > > > >> >> in the subject line of all future correspondence
about
> this
> > > > > issue.
> > > > > > To
> > > > > > > >> do
> > > > > > > >> >> so, you may reply to this message.
> > > > > > > >> >>
> > > > > > > >> >> For more information, please see:
> > > > > > > >> >>
> > > > > > > >> >> MET Online Tutorial:
> > > > > > > >> >>
> > > > > > > >>
> > > > >
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > > >> >>
> > > > > > > >> >> MET Users Guide:
> > > > > > > >> >>
https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > > >> >>
> > > > > > > >> >> MET FAQs:
> > > > > > > >> >>
> > https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > > >> >>
> > > > > > > >> >> MET-Help Email Archive:
> > > > > > > >> >> http://mailman.ucar.edu/pipermail/met_help
> > > > > > > >> >>
> > > > > > > >> >> Thank you,
> > > > > > > >> >> met_help at ucar.edu
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > >
> > >
>
-------------------------------------------------------------------------
> > > > > > > >> >> GOod evening,
> > > > > > > >> >>
> > > > > > > >> >> I thought I understood how to process the data
into
> masked
> > > > > regions,
> > > > > > > but
> > > > > > > >> >> apparently not. I'm using data that I processed
into a
> > > netcdf
> > > > > file
> > > > > > > >> that
> > > > > > > >> >> is
> > > > > > > >> >> now CF compatible. I'm processing the DLUSE field
from
> > this
> > > > > > > directory:
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>*
> > > > > > > >> >>
> > > > > > > >> >> I then run this line, which is inside a loop
within a
> > script
> > > > that
> > > > > > I'm
> > > > > > > >> >> using. The command looks like this:
> > > > > > > >> >> *gen_vx_mask
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > > > > > >> >> aqm.t06z.metcro2d.00.nc
> > > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > > > > > >> >> -type data -mask_field 'name="DLUSE"; level="Z0";'
> -thresh
> > > eq1
> > > > > > -name
> > > > > > > >> >> Urban*
> > > > > > > >> >>
> > > > > > > >> >> I determined that something was wrong when much of
my
> > masked
> > > > land
> > > > > > > >> plots -
> > > > > > > >> >> all 24 of them - looked more or less the same.
When
> > > inspecting
> > > > > > > Urban,
> > > > > > > >> I
> > > > > > > >> >> found that there were 588 matched pairs! Since I
realize
> > > that
> > > > we
> > > > > > > don't
> > > > > > > >> >> live on Coruscant, I decided to plot the data and
this is
> > > what
> > > > it
> > > > > > > >> looked
> > > > > > > >> >> like (see attached: fig 1). The original DLUSE
field
> looks
> > > > like
> > > > > > what
> > > > > > > >> you
> > > > > > > >> >> see in the second attachment. I'm pretty sure
that the
> > > > > gen_vx_mask
> > > > > > > was
> > > > > > > >> >> utilized correctly here. I have the file that I'm
using,
> > the
> > > > > file
> > > > > > > that
> > > > > > > >> >> I'm
> > > > > > > >> >> creating to store the result, the type (data), the
mask
> > field
> > > > > > (DLUSE
> > > > > > > at
> > > > > > > >> >> the
> > > > > > > >> >> surface), the threshold (set equal to 1 for
urban), and
> the
> > > > name
> > > > > of
> > > > > > > the
> > > > > > > >> >> field in the file (Urban). One of the things I
noticed
> was
> > > > that
> > > > > > the
> > > > > > > >> >> output
> > > > > > > >> >> from the netcdf file that I generated is
transposed from
> > the
> > > > > > > original.
> > > > > > > >> >> The
> > > > > > > >> >> matrix was originally 265 X 442. However, in the
file
> that
> > > was
> > > > > > > >> created it
> > > > > > > >> >> is 442X265. I'm not sure what's going on here at
all to
> be
> > > > > honest.
> > > > > > > I
> > > > > > > >> >> thought it would be as simple as running the
command with
> > the
> > > > > above
> > > > > > > >> >> settings. I may try to create netcdf files with
the
> matrix
> > > > being
> > > > > > > >> 442X265.
> > > > > > > >> >> Something clearly is off here.
> > > > > > > >> >>
> > > > > > > >> >> --
> > > > > > > >> >> Edward Strobach
> > > > > > > >> >> EMC/NCEP/NWS/
> > > > > > > >> >> IMSG Contractor
> > > > > > > >> >> Cubicle#: 2029
> > > > > > > >> >> 301-683-3717
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > Edward Strobach
> > > > > > > >> > EMC/NCEP/NWS/
> > > > > > > >> > IMSG Contractor
> > > > > > > >> > Cubicle#: 2029
> > > > > > > >> > 301-683-3717
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Edward Strobach
> > > > > > > >> EMC/NCEP/NWS/
> > > > > > > >> IMSG Contractor
> > > > > > > >> Cubicle#: 2029
> > > > > > > >> 301-683-3717
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Tue Aug 25 12:31:44 2020
I did confirm that it worked! I also want to extend my appreciation
for
all the help that you have provided me through these growing pains. I
have
learned a lot and your assistance has definitely gotten me there.
On Tue, Aug 25, 2020 at 8:37 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> Wow! Thanks John. I would have never guessed that to be the
solution!
>
> On Mon, Aug 24, 2020 at 5:23 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Ed,
>>
>> Sorry I missed seeing this remaining issue in your response! OK, I
was
>> able
>> to replicate the behavior you describe, and I understand what's
going on.
>> Here's what I'd recommend... just remove the "MET_tool" global
attribute
>> from your reformatted NetCDF file. For example,
>>
>> :FileOrigins = "NA" ;
>>
>> :MET_version = "V9.1" ;
>>
>> * :MET_tool = "gen_vx_mask" ;*
>>
>> :Projection = "Lambert Conformal" ;
>>
>> :hemisphere = "N" ;
>>
>> :accum = "00" ;
>>
>> And then gen_vx_mask will run as you'd expect.
>>
>> But why in the world would this matter? The gen_vx_mask tool is
designed
>> to
>> be run iteratively, passing the output from one run as the input to
the
>> next. The way the tool determines whether it's reading its own
output is
>> by
>> checking that "MET_tool" attribute. When it sees *MET_tool =
>> "gen_vx_mask";*,
>> it triggers some special logic. Rather than initializing the value
at each
>> grid point to 0, it instead initializes it to the values in the
input
>> file.
>> When it applies the threshold you defined, it correctly sets those
output
>> grid point values to 1... and it leaves the other grid point values
>> unchanged. So the problem is in initializing the grid points to
data
>> values
>> instead of 0's.
>>
>> This really is the behavior we want in gen_vx_mask. It makes
running the
>> tool iteratively much more convenient without having the specify
>> "-input_field" on the command line each time. But this is a
surprising
>> wrinkle I hadn't expected!
>>
>> Removing MET_tool (or setting it to anything else) should make it
work
>> fine.
>>
>> Thanks,
>> John
>>
>> On Mon, Aug 24, 2020 at 11:05 AM Edward Strobach - NOAA Affiliate
via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>> >
>> > Hi John,
>> >
>> > I guess that part is resolved, but I'm not sure why it seems to
produce
>> the
>> > output from the field itself. Copying something from the
previous
>> email..
>> >
>> > Following your earlier suggestion (e.g. gen_vx_mask
>> > wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212 *land_mask.nc
>> > <http://land_mask.nc>*
>> > -type data -mask_field 'name="LAND"; level="Z0";' -thresh eq1), I
went
>> > ahead and executed something similar for Urban type:
>> >
>> > gen_vx_mask
>> >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
>> > aqm.t06z.metcro2d.00.nc
>> >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
>> > aqm.t06z.metcro2d.00.nc
>> >
>> >
>>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
>> > -type
>> > data -name Urban -mask_field 'name="DLUSE"; level="Z0";' -thresh
'eq1'
>> >
>> > Here, I compared something that worked at assigning a 1 to
vegetation of
>> > type 1 (-thresh eq1 for top command) with what I did to isolate
urban
>> (also
>> > -thresh -eq1 for bottom command). Instead of getting 0's and 1's
for
>> > regions coinciding with masked threshold values, I get 1 through
24,
>> with
>> > the exact output as the input. I thought setting thresh to 1
would have
>> > worked. Both commands look pretty identical to me, so I'm not
sure why
>> > things would have happened differently. I find that to be an
unexpected
>> > result
>> >
>> > On Mon, Aug 24, 2020 at 12:59 PM John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > > Ed,
>> > >
>> > > Great, I'm glad you were able to get it working as expected.
Thanks
>> for
>> > > confirming. I'll go ahead and resolve this ticket.
>> > >
>> > > John
>> > >
>> > > On Sat, Aug 22, 2020 at 9:42 AM Edward Strobach - NOAA
Affiliate via
>> RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397
>
>> > > >
>> > > > Hi John,
>> > > >
>> > > > Rewriting the files to follow MET's convention helped produce
a
>> normal
>> > > > looking plot (see attached). That was made possible by
writing my
>> > files
>> > > in
>> > > > this way:
>> > > > Dataset('land_mask_Urban.nc')
>> > > > <class 'netCDF4._netCDF4.Dataset'>
>> > > > root group (NETCDF4 data model, file format HDF5):
>> > > > FileOrigins: File
>> > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc
>> > > > generated 20200822_152206 UTC on host m72a1 by the MET
gen_vx_mask
>> tool
>> > > > MET_version: V9.1
>> > > > MET_tool: gen_vx_mask
>> > > > Projection: Lambert Conformal
>> > > > hemisphere: N
>> > > > scale_lat_1: 33.000000
>> > > > scale_lat_2: 45.000000
>> > > > lat_pin: 21.821000
>> > > > lon_pin: -120.628000
>> > > > x_pin: 0.000000
>> > > > y_pin: 0.000000
>> > > > lon_orient: -97.000000
>> > > > d_km: 12.000000
>> > > > r_km: 6371.200000
>> > > >
>> > > >
>> > > >
>> > > > * nx: 442 ny: 265 grid_points dimensions(sizes):
lat(265),
>> > > > lon(442) variables(dimensions): float32 lat(lat,lon),
float32
>> > > > lon(lat,lon), float32 Urban(lat,lon)*
>> > > > groups:
>> > > >
>> > > > Following your earlier suggestion (e.g. gen_vx_mask
>> > > > wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212
*land_mask.nc
>> > > > <http://land_mask.nc>*
>> > > > -type data -mask_field 'name="LAND"; level="Z0";' -thresh
eq1), I
>> went
>> > > > ahead and executed something similar for Urban type:
>> > > >
>> > > > gen_vx_mask
>> > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
>> > > > aqm.t06z.metcro2d.00.nc
>> > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
>> > > > aqm.t06z.metcro2d.00.nc
>> > > >
>> > > >
>> > >
>> >
>>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
>> > > > -type
>> > > > data -name Urban -mask_field 'name="DLUSE"; level="Z0";'
-thresh
>> 'eq1'
>> > > >
>> > > > However, as you can see from the map, I retain all values
rather
>> than
>> > > > getting 0 for non-urban and 1 for urban. I don't see
anything
>> that's
>> > > > really different between these two examples other than I
specify the
>> > > > written name of the mask field as Urban. I've included - and
>> removing
>> > -
>> > > > the single quotes about eq1, but the result ended up being
the same.
>> > > >
>> > > >
>> > > >
>> > > > I also don't see anything alarmingly different when comparing
the
>> > header
>> > > > content from this file with the examples that had worked with
VGTYP.
>> > > >
>> > > > I'll continue trying a couple of other things if some ideas
come up.
>> > If I
>> > > > make traction, I'll let you know. Take care and have a good
>> weekend.
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Aug 21, 2020 at 4:46 PM John Halley Gotway via RT <
>> > > > met_help at ucar.edu>
>> > > > wrote:
>> > > >
>> > > > > Ed,
>> > > > >
>> > > > > Sounds good. Please let me know if reversing (lon,lat) to
>> (lat,lon)
>> > in
>> > > > the
>> > > > > NetCDF files solves the problem.
>> > > > >
>> > > > > Have a good weekend.
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > On Fri, Aug 21, 2020 at 10:51 AM Edward Strobach - NOAA
Affiliate
>> via
>> > > RT
>> > > > <
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>> > > > > >
>> > > > > > Hi John,
>> > > > > >
>> > > > > > Your plot looks just like mine. Thanks for confirming.
I do
>> > > believe I
>> > > > > > tried that, but I'll go ahead and try that suggestion
again.
>> I'll
>> > > > attach
>> > > > > > the result as well. When comparing with VGTYP - an
example that
>> > > worked
>> > > > > - I
>> > > > > > noticed the "lat, lon" ordering as opposed to "lon, lat",
which
>> is
>> > > what
>> > > > > > prompted me to go that route. I'll test this carefully.
Thanks
>> > > again.
>> > > > > >
>> > > > > > On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via
RT <
>> > > > > > met_help at ucar.edu> wrote:
>> > > > > >
>> > > > > > > Ed,
>> > > > > > >
>> > > > > > > I pulled that file and took a look. I've attached the
image
>> that
>> > > > > > > plot_data_plane generates with the following command:
>> > > > > > >
>> > > > > > > *plot_data_plane aqm.t06z.metcro2d.00.nc <
>> > > > > http://aqm.t06z.metcro2d.00.nc
>> > > > > > >
>> > > > > > > aqm.t06z.metcro2d_DLUSE.00.ps <
>> > > http://aqm.t06z.metcro2d_DLUSE.00.ps>
>> > > > > > > 'name="DLUSE"; level="(*,*)";'*
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > So there's clearly a problem here! Looking at that file
with
>> > > ncdump,
>> > > > I
>> > > > > > see
>> > > > > > > that this file is actually setup to follow MET's
internal
>> NetCDF
>> > > > > > > conventions rather the the NetCDF Climate-Forecast (CF)
>> > convention.
>> > > > > > >
>> > > > > > >
>> > > > > > > *ncdump -h aqm.t06z.metcro2d.00.nc <
>> > http://aqm.t06z.metcro2d.00.nc
>> > > >*
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > Even though we recommend the use of the CF-convention,
the MET
>> > > tools
>> > > > > > > actually don't write to that convention themselves. But
we
>> should
>> > > > > > probably
>> > > > > > > transition to it! For now, the tools write to a pretty
rigid
>> > > internal
>> > > > > > > format where the dimensions are always ordered (lat,
lon).
>> Your
>> > > data
>> > > > is
>> > > > > > > ordered (lon, lat) and I strongly suspect that's the
reason
>> for
>> > > this
>> > > > > > > behavior. Please try rearranging to write it as (lat,
lon) and
>> > then
>> > > > run
>> > > > > > > plot_data_plane on the result to see if it's fixed.
>> > > > > > >
>> > > > > > >
>> > > > > > > Hope that helps.
>> > > > > > >
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > John
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway <
>> > > johnhg at ucar.edu
>> > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Ed,
>> > > > > > > >
>> > > > > > > > Thanks for sending your sample commands and a
description of
>> > the
>> > > > > > > behavior.
>> > > > > > > > I see that you've prepared a NetCDF file that follows
the
>> > > > > > CF-convention,
>> > > > > > > > but are getting some odd output from gen_vx_mask. I
suspect
>> > that
>> > > > the
>> > > > > > odd
>> > > > > > > > output from gen_vx_mask is a red herring, and that
the real
>> > issue
>> > > > is
>> > > > > in
>> > > > > > > > reading the data from that input NetCDF file.
>> > > > > > > >
>> > > > > > > > Whenever, getting started with a new dataset in MET,
such as
>> > the
>> > > > > > > > CF-compliant NetCDF file, it's always a good idea to
plot
>> the
>> > > data
>> > > > > > using
>> > > > > > > > plot_data_plane. I suspect we'll find that
plot_data_plane
>> > > creates
>> > > > > the
>> > > > > > > same
>> > > > > > > > sort of odd plot when reading data directly from that
input
>> > file.
>> > > > And
>> > > > > > > > that's most likely due to either an issue in how the
data is
>> > > stored
>> > > > > in
>> > > > > > > that
>> > > > > > > > NetCDF file or perhaps some bug in the MET software.
>> > > > > > > >
>> > > > > > > > I'll go log on to WCOSS, retrieve that file, and do
some
>> > testing
>> > > > with
>> > > > > > it.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > John
>> > > > > > > >
>> > > > > > > > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach -
NOAA
>> > Affiliate
>> > > > via
>> > > > > > RT <
>> > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > >
>> > > > > > > >>
>> > > > > > > >> <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397
>> > >
>> > > > > > > >>
>> > > > > > > >> Originally, I had processed grib files with
vegetation
>> type.
>> > I
>> > > > used
>> > > > > > > this
>> > > > > > > >> record here: 880:Vegetation Type:Integer(0-13)
>> > > > > > > >> (instant):lambert:surface:level 0:fcst time 3
hrs:from
>> > > > 202008181800
>> > > > > > > >>
>> > > > > > > >> You can see the shape of that is this: >>>
>> > > > > > > >>
>> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
>> > > > > > > >> *(288, 468)*
>> > > > > > > >>
>> > > > > > > >> You can also see that upon processing this file
through
>> > > > gen_vx_mask
>> > > > > in
>> > > > > > > the
>> > > > > > > >> same way as done above produces the same sized
matrix - not
>> > > > > > transposed.
>> > > > > > > >>
>> > > > > > > >> Dataset('land_mask_15.nc').variables
>> > > > > > > >> OrderedDict([('lat', <class
'netCDF4._netCDF4.Variable'>
>> > > > > > > >> float32 lat(lat, lon)
>> > > > > > > >> long_name: latitude
>> > > > > > > >> units: degrees_north
>> > > > > > > >> standard_name: latitude
>> > > > > > > >> unlimited dimensions:
>> > > > > > > >> current shape = (288, 468)
>> > > > > > > >> filling on, default _FillValue of
9.969209968386869e+36
>> used
>> > > > > > > >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
>> > > > > > > >> float32 lon(lat, lon)
>> > > > > > > >> long_name: longitude
>> > > > > > > >> units: degrees_east
>> > > > > > > >> standard_name: longitude
>> > > > > > > >> unlimited dimensions:
>> > > > > > > >> current shape = (288, 468)
>> > > > > > > >> filling on, default _FillValue of
9.969209968386869e+36
>> used
>> > > > > > > >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
>> > > > > > > >> float32 data_mask(lat, lon)
>> > > > > > > >> long_name: data_mask masking region
>> > > > > > > >> _FillValue: -9999.0
>> > > > > > > >> mask_type: data==15
>> > > > > > > >> unlimited dimensions:
>> > > > > > > >> *current shape = (288, 468)*
>> > > > > > > >> filling on)])
>> > > > > > > >>
>> > > > > > > >> Moreover, given the mask_type, it identified points
having
>> 15
>> > > as 1
>> > > > > and
>> > > > > > > >> others as 0. Therefore I have a 0/1 dataset.
>> > > > > > > >>
>> > > > > > > >> This is different than when I process the newly
created
>> netcdf
>> > > > file.
>> > > > > > > The
>> > > > > > > >> netcdf file that was created has the Urban field
with a
>> shape
>> > of
>> > > > > > > (265,442)
>> > > > > > > >>
>> > > > > > > >> When processing it through gen_vx_mask, it gets
transposed.
>> > >>>
>> > > > > > > >>
Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
>> > > > > > > >> *(442, 265)*.
>> > > > > > > >>
>> > > > > > > >> So in the original case I have no change to the
>> dimensionality
>> > > of
>> > > > > the
>> > > > > > > >> dataset, but in the latter case I do. This happens
during
>> the
>> > > > > > > gen_vx_mask
>> > > > > > > >> step, and I suspect that this may somehow be related
to my
>> > > > issue. I
>> > > > > > > don't
>> > > > > > > >> know if there's anything else I can add that will
help, but
>> > if I
>> > > > > think
>> > > > > > > of
>> > > > > > > >> something I'll forward it along
>> > > > > > > >>
>> > > > > > > >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach -
NOAA
>> > > Affiliate <
>> > > > > > > >> edward.strobach at noaa.gov> wrote:
>> > > > > > > >>
>> > > > > > > >> > Transposing it didn't seem to do anything either.
Here's
>> > the
>> > > > > > > >> gen_vx_mask
>> > > > > > > >> > output. It looks okay at first glance:
>> > > > > > > >> >
>> > > > > > > >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > > > > > > >> > aqm.t06z.metcro2d.00.nc
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > > > > > > >> > aqm.t06z.metcro2d.00.nc
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> > > > > > > >> > -type data -mask_field 'name="DLUSE"; level="Z0";'
>> -thresh
>> > eq1
>> > > > > -name
>> > > > > > > >> Urban
>> > > > > > > >> > DEBUG 1: Input Grid:
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > > > > > > >> > aqm.t06z.metcro2d.00.nc
>> > > > > > > >> > DEBUG 1: Mask File:
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > > > > > > >> > aqm.t06z.metcro2d.00.nc
>> > > > > > > >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442
x 265)
>> > > > > > > >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442
x 265)
>> > > > > > > >> > DEBUG 1: Output File:
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> > > > > > > >> >
>> > > > > > > >> > I'm also attaching the header content of the file:
>> > > > > > > >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
>> > > > > > > >> > <class 'netCDF4._netCDF4.Dataset'>
>> > > > > > > >> > root group (NETCDF4 data model, file format HDF5):
>> > > > > > > >> > FileOrigins: NA
>> > > > > > > >> > MET_version: V9.1
>> > > > > > > >> > MET_tool: gen_vx_mask
>> > > > > > > >> > Projection: Lambert Conformal
>> > > > > > > >> > hemisphere: N
>> > > > > > > >> > accum: 00
>> > > > > > > >> > scale_lat_1: 33.000000
>> > > > > > > >> > scale_lat_2: 45.000000
>> > > > > > > >> > lat_pin: 21.821000
>> > > > > > > >> > lon_pin: -120.628000
>> > > > > > > >> > x_pin: 0.000000
>> > > > > > > >> > y_pin: 0.000000
>> > > > > > > >> > lon_orient: -97.000000
>> > > > > > > >> > d_km: 12.000000
>> > > > > > > >> > r_km: 6371.200000
>> > > > > > > >> > nx: 265
>> > > > > > > >> > ny: 442 grid_points
>> > > > > > > >> > dimensions(sizes): lon(265), lat(442)
>> > > > > > > >> > variables(dimensions): float32 LAT(lon,lat),
float32
>> > > > > > LON(lon,lat),
>> > > > > > > >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat),
float32
>> > > > > PBL(lon,lat),
>> > > > > > > >> float32
>> > > > > > > >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
>> > > > WSTAR(lon,lat),
>> > > > > > > >> float32
>> > > > > > > >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32
>> HFX(lon,lat),
>> > > > > float32
>> > > > > > > >> > GLW(lon,lat), float32 GSW(lon,lat), float32
>> RGRND(lon,lat),
>> > > > > float32
>> > > > > > > >> > Q2(lon,lat)
>> > > > > > > >> > groups:
>> > > > > > > >> >
>> > > > > > > >> > Nothing here seems troubling. You can see that
>> > > > > DLUSE(lon,lat).shape
>> > > > > > ==
>> > > > > > > >> > (265,442)
>> > > > > > > >> >
>> > > > > > > >> > The values in the matrix are of type float, not
integer;
>> > but,
>> > > I
>> > > > > > don't
>> > > > > > > >> > think that would be an issue.
>> > > > > > > >> >
Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
>> > > > > > > >> > masked_array(
>> > > > > > > >> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>> > > > > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>> > > > > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>> > > > > > > >> > ...,
>> > > > > > > >> > [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
>> > > > > > > >> > [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
>> > > > > > > >> > [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
>> > > > > > > >> > mask=[[False, False, False, ..., False, False,
False],
>> > > > > > > >> > [False, False, False, ..., False, False,
False],
>> > > > > > > >> > [False, False, False, ..., False, False,
False],
>> > > > > > > >> > ...,
>> > > > > > > >> > [False, False, False, ..., False, False,
False],
>> > > > > > > >> > [False, False, False, ..., False, False,
False],
>> > > > > > > >> > [False, False, False, ..., False, False,
False]],
>> > > > > > > >> > fill_value=9.96921e+36,
>> > > > > > > >> > dtype=float32)
>> > > > > > > >> >
>> > > > > > > >> > Do I need to change the dtype to integer?
>> > > > > > > >> >
>> > > > > > > >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu
via
>> RT <
>> > > > > > > >> > met_help at ucar.edu> wrote:
>> > > > > > > >> >
>> > > > > > > >> >>
>> > > > > > > >> >> Greetings,
>> > > > > > > >> >>
>> > > > > > > >> >> This message has been automatically generated in
>> response
>> > to
>> > > > the
>> > > > > > > >> creation
>> > > > > > > >> >> of a trouble ticket regarding:
>> > > > > > > >> >> "mask issues",
>> > > > > > > >> >> a summary of which appears below.
>> > > > > > > >> >>
>> > > > > > > >> >> There is no need to reply to this message right
now.
>> Your
>> > > > ticket
>> > > > > > has
>> > > > > > > >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
>> > > > > > > >> >>
>> > > > > > > >> >> Please include the string:
>> > > > > > > >> >>
>> > > > > > > >> >> [rt.rap.ucar.edu #96397]
>> > > > > > > >> >>
>> > > > > > > >> >> in the subject line of all future correspondence
about
>> this
>> > > > > issue.
>> > > > > > To
>> > > > > > > >> do
>> > > > > > > >> >> so, you may reply to this message.
>> > > > > > > >> >>
>> > > > > > > >> >> For more information, please see:
>> > > > > > > >> >>
>> > > > > > > >> >> MET Online Tutorial:
>> > > > > > > >> >>
>> > > > > > > >>
>> > > > >
>>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>> > > > > > > >> >>
>> > > > > > > >> >> MET Users Guide:
>> > > > > > > >> >>
https://www.dtcenter.org/met/users/docs/overview.php
>> > > > > > > >> >>
>> > > > > > > >> >> MET FAQs:
>> > > > > > > >> >>
>> > https://www.dtcenter.org/met/users/support/faqs/index.php
>> > > > > > > >> >>
>> > > > > > > >> >> MET-Help Email Archive:
>> > > > > > > >> >> http://mailman.ucar.edu/pipermail/met_help
>> > > > > > > >> >>
>> > > > > > > >> >> Thank you,
>> > > > > > > >> >> met_help at ucar.edu
>> > > > > > > >> >>
>> > > > > > > >> >>
>> > > > > > > >>
>> > > > > > >
>> > > > >
>> > >
>>
-------------------------------------------------------------------------
>> > > > > > > >> >> GOod evening,
>> > > > > > > >> >>
>> > > > > > > >> >> I thought I understood how to process the data
into
>> masked
>> > > > > regions,
>> > > > > > > but
>> > > > > > > >> >> apparently not. I'm using data that I processed
into a
>> > > netcdf
>> > > > > file
>> > > > > > > >> that
>> > > > > > > >> >> is
>> > > > > > > >> >> now CF compatible. I'm processing the DLUSE
field from
>> > this
>> > > > > > > directory:
>> > > > > > > >> >>
>> > > > > > > >> >>
>> > > > > > > >> >>
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > > > > > > >> >> aqm.t06z.metcro2d.00.nc
>> > > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>*
>> > > > > > > >> >>
>> > > > > > > >> >> I then run this line, which is inside a loop
within a
>> > script
>> > > > that
>> > > > > > I'm
>> > > > > > > >> >> using. The command looks like this:
>> > > > > > > >> >> *gen_vx_mask
>> > > > > > > >> >>
>> > > > > > > >> >>
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > > > > > > >> >> aqm.t06z.metcro2d.00.nc
>> > > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
>> > > > > > > >> >>
>> > > > > > > >> >>
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > > > > > > >> >> aqm.t06z.metcro2d.00.nc
>> > > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
>> > > > > > > >> >>
>> > > > > > > >> >>
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> > > > > > > >> >> -type data -mask_field 'name="DLUSE";
level="Z0";'
>> -thresh
>> > > eq1
>> > > > > > -name
>> > > > > > > >> >> Urban*
>> > > > > > > >> >>
>> > > > > > > >> >> I determined that something was wrong when much
of my
>> > masked
>> > > > land
>> > > > > > > >> plots -
>> > > > > > > >> >> all 24 of them - looked more or less the same.
When
>> > > inspecting
>> > > > > > > Urban,
>> > > > > > > >> I
>> > > > > > > >> >> found that there were 588 matched pairs! Since I
>> realize
>> > > that
>> > > > we
>> > > > > > > don't
>> > > > > > > >> >> live on Coruscant, I decided to plot the data and
this
>> is
>> > > what
>> > > > it
>> > > > > > > >> looked
>> > > > > > > >> >> like (see attached: fig 1). The original DLUSE
field
>> looks
>> > > > like
>> > > > > > what
>> > > > > > > >> you
>> > > > > > > >> >> see in the second attachment. I'm pretty sure
that the
>> > > > > gen_vx_mask
>> > > > > > > was
>> > > > > > > >> >> utilized correctly here. I have the file that
I'm
>> using,
>> > the
>> > > > > file
>> > > > > > > that
>> > > > > > > >> >> I'm
>> > > > > > > >> >> creating to store the result, the type (data),
the mask
>> > field
>> > > > > > (DLUSE
>> > > > > > > at
>> > > > > > > >> >> the
>> > > > > > > >> >> surface), the threshold (set equal to 1 for
urban), and
>> the
>> > > > name
>> > > > > of
>> > > > > > > the
>> > > > > > > >> >> field in the file (Urban). One of the things I
noticed
>> was
>> > > > that
>> > > > > > the
>> > > > > > > >> >> output
>> > > > > > > >> >> from the netcdf file that I generated is
transposed from
>> > the
>> > > > > > > original.
>> > > > > > > >> >> The
>> > > > > > > >> >> matrix was originally 265 X 442. However, in the
file
>> that
>> > > was
>> > > > > > > >> created it
>> > > > > > > >> >> is 442X265. I'm not sure what's going on here at
all
>> to be
>> > > > > honest.
>> > > > > > > I
>> > > > > > > >> >> thought it would be as simple as running the
command
>> with
>> > the
>> > > > > above
>> > > > > > > >> >> settings. I may try to create netcdf files with
the
>> matrix
>> > > > being
>> > > > > > > >> 442X265.
>> > > > > > > >> >> Something clearly is off here.
>> > > > > > > >> >>
>> > > > > > > >> >> --
>> > > > > > > >> >> Edward Strobach
>> > > > > > > >> >> EMC/NCEP/NWS/
>> > > > > > > >> >> IMSG Contractor
>> > > > > > > >> >> Cubicle#: 2029
>> > > > > > > >> >> 301-683-3717
>> > > > > > > >> >>
>> > > > > > > >> >>
>> > > > > > > >> >
>> > > > > > > >> > --
>> > > > > > > >> > Edward Strobach
>> > > > > > > >> > EMC/NCEP/NWS/
>> > > > > > > >> > IMSG Contractor
>> > > > > > > >> > Cubicle#: 2029
>> > > > > > > >> > 301-683-3717
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >> --
>> > > > > > > >> Edward Strobach
>> > > > > > > >> EMC/NCEP/NWS/
>> > > > > > > >> IMSG Contractor
>> > > > > > > >> Cubicle#: 2029
>> > > > > > > >> 301-683-3717
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Edward Strobach
>> > > > > > EMC/NCEP/NWS/
>> > > > > > IMSG Contractor
>> > > > > > Cubicle#: 2029
>> > > > > > 301-683-3717
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > Edward Strobach
>> > > > EMC/NCEP/NWS/
>> > > > IMSG Contractor
>> > > > Cubicle#: 2029
>> > > > 301-683-3717
>> > > >
>> > > >
>> > >
>> > >
>> >
>> > --
>> > Edward Strobach
>> > EMC/NCEP/NWS/
>> > IMSG Contractor
>> > Cubicle#: 2029
>> > 301-683-3717
>> >
>> >
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: mask issues
From: John Halley Gotway
Time: Tue Aug 25 12:36:24 2020
Ed,
Great, thanks for confirming. I'll go ahead and resolve this ticket
but
just let us know what else comes up.
Thanks,
John
On Tue, Aug 25, 2020 at 12:32 PM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>
> I did confirm that it worked! I also want to extend my appreciation
for
> all the help that you have provided me through these growing pains.
I have
> learned a lot and your assistance has definitely gotten me there.
>
> On Tue, Aug 25, 2020 at 8:37 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Wow! Thanks John. I would have never guessed that to be the
solution!
> >
> > On Mon, Aug 24, 2020 at 5:23 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Ed,
> >>
> >> Sorry I missed seeing this remaining issue in your response! OK,
I was
> >> able
> >> to replicate the behavior you describe, and I understand what's
going
> on.
> >> Here's what I'd recommend... just remove the "MET_tool" global
attribute
> >> from your reformatted NetCDF file. For example,
> >>
> >> :FileOrigins = "NA" ;
> >>
> >> :MET_version = "V9.1" ;
> >>
> >> * :MET_tool = "gen_vx_mask" ;*
> >>
> >> :Projection = "Lambert Conformal" ;
> >>
> >> :hemisphere = "N" ;
> >>
> >> :accum = "00" ;
> >>
> >> And then gen_vx_mask will run as you'd expect.
> >>
> >> But why in the world would this matter? The gen_vx_mask tool is
designed
> >> to
> >> be run iteratively, passing the output from one run as the input
to the
> >> next. The way the tool determines whether it's reading its own
output is
> >> by
> >> checking that "MET_tool" attribute. When it sees *MET_tool =
> >> "gen_vx_mask";*,
> >> it triggers some special logic. Rather than initializing the
value at
> each
> >> grid point to 0, it instead initializes it to the values in the
input
> >> file.
> >> When it applies the threshold you defined, it correctly sets
those
> output
> >> grid point values to 1... and it leaves the other grid point
values
> >> unchanged. So the problem is in initializing the grid points to
data
> >> values
> >> instead of 0's.
> >>
> >> This really is the behavior we want in gen_vx_mask. It makes
running the
> >> tool iteratively much more convenient without having the specify
> >> "-input_field" on the command line each time. But this is a
surprising
> >> wrinkle I hadn't expected!
> >>
> >> Removing MET_tool (or setting it to anything else) should make it
work
> >> fine.
> >>
> >> Thanks,
> >> John
> >>
> >> On Mon, Aug 24, 2020 at 11:05 AM Edward Strobach - NOAA Affiliate
via
> RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> >> >
> >> > Hi John,
> >> >
> >> > I guess that part is resolved, but I'm not sure why it seems to
> produce
> >> the
> >> > output from the field itself. Copying something from the
previous
> >> email..
> >> >
> >> > Following your earlier suggestion (e.g. gen_vx_mask
> >> > wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212
*land_mask.nc
> >> > <http://land_mask.nc>*
> >> > -type data -mask_field 'name="LAND"; level="Z0";' -thresh eq1),
I went
> >> > ahead and executed something similar for Urban type:
> >> >
> >> > gen_vx_mask
> >> >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> >> > aqm.t06z.metcro2d.00.nc
> >> >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> >> > aqm.t06z.metcro2d.00.nc
> >> >
> >> >
> >>
>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
> >> > -type
> >> > data -name Urban -mask_field 'name="DLUSE"; level="Z0";'
-thresh 'eq1'
> >> >
> >> > Here, I compared something that worked at assigning a 1 to
vegetation
> of
> >> > type 1 (-thresh eq1 for top command) with what I did to isolate
urban
> >> (also
> >> > -thresh -eq1 for bottom command). Instead of getting 0's and
1's for
> >> > regions coinciding with masked threshold values, I get 1
through 24,
> >> with
> >> > the exact output as the input. I thought setting thresh to 1
would
> have
> >> > worked. Both commands look pretty identical to me, so I'm not
sure
> why
> >> > things would have happened differently. I find that to be an
> unexpected
> >> > result
> >> >
> >> > On Mon, Aug 24, 2020 at 12:59 PM John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > > Ed,
> >> > >
> >> > > Great, I'm glad you were able to get it working as expected.
Thanks
> >> for
> >> > > confirming. I'll go ahead and resolve this ticket.
> >> > >
> >> > > John
> >> > >
> >> > > On Sat, Aug 22, 2020 at 9:42 AM Edward Strobach - NOAA
Affiliate via
> >> RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> >> > > >
> >> > > > Hi John,
> >> > > >
> >> > > > Rewriting the files to follow MET's convention helped
produce a
> >> normal
> >> > > > looking plot (see attached). That was made possible by
writing my
> >> > files
> >> > > in
> >> > > > this way:
> >> > > > Dataset('land_mask_Urban.nc')
> >> > > > <class 'netCDF4._netCDF4.Dataset'>
> >> > > > root group (NETCDF4 data model, file format HDF5):
> >> > > > FileOrigins: File
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc
> >> > > > generated 20200822_152206 UTC on host m72a1 by the MET
gen_vx_mask
> >> tool
> >> > > > MET_version: V9.1
> >> > > > MET_tool: gen_vx_mask
> >> > > > Projection: Lambert Conformal
> >> > > > hemisphere: N
> >> > > > scale_lat_1: 33.000000
> >> > > > scale_lat_2: 45.000000
> >> > > > lat_pin: 21.821000
> >> > > > lon_pin: -120.628000
> >> > > > x_pin: 0.000000
> >> > > > y_pin: 0.000000
> >> > > > lon_orient: -97.000000
> >> > > > d_km: 12.000000
> >> > > > r_km: 6371.200000
> >> > > >
> >> > > >
> >> > > >
> >> > > > * nx: 442 ny: 265 grid_points dimensions(sizes):
> lat(265),
> >> > > > lon(442) variables(dimensions): float32 lat(lat,lon),
float32
> >> > > > lon(lat,lon), float32 Urban(lat,lon)*
> >> > > > groups:
> >> > > >
> >> > > > Following your earlier suggestion (e.g. gen_vx_mask
> >> > > > wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212
*land_mask.nc
> >> > > > <http://land_mask.nc>*
> >> > > > -type data -mask_field 'name="LAND"; level="Z0";' -thresh
eq1), I
> >> went
> >> > > > ahead and executed something similar for Urban type:
> >> > > >
> >> > > > gen_vx_mask
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> >> > > > aqm.t06z.metcro2d.00.nc
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
> >> > > > aqm.t06z.metcro2d.00.nc
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
> >> > > > -type
> >> > > > data -name Urban -mask_field 'name="DLUSE"; level="Z0";'
-thresh
> >> 'eq1'
> >> > > >
> >> > > > However, as you can see from the map, I retain all values
rather
> >> than
> >> > > > getting 0 for non-urban and 1 for urban. I don't see
anything
> >> that's
> >> > > > really different between these two examples other than I
specify
> the
> >> > > > written name of the mask field as Urban. I've included -
and
> >> removing
> >> > -
> >> > > > the single quotes about eq1, but the result ended up being
the
> same.
> >> > > >
> >> > > >
> >> > > >
> >> > > > I also don't see anything alarmingly different when
comparing the
> >> > header
> >> > > > content from this file with the examples that had worked
with
> VGTYP.
> >> > > >
> >> > > > I'll continue trying a couple of other things if some ideas
come
> up.
> >> > If I
> >> > > > make traction, I'll let you know. Take care and have a
good
> >> weekend.
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Fri, Aug 21, 2020 at 4:46 PM John Halley Gotway via RT <
> >> > > > met_help at ucar.edu>
> >> > > > wrote:
> >> > > >
> >> > > > > Ed,
> >> > > > >
> >> > > > > Sounds good. Please let me know if reversing (lon,lat) to
> >> (lat,lon)
> >> > in
> >> > > > the
> >> > > > > NetCDF files solves the problem.
> >> > > > >
> >> > > > > Have a good weekend.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > John
> >> > > > >
> >> > > > > On Fri, Aug 21, 2020 at 10:51 AM Edward Strobach - NOAA
> Affiliate
> >> via
> >> > > RT
> >> > > > <
> >> > > > > met_help at ucar.edu> wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397
> >
> >> > > > > >
> >> > > > > > Hi John,
> >> > > > > >
> >> > > > > > Your plot looks just like mine. Thanks for confirming.
I do
> >> > > believe I
> >> > > > > > tried that, but I'll go ahead and try that suggestion
again.
> >> I'll
> >> > > > attach
> >> > > > > > the result as well. When comparing with VGTYP - an
example
> that
> >> > > worked
> >> > > > > - I
> >> > > > > > noticed the "lat, lon" ordering as opposed to "lon,
lat",
> which
> >> is
> >> > > what
> >> > > > > > prompted me to go that route. I'll test this
carefully.
> Thanks
> >> > > again.
> >> > > > > >
> >> > > > > > On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via
RT <
> >> > > > > > met_help at ucar.edu> wrote:
> >> > > > > >
> >> > > > > > > Ed,
> >> > > > > > >
> >> > > > > > > I pulled that file and took a look. I've attached the
image
> >> that
> >> > > > > > > plot_data_plane generates with the following command:
> >> > > > > > >
> >> > > > > > > *plot_data_plane aqm.t06z.metcro2d.00.nc <
> >> > > > > http://aqm.t06z.metcro2d.00.nc
> >> > > > > > >
> >> > > > > > > aqm.t06z.metcro2d_DLUSE.00.ps <
> >> > > http://aqm.t06z.metcro2d_DLUSE.00.ps>
> >> > > > > > > 'name="DLUSE"; level="(*,*)";'*
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > So there's clearly a problem here! Looking at that
file with
> >> > > ncdump,
> >> > > > I
> >> > > > > > see
> >> > > > > > > that this file is actually setup to follow MET's
internal
> >> NetCDF
> >> > > > > > > conventions rather the the NetCDF Climate-Forecast
(CF)
> >> > convention.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > *ncdump -h aqm.t06z.metcro2d.00.nc <
> >> > http://aqm.t06z.metcro2d.00.nc
> >> > > >*
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Even though we recommend the use of the CF-
convention, the
> MET
> >> > > tools
> >> > > > > > > actually don't write to that convention themselves.
But we
> >> should
> >> > > > > > probably
> >> > > > > > > transition to it! For now, the tools write to a
pretty rigid
> >> > > internal
> >> > > > > > > format where the dimensions are always ordered (lat,
lon).
> >> Your
> >> > > data
> >> > > > is
> >> > > > > > > ordered (lon, lat) and I strongly suspect that's the
reason
> >> for
> >> > > this
> >> > > > > > > behavior. Please try rearranging to write it as (lat,
lon)
> and
> >> > then
> >> > > > run
> >> > > > > > > plot_data_plane on the result to see if it's fixed.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Hope that helps.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > John
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway <
> >> > > johnhg at ucar.edu
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Ed,
> >> > > > > > > >
> >> > > > > > > > Thanks for sending your sample commands and a
description
> of
> >> > the
> >> > > > > > > behavior.
> >> > > > > > > > I see that you've prepared a NetCDF file that
follows the
> >> > > > > > CF-convention,
> >> > > > > > > > but are getting some odd output from gen_vx_mask. I
> suspect
> >> > that
> >> > > > the
> >> > > > > > odd
> >> > > > > > > > output from gen_vx_mask is a red herring, and that
the
> real
> >> > issue
> >> > > > is
> >> > > > > in
> >> > > > > > > > reading the data from that input NetCDF file.
> >> > > > > > > >
> >> > > > > > > > Whenever, getting started with a new dataset in
MET, such
> as
> >> > the
> >> > > > > > > > CF-compliant NetCDF file, it's always a good idea
to plot
> >> the
> >> > > data
> >> > > > > > using
> >> > > > > > > > plot_data_plane. I suspect we'll find that
plot_data_plane
> >> > > creates
> >> > > > > the
> >> > > > > > > same
> >> > > > > > > > sort of odd plot when reading data directly from
that
> input
> >> > file.
> >> > > > And
> >> > > > > > > > that's most likely due to either an issue in how
the data
> is
> >> > > stored
> >> > > > > in
> >> > > > > > > that
> >> > > > > > > > NetCDF file or perhaps some bug in the MET
software.
> >> > > > > > > >
> >> > > > > > > > I'll go log on to WCOSS, retrieve that file, and do
some
> >> > testing
> >> > > > with
> >> > > > > > it.
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > John
> >> > > > > > > >
> >> > > > > > > > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach -
NOAA
> >> > Affiliate
> >> > > > via
> >> > > > > > RT <
> >> > > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > > >
> >> > > > > > > >>
> >> > > > > > > >> <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397
> >> > >
> >> > > > > > > >>
> >> > > > > > > >> Originally, I had processed grib files with
vegetation
> >> type.
> >> > I
> >> > > > used
> >> > > > > > > this
> >> > > > > > > >> record here: 880:Vegetation Type:Integer(0-13)
> >> > > > > > > >> (instant):lambert:surface:level 0:fcst time 3
hrs:from
> >> > > > 202008181800
> >> > > > > > > >>
> >> > > > > > > >> You can see the shape of that is this: >>>
> >> > > > > > > >>
> >> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> >> > > > > > > >> *(288, 468)*
> >> > > > > > > >>
> >> > > > > > > >> You can also see that upon processing this file
through
> >> > > > gen_vx_mask
> >> > > > > in
> >> > > > > > > the
> >> > > > > > > >> same way as done above produces the same sized
matrix -
> not
> >> > > > > > transposed.
> >> > > > > > > >>
> >> > > > > > > >> Dataset('land_mask_15.nc').variables
> >> > > > > > > >> OrderedDict([('lat', <class
'netCDF4._netCDF4.Variable'>
> >> > > > > > > >> float32 lat(lat, lon)
> >> > > > > > > >> long_name: latitude
> >> > > > > > > >> units: degrees_north
> >> > > > > > > >> standard_name: latitude
> >> > > > > > > >> unlimited dimensions:
> >> > > > > > > >> current shape = (288, 468)
> >> > > > > > > >> filling on, default _FillValue of
9.969209968386869e+36
> >> used
> >> > > > > > > >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> >> > > > > > > >> float32 lon(lat, lon)
> >> > > > > > > >> long_name: longitude
> >> > > > > > > >> units: degrees_east
> >> > > > > > > >> standard_name: longitude
> >> > > > > > > >> unlimited dimensions:
> >> > > > > > > >> current shape = (288, 468)
> >> > > > > > > >> filling on, default _FillValue of
9.969209968386869e+36
> >> used
> >> > > > > > > >> ), ('data_mask', <class
'netCDF4._netCDF4.Variable'>
> >> > > > > > > >> float32 data_mask(lat, lon)
> >> > > > > > > >> long_name: data_mask masking region
> >> > > > > > > >> _FillValue: -9999.0
> >> > > > > > > >> mask_type: data==15
> >> > > > > > > >> unlimited dimensions:
> >> > > > > > > >> *current shape = (288, 468)*
> >> > > > > > > >> filling on)])
> >> > > > > > > >>
> >> > > > > > > >> Moreover, given the mask_type, it identified
points
> having
> >> 15
> >> > > as 1
> >> > > > > and
> >> > > > > > > >> others as 0. Therefore I have a 0/1 dataset.
> >> > > > > > > >>
> >> > > > > > > >> This is different than when I process the newly
created
> >> netcdf
> >> > > > file.
> >> > > > > > > The
> >> > > > > > > >> netcdf file that was created has the Urban field
with a
> >> shape
> >> > of
> >> > > > > > > (265,442)
> >> > > > > > > >>
> >> > > > > > > >> When processing it through gen_vx_mask, it gets
> transposed.
> >> > >>>
> >> > > > > > > >>
Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> >> > > > > > > >> *(442, 265)*.
> >> > > > > > > >>
> >> > > > > > > >> So in the original case I have no change to the
> >> dimensionality
> >> > > of
> >> > > > > the
> >> > > > > > > >> dataset, but in the latter case I do. This
happens
> during
> >> the
> >> > > > > > > gen_vx_mask
> >> > > > > > > >> step, and I suspect that this may somehow be
related to
> my
> >> > > > issue. I
> >> > > > > > > don't
> >> > > > > > > >> know if there's anything else I can add that will
help,
> but
> >> > if I
> >> > > > > think
> >> > > > > > > of
> >> > > > > > > >> something I'll forward it along
> >> > > > > > > >>
> >> > > > > > > >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach -
NOAA
> >> > > Affiliate <
> >> > > > > > > >> edward.strobach at noaa.gov> wrote:
> >> > > > > > > >>
> >> > > > > > > >> > Transposing it didn't seem to do anything
either.
> Here's
> >> > the
> >> > > > > > > >> gen_vx_mask
> >> > > > > > > >> > output. It looks okay at first glance:
> >> > > > > > > >> >
> >> > > > > > > >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> >> > > > > > > >> >
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > > > > > > >> > aqm.t06z.metcro2d.00.nc
> >> > > > > > > >> >
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > > > > > > >> > aqm.t06z.metcro2d.00.nc
> >> > > > > > > >> >
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> > > > > > > >> > -type data -mask_field 'name="DLUSE";
level="Z0";'
> >> -thresh
> >> > eq1
> >> > > > > -name
> >> > > > > > > >> Urban
> >> > > > > > > >> > DEBUG 1: Input Grid:
> >> > > > > > > >> >
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > > > > > > >> > aqm.t06z.metcro2d.00.nc
> >> > > > > > > >> > DEBUG 1: Mask File:
> >> > > > > > > >> >
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > > > > > > >> > aqm.t06z.metcro2d.00.nc
> >> > > > > > > >> > DEBUG 2: Parsed Input Grid: Lambert Conformal
(442 x
> 265)
> >> > > > > > > >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal
(442 x
> 265)
> >> > > > > > > >> > DEBUG 1: Output File:
> >> > > > > > > >> >
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> > > > > > > >> >
> >> > > > > > > >> > I'm also attaching the header content of the
file:
> >> > > > > > > >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> >> > > > > > > >> > <class 'netCDF4._netCDF4.Dataset'>
> >> > > > > > > >> > root group (NETCDF4 data model, file format
HDF5):
> >> > > > > > > >> > FileOrigins: NA
> >> > > > > > > >> > MET_version: V9.1
> >> > > > > > > >> > MET_tool: gen_vx_mask
> >> > > > > > > >> > Projection: Lambert Conformal
> >> > > > > > > >> > hemisphere: N
> >> > > > > > > >> > accum: 00
> >> > > > > > > >> > scale_lat_1: 33.000000
> >> > > > > > > >> > scale_lat_2: 45.000000
> >> > > > > > > >> > lat_pin: 21.821000
> >> > > > > > > >> > lon_pin: -120.628000
> >> > > > > > > >> > x_pin: 0.000000
> >> > > > > > > >> > y_pin: 0.000000
> >> > > > > > > >> > lon_orient: -97.000000
> >> > > > > > > >> > d_km: 12.000000
> >> > > > > > > >> > r_km: 6371.200000
> >> > > > > > > >> > nx: 265
> >> > > > > > > >> > ny: 442 grid_points
> >> > > > > > > >> > dimensions(sizes): lon(265), lat(442)
> >> > > > > > > >> > variables(dimensions): float32 LAT(lon,lat),
> float32
> >> > > > > > LON(lon,lat),
> >> > > > > > > >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat),
float32
> >> > > > > PBL(lon,lat),
> >> > > > > > > >> float32
> >> > > > > > > >> > WSPD10(lon,lat), float32 WDIR10(lon,lat),
float32
> >> > > > WSTAR(lon,lat),
> >> > > > > > > >> float32
> >> > > > > > > >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32
> >> HFX(lon,lat),
> >> > > > > float32
> >> > > > > > > >> > GLW(lon,lat), float32 GSW(lon,lat), float32
> >> RGRND(lon,lat),
> >> > > > > float32
> >> > > > > > > >> > Q2(lon,lat)
> >> > > > > > > >> > groups:
> >> > > > > > > >> >
> >> > > > > > > >> > Nothing here seems troubling. You can see that
> >> > > > > DLUSE(lon,lat).shape
> >> > > > > > ==
> >> > > > > > > >> > (265,442)
> >> > > > > > > >> >
> >> > > > > > > >> > The values in the matrix are of type float, not
> integer;
> >> > but,
> >> > > I
> >> > > > > > don't
> >> > > > > > > >> > think that would be an issue.
> >> > > > > > > >> > Dataset('aqm.t06z.metcro2d.00.nc
> ').variables['DLUSE'][:]
> >> > > > > > > >> > masked_array(
> >> > > > > > > >> > data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0,
16.0],
> >> > > > > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0,
16.0],
> >> > > > > > > >> > [16.0, 16.0, 16.0, ..., 16.0, 16.0,
16.0],
> >> > > > > > > >> > ...,
> >> > > > > > > >> > [16.0, 16.0, 16.0, ..., 12.0, 8.0,
14.0],
> >> > > > > > > >> > [16.0, 16.0, 16.0, ..., 8.0, 10.0,
10.0],
> >> > > > > > > >> > [16.0, 16.0, 16.0, ..., 15.0, 8.0,
10.0]],
> >> > > > > > > >> > mask=[[False, False, False, ..., False, False,
> False],
> >> > > > > > > >> > [False, False, False, ..., False, False,
> False],
> >> > > > > > > >> > [False, False, False, ..., False, False,
> False],
> >> > > > > > > >> > ...,
> >> > > > > > > >> > [False, False, False, ..., False, False,
> False],
> >> > > > > > > >> > [False, False, False, ..., False, False,
> False],
> >> > > > > > > >> > [False, False, False, ..., False, False,
> False]],
> >> > > > > > > >> > fill_value=9.96921e+36,
> >> > > > > > > >> > dtype=float32)
> >> > > > > > > >> >
> >> > > > > > > >> > Do I need to change the dtype to integer?
> >> > > > > > > >> >
> >> > > > > > > >> > On Fri, Aug 21, 2020 at 12:30 AM
met_help at ucar.edu via
> >> RT <
> >> > > > > > > >> > met_help at ucar.edu> wrote:
> >> > > > > > > >> >
> >> > > > > > > >> >>
> >> > > > > > > >> >> Greetings,
> >> > > > > > > >> >>
> >> > > > > > > >> >> This message has been automatically generated
in
> >> response
> >> > to
> >> > > > the
> >> > > > > > > >> creation
> >> > > > > > > >> >> of a trouble ticket regarding:
> >> > > > > > > >> >> "mask issues",
> >> > > > > > > >> >> a summary of which appears below.
> >> > > > > > > >> >>
> >> > > > > > > >> >> There is no need to reply to this message right
now.
> >> Your
> >> > > > ticket
> >> > > > > > has
> >> > > > > > > >> >> been assigned an ID of [rt.rap.ucar.edu
#96397].
> >> > > > > > > >> >>
> >> > > > > > > >> >> Please include the string:
> >> > > > > > > >> >>
> >> > > > > > > >> >> [rt.rap.ucar.edu #96397]
> >> > > > > > > >> >>
> >> > > > > > > >> >> in the subject line of all future
correspondence about
> >> this
> >> > > > > issue.
> >> > > > > > To
> >> > > > > > > >> do
> >> > > > > > > >> >> so, you may reply to this message.
> >> > > > > > > >> >>
> >> > > > > > > >> >> For more information, please see:
> >> > > > > > > >> >>
> >> > > > > > > >> >> MET Online Tutorial:
> >> > > > > > > >> >>
> >> > > > > > > >>
> >> > > > >
> >>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> >> > > > > > > >> >>
> >> > > > > > > >> >> MET Users Guide:
> >> > > > > > > >> >>
> https://www.dtcenter.org/met/users/docs/overview.php
> >> > > > > > > >> >>
> >> > > > > > > >> >> MET FAQs:
> >> > > > > > > >> >>
> >> > https://www.dtcenter.org/met/users/support/faqs/index.php
> >> > > > > > > >> >>
> >> > > > > > > >> >> MET-Help Email Archive:
> >> > > > > > > >> >> http://mailman.ucar.edu/pipermail/met_help
> >> > > > > > > >> >>
> >> > > > > > > >> >> Thank you,
> >> > > > > > > >> >> met_help at ucar.edu
> >> > > > > > > >> >>
> >> > > > > > > >> >>
> >> > > > > > > >>
> >> > > > > > >
> >> > > > >
> >> > >
> >>
>
-------------------------------------------------------------------------
> >> > > > > > > >> >> GOod evening,
> >> > > > > > > >> >>
> >> > > > > > > >> >> I thought I understood how to process the data
into
> >> masked
> >> > > > > regions,
> >> > > > > > > but
> >> > > > > > > >> >> apparently not. I'm using data that I
processed into
> a
> >> > > netcdf
> >> > > > > file
> >> > > > > > > >> that
> >> > > > > > > >> >> is
> >> > > > > > > >> >> now CF compatible. I'm processing the DLUSE
field
> from
> >> > this
> >> > > > > > > directory:
> >> > > > > > > >> >>
> >> > > > > > > >> >>
> >> > > > > > > >> >>
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > > > > > > >> >> aqm.t06z.metcro2d.00.nc
> >> > > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>*
> >> > > > > > > >> >>
> >> > > > > > > >> >> I then run this line, which is inside a loop
within a
> >> > script
> >> > > > that
> >> > > > > > I'm
> >> > > > > > > >> >> using. The command looks like this:
> >> > > > > > > >> >> *gen_vx_mask
> >> > > > > > > >> >>
> >> > > > > > > >> >>
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > > > > > > >> >> aqm.t06z.metcro2d.00.nc
> >> > > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> >> > > > > > > >> >>
> >> > > > > > > >> >>
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > > > > > > >> >> aqm.t06z.metcro2d.00.nc
> >> > > > > > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> >> > > > > > > >> >>
> >> > > > > > > >> >>
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> > > > > > > >> >> -type data -mask_field 'name="DLUSE";
level="Z0";'
> >> -thresh
> >> > > eq1
> >> > > > > > -name
> >> > > > > > > >> >> Urban*
> >> > > > > > > >> >>
> >> > > > > > > >> >> I determined that something was wrong when much
of my
> >> > masked
> >> > > > land
> >> > > > > > > >> plots -
> >> > > > > > > >> >> all 24 of them - looked more or less the same.
When
> >> > > inspecting
> >> > > > > > > Urban,
> >> > > > > > > >> I
> >> > > > > > > >> >> found that there were 588 matched pairs! Since
I
> >> realize
> >> > > that
> >> > > > we
> >> > > > > > > don't
> >> > > > > > > >> >> live on Coruscant, I decided to plot the data
and this
> >> is
> >> > > what
> >> > > > it
> >> > > > > > > >> looked
> >> > > > > > > >> >> like (see attached: fig 1). The original DLUSE
field
> >> looks
> >> > > > like
> >> > > > > > what
> >> > > > > > > >> you
> >> > > > > > > >> >> see in the second attachment. I'm pretty sure
that
> the
> >> > > > > gen_vx_mask
> >> > > > > > > was
> >> > > > > > > >> >> utilized correctly here. I have the file that
I'm
> >> using,
> >> > the
> >> > > > > file
> >> > > > > > > that
> >> > > > > > > >> >> I'm
> >> > > > > > > >> >> creating to store the result, the type (data),
the
> mask
> >> > field
> >> > > > > > (DLUSE
> >> > > > > > > at
> >> > > > > > > >> >> the
> >> > > > > > > >> >> surface), the threshold (set equal to 1 for
urban),
> and
> >> the
> >> > > > name
> >> > > > > of
> >> > > > > > > the
> >> > > > > > > >> >> field in the file (Urban). One of the things I
> noticed
> >> was
> >> > > > that
> >> > > > > > the
> >> > > > > > > >> >> output
> >> > > > > > > >> >> from the netcdf file that I generated is
transposed
> from
> >> > the
> >> > > > > > > original.
> >> > > > > > > >> >> The
> >> > > > > > > >> >> matrix was originally 265 X 442. However, in
the file
> >> that
> >> > > was
> >> > > > > > > >> created it
> >> > > > > > > >> >> is 442X265. I'm not sure what's going on here
at all
> >> to be
> >> > > > > honest.
> >> > > > > > > I
> >> > > > > > > >> >> thought it would be as simple as running the
command
> >> with
> >> > the
> >> > > > > above
> >> > > > > > > >> >> settings. I may try to create netcdf files
with the
> >> matrix
> >> > > > being
> >> > > > > > > >> 442X265.
> >> > > > > > > >> >> Something clearly is off here.
> >> > > > > > > >> >>
> >> > > > > > > >> >> --
> >> > > > > > > >> >> Edward Strobach
> >> > > > > > > >> >> EMC/NCEP/NWS/
> >> > > > > > > >> >> IMSG Contractor
> >> > > > > > > >> >> Cubicle#: 2029
> >> > > > > > > >> >> 301-683-3717
> >> > > > > > > >> >>
> >> > > > > > > >> >>
> >> > > > > > > >> >
> >> > > > > > > >> > --
> >> > > > > > > >> > Edward Strobach
> >> > > > > > > >> > EMC/NCEP/NWS/
> >> > > > > > > >> > IMSG Contractor
> >> > > > > > > >> > Cubicle#: 2029
> >> > > > > > > >> > 301-683-3717
> >> > > > > > > >> >
> >> > > > > > > >>
> >> > > > > > > >>
> >> > > > > > > >> --
> >> > > > > > > >> Edward Strobach
> >> > > > > > > >> EMC/NCEP/NWS/
> >> > > > > > > >> IMSG Contractor
> >> > > > > > > >> Cubicle#: 2029
> >> > > > > > > >> 301-683-3717
> >> > > > > > > >>
> >> > > > > > > >>
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Edward Strobach
> >> > > > > > EMC/NCEP/NWS/
> >> > > > > > IMSG Contractor
> >> > > > > > Cubicle#: 2029
> >> > > > > > 301-683-3717
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > > --
> >> > > > Edward Strobach
> >> > > > EMC/NCEP/NWS/
> >> > > > IMSG Contractor
> >> > > > Cubicle#: 2029
> >> > > > 301-683-3717
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > Edward Strobach
> >> > EMC/NCEP/NWS/
> >> > IMSG Contractor
> >> > Cubicle#: 2029
> >> > 301-683-3717
> >> >
> >> >
> >>
> >>
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
More information about the Met_help
mailing list