[Met_help] [rt.rap.ucar.edu #96397] History for mask issues

John Halley Gotway via RT met_help at ucar.edu
Mon Aug 24 10:58:56 MDT 2020


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

GOod evening,

I thought I understood how to process the data into masked regions, but
apparently not.  I'm using data that I processed into a netcdf file that is
now CF compatible.  I'm processing the DLUSE field from this directory:

*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>*

I then run this line, which is inside a loop within a script that I'm
using.  The command looks like this:
*gen_vx_mask
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
-type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1 -name Urban*

I determined that something was wrong when much of my masked land plots -
all 24 of them - looked more or less the same.  When inspecting Urban, I
found that there were 588 matched pairs!  Since I realize that we don't
live on Coruscant, I decided to plot the data and this is what it looked
like (see attached: fig 1).  The original DLUSE field looks like what you
see in the second attachment.  I'm pretty sure that the gen_vx_mask was
utilized correctly here.  I have the file that I'm using, the file that I'm
creating to store the result, the type (data), the mask field (DLUSE at the
surface), the threshold (set equal to 1 for urban), and the name of the
field in the file (Urban).  One of the things I noticed was that the output
from the netcdf file that I generated is transposed from the original.  The
matrix was originally 265 X 442.  However, in the file that was created it
is 442X265.  I'm not sure what's going on here at all to be honest.  I
thought it would be as simple as running the command with the above
settings.  I may try to create netcdf files with the matrix being 442X265.
Something clearly is off here.

-- 
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Fri Aug 21 07:28:27 2020

Transposing it didn't seem to do anything either.  Here's the
gen_vx_mask
output.  It looks okay at first glance:

[Edward.Strobach at m71a1 scripts]$ gen_vx_mask
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
aqm.t06z.metcro2d.00.nc
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
aqm.t06z.metcro2d.00.nc
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
-type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1 -name
Urban
DEBUG 1: Input Grid:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
aqm.t06z.metcro2d.00.nc
DEBUG 1: Mask File:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
aqm.t06z.metcro2d.00.nc
DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
DEBUG 1: Output File:
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc

I'm also attaching the header content of the file:
>>> Dataset('aqm.t06z.metcro2d.00.nc')
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):
    FileOrigins: NA
    MET_version: V9.1
    MET_tool: gen_vx_mask
    Projection: Lambert Conformal
    hemisphere: N
    accum: 00
    scale_lat_1: 33.000000
    scale_lat_2: 45.000000
    lat_pin: 21.821000
    lon_pin: -120.628000
    x_pin: 0.000000
    y_pin: 0.000000
    lon_orient: -97.000000
    d_km: 12.000000
    r_km: 6371.200000
    nx: 265
    ny: 442 grid_points
    dimensions(sizes): lon(265), lat(442)
    variables(dimensions): float32 LAT(lon,lat), float32 LON(lon,lat),
float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32 PBL(lon,lat),
float32
WSPD10(lon,lat), float32 WDIR10(lon,lat), float32 WSTAR(lon,lat),
float32
USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat), float32
GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat), float32
Q2(lon,lat)
    groups:

Nothing here seems troubling. You can see that DLUSE(lon,lat).shape ==
(265,442)

The values in the matrix are of type float, not integer; but, I don't
think
that would be an issue.
Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
masked_array(
  data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
        [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
        [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
        ...,
        [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
        [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
        [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
  mask=[[False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        ...,
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False],
        [False, False, False, ..., False, False, False]],
  fill_value=9.96921e+36,
  dtype=float32)

Do I need to change the dtype to integer?

On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT
<met_help at ucar.edu>
wrote:

>
> Greetings,
>
> This message has been automatically generated in response to the
creation
> of a trouble ticket regarding:
>         "mask issues",
> a summary of which appears below.
>
> There is no need to reply to this message right now.  Your ticket
has been
> assigned an ID of [rt.rap.ucar.edu #96397].
>
> Please include the string:
>
>          [rt.rap.ucar.edu #96397]
>
> in the subject line of all future correspondence about this issue.
To do
> so, you may reply to this message.
>
> For more information, please see:
>
> MET Online Tutorial:
>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>
> MET Users Guide:
>    https://www.dtcenter.org/met/users/docs/overview.php
>
> MET FAQs:
>    https://www.dtcenter.org/met/users/support/faqs/index.php
>
> MET-Help Email Archive:
>    http://mailman.ucar.edu/pipermail/met_help
>
>                         Thank you,
>                         met_help at ucar.edu
>
>
-------------------------------------------------------------------------
> GOod evening,
>
> I thought I understood how to process the data into masked regions,
but
> apparently not.  I'm using data that I processed into a netcdf file
that is
> now CF compatible.  I'm processing the DLUSE field from this
directory:
>
>
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
> <http://aqm.t06z.metcro2d.00.nc>*
>
> I then run this line, which is inside a loop within a script that
I'm
> using.  The command looks like this:
> *gen_vx_mask
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
> <http://aqm.t06z.metcro2d.00.nc>
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
> <http://aqm.t06z.metcro2d.00.nc>
>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1 -name
Urban*
>
> I determined that something was wrong when much of my masked land
plots -
> all 24 of them - looked more or less the same.  When inspecting
Urban, I
> found that there were 588 matched pairs!  Since I realize that we
don't
> live on Coruscant, I decided to plot the data and this is what it
looked
> like (see attached: fig 1).  The original DLUSE field looks like
what you
> see in the second attachment.  I'm pretty sure that the gen_vx_mask
was
> utilized correctly here.  I have the file that I'm using, the file
that I'm
> creating to store the result, the type (data), the mask field (DLUSE
at the
> surface), the threshold (set equal to 1 for urban), and the name of
the
> field in the file (Urban).  One of the things I noticed was that the
output
> from the netcdf file that I generated is transposed from the
original.  The
> matrix was originally 265 X 442.  However, in the file that was
created it
> is 442X265.  I'm not sure what's going on here at all to be honest.
I
> thought it would be as simple as running the command with the above
> settings.  I may try to create netcdf files with the matrix being
442X265.
> Something clearly is off here.
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Fri Aug 21 08:05:41 2020

Originally, I had processed grib files with vegetation type.  I used
this
record here: 880:Vegetation Type:Integer(0-13)
(instant):lambert:surface:level 0:fcst time 3 hrs:from 202008181800

You can see the shape of that is this: >>>
pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
*(288, 468)*

You can also see that upon processing this file through gen_vx_mask in
the
same way as done above produces the same sized matrix - not
transposed.

 Dataset('land_mask_15.nc').variables
OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
float32 lat(lat, lon)
    long_name: latitude
    units: degrees_north
    standard_name: latitude
unlimited dimensions:
current shape = (288, 468)
filling on, default _FillValue of 9.969209968386869e+36 used
), ('lon', <class 'netCDF4._netCDF4.Variable'>
float32 lon(lat, lon)
    long_name: longitude
    units: degrees_east
    standard_name: longitude
unlimited dimensions:
current shape = (288, 468)
filling on, default _FillValue of 9.969209968386869e+36 used
), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
float32 data_mask(lat, lon)
    long_name: data_mask masking region
    _FillValue: -9999.0
    mask_type: data==15
unlimited dimensions:
*current shape = (288, 468)*
filling on)])

Moreover, given the mask_type, it identified points having 15 as 1 and
others as 0.  Therefore I have a 0/1 dataset.

This is different than when I process the newly created netcdf file.
The
netcdf file that was created has the Urban field with a shape of
(265,442)

When processing it through gen_vx_mask, it gets transposed.  >>>
Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
*(442, 265)*.

So in the original case I have no change to the dimensionality of the
dataset, but in the latter case I do.  This happens during the
gen_vx_mask
step, and I suspect that this may somehow be related to my issue.  I
don't
know if there's anything else I can add that will help, but if I think
of
something I'll forward it along

On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> Transposing it didn't seem to do anything either.  Here's the
gen_vx_mask
> output.  It looks okay at first glance:
>
> [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1 -name
Urban
> DEBUG 1: Input Grid:
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
> DEBUG 1: Mask File:
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> aqm.t06z.metcro2d.00.nc
> DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> DEBUG 1: Output File:
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>
> I'm also attaching the header content of the file:
> >>> Dataset('aqm.t06z.metcro2d.00.nc')
> <class 'netCDF4._netCDF4.Dataset'>
> root group (NETCDF4 data model, file format HDF5):
>     FileOrigins: NA
>     MET_version: V9.1
>     MET_tool: gen_vx_mask
>     Projection: Lambert Conformal
>     hemisphere: N
>     accum: 00
>     scale_lat_1: 33.000000
>     scale_lat_2: 45.000000
>     lat_pin: 21.821000
>     lon_pin: -120.628000
>     x_pin: 0.000000
>     y_pin: 0.000000
>     lon_orient: -97.000000
>     d_km: 12.000000
>     r_km: 6371.200000
>     nx: 265
>     ny: 442 grid_points
>     dimensions(sizes): lon(265), lat(442)
>     variables(dimensions): float32 LAT(lon,lat), float32
LON(lon,lat),
> float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32 PBL(lon,lat),
float32
> WSPD10(lon,lat), float32 WDIR10(lon,lat), float32 WSTAR(lon,lat),
float32
> USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat), float32
> GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat), float32
> Q2(lon,lat)
>     groups:
>
> Nothing here seems troubling. You can see that DLUSE(lon,lat).shape
==
> (265,442)
>
> The values in the matrix are of type float, not integer; but, I
don't
> think that would be an issue.
> Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> masked_array(
>   data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>         ...,
>         [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
>         [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
>         [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
>   mask=[[False, False, False, ..., False, False, False],
>         [False, False, False, ..., False, False, False],
>         [False, False, False, ..., False, False, False],
>         ...,
>         [False, False, False, ..., False, False, False],
>         [False, False, False, ..., False, False, False],
>         [False, False, False, ..., False, False, False]],
>   fill_value=9.96921e+36,
>   dtype=float32)
>
> Do I need to change the dtype to integer?
>
> On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Greetings,
>>
>> This message has been automatically generated in response to the
creation
>> of a trouble ticket regarding:
>>         "mask issues",
>> a summary of which appears below.
>>
>> There is no need to reply to this message right now.  Your ticket
has
>> been assigned an ID of [rt.rap.ucar.edu #96397].
>>
>> Please include the string:
>>
>>          [rt.rap.ucar.edu #96397]
>>
>> in the subject line of all future correspondence about this issue.
To do
>> so, you may reply to this message.
>>
>> For more information, please see:
>>
>> MET Online Tutorial:
>>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>>
>> MET Users Guide:
>>    https://www.dtcenter.org/met/users/docs/overview.php
>>
>> MET FAQs:
>>    https://www.dtcenter.org/met/users/support/faqs/index.php
>>
>> MET-Help Email Archive:
>>    http://mailman.ucar.edu/pipermail/met_help
>>
>>                         Thank you,
>>                         met_help at ucar.edu
>>
>>
-------------------------------------------------------------------------
>> GOod evening,
>>
>> I thought I understood how to process the data into masked regions,
but
>> apparently not.  I'm using data that I processed into a netcdf file
that
>> is
>> now CF compatible.  I'm processing the DLUSE field from this
directory:
>>
>>
>>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> aqm.t06z.metcro2d.00.nc
>> <http://aqm.t06z.metcro2d.00.nc>*
>>
>> I then run this line, which is inside a loop within a script that
I'm
>> using.  The command looks like this:
>> *gen_vx_mask
>>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> aqm.t06z.metcro2d.00.nc
>> <http://aqm.t06z.metcro2d.00.nc>
>>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> aqm.t06z.metcro2d.00.nc
>> <http://aqm.t06z.metcro2d.00.nc>
>>
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
>> Urban*
>>
>> I determined that something was wrong when much of my masked land
plots -
>> all 24 of them - looked more or less the same.  When inspecting
Urban, I
>> found that there were 588 matched pairs!  Since I realize that we
don't
>> live on Coruscant, I decided to plot the data and this is what it
looked
>> like (see attached: fig 1).  The original DLUSE field looks like
what you
>> see in the second attachment.  I'm pretty sure that the gen_vx_mask
was
>> utilized correctly here.  I have the file that I'm using, the file
that
>> I'm
>> creating to store the result, the type (data), the mask field
(DLUSE at
>> the
>> surface), the threshold (set equal to 1 for urban), and the name of
the
>> field in the file (Urban).  One of the things I noticed was that
the
>> output
>> from the netcdf file that I generated is transposed from the
original.
>> The
>> matrix was originally 265 X 442.  However, in the file that was
created it
>> is 442X265.  I'm not sure what's going on here at all to be honest.
I
>> thought it would be as simple as running the command with the above
>> settings.  I may try to create netcdf files with the matrix being
442X265.
>> Something clearly is off here.
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>


--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: mask issues
From: John Halley Gotway
Time: Fri Aug 21 10:08:40 2020

Ed,

Thanks for sending your sample commands and a description of the
behavior.
I see that you've prepared a NetCDF file that follows the CF-
convention,
but are getting some odd output from gen_vx_mask. I suspect that the
odd
output from gen_vx_mask is a red herring, and that the real issue is
in
reading the data from that input NetCDF file.

Whenever, getting started with a new dataset in MET, such as the
CF-compliant NetCDF file, it's always a good idea to plot the data
using
plot_data_plane. I suspect we'll find that plot_data_plane creates the
same
sort of odd plot when reading data directly from that input file. And
that's most likely due to either an issue in how the data is stored in
that
NetCDF file or perhaps some bug in the MET software.

I'll go log on to WCOSS, retrieve that file, and do some testing with
it.

Thanks,
John

On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>
> Originally, I had processed grib files with vegetation type.  I used
this
> record here: 880:Vegetation Type:Integer(0-13)
> (instant):lambert:surface:level 0:fcst time 3 hrs:from 202008181800
>
> You can see the shape of that is this: >>>
> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> *(288, 468)*
>
> You can also see that upon processing this file through gen_vx_mask
in the
> same way as done above produces the same sized matrix - not
transposed.
>
>  Dataset('land_mask_15.nc').variables
> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> float32 lat(lat, lon)
>     long_name: latitude
>     units: degrees_north
>     standard_name: latitude
> unlimited dimensions:
> current shape = (288, 468)
> filling on, default _FillValue of 9.969209968386869e+36 used
> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> float32 lon(lat, lon)
>     long_name: longitude
>     units: degrees_east
>     standard_name: longitude
> unlimited dimensions:
> current shape = (288, 468)
> filling on, default _FillValue of 9.969209968386869e+36 used
> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> float32 data_mask(lat, lon)
>     long_name: data_mask masking region
>     _FillValue: -9999.0
>     mask_type: data==15
> unlimited dimensions:
> *current shape = (288, 468)*
> filling on)])
>
> Moreover, given the mask_type, it identified points having 15 as 1
and
> others as 0.  Therefore I have a 0/1 dataset.
>
> This is different than when I process the newly created netcdf file.
The
> netcdf file that was created has the Urban field with a shape of
(265,442)
>
> When processing it through gen_vx_mask, it gets transposed.  >>>
> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> *(442, 265)*.
>
> So in the original case I have no change to the dimensionality of
the
> dataset, but in the latter case I do.  This happens during the
gen_vx_mask
> step, and I suspect that this may somehow be related to my issue.  I
don't
> know if there's anything else I can add that will help, but if I
think of
> something I'll forward it along
>
> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Transposing it didn't seem to do anything either.  Here's the
gen_vx_mask
> > output.  It looks okay at first glance:
> >
> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > aqm.t06z.metcro2d.00.nc
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > aqm.t06z.metcro2d.00.nc
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> Urban
> > DEBUG 1: Input Grid:
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > aqm.t06z.metcro2d.00.nc
> > DEBUG 1: Mask File:
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > aqm.t06z.metcro2d.00.nc
> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> > DEBUG 1: Output File:
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >
> > I'm also attaching the header content of the file:
> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > <class 'netCDF4._netCDF4.Dataset'>
> > root group (NETCDF4 data model, file format HDF5):
> >     FileOrigins: NA
> >     MET_version: V9.1
> >     MET_tool: gen_vx_mask
> >     Projection: Lambert Conformal
> >     hemisphere: N
> >     accum: 00
> >     scale_lat_1: 33.000000
> >     scale_lat_2: 45.000000
> >     lat_pin: 21.821000
> >     lon_pin: -120.628000
> >     x_pin: 0.000000
> >     y_pin: 0.000000
> >     lon_orient: -97.000000
> >     d_km: 12.000000
> >     r_km: 6371.200000
> >     nx: 265
> >     ny: 442 grid_points
> >     dimensions(sizes): lon(265), lat(442)
> >     variables(dimensions): float32 LAT(lon,lat), float32
LON(lon,lat),
> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
PBL(lon,lat),
> float32
> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32 WSTAR(lon,lat),
float32
> > USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat),
float32
> > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
float32
> > Q2(lon,lat)
> >     groups:
> >
> > Nothing here seems troubling. You can see that
DLUSE(lon,lat).shape ==
> > (265,442)
> >
> > The values in the matrix are of type float, not integer; but, I
don't
> > think that would be an issue.
> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > masked_array(
> >   data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> >         ...,
> >         [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> >         [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> >         [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> >   mask=[[False, False, False, ..., False, False, False],
> >         [False, False, False, ..., False, False, False],
> >         [False, False, False, ..., False, False, False],
> >         ...,
> >         [False, False, False, ..., False, False, False],
> >         [False, False, False, ..., False, False, False],
> >         [False, False, False, ..., False, False, False]],
> >   fill_value=9.96921e+36,
> >   dtype=float32)
> >
> > Do I need to change the dtype to integer?
> >
> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> Greetings,
> >>
> >> This message has been automatically generated in response to the
> creation
> >> of a trouble ticket regarding:
> >>         "mask issues",
> >> a summary of which appears below.
> >>
> >> There is no need to reply to this message right now.  Your ticket
has
> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> >>
> >> Please include the string:
> >>
> >>          [rt.rap.ucar.edu #96397]
> >>
> >> in the subject line of all future correspondence about this
issue. To do
> >> so, you may reply to this message.
> >>
> >> For more information, please see:
> >>
> >> MET Online Tutorial:
> >>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> >>
> >> MET Users Guide:
> >>    https://www.dtcenter.org/met/users/docs/overview.php
> >>
> >> MET FAQs:
> >>    https://www.dtcenter.org/met/users/support/faqs/index.php
> >>
> >> MET-Help Email Archive:
> >>    http://mailman.ucar.edu/pipermail/met_help
> >>
> >>                         Thank you,
> >>                         met_help at ucar.edu
> >>
> >>
>
-------------------------------------------------------------------------
> >> GOod evening,
> >>
> >> I thought I understood how to process the data into masked
regions, but
> >> apparently not.  I'm using data that I processed into a netcdf
file that
> >> is
> >> now CF compatible.  I'm processing the DLUSE field from this
directory:
> >>
> >>
> >>
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> aqm.t06z.metcro2d.00.nc
> >> <http://aqm.t06z.metcro2d.00.nc>*
> >>
> >> I then run this line, which is inside a loop within a script that
I'm
> >> using.  The command looks like this:
> >> *gen_vx_mask
> >>
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> aqm.t06z.metcro2d.00.nc
> >> <http://aqm.t06z.metcro2d.00.nc>
> >>
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> aqm.t06z.metcro2d.00.nc
> >> <http://aqm.t06z.metcro2d.00.nc>
> >>
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> >> Urban*
> >>
> >> I determined that something was wrong when much of my masked land
plots
> -
> >> all 24 of them - looked more or less the same.  When inspecting
Urban, I
> >> found that there were 588 matched pairs!  Since I realize that we
don't
> >> live on Coruscant, I decided to plot the data and this is what it
looked
> >> like (see attached: fig 1).  The original DLUSE field looks like
what
> you
> >> see in the second attachment.  I'm pretty sure that the
gen_vx_mask was
> >> utilized correctly here.  I have the file that I'm using, the
file that
> >> I'm
> >> creating to store the result, the type (data), the mask field
(DLUSE at
> >> the
> >> surface), the threshold (set equal to 1 for urban), and the name
of the
> >> field in the file (Urban).  One of the things I noticed was that
the
> >> output
> >> from the netcdf file that I generated is transposed from the
original.
> >> The
> >> matrix was originally 265 X 442.  However, in the file that was
created
> it
> >> is 442X265.  I'm not sure what's going on here at all to be
honest.  I
> >> thought it would be as simple as running the command with the
above
> >> settings.  I may try to create netcdf files with the matrix being
> 442X265.
> >> Something clearly is off here.
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >>
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>

------------------------------------------------
Subject: mask issues
From: John Halley Gotway
Time: Fri Aug 21 10:41:41 2020

Ed,

I pulled that file and took a look. I've attached the image that
plot_data_plane generates with the following command:

*plot_data_plane aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>
aqm.t06z.metcro2d_DLUSE.00.ps <http://aqm.t06z.metcro2d_DLUSE.00.ps>
'name="DLUSE"; level="(*,*)";'*



So there's clearly a problem here! Looking at that file with ncdump, I
see
that this file is actually setup to follow MET's internal NetCDF
conventions rather the the NetCDF Climate-Forecast (CF) convention.


*ncdump -h aqm.t06z.metcro2d.00.nc <http://aqm.t06z.metcro2d.00.nc>*



Even though we recommend the use of the CF-convention, the MET tools
actually don't write to that convention themselves. But we should
probably
transition to it! For now, the tools write to a pretty rigid internal
format where the dimensions are always ordered (lat, lon). Your data
is
ordered (lon, lat) and I strongly suspect that's the reason for this
behavior. Please try rearranging to write it as (lat, lon) and then
run
plot_data_plane on the result to see if it's fixed.


Hope that helps.


Thanks,
John



On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway <johnhg at ucar.edu>
wrote:

> Ed,
>
> Thanks for sending your sample commands and a description of the
behavior.
> I see that you've prepared a NetCDF file that follows the CF-
convention,
> but are getting some odd output from gen_vx_mask. I suspect that the
odd
> output from gen_vx_mask is a red herring, and that the real issue is
in
> reading the data from that input NetCDF file.
>
> Whenever, getting started with a new dataset in MET, such as the
> CF-compliant NetCDF file, it's always a good idea to plot the data
using
> plot_data_plane. I suspect we'll find that plot_data_plane creates
the same
> sort of odd plot when reading data directly from that input file.
And
> that's most likely due to either an issue in how the data is stored
in that
> NetCDF file or perhaps some bug in the MET software.
>
> I'll go log on to WCOSS, retrieve that file, and do some testing
with it.
>
> Thanks,
> John
>
> On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>>
>> Originally, I had processed grib files with vegetation type.  I
used this
>> record here: 880:Vegetation Type:Integer(0-13)
>> (instant):lambert:surface:level 0:fcst time 3 hrs:from 202008181800
>>
>> You can see the shape of that is this: >>>
>> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
>> *(288, 468)*
>>
>> You can also see that upon processing this file through gen_vx_mask
in the
>> same way as done above produces the same sized matrix - not
transposed.
>>
>>  Dataset('land_mask_15.nc').variables
>> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
>> float32 lat(lat, lon)
>>     long_name: latitude
>>     units: degrees_north
>>     standard_name: latitude
>> unlimited dimensions:
>> current shape = (288, 468)
>> filling on, default _FillValue of 9.969209968386869e+36 used
>> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
>> float32 lon(lat, lon)
>>     long_name: longitude
>>     units: degrees_east
>>     standard_name: longitude
>> unlimited dimensions:
>> current shape = (288, 468)
>> filling on, default _FillValue of 9.969209968386869e+36 used
>> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
>> float32 data_mask(lat, lon)
>>     long_name: data_mask masking region
>>     _FillValue: -9999.0
>>     mask_type: data==15
>> unlimited dimensions:
>> *current shape = (288, 468)*
>> filling on)])
>>
>> Moreover, given the mask_type, it identified points having 15 as 1
and
>> others as 0.  Therefore I have a 0/1 dataset.
>>
>> This is different than when I process the newly created netcdf
file.  The
>> netcdf file that was created has the Urban field with a shape of
(265,442)
>>
>> When processing it through gen_vx_mask, it gets transposed.  >>>
>> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
>> *(442, 265)*.
>>
>> So in the original case I have no change to the dimensionality of
the
>> dataset, but in the latter case I do.  This happens during the
gen_vx_mask
>> step, and I suspect that this may somehow be related to my issue.
I don't
>> know if there's anything else I can add that will help, but if I
think of
>> something I'll forward it along
>>
>> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA Affiliate <
>> edward.strobach at noaa.gov> wrote:
>>
>> > Transposing it didn't seem to do anything either.  Here's the
>> gen_vx_mask
>> > output.  It looks okay at first glance:
>> >
>> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > aqm.t06z.metcro2d.00.nc
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > aqm.t06z.metcro2d.00.nc
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
>> Urban
>> > DEBUG 1: Input Grid:
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > aqm.t06z.metcro2d.00.nc
>> > DEBUG 1: Mask File:
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> > aqm.t06z.metcro2d.00.nc
>> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
>> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
>> > DEBUG 1: Output File:
>> >
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> >
>> > I'm also attaching the header content of the file:
>> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
>> > <class 'netCDF4._netCDF4.Dataset'>
>> > root group (NETCDF4 data model, file format HDF5):
>> >     FileOrigins: NA
>> >     MET_version: V9.1
>> >     MET_tool: gen_vx_mask
>> >     Projection: Lambert Conformal
>> >     hemisphere: N
>> >     accum: 00
>> >     scale_lat_1: 33.000000
>> >     scale_lat_2: 45.000000
>> >     lat_pin: 21.821000
>> >     lon_pin: -120.628000
>> >     x_pin: 0.000000
>> >     y_pin: 0.000000
>> >     lon_orient: -97.000000
>> >     d_km: 12.000000
>> >     r_km: 6371.200000
>> >     nx: 265
>> >     ny: 442 grid_points
>> >     dimensions(sizes): lon(265), lat(442)
>> >     variables(dimensions): float32 LAT(lon,lat), float32
LON(lon,lat),
>> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
PBL(lon,lat),
>> float32
>> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32 WSTAR(lon,lat),
>> float32
>> > USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat),
float32
>> > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
float32
>> > Q2(lon,lat)
>> >     groups:
>> >
>> > Nothing here seems troubling. You can see that
DLUSE(lon,lat).shape ==
>> > (265,442)
>> >
>> > The values in the matrix are of type float, not integer; but, I
don't
>> > think that would be an issue.
>> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
>> > masked_array(
>> >   data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>> >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>> >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
>> >         ...,
>> >         [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
>> >         [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
>> >         [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
>> >   mask=[[False, False, False, ..., False, False, False],
>> >         [False, False, False, ..., False, False, False],
>> >         [False, False, False, ..., False, False, False],
>> >         ...,
>> >         [False, False, False, ..., False, False, False],
>> >         [False, False, False, ..., False, False, False],
>> >         [False, False, False, ..., False, False, False]],
>> >   fill_value=9.96921e+36,
>> >   dtype=float32)
>> >
>> > Do I need to change the dtype to integer?
>> >
>> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >>
>> >> Greetings,
>> >>
>> >> This message has been automatically generated in response to the
>> creation
>> >> of a trouble ticket regarding:
>> >>         "mask issues",
>> >> a summary of which appears below.
>> >>
>> >> There is no need to reply to this message right now.  Your
ticket has
>> >> been assigned an ID of [rt.rap.ucar.edu #96397].
>> >>
>> >> Please include the string:
>> >>
>> >>          [rt.rap.ucar.edu #96397]
>> >>
>> >> in the subject line of all future correspondence about this
issue. To
>> do
>> >> so, you may reply to this message.
>> >>
>> >> For more information, please see:
>> >>
>> >> MET Online Tutorial:
>> >>
>>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>> >>
>> >> MET Users Guide:
>> >>    https://www.dtcenter.org/met/users/docs/overview.php
>> >>
>> >> MET FAQs:
>> >>    https://www.dtcenter.org/met/users/support/faqs/index.php
>> >>
>> >> MET-Help Email Archive:
>> >>    http://mailman.ucar.edu/pipermail/met_help
>> >>
>> >>                         Thank you,
>> >>                         met_help at ucar.edu
>> >>
>> >>
>>
-------------------------------------------------------------------------
>> >> GOod evening,
>> >>
>> >> I thought I understood how to process the data into masked
regions, but
>> >> apparently not.  I'm using data that I processed into a netcdf
file
>> that
>> >> is
>> >> now CF compatible.  I'm processing the DLUSE field from this
directory:
>> >>
>> >>
>> >>
>>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> >> aqm.t06z.metcro2d.00.nc
>> >> <http://aqm.t06z.metcro2d.00.nc>*
>> >>
>> >> I then run this line, which is inside a loop within a script
that I'm
>> >> using.  The command looks like this:
>> >> *gen_vx_mask
>> >>
>> >>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> >> aqm.t06z.metcro2d.00.nc
>> >> <http://aqm.t06z.metcro2d.00.nc>
>> >>
>> >>
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
>> >> aqm.t06z.metcro2d.00.nc
>> >> <http://aqm.t06z.metcro2d.00.nc>
>> >>
>> >>
>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
>> >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
>> >> Urban*
>> >>
>> >> I determined that something was wrong when much of my masked
land
>> plots -
>> >> all 24 of them - looked more or less the same.  When inspecting
Urban,
>> I
>> >> found that there were 588 matched pairs!  Since I realize that
we don't
>> >> live on Coruscant, I decided to plot the data and this is what
it
>> looked
>> >> like (see attached: fig 1).  The original DLUSE field looks like
what
>> you
>> >> see in the second attachment.  I'm pretty sure that the
gen_vx_mask was
>> >> utilized correctly here.  I have the file that I'm using, the
file that
>> >> I'm
>> >> creating to store the result, the type (data), the mask field
(DLUSE at
>> >> the
>> >> surface), the threshold (set equal to 1 for urban), and the name
of the
>> >> field in the file (Urban).  One of the things I noticed was that
the
>> >> output
>> >> from the netcdf file that I generated is transposed from the
original.
>> >> The
>> >> matrix was originally 265 X 442.  However, in the file that was
>> created it
>> >> is 442X265.  I'm not sure what's going on here at all to be
honest.  I
>> >> thought it would be as simple as running the command with the
above
>> >> settings.  I may try to create netcdf files with the matrix
being
>> 442X265.
>> >> Something clearly is off here.
>> >>
>> >> --
>> >> Edward Strobach
>> >> EMC/NCEP/NWS/
>> >> IMSG Contractor
>> >> Cubicle#: 2029
>> >> 301-683-3717
>> >>
>> >>
>> >
>> > --
>> > Edward Strobach
>> > EMC/NCEP/NWS/
>> > IMSG Contractor
>> > Cubicle#: 2029
>> > 301-683-3717
>> >
>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>>

------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Fri Aug 21 10:42:14 2020

Hi John,

Hmm, I thought I sent the attachments in the original email.  I don't
have
the colorbar attached in the screenshots, but the values are all
included
(1-24) for both plots.  The first is the original dominant landuse
field,
which looks like what you would expect.  The other is the result after
running gen_vx_mask.  It seems that this matrix got reordered in the
process, which is why I originally surmised it was related to the
matrix
being transposed when running that step.  I had expected only values
of 0's
and 1's, with 1's identifying urban; thus I thought that the map being
generated would be predominately 0's with isolated spots of 1's in the
major urban areas.  Hopefully the attachments are viewable

On Fri, Aug 21, 2020 at 12:08 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Ed,
>
> Thanks for sending your sample commands and a description of the
behavior.
> I see that you've prepared a NetCDF file that follows the CF-
convention,
> but are getting some odd output from gen_vx_mask. I suspect that the
odd
> output from gen_vx_mask is a red herring, and that the real issue is
in
> reading the data from that input NetCDF file.
>
> Whenever, getting started with a new dataset in MET, such as the
> CF-compliant NetCDF file, it's always a good idea to plot the data
using
> plot_data_plane. I suspect we'll find that plot_data_plane creates
the same
> sort of odd plot when reading data directly from that input file.
And
> that's most likely due to either an issue in how the data is stored
in that
> NetCDF file or perhaps some bug in the MET software.
>
> I'll go log on to WCOSS, retrieve that file, and do some testing
with it.
>
> Thanks,
> John
>
> On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> >
> > Originally, I had processed grib files with vegetation type.  I
used this
> > record here: 880:Vegetation Type:Integer(0-13)
> > (instant):lambert:surface:level 0:fcst time 3 hrs:from
202008181800
> >
> > You can see the shape of that is this: >>>
> > pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> > *(288, 468)*
> >
> > You can also see that upon processing this file through
gen_vx_mask in
> the
> > same way as done above produces the same sized matrix - not
transposed.
> >
> >  Dataset('land_mask_15.nc').variables
> > OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> > float32 lat(lat, lon)
> >     long_name: latitude
> >     units: degrees_north
> >     standard_name: latitude
> > unlimited dimensions:
> > current shape = (288, 468)
> > filling on, default _FillValue of 9.969209968386869e+36 used
> > ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> > float32 lon(lat, lon)
> >     long_name: longitude
> >     units: degrees_east
> >     standard_name: longitude
> > unlimited dimensions:
> > current shape = (288, 468)
> > filling on, default _FillValue of 9.969209968386869e+36 used
> > ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> > float32 data_mask(lat, lon)
> >     long_name: data_mask masking region
> >     _FillValue: -9999.0
> >     mask_type: data==15
> > unlimited dimensions:
> > *current shape = (288, 468)*
> > filling on)])
> >
> > Moreover, given the mask_type, it identified points having 15 as 1
and
> > others as 0.  Therefore I have a 0/1 dataset.
> >
> > This is different than when I process the newly created netcdf
file.  The
> > netcdf file that was created has the Urban field with a shape of
> (265,442)
> >
> > When processing it through gen_vx_mask, it gets transposed.  >>>
> > Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> > *(442, 265)*.
> >
> > So in the original case I have no change to the dimensionality of
the
> > dataset, but in the latter case I do.  This happens during the
> gen_vx_mask
> > step, and I suspect that this may somehow be related to my issue.
I
> don't
> > know if there's anything else I can add that will help, but if I
think of
> > something I'll forward it along
> >
> > On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > Transposing it didn't seem to do anything either.  Here's the
> gen_vx_mask
> > > output.  It looks okay at first glance:
> > >
> > > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > aqm.t06z.metcro2d.00.nc
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > aqm.t06z.metcro2d.00.nc
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> > Urban
> > > DEBUG 1: Input Grid:
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > aqm.t06z.metcro2d.00.nc
> > > DEBUG 1: Mask File:
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > aqm.t06z.metcro2d.00.nc
> > > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> > > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> > > DEBUG 1: Output File:
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > >
> > > I'm also attaching the header content of the file:
> > > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > > <class 'netCDF4._netCDF4.Dataset'>
> > > root group (NETCDF4 data model, file format HDF5):
> > >     FileOrigins: NA
> > >     MET_version: V9.1
> > >     MET_tool: gen_vx_mask
> > >     Projection: Lambert Conformal
> > >     hemisphere: N
> > >     accum: 00
> > >     scale_lat_1: 33.000000
> > >     scale_lat_2: 45.000000
> > >     lat_pin: 21.821000
> > >     lon_pin: -120.628000
> > >     x_pin: 0.000000
> > >     y_pin: 0.000000
> > >     lon_orient: -97.000000
> > >     d_km: 12.000000
> > >     r_km: 6371.200000
> > >     nx: 265
> > >     ny: 442 grid_points
> > >     dimensions(sizes): lon(265), lat(442)
> > >     variables(dimensions): float32 LAT(lon,lat), float32
LON(lon,lat),
> > > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
PBL(lon,lat),
> > float32
> > > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
WSTAR(lon,lat),
> float32
> > > USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat),
float32
> > > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
float32
> > > Q2(lon,lat)
> > >     groups:
> > >
> > > Nothing here seems troubling. You can see that
DLUSE(lon,lat).shape ==
> > > (265,442)
> > >
> > > The values in the matrix are of type float, not integer; but, I
don't
> > > think that would be an issue.
> > > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > > masked_array(
> > >   data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > >         ...,
> > >         [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > >         [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > >         [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > >   mask=[[False, False, False, ..., False, False, False],
> > >         [False, False, False, ..., False, False, False],
> > >         [False, False, False, ..., False, False, False],
> > >         ...,
> > >         [False, False, False, ..., False, False, False],
> > >         [False, False, False, ..., False, False, False],
> > >         [False, False, False, ..., False, False, False]],
> > >   fill_value=9.96921e+36,
> > >   dtype=float32)
> > >
> > > Do I need to change the dtype to integer?
> > >
> > > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> Greetings,
> > >>
> > >> This message has been automatically generated in response to
the
> > creation
> > >> of a trouble ticket regarding:
> > >>         "mask issues",
> > >> a summary of which appears below.
> > >>
> > >> There is no need to reply to this message right now.  Your
ticket has
> > >> been assigned an ID of [rt.rap.ucar.edu #96397].
> > >>
> > >> Please include the string:
> > >>
> > >>          [rt.rap.ucar.edu #96397]
> > >>
> > >> in the subject line of all future correspondence about this
issue. To
> do
> > >> so, you may reply to this message.
> > >>
> > >> For more information, please see:
> > >>
> > >> MET Online Tutorial:
> > >>
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > >>
> > >> MET Users Guide:
> > >>    https://www.dtcenter.org/met/users/docs/overview.php
> > >>
> > >> MET FAQs:
> > >>    https://www.dtcenter.org/met/users/support/faqs/index.php
> > >>
> > >> MET-Help Email Archive:
> > >>    http://mailman.ucar.edu/pipermail/met_help
> > >>
> > >>                         Thank you,
> > >>                         met_help at ucar.edu
> > >>
> > >>
> >
-------------------------------------------------------------------------
> > >> GOod evening,
> > >>
> > >> I thought I understood how to process the data into masked
regions,
> but
> > >> apparently not.  I'm using data that I processed into a netcdf
file
> that
> > >> is
> > >> now CF compatible.  I'm processing the DLUSE field from this
> directory:
> > >>
> > >>
> > >>
> >
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> aqm.t06z.metcro2d.00.nc
> > >> <http://aqm.t06z.metcro2d.00.nc>*
> > >>
> > >> I then run this line, which is inside a loop within a script
that I'm
> > >> using.  The command looks like this:
> > >> *gen_vx_mask
> > >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> aqm.t06z.metcro2d.00.nc
> > >> <http://aqm.t06z.metcro2d.00.nc>
> > >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> aqm.t06z.metcro2d.00.nc
> > >> <http://aqm.t06z.metcro2d.00.nc>
> > >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> > >> Urban*
> > >>
> > >> I determined that something was wrong when much of my masked
land
> plots
> > -
> > >> all 24 of them - looked more or less the same.  When inspecting
> Urban, I
> > >> found that there were 588 matched pairs!  Since I realize that
we
> don't
> > >> live on Coruscant, I decided to plot the data and this is what
it
> looked
> > >> like (see attached: fig 1).  The original DLUSE field looks
like what
> > you
> > >> see in the second attachment.  I'm pretty sure that the
gen_vx_mask
> was
> > >> utilized correctly here.  I have the file that I'm using, the
file
> that
> > >> I'm
> > >> creating to store the result, the type (data), the mask field
(DLUSE
> at
> > >> the
> > >> surface), the threshold (set equal to 1 for urban), and the
name of
> the
> > >> field in the file (Urban).  One of the things I noticed was
that the
> > >> output
> > >> from the netcdf file that I generated is transposed from the
original.
> > >> The
> > >> matrix was originally 265 X 442.  However, in the file that was
> created
> > it
> > >> is 442X265.  I'm not sure what's going on here at all to be
honest.  I
> > >> thought it would be as simple as running the command with the
above
> > >> settings.  I may try to create netcdf files with the matrix
being
> > 442X265.
> > >> Something clearly is off here.
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > >>
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Fri Aug 21 10:51:50 2020

Hi John,

Your plot looks just like mine.  Thanks for confirming.  I do believe
I
tried that, but I'll go ahead and try that suggestion again.  I'll
attach
the result as well.  When comparing with VGTYP - an example that
worked - I
noticed the "lat, lon" ordering as opposed to "lon, lat", which is
what
prompted me to go that route.  I'll test this carefully.  Thanks
again.

On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Ed,
>
> I pulled that file and took a look. I've attached the image that
> plot_data_plane generates with the following command:
>
> *plot_data_plane aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>
> aqm.t06z.metcro2d_DLUSE.00.ps <http://aqm.t06z.metcro2d_DLUSE.00.ps>
> 'name="DLUSE"; level="(*,*)";'*
>
>
>
> So there's clearly a problem here! Looking at that file with ncdump,
I see
> that this file is actually setup to follow MET's internal NetCDF
> conventions rather the the NetCDF Climate-Forecast (CF) convention.
>
>
> *ncdump -h aqm.t06z.metcro2d.00.nc <http://aqm.t06z.metcro2d.00.nc>*
>
>
>
> Even though we recommend the use of the CF-convention, the MET tools
> actually don't write to that convention themselves. But we should
probably
> transition to it! For now, the tools write to a pretty rigid
internal
> format where the dimensions are always ordered (lat, lon). Your data
is
> ordered (lon, lat) and I strongly suspect that's the reason for this
> behavior. Please try rearranging to write it as (lat, lon) and then
run
> plot_data_plane on the result to see if it's fixed.
>
>
> Hope that helps.
>
>
> Thanks,
> John
>
>
>
> On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Ed,
> >
> > Thanks for sending your sample commands and a description of the
> behavior.
> > I see that you've prepared a NetCDF file that follows the CF-
convention,
> > but are getting some odd output from gen_vx_mask. I suspect that
the odd
> > output from gen_vx_mask is a red herring, and that the real issue
is in
> > reading the data from that input NetCDF file.
> >
> > Whenever, getting started with a new dataset in MET, such as the
> > CF-compliant NetCDF file, it's always a good idea to plot the data
using
> > plot_data_plane. I suspect we'll find that plot_data_plane creates
the
> same
> > sort of odd plot when reading data directly from that input file.
And
> > that's most likely due to either an issue in how the data is
stored in
> that
> > NetCDF file or perhaps some bug in the MET software.
> >
> > I'll go log on to WCOSS, retrieve that file, and do some testing
with it.
> >
> > Thanks,
> > John
> >
> > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> >>
> >> Originally, I had processed grib files with vegetation type.  I
used
> this
> >> record here: 880:Vegetation Type:Integer(0-13)
> >> (instant):lambert:surface:level 0:fcst time 3 hrs:from
202008181800
> >>
> >> You can see the shape of that is this: >>>
> >> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> >> *(288, 468)*
> >>
> >> You can also see that upon processing this file through
gen_vx_mask in
> the
> >> same way as done above produces the same sized matrix - not
transposed.
> >>
> >>  Dataset('land_mask_15.nc').variables
> >> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> >> float32 lat(lat, lon)
> >>     long_name: latitude
> >>     units: degrees_north
> >>     standard_name: latitude
> >> unlimited dimensions:
> >> current shape = (288, 468)
> >> filling on, default _FillValue of 9.969209968386869e+36 used
> >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> >> float32 lon(lat, lon)
> >>     long_name: longitude
> >>     units: degrees_east
> >>     standard_name: longitude
> >> unlimited dimensions:
> >> current shape = (288, 468)
> >> filling on, default _FillValue of 9.969209968386869e+36 used
> >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> >> float32 data_mask(lat, lon)
> >>     long_name: data_mask masking region
> >>     _FillValue: -9999.0
> >>     mask_type: data==15
> >> unlimited dimensions:
> >> *current shape = (288, 468)*
> >> filling on)])
> >>
> >> Moreover, given the mask_type, it identified points having 15 as
1 and
> >> others as 0.  Therefore I have a 0/1 dataset.
> >>
> >> This is different than when I process the newly created netcdf
file.
> The
> >> netcdf file that was created has the Urban field with a shape of
> (265,442)
> >>
> >> When processing it through gen_vx_mask, it gets transposed.  >>>
> >> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> >> *(442, 265)*.
> >>
> >> So in the original case I have no change to the dimensionality of
the
> >> dataset, but in the latter case I do.  This happens during the
> gen_vx_mask
> >> step, and I suspect that this may somehow be related to my issue.
I
> don't
> >> know if there's anything else I can add that will help, but if I
think
> of
> >> something I'll forward it along
> >>
> >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA Affiliate
<
> >> edward.strobach at noaa.gov> wrote:
> >>
> >> > Transposing it didn't seem to do anything either.  Here's the
> >> gen_vx_mask
> >> > output.  It looks okay at first glance:
> >> >
> >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > aqm.t06z.metcro2d.00.nc
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > aqm.t06z.metcro2d.00.nc
> >> >
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> >> Urban
> >> > DEBUG 1: Input Grid:
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > aqm.t06z.metcro2d.00.nc
> >> > DEBUG 1: Mask File:
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> > aqm.t06z.metcro2d.00.nc
> >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> >> > DEBUG 1: Output File:
> >> >
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> >
> >> > I'm also attaching the header content of the file:
> >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> >> > <class 'netCDF4._netCDF4.Dataset'>
> >> > root group (NETCDF4 data model, file format HDF5):
> >> >     FileOrigins: NA
> >> >     MET_version: V9.1
> >> >     MET_tool: gen_vx_mask
> >> >     Projection: Lambert Conformal
> >> >     hemisphere: N
> >> >     accum: 00
> >> >     scale_lat_1: 33.000000
> >> >     scale_lat_2: 45.000000
> >> >     lat_pin: 21.821000
> >> >     lon_pin: -120.628000
> >> >     x_pin: 0.000000
> >> >     y_pin: 0.000000
> >> >     lon_orient: -97.000000
> >> >     d_km: 12.000000
> >> >     r_km: 6371.200000
> >> >     nx: 265
> >> >     ny: 442 grid_points
> >> >     dimensions(sizes): lon(265), lat(442)
> >> >     variables(dimensions): float32 LAT(lon,lat), float32
LON(lon,lat),
> >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
PBL(lon,lat),
> >> float32
> >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
WSTAR(lon,lat),
> >> float32
> >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat),
float32
> >> > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
float32
> >> > Q2(lon,lat)
> >> >     groups:
> >> >
> >> > Nothing here seems troubling. You can see that
DLUSE(lon,lat).shape ==
> >> > (265,442)
> >> >
> >> > The values in the matrix are of type float, not integer; but, I
don't
> >> > think that would be an issue.
> >> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> >> > masked_array(
> >> >   data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> >> >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> >> >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> >> >         ...,
> >> >         [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> >> >         [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> >> >         [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> >> >   mask=[[False, False, False, ..., False, False, False],
> >> >         [False, False, False, ..., False, False, False],
> >> >         [False, False, False, ..., False, False, False],
> >> >         ...,
> >> >         [False, False, False, ..., False, False, False],
> >> >         [False, False, False, ..., False, False, False],
> >> >         [False, False, False, ..., False, False, False]],
> >> >   fill_value=9.96921e+36,
> >> >   dtype=float32)
> >> >
> >> > Do I need to change the dtype to integer?
> >> >
> >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> >>
> >> >> Greetings,
> >> >>
> >> >> This message has been automatically generated in response to
the
> >> creation
> >> >> of a trouble ticket regarding:
> >> >>         "mask issues",
> >> >> a summary of which appears below.
> >> >>
> >> >> There is no need to reply to this message right now.  Your
ticket has
> >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> >> >>
> >> >> Please include the string:
> >> >>
> >> >>          [rt.rap.ucar.edu #96397]
> >> >>
> >> >> in the subject line of all future correspondence about this
issue. To
> >> do
> >> >> so, you may reply to this message.
> >> >>
> >> >> For more information, please see:
> >> >>
> >> >> MET Online Tutorial:
> >> >>
> >>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> >> >>
> >> >> MET Users Guide:
> >> >>    https://www.dtcenter.org/met/users/docs/overview.php
> >> >>
> >> >> MET FAQs:
> >> >>    https://www.dtcenter.org/met/users/support/faqs/index.php
> >> >>
> >> >> MET-Help Email Archive:
> >> >>    http://mailman.ucar.edu/pipermail/met_help
> >> >>
> >> >>                         Thank you,
> >> >>                         met_help at ucar.edu
> >> >>
> >> >>
> >>
>
-------------------------------------------------------------------------
> >> >> GOod evening,
> >> >>
> >> >> I thought I understood how to process the data into masked
regions,
> but
> >> >> apparently not.  I'm using data that I processed into a netcdf
file
> >> that
> >> >> is
> >> >> now CF compatible.  I'm processing the DLUSE field from this
> directory:
> >> >>
> >> >>
> >> >>
> >>
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> >> aqm.t06z.metcro2d.00.nc
> >> >> <http://aqm.t06z.metcro2d.00.nc>*
> >> >>
> >> >> I then run this line, which is inside a loop within a script
that I'm
> >> >> using.  The command looks like this:
> >> >> *gen_vx_mask
> >> >>
> >> >>
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> >> aqm.t06z.metcro2d.00.nc
> >> >> <http://aqm.t06z.metcro2d.00.nc>
> >> >>
> >> >>
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> >> >> aqm.t06z.metcro2d.00.nc
> >> >> <http://aqm.t06z.metcro2d.00.nc>
> >> >>
> >> >>
> >>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> >> >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh eq1
-name
> >> >> Urban*
> >> >>
> >> >> I determined that something was wrong when much of my masked
land
> >> plots -
> >> >> all 24 of them - looked more or less the same.  When
inspecting
> Urban,
> >> I
> >> >> found that there were 588 matched pairs!  Since I realize that
we
> don't
> >> >> live on Coruscant, I decided to plot the data and this is what
it
> >> looked
> >> >> like (see attached: fig 1).  The original DLUSE field looks
like what
> >> you
> >> >> see in the second attachment.  I'm pretty sure that the
gen_vx_mask
> was
> >> >> utilized correctly here.  I have the file that I'm using, the
file
> that
> >> >> I'm
> >> >> creating to store the result, the type (data), the mask field
(DLUSE
> at
> >> >> the
> >> >> surface), the threshold (set equal to 1 for urban), and the
name of
> the
> >> >> field in the file (Urban).  One of the things I noticed was
that the
> >> >> output
> >> >> from the netcdf file that I generated is transposed from the
> original.
> >> >> The
> >> >> matrix was originally 265 X 442.  However, in the file that
was
> >> created it
> >> >> is 442X265.  I'm not sure what's going on here at all to be
honest.
> I
> >> >> thought it would be as simple as running the command with the
above
> >> >> settings.  I may try to create netcdf files with the matrix
being
> >> 442X265.
> >> >> Something clearly is off here.
> >> >>
> >> >> --
> >> >> Edward Strobach
> >> >> EMC/NCEP/NWS/
> >> >> IMSG Contractor
> >> >> Cubicle#: 2029
> >> >> 301-683-3717
> >> >>
> >> >>
> >> >
> >> > --
> >> > Edward Strobach
> >> > EMC/NCEP/NWS/
> >> > IMSG Contractor
> >> > Cubicle#: 2029
> >> > 301-683-3717
> >> >
> >>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >>
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: mask issues
From: John Halley Gotway
Time: Fri Aug 21 14:46:09 2020

Ed,

Sounds good. Please let me know if reversing (lon,lat) to (lat,lon) in
the
NetCDF files solves the problem.

Have a good weekend.

Thanks,
John

On Fri, Aug 21, 2020 at 10:51 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
>
> Hi John,
>
> Your plot looks just like mine.  Thanks for confirming.  I do
believe I
> tried that, but I'll go ahead and try that suggestion again.  I'll
attach
> the result as well.  When comparing with VGTYP - an example that
worked - I
> noticed the "lat, lon" ordering as opposed to "lon, lat", which is
what
> prompted me to go that route.  I'll test this carefully.  Thanks
again.
>
> On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Ed,
> >
> > I pulled that file and took a look. I've attached the image that
> > plot_data_plane generates with the following command:
> >
> > *plot_data_plane aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc
> >
> > aqm.t06z.metcro2d_DLUSE.00.ps
<http://aqm.t06z.metcro2d_DLUSE.00.ps>
> > 'name="DLUSE"; level="(*,*)";'*
> >
> >
> >
> > So there's clearly a problem here! Looking at that file with
ncdump, I
> see
> > that this file is actually setup to follow MET's internal NetCDF
> > conventions rather the the NetCDF Climate-Forecast (CF)
convention.
> >
> >
> > *ncdump -h aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>*
> >
> >
> >
> > Even though we recommend the use of the CF-convention, the MET
tools
> > actually don't write to that convention themselves. But we should
> probably
> > transition to it! For now, the tools write to a pretty rigid
internal
> > format where the dimensions are always ordered (lat, lon). Your
data is
> > ordered (lon, lat) and I strongly suspect that's the reason for
this
> > behavior. Please try rearranging to write it as (lat, lon) and
then run
> > plot_data_plane on the result to see if it's fixed.
> >
> >
> > Hope that helps.
> >
> >
> > Thanks,
> > John
> >
> >
> >
> > On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Ed,
> > >
> > > Thanks for sending your sample commands and a description of the
> > behavior.
> > > I see that you've prepared a NetCDF file that follows the
> CF-convention,
> > > but are getting some odd output from gen_vx_mask. I suspect that
the
> odd
> > > output from gen_vx_mask is a red herring, and that the real
issue is in
> > > reading the data from that input NetCDF file.
> > >
> > > Whenever, getting started with a new dataset in MET, such as the
> > > CF-compliant NetCDF file, it's always a good idea to plot the
data
> using
> > > plot_data_plane. I suspect we'll find that plot_data_plane
creates the
> > same
> > > sort of odd plot when reading data directly from that input
file. And
> > > that's most likely due to either an issue in how the data is
stored in
> > that
> > > NetCDF file or perhaps some bug in the MET software.
> > >
> > > I'll go log on to WCOSS, retrieve that file, and do some testing
with
> it.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> > >>
> > >> Originally, I had processed grib files with vegetation type.  I
used
> > this
> > >> record here: 880:Vegetation Type:Integer(0-13)
> > >> (instant):lambert:surface:level 0:fcst time 3 hrs:from
202008181800
> > >>
> > >> You can see the shape of that is this: >>>
> > >> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> > >> *(288, 468)*
> > >>
> > >> You can also see that upon processing this file through
gen_vx_mask in
> > the
> > >> same way as done above produces the same sized matrix - not
> transposed.
> > >>
> > >>  Dataset('land_mask_15.nc').variables
> > >> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> > >> float32 lat(lat, lon)
> > >>     long_name: latitude
> > >>     units: degrees_north
> > >>     standard_name: latitude
> > >> unlimited dimensions:
> > >> current shape = (288, 468)
> > >> filling on, default _FillValue of 9.969209968386869e+36 used
> > >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> > >> float32 lon(lat, lon)
> > >>     long_name: longitude
> > >>     units: degrees_east
> > >>     standard_name: longitude
> > >> unlimited dimensions:
> > >> current shape = (288, 468)
> > >> filling on, default _FillValue of 9.969209968386869e+36 used
> > >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> > >> float32 data_mask(lat, lon)
> > >>     long_name: data_mask masking region
> > >>     _FillValue: -9999.0
> > >>     mask_type: data==15
> > >> unlimited dimensions:
> > >> *current shape = (288, 468)*
> > >> filling on)])
> > >>
> > >> Moreover, given the mask_type, it identified points having 15
as 1 and
> > >> others as 0.  Therefore I have a 0/1 dataset.
> > >>
> > >> This is different than when I process the newly created netcdf
file.
> > The
> > >> netcdf file that was created has the Urban field with a shape
of
> > (265,442)
> > >>
> > >> When processing it through gen_vx_mask, it gets transposed.
>>>
> > >> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> > >> *(442, 265)*.
> > >>
> > >> So in the original case I have no change to the dimensionality
of the
> > >> dataset, but in the latter case I do.  This happens during the
> > gen_vx_mask
> > >> step, and I suspect that this may somehow be related to my
issue.  I
> > don't
> > >> know if there's anything else I can add that will help, but if
I think
> > of
> > >> something I'll forward it along
> > >>
> > >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA
Affiliate <
> > >> edward.strobach at noaa.gov> wrote:
> > >>
> > >> > Transposing it didn't seem to do anything either.  Here's the
> > >> gen_vx_mask
> > >> > output.  It looks okay at first glance:
> > >> >
> > >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> > aqm.t06z.metcro2d.00.nc
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> > aqm.t06z.metcro2d.00.nc
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > >> > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh
eq1 -name
> > >> Urban
> > >> > DEBUG 1: Input Grid:
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> > aqm.t06z.metcro2d.00.nc
> > >> > DEBUG 1: Mask File:
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> > aqm.t06z.metcro2d.00.nc
> > >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> > >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> > >> > DEBUG 1: Output File:
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > >> >
> > >> > I'm also attaching the header content of the file:
> > >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > >> > <class 'netCDF4._netCDF4.Dataset'>
> > >> > root group (NETCDF4 data model, file format HDF5):
> > >> >     FileOrigins: NA
> > >> >     MET_version: V9.1
> > >> >     MET_tool: gen_vx_mask
> > >> >     Projection: Lambert Conformal
> > >> >     hemisphere: N
> > >> >     accum: 00
> > >> >     scale_lat_1: 33.000000
> > >> >     scale_lat_2: 45.000000
> > >> >     lat_pin: 21.821000
> > >> >     lon_pin: -120.628000
> > >> >     x_pin: 0.000000
> > >> >     y_pin: 0.000000
> > >> >     lon_orient: -97.000000
> > >> >     d_km: 12.000000
> > >> >     r_km: 6371.200000
> > >> >     nx: 265
> > >> >     ny: 442 grid_points
> > >> >     dimensions(sizes): lon(265), lat(442)
> > >> >     variables(dimensions): float32 LAT(lon,lat), float32
> LON(lon,lat),
> > >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
PBL(lon,lat),
> > >> float32
> > >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
WSTAR(lon,lat),
> > >> float32
> > >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32 HFX(lon,lat),
float32
> > >> > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
float32
> > >> > Q2(lon,lat)
> > >> >     groups:
> > >> >
> > >> > Nothing here seems troubling. You can see that
DLUSE(lon,lat).shape
> ==
> > >> > (265,442)
> > >> >
> > >> > The values in the matrix are of type float, not integer; but,
I
> don't
> > >> > think that would be an issue.
> > >> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > >> > masked_array(
> > >> >   data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > >> >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > >> >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > >> >         ...,
> > >> >         [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > >> >         [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > >> >         [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > >> >   mask=[[False, False, False, ..., False, False, False],
> > >> >         [False, False, False, ..., False, False, False],
> > >> >         [False, False, False, ..., False, False, False],
> > >> >         ...,
> > >> >         [False, False, False, ..., False, False, False],
> > >> >         [False, False, False, ..., False, False, False],
> > >> >         [False, False, False, ..., False, False, False]],
> > >> >   fill_value=9.96921e+36,
> > >> >   dtype=float32)
> > >> >
> > >> > Do I need to change the dtype to integer?
> > >> >
> > >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> >>
> > >> >> Greetings,
> > >> >>
> > >> >> This message has been automatically generated in response to
the
> > >> creation
> > >> >> of a trouble ticket regarding:
> > >> >>         "mask issues",
> > >> >> a summary of which appears below.
> > >> >>
> > >> >> There is no need to reply to this message right now.  Your
ticket
> has
> > >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> > >> >>
> > >> >> Please include the string:
> > >> >>
> > >> >>          [rt.rap.ucar.edu #96397]
> > >> >>
> > >> >> in the subject line of all future correspondence about this
issue.
> To
> > >> do
> > >> >> so, you may reply to this message.
> > >> >>
> > >> >> For more information, please see:
> > >> >>
> > >> >> MET Online Tutorial:
> > >> >>
> > >>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > >> >>
> > >> >> MET Users Guide:
> > >> >>    https://www.dtcenter.org/met/users/docs/overview.php
> > >> >>
> > >> >> MET FAQs:
> > >> >>    https://www.dtcenter.org/met/users/support/faqs/index.php
> > >> >>
> > >> >> MET-Help Email Archive:
> > >> >>    http://mailman.ucar.edu/pipermail/met_help
> > >> >>
> > >> >>                         Thank you,
> > >> >>                         met_help at ucar.edu
> > >> >>
> > >> >>
> > >>
> >
-------------------------------------------------------------------------
> > >> >> GOod evening,
> > >> >>
> > >> >> I thought I understood how to process the data into masked
regions,
> > but
> > >> >> apparently not.  I'm using data that I processed into a
netcdf file
> > >> that
> > >> >> is
> > >> >> now CF compatible.  I'm processing the DLUSE field from this
> > directory:
> > >> >>
> > >> >>
> > >> >>
> > >>
> >
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> >> aqm.t06z.metcro2d.00.nc
> > >> >> <http://aqm.t06z.metcro2d.00.nc>*
> > >> >>
> > >> >> I then run this line, which is inside a loop within a script
that
> I'm
> > >> >> using.  The command looks like this:
> > >> >> *gen_vx_mask
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> >> aqm.t06z.metcro2d.00.nc
> > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > >> >> aqm.t06z.metcro2d.00.nc
> > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > >> >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh
eq1
> -name
> > >> >> Urban*
> > >> >>
> > >> >> I determined that something was wrong when much of my masked
land
> > >> plots -
> > >> >> all 24 of them - looked more or less the same.  When
inspecting
> > Urban,
> > >> I
> > >> >> found that there were 588 matched pairs!  Since I realize
that we
> > don't
> > >> >> live on Coruscant, I decided to plot the data and this is
what it
> > >> looked
> > >> >> like (see attached: fig 1).  The original DLUSE field looks
like
> what
> > >> you
> > >> >> see in the second attachment.  I'm pretty sure that the
gen_vx_mask
> > was
> > >> >> utilized correctly here.  I have the file that I'm using,
the file
> > that
> > >> >> I'm
> > >> >> creating to store the result, the type (data), the mask
field
> (DLUSE
> > at
> > >> >> the
> > >> >> surface), the threshold (set equal to 1 for urban), and the
name of
> > the
> > >> >> field in the file (Urban).  One of the things I noticed was
that
> the
> > >> >> output
> > >> >> from the netcdf file that I generated is transposed from the
> > original.
> > >> >> The
> > >> >> matrix was originally 265 X 442.  However, in the file that
was
> > >> created it
> > >> >> is 442X265.  I'm not sure what's going on here at all to be
honest.
> > I
> > >> >> thought it would be as simple as running the command with
the above
> > >> >> settings.  I may try to create netcdf files with the matrix
being
> > >> 442X265.
> > >> >> Something clearly is off here.
> > >> >>
> > >> >> --
> > >> >> Edward Strobach
> > >> >> EMC/NCEP/NWS/
> > >> >> IMSG Contractor
> > >> >> Cubicle#: 2029
> > >> >> 301-683-3717
> > >> >>
> > >> >>
> > >> >
> > >> > --
> > >> > Edward Strobach
> > >> > EMC/NCEP/NWS/
> > >> > IMSG Contractor
> > >> > Cubicle#: 2029
> > >> > 301-683-3717
> > >> >
> > >>
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > >>
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>

------------------------------------------------
Subject: mask issues
From: Edward Strobach - NOAA Affiliate
Time: Sat Aug 22 09:42:10 2020

Hi John,

Rewriting the files to follow MET's convention helped produce a normal
looking plot (see attached).  That was made possible by writing my
files in
this way:
Dataset('land_mask_Urban.nc')
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):
    FileOrigins: File
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc
generated 20200822_152206 UTC on host m72a1 by the MET gen_vx_mask
tool
    MET_version: V9.1
    MET_tool: gen_vx_mask
    Projection: Lambert Conformal
    hemisphere: N
    scale_lat_1: 33.000000
    scale_lat_2: 45.000000
    lat_pin: 21.821000
    lon_pin: -120.628000
    x_pin: 0.000000
    y_pin: 0.000000
    lon_orient: -97.000000
    d_km: 12.000000
    r_km: 6371.200000



*    nx: 442    ny: 265 grid_points    dimensions(sizes): lat(265),
lon(442)    variables(dimensions): float32 lat(lat,lon), float32
lon(lat,lon), float32 Urban(lat,lon)*
    groups:

Following your earlier suggestion (e.g. gen_vx_mask
wrfprs_ruc13_12.tm00_G212 wrfprs_ruc13_12.tm00_G212 *land_mask.nc
<http://land_mask.nc>*
-type data -mask_field 'name="LAND"; level="Z0";' -thresh eq1), I went
ahead and executed something similar for Urban type:

gen_vx_mask
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
aqm.t06z.metcro2d.00.nc
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200819/
aqm.t06z.metcro2d.00.nc
*/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200819/land_mask_Urban.nc*
-type
data -name Urban -mask_field 'name="DLUSE"; level="Z0";' -thresh 'eq1'

However, as you can see from the map, I retain all values rather than
getting 0 for non-urban and 1 for urban.  I don't see anything that's
really different between these two examples other than I specify the
written name of the mask field as Urban.  I've included - and removing
-
the single quotes about eq1, but the result ended up being the same.



I also don't see anything alarmingly different when comparing the
header
content from this file with the examples that had worked with VGTYP.

I'll continue trying a couple of other things if some ideas come up.
If I
make traction, I'll let you know.  Take care and have a good weekend.



On Fri, Aug 21, 2020 at 4:46 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Ed,
>
> Sounds good. Please let me know if reversing (lon,lat) to (lat,lon)
in the
> NetCDF files solves the problem.
>
> Have a good weekend.
>
> Thanks,
> John
>
> On Fri, Aug 21, 2020 at 10:51 AM Edward Strobach - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397 >
> >
> > Hi John,
> >
> > Your plot looks just like mine.  Thanks for confirming.  I do
believe I
> > tried that, but I'll go ahead and try that suggestion again.  I'll
attach
> > the result as well.  When comparing with VGTYP - an example that
worked
> - I
> > noticed the "lat, lon" ordering as opposed to "lon, lat", which is
what
> > prompted me to go that route.  I'll test this carefully.  Thanks
again.
> >
> > On Fri, Aug 21, 2020 at 12:42 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Ed,
> > >
> > > I pulled that file and took a look. I've attached the image that
> > > plot_data_plane generates with the following command:
> > >
> > > *plot_data_plane aqm.t06z.metcro2d.00.nc <
> http://aqm.t06z.metcro2d.00.nc
> > >
> > > aqm.t06z.metcro2d_DLUSE.00.ps
<http://aqm.t06z.metcro2d_DLUSE.00.ps>
> > > 'name="DLUSE"; level="(*,*)";'*
> > >
> > >
> > >
> > > So there's clearly a problem here! Looking at that file with
ncdump, I
> > see
> > > that this file is actually setup to follow MET's internal NetCDF
> > > conventions rather the the NetCDF Climate-Forecast (CF)
convention.
> > >
> > >
> > > *ncdump -h aqm.t06z.metcro2d.00.nc
<http://aqm.t06z.metcro2d.00.nc>*
> > >
> > >
> > >
> > > Even though we recommend the use of the CF-convention, the MET
tools
> > > actually don't write to that convention themselves. But we
should
> > probably
> > > transition to it! For now, the tools write to a pretty rigid
internal
> > > format where the dimensions are always ordered (lat, lon). Your
data is
> > > ordered (lon, lat) and I strongly suspect that's the reason for
this
> > > behavior. Please try rearranging to write it as (lat, lon) and
then run
> > > plot_data_plane on the result to see if it's fixed.
> > >
> > >
> > > Hope that helps.
> > >
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Fri, Aug 21, 2020 at 10:08 AM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > Ed,
> > > >
> > > > Thanks for sending your sample commands and a description of
the
> > > behavior.
> > > > I see that you've prepared a NetCDF file that follows the
> > CF-convention,
> > > > but are getting some odd output from gen_vx_mask. I suspect
that the
> > odd
> > > > output from gen_vx_mask is a red herring, and that the real
issue is
> in
> > > > reading the data from that input NetCDF file.
> > > >
> > > > Whenever, getting started with a new dataset in MET, such as
the
> > > > CF-compliant NetCDF file, it's always a good idea to plot the
data
> > using
> > > > plot_data_plane. I suspect we'll find that plot_data_plane
creates
> the
> > > same
> > > > sort of odd plot when reading data directly from that input
file. And
> > > > that's most likely due to either an issue in how the data is
stored
> in
> > > that
> > > > NetCDF file or perhaps some bug in the MET software.
> > > >
> > > > I'll go log on to WCOSS, retrieve that file, and do some
testing with
> > it.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Aug 21, 2020 at 8:05 AM Edward Strobach - NOAA
Affiliate via
> > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96397
>
> > > >>
> > > >> Originally, I had processed grib files with vegetation type.
I used
> > > this
> > > >> record here: 880:Vegetation Type:Integer(0-13)
> > > >> (instant):lambert:surface:level 0:fcst time 3 hrs:from
202008181800
> > > >>
> > > >> You can see the shape of that is this: >>>
> > > >> pygrib.open('aqm.t18z.nmm03.tm00').select()[879].values.shape
> > > >> *(288, 468)*
> > > >>
> > > >> You can also see that upon processing this file through
gen_vx_mask
> in
> > > the
> > > >> same way as done above produces the same sized matrix - not
> > transposed.
> > > >>
> > > >>  Dataset('land_mask_15.nc').variables
> > > >> OrderedDict([('lat', <class 'netCDF4._netCDF4.Variable'>
> > > >> float32 lat(lat, lon)
> > > >>     long_name: latitude
> > > >>     units: degrees_north
> > > >>     standard_name: latitude
> > > >> unlimited dimensions:
> > > >> current shape = (288, 468)
> > > >> filling on, default _FillValue of 9.969209968386869e+36 used
> > > >> ), ('lon', <class 'netCDF4._netCDF4.Variable'>
> > > >> float32 lon(lat, lon)
> > > >>     long_name: longitude
> > > >>     units: degrees_east
> > > >>     standard_name: longitude
> > > >> unlimited dimensions:
> > > >> current shape = (288, 468)
> > > >> filling on, default _FillValue of 9.969209968386869e+36 used
> > > >> ), ('data_mask', <class 'netCDF4._netCDF4.Variable'>
> > > >> float32 data_mask(lat, lon)
> > > >>     long_name: data_mask masking region
> > > >>     _FillValue: -9999.0
> > > >>     mask_type: data==15
> > > >> unlimited dimensions:
> > > >> *current shape = (288, 468)*
> > > >> filling on)])
> > > >>
> > > >> Moreover, given the mask_type, it identified points having 15
as 1
> and
> > > >> others as 0.  Therefore I have a 0/1 dataset.
> > > >>
> > > >> This is different than when I process the newly created
netcdf file.
> > > The
> > > >> netcdf file that was created has the Urban field with a shape
of
> > > (265,442)
> > > >>
> > > >> When processing it through gen_vx_mask, it gets transposed.
>>>
> > > >> Dataset('land_mask_Urban.nc').variables['Urban'][:].shape
> > > >> *(442, 265)*.
> > > >>
> > > >> So in the original case I have no change to the
dimensionality of
> the
> > > >> dataset, but in the latter case I do.  This happens during
the
> > > gen_vx_mask
> > > >> step, and I suspect that this may somehow be related to my
issue.  I
> > > don't
> > > >> know if there's anything else I can add that will help, but
if I
> think
> > > of
> > > >> something I'll forward it along
> > > >>
> > > >> On Fri, Aug 21, 2020 at 9:27 AM Edward Strobach - NOAA
Affiliate <
> > > >> edward.strobach at noaa.gov> wrote:
> > > >>
> > > >> > Transposing it didn't seem to do anything either.  Here's
the
> > > >> gen_vx_mask
> > > >> > output.  It looks okay at first glance:
> > > >> >
> > > >> > [Edward.Strobach at m71a1 scripts]$ gen_vx_mask
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> > aqm.t06z.metcro2d.00.nc
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> > aqm.t06z.metcro2d.00.nc
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > >> > -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh
eq1
> -name
> > > >> Urban
> > > >> > DEBUG 1: Input Grid:
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> > aqm.t06z.metcro2d.00.nc
> > > >> > DEBUG 1: Mask File:
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> > aqm.t06z.metcro2d.00.nc
> > > >> > DEBUG 2: Parsed Input Grid: Lambert Conformal (442 x 265)
> > > >> > DEBUG 2: Parsed Mask Grid: Lambert Conformal (442 x 265)
> > > >> > DEBUG 1: Output File:
> > > >> >
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > >> >
> > > >> > I'm also attaching the header content of the file:
> > > >> > >>> Dataset('aqm.t06z.metcro2d.00.nc')
> > > >> > <class 'netCDF4._netCDF4.Dataset'>
> > > >> > root group (NETCDF4 data model, file format HDF5):
> > > >> >     FileOrigins: NA
> > > >> >     MET_version: V9.1
> > > >> >     MET_tool: gen_vx_mask
> > > >> >     Projection: Lambert Conformal
> > > >> >     hemisphere: N
> > > >> >     accum: 00
> > > >> >     scale_lat_1: 33.000000
> > > >> >     scale_lat_2: 45.000000
> > > >> >     lat_pin: 21.821000
> > > >> >     lon_pin: -120.628000
> > > >> >     x_pin: 0.000000
> > > >> >     y_pin: 0.000000
> > > >> >     lon_orient: -97.000000
> > > >> >     d_km: 12.000000
> > > >> >     r_km: 6371.200000
> > > >> >     nx: 265
> > > >> >     ny: 442 grid_points
> > > >> >     dimensions(sizes): lon(265), lat(442)
> > > >> >     variables(dimensions): float32 LAT(lon,lat), float32
> > LON(lon,lat),
> > > >> > float32 DLUSE(lon,lat), float32 TMP(lon,lat), float32
> PBL(lon,lat),
> > > >> float32
> > > >> > WSPD10(lon,lat), float32 WDIR10(lon,lat), float32
WSTAR(lon,lat),
> > > >> float32
> > > >> > USTAR(lon,lat), float32 MOLI(lon,lat), float32
HFX(lon,lat),
> float32
> > > >> > GLW(lon,lat), float32 GSW(lon,lat), float32 RGRND(lon,lat),
> float32
> > > >> > Q2(lon,lat)
> > > >> >     groups:
> > > >> >
> > > >> > Nothing here seems troubling. You can see that
> DLUSE(lon,lat).shape
> > ==
> > > >> > (265,442)
> > > >> >
> > > >> > The values in the matrix are of type float, not integer;
but, I
> > don't
> > > >> > think that would be an issue.
> > > >> > Dataset('aqm.t06z.metcro2d.00.nc').variables['DLUSE'][:]
> > > >> > masked_array(
> > > >> >   data=[[16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > >> >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > >> >         [16.0, 16.0, 16.0, ..., 16.0, 16.0, 16.0],
> > > >> >         ...,
> > > >> >         [16.0, 16.0, 16.0, ..., 12.0, 8.0, 14.0],
> > > >> >         [16.0, 16.0, 16.0, ..., 8.0, 10.0, 10.0],
> > > >> >         [16.0, 16.0, 16.0, ..., 15.0, 8.0, 10.0]],
> > > >> >   mask=[[False, False, False, ..., False, False, False],
> > > >> >         [False, False, False, ..., False, False, False],
> > > >> >         [False, False, False, ..., False, False, False],
> > > >> >         ...,
> > > >> >         [False, False, False, ..., False, False, False],
> > > >> >         [False, False, False, ..., False, False, False],
> > > >> >         [False, False, False, ..., False, False, False]],
> > > >> >   fill_value=9.96921e+36,
> > > >> >   dtype=float32)
> > > >> >
> > > >> > Do I need to change the dtype to integer?
> > > >> >
> > > >> > On Fri, Aug 21, 2020 at 12:30 AM met_help at ucar.edu via RT <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> >>
> > > >> >> Greetings,
> > > >> >>
> > > >> >> This message has been automatically generated in response
to the
> > > >> creation
> > > >> >> of a trouble ticket regarding:
> > > >> >>         "mask issues",
> > > >> >> a summary of which appears below.
> > > >> >>
> > > >> >> There is no need to reply to this message right now.  Your
ticket
> > has
> > > >> >> been assigned an ID of [rt.rap.ucar.edu #96397].
> > > >> >>
> > > >> >> Please include the string:
> > > >> >>
> > > >> >>          [rt.rap.ucar.edu #96397]
> > > >> >>
> > > >> >> in the subject line of all future correspondence about
this
> issue.
> > To
> > > >> do
> > > >> >> so, you may reply to this message.
> > > >> >>
> > > >> >> For more information, please see:
> > > >> >>
> > > >> >> MET Online Tutorial:
> > > >> >>
> > > >>
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > >> >>
> > > >> >> MET Users Guide:
> > > >> >>    https://www.dtcenter.org/met/users/docs/overview.php
> > > >> >>
> > > >> >> MET FAQs:
> > > >> >>
https://www.dtcenter.org/met/users/support/faqs/index.php
> > > >> >>
> > > >> >> MET-Help Email Archive:
> > > >> >>    http://mailman.ucar.edu/pipermail/met_help
> > > >> >>
> > > >> >>                         Thank you,
> > > >> >>                         met_help at ucar.edu
> > > >> >>
> > > >> >>
> > > >>
> > >
>
-------------------------------------------------------------------------
> > > >> >> GOod evening,
> > > >> >>
> > > >> >> I thought I understood how to process the data into masked
> regions,
> > > but
> > > >> >> apparently not.  I'm using data that I processed into a
netcdf
> file
> > > >> that
> > > >> >> is
> > > >> >> now CF compatible.  I'm processing the DLUSE field from
this
> > > directory:
> > > >> >>
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
>
*/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> >> aqm.t06z.metcro2d.00.nc
> > > >> >> <http://aqm.t06z.metcro2d.00.nc>*
> > > >> >>
> > > >> >> I then run this line, which is inside a loop within a
script that
> > I'm
> > > >> >> using.  The command looks like this:
> > > >> >> *gen_vx_mask
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> >> aqm.t06z.metcro2d.00.nc
> > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/prod/aqm.20200818/
> > > >> >> aqm.t06z.metcro2d.00.nc
> > > >> >> <http://aqm.t06z.metcro2d.00.nc>
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/polydata/land_mask/20200818/land_mask_Urban.nc
> > > >> >> -type data -mask_field 'name="DLUSE"; level="Z0";' -thresh
eq1
> > -name
> > > >> >> Urban*
> > > >> >>
> > > >> >> I determined that something was wrong when much of my
masked land
> > > >> plots -
> > > >> >> all 24 of them - looked more or less the same.  When
inspecting
> > > Urban,
> > > >> I
> > > >> >> found that there were 588 matched pairs!  Since I realize
that we
> > > don't
> > > >> >> live on Coruscant, I decided to plot the data and this is
what it
> > > >> looked
> > > >> >> like (see attached: fig 1).  The original DLUSE field
looks like
> > what
> > > >> you
> > > >> >> see in the second attachment.  I'm pretty sure that the
> gen_vx_mask
> > > was
> > > >> >> utilized correctly here.  I have the file that I'm using,
the
> file
> > > that
> > > >> >> I'm
> > > >> >> creating to store the result, the type (data), the mask
field
> > (DLUSE
> > > at
> > > >> >> the
> > > >> >> surface), the threshold (set equal to 1 for urban), and
the name
> of
> > > the
> > > >> >> field in the file (Urban).  One of the things I noticed
was that
> > the
> > > >> >> output
> > > >> >> from the netcdf file that I generated is transposed from
the
> > > original.
> > > >> >> The
> > > >> >> matrix was originally 265 X 442.  However, in the file
that was
> > > >> created it
> > > >> >> is 442X265.  I'm not sure what's going on here at all to
be
> honest.
> > > I
> > > >> >> thought it would be as simple as running the command with
the
> above
> > > >> >> settings.  I may try to create netcdf files with the
matrix being
> > > >> 442X265.
> > > >> >> Something clearly is off here.
> > > >> >>
> > > >> >> --
> > > >> >> Edward Strobach
> > > >> >> EMC/NCEP/NWS/
> > > >> >> IMSG Contractor
> > > >> >> Cubicle#: 2029
> > > >> >> 301-683-3717
> > > >> >>
> > > >> >>
> > > >> >
> > > >> > --
> > > >> > Edward Strobach
> > > >> > EMC/NCEP/NWS/
> > > >> > IMSG Contractor
> > > >> > Cubicle#: 2029
> > > >> > 301-683-3717
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> Edward Strobach
> > > >> EMC/NCEP/NWS/
> > > >> IMSG Contractor
> > > >> Cubicle#: 2029
> > > >> 301-683-3717
> > > >>
> > > >>
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------


More information about the Met_help mailing list