[Met_help] [rt.rap.ucar.edu #73124] History for Grid Stat Config File Question
John Halley Gotway via RT
met_help at ucar.edu
Fri Sep 4 09:56:55 MDT 2015
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I'm trying to figure out the right way to set the config file for grid_stat to do the following:
-- generate a mask for my data based on the contents of an external file, in NetCDF format
-- keying this mask off of a NetCDF variable named "End of day" within the .nc file
-- Keep only those grid points where "End of day" equals 24
I can't seem to find the right combination of equals, semicolons, etc., to do this.
Right now I have:
mask = {
grid = [
name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
field = [
{
name="End of day";
cat_thresh=[==24];
}
];
];
poly = [];
};
The header from the NetCDF file in question:
netcdf eodhours {
dimensions:
latitude = 360 ;
longitude = 720 ;
variables:
float latitude(latitude) ;
latitude:units = "degrees_north" ;
float longitude(longitude) ;
longitude:units = "degrees_east" ;
float End of day(latitude, longitude) ;
End of day:units = "hours" ;
}
The mask syntax I used is incorrect.
For future MET user guides, the following paragraph could really use an example to make this clear:
"Alternatively, any gridded data file that MET can read may be used to define a
verification masking region. Users must specify a description of the field to be used
from the input file and, optionally, may specify a threshold to be applied to that field.
Any grid point where the resulting field is 0, the mask is turned off. Any grid point where
it is non-zero, the mask is turned on."
Can you please help me get the syntax correct?
Thanks,
Matt
// SIGNED //
Matthew C. Sittel
University Corporation for Atmospheric Research
16 WS/WXN, 557 WW, Offutt AFB, NE
matthew.sittel.ctr at us.af.mil
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Tue Aug 25 10:39:25 2015
Matt,
Are you able to send me that end of day NetCDF file? Probably easiest
for
met to get it correct here and then let you know. Also, is this
METv4.1 or
met-5.0?
Thanks,
John
On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> Queue: met_help
> Subject: Grid Stat Config File Question
> Owner: Nobody
> Requestors: matthew.sittel.ctr at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
>
> Hi,
>
> I'm trying to figure out the right way to set the config file for
> grid_stat to do the following:
>
> -- generate a mask for my data based on the contents of an external
file,
> in NetCDF format
> -- keying this mask off of a NetCDF variable named "End of day"
within the
> .nc file
> -- Keep only those grid points where "End of day" equals 24
>
> I can't seem to find the right combination of equals, semicolons,
etc., to
> do this.
>
> Right now I have:
>
> mask = {
> grid = [
> name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> field = [
> {
> name="End of day";
> cat_thresh=[==24];
> }
> ];
> ];
> poly = [];
> };
>
> The header from the NetCDF file in question:
>
> netcdf eodhours {
> dimensions:
> latitude = 360 ;
> longitude = 720 ;
> variables:
> float latitude(latitude) ;
> latitude:units = "degrees_north" ;
> float longitude(longitude) ;
> longitude:units = "degrees_east" ;
> float End of day(latitude, longitude) ;
> End of day:units = "hours" ;
> }
>
> The mask syntax I used is incorrect.
>
> For future MET user guides, the following paragraph could really use
an
> example to make this clear:
>
> "Alternatively, any gridded data file that MET can read may be used
to
> define a
> verification masking region. Users must specify a description of the
field
> to be used
> from the input file and, optionally, may specify a threshold to be
applied
> to that field.
> Any grid point where the resulting field is 0, the mask is turned
off. Any
> grid point where
> it is non-zero, the mask is turned on."
>
> Can you please help me get the syntax correct?
>
> Thanks,
> Matt
>
> // SIGNED //
> Matthew C. Sittel
> University Corporation for Atmospheric Research
> 16 WS/WXN, 557 WW, Offutt AFB, NE
> matthew.sittel.ctr at us.af.mil
>
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Tue Aug 25 10:53:09 2015
Still 4.1. Attached is the NetCDF file.
I thought for sure I had this correct with:
mask = {
grid = [ "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
field = [
{
name = "End of day";
cat_thresh = [ ==24 ];
}
];
poly = [];
};
But I get the following syntax error (the carat points at the last
double quote, immediately after .nc:
ERROR :
ERROR : yyerror() -> syntax error in file
"/h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24"
ERROR :
ERROR : line = 48
ERROR :
ERROR : column = 58
ERROR :
ERROR : text = ";"
ERROR :
ERROR :
ERROR : grid = [ "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
ERROR : _________________________________________________________^__
ERROR :
Thanks John. I've spent most of the morning trying to figure this
out.
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, August 25, 2015 11:39 AM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
Are you able to send me that end of day NetCDF file? Probably easiest
for met to get it correct here and then let you know. Also, is this
METv4.1 or met-5.0?
Thanks,
John
On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> Queue: met_help
> Subject: Grid Stat Config File Question
> Owner: Nobody
> Requestors: matthew.sittel.ctr at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> >
>
>
> Hi,
>
> I'm trying to figure out the right way to set the config file for
> grid_stat to do the following:
>
> -- generate a mask for my data based on the contents of an external
> file, in NetCDF format
> -- keying this mask off of a NetCDF variable named "End of day"
within
> the .nc file
> -- Keep only those grid points where "End of day" equals 24
>
> I can't seem to find the right combination of equals, semicolons,
> etc., to do this.
>
> Right now I have:
>
> mask = {
> grid = [
> name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> field = [
> {
> name="End of day";
> cat_thresh=[==24];
> }
> ];
> ];
> poly = [];
> };
>
> The header from the NetCDF file in question:
>
> netcdf eodhours {
> dimensions:
> latitude = 360 ;
> longitude = 720 ;
> variables:
> float latitude(latitude) ;
> latitude:units = "degrees_north" ;
> float longitude(longitude) ;
> longitude:units = "degrees_east" ;
> float End of day(latitude, longitude) ;
> End of day:units = "hours" ; }
>
> The mask syntax I used is incorrect.
>
> For future MET user guides, the following paragraph could really use
> an example to make this clear:
>
> "Alternatively, any gridded data file that MET can read may be used
to
> define a verification masking region. Users must specify a
description
> of the field to be used from the input file and, optionally, may
> specify a threshold to be applied to that field.
> Any grid point where the resulting field is 0, the mask is turned
off.
> Any grid point where it is non-zero, the mask is turned on."
>
> Can you please help me get the syntax correct?
>
> Thanks,
> Matt
>
> // SIGNED //
> Matthew C. Sittel
> University Corporation for Atmospheric Research
> 16 WS/WXN, 557 WW, Offutt AFB, NE
> matthew.sittel.ctr at us.af.mil
>
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Tue Aug 25 12:05:58 2015
Matt,
Good news and bad news.
The good news is that I'm able to get MET to read this file, but the
bad
news is that it's with the development version of MET, not even a
released
version!
But I did modify your NetCDF file first in three ways:
(1) MET does not support variable names with embedded whitespace.
Those
variable names can ultimately be written to the ASCII output files of
MET
whose columns are whitespace delimited. Having whitespace in the
variable
names would throw off the number of columns. So I renamed "End of
day" to
"EndOfDay":
ncrename -v 'End of day',EndOfDay eodhours.nc eodhours_rename.nc
(2) Looks like you're using -999 for missing data, so I added the
_FillValue attribute.
(3) Also, MET needs to interpret this as a NetCDF file following the
CF
convention, so I added that as a global attribute:
ncatted -a _FillValue,EndOfDay,a,f,-999 -a
Conventions,global,a,c,"CF-1.0" \
eodhours_rename.nc eodhours_mod.nc
You're using METv4.1, but that doesn't include support for NetCDF CF
files. So this file won't work in METv4.1.
So here's what I did...
I ran the modified data file through the development "pcp_combine"
tool.
That creates a flavor of NetCDF that METv4.1 can actually read. I
plotted
it using this command:
METv4.1/bin/plot_data_plane \
eodhours_met.nc eodhours_met.ps \
'name="EndOfDay"; level="(*,*)";'
Both the NetCDF file and the resulting image are attached.
Try the following to use this as a mask in Grid-Stat. This defines 2
different masks, one for eod = 24 and another for eod = 12:
mask = {
grid = [];
poly = [
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc {
name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
{
name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ];
};
I agree, that's awfully convoluted!
Hope that helps.
John
On Tue, Aug 25, 2015 at 10:53 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Still 4.1. Attached is the NetCDF file.
>
> I thought for sure I had this correct with:
>
> mask = {
> grid = [ "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> field = [
> {
> name = "End of day";
> cat_thresh = [ ==24 ];
> }
> ];
> poly = [];
> };
>
> But I get the following syntax error (the carat points at the last
double
> quote, immediately after .nc:
>
> ERROR :
> ERROR : yyerror() -> syntax error in file
> "/h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24"
> ERROR :
> ERROR : line = 48
> ERROR :
> ERROR : column = 58
> ERROR :
> ERROR : text = ";"
> ERROR :
> ERROR :
> ERROR : grid = [ "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> ERROR :
_________________________________________________________^__
> ERROR :
>
> Thanks John. I've spent most of the morning trying to figure this
out.
>
> Matt
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, August 25, 2015 11:39 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Are you able to send me that end of day NetCDF file? Probably
easiest for
> met to get it correct here and then let you know. Also, is this
METv4.1 or
> met-5.0?
>
> Thanks,
> John
>
> On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > Queue: met_help
> > Subject: Grid Stat Config File Question
> > Owner: Nobody
> > Requestors: matthew.sittel.ctr at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > >
> >
> >
> > Hi,
> >
> > I'm trying to figure out the right way to set the config file for
> > grid_stat to do the following:
> >
> > -- generate a mask for my data based on the contents of an
external
> > file, in NetCDF format
> > -- keying this mask off of a NetCDF variable named "End of day"
within
> > the .nc file
> > -- Keep only those grid points where "End of day" equals 24
> >
> > I can't seem to find the right combination of equals, semicolons,
> > etc., to do this.
> >
> > Right now I have:
> >
> > mask = {
> > grid = [
> > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > field = [
> > {
> > name="End of day";
> > cat_thresh=[==24];
> > }
> > ];
> > ];
> > poly = [];
> > };
> >
> > The header from the NetCDF file in question:
> >
> > netcdf eodhours {
> > dimensions:
> > latitude = 360 ;
> > longitude = 720 ;
> > variables:
> > float latitude(latitude) ;
> > latitude:units = "degrees_north" ;
> > float longitude(longitude) ;
> > longitude:units = "degrees_east" ;
> > float End of day(latitude, longitude) ;
> > End of day:units = "hours" ; }
> >
> > The mask syntax I used is incorrect.
> >
> > For future MET user guides, the following paragraph could really
use
> > an example to make this clear:
> >
> > "Alternatively, any gridded data file that MET can read may be
used to
> > define a verification masking region. Users must specify a
description
> > of the field to be used from the input file and, optionally, may
> > specify a threshold to be applied to that field.
> > Any grid point where the resulting field is 0, the mask is turned
off.
> > Any grid point where it is non-zero, the mask is turned on."
> >
> > Can you please help me get the syntax correct?
> >
> > Thanks,
> > Matt
> >
> > // SIGNED //
> > Matthew C. Sittel
> > University Corporation for Atmospheric Research
> > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > matthew.sittel.ctr at us.af.mil
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Fri Aug 28 08:19:40 2015
Hey John,
Did you receive the NetCDF file I sent? I put it as an attachment to
an e-mail. AFWA E-mail has been really messed up here as of late so
it's possible that it never went out, or that if you replied I haven't
received it. I just wanted to check.
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, August 25, 2015 11:39 AM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
Are you able to send me that end of day NetCDF file? Probably easiest
for met to get it correct here and then let you know. Also, is this
METv4.1 or met-5.0?
Thanks,
John
On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> Queue: met_help
> Subject: Grid Stat Config File Question
> Owner: Nobody
> Requestors: matthew.sittel.ctr at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> >
>
>
> Hi,
>
> I'm trying to figure out the right way to set the config file for
> grid_stat to do the following:
>
> -- generate a mask for my data based on the contents of an external
> file, in NetCDF format
> -- keying this mask off of a NetCDF variable named "End of day"
within
> the .nc file
> -- Keep only those grid points where "End of day" equals 24
>
> I can't seem to find the right combination of equals, semicolons,
> etc., to do this.
>
> Right now I have:
>
> mask = {
> grid = [
> name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> field = [
> {
> name="End of day";
> cat_thresh=[==24];
> }
> ];
> ];
> poly = [];
> };
>
> The header from the NetCDF file in question:
>
> netcdf eodhours {
> dimensions:
> latitude = 360 ;
> longitude = 720 ;
> variables:
> float latitude(latitude) ;
> latitude:units = "degrees_north" ;
> float longitude(longitude) ;
> longitude:units = "degrees_east" ;
> float End of day(latitude, longitude) ;
> End of day:units = "hours" ; }
>
> The mask syntax I used is incorrect.
>
> For future MET user guides, the following paragraph could really use
> an example to make this clear:
>
> "Alternatively, any gridded data file that MET can read may be used
to
> define a verification masking region. Users must specify a
description
> of the field to be used from the input file and, optionally, may
> specify a threshold to be applied to that field.
> Any grid point where the resulting field is 0, the mask is turned
off.
> Any grid point where it is non-zero, the mask is turned on."
>
> Can you please help me get the syntax correct?
>
> Thanks,
> Matt
>
> // SIGNED //
> Matthew C. Sittel
> University Corporation for Atmospheric Research
> 16 WS/WXN, 557 WW, Offutt AFB, NE
> matthew.sittel.ctr at us.af.mil
>
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Fri Aug 28 09:33:41 2015
Matt,
I received an email from you on August 25 with a NetCDF file
attachment.
Then I replied to you on August 25th in an email with 2 attachments (
eodhours_met.nc and eodhours_met.ps) on August 25. Did you receive my
reply on 8/25?
Thanks,
John
On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hey John,
>
> Did you receive the NetCDF file I sent? I put it as an attachment
to an
> e-mail. AFWA E-mail has been really messed up here as of late so
it's
> possible that it never went out, or that if you replied I haven't
received
> it. I just wanted to check.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, August 25, 2015 11:39 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Are you able to send me that end of day NetCDF file? Probably
easiest for
> met to get it correct here and then let you know. Also, is this
METv4.1 or
> met-5.0?
>
> Thanks,
> John
>
> On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > Queue: met_help
> > Subject: Grid Stat Config File Question
> > Owner: Nobody
> > Requestors: matthew.sittel.ctr at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > >
> >
> >
> > Hi,
> >
> > I'm trying to figure out the right way to set the config file for
> > grid_stat to do the following:
> >
> > -- generate a mask for my data based on the contents of an
external
> > file, in NetCDF format
> > -- keying this mask off of a NetCDF variable named "End of day"
within
> > the .nc file
> > -- Keep only those grid points where "End of day" equals 24
> >
> > I can't seem to find the right combination of equals, semicolons,
> > etc., to do this.
> >
> > Right now I have:
> >
> > mask = {
> > grid = [
> > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > field = [
> > {
> > name="End of day";
> > cat_thresh=[==24];
> > }
> > ];
> > ];
> > poly = [];
> > };
> >
> > The header from the NetCDF file in question:
> >
> > netcdf eodhours {
> > dimensions:
> > latitude = 360 ;
> > longitude = 720 ;
> > variables:
> > float latitude(latitude) ;
> > latitude:units = "degrees_north" ;
> > float longitude(longitude) ;
> > longitude:units = "degrees_east" ;
> > float End of day(latitude, longitude) ;
> > End of day:units = "hours" ; }
> >
> > The mask syntax I used is incorrect.
> >
> > For future MET user guides, the following paragraph could really
use
> > an example to make this clear:
> >
> > "Alternatively, any gridded data file that MET can read may be
used to
> > define a verification masking region. Users must specify a
description
> > of the field to be used from the input file and, optionally, may
> > specify a threshold to be applied to that field.
> > Any grid point where the resulting field is 0, the mask is turned
off.
> > Any grid point where it is non-zero, the mask is turned on."
> >
> > Can you please help me get the syntax correct?
> >
> > Thanks,
> > Matt
> >
> > // SIGNED //
> > Matthew C. Sittel
> > University Corporation for Atmospheric Research
> > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > matthew.sittel.ctr at us.af.mil
> >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Fri Aug 28 09:37:37 2015
Can you resend please? It must still be out in cyberspace. This
morning I got all of yesterday's mail. Note I was here all day
yesterday... it's always something.
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, August 28, 2015 10:34 AM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
I received an email from you on August 25 with a NetCDF file
attachment.
Then I replied to you on August 25th in an email with 2 attachments (
eodhours_met.nc and eodhours_met.ps) on August 25. Did you receive my
reply on 8/25?
Thanks,
John
On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hey John,
>
> Did you receive the NetCDF file I sent? I put it as an attachment
to
> an e-mail. AFWA E-mail has been really messed up here as of late so
> it's possible that it never went out, or that if you replied I
haven't
> received it. I just wanted to check.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, August 25, 2015 11:39 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Are you able to send me that end of day NetCDF file? Probably
easiest
> for met to get it correct here and then let you know. Also, is this
> METv4.1 or met-5.0?
>
> Thanks,
> John
>
> On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > Queue: met_help
> > Subject: Grid Stat Config File Question
> > Owner: Nobody
> > Requestors: matthew.sittel.ctr at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > >
> >
> >
> > Hi,
> >
> > I'm trying to figure out the right way to set the config file for
> > grid_stat to do the following:
> >
> > -- generate a mask for my data based on the contents of an
external
> > file, in NetCDF format
> > -- keying this mask off of a NetCDF variable named "End of day"
> > within the .nc file
> > -- Keep only those grid points where "End of day" equals 24
> >
> > I can't seem to find the right combination of equals, semicolons,
> > etc., to do this.
> >
> > Right now I have:
> >
> > mask = {
> > grid = [
> > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > field = [
> > {
> > name="End of day";
> > cat_thresh=[==24];
> > }
> > ];
> > ];
> > poly = [];
> > };
> >
> > The header from the NetCDF file in question:
> >
> > netcdf eodhours {
> > dimensions:
> > latitude = 360 ;
> > longitude = 720 ;
> > variables:
> > float latitude(latitude) ;
> > latitude:units = "degrees_north" ;
> > float longitude(longitude) ;
> > longitude:units = "degrees_east" ;
> > float End of day(latitude, longitude) ;
> > End of day:units = "hours" ; }
> >
> > The mask syntax I used is incorrect.
> >
> > For future MET user guides, the following paragraph could really
use
> > an example to make this clear:
> >
> > "Alternatively, any gridded data file that MET can read may be
used
> > to define a verification masking region. Users must specify a
> > description of the field to be used from the input file and,
> > optionally, may specify a threshold to be applied to that field.
> > Any grid point where the resulting field is 0, the mask is turned
off.
> > Any grid point where it is non-zero, the mask is turned on."
> >
> > Can you please help me get the syntax correct?
> >
> > Thanks,
> > Matt
> >
> > // SIGNED //
> > Matthew C. Sittel
> > University Corporation for Atmospheric Research
> > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > matthew.sittel.ctr at us.af.mil
> >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Fri Aug 28 09:48:43 2015
Email from 8/25...
Matt,
Good news and bad news.
The good news is that I'm able to get MET to read this file, but the
bad
news is that it's with the development version of MET, not even a
released
version!
But I did modify your NetCDF file first in three ways:
(1) MET does not support variable names with embedded whitespace.
Those
variable names can ultimately be written to the ASCII output files of
MET
whose columns are whitespace delimited. Having whitespace in the
variable
names would throw off the number of columns. So I renamed "End of
day" to
"EndOfDay":
ncrename -v 'End of day',EndOfDay eodhours.nc eodhours_rename.nc
(2) Looks like you're using -999 for missing data, so I added the
_FillValue attribute.
(3) Also, MET needs to interpret this as a NetCDF file following the
CF
convention, so I added that as a global attribute:
ncatted -a _FillValue,EndOfDay,a,f,-999 -a
Conventions,global,a,c,"CF-1.0" \
eodhours_rename.nc eodhours_mod.nc
You're using METv4.1, but that version doesn't include support for
NetCDF-CF files. So this file won't work in METv4.1.
So here's what I did...
I ran the modified data file through the development "pcp_combine"
tool to
create a flavor of NetCDF that METv4.1 can actually read. I plotted
it
using this command:
METv4.1/bin/plot_data_plane \
eodhours_met.nc eodhours_met.ps \
'name="EndOfDay"; level="(*,*)";'
Both the NetCDF file (eodhours_met.nc) and the resulting image
(eodhour_met.png) are attached.
Try the following to use this as a mask in Grid-Stat. This defines 2
different masks, one for eod = 24 and another for eod = 12:
mask = {
grid = [];
poly = [
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc {
name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
{
name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ];
};
I agree, that's awfully convoluted!
Hope that helps.
John
On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Can you resend please? It must still be out in cyberspace. This
morning
> I got all of yesterday's mail. Note I was here all day yesterday...
it's
> always something.
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, August 28, 2015 10:34 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> I received an email from you on August 25 with a NetCDF file
attachment.
> Then I replied to you on August 25th in an email with 2 attachments
(
> eodhours_met.nc and eodhours_met.ps) on August 25. Did you receive
my
> reply on 8/25?
>
> Thanks,
> John
>
> On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hey John,
> >
> > Did you receive the NetCDF file I sent? I put it as an attachment
to
> > an e-mail. AFWA E-mail has been really messed up here as of late
so
> > it's possible that it never went out, or that if you replied I
haven't
> > received it. I just wanted to check.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Tuesday, August 25, 2015 11:39 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > Are you able to send me that end of day NetCDF file? Probably
easiest
> > for met to get it correct here and then let you know. Also, is
this
> > METv4.1 or met-5.0?
> >
> > Thanks,
> > John
> >
> > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > Queue: met_help
> > > Subject: Grid Stat Config File Question
> > > Owner: Nobody
> > > Requestors: matthew.sittel.ctr at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > >
> > >
> > >
> > > Hi,
> > >
> > > I'm trying to figure out the right way to set the config file
for
> > > grid_stat to do the following:
> > >
> > > -- generate a mask for my data based on the contents of an
external
> > > file, in NetCDF format
> > > -- keying this mask off of a NetCDF variable named "End of day"
> > > within the .nc file
> > > -- Keep only those grid points where "End of day" equals 24
> > >
> > > I can't seem to find the right combination of equals,
semicolons,
> > > etc., to do this.
> > >
> > > Right now I have:
> > >
> > > mask = {
> > > grid = [
> > > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > field = [
> > > {
> > > name="End of day";
> > > cat_thresh=[==24];
> > > }
> > > ];
> > > ];
> > > poly = [];
> > > };
> > >
> > > The header from the NetCDF file in question:
> > >
> > > netcdf eodhours {
> > > dimensions:
> > > latitude = 360 ;
> > > longitude = 720 ;
> > > variables:
> > > float latitude(latitude) ;
> > > latitude:units = "degrees_north" ;
> > > float longitude(longitude) ;
> > > longitude:units = "degrees_east" ;
> > > float End of day(latitude, longitude) ;
> > > End of day:units = "hours" ; }
> > >
> > > The mask syntax I used is incorrect.
> > >
> > > For future MET user guides, the following paragraph could really
use
> > > an example to make this clear:
> > >
> > > "Alternatively, any gridded data file that MET can read may be
used
> > > to define a verification masking region. Users must specify a
> > > description of the field to be used from the input file and,
> > > optionally, may specify a threshold to be applied to that field.
> > > Any grid point where the resulting field is 0, the mask is
turned off.
> > > Any grid point where it is non-zero, the mask is turned on."
> > >
> > > Can you please help me get the syntax correct?
> > >
> > > Thanks,
> > > Matt
> > >
> > > // SIGNED //
> > > Matthew C. Sittel
> > > University Corporation for Atmospheric Research
> > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > matthew.sittel.ctr at us.af.mil
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Fri Aug 28 10:08:01 2015
Just resent with 2 attachments. Did the message make it through?
In case not, here are the attachments on our ftp site...
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_met.nc
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_met.png
And here's the message...
Matt,
Good news and bad news.
The good news is that I'm able to get MET to read this file, but the
bad
news is that it's with the development version of MET, not even a
released
version!
But I did modify your NetCDF file first in three ways:
(1) MET does not support variable names with embedded whitespace.
Those
variable names can ultimately be written to the ASCII output files of
MET
whose columns are whitespace delimited. Having whitespace in the
variable
names would throw off the number of columns. So I renamed "End of
day" to
"EndOfDay":
ncrename -v 'End of day',EndOfDay eodhours.nc eodhours_rename.nc
(2) Looks like you're using -999 for missing data, so I added the
_FillValue attribute.
(3) Also, MET needs to interpret this as a NetCDF file following the
CF
convention, so I added that as a global attribute:
ncatted -a _FillValue,EndOfDay,a,f,-999 -a
Conventions,global,a,c,"CF-1.0" \
eodhours_rename.nc eodhours_mod.nc
You're using METv4.1, but that version doesn't include support for
NetCDF-CF files. So this file won't work in METv4.1.
So here's what I did...
I ran the modified data file through the development "pcp_combine"
tool to
create a flavor of NetCDF that METv4.1 can actually read. I plotted
it
using this command:
METv4.1/bin/plot_data_plane \
eodhours_met.nc eodhours_met.ps \
'name="EndOfDay"; level="(*,*)";'
Both the NetCDF file (eodhours_met.nc) and the resulting image
(eodhour_met.png) are attached.
John
On Fri, Aug 28, 2015 at 9:48 AM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Email from 8/25...
>
> Matt,
>
> Good news and bad news.
>
> The good news is that I'm able to get MET to read this file, but the
bad
> news is that it's with the development version of MET, not even a
released
> version!
>
> But I did modify your NetCDF file first in three ways:
>
> (1) MET does not support variable names with embedded whitespace.
Those
> variable names can ultimately be written to the ASCII output files
of MET
> whose columns are whitespace delimited. Having whitespace in the
variable
> names would throw off the number of columns. So I renamed "End of
day" to
> "EndOfDay":
> ncrename -v 'End of day',EndOfDay eodhours.nc eodhours_rename.nc
>
> (2) Looks like you're using -999 for missing data, so I added the
> _FillValue attribute.
>
> (3) Also, MET needs to interpret this as a NetCDF file following the
CF
> convention, so I added that as a global attribute:
>
> ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> Conventions,global,a,c,"CF-1.0" \
> eodhours_rename.nc eodhours_mod.nc
>
> You're using METv4.1, but that version doesn't include support for
> NetCDF-CF files. So this file won't work in METv4.1.
>
> So here's what I did...
>
> I ran the modified data file through the development "pcp_combine"
tool to
> create a flavor of NetCDF that METv4.1 can actually read. I plotted
it
> using this command:
>
> METv4.1/bin/plot_data_plane \
> eodhours_met.nc eodhours_met.ps \
> 'name="EndOfDay"; level="(*,*)";'
>
> Both the NetCDF file (eodhours_met.nc) and the resulting image
> (eodhour_met.png) are attached.
>
> Try the following to use this as a mask in Grid-Stat. This defines
2
> different masks, one for eod = 24 and another for eod = 12:
>
> mask = {
> grid = [];
> poly = [
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
>
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ];
> };
>
> I agree, that's awfully convoluted!
>
> Hope that helps.
>
> John
>
>
> On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>>
>> Can you resend please? It must still be out in cyberspace. This
morning
>> I got all of yesterday's mail. Note I was here all day
yesterday... it's
>> always something.
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Friday, August 28, 2015 10:34 AM
>> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
>> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
>>
>> Matt,
>>
>> I received an email from you on August 25 with a NetCDF file
attachment.
>> Then I replied to you on August 25th in an email with 2 attachments
(
>> eodhours_met.nc and eodhours_met.ps) on August 25. Did you receive
my
>> reply on 8/25?
>>
>> Thanks,
>> John
>>
>> On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
>> WS/WXN via RT <met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>> >
>> > Hey John,
>> >
>> > Did you receive the NetCDF file I sent? I put it as an
attachment to
>> > an e-mail. AFWA E-mail has been really messed up here as of late
so
>> > it's possible that it never went out, or that if you replied I
haven't
>> > received it. I just wanted to check.
>> >
>> > Matt
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> > Sent: Tuesday, August 25, 2015 11:39 AM
>> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
>> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
>> >
>> > Matt,
>> >
>> > Are you able to send me that end of day NetCDF file? Probably
easiest
>> > for met to get it correct here and then let you know. Also, is
this
>> > METv4.1 or met-5.0?
>> >
>> > Thanks,
>> > John
>> >
>> > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
>> > WS/WXN via RT <met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
>> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
>> > > Queue: met_help
>> > > Subject: Grid Stat Config File Question
>> > > Owner: Nobody
>> > > Requestors: matthew.sittel.ctr at us.af.mil
>> > > Status: new
>> > > Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>> > > >
>> > >
>> > >
>> > > Hi,
>> > >
>> > > I'm trying to figure out the right way to set the config file
for
>> > > grid_stat to do the following:
>> > >
>> > > -- generate a mask for my data based on the contents of an
external
>> > > file, in NetCDF format
>> > > -- keying this mask off of a NetCDF variable named "End of day"
>> > > within the .nc file
>> > > -- Keep only those grid points where "End of day" equals 24
>> > >
>> > > I can't seem to find the right combination of equals,
semicolons,
>> > > etc., to do this.
>> > >
>> > > Right now I have:
>> > >
>> > > mask = {
>> > > grid = [
>> > > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
>> > > field = [
>> > > {
>> > > name="End of day";
>> > > cat_thresh=[==24];
>> > > }
>> > > ];
>> > > ];
>> > > poly = [];
>> > > };
>> > >
>> > > The header from the NetCDF file in question:
>> > >
>> > > netcdf eodhours {
>> > > dimensions:
>> > > latitude = 360 ;
>> > > longitude = 720 ;
>> > > variables:
>> > > float latitude(latitude) ;
>> > > latitude:units = "degrees_north" ;
>> > > float longitude(longitude) ;
>> > > longitude:units = "degrees_east" ;
>> > > float End of day(latitude, longitude) ;
>> > > End of day:units = "hours" ; }
>> > >
>> > > The mask syntax I used is incorrect.
>> > >
>> > > For future MET user guides, the following paragraph could
really use
>> > > an example to make this clear:
>> > >
>> > > "Alternatively, any gridded data file that MET can read may be
used
>> > > to define a verification masking region. Users must specify a
>> > > description of the field to be used from the input file and,
>> > > optionally, may specify a threshold to be applied to that
field.
>> > > Any grid point where the resulting field is 0, the mask is
turned off.
>> > > Any grid point where it is non-zero, the mask is turned on."
>> > >
>> > > Can you please help me get the syntax correct?
>> > >
>> > > Thanks,
>> > > Matt
>> > >
>> > > // SIGNED //
>> > > Matthew C. Sittel
>> > > University Corporation for Atmospheric Research
>> > > 16 WS/WXN, 557 WW, Offutt AFB, NE
>> > > matthew.sittel.ctr at us.af.mil
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Fri Aug 28 10:12:28 2015
Hi John,
Thank you for re-sending the reply and the files.
I didn’t generate this NetCDF file, so I can get the author to make
this easier should I need to replicate this product. I was wondering
about the whitespace.
Bob Craig is getting the approval to install MET 5.0 on our new
development server. We have to go through a lot of people to get that
approved. So if a 5.1 release is imminent, we should probably wait so
we don't have to go through all of this twice just to update. AFWA
frowns on introducing any sort of external software onto their
systems, which makes it harder for us.
So I tried your fix and I get an error message of:
"NetCDF: Attribute not found".
Also, I'm wondering why this is a 'poly' and not a 'grid'.
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, August 28, 2015 10:49 AM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Email from 8/25...
Matt,
Good news and bad news.
The good news is that I'm able to get MET to read this file, but the
bad news is that it's with the development version of MET, not even a
released version!
But I did modify your NetCDF file first in three ways:
(1) MET does not support variable names with embedded whitespace.
Those variable names can ultimately be written to the ASCII output
files of MET whose columns are whitespace delimited. Having
whitespace in the variable names would throw off the number of
columns. So I renamed "End of day" to
"EndOfDay":
ncrename -v 'End of day',EndOfDay eodhours.nc eodhours_rename.nc
(2) Looks like you're using -999 for missing data, so I added the
_FillValue attribute.
(3) Also, MET needs to interpret this as a NetCDF file following the
CF convention, so I added that as a global attribute:
ncatted -a _FillValue,EndOfDay,a,f,-999 -a
Conventions,global,a,c,"CF-1.0" \
eodhours_rename.nc eodhours_mod.nc
You're using METv4.1, but that version doesn't include support for
NetCDF-CF files. So this file won't work in METv4.1.
So here's what I did...
I ran the modified data file through the development "pcp_combine"
tool to create a flavor of NetCDF that METv4.1 can actually read. I
plotted it using this command:
METv4.1/bin/plot_data_plane \
eodhours_met.nc eodhours_met.ps \
'name="EndOfDay"; level="(*,*)";'
Both the NetCDF file (eodhours_met.nc) and the resulting image
(eodhour_met.png) are attached.
Try the following to use this as a mask in Grid-Stat. This defines 2
different masks, one for eod = 24 and another for eod = 12:
mask = {
grid = [];
poly = [
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc { name =
\"EndOfDay\"; level = \"(*,*)\"; } =24",
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
{ name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
I agree, that's awfully convoluted!
Hope that helps.
John
On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Can you resend please? It must still be out in cyberspace. This
> morning I got all of yesterday's mail. Note I was here all day
> yesterday... it's always something.
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, August 28, 2015 10:34 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> I received an email from you on August 25 with a NetCDF file
attachment.
> Then I replied to you on August 25th in an email with 2 attachments
(
> eodhours_met.nc and eodhours_met.ps) on August 25. Did you receive
my
> reply on 8/25?
>
> Thanks,
> John
>
> On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hey John,
> >
> > Did you receive the NetCDF file I sent? I put it as an attachment
> > to an e-mail. AFWA E-mail has been really messed up here as of
late
> > so it's possible that it never went out, or that if you replied I
> > haven't received it. I just wanted to check.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Tuesday, August 25, 2015 11:39 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > Are you able to send me that end of day NetCDF file? Probably
> > easiest for met to get it correct here and then let you know.
Also,
> > is this
> > METv4.1 or met-5.0?
> >
> > Thanks,
> > John
> >
> > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > Queue: met_help
> > > Subject: Grid Stat Config File Question
> > > Owner: Nobody
> > > Requestors: matthew.sittel.ctr at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > >
> > >
> > >
> > > Hi,
> > >
> > > I'm trying to figure out the right way to set the config file
for
> > > grid_stat to do the following:
> > >
> > > -- generate a mask for my data based on the contents of an
> > > external file, in NetCDF format
> > > -- keying this mask off of a NetCDF variable named "End of day"
> > > within the .nc file
> > > -- Keep only those grid points where "End of day" equals 24
> > >
> > > I can't seem to find the right combination of equals,
semicolons,
> > > etc., to do this.
> > >
> > > Right now I have:
> > >
> > > mask = {
> > > grid = [
> > > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > field = [
> > > {
> > > name="End of day";
> > > cat_thresh=[==24];
> > > }
> > > ];
> > > ];
> > > poly = [];
> > > };
> > >
> > > The header from the NetCDF file in question:
> > >
> > > netcdf eodhours {
> > > dimensions:
> > > latitude = 360 ;
> > > longitude = 720 ;
> > > variables:
> > > float latitude(latitude) ;
> > > latitude:units = "degrees_north" ;
> > > float longitude(longitude) ;
> > > longitude:units = "degrees_east" ;
> > > float End of day(latitude, longitude) ;
> > > End of day:units = "hours" ; }
> > >
> > > The mask syntax I used is incorrect.
> > >
> > > For future MET user guides, the following paragraph could really
> > > use an example to make this clear:
> > >
> > > "Alternatively, any gridded data file that MET can read may be
> > > used to define a verification masking region. Users must specify
a
> > > description of the field to be used from the input file and,
> > > optionally, may specify a threshold to be applied to that field.
> > > Any grid point where the resulting field is 0, the mask is
turned off.
> > > Any grid point where it is non-zero, the mask is turned on."
> > >
> > > Can you please help me get the syntax correct?
> > >
> > > Thanks,
> > > Matt
> > >
> > > // SIGNED //
> > > Matthew C. Sittel
> > > University Corporation for Atmospheric Research
> > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > matthew.sittel.ctr at us.af.mil
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Fri Aug 28 10:45:50 2015
Matt,
Please try the following...
(1) Retrieve this file:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_met.nc
(2) Run METv4.1 to try to read this file:
METv4.1/bin/plot_data_plane eodhours_met.nc eodhours_met.ps
'name="EndOfDay"; level="(*,*)";'
Does that successfully create an output PostScript file as it does on
my
machine? If so, using it in the Grid-Stat configuration file should
work
fine.
As for defining masks in Grid-Stat, as you know, there are 2 options:
grid
and poly...
- Basically, "grid" handles named grid like "G212" for NCEP Grid 212
or
"G001" for NCEP grid 1. "FULL" is a special case for grid where we
just
use all points in the input grid.
- And "poly" handles everything else... like reading an ASCII lat/lon
polyline file or reading gridded data files and optionally applying a
threshold to them.
Were we to do it over again, we might split that out by using "poly"
to
specify ASCII lat/lon polyline files and adding "data" to specify
gridded
data files. I suppose we didn't do that because we wanted to read the
lat/lon polyline file or the NetCDF output of the gen_poly_mask tool
at the
same place. Oh well.
John
On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hi John,
>
> Thank you for re-sending the reply and the files.
>
> I didn’t generate this NetCDF file, so I can get the author to make
this
> easier should I need to replicate this product. I was wondering
about the
> whitespace.
>
> Bob Craig is getting the approval to install MET 5.0 on our new
> development server. We have to go through a lot of people to get
that
> approved. So if a 5.1 release is imminent, we should probably wait
so we
> don't have to go through all of this twice just to update. AFWA
frowns on
> introducing any sort of external software onto their systems, which
makes
> it harder for us.
>
> So I tried your fix and I get an error message of:
>
> "NetCDF: Attribute not found".
>
> Also, I'm wondering why this is a 'poly' and not a 'grid'.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, August 28, 2015 10:49 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Email from 8/25...
>
> Matt,
>
> Good news and bad news.
>
> The good news is that I'm able to get MET to read this file, but the
bad
> news is that it's with the development version of MET, not even a
released
> version!
>
> But I did modify your NetCDF file first in three ways:
>
> (1) MET does not support variable names with embedded whitespace.
Those
> variable names can ultimately be written to the ASCII output files
of MET
> whose columns are whitespace delimited. Having whitespace in the
variable
> names would throw off the number of columns. So I renamed "End of
day" to
> "EndOfDay":
> ncrename -v 'End of day',EndOfDay eodhours.nc eodhours_rename.nc
>
> (2) Looks like you're using -999 for missing data, so I added the
> _FillValue attribute.
>
> (3) Also, MET needs to interpret this as a NetCDF file following the
CF
> convention, so I added that as a global attribute:
>
> ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> Conventions,global,a,c,"CF-1.0" \
> eodhours_rename.nc eodhours_mod.nc
>
> You're using METv4.1, but that version doesn't include support for
> NetCDF-CF files. So this file won't work in METv4.1.
>
> So here's what I did...
>
> I ran the modified data file through the development "pcp_combine"
tool to
> create a flavor of NetCDF that METv4.1 can actually read. I plotted
it
> using this command:
>
> METv4.1/bin/plot_data_plane \
> eodhours_met.nc eodhours_met.ps \
> 'name="EndOfDay"; level="(*,*)";'
>
> Both the NetCDF file (eodhours_met.nc) and the resulting image
> (eodhour_met.png) are attached.
>
> Try the following to use this as a mask in Grid-Stat. This defines
2
> different masks, one for eod = 24 and another for eod = 12:
>
> mask = {
> grid = [];
> poly = [
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
>
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
>
> I agree, that's awfully convoluted!
>
> Hope that helps.
>
> John
>
> On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Can you resend please? It must still be out in cyberspace. This
> > morning I got all of yesterday's mail. Note I was here all day
> > yesterday... it's always something.
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, August 28, 2015 10:34 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > I received an email from you on August 25 with a NetCDF file
attachment.
> > Then I replied to you on August 25th in an email with 2
attachments (
> > eodhours_met.nc and eodhours_met.ps) on August 25. Did you
receive my
> > reply on 8/25?
> >
> > Thanks,
> > John
> >
> > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hey John,
> > >
> > > Did you receive the NetCDF file I sent? I put it as an
attachment
> > > to an e-mail. AFWA E-mail has been really messed up here as of
late
> > > so it's possible that it never went out, or that if you replied
I
> > > haven't received it. I just wanted to check.
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> > >
> > > Matt,
> > >
> > > Are you able to send me that end of day NetCDF file? Probably
> > > easiest for met to get it correct here and then let you know.
Also,
> > > is this
> > > METv4.1 or met-5.0?
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
AFWA 16
> > > WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > > Queue: met_help
> > > > Subject: Grid Stat Config File Question
> > > > Owner: Nobody
> > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > > I'm trying to figure out the right way to set the config file
for
> > > > grid_stat to do the following:
> > > >
> > > > -- generate a mask for my data based on the contents of an
> > > > external file, in NetCDF format
> > > > -- keying this mask off of a NetCDF variable named "End of
day"
> > > > within the .nc file
> > > > -- Keep only those grid points where "End of day" equals 24
> > > >
> > > > I can't seem to find the right combination of equals,
semicolons,
> > > > etc., to do this.
> > > >
> > > > Right now I have:
> > > >
> > > > mask = {
> > > > grid = [
> > > > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > field = [
> > > > {
> > > > name="End of day";
> > > > cat_thresh=[==24];
> > > > }
> > > > ];
> > > > ];
> > > > poly = [];
> > > > };
> > > >
> > > > The header from the NetCDF file in question:
> > > >
> > > > netcdf eodhours {
> > > > dimensions:
> > > > latitude = 360 ;
> > > > longitude = 720 ;
> > > > variables:
> > > > float latitude(latitude) ;
> > > > latitude:units = "degrees_north" ;
> > > > float longitude(longitude) ;
> > > > longitude:units = "degrees_east" ;
> > > > float End of day(latitude, longitude) ;
> > > > End of day:units = "hours" ; }
> > > >
> > > > The mask syntax I used is incorrect.
> > > >
> > > > For future MET user guides, the following paragraph could
really
> > > > use an example to make this clear:
> > > >
> > > > "Alternatively, any gridded data file that MET can read may be
> > > > used to define a verification masking region. Users must
specify a
> > > > description of the field to be used from the input file and,
> > > > optionally, may specify a threshold to be applied to that
field.
> > > > Any grid point where the resulting field is 0, the mask is
turned
> off.
> > > > Any grid point where it is non-zero, the mask is turned on."
> > > >
> > > > Can you please help me get the syntax correct?
> > > >
> > > > Thanks,
> > > > Matt
> > > >
> > > > // SIGNED //
> > > > Matthew C. Sittel
> > > > University Corporation for Atmospheric Research
> > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > matthew.sittel.ctr at us.af.mil
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Fri Aug 28 11:03:56 2015
It did generate a .ps file that is non-zero in size. Now if I can
remember how we view .ps files on our devel system I'll be all set?
Just to double check, here's what I have in the config file for
'mask':
mask = {
grid = [];
poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
\"EndOfDay\"; level=\"(*,*)\";} = 24"];
};
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, August 28, 2015 11:46 AM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
Please try the following...
(1) Retrieve this file:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_met.nc
(2) Run METv4.1 to try to read this file:
METv4.1/bin/plot_data_plane eodhours_met.nc eodhours_met.ps
'name="EndOfDay"; level="(*,*)";'
Does that successfully create an output PostScript file as it does on
my machine? If so, using it in the Grid-Stat configuration file
should work fine.
As for defining masks in Grid-Stat, as you know, there are 2 options:
grid and poly...
- Basically, "grid" handles named grid like "G212" for NCEP Grid 212
or "G001" for NCEP grid 1. "FULL" is a special case for grid where we
just use all points in the input grid.
- And "poly" handles everything else... like reading an ASCII lat/lon
polyline file or reading gridded data files and optionally applying a
threshold to them.
Were we to do it over again, we might split that out by using "poly"
to specify ASCII lat/lon polyline files and adding "data" to specify
gridded data files. I suppose we didn't do that because we wanted to
read the lat/lon polyline file or the NetCDF output of the
gen_poly_mask tool at the same place. Oh well.
John
On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hi John,
>
> Thank you for re-sending the reply and the files.
>
> I didn’t generate this NetCDF file, so I can get the author to make
> this easier should I need to replicate this product. I was
wondering
> about the whitespace.
>
> Bob Craig is getting the approval to install MET 5.0 on our new
> development server. We have to go through a lot of people to get
that
> approved. So if a 5.1 release is imminent, we should probably wait
so
> we don't have to go through all of this twice just to update. AFWA
> frowns on introducing any sort of external software onto their
> systems, which makes it harder for us.
>
> So I tried your fix and I get an error message of:
>
> "NetCDF: Attribute not found".
>
> Also, I'm wondering why this is a 'poly' and not a 'grid'.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, August 28, 2015 10:49 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Email from 8/25...
>
> Matt,
>
> Good news and bad news.
>
> The good news is that I'm able to get MET to read this file, but the
> bad news is that it's with the development version of MET, not even
a
> released version!
>
> But I did modify your NetCDF file first in three ways:
>
> (1) MET does not support variable names with embedded whitespace.
> Those variable names can ultimately be written to the ASCII output
> files of MET whose columns are whitespace delimited. Having
> whitespace in the variable names would throw off the number of
> columns. So I renamed "End of day" to
> "EndOfDay":
> ncrename -v 'End of day',EndOfDay eodhours.nc eodhours_rename.nc
>
> (2) Looks like you're using -999 for missing data, so I added the
> _FillValue attribute.
>
> (3) Also, MET needs to interpret this as a NetCDF file following the
> CF convention, so I added that as a global attribute:
>
> ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> Conventions,global,a,c,"CF-1.0" \
> eodhours_rename.nc eodhours_mod.nc
>
> You're using METv4.1, but that version doesn't include support for
> NetCDF-CF files. So this file won't work in METv4.1.
>
> So here's what I did...
>
> I ran the modified data file through the development "pcp_combine"
> tool to create a flavor of NetCDF that METv4.1 can actually read. I
> plotted it using this command:
>
> METv4.1/bin/plot_data_plane \
> eodhours_met.nc eodhours_met.ps \
> 'name="EndOfDay"; level="(*,*)";'
>
> Both the NetCDF file (eodhours_met.nc) and the resulting image
> (eodhour_met.png) are attached.
>
> Try the following to use this as a mask in Grid-Stat. This defines
2
> different masks, one for eod = 24 and another for eod = 12:
>
> mask = {
> grid = [];
> poly = [
> "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
>
> "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
>
> I agree, that's awfully convoluted!
>
> Hope that helps.
>
> John
>
> On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Can you resend please? It must still be out in cyberspace. This
> > morning I got all of yesterday's mail. Note I was here all day
> > yesterday... it's always something.
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, August 28, 2015 10:34 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > I received an email from you on August 25 with a NetCDF file
attachment.
> > Then I replied to you on August 25th in an email with 2
attachments
> > ( eodhours_met.nc and eodhours_met.ps) on August 25. Did you
> > receive my reply on 8/25?
> >
> > Thanks,
> > John
> >
> > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hey John,
> > >
> > > Did you receive the NetCDF file I sent? I put it as an
attachment
> > > to an e-mail. AFWA E-mail has been really messed up here as of
> > > late so it's possible that it never went out, or that if you
> > > replied I haven't received it. I just wanted to check.
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > Question
> > >
> > > Matt,
> > >
> > > Are you able to send me that end of day NetCDF file? Probably
> > > easiest for met to get it correct here and then let you know.
> > > Also, is this
> > > METv4.1 or met-5.0?
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > > Queue: met_help
> > > > Subject: Grid Stat Config File Question
> > > > Owner: Nobody
> > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > > I'm trying to figure out the right way to set the config file
> > > > for grid_stat to do the following:
> > > >
> > > > -- generate a mask for my data based on the contents of an
> > > > external file, in NetCDF format
> > > > -- keying this mask off of a NetCDF variable named "End of
day"
> > > > within the .nc file
> > > > -- Keep only those grid points where "End of day" equals 24
> > > >
> > > > I can't seem to find the right combination of equals,
> > > > semicolons, etc., to do this.
> > > >
> > > > Right now I have:
> > > >
> > > > mask = {
> > > > grid = [
> > > > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > field = [
> > > > {
> > > > name="End of day";
> > > > cat_thresh=[==24];
> > > > }
> > > > ];
> > > > ];
> > > > poly = [];
> > > > };
> > > >
> > > > The header from the NetCDF file in question:
> > > >
> > > > netcdf eodhours {
> > > > dimensions:
> > > > latitude = 360 ;
> > > > longitude = 720 ;
> > > > variables:
> > > > float latitude(latitude) ;
> > > > latitude:units = "degrees_north" ;
> > > > float longitude(longitude) ;
> > > > longitude:units = "degrees_east" ;
> > > > float End of day(latitude, longitude) ;
> > > > End of day:units = "hours" ; }
> > > >
> > > > The mask syntax I used is incorrect.
> > > >
> > > > For future MET user guides, the following paragraph could
really
> > > > use an example to make this clear:
> > > >
> > > > "Alternatively, any gridded data file that MET can read may be
> > > > used to define a verification masking region. Users must
specify
> > > > a description of the field to be used from the input file and,
> > > > optionally, may specify a threshold to be applied to that
field.
> > > > Any grid point where the resulting field is 0, the mask is
> > > > turned
> off.
> > > > Any grid point where it is non-zero, the mask is turned on."
> > > >
> > > > Can you please help me get the syntax correct?
> > > >
> > > > Thanks,
> > > > Matt
> > > >
> > > > // SIGNED //
> > > > Matthew C. Sittel
> > > > University Corporation for Atmospheric Research
> > > > 16 WS/WXN, 557 WW, Offutt AFB, NE matthew.sittel.ctr at us.af.mil
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Fri Aug 28 11:16:05 2015
Matt,
Try running the convert command to reformat from postscript to png:
convert -rotate 90 eodhours_met.ps eodhours_met.png
And then view that .png file either through a web browser or using the
display command:
display eodhours_met.png
In your settings it looks like you have a space between '=' and '24'.
That
probably won't parse correctly... try removing that space:
mask = {
grid = [];
poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
\"EndOfDay\"; level=\"(*,*)\";} =24"];
};
On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> It did generate a .ps file that is non-zero in size. Now if I can
> remember how we view .ps files on our devel system I'll be all set?
>
> Just to double check, here's what I have in the config file for
'mask':
>
> mask = {
> grid = [];
> poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
> \"EndOfDay\"; level=\"(*,*)\";} = 24"];
> };
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, August 28, 2015 11:46 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Please try the following...
>
> (1) Retrieve this file:
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_met.nc
> (2) Run METv4.1 to try to read this file:
> METv4.1/bin/plot_data_plane eodhours_met.nc eodhours_met.ps
> 'name="EndOfDay"; level="(*,*)";'
>
> Does that successfully create an output PostScript file as it does
on my
> machine? If so, using it in the Grid-Stat configuration file should
work
> fine.
>
> As for defining masks in Grid-Stat, as you know, there are 2
options: grid
> and poly...
> - Basically, "grid" handles named grid like "G212" for NCEP Grid 212
or
> "G001" for NCEP grid 1. "FULL" is a special case for grid where we
just
> use all points in the input grid.
> - And "poly" handles everything else... like reading an ASCII
lat/lon
> polyline file or reading gridded data files and optionally applying
a
> threshold to them.
>
> Were we to do it over again, we might split that out by using "poly"
to
> specify ASCII lat/lon polyline files and adding "data" to specify
gridded
> data files. I suppose we didn't do that because we wanted to read
the
> lat/lon polyline file or the NetCDF output of the gen_poly_mask tool
at the
> same place. Oh well.
>
> John
>
> On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hi John,
> >
> > Thank you for re-sending the reply and the files.
> >
> > I didn’t generate this NetCDF file, so I can get the author to
make
> > this easier should I need to replicate this product. I was
wondering
> > about the whitespace.
> >
> > Bob Craig is getting the approval to install MET 5.0 on our new
> > development server. We have to go through a lot of people to get
that
> > approved. So if a 5.1 release is imminent, we should probably
wait so
> > we don't have to go through all of this twice just to update.
AFWA
> > frowns on introducing any sort of external software onto their
> > systems, which makes it harder for us.
> >
> > So I tried your fix and I get an error message of:
> >
> > "NetCDF: Attribute not found".
> >
> > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, August 28, 2015 10:49 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Email from 8/25...
> >
> > Matt,
> >
> > Good news and bad news.
> >
> > The good news is that I'm able to get MET to read this file, but
the
> > bad news is that it's with the development version of MET, not
even a
> > released version!
> >
> > But I did modify your NetCDF file first in three ways:
> >
> > (1) MET does not support variable names with embedded whitespace.
> > Those variable names can ultimately be written to the ASCII output
> > files of MET whose columns are whitespace delimited. Having
> > whitespace in the variable names would throw off the number of
> > columns. So I renamed "End of day" to
> > "EndOfDay":
> > ncrename -v 'End of day',EndOfDay eodhours.nc
eodhours_rename.nc
> >
> > (2) Looks like you're using -999 for missing data, so I added the
> > _FillValue attribute.
> >
> > (3) Also, MET needs to interpret this as a NetCDF file following
the
> > CF convention, so I added that as a global attribute:
> >
> > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > Conventions,global,a,c,"CF-1.0" \
> > eodhours_rename.nc eodhours_mod.nc
> >
> > You're using METv4.1, but that version doesn't include support for
> > NetCDF-CF files. So this file won't work in METv4.1.
> >
> > So here's what I did...
> >
> > I ran the modified data file through the development "pcp_combine"
> > tool to create a flavor of NetCDF that METv4.1 can actually read.
I
> > plotted it using this command:
> >
> > METv4.1/bin/plot_data_plane \
> > eodhours_met.nc eodhours_met.ps \
> > 'name="EndOfDay"; level="(*,*)";'
> >
> > Both the NetCDF file (eodhours_met.nc) and the resulting image
> > (eodhour_met.png) are attached.
> >
> > Try the following to use this as a mask in Grid-Stat. This
defines 2
> > different masks, one for eod = 24 and another for eod = 12:
> >
> > mask = {
> > grid = [];
> > poly = [
> > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> >
> > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> >
> > I agree, that's awfully convoluted!
> >
> > Hope that helps.
> >
> > John
> >
> > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Can you resend please? It must still be out in cyberspace.
This
> > > morning I got all of yesterday's mail. Note I was here all day
> > > yesterday... it's always something.
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, August 28, 2015 10:34 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> > >
> > > Matt,
> > >
> > > I received an email from you on August 25 with a NetCDF file
> attachment.
> > > Then I replied to you on August 25th in an email with 2
attachments
> > > ( eodhours_met.nc and eodhours_met.ps) on August 25. Did you
> > > receive my reply on 8/25?
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > > WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Hey John,
> > > >
> > > > Did you receive the NetCDF file I sent? I put it as an
attachment
> > > > to an e-mail. AFWA E-mail has been really messed up here as
of
> > > > late so it's possible that it never went out, or that if you
> > > > replied I haven't received it. I just wanted to check.
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > Are you able to send me that end of day NetCDF file? Probably
> > > > easiest for met to get it correct here and then let you know.
> > > > Also, is this
> > > > METv4.1 or met-5.0?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > > > Queue: met_help
> > > > > Subject: Grid Stat Config File Question
> > > > > Owner: Nobody
> > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm trying to figure out the right way to set the config
file
> > > > > for grid_stat to do the following:
> > > > >
> > > > > -- generate a mask for my data based on the contents of an
> > > > > external file, in NetCDF format
> > > > > -- keying this mask off of a NetCDF variable named "End of
day"
> > > > > within the .nc file
> > > > > -- Keep only those grid points where "End of day" equals 24
> > > > >
> > > > > I can't seem to find the right combination of equals,
> > > > > semicolons, etc., to do this.
> > > > >
> > > > > Right now I have:
> > > > >
> > > > > mask = {
> > > > > grid = [
> > > > > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > > field = [
> > > > > {
> > > > > name="End of day";
> > > > > cat_thresh=[==24];
> > > > > }
> > > > > ];
> > > > > ];
> > > > > poly = [];
> > > > > };
> > > > >
> > > > > The header from the NetCDF file in question:
> > > > >
> > > > > netcdf eodhours {
> > > > > dimensions:
> > > > > latitude = 360 ;
> > > > > longitude = 720 ;
> > > > > variables:
> > > > > float latitude(latitude) ;
> > > > > latitude:units = "degrees_north" ;
> > > > > float longitude(longitude) ;
> > > > > longitude:units = "degrees_east" ;
> > > > > float End of day(latitude, longitude) ;
> > > > > End of day:units = "hours" ; }
> > > > >
> > > > > The mask syntax I used is incorrect.
> > > > >
> > > > > For future MET user guides, the following paragraph could
really
> > > > > use an example to make this clear:
> > > > >
> > > > > "Alternatively, any gridded data file that MET can read may
be
> > > > > used to define a verification masking region. Users must
specify
> > > > > a description of the field to be used from the input file
and,
> > > > > optionally, may specify a threshold to be applied to that
field.
> > > > > Any grid point where the resulting field is 0, the mask is
> > > > > turned
> > off.
> > > > > Any grid point where it is non-zero, the mask is turned on."
> > > > >
> > > > > Can you please help me get the syntax correct?
> > > > >
> > > > > Thanks,
> > > > > Matt
> > > > >
> > > > > // SIGNED //
> > > > > Matthew C. Sittel
> > > > > University Corporation for Atmospheric Research
> > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
matthew.sittel.ctr at us.af.mil
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Fri Aug 28 11:33:55 2015
Image looks just fine.
Made the change to "=24" and I still get the same error: " NetCDF:
Attribute not found".
To be clear this is in v4.1. Anything else you can suggest?
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, August 28, 2015 12:16 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
Try running the convert command to reformat from postscript to png:
convert -rotate 90 eodhours_met.ps eodhours_met.png
And then view that .png file either through a web browser or using the
display command:
display eodhours_met.png
In your settings it looks like you have a space between '=' and '24'.
That probably won't parse correctly... try removing that space:
mask = {
grid = [];
poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
\"EndOfDay\"; level=\"(*,*)\";} =24"]; };
On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> It did generate a .ps file that is non-zero in size. Now if I can
> remember how we view .ps files on our devel system I'll be all set?
>
> Just to double check, here's what I have in the config file for
'mask':
>
> mask = {
> grid = [];
> poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
> \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, August 28, 2015 11:46 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Please try the following...
>
> (1) Retrieve this file:
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_met
> .nc
> (2) Run METv4.1 to try to read this file:
> METv4.1/bin/plot_data_plane eodhours_met.nc eodhours_met.ps
> 'name="EndOfDay"; level="(*,*)";'
>
> Does that successfully create an output PostScript file as it does
on
> my machine? If so, using it in the Grid-Stat configuration file
> should work fine.
>
> As for defining masks in Grid-Stat, as you know, there are 2
options:
> grid and poly...
> - Basically, "grid" handles named grid like "G212" for NCEP Grid 212
> or "G001" for NCEP grid 1. "FULL" is a special case for grid where
we
> just use all points in the input grid.
> - And "poly" handles everything else... like reading an ASCII
lat/lon
> polyline file or reading gridded data files and optionally applying
a
> threshold to them.
>
> Were we to do it over again, we might split that out by using "poly"
> to specify ASCII lat/lon polyline files and adding "data" to specify
> gridded data files. I suppose we didn't do that because we wanted
to
> read the lat/lon polyline file or the NetCDF output of the
> gen_poly_mask tool at the same place. Oh well.
>
> John
>
> On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hi John,
> >
> > Thank you for re-sending the reply and the files.
> >
> > I didn’t generate this NetCDF file, so I can get the author to
make
> > this easier should I need to replicate this product. I was
> > wondering about the whitespace.
> >
> > Bob Craig is getting the approval to install MET 5.0 on our new
> > development server. We have to go through a lot of people to get
> > that approved. So if a 5.1 release is imminent, we should
probably
> > wait so we don't have to go through all of this twice just to
> > update. AFWA frowns on introducing any sort of external software
> > onto their systems, which makes it harder for us.
> >
> > So I tried your fix and I get an error message of:
> >
> > "NetCDF: Attribute not found".
> >
> > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, August 28, 2015 10:49 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Email from 8/25...
> >
> > Matt,
> >
> > Good news and bad news.
> >
> > The good news is that I'm able to get MET to read this file, but
the
> > bad news is that it's with the development version of MET, not
even
> > a released version!
> >
> > But I did modify your NetCDF file first in three ways:
> >
> > (1) MET does not support variable names with embedded whitespace.
> > Those variable names can ultimately be written to the ASCII output
> > files of MET whose columns are whitespace delimited. Having
> > whitespace in the variable names would throw off the number of
> > columns. So I renamed "End of day" to
> > "EndOfDay":
> > ncrename -v 'End of day',EndOfDay eodhours.nc
eodhours_rename.nc
> >
> > (2) Looks like you're using -999 for missing data, so I added the
> > _FillValue attribute.
> >
> > (3) Also, MET needs to interpret this as a NetCDF file following
the
> > CF convention, so I added that as a global attribute:
> >
> > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > Conventions,global,a,c,"CF-1.0" \
> > eodhours_rename.nc eodhours_mod.nc
> >
> > You're using METv4.1, but that version doesn't include support for
> > NetCDF-CF files. So this file won't work in METv4.1.
> >
> > So here's what I did...
> >
> > I ran the modified data file through the development "pcp_combine"
> > tool to create a flavor of NetCDF that METv4.1 can actually read.
I
> > plotted it using this command:
> >
> > METv4.1/bin/plot_data_plane \
> > eodhours_met.nc eodhours_met.ps \
> > 'name="EndOfDay"; level="(*,*)";'
> >
> > Both the NetCDF file (eodhours_met.nc) and the resulting image
> > (eodhour_met.png) are attached.
> >
> > Try the following to use this as a mask in Grid-Stat. This
defines
> > 2 different masks, one for eod = 24 and another for eod = 12:
> >
> > mask = {
> > grid = [];
> > poly = [
> > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> >
> > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> >
> > I agree, that's awfully convoluted!
> >
> > Hope that helps.
> >
> > John
> >
> > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Can you resend please? It must still be out in cyberspace.
This
> > > morning I got all of yesterday's mail. Note I was here all day
> > > yesterday... it's always something.
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, August 28, 2015 10:34 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > Question
> > >
> > > Matt,
> > >
> > > I received an email from you on August 25 with a NetCDF file
> attachment.
> > > Then I replied to you on August 25th in an email with 2
> > > attachments ( eodhours_met.nc and eodhours_met.ps) on August 25.
> > > Did you receive my reply on 8/25?
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Hey John,
> > > >
> > > > Did you receive the NetCDF file I sent? I put it as an
> > > > attachment to an e-mail. AFWA E-mail has been really messed
up
> > > > here as of late so it's possible that it never went out, or
that
> > > > if you replied I haven't received it. I just wanted to check.
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > Are you able to send me that end of day NetCDF file? Probably
> > > > easiest for met to get it correct here and then let you know.
> > > > Also, is this
> > > > METv4.1 or met-5.0?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
> > > > AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > > > Queue: met_help
> > > > > Subject: Grid Stat Config File Question
> > > > > Owner: Nobody
> > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm trying to figure out the right way to set the config
file
> > > > > for grid_stat to do the following:
> > > > >
> > > > > -- generate a mask for my data based on the contents of an
> > > > > external file, in NetCDF format
> > > > > -- keying this mask off of a NetCDF variable named "End of
day"
> > > > > within the .nc file
> > > > > -- Keep only those grid points where "End of day" equals 24
> > > > >
> > > > > I can't seem to find the right combination of equals,
> > > > > semicolons, etc., to do this.
> > > > >
> > > > > Right now I have:
> > > > >
> > > > > mask = {
> > > > > grid = [
> > > > > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > > field = [
> > > > > {
> > > > > name="End of day";
> > > > > cat_thresh=[==24];
> > > > > }
> > > > > ];
> > > > > ];
> > > > > poly = [];
> > > > > };
> > > > >
> > > > > The header from the NetCDF file in question:
> > > > >
> > > > > netcdf eodhours {
> > > > > dimensions:
> > > > > latitude = 360 ;
> > > > > longitude = 720 ;
> > > > > variables:
> > > > > float latitude(latitude) ;
> > > > > latitude:units = "degrees_north" ;
> > > > > float longitude(longitude) ;
> > > > > longitude:units = "degrees_east" ;
> > > > > float End of day(latitude, longitude) ;
> > > > > End of day:units = "hours" ; }
> > > > >
> > > > > The mask syntax I used is incorrect.
> > > > >
> > > > > For future MET user guides, the following paragraph could
> > > > > really use an example to make this clear:
> > > > >
> > > > > "Alternatively, any gridded data file that MET can read may
be
> > > > > used to define a verification masking region. Users must
> > > > > specify a description of the field to be used from the input
> > > > > file and, optionally, may specify a threshold to be applied
to that field.
> > > > > Any grid point where the resulting field is 0, the mask is
> > > > > turned
> > off.
> > > > > Any grid point where it is non-zero, the mask is turned on."
> > > > >
> > > > > Can you please help me get the syntax correct?
> > > > >
> > > > > Thanks,
> > > > > Matt
> > > > >
> > > > > // SIGNED //
> > > > > Matthew C. Sittel
> > > > > University Corporation for Atmospheric Research
> > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
matthew.sittel.ctr at us.af.mil
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Mon Aug 31 13:12:29 2015
Hey John,
Just checking to make sure we don't have e-mail problems here again.
Below is the last response I got from you.
The modelers are breathing down my neck wanting their data so I wanted
to check.
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, August 28, 2015 12:16 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
Try running the convert command to reformat from postscript to png:
convert -rotate 90 eodhours_met.ps eodhours_met.png
And then view that .png file either through a web browser or using the
display command:
display eodhours_met.png
In your settings it looks like you have a space between '=' and '24'.
That probably won't parse correctly... try removing that space:
mask = {
grid = [];
poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
\"EndOfDay\"; level=\"(*,*)\";} =24"]; };
On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> It did generate a .ps file that is non-zero in size. Now if I can
> remember how we view .ps files on our devel system I'll be all set?
>
> Just to double check, here's what I have in the config file for
'mask':
>
> mask = {
> grid = [];
> poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
> \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, August 28, 2015 11:46 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Please try the following...
>
> (1) Retrieve this file:
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_met
> .nc
> (2) Run METv4.1 to try to read this file:
> METv4.1/bin/plot_data_plane eodhours_met.nc eodhours_met.ps
> 'name="EndOfDay"; level="(*,*)";'
>
> Does that successfully create an output PostScript file as it does
on
> my machine? If so, using it in the Grid-Stat configuration file
> should work fine.
>
> As for defining masks in Grid-Stat, as you know, there are 2
options:
> grid and poly...
> - Basically, "grid" handles named grid like "G212" for NCEP Grid 212
> or "G001" for NCEP grid 1. "FULL" is a special case for grid where
we
> just use all points in the input grid.
> - And "poly" handles everything else... like reading an ASCII
lat/lon
> polyline file or reading gridded data files and optionally applying
a
> threshold to them.
>
> Were we to do it over again, we might split that out by using "poly"
> to specify ASCII lat/lon polyline files and adding "data" to specify
> gridded data files. I suppose we didn't do that because we wanted
to
> read the lat/lon polyline file or the NetCDF output of the
> gen_poly_mask tool at the same place. Oh well.
>
> John
>
> On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hi John,
> >
> > Thank you for re-sending the reply and the files.
> >
> > I didn’t generate this NetCDF file, so I can get the author to
make
> > this easier should I need to replicate this product. I was
> > wondering about the whitespace.
> >
> > Bob Craig is getting the approval to install MET 5.0 on our new
> > development server. We have to go through a lot of people to get
> > that approved. So if a 5.1 release is imminent, we should
probably
> > wait so we don't have to go through all of this twice just to
> > update. AFWA frowns on introducing any sort of external software
> > onto their systems, which makes it harder for us.
> >
> > So I tried your fix and I get an error message of:
> >
> > "NetCDF: Attribute not found".
> >
> > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, August 28, 2015 10:49 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Email from 8/25...
> >
> > Matt,
> >
> > Good news and bad news.
> >
> > The good news is that I'm able to get MET to read this file, but
the
> > bad news is that it's with the development version of MET, not
even
> > a released version!
> >
> > But I did modify your NetCDF file first in three ways:
> >
> > (1) MET does not support variable names with embedded whitespace.
> > Those variable names can ultimately be written to the ASCII output
> > files of MET whose columns are whitespace delimited. Having
> > whitespace in the variable names would throw off the number of
> > columns. So I renamed "End of day" to
> > "EndOfDay":
> > ncrename -v 'End of day',EndOfDay eodhours.nc
eodhours_rename.nc
> >
> > (2) Looks like you're using -999 for missing data, so I added the
> > _FillValue attribute.
> >
> > (3) Also, MET needs to interpret this as a NetCDF file following
the
> > CF convention, so I added that as a global attribute:
> >
> > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > Conventions,global,a,c,"CF-1.0" \
> > eodhours_rename.nc eodhours_mod.nc
> >
> > You're using METv4.1, but that version doesn't include support for
> > NetCDF-CF files. So this file won't work in METv4.1.
> >
> > So here's what I did...
> >
> > I ran the modified data file through the development "pcp_combine"
> > tool to create a flavor of NetCDF that METv4.1 can actually read.
I
> > plotted it using this command:
> >
> > METv4.1/bin/plot_data_plane \
> > eodhours_met.nc eodhours_met.ps \
> > 'name="EndOfDay"; level="(*,*)";'
> >
> > Both the NetCDF file (eodhours_met.nc) and the resulting image
> > (eodhour_met.png) are attached.
> >
> > Try the following to use this as a mask in Grid-Stat. This
defines
> > 2 different masks, one for eod = 24 and another for eod = 12:
> >
> > mask = {
> > grid = [];
> > poly = [
> > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> >
> > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> >
> > I agree, that's awfully convoluted!
> >
> > Hope that helps.
> >
> > John
> >
> > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Can you resend please? It must still be out in cyberspace.
This
> > > morning I got all of yesterday's mail. Note I was here all day
> > > yesterday... it's always something.
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, August 28, 2015 10:34 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > Question
> > >
> > > Matt,
> > >
> > > I received an email from you on August 25 with a NetCDF file
> attachment.
> > > Then I replied to you on August 25th in an email with 2
> > > attachments ( eodhours_met.nc and eodhours_met.ps) on August 25.
> > > Did you receive my reply on 8/25?
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Hey John,
> > > >
> > > > Did you receive the NetCDF file I sent? I put it as an
> > > > attachment to an e-mail. AFWA E-mail has been really messed
up
> > > > here as of late so it's possible that it never went out, or
that
> > > > if you replied I haven't received it. I just wanted to check.
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > Are you able to send me that end of day NetCDF file? Probably
> > > > easiest for met to get it correct here and then let you know.
> > > > Also, is this
> > > > METv4.1 or met-5.0?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
> > > > AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > > > Queue: met_help
> > > > > Subject: Grid Stat Config File Question
> > > > > Owner: Nobody
> > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm trying to figure out the right way to set the config
file
> > > > > for grid_stat to do the following:
> > > > >
> > > > > -- generate a mask for my data based on the contents of an
> > > > > external file, in NetCDF format
> > > > > -- keying this mask off of a NetCDF variable named "End of
day"
> > > > > within the .nc file
> > > > > -- Keep only those grid points where "End of day" equals 24
> > > > >
> > > > > I can't seem to find the right combination of equals,
> > > > > semicolons, etc., to do this.
> > > > >
> > > > > Right now I have:
> > > > >
> > > > > mask = {
> > > > > grid = [
> > > > > name = "/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > > field = [
> > > > > {
> > > > > name="End of day";
> > > > > cat_thresh=[==24];
> > > > > }
> > > > > ];
> > > > > ];
> > > > > poly = [];
> > > > > };
> > > > >
> > > > > The header from the NetCDF file in question:
> > > > >
> > > > > netcdf eodhours {
> > > > > dimensions:
> > > > > latitude = 360 ;
> > > > > longitude = 720 ;
> > > > > variables:
> > > > > float latitude(latitude) ;
> > > > > latitude:units = "degrees_north" ;
> > > > > float longitude(longitude) ;
> > > > > longitude:units = "degrees_east" ;
> > > > > float End of day(latitude, longitude) ;
> > > > > End of day:units = "hours" ; }
> > > > >
> > > > > The mask syntax I used is incorrect.
> > > > >
> > > > > For future MET user guides, the following paragraph could
> > > > > really use an example to make this clear:
> > > > >
> > > > > "Alternatively, any gridded data file that MET can read may
be
> > > > > used to define a verification masking region. Users must
> > > > > specify a description of the field to be used from the input
> > > > > file and, optionally, may specify a threshold to be applied
to that field.
> > > > > Any grid point where the resulting field is 0, the mask is
> > > > > turned
> > off.
> > > > > Any grid point where it is non-zero, the mask is turned on."
> > > > >
> > > > > Can you please help me get the syntax correct?
> > > > >
> > > > > Thanks,
> > > > > Matt
> > > > >
> > > > > // SIGNED //
> > > > > Matthew C. Sittel
> > > > > University Corporation for Atmospheric Research
> > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
matthew.sittel.ctr at us.af.mil
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Mon Aug 31 13:46:45 2015
Matt,
If you're able to successfully make an image with that plot_data_plane
command I sent, then MET is able to read that data file fine.
However, you're still receiving the very unhelpful error message:
"NetCDF: Attribute not found"
Seems to me there are a couple possible reasons...
(1) In your Grid-Stat configuration file in the "mask->poly" section,
are
you positive that you're pointing to the updated mask file I sent you?
Please make sure the path is correct and you're pointing to the file
on
which you ran that plot_data_plane command.
(2) Another possibility is that you're getting this error message from
Grid-Stat when trying to read some other gridded data file... like the
forecast or observation files.
If you're still having trouble, please rerun your grid_stat command
using
these options:
-v 4 -log run_grid_stat.log
That'll write the output from grid_stat to a log file named
"run_grid_stat.log".
Please send me that log file.
Thanks,
John
On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hey John,
>
> Just checking to make sure we don't have e-mail problems here again.
> Below is the last response I got from you.
>
> The modelers are breathing down my neck wanting their data so I
wanted to
> check.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, August 28, 2015 12:16 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Try running the convert command to reformat from postscript to png:
> convert -rotate 90 eodhours_met.ps eodhours_met.png
>
> And then view that .png file either through a web browser or using
the
> display command:
> display eodhours_met.png
>
> In your settings it looks like you have a space between '=' and
'24'.
> That probably won't parse correctly... try removing that space:
>
> mask = {
> grid = [];
> poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
> \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
>
> On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > It did generate a .ps file that is non-zero in size. Now if I can
> > remember how we view .ps files on our devel system I'll be all
set?
> >
> > Just to double check, here's what I have in the config file for
'mask':
> >
> > mask = {
> > grid = [];
> > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name
=
> > \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, August 28, 2015 11:46 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > Please try the following...
> >
> > (1) Retrieve this file:
> >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_met
> > .nc
> > (2) Run METv4.1 to try to read this file:
> > METv4.1/bin/plot_data_plane eodhours_met.nc eodhours_met.ps
> > 'name="EndOfDay"; level="(*,*)";'
> >
> > Does that successfully create an output PostScript file as it does
on
> > my machine? If so, using it in the Grid-Stat configuration file
> > should work fine.
> >
> > As for defining masks in Grid-Stat, as you know, there are 2
options:
> > grid and poly...
> > - Basically, "grid" handles named grid like "G212" for NCEP Grid
212
> > or "G001" for NCEP grid 1. "FULL" is a special case for grid
where we
> > just use all points in the input grid.
> > - And "poly" handles everything else... like reading an ASCII
lat/lon
> > polyline file or reading gridded data files and optionally
applying a
> > threshold to them.
> >
> > Were we to do it over again, we might split that out by using
"poly"
> > to specify ASCII lat/lon polyline files and adding "data" to
specify
> > gridded data files. I suppose we didn't do that because we wanted
to
> > read the lat/lon polyline file or the NetCDF output of the
> > gen_poly_mask tool at the same place. Oh well.
> >
> > John
> >
> > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hi John,
> > >
> > > Thank you for re-sending the reply and the files.
> > >
> > > I didn’t generate this NetCDF file, so I can get the author to
make
> > > this easier should I need to replicate this product. I was
> > > wondering about the whitespace.
> > >
> > > Bob Craig is getting the approval to install MET 5.0 on our new
> > > development server. We have to go through a lot of people to
get
> > > that approved. So if a 5.1 release is imminent, we should
probably
> > > wait so we don't have to go through all of this twice just to
> > > update. AFWA frowns on introducing any sort of external
software
> > > onto their systems, which makes it harder for us.
> > >
> > > So I tried your fix and I get an error message of:
> > >
> > > "NetCDF: Attribute not found".
> > >
> > > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, August 28, 2015 10:49 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> > >
> > > Email from 8/25...
> > >
> > > Matt,
> > >
> > > Good news and bad news.
> > >
> > > The good news is that I'm able to get MET to read this file, but
the
> > > bad news is that it's with the development version of MET, not
even
> > > a released version!
> > >
> > > But I did modify your NetCDF file first in three ways:
> > >
> > > (1) MET does not support variable names with embedded
whitespace.
> > > Those variable names can ultimately be written to the ASCII
output
> > > files of MET whose columns are whitespace delimited. Having
> > > whitespace in the variable names would throw off the number of
> > > columns. So I renamed "End of day" to
> > > "EndOfDay":
> > > ncrename -v 'End of day',EndOfDay eodhours.nc
eodhours_rename.nc
> > >
> > > (2) Looks like you're using -999 for missing data, so I added
the
> > > _FillValue attribute.
> > >
> > > (3) Also, MET needs to interpret this as a NetCDF file following
the
> > > CF convention, so I added that as a global attribute:
> > >
> > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > Conventions,global,a,c,"CF-1.0" \
> > > eodhours_rename.nc eodhours_mod.nc
> > >
> > > You're using METv4.1, but that version doesn't include support
for
> > > NetCDF-CF files. So this file won't work in METv4.1.
> > >
> > > So here's what I did...
> > >
> > > I ran the modified data file through the development
"pcp_combine"
> > > tool to create a flavor of NetCDF that METv4.1 can actually
read. I
> > > plotted it using this command:
> > >
> > > METv4.1/bin/plot_data_plane \
> > > eodhours_met.nc eodhours_met.ps \
> > > 'name="EndOfDay"; level="(*,*)";'
> > >
> > > Both the NetCDF file (eodhours_met.nc) and the resulting image
> > > (eodhour_met.png) are attached.
> > >
> > > Try the following to use this as a mask in Grid-Stat. This
defines
> > > 2 different masks, one for eod = 24 and another for eod = 12:
> > >
> > > mask = {
> > > grid = [];
> > > poly = [
> > > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > >
> > > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > >
> > > I agree, that's awfully convoluted!
> > >
> > > Hope that helps.
> > >
> > > John
> > >
> > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > > WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Can you resend please? It must still be out in cyberspace.
This
> > > > morning I got all of yesterday's mail. Note I was here all
day
> > > > yesterday... it's always something.
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > I received an email from you on August 25 with a NetCDF file
> > attachment.
> > > > Then I replied to you on August 25th in an email with 2
> > > > attachments ( eodhours_met.nc and eodhours_met.ps) on August
25.
> > > > Did you receive my reply on 8/25?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > > > >
> > > > > Hey John,
> > > > >
> > > > > Did you receive the NetCDF file I sent? I put it as an
> > > > > attachment to an e-mail. AFWA E-mail has been really messed
up
> > > > > here as of late so it's possible that it never went out, or
that
> > > > > if you replied I haven't received it. I just wanted to
check.
> > > > >
> > > > > Matt
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Matt,
> > > > >
> > > > > Are you able to send me that end of day NetCDF file?
Probably
> > > > > easiest for met to get it correct here and then let you
know.
> > > > > Also, is this
> > > > > METv4.1 or met-5.0?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > > > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > > > > > Queue: met_help
> > > > > > Subject: Grid Stat Config File Question
> > > > > > Owner: Nobody
> > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm trying to figure out the right way to set the config
file
> > > > > > for grid_stat to do the following:
> > > > > >
> > > > > > -- generate a mask for my data based on the contents of an
> > > > > > external file, in NetCDF format
> > > > > > -- keying this mask off of a NetCDF variable named "End of
day"
> > > > > > within the .nc file
> > > > > > -- Keep only those grid points where "End of day" equals
24
> > > > > >
> > > > > > I can't seem to find the right combination of equals,
> > > > > > semicolons, etc., to do this.
> > > > > >
> > > > > > Right now I have:
> > > > > >
> > > > > > mask = {
> > > > > > grid = [
> > > > > > name =
"/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > > > field = [
> > > > > > {
> > > > > > name="End of day";
> > > > > > cat_thresh=[==24];
> > > > > > }
> > > > > > ];
> > > > > > ];
> > > > > > poly = [];
> > > > > > };
> > > > > >
> > > > > > The header from the NetCDF file in question:
> > > > > >
> > > > > > netcdf eodhours {
> > > > > > dimensions:
> > > > > > latitude = 360 ;
> > > > > > longitude = 720 ;
> > > > > > variables:
> > > > > > float latitude(latitude) ;
> > > > > > latitude:units = "degrees_north" ;
> > > > > > float longitude(longitude) ;
> > > > > > longitude:units = "degrees_east" ;
> > > > > > float End of day(latitude, longitude) ;
> > > > > > End of day:units = "hours" ; }
> > > > > >
> > > > > > The mask syntax I used is incorrect.
> > > > > >
> > > > > > For future MET user guides, the following paragraph could
> > > > > > really use an example to make this clear:
> > > > > >
> > > > > > "Alternatively, any gridded data file that MET can read
may be
> > > > > > used to define a verification masking region. Users must
> > > > > > specify a description of the field to be used from the
input
> > > > > > file and, optionally, may specify a threshold to be
applied to
> that field.
> > > > > > Any grid point where the resulting field is 0, the mask is
> > > > > > turned
> > > off.
> > > > > > Any grid point where it is non-zero, the mask is turned
on."
> > > > > >
> > > > > > Can you please help me get the syntax correct?
> > > > > >
> > > > > > Thanks,
> > > > > > Matt
> > > > > >
> > > > > > // SIGNED //
> > > > > > Matthew C. Sittel
> > > > > > University Corporation for Atmospheric Research
> > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
matthew.sittel.ctr at us.af.mil
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Mon Aug 31 14:24:17 2015
Hey John,
Yes, I'm pointing to the right file, the one you created, and verified
that was the file that was an input for plot_data_plane.
As for the log, I don't get very much when I change from -v 2 to -v 4:
DEBUG 1: Default Config File:
/home/qcteam/METv4.1/data/config/GridStatConfig_default
DEBUG 1: User Config File:
/h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_Gb1".
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcMet"
But I tried something else and discovered the problem. Even though
I'm explicitly stating which config file to use, grid_stat is using
the default. I determined this by changing the default file name and
the code abruptly ended saying:
DEBUG 1: Default Config File:
/home/qcteam/METv4.1/data/config/GridStatConfig_default
DEBUG 1: User Config File:
/h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
ERROR :
ERROR : MetConfig::read(const char *) -> unable to open input file
"/home/qcteam/METv4.1/data/config/GridStatConfig_default"
ERROR :
I must be missing something. If MET *says* I have a user-defined
config file why wouldn't grid_stat read it? Or did MET read it, not
like the contents, and go back to read the default?
I checked the permissions on the file, and they match, and belong to
the same user and group as the default.
Thanks,
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, August 31, 2015 2:47 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
If you're able to successfully make an image with that plot_data_plane
command I sent, then MET is able to read that data file fine.
However, you're still receiving the very unhelpful error message:
"NetCDF: Attribute not found"
Seems to me there are a couple possible reasons...
(1) In your Grid-Stat configuration file in the "mask->poly" section,
are you positive that you're pointing to the updated mask file I sent
you?
Please make sure the path is correct and you're pointing to the file
on which you ran that plot_data_plane command.
(2) Another possibility is that you're getting this error message from
Grid-Stat when trying to read some other gridded data file... like the
forecast or observation files.
If you're still having trouble, please rerun your grid_stat command
using these options:
-v 4 -log run_grid_stat.log
That'll write the output from grid_stat to a log file named
"run_grid_stat.log".
Please send me that log file.
Thanks,
John
On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hey John,
>
> Just checking to make sure we don't have e-mail problems here again.
> Below is the last response I got from you.
>
> The modelers are breathing down my neck wanting their data so I
wanted
> to check.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, August 28, 2015 12:16 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Try running the convert command to reformat from postscript to png:
> convert -rotate 90 eodhours_met.ps eodhours_met.png
>
> And then view that .png file either through a web browser or using
the
> display command:
> display eodhours_met.png
>
> In your settings it looks like you have a space between '=' and
'24'.
> That probably won't parse correctly... try removing that space:
>
> mask = {
> grid = [];
> poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
> \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
>
> On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > It did generate a .ps file that is non-zero in size. Now if I can
> > remember how we view .ps files on our devel system I'll be all
set?
> >
> > Just to double check, here's what I have in the config file for
'mask':
> >
> > mask = {
> > grid = [];
> > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name
=
> > \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, August 28, 2015 11:46 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > Please try the following...
> >
> > (1) Retrieve this file:
> >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_m
> > et
> > .nc
> > (2) Run METv4.1 to try to read this file:
> > METv4.1/bin/plot_data_plane eodhours_met.nc eodhours_met.ps
> > 'name="EndOfDay"; level="(*,*)";'
> >
> > Does that successfully create an output PostScript file as it does
> > on my machine? If so, using it in the Grid-Stat configuration
file
> > should work fine.
> >
> > As for defining masks in Grid-Stat, as you know, there are 2
options:
> > grid and poly...
> > - Basically, "grid" handles named grid like "G212" for NCEP Grid
212
> > or "G001" for NCEP grid 1. "FULL" is a special case for grid
where
> > we just use all points in the input grid.
> > - And "poly" handles everything else... like reading an ASCII
> > lat/lon polyline file or reading gridded data files and optionally
> > applying a threshold to them.
> >
> > Were we to do it over again, we might split that out by using
"poly"
> > to specify ASCII lat/lon polyline files and adding "data" to
specify
> > gridded data files. I suppose we didn't do that because we wanted
> > to read the lat/lon polyline file or the NetCDF output of the
> > gen_poly_mask tool at the same place. Oh well.
> >
> > John
> >
> > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hi John,
> > >
> > > Thank you for re-sending the reply and the files.
> > >
> > > I didn’t generate this NetCDF file, so I can get the author to
> > > make this easier should I need to replicate this product. I was
> > > wondering about the whitespace.
> > >
> > > Bob Craig is getting the approval to install MET 5.0 on our new
> > > development server. We have to go through a lot of people to
get
> > > that approved. So if a 5.1 release is imminent, we should
> > > probably wait so we don't have to go through all of this twice
> > > just to update. AFWA frowns on introducing any sort of external
> > > software onto their systems, which makes it harder for us.
> > >
> > > So I tried your fix and I get an error message of:
> > >
> > > "NetCDF: Attribute not found".
> > >
> > > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, August 28, 2015 10:49 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > Question
> > >
> > > Email from 8/25...
> > >
> > > Matt,
> > >
> > > Good news and bad news.
> > >
> > > The good news is that I'm able to get MET to read this file, but
> > > the bad news is that it's with the development version of MET,
not
> > > even a released version!
> > >
> > > But I did modify your NetCDF file first in three ways:
> > >
> > > (1) MET does not support variable names with embedded
whitespace.
> > > Those variable names can ultimately be written to the ASCII
output
> > > files of MET whose columns are whitespace delimited. Having
> > > whitespace in the variable names would throw off the number of
> > > columns. So I renamed "End of day" to
> > > "EndOfDay":
> > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > eodhours_rename.nc
> > >
> > > (2) Looks like you're using -999 for missing data, so I added
the
> > > _FillValue attribute.
> > >
> > > (3) Also, MET needs to interpret this as a NetCDF file following
> > > the CF convention, so I added that as a global attribute:
> > >
> > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > Conventions,global,a,c,"CF-1.0" \
> > > eodhours_rename.nc eodhours_mod.nc
> > >
> > > You're using METv4.1, but that version doesn't include support
for
> > > NetCDF-CF files. So this file won't work in METv4.1.
> > >
> > > So here's what I did...
> > >
> > > I ran the modified data file through the development
"pcp_combine"
> > > tool to create a flavor of NetCDF that METv4.1 can actually
read.
> > > I plotted it using this command:
> > >
> > > METv4.1/bin/plot_data_plane \
> > > eodhours_met.nc eodhours_met.ps \
> > > 'name="EndOfDay"; level="(*,*)";'
> > >
> > > Both the NetCDF file (eodhours_met.nc) and the resulting image
> > > (eodhour_met.png) are attached.
> > >
> > > Try the following to use this as a mask in Grid-Stat. This
> > > defines
> > > 2 different masks, one for eod = 24 and another for eod = 12:
> > >
> > > mask = {
> > > grid = [];
> > > poly = [
> > > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > >
> > > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > >
> > > I agree, that's awfully convoluted!
> > >
> > > Hope that helps.
> > >
> > > John
> > >
> > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Can you resend please? It must still be out in cyberspace.
> > > > This morning I got all of yesterday's mail. Note I was here
all
> > > > day yesterday... it's always something.
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > I received an email from you on August 25 with a NetCDF file
> > attachment.
> > > > Then I replied to you on August 25th in an email with 2
> > > > attachments ( eodhours_met.nc and eodhours_met.ps) on August
25.
> > > > Did you receive my reply on 8/25?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > >
> > > > >
> > > > > Hey John,
> > > > >
> > > > > Did you receive the NetCDF file I sent? I put it as an
> > > > > attachment to an e-mail. AFWA E-mail has been really messed
> > > > > up here as of late so it's possible that it never went out,
or
> > > > > that if you replied I haven't received it. I just wanted to
check.
> > > > >
> > > > > Matt
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Matt,
> > > > >
> > > > > Are you able to send me that end of day NetCDF file?
Probably
> > > > > easiest for met to get it correct here and then let you
know.
> > > > > Also, is this
> > > > > METv4.1 or met-5.0?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > > > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > > > > > Queue: met_help
> > > > > > Subject: Grid Stat Config File Question
> > > > > > Owner: Nobody
> > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm trying to figure out the right way to set the config
> > > > > > file for grid_stat to do the following:
> > > > > >
> > > > > > -- generate a mask for my data based on the contents of an
> > > > > > external file, in NetCDF format
> > > > > > -- keying this mask off of a NetCDF variable named "End of
day"
> > > > > > within the .nc file
> > > > > > -- Keep only those grid points where "End of day" equals
24
> > > > > >
> > > > > > I can't seem to find the right combination of equals,
> > > > > > semicolons, etc., to do this.
> > > > > >
> > > > > > Right now I have:
> > > > > >
> > > > > > mask = {
> > > > > > grid = [
> > > > > > name =
"/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > > > field = [
> > > > > > {
> > > > > > name="End of day";
> > > > > > cat_thresh=[==24];
> > > > > > }
> > > > > > ];
> > > > > > ];
> > > > > > poly = [];
> > > > > > };
> > > > > >
> > > > > > The header from the NetCDF file in question:
> > > > > >
> > > > > > netcdf eodhours {
> > > > > > dimensions:
> > > > > > latitude = 360 ;
> > > > > > longitude = 720 ;
> > > > > > variables:
> > > > > > float latitude(latitude) ;
> > > > > > latitude:units = "degrees_north" ;
> > > > > > float longitude(longitude) ;
> > > > > > longitude:units = "degrees_east" ;
> > > > > > float End of day(latitude, longitude) ;
> > > > > > End of day:units = "hours" ; }
> > > > > >
> > > > > > The mask syntax I used is incorrect.
> > > > > >
> > > > > > For future MET user guides, the following paragraph could
> > > > > > really use an example to make this clear:
> > > > > >
> > > > > > "Alternatively, any gridded data file that MET can read
may
> > > > > > be used to define a verification masking region. Users
must
> > > > > > specify a description of the field to be used from the
input
> > > > > > file and, optionally, may specify a threshold to be
applied
> > > > > > to
> that field.
> > > > > > Any grid point where the resulting field is 0, the mask is
> > > > > > turned
> > > off.
> > > > > > Any grid point where it is non-zero, the mask is turned
on."
> > > > > >
> > > > > > Can you please help me get the syntax correct?
> > > > > >
> > > > > > Thanks,
> > > > > > Matt
> > > > > >
> > > > > > // SIGNED //
> > > > > > Matthew C. Sittel
> > > > > > University Corporation for Atmospheric Research
> > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > matthew.sittel.ctr at us.af.mil
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Mon Aug 31 14:33:14 2015
Hey John,
Okay, so it looks like that even if I don't use it, that
GridStatConfig_default must be present to even *run* grid_stat? If
so, that negates what I thought I found by changing its file name.
I know when I ran a separate version of grid_stat it did indeed uses
the user-defined config file because it used the modified model name I
placed in the user-defined version.
So what I did list before is still valid; it appears grid_stat reads
my config file and doesn't like something:
DEBUG 1: Default Config File:
/home/qcteam/METv4.1/data/config/GridStatConfig_default
DEBUG 1: User Config File:
/h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_Gb1".
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcMet".
NetCDF: Attribute not found
Recall (and note I am using the name you gave me!):
mask = {
grid = [];
poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
\"EndOfDay\"; level=\"(*,*)\";} =24"];
};
And I'm still confused.
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, August 31, 2015 2:47 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
If you're able to successfully make an image with that plot_data_plane
command I sent, then MET is able to read that data file fine.
However, you're still receiving the very unhelpful error message:
"NetCDF: Attribute not found"
Seems to me there are a couple possible reasons...
(1) In your Grid-Stat configuration file in the "mask->poly" section,
are you positive that you're pointing to the updated mask file I sent
you?
Please make sure the path is correct and you're pointing to the file
on which you ran that plot_data_plane command.
(2) Another possibility is that you're getting this error message from
Grid-Stat when trying to read some other gridded data file... like the
forecast or observation files.
If you're still having trouble, please rerun your grid_stat command
using these options:
-v 4 -log run_grid_stat.log
That'll write the output from grid_stat to a log file named
"run_grid_stat.log".
Please send me that log file.
Thanks,
John
On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hey John,
>
> Just checking to make sure we don't have e-mail problems here again.
> Below is the last response I got from you.
>
> The modelers are breathing down my neck wanting their data so I
wanted
> to check.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, August 28, 2015 12:16 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Try running the convert command to reformat from postscript to png:
> convert -rotate 90 eodhours_met.ps eodhours_met.png
>
> And then view that .png file either through a web browser or using
the
> display command:
> display eodhours_met.png
>
> In your settings it looks like you have a space between '=' and
'24'.
> That probably won't parse correctly... try removing that space:
>
> mask = {
> grid = [];
> poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
> \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
>
> On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > It did generate a .ps file that is non-zero in size. Now if I can
> > remember how we view .ps files on our devel system I'll be all
set?
> >
> > Just to double check, here's what I have in the config file for
'mask':
> >
> > mask = {
> > grid = [];
> > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name
=
> > \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, August 28, 2015 11:46 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > Please try the following...
> >
> > (1) Retrieve this file:
> >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_m
> > et
> > .nc
> > (2) Run METv4.1 to try to read this file:
> > METv4.1/bin/plot_data_plane eodhours_met.nc eodhours_met.ps
> > 'name="EndOfDay"; level="(*,*)";'
> >
> > Does that successfully create an output PostScript file as it does
> > on my machine? If so, using it in the Grid-Stat configuration
file
> > should work fine.
> >
> > As for defining masks in Grid-Stat, as you know, there are 2
options:
> > grid and poly...
> > - Basically, "grid" handles named grid like "G212" for NCEP Grid
212
> > or "G001" for NCEP grid 1. "FULL" is a special case for grid
where
> > we just use all points in the input grid.
> > - And "poly" handles everything else... like reading an ASCII
> > lat/lon polyline file or reading gridded data files and optionally
> > applying a threshold to them.
> >
> > Were we to do it over again, we might split that out by using
"poly"
> > to specify ASCII lat/lon polyline files and adding "data" to
specify
> > gridded data files. I suppose we didn't do that because we wanted
> > to read the lat/lon polyline file or the NetCDF output of the
> > gen_poly_mask tool at the same place. Oh well.
> >
> > John
> >
> > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hi John,
> > >
> > > Thank you for re-sending the reply and the files.
> > >
> > > I didn’t generate this NetCDF file, so I can get the author to
> > > make this easier should I need to replicate this product. I was
> > > wondering about the whitespace.
> > >
> > > Bob Craig is getting the approval to install MET 5.0 on our new
> > > development server. We have to go through a lot of people to
get
> > > that approved. So if a 5.1 release is imminent, we should
> > > probably wait so we don't have to go through all of this twice
> > > just to update. AFWA frowns on introducing any sort of external
> > > software onto their systems, which makes it harder for us.
> > >
> > > So I tried your fix and I get an error message of:
> > >
> > > "NetCDF: Attribute not found".
> > >
> > > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, August 28, 2015 10:49 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > Question
> > >
> > > Email from 8/25...
> > >
> > > Matt,
> > >
> > > Good news and bad news.
> > >
> > > The good news is that I'm able to get MET to read this file, but
> > > the bad news is that it's with the development version of MET,
not
> > > even a released version!
> > >
> > > But I did modify your NetCDF file first in three ways:
> > >
> > > (1) MET does not support variable names with embedded
whitespace.
> > > Those variable names can ultimately be written to the ASCII
output
> > > files of MET whose columns are whitespace delimited. Having
> > > whitespace in the variable names would throw off the number of
> > > columns. So I renamed "End of day" to
> > > "EndOfDay":
> > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > eodhours_rename.nc
> > >
> > > (2) Looks like you're using -999 for missing data, so I added
the
> > > _FillValue attribute.
> > >
> > > (3) Also, MET needs to interpret this as a NetCDF file following
> > > the CF convention, so I added that as a global attribute:
> > >
> > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > Conventions,global,a,c,"CF-1.0" \
> > > eodhours_rename.nc eodhours_mod.nc
> > >
> > > You're using METv4.1, but that version doesn't include support
for
> > > NetCDF-CF files. So this file won't work in METv4.1.
> > >
> > > So here's what I did...
> > >
> > > I ran the modified data file through the development
"pcp_combine"
> > > tool to create a flavor of NetCDF that METv4.1 can actually
read.
> > > I plotted it using this command:
> > >
> > > METv4.1/bin/plot_data_plane \
> > > eodhours_met.nc eodhours_met.ps \
> > > 'name="EndOfDay"; level="(*,*)";'
> > >
> > > Both the NetCDF file (eodhours_met.nc) and the resulting image
> > > (eodhour_met.png) are attached.
> > >
> > > Try the following to use this as a mask in Grid-Stat. This
> > > defines
> > > 2 different masks, one for eod = 24 and another for eod = 12:
> > >
> > > mask = {
> > > grid = [];
> > > poly = [
> > > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > >
> > > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > >
> > > I agree, that's awfully convoluted!
> > >
> > > Hope that helps.
> > >
> > > John
> > >
> > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Can you resend please? It must still be out in cyberspace.
> > > > This morning I got all of yesterday's mail. Note I was here
all
> > > > day yesterday... it's always something.
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > I received an email from you on August 25 with a NetCDF file
> > attachment.
> > > > Then I replied to you on August 25th in an email with 2
> > > > attachments ( eodhours_met.nc and eodhours_met.ps) on August
25.
> > > > Did you receive my reply on 8/25?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > >
> > > > >
> > > > > Hey John,
> > > > >
> > > > > Did you receive the NetCDF file I sent? I put it as an
> > > > > attachment to an e-mail. AFWA E-mail has been really messed
> > > > > up here as of late so it's possible that it never went out,
or
> > > > > that if you replied I haven't received it. I just wanted to
check.
> > > > >
> > > > > Matt
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Matt,
> > > > >
> > > > > Are you able to send me that end of day NetCDF file?
Probably
> > > > > easiest for met to get it correct here and then let you
know.
> > > > > Also, is this
> > > > > METv4.1 or met-5.0?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > > > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > > > > > Queue: met_help
> > > > > > Subject: Grid Stat Config File Question
> > > > > > Owner: Nobody
> > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm trying to figure out the right way to set the config
> > > > > > file for grid_stat to do the following:
> > > > > >
> > > > > > -- generate a mask for my data based on the contents of an
> > > > > > external file, in NetCDF format
> > > > > > -- keying this mask off of a NetCDF variable named "End of
day"
> > > > > > within the .nc file
> > > > > > -- Keep only those grid points where "End of day" equals
24
> > > > > >
> > > > > > I can't seem to find the right combination of equals,
> > > > > > semicolons, etc., to do this.
> > > > > >
> > > > > > Right now I have:
> > > > > >
> > > > > > mask = {
> > > > > > grid = [
> > > > > > name =
"/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > > > field = [
> > > > > > {
> > > > > > name="End of day";
> > > > > > cat_thresh=[==24];
> > > > > > }
> > > > > > ];
> > > > > > ];
> > > > > > poly = [];
> > > > > > };
> > > > > >
> > > > > > The header from the NetCDF file in question:
> > > > > >
> > > > > > netcdf eodhours {
> > > > > > dimensions:
> > > > > > latitude = 360 ;
> > > > > > longitude = 720 ;
> > > > > > variables:
> > > > > > float latitude(latitude) ;
> > > > > > latitude:units = "degrees_north" ;
> > > > > > float longitude(longitude) ;
> > > > > > longitude:units = "degrees_east" ;
> > > > > > float End of day(latitude, longitude) ;
> > > > > > End of day:units = "hours" ; }
> > > > > >
> > > > > > The mask syntax I used is incorrect.
> > > > > >
> > > > > > For future MET user guides, the following paragraph could
> > > > > > really use an example to make this clear:
> > > > > >
> > > > > > "Alternatively, any gridded data file that MET can read
may
> > > > > > be used to define a verification masking region. Users
must
> > > > > > specify a description of the field to be used from the
input
> > > > > > file and, optionally, may specify a threshold to be
applied
> > > > > > to
> that field.
> > > > > > Any grid point where the resulting field is 0, the mask is
> > > > > > turned
> > > off.
> > > > > > Any grid point where it is non-zero, the mask is turned
on."
> > > > > >
> > > > > > Can you please help me get the syntax correct?
> > > > > >
> > > > > > Thanks,
> > > > > > Matt
> > > > > >
> > > > > > // SIGNED //
> > > > > > Matthew C. Sittel
> > > > > > University Corporation for Atmospheric Research
> > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > matthew.sittel.ctr at us.af.mil
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Mon Aug 31 17:15:03 2015
Matt,
Hmm, sorry no, nothing else is jumping out at me. If I were able to
replicate the error you're getting, it'd probably jump out at me
really
quickly.
If you're able to post data somewhere I could grab it, I'd be happy to
try
to reproduce the error here. I'd need the grid_stat config file, the
forecast, and observation files, the mask file, and the exact command
you're running on the command line.
The reading of the config files is working as it should. MET reads
the
ConfigConstants file, followed by the default config file for that
tool,
followed by the user-specified config file. That's how we designed it
to
work.
Thanks,
John
On Mon, Aug 31, 2015 at 2:33 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hey John,
>
> Okay, so it looks like that even if I don't use it, that
> GridStatConfig_default must be present to even *run* grid_stat? If
so,
> that negates what I thought I found by changing its file name.
>
> I know when I ran a separate version of grid_stat it did indeed uses
the
> user-defined config file because it used the modified model name I
placed
> in the user-defined version.
>
> So what I did list before is still valid; it appears grid_stat reads
my
> config file and doesn't like something:
>
> DEBUG 1: Default Config File:
> /home/qcteam/METv4.1/data/config/GridStatConfig_default
> DEBUG 1: User Config File:
> /h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_Gb1".
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> NetCDF: Attribute not found
>
> Recall (and note I am using the name you gave me!):
>
> mask = {
> grid = [];
> poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
> \"EndOfDay\"; level=\"(*,*)\";} =24"];
> };
>
> And I'm still confused.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, August 31, 2015 2:47 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> If you're able to successfully make an image with that
plot_data_plane
> command I sent, then MET is able to read that data file fine.
>
> However, you're still receiving the very unhelpful error message:
> "NetCDF: Attribute not found"
>
> Seems to me there are a couple possible reasons...
>
> (1) In your Grid-Stat configuration file in the "mask->poly"
section, are
> you positive that you're pointing to the updated mask file I sent
you?
> Please make sure the path is correct and you're pointing to the file
on
> which you ran that plot_data_plane command.
>
> (2) Another possibility is that you're getting this error message
from
> Grid-Stat when trying to read some other gridded data file... like
the
> forecast or observation files.
>
> If you're still having trouble, please rerun your grid_stat command
using
> these options:
> -v 4 -log run_grid_stat.log
>
> That'll write the output from grid_stat to a log file named
> "run_grid_stat.log".
>
> Please send me that log file.
>
> Thanks,
> John
>
> On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hey John,
> >
> > Just checking to make sure we don't have e-mail problems here
again.
> > Below is the last response I got from you.
> >
> > The modelers are breathing down my neck wanting their data so I
wanted
> > to check.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, August 28, 2015 12:16 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > Try running the convert command to reformat from postscript to
png:
> > convert -rotate 90 eodhours_met.ps eodhours_met.png
> >
> > And then view that .png file either through a web browser or using
the
> > display command:
> > display eodhours_met.png
> >
> > In your settings it looks like you have a space between '=' and
'24'.
> > That probably won't parse correctly... try removing that space:
> >
> > mask = {
> > grid = [];
> > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name
=
> > \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> >
> > On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > It did generate a .ps file that is non-zero in size. Now if I
can
> > > remember how we view .ps files on our devel system I'll be all
set?
> > >
> > > Just to double check, here's what I have in the config file for
'mask':
> > >
> > > mask = {
> > > grid = [];
> > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
{name =
> > > \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, August 28, 2015 11:46 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> > >
> > > Matt,
> > >
> > > Please try the following...
> > >
> > > (1) Retrieve this file:
> > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours_m
> > > et
> > > .nc
> > > (2) Run METv4.1 to try to read this file:
> > > METv4.1/bin/plot_data_plane eodhours_met.nc eodhours_met.ps
> > > 'name="EndOfDay"; level="(*,*)";'
> > >
> > > Does that successfully create an output PostScript file as it
does
> > > on my machine? If so, using it in the Grid-Stat configuration
file
> > > should work fine.
> > >
> > > As for defining masks in Grid-Stat, as you know, there are 2
options:
> > > grid and poly...
> > > - Basically, "grid" handles named grid like "G212" for NCEP Grid
212
> > > or "G001" for NCEP grid 1. "FULL" is a special case for grid
where
> > > we just use all points in the input grid.
> > > - And "poly" handles everything else... like reading an ASCII
> > > lat/lon polyline file or reading gridded data files and
optionally
> > > applying a threshold to them.
> > >
> > > Were we to do it over again, we might split that out by using
"poly"
> > > to specify ASCII lat/lon polyline files and adding "data" to
specify
> > > gridded data files. I suppose we didn't do that because we
wanted
> > > to read the lat/lon polyline file or the NetCDF output of the
> > > gen_poly_mask tool at the same place. Oh well.
> > >
> > > John
> > >
> > > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF
AFWA 16
> > > WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Hi John,
> > > >
> > > > Thank you for re-sending the reply and the files.
> > > >
> > > > I didn’t generate this NetCDF file, so I can get the author to
> > > > make this easier should I need to replicate this product. I
was
> > > > wondering about the whitespace.
> > > >
> > > > Bob Craig is getting the approval to install MET 5.0 on our
new
> > > > development server. We have to go through a lot of people to
get
> > > > that approved. So if a 5.1 release is imminent, we should
> > > > probably wait so we don't have to go through all of this twice
> > > > just to update. AFWA frowns on introducing any sort of
external
> > > > software onto their systems, which makes it harder for us.
> > > >
> > > > So I tried your fix and I get an error message of:
> > > >
> > > > "NetCDF: Attribute not found".
> > > >
> > > > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, August 28, 2015 10:49 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Email from 8/25...
> > > >
> > > > Matt,
> > > >
> > > > Good news and bad news.
> > > >
> > > > The good news is that I'm able to get MET to read this file,
but
> > > > the bad news is that it's with the development version of MET,
not
> > > > even a released version!
> > > >
> > > > But I did modify your NetCDF file first in three ways:
> > > >
> > > > (1) MET does not support variable names with embedded
whitespace.
> > > > Those variable names can ultimately be written to the ASCII
output
> > > > files of MET whose columns are whitespace delimited. Having
> > > > whitespace in the variable names would throw off the number of
> > > > columns. So I renamed "End of day" to
> > > > "EndOfDay":
> > > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > > eodhours_rename.nc
> > > >
> > > > (2) Looks like you're using -999 for missing data, so I added
the
> > > > _FillValue attribute.
> > > >
> > > > (3) Also, MET needs to interpret this as a NetCDF file
following
> > > > the CF convention, so I added that as a global attribute:
> > > >
> > > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > > Conventions,global,a,c,"CF-1.0" \
> > > > eodhours_rename.nc eodhours_mod.nc
> > > >
> > > > You're using METv4.1, but that version doesn't include support
for
> > > > NetCDF-CF files. So this file won't work in METv4.1.
> > > >
> > > > So here's what I did...
> > > >
> > > > I ran the modified data file through the development
"pcp_combine"
> > > > tool to create a flavor of NetCDF that METv4.1 can actually
read.
> > > > I plotted it using this command:
> > > >
> > > > METv4.1/bin/plot_data_plane \
> > > > eodhours_met.nc eodhours_met.ps \
> > > > 'name="EndOfDay"; level="(*,*)";'
> > > >
> > > > Both the NetCDF file (eodhours_met.nc) and the resulting image
> > > > (eodhour_met.png) are attached.
> > > >
> > > > Try the following to use this as a mask in Grid-Stat. This
> > > > defines
> > > > 2 different masks, one for eod = 24 and another for eod = 12:
> > > >
> > > > mask = {
> > > > grid = [];
> > > > poly = [
> > > > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > > >
> > > > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > > >
> > > > I agree, that's awfully convoluted!
> > > >
> > > > Hope that helps.
> > > >
> > > > John
> > > >
> > > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > > > >
> > > > > Can you resend please? It must still be out in cyberspace.
> > > > > This morning I got all of yesterday's mail. Note I was here
all
> > > > > day yesterday... it's always something.
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Matt,
> > > > >
> > > > > I received an email from you on August 25 with a NetCDF file
> > > attachment.
> > > > > Then I replied to you on August 25th in an email with 2
> > > > > attachments ( eodhours_met.nc and eodhours_met.ps) on August
25.
> > > > > Did you receive my reply on 8/25?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > > Hey John,
> > > > > >
> > > > > > Did you receive the NetCDF file I sent? I put it as an
> > > > > > attachment to an e-mail. AFWA E-mail has been really
messed
> > > > > > up here as of late so it's possible that it never went
out, or
> > > > > > that if you replied I haven't received it. I just wanted
to
> check.
> > > > > >
> > > > > > Matt
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > Question
> > > > > >
> > > > > > Matt,
> > > > > >
> > > > > > Are you able to send me that end of day NetCDF file?
Probably
> > > > > > easiest for met to get it correct here and then let you
know.
> > > > > > Also, is this
> > > > > > METv4.1 or met-5.0?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > AFWA
> > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > > > > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > > > > > > Queue: met_help
> > > > > > > Subject: Grid Stat Config File Question
> > > > > > > Owner: Nobody
> > > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm trying to figure out the right way to set the config
> > > > > > > file for grid_stat to do the following:
> > > > > > >
> > > > > > > -- generate a mask for my data based on the contents of
an
> > > > > > > external file, in NetCDF format
> > > > > > > -- keying this mask off of a NetCDF variable named "End
of day"
> > > > > > > within the .nc file
> > > > > > > -- Keep only those grid points where "End of day" equals
24
> > > > > > >
> > > > > > > I can't seem to find the right combination of equals,
> > > > > > > semicolons, etc., to do this.
> > > > > > >
> > > > > > > Right now I have:
> > > > > > >
> > > > > > > mask = {
> > > > > > > grid = [
> > > > > > > name =
"/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > > > > field = [
> > > > > > > {
> > > > > > > name="End of day";
> > > > > > > cat_thresh=[==24];
> > > > > > > }
> > > > > > > ];
> > > > > > > ];
> > > > > > > poly = [];
> > > > > > > };
> > > > > > >
> > > > > > > The header from the NetCDF file in question:
> > > > > > >
> > > > > > > netcdf eodhours {
> > > > > > > dimensions:
> > > > > > > latitude = 360 ;
> > > > > > > longitude = 720 ;
> > > > > > > variables:
> > > > > > > float latitude(latitude) ;
> > > > > > > latitude:units = "degrees_north" ;
> > > > > > > float longitude(longitude) ;
> > > > > > > longitude:units = "degrees_east" ;
> > > > > > > float End of day(latitude, longitude) ;
> > > > > > > End of day:units = "hours" ; }
> > > > > > >
> > > > > > > The mask syntax I used is incorrect.
> > > > > > >
> > > > > > > For future MET user guides, the following paragraph
could
> > > > > > > really use an example to make this clear:
> > > > > > >
> > > > > > > "Alternatively, any gridded data file that MET can read
may
> > > > > > > be used to define a verification masking region. Users
must
> > > > > > > specify a description of the field to be used from the
input
> > > > > > > file and, optionally, may specify a threshold to be
applied
> > > > > > > to
> > that field.
> > > > > > > Any grid point where the resulting field is 0, the mask
is
> > > > > > > turned
> > > > off.
> > > > > > > Any grid point where it is non-zero, the mask is turned
on."
> > > > > > >
> > > > > > > Can you please help me get the syntax correct?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Matt
> > > > > > >
> > > > > > > // SIGNED //
> > > > > > > Matthew C. Sittel
> > > > > > > University Corporation for Atmospheric Research
> > > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > > matthew.sittel.ctr at us.af.mil
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Tue Sep 01 06:56:36 2015
Hey John,
Something finally occurred to me on the way home yesterday regarding
this problem. And I think I've found the issue.
The NetCDF file I'm reading in for observation data is NOT one MET
generates.
We are comparing two different cases, and in the first case I'm
summing up one-hour NCEP Stage IV data using 'pcp_combine'. So that
output format is correct since MET generated it.
The other one I'm using "as-is" (half-degree CPC analyses), and it's
already NetCDF and summed to 24 hours. But... you guessed it, it's
the same NetCDF format as that "EndOfDay" file.
So the solution is for me to get the modeler to reformat all the
observational data he generates, or reformat it myself using the
outline below. The problem is 'ncatted' doesn't appear to work on our
system. So I will pursue getting the modeler to change the format.
Can you help me explain to him what he needs to do to change the
NetCDF-CF format into something MET can read? Rather than converting
it as you did, is there something he can do at the creation phase to
write it out properly?
Also, what missing value does MET expect? '-9999.' Possibly? I'll
get him to change that too.
Thanks,
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, August 31, 2015 6:15 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
Hmm, sorry no, nothing else is jumping out at me. If I were able to
replicate the error you're getting, it'd probably jump out at me
really quickly.
If you're able to post data somewhere I could grab it, I'd be happy to
try to reproduce the error here. I'd need the grid_stat config file,
the forecast, and observation files, the mask file, and the exact
command you're running on the command line.
The reading of the config files is working as it should. MET reads
the ConfigConstants file, followed by the default config file for that
tool, followed by the user-specified config file. That's how we
designed it to work.
Thanks,
John
On Mon, Aug 31, 2015 at 2:33 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hey John,
>
> Okay, so it looks like that even if I don't use it, that
> GridStatConfig_default must be present to even *run* grid_stat? If
> so, that negates what I thought I found by changing its file name.
>
> I know when I ran a separate version of grid_stat it did indeed uses
> the user-defined config file because it used the modified model name
I
> placed in the user-defined version.
>
> So what I did list before is still valid; it appears grid_stat reads
> my config file and doesn't like something:
>
> DEBUG 1: Default Config File:
> /home/qcteam/METv4.1/data/config/GridStatConfig_default
> DEBUG 1: User Config File:
> /h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_Gb1".
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> NetCDF: Attribute not found
>
> Recall (and note I am using the name you gave me!):
>
> mask = {
> grid = [];
> poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name =
> \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
>
> And I'm still confused.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, August 31, 2015 2:47 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> If you're able to successfully make an image with that
plot_data_plane
> command I sent, then MET is able to read that data file fine.
>
> However, you're still receiving the very unhelpful error message:
> "NetCDF: Attribute not found"
>
> Seems to me there are a couple possible reasons...
>
> (1) In your Grid-Stat configuration file in the "mask->poly"
section,
> are you positive that you're pointing to the updated mask file I
sent you?
> Please make sure the path is correct and you're pointing to the file
> on which you ran that plot_data_plane command.
>
> (2) Another possibility is that you're getting this error message
from
> Grid-Stat when trying to read some other gridded data file... like
the
> forecast or observation files.
>
> If you're still having trouble, please rerun your grid_stat command
> using these options:
> -v 4 -log run_grid_stat.log
>
> That'll write the output from grid_stat to a log file named
> "run_grid_stat.log".
>
> Please send me that log file.
>
> Thanks,
> John
>
> On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hey John,
> >
> > Just checking to make sure we don't have e-mail problems here
again.
> > Below is the last response I got from you.
> >
> > The modelers are breathing down my neck wanting their data so I
> > wanted to check.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, August 28, 2015 12:16 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > Try running the convert command to reformat from postscript to
png:
> > convert -rotate 90 eodhours_met.ps eodhours_met.png
> >
> > And then view that .png file either through a web browser or using
> > the display command:
> > display eodhours_met.png
> >
> > In your settings it looks like you have a space between '=' and
'24'.
> > That probably won't parse correctly... try removing that space:
> >
> > mask = {
> > grid = [];
> > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name
=
> > \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> >
> > On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > It did generate a .ps file that is non-zero in size. Now if I
can
> > > remember how we view .ps files on our devel system I'll be all
set?
> > >
> > > Just to double check, here's what I have in the config file for
'mask':
> > >
> > > mask = {
> > > grid = [];
> > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
{name
> > > = \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, August 28, 2015 11:46 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > Question
> > >
> > > Matt,
> > >
> > > Please try the following...
> > >
> > > (1) Retrieve this file:
> > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours
> > > _m
> > > et
> > > .nc
> > > (2) Run METv4.1 to try to read this file:
> > > METv4.1/bin/plot_data_plane eodhours_met.nc eodhours_met.ps
> > > 'name="EndOfDay"; level="(*,*)";'
> > >
> > > Does that successfully create an output PostScript file as it
does
> > > on my machine? If so, using it in the Grid-Stat configuration
> > > file should work fine.
> > >
> > > As for defining masks in Grid-Stat, as you know, there are 2
options:
> > > grid and poly...
> > > - Basically, "grid" handles named grid like "G212" for NCEP Grid
> > > 212 or "G001" for NCEP grid 1. "FULL" is a special case for
grid
> > > where we just use all points in the input grid.
> > > - And "poly" handles everything else... like reading an ASCII
> > > lat/lon polyline file or reading gridded data files and
optionally
> > > applying a threshold to them.
> > >
> > > Were we to do it over again, we might split that out by using
"poly"
> > > to specify ASCII lat/lon polyline files and adding "data" to
> > > specify gridded data files. I suppose we didn't do that because
> > > we wanted to read the lat/lon polyline file or the NetCDF output
> > > of the gen_poly_mask tool at the same place. Oh well.
> > >
> > > John
> > >
> > > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Hi John,
> > > >
> > > > Thank you for re-sending the reply and the files.
> > > >
> > > > I didn’t generate this NetCDF file, so I can get the author to
> > > > make this easier should I need to replicate this product. I
was
> > > > wondering about the whitespace.
> > > >
> > > > Bob Craig is getting the approval to install MET 5.0 on our
new
> > > > development server. We have to go through a lot of people to
> > > > get that approved. So if a 5.1 release is imminent, we should
> > > > probably wait so we don't have to go through all of this twice
> > > > just to update. AFWA frowns on introducing any sort of
external
> > > > software onto their systems, which makes it harder for us.
> > > >
> > > > So I tried your fix and I get an error message of:
> > > >
> > > > "NetCDF: Attribute not found".
> > > >
> > > > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, August 28, 2015 10:49 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Email from 8/25...
> > > >
> > > > Matt,
> > > >
> > > > Good news and bad news.
> > > >
> > > > The good news is that I'm able to get MET to read this file,
but
> > > > the bad news is that it's with the development version of MET,
> > > > not even a released version!
> > > >
> > > > But I did modify your NetCDF file first in three ways:
> > > >
> > > > (1) MET does not support variable names with embedded
whitespace.
> > > > Those variable names can ultimately be written to the ASCII
> > > > output files of MET whose columns are whitespace delimited.
> > > > Having whitespace in the variable names would throw off the
> > > > number of columns. So I renamed "End of day" to
> > > > "EndOfDay":
> > > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > > eodhours_rename.nc
> > > >
> > > > (2) Looks like you're using -999 for missing data, so I added
> > > > the _FillValue attribute.
> > > >
> > > > (3) Also, MET needs to interpret this as a NetCDF file
following
> > > > the CF convention, so I added that as a global attribute:
> > > >
> > > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > > Conventions,global,a,c,"CF-1.0" \
> > > > eodhours_rename.nc eodhours_mod.nc
> > > >
> > > > You're using METv4.1, but that version doesn't include support
> > > > for NetCDF-CF files. So this file won't work in METv4.1.
> > > >
> > > > So here's what I did...
> > > >
> > > > I ran the modified data file through the development
"pcp_combine"
> > > > tool to create a flavor of NetCDF that METv4.1 can actually
read.
> > > > I plotted it using this command:
> > > >
> > > > METv4.1/bin/plot_data_plane \
> > > > eodhours_met.nc eodhours_met.ps \
> > > > 'name="EndOfDay"; level="(*,*)";'
> > > >
> > > > Both the NetCDF file (eodhours_met.nc) and the resulting image
> > > > (eodhour_met.png) are attached.
> > > >
> > > > Try the following to use this as a mask in Grid-Stat. This
> > > > defines
> > > > 2 different masks, one for eod = 24 and another for eod = 12:
> > > >
> > > > mask = {
> > > > grid = [];
> > > > poly = [
> > > > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > > >
> > > > "/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > > >
> > > > I agree, that's awfully convoluted!
> > > >
> > > > Hope that helps.
> > > >
> > > > John
> > > >
> > > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > >
> > > > >
> > > > > Can you resend please? It must still be out in cyberspace.
> > > > > This morning I got all of yesterday's mail. Note I was here
> > > > > all day yesterday... it's always something.
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Matt,
> > > > >
> > > > > I received an email from you on August 25 with a NetCDF file
> > > attachment.
> > > > > Then I replied to you on August 25th in an email with 2
> > > > > attachments ( eodhours_met.nc and eodhours_met.ps) on August
25.
> > > > > Did you receive my reply on 8/25?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > > Hey John,
> > > > > >
> > > > > > Did you receive the NetCDF file I sent? I put it as an
> > > > > > attachment to an e-mail. AFWA E-mail has been really
messed
> > > > > > up here as of late so it's possible that it never went
out,
> > > > > > or that if you replied I haven't received it. I just
wanted
> > > > > > to
> check.
> > > > > >
> > > > > > Matt
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > Question
> > > > > >
> > > > > > Matt,
> > > > > >
> > > > > > Are you able to send me that end of day NetCDF file?
> > > > > > Probably easiest for met to get it correct here and then
let you know.
> > > > > > Also, is this
> > > > > > METv4.1 or met-5.0?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > AFWA
> > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted upon.
> > > > > > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > > > > > > Queue: met_help
> > > > > > > Subject: Grid Stat Config File Question
> > > > > > > Owner: Nobody
> > > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm trying to figure out the right way to set the config
> > > > > > > file for grid_stat to do the following:
> > > > > > >
> > > > > > > -- generate a mask for my data based on the contents of
an
> > > > > > > external file, in NetCDF format
> > > > > > > -- keying this mask off of a NetCDF variable named "End
of day"
> > > > > > > within the .nc file
> > > > > > > -- Keep only those grid points where "End of day" equals
> > > > > > > 24
> > > > > > >
> > > > > > > I can't seem to find the right combination of equals,
> > > > > > > semicolons, etc., to do this.
> > > > > > >
> > > > > > > Right now I have:
> > > > > > >
> > > > > > > mask = {
> > > > > > > grid = [
> > > > > > > name =
"/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > > > > field = [
> > > > > > > {
> > > > > > > name="End of day";
> > > > > > > cat_thresh=[==24];
> > > > > > > }
> > > > > > > ];
> > > > > > > ];
> > > > > > > poly = [];
> > > > > > > };
> > > > > > >
> > > > > > > The header from the NetCDF file in question:
> > > > > > >
> > > > > > > netcdf eodhours {
> > > > > > > dimensions:
> > > > > > > latitude = 360 ;
> > > > > > > longitude = 720 ;
> > > > > > > variables:
> > > > > > > float latitude(latitude) ;
> > > > > > > latitude:units = "degrees_north" ;
> > > > > > > float longitude(longitude) ;
> > > > > > > longitude:units = "degrees_east" ;
> > > > > > > float End of day(latitude, longitude) ;
> > > > > > > End of day:units = "hours" ; }
> > > > > > >
> > > > > > > The mask syntax I used is incorrect.
> > > > > > >
> > > > > > > For future MET user guides, the following paragraph
could
> > > > > > > really use an example to make this clear:
> > > > > > >
> > > > > > > "Alternatively, any gridded data file that MET can read
> > > > > > > may be used to define a verification masking region.
Users
> > > > > > > must specify a description of the field to be used from
> > > > > > > the input file and, optionally, may specify a threshold
to
> > > > > > > be applied to
> > that field.
> > > > > > > Any grid point where the resulting field is 0, the mask
is
> > > > > > > turned
> > > > off.
> > > > > > > Any grid point where it is non-zero, the mask is turned
on."
> > > > > > >
> > > > > > > Can you please help me get the syntax correct?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Matt
> > > > > > >
> > > > > > > // SIGNED //
> > > > > > > Matthew C. Sittel
> > > > > > > University Corporation for Atmospheric Research
> > > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > > matthew.sittel.ctr at us.af.mil
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Tue Sep 01 10:06:19 2015
Matt,
METv4.1 does not include support for CF-compliant NetCDF. If you
continue
using METv4.1, you'll need to make this data look like the NetCDF
output of
the pcp_combine tool.
Another option would be finding the CPC analysis data in a different
format
that may be easier to read into MET. Is there something specific
about the
version of the CPC data you're using or could you use it from a
different
data source?
Thanks,
John
On Tue, Sep 1, 2015 at 6:56 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hey John,
>
> Something finally occurred to me on the way home yesterday regarding
this
> problem. And I think I've found the issue.
>
> The NetCDF file I'm reading in for observation data is NOT one MET
> generates.
>
> We are comparing two different cases, and in the first case I'm
summing up
> one-hour NCEP Stage IV data using 'pcp_combine'. So that output
format is
> correct since MET generated it.
>
> The other one I'm using "as-is" (half-degree CPC analyses), and it's
> already NetCDF and summed to 24 hours. But... you guessed it, it's
the
> same NetCDF format as that "EndOfDay" file.
>
> So the solution is for me to get the modeler to reformat all the
> observational data he generates, or reformat it myself using the
outline
> below. The problem is 'ncatted' doesn't appear to work on our
system. So
> I will pursue getting the modeler to change the format.
>
> Can you help me explain to him what he needs to do to change the
NetCDF-CF
> format into something MET can read? Rather than converting it as
you did,
> is there something he can do at the creation phase to write it out
properly?
>
> Also, what missing value does MET expect? '-9999.' Possibly? I'll
get
> him to change that too.
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, August 31, 2015 6:15 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Hmm, sorry no, nothing else is jumping out at me. If I were able to
> replicate the error you're getting, it'd probably jump out at me
really
> quickly.
>
> If you're able to post data somewhere I could grab it, I'd be happy
to try
> to reproduce the error here. I'd need the grid_stat config file,
the
> forecast, and observation files, the mask file, and the exact
command
> you're running on the command line.
>
> The reading of the config files is working as it should. MET reads
the
> ConfigConstants file, followed by the default config file for that
tool,
> followed by the user-specified config file. That's how we designed
it to
> work.
>
> Thanks,
> John
>
> On Mon, Aug 31, 2015 at 2:33 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hey John,
> >
> > Okay, so it looks like that even if I don't use it, that
> > GridStatConfig_default must be present to even *run* grid_stat?
If
> > so, that negates what I thought I found by changing its file name.
> >
> > I know when I ran a separate version of grid_stat it did indeed
uses
> > the user-defined config file because it used the modified model
name I
> > placed in the user-defined version.
> >
> > So what I did list before is still valid; it appears grid_stat
reads
> > my config file and doesn't like something:
> >
> > DEBUG 1: Default Config File:
> > /home/qcteam/METv4.1/data/config/GridStatConfig_default
> > DEBUG 1: User Config File:
> > /h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_Gb1".
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > NetCDF: Attribute not found
> >
> > Recall (and note I am using the name you gave me!):
> >
> > mask = {
> > grid = [];
> > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name
=
> > \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> >
> > And I'm still confused.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Monday, August 31, 2015 2:47 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > If you're able to successfully make an image with that
plot_data_plane
> > command I sent, then MET is able to read that data file fine.
> >
> > However, you're still receiving the very unhelpful error message:
> > "NetCDF: Attribute not found"
> >
> > Seems to me there are a couple possible reasons...
> >
> > (1) In your Grid-Stat configuration file in the "mask->poly"
section,
> > are you positive that you're pointing to the updated mask file I
sent
> you?
> > Please make sure the path is correct and you're pointing to the
file
> > on which you ran that plot_data_plane command.
> >
> > (2) Another possibility is that you're getting this error message
from
> > Grid-Stat when trying to read some other gridded data file... like
the
> > forecast or observation files.
> >
> > If you're still having trouble, please rerun your grid_stat
command
> > using these options:
> > -v 4 -log run_grid_stat.log
> >
> > That'll write the output from grid_stat to a log file named
> > "run_grid_stat.log".
> >
> > Please send me that log file.
> >
> > Thanks,
> > John
> >
> > On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hey John,
> > >
> > > Just checking to make sure we don't have e-mail problems here
again.
> > > Below is the last response I got from you.
> > >
> > > The modelers are breathing down my neck wanting their data so I
> > > wanted to check.
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, August 28, 2015 12:16 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> > >
> > > Matt,
> > >
> > > Try running the convert command to reformat from postscript to
png:
> > > convert -rotate 90 eodhours_met.ps eodhours_met.png
> > >
> > > And then view that .png file either through a web browser or
using
> > > the display command:
> > > display eodhours_met.png
> > >
> > > In your settings it looks like you have a space between '=' and
'24'.
> > > That probably won't parse correctly... try removing that space:
> > >
> > > mask = {
> > > grid = [];
> > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
{name =
> > > \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > >
> > > On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF
AFWA 16
> > > WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > It did generate a .ps file that is non-zero in size. Now if I
can
> > > > remember how we view .ps files on our devel system I'll be all
set?
> > > >
> > > > Just to double check, here's what I have in the config file
for
> 'mask':
> > > >
> > > > mask = {
> > > > grid = [];
> > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
{name
> > > > = \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, August 28, 2015 11:46 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > Please try the following...
> > > >
> > > > (1) Retrieve this file:
> > > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhours
> > > > _m
> > > > et
> > > > .nc
> > > > (2) Run METv4.1 to try to read this file:
> > > > METv4.1/bin/plot_data_plane eodhours_met.nc
eodhours_met.ps
> > > > 'name="EndOfDay"; level="(*,*)";'
> > > >
> > > > Does that successfully create an output PostScript file as it
does
> > > > on my machine? If so, using it in the Grid-Stat configuration
> > > > file should work fine.
> > > >
> > > > As for defining masks in Grid-Stat, as you know, there are 2
options:
> > > > grid and poly...
> > > > - Basically, "grid" handles named grid like "G212" for NCEP
Grid
> > > > 212 or "G001" for NCEP grid 1. "FULL" is a special case for
grid
> > > > where we just use all points in the input grid.
> > > > - And "poly" handles everything else... like reading an ASCII
> > > > lat/lon polyline file or reading gridded data files and
optionally
> > > > applying a threshold to them.
> > > >
> > > > Were we to do it over again, we might split that out by using
"poly"
> > > > to specify ASCII lat/lon polyline files and adding "data" to
> > > > specify gridded data files. I suppose we didn't do that
because
> > > > we wanted to read the lat/lon polyline file or the NetCDF
output
> > > > of the gen_poly_mask tool at the same place. Oh well.
> > > >
> > > > John
> > > >
> > > > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Thank you for re-sending the reply and the files.
> > > > >
> > > > > I didn’t generate this NetCDF file, so I can get the author
to
> > > > > make this easier should I need to replicate this product. I
was
> > > > > wondering about the whitespace.
> > > > >
> > > > > Bob Craig is getting the approval to install MET 5.0 on our
new
> > > > > development server. We have to go through a lot of people
to
> > > > > get that approved. So if a 5.1 release is imminent, we
should
> > > > > probably wait so we don't have to go through all of this
twice
> > > > > just to update. AFWA frowns on introducing any sort of
external
> > > > > software onto their systems, which makes it harder for us.
> > > > >
> > > > > So I tried your fix and I get an error message of:
> > > > >
> > > > > "NetCDF: Attribute not found".
> > > > >
> > > > > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> > > > >
> > > > > Matt
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Friday, August 28, 2015 10:49 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Email from 8/25...
> > > > >
> > > > > Matt,
> > > > >
> > > > > Good news and bad news.
> > > > >
> > > > > The good news is that I'm able to get MET to read this file,
but
> > > > > the bad news is that it's with the development version of
MET,
> > > > > not even a released version!
> > > > >
> > > > > But I did modify your NetCDF file first in three ways:
> > > > >
> > > > > (1) MET does not support variable names with embedded
whitespace.
> > > > > Those variable names can ultimately be written to the ASCII
> > > > > output files of MET whose columns are whitespace delimited.
> > > > > Having whitespace in the variable names would throw off the
> > > > > number of columns. So I renamed "End of day" to
> > > > > "EndOfDay":
> > > > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > > > eodhours_rename.nc
> > > > >
> > > > > (2) Looks like you're using -999 for missing data, so I
added
> > > > > the _FillValue attribute.
> > > > >
> > > > > (3) Also, MET needs to interpret this as a NetCDF file
following
> > > > > the CF convention, so I added that as a global attribute:
> > > > >
> > > > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > > > Conventions,global,a,c,"CF-1.0" \
> > > > > eodhours_rename.nc eodhours_mod.nc
> > > > >
> > > > > You're using METv4.1, but that version doesn't include
support
> > > > > for NetCDF-CF files. So this file won't work in METv4.1.
> > > > >
> > > > > So here's what I did...
> > > > >
> > > > > I ran the modified data file through the development
"pcp_combine"
> > > > > tool to create a flavor of NetCDF that METv4.1 can actually
read.
> > > > > I plotted it using this command:
> > > > >
> > > > > METv4.1/bin/plot_data_plane \
> > > > > eodhours_met.nc eodhours_met.ps \
> > > > > 'name="EndOfDay"; level="(*,*)";'
> > > > >
> > > > > Both the NetCDF file (eodhours_met.nc) and the resulting
image
> > > > > (eodhour_met.png) are attached.
> > > > >
> > > > > Try the following to use this as a mask in Grid-Stat. This
> > > > > defines
> > > > > 2 different masks, one for eod = 24 and another for eod =
12:
> > > > >
> > > > > mask = {
> > > > > grid = [];
> > > > > poly = [
> > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > > > >
> > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > > > >
> > > > > I agree, that's awfully convoluted!
> > > > >
> > > > > Hope that helps.
> > > > >
> > > > > John
> > > > >
> > > > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > > Can you resend please? It must still be out in
cyberspace.
> > > > > > This morning I got all of yesterday's mail. Note I was
here
> > > > > > all day yesterday... it's always something.
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > Question
> > > > > >
> > > > > > Matt,
> > > > > >
> > > > > > I received an email from you on August 25 with a NetCDF
file
> > > > attachment.
> > > > > > Then I replied to you on August 25th in an email with 2
> > > > > > attachments ( eodhours_met.nc and eodhours_met.ps) on
August 25.
> > > > > > Did you receive my reply on 8/25?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > AFWA
> > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > >
> > > > > > >
> > > > > > > Hey John,
> > > > > > >
> > > > > > > Did you receive the NetCDF file I sent? I put it as an
> > > > > > > attachment to an e-mail. AFWA E-mail has been really
messed
> > > > > > > up here as of late so it's possible that it never went
out,
> > > > > > > or that if you replied I haven't received it. I just
wanted
> > > > > > > to
> > check.
> > > > > > >
> > > > > > > Matt
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > > Question
> > > > > > >
> > > > > > > Matt,
> > > > > > >
> > > > > > > Are you able to send me that end of day NetCDF file?
> > > > > > > Probably easiest for met to get it correct here and then
let
> you know.
> > > > > > > Also, is this
> > > > > > > METv4.1 or met-5.0?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > > AFWA
> > > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted
upon.
> > > > > > > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > > > > > > > Queue: met_help
> > > > > > > > Subject: Grid Stat Config File Question
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I'm trying to figure out the right way to set the
config
> > > > > > > > file for grid_stat to do the following:
> > > > > > > >
> > > > > > > > -- generate a mask for my data based on the contents
of an
> > > > > > > > external file, in NetCDF format
> > > > > > > > -- keying this mask off of a NetCDF variable named
"End of
> day"
> > > > > > > > within the .nc file
> > > > > > > > -- Keep only those grid points where "End of day"
equals
> > > > > > > > 24
> > > > > > > >
> > > > > > > > I can't seem to find the right combination of equals,
> > > > > > > > semicolons, etc., to do this.
> > > > > > > >
> > > > > > > > Right now I have:
> > > > > > > >
> > > > > > > > mask = {
> > > > > > > > grid = [
> > > > > > > > name =
"/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > > > > > field = [
> > > > > > > > {
> > > > > > > > name="End of day";
> > > > > > > > cat_thresh=[==24];
> > > > > > > > }
> > > > > > > > ];
> > > > > > > > ];
> > > > > > > > poly = [];
> > > > > > > > };
> > > > > > > >
> > > > > > > > The header from the NetCDF file in question:
> > > > > > > >
> > > > > > > > netcdf eodhours {
> > > > > > > > dimensions:
> > > > > > > > latitude = 360 ;
> > > > > > > > longitude = 720 ;
> > > > > > > > variables:
> > > > > > > > float latitude(latitude) ;
> > > > > > > > latitude:units = "degrees_north" ;
> > > > > > > > float longitude(longitude) ;
> > > > > > > > longitude:units = "degrees_east" ;
> > > > > > > > float End of day(latitude, longitude) ;
> > > > > > > > End of day:units = "hours" ; }
> > > > > > > >
> > > > > > > > The mask syntax I used is incorrect.
> > > > > > > >
> > > > > > > > For future MET user guides, the following paragraph
could
> > > > > > > > really use an example to make this clear:
> > > > > > > >
> > > > > > > > "Alternatively, any gridded data file that MET can
read
> > > > > > > > may be used to define a verification masking region.
Users
> > > > > > > > must specify a description of the field to be used
from
> > > > > > > > the input file and, optionally, may specify a
threshold to
> > > > > > > > be applied to
> > > that field.
> > > > > > > > Any grid point where the resulting field is 0, the
mask is
> > > > > > > > turned
> > > > > off.
> > > > > > > > Any grid point where it is non-zero, the mask is
turned on."
> > > > > > > >
> > > > > > > > Can you please help me get the syntax correct?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Matt
> > > > > > > >
> > > > > > > > // SIGNED //
> > > > > > > > Matthew C. Sittel
> > > > > > > > University Corporation for Atmospheric Research
> > > > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > > > matthew.sittel.ctr at us.af.mil
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Tue Sep 01 10:15:20 2015
The modeler downloaded a software package off of the web that creates
NetCDF from binary data. So the NetCDF 'construction' is done in
house, by converting that original binary to NetCDF format.
Is there a simple way to determine when a NetCDF file is/isn't CF-
compliant?
If we can download/install NCO here we can use 'ncatted' as you did.
When you used the following command:
ncatted -a _FillValue,EndOfDay,a,f,-999 -a Conventions,global,a,c,"CF-
1.0" eodhours_rename.nc eodhours_mod.nc
Did this change the NetCDF file version from "CF-1.something" to "CF-
1.0", or is that the CF version of the NetCDF input you are trying to
undo?
Bob Craig wants to know when MET 5.1 will be released.
Thanks,
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, September 01, 2015 11:06 AM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
METv4.1 does not include support for CF-compliant NetCDF. If you
continue using METv4.1, you'll need to make this data look like the
NetCDF output of the pcp_combine tool.
Another option would be finding the CPC analysis data in a different
format that may be easier to read into MET. Is there something
specific about the version of the CPC data you're using or could you
use it from a different data source?
Thanks,
John
On Tue, Sep 1, 2015 at 6:56 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hey John,
>
> Something finally occurred to me on the way home yesterday regarding
> this problem. And I think I've found the issue.
>
> The NetCDF file I'm reading in for observation data is NOT one MET
> generates.
>
> We are comparing two different cases, and in the first case I'm
> summing up one-hour NCEP Stage IV data using 'pcp_combine'. So that
> output format is correct since MET generated it.
>
> The other one I'm using "as-is" (half-degree CPC analyses), and it's
> already NetCDF and summed to 24 hours. But... you guessed it, it's
> the same NetCDF format as that "EndOfDay" file.
>
> So the solution is for me to get the modeler to reformat all the
> observational data he generates, or reformat it myself using the
> outline below. The problem is 'ncatted' doesn't appear to work on
our
> system. So I will pursue getting the modeler to change the format.
>
> Can you help me explain to him what he needs to do to change the
> NetCDF-CF format into something MET can read? Rather than
converting
> it as you did, is there something he can do at the creation phase to
write it out properly?
>
> Also, what missing value does MET expect? '-9999.' Possibly? I'll
> get him to change that too.
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, August 31, 2015 6:15 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> Hmm, sorry no, nothing else is jumping out at me. If I were able to
> replicate the error you're getting, it'd probably jump out at me
> really quickly.
>
> If you're able to post data somewhere I could grab it, I'd be happy
to
> try to reproduce the error here. I'd need the grid_stat config
file,
> the forecast, and observation files, the mask file, and the exact
> command you're running on the command line.
>
> The reading of the config files is working as it should. MET reads
> the ConfigConstants file, followed by the default config file for
that
> tool, followed by the user-specified config file. That's how we
> designed it to work.
>
> Thanks,
> John
>
> On Mon, Aug 31, 2015 at 2:33 PM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hey John,
> >
> > Okay, so it looks like that even if I don't use it, that
> > GridStatConfig_default must be present to even *run* grid_stat?
If
> > so, that negates what I thought I found by changing its file name.
> >
> > I know when I ran a separate version of grid_stat it did indeed
uses
> > the user-defined config file because it used the modified model
name
> > I placed in the user-defined version.
> >
> > So what I did list before is still valid; it appears grid_stat
reads
> > my config file and doesn't like something:
> >
> > DEBUG 1: Default Config File:
> > /home/qcteam/METv4.1/data/config/GridStatConfig_default
> > DEBUG 1: User Config File:
> > /h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_Gb1".
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > NetCDF: Attribute not found
> >
> > Recall (and note I am using the name you gave me!):
> >
> > mask = {
> > grid = [];
> > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc {name
=
> > \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> >
> > And I'm still confused.
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Monday, August 31, 2015 2:47 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > If you're able to successfully make an image with that
> > plot_data_plane command I sent, then MET is able to read that data
file fine.
> >
> > However, you're still receiving the very unhelpful error message:
> > "NetCDF: Attribute not found"
> >
> > Seems to me there are a couple possible reasons...
> >
> > (1) In your Grid-Stat configuration file in the "mask->poly"
> > section, are you positive that you're pointing to the updated mask
> > file I sent
> you?
> > Please make sure the path is correct and you're pointing to the
file
> > on which you ran that plot_data_plane command.
> >
> > (2) Another possibility is that you're getting this error message
> > from Grid-Stat when trying to read some other gridded data file...
> > like the forecast or observation files.
> >
> > If you're still having trouble, please rerun your grid_stat
command
> > using these options:
> > -v 4 -log run_grid_stat.log
> >
> > That'll write the output from grid_stat to a log file named
> > "run_grid_stat.log".
> >
> > Please send me that log file.
> >
> > Thanks,
> > John
> >
> > On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hey John,
> > >
> > > Just checking to make sure we don't have e-mail problems here
again.
> > > Below is the last response I got from you.
> > >
> > > The modelers are breathing down my neck wanting their data so I
> > > wanted to check.
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, August 28, 2015 12:16 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > Question
> > >
> > > Matt,
> > >
> > > Try running the convert command to reformat from postscript to
png:
> > > convert -rotate 90 eodhours_met.ps eodhours_met.png
> > >
> > > And then view that .png file either through a web browser or
using
> > > the display command:
> > > display eodhours_met.png
> > >
> > > In your settings it looks like you have a space between '=' and
'24'.
> > > That probably won't parse correctly... try removing that space:
> > >
> > > mask = {
> > > grid = [];
> > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
{name
> > > = \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > >
> > > On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > It did generate a .ps file that is non-zero in size. Now if I
> > > > can remember how we view .ps files on our devel system I'll be
all set?
> > > >
> > > > Just to double check, here's what I have in the config file
for
> 'mask':
> > > >
> > > > mask = {
> > > > grid = [];
> > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > {name = \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, August 28, 2015 11:46 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > Please try the following...
> > > >
> > > > (1) Retrieve this file:
> > > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhou
> > > > rs
> > > > _m
> > > > et
> > > > .nc
> > > > (2) Run METv4.1 to try to read this file:
> > > > METv4.1/bin/plot_data_plane eodhours_met.nc
eodhours_met.ps
> > > > 'name="EndOfDay"; level="(*,*)";'
> > > >
> > > > Does that successfully create an output PostScript file as it
> > > > does on my machine? If so, using it in the Grid-Stat
> > > > configuration file should work fine.
> > > >
> > > > As for defining masks in Grid-Stat, as you know, there are 2
options:
> > > > grid and poly...
> > > > - Basically, "grid" handles named grid like "G212" for NCEP
Grid
> > > > 212 or "G001" for NCEP grid 1. "FULL" is a special case for
> > > > grid where we just use all points in the input grid.
> > > > - And "poly" handles everything else... like reading an ASCII
> > > > lat/lon polyline file or reading gridded data files and
> > > > optionally applying a threshold to them.
> > > >
> > > > Were we to do it over again, we might split that out by using
"poly"
> > > > to specify ASCII lat/lon polyline files and adding "data" to
> > > > specify gridded data files. I suppose we didn't do that
because
> > > > we wanted to read the lat/lon polyline file or the NetCDF
output
> > > > of the gen_poly_mask tool at the same place. Oh well.
> > > >
> > > > John
> > > >
> > > > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF
> > > > AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Thank you for re-sending the reply and the files.
> > > > >
> > > > > I didn’t generate this NetCDF file, so I can get the author
to
> > > > > make this easier should I need to replicate this product. I
> > > > > was wondering about the whitespace.
> > > > >
> > > > > Bob Craig is getting the approval to install MET 5.0 on our
> > > > > new development server. We have to go through a lot of
people
> > > > > to get that approved. So if a 5.1 release is imminent, we
> > > > > should probably wait so we don't have to go through all of
> > > > > this twice just to update. AFWA frowns on introducing any
> > > > > sort of external software onto their systems, which makes it
harder for us.
> > > > >
> > > > > So I tried your fix and I get an error message of:
> > > > >
> > > > > "NetCDF: Attribute not found".
> > > > >
> > > > > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> > > > >
> > > > > Matt
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Friday, August 28, 2015 10:49 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Email from 8/25...
> > > > >
> > > > > Matt,
> > > > >
> > > > > Good news and bad news.
> > > > >
> > > > > The good news is that I'm able to get MET to read this file,
> > > > > but the bad news is that it's with the development version
of
> > > > > MET, not even a released version!
> > > > >
> > > > > But I did modify your NetCDF file first in three ways:
> > > > >
> > > > > (1) MET does not support variable names with embedded
whitespace.
> > > > > Those variable names can ultimately be written to the ASCII
> > > > > output files of MET whose columns are whitespace delimited.
> > > > > Having whitespace in the variable names would throw off the
> > > > > number of columns. So I renamed "End of day" to
> > > > > "EndOfDay":
> > > > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > > > eodhours_rename.nc
> > > > >
> > > > > (2) Looks like you're using -999 for missing data, so I
added
> > > > > the _FillValue attribute.
> > > > >
> > > > > (3) Also, MET needs to interpret this as a NetCDF file
> > > > > following the CF convention, so I added that as a global
attribute:
> > > > >
> > > > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > > > Conventions,global,a,c,"CF-1.0" \
> > > > > eodhours_rename.nc eodhours_mod.nc
> > > > >
> > > > > You're using METv4.1, but that version doesn't include
support
> > > > > for NetCDF-CF files. So this file won't work in METv4.1.
> > > > >
> > > > > So here's what I did...
> > > > >
> > > > > I ran the modified data file through the development
"pcp_combine"
> > > > > tool to create a flavor of NetCDF that METv4.1 can actually
read.
> > > > > I plotted it using this command:
> > > > >
> > > > > METv4.1/bin/plot_data_plane \
> > > > > eodhours_met.nc eodhours_met.ps \
> > > > > 'name="EndOfDay"; level="(*,*)";'
> > > > >
> > > > > Both the NetCDF file (eodhours_met.nc) and the resulting
image
> > > > > (eodhour_met.png) are attached.
> > > > >
> > > > > Try the following to use this as a mask in Grid-Stat. This
> > > > > defines
> > > > > 2 different masks, one for eod = 24 and another for eod =
12:
> > > > >
> > > > > mask = {
> > > > > grid = [];
> > > > > poly = [
> > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > > > >
> > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > > > >
> > > > > I agree, that's awfully convoluted!
> > > > >
> > > > > Hope that helps.
> > > > >
> > > > > John
> > > > >
> > > > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > > Can you resend please? It must still be out in
cyberspace.
> > > > > > This morning I got all of yesterday's mail. Note I was
here
> > > > > > all day yesterday... it's always something.
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > Question
> > > > > >
> > > > > > Matt,
> > > > > >
> > > > > > I received an email from you on August 25 with a NetCDF
file
> > > > attachment.
> > > > > > Then I replied to you on August 25th in an email with 2
> > > > > > attachments ( eodhours_met.nc and eodhours_met.ps) on
August 25.
> > > > > > Did you receive my reply on 8/25?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > AFWA
> > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > >
> > > > > > >
> > > > > > > Hey John,
> > > > > > >
> > > > > > > Did you receive the NetCDF file I sent? I put it as an
> > > > > > > attachment to an e-mail. AFWA E-mail has been really
> > > > > > > messed up here as of late so it's possible that it never
> > > > > > > went out, or that if you replied I haven't received it.
I
> > > > > > > just wanted to
> > check.
> > > > > > >
> > > > > > > Matt
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
> > > > > > > File Question
> > > > > > >
> > > > > > > Matt,
> > > > > > >
> > > > > > > Are you able to send me that end of day NetCDF file?
> > > > > > > Probably easiest for met to get it correct here and then
> > > > > > > let
> you know.
> > > > > > > Also, is this
> > > > > > > METv4.1 or met-5.0?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C CTR
> > > > > > > USAF AFWA
> > > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted
upon.
> > > > > > > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > > > > > > > Queue: met_help
> > > > > > > > Subject: Grid Stat Config File Question
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I'm trying to figure out the right way to set the
config
> > > > > > > > file for grid_stat to do the following:
> > > > > > > >
> > > > > > > > -- generate a mask for my data based on the contents
of
> > > > > > > > an external file, in NetCDF format
> > > > > > > > -- keying this mask off of a NetCDF variable named
"End
> > > > > > > > of
> day"
> > > > > > > > within the .nc file
> > > > > > > > -- Keep only those grid points where "End of day"
equals
> > > > > > > > 24
> > > > > > > >
> > > > > > > > I can't seem to find the right combination of equals,
> > > > > > > > semicolons, etc., to do this.
> > > > > > > >
> > > > > > > > Right now I have:
> > > > > > > >
> > > > > > > > mask = {
> > > > > > > > grid = [
> > > > > > > > name =
"/h/data/global/WXQC/data/met/cfg/eodhours.nc";
> > > > > > > > field = [
> > > > > > > > {
> > > > > > > > name="End of day";
> > > > > > > > cat_thresh=[==24];
> > > > > > > > }
> > > > > > > > ];
> > > > > > > > ];
> > > > > > > > poly = [];
> > > > > > > > };
> > > > > > > >
> > > > > > > > The header from the NetCDF file in question:
> > > > > > > >
> > > > > > > > netcdf eodhours {
> > > > > > > > dimensions:
> > > > > > > > latitude = 360 ;
> > > > > > > > longitude = 720 ;
> > > > > > > > variables:
> > > > > > > > float latitude(latitude) ;
> > > > > > > > latitude:units = "degrees_north" ;
> > > > > > > > float longitude(longitude) ;
> > > > > > > > longitude:units = "degrees_east" ;
> > > > > > > > float End of day(latitude, longitude) ;
> > > > > > > > End of day:units = "hours" ; }
> > > > > > > >
> > > > > > > > The mask syntax I used is incorrect.
> > > > > > > >
> > > > > > > > For future MET user guides, the following paragraph
> > > > > > > > could really use an example to make this clear:
> > > > > > > >
> > > > > > > > "Alternatively, any gridded data file that MET can
read
> > > > > > > > may be used to define a verification masking region.
> > > > > > > > Users must specify a description of the field to be
used
> > > > > > > > from the input file and, optionally, may specify a
> > > > > > > > threshold to be applied to
> > > that field.
> > > > > > > > Any grid point where the resulting field is 0, the
mask
> > > > > > > > is turned
> > > > > off.
> > > > > > > > Any grid point where it is non-zero, the mask is
turned on."
> > > > > > > >
> > > > > > > > Can you please help me get the syntax correct?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Matt
> > > > > > > >
> > > > > > > > // SIGNED //
> > > > > > > > Matthew C. Sittel
> > > > > > > > University Corporation for Atmospheric Research
> > > > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > > > matthew.sittel.ctr at us.af.mil
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Tue Sep 01 11:03:06 2015
Matt,
I asked around here, and we've used the CPC analysis data in the past.
In
particular, we used on a project comparing GFS precipitation to NAM
precipitation. Part of that work was preparing the binary CPC data
for use
in MET.
To do so, we ran a FORTRAN utility called grbrfc8, which is available
in
this directory:
http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/sorc/nam_cpc2grbrfc8.fd
It writes a GRIB output file which MET can easily handle. It looks
like
this is set up to read 1/8 degree CPC data, and you mention that you
have
1/2 degree data. So I'm not sure if this would help.
However, we no longer use the CPC analysis data for verification, and
I'm
told that people are generally moving away from it and using StageII,
StageIV, or CCPA data instead.
Things would be easiest if we could get the data into GRIB format.
Short
of that, we'd need to make it look like the NetCDF output of
pcp_combine to
use it in METv4.1.
We expect version 5.1 to be available by the end of September. We've
been
instructed to add support for computing anomaly correlation in the
same way
NCEP is doing it which is pushing development out a little longer.
John
On Tue, Sep 1, 2015 at 10:15 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> The modeler downloaded a software package off of the web that
creates
> NetCDF from binary data. So the NetCDF 'construction' is done in
house, by
> converting that original binary to NetCDF format.
>
> Is there a simple way to determine when a NetCDF file is/isn't
> CF-compliant?
>
> If we can download/install NCO here we can use 'ncatted' as you did.
When
> you used the following command:
>
> ncatted -a _FillValue,EndOfDay,a,f,-999 -a
Conventions,global,a,c,"CF-1.0"
> eodhours_rename.nc eodhours_mod.nc
>
> Did this change the NetCDF file version from "CF-1.something" to
"CF-1.0",
> or is that the CF version of the NetCDF input you are trying to
undo?
>
> Bob Craig wants to know when MET 5.1 will be released.
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, September 01, 2015 11:06 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> METv4.1 does not include support for CF-compliant NetCDF. If you
continue
> using METv4.1, you'll need to make this data look like the NetCDF
output of
> the pcp_combine tool.
>
> Another option would be finding the CPC analysis data in a different
> format that may be easier to read into MET. Is there something
specific
> about the version of the CPC data you're using or could you use it
from a
> different data source?
>
> Thanks,
> John
>
> On Tue, Sep 1, 2015 at 6:56 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hey John,
> >
> > Something finally occurred to me on the way home yesterday
regarding
> > this problem. And I think I've found the issue.
> >
> > The NetCDF file I'm reading in for observation data is NOT one
MET
> > generates.
> >
> > We are comparing two different cases, and in the first case I'm
> > summing up one-hour NCEP Stage IV data using 'pcp_combine'. So
that
> > output format is correct since MET generated it.
> >
> > The other one I'm using "as-is" (half-degree CPC analyses), and
it's
> > already NetCDF and summed to 24 hours. But... you guessed it,
it's
> > the same NetCDF format as that "EndOfDay" file.
> >
> > So the solution is for me to get the modeler to reformat all the
> > observational data he generates, or reformat it myself using the
> > outline below. The problem is 'ncatted' doesn't appear to work on
our
> > system. So I will pursue getting the modeler to change the
format.
> >
> > Can you help me explain to him what he needs to do to change the
> > NetCDF-CF format into something MET can read? Rather than
converting
> > it as you did, is there something he can do at the creation phase
to
> write it out properly?
> >
> > Also, what missing value does MET expect? '-9999.' Possibly?
I'll
> > get him to change that too.
> >
> > Thanks,
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Monday, August 31, 2015 6:15 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > Hmm, sorry no, nothing else is jumping out at me. If I were able
to
> > replicate the error you're getting, it'd probably jump out at me
> > really quickly.
> >
> > If you're able to post data somewhere I could grab it, I'd be
happy to
> > try to reproduce the error here. I'd need the grid_stat config
file,
> > the forecast, and observation files, the mask file, and the exact
> > command you're running on the command line.
> >
> > The reading of the config files is working as it should. MET
reads
> > the ConfigConstants file, followed by the default config file for
that
> > tool, followed by the user-specified config file. That's how we
> > designed it to work.
> >
> > Thanks,
> > John
> >
> > On Mon, Aug 31, 2015 at 2:33 PM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hey John,
> > >
> > > Okay, so it looks like that even if I don't use it, that
> > > GridStatConfig_default must be present to even *run* grid_stat?
If
> > > so, that negates what I thought I found by changing its file
name.
> > >
> > > I know when I ran a separate version of grid_stat it did indeed
uses
> > > the user-defined config file because it used the modified model
name
> > > I placed in the user-defined version.
> > >
> > > So what I did list before is still valid; it appears grid_stat
reads
> > > my config file and doesn't like something:
> > >
> > > DEBUG 1: Default Config File:
> > > /home/qcteam/METv4.1/data/config/GridStatConfig_default
> > > DEBUG 1: User Config File:
> > > /h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_Gb1".
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_NcMet".
> > > NetCDF: Attribute not found
> > >
> > > Recall (and note I am using the name you gave me!):
> > >
> > > mask = {
> > > grid = [];
> > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
{name =
> > > \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > >
> > > And I'm still confused.
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Monday, August 31, 2015 2:47 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> > >
> > > Matt,
> > >
> > > If you're able to successfully make an image with that
> > > plot_data_plane command I sent, then MET is able to read that
data
> file fine.
> > >
> > > However, you're still receiving the very unhelpful error
message:
> > > "NetCDF: Attribute not found"
> > >
> > > Seems to me there are a couple possible reasons...
> > >
> > > (1) In your Grid-Stat configuration file in the "mask->poly"
> > > section, are you positive that you're pointing to the updated
mask
> > > file I sent
> > you?
> > > Please make sure the path is correct and you're pointing to the
file
> > > on which you ran that plot_data_plane command.
> > >
> > > (2) Another possibility is that you're getting this error
message
> > > from Grid-Stat when trying to read some other gridded data
file...
> > > like the forecast or observation files.
> > >
> > > If you're still having trouble, please rerun your grid_stat
command
> > > using these options:
> > > -v 4 -log run_grid_stat.log
> > >
> > > That'll write the output from grid_stat to a log file named
> > > "run_grid_stat.log".
> > >
> > > Please send me that log file.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > > WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Hey John,
> > > >
> > > > Just checking to make sure we don't have e-mail problems here
again.
> > > > Below is the last response I got from you.
> > > >
> > > > The modelers are breathing down my neck wanting their data so
I
> > > > wanted to check.
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, August 28, 2015 12:16 PM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > Try running the convert command to reformat from postscript to
png:
> > > > convert -rotate 90 eodhours_met.ps eodhours_met.png
> > > >
> > > > And then view that .png file either through a web browser or
using
> > > > the display command:
> > > > display eodhours_met.png
> > > >
> > > > In your settings it looks like you have a space between '='
and '24'.
> > > > That probably won't parse correctly... try removing that
space:
> > > >
> > > > mask = {
> > > > grid = [];
> > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
{name
> > > > = \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > > >
> > > > On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > > > >
> > > > > It did generate a .ps file that is non-zero in size. Now if
I
> > > > > can remember how we view .ps files on our devel system I'll
be all
> set?
> > > > >
> > > > > Just to double check, here's what I have in the config file
for
> > 'mask':
> > > > >
> > > > > mask = {
> > > > > grid = [];
> > > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > > {name = \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Friday, August 28, 2015 11:46 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Matt,
> > > > >
> > > > > Please try the following...
> > > > >
> > > > > (1) Retrieve this file:
> > > > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodhou
> > > > > rs
> > > > > _m
> > > > > et
> > > > > .nc
> > > > > (2) Run METv4.1 to try to read this file:
> > > > > METv4.1/bin/plot_data_plane eodhours_met.nc
eodhours_met.ps
> > > > > 'name="EndOfDay"; level="(*,*)";'
> > > > >
> > > > > Does that successfully create an output PostScript file as
it
> > > > > does on my machine? If so, using it in the Grid-Stat
> > > > > configuration file should work fine.
> > > > >
> > > > > As for defining masks in Grid-Stat, as you know, there are 2
> options:
> > > > > grid and poly...
> > > > > - Basically, "grid" handles named grid like "G212" for NCEP
Grid
> > > > > 212 or "G001" for NCEP grid 1. "FULL" is a special case for
> > > > > grid where we just use all points in the input grid.
> > > > > - And "poly" handles everything else... like reading an
ASCII
> > > > > lat/lon polyline file or reading gridded data files and
> > > > > optionally applying a threshold to them.
> > > > >
> > > > > Were we to do it over again, we might split that out by
using
> "poly"
> > > > > to specify ASCII lat/lon polyline files and adding "data" to
> > > > > specify gridded data files. I suppose we didn't do that
because
> > > > > we wanted to read the lat/lon polyline file or the NetCDF
output
> > > > > of the gen_poly_mask tool at the same place. Oh well.
> > > > >
> > > > > John
> > > > >
> > > > > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Thank you for re-sending the reply and the files.
> > > > > >
> > > > > > I didn’t generate this NetCDF file, so I can get the
author to
> > > > > > make this easier should I need to replicate this product.
I
> > > > > > was wondering about the whitespace.
> > > > > >
> > > > > > Bob Craig is getting the approval to install MET 5.0 on
our
> > > > > > new development server. We have to go through a lot of
people
> > > > > > to get that approved. So if a 5.1 release is imminent, we
> > > > > > should probably wait so we don't have to go through all of
> > > > > > this twice just to update. AFWA frowns on introducing any
> > > > > > sort of external software onto their systems, which makes
it
> harder for us.
> > > > > >
> > > > > > So I tried your fix and I get an error message of:
> > > > > >
> > > > > > "NetCDF: Attribute not found".
> > > > > >
> > > > > > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> > > > > >
> > > > > > Matt
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Friday, August 28, 2015 10:49 AM
> > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > Question
> > > > > >
> > > > > > Email from 8/25...
> > > > > >
> > > > > > Matt,
> > > > > >
> > > > > > Good news and bad news.
> > > > > >
> > > > > > The good news is that I'm able to get MET to read this
file,
> > > > > > but the bad news is that it's with the development version
of
> > > > > > MET, not even a released version!
> > > > > >
> > > > > > But I did modify your NetCDF file first in three ways:
> > > > > >
> > > > > > (1) MET does not support variable names with embedded
whitespace.
> > > > > > Those variable names can ultimately be written to the
ASCII
> > > > > > output files of MET whose columns are whitespace
delimited.
> > > > > > Having whitespace in the variable names would throw off
the
> > > > > > number of columns. So I renamed "End of day" to
> > > > > > "EndOfDay":
> > > > > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > > > > eodhours_rename.nc
> > > > > >
> > > > > > (2) Looks like you're using -999 for missing data, so I
added
> > > > > > the _FillValue attribute.
> > > > > >
> > > > > > (3) Also, MET needs to interpret this as a NetCDF file
> > > > > > following the CF convention, so I added that as a global
> attribute:
> > > > > >
> > > > > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > > > > Conventions,global,a,c,"CF-1.0" \
> > > > > > eodhours_rename.nc eodhours_mod.nc
> > > > > >
> > > > > > You're using METv4.1, but that version doesn't include
support
> > > > > > for NetCDF-CF files. So this file won't work in METv4.1.
> > > > > >
> > > > > > So here's what I did...
> > > > > >
> > > > > > I ran the modified data file through the development
> "pcp_combine"
> > > > > > tool to create a flavor of NetCDF that METv4.1 can
actually read.
> > > > > > I plotted it using this command:
> > > > > >
> > > > > > METv4.1/bin/plot_data_plane \
> > > > > > eodhours_met.nc eodhours_met.ps \
> > > > > > 'name="EndOfDay"; level="(*,*)";'
> > > > > >
> > > > > > Both the NetCDF file (eodhours_met.nc) and the resulting
image
> > > > > > (eodhour_met.png) are attached.
> > > > > >
> > > > > > Try the following to use this as a mask in Grid-Stat.
This
> > > > > > defines
> > > > > > 2 different masks, one for eod = 24 and another for eod =
12:
> > > > > >
> > > > > > mask = {
> > > > > > grid = [];
> > > > > > poly = [
> > > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > > > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > > > > >
> > > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.nc
> > > > > > { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > > > > >
> > > > > > I agree, that's awfully convoluted!
> > > > > >
> > > > > > Hope that helps.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > AFWA
> > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > >
> > > > > > >
> > > > > > > Can you resend please? It must still be out in
cyberspace.
> > > > > > > This morning I got all of yesterday's mail. Note I was
here
> > > > > > > all day yesterday... it's always something.
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > > Question
> > > > > > >
> > > > > > > Matt,
> > > > > > >
> > > > > > > I received an email from you on August 25 with a NetCDF
file
> > > > > attachment.
> > > > > > > Then I replied to you on August 25th in an email with 2
> > > > > > > attachments ( eodhours_met.nc and eodhours_met.ps) on
August
> 25.
> > > > > > > Did you receive my reply on 8/25?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > > AFWA
> > > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hey John,
> > > > > > > >
> > > > > > > > Did you receive the NetCDF file I sent? I put it as
an
> > > > > > > > attachment to an e-mail. AFWA E-mail has been really
> > > > > > > > messed up here as of late so it's possible that it
never
> > > > > > > > went out, or that if you replied I haven't received
it. I
> > > > > > > > just wanted to
> > > check.
> > > > > > > >
> > > > > > > > Matt
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
> > > > > > > > File Question
> > > > > > > >
> > > > > > > > Matt,
> > > > > > > >
> > > > > > > > Are you able to send me that end of day NetCDF file?
> > > > > > > > Probably easiest for met to get it correct here and
then
> > > > > > > > let
> > you know.
> > > > > > > > Also, is this
> > > > > > > > METv4.1 or met-5.0?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C
CTR
> > > > > > > > USAF AFWA
> > > > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted
upon.
> > > > > > > > > Transaction: Ticket created by
> matthew.sittel.ctr at us.af.mil
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: Grid Stat Config File Question
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I'm trying to figure out the right way to set the
config
> > > > > > > > > file for grid_stat to do the following:
> > > > > > > > >
> > > > > > > > > -- generate a mask for my data based on the contents
of
> > > > > > > > > an external file, in NetCDF format
> > > > > > > > > -- keying this mask off of a NetCDF variable named
"End
> > > > > > > > > of
> > day"
> > > > > > > > > within the .nc file
> > > > > > > > > -- Keep only those grid points where "End of day"
equals
> > > > > > > > > 24
> > > > > > > > >
> > > > > > > > > I can't seem to find the right combination of
equals,
> > > > > > > > > semicolons, etc., to do this.
> > > > > > > > >
> > > > > > > > > Right now I have:
> > > > > > > > >
> > > > > > > > > mask = {
> > > > > > > > > grid = [
> > > > > > > > > name =
"/h/data/global/WXQC/data/met/cfg/eodhours.nc
> ";
> > > > > > > > > field = [
> > > > > > > > > {
> > > > > > > > > name="End of day";
> > > > > > > > > cat_thresh=[==24];
> > > > > > > > > }
> > > > > > > > > ];
> > > > > > > > > ];
> > > > > > > > > poly = [];
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > > The header from the NetCDF file in question:
> > > > > > > > >
> > > > > > > > > netcdf eodhours {
> > > > > > > > > dimensions:
> > > > > > > > > latitude = 360 ;
> > > > > > > > > longitude = 720 ;
> > > > > > > > > variables:
> > > > > > > > > float latitude(latitude) ;
> > > > > > > > > latitude:units = "degrees_north" ;
> > > > > > > > > float longitude(longitude) ;
> > > > > > > > > longitude:units = "degrees_east" ;
> > > > > > > > > float End of day(latitude, longitude) ;
> > > > > > > > > End of day:units = "hours" ; }
> > > > > > > > >
> > > > > > > > > The mask syntax I used is incorrect.
> > > > > > > > >
> > > > > > > > > For future MET user guides, the following paragraph
> > > > > > > > > could really use an example to make this clear:
> > > > > > > > >
> > > > > > > > > "Alternatively, any gridded data file that MET can
read
> > > > > > > > > may be used to define a verification masking region.
> > > > > > > > > Users must specify a description of the field to be
used
> > > > > > > > > from the input file and, optionally, may specify a
> > > > > > > > > threshold to be applied to
> > > > that field.
> > > > > > > > > Any grid point where the resulting field is 0, the
mask
> > > > > > > > > is turned
> > > > > > off.
> > > > > > > > > Any grid point where it is non-zero, the mask is
turned
> on."
> > > > > > > > >
> > > > > > > > > Can you please help me get the syntax correct?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Matt
> > > > > > > > >
> > > > > > > > > // SIGNED //
> > > > > > > > > Matthew C. Sittel
> > > > > > > > > University Corporation for Atmospheric Research
> > > > > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > > > > matthew.sittel.ctr at us.af.mil
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Tue Sep 01 12:42:17 2015
I'm not sure why CPC is being used, other than it's a second opinion
to the Stage IV data.
Let me see if the modeler can easily make the binary data into GriB.
I didn't even know until today this product wasn't coming into the
building in NetCDF.
Thanks for the info on 5.1. Bob is trying to get approval to formally
install MET on our development server... since we aren't allowed to do
that ourselves anymore. Sigh...
Thanks,
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, September 01, 2015 12:03 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
I asked around here, and we've used the CPC analysis data in the past.
In particular, we used on a project comparing GFS precipitation to NAM
precipitation. Part of that work was preparing the binary CPC data
for use in MET.
To do so, we ran a FORTRAN utility called grbrfc8, which is available
in this directory:
http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/sorc/nam_cpc2grbrfc8.fd
It writes a GRIB output file which MET can easily handle. It looks
like this is set up to read 1/8 degree CPC data, and you mention that
you have
1/2 degree data. So I'm not sure if this would help.
However, we no longer use the CPC analysis data for verification, and
I'm told that people are generally moving away from it and using
StageII, StageIV, or CCPA data instead.
Things would be easiest if we could get the data into GRIB format.
Short of that, we'd need to make it look like the NetCDF output of
pcp_combine to use it in METv4.1.
We expect version 5.1 to be available by the end of September. We've
been instructed to add support for computing anomaly correlation in
the same way NCEP is doing it which is pushing development out a
little longer.
John
On Tue, Sep 1, 2015 at 10:15 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> The modeler downloaded a software package off of the web that
creates
> NetCDF from binary data. So the NetCDF 'construction' is done in
> house, by converting that original binary to NetCDF format.
>
> Is there a simple way to determine when a NetCDF file is/isn't
> CF-compliant?
>
> If we can download/install NCO here we can use 'ncatted' as you did.
> When you used the following command:
>
> ncatted -a _FillValue,EndOfDay,a,f,-999 -a
Conventions,global,a,c,"CF-1.0"
> eodhours_rename.nc eodhours_mod.nc
>
> Did this change the NetCDF file version from "CF-1.something" to
> "CF-1.0", or is that the CF version of the NetCDF input you are
trying to undo?
>
> Bob Craig wants to know when MET 5.1 will be released.
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, September 01, 2015 11:06 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> METv4.1 does not include support for CF-compliant NetCDF. If you
> continue using METv4.1, you'll need to make this data look like the
> NetCDF output of the pcp_combine tool.
>
> Another option would be finding the CPC analysis data in a different
> format that may be easier to read into MET. Is there something
> specific about the version of the CPC data you're using or could you
> use it from a different data source?
>
> Thanks,
> John
>
> On Tue, Sep 1, 2015 at 6:56 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hey John,
> >
> > Something finally occurred to me on the way home yesterday
regarding
> > this problem. And I think I've found the issue.
> >
> > The NetCDF file I'm reading in for observation data is NOT one
MET
> > generates.
> >
> > We are comparing two different cases, and in the first case I'm
> > summing up one-hour NCEP Stage IV data using 'pcp_combine'. So
that
> > output format is correct since MET generated it.
> >
> > The other one I'm using "as-is" (half-degree CPC analyses), and
it's
> > already NetCDF and summed to 24 hours. But... you guessed it,
it's
> > the same NetCDF format as that "EndOfDay" file.
> >
> > So the solution is for me to get the modeler to reformat all the
> > observational data he generates, or reformat it myself using the
> > outline below. The problem is 'ncatted' doesn't appear to work on
> > our system. So I will pursue getting the modeler to change the
format.
> >
> > Can you help me explain to him what he needs to do to change the
> > NetCDF-CF format into something MET can read? Rather than
> > converting it as you did, is there something he can do at the
> > creation phase to
> write it out properly?
> >
> > Also, what missing value does MET expect? '-9999.' Possibly?
I'll
> > get him to change that too.
> >
> > Thanks,
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Monday, August 31, 2015 6:15 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > Hmm, sorry no, nothing else is jumping out at me. If I were able
to
> > replicate the error you're getting, it'd probably jump out at me
> > really quickly.
> >
> > If you're able to post data somewhere I could grab it, I'd be
happy
> > to try to reproduce the error here. I'd need the grid_stat config
> > file, the forecast, and observation files, the mask file, and the
> > exact command you're running on the command line.
> >
> > The reading of the config files is working as it should. MET
reads
> > the ConfigConstants file, followed by the default config file for
> > that tool, followed by the user-specified config file. That's how
> > we designed it to work.
> >
> > Thanks,
> > John
> >
> > On Mon, Aug 31, 2015 at 2:33 PM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hey John,
> > >
> > > Okay, so it looks like that even if I don't use it, that
> > > GridStatConfig_default must be present to even *run* grid_stat?
> > > If so, that negates what I thought I found by changing its file
name.
> > >
> > > I know when I ran a separate version of grid_stat it did indeed
> > > uses the user-defined config file because it used the modified
> > > model name I placed in the user-defined version.
> > >
> > > So what I did list before is still valid; it appears grid_stat
> > > reads my config file and doesn't like something:
> > >
> > > DEBUG 1: Default Config File:
> > > /home/qcteam/METv4.1/data/config/GridStatConfig_default
> > > DEBUG 1: User Config File:
> > > /h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
> > > new Met2dDataFile object of type "FileType_Gb1".
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
> > > new Met2dDataFile object of type "FileType_NcMet".
> > > NetCDF: Attribute not found
> > >
> > > Recall (and note I am using the name you gave me!):
> > >
> > > mask = {
> > > grid = [];
> > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
{name
> > > = \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > >
> > > And I'm still confused.
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Monday, August 31, 2015 2:47 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > Question
> > >
> > > Matt,
> > >
> > > If you're able to successfully make an image with that
> > > plot_data_plane command I sent, then MET is able to read that
data
> file fine.
> > >
> > > However, you're still receiving the very unhelpful error
message:
> > > "NetCDF: Attribute not found"
> > >
> > > Seems to me there are a couple possible reasons...
> > >
> > > (1) In your Grid-Stat configuration file in the "mask->poly"
> > > section, are you positive that you're pointing to the updated
mask
> > > file I sent
> > you?
> > > Please make sure the path is correct and you're pointing to the
> > > file on which you ran that plot_data_plane command.
> > >
> > > (2) Another possibility is that you're getting this error
message
> > > from Grid-Stat when trying to read some other gridded data
file...
> > > like the forecast or observation files.
> > >
> > > If you're still having trouble, please rerun your grid_stat
> > > command using these options:
> > > -v 4 -log run_grid_stat.log
> > >
> > > That'll write the output from grid_stat to a log file named
> > > "run_grid_stat.log".
> > >
> > > Please send me that log file.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Hey John,
> > > >
> > > > Just checking to make sure we don't have e-mail problems here
again.
> > > > Below is the last response I got from you.
> > > >
> > > > The modelers are breathing down my neck wanting their data so
I
> > > > wanted to check.
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, August 28, 2015 12:16 PM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > Try running the convert command to reformat from postscript to
png:
> > > > convert -rotate 90 eodhours_met.ps eodhours_met.png
> > > >
> > > > And then view that .png file either through a web browser or
> > > > using the display command:
> > > > display eodhours_met.png
> > > >
> > > > In your settings it looks like you have a space between '='
and '24'.
> > > > That probably won't parse correctly... try removing that
space:
> > > >
> > > > mask = {
> > > > grid = [];
> > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > {name = \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > > >
> > > > On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF
> > > > AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > >
> > > > >
> > > > > It did generate a .ps file that is non-zero in size. Now if
I
> > > > > can remember how we view .ps files on our devel system I'll
be
> > > > > all
> set?
> > > > >
> > > > > Just to double check, here's what I have in the config file
> > > > > for
> > 'mask':
> > > > >
> > > > > mask = {
> > > > > grid = [];
> > > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > > {name = \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Friday, August 28, 2015 11:46 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Matt,
> > > > >
> > > > > Please try the following...
> > > > >
> > > > > (1) Retrieve this file:
> > > > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodh
> > > > > ou
> > > > > rs
> > > > > _m
> > > > > et
> > > > > .nc
> > > > > (2) Run METv4.1 to try to read this file:
> > > > > METv4.1/bin/plot_data_plane eodhours_met.nc
> > > > > eodhours_met.ps 'name="EndOfDay"; level="(*,*)";'
> > > > >
> > > > > Does that successfully create an output PostScript file as
it
> > > > > does on my machine? If so, using it in the Grid-Stat
> > > > > configuration file should work fine.
> > > > >
> > > > > As for defining masks in Grid-Stat, as you know, there are 2
> options:
> > > > > grid and poly...
> > > > > - Basically, "grid" handles named grid like "G212" for NCEP
> > > > > Grid
> > > > > 212 or "G001" for NCEP grid 1. "FULL" is a special case for
> > > > > grid where we just use all points in the input grid.
> > > > > - And "poly" handles everything else... like reading an
ASCII
> > > > > lat/lon polyline file or reading gridded data files and
> > > > > optionally applying a threshold to them.
> > > > >
> > > > > Were we to do it over again, we might split that out by
using
> "poly"
> > > > > to specify ASCII lat/lon polyline files and adding "data" to
> > > > > specify gridded data files. I suppose we didn't do that
> > > > > because we wanted to read the lat/lon polyline file or the
> > > > > NetCDF output of the gen_poly_mask tool at the same place.
Oh well.
> > > > >
> > > > > John
> > > > >
> > > > > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Thank you for re-sending the reply and the files.
> > > > > >
> > > > > > I didn’t generate this NetCDF file, so I can get the
author
> > > > > > to make this easier should I need to replicate this
product.
> > > > > > I was wondering about the whitespace.
> > > > > >
> > > > > > Bob Craig is getting the approval to install MET 5.0 on
our
> > > > > > new development server. We have to go through a lot of
> > > > > > people to get that approved. So if a 5.1 release is
> > > > > > imminent, we should probably wait so we don't have to go
> > > > > > through all of this twice just to update. AFWA frowns on
> > > > > > introducing any sort of external software onto their
> > > > > > systems, which makes it
> harder for us.
> > > > > >
> > > > > > So I tried your fix and I get an error message of:
> > > > > >
> > > > > > "NetCDF: Attribute not found".
> > > > > >
> > > > > > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> > > > > >
> > > > > > Matt
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Friday, August 28, 2015 10:49 AM
> > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > Question
> > > > > >
> > > > > > Email from 8/25...
> > > > > >
> > > > > > Matt,
> > > > > >
> > > > > > Good news and bad news.
> > > > > >
> > > > > > The good news is that I'm able to get MET to read this
file,
> > > > > > but the bad news is that it's with the development version
> > > > > > of MET, not even a released version!
> > > > > >
> > > > > > But I did modify your NetCDF file first in three ways:
> > > > > >
> > > > > > (1) MET does not support variable names with embedded
whitespace.
> > > > > > Those variable names can ultimately be written to the
ASCII
> > > > > > output files of MET whose columns are whitespace
delimited.
> > > > > > Having whitespace in the variable names would throw off
the
> > > > > > number of columns. So I renamed "End of day" to
> > > > > > "EndOfDay":
> > > > > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > > > > eodhours_rename.nc
> > > > > >
> > > > > > (2) Looks like you're using -999 for missing data, so I
> > > > > > added the _FillValue attribute.
> > > > > >
> > > > > > (3) Also, MET needs to interpret this as a NetCDF file
> > > > > > following the CF convention, so I added that as a global
> attribute:
> > > > > >
> > > > > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > > > > Conventions,global,a,c,"CF-1.0" \
> > > > > > eodhours_rename.nc eodhours_mod.nc
> > > > > >
> > > > > > You're using METv4.1, but that version doesn't include
> > > > > > support for NetCDF-CF files. So this file won't work in
METv4.1.
> > > > > >
> > > > > > So here's what I did...
> > > > > >
> > > > > > I ran the modified data file through the development
> "pcp_combine"
> > > > > > tool to create a flavor of NetCDF that METv4.1 can
actually read.
> > > > > > I plotted it using this command:
> > > > > >
> > > > > > METv4.1/bin/plot_data_plane \
> > > > > > eodhours_met.nc eodhours_met.ps \
> > > > > > 'name="EndOfDay"; level="(*,*)";'
> > > > > >
> > > > > > Both the NetCDF file (eodhours_met.nc) and the resulting
> > > > > > image
> > > > > > (eodhour_met.png) are attached.
> > > > > >
> > > > > > Try the following to use this as a mask in Grid-Stat.
This
> > > > > > defines
> > > > > > 2 different masks, one for eod = 24 and another for eod =
12:
> > > > > >
> > > > > > mask = {
> > > > > > grid = [];
> > > > > > poly = [
> > > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.n
> > > > > > c { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > > > > >
> > > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.n
> > > > > > c { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > > > > >
> > > > > > I agree, that's awfully convoluted!
> > > > > >
> > > > > > Hope that helps.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > AFWA
> > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > >
> > > > > > >
> > > > > > > Can you resend please? It must still be out in
cyberspace.
> > > > > > > This morning I got all of yesterday's mail. Note I was
> > > > > > > here all day yesterday... it's always something.
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
> > > > > > > File Question
> > > > > > >
> > > > > > > Matt,
> > > > > > >
> > > > > > > I received an email from you on August 25 with a NetCDF
> > > > > > > file
> > > > > attachment.
> > > > > > > Then I replied to you on August 25th in an email with 2
> > > > > > > attachments ( eodhours_met.nc and eodhours_met.ps) on
> > > > > > > August
> 25.
> > > > > > > Did you receive my reply on 8/25?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR
> > > > > > > USAF AFWA
> > > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hey John,
> > > > > > > >
> > > > > > > > Did you receive the NetCDF file I sent? I put it as
an
> > > > > > > > attachment to an e-mail. AFWA E-mail has been really
> > > > > > > > messed up here as of late so it's possible that it
never
> > > > > > > > went out, or that if you replied I haven't received
it.
> > > > > > > > I just wanted to
> > > check.
> > > > > > > >
> > > > > > > > Matt
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: John Halley Gotway via RT
> > > > > > > > [mailto:met_help at ucar.edu]
> > > > > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
> > > > > > > > File Question
> > > > > > > >
> > > > > > > > Matt,
> > > > > > > >
> > > > > > > > Are you able to send me that end of day NetCDF file?
> > > > > > > > Probably easiest for met to get it correct here and
then
> > > > > > > > let
> > you know.
> > > > > > > > Also, is this
> > > > > > > > METv4.1 or met-5.0?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C
CTR
> > > > > > > > USAF AFWA
> > > > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted
upon.
> > > > > > > > > Transaction: Ticket created by
> matthew.sittel.ctr at us.af.mil
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: Grid Stat Config File Question
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=7312
> > > > > > > > > 4
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I'm trying to figure out the right way to set the
> > > > > > > > > config file for grid_stat to do the following:
> > > > > > > > >
> > > > > > > > > -- generate a mask for my data based on the contents
> > > > > > > > > of an external file, in NetCDF format
> > > > > > > > > -- keying this mask off of a NetCDF variable named
> > > > > > > > > "End of
> > day"
> > > > > > > > > within the .nc file
> > > > > > > > > -- Keep only those grid points where "End of day"
> > > > > > > > > equals
> > > > > > > > > 24
> > > > > > > > >
> > > > > > > > > I can't seem to find the right combination of
equals,
> > > > > > > > > semicolons, etc., to do this.
> > > > > > > > >
> > > > > > > > > Right now I have:
> > > > > > > > >
> > > > > > > > > mask = {
> > > > > > > > > grid = [
> > > > > > > > > name =
> > > > > > > > > "/h/data/global/WXQC/data/met/cfg/eodhours.nc
> ";
> > > > > > > > > field = [
> > > > > > > > > {
> > > > > > > > > name="End of day";
> > > > > > > > > cat_thresh=[==24];
> > > > > > > > > }
> > > > > > > > > ];
> > > > > > > > > ];
> > > > > > > > > poly = [];
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > > The header from the NetCDF file in question:
> > > > > > > > >
> > > > > > > > > netcdf eodhours {
> > > > > > > > > dimensions:
> > > > > > > > > latitude = 360 ;
> > > > > > > > > longitude = 720 ;
> > > > > > > > > variables:
> > > > > > > > > float latitude(latitude) ;
> > > > > > > > > latitude:units = "degrees_north" ;
> > > > > > > > > float longitude(longitude) ;
> > > > > > > > > longitude:units = "degrees_east" ;
> > > > > > > > > float End of day(latitude, longitude) ;
> > > > > > > > > End of day:units = "hours" ; }
> > > > > > > > >
> > > > > > > > > The mask syntax I used is incorrect.
> > > > > > > > >
> > > > > > > > > For future MET user guides, the following paragraph
> > > > > > > > > could really use an example to make this clear:
> > > > > > > > >
> > > > > > > > > "Alternatively, any gridded data file that MET can
> > > > > > > > > read may be used to define a verification masking
region.
> > > > > > > > > Users must specify a description of the field to be
> > > > > > > > > used from the input file and, optionally, may
specify
> > > > > > > > > a threshold to be applied to
> > > > that field.
> > > > > > > > > Any grid point where the resulting field is 0, the
> > > > > > > > > mask is turned
> > > > > > off.
> > > > > > > > > Any grid point where it is non-zero, the mask is
> > > > > > > > > turned
> on."
> > > > > > > > >
> > > > > > > > > Can you please help me get the syntax correct?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Matt
> > > > > > > > >
> > > > > > > > > // SIGNED //
> > > > > > > > > Matthew C. Sittel
> > > > > > > > > University Corporation for Atmospheric Research
> > > > > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > > > > matthew.sittel.ctr at us.af.mil
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Sep 03 06:53:09 2015
Hi John,
We're still trying to replicate the NetCDF format MET will accept.
It's not going quickly since that "NetCDF: Attribute not found" isn't
giving us hints as to what we're doing wrong.
I think we will ultimately go to GriB, but in the meantime we are
testing format modifications before abandoning that altogether.
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, September 01, 2015 12:03 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
I asked around here, and we've used the CPC analysis data in the past.
In particular, we used on a project comparing GFS precipitation to NAM
precipitation. Part of that work was preparing the binary CPC data
for use in MET.
To do so, we ran a FORTRAN utility called grbrfc8, which is available
in this directory:
http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/sorc/nam_cpc2grbrfc8.fd
It writes a GRIB output file which MET can easily handle. It looks
like this is set up to read 1/8 degree CPC data, and you mention that
you have
1/2 degree data. So I'm not sure if this would help.
However, we no longer use the CPC analysis data for verification, and
I'm told that people are generally moving away from it and using
StageII, StageIV, or CCPA data instead.
Things would be easiest if we could get the data into GRIB format.
Short of that, we'd need to make it look like the NetCDF output of
pcp_combine to use it in METv4.1.
We expect version 5.1 to be available by the end of September. We've
been instructed to add support for computing anomaly correlation in
the same way NCEP is doing it which is pushing development out a
little longer.
John
On Tue, Sep 1, 2015 at 10:15 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> The modeler downloaded a software package off of the web that
creates
> NetCDF from binary data. So the NetCDF 'construction' is done in
> house, by converting that original binary to NetCDF format.
>
> Is there a simple way to determine when a NetCDF file is/isn't
> CF-compliant?
>
> If we can download/install NCO here we can use 'ncatted' as you did.
> When you used the following command:
>
> ncatted -a _FillValue,EndOfDay,a,f,-999 -a
Conventions,global,a,c,"CF-1.0"
> eodhours_rename.nc eodhours_mod.nc
>
> Did this change the NetCDF file version from "CF-1.something" to
> "CF-1.0", or is that the CF version of the NetCDF input you are
trying to undo?
>
> Bob Craig wants to know when MET 5.1 will be released.
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, September 01, 2015 11:06 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> METv4.1 does not include support for CF-compliant NetCDF. If you
> continue using METv4.1, you'll need to make this data look like the
> NetCDF output of the pcp_combine tool.
>
> Another option would be finding the CPC analysis data in a different
> format that may be easier to read into MET. Is there something
> specific about the version of the CPC data you're using or could you
> use it from a different data source?
>
> Thanks,
> John
>
> On Tue, Sep 1, 2015 at 6:56 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > Hey John,
> >
> > Something finally occurred to me on the way home yesterday
regarding
> > this problem. And I think I've found the issue.
> >
> > The NetCDF file I'm reading in for observation data is NOT one
MET
> > generates.
> >
> > We are comparing two different cases, and in the first case I'm
> > summing up one-hour NCEP Stage IV data using 'pcp_combine'. So
that
> > output format is correct since MET generated it.
> >
> > The other one I'm using "as-is" (half-degree CPC analyses), and
it's
> > already NetCDF and summed to 24 hours. But... you guessed it,
it's
> > the same NetCDF format as that "EndOfDay" file.
> >
> > So the solution is for me to get the modeler to reformat all the
> > observational data he generates, or reformat it myself using the
> > outline below. The problem is 'ncatted' doesn't appear to work on
> > our system. So I will pursue getting the modeler to change the
format.
> >
> > Can you help me explain to him what he needs to do to change the
> > NetCDF-CF format into something MET can read? Rather than
> > converting it as you did, is there something he can do at the
> > creation phase to
> write it out properly?
> >
> > Also, what missing value does MET expect? '-9999.' Possibly?
I'll
> > get him to change that too.
> >
> > Thanks,
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Monday, August 31, 2015 6:15 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > Hmm, sorry no, nothing else is jumping out at me. If I were able
to
> > replicate the error you're getting, it'd probably jump out at me
> > really quickly.
> >
> > If you're able to post data somewhere I could grab it, I'd be
happy
> > to try to reproduce the error here. I'd need the grid_stat config
> > file, the forecast, and observation files, the mask file, and the
> > exact command you're running on the command line.
> >
> > The reading of the config files is working as it should. MET
reads
> > the ConfigConstants file, followed by the default config file for
> > that tool, followed by the user-specified config file. That's how
> > we designed it to work.
> >
> > Thanks,
> > John
> >
> > On Mon, Aug 31, 2015 at 2:33 PM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hey John,
> > >
> > > Okay, so it looks like that even if I don't use it, that
> > > GridStatConfig_default must be present to even *run* grid_stat?
> > > If so, that negates what I thought I found by changing its file
name.
> > >
> > > I know when I ran a separate version of grid_stat it did indeed
> > > uses the user-defined config file because it used the modified
> > > model name I placed in the user-defined version.
> > >
> > > So what I did list before is still valid; it appears grid_stat
> > > reads my config file and doesn't like something:
> > >
> > > DEBUG 1: Default Config File:
> > > /home/qcteam/METv4.1/data/config/GridStatConfig_default
> > > DEBUG 1: User Config File:
> > > /h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
> > > new Met2dDataFile object of type "FileType_Gb1".
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
> > > new Met2dDataFile object of type "FileType_NcMet".
> > > NetCDF: Attribute not found
> > >
> > > Recall (and note I am using the name you gave me!):
> > >
> > > mask = {
> > > grid = [];
> > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
{name
> > > = \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > >
> > > And I'm still confused.
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Monday, August 31, 2015 2:47 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > Question
> > >
> > > Matt,
> > >
> > > If you're able to successfully make an image with that
> > > plot_data_plane command I sent, then MET is able to read that
data
> file fine.
> > >
> > > However, you're still receiving the very unhelpful error
message:
> > > "NetCDF: Attribute not found"
> > >
> > > Seems to me there are a couple possible reasons...
> > >
> > > (1) In your Grid-Stat configuration file in the "mask->poly"
> > > section, are you positive that you're pointing to the updated
mask
> > > file I sent
> > you?
> > > Please make sure the path is correct and you're pointing to the
> > > file on which you ran that plot_data_plane command.
> > >
> > > (2) Another possibility is that you're getting this error
message
> > > from Grid-Stat when trying to read some other gridded data
file...
> > > like the forecast or observation files.
> > >
> > > If you're still having trouble, please rerun your grid_stat
> > > command using these options:
> > > -v 4 -log run_grid_stat.log
> > >
> > > That'll write the output from grid_stat to a log file named
> > > "run_grid_stat.log".
> > >
> > > Please send me that log file.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Hey John,
> > > >
> > > > Just checking to make sure we don't have e-mail problems here
again.
> > > > Below is the last response I got from you.
> > > >
> > > > The modelers are breathing down my neck wanting their data so
I
> > > > wanted to check.
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, August 28, 2015 12:16 PM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > Try running the convert command to reformat from postscript to
png:
> > > > convert -rotate 90 eodhours_met.ps eodhours_met.png
> > > >
> > > > And then view that .png file either through a web browser or
> > > > using the display command:
> > > > display eodhours_met.png
> > > >
> > > > In your settings it looks like you have a space between '='
and '24'.
> > > > That probably won't parse correctly... try removing that
space:
> > > >
> > > > mask = {
> > > > grid = [];
> > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > {name = \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > > >
> > > > On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF
> > > > AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > >
> > > > >
> > > > > It did generate a .ps file that is non-zero in size. Now if
I
> > > > > can remember how we view .ps files on our devel system I'll
be
> > > > > all
> set?
> > > > >
> > > > > Just to double check, here's what I have in the config file
> > > > > for
> > 'mask':
> > > > >
> > > > > mask = {
> > > > > grid = [];
> > > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > > {name = \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Friday, August 28, 2015 11:46 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Matt,
> > > > >
> > > > > Please try the following...
> > > > >
> > > > > (1) Retrieve this file:
> > > > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodh
> > > > > ou
> > > > > rs
> > > > > _m
> > > > > et
> > > > > .nc
> > > > > (2) Run METv4.1 to try to read this file:
> > > > > METv4.1/bin/plot_data_plane eodhours_met.nc
> > > > > eodhours_met.ps 'name="EndOfDay"; level="(*,*)";'
> > > > >
> > > > > Does that successfully create an output PostScript file as
it
> > > > > does on my machine? If so, using it in the Grid-Stat
> > > > > configuration file should work fine.
> > > > >
> > > > > As for defining masks in Grid-Stat, as you know, there are 2
> options:
> > > > > grid and poly...
> > > > > - Basically, "grid" handles named grid like "G212" for NCEP
> > > > > Grid
> > > > > 212 or "G001" for NCEP grid 1. "FULL" is a special case for
> > > > > grid where we just use all points in the input grid.
> > > > > - And "poly" handles everything else... like reading an
ASCII
> > > > > lat/lon polyline file or reading gridded data files and
> > > > > optionally applying a threshold to them.
> > > > >
> > > > > Were we to do it over again, we might split that out by
using
> "poly"
> > > > > to specify ASCII lat/lon polyline files and adding "data" to
> > > > > specify gridded data files. I suppose we didn't do that
> > > > > because we wanted to read the lat/lon polyline file or the
> > > > > NetCDF output of the gen_poly_mask tool at the same place.
Oh well.
> > > > >
> > > > > John
> > > > >
> > > > > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Thank you for re-sending the reply and the files.
> > > > > >
> > > > > > I didn’t generate this NetCDF file, so I can get the
author
> > > > > > to make this easier should I need to replicate this
product.
> > > > > > I was wondering about the whitespace.
> > > > > >
> > > > > > Bob Craig is getting the approval to install MET 5.0 on
our
> > > > > > new development server. We have to go through a lot of
> > > > > > people to get that approved. So if a 5.1 release is
> > > > > > imminent, we should probably wait so we don't have to go
> > > > > > through all of this twice just to update. AFWA frowns on
> > > > > > introducing any sort of external software onto their
> > > > > > systems, which makes it
> harder for us.
> > > > > >
> > > > > > So I tried your fix and I get an error message of:
> > > > > >
> > > > > > "NetCDF: Attribute not found".
> > > > > >
> > > > > > Also, I'm wondering why this is a 'poly' and not a 'grid'.
> > > > > >
> > > > > > Matt
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > > > Sent: Friday, August 28, 2015 10:49 AM
> > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > Question
> > > > > >
> > > > > > Email from 8/25...
> > > > > >
> > > > > > Matt,
> > > > > >
> > > > > > Good news and bad news.
> > > > > >
> > > > > > The good news is that I'm able to get MET to read this
file,
> > > > > > but the bad news is that it's with the development version
> > > > > > of MET, not even a released version!
> > > > > >
> > > > > > But I did modify your NetCDF file first in three ways:
> > > > > >
> > > > > > (1) MET does not support variable names with embedded
whitespace.
> > > > > > Those variable names can ultimately be written to the
ASCII
> > > > > > output files of MET whose columns are whitespace
delimited.
> > > > > > Having whitespace in the variable names would throw off
the
> > > > > > number of columns. So I renamed "End of day" to
> > > > > > "EndOfDay":
> > > > > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > > > > eodhours_rename.nc
> > > > > >
> > > > > > (2) Looks like you're using -999 for missing data, so I
> > > > > > added the _FillValue attribute.
> > > > > >
> > > > > > (3) Also, MET needs to interpret this as a NetCDF file
> > > > > > following the CF convention, so I added that as a global
> attribute:
> > > > > >
> > > > > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > > > > Conventions,global,a,c,"CF-1.0" \
> > > > > > eodhours_rename.nc eodhours_mod.nc
> > > > > >
> > > > > > You're using METv4.1, but that version doesn't include
> > > > > > support for NetCDF-CF files. So this file won't work in
METv4.1.
> > > > > >
> > > > > > So here's what I did...
> > > > > >
> > > > > > I ran the modified data file through the development
> "pcp_combine"
> > > > > > tool to create a flavor of NetCDF that METv4.1 can
actually read.
> > > > > > I plotted it using this command:
> > > > > >
> > > > > > METv4.1/bin/plot_data_plane \
> > > > > > eodhours_met.nc eodhours_met.ps \
> > > > > > 'name="EndOfDay"; level="(*,*)";'
> > > > > >
> > > > > > Both the NetCDF file (eodhours_met.nc) and the resulting
> > > > > > image
> > > > > > (eodhour_met.png) are attached.
> > > > > >
> > > > > > Try the following to use this as a mask in Grid-Stat.
This
> > > > > > defines
> > > > > > 2 different masks, one for eod = 24 and another for eod =
12:
> > > > > >
> > > > > > mask = {
> > > > > > grid = [];
> > > > > > poly = [
> > > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.n
> > > > > > c { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > > > > >
> > > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.n
> > > > > > c { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > > > > >
> > > > > > I agree, that's awfully convoluted!
> > > > > >
> > > > > > Hope that helps.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > AFWA
> > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > >
> > > > > > >
> > > > > > > Can you resend please? It must still be out in
cyberspace.
> > > > > > > This morning I got all of yesterday's mail. Note I was
> > > > > > > here all day yesterday... it's always something.
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu]
> > > > > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
> > > > > > > File Question
> > > > > > >
> > > > > > > Matt,
> > > > > > >
> > > > > > > I received an email from you on August 25 with a NetCDF
> > > > > > > file
> > > > > attachment.
> > > > > > > Then I replied to you on August 25th in an email with 2
> > > > > > > attachments ( eodhours_met.nc and eodhours_met.ps) on
> > > > > > > August
> 25.
> > > > > > > Did you receive my reply on 8/25?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR
> > > > > > > USAF AFWA
> > > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hey John,
> > > > > > > >
> > > > > > > > Did you receive the NetCDF file I sent? I put it as
an
> > > > > > > > attachment to an e-mail. AFWA E-mail has been really
> > > > > > > > messed up here as of late so it's possible that it
never
> > > > > > > > went out, or that if you replied I haven't received
it.
> > > > > > > > I just wanted to
> > > check.
> > > > > > > >
> > > > > > > > Matt
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: John Halley Gotway via RT
> > > > > > > > [mailto:met_help at ucar.edu]
> > > > > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
> > > > > > > > File Question
> > > > > > > >
> > > > > > > > Matt,
> > > > > > > >
> > > > > > > > Are you able to send me that end of day NetCDF file?
> > > > > > > > Probably easiest for met to get it correct here and
then
> > > > > > > > let
> > you know.
> > > > > > > > Also, is this
> > > > > > > > METv4.1 or met-5.0?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C
CTR
> > > > > > > > USAF AFWA
> > > > > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted
upon.
> > > > > > > > > Transaction: Ticket created by
> matthew.sittel.ctr at us.af.mil
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: Grid Stat Config File Question
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=7312
> > > > > > > > > 4
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I'm trying to figure out the right way to set the
> > > > > > > > > config file for grid_stat to do the following:
> > > > > > > > >
> > > > > > > > > -- generate a mask for my data based on the contents
> > > > > > > > > of an external file, in NetCDF format
> > > > > > > > > -- keying this mask off of a NetCDF variable named
> > > > > > > > > "End of
> > day"
> > > > > > > > > within the .nc file
> > > > > > > > > -- Keep only those grid points where "End of day"
> > > > > > > > > equals
> > > > > > > > > 24
> > > > > > > > >
> > > > > > > > > I can't seem to find the right combination of
equals,
> > > > > > > > > semicolons, etc., to do this.
> > > > > > > > >
> > > > > > > > > Right now I have:
> > > > > > > > >
> > > > > > > > > mask = {
> > > > > > > > > grid = [
> > > > > > > > > name =
> > > > > > > > > "/h/data/global/WXQC/data/met/cfg/eodhours.nc
> ";
> > > > > > > > > field = [
> > > > > > > > > {
> > > > > > > > > name="End of day";
> > > > > > > > > cat_thresh=[==24];
> > > > > > > > > }
> > > > > > > > > ];
> > > > > > > > > ];
> > > > > > > > > poly = [];
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > > The header from the NetCDF file in question:
> > > > > > > > >
> > > > > > > > > netcdf eodhours {
> > > > > > > > > dimensions:
> > > > > > > > > latitude = 360 ;
> > > > > > > > > longitude = 720 ;
> > > > > > > > > variables:
> > > > > > > > > float latitude(latitude) ;
> > > > > > > > > latitude:units = "degrees_north" ;
> > > > > > > > > float longitude(longitude) ;
> > > > > > > > > longitude:units = "degrees_east" ;
> > > > > > > > > float End of day(latitude, longitude) ;
> > > > > > > > > End of day:units = "hours" ; }
> > > > > > > > >
> > > > > > > > > The mask syntax I used is incorrect.
> > > > > > > > >
> > > > > > > > > For future MET user guides, the following paragraph
> > > > > > > > > could really use an example to make this clear:
> > > > > > > > >
> > > > > > > > > "Alternatively, any gridded data file that MET can
> > > > > > > > > read may be used to define a verification masking
region.
> > > > > > > > > Users must specify a description of the field to be
> > > > > > > > > used from the input file and, optionally, may
specify
> > > > > > > > > a threshold to be applied to
> > > > that field.
> > > > > > > > > Any grid point where the resulting field is 0, the
> > > > > > > > > mask is turned
> > > > > > off.
> > > > > > > > > Any grid point where it is non-zero, the mask is
> > > > > > > > > turned
> on."
> > > > > > > > >
> > > > > > > > > Can you please help me get the syntax correct?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Matt
> > > > > > > > >
> > > > > > > > > // SIGNED //
> > > > > > > > > Matthew C. Sittel
> > > > > > > > > University Corporation for Atmospheric Research
> > > > > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > > > > matthew.sittel.ctr at us.af.mil
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Grid Stat Config File Question
From: John Halley Gotway
Time: Thu Sep 03 10:16:39 2015
Matt,
Yes, that is particularly unhelpful! Might be quickest if you send
me a
sample file, I update it, and sen you a list of the required changes.
I'm out of the office today, but could do it tomorrow.
John
On Thursday, September 3, 2015, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hi John,
>
> We're still trying to replicate the NetCDF format MET will accept.
It's
> not going quickly since that "NetCDF: Attribute not found" isn't
giving us
> hints as to what we're doing wrong.
>
> I think we will ultimately go to GriB, but in the meantime we are
testing
> format modifications before abandoning that altogether.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu
<javascript:;>]
> Sent: Tuesday, September 01, 2015 12:03 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> I asked around here, and we've used the CPC analysis data in the
past. In
> particular, we used on a project comparing GFS precipitation to NAM
> precipitation. Part of that work was preparing the binary CPC data
for use
> in MET.
>
> To do so, we ran a FORTRAN utility called grbrfc8, which is
available in
> this directory:
>
http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/sorc/nam_cpc2grbrfc8.fd
>
> It writes a GRIB output file which MET can easily handle. It looks
like
> this is set up to read 1/8 degree CPC data, and you mention that you
have
> 1/2 degree data. So I'm not sure if this would help.
>
> However, we no longer use the CPC analysis data for verification,
and I'm
> told that people are generally moving away from it and using
StageII,
> StageIV, or CCPA data instead.
>
> Things would be easiest if we could get the data into GRIB format.
Short
> of that, we'd need to make it look like the NetCDF output of
pcp_combine to
> use it in METv4.1.
>
> We expect version 5.1 to be available by the end of September.
We've been
> instructed to add support for computing anomaly correlation in the
same way
> NCEP is doing it which is pushing development out a little longer.
>
> John
>
> On Tue, Sep 1, 2015 at 10:15 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu <javascript:;>> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > The modeler downloaded a software package off of the web that
creates
> > NetCDF from binary data. So the NetCDF 'construction' is done in
> > house, by converting that original binary to NetCDF format.
> >
> > Is there a simple way to determine when a NetCDF file is/isn't
> > CF-compliant?
> >
> > If we can download/install NCO here we can use 'ncatted' as you
did.
> > When you used the following command:
> >
> > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> Conventions,global,a,c,"CF-1.0"
> > eodhours_rename.nc eodhours_mod.nc
> >
> > Did this change the NetCDF file version from "CF-1.something" to
> > "CF-1.0", or is that the CF version of the NetCDF input you are
trying
> to undo?
> >
> > Bob Craig wants to know when MET 5.1 will be released.
> >
> > Thanks,
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu
<javascript:;>
> ]
> > Sent: Tuesday, September 01, 2015 11:06 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > METv4.1 does not include support for CF-compliant NetCDF. If you
> > continue using METv4.1, you'll need to make this data look like
the
> > NetCDF output of the pcp_combine tool.
> >
> > Another option would be finding the CPC analysis data in a
different
> > format that may be easier to read into MET. Is there something
> > specific about the version of the CPC data you're using or could
you
> > use it from a different data source?
> >
> > Thanks,
> > John
> >
> > On Tue, Sep 1, 2015 at 6:56 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> > WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hey John,
> > >
> > > Something finally occurred to me on the way home yesterday
regarding
> > > this problem. And I think I've found the issue.
> > >
> > > The NetCDF file I'm reading in for observation data is NOT one
MET
> > > generates.
> > >
> > > We are comparing two different cases, and in the first case I'm
> > > summing up one-hour NCEP Stage IV data using 'pcp_combine'. So
that
> > > output format is correct since MET generated it.
> > >
> > > The other one I'm using "as-is" (half-degree CPC analyses), and
it's
> > > already NetCDF and summed to 24 hours. But... you guessed it,
it's
> > > the same NetCDF format as that "EndOfDay" file.
> > >
> > > So the solution is for me to get the modeler to reformat all the
> > > observational data he generates, or reformat it myself using the
> > > outline below. The problem is 'ncatted' doesn't appear to work
on
> > > our system. So I will pursue getting the modeler to change the
format.
> > >
> > > Can you help me explain to him what he needs to do to change the
> > > NetCDF-CF format into something MET can read? Rather than
> > > converting it as you did, is there something he can do at the
> > > creation phase to
> > write it out properly?
> > >
> > > Also, what missing value does MET expect? '-9999.' Possibly?
I'll
> > > get him to change that too.
> > >
> > > Thanks,
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu
> <javascript:;>]
> > > Sent: Monday, August 31, 2015 6:15 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> > >
> > > Matt,
> > >
> > > Hmm, sorry no, nothing else is jumping out at me. If I were
able to
> > > replicate the error you're getting, it'd probably jump out at me
> > > really quickly.
> > >
> > > If you're able to post data somewhere I could grab it, I'd be
happy
> > > to try to reproduce the error here. I'd need the grid_stat
config
> > > file, the forecast, and observation files, the mask file, and
the
> > > exact command you're running on the command line.
> > >
> > > The reading of the config files is working as it should. MET
reads
> > > the ConfigConstants file, followed by the default config file
for
> > > that tool, followed by the user-specified config file. That's
how
> > > we designed it to work.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Aug 31, 2015 at 2:33 PM, SITTEL, MATTHEW C CTR USAF AFWA
16
> > > WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Hey John,
> > > >
> > > > Okay, so it looks like that even if I don't use it, that
> > > > GridStatConfig_default must be present to even *run*
grid_stat?
> > > > If so, that negates what I thought I found by changing its
file name.
> > > >
> > > > I know when I ran a separate version of grid_stat it did
indeed
> > > > uses the user-defined config file because it used the modified
> > > > model name I placed in the user-defined version.
> > > >
> > > > So what I did list before is still valid; it appears grid_stat
> > > > reads my config file and doesn't like something:
> > > >
> > > > DEBUG 1: Default Config File:
> > > > /home/qcteam/METv4.1/data/config/GridStatConfig_default
> > > > DEBUG 1: User Config File:
> > > > /h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_Gb1".
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > NetCDF: Attribute not found
> > > >
> > > > Recall (and note I am using the name you gave me!):
> > > >
> > > > mask = {
> > > > grid = [];
> > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
{name
> > > > = \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > > >
> > > > And I'm still confused.
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu
> <javascript:;>]
> > > > Sent: Monday, August 31, 2015 2:47 PM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > If you're able to successfully make an image with that
> > > > plot_data_plane command I sent, then MET is able to read that
data
> > file fine.
> > > >
> > > > However, you're still receiving the very unhelpful error
message:
> > > > "NetCDF: Attribute not found"
> > > >
> > > > Seems to me there are a couple possible reasons...
> > > >
> > > > (1) In your Grid-Stat configuration file in the "mask->poly"
> > > > section, are you positive that you're pointing to the updated
mask
> > > > file I sent
> > > you?
> > > > Please make sure the path is correct and you're pointing to
the
> > > > file on which you ran that plot_data_plane command.
> > > >
> > > > (2) Another possibility is that you're getting this error
message
> > > > from Grid-Stat when trying to read some other gridded data
file...
> > > > like the forecast or observation files.
> > > >
> > > > If you're still having trouble, please rerun your grid_stat
> > > > command using these options:
> > > > -v 4 -log run_grid_stat.log
> > > >
> > > > That'll write the output from grid_stat to a log file named
> > > > "run_grid_stat.log".
> > > >
> > > > Please send me that log file.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > > > >
> > > > > Hey John,
> > > > >
> > > > > Just checking to make sure we don't have e-mail problems
here
> again.
> > > > > Below is the last response I got from you.
> > > > >
> > > > > The modelers are breathing down my neck wanting their data
so I
> > > > > wanted to check.
> > > > >
> > > > > Matt
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu
> <javascript:;>]
> > > > > Sent: Friday, August 28, 2015 12:16 PM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Matt,
> > > > >
> > > > > Try running the convert command to reformat from postscript
to png:
> > > > > convert -rotate 90 eodhours_met.ps eodhours_met.png
> > > > >
> > > > > And then view that .png file either through a web browser or
> > > > > using the display command:
> > > > > display eodhours_met.png
> > > > >
> > > > > In your settings it looks like you have a space between '='
and
> '24'.
> > > > > That probably won't parse correctly... try removing that
space:
> > > > >
> > > > > mask = {
> > > > > grid = [];
> > > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > > {name = \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > > > >
> > > > > On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > > It did generate a .ps file that is non-zero in size. Now
if I
> > > > > > can remember how we view .ps files on our devel system
I'll be
> > > > > > all
> > set?
> > > > > >
> > > > > > Just to double check, here's what I have in the config
file
> > > > > > for
> > > 'mask':
> > > > > >
> > > > > > mask = {
> > > > > > grid = [];
> > > > > > poly =
["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > > > {name = \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu
> <javascript:;>]
> > > > > > Sent: Friday, August 28, 2015 11:46 AM
> > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > Question
> > > > > >
> > > > > > Matt,
> > > > > >
> > > > > > Please try the following...
> > > > > >
> > > > > > (1) Retrieve this file:
> > > > > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eodh
> > > > > > ou
> > > > > > rs
> > > > > > _m
> > > > > > et
> > > > > > .nc
> > > > > > (2) Run METv4.1 to try to read this file:
> > > > > > METv4.1/bin/plot_data_plane eodhours_met.nc
> > > > > > eodhours_met.ps 'name="EndOfDay"; level="(*,*)";'
> > > > > >
> > > > > > Does that successfully create an output PostScript file as
it
> > > > > > does on my machine? If so, using it in the Grid-Stat
> > > > > > configuration file should work fine.
> > > > > >
> > > > > > As for defining masks in Grid-Stat, as you know, there are
2
> > options:
> > > > > > grid and poly...
> > > > > > - Basically, "grid" handles named grid like "G212" for
NCEP
> > > > > > Grid
> > > > > > 212 or "G001" for NCEP grid 1. "FULL" is a special case
for
> > > > > > grid where we just use all points in the input grid.
> > > > > > - And "poly" handles everything else... like reading an
ASCII
> > > > > > lat/lon polyline file or reading gridded data files and
> > > > > > optionally applying a threshold to them.
> > > > > >
> > > > > > Were we to do it over again, we might split that out by
using
> > "poly"
> > > > > > to specify ASCII lat/lon polyline files and adding "data"
to
> > > > > > specify gridded data files. I suppose we didn't do that
> > > > > > because we wanted to read the lat/lon polyline file or the
> > > > > > NetCDF output of the gen_poly_mask tool at the same place.
Oh
> well.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > AFWA
> > > > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Thank you for re-sending the reply and the files.
> > > > > > >
> > > > > > > I didn’t generate this NetCDF file, so I can get the
author
> > > > > > > to make this easier should I need to replicate this
product.
> > > > > > > I was wondering about the whitespace.
> > > > > > >
> > > > > > > Bob Craig is getting the approval to install MET 5.0 on
our
> > > > > > > new development server. We have to go through a lot of
> > > > > > > people to get that approved. So if a 5.1 release is
> > > > > > > imminent, we should probably wait so we don't have to go
> > > > > > > through all of this twice just to update. AFWA frowns
on
> > > > > > > introducing any sort of external software onto their
> > > > > > > systems, which makes it
> > harder for us.
> > > > > > >
> > > > > > > So I tried your fix and I get an error message of:
> > > > > > >
> > > > > > > "NetCDF: Attribute not found".
> > > > > > >
> > > > > > > Also, I'm wondering why this is a 'poly' and not a
'grid'.
> > > > > > >
> > > > > > > Matt
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu
> <javascript:;>]
> > > > > > > Sent: Friday, August 28, 2015 10:49 AM
> > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > > Question
> > > > > > >
> > > > > > > Email from 8/25...
> > > > > > >
> > > > > > > Matt,
> > > > > > >
> > > > > > > Good news and bad news.
> > > > > > >
> > > > > > > The good news is that I'm able to get MET to read this
file,
> > > > > > > but the bad news is that it's with the development
version
> > > > > > > of MET, not even a released version!
> > > > > > >
> > > > > > > But I did modify your NetCDF file first in three ways:
> > > > > > >
> > > > > > > (1) MET does not support variable names with embedded
> whitespace.
> > > > > > > Those variable names can ultimately be written to the
ASCII
> > > > > > > output files of MET whose columns are whitespace
delimited.
> > > > > > > Having whitespace in the variable names would throw off
the
> > > > > > > number of columns. So I renamed "End of day" to
> > > > > > > "EndOfDay":
> > > > > > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > > > > > eodhours_rename.nc
> > > > > > >
> > > > > > > (2) Looks like you're using -999 for missing data, so I
> > > > > > > added the _FillValue attribute.
> > > > > > >
> > > > > > > (3) Also, MET needs to interpret this as a NetCDF file
> > > > > > > following the CF convention, so I added that as a global
> > attribute:
> > > > > > >
> > > > > > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > > > > > Conventions,global,a,c,"CF-1.0" \
> > > > > > > eodhours_rename.nc eodhours_mod.nc
> > > > > > >
> > > > > > > You're using METv4.1, but that version doesn't include
> > > > > > > support for NetCDF-CF files. So this file won't work in
> METv4.1.
> > > > > > >
> > > > > > > So here's what I did...
> > > > > > >
> > > > > > > I ran the modified data file through the development
> > "pcp_combine"
> > > > > > > tool to create a flavor of NetCDF that METv4.1 can
actually
> read.
> > > > > > > I plotted it using this command:
> > > > > > >
> > > > > > > METv4.1/bin/plot_data_plane \
> > > > > > > eodhours_met.nc eodhours_met.ps \
> > > > > > > 'name="EndOfDay"; level="(*,*)";'
> > > > > > >
> > > > > > > Both the NetCDF file (eodhours_met.nc) and the resulting
> > > > > > > image
> > > > > > > (eodhour_met.png) are attached.
> > > > > > >
> > > > > > > Try the following to use this as a mask in Grid-Stat.
This
> > > > > > > defines
> > > > > > > 2 different masks, one for eod = 24 and another for eod
= 12:
> > > > > > >
> > > > > > > mask = {
> > > > > > > grid = [];
> > > > > > > poly = [
> > > > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.n
> > > > > > > c { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > > > > > >
> > > > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met.n
> > > > > > > c { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ]; };
> > > > > > >
> > > > > > > I agree, that's awfully convoluted!
> > > > > > >
> > > > > > > Hope that helps.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > > AFWA
> > > > > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>>
wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > > >
> > > > > > > >
> > > > > > > > Can you resend please? It must still be out in
cyberspace.
> > > > > > > > This morning I got all of yesterday's mail. Note I
was
> > > > > > > > here all day yesterday... it's always something.
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu
> <javascript:;>]
> > > > > > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
> > > > > > > > File Question
> > > > > > > >
> > > > > > > > Matt,
> > > > > > > >
> > > > > > > > I received an email from you on August 25 with a
NetCDF
> > > > > > > > file
> > > > > > attachment.
> > > > > > > > Then I replied to you on August 25th in an email with
2
> > > > > > > > attachments ( eodhours_met.nc and eodhours_met.ps) on
> > > > > > > > August
> > 25.
> > > > > > > > Did you receive my reply on 8/25?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR
> > > > > > > > USAF AFWA
> > > > > > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>>
wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hey John,
> > > > > > > > >
> > > > > > > > > Did you receive the NetCDF file I sent? I put it as
an
> > > > > > > > > attachment to an e-mail. AFWA E-mail has been
really
> > > > > > > > > messed up here as of late so it's possible that it
never
> > > > > > > > > went out, or that if you replied I haven't received
it.
> > > > > > > > > I just wanted to
> > > > check.
> > > > > > > > >
> > > > > > > > > Matt
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: John Halley Gotway via RT
> > > > > > > > > [mailto:met_help at ucar.edu <javascript:;>]
> > > > > > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat
Config
> > > > > > > > > File Question
> > > > > > > > >
> > > > > > > > > Matt,
> > > > > > > > >
> > > > > > > > > Are you able to send me that end of day NetCDF file?
> > > > > > > > > Probably easiest for met to get it correct here and
then
> > > > > > > > > let
> > > you know.
> > > > > > > > > Also, is this
> > > > > > > > > METv4.1 or met-5.0?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C
CTR
> > > > > > > > > USAF AFWA
> > > > > > > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>>
wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted
upon.
> > > > > > > > > > Transaction: Ticket created by
> > matthew.sittel.ctr at us.af.mil <javascript:;>
> > > > > > > > > > Queue: met_help
> > > > > > > > > > Subject: Grid Stat Config File Question
> > > > > > > > > > Owner: Nobody
> > > > > > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> <javascript:;>
> > > > > > > > > > Status: new
> > > > > > > > > > Ticket <URL:
> > > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=7312
> > > > > > > > > > 4
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I'm trying to figure out the right way to set the
> > > > > > > > > > config file for grid_stat to do the following:
> > > > > > > > > >
> > > > > > > > > > -- generate a mask for my data based on the
contents
> > > > > > > > > > of an external file, in NetCDF format
> > > > > > > > > > -- keying this mask off of a NetCDF variable named
> > > > > > > > > > "End of
> > > day"
> > > > > > > > > > within the .nc file
> > > > > > > > > > -- Keep only those grid points where "End of day"
> > > > > > > > > > equals
> > > > > > > > > > 24
> > > > > > > > > >
> > > > > > > > > > I can't seem to find the right combination of
equals,
> > > > > > > > > > semicolons, etc., to do this.
> > > > > > > > > >
> > > > > > > > > > Right now I have:
> > > > > > > > > >
> > > > > > > > > > mask = {
> > > > > > > > > > grid = [
> > > > > > > > > > name =
> > > > > > > > > > "/h/data/global/WXQC/data/met/cfg/eodhours.nc
> > ";
> > > > > > > > > > field = [
> > > > > > > > > > {
> > > > > > > > > > name="End of day";
> > > > > > > > > > cat_thresh=[==24];
> > > > > > > > > > }
> > > > > > > > > > ];
> > > > > > > > > > ];
> > > > > > > > > > poly = [];
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > > The header from the NetCDF file in question:
> > > > > > > > > >
> > > > > > > > > > netcdf eodhours {
> > > > > > > > > > dimensions:
> > > > > > > > > > latitude = 360 ;
> > > > > > > > > > longitude = 720 ;
> > > > > > > > > > variables:
> > > > > > > > > > float latitude(latitude) ;
> > > > > > > > > > latitude:units = "degrees_north" ;
> > > > > > > > > > float longitude(longitude) ;
> > > > > > > > > > longitude:units = "degrees_east" ;
> > > > > > > > > > float End of day(latitude, longitude) ;
> > > > > > > > > > End of day:units = "hours" ; }
> > > > > > > > > >
> > > > > > > > > > The mask syntax I used is incorrect.
> > > > > > > > > >
> > > > > > > > > > For future MET user guides, the following
paragraph
> > > > > > > > > > could really use an example to make this clear:
> > > > > > > > > >
> > > > > > > > > > "Alternatively, any gridded data file that MET can
> > > > > > > > > > read may be used to define a verification masking
region.
> > > > > > > > > > Users must specify a description of the field to
be
> > > > > > > > > > used from the input file and, optionally, may
specify
> > > > > > > > > > a threshold to be applied to
> > > > > that field.
> > > > > > > > > > Any grid point where the resulting field is 0, the
> > > > > > > > > > mask is turned
> > > > > > > off.
> > > > > > > > > > Any grid point where it is non-zero, the mask is
> > > > > > > > > > turned
> > on."
> > > > > > > > > >
> > > > > > > > > > Can you please help me get the syntax correct?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Matt
> > > > > > > > > >
> > > > > > > > > > // SIGNED //
> > > > > > > > > > Matthew C. Sittel
> > > > > > > > > > University Corporation for Atmospheric Research
> > > > > > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > > > > > matthew.sittel.ctr at us.af.mil <javascript:;>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Sep 03 10:34:09 2015
We have a workaround, believe it or not.
We finally got MET to read the NetCDF files by simply emulating what
you were using in that "eodhours_met.nc". Adding in the same header
information, including the comments specific to MET. So we're
running, albeit in a less than elegant manner.
Thank you for all your help! I think we can close this one out now.
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, September 03, 2015 11:17 AM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
Matt,
Yes, that is particularly unhelpful! Might be quickest if you send
me a sample file, I update it, and sen you a list of the required
changes.
I'm out of the office today, but could do it tomorrow.
John
On Thursday, September 3, 2015, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
>
> Hi John,
>
> We're still trying to replicate the NetCDF format MET will accept.
> It's not going quickly since that "NetCDF: Attribute not found"
isn't
> giving us hints as to what we're doing wrong.
>
> I think we will ultimately go to GriB, but in the meantime we are
> testing format modifications before abandoning that altogether.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu
> <javascript:;>]
> Sent: Tuesday, September 01, 2015 12:03 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File Question
>
> Matt,
>
> I asked around here, and we've used the CPC analysis data in the
past.
> In particular, we used on a project comparing GFS precipitation to
NAM
> precipitation. Part of that work was preparing the binary CPC data
> for use in MET.
>
> To do so, we ran a FORTRAN utility called grbrfc8, which is
available
> in this directory:
>
>
http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/sorc/nam_cpc2grbrfc8.fd
>
> It writes a GRIB output file which MET can easily handle. It looks
> like this is set up to read 1/8 degree CPC data, and you mention
that
> you have
> 1/2 degree data. So I'm not sure if this would help.
>
> However, we no longer use the CPC analysis data for verification,
and
> I'm told that people are generally moving away from it and using
> StageII, StageIV, or CCPA data instead.
>
> Things would be easiest if we could get the data into GRIB format.
> Short of that, we'd need to make it look like the NetCDF output of
> pcp_combine to use it in METv4.1.
>
> We expect version 5.1 to be available by the end of September.
We've
> been instructed to add support for computing anomaly correlation in
> the same way NCEP is doing it which is pushing development out a
little longer.
>
> John
>
> On Tue, Sep 1, 2015 at 10:15 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> >
> > The modeler downloaded a software package off of the web that
> > creates NetCDF from binary data. So the NetCDF 'construction' is
> > done in house, by converting that original binary to NetCDF
format.
> >
> > Is there a simple way to determine when a NetCDF file is/isn't
> > CF-compliant?
> >
> > If we can download/install NCO here we can use 'ncatted' as you
did.
> > When you used the following command:
> >
> > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> Conventions,global,a,c,"CF-1.0"
> > eodhours_rename.nc eodhours_mod.nc
> >
> > Did this change the NetCDF file version from "CF-1.something" to
> > "CF-1.0", or is that the CF version of the NetCDF input you are
> > trying
> to undo?
> >
> > Bob Craig wants to know when MET 5.1 will be released.
> >
> > Thanks,
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu
> > <javascript:;>
> ]
> > Sent: Tuesday, September 01, 2015 11:06 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
Question
> >
> > Matt,
> >
> > METv4.1 does not include support for CF-compliant NetCDF. If you
> > continue using METv4.1, you'll need to make this data look like
the
> > NetCDF output of the pcp_combine tool.
> >
> > Another option would be finding the CPC analysis data in a
different
> > format that may be easier to read into MET. Is there something
> > specific about the version of the CPC data you're using or could
you
> > use it from a different data source?
> >
> > Thanks,
> > John
> >
> > On Tue, Sep 1, 2015 at 6:56 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
> > WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124 >
> > >
> > > Hey John,
> > >
> > > Something finally occurred to me on the way home yesterday
> > > regarding this problem. And I think I've found the issue.
> > >
> > > The NetCDF file I'm reading in for observation data is NOT one
> > > MET generates.
> > >
> > > We are comparing two different cases, and in the first case I'm
> > > summing up one-hour NCEP Stage IV data using 'pcp_combine'. So
> > > that output format is correct since MET generated it.
> > >
> > > The other one I'm using "as-is" (half-degree CPC analyses), and
> > > it's already NetCDF and summed to 24 hours. But... you guessed
> > > it, it's the same NetCDF format as that "EndOfDay" file.
> > >
> > > So the solution is for me to get the modeler to reformat all the
> > > observational data he generates, or reformat it myself using the
> > > outline below. The problem is 'ncatted' doesn't appear to work
on
> > > our system. So I will pursue getting the modeler to change the
format.
> > >
> > > Can you help me explain to him what he needs to do to change the
> > > NetCDF-CF format into something MET can read? Rather than
> > > converting it as you did, is there something he can do at the
> > > creation phase to
> > write it out properly?
> > >
> > > Also, what missing value does MET expect? '-9999.' Possibly?
> > > I'll get him to change that too.
> > >
> > > Thanks,
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu
> <javascript:;>]
> > > Sent: Monday, August 31, 2015 6:15 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > Question
> > >
> > > Matt,
> > >
> > > Hmm, sorry no, nothing else is jumping out at me. If I were
able
> > > to replicate the error you're getting, it'd probably jump out at
> > > me really quickly.
> > >
> > > If you're able to post data somewhere I could grab it, I'd be
> > > happy to try to reproduce the error here. I'd need the
grid_stat
> > > config file, the forecast, and observation files, the mask file,
> > > and the exact command you're running on the command line.
> > >
> > > The reading of the config files is working as it should. MET
> > > reads the ConfigConstants file, followed by the default config
> > > file for that tool, followed by the user-specified config file.
> > > That's how we designed it to work.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Aug 31, 2015 at 2:33 PM, SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
>
> > > >
> > > > Hey John,
> > > >
> > > > Okay, so it looks like that even if I don't use it, that
> > > > GridStatConfig_default must be present to even *run*
grid_stat?
> > > > If so, that negates what I thought I found by changing its
file name.
> > > >
> > > > I know when I ran a separate version of grid_stat it did
indeed
> > > > uses the user-defined config file because it used the modified
> > > > model name I placed in the user-defined version.
> > > >
> > > > So what I did list before is still valid; it appears grid_stat
> > > > reads my config file and doesn't like something:
> > > >
> > > > DEBUG 1: Default Config File:
> > > > /home/qcteam/METv4.1/data/config/GridStatConfig_default
> > > > DEBUG 1: User Config File:
> > > > /h/data/global/WXQC/data/met/cfg/lis/lis_CPC_grid_stat.cfg.24
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_Gb1".
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_NcMet".
> > > > NetCDF: Attribute not found
> > > >
> > > > Recall (and note I am using the name you gave me!):
> > > >
> > > > mask = {
> > > > grid = [];
> > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > {name = \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > > >
> > > > And I'm still confused.
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu
> <javascript:;>]
> > > > Sent: Monday, August 31, 2015 2:47 PM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > Question
> > > >
> > > > Matt,
> > > >
> > > > If you're able to successfully make an image with that
> > > > plot_data_plane command I sent, then MET is able to read that
> > > > data
> > file fine.
> > > >
> > > > However, you're still receiving the very unhelpful error
message:
> > > > "NetCDF: Attribute not found"
> > > >
> > > > Seems to me there are a couple possible reasons...
> > > >
> > > > (1) In your Grid-Stat configuration file in the "mask->poly"
> > > > section, are you positive that you're pointing to the updated
> > > > mask file I sent
> > > you?
> > > > Please make sure the path is correct and you're pointing to
the
> > > > file on which you ran that plot_data_plane command.
> > > >
> > > > (2) Another possibility is that you're getting this error
> > > > message from Grid-Stat when trying to read some other gridded
data file...
> > > > like the forecast or observation files.
> > > >
> > > > If you're still having trouble, please rerun your grid_stat
> > > > command using these options:
> > > > -v 4 -log run_grid_stat.log
> > > >
> > > > That'll write the output from grid_stat to a log file named
> > > > "run_grid_stat.log".
> > > >
> > > > Please send me that log file.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Aug 31, 2015 at 1:12 PM, SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > >
> > > > >
> > > > > Hey John,
> > > > >
> > > > > Just checking to make sure we don't have e-mail problems
here
> again.
> > > > > Below is the last response I got from you.
> > > > >
> > > > > The modelers are breathing down my neck wanting their data
so
> > > > > I wanted to check.
> > > > >
> > > > > Matt
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu
> <javascript:;>]
> > > > > Sent: Friday, August 28, 2015 12:16 PM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config File
> > > > > Question
> > > > >
> > > > > Matt,
> > > > >
> > > > > Try running the convert command to reformat from postscript
to png:
> > > > > convert -rotate 90 eodhours_met.ps eodhours_met.png
> > > > >
> > > > > And then view that .png file either through a web browser or
> > > > > using the display command:
> > > > > display eodhours_met.png
> > > > >
> > > > > In your settings it looks like you have a space between '='
> > > > > and
> '24'.
> > > > > That probably won't parse correctly... try removing that
space:
> > > > >
> > > > > mask = {
> > > > > grid = [];
> > > > > poly = ["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > > {name = \"EndOfDay\"; level=\"(*,*)\";} =24"]; };
> > > > >
> > > > > On Fri, Aug 28, 2015 at 11:03 AM, SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > >
> > > > > >
> > > > > > It did generate a .ps file that is non-zero in size. Now
if
> > > > > > I can remember how we view .ps files on our devel system
> > > > > > I'll be all
> > set?
> > > > > >
> > > > > > Just to double check, here's what I have in the config
file
> > > > > > for
> > > 'mask':
> > > > > >
> > > > > > mask = {
> > > > > > grid = [];
> > > > > > poly =
["/h/data/global/WXQC/data/met/cfg/eodhours_met.nc
> > > > > > {name = \"EndOfDay\"; level=\"(*,*)\";} = 24"]; };
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu
> <javascript:;>]
> > > > > > Sent: Friday, August 28, 2015 11:46 AM
> > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
File
> > > > > > Question
> > > > > >
> > > > > > Matt,
> > > > > >
> > > > > > Please try the following...
> > > > > >
> > > > > > (1) Retrieve this file:
> > > > > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/sittel_data/eo
> > > > > > dh
> > > > > > ou
> > > > > > rs
> > > > > > _m
> > > > > > et
> > > > > > .nc
> > > > > > (2) Run METv4.1 to try to read this file:
> > > > > > METv4.1/bin/plot_data_plane eodhours_met.nc
> > > > > > eodhours_met.ps 'name="EndOfDay"; level="(*,*)";'
> > > > > >
> > > > > > Does that successfully create an output PostScript file as
> > > > > > it does on my machine? If so, using it in the Grid-Stat
> > > > > > configuration file should work fine.
> > > > > >
> > > > > > As for defining masks in Grid-Stat, as you know, there are
2
> > options:
> > > > > > grid and poly...
> > > > > > - Basically, "grid" handles named grid like "G212" for
NCEP
> > > > > > Grid
> > > > > > 212 or "G001" for NCEP grid 1. "FULL" is a special case
for
> > > > > > grid where we just use all points in the input grid.
> > > > > > - And "poly" handles everything else... like reading an
> > > > > > ASCII lat/lon polyline file or reading gridded data files
> > > > > > and optionally applying a threshold to them.
> > > > > >
> > > > > > Were we to do it over again, we might split that out by
> > > > > > using
> > "poly"
> > > > > > to specify ASCII lat/lon polyline files and adding "data"
to
> > > > > > specify gridded data files. I suppose we didn't do that
> > > > > > because we wanted to read the lat/lon polyline file or the
> > > > > > NetCDF output of the gen_poly_mask tool at the same place.
> > > > > > Oh
> well.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Fri, Aug 28, 2015 at 10:12 AM, SITTEL, MATTHEW C CTR
USAF
> > > > > > AFWA
> > > > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Thank you for re-sending the reply and the files.
> > > > > > >
> > > > > > > I didn’t generate this NetCDF file, so I can get the
> > > > > > > author to make this easier should I need to replicate
this product.
> > > > > > > I was wondering about the whitespace.
> > > > > > >
> > > > > > > Bob Craig is getting the approval to install MET 5.0 on
> > > > > > > our new development server. We have to go through a lot
> > > > > > > of people to get that approved. So if a 5.1 release is
> > > > > > > imminent, we should probably wait so we don't have to go
> > > > > > > through all of this twice just to update. AFWA frowns
on
> > > > > > > introducing any sort of external software onto their
> > > > > > > systems, which makes it
> > harder for us.
> > > > > > >
> > > > > > > So I tried your fix and I get an error message of:
> > > > > > >
> > > > > > > "NetCDF: Attribute not found".
> > > > > > >
> > > > > > > Also, I'm wondering why this is a 'poly' and not a
'grid'.
> > > > > > >
> > > > > > > Matt
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: John Halley Gotway via RT
[mailto:met_help at ucar.edu
> <javascript:;>]
> > > > > > > Sent: Friday, August 28, 2015 10:49 AM
> > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
> > > > > > > File Question
> > > > > > >
> > > > > > > Email from 8/25...
> > > > > > >
> > > > > > > Matt,
> > > > > > >
> > > > > > > Good news and bad news.
> > > > > > >
> > > > > > > The good news is that I'm able to get MET to read this
> > > > > > > file, but the bad news is that it's with the development
> > > > > > > version of MET, not even a released version!
> > > > > > >
> > > > > > > But I did modify your NetCDF file first in three ways:
> > > > > > >
> > > > > > > (1) MET does not support variable names with embedded
> whitespace.
> > > > > > > Those variable names can ultimately be written to the
> > > > > > > ASCII output files of MET whose columns are whitespace
delimited.
> > > > > > > Having whitespace in the variable names would throw off
> > > > > > > the number of columns. So I renamed "End of day" to
> > > > > > > "EndOfDay":
> > > > > > > ncrename -v 'End of day',EndOfDay eodhours.nc
> > > > > > > eodhours_rename.nc
> > > > > > >
> > > > > > > (2) Looks like you're using -999 for missing data, so I
> > > > > > > added the _FillValue attribute.
> > > > > > >
> > > > > > > (3) Also, MET needs to interpret this as a NetCDF file
> > > > > > > following the CF convention, so I added that as a global
> > attribute:
> > > > > > >
> > > > > > > ncatted -a _FillValue,EndOfDay,a,f,-999 -a
> > > > > > > Conventions,global,a,c,"CF-1.0" \
> > > > > > > eodhours_rename.nc eodhours_mod.nc
> > > > > > >
> > > > > > > You're using METv4.1, but that version doesn't include
> > > > > > > support for NetCDF-CF files. So this file won't work in
> METv4.1.
> > > > > > >
> > > > > > > So here's what I did...
> > > > > > >
> > > > > > > I ran the modified data file through the development
> > "pcp_combine"
> > > > > > > tool to create a flavor of NetCDF that METv4.1 can
> > > > > > > actually
> read.
> > > > > > > I plotted it using this command:
> > > > > > >
> > > > > > > METv4.1/bin/plot_data_plane \
> > > > > > > eodhours_met.nc eodhours_met.ps \
> > > > > > > 'name="EndOfDay"; level="(*,*)";'
> > > > > > >
> > > > > > > Both the NetCDF file (eodhours_met.nc) and the resulting
> > > > > > > image
> > > > > > > (eodhour_met.png) are attached.
> > > > > > >
> > > > > > > Try the following to use this as a mask in Grid-Stat.
> > > > > > > This defines
> > > > > > > 2 different masks, one for eod = 24 and another for eod
= 12:
> > > > > > >
> > > > > > > mask = {
> > > > > > > grid = [];
> > > > > > > poly = [
> > > > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met
> > > > > > > .n c { name = \"EndOfDay\"; level = \"(*,*)\"; } =24",
> > > > > > >
> > > > > > >
"/d1/johnhg/MET/MET_Help/sittel_data_20150825/eodhours_met
> > > > > > > .n c { name = \"EndOfDay\"; level = \"(*,*)\"; } =12" ];
> > > > > > > };
> > > > > > >
> > > > > > > I agree, that's awfully convoluted!
> > > > > > >
> > > > > > > Hope that helps.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Fri, Aug 28, 2015 at 9:37 AM, SITTEL, MATTHEW C CTR
> > > > > > > USAF AFWA
> > > > > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>>
wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73124
> > > > > > > > >
> > > > > > > >
> > > > > > > > Can you resend please? It must still be out in
cyberspace.
> > > > > > > > This morning I got all of yesterday's mail. Note I
was
> > > > > > > > here all day yesterday... it's always something.
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: John Halley Gotway via RT
> > > > > > > > [mailto:met_help at ucar.edu
> <javascript:;>]
> > > > > > > > Sent: Friday, August 28, 2015 10:34 AM
> > > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat Config
> > > > > > > > File Question
> > > > > > > >
> > > > > > > > Matt,
> > > > > > > >
> > > > > > > > I received an email from you on August 25 with a
NetCDF
> > > > > > > > file
> > > > > > attachment.
> > > > > > > > Then I replied to you on August 25th in an email with
2
> > > > > > > > attachments ( eodhours_met.nc and eodhours_met.ps) on
> > > > > > > > August
> > 25.
> > > > > > > > Did you receive my reply on 8/25?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Fri, Aug 28, 2015 at 8:19 AM, SITTEL, MATTHEW C CTR
> > > > > > > > USAF AFWA
> > > > > > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>>
wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=7312
> > > > > > > > > 4
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hey John,
> > > > > > > > >
> > > > > > > > > Did you receive the NetCDF file I sent? I put it as
> > > > > > > > > an attachment to an e-mail. AFWA E-mail has been
> > > > > > > > > really messed up here as of late so it's possible
that
> > > > > > > > > it never went out, or that if you replied I haven't
received it.
> > > > > > > > > I just wanted to
> > > > check.
> > > > > > > > >
> > > > > > > > > Matt
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: John Halley Gotway via RT
> > > > > > > > > [mailto:met_help at ucar.edu <javascript:;>]
> > > > > > > > > Sent: Tuesday, August 25, 2015 11:39 AM
> > > > > > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > > > > > > > Subject: Re: [rt.rap.ucar.edu #73124] Grid Stat
Config
> > > > > > > > > File Question
> > > > > > > > >
> > > > > > > > > Matt,
> > > > > > > > >
> > > > > > > > > Are you able to send me that end of day NetCDF file?
> > > > > > > > > Probably easiest for met to get it correct here and
> > > > > > > > > then let
> > > you know.
> > > > > > > > > Also, is this
> > > > > > > > > METv4.1 or met-5.0?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Tue, Aug 25, 2015 at 10:05 AM, SITTEL, MATTHEW C
> > > > > > > > > CTR USAF AFWA
> > > > > > > > > 16 WS/WXN via RT <met_help at ucar.edu <javascript:;>>
wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Tue Aug 25 10:05:00 2015: Request 73124 was acted
upon.
> > > > > > > > > > Transaction: Ticket created by
> > matthew.sittel.ctr at us.af.mil <javascript:;>
> > > > > > > > > > Queue: met_help
> > > > > > > > > > Subject: Grid Stat Config File Question
> > > > > > > > > > Owner: Nobody
> > > > > > > > > > Requestors: matthew.sittel.ctr at us.af.mil
> <javascript:;>
> > > > > > > > > > Status: new
> > > > > > > > > > Ticket <URL:
> > > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=73
> > > > > > > > > > 12
> > > > > > > > > > 4
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I'm trying to figure out the right way to set the
> > > > > > > > > > config file for grid_stat to do the following:
> > > > > > > > > >
> > > > > > > > > > -- generate a mask for my data based on the
contents
> > > > > > > > > > of an external file, in NetCDF format
> > > > > > > > > > -- keying this mask off of a NetCDF variable named
> > > > > > > > > > "End of
> > > day"
> > > > > > > > > > within the .nc file
> > > > > > > > > > -- Keep only those grid points where "End of day"
> > > > > > > > > > equals
> > > > > > > > > > 24
> > > > > > > > > >
> > > > > > > > > > I can't seem to find the right combination of
> > > > > > > > > > equals, semicolons, etc., to do this.
> > > > > > > > > >
> > > > > > > > > > Right now I have:
> > > > > > > > > >
> > > > > > > > > > mask = {
> > > > > > > > > > grid = [
> > > > > > > > > > name =
> > > > > > > > > > "/h/data/global/WXQC/data/met/cfg/eodhours.nc
> > ";
> > > > > > > > > > field = [
> > > > > > > > > > {
> > > > > > > > > > name="End of day";
> > > > > > > > > > cat_thresh=[==24];
> > > > > > > > > > }
> > > > > > > > > > ];
> > > > > > > > > > ];
> > > > > > > > > > poly = [];
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > > The header from the NetCDF file in question:
> > > > > > > > > >
> > > > > > > > > > netcdf eodhours {
> > > > > > > > > > dimensions:
> > > > > > > > > > latitude = 360 ;
> > > > > > > > > > longitude = 720 ;
> > > > > > > > > > variables:
> > > > > > > > > > float latitude(latitude) ;
> > > > > > > > > > latitude:units = "degrees_north" ;
> > > > > > > > > > float longitude(longitude) ;
> > > > > > > > > > longitude:units = "degrees_east" ;
> > > > > > > > > > float End of day(latitude, longitude) ;
> > > > > > > > > > End of day:units = "hours" ; }
> > > > > > > > > >
> > > > > > > > > > The mask syntax I used is incorrect.
> > > > > > > > > >
> > > > > > > > > > For future MET user guides, the following
paragraph
> > > > > > > > > > could really use an example to make this clear:
> > > > > > > > > >
> > > > > > > > > > "Alternatively, any gridded data file that MET can
> > > > > > > > > > read may be used to define a verification masking
region.
> > > > > > > > > > Users must specify a description of the field to
be
> > > > > > > > > > used from the input file and, optionally, may
> > > > > > > > > > specify a threshold to be applied to
> > > > > that field.
> > > > > > > > > > Any grid point where the resulting field is 0, the
> > > > > > > > > > mask is turned
> > > > > > > off.
> > > > > > > > > > Any grid point where it is non-zero, the mask is
> > > > > > > > > > turned
> > on."
> > > > > > > > > >
> > > > > > > > > > Can you please help me get the syntax correct?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Matt
> > > > > > > > > >
> > > > > > > > > > // SIGNED //
> > > > > > > > > > Matthew C. Sittel
> > > > > > > > > > University Corporation for Atmospheric Research
> > > > > > > > > > 16 WS/WXN, 557 WW, Offutt AFB, NE
> > > > > > > > > > matthew.sittel.ctr at us.af.mil <javascript:;>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list