[Met_help] Re: Applying grid_stat to custom fields

[John Halley Gotway]

Jan,

Let me address a few of the points you mentioned.

Regarding the lat/lon values included in the NetCDF files.  MET does not actually use those lat/lon fields at all.  We included them in the test data files for easy plotting with NcView.  So you may 
omit them in your data - which for large datesets will save some significant space.  Instead, MET reads the grid definition parameters in the attributes section.  It instantiates the grid defined by 
those parameters and uses the grid to do a lot of conversions: lat/lon to grid x/y and vice versa.  Unfortunately, there's no easy way to read a field of lat/lon's and "recover" the grid definition 
parameters from them.  So unless we discover a slick way to do that, we'll continue to need them specified in the attributes.

I've been talking to the WRF-ARW folks, and for the release following METv1.1, we'll likely be adding support for the NetCDF output of ARW-Post.  At that point, we'll try to generalize the NetCDF 
support to include a variety of flavors of NetCDF.

For METv1.1, we have required the GRIB code abbreviations to be used for the names of the fields simply for convenience.  It didn't break the existing logic for specifying which fields to verify based 
on grib codes, and it determined the units for the data.  In our plans for the next release, we'll try to revise that requirement and make the list of required elements for NetCDF as short as 
possible, as you suggested.

As for what those grid definition parameters mean, let me first point you to a whole page of examples: http://www.nco.ncep.noaa.gov/pmb/docs/on388/tableb.html
These are predefined grids from NCEP.  Scroll down to the bottom of the page, and you'll see several examples of these types of specifications.  The grid specification parameters differ depending on 
the projection type: lat/lon, lambert conformal, or polar stereographic.

Here's an excerpt from the file where the Lambert Conformal data structure is defined (METv1.0/lib/vx_data_grids/grid_base.h):
struct LambertData {

    char *name;         //identifier

    double p1_deg;      // first latitude where cone intersects earth.
    double p2_deg;      // second latitude where cone intersects earth.

    double p0_deg;      // latitude of lower left (southwest most) gridpoint
    double l0_deg;      // longitude of lower left (southwest most) gridpoint

    double lcen_deg;    // longitude that looks vertical on the grid.

    double d_km;        // "resolution" of grid in km.
    double r_km;        // Radius of sphere.

    int nx;             // number of x gridpoints.
    int ny;             // number of y gridpoints.
};

Reading the comments, you can see a description for what each parameter defines.

Hope that helps.  And thanks for all of your feedback and suggestions.  We really appreciate it.  With each release, we hope to make MET a more general and useful set of tools.

Thanks,
John


Jan Ploski wrote:
> John Halley Gotway wrote:
>> Jan,
>>
>> I understand your frustration.  There's always a desire to support 
>> more data formats.  The reason we chose to begin by supporting GRIB is 
>> that our original statement of work required us to replicate and 
>> extend the existing capabilities of a verification package from NCEP.  
>> Their package only reads GRIB data, so that's where we had to start.
> 
> John,
> 
> Thanks for the explanations. I'm not frustrated, just trying to figure 
> out how things work and (especially) why. These questions I ask are 
> driven by my own ignorance of the meteorological file formats intricacies.
> 
>> I would like to support a more generalized NetCDF format for gridded 
>> data in the future.  However, we will need the grid information 
>> encoded into the NetCDF file for MET to read.  The output of 
>> PCP-Combine is gridded NetCDF, and we do encode the grid definitions 
>> into the header.
> 
> I still do not understand why this information is needed when you get 
> the lat, lon fields in a NetCDF file? Is it simply due to the current 
> implementation of grid_stat/point_stat (they expect the grid definitions 
> to be present, even though they are not used for computing scores) or is 
> there some fundamental dependency on this information somewhere?
> 
> I suspect that this NetCDF/GRIB issue might not be just a matter of 
> supporting NetCDF as another input data format (GRIB being the original 
> one due to your internal requirements, as you described), but rather a 
> deeper design decision with implications on the internal code structure, 
> interfaces, and possible application areas.
> 
> As I see it (possibly incorrectly), one can always convert a GRIB file 
> to the more general (and vague) NetCDF format, but not vice versa. Also, 
> the core of the MET tool suite doesn't really depend on GRIB features - 
> it merely "supports" them. So if you built the tool suite around NetCDF, 
> then taking GRIB input would just seem like a special case - an 
> important one, no doubt. However, it would not artificially restrict the 
> applicability of the tool (I believe it would be even useful for 
> non-meteorological applications).
> 
>> As for a workaround... since PCP-Combine writes out gridded NetCDF 
>> data that Grid-Stat can read, I'd suggest reformatting your NetCDF 
>> files to look like the output of PCP-Combine.  However, rather than 
>> working with METv1.0, I'll give you a beta release of METv1.1 to work 
>> with.  We've generalized the output of PCP-Combine a bit for METv1.1.  
>> I haven't done this myself, but I do know of other users who have.  So 
>> hopefully it'll work alright.
> 
> I suppose that I have to supply the values for these fields:
> 
>                 :Projection = "Lambert Conformal" ;
>                 :p1_deg = "25.000000 degrees_north" ;
>                 :p2_deg = "25.000000 degrees_north" ;
>                 :p0_deg = "12.190000 degrees_north" ;
>                 :l0_deg = "-133.459000 degrees_east" ;
>                 :lcen_deg = "-95.000000 degrees_east" ;
>                 :d_km = "40.635000 km" ;
>                 :r_km = "6367.470000 km" ;
> 
> Can you describe the meaning of these attributes? Do you happen to have 
> a recipe for obtaining their values based on WRF's namelist.wps content?
> 
>> In your NetCDF file, you'll need to name the field "WIND" since that's 
>> the GRIB abbreviation for wind speed 
>> (http://www.nco.ncep.noaa.gov/pmb/docs/on388/table2.html),
> 
> That's a reasonable suggestion, but do I really *have* to, that is, does 
> MET really care whether the variable names are GRIB-compliant?
> 
>> Also, I should mention that we will be supporting additional NetCDF 
>> formats in the future.  Specifically, the WRF-ARW folks have come up 
>> with a PostProcessor that reads WRF-ARW output and writes NetCDF.  
>> It's meant as an alternative to the WRF-PostProcessor (don't ask me 
>> why).  We plan to support their NetCDF output in coming releases.
> 
> Sounds nice. It would be even nicer if you supported anyone's NetCDF 
> output ;-). Just describe what you need as your input interface (and 
> keep it minimal), then everyone, including yourself, can write adapters 
> to plug in their favorite data source!
> 
>> Hope that helps.
>>
>> You can access the latest beta release of METv1.1 at: 
>> ftp://ftp.rap.ucar.edu/incoming/irap/johnhg/METv1.1beta6.20080613.tar.gz
> 
> Thanks, it compiled without problems. The 64bit BUFR issue is still 
> there and I understand your instructions regarding 32bit recompilation, 
> but as we don't plan using PREPBUFR in the near future and it doesn't 
> seem to affect the other tools, I haven't tried following them.
> 
>> While the code is in good shape and we don't anticipate making many 
>> more changes before the release, let me caution you that the MET 
>> User's Guide included in there has not yet been updated to reflect the 
>> additions/fixes.  We're still working on that.
> 
> That sounds great. I especially like your comprehensive documentation, 
> including the meanings of the individual output statistics.
> 
> Regards,
> Jan Ploski
>