[Met_help] NetCDF format
John Halley Gotway
johnhg at ucar.edu
Thu Sep 17 07:45:53 MDT 2009
Chad,
Here you go...
> 1. How does MET expect the grid to be oriented? For example. What corner is the first grid point. Do the rows run north to south or south to north> Do the columns run west to east or east to west?
In the NetCDF file, the variables are sized as (lat, lon)... so really that's (y, x). The lower-left grid point is (0, 0) and the upper-right grid point is (nlat, nlon), where nlat and nlon indicates
the size of the grid. The lat increases from south to north and the lon increases from west to east.
> 2. Is the grid row major (C style) or column major (Fortran style).
I believe it'd be C style row major. When packing all of the data to write into the NetCDF file, we use the following code (taken from pcp_combine.cc):
for(x=0; x<grid.nx(); x++) {
for(y=0; y<grid.ny(); y++) {
pcp_data[wd.two_to_one(x, y)] = wd.get_xy_double(x, y);
}
}
And that "two_to_one" function uses the following equation:
n = y*nx + x;
> 3. I know whether MET is able to eat the NetCDF file, but do you know of any way I can verify that MET is interpreting the data correctly. For example, I might have the grid flipped or rotated, or
the navigation parameters defined incorrectly, even though it is technically correct and is read correctly. Can you recommend any visualization tools that understand the MET NetCDF variables and
global attributes?
Once you've written the NetCDF file, I'd suggest running it through the MODE tool in MET. You could even pass the same file for the "forecast" and "observation". One of MODE's outputs is a
PostScript file that will show you a plot of raw input data. That'll tell you how MET is reading the data to tell whether anything is flipped or rotated.
Hope that helps.
John
Chad Johnson wrote:
> Hi John,
>
> Thanks for all your help the last couple weeks.
>
> I've been working with Holly on trying to get the NDFD data into a format acceptable to the MET software. My first approach was to use cnvgrib to convert the NDFD data to grib version 1. Cnvgrib appeared to perform the conversion and produced output in GRIB1 format. The MET software appears to chew on the data just fine. However, when we attempted to perform verifications on it, the data verified very poorly. Visual spot checks with the original GRIB2 NDFD, which we are able to render and navigate, seemed to verify just fine. This lead me to believe that there is a problem with the conversion from GRIB2 to GRIB1. After further investigations, we discovered that we couldn't view the converted GRIB1 NDFD grid with our GRIB viewing software (which works with all other GRIB1 formats we've come across, including other grids that we convert to grib1 with cnvgrib on a regular basis). The grib C library we are using complains that the BDS uses an unrecognized packing algorithm. Tec
hnically speaking, all of bits 1 through 4 in BDS Octet 4 are set to 1. Referencing table 11, this would indicate the data is "Spherical Harmonic Coefficients" with second order ("Complex") packing, the original data values were integer values and octet 14 contains flag bits 5-12. The grib1 decoding library we use (from NCEP) expects all these bits to be zero. To make matters even more interesting, I know for a fact that the points in the NDFD data are ordered in the grid in a "S" shaped. Meaning, the next row/column does not start at the beginning of the column/row (like a carriage return), but rather proceeds down the next row/column in the opposite direction. NDFD is the only GRIB2 dataset I've come across that orders the points this way. Let me just say, I'm skeptical if cnvgrib takes that into account when it performs the conversions. Anyway...to make a long story short, I'm not sure whether I can trust the conversion of GRIB2 NDFD to GRIB1 with cnvgrib.
>
> I just tested the converted files you sent us and it appears to exhibit the same characteristics as the files we've converted.
>
> So...the next approach I am currently implementing is to write my own converter that will convert the NDFD from GRIB2 directly to the MET NetCDF format. I'm pretty close to writing the NetCDF files right now, but I have a couple questions....
>
> 1. How does MET expect the grid to be oriented? For example. What corner is the first grid point. Do the rows run north to south or south to north> Do the columns run west to east or east to west?
> 2. Is the grid row major (C style) or column major (Fortran style).
> 3. I know whether MET is able to eat the NetCDF file, but do you know of any way I can verify that MET is interpreting the data correctly. For example, I might have the grid flipped or rotated, or the navigation parameters defined incorrectly, even though it is technically correct and is read correctly. Can you recommend any visualization tools that understand the MET NetCDF variables and global attributes?
>
> Thanks for any further assistance you can provide.
>
> -Chad
>
> Chad Johnson
> Software Engineer/Meteorologist
> 608-274-5789
> Weather Central, Inc.
> 401 Charmany Drive
> Madison, WI 53719
> (608) 274-5789
> (608) 273-5854 Fax
> www.weathercentral.tv
> CONFIDENTIALITY NOTICE: THE INFORMATION CONTAINED IN THIS DOCUMENT IS INTENDED ONLY FOR THE PERSONAL AND CONFIDENTIAL USE OF THE DESIGNATED RECIPIENT(S). IF THE READER OF THIS DOCUMENT IS NOT THE INTENDED RECIPIENT OR ANY AGENT RESPONSIBLE FOR DELIVERING IT TO THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED THAT YOU HAVE RECEIVED THIS DOCUMENT IN ERROR, AND THAT ANY REVIEW, DISSEMINATION, DISTRIBUTION OR COPYING OF THIS MESSAGE OR USE OF ITS CONTENTS IS STRICTLY PROHIBITED.
>
>
>
> -----Original Message-----
> From: Holly Hassenzahl
> Sent: Wednesday, September 16, 2009 12:53 PM
> To: Chad Johnson
> Subject: FW: NetCDF format
>
>
>
> ___________________________________
> Holly C. Hassenzahl
> Meteorologist, Science Analyst
> Data Products Group
>
> Weather Central, Inc.
> 401 Charmany Drive Suite 200
> Madison, WI 53711
> +1.608.274.5789
> +1.608.276.4600
> http://www.weathercentral.tv
>
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at ucar.edu]
> Sent: Wednesday, September 16, 2009 10:55 AM
> To: Holly Hassenzahl
> Cc: met_help
> Subject: Re: NetCDF format
>
> Holly,
>
> How are you converting from GRIB2 to GRIB1? We've been using the "cnvgrib" tool to do this conversion (http://www.nco.ncep.noaa.gov/pmb/codes/GRIB2/). There are several steps required to build this
> tool, but once it's built, it seems to work well.
>
> I retrieved a sample file NDFD file from: ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.001-003/ds.maxt.bin
> And I ran the following command to convert from GRIB2 to 1: cnvgrib -g21 ds.maxt.bin ds.maxt.gr
> It seemed to do the trick. I've posted 2 files to the anonymous ftp site:
>
> Original NDFD GRIB2 File: ftp.rap.ucar.edu/incoming/irap/met_help/grib_data/ds.maxt.gr2
> Output of cnvgrib tool: ftp.rap.ucar.edu/incoming/irap/met_help/grib_data/ds.maxt.gr
>
> Feel free to look at them to see whether or not they have the same problem with the data that you're seeing.
>
> I'm looking into when we're planning to support GRIB2 directly - whether it's the next release in winter 2010 or later. I'll let you know what I find out.
>
> Thanks,
> John
>
> Holly Hassenzahl wrote:
>> Hi John-
>>
>> Hope you had a nice time away from the office!
>>
>> I've read here and there that you guys are working on giving MET the
>> ability to read Grib2. Do you have an estimation on when that version
>> might be released (if that's still the plan)?
>>
>> We are having a really tough time getting the NDFD grib2 data converted.
>> We are currently converting to grib1, but just realized that the grib2
>> arrays are packed strangely (like an 'S' structure instead of line by
>> line). Thus when stats are computed, half are correct and half are not.
>> So now we're sort of back at square one on that front. Just got us
>> curious about where things are at with grib2 readability!
>>
>> Thanks,
>> Holly
>>
>> ___________________________________
>> Holly C. Hassenzahl
>> Meteorologist, Science Analyst
>> Data Products Group
>>
>> Weather Central, Inc.
>> 401 Charmany Drive Suite 200
>> Madison, WI 53711
>> +1.608.274.5789
>> +1.608.276.4600
>> http://www.weathercentral.tv
>>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>> Sent: Thursday, September 10, 2009 4:03 PM
>> To: Holly Hassenzahl
>> Cc: met_help at ucar.edu
>> Subject: RE: NetCDF format
>>
>> Holly,
>>
>> I've attached a sample NetCDF header as an example of the global
>> attributes to define the lat/lon grid. This example is a global grid
>> for
>> GFS data. Looks like it consists of the lat/lon for the lower-left
>> corner, the lat/lon increments in degrees, and the total number of
>> lat/lon
>> points. Pretty straightforward.
>>
>> Hopefully that gets you what you need. I'll be out of town Friday
>> through
>> Tuesday of next week. If you have more questions, I'll try to get to
>> them
>> later next week.
>>
>> Good luck.
>>
>> John
>>
>>> Hi John-
>>>
>>> P&G is a rotated equidistant projection. We can remap it to Plate
>>> Caree. Do you have any ncdump output of a MET fcst file using the
>> Plate
>>> Caree projection? We just need to know what all the parameters need
>> to
>>> be called, etc.
>>>
>>> Thanks,
>>> Holly
>>>
>>> ___________________________________
>>> Holly C. Hassenzahl
>>> Meteorologist, Science Analyst
>>> Data Products Group
>>>
>>> Weather Central, Inc.
>>> 401 Charmany Drive Suite 200
>>> Madison, WI 53711
>>> +1.608.274.5789
>>> +1.608.276.4600
>>> http://www.weathercentral.tv
>>>
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>> Sent: Wednesday, September 09, 2009 9:13 PM
>>> To: Holly Hassenzahl
>>> Cc: met_help at ucar.edu
>>> Subject: RE: NetCDF format
>>>
>>> Holly,
>>>
>>> I'm not familiar with "P&G". What is that? The projections currently
>>> implemented in MET are lat/lon (i.e. Plate Carree), lambert conformal,
>>> mercator, and polar stereographic. Would one of those work for you
>>> guys?
>>>
>>> John
>>>
>>>> Thanks, John. I really appreciate your feedback. We will work on
>>>> reformatting the file tomorrow.
>>>>
>>>> The projection we are working with is P&G. One of my colleagues said
>>> he
>>>> could try to remap it to Lambert Conformal, but that it would be
>>> easier
>>>> to remap it to a pseudo-mercator grid. Does MET support
>>>> pseudo-mercator? And if so, what are the global variables that we
>>> would
>>>> need to specify for that projection?
>>>>
>>>> Thanks,
>>>> Holly
>>>>
>>>> ___________________________________
>>>> Holly C. Hassenzahl
>>>> Meteorologist, Science Analyst
>>>> Data Products Group
>>>>
>>>> Weather Central, Inc.
>>>> 401 Charmany Drive Suite 200
>>>> Madison, WI 53711
>>>> +1.608.274.5789
>>>> +1.608.276.4600
>>>> http://www.weathercentral.tv
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>>> Sent: Wednesday, September 09, 2009 3:40 PM
>>>> To: Holly Hassenzahl
>>>> Cc: met_help at ucar.edu
>>>> Subject: Re: NetCDF format
>>>>
>>>> Holly,
>>>>
>>>> Sure thing. Take a look in the directory METv2.0/out/pcp_combine for
>>>> the
>>>> output of the PCP-Combine tool. These were generated in by the MET
>>> test
>>>> scripts. The NetCDF output from PCP-Combine is the format you should
>>>> copy.
>>>>
>>>> The NetCDF format in METv2.0 is rather narrowly defined and wasn't
>>>> originally intended for this sort of use. We're working to provide
>>> more
>>>> general support for NetCDF in the next release.
>>>>
>>>> But as long as you make it look like the output of PCP-Combine, you
>>>> should
>>>> be fine. Here are a few pointers:
>>>>
>>>> - Be sure to name the NetCDF data variables using the GRIB code
>>>> abbreviations listed in this table:
>>>> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table2.html
>>>> - Any spaces in the abbreviation in that table should be removed
>> in
>>>> the
>>>> variable name.
>>>> - For accumulation intervals, append the number of accumulation
>>> hours
>>>> "_NN" to the variable name. For example, 3-hours accumulated precip
>>>> would be APCP_03.
>>>>
>>>> - Be sure to define all the variable attributes you see in the output
>>> of
>>>> PCP-Combine. The "_ut" means unix time, the number of seconds since
>>> Jan
>>>> 1
>>>> 1970. FYI, the command "date +%s" will give you the current unix
>>> time.
>>>> - Lastly, you need to define a bunch of global attributes that define
>>>> the
>>>> grid on which the data resides. The PCP-Combine output gives an
>>> example
>>>> of data on a Lambert Conformal grid. If your data is on a different
>>>> projection, you could email me for help. Or you could find a Grib
>>> file
>>>> that uses that type of projection, run it though PCP-Combine, and
>> then
>>>> look at the global attributes to see how they're defined.
>>>>
>>>> As I mentioned, we realize these conventions are overly restrictive.
>>>> We're working to make it a lot easier to use NetCDF CF-convention
>> data
>>>> in
>>>> future releases of MET.
>>>>
>>>> Thanks,
>>>> John
>>>>> Hello John-
>>>>>
>>>>>
>>>>>
>>>>> I hope this email finds you well. We are hoping to convert one of
>>> our
>>>>> in-house models into NetCDF format so it can be read by Point Stat
>> in
>>>>> MET. Our first attempt gave us the following error: "NetCDF:
>>>>> Attribute not found."
>>>>>
>>>>>
>>>>>
>>>>> Is there a sample NetCDF file you could send me so we have something
>>>> to
>>>>> go off of? Or a list of attributes/variables that must be defined
>> in
>>>>> the files in order to be read by MET?
>>>>>
>>>>>
>>>>>
>>>>> Thank you!
>>>>>
>>>>> Holly
>>>>>
>>>>>
>>>>>
>>>>> ___________________________________
>>>>>
>>>>> Holly C. Hassenzahl
>>>>>
>>>>> Meteorologist, Science Analyst
>>>>>
>>>>> Data Products Group
>>>>>
>>>>>
>>>>>
>>>>> Weather Central, Inc.
>>>>>
>>>>> 401 Charmany Drive Suite 200
>>>>>
>>>>> Madison, WI 53711
>>>>>
>>>>> +1.608.274.5789
>>>>>
>>>>> +1.608.276.4600
>>>>>
>>>>> http://www.weathercentral.tv
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
More information about the Met_help
mailing list