[Met_help] NetCDF format
John Halley Gotway
johnhg at ucar.edu
Fri Sep 18 11:58:42 MDT 2009
Holly,
Here's a section of the header for that NetCDF file. In here, you have an accumulation interval for TMIN set to 1 hour. However, temperature isn't really an "accumulation". The accumulation
attribute is meant for quantities like precipitation... to distinguish between 3 hours of accumulated precip and 6 hours, for example.
float TMIN(lat, lon) ;
TMIN:grib_code = 16 ;
TMIN:units = "kelvin" ;
TMIN:long_name = "Minimum temperature" ;
TMIN:level = "SFC" ;
TMIN:_FillValue = 9999.f ;
TMIN:init_time = "20090911_000000" ;
TMIN:init_time_ut = 1252627200 ;
TMIN:valid_time = "20090911_120000" ;
TMIN:valid_time_ut = 1252670400 ;
TMIN:accum_time = "1 hours" ;
TMIN:accum_time_sec = 3600 ;
If you change the accumulation attributes to 0, MODE should run fine on it:
TMIN:accum_time = "0 hours" ;
TMIN:accum_time_sec = 0 ;
Thanks,
John
Holly Hassenzahl wrote:
> Hi John-
>
> Thanks for your patience, I was out of the office for a bit. The files
> you requested should be on the ftp site in the hassenzahl_data folder.
>
> Thanks!
> Holly
>
> ___________________________________
> Holly C. Hassenzahl
> Meteorologist, Science Analyst
> Data Products Group
>
> Weather Central, Inc.
> 401 Charmany Drive Suite 200
> Madison, WI 53711
> +1.608.274.5789
> +1.608.276.4600
> http://www.weathercentral.tv
>
>
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at ucar.edu]
> Sent: Friday, September 18, 2009 10:04 AM
> To: Holly Hassenzahl
> Subject: Re: NetCDF format
>
> Holly,
>
> Please send me the NetCDF file (data/fcst/ndfd/Day1/fcst00z_12L_TMIN.nc)
> and the MODE configuration file you're trying to use. And I'll be happy
> to take a look.
>
> John
>
> Holly Hassenzahl wrote:
>> Hi John-
>>
>> One quick question for you. I tried running our new NetCDF file
> through
>> MODE and it's giving me the following error:
>>
>> ERROR: process_fcst_obs_files() -> accumulation time of 1 hours in the
>> forecast file (data/fcst/ndfd/Day1/fcst00z_12L_TMIN.nc) does not match
>> the 0 hour accumulation time requested in the config file.
>>
>> I am using the same forecast file as both the "fcst" and "obs" input.
> I
>> didn't see any time-related variables in the MODE config file that I
>> could adjust, so I'm not quite sure where to go from here.
>>
>> Any suggestions?
>> Thanks,
>> Holly
>>
>> ___________________________________
>> Holly C. Hassenzahl
>> Meteorologist, Science Analyst
>> Data Products Group
>>
>> Weather Central, Inc.
>> 401 Charmany Drive Suite 200
>> Madison, WI 53711
>> +1.608.274.5789
>> +1.608.276.4600
>> http://www.weathercentral.tv
>>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>> Sent: Thursday, September 17, 2009 8:46 AM
>> To: Chad Johnson
>> Cc: Holly Hassenzahl; met_help
>> Subject: Re: NetCDF format
>>
>> Chad,
>>
>> Here you go...
>>
>>> 1. How does MET expect the grid to be oriented? For example. What
>> corner is the first grid point. Do the rows run north to south or
> south
>> to north> Do the columns run west to east or east to west?
>>
>> In the NetCDF file, the variables are sized as (lat, lon)... so really
>> that's (y, x). The lower-left grid point is (0, 0) and the
> upper-right
>> grid point is (nlat, nlon), where nlat and nlon indicates
>> the size of the grid. The lat increases from south to north and the
> lon
>> increases from west to east.
>>
>>> 2. Is the grid row major (C style) or column major (Fortran style).
>> I believe it'd be C style row major. When packing all of the data to
>> write into the NetCDF file, we use the following code (taken from
>> pcp_combine.cc):
>>
>> for(x=0; x<grid.nx(); x++) {
>> for(y=0; y<grid.ny(); y++) {
>> pcp_data[wd.two_to_one(x, y)] = wd.get_xy_double(x, y);
>> }
>> }
>> And that "two_to_one" function uses the following equation:
>> n = y*nx + x;
>>
>>> 3. I know whether MET is able to eat the NetCDF file, but do you know
>> of any way I can verify that MET is interpreting the data correctly.
> For
>> example, I might have the grid flipped or rotated, or
>> the navigation parameters defined incorrectly, even though it is
>> technically correct and is read correctly. Can you recommend any
>> visualization tools that understand the MET NetCDF variables and
>> global attributes?
>>
>> Once you've written the NetCDF file, I'd suggest running it through
> the
>> MODE tool in MET. You could even pass the same file for the
> "forecast"
>> and "observation". One of MODE's outputs is a
>> PostScript file that will show you a plot of raw input data. That'll
>> tell you how MET is reading the data to tell whether anything is
> flipped
>> or rotated.
>>
>> Hope that helps.
>>
>> John
>>
>> Chad Johnson wrote:
>>> Hi John,
>>>
>>> Thanks for all your help the last couple weeks.
>>>
>>> I've been working with Holly on trying to get the NDFD data into a
>> format acceptable to the MET software. My first approach was to use
>> cnvgrib to convert the NDFD data to grib version 1. Cnvgrib appeared
> to
>> perform the conversion and produced output in GRIB1 format. The MET
>> software appears to chew on the data just fine. However, when we
>> attempted to perform verifications on it, the data verified very
> poorly.
>> Visual spot checks with the original GRIB2 NDFD, which we are able to
>> render and navigate, seemed to verify just fine. This lead me to
> believe
>> that there is a problem with the conversion from GRIB2 to GRIB1. After
>> further investigations, we discovered that we couldn't view the
>> converted GRIB1 NDFD grid with our GRIB viewing software (which works
>> with all other GRIB1 formats we've come across, including other grids
>> that we convert to grib1 with cnvgrib on a regular basis). The grib C
>> library we are using complains that the BDS uses an unrecognized
> packing
>> algorithm. Tec
>> hnically speaking, all of bits 1 through 4 in BDS Octet 4 are set to
> 1.
>> Referencing table 11, this would indicate the data is "Spherical
>> Harmonic Coefficients" with second order ("Complex") packing, the
>> original data values were integer values and octet 14 contains flag
> bits
>> 5-12. The grib1 decoding library we use (from NCEP) expects all these
>> bits to be zero. To make matters even more interesting, I know for a
>> fact that the points in the NDFD data are ordered in the grid in a "S"
>> shaped. Meaning, the next row/column does not start at the beginning
> of
>> the column/row (like a carriage return), but rather proceeds down the
>> next row/column in the opposite direction. NDFD is the only GRIB2
>> dataset I've come across that orders the points this way. Let me just
>> say, I'm skeptical if cnvgrib takes that into account when it performs
>> the conversions. Anyway...to make a long story short, I'm not sure
>> whether I can trust the conversion of GRIB2 NDFD to GRIB1 with
> cnvgrib.
>>> I just tested the converted files you sent us and it appears to
>> exhibit the same characteristics as the files we've converted.
>>> So...the next approach I am currently implementing is to write my own
>> converter that will convert the NDFD from GRIB2 directly to the MET
>> NetCDF format. I'm pretty close to writing the NetCDF files right now,
>> but I have a couple questions....
>>> 1. How does MET expect the grid to be oriented? For example. What
>> corner is the first grid point. Do the rows run north to south or
> south
>> to north> Do the columns run west to east or east to west?
>>> 2. Is the grid row major (C style) or column major (Fortran style).
>>> 3. I know whether MET is able to eat the NetCDF file, but do you know
>> of any way I can verify that MET is interpreting the data correctly.
> For
>> example, I might have the grid flipped or rotated, or the navigation
>> parameters defined incorrectly, even though it is technically correct
>> and is read correctly. Can you recommend any visualization tools that
>> understand the MET NetCDF variables and global attributes?
>>> Thanks for any further assistance you can provide.
>>>
>>> -Chad
>>>
>>> Chad Johnson
>>> Software Engineer/Meteorologist
>>> 608-274-5789
>>> Weather Central, Inc.
>>> 401 Charmany Drive
>>> Madison, WI 53719
>>> (608) 274-5789
>>> (608) 273-5854 Fax
>>> www.weathercentral.tv
>>> CONFIDENTIALITY NOTICE: THE INFORMATION CONTAINED IN THIS DOCUMENT IS
>> INTENDED ONLY FOR THE PERSONAL AND CONFIDENTIAL USE OF THE DESIGNATED
>> RECIPIENT(S). IF THE READER OF THIS DOCUMENT IS NOT THE INTENDED
>> RECIPIENT OR ANY AGENT RESPONSIBLE FOR DELIVERING IT TO THE INTENDED
>> RECIPIENT, YOU ARE HEREBY NOTIFIED THAT YOU HAVE RECEIVED THIS
> DOCUMENT
>> IN ERROR, AND THAT ANY REVIEW, DISSEMINATION, DISTRIBUTION OR COPYING
> OF
>> THIS MESSAGE OR USE OF ITS CONTENTS IS STRICTLY PROHIBITED.
>>>
>>> -----Original Message-----
>>> From: Holly Hassenzahl
>>> Sent: Wednesday, September 16, 2009 12:53 PM
>>> To: Chad Johnson
>>> Subject: FW: NetCDF format
>>>
>>>
>>>
>>> ___________________________________
>>> Holly C. Hassenzahl
>>> Meteorologist, Science Analyst
>>> Data Products Group
>>>
>>> Weather Central, Inc.
>>> 401 Charmany Drive Suite 200
>>> Madison, WI 53711
>>> +1.608.274.5789
>>> +1.608.276.4600
>>> http://www.weathercentral.tv
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>> Sent: Wednesday, September 16, 2009 10:55 AM
>>> To: Holly Hassenzahl
>>> Cc: met_help
>>> Subject: Re: NetCDF format
>>>
>>> Holly,
>>>
>>> How are you converting from GRIB2 to GRIB1? We've been using the
>> "cnvgrib" tool to do this conversion
>> (http://www.nco.ncep.noaa.gov/pmb/codes/GRIB2/). There are several
>> steps required to build this
>>> tool, but once it's built, it seems to work well.
>>>
>>> I retrieved a sample file NDFD file from:
> ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.
>> 001-003/ds.maxt.bin
>>> And I ran the following command to convert from GRIB2 to 1: cnvgrib
>> -g21 ds.maxt.bin ds.maxt.gr
>>> It seemed to do the trick. I've posted 2 files to the anonymous ftp
>> site:
>>> Original NDFD GRIB2 File:
>> ftp.rap.ucar.edu/incoming/irap/met_help/grib_data/ds.maxt.gr2
>>> Output of cnvgrib tool:
>> ftp.rap.ucar.edu/incoming/irap/met_help/grib_data/ds.maxt.gr
>>> Feel free to look at them to see whether or not they have the same
>> problem with the data that you're seeing.
>>> I'm looking into when we're planning to support GRIB2 directly -
>> whether it's the next release in winter 2010 or later. I'll let you
>> know what I find out.
>>> Thanks,
>>> John
>>>
>>> Holly Hassenzahl wrote:
>>>> Hi John-
>>>>
>>>> Hope you had a nice time away from the office!
>>>>
>>>> I've read here and there that you guys are working on giving MET the
>>>> ability to read Grib2. Do you have an estimation on when that
>> version
>>>> might be released (if that's still the plan)?
>>>>
>>>> We are having a really tough time getting the NDFD grib2 data
>> converted.
>>>> We are currently converting to grib1, but just realized that the
>> grib2
>>>> arrays are packed strangely (like an 'S' structure instead of line
> by
>>>> line). Thus when stats are computed, half are correct and half are
>> not.
>>>> So now we're sort of back at square one on that front. Just got us
>>>> curious about where things are at with grib2 readability!
>>>>
>>>> Thanks,
>>>> Holly
>>>>
>>>> ___________________________________
>>>> Holly C. Hassenzahl
>>>> Meteorologist, Science Analyst
>>>> Data Products Group
>>>>
>>>> Weather Central, Inc.
>>>> 401 Charmany Drive Suite 200
>>>> Madison, WI 53711
>>>> +1.608.274.5789
>>>> +1.608.276.4600
>>>> http://www.weathercentral.tv
>>>>
>>>> -----Original Message-----
>>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>>> Sent: Thursday, September 10, 2009 4:03 PM
>>>> To: Holly Hassenzahl
>>>> Cc: met_help at ucar.edu
>>>> Subject: RE: NetCDF format
>>>>
>>>> Holly,
>>>>
>>>> I've attached a sample NetCDF header as an example of the global
>>>> attributes to define the lat/lon grid. This example is a global
> grid
>>>> for
>>>> GFS data. Looks like it consists of the lat/lon for the lower-left
>>>> corner, the lat/lon increments in degrees, and the total number of
>>>> lat/lon
>>>> points. Pretty straightforward.
>>>>
>>>> Hopefully that gets you what you need. I'll be out of town Friday
>>>> through
>>>> Tuesday of next week. If you have more questions, I'll try to get
> to
>>>> them
>>>> later next week.
>>>>
>>>> Good luck.
>>>>
>>>> John
>>>>
>>>>> Hi John-
>>>>>
>>>>> P&G is a rotated equidistant projection. We can remap it to Plate
>>>>> Caree. Do you have any ncdump output of a MET fcst file using the
>>>> Plate
>>>>> Caree projection? We just need to know what all the parameters
> need
>>>> to
>>>>> be called, etc.
>>>>>
>>>>> Thanks,
>>>>> Holly
>>>>>
>>>>> ___________________________________
>>>>> Holly C. Hassenzahl
>>>>> Meteorologist, Science Analyst
>>>>> Data Products Group
>>>>>
>>>>> Weather Central, Inc.
>>>>> 401 Charmany Drive Suite 200
>>>>> Madison, WI 53711
>>>>> +1.608.274.5789
>>>>> +1.608.276.4600
>>>>> http://www.weathercentral.tv
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>>>> Sent: Wednesday, September 09, 2009 9:13 PM
>>>>> To: Holly Hassenzahl
>>>>> Cc: met_help at ucar.edu
>>>>> Subject: RE: NetCDF format
>>>>>
>>>>> Holly,
>>>>>
>>>>> I'm not familiar with "P&G". What is that? The projections
>> currently
>>>>> implemented in MET are lat/lon (i.e. Plate Carree), lambert
>> conformal,
>>>>> mercator, and polar stereographic. Would one of those work for you
>>>>> guys?
>>>>>
>>>>> John
>>>>>
>>>>>> Thanks, John. I really appreciate your feedback. We will work on
>>>>>> reformatting the file tomorrow.
>>>>>>
>>>>>> The projection we are working with is P&G. One of my colleagues
>> said
>>>>> he
>>>>>> could try to remap it to Lambert Conformal, but that it would be
>>>>> easier
>>>>>> to remap it to a pseudo-mercator grid. Does MET support
>>>>>> pseudo-mercator? And if so, what are the global variables that we
>>>>> would
>>>>>> need to specify for that projection?
>>>>>>
>>>>>> Thanks,
>>>>>> Holly
>>>>>>
>>>>>> ___________________________________
>>>>>> Holly C. Hassenzahl
>>>>>> Meteorologist, Science Analyst
>>>>>> Data Products Group
>>>>>>
>>>>>> Weather Central, Inc.
>>>>>> 401 Charmany Drive Suite 200
>>>>>> Madison, WI 53711
>>>>>> +1.608.274.5789
>>>>>> +1.608.276.4600
>>>>>> http://www.weathercentral.tv
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>>>>> Sent: Wednesday, September 09, 2009 3:40 PM
>>>>>> To: Holly Hassenzahl
>>>>>> Cc: met_help at ucar.edu
>>>>>> Subject: Re: NetCDF format
>>>>>>
>>>>>> Holly,
>>>>>>
>>>>>> Sure thing. Take a look in the directory METv2.0/out/pcp_combine
>> for
>>>>>> the
>>>>>> output of the PCP-Combine tool. These were generated in by the
> MET
>>>>> test
>>>>>> scripts. The NetCDF output from PCP-Combine is the format you
>> should
>>>>>> copy.
>>>>>>
>>>>>> The NetCDF format in METv2.0 is rather narrowly defined and wasn't
>>>>>> originally intended for this sort of use. We're working to
> provide
>>>>> more
>>>>>> general support for NetCDF in the next release.
>>>>>>
>>>>>> But as long as you make it look like the output of PCP-Combine,
> you
>>>>>> should
>>>>>> be fine. Here are a few pointers:
>>>>>>
>>>>>> - Be sure to name the NetCDF data variables using the GRIB code
>>>>>> abbreviations listed in this table:
>>>>>> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table2.html
>>>>>> - Any spaces in the abbreviation in that table should be
> removed
>>>> in
>>>>>> the
>>>>>> variable name.
>>>>>> - For accumulation intervals, append the number of accumulation
>>>>> hours
>>>>>> "_NN" to the variable name. For example, 3-hours accumulated
>> precip
>>>>>> would be APCP_03.
>>>>>>
>>>>>> - Be sure to define all the variable attributes you see in the
>> output
>>>>> of
>>>>>> PCP-Combine. The "_ut" means unix time, the number of seconds
>> since
>>>>> Jan
>>>>>> 1
>>>>>> 1970. FYI, the command "date +%s" will give you the current unix
>>>>> time.
>>>>>> - Lastly, you need to define a bunch of global attributes that
>> define
>>>>>> the
>>>>>> grid on which the data resides. The PCP-Combine output gives an
>>>>> example
>>>>>> of data on a Lambert Conformal grid. If your data is on a
>> different
>>>>>> projection, you could email me for help. Or you could find a Grib
>>>>> file
>>>>>> that uses that type of projection, run it though PCP-Combine, and
>>>> then
>>>>>> look at the global attributes to see how they're defined.
>>>>>>
>>>>>> As I mentioned, we realize these conventions are overly
>> restrictive.
>>>>>> We're working to make it a lot easier to use NetCDF CF-convention
>>>> data
>>>>>> in
>>>>>> future releases of MET.
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>> Hello John-
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I hope this email finds you well. We are hoping to convert one
> of
>>>>> our
>>>>>>> in-house models into NetCDF format so it can be read by Point
> Stat
>>>> in
>>>>>>> MET. Our first attempt gave us the following error: "NetCDF:
>>>>>>> Attribute not found."
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Is there a sample NetCDF file you could send me so we have
>> something
>>>>>> to
>>>>>>> go off of? Or a list of attributes/variables that must be
> defined
>>>> in
>>>>>>> the files in order to be read by MET?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thank you!
>>>>>>>
>>>>>>> Holly
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ___________________________________
>>>>>>>
>>>>>>> Holly C. Hassenzahl
>>>>>>>
>>>>>>> Meteorologist, Science Analyst
>>>>>>>
>>>>>>> Data Products Group
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Weather Central, Inc.
>>>>>>>
>>>>>>> 401 Charmany Drive Suite 200
>>>>>>>
>>>>>>> Madison, WI 53711
>>>>>>>
>>>>>>> +1.608.274.5789
>>>>>>>
>>>>>>> +1.608.276.4600
>>>>>>>
>>>>>>> http://www.weathercentral.tv
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
More information about the Met_help
mailing list