[Met_help] NetCDF format
Holly Hassenzahl
hhassenzahl at wxc.com
Fri Sep 18 09:02:04 MDT 2009
Hi John-
One quick question for you. I tried running our new NetCDF file through
MODE and it's giving me the following error:
ERROR: process_fcst_obs_files() -> accumulation time of 1 hours in the
forecast file (data/fcst/ndfd/Day1/fcst00z_12L_TMIN.nc) does not match
the 0 hour accumulation time requested in the config file.
I am using the same forecast file as both the "fcst" and "obs" input. I
didn't see any time-related variables in the MODE config file that I
could adjust, so I'm not quite sure where to go from here.
Any suggestions?
Thanks,
Holly
___________________________________
Holly C. Hassenzahl
Meteorologist, Science Analyst
Data Products Group
Weather Central, Inc.
401 Charmany Drive Suite 200
Madison, WI 53711
+1.608.274.5789
+1.608.276.4600
http://www.weathercentral.tv
-----Original Message-----
From: John Halley Gotway [mailto:johnhg at ucar.edu]
Sent: Thursday, September 17, 2009 8:46 AM
To: Chad Johnson
Cc: Holly Hassenzahl; met_help
Subject: Re: NetCDF format
Chad,
Here you go...
> 1. How does MET expect the grid to be oriented? For example. What
corner is the first grid point. Do the rows run north to south or south
to north> Do the columns run west to east or east to west?
In the NetCDF file, the variables are sized as (lat, lon)... so really
that's (y, x). The lower-left grid point is (0, 0) and the upper-right
grid point is (nlat, nlon), where nlat and nlon indicates
the size of the grid. The lat increases from south to north and the lon
increases from west to east.
> 2. Is the grid row major (C style) or column major (Fortran style).
I believe it'd be C style row major. When packing all of the data to
write into the NetCDF file, we use the following code (taken from
pcp_combine.cc):
for(x=0; x<grid.nx(); x++) {
for(y=0; y<grid.ny(); y++) {
pcp_data[wd.two_to_one(x, y)] = wd.get_xy_double(x, y);
}
}
And that "two_to_one" function uses the following equation:
n = y*nx + x;
> 3. I know whether MET is able to eat the NetCDF file, but do you know
of any way I can verify that MET is interpreting the data correctly. For
example, I might have the grid flipped or rotated, or
the navigation parameters defined incorrectly, even though it is
technically correct and is read correctly. Can you recommend any
visualization tools that understand the MET NetCDF variables and
global attributes?
Once you've written the NetCDF file, I'd suggest running it through the
MODE tool in MET. You could even pass the same file for the "forecast"
and "observation". One of MODE's outputs is a
PostScript file that will show you a plot of raw input data. That'll
tell you how MET is reading the data to tell whether anything is flipped
or rotated.
Hope that helps.
John
Chad Johnson wrote:
> Hi John,
>
> Thanks for all your help the last couple weeks.
>
> I've been working with Holly on trying to get the NDFD data into a
format acceptable to the MET software. My first approach was to use
cnvgrib to convert the NDFD data to grib version 1. Cnvgrib appeared to
perform the conversion and produced output in GRIB1 format. The MET
software appears to chew on the data just fine. However, when we
attempted to perform verifications on it, the data verified very poorly.
Visual spot checks with the original GRIB2 NDFD, which we are able to
render and navigate, seemed to verify just fine. This lead me to believe
that there is a problem with the conversion from GRIB2 to GRIB1. After
further investigations, we discovered that we couldn't view the
converted GRIB1 NDFD grid with our GRIB viewing software (which works
with all other GRIB1 formats we've come across, including other grids
that we convert to grib1 with cnvgrib on a regular basis). The grib C
library we are using complains that the BDS uses an unrecognized packing
algorithm. Tec
hnically speaking, all of bits 1 through 4 in BDS Octet 4 are set to 1.
Referencing table 11, this would indicate the data is "Spherical
Harmonic Coefficients" with second order ("Complex") packing, the
original data values were integer values and octet 14 contains flag bits
5-12. The grib1 decoding library we use (from NCEP) expects all these
bits to be zero. To make matters even more interesting, I know for a
fact that the points in the NDFD data are ordered in the grid in a "S"
shaped. Meaning, the next row/column does not start at the beginning of
the column/row (like a carriage return), but rather proceeds down the
next row/column in the opposite direction. NDFD is the only GRIB2
dataset I've come across that orders the points this way. Let me just
say, I'm skeptical if cnvgrib takes that into account when it performs
the conversions. Anyway...to make a long story short, I'm not sure
whether I can trust the conversion of GRIB2 NDFD to GRIB1 with cnvgrib.
>
> I just tested the converted files you sent us and it appears to
exhibit the same characteristics as the files we've converted.
>
> So...the next approach I am currently implementing is to write my own
converter that will convert the NDFD from GRIB2 directly to the MET
NetCDF format. I'm pretty close to writing the NetCDF files right now,
but I have a couple questions....
>
> 1. How does MET expect the grid to be oriented? For example. What
corner is the first grid point. Do the rows run north to south or south
to north> Do the columns run west to east or east to west?
> 2. Is the grid row major (C style) or column major (Fortran style).
> 3. I know whether MET is able to eat the NetCDF file, but do you know
of any way I can verify that MET is interpreting the data correctly. For
example, I might have the grid flipped or rotated, or the navigation
parameters defined incorrectly, even though it is technically correct
and is read correctly. Can you recommend any visualization tools that
understand the MET NetCDF variables and global attributes?
>
> Thanks for any further assistance you can provide.
>
> -Chad
>
> Chad Johnson
> Software Engineer/Meteorologist
> 608-274-5789
> Weather Central, Inc.
> 401 Charmany Drive
> Madison, WI 53719
> (608) 274-5789
> (608) 273-5854 Fax
> www.weathercentral.tv
> CONFIDENTIALITY NOTICE: THE INFORMATION CONTAINED IN THIS DOCUMENT IS
INTENDED ONLY FOR THE PERSONAL AND CONFIDENTIAL USE OF THE DESIGNATED
RECIPIENT(S). IF THE READER OF THIS DOCUMENT IS NOT THE INTENDED
RECIPIENT OR ANY AGENT RESPONSIBLE FOR DELIVERING IT TO THE INTENDED
RECIPIENT, YOU ARE HEREBY NOTIFIED THAT YOU HAVE RECEIVED THIS DOCUMENT
IN ERROR, AND THAT ANY REVIEW, DISSEMINATION, DISTRIBUTION OR COPYING OF
THIS MESSAGE OR USE OF ITS CONTENTS IS STRICTLY PROHIBITED.
>
>
>
> -----Original Message-----
> From: Holly Hassenzahl
> Sent: Wednesday, September 16, 2009 12:53 PM
> To: Chad Johnson
> Subject: FW: NetCDF format
>
>
>
> ___________________________________
> Holly C. Hassenzahl
> Meteorologist, Science Analyst
> Data Products Group
>
> Weather Central, Inc.
> 401 Charmany Drive Suite 200
> Madison, WI 53711
> +1.608.274.5789
> +1.608.276.4600
> http://www.weathercentral.tv
>
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at ucar.edu]
> Sent: Wednesday, September 16, 2009 10:55 AM
> To: Holly Hassenzahl
> Cc: met_help
> Subject: Re: NetCDF format
>
> Holly,
>
> How are you converting from GRIB2 to GRIB1? We've been using the
"cnvgrib" tool to do this conversion
(http://www.nco.ncep.noaa.gov/pmb/codes/GRIB2/). There are several
steps required to build this
> tool, but once it's built, it seems to work well.
>
> I retrieved a sample file NDFD file from:
ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.
001-003/ds.maxt.bin
> And I ran the following command to convert from GRIB2 to 1: cnvgrib
-g21 ds.maxt.bin ds.maxt.gr
> It seemed to do the trick. I've posted 2 files to the anonymous ftp
site:
>
> Original NDFD GRIB2 File:
ftp.rap.ucar.edu/incoming/irap/met_help/grib_data/ds.maxt.gr2
> Output of cnvgrib tool:
ftp.rap.ucar.edu/incoming/irap/met_help/grib_data/ds.maxt.gr
>
> Feel free to look at them to see whether or not they have the same
problem with the data that you're seeing.
>
> I'm looking into when we're planning to support GRIB2 directly -
whether it's the next release in winter 2010 or later. I'll let you
know what I find out.
>
> Thanks,
> John
>
> Holly Hassenzahl wrote:
>> Hi John-
>>
>> Hope you had a nice time away from the office!
>>
>> I've read here and there that you guys are working on giving MET the
>> ability to read Grib2. Do you have an estimation on when that
version
>> might be released (if that's still the plan)?
>>
>> We are having a really tough time getting the NDFD grib2 data
converted.
>> We are currently converting to grib1, but just realized that the
grib2
>> arrays are packed strangely (like an 'S' structure instead of line by
>> line). Thus when stats are computed, half are correct and half are
not.
>> So now we're sort of back at square one on that front. Just got us
>> curious about where things are at with grib2 readability!
>>
>> Thanks,
>> Holly
>>
>> ___________________________________
>> Holly C. Hassenzahl
>> Meteorologist, Science Analyst
>> Data Products Group
>>
>> Weather Central, Inc.
>> 401 Charmany Drive Suite 200
>> Madison, WI 53711
>> +1.608.274.5789
>> +1.608.276.4600
>> http://www.weathercentral.tv
>>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>> Sent: Thursday, September 10, 2009 4:03 PM
>> To: Holly Hassenzahl
>> Cc: met_help at ucar.edu
>> Subject: RE: NetCDF format
>>
>> Holly,
>>
>> I've attached a sample NetCDF header as an example of the global
>> attributes to define the lat/lon grid. This example is a global grid
>> for
>> GFS data. Looks like it consists of the lat/lon for the lower-left
>> corner, the lat/lon increments in degrees, and the total number of
>> lat/lon
>> points. Pretty straightforward.
>>
>> Hopefully that gets you what you need. I'll be out of town Friday
>> through
>> Tuesday of next week. If you have more questions, I'll try to get to
>> them
>> later next week.
>>
>> Good luck.
>>
>> John
>>
>>> Hi John-
>>>
>>> P&G is a rotated equidistant projection. We can remap it to Plate
>>> Caree. Do you have any ncdump output of a MET fcst file using the
>> Plate
>>> Caree projection? We just need to know what all the parameters need
>> to
>>> be called, etc.
>>>
>>> Thanks,
>>> Holly
>>>
>>> ___________________________________
>>> Holly C. Hassenzahl
>>> Meteorologist, Science Analyst
>>> Data Products Group
>>>
>>> Weather Central, Inc.
>>> 401 Charmany Drive Suite 200
>>> Madison, WI 53711
>>> +1.608.274.5789
>>> +1.608.276.4600
>>> http://www.weathercentral.tv
>>>
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>> Sent: Wednesday, September 09, 2009 9:13 PM
>>> To: Holly Hassenzahl
>>> Cc: met_help at ucar.edu
>>> Subject: RE: NetCDF format
>>>
>>> Holly,
>>>
>>> I'm not familiar with "P&G". What is that? The projections
currently
>>> implemented in MET are lat/lon (i.e. Plate Carree), lambert
conformal,
>>> mercator, and polar stereographic. Would one of those work for you
>>> guys?
>>>
>>> John
>>>
>>>> Thanks, John. I really appreciate your feedback. We will work on
>>>> reformatting the file tomorrow.
>>>>
>>>> The projection we are working with is P&G. One of my colleagues
said
>>> he
>>>> could try to remap it to Lambert Conformal, but that it would be
>>> easier
>>>> to remap it to a pseudo-mercator grid. Does MET support
>>>> pseudo-mercator? And if so, what are the global variables that we
>>> would
>>>> need to specify for that projection?
>>>>
>>>> Thanks,
>>>> Holly
>>>>
>>>> ___________________________________
>>>> Holly C. Hassenzahl
>>>> Meteorologist, Science Analyst
>>>> Data Products Group
>>>>
>>>> Weather Central, Inc.
>>>> 401 Charmany Drive Suite 200
>>>> Madison, WI 53711
>>>> +1.608.274.5789
>>>> +1.608.276.4600
>>>> http://www.weathercentral.tv
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>>> Sent: Wednesday, September 09, 2009 3:40 PM
>>>> To: Holly Hassenzahl
>>>> Cc: met_help at ucar.edu
>>>> Subject: Re: NetCDF format
>>>>
>>>> Holly,
>>>>
>>>> Sure thing. Take a look in the directory METv2.0/out/pcp_combine
for
>>>> the
>>>> output of the PCP-Combine tool. These were generated in by the MET
>>> test
>>>> scripts. The NetCDF output from PCP-Combine is the format you
should
>>>> copy.
>>>>
>>>> The NetCDF format in METv2.0 is rather narrowly defined and wasn't
>>>> originally intended for this sort of use. We're working to provide
>>> more
>>>> general support for NetCDF in the next release.
>>>>
>>>> But as long as you make it look like the output of PCP-Combine, you
>>>> should
>>>> be fine. Here are a few pointers:
>>>>
>>>> - Be sure to name the NetCDF data variables using the GRIB code
>>>> abbreviations listed in this table:
>>>> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table2.html
>>>> - Any spaces in the abbreviation in that table should be removed
>> in
>>>> the
>>>> variable name.
>>>> - For accumulation intervals, append the number of accumulation
>>> hours
>>>> "_NN" to the variable name. For example, 3-hours accumulated
precip
>>>> would be APCP_03.
>>>>
>>>> - Be sure to define all the variable attributes you see in the
output
>>> of
>>>> PCP-Combine. The "_ut" means unix time, the number of seconds
since
>>> Jan
>>>> 1
>>>> 1970. FYI, the command "date +%s" will give you the current unix
>>> time.
>>>> - Lastly, you need to define a bunch of global attributes that
define
>>>> the
>>>> grid on which the data resides. The PCP-Combine output gives an
>>> example
>>>> of data on a Lambert Conformal grid. If your data is on a
different
>>>> projection, you could email me for help. Or you could find a Grib
>>> file
>>>> that uses that type of projection, run it though PCP-Combine, and
>> then
>>>> look at the global attributes to see how they're defined.
>>>>
>>>> As I mentioned, we realize these conventions are overly
restrictive.
>>>> We're working to make it a lot easier to use NetCDF CF-convention
>> data
>>>> in
>>>> future releases of MET.
>>>>
>>>> Thanks,
>>>> John
>>>>> Hello John-
>>>>>
>>>>>
>>>>>
>>>>> I hope this email finds you well. We are hoping to convert one of
>>> our
>>>>> in-house models into NetCDF format so it can be read by Point Stat
>> in
>>>>> MET. Our first attempt gave us the following error: "NetCDF:
>>>>> Attribute not found."
>>>>>
>>>>>
>>>>>
>>>>> Is there a sample NetCDF file you could send me so we have
something
>>>> to
>>>>> go off of? Or a list of attributes/variables that must be defined
>> in
>>>>> the files in order to be read by MET?
>>>>>
>>>>>
>>>>>
>>>>> Thank you!
>>>>>
>>>>> Holly
>>>>>
>>>>>
>>>>>
>>>>> ___________________________________
>>>>>
>>>>> Holly C. Hassenzahl
>>>>>
>>>>> Meteorologist, Science Analyst
>>>>>
>>>>> Data Products Group
>>>>>
>>>>>
>>>>>
>>>>> Weather Central, Inc.
>>>>>
>>>>> 401 Charmany Drive Suite 200
>>>>>
>>>>> Madison, WI 53711
>>>>>
>>>>> +1.608.274.5789
>>>>>
>>>>> +1.608.276.4600
>>>>>
>>>>> http://www.weathercentral.tv
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
More information about the Met_help
mailing list