[Met_help] BUFR, NetCDF files in MET
John Halley Gotway
johnhg at rap.ucar.edu
Thu Dec 6 09:21:59 MST 2007
Good question. Providing the fill value of -9999 for most of those will be fine. MET does not use most of them. I've retained that info since it's present in the PrepBufr data and NCEP has some use
As for the forecast value... apparently, in the verification code that NCEP uses, it processes through the observation files and appends the nearest forecast value to each observation. So eventually,
the PrepBufr file they're using contains matched pair data. MET, on the other hand, just uses the observations from the PrepBufr files. We don't "add" data them during the processing.
As for the analyzed value... to be honest I'm not sure how NCEP uses this. Let me know if you're interested though, and I could find out.
The MET utility "pb2nc" allows users to filter out the observation values they'd like to used based on all of these parameters. For example, the user could specify a quality mark threshold to apply
in pb2nc. Then, only those observations which meet that quality mark criteria will be retained. So that's why we write out all of these fields.
Let me attempt to indicate which fields actually need to contain valid data to be used by "point_stat":
obs_arr:hdr_id = header index -> YES
obs_arr:level = vertical level -> NO
obs_arr:p_level = pressure level -> YES
obs_arr:gc = grib code corresponding to the observation -> YES
obs_arr:ob = observation value (in the units specified by the grib code documentation) -> YES
obs_arr:qm = quality mark -> NO
obs_arr:pc = program code -> NO
obs_arr:rc = reason code -> NO
obs_arr:fc = forecast value -> NO
obs_arr:an = analyzed value -> NO
obs_arr:cat = data level category -> NO
hdr_typ = header type sting -> YES
hdr_sid = header station id -> YES
hdr_vld = header valid time (YYYY-MM-DD_HH:MM:SS UTC) -> YES
hdr_arr:lon = longitude of obs -> YES
hdr_arr:lat = latitude of obs -> YES
hdr_arr:dhr = obs time - cycle time -> NO
hdr_arr:elv = elevation of obs location -> NO
hdr_arr:typ = report type -> NO
hdr_arr:t29 = input report type -> NO
hdr_arr:itp = instrument type -> NO
Hopefully, I got all of these correct. Let me know if you need any other assistance. We're eager to find out how this conversion goes for you. If it works well (or even works at all!) we may
recommend it to other users.
> I am still in the process of attempting to reformat my point data to
> NetCDF based on the .cdl file you provided me. However, I have a
> question about what data the MET software actually needs. There are many
> variables listed in the .cdl file from the PrepBufr data. I assume the
> MET software will need all the basics such as observation value, grib
> code that corresponds to the observation type, pressure level, lat, lon,
> date/time, etc, but how about variables such as quality mark, program
> code, reason code, forecast value, analyzed value (what exactly are
> these last 2 values??), and data level category? Will MET accept -9999
> (fill values) for these variables or does it somehow use these?
> Leela Watson
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
> Sent: Thursday, November 29, 2007 12:32 PM
> To: Watson.Leela
> Cc: met_help at ucar.edu
> Subject: Re: [Met_help] BUFR, NetCDF files in MET
> Sorry for the delay in getting back to you. My name is John Halley
> Gotway, and I'm a developer working on MET.
> I see that you're trying to reformat your ASCII point observations into
> a netCDF file for input into the "point_stat" program. I have attached
> a CDL file explaining the format of the netCDF file that
> point_stat is expecting. I generated this CDL file by running the
> ncdump utility on the sample output of PB2NC that it generated by the
> test script in MET. If you were able to run the test scripts,
> you could find the sample output of pb2nc:
> The attached CDL file was generated by running: ncdump -h sample_pb.nc
> I assume that you'd like to use the ncgen utility to reformat your
> observations into netCDF. To be honest, I've never used ncgen myself.
> So I'll be interested to hear how it goes.
> Unfortunately, I expect that you'll find the format of the netCDF file
> rather convoluted. This is due to the input PrepBufr data from NCEP
> that we had to work with. Let me explain it a little.
> PrepBufr consists of a number of "messages". For each message, things
> like message type, station id, valid time, lat, lon, elevation, and a
> couple other things are defined.
> Each PrepBufr message consists of 1 or more "observations". For each
> observation value, several things are specified, like: variable type (P,
> Q, T, Z, U, V), pressure level, quality mark, and a few
> other things used by NCEP.
> In the file sample_pb.nc (and the attached CDL), you'll see that there
> were 9,716 PrepBufr messages (nmsg dimension), and 56,630 actual
> observations (nobs dimension). To save some space, we decided
> not to repeat the message header information for each of the 56,630
> observations. Instead, we just write out the message header info 9,716
> times. It's stored in the variables hdr_typ, hdr_sid,
> hdr_vld, and hdr_arr. The observation values are stored in the variable
> obs_arr. The first element of the obs_arr is hdr_id which points to the
> index of the header values which correspond to this
> observation value. Pretty convoluted.
> So I'm not sure how easy it will be to work within this format. Since
> I've never used ncgen, I'm not sure what's involved. But I'm happy to
> answer whatever questions arise as best I can.
> Good luck and please let us know how things go.
> John Halley Gotway
> johnhg at ucar.edu
> Watson.Leela wrote:
>> Hi Lacey,
>> Just a follow up on my previous message... Is it possible for someone
>> provide a .cdl file for converting ASCII point data to NetCDF for use
>> Leela Watson
>> -----Original Message-----
>> From: Lacey Holland [mailto:lholland at ucar.edu]
>> Sent: Wednesday, November 21, 2007 12:25 PM
>> To: Watson.Leela
>> Cc: met_help at ucar.edu
>> Subject: Re: [Met_help] BUFR, NetCDF files in MET
>> Hi, Leela!
>> Thanks for the questions! I'm glad to hear you were able to install
>> on your Linux machine!
>> Yes, the BUFR files must be written like the NCEP Prepbufr files.
>> Unfortunately, this isn't an easy format to work with, but it was
>> something we were tied to using for the initial beta release (v0.9).
>> an alternative, I think you would have better luck reformatting your
>> data into NetCDF format. Today, a lot of our staff is out for the
>> holiday, but you'll get a more definite response early next week with
>> respect to the specific NetCDF format and the availability of a .cdl
>> That's a good question about ASCII. We are currently planning to
>> ASCII files in METv1.1, which is being planned for the early spring
>> Feb 2008). The specific ASCII format will likely be the same
>> format used by MM5 and WRF-Var.
>> Thanks for your interest and for the questions!
>> --Lacey Holland
>> Watson.Leela wrote:
>>> I have recently successfully installed METv0.9 on my Linux machine
>>> now have a couple of question regarding the acceptable data formats
>>> for MET. I have formatted ASCII wind tower data that I would like to
>>> use to verify my WRF forecasts, but I know that MET does not
>>> accept this format. I have looked into converting this data into BUFR
>>> format, however, I am wondering if the BUFR files must be written
>>> exactly like the NCEP PrepBufr files (which, according to NCEP's
>>> website, are 'special' BUFR files)?
>>> It is also stated in the MET User's Guide that the PrepBufr files are
>>> converted into intermediate NetCDF files. Is there a .cdl file
>>> available for converting point data to NetCDF? Lastly, is there any
>>> sort of timeline for a MET version that allows formatted ASCII files
>>> (if at all)?
>>> Leela R. Watson, Sr Scientist/Meteorologist
>>> ENSCO, Inc. / Applied Meteorology Unit
>>> 1980 N. Atlantic Ave., Suite 230
>>> Cocoa Beach, FL 32931
>>> Voice: (321) 853-8264
>>> Fax: (321) 853-8415
>>> Email: watson.leela at ensco.com
>>> The information contained in this email message is intended only for
>>> the use of the individuals to whom it is
>>> addressed and may contain information that is privileged and
>>> sensitive. If the reader of this message is not
>>> the intended recipient, you are hereby notified that any
>>> dissemination, distribution or copying of this
>>> communication is strictly prohibited. If you have received this
>>> communication in error, please notify the
>>> sender immediately by email at the above referenced address. Thank
>>> Met_help mailing list
>>> Met_help at mailman.ucar.edu
More information about the Met_help