[Met_help] BUFR, NetCDF files in MET

[John Halley Gotway]

Leela,

Thanks for the feedback.  I'm really glad to hear that it worked for you.

As for the results that you're getting from point_stat, let me caution you about them a little.  We're working toward a release of MET by the end of December, and during our testing we've run across
some issues with point_stat.  They're mostly related to unit conversion, but they'll all be fixed in the next release.

In particular, prior to comparing with the model output, for the point observations in the PrepBufr files...
- specific humidity needs to be converted from g/kg to kg/kg
- temperature needs to be converted from C to K
- the U and V components of the wind likely need to be rotated from grid relative to earth relative

I just wanted to make you aware of those issues and know that we'll be fixing them for the next release.  We're also adding the ability (in pb2nc) of deriving wind speed, wind direction, relative
humidity, mixing ratio, and dewpoint temp.  And since the WRF-PostProcessor doesn't dump out wind speed and direction, we're adding the ability to produce those fields in point_stat and grid_stat.

To be clear, were you able to use the "ncgen" tool to generate that fortran program you used in the conversion, or did you have to write the program manually?  If a user has observations in some ASCII
format, I'm wondering how much we'd be able to help them streamline the process of converting to NetCDF?

Thanks,
John

Watson.Leela wrote:
> John & Lacey,
> 
> It seems I was able to get point_stat to work on my ASCII data that I
> converted to NetCDF. I say 'seems' because the program ran, did not
> produce any errors, and produced what seems to be reasonable values.
> However, I have not yet done a thorough check of the results, but some
> quick calculations of my own produced somewhat similar results. 
> 
> To create the NetCDF file I wrote a fortran program that created a .cdl
> file with the same format as the output from 'ncdump sample_pb.nc', with
> my data filled in the appropriate fields. I was able to use -9999 for
> all fields that you indicated were not needed (below). I then used ncgen
> to generate a NetCDF file that was then used in the point_stat program.
> It was actually a fairly easy task.
> 
> If you would like any more information, feel free to email me. If I come
> across any issues/problems with this conversion, I will let you know.
> 
> Regards,
> 
> Leela Watson
> 
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu] 
> Sent: Thursday, December 06, 2007 11:22 AM
> To: Watson.Leela
> Cc: met_help at ucar.edu
> Subject: Re: [Met_help] BUFR, NetCDF files in MET
> 
> Leela,
> 
> Good question.  Providing the fill value of -9999 for most of those will
> be fine.  MET does not use most of them.  I've retained that info since
> it's present in the PrepBufr data and NCEP has some use
> for it.
> 
> As for the forecast value... apparently, in the verification code that
> NCEP uses, it processes through the observation files and appends the
> nearest forecast value to each observation.  So eventually,
> the PrepBufr file they're using contains matched pair data.  MET, on the
> other hand, just uses the observations from the PrepBufr files.  We
> don't "add" data them during the processing.
> 
> As for the analyzed value... to be honest I'm not sure how NCEP uses
> this.  Let me know if you're interested though, and I could find out.
> 
> The MET utility "pb2nc" allows users to filter out the observation
> values they'd like to used based on all of these parameters.  For
> example, the user could specify a quality mark threshold to apply
> in pb2nc.  Then, only those observations which meet that quality mark
> criteria will be retained.  So that's why we write out all of these
> fields.
> 
> Let me attempt to indicate which fields actually need to contain valid
> data to be used by "point_stat":
> 
> obs_arr:hdr_id = header index -> YES
> obs_arr:level = vertical level -> NO
> obs_arr:p_level = pressure level -> YES
> obs_arr:gc = grib code corresponding to the observation -> YES
> obs_arr:ob = observation value (in the units specified by the grib code
> documentation) -> YES
> obs_arr:qm = quality mark -> NO
> obs_arr:pc = program code -> NO
> obs_arr:rc = reason code -> NO
> obs_arr:fc = forecast value -> NO
> obs_arr:an = analyzed value -> NO
> obs_arr:cat = data level category -> NO
> hdr_typ = header type sting -> YES
> hdr_sid = header station id -> YES
> hdr_vld = header valid time (YYYY-MM-DD_HH:MM:SS UTC) -> YES
> hdr_arr:lon = longitude of obs -> YES
> hdr_arr:lat = latitude of obs -> YES
> hdr_arr:dhr = obs time - cycle time -> NO
> hdr_arr:elv = elevation of obs location -> NO
> hdr_arr:typ = report type -> NO
> hdr_arr:t29 = input report type -> NO
> hdr_arr:itp = instrument type -> NO
> 
> Hopefully, I got all of these correct.  Let me know if you need any
> other assistance.  We're eager to find out how this conversion goes for
> you.  If it works well (or even works at all!) we may
> recommend it to other users.
> 
> Thanks,
> John
> 
> Watson.Leela wrote:
>> John,
>>
>> I am still in the process of attempting to reformat my point data to
>> NetCDF based on the .cdl file you provided me. However, I have a
>> question about what data the MET software actually needs. There are
> many
>> variables listed in the .cdl file from the PrepBufr data. I assume the
>> MET software will need all the basics such as observation value, grib
>> code that corresponds to the observation type, pressure level, lat,
> lon,
>> date/time, etc, but how about variables such as quality mark, program
>> code, reason code, forecast value, analyzed value (what exactly are
>> these last 2 values??), and data level category? Will MET accept -9999
>> (fill values) for these variables or does it somehow use these?
>>
>> Regards,
>>
>> Leela Watson
>>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu] 
>> Sent: Thursday, November 29, 2007 12:32 PM
>> To: Watson.Leela
>> Cc: met_help at ucar.edu
>> Subject: Re: [Met_help] BUFR, NetCDF files in MET
>>
>> Leela,
>>
>> Sorry for the delay in getting back to you.  My name is John Halley
>> Gotway, and I'm a developer working on MET.
>>
>> I see that you're trying to reformat your ASCII point observations
> into
>> a netCDF file for input into the "point_stat" program.  I have
> attached
>> a CDL file explaining the format of the netCDF file that
>> point_stat is expecting.  I generated this CDL file by running the
>> ncdump utility on the sample output of PB2NC that it generated by the
>> test script in MET.  If you were able to run the test scripts,
>> you could find the sample output of pb2nc:
>> .../METv0.9/out/pb2nc/sample_pb.nc
>>
>> The attached CDL file was generated by running: ncdump -h sample_pb.nc
>>
>> I assume that you'd like to use the ncgen utility to reformat your
>> observations into netCDF.  To be honest, I've never used ncgen myself.
>> So I'll be interested to hear how it goes.
>>
>> Unfortunately, I expect that you'll find the format of the netCDF file
>> rather convoluted.  This is due to the input PrepBufr data from NCEP
>> that we had to work with.  Let me explain it a little.
>>
>> PrepBufr consists of a number of "messages".  For each message, things
>> like message type, station id, valid time, lat, lon, elevation, and a
>> couple other things are defined.
>> Each PrepBufr message consists of 1 or more "observations".  For each
>> observation value, several things are specified, like: variable type
> (P,
>> Q, T, Z, U, V), pressure level, quality mark, and a few
>> other things used by NCEP.
>>
>> In the file sample_pb.nc (and the attached CDL), you'll see that there
>> were 9,716 PrepBufr messages (nmsg dimension), and 56,630 actual
>> observations (nobs dimension).  To save some space, we decided
>> not to repeat the message header information for each of the 56,630
>> observations.  Instead, we just write out the message header info
> 9,716
>> times.  It's stored in the variables hdr_typ, hdr_sid,
>> hdr_vld, and hdr_arr.  The observation values are stored in the
> variable
>> obs_arr.  The first element of the obs_arr is hdr_id which points to
> the
>> index of the header values which correspond to this
>> observation value.  Pretty convoluted.
>>
>> So I'm not sure how easy it will be to work within this format.  Since
>> I've never used ncgen, I'm not sure what's involved.  But I'm happy to
>> answer whatever questions arise as best I can.
>>
>> Good luck and please let us know how things go.
>>
>> Thanks,
>> John Halley Gotway
>> johnhg at ucar.edu
>>
>>
>>
>> Watson.Leela wrote:
>>> Hi Lacey,
>>>
>>> Just a follow up on my previous message... Is it possible for someone
>> to
>>> provide a .cdl file for converting ASCII point data to NetCDF for use
>> in
>>> Metv0.9? 
>>>
>>> Thanks!
>>>
>>> Leela Watson 
>>>
>>> -----Original Message-----
>>> From: Lacey Holland [mailto:lholland at ucar.edu] 
>>> Sent: Wednesday, November 21, 2007 12:25 PM
>>> To: Watson.Leela
>>> Cc: met_help at ucar.edu
>>> Subject: Re: [Met_help] BUFR, NetCDF files in MET
>>>
>>> Hi, Leela!
>>>
>>>
>>> Thanks for the questions! I'm glad to hear you were able to install
>> MET 
>>> on your Linux machine!
>>>
>>> Yes, the BUFR files must be written like the NCEP Prepbufr files. 
>>> Unfortunately, this isn't an easy format to work with, but it was 
>>> something we were tied to using for the initial beta release (v0.9).
>> As 
>>> an alternative, I think you would have better luck reformatting your 
>>> data into NetCDF format. Today, a lot of our staff is out for the 
>>> holiday, but you'll get a more definite response early next week with
> 
>>> respect to the specific NetCDF format and the availability of a .cdl
>>> file.
>>>
>>> That's a good question about ASCII. We are currently planning to
>> support
>>> ASCII files in METv1.1, which is being planned for the early spring
>> (29 
>>> Feb 2008). The specific ASCII format will likely be the same
>> "little_r" 
>>> format used by MM5 and WRF-Var.
>>>
>>>
>>> Thanks for your interest and for the questions!
>>> --Lacey Holland
>>>
>>>
>>>
>>>
>>>
>>> Watson.Leela wrote:
>>>> Hello,
>>>>
>>>> I have recently successfully installed METv0.9 on my Linux machine
>> and
>>>> now have a couple of question regarding the acceptable data formats 
>>>> for MET. I have formatted ASCII wind tower data that I would like to
> 
>>>> use to verify my WRF forecasts, but I know that MET does not
>> currently
>>>> accept this format. I have looked into converting this data into
> BUFR
>>>> format, however, I am wondering if the BUFR files must be written 
>>>> exactly like the NCEP PrepBufr files (which, according to NCEP's 
>>>> website, are 'special' BUFR files)?
>>>>
>>>> It is also stated in the MET User's Guide that the PrepBufr files
> are
>>>> converted into intermediate NetCDF files. Is there a .cdl file 
>>>> available for converting point data to NetCDF? Lastly, is there any 
>>>> sort of timeline for a MET version that allows formatted ASCII files
> 
>>>> (if at all)?
>>>>
>>>> Regards,
>>>>
>>>> Leela
>>>>
>>>> *******************************************************************
>>>>
>>>> Leela R. Watson, Sr Scientist/Meteorologist
>>>>
>>>> ENSCO, Inc. / Applied Meteorology Unit
>>>>
>>>> 1980 N. Atlantic Ave., Suite 230
>>>>
>>>> Cocoa Beach, FL 32931
>>>>
>>>> Voice: (321) 853-8264
>>>>
>>>> Fax: (321) 853-8415
>>>>
>>>> Email: watson.leela at ensco.com
>>>>
>>>>
> ------------------------------------------------------------------------
>>>> The information contained in this email message is intended only for
> 
>>>> the use of the individuals to whom it is
>>>> addressed and may contain information that is privileged and 
>>>> sensitive. If the reader of this message is not
>>>> the intended recipient, you are hereby notified that any 
>>>> dissemination, distribution or copying of this
>>>> communication is strictly prohibited. If you have received this 
>>>> communication in error, please notify the
>>>> sender immediately by email at the above referenced address. Thank
>>> you.
>>>
> ------------------------------------------------------------------------
>>>> _______________________________________________
>>>> Met_help mailing list
>>>> Met_help at mailman.ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>   
>