[Met_help] BUFR, NetCDF files in MET

[John Halley Gotway]

Leela,

Sorry for the delay in getting back to you.  My name is John Halley Gotway, and I'm a developer working on MET.

I see that you're trying to reformat your ASCII point observations into a netCDF file for input into the "point_stat" program.  I have attached a CDL file explaining the format of the netCDF file that
point_stat is expecting.  I generated this CDL file by running the ncdump utility on the sample output of PB2NC that it generated by the test script in MET.  If you were able to run the test scripts,
you could find the sample output of pb2nc: .../METv0.9/out/pb2nc/sample_pb.nc

The attached CDL file was generated by running: ncdump -h sample_pb.nc

I assume that you'd like to use the ncgen utility to reformat your observations into netCDF.  To be honest, I've never used ncgen myself.  So I'll be interested to hear how it goes.

Unfortunately, I expect that you'll find the format of the netCDF file rather convoluted.  This is due to the input PrepBufr data from NCEP that we had to work with.  Let me explain it a little.

PrepBufr consists of a number of "messages".  For each message, things like message type, station id, valid time, lat, lon, elevation, and a couple other things are defined.
Each PrepBufr message consists of 1 or more "observations".  For each observation value, several things are specified, like: variable type (P, Q, T, Z, U, V), pressure level, quality mark, and a few
other things used by NCEP.

In the file sample_pb.nc (and the attached CDL), you'll see that there were 9,716 PrepBufr messages (nmsg dimension), and 56,630 actual observations (nobs dimension).  To save some space, we decided
not to repeat the message header information for each of the 56,630 observations.  Instead, we just write out the message header info 9,716 times.  It's stored in the variables hdr_typ, hdr_sid,
hdr_vld, and hdr_arr.  The observation values are stored in the variable obs_arr.  The first element of the obs_arr is hdr_id which points to the index of the header values which correspond to this
observation value.  Pretty convoluted.

So I'm not sure how easy it will be to work within this format.  Since I've never used ncgen, I'm not sure what's involved.  But I'm happy to answer whatever questions arise as best I can.

Good luck and please let us know how things go.

Thanks,
John Halley Gotway
johnhg at ucar.edu



Watson.Leela wrote:
> Hi Lacey,
> 
> Just a follow up on my previous message... Is it possible for someone to
> provide a .cdl file for converting ASCII point data to NetCDF for use in
> Metv0.9? 
> 
> Thanks!
> 
> Leela Watson 
> 
> -----Original Message-----
> From: Lacey Holland [mailto:lholland at ucar.edu] 
> Sent: Wednesday, November 21, 2007 12:25 PM
> To: Watson.Leela
> Cc: met_help at ucar.edu
> Subject: Re: [Met_help] BUFR, NetCDF files in MET
> 
> Hi, Leela!
> 
> 
> Thanks for the questions! I'm glad to hear you were able to install MET 
> on your Linux machine!
> 
> Yes, the BUFR files must be written like the NCEP Prepbufr files. 
> Unfortunately, this isn't an easy format to work with, but it was 
> something we were tied to using for the initial beta release (v0.9). As 
> an alternative, I think you would have better luck reformatting your 
> data into NetCDF format. Today, a lot of our staff is out for the 
> holiday, but you'll get a more definite response early next week with 
> respect to the specific NetCDF format and the availability of a .cdl
> file.
> 
> That's a good question about ASCII. We are currently planning to support
> 
> ASCII files in METv1.1, which is being planned for the early spring (29 
> Feb 2008). The specific ASCII format will likely be the same "little_r" 
> format used by MM5 and WRF-Var.
> 
> 
> Thanks for your interest and for the questions!
> --Lacey Holland
> 
> 
> 
> 
> 
> Watson.Leela wrote:
>> Hello,
>>
>> I have recently successfully installed METv0.9 on my Linux machine and
> 
>> now have a couple of question regarding the acceptable data formats 
>> for MET. I have formatted ASCII wind tower data that I would like to 
>> use to verify my WRF forecasts, but I know that MET does not currently
> 
>> accept this format. I have looked into converting this data into BUFR 
>> format, however, I am wondering if the BUFR files must be written 
>> exactly like the NCEP PrepBufr files (which, according to NCEP's 
>> website, are 'special' BUFR files)?
>>
>> It is also stated in the MET User's Guide that the PrepBufr files are 
>> converted into intermediate NetCDF files. Is there a .cdl file 
>> available for converting point data to NetCDF? Lastly, is there any 
>> sort of timeline for a MET version that allows formatted ASCII files 
>> (if at all)?
>>
>> Regards,
>>
>> Leela
>>
>> *******************************************************************
>>
>> Leela R. Watson, Sr Scientist/Meteorologist
>>
>> ENSCO, Inc. / Applied Meteorology Unit
>>
>> 1980 N. Atlantic Ave., Suite 230
>>
>> Cocoa Beach, FL 32931
>>
>> Voice: (321) 853-8264
>>
>> Fax: (321) 853-8415
>>
>> Email: watson.leela at ensco.com
>>
>>
> ------------------------------------------------------------------------
>> The information contained in this email message is intended only for 
>> the use of the individuals to whom it is
>> addressed and may contain information that is privileged and 
>> sensitive. If the reader of this message is not
>> the intended recipient, you are hereby notified that any 
>> dissemination, distribution or copying of this
>> communication is strictly prohibited. If you have received this 
>> communication in error, please notify the
>> sender immediately by email at the above referenced address. Thank
> you.
> ------------------------------------------------------------------------
>> _______________________________________________
>> Met_help mailing list
>> Met_help at mailman.ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>   
> 
> 
-------------- next part --------------
netcdf sample_pb {
dimensions:
	mxstr = 20 ;
	hdr_arr_len = 7 ;
	obs_arr_len = 11 ;
	nobs = UNLIMITED ; // (56630 currently)
	nmsg = 9716 ;
variables:
	float obs_arr(nobs, obs_arr_len) ;
		obs_arr:long_name = "array of observation values" ;
		obs_arr:_fill_value = -9999.f ;
		obs_arr:columns = "hdr_id level p_level gc ob qm pc rc fc an cat" ;
		obs_arr:hdr_id_long_name = "index of matching header data" ;
		obs_arr:level_long_name = "vertical level" ;
		obs_arr:p_level_long_name = "pressure level (hPa or mb)" ;
		obs_arr:gc_long_name = "grib code corresponding to the observation type" ;
		obs_arr:ob_long_name = "observation value" ;
		obs_arr:qm_long_name = "quality mark" ;
		obs_arr:pc_long_name = "program code" ;
		obs_arr:rc_long_name = "reason code" ;
		obs_arr:fc_long_name = "forecast value" ;
		obs_arr:an_long_name = "analyzed value" ;
		obs_arr:cat_long_name = "data level category" ;
	char hdr_typ(nmsg, mxstr) ;
		hdr_typ:long_name = "message type" ;
	char hdr_sid(nmsg, mxstr) ;
		hdr_sid:long_name = "station identification" ;
	char hdr_vld(nmsg, mxstr) ;
		hdr_vld:long_name = "valid time (observation time plus dhr)" ;
		hdr_vld:units = "YYYY-MM-DD_HH:MM:SS UTC" ;
	float hdr_arr(nmsg, hdr_arr_len) ;
		hdr_arr:long_name = "array of observation station header values" ;
		hdr_arr:_fill_value = -9999.f ;
		hdr_arr:columns = "lon lat dhr elv typ t29 itp" ;
		hdr_arr:lon_long_name = "longitude" ;
		hdr_arr:lon_units = "degrees_east" ;
		hdr_arr:lat_long_name = "latitude" ;
		hdr_arr:lat_units = "degrees_north" ;
		hdr_arr:dhr_long_name = "observation time minus cycle time" ;
		hdr_arr:dhr_units = "hours" ;
		hdr_arr:elv_long_name = "elevation" ;
		hdr_arr:elv_units = "meters" ;
		hdr_arr:typ_long_name = "prepbufr report type" ;
		hdr_arr:t29_long_name = "input report type" ;
		hdr_arr:itp_long_name = "instrument type" ;

// global attributes:
		:FileOrigins = "File ../out/pb2nc/sample_pb.nc generated 2007-07-16_19:47:43 UTC on host pigpen" ;
		:obs_time = "2007-03-31_12:00:00" ;
}