[Met_help] Re: Data interface about met program?
John Halley Gotway
johnhg at rap.ucar.edu
Tue Feb 26 08:38:29 MST 2008
We're currently working on the next release of MET, METv1.1. We've posted a list of main things that will be included in METv1.1:
http://www.dtcenter.org/met/users/support/future_releases/METv1.1/METv1.1_plans.php
In particular, you may be interested in the addition of the ASCII2NC tool. This tool will enable you to reformat point observations from a simple ASCII format to the NetCDF format that Point-Stat is
expecting. At the same time, we've simplified that intermediate NetCDF format somewhat. Here's the header of what that NetCDF file will look like in version 1.1:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
mxstr = 15 ;
hdr_arr_len = 3 ;
obs_arr_len = 5 ;
nobs = UNLIMITED ; // (51603 currently)
nhdr = 17205 ;
variables:
float obs_arr(nobs, obs_arr_len) ;
obs_arr:long_name = "array of observation values" ;
obs_arr:_fill_value = -9999.f ;
obs_arr:columns = "hdr_id gc lvl hgt ob" ;
obs_arr:hdr_id_long_name = "index of matching header data" ;
obs_arr:gc_long_name = "grib code corresponding to the observation type" ;
obs_arr:lvl_long_name = "pressure level (hPa) or accumulation interval (h)" ;
obs_arr:hgt_long_name = "geopotential height (m)" ;
obs_arr:ob_long_name = "observation value" ;
char hdr_typ(nhdr, mxstr) ;
hdr_typ:long_name = "message type" ;
char hdr_sid(nhdr, mxstr) ;
hdr_sid:long_name = "station identification" ;
char hdr_vld(nhdr, mxstr) ;
hdr_vld:long_name = "valid time" ;
hdr_vld:units = "YYYYMMDD_HHMMSS" ;
float hdr_arr(nhdr, hdr_arr_len) ;
hdr_arr:long_name = "array of observation station header values" ;
hdr_arr:_fill_value = -9999.f ;
hdr_arr:columns = "lat lon elv" ;
hdr_arr:lat_long_name = "latitude" ;
hdr_arr:lat_units = "degrees_north" ;
hdr_arr:lon_long_name = "longitude" ;
hdr_arr:lon_units = "degrees_east" ;
hdr_arr:elv_long_name = "elevation" ;
hdr_arr:elv_units = "meters" ;
// global attributes:
:FileOrigins = "File test_pb2nc/sample_pb_02.nc generated 20080204_185610 UTC on host cardinal by the PB2NC tool" ;
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The things we removed were present in the input PREPBUFR file but were not actually used by Point-Stat.
Regarding the things that remain though, here are links to where you can find lists of their values:
obs_arr:gc_long_name = "grib code corresponding to the observation type" ; http://www.nco.ncep.noaa.gov/pmb/docs/on388/table2.html
hdr_typ:long_name = "message type" ; Table 1.a in http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
Lastly, let me point you to a write-up for people interested in reformatting their observations into NetCDF format for METv1.0:
http://www.dtcenter.org/met/users/docs/write_ups/netcdf_point_obs_format.pdf
However, again let me caution you that the format will be changing for METv1.1 as described above.
Hope that helps.
John
Coffee.Z wrote:
> Dear John:
>
> I want to write the netcdf code myself. And i have some quesitions about the data columns:
>
> obs_arr:level_long_name = "vertical level" ;
> obs_arr:gc_long_name = "grib code corresponding to the observation type" ;
> obs_arr:qm_long_name = "quality mark" ;
> obs_arr:pc_long_name = "program code" ;
> obs_arr:rc_long_name = "reason code" ;
> obs_arr:fc_long_name = "forecast value" ;
> obs_arr:an_long_name = "analyzed value" ;
> obs_arr:cat_long_name = "data level category" ;
>
> hdr_arr:typ_long_name = "prepbufr report type" ;
> hdr_arr:t29_long_name = "input report type" ;
> hdr_arr:itp_long_name = "instrument type" ;
>
> Do you have the table that i could find the above vaule according to my own data? Besides, in your given sample data, the vertical level is integer. I'm wondering how can i determine my data. And i noticed the forecast value is not always missing. Is that mean the observational data have forecast value?
> Thanks!
>
> User: Coffee.Z
>
>
>
> ·¢¼þÈË£º John Halley Gotway
> ·¢ËÍʱ¼ä£º 2008-02-26 00:53:28
> ÊÕ¼þÈË£º Coffee.Z; met_help
> ³ËÍ£º
> Ö÷Ì⣺ Re: Data interface about met program?
>
> That's interesting. That's the first time I've seen that error message.
>
> Can you tell me what version of the PGI compilers you're using? Just type "pgCC --version" (that's with two '-' in front of version) to get the version number.
>
> I really don't know why the compiler would have a problem with that line of code. But let's try the following. I'd like you to do 2 things:
> (1) First, replace the file METv1.0/lib/vx_met_util/apply_mask.cc with the version of that file attached to this message.
> (2) Second, replace the file METv1.0/lib/vx_met_util/constants.h with the version of that file attached to this message.
>
> Then try rebuilding MET, and please let me know if that get's you past this error message. If so, we'll need to edit a 3rd file in a similar way to get the "vx_wrfmode" library to build.
>
> Thanks and please do let me know what compiler version you're using.
>
> John
>
>
>
> Coffee.Z wrote:
>> Dear John:
>>
>> I'm back to school now. I have tried the pgi compiler, and it is still failed. This time it seems syntax error, but i cannot figure out why. Here is the log. I have already updated all the file according to your "know issues".
>> Thanks!
>>
>> user: CoffeeZ
>>
>>
>>
>>
>> ÄúºÃ£¡
>>
>>
>>
>>
>> ÖÂ
>> Àñ£¡
>>
>>
>> СÖì
>> 2008-02-24
>>
>>
>>
>> ·¢¼þÈË£º John Halley Gotway
>> ·¢ËÍʱ¼ä£º 2008-01-26 00:23:10
>> ÊÕ¼þÈË£º Coffee.Z
>> ³ËÍ£º met_help
>> Ö÷Ì⣺ Re: Data interface about met program?
>>
>> Hello,
>>
>> Based on the files you sent, I see that the error you're encountering is due to NetCDF. The linker isn't finding the NetCDF library archive files it's looking for.
>>
>> I'm hoping that if you make a simple fix in the Makefile and recompile, it'll get past this error.
>>
>> In the Makefile, look at how the variable NETCDF_LIBS is defined. List out those directories:
>> ls /home/users/zhucoffee/wrfnew/netcdf/libs
>> ls /home/users/zhucoffee/wrfnew/netcdf/include
>>
>> Do they contain the files libnetcdf.a and libnetcdf_c++.a?
>>
>> If not, locate those two files in the NetCDF directory. You can try something like:
>> cd /home/users/zhucoffee/wrfnew/netcdf
>> find ./ -name libnetcdf.a
>> find ./ -name libnetcdf_c++.a
>>
>> Then reset NETCDF_LIBS to the directory (or directories) that contain them. And then rebuild MET with something like:
>> make clean
>> make >& make_met.log&
>>
>> Please let me know if that did the trick.
>>
>> Thanks,
>> John Halley Gotway
>>
>> So I'd suggest checking whether the version of NetCDF in /home/users/zhucoffee/wrfnew/netcdf was compiled using the PGI or the GNU compilers.
>>
>> Coffee.Z wrote:
>>> Dear John:
>>>
>>>
>>> Here is the log. Although i have changed the PGI compiler to the 32 bit and all other external lib to 32 bit. It is still failed.
>>>
>>>
>>>
>>> User: CoffeeZ
>>>
>>>
>>>
>>>
>>> ÄúºÃ£¡
>>>
>>>
>>>
>>>
>>> ÖÂ
>>> Àñ£¡
>>>
>>>
>>> СÖì
>>> 2008-01-25
>>>
>>>
>>>
>>> ·¢¼þÈË£º John Halley Gotway
>>> ·¢ËÍʱ¼ä£º 2008-01-25 12:44:42
>>> ÊÕ¼þÈË£º Coffee.Z
>>> ³ËÍ£º met_help at ucar.edu
>>> Ö÷Ì⣺ Re: Data interface about met program?
>>>
>>>
>>> Sorry to hear about the PGI compilers. If you'd like to send the Makefile
>>> and compilation log file, I could take a look and see if there's anything
>>> obviously wrong.
>>>
>>> What I mean is, when you build MET using a command like...
>>> make -f Makefile_pgi >& make_met.log
>>> ... please send me a copy of the "Makefile_pgi" that you're using and a
>>> copy of the "make_met.log" logfile.
>>>
>>> Regarding the issue of using point observations in ASCII format, several
>>> users are encountering the same obstacle. For the next release of MET,
>>> I'd like to add a utility that will convert formatted ASCII point
>>> observations into the NetCDF format that MET is expecting.
>>>
>>> However, that release won't be ready for a couple of months I'd say. In
>>> the meantime, I need to write up some instructions describing what the
>>> NetCDF point observations should look like for those users who would like
>>> to reformat their observations into NetCDF themselves.
>>>
>>> So I'll work on those basic instructions tomorrow, post that info to the
>>> MET website, and let you know when it's available for you to take a look
>>> at.
>>>
>>> Thanks,
>>> John
>>>
>>>
>>>
>>>
>>>> Dear John:
>>>>
>>>> Thanks very much for your kind guidance! I have try the PGI compiler,
>>>> but still failed. I will try it again when i back to my school.
>>>>
>>>> Right now, I have some question about the met data interface. For in
>>>> my country, most of the observational data are stored in ASCII
>>>> format. That mean i can't use the 'pb2nc' util. So I'm thinking to
>>>> transfer my observational data to the netcdf format, instead of to
>>>> the prebufr format. My question is do i need to pay special
>>>> attention when i write the netcdf code?
>>>>
>>>> Another question is about the gridded input of the observational
>>>> data. I'm wondering if i use the data like 'ncep fnl global 1 degree
>>>> * 1 degree', do i need to intepolate the data into the domain just as
>>>> the wrf output?
>>>>
>>>> Thanks!
>>>>
>>>> users: CoffeeZ
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ÄúºÃ£¡
>>>>
>>>>
>>>>
>>>>
>>>> ÖÂ
>>>> Àñ£¡
>>>>
>>>>
>>>> СÖì
>>>> 2008-01-25
>>>>
>>>>
>>>>
>>>> ·¢¼þÈË£º John Halley Gotway
>>>> ·¢ËÍʱ¼ä£º 2008-01-23 09:35:59
>>>> ÊÕ¼þÈË£º Coffee.Z
>>>> ³ËÍ£º met_help
>>>> Ö÷Ì⣺ Re: [Met_help] Error when running met with the test_all.csh?
>>>>
>>>> In regards to compiling MET using the PGI compilers, I wanted to point
>>>> you
>>>> to a couple of known issues posted on the MET website.
>>>>
>>>> The "Makefile_pgi" file needs to be modified prior to compiling with the
>>>> PGI compilers, especially version 7 of the PGI compilers:
>>>> http://www.dtcenter.org/met/users/support/known_issues/METv1.0/METv1.0_known_issues.php
>>>>
>>>> So if you'd like to try using the PGI compilers again please refer to the
>>>> fixes posted.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm guessing that the problems you're encountering are related to
>>>>> running
>>>>> MET on a 64-bit machine. Currently, BUFRLIB a library on which MET
>>>>> depends, doesn't work well on a 64-bit machine. That's
>>>>> the reason for the error message you're seeing from PB2NC:
>>>>>
>>>>> **************BUFR ARCHIVE LIBRARY ABORT*****************
>>>>> BUFRLIB: OPENBF - FIRST 4 BYTES READ FROM RECORD IN INPUT FILE
>>>>> CONNECTED
>>>>> TO UNIT 11 NOT 'BUFR', DOES NOT CONTAIN BUFR DATA
>>>>> **************BUFR ARCHIVE LIBRARY ABORT*****************
>>>>>
>>>>> However, for the other error messages you're seeing about the NetCDF
>>>>> file
>>>>> containing illegal characters... that may also be related to running on
>>>>> a
>>>>> 64-bit machine.
>>>>>
>>>>> I'm in the process of trying to get MET to run on a 64-bit machine we
>>>>> have
>>>>> here at NCAR. What I'm trying to do is compile MET (and the libraries
>>>>> on
>>>>> which it depends) using the GNU -m32 compiler flag
>>>>> which should build 32-bit code. But I'm still working on it.
>>>>>
>>>>> In the top-level MET directory, edit the file "Makefile".
>>>>> Replace the line:
>>>>> FLAGS = -g -Wall -Wshadow -static
>>>>> with
>>>>> FLAGS = -g -Wall -Wshadow -static -m32
>>>>>
>>>>> But the trick is to get the libraries on which MET depends to compile
>>>>> as
>>>>> 32-bit as well.
>>>>>
>>>>> Feel free to give it a shot yourself. I'll let you know when I'm able
>>>>> to
>>>>> compile it on a 64-bit machine myself. If you have access to a 32-bit
>>>>> machine, I'd suggest trying it over there as well.
>>>>>
>>>>> Thanks,
>>>>> John Halley Gotway
>>>>>
>>>>> Coffee.Z wrote:
>>>>> > Hello,met_help:
>>>>> >
>>>>> > i have been trying to run the met program with the sample data,
>>>>> but
>>>>> > encountered the following problem, for more detail, you can see
>>>>> > further in test.log:
>>>>> >
>>>>> > *** Running GRID_STAT using netCDF input for both forecast and
>>>>> > observation ***
>>>>> > Forecast File: ../out/pcp_combine/sample_fcst_12L_2005080712V_12A.nc
>>>>> > Observation File: ../out/pcp_combine/sample_obs_2005080712V_12A.nc
>>>>> > Configuration File: config/GridStatConfig_APCP_12
>>>>> > NetCDF Precipitation (min, max) = (0, 38.8841)
>>>>> > NetCDF Precipitation (min, max) = (0, 104.1)
>>>>> > NetCDF: Name contains illegal characters
>>>>> > ....
>>>>> > ERROR: process_obs_file() - > can't open observation netCDF file:
>>>>> > ../out/pb2nc/sample_pb.nc
>>>>> >
>>>>> > I have been tried to build the source code with pgf77, but failed.
>>>>> > The above is the result build with g++ and g77. Since the netcdf is
>>>>> > build with the pgf90 and the system is 64 bit, and i'm wondering if i
>>>>> > need to build the source code with pgi.
>>>>> >
>>>>> > Thanks!
>>>>> >
>>>>> > user: CoffeeZ
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > ÄúºÃ£¡
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > ÖÂ
>>>>> > Àñ£¡
>>>>> >
>>>>> >
>>>>> > СÖì
>>>>> > 2008-01-23
>>>>> >
>>>>> >
>>>>> > ------------------------------------------------------------------------
>>>>> >
>>>>> > _______________________________________________
>>>>> > Met_help mailing list
>>>>> > Met_help at mailman.ucar.edu
>>>>> > http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>> _______________________________________________
>>>>> Met_help mailing list
>>>>> Met_help at mailman.ucar.edu
>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>>
>>> .
More information about the Met_help
mailing list