[Met_help] Re: Netcdf file
John Halley Gotway
johnhg at rap.ucar.edu
Wed Jul 30 13:34:28 MDT 2008
Erik,
MET is expecting to read a 2D variable, i.e. APCP(lat, lon).
However, your NetCDF variable is a 3D one: APCP(time, lat, lon)
I'm pretty sure that would be a problem. I'd suggest writing each 2D APCP field out to it's own file for each time. In addition, MET will be looking for the following attributes:
APCP:init_time_ut (model initialization time in unixtime)
APCP:valid_time_ut (forecast valid time in unixtime)
APCP:accum_time (accumulation time in seconds)
I realize that converting the data to NetCDF like this is a hassle. I hope that once we implement code to support a more generalized NetCDF format, do things like this will be a lot easier.
John
Erik Noble wrote:
> Dear john, I am almost there.
>
> Do I need to include any other attributes here? Will this be enough for MET?
>
> Erik at noble:/Volumes/Data_and_Models/ncl_scripts:ncdump -h
> /Volumes/Data_and_Models/Precipitation_data/observed/TRMM/3B42.2006082612.nc
> netcdf 3B42.2006082612 {
> dimensions:
> time = UNLIMITED ; // (1 currently)
> lat = 400 ;
> lon = 1440 ;
> variables:
> double time(time) ;
> time:units = "hours since 1998-01-01 00:00:0.0" ;
> float lat(lat) ;
> lat:units = "degrees_north" ;
> lat:long_name = "latitude" ;
> float lon(lon) ;
> lon:units = "degrees_east" ;
> lon:long_name = "longitude" ;
> int date(time) ;
> int datesec(time) ;
> float APCP(time, lat, lon) ;
> APCP:long_name = "Total Precipitation" ;
> APCP:missing_value = -9999.f ;
> APCP:_FillValue = -9999.f ;
> APCP:Conventions = "None" ;
> APCP:level = "SFC" ;
> APCP:units = "mm" ;
> APCP:grib_code = 61 ;
> APCP:avg_period = "0000-00-00 01:00:00" ;
> APCP:delta_t = "0000-00-00 03:00:00" ;
> APCP:hdf_name = "precipitation" ;
>
> // global attributes:
> :Projection = "Lambert Conformal" ;
> :creation_date = "Wed Jul 30 14:58:04 EDT 2008" ;
> :Conventions = "None" ;
> :source_html = "\n",
>
> "http://disc.sci.gsfc.nasa.gov/data/datapool/TRMM/01_Data_Products/02_Gridde
> d/06_3-hour_Gpi_Cal_3B_42/index.html\n",
> "" ;
> :title = "TRMM: 3B42" ;
> }
> Erik at noble:/Volumes/Data_and_Models/ncl_scripts:
>
>
>
> On 7/29/08 4:53 PM, "John Halley Gotway" <johnhg at rap.ucar.edu> wrote:
>
>> Erik,
>>
>> MET currently supports reading gridded data in GRIB format. If you have
>> gridded data you'd like to use in MET, that isn't in GRIB format, you have 2
>> choices. The first is to convert it to GRIB
>> format (NCL may be able to do this). The second is to convert it a NetCDF
>> format that mimics the output of the PCP-Combine tool.
>>
>> I was giving you info about how to mimic the output of PCP-Combine. You can
>> see sample output of PCP-Combine in the METv1.1/out/pcp_combine directory.
>>
>> As for what you choose and how you go about converting your data, that'll be
>> up to you. I have limited experience with NCL. Starting with an existing
>> conversion script like you mentioned seems like
>> a good way to go, but I'm guessing you'll need to make some modifications to
>> get it in a format that MET will like.
>>
>> I'm busy the remainder of this week and will be out of the office for 10 days
>> starting on Monday. I'll be returning August 14th.
>>
>> If you choose to convert your data to NetCDF, let me point you to the routine
>> in MET that reads those NetCDF files. The routine is called
>> read_pcp_combine_netcdf in the file named
>> METv1.1/lib/vx_met_util/read_pcp_combine_netcdf.cc.
>>
>> In that routine, you'll see that MET reads the following from the NetCDF file:
>> (1) Grid projection information from the global attributes.
>> (2) The precip variable named using the GRIB code abbreviation for the field.
>> (3) The "level" attribute of the precip variable.
>> (4) The "init_time_ut" attribute of the precip variable.
>> (5) The "valid_time_ut" attribute of the precip variable.
>> (6) The "accum_time" attribute of the precip variable.
>>
>> John
>>
>> Erik Noble wrote:
>>> Oh.......wait. So to clarify, we are doing this step in order to avoid using
>>> PCP Combine? I missed that.
>>> -Erik
>>>
>>>
>>> On 7/29/08 3:36 PM, "John Halley Gotway" <johnhg at rap.ucar.edu> wrote:
>>>
>>>> Erik,
>>>>
>>>> Listed below is a NetCDF header dump of one of the output files from the
>>>> METv1.1 test script for pcp_combine.
>>>>
>>>> Regarding (2), I'd suggest adding the same type of variable attributes that
>>>> are listed here:
>>>> - grib_code = 61
>>>> - units = "kg/m^2" or "mm"
>>>> - level = "SFC"
>>>> - _FillValue = -9999.f
>>>> - init_time = "20050807_000000"
>>>> - init_time_ut = 1123372800
>>>> - valid_time = "20050808_000000"
>>>> - valid_time_ut = 1123459200
>>>> - accum_time = "24 hours"
>>>>
>>>> The "_ut" times listed above are in unixtime - the number of seconds that
>>>> have
>>>> elapsed since Jan 1, 1970. A fancy "date" command can be used to give you
>>>> the
>>>> unixtime.
>>>> For example, the unixtime corresponding to 8am on July 29, 2008 is given by
>>>> the following command: date -ud '2008-07-29 UTC 08:00:00' +%s
>>>>
>>>> Regarding (3), please the following global attributes define the grid on
>>>> which
>>>> this data resides:
>>>> - Projection = "Lambert Conformal" ;
>>>> - p1_deg = "25.000000 degrees_north" ;
>>>> - p2_deg = "25.000000 degrees_north" ;
>>>> - p0_deg = "12.190000 degrees_north" ;
>>>> - l0_deg = "-133.459000 degrees_east" ;
>>>> - lcen_deg = "-95.000000 degrees_east" ;
>>>> - d_km = "40.635000 km" ;
>>>> - r_km = "6367.470000 km" ;
>>>> - nx = "185 grid_points" ;
>>>> - ny = "129 grid_points" ;
>>>>
>>>> This example is for a lat/lon grid. You could also use lambert conformal or
>>>> polar stereographic. If you have a sample GRIB file already on the grid
>>>> you're using, please send it to me, and I can help
>>>> you figure out how to specify it in the NetCDF file.
>>>>
>>>> Hope that helps,
>>>> John
>>>>
>>>> netcdf sample_fcst_24L_2005080800V_24A {
>>>> dimensions:
>>>> lat = 129 ;
>>>> lon = 185 ;
>>>> variables:
>>>> float lat(lat, lon) ;
>>>> lat:units = "degrees_north" ;
>>>> lat:long_name = "Latitude" ;
>>>> lat:_FillValue = -9999.f ;
>>>> float lon(lat, lon) ;
>>>> lon:units = "degrees_east" ;
>>>> lon:long_name = "Longitude" ;
>>>> lon:_FillValue = -9999.f ;
>>>> float APCP(lat, lon) ;
>>>> APCP:grib_code = 61 ;
>>>> APCP:units = "kg/m^2" ;
>>>> APCP:long_name = "Total precipitation" ;
>>>> APCP:level = "SFC" ;
>>>> APCP:_FillValue = -9999.f ;
>>>> APCP:init_time = "20050807_000000" ;
>>>> APCP:init_time_ut = 1123372800 ;
>>>> APCP:valid_time = "20050808_000000" ;
>>>> APCP:valid_time_ut = 1123459200 ;
>>>> APCP:accum_time = "24 hours" ;
>>>>
>>>> // global attributes:
>>>> :FileOrigins = "File
>>>> ../out/pcp_combine/sample_fcst_24L_2005080800V_24A.nc generated
>>>> 20080722_174544 UTC on host pigpen by the PCP-Combine tool" ;
>>>> :RunCommand = "Sum: Files from
>>>> ../data/sample_fcst/2005080700
>>>> with accumulations of 3 hours." ;
>>>> :Projection = "Lambert Conformal" ;
>>>> :p1_deg = "25.000000 degrees_north" ;
>>>> :p2_deg = "25.000000 degrees_north" ;
>>>> :p0_deg = "12.190000 degrees_north" ;
>>>> :l0_deg = "-133.459000 degrees_east" ;
>>>> :lcen_deg = "-95.000000 degrees_east" ;
>>>> :d_km = "40.635000 km" ;
>>>> :r_km = "6367.470000 km" ;
>>>> :nx = "185 grid_points" ;
>>>> :ny = "129 grid_points" ;
>>>> }
>>>>
>>>>
>>>> Erik Noble wrote:
>>>>> Dear John,
>>>>> I was told today to not use the CMORPH data.
>>>>> So I am changing the TRMM data set that I am using into netcdf.
>>>>>
>>>>> You gave me these steps to follow earlier:
>>>>> (1) Make sure that the precip fields are named APCP to match the GRIB code
>>>>> abbreviation for accumulated precip. And ensure that the units are in "mm"
>>>>> which is the prescribed unit for APCP.
>>>>>
>>>>> "OK"
>>>>>
>>>>> (2) Add variable attributes to the APCP variable to match the attributes
>>>>> that are included for the variables created by the PCP-Combine tool.
>>>>> "Could you clarify this?"
>>>>>
>>>>> (3) Add global attributes defining the grid on which the data resides.
>>>>> "Do you mean, specify a region?"
>>>>>
>>>>> On Mon, Jul 14, 2008 at 3:57 PM, John Halley Gotway
>>>>> <johnhg at rap.ucar.edu>wrote:
>>>>>
>>>>>> Erik,
>>>>>>
>>>>>> Good luck with your upcoming defense.
>>>>>>
>>>>>> Generally speaking, the Grid-Stat and MODE tools in MET may be used to
>>>>>> compare any two gridded fields that are in GRIB format or the NetCDF
>>>>>> format
>>>>>> MET expects. So from a technical standpoint, you could certainly compare
>>>>>> your model output to reanalysis data - which I believe is available GRIB
>>>>>> format.
>>>>>>
>>>>>> From a scientific standpoint, as to whether or not that's a worthwhile
>>>>>> thing to do, I can't provide much advice. In MET we're trying to provide
>>>>>> a
>>>>>> set of verification tools that's as flexible as possible. Exactly how you
>>>>>> use them though is up to you.
>>>>>>
>>>>>> John
>>>>>>
>>>>>> Erik Noble wrote:
>>>>>>
>>>>>>> This will be of great help.......Thank you!
>>>>>>>
>>>>>>> John, I have not been able to fully do the tutorial because I have to
>>>>>>> defend
>>>>>>> my thesis proposal in 12 days. My time with it has been sporadic.
>>>>>>>
>>>>>>> Does one need precip files in order to verify other variables in the WRF
>>>>>>> model like wind or temps? Can a user use reanalaysis with MET to verify a
>>>>>>> model?
>>>>>>> -Erik
>>>>>>>
>>>>>>> On 7/14/08 3:08 PM, "John Halley Gotway" <johnhg at rap.ucar.edu> wrote:
>>>>>>>
>>>>>>> Erik,
>>>>>>>> I apologize for the delay in getting back to you. Last week and this
>>>>>>>> week
>>>>>>>> have been very busy getting the next release of MET ready and getting
>>>>>>>> ready
>>>>>>>> for the WRF-Tutorial.
>>>>>>>>
>>>>>>>> Regarding precip data outside the US, I searched through the previous
>>>>>>>> MET
>>>>>>>> Help
>>>>>>>> emails, and did find someone with a problem very similar to yours. He
>>>>>>>> was
>>>>>>>> trying to use CMORPH precip satellite data
>>>>>>>> with MET. I've emailed him to find out if he'd be willing to provide
>>>>>>>> you
>>>>>>>> with
>>>>>>>> some basic instructions on how he went about doing it. Perhaps one of
>>>>>>>> you
>>>>>>>> could provide me with some information I could
>>>>>>>> post to the MET website describing how to use CMORPH data.
>>>>>>>>
>>>>>>>> I'll let you know when I hear back from him.
>>>>>>>>
>>>>>>>> John
>>>>>>>>
>>>>>>>> Erik Noble wrote:
>>>>>>>>
>>>>>>>>> Dear John, has anyone else used Met with precip data to verify models
>>>>>>>>> simulations outside of the united states?
>>>>>>>>>
>>>>>>>>> i am still getting stuck on finding precipitation data in Grib format
>>>>>>>>> and the conversion to Netcdf.
>>>>>>>>> -Erik
>>>>>>>>>
>>>>>>>>> On Tue, Jul 8, 2008 at 4:02 PM, John Halley Gotway <johnhg at rap.ucar.edu
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Erik,
>>>>>>>>>>
>>>>>>>>>> The ones
>>>>>>>>>> In future releases of MET we hope to more generally support the NetCDF
>>>>>>>>>> CF
>>>>>>>>>> convention. Then we could at least remove the variable name
>>>>>>>>>> requirement.
>>>>>>>>>> But as MET currently stands (for version 1.1), you'd need to set it
>>>>>>>>>> up
>>>>>>>>>> as
>>>>>>>>>> listed above.
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> Erik Noble wrote:
>>>>>>>>>>
>>>>>>>>>>> Dear John,
>>>>>>>>>>> I have asked for this data in GRIB but no response.
>>>>>>>>>>>
>>>>>>>>>>> "I could help you set up the format for the NetCDF files so that MET
>>>>>>>>>>> would be able to read it."
>>>>>>>>>>> What do I need to do once i provide you an netcdf file?
>>>>>>>>>>> -Erik
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jul 3, 2008 at 1:52 AM, John Halley Gotway <
>>>>>>>>>>> johnhg at rap.ucar.edu>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Erik,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm sorry, I'm not familiar with global precip analyses in GRIB. I
>>>>>>>>>>>> know
>>>>>>>>>>>> that StageII and StageIV precip analyses are available in GRIB
>>>>>>>>>>>> format.
>>>>>>>>>>>> I'd
>>>>>>>>>>>> refer you to Lacey Holland on this but she recently left NCAR for a
>>>>>>>>>>>> new
>>>>>>>>>>>> job.
>>>>>>>>>>>>
>>>>>>>>>>>> I would guess that there would be HDF -> NetCDF converters out
>>>>>>>>>>>> there.
>>>>>>>>>>>>
>>>>>>>>>>>> If you do find a good source for this data in GRIB, let me know and
>>>>>>>>>>>> I
>>>>>>>>>>>> can
>>>>>>>>>>>> add a link to it on the MET website.
>>>>>>>>>>>>
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>> Erik Noble wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe you know more about this than me......
>>>>>>>>>>>>> What global precipitation data is available in GRIB already? All
>>>>>>>>>>>>> the
>>>>>>>>>>>>> ones I encounter are ither binary or HDF format.
>>>>>>>>>>>>> -Erik
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jul 2, 2008 at 11:51 AM, John Halley Gotway
>>>>>>>>>>>>> <johnhg at rap.ucar.edu>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Erik,
>>>>>>>>>>>>>
>>>>>>>>>>>>> For use with MET, the easiest thing would be to get the data into
>>>>>>>>>>>>> GRIB1
>>>>>>>>>>>>> format. Then you'd be able to use it directly in MET.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If that's too difficult, you could get it into NetCDF format, and I
>>>>>>>>>>>>> could
>>>>>>>>>>>>> help you set up the format for the NetCDF files so that MET would
>>>>>>>>>>>>> be
>>>>>>>>>>>>> able
>>>>>>>>>>>>> to
>>>>>>>>>>>>> read it. In future releases of MET, we hope to more fully support
>>>>>>>>>>>>> the
>>>>>>>>>>>>> NetCDF CF conventions. For the current release though, we'd just
>>>>>>>>>>>>> need
>>>>>>>>>>>>> to
>>>>>>>>>>>>> make your NetCDF file look like the output of the PCP-Combine tool
>>>>>>>>>>>>> (in
>>>>>>>>>>>>> MET).
>>>>>>>>>>>>> Then the stats tools would be able to read the NetCDF file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> Erik Noble wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dear MET help,
>>>>>>>>>>>>> If I have access to precipitation observation data that is only
>>>>>>>>>>>>> available in binary, can I still use it with MET.
>>>>>>>>>>>>> This site says that it has a fortran code that I can use to "read"
>>>>>>>>>>>>> the
>>>>>>>>>>>>> data, but I ahve to figure out what format I want the output in.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do I need to do in oder to use it with MET?
>>>>>>>>>>>>> -Erik
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>> From: Gareth Berry <gareth at atmos.albany.edu>
>>>>>>>>>>>>> Date: Mon, Jun 30, 2008 at 4:39 PM
>>>>>>>>>>>>> Subject: Re: WAF submission
>>>>>>>>>>>>> To: enoble at giss.nasa.gov
>>>>>>>>>>>>> Cc: chris at atmos.albany.edu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Erik,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Chris asked me to reply to you about the cmorph data.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The data is in a binary format, there is fortran code in the
>>>>>>>>>>>>> readme
>>>>>>>>>>>>> file that you can use to read it
>>>>>>>>>>>>> and export to your favourite data format (netcdf etc).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ftp://ftp.cpc.ncep.noaa.gov/precip/global_CMORPH/3-hourly_025deg/RE
>>>>>>>>>>>>> ADME
>>>>>>>>>>>>> .cmorph.025deg_3-hourly
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gareth.
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Met_help mailing list
>>>>>>>>>>>>> Met_help at mailman.ucar.edu
>>>>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>>>>>>>>>>
>
More information about the Met_help
mailing list