[Met_help] Re: Netcdf file
John Halley Gotway
johnhg at rap.ucar.edu
Thu Jul 31 13:23:38 MDT 2008
Erik,
Currently, there's no way to search the MET_Help emails publicly. However, I searched though them myself and did not see any questions related to using TRMM data other than yours.
I searched on the NCL website and found a way of computing Unix Time using the "ut_inv_calendar" NCL routine. This is taken from an NCL talk posting
(http://www.ncl.ucar.edu/Support/talk_archives/2006/0191.html):
(Note that the -7:00 is to adjust from Mountain Time to UTC)
Example:
units = "seconds since 1970-1-1 00:00:0.0 -7:00"
Time = ut_inv_calendar(2006, 2, 26 ,9 , 14, 37.0 ,units, 0)
print(Time)
The value of "Time" should be the unix time corresponding to the date/time of 02/26/2006 09:14:37.
So you could compute the unix time for the model initialization and valid times and then add those as variable attributes to the NetCDF files.
Hope that helps.
John
Erik Noble wrote:
> Hi. I spent all day on this. I couldn't figure out how else to include the
> other variables (except for accumulation time = 3 hours = 3600sec X3).
>
> How can I search the archives to see if anyone else has used TRMM?
> -Erik
>
>
>
> On 7/30/08 3:34 PM, "John Halley Gotway" <johnhg at rap.ucar.edu> wrote:
>
>> Erik,
>>
>> MET is expecting to read a 2D variable, i.e. APCP(lat, lon).
>>
>> However, your NetCDF variable is a 3D one: APCP(time, lat, lon)
>> I'm pretty sure that would be a problem. I'd suggest writing each 2D APCP
>> field out to it's own file for each time. In addition, MET will be looking
>> for the following attributes:
>> APCP:init_time_ut (model initialization time in unixtime)
>> APCP:valid_time_ut (forecast valid time in unixtime)
>> APCP:accum_time (accumulation time in seconds)
>>
>> I realize that converting the data to NetCDF like this is a hassle. I hope
>> that once we implement code to support a more generalized NetCDF format, do
>> things like this will be a lot easier.
>>
>> John
>>
>> Erik Noble wrote:
>>> Dear john, I am almost there.
>>>
>>> Do I need to include any other attributes here? Will this be enough for MET?
>>>
>>> Erik at noble:/Volumes/Data_and_Models/ncl_scripts:ncdump -h
>>> /Volumes/Data_and_Models/Precipitation_data/observed/TRMM/3B42.2006082612.nc
>>> netcdf 3B42.2006082612 {
>>> dimensions:
>>> time = UNLIMITED ; // (1 currently)
>>> lat = 400 ;
>>> lon = 1440 ;
>>> variables:
>>> double time(time) ;
>>> time:units = "hours since 1998-01-01 00:00:0.0" ;
>>> float lat(lat) ;
>>> lat:units = "degrees_north" ;
>>> lat:long_name = "latitude" ;
>>> float lon(lon) ;
>>> lon:units = "degrees_east" ;
>>> lon:long_name = "longitude" ;
>>> int date(time) ;
>>> int datesec(time) ;
>>> float APCP(time, lat, lon) ;
>>> APCP:long_name = "Total Precipitation" ;
>>> APCP:missing_value = -9999.f ;
>>> APCP:_FillValue = -9999.f ;
>>> APCP:Conventions = "None" ;
>>> APCP:level = "SFC" ;
>>> APCP:units = "mm" ;
>>> APCP:grib_code = 61 ;
>>> APCP:avg_period = "0000-00-00 01:00:00" ;
>>> APCP:delta_t = "0000-00-00 03:00:00" ;
>>> APCP:hdf_name = "precipitation" ;
>>>
>>> // global attributes:
>>> :Projection = "Lambert Conformal" ;
>>> :creation_date = "Wed Jul 30 14:58:04 EDT 2008" ;
>>> :Conventions = "None" ;
>>> :source_html = "\n",
>>>
>>> "http://disc.sci.gsfc.nasa.gov/data/datapool/TRMM/01_Data_Products/02_Gridde
>>> d/06_3-hour_Gpi_Cal_3B_42/index.html\n",
>>> "" ;
>>> :title = "TRMM: 3B42" ;
>>> }
>>> Erik at noble:/Volumes/Data_and_Models/ncl_scripts:
>>>
>>>
>>>
>>> On 7/29/08 4:53 PM, "John Halley Gotway" <johnhg at rap.ucar.edu> wrote:
>>>
>>>> Erik,
>>>>
>>>> MET currently supports reading gridded data in GRIB format. If you have
>>>> gridded data you'd like to use in MET, that isn't in GRIB format, you have 2
>>>> choices. The first is to convert it to GRIB
>>>> format (NCL may be able to do this). The second is to convert it a NetCDF
>>>> format that mimics the output of the PCP-Combine tool.
>>>>
>>>> I was giving you info about how to mimic the output of PCP-Combine. You can
>>>> see sample output of PCP-Combine in the METv1.1/out/pcp_combine directory.
>>>>
>>>> As for what you choose and how you go about converting your data, that'll be
>>>> up to you. I have limited experience with NCL. Starting with an existing
>>>> conversion script like you mentioned seems like
>>>> a good way to go, but I'm guessing you'll need to make some modifications to
>>>> get it in a format that MET will like.
>>>>
>>>> I'm busy the remainder of this week and will be out of the office for 10
>>>> days
>>>> starting on Monday. I'll be returning August 14th.
>>>>
>>>> If you choose to convert your data to NetCDF, let me point you to the
>>>> routine
>>>> in MET that reads those NetCDF files. The routine is called
>>>> read_pcp_combine_netcdf in the file named
>>>> METv1.1/lib/vx_met_util/read_pcp_combine_netcdf.cc.
>>>>
>>>> In that routine, you'll see that MET reads the following from the NetCDF
>>>> file:
>>>> (1) Grid projection information from the global attributes.
>>>> (2) The precip variable named using the GRIB code abbreviation for the
>>>> field.
>>>> (3) The "level" attribute of the precip variable.
>>>> (4) The "init_time_ut" attribute of the precip variable.
>>>> (5) The "valid_time_ut" attribute of the precip variable.
>>>> (6) The "accum_time" attribute of the precip variable.
>>>>
>>>> John
>>>>
>>>> Erik Noble wrote:
>>>>> Oh.......wait. So to clarify, we are doing this step in order to avoid
>>>>> using
>>>>> PCP Combine? I missed that.
>>>>> -Erik
>>>>>
>>>>>
>>>>> On 7/29/08 3:36 PM, "John Halley Gotway" <johnhg at rap.ucar.edu> wrote:
>>>>>
>>>>>> Erik,
>>>>>>
>>>>>> Listed below is a NetCDF header dump of one of the output files from the
>>>>>> METv1.1 test script for pcp_combine.
>>>>>>
>>>>>> Regarding (2), I'd suggest adding the same type of variable attributes
>>>>>> that
>>>>>> are listed here:
>>>>>> - grib_code = 61
>>>>>> - units = "kg/m^2" or "mm"
>>>>>> - level = "SFC"
>>>>>> - _FillValue = -9999.f
>>>>>> - init_time = "20050807_000000"
>>>>>> - init_time_ut = 1123372800
>>>>>> - valid_time = "20050808_000000"
>>>>>> - valid_time_ut = 1123459200
>>>>>> - accum_time = "24 hours"
>>>>>>
>>>>>> The "_ut" times listed above are in unixtime - the number of seconds that
>>>>>> have
>>>>>> elapsed since Jan 1, 1970. A fancy "date" command can be used to give you
>>>>>> the
>>>>>> unixtime.
>>>>>> For example, the unixtime corresponding to 8am on July 29, 2008 is given
>>>>>> by
>>>>>> the following command: date -ud '2008-07-29 UTC 08:00:00' +%s
>>>>>>
>>>>>> Regarding (3), please the following global attributes define the grid on
>>>>>> which
>>>>>> this data resides:
>>>>>> - Projection = "Lambert Conformal" ;
>>>>>> - p1_deg = "25.000000 degrees_north" ;
>>>>>> - p2_deg = "25.000000 degrees_north" ;
>>>>>> - p0_deg = "12.190000 degrees_north" ;
>>>>>> - l0_deg = "-133.459000 degrees_east" ;
>>>>>> - lcen_deg = "-95.000000 degrees_east" ;
>>>>>> - d_km = "40.635000 km" ;
>>>>>> - r_km = "6367.470000 km" ;
>>>>>> - nx = "185 grid_points" ;
>>>>>> - ny = "129 grid_points" ;
>>>>>>
>>>>>> This example is for a lat/lon grid. You could also use lambert conformal
>>>>>> or
>>>>>> polar stereographic. If you have a sample GRIB file already on the grid
>>>>>> you're using, please send it to me, and I can help
>>>>>> you figure out how to specify it in the NetCDF file.
>>>>>>
>>>>>> Hope that helps,
>>>>>> John
>>>>>>
>>>>>> netcdf sample_fcst_24L_2005080800V_24A {
>>>>>> dimensions:
>>>>>> lat = 129 ;
>>>>>> lon = 185 ;
>>>>>> variables:
>>>>>> float lat(lat, lon) ;
>>>>>> lat:units = "degrees_north" ;
>>>>>> lat:long_name = "Latitude" ;
>>>>>> lat:_FillValue = -9999.f ;
>>>>>> float lon(lat, lon) ;
>>>>>> lon:units = "degrees_east" ;
>>>>>> lon:long_name = "Longitude" ;
>>>>>> lon:_FillValue = -9999.f ;
>>>>>> float APCP(lat, lon) ;
>>>>>> APCP:grib_code = 61 ;
>>>>>> APCP:units = "kg/m^2" ;
>>>>>> APCP:long_name = "Total precipitation" ;
>>>>>> APCP:level = "SFC" ;
>>>>>> APCP:_FillValue = -9999.f ;
>>>>>> APCP:init_time = "20050807_000000" ;
>>>>>> APCP:init_time_ut = 1123372800 ;
>>>>>> APCP:valid_time = "20050808_000000" ;
>>>>>> APCP:valid_time_ut = 1123459200 ;
>>>>>> APCP:accum_time = "24 hours" ;
>>>>>>
>>>>>> // global attributes:
>>>>>> :FileOrigins = "File
>>>>>> ../out/pcp_combine/sample_fcst_24L_2005080800V_24A.nc generated
>>>>>> 20080722_174544 UTC on host pigpen by the PCP-Combine tool" ;
>>>>>> :RunCommand = "Sum: Files from
>>>>>> ../data/sample_fcst/2005080700
>>>>>> with accumulations of 3 hours." ;
>>>>>> :Projection = "Lambert Conformal" ;
>>>>>> :p1_deg = "25.000000 degrees_north" ;
>>>>>> :p2_deg = "25.000000 degrees_north" ;
>>>>>> :p0_deg = "12.190000 degrees_north" ;
>>>>>> :l0_deg = "-133.459000 degrees_east" ;
>>>>>> :lcen_deg = "-95.000000 degrees_east" ;
>>>>>> :d_km = "40.635000 km" ;
>>>>>> :r_km = "6367.470000 km" ;
>>>>>> :nx = "185 grid_points" ;
>>>>>> :ny = "129 grid_points" ;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Erik Noble wrote:
>>>>>>> Dear John,
>>>>>>> I was told today to not use the CMORPH data.
>>>>>>> So I am changing the TRMM data set that I am using into netcdf.
>>>>>>>
>>>>>>> You gave me these steps to follow earlier:
>>>>>>> (1) Make sure that the precip fields are named APCP to match the GRIB
>>>>>>> code
>>>>>>> abbreviation for accumulated precip. And ensure that the units are in
>>>>>>> "mm"
>>>>>>> which is the prescribed unit for APCP.
>>>>>>>
>>>>>>> "OK"
>>>>>>>
>>>>>>> (2) Add variable attributes to the APCP variable to match the attributes
>>>>>>> that are included for the variables created by the PCP-Combine tool.
>>>>>>> "Could you clarify this?"
>>>>>>>
>>>>>>> (3) Add global attributes defining the grid on which the data resides.
>>>>>>> "Do you mean, specify a region?"
>>>>>>>
>>>>>>> On Mon, Jul 14, 2008 at 3:57 PM, John Halley Gotway
>>>>>>> <johnhg at rap.ucar.edu>wrote:
>>>>>>>
>>>>>>>> Erik,
>>>>>>>>
>>>>>>>> Good luck with your upcoming defense.
>>>>>>>>
>>>>>>>> Generally speaking, the Grid-Stat and MODE tools in MET may be used to
>>>>>>>> compare any two gridded fields that are in GRIB format or the NetCDF
>>>>>>>> format
>>>>>>>> MET expects. So from a technical standpoint, you could certainly
>>>>>>>> compare
>>>>>>>> your model output to reanalysis data - which I believe is available GRIB
>>>>>>>> format.
>>>>>>>>
>>>>>>>> From a scientific standpoint, as to whether or not that's a worthwhile
>>>>>>>> thing to do, I can't provide much advice. In MET we're trying to
>>>>>>>> provide
>>>>>>>> a
>>>>>>>> set of verification tools that's as flexible as possible. Exactly how
>>>>>>>> you
>>>>>>>> use them though is up to you.
>>>>>>>>
>>>>>>>> John
>>>>>>>>
>>>>>>>> Erik Noble wrote:
>>>>>>>>
>>>>>>>>> This will be of great help.......Thank you!
>>>>>>>>>
>>>>>>>>> John, I have not been able to fully do the tutorial because I have to
>>>>>>>>> defend
>>>>>>>>> my thesis proposal in 12 days. My time with it has been sporadic.
>>>>>>>>>
>>>>>>>>> Does one need precip files in order to verify other variables in the
>>>>>>>>> WRF
>>>>>>>>> model like wind or temps? Can a user use reanalaysis with MET to verify
>>>>>>>>> a
>>>>>>>>> model?
>>>>>>>>> -Erik
>>>>>>>>>
>>>>>>>>> On 7/14/08 3:08 PM, "John Halley Gotway" <johnhg at rap.ucar.edu> wrote:
>>>>>>>>>
>>>>>>>>> Erik,
>>>>>>>>>> I apologize for the delay in getting back to you. Last week and this
>>>>>>>>>> week
>>>>>>>>>> have been very busy getting the next release of MET ready and getting
>>>>>>>>>> ready
>>>>>>>>>> for the WRF-Tutorial.
>>>>>>>>>>
>>>>>>>>>> Regarding precip data outside the US, I searched through the previous
>>>>>>>>>> MET
>>>>>>>>>> Help
>>>>>>>>>> emails, and did find someone with a problem very similar to yours. He
>>>>>>>>>> was
>>>>>>>>>> trying to use CMORPH precip satellite data
>>>>>>>>>> with MET. I've emailed him to find out if he'd be willing to provide
>>>>>>>>>> you
>>>>>>>>>> with
>>>>>>>>>> some basic instructions on how he went about doing it. Perhaps one of
>>>>>>>>>> you
>>>>>>>>>> could provide me with some information I could
>>>>>>>>>> post to the MET website describing how to use CMORPH data.
>>>>>>>>>>
>>>>>>>>>> I'll let you know when I hear back from him.
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> Erik Noble wrote:
>>>>>>>>>>
>>>>>>>>>>> Dear John, has anyone else used Met with precip data to verify models
>>>>>>>>>>> simulations outside of the united states?
>>>>>>>>>>>
>>>>>>>>>>> i am still getting stuck on finding precipitation data in Grib format
>>>>>>>>>>> and the conversion to Netcdf.
>>>>>>>>>>> -Erik
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jul 8, 2008 at 4:02 PM, John Halley Gotway
>>>>>>>>>>> <johnhg at rap.ucar.edu
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Erik,
>>>>>>>>>>>>
>>>>>>>>>>>> The ones
>>>>>>>>>>>> In future releases of MET we hope to more generally support the
>>>>>>>>>>>> NetCDF
>>>>>>>>>>>> CF
>>>>>>>>>>>> convention. Then we could at least remove the variable name
>>>>>>>>>>>> requirement.
>>>>>>>>>>>> But as MET currently stands (for version 1.1), you'd need to set it
>>>>>>>>>>>> up
>>>>>>>>>>>> as
>>>>>>>>>>>> listed above.
>>>>>>>>>>>>
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>> Erik Noble wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear John,
>>>>>>>>>>>>> I have asked for this data in GRIB but no response.
>>>>>>>>>>>>>
>>>>>>>>>>>>> "I could help you set up the format for the NetCDF files so that
>>>>>>>>>>>>> MET
>>>>>>>>>>>>> would be able to read it."
>>>>>>>>>>>>> What do I need to do once i provide you an netcdf file?
>>>>>>>>>>>>> -Erik
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jul 3, 2008 at 1:52 AM, John Halley Gotway <
>>>>>>>>>>>>> johnhg at rap.ucar.edu>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Erik,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm sorry, I'm not familiar with global precip analyses in GRIB. I
>>>>>>>>>>>>> know
>>>>>>>>>>>>> that StageII and StageIV precip analyses are available in GRIB
>>>>>>>>>>>>> format.
>>>>>>>>>>>>> I'd
>>>>>>>>>>>>> refer you to Lacey Holland on this but she recently left NCAR for a
>>>>>>>>>>>>> new
>>>>>>>>>>>>> job.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would guess that there would be HDF -> NetCDF converters out
>>>>>>>>>>>>> there.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you do find a good source for this data in GRIB, let me know and
>>>>>>>>>>>>> I
>>>>>>>>>>>>> can
>>>>>>>>>>>>> add a link to it on the MET website.
>>>>>>>>>>>>>
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> Erik Noble wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe you know more about this than me......
>>>>>>>>>>>>> What global precipitation data is available in GRIB already? All
>>>>>>>>>>>>> the
>>>>>>>>>>>>> ones I encounter are ither binary or HDF format.
>>>>>>>>>>>>> -Erik
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jul 2, 2008 at 11:51 AM, John Halley Gotway
>>>>>>>>>>>>> <johnhg at rap.ucar.edu>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Erik,
>>>>>>>>>>>>>
>>>>>>>>>>>>> For use with MET, the easiest thing would be to get the data into
>>>>>>>>>>>>> GRIB1
>>>>>>>>>>>>> format. Then you'd be able to use it directly in MET.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If that's too difficult, you could get it into NetCDF format, and I
>>>>>>>>>>>>> could
>>>>>>>>>>>>> help you set up the format for the NetCDF files so that MET would
>>>>>>>>>>>>> be
>>>>>>>>>>>>> able
>>>>>>>>>>>>> to
>>>>>>>>>>>>> read it. In future releases of MET, we hope to more fully support
>>>>>>>>>>>>> the
>>>>>>>>>>>>> NetCDF CF conventions. For the current release though, we'd just
>>>>>>>>>>>>> need
>>>>>>>>>>>>> to
>>>>>>>>>>>>> make your NetCDF file look like the output of the PCP-Combine tool
>>>>>>>>>>>>> (in
>>>>>>>>>>>>> MET).
>>>>>>>>>>>>> Then the stats tools would be able to read the NetCDF file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> Erik Noble wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dear MET help,
>>>>>>>>>>>>> If I have access to precipitation observation data that is only
>>>>>>>>>>>>> available in binary, can I still use it with MET.
>>>>>>>>>>>>> This site says that it has a fortran code that I can use to "read"
>>>>>>>>>>>>> the
>>>>>>>>>>>>> data, but I ahve to figure out what format I want the output in.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do I need to do in oder to use it with MET?
>>>>>>>>>>>>> -Erik
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>> From: Gareth Berry <gareth at atmos.albany.edu>
>>>>>>>>>>>>> Date: Mon, Jun 30, 2008 at 4:39 PM
>>>>>>>>>>>>> Subject: Re: WAF submission
>>>>>>>>>>>>> To: enoble at giss.nasa.gov
>>>>>>>>>>>>> Cc: chris at atmos.albany.edu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Erik,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Chris asked me to reply to you about the cmorph data.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The data is in a binary format, there is fortran code in the
>>>>>>>>>>>>> readme
>>>>>>>>>>>>> file that you can use to read it
>>>>>>>>>>>>> and export to your favourite data format (netcdf etc).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ftp://ftp.cpc.ncep.noaa.gov/precip/global_CMORPH/3-hourly_025deg/RE
>>>>>>>>>>>>> ADME
>>>>>>>>>>>>> .cmorph.025deg_3-hourly
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gareth.
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Met_help mailing list
>>>>>>>>>>>>> Met_help at mailman.ucar.edu
>>>>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>>>>>>>>>>
>
More information about the Met_help
mailing list