[Met_help] Fwd: [Fwd: Re: Points of clarification for data conversion/usage in MODE]
Tressa Fowler
tressa at ucar.edu
Thu Mar 18 17:04:29 MDT 2010
Matt,
I asked Jamie Wolff, who is better with copygb than I am, for some
help. Here she gives an example of the output and grid specifications
when she runs a file through copygb and outputs grid 27.
Do you have the IDV software? I use it a lot to look at my grib files.
Also, you might try ncview to look at your nc files.
Tressa
Begin forwarded message:
> From: Jamie Wolff <jwolff at ucar.edu>
> Date: March 18, 2010 4:57:39 PM MDT
> To: Tressa Fowler <tressa at ucar.edu>
> Subject: Re: Fwd: [Met_help] [Fwd: Re: Points of clarification for
> data conversion/usage in MODE]
>
> Hi Tressa,
>
> I grabbed an old GFS grib1 file I had laying around and if I do a
> wgrib
> -V on the original file I get (just for reference):
> lefse:/d1/jwolff/v3.2/prerelease/NMM/DOMAINS/wrfhelp/metuser> wgrib -V
> -d 1 20051230_i12_f000_GFS003.grb
> rec 1:0:date 2005123012 HGT kpds5=7 kpds6=100 kpds7=950 levels=(3,182)
> grid=3 950 mb anl:
> HGT=Geopotential height [gpm]
> timerange 10 P1 0 P2 0 TimeU 1 nx 360 ny 181 GDS grid 0 num_in_ave 0
> missing 0
> center 7 subcenter 0 process 81 Table 2 scan: WE:NS winds(N/S)
> latlon: lat 90.000000 to -90.000000 by 1.000000 nxny 65160
> long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 0
> mode 128 bdsgrid 1
> min/max data 40.3 772.1 num bits 13 BDS_Ref 403 DecScale 1
> BinScale 0
>
> Then, if I run the command:
> copygb.exe -xg27 20051230_i12_f000_GFS003.grb
> grid27_20051230_i12_f000_GFS003.grb
>
> and do wgrib -V again I get:
> lefse:/d1/jwolff/v3.2/prerelease/NMM/DOMAINS/wrfhelp/metuser> wgrib -V
> -d 1 grid27_20051230_i12_f000_GFS003.grb
> rec 1:0:date 2005123012 HGT kpds5=7 kpds6=100 kpds7=950 levels=(3,182)
> grid=27 950 mb anl:
> HGT=Geopotential height [gpm]
> timerange 10 P1 0 P2 0 TimeU 1 nx 65 ny 65 GDS grid 5 num_in_ave 0
> missing 0
> center 7 subcenter 0 process 81 Table 2 scan: WE:SN winds(grid)
> polar stereo: Lat1 -20.826000 Long1 -125.000000 Orient -80.000000
> north pole (65 x 65) Dx 381000 Dy 381000 scan 64 mode 9
> min/max data 59.3 764.7 num bits 13 BDS_Ref 593 DecScale 1
> BinScale 0
>
> I have also attached two images one from the original GFS grib file
> (gfs.png) and one from the interpolated GFS file to grid 27
> (grid27.png). So, you can see that my grib file contains the domain I
> would expect. I am not sure how this user is viewing his domain (or
> if
> he even has). I have used idv for creating the images I have
> attached.
> I think maybe he is just looking at the wgrib output and
> misinterpreting
> the values printed from that (I don't see how he gets the lat to run
> from -20 to 60). I am not familiar with how polar stereo domains are
> encoded in grib so I am not exactly sure how to interpret those
> numbers
> (in particular I am not quite sure what Lat1 and Long1 represent)
> but I
> know that Orient is the longitude that runs from the pole to the south
> and appears "right-side-up" in the domain plot (centered around that
> longitude)). Hopefully that makes a little bit of sense! I just look
> at the final domain image and it looks appropriate to me!
>
> jamie
>
> Tressa Fowler wrote:
>> Hi Jamie,
>>
>> This user is having trouble with copygb, and he's past what we can
>> help him with while John is out. Can you maybe help?
>>
>> Tressa
>>
>> Begin forwarded message:
>>
>>> *From: *Matthew Souders <matt.souders at gmail.com
>>> <mailto:matt.souders at gmail.com>>
>>> *Date: *March 18, 2010 3:16:54 PM MDT
>>> *To: *Tressa Fowler <tressa at ucar.edu <mailto:tressa at ucar.edu>>
>>> *Subject: **Re: [Met_help] [Fwd: Re: Points of clarification for
>>> data conversion/usage in MODE]*
>>>
>>> The default specifications for a grid-27 NCEP grid say the north
>>> pole
>>> should be at grid point 33,33...but when I look at the ncdump file,
>>> it says it's a 65X65 grid as you would expect....it says it's polar
>>> stereographic as you would expect, but the boundaries run from
>>> -20.8N
>>> (20 South??) to 60 north and from -80 west to -125 W
>>> (roughly)...that
>>> doesn't make any sense. The pole is nowhere in that projection. I
>>> know you aren't expert in gribw usage...unfortunately, the person
>>> who
>>> wrote the code is difficult to track down in e-mail to ask these
>>> sorts of questions - he basically maintains at least a half dozen
>>> very important script libraries for processing GRIB files and doing
>>> other model-related work and therefore doesn't have much time to
>>> address day to day problems like this. Do you know of anyone who
>>> has
>>> extensive experience working with gribw?
>>>
>>> On 3/16/10, *Tressa Fowler* <tressa at ucar.edu
>>> <mailto:tressa at ucar.edu>> wrote:
>>>
>>> Hi Matt,
>>>
>>> I'm not an expert at copygb, but I think you are not specifying
>>> the grid you actually want. Your command line asks for NCEP grid
>>> #27. Here's the description of that grid.
>>> 4225-point (65x65) N. Hemisphere polar stereographic grid
>>> oriented 80W; Pole at (33,33).
>>>
>>> Perhaps you should see the NCEP web page to choose a different
>>> grid? http://www.nco.ncep.noaa.gov/pmb/docs/on388/tableb.html
>>>
>>> Thanks,
>>>
>>> Tressa
>>>
>>> On Mar 16, 2010, at 4:07 PM, John Halley Gotway wrote:
>>>
>>>> Matt,
>>>>
>>>> I'm forwarding your message to the "met_help at ucar.edu
>>>> <mailto:met_help at ucar.edu>" email address.
>>>> Unfortunately, I'm out of the office at all week. I'm with my
>>>> son's who's
>>>> recovering from a burst appendix. He's recovering nicely, but
>>>> we'll be
>>>> here with him all week.
>>>>
>>>> Perhaps Tressa, Randy, or Paul could help you debug the issues
>>>> you're
>>>> experiencing. I should be available to help out sometime next
>>>> week.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> ---------------------------- Original Message
>>>> ----------------------------
>>>> Subject: Re: Points of clarification for data conversion/usage
>>>> in MODE
>>>> From: "Matthew Souders" <matt.souders at gmail.com
>>>> <mailto:matt.souders at gmail.com>>
>>>> Date: Tue, March 16, 2010 1:29 pm
>>>> To: "John Halley Gotway" <johnhg at ucar.edu
>>>> <mailto:johnhg at ucar.edu>>
>>>>
>>>> --------------------------------------------------------------------------
>>>>
>>>> OK...never mind about the last problem...my apologies for
>>>> bothering you. I
>>>> got PCP_Combine to run, but there is definitely something wrong
>>>> with the
>>>> copygb-converted GRIB file, because I am not getting all of the
>>>> northern
>>>> hemisphere when I use NCView to look at the resulting .nc
>>>> file...it should
>>>> produce a 91 by 360 pole-centered grid...instead it produces a
>>>> 65X65 square
>>>> centered on Italy (roughly) that only reveals most of Europe and
>>>> north
>>>> Africa (and the Middle East).
>>>>
>>>> The precip data looks odd too...3-hourly values with little
>>>> areas of 5+
>>>> inches...hard to fathom that.
>>>>
>>>> On 3/15/10, Matthew Souders <matt.souders at gmail.com
>>>> <mailto:matt.souders at gmail.com>> wrote:
>>>>>
>>>>> I'm probably missing something ridiculously obvious, but I am
>>>>> trying to run
>>>>> a quick experiment using a GFS .grb file I acquired today. I
>>>>> ran the
>>>> copygb
>>>>> utility on grid 003 GFS global domain (gave it the following
>>>>> command:
>>>>> copygb -xg 27 gfs_3_20100313_0000_003.grb
>>>>> gfs_27_20100313_0000_003.grb),
>>>> and
>>>>> that may or may have worked properly (it produced a file, but
>>>>> the size of
>>>>> that file was an order of magnitude smaller...which I don't
>>>>> think is
>>>>> right...and it gave me a handful of messages about remaining
>>>>> memory being
>>>>> allocated to some file called polatev0.f)...but I decided to
>>>>> see what it
>>>>> looked like by running the PCP_Combine utility...I got into the
>>>>> METv2.0
>>>> main
>>>>> directory and issued this command:
>>>>>
>>>>> bin/pcp_combine \ -sum 20100313_030000 3 20100313_030000 3 \
>>>>> test_ncfile_gfs.nc <http://test_ncfile_gfs.nc>
>>>>>
>>>>> I only have a single time step in my .grb, so all this should
>>>>> do is add up
>>>>> one time step and re-output the data in .nc format.
>>>>>
>>>>> It's giving me this error though:
>>>>>
>>>>> ERROR: process_sum_args() -> The init_time and valid_time
>>>>> command line
>>>>> arguments must be in the form YYYYMMDD_HHMMSS
>>>>>
>>>>> What confuses me...is that my arguments ARE in that format.
>>>>> Not sure why
>>>>> it's giving me that error. Any ideas?
>>>>>
>>>>>
>>>>> On 3/12/10, John Halley Gotway <johnhg at ucar.edu
>>>>> <mailto:johnhg at ucar.edu>> wrote:
>>>>>>
>>>>>> Matt,
>>>>>>
>>>>>> Yeah, polar stereographic is fine. Actually, *northern* polar
>>>>>> stereographic is currently supported in MET - not southern yet
>>>>>> for some
>>>>>> reason. You just need to figure out what to put in the NetCDF
>>>>>> header to define it. If you can get your hands on a GRIB
>>>>>> file, you could
>>>>>> run it through the copygb tool to interpolate it a polar
>>>>>> stereographic
>>>>>> projection. Then run it through the PCP-Combine tool
>>>>>> (the operation you do with PCP-Combine doesn't really matter).
>>>>>> And then
>>>>>> look at the NetCDF output file.
>>>>>>
>>>>>> Just let me know if you get stuck anywhere.
>>>>>>
>>>>>>
>>>>>> John
>>>>>>
>>>>>>
>>>>>> Matthew Souders wrote:
>>>>>>> Boy...it seems like no matter what alley I drive down,
>>>>>>> there's an acme
>>>>>> anvil
>>>>>>> rigged to fall on my head. I should change my last name to
>>>>>>> Coyote.
>>>>>>>
>>>>>>> In all seriousness, I do appreciate you going through the
>>>>>>> various tools
>>>>>> and
>>>>>>> ideas with me...at this point...I am thinking it would be a
>>>>>>> less painful
>>>>>>> task to change map projection of my data BEFORE I try to
>>>>>>> create the
>>>>>> netCDF
>>>>>>> than it would be to convert it later. I al going to have to
>>>>>>> write some
>>>>>> kind
>>>>>>> of MATLAB script to convert every piece of data I have to a
>>>>>>> projection I
>>>>>> can
>>>>>>> use...at which point I can then convert to netCDF and read
>>>>>>> the thing
>>>>>> into
>>>>>>> MODE, I assume? Will MODE recognize a netCDF file in a polar
>>>>>> stereographic
>>>>>>> projection?
>>>>>>>
>>>>>>> On 3/12/10, John Halley Gotway <johnhg at ucar.edu
>>>>>>> <mailto:johnhg at ucar.edu>> wrote:
>>>>>>>> Matt,
>>>>>>>>
>>>>>>>> Here's a link regarding that:
>>>>>>>> http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#grib
>>>>>>>>
>>>>>>>> Converting GRIB to NetCDF is pretty easy. The tool
>>>>>>>> "ncl_convert2nc"
>>>>>> does
>>>>>>>> it for you. Going the other way is much more difficult.
>>>>>>>> And really,
>>>>>> I've
>>>>>>>> never done it before.
>>>>>>>>
>>>>>>>>
>>>>>>>> John
>>>>>>>>
>>>>>>>> Matthew Souders wrote:
>>>>>>>>> Unfortunately, I have not used NCL before. Given my time
>>>>>>>>> limmitations
>>>>>> in
>>>>>>>>> completely this analysis, I am thinking that will be
>>>>>>>> impractical. regular
>>>>>>>>> lat/lon doesn't really work because I have to split the
>>>>>>>>> grid in half
>>>>>> and
>>>>>>>>> then track objects that straddle the divide somehow...which
>>>>>>>>> causes
>>>>>> lots
>>>>>>>> of
>>>>>>>>> problems. There is software out there that can convert a
>>>>>>>>> netCDF to a
>>>>>>>> GRIB,
>>>>>>>>> isn't there? I seem to recall that being out there in some
>>>>>>>>> form.
>>>>>>>>>
>>>>>>>>> On 3/12/10, John Halley Gotway <johnhg at ucar.edu
>>>>>>>>> <mailto:johnhg at ucar.edu>> wrote:
>>>>>>>>>> Matt,
>>>>>>>>>>
>>>>>>>>>> There is no general tool for regridding NetCDF files from
>>>>>>>>>> one grid to
>>>>>>>>>> another. You may be able to write an NCL (NCAR Command
>>>>>>>>>> Language)
>>>>>> script
>>>>>>>> to
>>>>>>>>>> do so, but there's a learning curve if you haven't
>>>>>>>>>> used NCL before.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Matthew Souders wrote:
>>>>>>>>>>> One reason I started down the long and scary road of
>>>>>>>>>>> trying to
>>>>>> create
>>>>>>>>>> GRIB
>>>>>>>>>>> files was to gain access to the copygb utility in order
>>>>>>>>>>> to convert
>>>>>> my
>>>>>>>>>>> regular lat/lon grid to polar stereographic without too
>>>>>>>>>>> much
>>>>>>>>>> hardship. Is
>>>>>>>>>>> there a way to convert a netCDF file in regular lat/lon
>>>>>>>>>>> to polar
>>>>>>>>>>> stereographic simply?
>>>>>>>>>>>
>>>>>>>>>>> It does seem like using netCDF is a better way to go...I
>>>>>>>>>>> believe
>>>>>> even
>>>>>>>>>> MATLAB
>>>>>>>>>>> (my main dta interface tool) has a netCDF writer...it
>>>>>>>>>>> definitely has
>>>>>>>> (and
>>>>>>>>>> I
>>>>>>>>>>> use) a netCDF reader. But I need to figure out how to
>>>>>>>>>>> change map
>>>>>>>>>>> projections and how exactly to set up the netCDF file so
>>>>>>>>>>> that MODE
>>>>>> can
>>>>>>>>>>> understand it.
>>>>>>>>>>>
>>>>>>>>>>> I will take a look at the output from
>>>>>>>>>>> P2P_Combine...that file
>>>>>>>> definitely
>>>>>>>>>>> created successfully when I ran the test script...I can
>>>>>>>>>>> see it in my
>>>>>>>>>>> directory.
>>>>>>>>>>>
>>>>>>>>>>> On 3/12/10, John Halley Gotway <johnhg at ucar.edu
>>>>>>>>>>> <mailto:johnhg at ucar.edu>> wrote:
>>>>>>>>>>>> Matt,
>>>>>>>>>>>>
>>>>>>>>>>>> Creating a GRIB file is a rather ambitious task I have
>>>>>>>>>>>> never
>>>>>> actually
>>>>>>>>>> done
>>>>>>>>>>>> myself. If I were in your position, I'd actually
>>>>>>>>>>>> reformat your
>>>>>> binary
>>>>>>>>>> data
>>>>>>>>>>>> into a NetCDF file that looks like the output of
>>>>>>>>>>>> the PCP-Combine tool. MODE can read either GRIB data or
>>>>>>>>>>>> the NetCDF
>>>>>>>>>> output
>>>>>>>>>>>> of PCP-Combine. There are a lot more utilities out
>>>>>>>>>>>> there for
>>>>>> writing
>>>>>>>>>> NetCDF
>>>>>>>>>>>> files than there are for writing GRIB files.
>>>>>>>>>>>>
>>>>>>>>>>>> Use the "ncdump" utility to view the NetCDF output from
>>>>>>>>>>>> PCP-Combine
>>>>>> in
>>>>>>>>>> the
>>>>>>>>>>>> "METv2.0/out/pcp_combine" directory. These files are
>>>>>>>>>>>> created when
>>>>>> you
>>>>>>>>>> run
>>>>>>>>>>>> the test script included with MET. And just
>>>>>>>>>>>> mimick the file structure with your data.
>>>>>>>>>>>>
>>>>>>>>>>>> If you choose to go that route, you'll probably need
>>>>>>>>>>>> some help
>>>>>>>> choosing
>>>>>>>>>> the
>>>>>>>>>>>> variable name and setting up the grid definition
>>>>>>>>>>>> information in
>>>>>> NetCDF
>>>>>>>>>>>> header. We're working on expanding our support for
>>>>>>>>>>>> NetCDF files for the next release of MET. They're a lot
>>>>>>>>>>>> easier to
>>>>>>>> work
>>>>>>>>>>>> with than GRIB files are.
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know how you plan to proceed and what other
>>>>>>>>>>>> questions you
>>>>>> have.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Matthew Souders wrote:
>>>>>>>>>>>>> Hello John,
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is Matthew Souders (Stony Brook University) - I
>>>>>>>>>>>>> apologize for
>>>>>>>>>> taking
>>>>>>>>>>>> up
>>>>>>>>>>>>> a bit more of your time, but I am running into some
>>>>>>>>>>>>> trouble
>>>>>> getting
>>>>>>>>>> data
>>>>>>>>>>>>> read into MODE for which I thought you might have a
>>>>>>>>>>>>> solution on
>>>>>> hand
>>>>>>>> to
>>>>>>>>>>>>> help. My basic data is a single variable on a 2.5 by
>>>>>>>>>>>>> 2.5 degree
>>>>>>>>>> regular
>>>>>>>>>>>>> lat/lon grid in fortran direct access BINARY format. I
>>>>>>>>>>>>> have been
>>>>>>>>>> trying
>>>>>>>>>>>> to
>>>>>>>>>>>>> get it intelligibly read into GRIB format (the form
>>>>>>>>>>>>> MODE needs)
>>>>>> for a
>>>>>>>>>>>> month
>>>>>>>>>>>>> now using Wesley Ebusizaki's gribw utility; however,
>>>>>>>>>>>>> you need a
>>>>>>>> rather
>>>>>>>>>>>>> significant background in C scripting to rewrite his
>>>>>>>>>>>>> basic code
>>>>>> and
>>>>>>>>>> make
>>>>>>>>>>>>> proper use of the PDS file writer. As I understand
>>>>>>>>>>>>> it...MODE
>>>>>> needs
>>>>>>>>>> this
>>>>>>>>>>>> PDS
>>>>>>>>>>>>> file to operate properly. I can create a GRIB using
>>>>>>>>>>>>> the default
>>>>>> PDS
>>>>>>>>>>>> file,
>>>>>>>>>>>>> but that doesn't accurately describe the data in the
>>>>>>>>>>>>> GRIB.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I tried e-mail Wesley on this matter but he is
>>>>>>>>>>>>> understandably
>>>>>>>> extremely
>>>>>>>>>>>> busy
>>>>>>>>>>>>> and limits support to installation and debugging
>>>>>>>>>>>>> issues. He
>>>>>>>> basically
>>>>>>>>>>>> just
>>>>>>>>>>>>> directed me to the GRIB standard release. I was
>>>>>>>>>>>>> wondering if you
>>>>>> had
>>>>>>>>>> any
>>>>>>>>>>>>> experience getting binary data read into MODE or if you
>>>>>>>>>>>>> knew
>>>>>> someone
>>>>>>>>>> who
>>>>>>>>>>>>> might be able to help?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I also would like to know how exactly the MODE tool
>>>>>>>>>>>>> uses the PDS
>>>>>>>>>> file? I
>>>>>>>>>>>>> ask because I am trying to figure out how picky it is
>>>>>>>>>>>>> w/r/t what
>>>>>> is
>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>>> PDS file vs. what is in the actual GRIB file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks in advance for your time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Matt Souders.
>>>>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>> <untitled-
>>>> [2].html>_______________________________________________
>>>> Met_help mailing list
>>>> Met_help at mailman.ucar.edu <mailto:Met_help at mailman.ucar.edu>
>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/met_help/attachments/20100318/0be5e3ed/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gfs.png
Type: image/png
Size: 20674 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/met_help/attachments/20100318/0be5e3ed/attachment-0002.png
-------------- next part --------------
A non-text attachment was scrubbed...
Name: grid27.png
Type: image/png
Size: 39378 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/met_help/attachments/20100318/0be5e3ed/attachment-0003.png
More information about the Met_help
mailing list