[Met_help] Using MET with MADIS data
John Henderson
jhenders at aer.com
Tue Mar 2 10:42:46 MST 2010
John,
I've uploaded a sample file that you have requested. The filename is not
very informative (20061022_1000), but it is a ncdf file straight from
the MADIS server.
Please let me know if the line of thinking that prompted this email
might be fruitful for our immediate needs. Otherwise, we may actually
start to write a MADIS ncdf -> ASCII script for ingest into MET. We
really would like to use mesonet obs in MET and I've been told that
retrieving multiple ncdf format files from MADIS is much cleaner than
retrieving ASCII format.
John
On 3/1/10 3:57 PM, John Halley Gotway wrote:
> John,
>
> Our of curiosity, I'm wondering if you could send me a sample MADIS NetCDF file that you'd like to use in MET. I'd like to see how it compares to some AWIPS point observation data we'd read in the past.
>
> You could post a sample file to our anonymous ftp site:
> ftp ftp.rap.ucar.edu
> username = anonymous
> password = "your email address"
> cd incoming/irap/met_help
> put "your file(s)"
> bye
>
> I'd appreciate it.
>
> Thanks,
> John
>
> John Halley Gotway wrote:
>
>> John,
>>
>> I looked back through our MET-Help emails and did find another user going through the same process as you. And he actually sent us some the scripts he was using for converting MADIS to ASCII for
>> surface observations. I wanted to pass them along to you in case they're of use. However I think he was starting with an ASCII version of MADIS data instead of NetCDF.
>>
>> His scripts are attached.
>>
>> Thanks,
>> John
>>
>> -------- Original Message --------
>> Subject: MADIS_ASCII2NC scripts/files
>> Date: Wed, 4 Mar 2009 12:49:06 -0600
>> From: Case, Jonathan (MSFC-VP61)[Other]<jonathan.case-1 at nasa.gov>
>> To: John Halley Gotway<johnhg at rap.ucar.edu>
>> References:<A9BB815E34BC6947A9DF9EABB15F147101B5EB18 at NDMSEVS31A.ndc.nasa.gov> <53403.128.117.65.62.1236126600.squirrel at mail.rap.ucar.edu>
>> <A9BB815E34BC6947A9DF9EABB15F147101B5EB5F at NDMSEVS31A.ndc.nasa.gov> <49AEA2B6.7080403 at rap.ucar.edu>
>>
>> John,
>>
>> I've bundled up a bunch of scripts/files I used to re-format and process
>> the MADIS sfc observations for ascii2nc. The file is attached to this
>> email.
>>
>> Please keep in mind that these scripts were just written (except for
>> dayadd, which I've used in many other scripts), so it's still "drafty".
>> Also, it's only set up to work on sfc observations only, not any
>> upper-air data yet.
>>
>> Take care,
>> Jonathan
>>
>> John Henderson wrote:
>>
>>> John,
>>>
>>> Yes, of course, reformatting into the catch-all ASCII format seems like
>>> the best start. Thanks for reminding me about that option. Will let you
>>> know how the MADIS->ASCII script works if I choose to jump into the task!
>>>
>>> John
>>>
>>> On 2/26/10 1:38 PM, John Halley Gotway wrote:
>>>
>>>> John,
>>>>
>>>> I can tell you that unfortunately it won't be as simple as renaming
>>>> some variables. You know, rather than modifying the NetCDF file
>>>> directly and trying to figure out the details of what the PB2NC
>>>> output looks like, I'd suggest using an intermediate ASCII file.
>>>>
>>>> It'd probably be easiest to parse the MADIS observations and dump them
>>>> out in the ASCII format that the ASCII2NC tool can read. Then run
>>>> them through ASCII2NC and use them in Point-Stat. I know
>>>> that's a lot of steps, but I do think it'd be the most
>>>> straight-forward way to go.
>>>>
>>>> You can find details of what the supported ASCII format in the MET
>>>> User's Guide:
>>>>
>>>> http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v2.0_rev2.pdf
>>>>
>>>>
>>>> Keep us posted on know how it goes. And if you're MADIS -> ASCII
>>>> script works well, we'd be happy to post it on the MET website for
>>>> other users.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> John Henderson wrote:
>>>>
>>>>
>>>>> Hi John,
>>>>>
>>>>> Thanks for your quick response.
>>>>>
>>>>> Would anyone be able to provide some basic details (e.g., what PB2NC
>>>>> fields are required) of the PB2NC ncdf files to help guide me with
>>>>> trying to match up the MADIS ncdf and PB2NC ncdf files? Perhaps it's a
>>>>> simple as a renaming of a field or two...
>>>>>
>>>>> Thanks.
>>>>>
>>>>> John
>>>>>
>>>>> On 2/26/10 12:34 PM, John Halley Gotway wrote:
>>>>>
>>>>>
>>>>>> John,
>>>>>>
>>>>>> Yes, the file formats are different, but I have never worked directly
>>>>>> with MADIS data myself. I imagine you'd need to reformat the MADIS
>>>>>> observations to make them look like the NetCDF output of
>>>>>> PB2NC. There have been other MET users who would like to use MADIS
>>>>>> observations, but we haven't had time to add direct support for their
>>>>>> use in MET.
>>>>>>
>>>>>> I'll forward your request to others in our group to see where we can
>>>>>> put it in our development timeline.
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>> John Henderson wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hello again John,
>>>>>>>
>>>>>>> Thanks again for your previous help.
>>>>>>>
>>>>>>> Perhaps you could also save me a little time by explaining whether
>>>>>>> MADIS
>>>>>>> data can be easily (or not so easily) used in WRF-MET.
>>>>>>> Specifically, is
>>>>>>> the ncdf file format generated by pb2nc different from what might be
>>>>>>> expected in, say, a MADIS mesonet ncdf file?
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> John
>>>>>>>
>>>>>>> On 1/7/10 2:39 PM, John Halley Gotway wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> John,
>>>>>>>>
>>>>>>>> Unfortunately, no. The PB2NC tool is not set up to extract
>>>>>>>> precipitation observations from PREPBUFR files. It's only set up to
>>>>>>>> extract observations of Pressure, Specific Humidity, Temperature,
>>>>>>>> Height, and U/V-components of wind.
>>>>>>>>
>>>>>>>> For verification of QPF, you'll either need to use a gridded
>>>>>>>> analysis,
>>>>>>>> like StageII/IV (http://www.emc.ncep.noaa.gov/mmb/ylin/pcpanl/stage2
>>>>>>>> and http://www.emc.ncep.noaa.gov/mmb/ylin/pcpanl/stage4) or
>>>>>>>> reformat ASCII versions of gauge data using the ASCII2NC tool for
>>>>>>>> point verification.
>>>>>>>>
>>>>>>>> I've done both of these.
>>>>>>>>
>>>>>>>> We would like to expand our support for the observation types packed
>>>>>>>> into the PREPBUFR files, but right now I'm really not sure what is/is
>>>>>>>> not in there.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>> John Henderson wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> John,
>>>>>>>>>
>>>>>>>>> Oh, okay. Just thought I'd point it out! Thanks for explaining...
>>>>>>>>> Unfortunately, right at the moment I don't have any long (>99-h)
>>>>>>>>> forecast fields handy to investigate my concerns directly.
>>>>>>>>>
>>>>>>>>> There is one other somewhat-related item you might be able to
>>>>>>>>> help me
>>>>>>>>> out with. I'm having trouble finding precipitation in
>>>>>>>>> pb2nc-generated
>>>>>>>>> netcdf files. I've converted a number of PrepBufr obs files but
>>>>>>>>> cannot
>>>>>>>>> find a grib code that should contain precip. Do you have any
>>>>>>>>> experience
>>>>>>>>> working with precip in PrepBufr format?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On 1/7/10 2:14 PM, John Halley Gotway wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> John,
>>>>>>>>>>
>>>>>>>>>> No those "HH" arguments to PCP-Combine should be fine. Perhaps
>>>>>>>>>> listing the HH as 2-digits is misleading, but it'll read the
>>>>>>>>>> integer
>>>>>>>>>> number of hours (any number of digits) and handle them correctly.
>>>>>>>>>>
>>>>>>>>>> To make sure, I ran PCP-Combine on some that that included
>>>>>>>>>> accumulations intervals> 100 hours. All three commands
>>>>>>>>>> (sum, add,
>>>>>>>>>> and subtract) worked fine even when the accumulation interval on
>>>>>>>>>> the
>>>>>>>>>> command line was> 100.
>>>>>>>>>>
>>>>>>>>>> Please let me know if anything else comes up.
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> John Henderson wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hello again John,
>>>>>>>>>>>
>>>>>>>>>>> Regarding the> 99-h forecast lead time problem from a
>>>>>>>>>>> couple of
>>>>>>>>>>> weeks
>>>>>>>>>>> ago... I believe that PCP_COMBINE will also need to be modified
>>>>>>>>>>> since
>>>>>>>>>>> there is an argument to the code that is of the format HH (for the
>>>>>>>>>>> 'sum'
>>>>>>>>>>> option: out_accum).
>>>>>>>>>>>
>>>>>>>>>>> Please let me know if this is actually the case.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> John Henderson
>>>>>>>>>>>
>>>>>>>>>>> On 12/23/09 5:46 PM, John Halley Gotway wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> John,
>>>>>>>>>>>>
>>>>>>>>>>>> I checked and found out that this really is a problem for the
>>>>>>>>>>>> STAT-Analysis and MODE-Analysis tools. When I ran STAT-Analysis
>>>>>>>>>>>> and
>>>>>>>>>>>> specified a lead time of "-fcst_lead 114", it errored out with an
>>>>>>>>>>>> error
>>>>>>>>>>>> message about not being able to parse that time.
>>>>>>>>>>>>
>>>>>>>>>>>> So I modified the library code which parses these strings to
>>>>>>>>>>>> allow
>>>>>>>>>>>> for
>>>>>>>>>>>> the
>>>>>>>>>>>> lead hours to be either 2 or 3 digits long.
>>>>>>>>>>>>
>>>>>>>>>>>> I've posted the fix to:
>>>>>>>>>>>> http://www.dtcenter.org/met/users/support/known_issues/METv2.0/index.php
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for pointing out this problem, and please let me know
>>>>>>>>>>>> if you
>>>>>>>>>>>> uncover any more issues.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> John,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the clarification of everything.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll let you know if I uncover any other problems with
>>>>>>>>>>>>> triple-digit
>>>>>>>>>>>>> lead
>>>>>>>>>>>>> times, but, otherwise, I'll await your fix.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the quick responses!
>>>>>>>>>>>>>
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 12/23/09 4:37 PM, John Halley Gotway wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> John,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding WPP, I really don't think that you'll have a problem.
>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>> currently working on a project with folks from NOAA where we're
>>>>>>>>>>>>>> creating
>>>>>>>>>>>>>> forecasts out to 114 hours and are running the most recently
>>>>>>>>>>>>>> released
>>>>>>>>>>>>>> version of WPP just fine. You would only potentially have an
>>>>>>>>>>>>>> issue if
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> were running your forecasts out> 255 hours.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding issues with MET, you're probably correct. I think
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>> come up in your use of the MODE-Analysis and STAT-Analysis
>>>>>>>>>>>>>> tools.
>>>>>>>>>>>>>> That's
>>>>>>>>>>>>>> the place you'd be specifying the forecast lead time in the
>>>>>>>>>>>>>> config
>>>>>>>>>>>>>> file
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>> on the command line. I believe it would be a pretty
>>>>>>>>>>>>>> easy fix
>>>>>>>>>>>>>> involving
>>>>>>>>>>>>>> changes to the "timestring_to_sec" routine in the file
>>>>>>>>>>>>>> METv2.0/lib/vx_cal/time_strings.cc.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However, before making any changes, I'd like to do some
>>>>>>>>>>>>>> testing to
>>>>>>>>>>>>>> make
>>>>>>>>>>>>>> sure it really is a problem. And once I verify that, I'll work
>>>>>>>>>>>>>> on a
>>>>>>>>>>>>>> fix.
>>>>>>>>>>>>>> If I isolate the problem and come up with a fix, I'd post it to
>>>>>>>>>>>>>> the MET
>>>>>>>>>>>>>> website and update the package of bug fixes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'll let you know how it pans out. Please let me know if, in
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>> use
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> MET, you find another place where it's a problem to have
>>>>>>>>>>>>>> forecast
>>>>>>>>>>>>>> hours>
>>>>>>>>>>>>>> 99.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello John,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the prompt response. Perhaps I had briefly come
>>>>>>>>>>>>>>> across
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> WPP email and thought it was actually related to MET...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I believe - according to the user manual - that MET allows the
>>>>>>>>>>>>>>> user to
>>>>>>>>>>>>>>> specify forecast lead time as HH[MMSS], which seems to me to
>>>>>>>>>>>>>>> prevent
>>>>>>>>>>>>>>> leads times> 99h. Am I mistaken?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding WPP...I intend to verify precip accumulation in MET
>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>> year-long series of 5.5-day forecasts that overlap at the
>>>>>>>>>>>>>>> beginning of
>>>>>>>>>>>>>>> each forecast run to avoid spin-up problems. It's a pretty
>>>>>>>>>>>>>>> standard
>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>> of generating year-long WRF simulations. However, that means
>>>>>>>>>>>>>>> that I
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>> be applying WPP to WRF forecasts with forecast lead times
>>>>>>>>>>>>>>> of 12
>>>>>>>>>>>>>>> h to
>>>>>>>>>>>>>>> 132
>>>>>>>>>>>>>>> h. I hadn't anticipated any problems with WPP,
>>>>>>>>>>>>>>> unfortunately. I
>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>> vague memory that someone had come up with a fix for the
>>>>>>>>>>>>>>> long-lead
>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>> problem, but I'm not sure. Would you be able to investigate
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> status
>>>>>>>>>>>>>>> for me?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 12/23/09 2:51 PM, John Halley Gotway wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> John,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I looked back through the old MET-Help emails and couldn't
>>>>>>>>>>>>>>>> find
>>>>>>>>>>>>>>>> anything
>>>>>>>>>>>>>>>> that would explain a problem of using fcst_lead>= 100
>>>>>>>>>>>>>>>> hours. So
>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> aware of a problem with that. I did find one email
>>>>>>>>>>>>>>>> related to this but I believe that was due to an incorrectly
>>>>>>>>>>>>>>>> packed
>>>>>>>>>>>>>>>> GRIB
>>>>>>>>>>>>>>>> file, not a problem in MET itself.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, there are some issues in WPP (WRF-PostProcessor) for
>>>>>>>>>>>>>>>> accumulated precip when the forecast lead time extends
>>>>>>>>>>>>>>>> beyond 255
>>>>>>>>>>>>>>>> hours.
>>>>>>>>>>>>>>>> The time value of 255+ overflows the single byte
>>>>>>>>>>>>>>>> that's
>>>>>>>>>>>>>>>> allocated in the GRIB record to store that time info.
>>>>>>>>>>>>>>>> This is
>>>>>>>>>>>>>>>> mostly
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>> issue for accumulated precip since you have to store 2 times
>>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>> GRIB
>>>>>>>>>>>>>>>> record - accumulation starting and ending times. I'm
>>>>>>>>>>>>>>>> not sure of the status of this issue, but if you run
>>>>>>>>>>>>>>>> across it,
>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>> look into it more.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd say, proceed with trying to use MET to verify
>>>>>>>>>>>>>>>> forecasts>= 100
>>>>>>>>>>>>>>>> hours.
>>>>>>>>>>>>>>>> And if you run into any problems, just let us know
>>>>>>>>>>>>>>>> through
>>>>>>>>>>>>>>>> met_help at ucar.edu.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If you haven't already done so, I would suggest retrieving
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> latest
>>>>>>>>>>>>>>>> set of bug fixes for METv2.0 from:
>>>>>>>>>>>>>>>> http://www.dtcenter.org/met/users/support/known_issues/METv2.0/index.php
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hope that helps,
>>>>>>>>>>>>>>>> John Halley Gotway
>>>>>>>>>>>>>>>> johnhg at ucar.edu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> John Henderson wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When first I installed WRF-MET I believe I saw a discussion
>>>>>>>>>>>>>>>>> somewhere
>>>>>>>>>>>>>>>>> about how to apply WRF-MET to forecasts that require a
>>>>>>>>>>>>>>>>> fcst_lead of
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> least 100 h, however, I can't seem to find it now. Can you
>>>>>>>>>>>>>>>>> point me
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the right direction?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> John Henderson
>>>>>>>>>>>>>>>>> AER, Inc.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Met_help mailing list
>>>>>>>>>>>>>>>>> Met_help at mailman.ucar.edu
>>>>>>>>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
More information about the Met_help
mailing list