[Met_help] Using MET with MADIS data
John Halley Gotway
johnhg at ucar.edu
Fri Feb 26 11:38:14 MST 2010
John,
I can tell you that unfortunately it won't be as simple as renaming some variables. You know, rather than modifying the NetCDF file directly and trying to figure out the details of what the PB2NC
output looks like, I'd suggest using an intermediate ASCII file.
It'd probably be easiest to parse the MADIS observations and dump them out in the ASCII format that the ASCII2NC tool can read. Then run them through ASCII2NC and use them in Point-Stat. I know
that's a lot of steps, but I do think it'd be the most straight-forward way to go.
You can find details of what the supported ASCII format in the MET User's Guide:
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v2.0_rev2.pdf
Keep us posted on know how it goes. And if you're MADIS -> ASCII script works well, we'd be happy to post it on the MET website for other users.
Thanks,
John
John Henderson wrote:
> Hi John,
>
> Thanks for your quick response.
>
> Would anyone be able to provide some basic details (e.g., what PB2NC
> fields are required) of the PB2NC ncdf files to help guide me with
> trying to match up the MADIS ncdf and PB2NC ncdf files? Perhaps it's a
> simple as a renaming of a field or two...
>
> Thanks.
>
> John
>
> On 2/26/10 12:34 PM, John Halley Gotway wrote:
>> John,
>>
>> Yes, the file formats are different, but I have never worked directly
>> with MADIS data myself. I imagine you'd need to reformat the MADIS
>> observations to make them look like the NetCDF output of
>> PB2NC. There have been other MET users who would like to use MADIS
>> observations, but we haven't had time to add direct support for their
>> use in MET.
>>
>> I'll forward your request to others in our group to see where we can
>> put it in our development timeline.
>>
>> Thanks,
>> John
>>
>> John Henderson wrote:
>>
>>> Hello again John,
>>>
>>> Thanks again for your previous help.
>>>
>>> Perhaps you could also save me a little time by explaining whether MADIS
>>> data can be easily (or not so easily) used in WRF-MET. Specifically, is
>>> the ncdf file format generated by pb2nc different from what might be
>>> expected in, say, a MADIS mesonet ncdf file?
>>>
>>> Thanks.
>>>
>>> John
>>>
>>> On 1/7/10 2:39 PM, John Halley Gotway wrote:
>>>
>>>> John,
>>>>
>>>> Unfortunately, no. The PB2NC tool is not set up to extract
>>>> precipitation observations from PREPBUFR files. It's only set up to
>>>> extract observations of Pressure, Specific Humidity, Temperature,
>>>> Height, and U/V-components of wind.
>>>>
>>>> For verification of QPF, you'll either need to use a gridded analysis,
>>>> like StageII/IV (http://www.emc.ncep.noaa.gov/mmb/ylin/pcpanl/stage2
>>>> and http://www.emc.ncep.noaa.gov/mmb/ylin/pcpanl/stage4) or
>>>> reformat ASCII versions of gauge data using the ASCII2NC tool for
>>>> point verification.
>>>>
>>>> I've done both of these.
>>>>
>>>> We would like to expand our support for the observation types packed
>>>> into the PREPBUFR files, but right now I'm really not sure what is/is
>>>> not in there.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> John Henderson wrote:
>>>>
>>>>
>>>>> John,
>>>>>
>>>>> Oh, okay. Just thought I'd point it out! Thanks for explaining...
>>>>> Unfortunately, right at the moment I don't have any long (>99-h)
>>>>> forecast fields handy to investigate my concerns directly.
>>>>>
>>>>> There is one other somewhat-related item you might be able to help me
>>>>> out with. I'm having trouble finding precipitation in pb2nc-generated
>>>>> netcdf files. I've converted a number of PrepBufr obs files but cannot
>>>>> find a grib code that should contain precip. Do you have any
>>>>> experience
>>>>> working with precip in PrepBufr format?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> John
>>>>>
>>>>> On 1/7/10 2:14 PM, John Halley Gotway wrote:
>>>>>
>>>>>
>>>>>> John,
>>>>>>
>>>>>> No those "HH" arguments to PCP-Combine should be fine. Perhaps
>>>>>> listing the HH as 2-digits is misleading, but it'll read the integer
>>>>>> number of hours (any number of digits) and handle them correctly.
>>>>>>
>>>>>> To make sure, I ran PCP-Combine on some that that included
>>>>>> accumulations intervals> 100 hours. All three commands (sum, add,
>>>>>> and subtract) worked fine even when the accumulation interval on the
>>>>>> command line was> 100.
>>>>>>
>>>>>> Please let me know if anything else comes up.
>>>>>>
>>>>>> John
>>>>>>
>>>>>> John Henderson wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hello again John,
>>>>>>>
>>>>>>> Regarding the> 99-h forecast lead time problem from a couple of
>>>>>>> weeks
>>>>>>> ago... I believe that PCP_COMBINE will also need to be modified
>>>>>>> since
>>>>>>> there is an argument to the code that is of the format HH (for the
>>>>>>> 'sum'
>>>>>>> option: out_accum).
>>>>>>>
>>>>>>> Please let me know if this is actually the case.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> John Henderson
>>>>>>>
>>>>>>> On 12/23/09 5:46 PM, John Halley Gotway wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> John,
>>>>>>>>
>>>>>>>> I checked and found out that this really is a problem for the
>>>>>>>> STAT-Analysis and MODE-Analysis tools. When I ran STAT-Analysis
>>>>>>>> and
>>>>>>>> specified a lead time of "-fcst_lead 114", it errored out with an
>>>>>>>> error
>>>>>>>> message about not being able to parse that time.
>>>>>>>>
>>>>>>>> So I modified the library code which parses these strings to allow
>>>>>>>> for
>>>>>>>> the
>>>>>>>> lead hours to be either 2 or 3 digits long.
>>>>>>>>
>>>>>>>> I've posted the fix to:
>>>>>>>> http://www.dtcenter.org/met/users/support/known_issues/METv2.0/index.php
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks for pointing out this problem, and please let me know if you
>>>>>>>> uncover any more issues.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> John,
>>>>>>>>>
>>>>>>>>> Thanks for the clarification of everything.
>>>>>>>>>
>>>>>>>>> I'll let you know if I uncover any other problems with
>>>>>>>>> triple-digit
>>>>>>>>> lead
>>>>>>>>> times, but, otherwise, I'll await your fix.
>>>>>>>>>
>>>>>>>>> Thanks for the quick responses!
>>>>>>>>>
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On 12/23/09 4:37 PM, John Halley Gotway wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> John,
>>>>>>>>>>
>>>>>>>>>> Regarding WPP, I really don't think that you'll have a problem.
>>>>>>>>>> I'm
>>>>>>>>>> currently working on a project with folks from NOAA where we're
>>>>>>>>>> creating
>>>>>>>>>> forecasts out to 114 hours and are running the most recently
>>>>>>>>>> released
>>>>>>>>>> version of WPP just fine. You would only potentially have an
>>>>>>>>>> issue if
>>>>>>>>>> you
>>>>>>>>>> were running your forecasts out> 255 hours.
>>>>>>>>>>
>>>>>>>>>> Regarding issues with MET, you're probably correct. I think this
>>>>>>>>>> would
>>>>>>>>>> come up in your use of the MODE-Analysis and STAT-Analysis tools.
>>>>>>>>>> That's
>>>>>>>>>> the place you'd be specifying the forecast lead time in the
>>>>>>>>>> config
>>>>>>>>>> file
>>>>>>>>>> or
>>>>>>>>>> on the command line. I believe it would be a pretty
>>>>>>>>>> easy fix
>>>>>>>>>> involving
>>>>>>>>>> changes to the "timestring_to_sec" routine in the file
>>>>>>>>>> METv2.0/lib/vx_cal/time_strings.cc.
>>>>>>>>>>
>>>>>>>>>> However, before making any changes, I'd like to do some
>>>>>>>>>> testing to
>>>>>>>>>> make
>>>>>>>>>> sure it really is a problem. And once I verify that, I'll work
>>>>>>>>>> on a
>>>>>>>>>> fix.
>>>>>>>>>> If I isolate the problem and come up with a fix, I'd post it to
>>>>>>>>>> the MET
>>>>>>>>>> website and update the package of bug fixes.
>>>>>>>>>>
>>>>>>>>>> I'll let you know how it pans out. Please let me know if, in
>>>>>>>>>> your
>>>>>>>>>> use
>>>>>>>>>> of
>>>>>>>>>> MET, you find another place where it's a problem to have forecast
>>>>>>>>>> hours>
>>>>>>>>>> 99.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hello John,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the prompt response. Perhaps I had briefly come
>>>>>>>>>>> across
>>>>>>>>>>> the
>>>>>>>>>>> WPP email and thought it was actually related to MET...
>>>>>>>>>>>
>>>>>>>>>>> I believe - according to the user manual - that MET allows the
>>>>>>>>>>> user to
>>>>>>>>>>> specify forecast lead time as HH[MMSS], which seems to me to
>>>>>>>>>>> prevent
>>>>>>>>>>> leads times> 99h. Am I mistaken?
>>>>>>>>>>>
>>>>>>>>>>> Regarding WPP...I intend to verify precip accumulation in MET
>>>>>>>>>>> for a
>>>>>>>>>>> year-long series of 5.5-day forecasts that overlap at the
>>>>>>>>>>> beginning of
>>>>>>>>>>> each forecast run to avoid spin-up problems. It's a pretty
>>>>>>>>>>> standard
>>>>>>>>>>> way
>>>>>>>>>>> of generating year-long WRF simulations. However, that means
>>>>>>>>>>> that I
>>>>>>>>>>> will
>>>>>>>>>>> be applying WPP to WRF forecasts with forecast lead times of 12
>>>>>>>>>>> h to
>>>>>>>>>>> 132
>>>>>>>>>>> h. I hadn't anticipated any problems with WPP, unfortunately. I
>>>>>>>>>>> have a
>>>>>>>>>>> vague memory that someone had come up with a fix for the
>>>>>>>>>>> long-lead
>>>>>>>>>>> time
>>>>>>>>>>> problem, but I'm not sure. Would you be able to investigate the
>>>>>>>>>>> status
>>>>>>>>>>> for me?
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> On 12/23/09 2:51 PM, John Halley Gotway wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> John,
>>>>>>>>>>>>
>>>>>>>>>>>> I looked back through the old MET-Help emails and couldn't find
>>>>>>>>>>>> anything
>>>>>>>>>>>> that would explain a problem of using fcst_lead>= 100
>>>>>>>>>>>> hours. So
>>>>>>>>>>>> I'm
>>>>>>>>>>>> not
>>>>>>>>>>>> aware of a problem with that. I did find one email
>>>>>>>>>>>> related to this but I believe that was due to an incorrectly
>>>>>>>>>>>> packed
>>>>>>>>>>>> GRIB
>>>>>>>>>>>> file, not a problem in MET itself.
>>>>>>>>>>>>
>>>>>>>>>>>> However, there are some issues in WPP (WRF-PostProcessor) for
>>>>>>>>>>>> accumulated precip when the forecast lead time extends
>>>>>>>>>>>> beyond 255
>>>>>>>>>>>> hours.
>>>>>>>>>>>> The time value of 255+ overflows the single byte that's
>>>>>>>>>>>> allocated in the GRIB record to store that time info. This is
>>>>>>>>>>>> mostly
>>>>>>>>>>>> an
>>>>>>>>>>>> issue for accumulated precip since you have to store 2 times
>>>>>>>>>>>> in the
>>>>>>>>>>>> GRIB
>>>>>>>>>>>> record - accumulation starting and ending times. I'm
>>>>>>>>>>>> not sure of the status of this issue, but if you run across it,
>>>>>>>>>>>> I can
>>>>>>>>>>>> look into it more.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd say, proceed with trying to use MET to verify
>>>>>>>>>>>> forecasts>= 100
>>>>>>>>>>>> hours.
>>>>>>>>>>>> And if you run into any problems, just let us know
>>>>>>>>>>>> through
>>>>>>>>>>>> met_help at ucar.edu.
>>>>>>>>>>>>
>>>>>>>>>>>> If you haven't already done so, I would suggest retrieving the
>>>>>>>>>>>> latest
>>>>>>>>>>>> set of bug fixes for METv2.0 from:
>>>>>>>>>>>> http://www.dtcenter.org/met/users/support/known_issues/METv2.0/index.php
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hope that helps,
>>>>>>>>>>>> John Halley Gotway
>>>>>>>>>>>> johnhg at ucar.edu
>>>>>>>>>>>>
>>>>>>>>>>>> John Henderson wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> When first I installed WRF-MET I believe I saw a discussion
>>>>>>>>>>>>> somewhere
>>>>>>>>>>>>> about how to apply WRF-MET to forecasts that require a
>>>>>>>>>>>>> fcst_lead of
>>>>>>>>>>>>> at
>>>>>>>>>>>>> least 100 h, however, I can't seem to find it now. Can you
>>>>>>>>>>>>> point me
>>>>>>>>>>>>> in
>>>>>>>>>>>>> the right direction?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> John Henderson
>>>>>>>>>>>>> AER, Inc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Met_help mailing list
>>>>>>>>>>>>> Met_help at mailman.ucar.edu
>>>>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
More information about the Met_help
mailing list