[Met_help] [Fwd: Re: point_stat]

Tue May 5 09:32:48 MDT 2009

Mark,

John forwarded your message in the hope that I might shed a little  
light on your question regarding correlation calculations in MET.

The correlations in MET are not calculated over space or time. They  
are not within field correlation values, but between two fields. They  
are calculated at a single time, between each forecast and observation  
pair in the domain, treating each pair as if it is independent from  
all other pairs (obviously false). The information you get from this  
is how well the forecasts and observations "match up" in a linear way.  
It gives you no information about how forecasts are related to the  
other surrounding forecasts in either space or time.

Hope that helps. Please let me know if you have further questions.

Tressa

On Apr 30, 2009, at 12:53 PM, John Halley Gotway wrote:

> Tressa and Barb,
>
> FYI - Here's a user who is trying to use MET to verify a single  
> station over the course of a month.  Please see the message traffic  
> below.
>
> I explained that MET isn't set up well to handle that case, but  
> outlined what he'd have to do to accomplish it.  He's decided not to  
> use MET because the steps are too cumbersome to accomplish what he
> wants to do.
>
> Tressa, there's a question in there about the CI's being computed in  
> space rather than in time.
>
> Feel free to send any advice or suggestions you might have.
>
> Thanks,
> John
>
> -------- Original Message --------
> Subject: Re: [Met_help] point_stat
> Date: Thu, 30 Apr 2009 12:48:37 -0600
> From: Mark Seefeldt <mark.seefeldt at colorado.edu>
> To: John Halley Gotway <johnhg at rap.ucar.edu>
> References: <49F64DAE.6030402 at colorado.edu> <49F6FEB3.70306 at rap.ucar.edu 
> >	<49F73450.9060501 at colorado.edu> <49F73F6D.9060804 at rap.ucar.edu> <49F74659.3080005 at colorado.edu 
> >
> <49F85BA8.3080307 at rap.ucar.edu> <49F8A2F6.5060507 at colorado.edu> <49F9A7A3.7060502 at rap.ucar.edu 
> >
>
> John,
>
> Thank you for the thorough description of MET in relation to my  
> current
> application.  Naturally, I am quite disappointed that it can not be  
> used
> for my current evaluation of the performance of WRF over time.  I
> appreciate the steps which you outlined.  Unfortunately they are just
> too cumbersome for this evaluation.  I have run 120 different WRF
> simulations with variations in the physics parameterizations.  Each
> simulation has a 50km and a 10km domain.  The simulations are one  
> month
> in length, or 720/744  3-hourly values.  To run point_stat over all of
> those values (120 * 720/744) for the two different observation  
> locations
> would make things quickly unbearable.  I will now return to my own
> methods of calculating model evaluation statistics, which  
> unfortunately
> do not include confidence intervals.
>
> I am a little confused as to how you can get a correlation value if  
> the
> verification if it is only done at a single point in time.  That would
> seem to indicate to me that the correlation is a spatial value, which
> has limited meaning as it depends on how one progresses through the
> observations spatially.  I'll take a closer look at the  
> documentation to
> answer that question.
>
> Thanks again for your assistance and providing clear answers.
> Unfortunately, I am going to have to shelve MET as a post-processing
> tool for WRF.
>
> Mark
>
> John Halley Gotway wrote:
>> Mark,
>>
>> Glad it's working now.  I did notice how your observations were laid
>> out and was wondering what type of verification you were trying to
>> do.
>>
>> Basically, you'd like to collect matched forecast/observation pairs
>> at a single location through time, and then compute statistics on
>> that set of matched pairs.  Unfortunately, MET isn't set up to handle
>> that type of task well.  You can use MET to do it, but at this point,
>> it's a bit more cumbersome than I'd like.
>>
>> Point-Stat is designed to compare a forecast field to a set of
>> observations at a SINGLE point in time.  Point-Stat able to aggregate
>> matched pairs in space but not in time, as you'd like. Unfortunately,
>> 'cat'ing together all of your forecast files does not have the
>> desired effect.  Since the output of WPP is one file for each valid
>> time, that's the type of data that MET expects.  When you specify the
>> forecast field as "PRES/Z0" in the configuration file, Point-Stat
>> looks in the input forecast file for a matching record.  It uses the
>> first one it finds, so it'd only use the data for the first valid
>> time in your file and ignore the rest of the records.
>>
>> Here's how you'd need to do this: (1) Do NOT cat together your
>> forecast GRIB files - keep them separate.  But all your observation
>> points can be in the same file. (2) Create a masking station id file
>> that lists the stations you'd like to verify (just "Barro", I
>> suppose). (3) In the Point-Stat config file, set the following:
>>
>> - Set "mask_sid" variable to point to that station id file. - Set
>> "beg_ds" and "end_ds" to define a matching time window around each
>> forecast valid time.  This should be set carefully so that you get
>> exactly one matched pair for Barro for each run.  You don't want to
>> accidentally include the one from the day before or the day after. -
>> Set the output_flag as follows to dump out only the matched pair
>> data: output_flag[] = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1 ]; -
>> Set interp_method[] = [ "UW_MEAN" ]; for unweighted mean. - Set
>> interp_width[] = [ 1 ]; to only use the nearest neighbor. - You may
>> want to consider using additional interpolation methods and widths.
>> That way you could see how the results change by smoothing over
>> larger areas.
>>
>> (4) For each forecast valid time, run Point-Stat once.  You need to
>> pass it the forecast file for that time and the point observation
>> file. (5) Now you've run Point-Stat about 30 times and have generated
>> about 30 STAT files - each only containing one MPR line (or multiple
>> if you use multiple interpolation methods).  To aggregate through
>> time, you can run the STAT-Analysis tool, passing to it the directory
>> containing those 30 STAT files "-lookin stat_dir".  You'll want to
>> run the "-job aggregate_stat" job with "-line_type MPR".  And you can
>> select the type of stats you want computed by setting the
>> "-out_line_type" argument.  If you've using multiple interpolation
>> methods, you can use the "-interp_mthd" and "-interp_pnts" arguments
>> to specify which matched pairs should go into the calculations.
>>
>> So that'd be the way to do it.  Sorry it's so cumbersome.  We do
>> realize that it'd be nice to perform this type of verification more
>> directly in a single step.  We're considering how best to support
>> this type of verification through time.
>>
>> Thanks, John
>>
>> Mark Seefeldt wrote:
>>> John,
>>>
>>> Thank you for posting the fix so quickly.  I have retrieved the set
>>> of patches and have recompiled MET.  I am now getting matched
>>> pairs.
>>>
>>> I appear to still be struggling in producing the output which I
>>> desire. I am guessing this is more of a user problem.  I'd like to
>>> step you through what I am trying to do and see if you have any
>>> tips to improve what I am doing.
>>>
>>> I have a month-long simulation from WRF.  The simulation runs for
>>> one month plus one-day, starting the last day of the previous
>>> month.  I have created GRIB files using WPP from the wrfout files.
>>> The GRIB files were created in the three-hour intervals of the
>>> history file.  I have cat'ed the GRIB files together making a
>>> single file which represents the entire month long simulation.
>>> That GRIB file is:  wppout_d01_1998-04-30_00.grb
>>>
>>> I have one month of observations for a given observation location.
>>> In this case the Barrow Baseline Surface Radiation Network (BSRN)
>>> observations for May 1998.  From the original observation file I
>>> have a program which creates a text file of the observations.  That
>>> text file is then processed by ascii2nc to create the netcdf input
>>> file for point_stat.  The netcdf file is:
>>> phy_sheba-barrow-d01-199805.nc
>>>
>>> The end desired result is a list of the continuous statistics for
>>> the entire month between the WRF model simulations for the nearest
>>> point to the Barrow observations in comparison to the actual Barrow
>>> observations. There would be a maximum of 744 observation/forecast
>>> pairs (31 days x 24 hours).  This represents a maximum and not the
>>> expected because there are some missing values.
>>>
>>> When I initially ran point_stat I only got 2 matching pairs.  I
>>> added the command-line options -valid_beg 19980501_00 and
>>> -valid_end 19980531_23.  I then got 743, 741, 703, 731, and 736
>>> matching pairs, depending on the variable of interest (i.e. T_2m).
>>> That is what I would expect.
>>>
>>> When looking at the CNT file things became a little more suspect.
>>> -I initially noticed the following: FCST_VALID_BEG :
>>> 19980501_000000 FCST_VALID_END : 19980501_000000 OBS_VALID_BEG :
>>> 19980501_000000 OBS_VALID_END : 19980531_230000 The OBS fields are
>>> what I would expect.  I would expect FCST_END to be 19980531_2300.
>>>
>>> -I also noticed that I do not have FSTDEV, FSTDEV_NCL, FSTDEV_NCU,
>>> FSTDEV_BCL, and FSTDEV_BCU values (all are listed as NA).  I am
>>> wondering if it is only using the forecast value for 19980501_0000,
>>> therefore it cannot calculate a FSTDEV.
>>>
>>> My questions lie in: -Is what I am trying to do reasonable? -Is the
>>> methodology which I am doing correct? -Why does the FCST_VALID_END
>>> not go to the end of the month? -Why do I not have FSTDEV values?
>>>
>>> Thanks
>>>
>>> Mark
>>>
>>> John Halley Gotway wrote:
>>>> Mark,
>>>>
>>>> I posted a fix for this issue.  Please retrieve the fix from the
>>>> MET Known Issues page:
>>>> http://www.dtcenter.org/met/users/support/known_issues/METv2.0/index.php
>>>>
>>>>
>>>> I'd suggest following the instructions in the "All Recommended
>>>> Updates" section.  There are now two bug fixes available and some
>>>> minor updates to the user's guide, and doing it this way, you'll
>>>> grab all of the updates.
>>>>
>>>> Feel free to write with any more questions or problems.
>>>>
>>>> Thanks, John
>>>>
>>>> Mark Seefeldt wrote:
>>>>> John,
>>>>>
>>>>> Thanks for the update.  Please pass around the fix when you
>>>>> have it completed.  The information you provided is valuable as
>>>>> it means that I can start processing the GRIB files for the
>>>>> complete evaluation.
>>>>>
>>>>> Mark
>>>>>
>>>>> John Halley Gotway wrote:
>>>>>> Mark,
>>>>>>
>>>>>> Thanks for sending the data.  I see what the problem is -
>>>>>> there's a bug in the library code that reads the valid time
>>>>>> of the GRIB forecast file.  It thinks it 2098 as opposed to
>>>>>> 1998.  So Point-Stat is looking for observation values that
>>>>>> are in the time window 20980430 +/- 5400 seconds.  And of
>>>>>> course, it doesn't find any!
>>>>>>
>>>>>> I'm headed out for the day, but I'll put together a fix and
>>>>>> send it to you tomorrow.
>>>>>>
>>>>>> In the meantime, try using the "-valid_beg" and "-valid_end"
>>>>>> command line options to manually set the matching time
>>>>>> window.  That should get you non-zero matched pairs.
>>>>>>
>>>>>> Thanks for finding this issue!
>>>>>>
>>>>>> John
>>>>>>
>>>>>> Mark Seefeldt wrote:
>>>>>>> John,
>>>>>>>
>>>>>>> Thank you for all of the tips and suggestions which you
>>>>>>> have provided. I have worked through the different items
>>>>>>> and I am still not getting matched pairs when I should be.
>>>>>>>
>>>>>>> I have uploaded the following files to the anonymous ftp:
>>>>>>> phy_sheba-barrow-d01-199805.nc - nc observation file
>>>>>>> phy_sheba-barrow-d01-199805.txt - text observation file
>>>>>>> wppout_d01_1998-04-30_00.grb - GRIB output from using
>>>>>>> WPPv3.1 PointStatConfig-phy_sheba - point_stat
>>>>>>> configuration file
>>>>>>>
>>>>>>> The WRF simulation is for an entire month, centered over
>>>>>>> Alaska. The observations are surface pressure, temperature,
>>>>>>> relative humidity, downwelling shortwave, and downwelling
>>>>>>> longwave radiation for a single site, Barrow, Alaska.
>>>>>>>
>>>>>>> Let me know if you have any additional questions.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>> John Halley Gotway wrote:
>>>>>>>> Mark,
>>>>>>>>
>>>>>>>> Let me make a few comments about this.
>>>>>>>>
>>>>>>>> First, depending on how you configure Point-Stat, getting
>>>>>>>> 0 matched pairs for certain combinations of
>>>>>>>> variables/message type may be fine. For example, if you
>>>>>>>> configure Point-Stat to verify Temperature at 2-meters
>>>>>>>> above the surface (TMP/Z2) and at 500mb (TMP/P500) using
>>>>>>>> message types of ADPSFC (surface obs) and APDUPA (upper
>>>>>>>> air obs), you would actually expect to get 0 matched
>>>>>>>> pairs for TMP/Z2 vs APDUPA and 0 matched pairs for
>>>>>>>> TMP/P500 vs ADPSFC. So sometimes having 0 matched pairs
>>>>>>>> is fine.
>>>>>>>>
>>>>>>>> However, if you're getting 0 matched pairs when you
>>>>>>>> expect that you should actually be finding some, here's
>>>>>>>> what I'd ask myself:
>>>>>>>>
>>>>>>>> - Am I applying some masking region (a grid or a
>>>>>>>> polyline) that is perhaps not working like I expect?  Try
>>>>>>>> rerunning with the masking grid set to FULL to verify
>>>>>>>> over the whole domain.
>>>>>>>>
>>>>>>>> - Does my forecast field contain valid data?  Clearly
>>>>>>>> Point-Stat is finding the fields you'd like to verify,
>>>>>>>> otherwise it'd error out. But if what it's finding
>>>>>>>> contains only bad data, it won't find any matched pairs.
>>>>>>>> Can you view the forecast field with some other tool to
>>>>>>>> check that the field contains valid data?  For NetCDF
>>>>>>>> format, use ncview.  For GRIB, "wgrib -V" will tell you
>>>>>>>> the min/max data values. Or you could view the GRIB file
>>>>>>>> using NCL or IDV.  Or you could run it through the
>>>>>>>> MET-MODE tool and look at the output plot.
>>>>>>>>
>>>>>>>> - Do I have my valid times correct?  Am I using
>>>>>>>> observations that are valid around the same time that my
>>>>>>>> forecast file is valid? In the Point-Stat config file,
>>>>>>>> you could set the "beg_ds" and "end_ds" values to define
>>>>>>>> a VERY large time window to see if you can get some
>>>>>>>> matched pairs.
>>>>>>>>
>>>>>>>> - Lastly, do the observations I'm using not match my
>>>>>>>> forecast for some other reason?  For example, are the
>>>>>>>> message types for the observations correct?  You could
>>>>>>>> try doing an ncdump to see what message types are in your
>>>>>>>> point observation file (ncdump -v hdr_typ file_name.nc |
>>>>>>>> sort -u).  Or are the observations not matching for some
>>>>>>>> other reason?  This would be the most difficult to
>>>>>>>> determine!
>>>>>>>>
>>>>>>>> Hopefully that'll help you figure out what's going on
>>>>>>>> with your data. I'd suggest "opening" things up as much
>>>>>>>> as possible (mask grid = FULL and set beg_ds/end_ds very
>>>>>>>> large) to try to get non-zero matched pairs, and go from
>>>>>>>> there.
>>>>>>>>
>>>>>>>> If you're still having problems after trying these
>>>>>>>> things, feel free to send me some sample files, and I
>>>>>>>> could take a look to see what going on.  You'd need to
>>>>>>>> send me: (1) Forecast file input for Point-Stat. (2)
>>>>>>>> Observation file input for Point-Stat. (3) Configuration
>>>>>>>> file input for Point-Stat. And you could post those files
>>>>>>>> to RAL's anonymous ftp site: ftp ftp.rap.ucar.edu
>>>>>>>> username = anonymous password = "your email address" cd
>>>>>>>> incoming/irap/johnhg put "those 3 files" bye (to exit
>>>>>>>> ftp)
>>>>>>>>
>>>>>>>> Thanks and good luck, John
>>>>>>>>
>>>>>>>>
>>>>>>>> Mark Seefeldt wrote:
>>>>>>>>> I am working on a model evaluation using point_stat in
>>>>>>>>> MET.  As it processes I am getting 0 pairs matched,
>>>>>>>>> therefore no statistics. Is there a preferred method to
>>>>>>>>> identify if it is the observation file, the forecast
>>>>>>>>> file, or the configuration file where the error resides
>>>>>>>>> resulting in the lack of matched obs/fcst values?  I am
>>>>>>>>> at a loss as to what is wrong in my setup which is
>>>>>>>>> preventing the obs/fcst pairs to be matched and to
>>>>>>>>> create the output.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Mark _______________________________________________
>>>>>>>>> Met_help mailing list Met_help at mailman.ucar.edu
>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>> _______________________________________________ Met_help
>>>>> mailing list Met_help at mailman.ucar.edu
>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>