[Met_help] [rt.rap.ucar.edu #45536] History for About the rmse calculation.

[RAL HelpDesk {for John Halley Gotway}]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Dear Sir:

      When the observation time is not exactly match the verification time, how the MET calculate the RMSE in the point_stat analysis? Will it consider the time diff weight during the calculation? Or as long as the observation within the time window, it treated the observation just as the one that exactly match the verification time? 

      Many thanks!

Kefeng


2011-03-23 



kefeng 


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #45536] About the rmse calculation.
From: John Halley Gotway
Time: Wed Mar 23 09:38:18 2011

Kefeng,

No, MET does not consider the time offsets of the observations when
computing statistics.  When you run Point-Stat, you use the "beg_ds"
and "end_ds" options in the config file (or the "obs_valid_beg"
and "obs_valid_end" command line options) to define the matching
observation time window.  Any observations that fall within that time
window are retained, compared to the forecast data, and used in
the computation of statistics.  Even if there are multiple
observations at the same observing location that fall within the time
window, all of them are used.

So you should think carefully about how you want to define the
observation time window based on the reporting frequency for the type
of observations you're verifying against.  For example, sounding
data may only show up once every 12 hours, whereas aircraft data may
be every minute.

If you want to play around with how changing the time window affect
your statistics, here's an approach you could try:
- Run Point-Stat once setting beg_ds and end_ds to relatively large
values, and dump out the MPR line type.
- Next, set up a STAT-Analysis job on the command line to read those
MPR lines and recompute continuous statistics.  Something like the
following:
   stat_analysis -looking out/point_stat/*.stat -fcst_var TMP
-fcst_lev Z2 -job aggregate_stat -line_type MPR -out_line_type CNT

The example job listed above will look for MPR lines for 2-meter
temperature, and use them to recompute continuous statistics.  Then
you could add in the "-obs_valid_beg" and "-obs_valid_end" command
line options to set the matching time window.  Only those MPR lines
whose observation time falls in that window will be used.  So you
could rerun that job many times, varying the time window, to see
how the resulting continuous statistics change for a single case.

Hope that helps clarify.

John Halley Gotway
met_help at ucar.edu

On 03/22/2011 11:09 PM, RAL HelpDesk {for kefeng} wrote:
>
> Tue Mar 22 23:09:52 2011: Request 45536 was acted upon.
> Transaction: Ticket created by kefeng at ou.edu
>        Queue: met_help
>      Subject: About the rmse calculation.
>        Owner: Nobody
>   Requestors: kefeng at ou.edu
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45536 >
>
>
> Dear Sir:
>
>       When the observation time is not exactly match the
verification time, how the MET calculate the RMSE in the point_stat
analysis? Will it consider the time diff weight during the
calculation? Or as long as the observation within the time window, it
treated the observation just as the one that exactly match the
verification time?
>
>       Many thanks!
>
> Kefeng
>
>
> 2011-03-23
>
>
>
> kefeng

------------------------------------------------
Subject: About the rmse calculation.
From: kefeng
Time: Wed Mar 23 10:04:19 2011

Hi, John:

    Many thanks!
   
    It helped a lot!

kefeng
2011-03-23 



kefeng 



发件人： RAL HelpDesk {for John Halley
Gotway} 
发送时间： 2011-03-23  10:38:24 
收件人： kefeng at ou.edu 
抄送：
met_help at ucar.edu 
主题： Re: [rt.rap.ucar.edu #45536] About the rmse
calculation. 
 
Kefeng,
No, MET does not consider the time offsets
of the observations when computing statistics.  When you run Point-
Stat, you use the "beg_ds" and "end_ds" options in the config file (or
the "obs_valid_beg"
and "obs_valid_end" command line options) to
define the matching observation time window.  Any observations that
fall within that time window are retained, compared to the forecast
data, and used in
the computation of statistics.  Even if there are
multiple observations at the same observing location that fall within
the time window, all of them are used.
So you should think carefully
about how you want to define the observation time window based on the
reporting frequency for the type of observations you're verifying
against.  For example, sounding
data may only show up once every 12
hours, whereas aircraft data may be every minute.
If you want to play
around with how changing the time window affect your statistics,
here's an approach you could try:
- Run Point-Stat once setting
beg_ds and end_ds to relatively large values, and dump out the MPR
line type.
- Next, set up a STAT-Analysis job on the command line to
read those MPR lines and recompute continuous statistics.  Something
like the following:
   stat_analysis -looking out/point_stat/*.stat
-fcst_var TMP -fcst_lev Z2 -job aggregate_stat -line_type MPR
-out_line_type CNT
The example job listed above will look for MPR
lines for 2-meter temperature, and use them to recompute continuous
statistics.  Then you could add in the "-obs_valid_beg" and "-
obs_valid_end" command
line options to set the matching time window.
Only those MPR lines whose observation time falls in that window will
be used.  So you could rerun that job many times, varying the time
window, to see
how the resulting continuous statistics change for a
single case.
Hope that helps clarify.
John Halley Gotway
met_help at ucar.edu
On 03/22/2011 11:09 PM, RAL HelpDesk {for kefeng}
wrote:
> 
> Tue Mar 22 23:09:52 2011: Request 45536 was acted upon.
> Transaction: Ticket created by kefeng at ou.edu
>        Queue:
met_help
>      Subject: About the rmse calculation.
>        Owner:
Nobody
>   Requestors: kefeng at ou.edu
>       Status: new
>  Ticket
<URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45536 >
> 
>
> Dear Sir:
> 
>       When the observation time is not exactly
match the verification time, how the MET calculate the RMSE in the
point_stat analysis? Will it consider the time diff weight during the
calculation? Or as long as the observation within the time window, it
treated the observation just as the one that exactly match the
verification time? 
> 
>       Many thanks!
> 
> Kefeng
> 
> 
>
2011-03-23 
> 
> 
> 
> kefeng

------------------------------------------------