[Met_help] [rt.rap.ucar.edu #52210] History for about the ADPUPA verification

John Halley Gotway via RT met_help at ucar.edu
Thu Jan 5 13:36:42 MST 2012


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Dear Sir:

 Here is the output from one of "*mpr.txt".  In the text below, we can see the forecast vaule is not varying. I'm not sure why. One reason i can guess from the text is that most of observation points level is missing "NA". This unchanged forecast value will lead large RMSE in that level. The level i posted is between 150-10hPa. 
  
 Here are also the common session of this section"
 V3.0    WRF   120000    20100515_120000 20100515_120000 000000   20100515_120000 20100515_120000 TMP      P150-10   TMP     P150-10   ADPUPA FULL    DW_MEAN     4           NA          NA         NA         NA    MPR       2367"   

I have checked *cnt.txt files. The "TOTAL"  in that file is 2367 which means all the pairs below is counted in the verification. I'm just wondering how the forecast value calculated when the observation level is missing? And can default program get rid of these points in the calculation of RMSE or other verification scores?  

Many thanks!



 INDEX OBS_SID  OBS_LAT  OBS_LON    OBS_LVL    OBS_ELV     FCST         OBS          CLIMO 
    
 439   72201    24.55000 -81.79000  50.00000   20731.58008 214.66865    209.85001    NA
 440   72201    24.55000 -81.79000  47.00000   NA          214.66865    211.25000    NA
 441   72201    24.55000 -81.79000  45.10000   NA          214.66865    210.25000    NA
 442   72201    24.55000 -81.79000  39.70000   NA          214.66865    215.85001    NA
 443   72201    24.55000 -81.79000  36.10000   NA          214.66865    217.05000    NA
 444   72201    24.55000 -81.79000  30.00000   23934.91602 214.66865    214.05000    NA
 445   72201    24.55000 -81.79000  29.00000   NA          214.66865    214.05000    NA
 446   72201    24.55000 -81.79000  27.30000   NA          214.66865    217.85001    NA
 448   72201    24.55000 -81.79000  25.60000   NA          214.66865    218.64999    NA
 449   72201    24.55000 -81.79000  24.90000   NA          214.66865    220.64999    NA
 450   72201    24.55000 -81.79000  24.00000   NA          214.66865    219.85001    NA
 451   72201    24.55000 -81.79000  22.30000   NA          214.66865    221.05000    NA
 452   72201    24.55000 -81.79000  20.90000   NA          214.66865    224.45000    NA
 453   72201    24.55000 -81.79000  20.00000   26547.63477 214.66865    224.85001    NA
 454   72201    24.55000 -81.79000  16.60000   27765.90430 214.66865    227.05000    NA
 455   72201    24.55000 -81.79000  13.50000   NA          214.66865    232.45000    NA
 456   72201    24.55000 -81.79000  12.80000   NA          214.66865    231.45000    NA
 457   72201    24.55000 -81.79000  10.00000   31212.49219 214.66865    232.25000    NA



kefeng

2011-12-21 



kefeng 


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #52210] about the ADPUPA verification
From: John Halley Gotway
Time: Thu Dec 22 09:45:12 2011

Kefeng,

Hmmm, interesting.  This a good question.

The NA values in the OBS_ELV column do not impact the computation of
the statistics, and it doesn't make any sense to throw them away.  And
here's why...

In the header columns, I see that you're verifying temperature against
upper-air observations between 150mb and 10mb.  In the sample MPR
output you sent, I see that this is basically sounding data -
the station ID and lat/lon remain constant while the contents of the
"OBS_LVL" column changes.  That "OBS_LVL" column indicates the
pressure level at which the observation occurred.  Sine you're
verifying a range of pressure levels (150 to 10mb), these observations
are used because they fall in that range.  The "OBS_ELV" column is the
meters above sea level at which the observations occurred.
 Since you're verifying on pressure levels in this case, that
elevation information is not used at all by Point-Stat.

The real question is, why does the forecast value remain constant for
all of these observations.  If I were you, I would look at your model
output to see at what levels do you have temperature
records?  My guess is that you don't have any model output higher than
50 mb.  So all of the observations higher than 50 mb are getting
compared to the forecast field at 50mb.  That would explain why
the forecast value doesn't change.

Ultimately, it's your responsibility to know your data and make
sensible choices about what and how to do the verification.  If you
don't have any model output higher than 50mb, it doesn't make sense
to verify there.

Hope that helps.  If you have more questions about your particular
dataset, it would probably be easiest to just have you send us a
sample of data along with your question.

We'd need your Point-Stat forecast file, observation file,
configuration file, and a the command line you use to call Point-Stat.
You can follow the directions for sending us data listed here:
   http://www.dtcenter.org/met/users/support/met_help.php#ftp

By the way, I thought I'd mention that there is in fact a way to use
the STAT-Analysis tool to filter out data and recompute statistics.
That isn't really the point of the issue you're having, but I
thought I'd mention it.  The following type of job would read the data
MPR lines, filter out only those lines where OBS_ELV > 0, and then
recompute CNT statistics.  That would basically throw out the
lines where OBS_ELV = NA:
   stat_analysis -lookin /path/to/stat/file/with/MPR/lines -job
aggregate_stat -line_type MPR -out_line_type CNT -column_min OBS_ELV 0

Hope that helps.

Thanks,
John

On 12/21/2011 10:39 AM, kefeng via RT wrote:
>
> Wed Dec 21 10:39:23 2011: Request 52210 was acted upon.
> Transaction: Ticket created by kefeng at ou.edu
>        Queue: met_help
>      Subject: about the ADPUPA verification
>        Owner: Nobody
>   Requestors: kefeng at ou.edu
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=52210 >
>
>
> Dear Sir:
>
>  Here is the output from one of "*mpr.txt".  In the text below, we
can see the forecast vaule is not varying. I'm not sure why. One
reason i can guess from the text is that most of observation points
level is missing "NA". This unchanged forecast value will lead large
RMSE in that level. The level i posted is between 150-10hPa.
>
>  Here are also the common session of this section"
>  V3.0    WRF   120000    20100515_120000 20100515_120000 000000
20100515_120000 20100515_120000 TMP      P150-10   TMP     P150-10
ADPUPA FULL    DW_MEAN     4           NA          NA         NA
NA    MPR       2367"
>
> I have checked *cnt.txt files. The "TOTAL"  in that file is 2367
which means all the pairs below is counted in the verification. I'm
just wondering how the forecast value calculated when the observation
level is missing? And can default program get rid of these points in
the calculation of RMSE or other verification scores?
>
> Many thanks!
>
>
>
>  INDEX OBS_SID  OBS_LAT  OBS_LON    OBS_LVL    OBS_ELV     FCST
OBS          CLIMO
>
>  439   72201    24.55000 -81.79000  50.00000   20731.58008 214.66865
209.85001    NA
>  440   72201    24.55000 -81.79000  47.00000   NA          214.66865
211.25000    NA
>  441   72201    24.55000 -81.79000  45.10000   NA          214.66865
210.25000    NA
>  442   72201    24.55000 -81.79000  39.70000   NA          214.66865
215.85001    NA
>  443   72201    24.55000 -81.79000  36.10000   NA          214.66865
217.05000    NA
>  444   72201    24.55000 -81.79000  30.00000   23934.91602 214.66865
214.05000    NA
>  445   72201    24.55000 -81.79000  29.00000   NA          214.66865
214.05000    NA
>  446   72201    24.55000 -81.79000  27.30000   NA          214.66865
217.85001    NA
>  448   72201    24.55000 -81.79000  25.60000   NA          214.66865
218.64999    NA
>  449   72201    24.55000 -81.79000  24.90000   NA          214.66865
220.64999    NA
>  450   72201    24.55000 -81.79000  24.00000   NA          214.66865
219.85001    NA
>  451   72201    24.55000 -81.79000  22.30000   NA          214.66865
221.05000    NA
>  452   72201    24.55000 -81.79000  20.90000   NA          214.66865
224.45000    NA
>  453   72201    24.55000 -81.79000  20.00000   26547.63477 214.66865
224.85001    NA
>  454   72201    24.55000 -81.79000  16.60000   27765.90430 214.66865
227.05000    NA
>  455   72201    24.55000 -81.79000  13.50000   NA          214.66865
232.45000    NA
>  456   72201    24.55000 -81.79000  12.80000   NA          214.66865
231.45000    NA
>  457   72201    24.55000 -81.79000  10.00000   31212.49219 214.66865
232.25000    NA
>
>
>
> kefeng
>
> 2011-12-21
>
>
>
> kefeng

------------------------------------------------
Subject: about the ADPUPA verification
From: kefeng
Time: Thu Jan 05 10:27:06 2012



    Thanks, John. 

    I was on vaction these days and just saw
the confirmation email. 

    The problem have been solved. It was
due to the WPP process. In that procedure, my default configuration
turned off the level above 50hPa. Therefore, the level above 50mb
always have the same value. I suggest to add a warning in the log in
the furture version when the observation level exceed model topest
level after WPP. 

     
    Kefeng
    


2012-01-05
kefeng 



发件人: John Halley Gotway via RT 
发送时间: 2011-12-22
10:45:13 
收件人: kefeng at ou.edu 
抄送: 
主题: Re: [rt.rap.ucar.edu #52210]
about the ADPUPA verification 
 
Kefeng,
Hmmm, interesting.  This a
good question.
The NA values in the OBS_ELV column do not impact the
computation of the statistics, and it doesn't make any sense to throw
them away.  And here's why...
In the header columns, I see that
you're verifying temperature against upper-air observations between
150mb and 10mb.  In the sample MPR output you sent, I see that this is
basically sounding data -
the station ID and lat/lon remain constant
while the contents of the "OBS_LVL" column changes.  That "OBS_LVL"
column indicates the pressure level at which the observation occurred.
Sine you're
verifying a range of pressure levels (150 to 10mb), these
observations are used because they fall in that range.  The "OBS_ELV"
column is the meters above sea level at which the observations
occurred.
 Since you're verifying on pressure levels in this case,
that elevation information is not used at all by Point-Stat.
The real
question is, why does the forecast value remain constant for all of
these observations.  If I were you, I would look at your model output
to see at what levels do you have temperature
records?  My guess is
that you don't have any model output higher than 50 mb.  So all of the
observations higher than 50 mb are getting compared to the forecast
field at 50mb.  That would explain why
the forecast value doesn't
change.
Ultimately, it's your responsibility to know your data and
make sensible choices about what and how to do the verification.  If
you don't have any model output higher than 50mb, it doesn't make
sense
to verify there.
Hope that helps.  If you have more questions
about your particular dataset, it would probably be easiest to just
have you send us a sample of data along with your question.
We'd need
your Point-Stat forecast file, observation file, configuration file,
and a the command line you use to call Point-Stat.  You can follow the
directions for sending us data listed here:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
By the
way, I thought I'd mention that there is in fact a way to use the
STAT-Analysis tool to filter out data and recompute statistics.  That
isn't really the point of the issue you're having, but I
thought I'd
mention it.  The following type of job would read the data MPR lines,
filter out only those lines where OBS_ELV > 0, and then recompute CNT
statistics.  That would basically throw out the
lines where OBS_ELV =
NA:
   stat_analysis -lookin /path/to/stat/file/with/MPR/lines -job
aggregate_stat -line_type MPR -out_line_type CNT -column_min OBS_ELV 0
Hope that helps.
Thanks,
John
On 12/21/2011 10:39 AM, kefeng via RT
wrote:
> 
> Wed Dec 21 10:39:23 2011: Request 52210 was acted upon.
> Transaction: Ticket created by kefeng at ou.edu
>        Queue:
met_help
>      Subject: about the ADPUPA verification
>
Owner: Nobody
>   Requestors: kefeng at ou.edu
>       Status: new
>
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=52210 >
> 
> 
> Dear Sir:
> 
>  Here is the output from one of "*mpr.txt".
In the text below, we can see the forecast vaule is not varying. I'm
not sure why. One reason i can guess from the text is that most of
observation points level is missing "NA". This unchanged forecast
value will lead large RMSE in that level. The level i posted is
between 150-10hPa. 
>   
>  Here are also the common session of this
section"
>  V3.0    WRF   120000    20100515_120000 20100515_120000
000000   20100515_120000 20100515_120000 TMP      P150-10   TMP
P150-10   ADPUPA FULL    DW_MEAN     4           NA          NA
NA         NA    MPR       2367"   
> 
> I have checked *cnt.txt
files. The "TOTAL"  in that file is 2367 which means all the pairs
below is counted in the verification. I'm just wondering how the
forecast value calculated when the observation level is missing? And
can default program get rid of these points in the calculation of RMSE
or other verification scores?  
> 
> Many thanks!
> 
> 
> 
>
INDEX OBS_SID  OBS_LAT  OBS_LON    OBS_LVL    OBS_ELV     FCST
OBS          CLIMO 
>     
>  439   72201    24.55000 -81.79000
50.00000   20731.58008 214.66865    209.85001    NA
>  440   72201
24.55000 -81.79000  47.00000   NA          214.66865    211.25000
NA
>  441   72201    24.55000 -81.79000  45.10000   NA
214.66865    210.25000    NA
>  442   72201    24.55000 -81.79000
39.70000   NA          214.66865    215.85001    NA
>  443   72201
24.55000 -81.79000  36.10000   NA          214.66865    217.05000
NA
>  444   72201    24.55000 -81.79000  30.00000   23934.91602
214.66865    214.05000    NA
>  445   72201    24.55000 -81.79000
29.00000   NA          214.66865    214.05000    NA
>  446   72201
24.55000 -81.79000  27.30000   NA          214.66865    217.85001
NA
>  448   72201    24.55000 -81.79000  25.60000   NA
214.66865    218.64999    NA
>  449   72201    24.55000 -81.79000
24.90000   NA          214.66865    220.64999    NA
>  450   72201
24.55000 -81.79000  24.00000   NA          214.66865    219.85001
NA
>  451   72201    24.55000 -81.79000  22.30000   NA
214.66865    221.05000    NA
>  452   72201    24.55000 -81.79000
20.90000   NA          214.66865    224.45000    NA
>  453   72201
24.55000 -81.79000  20.00000   26547.63477 214.66865    224.85001
NA
>  454   72201    24.55000 -81.79000  16.60000   27765.90430
214.66865    227.05000    NA
>  455   72201    24.55000 -81.79000
13.50000   NA          214.66865    232.45000    NA
>  456   72201
24.55000 -81.79000  12.80000   NA          214.66865    231.45000
NA
>  457   72201    24.55000 -81.79000  10.00000   31212.49219
214.66865    232.25000    NA
> 
> 
> 
> kefeng
> 
> 2011-12-21
> 
> 
> 
> kefeng

------------------------------------------------


More information about the Met_help mailing list