[Met_help] [rt.rap.ucar.edu #42337] History for using point-stat with METv3.0 : question on threshold
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Tue Nov 23 13:42:44 MST 2010
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi Met_Help,
Sorry for submitting another ticket. But this question is regarding
point-stat.
While looking over the results from point-stat (METv3.0) on my datasets,
I noticed that results with fcst_thresh (>=1.000) does not show up in
CNT, and some other files. Does this mean that parameters in CNT (e.g.,
fcst mean, obs mean, mean error, etc.) are computed without applying the
threshold?
I have included in this email a tar file of my point-stat result from
one sample input file.
PointStatConfig -- my config file
nowcast_point_stat.sh -- shell script for running point-stat
20080809_QAQC_UDFCD.nc --- point data
chill_080809_0201_mp_precip.cdf --- gridded data
point_stat_test_run_log -- log file from running the shell script
Output files:
point_stat_010000L_20080809_020100V.stat
point_stat_010000L_20080809_020100V_cnt.txt
point_stat_010000L_20080809_020100V_ctc.txt
point_stat_010000L_20080809_020100V_cts.txt
point_stat_010000L_20080809_020100V_fho.txt
point_stat_010000L_20080809_020100V_mpr.txt
point_stat_010000L_20080809_020100V_sal1l2.txt
point_stat_010000L_20080809_020100V_sl1l2.txt
point_stat_010000L_20080809_020100V_val1l2.txt
point_stat_010000L_20080809_020100V_vl1l2.txt
Please let me know if you have any questions.
Thank you.
Kyoko
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #42337] using point-stat with METv3.0 : question on threshold
From: John Halley Gotway
Time: Tue Nov 23 12:41:01 2010
Kyoko,
Yes, that is true. The thresholds specified in "fcst_thresh" and
"obs_thresh" are used to discriminate between events and non-events
and create a contingency tables. So these thresholds are applied
in the computation of the contingency table counts and contingency
table statistics (CTC and CTS) lines.
The continuous statistics are always computed over the raw forecast
and observation values - no threshold or filtering is applied. What
you're asking for is called conditional verification. For
example, what is the RMSE for all pairs where the observation value of
temperature is greater than freezing? Currently there is no straight-
forward way to answer that question directly. However,
there are some options for ways of getting at it:
(1) First, when you define the masking region in the Point-Stat config
file, you can actually pass it a gridded data file, the name of a
field in that file, and a threshold value to use. For example,
you may choose to pass it a field of topography and look threshold it
< 1000m. That would mean you'd only be considering points whose
elevation is < 1000m. However, you could really pass it any
field - such as temperature - and threshold it so that you're only
looking at points where the forecast value is freezing.
(2) Second, when you run Point-Stat, you could choose to dump out the
MPR matched pairs line type. Then you could run STAT-Analysis jobs
that do the type of conditional verification you're
requesting. For example, I ran the following job on the Point-Stat
output you sent to me. This looks at all MPR lines in the "FULL"
region, picks out the ones that use the nearest neighbor interp
method (-interp_mthd UW_MEAN -interp_pnts 1) and whose forecast AND
observation values are >= 1.0, and computes continuous statistics:
stat_analysis -lookin . -job aggregate_stat -line_type MPR
-out_line_type CNT -vx_mask FULL -interp_mthd UW_MEAN -interp_pnts 1
-column_min FCST 1.0 -column_min OBS 1.0
We've had a couple of users ask about supporting this type of
conditional verification directly in the MET tools, but we haven't
implemented it yet. Is this something that would be useful in your
work?
Thanks,
John
On 11/23/2010 12:26 PM, RAL HelpDesk {for Kyoko Ikeda} wrote:
>
> Tue Nov 23 12:26:45 2010: Request 42337 was acted upon.
> Transaction: Ticket created by kyoko
> Queue: met_help
> Subject: using point-stat with METv3.0 : question on threshold
> Owner: Nobody
> Requestors: kyoko at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42337 >
>
>
> Hi Met_Help,
>
> Sorry for submitting another ticket. But this question is regarding
> point-stat.
> While looking over the results from point-stat (METv3.0) on my
datasets,
> I noticed that results with fcst_thresh (>=1.000) does not show up
in
> CNT, and some other files. Does this mean that parameters in CNT
(e.g.,
> fcst mean, obs mean, mean error, etc.) are computed without applying
the
> threshold?
>
> I have included in this email a tar file of my point-stat result
from
> one sample input file.
>
> PointStatConfig -- my config file
> nowcast_point_stat.sh -- shell script for running point-stat
>
> 20080809_QAQC_UDFCD.nc --- point data
> chill_080809_0201_mp_precip.cdf --- gridded data
>
> point_stat_test_run_log -- log file from running the shell script
>
> Output files:
> point_stat_010000L_20080809_020100V.stat
> point_stat_010000L_20080809_020100V_cnt.txt
> point_stat_010000L_20080809_020100V_ctc.txt
> point_stat_010000L_20080809_020100V_cts.txt
> point_stat_010000L_20080809_020100V_fho.txt
> point_stat_010000L_20080809_020100V_mpr.txt
> point_stat_010000L_20080809_020100V_sal1l2.txt
> point_stat_010000L_20080809_020100V_sl1l2.txt
> point_stat_010000L_20080809_020100V_val1l2.txt
> point_stat_010000L_20080809_020100V_vl1l2.txt
>
>
> Please let me know if you have any questions.
>
> Thank you.
>
> Kyoko
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #42337] using point-stat with METv3.0 : question on threshold
From: Kyoko Ikeda
Time: Tue Nov 23 13:41:11 2010
These are very good tips. I would never have figured out this out by
myself.
Thank you for your thorough explanation on conditional verification. I
think in the future, it would be very helpful to be able to do
conditional verification.
Kyoko
On 11/23/10 12:41 PM, RAL HelpDesk {for John Halley Gotway} wrote:
> Kyoko,
>
> Yes, that is true. The thresholds specified in "fcst_thresh" and
"obs_thresh" are used to discriminate between events and non-events
and create a contingency tables. So these thresholds are applied
> in the computation of the contingency table counts and contingency
table statistics (CTC and CTS) lines.
>
> The continuous statistics are always computed over the raw forecast
and observation values - no threshold or filtering is applied. What
you're asking for is called conditional verification. For
> example, what is the RMSE for all pairs where the observation value
of temperature is greater than freezing? Currently there is no
straight-forward way to answer that question directly. However,
> there are some options for ways of getting at it:
>
> (1) First, when you define the masking region in the Point-Stat
config file, you can actually pass it a gridded data file, the name of
a field in that file, and a threshold value to use. For example,
> you may choose to pass it a field of topography and look threshold
it< 1000m. That would mean you'd only be considering points whose
elevation is< 1000m. However, you could really pass it any
> field - such as temperature - and threshold it so that you're only
looking at points where the forecast value is freezing.
>
> (2) Second, when you run Point-Stat, you could choose to dump out
the MPR matched pairs line type. Then you could run STAT-Analysis
jobs that do the type of conditional verification you're
> requesting. For example, I ran the following job on the Point-Stat
output you sent to me. This looks at all MPR lines in the "FULL"
region, picks out the ones that use the nearest neighbor interp
> method (-interp_mthd UW_MEAN -interp_pnts 1) and whose forecast AND
observation values are>= 1.0, and computes continuous statistics:
>
> stat_analysis -lookin . -job aggregate_stat -line_type MPR
-out_line_type CNT -vx_mask FULL -interp_mthd UW_MEAN -interp_pnts 1
-column_min FCST 1.0 -column_min OBS 1.0
>
> We've had a couple of users ask about supporting this type of
conditional verification directly in the MET tools, but we haven't
implemented it yet. Is this something that would be useful in your
work?
>
> Thanks,
> John
>
> On 11/23/2010 12:26 PM, RAL HelpDesk {for Kyoko Ikeda} wrote:
>> Tue Nov 23 12:26:45 2010: Request 42337 was acted upon.
>> Transaction: Ticket created by kyoko
>> Queue: met_help
>> Subject: using point-stat with METv3.0 : question on
threshold
>> Owner: Nobody
>> Requestors: kyoko at ucar.edu
>> Status: new
>> Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42337>
>>
>>
>> Hi Met_Help,
>>
>> Sorry for submitting another ticket. But this question is regarding
>> point-stat.
>> While looking over the results from point-stat (METv3.0) on my
datasets,
>> I noticed that results with fcst_thresh (>=1.000) does not show up
in
>> CNT, and some other files. Does this mean that parameters in CNT
(e.g.,
>> fcst mean, obs mean, mean error, etc.) are computed without
applying the
>> threshold?
>>
>> I have included in this email a tar file of my point-stat result
from
>> one sample input file.
>>
>> PointStatConfig -- my config file
>> nowcast_point_stat.sh -- shell script for running point-stat
>>
>> 20080809_QAQC_UDFCD.nc --- point data
>> chill_080809_0201_mp_precip.cdf --- gridded data
>>
>> point_stat_test_run_log -- log file from running the shell script
>>
>> Output files:
>> point_stat_010000L_20080809_020100V.stat
>> point_stat_010000L_20080809_020100V_cnt.txt
>> point_stat_010000L_20080809_020100V_ctc.txt
>> point_stat_010000L_20080809_020100V_cts.txt
>> point_stat_010000L_20080809_020100V_fho.txt
>> point_stat_010000L_20080809_020100V_mpr.txt
>> point_stat_010000L_20080809_020100V_sal1l2.txt
>> point_stat_010000L_20080809_020100V_sl1l2.txt
>> point_stat_010000L_20080809_020100V_val1l2.txt
>> point_stat_010000L_20080809_020100V_vl1l2.txt
>>
>>
>> Please let me know if you have any questions.
>>
>> Thank you.
>>
>> Kyoko
>>
>>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #42337] using point-stat with METv3.0 : question on threshold
From: John Halley Gotway
Time: Tue Nov 23 13:42:24 2010
Kyoko,
OK, thanks for your feedback. I'll forward this on to the MET project
managers for consideration.
Thanks,
John
On 11/23/2010 01:41 PM, RAL HelpDesk {for Kyoko Ikeda} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42337 >
>
> These are very good tips. I would never have figured out this out
by
> myself.
>
> Thank you for your thorough explanation on conditional verification.
I
> think in the future, it would be very helpful to be able to do
> conditional verification.
>
> Kyoko
>
>
> On 11/23/10 12:41 PM, RAL HelpDesk {for John Halley Gotway} wrote:
>> Kyoko,
>>
>> Yes, that is true. The thresholds specified in "fcst_thresh" and
"obs_thresh" are used to discriminate between events and non-events
and create a contingency tables. So these thresholds are applied
>> in the computation of the contingency table counts and contingency
table statistics (CTC and CTS) lines.
>>
>> The continuous statistics are always computed over the raw forecast
and observation values - no threshold or filtering is applied. What
you're asking for is called conditional verification. For
>> example, what is the RMSE for all pairs where the observation value
of temperature is greater than freezing? Currently there is no
straight-forward way to answer that question directly. However,
>> there are some options for ways of getting at it:
>>
>> (1) First, when you define the masking region in the Point-Stat
config file, you can actually pass it a gridded data file, the name of
a field in that file, and a threshold value to use. For example,
>> you may choose to pass it a field of topography and look threshold
it< 1000m. That would mean you'd only be considering points whose
elevation is< 1000m. However, you could really pass it any
>> field - such as temperature - and threshold it so that you're only
looking at points where the forecast value is freezing.
>>
>> (2) Second, when you run Point-Stat, you could choose to dump out
the MPR matched pairs line type. Then you could run STAT-Analysis
jobs that do the type of conditional verification you're
>> requesting. For example, I ran the following job on the Point-Stat
output you sent to me. This looks at all MPR lines in the "FULL"
region, picks out the ones that use the nearest neighbor interp
>> method (-interp_mthd UW_MEAN -interp_pnts 1) and whose forecast AND
observation values are>= 1.0, and computes continuous statistics:
>>
>> stat_analysis -lookin . -job aggregate_stat -line_type MPR
-out_line_type CNT -vx_mask FULL -interp_mthd UW_MEAN -interp_pnts 1
-column_min FCST 1.0 -column_min OBS 1.0
>>
>> We've had a couple of users ask about supporting this type of
conditional verification directly in the MET tools, but we haven't
implemented it yet. Is this something that would be useful in your
work?
>>
>> Thanks,
>> John
>>
>> On 11/23/2010 12:26 PM, RAL HelpDesk {for Kyoko Ikeda} wrote:
>>> Tue Nov 23 12:26:45 2010: Request 42337 was acted upon.
>>> Transaction: Ticket created by kyoko
>>> Queue: met_help
>>> Subject: using point-stat with METv3.0 : question on
threshold
>>> Owner: Nobody
>>> Requestors: kyoko at ucar.edu
>>> Status: new
>>> Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42337>
>>>
>>>
>>> Hi Met_Help,
>>>
>>> Sorry for submitting another ticket. But this question is
regarding
>>> point-stat.
>>> While looking over the results from point-stat (METv3.0) on my
datasets,
>>> I noticed that results with fcst_thresh (>=1.000) does not show up
in
>>> CNT, and some other files. Does this mean that parameters in CNT
(e.g.,
>>> fcst mean, obs mean, mean error, etc.) are computed without
applying the
>>> threshold?
>>>
>>> I have included in this email a tar file of my point-stat result
from
>>> one sample input file.
>>>
>>> PointStatConfig -- my config file
>>> nowcast_point_stat.sh -- shell script for running point-stat
>>>
>>> 20080809_QAQC_UDFCD.nc --- point data
>>> chill_080809_0201_mp_precip.cdf --- gridded data
>>>
>>> point_stat_test_run_log -- log file from running the shell script
>>>
>>> Output files:
>>> point_stat_010000L_20080809_020100V.stat
>>> point_stat_010000L_20080809_020100V_cnt.txt
>>> point_stat_010000L_20080809_020100V_ctc.txt
>>> point_stat_010000L_20080809_020100V_cts.txt
>>> point_stat_010000L_20080809_020100V_fho.txt
>>> point_stat_010000L_20080809_020100V_mpr.txt
>>> point_stat_010000L_20080809_020100V_sal1l2.txt
>>> point_stat_010000L_20080809_020100V_sl1l2.txt
>>> point_stat_010000L_20080809_020100V_val1l2.txt
>>> point_stat_010000L_20080809_020100V_vl1l2.txt
>>>
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thank you.
>>>
>>> Kyoko
>>>
>>>
>
------------------------------------------------
More information about the Met_help
mailing list