[Met_help] [rt.rap.ucar.edu #63416] History for about cat_thresh setting
Tressa Fowler via RT
met_help at ucar.edu
Tue Jan 27 08:33:41 MST 2015
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello!
I have verified my automatic weather stations rainfall with using the MET pointstat library, the attached is ETS and TS
scores that I have ploted.
There are some different about greater and equal 0.1 mm, and there is big different between ETS and TS from 18:00 UTC. The TS scores greater than the ETS scores, and the others thresh are normal. I think the reason maybe be that there are many 0.1 mm in my observation rainfall data. I am not sure that.
Another question is about the cat_thresh, the example in MET sets a sides of cat_thresh, for example just setting >=0.1,>=10,>=20 and so on. I do not know whether the >=0.1 thresh scores includes the >=10 thresh scores.
If I just want to verify the thresh between 0.1 mm and 10 mm, what should I do ?
Best wishes!
look forward to your replying!
Qiang Li
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #63416] about cat_thresh setting
From: John Halley Gotway
Time: Tue Oct 15 11:41:03 2013
Qiang Li,
When you select multiple categorical thresholds in Point-Stat, several
things show up in the output. Each threshold you choose
(>=0.1,>=10,>=20) is used individually to compute a 2x2 contingency
table, from which statistics (like ETS and TS) are computed. The ETS
and TS values in your plot are computed for each of those thresholds
separately. For example, for the >=0.1 thresholds, any
values greater than 0.1 are considered an "event", while any values
less than 0.1 are a "non-event". So yes, values >=10 are definitely
considered an event when using the >=0.1 threshold.
However, there is another type of output available from Point-Stat
called the multi-category contingency table. The 3 thresholds you
selected partition the forecast/observation space into 4
categories. If you turn on the MCTC and MCTS output types in the
Point-Stat config file, you should get a single output line for each.
The MCTC line will contain the 16 counts for a 4-by-4
contingency table, and the MCTS line will contain statistics derived
from that table. Unfortunately, there are relatively few statistics
that are defined for multi-category contingency tables.
Currently, there is no way in MET to define an event between 0.1 and
10mm and a non-event outside that range. I don't think that is the
statistically preferred way of doing the analysis.
After I send this message, I'll assign this help desk ticket to Tressa
Fowler, our resident statistician. She can answer your questions
about interpreting the MET output.
Hope that helps.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 10/15/2013 09:05 AM, Li, Qiang via RT wrote:
>
> Tue Oct 15 09:05:44 2013: Request 63416 was acted upon.
> Transaction: Ticket created by liqiang at ou.edu
> Queue: met_help
> Subject: about cat_thresh setting
> Owner: Nobody
> Requestors: liqiang at ou.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=63416 >
>
>
> Hello!
>
> I have verified my automatic weather stations rainfall with using
the MET pointstat library, the attached is ETS and TS
> scores that I have ploted.
> There are some different about greater and equal 0.1 mm, and there
is big different between ETS and TS from 18:00 UTC. The TS scores
greater than the ETS scores, and the others thresh are normal. I
think the reason maybe be that there are many 0.1 mm in my observation
rainfall data. I am not sure that.
> Another question is about the cat_thresh, the example in MET sets a
sides of cat_thresh, for example just setting >=0.1,>=10,>=20 and so
on. I do not know whether the >=0.1 thresh scores includes the >=10
thresh scores.
> If I just want to verify the thresh between 0.1 mm and 10 mm, what
should I do ?
>
> Best wishes!
>
> look forward to your replying!
>
>
>
> Qiang Li
>
>
>
------------------------------------------------
Subject: about cat_thresh setting
From: Tressa Fowler
Time: Mon Oct 21 13:44:35 2013
Hi Qiang,
I looked at your plots, but in order to help you interpret, I need
more information. How many forecasts are represented by each point on
the X-axis time? If it is just a single occurrence with several point
observations, this would explain the large differences. Statistics for
a single case are highly variable and nearly useless. However, if you
have accumulated several months of data valid at 18Z, then it seems
odd for the statistics to change so drastically. For that case, I
would then see if most of your precipitation cases occur at 18Z.
Let me know if you have further questions.
Thanks,
Tressa
On Tue Oct 15 09:05:44 2013, liqiang at ou.edu wrote:
> Hello!
>
> I have verified my automatic weather stations rainfall with using
> the MET pointstat library, the attached is ETS and TS
> scores that I have ploted.
> There are some different about greater and equal 0.1 mm, and there
is
> big different between ETS and TS from 18:00 UTC. The TS scores
> greater than the ETS scores, and the others thresh are normal. I
> think the reason maybe be that there are many 0.1 mm in my
> observation rainfall data. I am not sure that.
> Another question is about the cat_thresh, the example in MET sets a
> sides of cat_thresh, for example just setting >=0.1,>=10,>=20 and
> so on. I do not know whether the >=0.1 thresh scores includes
the
> >=10 thresh scores.
> If I just want to verify the thresh between 0.1 mm and 10 mm, what
> should I do ?
>
> Best wishes!
>
> look forward to your replying!
>
>
>
> Qiang Li
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #63416] about cat_thresh setting
From: Li, Qiang
Time: Mon Oct 21 14:41:30 2013
Hello:
each point represents 3 hour interval. In my case there are about one
thousand point observations that are verified , so I do not know why
the big different between TS and ETS and the others' hour interval is
the same. so I want to export the CTC output and then calculate by
excel.
by the way I have other question about the long time verification.
I will verify my forecast output data from ARPS-WRF forecast system
in summer, but the MET just can verify a point time one time.
there are two times forecast in my system data one day, so it will
cost long time verification all the summer day by day.
so whether can it verify long time one times?
Could you give me some suggestion?
Best wishes!
Qiang Li
________________________________________
From: Tressa Fowler via RT [met_help at ucar.edu]
Sent: Monday, October 21, 2013 2:44 PM
To: Li, Qiang
Subject: [rt.rap.ucar.edu #63416] about cat_thresh setting
Hi Qiang,
I looked at your plots, but in order to help you interpret, I need
more information. How many forecasts are represented by each point on
the X-axis time? If it is just a single occurrence with several point
observations, this would explain the large differences. Statistics for
a single case are highly variable and nearly useless. However, if you
have accumulated several months of data valid at 18Z, then it seems
odd for the statistics to change so drastically. For that case, I
would then see if most of your precipitation cases occur at 18Z.
Let me know if you have further questions.
Thanks,
Tressa
On Tue Oct 15 09:05:44 2013, liqiang at ou.edu wrote:
> Hello!
>
> I have verified my automatic weather stations rainfall with using
> the MET pointstat library, the attached is ETS and TS
> scores that I have ploted.
> There are some different about greater and equal 0.1 mm, and there
is
> big different between ETS and TS from 18:00 UTC. The TS scores
> greater than the ETS scores, and the others thresh are normal. I
> think the reason maybe be that there are many 0.1 mm in my
> observation rainfall data. I am not sure that.
> Another question is about the cat_thresh, the example in MET sets a
> sides of cat_thresh, for example just setting >=0.1,>=10,>=20 and
> so on. I do not know whether the >=0.1 thresh scores includes
the
> >=10 thresh scores.
> If I just want to verify the thresh between 0.1 mm and 10 mm, what
> should I do ?
>
> Best wishes!
>
> look forward to your replying!
>
>
>
> Qiang Li
>
>
------------------------------------------------
Subject: about cat_thresh setting
From: Tressa Fowler
Time: Mon Oct 21 15:01:34 2013
Qiang,
Even with 1000 point observations, if there are only a couple of them
with precipitation, then your cases will be small. How many of the
1000 have precip greater than your threshold?
Most MET tools only verify 1 valid time per run. For verifying long
periods, we recommend scripting each call of point_stat and then
summarizing results using the stat_analysis tool. If you need details
about these, please see the Users' Guide.
Thanks,
Tressa
On Mon Oct 21 14:41:30 2013, liqiang at ou.edu wrote:
> Hello:
>
>
> each point represents 3 hour interval. In my case there are about
one
> thousand point observations that are verified , so I do not know
> why the big different between TS and ETS and the others' hour
> interval is the same. so I want to export the CTC output and then
> calculate by excel.
>
> by the way I have other question about the long time verification.
> I will verify my forecast output data from ARPS-WRF forecast system
> in summer, but the MET just can verify a point time one time.
> there are two times forecast in my system data one day, so it will
> cost long time verification all the summer day by day.
> so whether can it verify long time one times?
>
> Could you give me some suggestion?
>
> Best wishes!
>
> Qiang Li
> ________________________________________
> From: Tressa Fowler via RT [met_help at ucar.edu]
> Sent: Monday, October 21, 2013 2:44 PM
> To: Li, Qiang
> Subject: [rt.rap.ucar.edu #63416] about cat_thresh setting
>
> Hi Qiang,
>
> I looked at your plots, but in order to help you interpret, I need
> more information. How many forecasts are represented by each
point
> on the X-axis time? If it is just a single occurrence with
several
> point observations, this would explain the large differences.
> Statistics for a single case are highly variable and nearly
> useless. However, if you have accumulated several months of data
> valid at 18Z, then it seems odd for the statistics to change so
> drastically. For that case, I would then see if most of your
> precipitation cases occur at 18Z.
>
> Let me know if you have further questions.
>
> Thanks,
>
> Tressa
>
> On Tue Oct 15 09:05:44 2013, liqiang at ou.edu wrote:
> > Hello!
> >
> > I have verified my automatic weather stations rainfall with
using
> > the MET pointstat library, the attached is ETS and TS
> > scores that I have ploted.
> > There are some different about greater and equal 0.1 mm, and
there
> is
> > big different between ETS and TS from 18:00 UTC. The TS
scores
> > greater than the ETS scores, and the others thresh are normal.
I
> > think the reason maybe be that there are many 0.1 mm in my
> > observation rainfall data. I am not sure that.
> > Another question is about the cat_thresh, the example in MET sets
a
> > sides of cat_thresh, for example just setting >=0.1,>=10,>=20
and
> > so on. I do not know whether the >=0.1 thresh scores includes
> the
> > >=10 thresh scores.
> > If I just want to verify the thresh between 0.1 mm and 10 mm,
what
> > should I do ?
> >
> > Best wishes!
> >
> > look forward to your replying!
> >
> >
> >
> > Qiang Li
> >
> >
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #63416] about cat_thresh setting
From: Li, Qiang
Time: Mon Oct 21 15:16:58 2013
Tressa:
1000 point are matched.
most value of the 1000 point is zero, but the point number threshold
of greater than 0.1 mm is also several hundreds.
Qiang
________________________________________
From: Tressa Fowler via RT [met_help at ucar.edu]
Sent: Monday, October 21, 2013 4:01 PM
To: Li, Qiang
Subject: [rt.rap.ucar.edu #63416] about cat_thresh setting
Qiang,
Even with 1000 point observations, if there are only a couple of them
with precipitation, then your cases will be small. How many of the
1000 have precip greater than your threshold?
Most MET tools only verify 1 valid time per run. For verifying long
periods, we recommend scripting each call of point_stat and then
summarizing results using the stat_analysis tool. If you need details
about these, please see the Users' Guide.
Thanks,
Tressa
On Mon Oct 21 14:41:30 2013, liqiang at ou.edu wrote:
> Hello:
>
>
> each point represents 3 hour interval. In my case there are about
one
> thousand point observations that are verified , so I do not know
> why the big different between TS and ETS and the others' hour
> interval is the same. so I want to export the CTC output and then
> calculate by excel.
>
> by the way I have other question about the long time verification.
> I will verify my forecast output data from ARPS-WRF forecast system
> in summer, but the MET just can verify a point time one time.
> there are two times forecast in my system data one day, so it will
> cost long time verification all the summer day by day.
> so whether can it verify long time one times?
>
> Could you give me some suggestion?
>
> Best wishes!
>
> Qiang Li
> ________________________________________
> From: Tressa Fowler via RT [met_help at ucar.edu]
> Sent: Monday, October 21, 2013 2:44 PM
> To: Li, Qiang
> Subject: [rt.rap.ucar.edu #63416] about cat_thresh setting
>
> Hi Qiang,
>
> I looked at your plots, but in order to help you interpret, I need
> more information. How many forecasts are represented by each
point
> on the X-axis time? If it is just a single occurrence with
several
> point observations, this would explain the large differences.
> Statistics for a single case are highly variable and nearly
> useless. However, if you have accumulated several months of data
> valid at 18Z, then it seems odd for the statistics to change so
> drastically. For that case, I would then see if most of your
> precipitation cases occur at 18Z.
>
> Let me know if you have further questions.
>
> Thanks,
>
> Tressa
>
> On Tue Oct 15 09:05:44 2013, liqiang at ou.edu wrote:
> > Hello!
> >
> > I have verified my automatic weather stations rainfall with
using
> > the MET pointstat library, the attached is ETS and TS
> > scores that I have ploted.
> > There are some different about greater and equal 0.1 mm, and
there
> is
> > big different between ETS and TS from 18:00 UTC. The TS
scores
> > greater than the ETS scores, and the others thresh are normal.
I
> > think the reason maybe be that there are many 0.1 mm in my
> > observation rainfall data. I am not sure that.
> > Another question is about the cat_thresh, the example in MET sets
a
> > sides of cat_thresh, for example just setting >=0.1,>=10,>=20
and
> > so on. I do not know whether the >=0.1 thresh scores includes
> the
> > >=10 thresh scores.
> > If I just want to verify the thresh between 0.1 mm and 10 mm,
what
> > should I do ?
> >
> > Best wishes!
> >
> > look forward to your replying!
> >
> >
> >
> > Qiang Li
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list