[Met_help] [rt.rap.ucar.edu #79763] History for Grid Stat Ensemble Probabiltity Verification Climo Data
John Halley Gotway via RT
met_help at ucar.edu
Thu Apr 6 08:52:59 MDT 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John, I am converting our ensemble verification over to MET software. In grid stat, I will be verifying probability data and will use NCEP climo for the reference. The question I have and I did not find it in the user guide: will I have to convert the climo file to 1 or 0s - 1 being the threshold in question was met and 0 meaning is wasn't or does MET do this for me automatically? So if I am verifying wind speeds > 25 knots, can I just provide a climo average wind speed file or do I need to threshold it against 25 knots?
Also, could you place the MET5.2 guide on the FTP server in my directory. I can't seem to down load it from the met help web site.
Thanks
Bob
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Grid Stat Ensemble Probabiltity Verification Climo Data
From: John Halley Gotway
Time: Fri Mar 10 14:14:49 2017
Bob,
I posted the user's guide to our ftp site:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/craigr_data/
MET_Users_Guide.pdf
It sounds to me like you're running Grid-Stat to evaluate
probabilistic
forecasts and you're wondering how to define a climatology for that
probabilistic data. I can tell you that MET does not include any
special
logic for automatically deriving a probabilistic climatology for you
on the
fly. But I'm not the right person to make suggestions for how you
*should*
be defining a reference climatology for probabilities.
I'm going to assign this ticket to our resident statistician, Tressa,
and
hopefully she'll have some advice for both you and me!
Thanks,
John Halley Gotway
On Thu, Mar 9, 2017 at 7:13 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu Mar 09 07:13:13 2017: Request 79763 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Grid Stat Ensemble Probabiltity Verification Climo
Data
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79763 >
>
>
> John, I am converting our ensemble verification over to MET
software. In
> grid stat, I will be verifying probability data and will use NCEP
climo for
> the reference. The question I have and I did not find it in the
user
> guide: will I have to convert the climo file to 1 or 0s - 1 being
the
> threshold in question was met and 0 meaning is wasn't or does MET do
this
> for me automatically? So if I am verifying wind speeds > 25 knots,
can I
> just provide a climo average wind speed file or do I need to
threshold it
> against 25 knots?
>
> Also, could you place the MET5.2 guide on the FTP server in my
directory.
> I can't seem to down load it from the met help web site.
>
> Thanks
> Bob
>
>
>
------------------------------------------------
Subject: Grid Stat Ensemble Probabiltity Verification Climo Data
From: Tressa Fowler
Time: Mon Mar 13 11:08:02 2017
Hi Bob,
In all cases, when a reference climatology is used, the statistic
using the reference assumes that the climatology is a typical value of
the same type as the forecast. So if you are looking at errors in
surface temperature, you want your climatology to be the average
surface temperature. In this case, you want probability of winds
exceeding 25 knots, so that is what your climatology should represent.
If for some reason your climatology is different in form or character
from the forecast, but you think it makes sense to do a comparison, I
would treat the 'climatology' as just another forecast, running MET
separately on it with the same observations set, then compare some
output stats. By putting it in the climo field in MET, it will be used
in skill scores and such where it must match to make sense.
Does that help?
If you need to compare the NCEP average to your probability using the
same stats, perhaps Marion's HIRA method would give you a way to do
that?
Tressa
On Thu Mar 09 07:13:13 2017, robert.craig.2 at us.af.mil wrote:
> John, I am converting our ensemble verification over to MET
software.
> In grid stat, I will be verifying probability data and will use NCEP
> climo for the reference. The question I have and I did not find it
in
> the user guide: will I have to convert the climo file to 1 or 0s -
1
> being the threshold in question was met and 0 meaning is wasn't or
> does MET do this for me automatically? So if I am verifying wind
> speeds > 25 knots, can I just provide a climo average wind speed
file
> or do I need to threshold it against 25 knots?
>
> Also, could you place the MET5.2 guide on the FTP server in my
> directory. I can't seem to down load it from the met help web site.
>
> Thanks
> Bob
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #79763] Grid Stat Ensemble Probabiltity Verification Climo Data
From: robert.craig.2 at us.af.mil
Time: Mon Mar 13 11:54:44 2017
Hi Teresa,
In this case we are verifying the probability of the model forecasting
sustained winds > 25kts. I have NCEP surface wind climo and I was
planning to threshold it to the 25 knot criteria. What John told me
is that I will have to convert this file to 1 or 0 (1 being >25kts)
since MET doesn't do so for climo. We are generating these
statistics for comparison with NCEP NAEFS. Talking to Yuejian Zhu,
he indicated they were using the 2.5 degree climo for all there
reference forecasts. If they have some climo specific to these
thresholds, I would love to get ahold of it. If this isn't a good
idea, then I can treat climo as a separate forecast. I hate this
approach since I have to carry around a second set of "forecasts" but
I want the results to make sense.
We are currently using HIRA for high resolution model comparisons. I
can't use it for this since we should trying to duplicate what NCEP
does for comparison purposes. By the way, will the HIRA technique
be include in MET 6.0? I had heard such a rumor.
Thanks for your help
Bob Craig
-----Original Message-----
From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
Sent: Monday, March 13, 2017 12:08 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
<matthew.sittel.ctr at us.af.mil>
Subject: [rt.rap.ucar.edu #79763] Grid Stat Ensemble Probabiltity
Verification Climo Data
Hi Bob,
In all cases, when a reference climatology is used, the statistic
using the reference assumes that the climatology is a typical value of
the same type as the forecast. So if you are looking at errors in
surface temperature, you want your climatology to be the average
surface temperature. In this case, you want probability of winds
exceeding 25 knots, so that is what your climatology should represent.
If for some reason your climatology is different in form or character
from the forecast, but you think it makes sense to do a comparison, I
would treat the 'climatology' as just another forecast, running MET
separately on it with the same observations set, then compare some
output stats. By putting it in the climo field in MET, it will be used
in skill scores and such where it must match to make sense.
Does that help?
If you need to compare the NCEP average to your probability using the
same stats, perhaps Marion's HIRA method would give you a way to do
that?
Tressa
On Thu Mar 09 07:13:13 2017, robert.craig.2 at us.af.mil wrote:
> John, I am converting our ensemble verification over to MET
software.
> In grid stat, I will be verifying probability data and will use NCEP
> climo for the reference. The question I have and I did not find it
in
> the user guide: will I have to convert the climo file to 1 or 0s -
1
> being the threshold in question was met and 0 meaning is wasn't or
> does MET do this for me automatically? So if I am verifying wind
> speeds > 25 knots, can I just provide a climo average wind speed
file
> or do I need to threshold it against 25 knots?
>
> Also, could you place the MET5.2 guide on the FTP server in my
> directory. I can't seem to down load it from the met help web site.
>
> Thanks
> Bob
------------------------------------------------
Subject: Grid Stat Ensemble Probabiltity Verification Climo Data
From: John Halley Gotway
Time: Wed Mar 22 10:29:06 2017
Bob,
Regarding your last question, yes, we've added HiRA to Point-Stat in
version 6.0 of MET. You define thresholds and neighborhood sizes of
interest. For each observation value, Point-Stat looks at all the
forecast
points in the neighborhood around the observation, applies the
threshold,
and computes the ratio of forecast points meeting that threshold
criteria.
Those ratios are numbers between 0 and 1 which Point-Stat then
verifies as
if they were probabilities.
Regarding, the climatologies for probabilistic verification, there are
a
lot of details I don't fully understand...
If you're verifying forecast probabilities with a range of 0 to 1, I
would
think you'd need climatological probabilities in that range as well.
So
not just values of 0 or 1 but values *between* 0 and 1. And you could
use
historical data to determine that... for example, in the last 50
years,
what percentage of the time does a grid point experience sustained
winds >
25kts. But you'd need a lot of data to determine that!
There's likely some confusion going on here about the 2.5 degree climo
data
from NCEP. The "GRIB" version of that data is a mean value for each
grid
point along with a standard deviation value. The NCEP binary version
contains 101 (or 100?) values at each grid point representing the
distribution of values at each point. And those are obviously much
larger
files. It's possible that they're using those 101 values at each grid
point to determine the frequency at which wind speeds exceed 25kts.
MET
does not currently include logic for processing those NCEP binary
files.
Tressa and Tara, please correct me if I'm wrong on any of these
points.
Thanks,
John
On Mon, Mar 13, 2017 at 11:54 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79763 >
>
> Hi Teresa,
> In this case we are verifying the probability of the model
forecasting
> sustained winds > 25kts. I have NCEP surface wind climo and I was
planning
> to threshold it to the 25 knot criteria. What John told me is that
I will
> have to convert this file to 1 or 0 (1 being >25kts) since MET
doesn't do
> so for climo. We are generating these statistics for comparison
with
> NCEP NAEFS. Talking to Yuejian Zhu, he indicated they were using
the 2.5
> degree climo for all there reference forecasts. If they have some
climo
> specific to these thresholds, I would love to get ahold of it. If
this
> isn't a good idea, then I can treat climo as a separate forecast. I
hate
> this approach since I have to carry around a second set of
"forecasts" but
> I want the results to make sense.
>
> We are currently using HIRA for high resolution model comparisons.
I
> can't use it for this since we should trying to duplicate what NCEP
does
> for comparison purposes. By the way, will the HIRA technique be
include
> in MET 6.0? I had heard such a rumor.
>
> Thanks for your help
> Bob Craig
>
>
> -----Original Message-----
> From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> Sent: Monday, March 13, 2017 12:08 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: [rt.rap.ucar.edu #79763] Grid Stat Ensemble Probabiltity
> Verification Climo Data
>
> Hi Bob,
>
> In all cases, when a reference climatology is used, the statistic
using
> the reference assumes that the climatology is a typical value of the
same
> type as the forecast. So if you are looking at errors in surface
> temperature, you want your climatology to be the average surface
> temperature. In this case, you want probability of winds exceeding
25
> knots, so that is what your climatology should represent. If for
some
> reason your climatology is different in form or character from the
> forecast, but you think it makes sense to do a comparison, I would
treat
> the 'climatology' as just another forecast, running MET separately
on it
> with the same observations set, then compare some output stats. By
putting
> it in the climo field in MET, it will be used in skill scores and
such
> where it must match to make sense.
>
> Does that help?
>
> If you need to compare the NCEP average to your probability using
the same
> stats, perhaps Marion's HIRA method would give you a way to do that?
>
> Tressa
>
> On Thu Mar 09 07:13:13 2017, robert.craig.2 at us.af.mil wrote:
> > John, I am converting our ensemble verification over to MET
software.
> > In grid stat, I will be verifying probability data and will use
NCEP
> > climo for the reference. The question I have and I did not find
it in
> > the user guide: will I have to convert the climo file to 1 or 0s
- 1
> > being the threshold in question was met and 0 meaning is wasn't or
> > does MET do this for me automatically? So if I am verifying wind
> > speeds > 25 knots, can I just provide a climo average wind speed
file
> > or do I need to threshold it against 25 knots?
> >
> > Also, could you place the MET5.2 guide on the FTP server in my
> > directory. I can't seem to down load it from the met help web
site.
> >
> > Thanks
> > Bob
>
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list