[Met_help] [rt.rap.ucar.edu #77000] History for CRPS Calculations
John Halley Gotway via RT
met_help at ucar.edu
Fri Aug 12 11:46:17 MDT 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi John or Julie, as part of HiRA set up, I need to verify temperature
neighborhoods and generate a CRPS skill score. So, I generate a MET ORANK
file containing my ensemble values and then use MET to generate the RHIST
file. I use persistence value for the climo field.
To test the CRPS calculation, I limited the ORANK file to one line of data
and ran Statanalysis to generate the RHIST file. I attached both the
ORANK file and the RHIST file. I also put the CRPS calculation from the
HERSBACK paper (October 2000) into a spreadsheet (attached). When I run the
numbers in the ORANK file through the spreadsheet, I get a CRPS value of
.178. The CRPS value in the RHIST file is .162. So, we have a
discrepancy. Was the HERSBACK paper used to code CRPS for MET or some
other reference? Also, CRPSS is -1.59 which is terrible. But the climo
value of 7.17 falls within the ensemble members so should it be so bad.
How is the climo value used to calculate the CRPSS in MET?
The job I am running to calculate the RHIST file is:
/home/qcteam/met-5.1/bin/stat_analysis -lookin
/h/data/global/WXQC/data/met/mdlob_pairs/TT/temp -out
/h/data/global/WXQC/data/met/summary/GALWEM_TMP_00hr_9_RHIST_0Z -job
aggregate_stat -line_type ORANK -out_line_type RHIST -fcst_lead 000000 -v
6 -dump_row /h/data/global/WXQC/data/met/filter_job.stat
Thanks
Bob Craig
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: CRPS Calculations
From: John Halley Gotway
Time: Wed Jul 06 14:58:09 2016
Hello Bob,
I see you have questions about how MET is computing CRPS for a single
point. This is done in the function named compute_crps_ign_pit()
which
starts on line 1327 of this file:
src/libcode/vx_statistics/pair_data_ensemble.cc
Here's the reference for this equation:
http://journals.ametsoc.org/doi/full/10.1175/MWR2904.1
Specifically, look at equation (5) in that paper.
FYI, this is also listed in Appendix C of the MET User's Guide:
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.1.pdf
Hope that helps clarify. If you have more questions about Gneiting vs
Hersback, I could refer you to a statistician here.
Thanks,
John
On Tue, Jul 5, 2016 at 8:55 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Tue Jul 05 08:55:24 2016: Request 77000 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: CRPS Calculations
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000 >
>
>
> Hi John or Julie, as part of HiRA set up, I need to verify
temperature
> neighborhoods and generate a CRPS skill score. So, I generate a MET
ORANK
> file containing my ensemble values and then use MET to generate the
RHIST
> file. I use persistence value for the climo field.
>
> To test the CRPS calculation, I limited the ORANK file to one line
of data
> and ran Statanalysis to generate the RHIST file. I attached both
the
> ORANK file and the RHIST file. I also put the CRPS calculation
from the
> HERSBACK paper (October 2000) into a spreadsheet (attached). When I
run
> the
> numbers in the ORANK file through the spreadsheet, I get a CRPS
value of
> .178. The CRPS value in the RHIST file is .162. So, we have a
> discrepancy. Was the HERSBACK paper used to code CRPS for MET or
some
> other reference? Also, CRPSS is -1.59 which is terrible. But the
climo
> value of 7.17 falls within the ensemble members so should it be so
bad.
> How is the climo value used to calculate the CRPSS in MET?
>
> The job I am running to calculate the RHIST file is:
>
> /home/qcteam/met-5.1/bin/stat_analysis -lookin
> /h/data/global/WXQC/data/met/mdlob_pairs/TT/temp -out
> /h/data/global/WXQC/data/met/summary/GALWEM_TMP_00hr_9_RHIST_0Z
-job
> aggregate_stat -line_type ORANK -out_line_type RHIST -fcst_lead
000000 -v
> 6 -dump_row /h/data/global/WXQC/data/met/filter_job.stat
>
> Thanks
> Bob Craig
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #77000] CRPS Calculations
From: robert.craig.2 at us.af.mil
Time: Wed Jul 06 15:00:47 2016
Thanks John, the real questions is how you compute CRPSS?
Thanks
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, July 06, 2016 3:58 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #77000] CRPS Calculations
Hello Bob,
I see you have questions about how MET is computing CRPS for a single
point. This is done in the function named compute_crps_ign_pit()
which starts on line 1327 of this file:
src/libcode/vx_statistics/pair_data_ensemble.cc
Here's the reference for this equation:
http://journals.ametsoc.org/doi/full/10.1175/MWR2904.1
Specifically, look at equation (5) in that paper.
FYI, this is also listed in Appendix C of the MET User's Guide:
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.1.pdf
Hope that helps clarify. If you have more questions about Gneiting vs
Hersback, I could refer you to a statistician here.
Thanks,
John
On Tue, Jul 5, 2016 at 8:55 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Tue Jul 05 08:55:24 2016: Request 77000 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: CRPS Calculations
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000
> >
>
>
> Hi John or Julie, as part of HiRA set up, I need to verify
> temperature neighborhoods and generate a CRPS skill score. So, I
> generate a MET ORANK file containing my ensemble values and then use
> MET to generate the RHIST file. I use persistence value for the
climo field.
>
> To test the CRPS calculation, I limited the ORANK file to one line
of data
> and ran Statanalysis to generate the RHIST file. I attached both
the
> ORANK file and the RHIST file. I also put the CRPS calculation
from the
> HERSBACK paper (October 2000) into a spreadsheet (attached). When I
> run the numbers in the ORANK file through the spreadsheet, I get a
> CRPS value of
> .178. The CRPS value in the RHIST file is .162. So, we have a
> discrepancy. Was the HERSBACK paper used to code CRPS for MET or
some
> other reference? Also, CRPSS is -1.59 which is terrible. But the
climo
> value of 7.17 falls within the ensemble members so should it be so
bad.
> How is the climo value used to calculate the CRPSS in MET?
>
> The job I am running to calculate the RHIST file is:
>
> /home/qcteam/met-5.1/bin/stat_analysis -lookin
> /h/data/global/WXQC/data/met/mdlob_pairs/TT/temp -out
> /h/data/global/WXQC/data/met/summary/GALWEM_TMP_00hr_9_RHIST_0Z
-job
> aggregate_stat -line_type ORANK -out_line_type RHIST -fcst_lead
000000 -v
> 6 -dump_row /h/data/global/WXQC/data/met/filter_job.stat
>
> Thanks
> Bob Craig
>
>
>
>
------------------------------------------------
Subject: CRPS Calculations
From: John Halley Gotway
Time: Wed Jul 06 16:28:52 2016
Bob,
FYI, I also used the "verification" package in R to double-check the
CRPS
number we're getting from MET for your sample data...
R
library(verification)
ens = c(6.47018, 6.47018, 6.67020, 6.97018, 7.17020, 7.27020, 7.37018,
7.87018, 7.97018)
obs = 6.91998
crps(obs, c(mean(ens), sd(ens)))
$crps
[1] 0.1626081
Now as for the computation of CRPSS, that gets a lot messier! Tressa
and I
looked at the code for a while and do not think what it is doing now
is
very good. Specifically, the computation of climatological CRPS isn't
very
good, and thus, the CRPSS isn't very good.
Tressa tells me that what is really required is the spread of the
climatology values. However, there is no slot for that in the ORANK
line
type, as of now. And we can't break any of the MET line types until
the
6.0 release. Also, coming in to play, is that we need to replicate
the
logic that NCEP is using. I have their code, but am still working
through
it.
Needless to say, we have some more details to hammer out.
John
On Wed, Jul 6, 2016 at 3:00 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000 >
>
> Thanks John, the real questions is how you compute CRPSS?
>
> Thanks
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, July 06, 2016 3:58 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #77000] CRPS Calculations
>
> Hello Bob,
>
> I see you have questions about how MET is computing CRPS for a
single
> point. This is done in the function named compute_crps_ign_pit()
which
> starts on line 1327 of this file:
> src/libcode/vx_statistics/pair_data_ensemble.cc
>
> Here's the reference for this equation:
> http://journals.ametsoc.org/doi/full/10.1175/MWR2904.1
>
> Specifically, look at equation (5) in that paper.
>
> FYI, this is also listed in Appendix C of the MET User's Guide:
>
>
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.1.pdf
>
> Hope that helps clarify. If you have more questions about Gneiting
vs
> Hersback, I could refer you to a statistician here.
>
> Thanks,
> John
>
>
> On Tue, Jul 5, 2016 at 8:55 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jul 05 08:55:24 2016: Request 77000 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: CRPS Calculations
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000
> > >
> >
> >
> > Hi John or Julie, as part of HiRA set up, I need to verify
> > temperature neighborhoods and generate a CRPS skill score. So, I
> > generate a MET ORANK file containing my ensemble values and then
use
> > MET to generate the RHIST file. I use persistence value for the
climo
> field.
> >
> > To test the CRPS calculation, I limited the ORANK file to one line
of
> data
> > and ran Statanalysis to generate the RHIST file. I attached
both the
> > ORANK file and the RHIST file. I also put the CRPS calculation
from
> the
> > HERSBACK paper (October 2000) into a spreadsheet (attached). When
I
> > run the numbers in the ORANK file through the spreadsheet, I get a
> > CRPS value of
> > .178. The CRPS value in the RHIST file is .162. So, we have a
> > discrepancy. Was the HERSBACK paper used to code CRPS for MET
or some
> > other reference? Also, CRPSS is -1.59 which is terrible. But
the climo
> > value of 7.17 falls within the ensemble members so should it be so
bad.
> > How is the climo value used to calculate the CRPSS in MET?
> >
> > The job I am running to calculate the RHIST file is:
> >
> > /home/qcteam/met-5.1/bin/stat_analysis -lookin
> > /h/data/global/WXQC/data/met/mdlob_pairs/TT/temp -out
> > /h/data/global/WXQC/data/met/summary/GALWEM_TMP_00hr_9_RHIST_0Z
-job
> > aggregate_stat -line_type ORANK -out_line_type RHIST -fcst_lead
000000
> -v
> > 6 -dump_row /h/data/global/WXQC/data/met/filter_job.stat
> >
> > Thanks
> > Bob Craig
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #77000] CRPS Calculations
From: robert.craig.2 at us.af.mil
Time: Thu Jul 07 07:58:14 2016
Thanks for checking. Are you confident that your use of climatology
for Brier Skill Score is good? I will use that heavily in HiRA.
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, July 06, 2016 5:29 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #77000] CRPS Calculations
Bob,
FYI, I also used the "verification" package in R to double-check the
CRPS number we're getting from MET for your sample data...
R
library(verification)
ens = c(6.47018, 6.47018, 6.67020, 6.97018, 7.17020, 7.27020, 7.37018,
7.87018, 7.97018) obs = 6.91998 crps(obs, c(mean(ens), sd(ens))) $crps
[1] 0.1626081
Now as for the computation of CRPSS, that gets a lot messier! Tressa
and I looked at the code for a while and do not think what it is doing
now is very good. Specifically, the computation of climatological
CRPS isn't very good, and thus, the CRPSS isn't very good.
Tressa tells me that what is really required is the spread of the
climatology values. However, there is no slot for that in the ORANK
line type, as of now. And we can't break any of the MET line types
until the
6.0 release. Also, coming in to play, is that we need to replicate
the logic that NCEP is using. I have their code, but am still working
through it.
Needless to say, we have some more details to hammer out.
John
On Wed, Jul 6, 2016 at 3:00 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000 >
>
> Thanks John, the real questions is how you compute CRPSS?
>
> Thanks
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, July 06, 2016 3:58 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #77000] CRPS Calculations
>
> Hello Bob,
>
> I see you have questions about how MET is computing CRPS for a
single
> point. This is done in the function named compute_crps_ign_pit()
> which starts on line 1327 of this file:
> src/libcode/vx_statistics/pair_data_ensemble.cc
>
> Here's the reference for this equation:
> http://journals.ametsoc.org/doi/full/10.1175/MWR2904.1
>
> Specifically, look at equation (5) in that paper.
>
> FYI, this is also listed in Appendix C of the MET User's Guide:
>
>
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.
> 1.pdf
>
> Hope that helps clarify. If you have more questions about Gneiting
vs
> Hersback, I could refer you to a statistician here.
>
> Thanks,
> John
>
>
> On Tue, Jul 5, 2016 at 8:55 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jul 05 08:55:24 2016: Request 77000 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: CRPS Calculations
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000
> > >
> >
> >
> > Hi John or Julie, as part of HiRA set up, I need to verify
> > temperature neighborhoods and generate a CRPS skill score. So, I
> > generate a MET ORANK file containing my ensemble values and then
use
> > MET to generate the RHIST file. I use persistence value for the
> > climo
> field.
> >
> > To test the CRPS calculation, I limited the ORANK file to one line
> > of
> data
> > and ran Statanalysis to generate the RHIST file. I attached
both the
> > ORANK file and the RHIST file. I also put the CRPS calculation
from
> the
> > HERSBACK paper (October 2000) into a spreadsheet (attached). When
I
> > run the numbers in the ORANK file through the spreadsheet, I get a
> > CRPS value of
> > .178. The CRPS value in the RHIST file is .162. So, we have a
> > discrepancy. Was the HERSBACK paper used to code CRPS for MET
or some
> > other reference? Also, CRPSS is -1.59 which is terrible. But
the climo
> > value of 7.17 falls within the ensemble members so should it be so
bad.
> > How is the climo value used to calculate the CRPSS in MET?
> >
> > The job I am running to calculate the RHIST file is:
> >
> > /home/qcteam/met-5.1/bin/stat_analysis -lookin
> > /h/data/global/WXQC/data/met/mdlob_pairs/TT/temp -out
> > /h/data/global/WXQC/data/met/summary/GALWEM_TMP_00hr_9_RHIST_0Z
-job
> > aggregate_stat -line_type ORANK -out_line_type RHIST -fcst_lead
000000
> -v
> > 6 -dump_row /h/data/global/WXQC/data/met/filter_job.stat
> >
> > Thanks
> > Bob Craig
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: CRPS Calculations
From: John Halley Gotway
Time: Thu Jul 07 10:21:03 2016
Bob,
The brier score, and thus brier skill score, is much simpler than
CRPS.
For each observation, we have a forecast probability value along with
a
"climatological" (or reference) probability value. We compute the
brier
score in the same way for the forecast probabilities as we do for the
climo
probabilities, and use those scores to compute BSS. The bug you found
was
in that last step, computing the skill score. But yes, I feel pretty
confident that we're using the climo values appropriately in the
computation of BSS.
CRPS is messier because we aren't using a single forecast value.
Instead,
it's computed using an ensemble of forecast values (9 in your
example).
We compute the ensemble mean and spread of the ensemble at that point.
The
forecast CRPS is computed using the ensemble mean and spread along
with the
observation value.
Now for the climatology, all we have at this point is the
climatological
mean. We are missing the spread. According to Tressa, adding the
spread
and using it in the computation would be the most appropriate thing to
do.
However, we can't add that to the ORANK line type until version 6.0.
And
we also need to check to make sure this would be consistent with the
logic
used at NCEP. So we have some details to iron out.
Tressa, please correct me if that description isn't accurate.
Thanks,
John
On Thu, Jul 7, 2016 at 7:58 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000 >
>
> Thanks for checking. Are you confident that your use of climatology
for
> Brier Skill Score is good? I will use that heavily in HiRA.
>
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, July 06, 2016 5:29 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #77000] CRPS Calculations
>
> Bob,
>
> FYI, I also used the "verification" package in R to double-check the
CRPS
> number we're getting from MET for your sample data...
>
> R
> library(verification)
> ens = c(6.47018, 6.47018, 6.67020, 6.97018, 7.17020, 7.27020,
7.37018,
> 7.87018, 7.97018) obs = 6.91998 crps(obs, c(mean(ens), sd(ens)))
$crps [1]
> 0.1626081
>
> Now as for the computation of CRPSS, that gets a lot messier!
Tressa and
> I looked at the code for a while and do not think what it is doing
now is
> very good. Specifically, the computation of climatological CRPS
isn't very
> good, and thus, the CRPSS isn't very good.
>
> Tressa tells me that what is really required is the spread of the
> climatology values. However, there is no slot for that in the ORANK
line
> type, as of now. And we can't break any of the MET line types until
the
> 6.0 release. Also, coming in to play, is that we need to replicate
the
> logic that NCEP is using. I have their code, but am still working
through
> it.
>
> Needless to say, we have some more details to hammer out.
>
> John
>
>
> On Wed, Jul 6, 2016 at 3:00 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000 >
> >
> > Thanks John, the real questions is how you compute CRPSS?
> >
> > Thanks
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, July 06, 2016 3:58 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [rt.rap.ucar.edu #77000] CRPS Calculations
> >
> > Hello Bob,
> >
> > I see you have questions about how MET is computing CRPS for a
single
> > point. This is done in the function named compute_crps_ign_pit()
> > which starts on line 1327 of this file:
> > src/libcode/vx_statistics/pair_data_ensemble.cc
> >
> > Here's the reference for this equation:
> > http://journals.ametsoc.org/doi/full/10.1175/MWR2904.1
> >
> > Specifically, look at equation (5) in that paper.
> >
> > FYI, this is also listed in Appendix C of the MET User's Guide:
> >
> >
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.
> > 1.pdf
> >
> > Hope that helps clarify. If you have more questions about
Gneiting vs
> > Hersback, I could refer you to a statistician here.
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Jul 5, 2016 at 8:55 AM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jul 05 08:55:24 2016: Request 77000 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: CRPS Calculations
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000
> > > >
> > >
> > >
> > > Hi John or Julie, as part of HiRA set up, I need to verify
> > > temperature neighborhoods and generate a CRPS skill score. So,
I
> > > generate a MET ORANK file containing my ensemble values and then
use
> > > MET to generate the RHIST file. I use persistence value for the
> > > climo
> > field.
> > >
> > > To test the CRPS calculation, I limited the ORANK file to one
line
> > > of
> > data
> > > and ran Statanalysis to generate the RHIST file. I attached
both
> the
> > > ORANK file and the RHIST file. I also put the CRPS
calculation from
> > the
> > > HERSBACK paper (October 2000) into a spreadsheet (attached).
When I
> > > run the numbers in the ORANK file through the spreadsheet, I get
a
> > > CRPS value of
> > > .178. The CRPS value in the RHIST file is .162. So, we have
a
> > > discrepancy. Was the HERSBACK paper used to code CRPS for MET
or
> some
> > > other reference? Also, CRPSS is -1.59 which is terrible. But
the
> climo
> > > value of 7.17 falls within the ensemble members so should it be
so bad.
> > > How is the climo value used to calculate the CRPSS in MET?
> > >
> > > The job I am running to calculate the RHIST file is:
> > >
> > > /home/qcteam/met-5.1/bin/stat_analysis -lookin
> > > /h/data/global/WXQC/data/met/mdlob_pairs/TT/temp -out
> > > /h/data/global/WXQC/data/met/summary/GALWEM_TMP_00hr_9_RHIST_0Z
-job
> > > aggregate_stat -line_type ORANK -out_line_type RHIST
-fcst_lead
> 000000
> > -v
> > > 6 -dump_row /h/data/global/WXQC/data/met/filter_job.stat
> > >
> > > Thanks
> > > Bob Craig
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #77000] CRPS Calculations
From: robert.craig.2 at us.af.mil
Time: Thu Jul 07 10:35:51 2016
Got it. I changed my code to calculate a CRPS for model and a CRPS
for persistence in MET. Then I will calculate the skill scores
outside of MET. I look forward to your fix with 6.0 since I won't
have to carry two 0RANK and RHIST files around, but only one.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, July 07, 2016 11:21 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #77000] CRPS Calculations
Bob,
The brier score, and thus brier skill score, is much simpler than
CRPS.
For each observation, we have a forecast probability value along with
a "climatological" (or reference) probability value. We compute the
brier score in the same way for the forecast probabilities as we do
for the climo probabilities, and use those scores to compute BSS. The
bug you found was in that last step, computing the skill score. But
yes, I feel pretty confident that we're using the climo values
appropriately in the computation of BSS.
CRPS is messier because we aren't using a single forecast value.
Instead, it's computed using an ensemble of forecast values (9 in
your example).
We compute the ensemble mean and spread of the ensemble at that point.
The forecast CRPS is computed using the ensemble mean and spread along
with the observation value.
Now for the climatology, all we have at this point is the
climatological mean. We are missing the spread. According to Tressa,
adding the spread and using it in the computation would be the most
appropriate thing to do.
However, we can't add that to the ORANK line type until version 6.0.
And we also need to check to make sure this would be consistent with
the logic used at NCEP. So we have some details to iron out.
Tressa, please correct me if that description isn't accurate.
Thanks,
John
On Thu, Jul 7, 2016 at 7:58 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000 >
>
> Thanks for checking. Are you confident that your use of climatology
> for Brier Skill Score is good? I will use that heavily in HiRA.
>
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, July 06, 2016 5:29 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #77000] CRPS Calculations
>
> Bob,
>
> FYI, I also used the "verification" package in R to double-check the
> CRPS number we're getting from MET for your sample data...
>
> R
> library(verification)
> ens = c(6.47018, 6.47018, 6.67020, 6.97018, 7.17020, 7.27020,
7.37018,
> 7.87018, 7.97018) obs = 6.91998 crps(obs, c(mean(ens), sd(ens)))
$crps
> [1]
> 0.1626081
>
> Now as for the computation of CRPSS, that gets a lot messier!
Tressa
> and I looked at the code for a while and do not think what it is
doing
> now is very good. Specifically, the computation of climatological
> CRPS isn't very good, and thus, the CRPSS isn't very good.
>
> Tressa tells me that what is really required is the spread of the
> climatology values. However, there is no slot for that in the ORANK
> line type, as of now. And we can't break any of the MET line types
> until the
> 6.0 release. Also, coming in to play, is that we need to replicate
> the logic that NCEP is using. I have their code, but am still
working
> through it.
>
> Needless to say, we have some more details to hammer out.
>
> John
>
>
> On Wed, Jul 6, 2016 at 3:00 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000 >
> >
> > Thanks John, the real questions is how you compute CRPSS?
> >
> > Thanks
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, July 06, 2016 3:58 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [rt.rap.ucar.edu #77000] CRPS Calculations
> >
> > Hello Bob,
> >
> > I see you have questions about how MET is computing CRPS for a
> > single point. This is done in the function named
> > compute_crps_ign_pit() which starts on line 1327 of this file:
> > src/libcode/vx_statistics/pair_data_ensemble.cc
> >
> > Here's the reference for this equation:
> > http://journals.ametsoc.org/doi/full/10.1175/MWR2904.1
> >
> > Specifically, look at equation (5) in that paper.
> >
> > FYI, this is also listed in Appendix C of the MET User's Guide:
> >
> >
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.
> > 1.pdf
> >
> > Hope that helps clarify. If you have more questions about
Gneiting
> > vs Hersback, I could refer you to a statistician here.
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Jul 5, 2016 at 8:55 AM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jul 05 08:55:24 2016: Request 77000 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: CRPS Calculations
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=77000
> > > >
> > >
> > >
> > > Hi John or Julie, as part of HiRA set up, I need to verify
> > > temperature neighborhoods and generate a CRPS skill score. So,
I
> > > generate a MET ORANK file containing my ensemble values and then
> > > use MET to generate the RHIST file. I use persistence value for
> > > the climo
> > field.
> > >
> > > To test the CRPS calculation, I limited the ORANK file to one
line
> > > of
> > data
> > > and ran Statanalysis to generate the RHIST file. I attached
both
> the
> > > ORANK file and the RHIST file. I also put the CRPS
calculation from
> > the
> > > HERSBACK paper (October 2000) into a spreadsheet (attached).
When
> > > I run the numbers in the ORANK file through the spreadsheet, I
get
> > > a CRPS value of
> > > .178. The CRPS value in the RHIST file is .162. So, we have
a
> > > discrepancy. Was the HERSBACK paper used to code CRPS for MET
or
> some
> > > other reference? Also, CRPSS is -1.59 which is terrible. But
the
> climo
> > > value of 7.17 falls within the ensemble members so should it be
so bad.
> > > How is the climo value used to calculate the CRPSS in MET?
> > >
> > > The job I am running to calculate the RHIST file is:
> > >
> > > /home/qcteam/met-5.1/bin/stat_analysis -lookin
> > > /h/data/global/WXQC/data/met/mdlob_pairs/TT/temp -out
> > > /h/data/global/WXQC/data/met/summary/GALWEM_TMP_00hr_9_RHIST_0Z
-job
> > > aggregate_stat -line_type ORANK -out_line_type RHIST
-fcst_lead
> 000000
> > -v
> > > 6 -dump_row /h/data/global/WXQC/data/met/filter_job.stat
> > >
> > > Thanks
> > > Bob Craig
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list