[Met_help] [rt.rap.ucar.edu #71420] History for BCRMSE
John Halley Gotway via RT
met_help at ucar.edu
Thu Apr 16 15:04:28 MDT 2015
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I have a question about BCRMSE. We were shown some WRF MET verification comparisons of 3.5.1 vs 3.6.1. On a couple of the slides we saw the result that the bias value was reduced but the BCRMSE increased. Considering that the RMSE is calculated from the squared sum of the biases, how can the BCRMSE go up when the bias goes down?
Thanks
Bob Craig
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: BCRMSE
From: John Halley Gotway
Time: Tue Apr 14 09:57:05 2015
Hi Bob,
This is John. We received your question yesterday, and I've been
talking
to the statisticians trying to get a good answer for you. But I think
it'd
help if I could narrow down to the specific plots in question. I
checked
our wiki site and we have *many* different presentations with
verification
results. Are you able to send me the presentation that shows this
behavior? Or the specific plots that show this?
Thanks,
John
On Mon, Apr 13, 2015 at 9:31 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Mon Apr 13 09:31:27 2015: Request 71420 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: BCRMSE
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71420 >
>
>
> I have a question about BCRMSE. We were shown some WRF MET
verification
> comparisons of 3.5.1 vs 3.6.1. On a couple of the slides we saw the
result
> that the bias value was reduced but the BCRMSE increased.
Considering
> that the RMSE is calculated from the squared sum of the biases, how
can the
> BCRMSE go up when the bias goes down?
>
> Thanks
>
> Bob Craig
>
>
>
------------------------------------------------
Subject: BCRMSE
From: John Halley Gotway
Time: Tue Apr 14 10:45:23 2015
Hello Bob,
I looked at some MET output for 2-m temperature computed over CONUS
for a
single time, the 48-hour lead time for the 2011070400 initialization.
I
compared the output from WRFv3.5.1 to WRFv3.6. Listed below are the
ME,
MSE, and BCMSE values.
MODEL ME MSE BCMSE
AFWAOC_WRFv3.5.1_d01 -1.19469 10.72606 9.29877
AFWAOC_WRFv3.6_d01 -0.80111 10.51906 9.87728
MET doesn't actually computed BCRMSE. Instead we just take the square
root
of BCMSE when we plot it. The decomposition of the mean-squared error
works like this:
MSE = ME*ME + BCMSE
In the above example, we see that from v3.5.1 to v3.6, the ME (i.e.
BIAS)
improved from -1.19 to -0.80. So ME*ME decreased. The BCMSE values
increased from 9.30 to 9.88.
Your question was, how can the bias decrease when the BCMSE increases?
Looking at the above equation, those components are both on the right
side
of the equation. So one can easily go up while the other goes down.
However, if both ME and BCMSE increased, then it would be impossible
for
MSE to decrease.
Does that help explain?
Thanks,
John
On Tue, Apr 14, 2015 at 9:56 AM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Hi Bob,
>
> This is John. We received your question yesterday, and I've been
talking
> to the statisticians trying to get a good answer for you. But I
think it'd
> help if I could narrow down to the specific plots in question. I
checked
> our wiki site and we have *many* different presentations with
verification
> results. Are you able to send me the presentation that shows this
> behavior? Or the specific plots that show this?
>
> Thanks,
> John
>
> On Mon, Apr 13, 2015 at 9:31 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Mon Apr 13 09:31:27 2015: Request 71420 was acted upon.
>> Transaction: Ticket created by robert.craig.2 at us.af.mil
>> Queue: met_help
>> Subject: BCRMSE
>> Owner: Nobody
>> Requestors: robert.craig.2 at us.af.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71420 >
>>
>>
>> I have a question about BCRMSE. We were shown some WRF MET
verification
>> comparisons of 3.5.1 vs 3.6.1. On a couple of the slides we saw
the result
>> that the bias value was reduced but the BCRMSE increased.
Considering
>> that the RMSE is calculated from the squared sum of the biases, how
can the
>> BCRMSE go up when the bias goes down?
>>
>> Thanks
>>
>> Bob Craig
>>
>>
>>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #71420] BCRMSE
From: robert.craig.2 at us.af.mil
Time: Wed Apr 15 06:55:18 2015
Thanks John, that explains it.
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, April 14, 2015 11:45 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #71420] BCRMSE
Hello Bob,
I looked at some MET output for 2-m temperature computed over CONUS
for a single time, the 48-hour lead time for the 2011070400
initialization. I compared the output from WRFv3.5.1 to WRFv3.6.
Listed below are the ME, MSE, and BCMSE values.
MODEL ME MSE BCMSE
AFWAOC_WRFv3.5.1_d01 -1.19469 10.72606 9.29877
AFWAOC_WRFv3.6_d01 -0.80111 10.51906 9.87728
MET doesn't actually computed BCRMSE. Instead we just take the square
root of BCMSE when we plot it. The decomposition of the mean-squared
error works like this:
MSE = ME*ME + BCMSE
In the above example, we see that from v3.5.1 to v3.6, the ME (i.e.
BIAS) improved from -1.19 to -0.80. So ME*ME decreased. The BCMSE
values increased from 9.30 to 9.88.
Your question was, how can the bias decrease when the BCMSE increases?
Looking at the above equation, those components are both on the right
side of the equation. So one can easily go up while the other goes
down.
However, if both ME and BCMSE increased, then it would be impossible
for MSE to decrease.
Does that help explain?
Thanks,
John
On Tue, Apr 14, 2015 at 9:56 AM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Hi Bob,
>
> This is John. We received your question yesterday, and I've been
> talking to the statisticians trying to get a good answer for you.
But
> I think it'd help if I could narrow down to the specific plots in
> question. I checked our wiki site and we have *many* different
> presentations with verification results. Are you able to send me
the
> presentation that shows this behavior? Or the specific plots that
show this?
>
> Thanks,
> John
>
> On Mon, Apr 13, 2015 at 9:31 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Mon Apr 13 09:31:27 2015: Request 71420 was acted upon.
>> Transaction: Ticket created by robert.craig.2 at us.af.mil
>> Queue: met_help
>> Subject: BCRMSE
>> Owner: Nobody
>> Requestors: robert.craig.2 at us.af.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71420
>> >
>>
>>
>> I have a question about BCRMSE. We were shown some WRF MET
verification
>> comparisons of 3.5.1 vs 3.6.1. On a couple of the slides we saw
the result
>> that the bias value was reduced but the BCRMSE increased.
Considering
>> that the RMSE is calculated from the squared sum of the biases, how
>> can the BCRMSE go up when the bias goes down?
>>
>> Thanks
>>
>> Bob Craig
>>
>>
>>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #71420] BCRMSE
From: robert.craig.2 at us.af.mil
Time: Wed Apr 15 08:12:20 2015
John, based on your answer, Steve would like to know how should we
(AFWA) be interpreting the results for the 3.5.1 vs 3.6.1 comparison.
This is probably a question to one of the authors of the comparison.
If one WRF version has a lower MSE, but a higher BCMSE, compared to
the other version, should we be looking more at the MSE to decide
which version did better.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, April 14, 2015 11:45 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #71420] BCRMSE
Hello Bob,
I looked at some MET output for 2-m temperature computed over CONUS
for a single time, the 48-hour lead time for the 2011070400
initialization. I compared the output from WRFv3.5.1 to WRFv3.6.
Listed below are the ME, MSE, and BCMSE values.
MODEL ME MSE BCMSE
AFWAOC_WRFv3.5.1_d01 -1.19469 10.72606 9.29877
AFWAOC_WRFv3.6_d01 -0.80111 10.51906 9.87728
MET doesn't actually computed BCRMSE. Instead we just take the square
root of BCMSE when we plot it. The decomposition of the mean-squared
error works like this:
MSE = ME*ME + BCMSE
In the above example, we see that from v3.5.1 to v3.6, the ME (i.e.
BIAS) improved from -1.19 to -0.80. So ME*ME decreased. The BCMSE
values increased from 9.30 to 9.88.
Your question was, how can the bias decrease when the BCMSE increases?
Looking at the above equation, those components are both on the right
side of the equation. So one can easily go up while the other goes
down.
However, if both ME and BCMSE increased, then it would be impossible
for MSE to decrease.
Does that help explain?
Thanks,
John
On Tue, Apr 14, 2015 at 9:56 AM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Hi Bob,
>
> This is John. We received your question yesterday, and I've been
> talking to the statisticians trying to get a good answer for you.
But
> I think it'd help if I could narrow down to the specific plots in
> question. I checked our wiki site and we have *many* different
> presentations with verification results. Are you able to send me
the
> presentation that shows this behavior? Or the specific plots that
show this?
>
> Thanks,
> John
>
> On Mon, Apr 13, 2015 at 9:31 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Mon Apr 13 09:31:27 2015: Request 71420 was acted upon.
>> Transaction: Ticket created by robert.craig.2 at us.af.mil
>> Queue: met_help
>> Subject: BCRMSE
>> Owner: Nobody
>> Requestors: robert.craig.2 at us.af.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71420
>> >
>>
>>
>> I have a question about BCRMSE. We were shown some WRF MET
verification
>> comparisons of 3.5.1 vs 3.6.1. On a couple of the slides we saw
the result
>> that the bias value was reduced but the BCRMSE increased.
Considering
>> that the RMSE is calculated from the squared sum of the biases, how
>> can the BCRMSE go up when the bias goes down?
>>
>> Thanks
>>
>> Bob Craig
>>
>>
>>
>
------------------------------------------------
Subject: BCRMSE
From: John Halley Gotway
Time: Wed Apr 15 09:41:36 2015
Bob,
To clarify, I believe the comparison that we presented was between
versions
3.5.1 and 3.6, not 3.6.1. May be a minor point, but I just wanted to
clear
that up.
You're correct, I'm not the right person to talk to about interpreting
the
results. I'll go ahead and resolve this ticket, but then I'll forward
it
to Jamie Wolff, who led the comparison.
Thanks,
John
On Wed, Apr 15, 2015 at 8:12 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71420 >
>
> John, based on your answer, Steve would like to know how should we
(AFWA)
> be interpreting the results for the 3.5.1 vs 3.6.1 comparison.
This is
> probably a question to one of the authors of the comparison. If
one WRF
> version has a lower MSE, but a higher BCMSE, compared to the other
version,
> should we be looking more at the MSE to decide which version did
better.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, April 14, 2015 11:45 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #71420] BCRMSE
>
> Hello Bob,
>
> I looked at some MET output for 2-m temperature computed over CONUS
for a
> single time, the 48-hour lead time for the 2011070400
initialization. I
> compared the output from WRFv3.5.1 to WRFv3.6. Listed below are the
ME,
> MSE, and BCMSE values.
>
> MODEL ME MSE BCMSE
> AFWAOC_WRFv3.5.1_d01 -1.19469 10.72606 9.29877
> AFWAOC_WRFv3.6_d01 -0.80111 10.51906 9.87728
>
> MET doesn't actually computed BCRMSE. Instead we just take the
square
> root of BCMSE when we plot it. The decomposition of the mean-
squared error
> works like this:
> MSE = ME*ME + BCMSE
>
> In the above example, we see that from v3.5.1 to v3.6, the ME (i.e.
BIAS)
> improved from -1.19 to -0.80. So ME*ME decreased. The BCMSE values
> increased from 9.30 to 9.88.
>
> Your question was, how can the bias decrease when the BCMSE
increases?
> Looking at the above equation, those components are both on the
right side
> of the equation. So one can easily go up while the other goes down.
> However, if both ME and BCMSE increased, then it would be impossible
for
> MSE to decrease.
>
> Does that help explain?
>
> Thanks,
> John
>
>
> On Tue, Apr 14, 2015 at 9:56 AM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Hi Bob,
> >
> > This is John. We received your question yesterday, and I've been
> > talking to the statisticians trying to get a good answer for you.
But
> > I think it'd help if I could narrow down to the specific plots in
> > question. I checked our wiki site and we have *many* different
> > presentations with verification results. Are you able to send me
the
> > presentation that shows this behavior? Or the specific plots that
show
> this?
> >
> > Thanks,
> > John
> >
> > On Mon, Apr 13, 2015 at 9:31 AM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> Mon Apr 13 09:31:27 2015: Request 71420 was acted upon.
> >> Transaction: Ticket created by robert.craig.2 at us.af.mil
> >> Queue: met_help
> >> Subject: BCRMSE
> >> Owner: Nobody
> >> Requestors: robert.craig.2 at us.af.mil
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71420
> >> >
> >>
> >>
> >> I have a question about BCRMSE. We were shown some WRF MET
> verification
> >> comparisons of 3.5.1 vs 3.6.1. On a couple of the slides we saw
the
> result
> >> that the bias value was reduced but the BCRMSE increased.
Considering
> >> that the RMSE is calculated from the squared sum of the biases,
how
> >> can the BCRMSE go up when the bias goes down?
> >>
> >> Thanks
> >>
> >> Bob Craig
> >>
> >>
> >>
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list