[Met_help] [rt.rap.ucar.edu #53444] History for Other very minor differences (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Tue Feb 14 09:51:36 MST 2012
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Classification: UNCLASSIFIED
Caveats: NONE
I am doing some benchmark testing of MET V3.0, V3.0.1 to verify the patch we
needed to Point-Stat to allow error-free use of 2 sets of observations files
(PrepBUFR and MADIS) which, at times, contains missing files for certain
hours and which was causing errors before you (John Halley Gotway) wrote the
patch for us which changes the error to just a warning. I haven't used
Point-Stat since last summer and wanted to make sure this was retained
before migrating to V3.1.
I confirmed that V3.0.1 contains this patch or includes this fix as part of
the release. Either way the functionality works fine.
The only thing I did notice when using V3.0 Point-Stat and V3.0.1 Point-Stat
is that my stats (ME, MAE, RMSE) are slightly different when generated by
each version. I was comparing stats I had generated last summer with V3.0 to
some I generated today with V3.0.1.
I made sure that the same input observations and forecasts were used each
time by comparing my log files which tracks the verbosity of Point-Stat as
it runs.
I went back and changed my current MET version from V3.0.1 back to MET V3.0
and then reran Point-Stat and I was able to regenerate the same results
identically as those I generated last summer.
Is there any chance that something was changed in Point-Stat between V3.0
and V3.0.1 which might account for the differences I'm seeing?
Thanks.
R/
John
Classification: UNCLASSIFIED
Caveats: NONE
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #53444] Other very minor differences (UNCLASSIFIED)
From: John Halley Gotway
Time: Mon Feb 13 16:08:55 2012
John,
I'll refer you to the release notes for METv3.0.1:
http://www.dtcenter.org/met/users/support/release_notes/METv3.0.1_release_notes.php
The change listed there that would likely produce very small
differences in statistics is changing the radius of the earth. You
know, it seems like we would have decided on one of these by now, but
there a couple of different values in common use. After going through
detailed process of comparing the output of MET to the output of the
NCEP Verification tools on which MET was originally based,
we changed the value used by MET to be consistent the value used by
the NCEP code.
We've considered making the radius of the earth a configurable
environment setting for MET, but have not done so yet. Would having
that flexibility be beneficial in your work?
Thanks,
John
On 02/13/2012 03:36 PM, Raby, John W USA CIV via RT wrote:
>
> Mon Feb 13 15:36:31 2012: Request 53444 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Other very minor differences (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=53444>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I am doing some benchmark testing of MET V3.0, V3.0.1 to verify the
patch we
> needed to Point-Stat to allow error-free use of 2 sets of
observations files
> (PrepBUFR and MADIS) which, at times, contains missing files for
certain
> hours and which was causing errors before you (John Halley Gotway)
wrote the
> patch for us which changes the error to just a warning. I haven't
used
> Point-Stat since last summer and wanted to make sure this was
retained
> before migrating to V3.1.
>
> I confirmed that V3.0.1 contains this patch or includes this fix as
part of
> the release. Either way the functionality works fine.
>
> The only thing I did notice when using V3.0 Point-Stat and V3.0.1
Point-Stat
> is that my stats (ME, MAE, RMSE) are slightly different when
generated by
> each version. I was comparing stats I had generated last summer with
V3.0 to
> some I generated today with V3.0.1.
>
> I made sure that the same input observations and forecasts were used
each
> time by comparing my log files which tracks the verbosity of Point-
Stat as
> it runs.
>
> I went back and changed my current MET version from V3.0.1 back to
MET V3.0
> and then reran Point-Stat and I was able to regenerate the same
results
> identically as those I generated last summer.
>
> Is there any chance that something was changed in Point-Stat between
V3.0
> and V3.0.1 which might account for the differences I'm seeing?
>
> Thanks.
>
> R/
> John
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
------------------------------------------------
Subject: Other very minor differences (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Feb 14 07:30:18 2012
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for your quick response.
I did not check in the release notes which describe the radius of the
earth
change as you point out. In answer to your question on whether this
should be
changed to configurable, I just can't envision my future concern over
this
parameter being in need of frequent changing. If I did have the
ability to
change the radius back to that used previously, I guess I would use it
to
preserve internal consistency among my stats generated using previous
MET
versions. The differences in the stats are just not big enough to be
of real
concern. Seeing the difference only caused a question to come up as to
"why" I
see this difference.
The only thing I would suggest for the future would be to include the
release
notes somewhere in the User's Guide. Yesterday I checked the section
"New for
V 3.0.1" on p vii in that User's Guide to see if something there would
explain
why Point-Stat results might be affected and nothing there did. Maybe
the
release notes could be included on p vii.
I'll make it a point to check the release notes on the website in
addition to
the User's Guide for clearing up questions like this.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, February 13, 2012 4:09 PM
To: Raby, John W CIV (US)
Subject: Re: [rt.rap.ucar.edu #53444] Other very minor differences
(UNCLASSIFIED)
John,
I'll refer you to the release notes for METv3.0.1:
http://www.dtcenter.org/met/users/support/release_notes/METv3.0.1_release_notes.php
The change listed there that would likely produce very small
differences in
statistics is changing the radius of the earth. You know, it seems
like we
would have decided on one of these by now, but
there a couple of different values in common use. After going through
detailed process of comparing the output of MET to the output of the
NCEP
Verification tools on which MET was originally based,
we changed the value used by MET to be consistent the value used by
the NCEP
code.
We've considered making the radius of the earth a configurable
environment
setting for MET, but have not done so yet. Would having that
flexibility be
beneficial in your work?
Thanks,
John
On 02/13/2012 03:36 PM, Raby, John W USA CIV via RT wrote:
>
> Mon Feb 13 15:36:31 2012: Request 53444 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Other very minor differences (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=53444>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I am doing some benchmark testing of MET V3.0, V3.0.1 to verify the
patch we
> needed to Point-Stat to allow error-free use of 2 sets of
observations files
> (PrepBUFR and MADIS) which, at times, contains missing files for
certain
> hours and which was causing errors before you (John Halley Gotway)
wrote the
> patch for us which changes the error to just a warning. I haven't
used
> Point-Stat since last summer and wanted to make sure this was
retained
> before migrating to V3.1.
>
> I confirmed that V3.0.1 contains this patch or includes this fix as
part of
> the release. Either way the functionality works fine.
>
> The only thing I did notice when using V3.0 Point-Stat and V3.0.1
Point-Stat
> is that my stats (ME, MAE, RMSE) are slightly different when
generated by
> each version. I was comparing stats I had generated last summer with
V3.0 to
> some I generated today with V3.0.1.
>
> I made sure that the same input observations and forecasts were used
each
> time by comparing my log files which tracks the verbosity of Point-
Stat as
> it runs.
>
> I went back and changed my current MET version from V3.0.1 back to
MET V3.0
> and then reran Point-Stat and I was able to regenerate the same
results
> identically as those I generated last summer.
>
> Is there any chance that something was changed in Point-Stat between
V3.0
> and V3.0.1 which might account for the differences I'm seeing?
>
> Thanks.
>
> R/
> John
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Other very minor differences (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Feb 14 08:56:58 2012
Classification: UNCLASSIFIED
Caveats: NONE
John -
I thought you might want to see some more results when I did a similar
test to
compare V3.0.1 with V3.1.
The Point-Stat (CNT) results were different but this time the
differences were
much smaller than those between V3.0 and V3.0.1.
These differences are close to negligible and are out in the 4th and
5th
decimal place.
Also, I was able to read in observations from 2 separate files for
each hour
with only a warning one of the files was not available which means
that fix is
effective.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, February 13, 2012 4:09 PM
To: Raby, John W CIV (US)
Subject: Re: [rt.rap.ucar.edu #53444] Other very minor differences
(UNCLASSIFIED)
John,
I'll refer you to the release notes for METv3.0.1:
http://www.dtcenter.org/met/users/support/release_notes/METv3.0.1_release_notes.php
The change listed there that would likely produce very small
differences in
statistics is changing the radius of the earth. You know, it seems
like we
would have decided on one of these by now, but there a couple of
different
values in common use. After going through detailed process of
comparing the
output of MET to the output of the NCEP Verification tools on which
MET was
originally based, we changed the value used by MET to be consistent
the value
used by the NCEP code.
We've considered making the radius of the earth a configurable
environment
setting for MET, but have not done so yet. Would having that
flexibility be
beneficial in your work?
Thanks,
John
On 02/13/2012 03:36 PM, Raby, John W USA CIV via RT wrote:
>
> Mon Feb 13 15:36:31 2012: Request 53444 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Other very minor differences (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=53444>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I am doing some benchmark testing of MET V3.0, V3.0.1 to verify the
> patch we needed to Point-Stat to allow error-free use of 2 sets of
> observations files (PrepBUFR and MADIS) which, at times, contains
> missing files for certain hours and which was causing errors before
> you (John Halley Gotway) wrote the patch for us which changes the
> error to just a warning. I haven't used Point-Stat since last summer
> and wanted to make sure this was retained before migrating to V3.1.
>
> I confirmed that V3.0.1 contains this patch or includes this fix as
> part of the release. Either way the functionality works fine.
>
> The only thing I did notice when using V3.0 Point-Stat and V3.0.1
> Point-Stat is that my stats (ME, MAE, RMSE) are slightly different
> when generated by each version. I was comparing stats I had
generated
> last summer with V3.0 to some I generated today with V3.0.1.
>
> I made sure that the same input observations and forecasts were used
> each time by comparing my log files which tracks the verbosity of
> Point-Stat as it runs.
>
> I went back and changed my current MET version from V3.0.1 back to
MET
> V3.0 and then reran Point-Stat and I was able to regenerate the same
> results identically as those I generated last summer.
>
> Is there any chance that something was changed in Point-Stat between
> V3.0 and V3.0.1 which might account for the differences I'm seeing?
>
> Thanks.
>
> R/
> John
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #53444] Other very minor differences (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Feb 14 09:27:27 2012
John,
Great. Glad you were able to isolate the differences out in the 4th
or 5th decimal place. That's exactly the impact we expected when
resolving the scale and offset issue. As you've already found
out though, the differences are more apparent when applying a
threshold - as in the computation of contingency table counts and
statistics and when defining MODE objects. And this is largely based
on
the choice of threshold you apply.
As for comparing results from different versions of MET, a more
conservative approach would be to only do direct comparisons of
statistics from the same version of MET. In some cases - like
tracking
long-term performance of a single model - that isn't practical. But
it's a good goal to shoot for.
Thanks,
John
On 02/14/2012 08:56 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=53444>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> I thought you might want to see some more results when I did a
similar test to
> compare V3.0.1 with V3.1.
>
> The Point-Stat (CNT) results were different but this time the
differences were
> much smaller than those between V3.0 and V3.0.1.
>
> These differences are close to negligible and are out in the 4th and
5th
> decimal place.
>
> Also, I was able to read in observations from 2 separate files for
each hour
> with only a warning one of the files was not available which means
that fix is
> effective.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, February 13, 2012 4:09 PM
> To: Raby, John W CIV (US)
> Subject: Re: [rt.rap.ucar.edu #53444] Other very minor differences
> (UNCLASSIFIED)
>
> John,
>
> I'll refer you to the release notes for METv3.0.1:
>
http://www.dtcenter.org/met/users/support/release_notes/METv3.0.1_release_notes.php
>
> The change listed there that would likely produce very small
differences in
> statistics is changing the radius of the earth. You know, it seems
like we
> would have decided on one of these by now, but there a couple of
different
> values in common use. After going through detailed process of
comparing the
> output of MET to the output of the NCEP Verification tools on which
MET was
> originally based, we changed the value used by MET to be consistent
the value
> used by the NCEP code.
>
> We've considered making the radius of the earth a configurable
environment
> setting for MET, but have not done so yet. Would having that
flexibility be
> beneficial in your work?
>
> Thanks,
> John
>
>
> On 02/13/2012 03:36 PM, Raby, John W USA CIV via RT wrote:
>>
>> Mon Feb 13 15:36:31 2012: Request 53444 was acted upon.
>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>> Queue: met_help
>> Subject: Other very minor differences (UNCLASSIFIED)
>> Owner: Nobody
>> Requestors: john.w.raby2.civ at mail.mil
>> Status: new
>> Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=53444>
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> I am doing some benchmark testing of MET V3.0, V3.0.1 to verify the
>> patch we needed to Point-Stat to allow error-free use of 2 sets of
>> observations files (PrepBUFR and MADIS) which, at times, contains
>> missing files for certain hours and which was causing errors before
>> you (John Halley Gotway) wrote the patch for us which changes the
>> error to just a warning. I haven't used Point-Stat since last
summer
>> and wanted to make sure this was retained before migrating to V3.1.
>>
>> I confirmed that V3.0.1 contains this patch or includes this fix as
>> part of the release. Either way the functionality works fine.
>>
>> The only thing I did notice when using V3.0 Point-Stat and V3.0.1
>> Point-Stat is that my stats (ME, MAE, RMSE) are slightly different
>> when generated by each version. I was comparing stats I had
generated
>> last summer with V3.0 to some I generated today with V3.0.1.
>>
>> I made sure that the same input observations and forecasts were
used
>> each time by comparing my log files which tracks the verbosity of
>> Point-Stat as it runs.
>>
>> I went back and changed my current MET version from V3.0.1 back to
MET
>> V3.0 and then reran Point-Stat and I was able to regenerate the
same
>> results identically as those I generated last summer.
>>
>> Is there any chance that something was changed in Point-Stat
between
>> V3.0 and V3.0.1 which might account for the differences I'm seeing?
>>
>> Thanks.
>>
>> R/
>> John
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
>>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
------------------------------------------------
Subject: Other very minor differences (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Feb 14 09:50:56 2012
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for your evaluation of the results I'm getting. It really helps
in
putting the perspective on version differences.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, February 14, 2012 9:27 AM
To: Raby, John W CIV (US)
Subject: Re: [rt.rap.ucar.edu #53444] Other very minor differences
(UNCLASSIFIED)
John,
Great. Glad you were able to isolate the differences out in the 4th
or 5th
decimal place. That's exactly the impact we expected when resolving
the scale
and offset issue. As you've already found out though, the differences
are
more apparent when applying a threshold - as in the computation of
contingency
table counts and statistics and when defining MODE objects. And this
is
largely based on the choice of threshold you apply.
As for comparing results from different versions of MET, a more
conservative
approach would be to only do direct comparisons of statistics from the
same
version of MET. In some cases - like tracking long-term performance
of a
single model - that isn't practical. But it's a good goal to shoot
for.
Thanks,
John
On 02/14/2012 08:56 AM, Raby, John W USA CIV via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=53444>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> I thought you might want to see some more results when I did a
similar
> test to compare V3.0.1 with V3.1.
>
> The Point-Stat (CNT) results were different but this time the
> differences were much smaller than those between V3.0 and V3.0.1.
>
> These differences are close to negligible and are out in the 4th and
> 5th decimal place.
>
> Also, I was able to read in observations from 2 separate files for
> each hour with only a warning one of the files was not available
which
> means that fix is effective.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, February 13, 2012 4:09 PM
> To: Raby, John W CIV (US)
> Subject: Re: [rt.rap.ucar.edu #53444] Other very minor differences
> (UNCLASSIFIED)
>
> John,
>
> I'll refer you to the release notes for METv3.0.1:
>
>
http://www.dtcenter.org/met/users/support/release_notes/METv3.0.1_rele
> ase_notes.php
>
> The change listed there that would likely produce very small
> differences in statistics is changing the radius of the earth. You
> know, it seems like we would have decided on one of these by now,
but
> there a couple of different values in common use. After going
through
> detailed process of comparing the output of MET to the output of the
> NCEP Verification tools on which MET was originally based, we
changed
> the value used by MET to be consistent the value used by the NCEP
code.
>
> We've considered making the radius of the earth a configurable
> environment setting for MET, but have not done so yet. Would having
> that flexibility be beneficial in your work?
>
> Thanks,
> John
>
>
> On 02/13/2012 03:36 PM, Raby, John W USA CIV via RT wrote:
>>
>> Mon Feb 13 15:36:31 2012: Request 53444 was acted upon.
>> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>> Queue: met_help
>> Subject: Other very minor differences (UNCLASSIFIED)
>> Owner: Nobody
>> Requestors: john.w.raby2.civ at mail.mil
>> Status: new
>> Ticket<URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=53444>
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> I am doing some benchmark testing of MET V3.0, V3.0.1 to verify the
>> patch we needed to Point-Stat to allow error-free use of 2 sets of
>> observations files (PrepBUFR and MADIS) which, at times, contains
>> missing files for certain hours and which was causing errors before
>> you (John Halley Gotway) wrote the patch for us which changes the
>> error to just a warning. I haven't used Point-Stat since last
summer
>> and wanted to make sure this was retained before migrating to V3.1.
>>
>> I confirmed that V3.0.1 contains this patch or includes this fix as
>> part of the release. Either way the functionality works fine.
>>
>> The only thing I did notice when using V3.0 Point-Stat and V3.0.1
>> Point-Stat is that my stats (ME, MAE, RMSE) are slightly different
>> when generated by each version. I was comparing stats I had
generated
>> last summer with V3.0 to some I generated today with V3.0.1.
>>
>> I made sure that the same input observations and forecasts were
used
>> each time by comparing my log files which tracks the verbosity of
>> Point-Stat as it runs.
>>
>> I went back and changed my current MET version from V3.0.1 back to
>> MET V3.0 and then reran Point-Stat and I was able to regenerate the
>> same results identically as those I generated last summer.
>>
>> Is there any chance that something was changed in Point-Stat
between
>> V3.0 and V3.0.1 which might account for the differences I'm seeing?
>>
>> Thanks.
>>
>> R/
>> John
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
>>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
More information about the Met_help
mailing list