[Met_help] [rt.rap.ucar.edu #69098] History for Stat-Analysis wind direction verification (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Mon Oct 20 13:21:39 MDT 2014
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Classification: UNCLASSIFIED
Caveats: NONE
I need to know more specifically how wind direction errors are calculated. I
ran a case using METV4.1 using a forecast and observations and I followed
the guidance to assure that the Point-Stat VL1L2 lines were generated. When
I ran Stat-Analysis I specified the job as aggregate_stat, the -line_type as
VL1L2 and the -out_line_type as WDIR. The attached document provides samples
of the output and my attempts to calculate the errors by hand so I could try
to understand how MET calculates them. I also highlighted the questions I
have in the document. I would appreciate having answers to my questions if
possible. Thanks.
R/
John
Classification: UNCLASSIFIED
Caveats: NONE
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Sep 25 08:54:05 2014
John
I'm out of the office today, but will take a close look at this on
Friday.
Thanks
John
On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
<javascript:;>
> Queue: met_help
> Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I need to know more specifically how wind direction errors are
calculated.
> I
> ran a case using METV4.1 using a forecast and observations and I
followed
> the guidance to assure that the Point-Stat VL1L2 lines were
generated. When
> I ran Stat-Analysis I specified the job as aggregate_stat, the
-line_type
> as
> VL1L2 and the -out_line_type as WDIR. The attached document provides
> samples
> of the output and my attempts to calculate the errors by hand so I
could
> try
> to understand how MET calculates them. I also highlighted the
questions I
> have in the document. I would appreciate having answers to my
questions if
> possible. Thanks.
>
> R/
> John
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Sep 25 09:24:34 2014
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for the update. I will be out Friday and Monday, returning on
Tuesday.
Have a great weekend!
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, September 25, 2014 8:54 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
verification (UNCLASSIFIED)
John
I'm out of the office today, but will take a close look at this on
Friday.
Thanks
John
On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
<javascript:;>
> Queue: met_help
> Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I need to know more specifically how wind direction errors are
calculated.
> I
> ran a case using METV4.1 using a forecast and observations and I
> followed the guidance to assure that the Point-Stat VL1L2 lines were
> generated. When I ran Stat-Analysis I specified the job as
> aggregate_stat, the -line_type as
> VL1L2 and the -out_line_type as WDIR. The attached document provides
> samples of the output and my attempts to calculate the errors by
hand
> so I could try to understand how MET calculates them. I also
> highlighted the questions I have in the document. I would appreciate
> having answers to my questions if possible. Thanks.
>
> R/
> John
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Sep 26 13:05:14 2014
John,
I read through the document you sent about the output of wind
direction
errors and will respond to your questions below...
I see that you've run a single VL1L2 line (computed over 103 matched
pairs)
through STAT-Analysis to compute wind direction errors. In the output
you
see the same values for FBAR (94.25) and OBAR (89.09) repeated in both
lines (ROW_MEAN_WDIR and AGGR_WDIR) of the output. You see that
because
this job is not actually aggregating multiple lines together. It's
just
being run over a single VL1L2 line. If your job was run over multiple
VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
AGGR_WDIR lines would differ.
Your first question is - how is the MAE value calculated? The FBAR
and
OBAR columns contain the forecast and observed wind directions, in
degrees. The error is the directed difference between those angles (f
-
o). The mean error (ME) is the average of those angular differences.
The
range of this column is -180 to +180. The MAE is the average of the
*absolute value* of those angular differences. So the range of the
MAE
column is 0 to +180.
You're wondering where in the MET code this logic is implemented. In
the
file "src/basic/vx_math/angles.cc" the function named
"convert_u_v_to_wdir"
handles the conversion of the U and V components to wind direction.
Also,
the file "src/tools/core/stat_analysis/stat_analysis_job.cc" contains
the
function "write_job_aggr_wind" which creates the output for STAT-
Analysis.
Your next question is why you the numbers are the same in
ROW_MEAN_WDIR And
AGGR_WDIR - again, that's because you're running over a single VL1L2
line.
If you run on multiple, they will likely differ. It's difficult to
describe the difference with words, but I'll give it a shot. Suppose
you're running this job over 10 different VL1L2 lines that represent
output
for 10 different days over the same region. And let's suppose 5 days
had
50 stations reporting and 5 days had 100 stations reporting...
- For ROW_MEAN_WDIR, we do the following...
- For each VL1L2 line (meaning, for each day), compute the average
forecast and observation direction and compute the angular difference.
- Compute the average of those 10 angular differences and write it
in the
ME column. Also, put the average of the absolute value of those
differences in the MAE column.
- This is the daily average of the wind direction errors. Even
though
some of the days were computed over 50 matched pairs and others were
computed over 100.
- For the AGGR_WDIR, aggregate together those 10 different VL1L2 lines
into
a single VL1L2 line. When we aggregate VL1L2 lines, we do a weighted
average, weighted by the "TOTAL" columns. Relatively speaking, those
5
days with a 100 matched pairs will count twice as much as the other 5
days
with only 50 matched pairs.
- From that single aggregated VL1L2 line, compute forecast and
observation directions. The ME is the angular difference between
them.
- We could write out the MAE which would just be the absolute value
of
the the ME, but we're just writing NA.
- This is an aggregated wind direction difference.
Both are reasonable ways of processing the data - but the computations
are
slightly different.
The real short-coming here is that we're only computing wind direction
errors over points that have already been aggregated through space.
In
METv4.1 STAT-Analysis, the only supported job type is "-job
aggregate_stat
-line_type VL1L2 -out_line_type MPR".
To address this short-coming in met-5.0, we added a similar job that
operates over matched pair values instead. Here's a selection from
the
met-5.0 release notes:
"Enhanced STAT-Analysis by adding support for an aggregate_stat job
type
conversion of MPR to WDIR that reads UGRD and VGRD matched pair lines
and
computes aggregated wind direction errors. Added -out_fcst_wind_thresh
and
-out_obs_wind_thresh options to filter the U/V pairs by wind speed
prior to
computing wind direction errors."
For example, the following job would compute wind direction errors for
each
station individually...
-job aggregate_wdir -line_type MPR -out_line_type WDIR -by OBS_SID
-out_fcst_wind_thresh \>+2.0
That'll compute wind direction errors for each unique station name in
the
OBS_SID column of the MPR line, but only when the forecast wind speed
exceeds 2 m/s.
This should make it easier to look at wind direction errors
station-by-station, rather than have to aggregate over a much larger
area.
Please let me know if this answers your questions. If not, I'll
probably
refer you to a scientist here to address the interpretation of the
output.
I can help you running the tools and describing the options, but I
think a
scientist would do a better job helping you interpret the results.
Thanks,
John
On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for the update. I will be out Friday and Monday, returning on
> Tuesday.
> Have a great weekend!
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, September 25, 2014 8:54 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John
>
> I'm out of the office today, but will take a close look at this on
Friday.
>
> Thanks
> John
>
> On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
<javascript:;>
> > Queue: met_help
> > Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I need to know more specifically how wind direction errors are
> calculated.
> > I
> > ran a case using METV4.1 using a forecast and observations and I
> > followed the guidance to assure that the Point-Stat VL1L2 lines
were
> > generated. When I ran Stat-Analysis I specified the job as
> > aggregate_stat, the -line_type as
> > VL1L2 and the -out_line_type as WDIR. The attached document
provides
> > samples of the output and my attempts to calculate the errors by
hand
> > so I could try to understand how MET calculates them. I also
> > highlighted the questions I have in the document. I would
appreciate
> > having answers to my questions if possible. Thanks.
> >
> > R/
> > John
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Oct 01 10:09:57 2014
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for your detailed explanations for my questions. I think it
confirmed
how the Stat-Analysis output is generated for me. Now it raises some
questions
about how the vector partial sums in the Point-Stat VL1L2 line type
are
generated.
I was able to trace how the partial sums UFBAR, VFBAR and UOBAR, VOBAR
were
generated from the UGRD, VGRD matched pair data in the MPR file in a
spreadsheet. I simply took the mean of all 103 values of the UGRD and
VGRD for
the Z10 level and the results were identical. For example: for UFBAR,
I took
the average of all 103 forecast values of UGRD and the result matched
the
value for UFBAR in the VL1L2 output. From your answer below, I gather
that you
use the code "convert_u_v_to_wdir" to convert UFBAR, VFBAR to generate
the
forecast wind direction (FBAR in the Stat-Analysis output) and the
same code
is used to convert UOBAR, VOBAR to generate the observed wind
direction (OBAR
in the Stat-analysis output). Correct so far?
So, based on your answer and the above I can trace the data in the MPR
file
all the way through the Stat-Analysis output of FBAR, OBAR, ME and
MAE. The
only remaining question, I have is about the vector partial sums:
UVFOBAR,
UVFFBAR and UVOOBAR. I looked in the User's Guide where the VL1L2 line
type is
described and I see the formulas for how these are calculated, but I
don't see
where these partial sums are used for generating the values in the
Stat-Analysis output. How are these partial sums used?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, September 26, 2014 1:05 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
verification (UNCLASSIFIED)
John,
I read through the document you sent about the output of wind
direction errors
and will respond to your questions below...
I see that you've run a single VL1L2 line (computed over 103 matched
pairs)
through STAT-Analysis to compute wind direction errors. In the output
you see
the same values for FBAR (94.25) and OBAR (89.09) repeated in both
lines
(ROW_MEAN_WDIR and AGGR_WDIR) of the output. You see that because
this job is
not actually aggregating multiple lines together. It's just being run
over a
single VL1L2 line. If your job was run over multiple
VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
AGGR_WDIR
lines would differ.
Your first question is - how is the MAE value calculated? The FBAR
and OBAR
columns contain the forecast and observed wind directions, in degrees.
The
error is the directed difference between those angles (f - o). The
mean error
(ME) is the average of those angular differences. The range of this
column
is -180 to +180. The MAE is the average of the *absolute value* of
those
angular differences. So the range of the MAE column is 0 to +180.
You're wondering where in the MET code this logic is implemented. In
the file
"src/basic/vx_math/angles.cc" the function named "convert_u_v_to_wdir"
handles the conversion of the U and V components to wind direction.
Also, the
file "src/tools/core/stat_analysis/stat_analysis_job.cc" contains the
function
"write_job_aggr_wind" which creates the output for STAT-Analysis.
Your next question is why you the numbers are the same in
ROW_MEAN_WDIR And
AGGR_WDIR - again, that's because you're running over a single VL1L2
line.
If you run on multiple, they will likely differ. It's difficult to
describe
the difference with words, but I'll give it a shot. Suppose you're
running
this job over 10 different VL1L2 lines that represent output for 10
different
days over the same region. And let's suppose 5 days had
50 stations reporting and 5 days had 100 stations reporting...
- For ROW_MEAN_WDIR, we do the following...
- For each VL1L2 line (meaning, for each day), compute the average
forecast
and observation direction and compute the angular difference.
- Compute the average of those 10 angular differences and write it
in the ME
column. Also, put the average of the absolute value of those
differences in
the MAE column.
- This is the daily average of the wind direction errors. Even
though some
of the days were computed over 50 matched pairs and others were
computed over
100.
- For the AGGR_WDIR, aggregate together those 10 different VL1L2 lines
into a
single VL1L2 line. When we aggregate VL1L2 lines, we do a weighted
average,
weighted by the "TOTAL" columns. Relatively speaking, those 5 days
with a 100
matched pairs will count twice as much as the other 5 days with only
50
matched pairs.
- From that single aggregated VL1L2 line, compute forecast and
observation
directions. The ME is the angular difference between them.
- We could write out the MAE which would just be the absolute value
of the
the ME, but we're just writing NA.
- This is an aggregated wind direction difference.
Both are reasonable ways of processing the data - but the computations
are
slightly different.
The real short-coming here is that we're only computing wind direction
errors
over points that have already been aggregated through space. In
METv4.1 STAT-Analysis, the only supported job type is "-job
aggregate_stat -line_type VL1L2 -out_line_type MPR".
To address this short-coming in met-5.0, we added a similar job that
operates
over matched pair values instead. Here's a selection from the
met-5.0 release notes:
"Enhanced STAT-Analysis by adding support for an aggregate_stat job
type
conversion of MPR to WDIR that reads UGRD and VGRD matched pair lines
and
computes aggregated wind direction errors. Added -out_fcst_wind_thresh
and -out_obs_wind_thresh options to filter the U/V pairs by wind speed
prior
to computing wind direction errors."
For example, the following job would compute wind direction errors for
each
station individually...
-job aggregate_wdir -line_type MPR -out_line_type WDIR -by
OBS_SID -out_fcst_wind_thresh \>+2.0
That'll compute wind direction errors for each unique station name in
the
OBS_SID column of the MPR line, but only when the forecast wind speed
exceeds
2 m/s.
This should make it easier to look at wind direction errors
station-by-station, rather than have to aggregate over a much larger
area.
Please let me know if this answers your questions. If not, I'll
probably
refer you to a scientist here to address the interpretation of the
output.
I can help you running the tools and describing the options, but I
think a
scientist would do a better job helping you interpret the results.
Thanks,
John
On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for the update. I will be out Friday and Monday, returning on
> Tuesday.
> Have a great weekend!
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, September 25, 2014 8:54 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John
>
> I'm out of the office today, but will take a close look at this on
Friday.
>
> Thanks
> John
>
> On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
<javascript:;>
> > Queue: met_help
> > Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I need to know more specifically how wind direction errors are
> calculated.
> > I
> > ran a case using METV4.1 using a forecast and observations and I
> > followed the guidance to assure that the Point-Stat VL1L2 lines
were
> > generated. When I ran Stat-Analysis I specified the job as
> > aggregate_stat, the -line_type as
> > VL1L2 and the -out_line_type as WDIR. The attached document
provides
> > samples of the output and my attempts to calculate the errors by
> > hand so I could try to understand how MET calculates them. I also
> > highlighted the questions I have in the document. I would
appreciate
> > having answers to my questions if possible. Thanks.
> >
> > R/
> > John
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Julie Prestopnik
Time: Wed Oct 01 10:13:26 2014
Hi John. John is currently out of the office, but will hopefully be
able
to respond to your questions later this week. I do not have enough
knowledge to answer your questions.
Thanks,
Julie
On Wed, Oct 1, 2014 at 10:09 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for your detailed explanations for my questions. I think it
> confirmed
> how the Stat-Analysis output is generated for me. Now it raises some
> questions
> about how the vector partial sums in the Point-Stat VL1L2 line type
are
> generated.
>
> I was able to trace how the partial sums UFBAR, VFBAR and UOBAR,
VOBAR were
> generated from the UGRD, VGRD matched pair data in the MPR file in a
> spreadsheet. I simply took the mean of all 103 values of the UGRD
and VGRD
> for
> the Z10 level and the results were identical. For example: for
UFBAR, I
> took
> the average of all 103 forecast values of UGRD and the result
matched the
> value for UFBAR in the VL1L2 output. From your answer below, I
gather that
> you
> use the code "convert_u_v_to_wdir" to convert UFBAR, VFBAR to
generate the
> forecast wind direction (FBAR in the Stat-Analysis output) and the
same
> code
> is used to convert UOBAR, VOBAR to generate the observed wind
direction
> (OBAR
> in the Stat-analysis output). Correct so far?
>
> So, based on your answer and the above I can trace the data in the
MPR file
> all the way through the Stat-Analysis output of FBAR, OBAR, ME and
MAE. The
> only remaining question, I have is about the vector partial sums:
UVFOBAR,
> UVFFBAR and UVOOBAR. I looked in the User's Guide where the VL1L2
line
> type is
> described and I see the formulas for how these are calculated, but I
don't
> see
> where these partial sums are used for generating the values in the
> Stat-Analysis output. How are these partial sums used?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, September 26, 2014 1:05 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John,
>
> I read through the document you sent about the output of wind
direction
> errors
> and will respond to your questions below...
>
> I see that you've run a single VL1L2 line (computed over 103 matched
pairs)
> through STAT-Analysis to compute wind direction errors. In the
output you
> see
> the same values for FBAR (94.25) and OBAR (89.09) repeated in both
lines
> (ROW_MEAN_WDIR and AGGR_WDIR) of the output. You see that because
this
> job is
> not actually aggregating multiple lines together. It's just being
run
> over a
> single VL1L2 line. If your job was run over multiple
> VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
> AGGR_WDIR
> lines would differ.
>
> Your first question is - how is the MAE value calculated? The FBAR
and
> OBAR
> columns contain the forecast and observed wind directions, in
degrees. The
> error is the directed difference between those angles (f - o). The
mean
> error
> (ME) is the average of those angular differences. The range of this
column
> is -180 to +180. The MAE is the average of the *absolute value* of
those
> angular differences. So the range of the MAE column is 0 to +180.
>
> You're wondering where in the MET code this logic is implemented.
In the
> file
> "src/basic/vx_math/angles.cc" the function named
"convert_u_v_to_wdir"
> handles the conversion of the U and V components to wind direction.
Also,
> the
> file "src/tools/core/stat_analysis/stat_analysis_job.cc" contains
the
> function
> "write_job_aggr_wind" which creates the output for STAT-Analysis.
>
> Your next question is why you the numbers are the same in
ROW_MEAN_WDIR And
> AGGR_WDIR - again, that's because you're running over a single VL1L2
line.
> If you run on multiple, they will likely differ. It's difficult to
> describe
> the difference with words, but I'll give it a shot. Suppose you're
running
> this job over 10 different VL1L2 lines that represent output for 10
> different
> days over the same region. And let's suppose 5 days had
> 50 stations reporting and 5 days had 100 stations reporting...
>
> - For ROW_MEAN_WDIR, we do the following...
> - For each VL1L2 line (meaning, for each day), compute the average
> forecast
> and observation direction and compute the angular difference.
> - Compute the average of those 10 angular differences and write it
in
> the ME
> column. Also, put the average of the absolute value of those
differences
> in
> the MAE column.
> - This is the daily average of the wind direction errors. Even
though
> some
> of the days were computed over 50 matched pairs and others were
computed
> over
> 100.
>
> - For the AGGR_WDIR, aggregate together those 10 different VL1L2
lines
> into a
> single VL1L2 line. When we aggregate VL1L2 lines, we do a weighted
> average,
> weighted by the "TOTAL" columns. Relatively speaking, those 5 days
with a
> 100
> matched pairs will count twice as much as the other 5 days with only
50
> matched pairs.
> - From that single aggregated VL1L2 line, compute forecast and
> observation
> directions. The ME is the angular difference between them.
> - We could write out the MAE which would just be the absolute
value of
> the
> the ME, but we're just writing NA.
> - This is an aggregated wind direction difference.
>
> Both are reasonable ways of processing the data - but the
computations are
> slightly different.
>
> The real short-coming here is that we're only computing wind
direction
> errors
> over points that have already been aggregated through space. In
> METv4.1 STAT-Analysis, the only supported job type is "-job
> aggregate_stat -line_type VL1L2 -out_line_type MPR".
>
> To address this short-coming in met-5.0, we added a similar job that
> operates
> over matched pair values instead. Here's a selection from the
> met-5.0 release notes:
>
> "Enhanced STAT-Analysis by adding support for an aggregate_stat job
type
> conversion of MPR to WDIR that reads UGRD and VGRD matched pair
lines and
> computes aggregated wind direction errors. Added
-out_fcst_wind_thresh
> and -out_obs_wind_thresh options to filter the U/V pairs by wind
speed
> prior
> to computing wind direction errors."
>
> For example, the following job would compute wind direction errors
for each
> station individually...
> -job aggregate_wdir -line_type MPR -out_line_type WDIR -by
> OBS_SID -out_fcst_wind_thresh \>+2.0
>
> That'll compute wind direction errors for each unique station name
in the
> OBS_SID column of the MPR line, but only when the forecast wind
speed
> exceeds
> 2 m/s.
>
> This should make it easier to look at wind direction errors
> station-by-station, rather than have to aggregate over a much larger
area.
>
> Please let me know if this answers your questions. If not, I'll
probably
> refer you to a scientist here to address the interpretation of the
output.
> I can help you running the tools and describing the options, but I
think a
> scientist would do a better job helping you interpret the results.
>
> Thanks,
> John
>
> On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for the update. I will be out Friday and Monday, returning
on
> > Tuesday.
> > Have a great weekend!
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, September 25, 2014 8:54 AM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> > verification (UNCLASSIFIED)
> >
> > John
> >
> > I'm out of the office today, but will take a close look at this on
> Friday.
> >
> > Thanks
> > John
> >
> > On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> <javascript:;>
> > > Queue: met_help
> > > Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > I need to know more specifically how wind direction errors are
> > calculated.
> > > I
> > > ran a case using METV4.1 using a forecast and observations and I
> > > followed the guidance to assure that the Point-Stat VL1L2 lines
were
> > > generated. When I ran Stat-Analysis I specified the job as
> > > aggregate_stat, the -line_type as
> > > VL1L2 and the -out_line_type as WDIR. The attached document
provides
> > > samples of the output and my attempts to calculate the errors by
> > > hand so I could try to understand how MET calculates them. I
also
> > > highlighted the questions I have in the document. I would
appreciate
> > > having answers to my questions if possible. Thanks.
> > >
> > > R/
> > > John
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
--
Julie Prestopnik
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Oct 01 10:21:46 2014
Classification: UNCLASSIFIED
Caveats: NONE
Thanks for your response. No problem with waiting until he gets back.
R/
John
-----Original Message-----
From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, October 01, 2014 10:13 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
verification (UNCLASSIFIED)
Hi John. John is currently out of the office, but will hopefully be
able to
respond to your questions later this week. I do not have enough
knowledge to
answer your questions.
Thanks,
Julie
On Wed, Oct 1, 2014 at 10:09 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for your detailed explanations for my questions. I think it
> confirmed how the Stat-Analysis output is generated for me. Now it
> raises some questions about how the vector partial sums in the
> Point-Stat VL1L2 line type are generated.
>
> I was able to trace how the partial sums UFBAR, VFBAR and UOBAR,
VOBAR
> were generated from the UGRD, VGRD matched pair data in the MPR file
> in a spreadsheet. I simply took the mean of all 103 values of the
UGRD
> and VGRD for the Z10 level and the results were identical. For
> example: for UFBAR, I took the average of all 103 forecast values of
> UGRD and the result matched the value for UFBAR in the VL1L2 output.
> From your answer below, I gather that you use the code
> "convert_u_v_to_wdir" to convert UFBAR, VFBAR to generate the
forecast
> wind direction (FBAR in the Stat-Analysis output) and the same code
is
> used to convert UOBAR, VOBAR to generate the observed wind direction
> (OBAR in the Stat-analysis output). Correct so far?
>
> So, based on your answer and the above I can trace the data in the
MPR
> file all the way through the Stat-Analysis output of FBAR, OBAR, ME
> and MAE. The only remaining question, I have is about the vector
> partial sums: UVFOBAR, UVFFBAR and UVOOBAR. I looked in the User's
> Guide where the VL1L2 line type is described and I see the formulas
> for how these are calculated, but I don't see where these partial
sums
> are used for generating the values in the Stat-Analysis output. How
> are these partial sums used?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, September 26, 2014 1:05 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John,
>
> I read through the document you sent about the output of wind
> direction errors and will respond to your questions below...
>
> I see that you've run a single VL1L2 line (computed over 103 matched
> pairs) through STAT-Analysis to compute wind direction errors. In
the
> output you see the same values for FBAR (94.25) and OBAR (89.09)
> repeated in both lines (ROW_MEAN_WDIR and AGGR_WDIR) of the output.
> You see that because this job is not actually aggregating multiple
> lines together. It's just being run over a single VL1L2 line. If
> your job was run over multiple
> VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
> AGGR_WDIR lines would differ.
>
> Your first question is - how is the MAE value calculated? The FBAR
> and OBAR columns contain the forecast and observed wind directions,
in
> degrees. The error is the directed difference between those angles
(f
> - o). The mean error
> (ME) is the average of those angular differences. The range of this
> column is -180 to +180. The MAE is the average of the *absolute
> value* of those angular differences. So the range of the MAE column
is 0 to
> +180.
>
> You're wondering where in the MET code this logic is implemented.
In
> the file "src/basic/vx_math/angles.cc" the function named
> "convert_u_v_to_wdir"
> handles the conversion of the U and V components to wind direction.
> Also, the file "src/tools/core/stat_analysis/stat_analysis_job.cc"
> contains the function "write_job_aggr_wind" which creates the output
> for STAT-Analysis.
>
> Your next question is why you the numbers are the same in
> ROW_MEAN_WDIR And AGGR_WDIR - again, that's because you're running
over a
> single VL1L2 line.
> If you run on multiple, they will likely differ. It's difficult to
> describe the difference with words, but I'll give it a shot.
Suppose
> you're running this job over 10 different VL1L2 lines that represent
> output for 10 different days over the same region. And let's
suppose
> 5 days had
> 50 stations reporting and 5 days had 100 stations reporting...
>
> - For ROW_MEAN_WDIR, we do the following...
> - For each VL1L2 line (meaning, for each day), compute the average
> forecast and observation direction and compute the angular
difference.
> - Compute the average of those 10 angular differences and write it
> in the ME column. Also, put the average of the absolute value of
> those differences in the MAE column.
> - This is the daily average of the wind direction errors. Even
> though some of the days were computed over 50 matched pairs and
others
> were computed over 100.
>
> - For the AGGR_WDIR, aggregate together those 10 different VL1L2
lines
> into a single VL1L2 line. When we aggregate VL1L2 lines, we do a
> weighted average, weighted by the "TOTAL" columns. Relatively
> speaking, those 5 days with a
> 100
> matched pairs will count twice as much as the other 5 days with only
> 50 matched pairs.
> - From that single aggregated VL1L2 line, compute forecast and
> observation directions. The ME is the angular difference between
> them.
> - We could write out the MAE which would just be the absolute
value
> of the the ME, but we're just writing NA.
> - This is an aggregated wind direction difference.
>
> Both are reasonable ways of processing the data - but the
computations
> are slightly different.
>
> The real short-coming here is that we're only computing wind
direction
> errors over points that have already been aggregated through space.
> In
> METv4.1 STAT-Analysis, the only supported job type is "-job
> aggregate_stat -line_type VL1L2 -out_line_type MPR".
>
> To address this short-coming in met-5.0, we added a similar job that
> operates over matched pair values instead. Here's a selection from
> the
> met-5.0 release notes:
>
> "Enhanced STAT-Analysis by adding support for an aggregate_stat job
> type conversion of MPR to WDIR that reads UGRD and VGRD matched pair
> lines and computes aggregated wind direction errors. Added
> -out_fcst_wind_thresh and -out_obs_wind_thresh options to filter the
> U/V pairs by wind speed prior to computing wind direction errors."
>
> For example, the following job would compute wind direction errors
for
> each station individually...
> -job aggregate_wdir -line_type MPR -out_line_type WDIR -by OBS_SID
> -out_fcst_wind_thresh \>+2.0
>
> That'll compute wind direction errors for each unique station name
in
> the OBS_SID column of the MPR line, but only when the forecast wind
> speed exceeds
> 2 m/s.
>
> This should make it easier to look at wind direction errors
> station-by-station, rather than have to aggregate over a much larger
area.
>
> Please let me know if this answers your questions. If not, I'll
> probably refer you to a scientist here to address the interpretation
of the
> output.
> I can help you running the tools and describing the options, but I
> think a scientist would do a better job helping you interpret the
results.
>
> Thanks,
> John
>
> On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for the update. I will be out Friday and Monday, returning
on
> > Tuesday.
> > Have a great weekend!
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, September 25, 2014 8:54 AM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> > verification (UNCLASSIFIED)
> >
> > John
> >
> > I'm out of the office today, but will take a close look at this on
> Friday.
> >
> > Thanks
> > John
> >
> > On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> <javascript:;>
> > > Queue: met_help
> > > Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > I need to know more specifically how wind direction errors are
> > calculated.
> > > I
> > > ran a case using METV4.1 using a forecast and observations and I
> > > followed the guidance to assure that the Point-Stat VL1L2 lines
> > > were generated. When I ran Stat-Analysis I specified the job as
> > > aggregate_stat, the -line_type as
> > > VL1L2 and the -out_line_type as WDIR. The attached document
> > > provides samples of the output and my attempts to calculate the
> > > errors by hand so I could try to understand how MET calculates
> > > them. I also highlighted the questions I have in the document. I
> > > would appreciate having answers to my questions if possible.
Thanks.
> > >
> > > R/
> > > John
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
--
Julie Prestopnik
National Center for Atmospheric Research Research Applications
Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Oct 02 13:45:24 2014
John,
Here's the contents of the VL1L2 line: UFBAR, VFBAR, UOBAR, VOBAR,
UVFOBAR,
UVFFBAR, UVOOBAR
Yes, the forecast average U and V (UFBAR and VFBAR) are used in
computing
the forecast wind speed and direction. While UOBAR and VOBAR are used
in
computing the observed wind speed and direction. And you're wondering
how
the UVFOBAR, UVFFBAR, and UVOOBAR data is used. The answer is that
they're
not currently being used in the computation any statistics!
The MET software was derived from the NCEP verification tools which
include
the VL1L2 output line. I spoke with statisticians here, and we
believe
that NCEP uses the data in those 3 columns to derive a combined U and
V
vector-based RMSE score. FVS is the name of the plotting tools
associated
with that package, and we'll check it's documentation to see if/how
NCEP is
using those values to derive statistics.
If so, we could potentially add that capability to MET, deriving
vector-based scores. And I'll forward whatever equations we find.
Thanks for the question, and we'll let you know what we turn up.
Thanks,
John
On Wed, Oct 1, 2014 at 10:13 AM, Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Hi John. John is currently out of the office, but will hopefully be
able
> to respond to your questions later this week. I do not have enough
> knowledge to answer your questions.
>
> Thanks,
> Julie
>
> On Wed, Oct 1, 2014 at 10:09 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for your detailed explanations for my questions. I think it
> > confirmed
> > how the Stat-Analysis output is generated for me. Now it raises
some
> > questions
> > about how the vector partial sums in the Point-Stat VL1L2 line
type are
> > generated.
> >
> > I was able to trace how the partial sums UFBAR, VFBAR and UOBAR,
VOBAR
> were
> > generated from the UGRD, VGRD matched pair data in the MPR file in
a
> > spreadsheet. I simply took the mean of all 103 values of the UGRD
and
> VGRD
> > for
> > the Z10 level and the results were identical. For example: for
UFBAR, I
> > took
> > the average of all 103 forecast values of UGRD and the result
matched the
> > value for UFBAR in the VL1L2 output. From your answer below, I
gather
> that
> > you
> > use the code "convert_u_v_to_wdir" to convert UFBAR, VFBAR to
generate
> the
> > forecast wind direction (FBAR in the Stat-Analysis output) and the
same
> > code
> > is used to convert UOBAR, VOBAR to generate the observed wind
direction
> > (OBAR
> > in the Stat-analysis output). Correct so far?
> >
> > So, based on your answer and the above I can trace the data in the
MPR
> file
> > all the way through the Stat-Analysis output of FBAR, OBAR, ME and
MAE.
> The
> > only remaining question, I have is about the vector partial sums:
> UVFOBAR,
> > UVFFBAR and UVOOBAR. I looked in the User's Guide where the VL1L2
line
> > type is
> > described and I see the formulas for how these are calculated, but
I
> don't
> > see
> > where these partial sums are used for generating the values in the
> > Stat-Analysis output. How are these partial sums used?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, September 26, 2014 1:05 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> > verification (UNCLASSIFIED)
> >
> > John,
> >
> > I read through the document you sent about the output of wind
direction
> > errors
> > and will respond to your questions below...
> >
> > I see that you've run a single VL1L2 line (computed over 103
matched
> pairs)
> > through STAT-Analysis to compute wind direction errors. In the
output
> you
> > see
> > the same values for FBAR (94.25) and OBAR (89.09) repeated in both
lines
> > (ROW_MEAN_WDIR and AGGR_WDIR) of the output. You see that because
this
> > job is
> > not actually aggregating multiple lines together. It's just being
run
> > over a
> > single VL1L2 line. If your job was run over multiple
> > VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
> > AGGR_WDIR
> > lines would differ.
> >
> > Your first question is - how is the MAE value calculated? The
FBAR and
> > OBAR
> > columns contain the forecast and observed wind directions, in
degrees.
> The
> > error is the directed difference between those angles (f - o).
The mean
> > error
> > (ME) is the average of those angular differences. The range of
this
> column
> > is -180 to +180. The MAE is the average of the *absolute value*
of those
> > angular differences. So the range of the MAE column is 0 to +180.
> >
> > You're wondering where in the MET code this logic is implemented.
In the
> > file
> > "src/basic/vx_math/angles.cc" the function named
"convert_u_v_to_wdir"
> > handles the conversion of the U and V components to wind
direction.
> Also,
> > the
> > file "src/tools/core/stat_analysis/stat_analysis_job.cc" contains
the
> > function
> > "write_job_aggr_wind" which creates the output for STAT-Analysis.
> >
> > Your next question is why you the numbers are the same in
ROW_MEAN_WDIR
> And
> > AGGR_WDIR - again, that's because you're running over a single
VL1L2
> line.
> > If you run on multiple, they will likely differ. It's difficult
to
> > describe
> > the difference with words, but I'll give it a shot. Suppose
you're
> running
> > this job over 10 different VL1L2 lines that represent output for
10
> > different
> > days over the same region. And let's suppose 5 days had
> > 50 stations reporting and 5 days had 100 stations reporting...
> >
> > - For ROW_MEAN_WDIR, we do the following...
> > - For each VL1L2 line (meaning, for each day), compute the
average
> > forecast
> > and observation direction and compute the angular difference.
> > - Compute the average of those 10 angular differences and write
it in
> > the ME
> > column. Also, put the average of the absolute value of those
differences
> > in
> > the MAE column.
> > - This is the daily average of the wind direction errors. Even
though
> > some
> > of the days were computed over 50 matched pairs and others were
computed
> > over
> > 100.
> >
> > - For the AGGR_WDIR, aggregate together those 10 different VL1L2
lines
> > into a
> > single VL1L2 line. When we aggregate VL1L2 lines, we do a
weighted
> > average,
> > weighted by the "TOTAL" columns. Relatively speaking, those 5
days with
> a
> > 100
> > matched pairs will count twice as much as the other 5 days with
only 50
> > matched pairs.
> > - From that single aggregated VL1L2 line, compute forecast and
> > observation
> > directions. The ME is the angular difference between them.
> > - We could write out the MAE which would just be the absolute
value of
> > the
> > the ME, but we're just writing NA.
> > - This is an aggregated wind direction difference.
> >
> > Both are reasonable ways of processing the data - but the
computations
> are
> > slightly different.
> >
> > The real short-coming here is that we're only computing wind
direction
> > errors
> > over points that have already been aggregated through space. In
> > METv4.1 STAT-Analysis, the only supported job type is "-job
> > aggregate_stat -line_type VL1L2 -out_line_type MPR".
> >
> > To address this short-coming in met-5.0, we added a similar job
that
> > operates
> > over matched pair values instead. Here's a selection from the
> > met-5.0 release notes:
> >
> > "Enhanced STAT-Analysis by adding support for an aggregate_stat
job type
> > conversion of MPR to WDIR that reads UGRD and VGRD matched pair
lines and
> > computes aggregated wind direction errors. Added
-out_fcst_wind_thresh
> > and -out_obs_wind_thresh options to filter the U/V pairs by wind
speed
> > prior
> > to computing wind direction errors."
> >
> > For example, the following job would compute wind direction errors
for
> each
> > station individually...
> > -job aggregate_wdir -line_type MPR -out_line_type WDIR -by
> > OBS_SID -out_fcst_wind_thresh \>+2.0
> >
> > That'll compute wind direction errors for each unique station name
in the
> > OBS_SID column of the MPR line, but only when the forecast wind
speed
> > exceeds
> > 2 m/s.
> >
> > This should make it easier to look at wind direction errors
> > station-by-station, rather than have to aggregate over a much
larger
> area.
> >
> > Please let me know if this answers your questions. If not, I'll
probably
> > refer you to a scientist here to address the interpretation of the
> output.
> > I can help you running the tools and describing the options, but I
think
> a
> > scientist would do a better job helping you interpret the results.
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > John -
> > >
> > > Thanks for the update. I will be out Friday and Monday,
returning on
> > > Tuesday.
> > > Have a great weekend!
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, September 25, 2014 8:54 AM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind
direction
> > > verification (UNCLASSIFIED)
> > >
> > > John
> > >
> > > I'm out of the office today, but will take a close look at this
on
> > Friday.
> > >
> > > Thanks
> > > John
> > >
> > > On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > <javascript:;>
> > > > Queue: met_help
> > > > Subject: Stat-Analysis wind direction verification
> (UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > I need to know more specifically how wind direction errors are
> > > calculated.
> > > > I
> > > > ran a case using METV4.1 using a forecast and observations and
I
> > > > followed the guidance to assure that the Point-Stat VL1L2
lines were
> > > > generated. When I ran Stat-Analysis I specified the job as
> > > > aggregate_stat, the -line_type as
> > > > VL1L2 and the -out_line_type as WDIR. The attached document
provides
> > > > samples of the output and my attempts to calculate the errors
by
> > > > hand so I could try to understand how MET calculates them. I
also
> > > > highlighted the questions I have in the document. I would
appreciate
> > > > having answers to my questions if possible. Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Oct 02 13:49:37 2014
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for clarifying that.
I had a couple of questions from Dr. Brian Reen also on our team:
So, from this I have two questions for possible inclusion in the email
you
noted you were planning on sending to John Halley Gotway:
1) Anytime that you subtract observed wind direction from model wind
direction
(e.g., for AGGR_WIND, ROW_MEAN_WDIR, and the new capability to
calculate wind
direction error for individual obs) and the result is less than -180
or
greater than +180 do you add 360 in the former case and subtract 360
in the
latter case?
2) With the new capability to calculate wind direction errors at
individual
stations, is it now possible to calculate mean wind direction errors
based on
these individual stations wind direction errors? In other words, if at
12 UTC
we have station A with a wind direction error of +50 and station B
with a wind
direction error of -70, will MET tell us that the mean wind direction
error
is -20 degrees and the mean absolute wind direction error is 60
degrees?
Brian
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, October 02, 2014 1:45 PM
To: Raby, John W CIV USARMY ARL (US)
Cc: bgb at ucar.edu
Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
verification (UNCLASSIFIED)
John,
Here's the contents of the VL1L2 line: UFBAR, VFBAR, UOBAR, VOBAR,
UVFOBAR,
UVFFBAR, UVOOBAR
Yes, the forecast average U and V (UFBAR and VFBAR) are used in
computing the
forecast wind speed and direction. While UOBAR and VOBAR are used in
computing the observed wind speed and direction. And you're wondering
how the
UVFOBAR, UVFFBAR, and UVOOBAR data is used. The answer is that
they're not
currently being used in the computation any statistics!
The MET software was derived from the NCEP verification tools which
include
the VL1L2 output line. I spoke with statisticians here, and we
believe that
NCEP uses the data in those 3 columns to derive a combined U and V
vector-based RMSE score. FVS is the name of the plotting tools
associated
with that package, and we'll check it's documentation to see if/how
NCEP is
using those values to derive statistics.
If so, we could potentially add that capability to MET, deriving
vector-based
scores. And I'll forward whatever equations we find.
Thanks for the question, and we'll let you know what we turn up.
Thanks,
John
On Wed, Oct 1, 2014 at 10:13 AM, Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Hi John. John is currently out of the office, but will hopefully be
> able to respond to your questions later this week. I do not have
> enough knowledge to answer your questions.
>
> Thanks,
> Julie
>
> On Wed, Oct 1, 2014 at 10:09 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for your detailed explanations for my questions. I think it
> > confirmed how the Stat-Analysis output is generated for me. Now it
> > raises some questions about how the vector partial sums in the
> > Point-Stat VL1L2 line type are generated.
> >
> > I was able to trace how the partial sums UFBAR, VFBAR and UOBAR,
> > VOBAR
> were
> > generated from the UGRD, VGRD matched pair data in the MPR file in
a
> > spreadsheet. I simply took the mean of all 103 values of the UGRD
> > and
> VGRD
> > for
> > the Z10 level and the results were identical. For example: for
> > UFBAR, I took the average of all 103 forecast values of UGRD and
the
> > result matched the value for UFBAR in the VL1L2 output. From your
> > answer below, I gather
> that
> > you
> > use the code "convert_u_v_to_wdir" to convert UFBAR, VFBAR to
> > generate
> the
> > forecast wind direction (FBAR in the Stat-Analysis output) and the
> > same code is used to convert UOBAR, VOBAR to generate the observed
> > wind direction (OBAR in the Stat-analysis output). Correct so far?
> >
> > So, based on your answer and the above I can trace the data in the
> > MPR
> file
> > all the way through the Stat-Analysis output of FBAR, OBAR, ME and
MAE.
> The
> > only remaining question, I have is about the vector partial sums:
> UVFOBAR,
> > UVFFBAR and UVOOBAR. I looked in the User's Guide where the VL1L2
> > line type is described and I see the formulas for how these are
> > calculated, but I
> don't
> > see
> > where these partial sums are used for generating the values in the
> > Stat-Analysis output. How are these partial sums used?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, September 26, 2014 1:05 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> > verification (UNCLASSIFIED)
> >
> > John,
> >
> > I read through the document you sent about the output of wind
> > direction errors and will respond to your questions below...
> >
> > I see that you've run a single VL1L2 line (computed over 103
matched
> pairs)
> > through STAT-Analysis to compute wind direction errors. In the
> > output
> you
> > see
> > the same values for FBAR (94.25) and OBAR (89.09) repeated in both
> > lines (ROW_MEAN_WDIR and AGGR_WDIR) of the output. You see that
> > because this job is not actually aggregating multiple lines
> > together. It's just being run over a single VL1L2 line. If your
> > job was run over multiple
> > VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
> > AGGR_WDIR lines would differ.
> >
> > Your first question is - how is the MAE value calculated? The
FBAR
> > and OBAR columns contain the forecast and observed wind
directions,
> > in degrees.
> The
> > error is the directed difference between those angles (f - o).
The
> > mean error
> > (ME) is the average of those angular differences. The range of
this
> column
> > is -180 to +180. The MAE is the average of the *absolute value*
of
> > those angular differences. So the range of the MAE column is 0 to
+180.
> >
> > You're wondering where in the MET code this logic is implemented.
> > In the file "src/basic/vx_math/angles.cc" the function named
> > "convert_u_v_to_wdir"
> > handles the conversion of the U and V components to wind
direction.
> Also,
> > the
> > file "src/tools/core/stat_analysis/stat_analysis_job.cc" contains
> > the function "write_job_aggr_wind" which creates the output for
> > STAT-Analysis.
> >
> > Your next question is why you the numbers are the same in
> > ROW_MEAN_WDIR
> And
> > AGGR_WDIR - again, that's because you're running over a single
VL1L2
> line.
> > If you run on multiple, they will likely differ. It's difficult
to
> > describe the difference with words, but I'll give it a shot.
> > Suppose you're
> running
> > this job over 10 different VL1L2 lines that represent output for
10
> > different days over the same region. And let's suppose 5 days had
> > 50 stations reporting and 5 days had 100 stations reporting...
> >
> > - For ROW_MEAN_WDIR, we do the following...
> > - For each VL1L2 line (meaning, for each day), compute the
average
> > forecast and observation direction and compute the angular
> > difference.
> > - Compute the average of those 10 angular differences and write
it
> > in the ME column. Also, put the average of the absolute value of
> > those differences in the MAE column.
> > - This is the daily average of the wind direction errors. Even
> > though some of the days were computed over 50 matched pairs and
> > others were computed over 100.
> >
> > - For the AGGR_WDIR, aggregate together those 10 different VL1L2
> > lines into a single VL1L2 line. When we aggregate VL1L2 lines, we
> > do a weighted average, weighted by the "TOTAL" columns.
Relatively
> > speaking, those 5 days with
> a
> > 100
> > matched pairs will count twice as much as the other 5 days with
only
> > 50 matched pairs.
> > - From that single aggregated VL1L2 line, compute forecast and
> > observation directions. The ME is the angular difference between
> > them.
> > - We could write out the MAE which would just be the absolute
> > value of the the ME, but we're just writing NA.
> > - This is an aggregated wind direction difference.
> >
> > Both are reasonable ways of processing the data - but the
> > computations
> are
> > slightly different.
> >
> > The real short-coming here is that we're only computing wind
> > direction errors over points that have already been aggregated
> > through space. In
> > METv4.1 STAT-Analysis, the only supported job type is "-job
> > aggregate_stat -line_type VL1L2 -out_line_type MPR".
> >
> > To address this short-coming in met-5.0, we added a similar job
that
> > operates over matched pair values instead. Here's a selection
from
> > the
> > met-5.0 release notes:
> >
> > "Enhanced STAT-Analysis by adding support for an aggregate_stat
job
> > type conversion of MPR to WDIR that reads UGRD and VGRD matched
pair
> > lines and computes aggregated wind direction errors. Added
> > -out_fcst_wind_thresh and -out_obs_wind_thresh options to filter
the
> > U/V pairs by wind speed prior to computing wind direction errors."
> >
> > For example, the following job would compute wind direction errors
> > for
> each
> > station individually...
> > -job aggregate_wdir -line_type MPR -out_line_type WDIR -by
OBS_SID
> > -out_fcst_wind_thresh \>+2.0
> >
> > That'll compute wind direction errors for each unique station name
> > in the OBS_SID column of the MPR line, but only when the forecast
> > wind speed exceeds
> > 2 m/s.
> >
> > This should make it easier to look at wind direction errors
> > station-by-station, rather than have to aggregate over a much
larger
> area.
> >
> > Please let me know if this answers your questions. If not, I'll
> > probably refer you to a scientist here to address the
interpretation
> > of the
> output.
> > I can help you running the tools and describing the options, but I
> > think
> a
> > scientist would do a better job helping you interpret the results.
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > John -
> > >
> > > Thanks for the update. I will be out Friday and Monday,
returning
> > > on Tuesday.
> > > Have a great weekend!
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, September 25, 2014 8:54 AM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind
direction
> > > verification (UNCLASSIFIED)
> > >
> > > John
> > >
> > > I'm out of the office today, but will take a close look at this
on
> > Friday.
> > >
> > > Thanks
> > > John
> > >
> > > On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > <javascript:;>
> > > > Queue: met_help
> > > > Subject: Stat-Analysis wind direction verification
> (UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > I need to know more specifically how wind direction errors are
> > > calculated.
> > > > I
> > > > ran a case using METV4.1 using a forecast and observations and
I
> > > > followed the guidance to assure that the Point-Stat VL1L2
lines
> > > > were generated. When I ran Stat-Analysis I specified the job
as
> > > > aggregate_stat, the -line_type as
> > > > VL1L2 and the -out_line_type as WDIR. The attached document
> > > > provides samples of the output and my attempts to calculate
the
> > > > errors by hand so I could try to understand how MET calculates
> > > > them. I also highlighted the questions I have in the document.
I
> > > > would appreciate having answers to my questions if possible.
Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research Research Applications
> Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Oct 03 10:09:17 2014
John,
The directed difference between two angles is computed in MET by the
function named "angle_difference" in the file
src/basic/vx_math/angles.cc.
It ultimately calls the "atan2d" function (arctangent) which returns
values
between -180 and 180. So angle differences reported by stat-analysis
should always be in that range.
And I believe the answer to the second question is yes. To test this
out I
ran the following 2 jobs on the output of the MET test scripts:
bin/stat_analysis -lookin out/point_stat \
-job aggregate_stat -line_type MPR -out_line_type WDIR \
-vx_mask LMV -interp_mthd UW_MEAN \
-column_str OBS_SID KUNO,KSDF -by OBS_SID
This says pick compute wind direction errors for the 2 stations
listed.
And here's the output:
COL_NAME: OBS_SID TOTAL FBAR OBAR ME
MAE
ROW_MEAN_WDIR: KSDF 1 183.92113 188.74616 -4.82503 4.82503
AGGR_WDIR: KSDF 1 183.92113 188.74616 -4.82503
NA
ROW_MEAN_WDIR: KUNO 1 165.37725 159.94391 5.43334 5.43334
AGGR_WDIR: KUNO 1 165.37725 159.94391 5.43334 NA
KSDF has a ME of -4.825 degrees and KUNO has a ME of 5.4333 degrees.
Then I ran this job, removing the "-by OBS_SID" option:
bin/stat_analysis -lookin out/point_stat \
-job aggregate_stat -line_type MPR -out_line_type WDIR \
-vx_mask LMV -interp_mthd UW_MEAN \
-column_str OBS_SID KUNO,KSDF
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 2 174.64919 174.34503 0.30416 5.12919
AGGR_WDIR: 2 170.76076 167.94922 2.81154 NA
The ME in the ROW_MEAN_WDIR is now listed as 0.304 which is the
average of
the two. So if you compute wind direction errors station by station,
the
ROW_MEAN_WDIR line will contain the average of the errors across the
stations.
Thanks,
John
On Thu, Oct 2, 2014 at 1:49 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for clarifying that.
>
> I had a couple of questions from Dr. Brian Reen also on our team:
>
> So, from this I have two questions for possible inclusion in the
email you
> noted you were planning on sending to John Halley Gotway:
>
> 1) Anytime that you subtract observed wind direction from model wind
> direction
> (e.g., for AGGR_WIND, ROW_MEAN_WDIR, and the new capability to
calculate
> wind
> direction error for individual obs) and the result is less than -180
or
> greater than +180 do you add 360 in the former case and subtract 360
in the
> latter case?
>
> 2) With the new capability to calculate wind direction errors at
individual
> stations, is it now possible to calculate mean wind direction errors
based
> on
> these individual stations wind direction errors? In other words, if
at 12
> UTC
> we have station A with a wind direction error of +50 and station B
with a
> wind
> direction error of -70, will MET tell us that the mean wind
direction error
> is -20 degrees and the mean absolute wind direction error is 60
degrees?
>
> Brian
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, October 02, 2014 1:45 PM
> To: Raby, John W CIV USARMY ARL (US)
> Cc: bgb at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John,
>
> Here's the contents of the VL1L2 line: UFBAR, VFBAR, UOBAR, VOBAR,
UVFOBAR,
> UVFFBAR, UVOOBAR
>
> Yes, the forecast average U and V (UFBAR and VFBAR) are used in
computing
> the
> forecast wind speed and direction. While UOBAR and VOBAR are used
in
> computing the observed wind speed and direction. And you're
wondering how
> the
> UVFOBAR, UVFFBAR, and UVOOBAR data is used. The answer is that
they're not
> currently being used in the computation any statistics!
>
> The MET software was derived from the NCEP verification tools which
include
> the VL1L2 output line. I spoke with statisticians here, and we
believe
> that
> NCEP uses the data in those 3 columns to derive a combined U and V
> vector-based RMSE score. FVS is the name of the plotting tools
associated
> with that package, and we'll check it's documentation to see if/how
NCEP is
> using those values to derive statistics.
>
> If so, we could potentially add that capability to MET, deriving
> vector-based
> scores. And I'll forward whatever equations we find.
>
> Thanks for the question, and we'll let you know what we turn up.
>
> Thanks,
> John
>
> On Wed, Oct 1, 2014 at 10:13 AM, Julie Prestopnik via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Hi John. John is currently out of the office, but will hopefully
be
> > able to respond to your questions later this week. I do not have
> > enough knowledge to answer your questions.
> >
> > Thanks,
> > Julie
> >
> > On Wed, Oct 1, 2014 at 10:09 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > John -
> > >
> > > Thanks for your detailed explanations for my questions. I think
it
> > > confirmed how the Stat-Analysis output is generated for me. Now
it
> > > raises some questions about how the vector partial sums in the
> > > Point-Stat VL1L2 line type are generated.
> > >
> > > I was able to trace how the partial sums UFBAR, VFBAR and UOBAR,
> > > VOBAR
> > were
> > > generated from the UGRD, VGRD matched pair data in the MPR file
in a
> > > spreadsheet. I simply took the mean of all 103 values of the
UGRD
> > > and
> > VGRD
> > > for
> > > the Z10 level and the results were identical. For example: for
> > > UFBAR, I took the average of all 103 forecast values of UGRD and
the
> > > result matched the value for UFBAR in the VL1L2 output. From
your
> > > answer below, I gather
> > that
> > > you
> > > use the code "convert_u_v_to_wdir" to convert UFBAR, VFBAR to
> > > generate
> > the
> > > forecast wind direction (FBAR in the Stat-Analysis output) and
the
> > > same code is used to convert UOBAR, VOBAR to generate the
observed
> > > wind direction (OBAR in the Stat-analysis output). Correct so
far?
> > >
> > > So, based on your answer and the above I can trace the data in
the
> > > MPR
> > file
> > > all the way through the Stat-Analysis output of FBAR, OBAR, ME
and MAE.
> > The
> > > only remaining question, I have is about the vector partial
sums:
> > UVFOBAR,
> > > UVFFBAR and UVOOBAR. I looked in the User's Guide where the
VL1L2
> > > line type is described and I see the formulas for how these are
> > > calculated, but I
> > don't
> > > see
> > > where these partial sums are used for generating the values in
the
> > > Stat-Analysis output. How are these partial sums used?
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, September 26, 2014 1:05 PM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind
direction
> > > verification (UNCLASSIFIED)
> > >
> > > John,
> > >
> > > I read through the document you sent about the output of wind
> > > direction errors and will respond to your questions below...
> > >
> > > I see that you've run a single VL1L2 line (computed over 103
matched
> > pairs)
> > > through STAT-Analysis to compute wind direction errors. In the
> > > output
> > you
> > > see
> > > the same values for FBAR (94.25) and OBAR (89.09) repeated in
both
> > > lines (ROW_MEAN_WDIR and AGGR_WDIR) of the output. You see that
> > > because this job is not actually aggregating multiple lines
> > > together. It's just being run over a single VL1L2 line. If
your
> > > job was run over multiple
> > > VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR
and
> > > AGGR_WDIR lines would differ.
> > >
> > > Your first question is - how is the MAE value calculated? The
FBAR
> > > and OBAR columns contain the forecast and observed wind
directions,
> > > in degrees.
> > The
> > > error is the directed difference between those angles (f - o).
The
> > > mean error
> > > (ME) is the average of those angular differences. The range of
this
> > column
> > > is -180 to +180. The MAE is the average of the *absolute value*
of
> > > those angular differences. So the range of the MAE column is 0
to
> +180.
> > >
> > > You're wondering where in the MET code this logic is
implemented.
> > > In the file "src/basic/vx_math/angles.cc" the function named
> > > "convert_u_v_to_wdir"
> > > handles the conversion of the U and V components to wind
direction.
> > Also,
> > > the
> > > file "src/tools/core/stat_analysis/stat_analysis_job.cc"
contains
> > > the function "write_job_aggr_wind" which creates the output for
> > > STAT-Analysis.
> > >
> > > Your next question is why you the numbers are the same in
> > > ROW_MEAN_WDIR
> > And
> > > AGGR_WDIR - again, that's because you're running over a single
VL1L2
> > line.
> > > If you run on multiple, they will likely differ. It's difficult
to
> > > describe the difference with words, but I'll give it a shot.
> > > Suppose you're
> > running
> > > this job over 10 different VL1L2 lines that represent output for
10
> > > different days over the same region. And let's suppose 5 days
had
> > > 50 stations reporting and 5 days had 100 stations reporting...
> > >
> > > - For ROW_MEAN_WDIR, we do the following...
> > > - For each VL1L2 line (meaning, for each day), compute the
average
> > > forecast and observation direction and compute the angular
> > > difference.
> > > - Compute the average of those 10 angular differences and
write it
> > > in the ME column. Also, put the average of the absolute value
of
> > > those differences in the MAE column.
> > > - This is the daily average of the wind direction errors.
Even
> > > though some of the days were computed over 50 matched pairs and
> > > others were computed over 100.
> > >
> > > - For the AGGR_WDIR, aggregate together those 10 different VL1L2
> > > lines into a single VL1L2 line. When we aggregate VL1L2 lines,
we
> > > do a weighted average, weighted by the "TOTAL" columns.
Relatively
> > > speaking, those 5 days with
> > a
> > > 100
> > > matched pairs will count twice as much as the other 5 days with
only
> > > 50 matched pairs.
> > > - From that single aggregated VL1L2 line, compute forecast and
> > > observation directions. The ME is the angular difference
between
> > > them.
> > > - We could write out the MAE which would just be the absolute
> > > value of the the ME, but we're just writing NA.
> > > - This is an aggregated wind direction difference.
> > >
> > > Both are reasonable ways of processing the data - but the
> > > computations
> > are
> > > slightly different.
> > >
> > > The real short-coming here is that we're only computing wind
> > > direction errors over points that have already been aggregated
> > > through space. In
> > > METv4.1 STAT-Analysis, the only supported job type is "-job
> > > aggregate_stat -line_type VL1L2 -out_line_type MPR".
> > >
> > > To address this short-coming in met-5.0, we added a similar job
that
> > > operates over matched pair values instead. Here's a selection
from
> > > the
> > > met-5.0 release notes:
> > >
> > > "Enhanced STAT-Analysis by adding support for an aggregate_stat
job
> > > type conversion of MPR to WDIR that reads UGRD and VGRD matched
pair
> > > lines and computes aggregated wind direction errors. Added
> > > -out_fcst_wind_thresh and -out_obs_wind_thresh options to filter
the
> > > U/V pairs by wind speed prior to computing wind direction
errors."
> > >
> > > For example, the following job would compute wind direction
errors
> > > for
> > each
> > > station individually...
> > > -job aggregate_wdir -line_type MPR -out_line_type WDIR -by
OBS_SID
> > > -out_fcst_wind_thresh \>+2.0
> > >
> > > That'll compute wind direction errors for each unique station
name
> > > in the OBS_SID column of the MPR line, but only when the
forecast
> > > wind speed exceeds
> > > 2 m/s.
> > >
> > > This should make it easier to look at wind direction errors
> > > station-by-station, rather than have to aggregate over a much
larger
> > area.
> > >
> > > Please let me know if this answers your questions. If not, I'll
> > > probably refer you to a scientist here to address the
interpretation
> > > of the
> > output.
> > > I can help you running the tools and describing the options, but
I
> > > think
> > a
> > > scientist would do a better job helping you interpret the
results.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
>
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > John -
> > > >
> > > > Thanks for the update. I will be out Friday and Monday,
returning
> > > > on Tuesday.
> > > > Have a great weekend!
> > > >
> > > > R/
> > > > John
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Thursday, September 25, 2014 8:54 AM
> > > > To: Raby, John W CIV USARMY ARL (US)
> > > > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind
direction
> > > > verification (UNCLASSIFIED)
> > > >
> > > > John
> > > >
> > > > I'm out of the office today, but will take a close look at
this on
> > > Friday.
> > > >
> > > > Thanks
> > > > John
> > > >
> > > > On Wednesday, September 24, 2014, Raby, John W USA CIV via RT
<
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > <javascript:;>
> > > > > Queue: met_help
> > > > > Subject: Stat-Analysis wind direction verification
> > (UNCLASSIFIED)
> > > > > Owner: Nobody
> > > > > Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > > > > >
> > > > >
> > > > >
> > > > > Classification: UNCLASSIFIED
> > > > > Caveats: NONE
> > > > >
> > > > > I need to know more specifically how wind direction errors
are
> > > > calculated.
> > > > > I
> > > > > ran a case using METV4.1 using a forecast and observations
and I
> > > > > followed the guidance to assure that the Point-Stat VL1L2
lines
> > > > > were generated. When I ran Stat-Analysis I specified the job
as
> > > > > aggregate_stat, the -line_type as
> > > > > VL1L2 and the -out_line_type as WDIR. The attached document
> > > > > provides samples of the output and my attempts to calculate
the
> > > > > errors by hand so I could try to understand how MET
calculates
> > > > > them. I also highlighted the questions I have in the
document. I
> > > > > would appreciate having answers to my questions if possible.
> Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > >
> > > > > Classification: UNCLASSIFIED
> > > > > Caveats: NONE
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Julie Prestopnik
> > National Center for Atmospheric Research Research Applications
> > Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri Oct 03 12:28:21 2014
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for addressing all our questions.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, October 03, 2014 12:24 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: [rt.rap.ucar.edu #69098] Resolved: Stat-Analysis wind
direction
verification (UNCLASSIFIED)
According to our records, your request has been resolved. If you have
any
further questions or concerns, please respond to this message.
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Oct 14 15:46:50 2014
Classification: UNCLASSIFIED
Caveats: NONE
Can I use the new wind direction verification methodology to construct
domain-wide wind direction errors that are based on averages of the
wind
direction errors of individual observations, and thus involve no
averaging of
U/V wind components? It appears that one can specify a list of
stations and
have the wind direction errors of those stations averaged together,
rather
than averaging the U/V components from all of the stations together
and find
wind direction error from that. However, for use in domain-wide
statistics a
few aspects remain unclear to me:
1) Instead of listing individual stations, can I specify that this be
done for
all stations? This would allow the user to find the domain-wide wind
direction error, rather than finding the wind direction difference
between the
domain-wide mean observed and mean modeled wind.
2) If a statistic involves multiple observations from a given station
(i.e.,
multiple hours of obs from a station), will the wind direction errors
from
each observation from that station be averaged together or will the
U/V
components be averaged first?
3) Will the ROW_MEAN_WDIR and the AGGR_WDIR both calculate wind
direction
errors by using an average of the wind direction errors of individual
stations, but just have the AGGR_WDIR weight the statistics based on
the
number of observations in each ROW_MEAN_WDIR line that it is
averaging?
As an example, assume we run a job over 2 hours, one of which has 3
stations
reporting, and one of which has 2 stations reporting. Without
specifying the
station names to MET, can we calculate statistics based on the
following
observations as described below:
Hour 1
Station 1: Observed wind direction = 040, Model wind direction = 100
-> Error
= -060 -> Absolute Error = 060
Station 2: Observed wind direction = 130, Model wind direction = 320
-> Error
= +170 -> Absolute Error = 170
Station 3: Observed wind direction = 320, Model wind direction = 130
-> Error
= -170 -> Absolute Error = 170
Mean Error: -20, Mean Absolute Error: 133.3
Hour 2
Station 1: Observed wind direction = 100, Model wind direction = 040
-> Error
= +060 -> Absolute Error = 060
Station 2: Observed wind direction = 130, Model wind direction = 160
-> Error
= -030 -> Absolute Error = 030
Mean Error: +15, Mean Absolute Error: 45
It we verify hourly, it would seem that ROW_MEAN_WDIR would have:
Mean Error: (-20 + 15) / 2 = -2.5
Mean Absolute Error: (133.3 + 45 ) / 2 = 89.15
since ROW_MEAN_WDIR would simply average the two hours without regard
to the
number of observations in that hour.
It would seem that AGGR_WDIR would be:
Mean Error: ([3*-20]+[2*15])/(3+2) = -6
Mean Absolute Error: ([3*133.3]+[2*45])/(3+2) = 97.98
Since AGGR_WDIR would weight the errors from the two hours based on
the number
of observations each hour's value is based on.
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Oct 15 11:56:07 2014
John,
(1) Yes, you can use all stations in the input dataset. When you run
this
job, STAT-Analysis reads and process all the input MPR lines. If you
haven't told it to use a specific station, or set of stations, it'll
use
all of them that are in the input.
(2) When STAT-Analysis converts MPR lines to wind direction errors,
for
each UGRD line it finds, it needs to find the matching VGRD line. It
matches them up by making sure all the header columns match, including
the
model name, lead time, valid time, vertical level, observation type,
masking region, and interpolation method. It also makes sure the
station
id, latitude, longitude, level, and elevation are the same. It uses
the
matching U/V pairs to derive wind direction errors. So to answer your
question, it computes wind direction errors separately for each U/V
pair
(even for different hours of the same station).
If you do NOT want to include multiple hours in the computations, you
could
either limit the analysis to a single hour (-fcst_lead 12), or
stratify the
results by lead time (-by FCST_LEAD).
(3) Try running this sample job using the output of the MET test
scripts:
bin/stat_analysis \
-job aggregate_stat -line_type MPR -out_line_type WDIR \
-lookin out/point_stat/point_stat_360000L_20070331_120000V.stat \
-interp_mthd UW_MEAN
Here's the output:
JOB_LIST: -job aggregate_stat -interp_mthd UW_MEAN -line_type MPR
-out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 3521 134.83504 113.91382 -5.26449 32.86718
AGGR_WDIR: 4252 130.56134 110.44698 20.11436 NA
Notice that the TOTAL column is smaller in ROW_MEAN_WDIR than it is in
AGGR_WDIR. When we aggregate U/V vectors together for the AGGR_WDIR
lines,
we can include cases where U/V are both 0. So all the input lines are
represented in AGGR_WDIR, even the 0 cases. However, when we define
wind
direction errors point-by-point for the ROW_MEAN_WDIR output, when U/V
are
both 0, the wind direction is undefined. So we can't compute a wind
direction error. Based on this output, I'd say that in 731 cases (=
4252 -
3521) either the forecast U/V or the observed U/V were both 0.
I'm not sure if this answers your question or not. The ROW_MEAN_WDIR
and
AGGR_WDIR are just two different ways of processing the data. Over a
large
sample, the inclusion/exclusion of 0 U/V cases would have a pretty big
impact on the computations.
Thanks,
John
On Tue, Oct 14, 2014 at 3:46 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Can I use the new wind direction verification methodology to
construct
> domain-wide wind direction errors that are based on averages of the
wind
> direction errors of individual observations, and thus involve no
averaging
> of
> U/V wind components? It appears that one can specify a list of
stations
> and
> have the wind direction errors of those stations averaged together,
rather
> than averaging the U/V components from all of the stations together
and
> find
> wind direction error from that. However, for use in domain-wide
> statistics a
> few aspects remain unclear to me:
>
> 1) Instead of listing individual stations, can I specify that this
be done
> for
> all stations? This would allow the user to find the domain-wide
wind
> direction error, rather than finding the wind direction difference
between
> the
> domain-wide mean observed and mean modeled wind.
>
> 2) If a statistic involves multiple observations from a given
station
> (i.e.,
> multiple hours of obs from a station), will the wind direction
errors from
> each observation from that station be averaged together or will the
U/V
> components be averaged first?
>
> 3) Will the ROW_MEAN_WDIR and the AGGR_WDIR both calculate wind
direction
> errors by using an average of the wind direction errors of
individual
> stations, but just have the AGGR_WDIR weight the statistics based on
the
> number of observations in each ROW_MEAN_WDIR line that it is
averaging?
>
> As an example, assume we run a job over 2 hours, one of which has 3
> stations
> reporting, and one of which has 2 stations reporting. Without
specifying
> the
> station names to MET, can we calculate statistics based on the
following
> observations as described below:
>
> Hour 1
> Station 1: Observed wind direction = 040, Model wind direction = 100
->
> Error
> = -060 -> Absolute Error = 060
> Station 2: Observed wind direction = 130, Model wind direction = 320
->
> Error
> = +170 -> Absolute Error = 170
> Station 3: Observed wind direction = 320, Model wind direction = 130
->
> Error
> = -170 -> Absolute Error = 170
> Mean Error: -20, Mean Absolute Error: 133.3
>
> Hour 2
> Station 1: Observed wind direction = 100, Model wind direction = 040
->
> Error
> = +060 -> Absolute Error = 060
> Station 2: Observed wind direction = 130, Model wind direction = 160
->
> Error
> = -030 -> Absolute Error = 030
> Mean Error: +15, Mean Absolute Error: 45
>
> It we verify hourly, it would seem that ROW_MEAN_WDIR would have:
> Mean Error: (-20 + 15) / 2 = -2.5
> Mean Absolute Error: (133.3 + 45 ) / 2 = 89.15
> since ROW_MEAN_WDIR would simply average the two hours without
regard to
> the
> number of observations in that hour.
>
> It would seem that AGGR_WDIR would be:
> Mean Error: ([3*-20]+[2*15])/(3+2) = -6
> Mean Absolute Error: ([3*133.3]+[2*45])/(3+2) = 97.98
> Since AGGR_WDIR would weight the errors from the two hours based on
the
> number
> of observations each hour's value is based on.
>
>
>
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Oct 15 12:12:11 2014
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for fielding more questions. This is really helping us. We want
to
continue the ability to verify direction because there are certain
type of
information which can only come from that, but we are also working on
a
bivariate approach which takes into account U and V separately to
calculate
the errors in a way which avoids the problem of working with circular
values.
The logic you mentioned in the MET code (angles.cc) where the
conversion of
U,V to wdir occurs, does it add/subtract 360 deg to make sure that the
error
never exceeds 180 deg?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, October 15, 2014 11:56 AM
To: Raby, John W CIV USARMY ARL (US)
Cc: bgb at ucar.edu
Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
verification (UNCLASSIFIED)
John,
(1) Yes, you can use all stations in the input dataset. When you run
this
job, STAT-Analysis reads and process all the input MPR lines. If you
haven't
told it to use a specific station, or set of stations, it'll use all
of them
that are in the input.
(2) When STAT-Analysis converts MPR lines to wind direction errors,
for each
UGRD line it finds, it needs to find the matching VGRD line. It
matches them
up by making sure all the header columns match, including the model
name, lead
time, valid time, vertical level, observation type, masking region,
and
interpolation method. It also makes sure the station id, latitude,
longitude,
level, and elevation are the same. It uses the matching U/V pairs to
derive
wind direction errors. So to answer your question, it computes wind
direction
errors separately for each U/V pair (even for different hours of the
same
station).
If you do NOT want to include multiple hours in the computations, you
could
either limit the analysis to a single hour (-fcst_lead 12), or
stratify the
results by lead time (-by FCST_LEAD).
(3) Try running this sample job using the output of the MET test
scripts:
bin/stat_analysis \
-job aggregate_stat -line_type MPR -out_line_type WDIR \
-lookin out/point_stat/point_stat_360000L_20070331_120000V.stat \
-interp_mthd UW_MEAN
Here's the output:
JOB_LIST: -job aggregate_stat -interp_mthd UW_MEAN -line_type MPR
-out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 3521 134.83504 113.91382 -5.26449 32.86718
AGGR_WDIR: 4252 130.56134 110.44698 20.11436 NA
Notice that the TOTAL column is smaller in ROW_MEAN_WDIR than it is in
AGGR_WDIR. When we aggregate U/V vectors together for the AGGR_WDIR
lines, we
can include cases where U/V are both 0. So all the input lines are
represented in AGGR_WDIR, even the 0 cases. However, when we define
wind
direction errors point-by-point for the ROW_MEAN_WDIR output, when U/V
are
both 0, the wind direction is undefined. So we can't compute a wind
direction
error. Based on this output, I'd say that in 731 cases (= 4252 -
3521) either the forecast U/V or the observed U/V were both 0.
I'm not sure if this answers your question or not. The ROW_MEAN_WDIR
and
AGGR_WDIR are just two different ways of processing the data. Over a
large
sample, the inclusion/exclusion of 0 U/V cases would have a pretty big
impact
on the computations.
Thanks,
John
On Tue, Oct 14, 2014 at 3:46 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Can I use the new wind direction verification methodology to
construct
> domain-wide wind direction errors that are based on averages of the
> wind direction errors of individual observations, and thus involve
no
> averaging of U/V wind components? It appears that one can specify a
> list of stations and have the wind direction errors of those
stations
> averaged together, rather than averaging the U/V components from all
> of the stations together and find wind direction error from that.
> However, for use in domain-wide statistics a few aspects remain
> unclear to me:
>
> 1) Instead of listing individual stations, can I specify that this
be
> done for all stations? This would allow the user to find the
> domain-wide wind direction error, rather than finding the wind
> direction difference between the domain-wide mean observed and mean
> modeled wind.
>
> 2) If a statistic involves multiple observations from a given
station
> (i.e., multiple hours of obs from a station), will the wind
direction
> errors from each observation from that station be averaged together
or
> will the U/V components be averaged first?
>
> 3) Will the ROW_MEAN_WDIR and the AGGR_WDIR both calculate wind
> direction errors by using an average of the wind direction errors of
> individual stations, but just have the AGGR_WDIR weight the
statistics
> based on the number of observations in each ROW_MEAN_WDIR line that
it is
> averaging?
>
> As an example, assume we run a job over 2 hours, one of which has 3
> stations reporting, and one of which has 2 stations reporting.
> Without specifying the station names to MET, can we calculate
> statistics based on the following observations as described below:
>
> Hour 1
> Station 1: Observed wind direction = 040, Model wind direction = 100
> -> Error = -060 -> Absolute Error = 060 Station 2: Observed wind
> direction = 130, Model wind direction = 320 -> Error = +170 ->
> Absolute Error = 170 Station 3: Observed wind direction = 320, Model
> wind direction = 130 -> Error = -170 -> Absolute Error = 170 Mean
> Error: -20, Mean Absolute Error: 133.3
>
> Hour 2
> Station 1: Observed wind direction = 100, Model wind direction = 040
> -> Error = +060 -> Absolute Error = 060 Station 2: Observed wind
> direction = 130, Model wind direction = 160 -> Error = -030 ->
> Absolute Error = 030 Mean Error: +15, Mean Absolute Error: 45
>
> It we verify hourly, it would seem that ROW_MEAN_WDIR would have:
> Mean Error: (-20 + 15) / 2 = -2.5
> Mean Absolute Error: (133.3 + 45 ) / 2 = 89.15 since ROW_MEAN_WDIR
> would simply average the two hours without regard to the number of
> observations in that hour.
>
> It would seem that AGGR_WDIR would be:
> Mean Error: ([3*-20]+[2*15])/(3+2) = -6 Mean Absolute Error:
> ([3*133.3]+[2*45])/(3+2) = 97.98 Since AGGR_WDIR would weight the
> errors from the two hours based on the number of observations each
> hour's value is based on.
>
>
>
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Oct 15 12:25:05 2014
John,
Yep, it calls and returns the value from atan2d, whose return value is
in
the range of -180 to 180.
See line 171 of the file met-5.0/src/basic/vx_math/angles.cc
John
On Wed, Oct 15, 2014 at 12:12 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for fielding more questions. This is really helping us. We
want to
> continue the ability to verify direction because there are certain
type of
> information which can only come from that, but we are also working
on a
> bivariate approach which takes into account U and V separately to
calculate
> the errors in a way which avoids the problem of working with
circular
> values.
>
> The logic you mentioned in the MET code (angles.cc) where the
conversion of
> U,V to wdir occurs, does it add/subtract 360 deg to make sure that
the
> error
> never exceeds 180 deg?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, October 15, 2014 11:56 AM
> To: Raby, John W CIV USARMY ARL (US)
> Cc: bgb at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John,
>
> (1) Yes, you can use all stations in the input dataset. When you
run this
> job, STAT-Analysis reads and process all the input MPR lines. If
you
> haven't
> told it to use a specific station, or set of stations, it'll use all
of
> them
> that are in the input.
>
> (2) When STAT-Analysis converts MPR lines to wind direction errors,
for
> each
> UGRD line it finds, it needs to find the matching VGRD line. It
matches
> them
> up by making sure all the header columns match, including the model
name,
> lead
> time, valid time, vertical level, observation type, masking region,
and
> interpolation method. It also makes sure the station id, latitude,
> longitude,
> level, and elevation are the same. It uses the matching U/V pairs
to
> derive
> wind direction errors. So to answer your question, it computes wind
> direction
> errors separately for each U/V pair (even for different hours of the
same
> station).
>
> If you do NOT want to include multiple hours in the computations,
you could
> either limit the analysis to a single hour (-fcst_lead 12), or
stratify the
> results by lead time (-by FCST_LEAD).
>
> (3) Try running this sample job using the output of the MET test
scripts:
> bin/stat_analysis \
> -job aggregate_stat -line_type MPR -out_line_type WDIR \
> -lookin out/point_stat/point_stat_360000L_20070331_120000V.stat
\
> -interp_mthd UW_MEAN
>
> Here's the output:
> JOB_LIST: -job aggregate_stat -interp_mthd UW_MEAN -line_type
MPR
> -out_line_type WDIR
> COL_NAME: TOTAL FBAR OBAR ME MAE
> ROW_MEAN_WDIR: 3521 134.83504 113.91382 -5.26449 32.86718
> AGGR_WDIR: 4252 130.56134 110.44698 20.11436 NA
>
> Notice that the TOTAL column is smaller in ROW_MEAN_WDIR than it is
in
> AGGR_WDIR. When we aggregate U/V vectors together for the AGGR_WDIR
> lines, we
> can include cases where U/V are both 0. So all the input lines are
> represented in AGGR_WDIR, even the 0 cases. However, when we define
wind
> direction errors point-by-point for the ROW_MEAN_WDIR output, when
U/V are
> both 0, the wind direction is undefined. So we can't compute a wind
> direction
> error. Based on this output, I'd say that in 731 cases (= 4252 -
> 3521) either the forecast U/V or the observed U/V were both 0.
>
> I'm not sure if this answers your question or not. The
ROW_MEAN_WDIR and
> AGGR_WDIR are just two different ways of processing the data. Over
a large
> sample, the inclusion/exclusion of 0 U/V cases would have a pretty
big
> impact
> on the computations.
>
> Thanks,
> John
>
>
>
>
> On Tue, Oct 14, 2014 at 3:46 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > Can I use the new wind direction verification methodology to
construct
> > domain-wide wind direction errors that are based on averages of
the
> > wind direction errors of individual observations, and thus involve
no
> > averaging of U/V wind components? It appears that one can specify
a
> > list of stations and have the wind direction errors of those
stations
> > averaged together, rather than averaging the U/V components from
all
> > of the stations together and find wind direction error from that.
> > However, for use in domain-wide statistics a few aspects remain
> > unclear to me:
> >
> > 1) Instead of listing individual stations, can I specify that this
be
> > done for all stations? This would allow the user to find the
> > domain-wide wind direction error, rather than finding the wind
> > direction difference between the domain-wide mean observed and
mean
> > modeled wind.
> >
> > 2) If a statistic involves multiple observations from a given
station
> > (i.e., multiple hours of obs from a station), will the wind
direction
> > errors from each observation from that station be averaged
together or
> > will the U/V components be averaged first?
> >
> > 3) Will the ROW_MEAN_WDIR and the AGGR_WDIR both calculate wind
> > direction errors by using an average of the wind direction errors
of
> > individual stations, but just have the AGGR_WDIR weight the
statistics
> > based on the number of observations in each ROW_MEAN_WDIR line
that it is
> > averaging?
> >
> > As an example, assume we run a job over 2 hours, one of which has
3
> > stations reporting, and one of which has 2 stations reporting.
> > Without specifying the station names to MET, can we calculate
> > statistics based on the following observations as described below:
> >
> > Hour 1
> > Station 1: Observed wind direction = 040, Model wind direction =
100
> > -> Error = -060 -> Absolute Error = 060 Station 2: Observed wind
> > direction = 130, Model wind direction = 320 -> Error = +170 ->
> > Absolute Error = 170 Station 3: Observed wind direction = 320,
Model
> > wind direction = 130 -> Error = -170 -> Absolute Error = 170 Mean
> > Error: -20, Mean Absolute Error: 133.3
> >
> > Hour 2
> > Station 1: Observed wind direction = 100, Model wind direction =
040
> > -> Error = +060 -> Absolute Error = 060 Station 2: Observed wind
> > direction = 130, Model wind direction = 160 -> Error = -030 ->
> > Absolute Error = 030 Mean Error: +15, Mean Absolute Error: 45
> >
> > It we verify hourly, it would seem that ROW_MEAN_WDIR would have:
> > Mean Error: (-20 + 15) / 2 = -2.5
> > Mean Absolute Error: (133.3 + 45 ) / 2 = 89.15 since ROW_MEAN_WDIR
> > would simply average the two hours without regard to the number of
> > observations in that hour.
> >
> > It would seem that AGGR_WDIR would be:
> > Mean Error: ([3*-20]+[2*15])/(3+2) = -6 Mean Absolute Error:
> > ([3*133.3]+[2*45])/(3+2) = 97.98 Since AGGR_WDIR would weight the
> > errors from the two hours based on the number of observations each
> > hour's value is based on.
> >
> >
> >
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list