[Met_help] [rt.rap.ucar.edu #69098] History for Stat-Analysis wind direction verification (UNCLASSIFIED)

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Classification: UNCLASSIFIED
Caveats: NONE

I need to know more specifically how wind direction errors are calculated. I
ran a case using METV4.1 using a forecast and observations and I followed
the guidance to assure that the Point-Stat VL1L2 lines were generated. When
I ran Stat-Analysis I specified the job as aggregate_stat, the -line_type as
VL1L2 and the -out_line_type as WDIR. The attached document provides samples
of the output and my attempts to calculate the errors by hand so I could try
to understand how MET calculates them. I also highlighted the questions I
have in the document. I would appreciate having answers to my questions if
possible. Thanks.

R/
John


Classification: UNCLASSIFIED
Caveats: NONE




----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Sep 25 08:54:05 2014

John

I'm out of the office today, but will take a close look at this on
Friday.

Thanks
John

On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
<javascript:;>
>        Queue: met_help
>      Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
>        Owner: Nobody
>   Requestors: john.w.raby2.civ at mail.mil <javascript:;>
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I need to know more specifically how wind direction errors are
calculated.
> I
> ran a case using METV4.1 using a forecast and observations and I
followed
> the guidance to assure that the Point-Stat VL1L2 lines were
generated. When
> I ran Stat-Analysis I specified the job as aggregate_stat, the
-line_type
> as
> VL1L2 and the -out_line_type as WDIR. The attached document provides
> samples
> of the output and my attempts to calculate the errors by hand so I
could
> try
> to understand how MET calculates them. I also highlighted the
questions I
> have in the document. I would appreciate having answers to my
questions if
> possible. Thanks.
>
> R/
> John
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>

------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Sep 25 09:24:34 2014

Classification: UNCLASSIFIED
Caveats: NONE

John -

Thanks for the update. I will be out Friday and Monday, returning on
Tuesday.
Have a great weekend!

R/
John

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, September 25, 2014 8:54 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
verification (UNCLASSIFIED)

John

I'm out of the office today, but will take a close look at this on
Friday.

Thanks
John

On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
<javascript:;>
>        Queue: met_help
>      Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
>        Owner: Nobody
>   Requestors: john.w.raby2.civ at mail.mil <javascript:;>
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I need to know more specifically how wind direction errors are
calculated.
> I
> ran a case using METV4.1 using a forecast and observations and I
> followed the guidance to assure that the Point-Stat VL1L2 lines were
> generated. When I ran Stat-Analysis I specified the job as
> aggregate_stat, the -line_type as
> VL1L2 and the -out_line_type as WDIR. The attached document provides
> samples of the output and my attempts to calculate the errors by
hand
> so I could try to understand how MET calculates them. I also
> highlighted the questions I have in the document. I would appreciate
> having answers to my questions if possible. Thanks.
>
> R/
> John
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>


Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Sep 26 13:05:14 2014

John,

I read through the document you sent about the output of wind
direction
errors and will respond to your questions below...

I see that you've run a single VL1L2 line (computed over 103 matched
pairs)
through STAT-Analysis to compute wind direction errors.  In the output
you
see the same values for FBAR (94.25) and OBAR (89.09) repeated in both
lines (ROW_MEAN_WDIR and AGGR_WDIR) of the output.  You see that
because
this job is not actually aggregating multiple lines together.  It's
just
being run over a single VL1L2 line.  If your job was run over multiple
VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
AGGR_WDIR lines would differ.

Your first question is - how is the MAE value calculated?  The FBAR
and
OBAR columns contain the forecast and observed wind directions, in
degrees.  The error is the directed difference between those angles (f
-
o).  The mean error (ME) is the average of those angular differences.
The
range of this column is -180 to +180.  The MAE is the average of the
*absolute value* of those angular differences.  So the range of the
MAE
column is 0 to +180.

You're wondering where in the MET code this logic is implemented.  In
the
file "src/basic/vx_math/angles.cc" the function named
"convert_u_v_to_wdir"
handles the conversion of the U and V components to wind direction.
Also,
the file "src/tools/core/stat_analysis/stat_analysis_job.cc" contains
the
function "write_job_aggr_wind" which creates the output for STAT-
Analysis.

Your next question is why you the numbers are the same in
ROW_MEAN_WDIR And
AGGR_WDIR - again, that's because you're running over a single VL1L2
line.
If you run on multiple, they will likely differ.  It's difficult to
describe the difference with words, but I'll give it a shot.  Suppose
you're running this job over 10 different VL1L2 lines that represent
output
for 10 different days over the same region.  And let's suppose 5 days
had
50 stations reporting and 5 days had 100 stations reporting...

- For ROW_MEAN_WDIR, we do the following...
  - For each VL1L2 line (meaning, for each day), compute the average
forecast and observation direction and compute the angular difference.
  - Compute the average of those 10 angular differences and write it
in the
ME column.  Also, put the average of the absolute value of those
differences in the MAE column.
  - This is the daily average of the wind direction errors.  Even
though
some of the days were computed over 50 matched pairs and others were
computed over 100.

- For the AGGR_WDIR, aggregate together those 10 different VL1L2 lines
into
a single VL1L2 line.  When we aggregate VL1L2 lines, we do a weighted
average, weighted by the "TOTAL" columns.  Relatively speaking, those
5
days with a 100 matched pairs will count twice as much as the other 5
days
with only 50 matched pairs.
  - From that single aggregated VL1L2 line, compute forecast and
observation directions.  The ME is the angular difference between
them.
  - We could write out the MAE which would just be the absolute value
of
the the ME, but we're just writing NA.
  - This is an aggregated wind direction difference.

Both are reasonable ways of processing the data - but the computations
are
slightly different.

The real short-coming here is that we're only computing wind direction
errors over points that have already been aggregated through space.
In
METv4.1 STAT-Analysis, the only supported job type is "-job
aggregate_stat
-line_type VL1L2 -out_line_type MPR".

To address this short-coming in met-5.0, we added a similar job that
operates over matched pair values instead.  Here's a selection from
the
met-5.0 release notes:

"Enhanced STAT-Analysis by adding support for an aggregate_stat job
type
conversion of MPR to WDIR that reads UGRD and VGRD matched pair lines
and
computes aggregated wind direction errors. Added -out_fcst_wind_thresh
and
-out_obs_wind_thresh options to filter the U/V pairs by wind speed
prior to
computing wind direction errors."

For example, the following job would compute wind direction errors for
each
station individually...
  -job aggregate_wdir -line_type MPR -out_line_type WDIR -by OBS_SID
-out_fcst_wind_thresh \>+2.0

That'll compute wind direction errors for each unique station name in
the
OBS_SID column of the MPR line, but only when the forecast wind speed
exceeds 2 m/s.

This should make it easier to look at wind direction errors
station-by-station, rather than have to aggregate over a much larger
area.

Please let me know if this answers your questions.  If not, I'll
probably
refer you to a scientist here to address the interpretation of the
output.
I can help you running the tools and describing the options, but I
think a
scientist would do a better job helping you interpret the results.

Thanks,
John

On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for the update. I will be out Friday and Monday, returning on
> Tuesday.
> Have a great weekend!
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, September 25, 2014 8:54 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John
>
> I'm out of the office today, but will take a close look at this on
Friday.
>
> Thanks
> John
>
> On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
<javascript:;>
> >        Queue: met_help
> >      Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
> >        Owner: Nobody
> >   Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I need to know more specifically how wind direction errors are
> calculated.
> > I
> > ran a case using METV4.1 using a forecast and observations and I
> > followed the guidance to assure that the Point-Stat VL1L2 lines
were
> > generated. When I ran Stat-Analysis I specified the job as
> > aggregate_stat, the -line_type as
> > VL1L2 and the -out_line_type as WDIR. The attached document
provides
> > samples of the output and my attempts to calculate the errors by
hand
> > so I could try to understand how MET calculates them. I also
> > highlighted the questions I have in the document. I would
appreciate
> > having answers to my questions if possible. Thanks.
> >
> > R/
> > John
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>

------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Oct 01 10:09:57 2014

Classification: UNCLASSIFIED
Caveats: NONE

John -

Thanks for your detailed explanations for my questions. I think it
confirmed
how the Stat-Analysis output is generated for me. Now it raises some
questions
about how the vector partial sums in the Point-Stat VL1L2 line type
are
generated.

I was able to trace how the partial sums UFBAR, VFBAR and UOBAR, VOBAR
were
generated from the UGRD, VGRD matched pair data in the MPR file in a
spreadsheet. I simply took the mean of all 103 values of the UGRD and
VGRD for
the Z10 level and the results were identical. For example: for UFBAR,
I took
the average of all 103 forecast values of UGRD and the result matched
the
value for UFBAR in the VL1L2 output. From your answer below, I gather
that you
use the code "convert_u_v_to_wdir" to convert UFBAR, VFBAR to generate
the
forecast wind direction (FBAR in the Stat-Analysis output) and the
same code
is used to convert UOBAR, VOBAR to generate the observed wind
direction (OBAR
in the Stat-analysis output). Correct so far?

So, based on your answer and the above I can trace the data in the MPR
file
all the way through the Stat-Analysis output of FBAR, OBAR, ME and
MAE. The
only remaining question, I have is about the vector partial sums:
UVFOBAR,
UVFFBAR and UVOOBAR. I looked in the User's Guide where the VL1L2 line
type is
described and I see the formulas for how these are calculated, but I
don't see
where these partial sums are used for generating the values in the
Stat-Analysis output. How are these partial sums used?

R/
John

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, September 26, 2014 1:05 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
verification (UNCLASSIFIED)

John,

I read through the document you sent about the output of wind
direction errors
and will respond to your questions below...

I see that you've run a single VL1L2 line (computed over 103 matched
pairs)
through STAT-Analysis to compute wind direction errors.  In the output
you see
the same values for FBAR (94.25) and OBAR (89.09) repeated in both
lines
(ROW_MEAN_WDIR and AGGR_WDIR) of the output.  You see that because
this job is
not actually aggregating multiple lines together.  It's just being run
over a
single VL1L2 line.  If your job was run over multiple
VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
AGGR_WDIR
lines would differ.

Your first question is - how is the MAE value calculated?  The FBAR
and OBAR
columns contain the forecast and observed wind directions, in degrees.
The
error is the directed difference between those angles (f - o).  The
mean error
(ME) is the average of those angular differences. The range of this
column
is -180 to +180.  The MAE is the average of the *absolute value* of
those
angular differences.  So the range of the MAE column is 0 to +180.

You're wondering where in the MET code this logic is implemented.  In
the file
"src/basic/vx_math/angles.cc" the function named "convert_u_v_to_wdir"
handles the conversion of the U and V components to wind direction.
Also, the
file "src/tools/core/stat_analysis/stat_analysis_job.cc" contains the
function
"write_job_aggr_wind" which creates the output for STAT-Analysis.

Your next question is why you the numbers are the same in
ROW_MEAN_WDIR And
AGGR_WDIR - again, that's because you're running over a single VL1L2
line.
If you run on multiple, they will likely differ.  It's difficult to
describe
the difference with words, but I'll give it a shot.  Suppose you're
running
this job over 10 different VL1L2 lines that represent output for 10
different
days over the same region.  And let's suppose 5 days had
50 stations reporting and 5 days had 100 stations reporting...

- For ROW_MEAN_WDIR, we do the following...
  - For each VL1L2 line (meaning, for each day), compute the average
forecast
and observation direction and compute the angular difference.
  - Compute the average of those 10 angular differences and write it
in the ME
column.  Also, put the average of the absolute value of those
differences in
the MAE column.
  - This is the daily average of the wind direction errors.  Even
though some
of the days were computed over 50 matched pairs and others were
computed over
100.

- For the AGGR_WDIR, aggregate together those 10 different VL1L2 lines
into a
single VL1L2 line.  When we aggregate VL1L2 lines, we do a weighted
average,
weighted by the "TOTAL" columns.  Relatively speaking, those 5 days
with a 100
matched pairs will count twice as much as the other 5 days with only
50
matched pairs.
  - From that single aggregated VL1L2 line, compute forecast and
observation
directions.  The ME is the angular difference between them.
  - We could write out the MAE which would just be the absolute value
of the
the ME, but we're just writing NA.
  - This is an aggregated wind direction difference.

Both are reasonable ways of processing the data - but the computations
are
slightly different.

The real short-coming here is that we're only computing wind direction
errors
over points that have already been aggregated through space.  In
METv4.1 STAT-Analysis, the only supported job type is "-job
aggregate_stat -line_type VL1L2 -out_line_type MPR".

To address this short-coming in met-5.0, we added a similar job that
operates
over matched pair values instead.  Here's a selection from the
met-5.0 release notes:

"Enhanced STAT-Analysis by adding support for an aggregate_stat job
type
conversion of MPR to WDIR that reads UGRD and VGRD matched pair lines
and
computes aggregated wind direction errors. Added -out_fcst_wind_thresh
and -out_obs_wind_thresh options to filter the U/V pairs by wind speed
prior
to computing wind direction errors."

For example, the following job would compute wind direction errors for
each
station individually...
  -job aggregate_wdir -line_type MPR -out_line_type WDIR -by
OBS_SID -out_fcst_wind_thresh \>+2.0

That'll compute wind direction errors for each unique station name in
the
OBS_SID column of the MPR line, but only when the forecast wind speed
exceeds
2 m/s.

This should make it easier to look at wind direction errors
station-by-station, rather than have to aggregate over a much larger
area.

Please let me know if this answers your questions.  If not, I'll
probably
refer you to a scientist here to address the interpretation of the
output.
I can help you running the tools and describing the options, but I
think a
scientist would do a better job helping you interpret the results.

Thanks,
John

On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for the update. I will be out Friday and Monday, returning on
> Tuesday.
> Have a great weekend!
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, September 25, 2014 8:54 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John
>
> I'm out of the office today, but will take a close look at this on
Friday.
>
> Thanks
> John
>
> On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
<javascript:;>
> >        Queue: met_help
> >      Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
> >        Owner: Nobody
> >   Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> >       Status: new
> >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I need to know more specifically how wind direction errors are
> calculated.
> > I
> > ran a case using METV4.1 using a forecast and observations and I
> > followed the guidance to assure that the Point-Stat VL1L2 lines
were
> > generated. When I ran Stat-Analysis I specified the job as
> > aggregate_stat, the -line_type as
> > VL1L2 and the -out_line_type as WDIR. The attached document
provides
> > samples of the output and my attempts to calculate the errors by
> > hand so I could try to understand how MET calculates them. I also
> > highlighted the questions I have in the document. I would
appreciate
> > having answers to my questions if possible. Thanks.
> >
> > R/
> > John
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>


Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Julie Prestopnik
Time: Wed Oct 01 10:13:26 2014

Hi John.  John is currently out of the office, but will hopefully be
able
to respond to your questions later this week.  I do not have enough
knowledge to answer your questions.

Thanks,
Julie

On Wed, Oct 1, 2014 at 10:09 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for your detailed explanations for my questions. I think it
> confirmed
> how the Stat-Analysis output is generated for me. Now it raises some
> questions
> about how the vector partial sums in the Point-Stat VL1L2 line type
are
> generated.
>
> I was able to trace how the partial sums UFBAR, VFBAR and UOBAR,
VOBAR were
> generated from the UGRD, VGRD matched pair data in the MPR file in a
> spreadsheet. I simply took the mean of all 103 values of the UGRD
and VGRD
> for
> the Z10 level and the results were identical. For example: for
UFBAR, I
> took
> the average of all 103 forecast values of UGRD and the result
matched the
> value for UFBAR in the VL1L2 output. From your answer below, I
gather that
> you
> use the code "convert_u_v_to_wdir" to convert UFBAR, VFBAR to
generate the
> forecast wind direction (FBAR in the Stat-Analysis output) and the
same
> code
> is used to convert UOBAR, VOBAR to generate the observed wind
direction
> (OBAR
> in the Stat-analysis output). Correct so far?
>
> So, based on your answer and the above I can trace the data in the
MPR file
> all the way through the Stat-Analysis output of FBAR, OBAR, ME and
MAE. The
> only remaining question, I have is about the vector partial sums:
UVFOBAR,
> UVFFBAR and UVOOBAR. I looked in the User's Guide where the VL1L2
line
> type is
> described and I see the formulas for how these are calculated, but I
don't
> see
> where these partial sums are used for generating the values in the
> Stat-Analysis output. How are these partial sums used?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, September 26, 2014 1:05 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John,
>
> I read through the document you sent about the output of wind
direction
> errors
> and will respond to your questions below...
>
> I see that you've run a single VL1L2 line (computed over 103 matched
pairs)
> through STAT-Analysis to compute wind direction errors.  In the
output you
> see
> the same values for FBAR (94.25) and OBAR (89.09) repeated in both
lines
> (ROW_MEAN_WDIR and AGGR_WDIR) of the output.  You see that because
this
> job is
> not actually aggregating multiple lines together.  It's just being
run
> over a
> single VL1L2 line.  If your job was run over multiple
> VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
> AGGR_WDIR
> lines would differ.
>
> Your first question is - how is the MAE value calculated?  The FBAR
and
> OBAR
> columns contain the forecast and observed wind directions, in
degrees.  The
> error is the directed difference between those angles (f - o).  The
mean
> error
> (ME) is the average of those angular differences. The range of this
column
> is -180 to +180.  The MAE is the average of the *absolute value* of
those
> angular differences.  So the range of the MAE column is 0 to +180.
>
> You're wondering where in the MET code this logic is implemented.
In the
> file
> "src/basic/vx_math/angles.cc" the function named
"convert_u_v_to_wdir"
> handles the conversion of the U and V components to wind direction.
Also,
> the
> file "src/tools/core/stat_analysis/stat_analysis_job.cc" contains
the
> function
> "write_job_aggr_wind" which creates the output for STAT-Analysis.
>
> Your next question is why you the numbers are the same in
ROW_MEAN_WDIR And
> AGGR_WDIR - again, that's because you're running over a single VL1L2
line.
> If you run on multiple, they will likely differ.  It's difficult to
> describe
> the difference with words, but I'll give it a shot.  Suppose you're
running
> this job over 10 different VL1L2 lines that represent output for 10
> different
> days over the same region.  And let's suppose 5 days had
> 50 stations reporting and 5 days had 100 stations reporting...
>
> - For ROW_MEAN_WDIR, we do the following...
>   - For each VL1L2 line (meaning, for each day), compute the average
> forecast
> and observation direction and compute the angular difference.
>   - Compute the average of those 10 angular differences and write it
in
> the ME
> column.  Also, put the average of the absolute value of those
differences
> in
> the MAE column.
>   - This is the daily average of the wind direction errors.  Even
though
> some
> of the days were computed over 50 matched pairs and others were
computed
> over
> 100.
>
> - For the AGGR_WDIR, aggregate together those 10 different VL1L2
lines
> into a
> single VL1L2 line.  When we aggregate VL1L2 lines, we do a weighted
> average,
> weighted by the "TOTAL" columns.  Relatively speaking, those 5 days
with a
> 100
> matched pairs will count twice as much as the other 5 days with only
50
> matched pairs.
>   - From that single aggregated VL1L2 line, compute forecast and
> observation
> directions.  The ME is the angular difference between them.
>   - We could write out the MAE which would just be the absolute
value of
> the
> the ME, but we're just writing NA.
>   - This is an aggregated wind direction difference.
>
> Both are reasonable ways of processing the data - but the
computations are
> slightly different.
>
> The real short-coming here is that we're only computing wind
direction
> errors
> over points that have already been aggregated through space.  In
> METv4.1 STAT-Analysis, the only supported job type is "-job
> aggregate_stat -line_type VL1L2 -out_line_type MPR".
>
> To address this short-coming in met-5.0, we added a similar job that
> operates
> over matched pair values instead.  Here's a selection from the
> met-5.0 release notes:
>
> "Enhanced STAT-Analysis by adding support for an aggregate_stat job
type
> conversion of MPR to WDIR that reads UGRD and VGRD matched pair
lines and
> computes aggregated wind direction errors. Added
-out_fcst_wind_thresh
> and -out_obs_wind_thresh options to filter the U/V pairs by wind
speed
> prior
> to computing wind direction errors."
>
> For example, the following job would compute wind direction errors
for each
> station individually...
>   -job aggregate_wdir -line_type MPR -out_line_type WDIR -by
> OBS_SID -out_fcst_wind_thresh \>+2.0
>
> That'll compute wind direction errors for each unique station name
in the
> OBS_SID column of the MPR line, but only when the forecast wind
speed
> exceeds
> 2 m/s.
>
> This should make it easier to look at wind direction errors
> station-by-station, rather than have to aggregate over a much larger
area.
>
> Please let me know if this answers your questions.  If not, I'll
probably
> refer you to a scientist here to address the interpretation of the
output.
> I can help you running the tools and describing the options, but I
think a
> scientist would do a better job helping you interpret the results.
>
> Thanks,
> John
>
> On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for the update. I will be out Friday and Monday, returning
on
> > Tuesday.
> > Have a great weekend!
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, September 25, 2014 8:54 AM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> > verification (UNCLASSIFIED)
> >
> > John
> >
> > I'm out of the office today, but will take a close look at this on
> Friday.
> >
> > Thanks
> > John
> >
> > On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> <javascript:;>
> > >        Queue: met_help
> > >      Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
> > >        Owner: Nobody
> > >   Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > >       Status: new
> > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > I need to know more specifically how wind direction errors are
> > calculated.
> > > I
> > > ran a case using METV4.1 using a forecast and observations and I
> > > followed the guidance to assure that the Point-Stat VL1L2 lines
were
> > > generated. When I ran Stat-Analysis I specified the job as
> > > aggregate_stat, the -line_type as
> > > VL1L2 and the -out_line_type as WDIR. The attached document
provides
> > > samples of the output and my attempts to calculate the errors by
> > > hand so I could try to understand how MET calculates them. I
also
> > > highlighted the questions I have in the document. I would
appreciate
> > > having answers to my questions if possible. Thanks.
> > >
> > > R/
> > > John
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>


--
Julie Prestopnik
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu

------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Oct 01 10:21:46 2014

Classification: UNCLASSIFIED
Caveats: NONE

Thanks for your response. No problem with waiting until he gets back.

R/
John

-----Original Message-----
From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, October 01, 2014 10:13 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
verification (UNCLASSIFIED)

Hi John.  John is currently out of the office, but will hopefully be
able to
respond to your questions later this week.  I do not have enough
knowledge to
answer your questions.

Thanks,
Julie

On Wed, Oct 1, 2014 at 10:09 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for your detailed explanations for my questions. I think it
> confirmed how the Stat-Analysis output is generated for me. Now it
> raises some questions about how the vector partial sums in the
> Point-Stat VL1L2 line type are generated.
>
> I was able to trace how the partial sums UFBAR, VFBAR and UOBAR,
VOBAR
> were generated from the UGRD, VGRD matched pair data in the MPR file
> in a spreadsheet. I simply took the mean of all 103 values of the
UGRD
> and VGRD for the Z10 level and the results were identical. For
> example: for UFBAR, I took the average of all 103 forecast values of
> UGRD and the result matched the value for UFBAR in the VL1L2 output.
> From your answer below, I gather that you use the code
> "convert_u_v_to_wdir" to convert UFBAR, VFBAR to generate the
forecast
> wind direction (FBAR in the Stat-Analysis output) and the same code
is
> used to convert UOBAR, VOBAR to generate the observed wind direction
> (OBAR in the Stat-analysis output). Correct so far?
>
> So, based on your answer and the above I can trace the data in the
MPR
> file all the way through the Stat-Analysis output of FBAR, OBAR, ME
> and MAE. The only remaining question, I have is about the vector
> partial sums: UVFOBAR, UVFFBAR and UVOOBAR. I looked in the User's
> Guide where the VL1L2 line type is described and I see the formulas
> for how these are calculated, but I don't see where these partial
sums
> are used for generating the values in the Stat-Analysis output. How
> are these partial sums used?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, September 26, 2014 1:05 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John,
>
> I read through the document you sent about the output of wind
> direction errors and will respond to your questions below...
>
> I see that you've run a single VL1L2 line (computed over 103 matched
> pairs) through STAT-Analysis to compute wind direction errors.  In
the
> output you see the same values for FBAR (94.25) and OBAR (89.09)
> repeated in both lines (ROW_MEAN_WDIR and AGGR_WDIR) of the output.
> You see that because this job is not actually aggregating multiple
> lines together.  It's just being run over a single VL1L2 line.  If
> your job was run over multiple
> VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
> AGGR_WDIR lines would differ.
>
> Your first question is - how is the MAE value calculated?  The FBAR
> and OBAR columns contain the forecast and observed wind directions,
in
> degrees.  The error is the directed difference between those angles
(f
> - o).  The mean error
> (ME) is the average of those angular differences. The range of this
> column is -180 to +180.  The MAE is the average of the *absolute
> value* of those angular differences.  So the range of the MAE column
is 0 to
> +180.
>
> You're wondering where in the MET code this logic is implemented.
In
> the file "src/basic/vx_math/angles.cc" the function named
> "convert_u_v_to_wdir"
> handles the conversion of the U and V components to wind direction.
> Also, the file "src/tools/core/stat_analysis/stat_analysis_job.cc"
> contains the function "write_job_aggr_wind" which creates the output
> for STAT-Analysis.
>
> Your next question is why you the numbers are the same in
> ROW_MEAN_WDIR And AGGR_WDIR - again, that's because you're running
over a
> single VL1L2 line.
> If you run on multiple, they will likely differ.  It's difficult to
> describe the difference with words, but I'll give it a shot.
Suppose
> you're running this job over 10 different VL1L2 lines that represent
> output for 10 different days over the same region.  And let's
suppose
> 5 days had
> 50 stations reporting and 5 days had 100 stations reporting...
>
> - For ROW_MEAN_WDIR, we do the following...
>   - For each VL1L2 line (meaning, for each day), compute the average
> forecast and observation direction and compute the angular
difference.
>   - Compute the average of those 10 angular differences and write it
> in the ME column.  Also, put the average of the absolute value of
> those differences in the MAE column.
>   - This is the daily average of the wind direction errors.  Even
> though some of the days were computed over 50 matched pairs and
others
> were computed over 100.
>
> - For the AGGR_WDIR, aggregate together those 10 different VL1L2
lines
> into a single VL1L2 line.  When we aggregate VL1L2 lines, we do a
> weighted average, weighted by the "TOTAL" columns.  Relatively
> speaking, those 5 days with a
> 100
> matched pairs will count twice as much as the other 5 days with only
> 50 matched pairs.
>   - From that single aggregated VL1L2 line, compute forecast and
> observation directions.  The ME is the angular difference between
> them.
>   - We could write out the MAE which would just be the absolute
value
> of the the ME, but we're just writing NA.
>   - This is an aggregated wind direction difference.
>
> Both are reasonable ways of processing the data - but the
computations
> are slightly different.
>
> The real short-coming here is that we're only computing wind
direction
> errors over points that have already been aggregated through space.
> In
> METv4.1 STAT-Analysis, the only supported job type is "-job
> aggregate_stat -line_type VL1L2 -out_line_type MPR".
>
> To address this short-coming in met-5.0, we added a similar job that
> operates over matched pair values instead.  Here's a selection from
> the
> met-5.0 release notes:
>
> "Enhanced STAT-Analysis by adding support for an aggregate_stat job
> type conversion of MPR to WDIR that reads UGRD and VGRD matched pair
> lines and computes aggregated wind direction errors. Added
> -out_fcst_wind_thresh and -out_obs_wind_thresh options to filter the
> U/V pairs by wind speed prior to computing wind direction errors."
>
> For example, the following job would compute wind direction errors
for
> each station individually...
>   -job aggregate_wdir -line_type MPR -out_line_type WDIR -by OBS_SID
> -out_fcst_wind_thresh \>+2.0
>
> That'll compute wind direction errors for each unique station name
in
> the OBS_SID column of the MPR line, but only when the forecast wind
> speed exceeds
> 2 m/s.
>
> This should make it easier to look at wind direction errors
> station-by-station, rather than have to aggregate over a much larger
area.
>
> Please let me know if this answers your questions.  If not, I'll
> probably refer you to a scientist here to address the interpretation
of the
> output.
> I can help you running the tools and describing the options, but I
> think a scientist would do a better job helping you interpret the
results.
>
> Thanks,
> John
>
> On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for the update. I will be out Friday and Monday, returning
on
> > Tuesday.
> > Have a great weekend!
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, September 25, 2014 8:54 AM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> > verification (UNCLASSIFIED)
> >
> > John
> >
> > I'm out of the office today, but will take a close look at this on
> Friday.
> >
> > Thanks
> > John
> >
> > On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> <javascript:;>
> > >        Queue: met_help
> > >      Subject: Stat-Analysis wind direction verification
(UNCLASSIFIED)
> > >        Owner: Nobody
> > >   Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > >       Status: new
> > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > I need to know more specifically how wind direction errors are
> > calculated.
> > > I
> > > ran a case using METV4.1 using a forecast and observations and I
> > > followed the guidance to assure that the Point-Stat VL1L2 lines
> > > were generated. When I ran Stat-Analysis I specified the job as
> > > aggregate_stat, the -line_type as
> > > VL1L2 and the -out_line_type as WDIR. The attached document
> > > provides samples of the output and my attempts to calculate the
> > > errors by hand so I could try to understand how MET calculates
> > > them. I also highlighted the questions I have in the document. I
> > > would appreciate having answers to my questions if possible.
Thanks.
> > >
> > > R/
> > > John
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>


--
Julie Prestopnik
National Center for Atmospheric Research Research Applications
Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu


Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Oct 02 13:45:24 2014

John,

Here's the contents of the VL1L2 line: UFBAR, VFBAR, UOBAR, VOBAR,
UVFOBAR,
UVFFBAR, UVOOBAR

Yes, the forecast average U and V (UFBAR and VFBAR) are used in
computing
the forecast wind speed and direction.  While UOBAR and VOBAR are used
in
computing the observed wind speed and direction.  And you're wondering
how
the UVFOBAR, UVFFBAR, and UVOOBAR data is used.  The answer is that
they're
not currently being used in the computation any statistics!

The MET software was derived from the NCEP verification tools which
include
the VL1L2 output line.  I spoke with statisticians here, and we
believe
that NCEP uses the data in those 3 columns to derive a combined U and
V
vector-based RMSE score.  FVS is the name of the plotting tools
associated
with that package, and we'll check it's documentation to see if/how
NCEP is
using those values to derive statistics.

If so, we could potentially add that capability to MET, deriving
vector-based scores.  And I'll forward whatever equations we find.

Thanks for the question, and we'll let you know what we turn up.

Thanks,
John

On Wed, Oct 1, 2014 at 10:13 AM, Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Hi John.  John is currently out of the office, but will hopefully be
able
> to respond to your questions later this week.  I do not have enough
> knowledge to answer your questions.
>
> Thanks,
> Julie
>
> On Wed, Oct 1, 2014 at 10:09 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for your detailed explanations for my questions. I think it
> > confirmed
> > how the Stat-Analysis output is generated for me. Now it raises
some
> > questions
> > about how the vector partial sums in the Point-Stat VL1L2 line
type are
> > generated.
> >
> > I was able to trace how the partial sums UFBAR, VFBAR and UOBAR,
VOBAR
> were
> > generated from the UGRD, VGRD matched pair data in the MPR file in
a
> > spreadsheet. I simply took the mean of all 103 values of the UGRD
and
> VGRD
> > for
> > the Z10 level and the results were identical. For example: for
UFBAR, I
> > took
> > the average of all 103 forecast values of UGRD and the result
matched the
> > value for UFBAR in the VL1L2 output. From your answer below, I
gather
> that
> > you
> > use the code "convert_u_v_to_wdir" to convert UFBAR, VFBAR to
generate
> the
> > forecast wind direction (FBAR in the Stat-Analysis output) and the
same
> > code
> > is used to convert UOBAR, VOBAR to generate the observed wind
direction
> > (OBAR
> > in the Stat-analysis output). Correct so far?
> >
> > So, based on your answer and the above I can trace the data in the
MPR
> file
> > all the way through the Stat-Analysis output of FBAR, OBAR, ME and
MAE.
> The
> > only remaining question, I have is about the vector partial sums:
> UVFOBAR,
> > UVFFBAR and UVOOBAR. I looked in the User's Guide where the VL1L2
line
> > type is
> > described and I see the formulas for how these are calculated, but
I
> don't
> > see
> > where these partial sums are used for generating the values in the
> > Stat-Analysis output. How are these partial sums used?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, September 26, 2014 1:05 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> > verification (UNCLASSIFIED)
> >
> > John,
> >
> > I read through the document you sent about the output of wind
direction
> > errors
> > and will respond to your questions below...
> >
> > I see that you've run a single VL1L2 line (computed over 103
matched
> pairs)
> > through STAT-Analysis to compute wind direction errors.  In the
output
> you
> > see
> > the same values for FBAR (94.25) and OBAR (89.09) repeated in both
lines
> > (ROW_MEAN_WDIR and AGGR_WDIR) of the output.  You see that because
this
> > job is
> > not actually aggregating multiple lines together.  It's just being
run
> > over a
> > single VL1L2 line.  If your job was run over multiple
> > VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
> > AGGR_WDIR
> > lines would differ.
> >
> > Your first question is - how is the MAE value calculated?  The
FBAR and
> > OBAR
> > columns contain the forecast and observed wind directions, in
degrees.
> The
> > error is the directed difference between those angles (f - o).
The mean
> > error
> > (ME) is the average of those angular differences. The range of
this
> column
> > is -180 to +180.  The MAE is the average of the *absolute value*
of those
> > angular differences.  So the range of the MAE column is 0 to +180.
> >
> > You're wondering where in the MET code this logic is implemented.
In the
> > file
> > "src/basic/vx_math/angles.cc" the function named
"convert_u_v_to_wdir"
> > handles the conversion of the U and V components to wind
direction.
> Also,
> > the
> > file "src/tools/core/stat_analysis/stat_analysis_job.cc" contains
the
> > function
> > "write_job_aggr_wind" which creates the output for STAT-Analysis.
> >
> > Your next question is why you the numbers are the same in
ROW_MEAN_WDIR
> And
> > AGGR_WDIR - again, that's because you're running over a single
VL1L2
> line.
> > If you run on multiple, they will likely differ.  It's difficult
to
> > describe
> > the difference with words, but I'll give it a shot.  Suppose
you're
> running
> > this job over 10 different VL1L2 lines that represent output for
10
> > different
> > days over the same region.  And let's suppose 5 days had
> > 50 stations reporting and 5 days had 100 stations reporting...
> >
> > - For ROW_MEAN_WDIR, we do the following...
> >   - For each VL1L2 line (meaning, for each day), compute the
average
> > forecast
> > and observation direction and compute the angular difference.
> >   - Compute the average of those 10 angular differences and write
it in
> > the ME
> > column.  Also, put the average of the absolute value of those
differences
> > in
> > the MAE column.
> >   - This is the daily average of the wind direction errors.  Even
though
> > some
> > of the days were computed over 50 matched pairs and others were
computed
> > over
> > 100.
> >
> > - For the AGGR_WDIR, aggregate together those 10 different VL1L2
lines
> > into a
> > single VL1L2 line.  When we aggregate VL1L2 lines, we do a
weighted
> > average,
> > weighted by the "TOTAL" columns.  Relatively speaking, those 5
days with
> a
> > 100
> > matched pairs will count twice as much as the other 5 days with
only 50
> > matched pairs.
> >   - From that single aggregated VL1L2 line, compute forecast and
> > observation
> > directions.  The ME is the angular difference between them.
> >   - We could write out the MAE which would just be the absolute
value of
> > the
> > the ME, but we're just writing NA.
> >   - This is an aggregated wind direction difference.
> >
> > Both are reasonable ways of processing the data - but the
computations
> are
> > slightly different.
> >
> > The real short-coming here is that we're only computing wind
direction
> > errors
> > over points that have already been aggregated through space.  In
> > METv4.1 STAT-Analysis, the only supported job type is "-job
> > aggregate_stat -line_type VL1L2 -out_line_type MPR".
> >
> > To address this short-coming in met-5.0, we added a similar job
that
> > operates
> > over matched pair values instead.  Here's a selection from the
> > met-5.0 release notes:
> >
> > "Enhanced STAT-Analysis by adding support for an aggregate_stat
job type
> > conversion of MPR to WDIR that reads UGRD and VGRD matched pair
lines and
> > computes aggregated wind direction errors. Added
-out_fcst_wind_thresh
> > and -out_obs_wind_thresh options to filter the U/V pairs by wind
speed
> > prior
> > to computing wind direction errors."
> >
> > For example, the following job would compute wind direction errors
for
> each
> > station individually...
> >   -job aggregate_wdir -line_type MPR -out_line_type WDIR -by
> > OBS_SID -out_fcst_wind_thresh \>+2.0
> >
> > That'll compute wind direction errors for each unique station name
in the
> > OBS_SID column of the MPR line, but only when the forecast wind
speed
> > exceeds
> > 2 m/s.
> >
> > This should make it easier to look at wind direction errors
> > station-by-station, rather than have to aggregate over a much
larger
> area.
> >
> > Please let me know if this answers your questions.  If not, I'll
probably
> > refer you to a scientist here to address the interpretation of the
> output.
> > I can help you running the tools and describing the options, but I
think
> a
> > scientist would do a better job helping you interpret the results.
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > John -
> > >
> > > Thanks for the update. I will be out Friday and Monday,
returning on
> > > Tuesday.
> > > Have a great weekend!
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, September 25, 2014 8:54 AM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind
direction
> > > verification (UNCLASSIFIED)
> > >
> > > John
> > >
> > > I'm out of the office today, but will take a close look at this
on
> > Friday.
> > >
> > > Thanks
> > > John
> > >
> > > On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > <javascript:;>
> > > >        Queue: met_help
> > > >      Subject: Stat-Analysis wind direction verification
> (UNCLASSIFIED)
> > > >        Owner: Nobody
> > > >   Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > > >       Status: new
> > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > I need to know more specifically how wind direction errors are
> > > calculated.
> > > > I
> > > > ran a case using METV4.1 using a forecast and observations and
I
> > > > followed the guidance to assure that the Point-Stat VL1L2
lines were
> > > > generated. When I ran Stat-Analysis I specified the job as
> > > > aggregate_stat, the -line_type as
> > > > VL1L2 and the -out_line_type as WDIR. The attached document
provides
> > > > samples of the output and my attempts to calculate the errors
by
> > > > hand so I could try to understand how MET calculates them. I
also
> > > > highlighted the questions I have in the document. I would
appreciate
> > > > having answers to my questions if possible. Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>

------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Oct 02 13:49:37 2014

Classification: UNCLASSIFIED
Caveats: NONE

John -

Thanks for clarifying that.

I had a couple of questions from Dr. Brian Reen also on our team:

So, from this I have two questions for possible inclusion in the email
you
noted you were planning on sending to John Halley Gotway:

1) Anytime that you subtract observed wind direction from model wind
direction
(e.g., for AGGR_WIND, ROW_MEAN_WDIR, and the new capability to
calculate wind
direction error for individual obs) and the result is less than -180
or
greater than +180 do you add 360 in the former case and subtract 360
in the
latter case?

2) With the new capability to calculate wind direction errors at
individual
stations, is it now possible to calculate mean wind direction errors
based on
these individual stations wind direction errors? In other words, if at
12 UTC
we have station A with a wind direction error of +50 and station B
with a wind
direction error of -70, will MET tell us that the mean wind direction
error
is -20 degrees and the mean absolute wind direction error is 60
degrees?

Brian


-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, October 02, 2014 1:45 PM
To: Raby, John W CIV USARMY ARL (US)
Cc: bgb at ucar.edu
Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
verification (UNCLASSIFIED)

John,

Here's the contents of the VL1L2 line: UFBAR, VFBAR, UOBAR, VOBAR,
UVFOBAR,
UVFFBAR, UVOOBAR

Yes, the forecast average U and V (UFBAR and VFBAR) are used in
computing the
forecast wind speed and direction.  While UOBAR and VOBAR are used in
computing the observed wind speed and direction.  And you're wondering
how the
UVFOBAR, UVFFBAR, and UVOOBAR data is used.  The answer is that
they're not
currently being used in the computation any statistics!

The MET software was derived from the NCEP verification tools which
include
the VL1L2 output line.  I spoke with statisticians here, and we
believe that
NCEP uses the data in those 3 columns to derive a combined U and V
vector-based RMSE score.  FVS is the name of the plotting tools
associated
with that package, and we'll check it's documentation to see if/how
NCEP is
using those values to derive statistics.

If so, we could potentially add that capability to MET, deriving
vector-based
scores.  And I'll forward whatever equations we find.

Thanks for the question, and we'll let you know what we turn up.

Thanks,
John

On Wed, Oct 1, 2014 at 10:13 AM, Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Hi John.  John is currently out of the office, but will hopefully be
> able to respond to your questions later this week.  I do not have
> enough knowledge to answer your questions.
>
> Thanks,
> Julie
>
> On Wed, Oct 1, 2014 at 10:09 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for your detailed explanations for my questions. I think it
> > confirmed how the Stat-Analysis output is generated for me. Now it
> > raises some questions about how the vector partial sums in the
> > Point-Stat VL1L2 line type are generated.
> >
> > I was able to trace how the partial sums UFBAR, VFBAR and UOBAR,
> > VOBAR
> were
> > generated from the UGRD, VGRD matched pair data in the MPR file in
a
> > spreadsheet. I simply took the mean of all 103 values of the UGRD
> > and
> VGRD
> > for
> > the Z10 level and the results were identical. For example: for
> > UFBAR, I took the average of all 103 forecast values of UGRD and
the
> > result matched the value for UFBAR in the VL1L2 output. From your
> > answer below, I gather
> that
> > you
> > use the code "convert_u_v_to_wdir" to convert UFBAR, VFBAR to
> > generate
> the
> > forecast wind direction (FBAR in the Stat-Analysis output) and the
> > same code is used to convert UOBAR, VOBAR to generate the observed
> > wind direction (OBAR in the Stat-analysis output). Correct so far?
> >
> > So, based on your answer and the above I can trace the data in the
> > MPR
> file
> > all the way through the Stat-Analysis output of FBAR, OBAR, ME and
MAE.
> The
> > only remaining question, I have is about the vector partial sums:
> UVFOBAR,
> > UVFFBAR and UVOOBAR. I looked in the User's Guide where the VL1L2
> > line type is described and I see the formulas for how these are
> > calculated, but I
> don't
> > see
> > where these partial sums are used for generating the values in the
> > Stat-Analysis output. How are these partial sums used?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, September 26, 2014 1:05 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> > verification (UNCLASSIFIED)
> >
> > John,
> >
> > I read through the document you sent about the output of wind
> > direction errors and will respond to your questions below...
> >
> > I see that you've run a single VL1L2 line (computed over 103
matched
> pairs)
> > through STAT-Analysis to compute wind direction errors.  In the
> > output
> you
> > see
> > the same values for FBAR (94.25) and OBAR (89.09) repeated in both
> > lines (ROW_MEAN_WDIR and AGGR_WDIR) of the output.  You see that
> > because this job is not actually aggregating multiple lines
> > together.  It's just being run over a single VL1L2 line.  If your
> > job was run over multiple
> > VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR and
> > AGGR_WDIR lines would differ.
> >
> > Your first question is - how is the MAE value calculated?  The
FBAR
> > and OBAR columns contain the forecast and observed wind
directions,
> > in degrees.
> The
> > error is the directed difference between those angles (f - o).
The
> > mean error
> > (ME) is the average of those angular differences. The range of
this
> column
> > is -180 to +180.  The MAE is the average of the *absolute value*
of
> > those angular differences.  So the range of the MAE column is 0 to
+180.
> >
> > You're wondering where in the MET code this logic is implemented.
> > In the file "src/basic/vx_math/angles.cc" the function named
> > "convert_u_v_to_wdir"
> > handles the conversion of the U and V components to wind
direction.
> Also,
> > the
> > file "src/tools/core/stat_analysis/stat_analysis_job.cc" contains
> > the function "write_job_aggr_wind" which creates the output for
> > STAT-Analysis.
> >
> > Your next question is why you the numbers are the same in
> > ROW_MEAN_WDIR
> And
> > AGGR_WDIR - again, that's because you're running over a single
VL1L2
> line.
> > If you run on multiple, they will likely differ.  It's difficult
to
> > describe the difference with words, but I'll give it a shot.
> > Suppose you're
> running
> > this job over 10 different VL1L2 lines that represent output for
10
> > different days over the same region.  And let's suppose 5 days had
> > 50 stations reporting and 5 days had 100 stations reporting...
> >
> > - For ROW_MEAN_WDIR, we do the following...
> >   - For each VL1L2 line (meaning, for each day), compute the
average
> > forecast and observation direction and compute the angular
> > difference.
> >   - Compute the average of those 10 angular differences and write
it
> > in the ME column.  Also, put the average of the absolute value of
> > those differences in the MAE column.
> >   - This is the daily average of the wind direction errors.  Even
> > though some of the days were computed over 50 matched pairs and
> > others were computed over 100.
> >
> > - For the AGGR_WDIR, aggregate together those 10 different VL1L2
> > lines into a single VL1L2 line.  When we aggregate VL1L2 lines, we
> > do a weighted average, weighted by the "TOTAL" columns.
Relatively
> > speaking, those 5 days with
> a
> > 100
> > matched pairs will count twice as much as the other 5 days with
only
> > 50 matched pairs.
> >   - From that single aggregated VL1L2 line, compute forecast and
> > observation directions.  The ME is the angular difference between
> > them.
> >   - We could write out the MAE which would just be the absolute
> > value of the the ME, but we're just writing NA.
> >   - This is an aggregated wind direction difference.
> >
> > Both are reasonable ways of processing the data - but the
> > computations
> are
> > slightly different.
> >
> > The real short-coming here is that we're only computing wind
> > direction errors over points that have already been aggregated
> > through space.  In
> > METv4.1 STAT-Analysis, the only supported job type is "-job
> > aggregate_stat -line_type VL1L2 -out_line_type MPR".
> >
> > To address this short-coming in met-5.0, we added a similar job
that
> > operates over matched pair values instead.  Here's a selection
from
> > the
> > met-5.0 release notes:
> >
> > "Enhanced STAT-Analysis by adding support for an aggregate_stat
job
> > type conversion of MPR to WDIR that reads UGRD and VGRD matched
pair
> > lines and computes aggregated wind direction errors. Added
> > -out_fcst_wind_thresh and -out_obs_wind_thresh options to filter
the
> > U/V pairs by wind speed prior to computing wind direction errors."
> >
> > For example, the following job would compute wind direction errors
> > for
> each
> > station individually...
> >   -job aggregate_wdir -line_type MPR -out_line_type WDIR -by
OBS_SID
> > -out_fcst_wind_thresh \>+2.0
> >
> > That'll compute wind direction errors for each unique station name
> > in the OBS_SID column of the MPR line, but only when the forecast
> > wind speed exceeds
> > 2 m/s.
> >
> > This should make it easier to look at wind direction errors
> > station-by-station, rather than have to aggregate over a much
larger
> area.
> >
> > Please let me know if this answers your questions.  If not, I'll
> > probably refer you to a scientist here to address the
interpretation
> > of the
> output.
> > I can help you running the tools and describing the options, but I
> > think
> a
> > scientist would do a better job helping you interpret the results.
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > John -
> > >
> > > Thanks for the update. I will be out Friday and Monday,
returning
> > > on Tuesday.
> > > Have a great weekend!
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, September 25, 2014 8:54 AM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind
direction
> > > verification (UNCLASSIFIED)
> > >
> > > John
> > >
> > > I'm out of the office today, but will take a close look at this
on
> > Friday.
> > >
> > > Thanks
> > > John
> > >
> > > On Wednesday, September 24, 2014, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > <javascript:;>
> > > >        Queue: met_help
> > > >      Subject: Stat-Analysis wind direction verification
> (UNCLASSIFIED)
> > > >        Owner: Nobody
> > > >   Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > > >       Status: new
> > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > I need to know more specifically how wind direction errors are
> > > calculated.
> > > > I
> > > > ran a case using METV4.1 using a forecast and observations and
I
> > > > followed the guidance to assure that the Point-Stat VL1L2
lines
> > > > were generated. When I ran Stat-Analysis I specified the job
as
> > > > aggregate_stat, the -line_type as
> > > > VL1L2 and the -out_line_type as WDIR. The attached document
> > > > provides samples of the output and my attempts to calculate
the
> > > > errors by hand so I could try to understand how MET calculates
> > > > them. I also highlighted the questions I have in the document.
I
> > > > would appreciate having answers to my questions if possible.
Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research Research Applications
> Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>


Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: Stat-Analysis wind direction verification (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Oct 03 10:09:17 2014

John,

The directed difference between two angles is computed in MET by the
function named "angle_difference" in the file
src/basic/vx_math/angles.cc.
It ultimately calls the "atan2d" function (arctangent) which returns
values
between -180 and 180.  So angle differences reported by stat-analysis
should always be in that range.

And I believe the answer to the second question is yes.  To test this
out I
ran the following 2 jobs on the output of the MET test scripts:

bin/stat_analysis -lookin out/point_stat \
-job aggregate_stat -line_type MPR -out_line_type WDIR \
-vx_mask LMV -interp_mthd UW_MEAN \
-column_str OBS_SID KUNO,KSDF -by OBS_SID

This says pick compute wind direction errors for the 2 stations
listed.
And here's the output:
           COL_NAME: OBS_SID TOTAL      FBAR      OBAR       ME
MAE
ROW_MEAN_WDIR: KSDF        1 183.92113 188.74616 -4.82503 4.82503
          AGGR_WDIR: KSDF        1 183.92113 188.74616 -4.82503
NA
ROW_MEAN_WDIR: KUNO        1 165.37725 159.94391  5.43334 5.43334
         AGGR_WDIR: KUNO        1 165.37725 159.94391  5.43334      NA

KSDF has a ME of -4.825 degrees and KUNO has a ME of 5.4333 degrees.

Then I ran this job, removing the "-by OBS_SID" option:
bin/stat_analysis -lookin out/point_stat \
-job aggregate_stat -line_type MPR -out_line_type WDIR \
-vx_mask LMV -interp_mthd UW_MEAN \
-column_str OBS_SID KUNO,KSDF

            COL_NAME: TOTAL      FBAR      OBAR      ME     MAE
ROW_MEAN_WDIR:     2 174.64919 174.34503 0.30416 5.12919
          AGGR_WDIR:     2 170.76076 167.94922 2.81154      NA

The ME in the ROW_MEAN_WDIR is now listed as 0.304 which is the
average of
the two.   So if you compute wind direction errors station by station,
the
ROW_MEAN_WDIR line will contain the average of the errors across the
stations.

Thanks,
John





On Thu, Oct 2, 2014 at 1:49 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for clarifying that.
>
> I had a couple of questions from Dr. Brian Reen also on our team:
>
> So, from this I have two questions for possible inclusion in the
email you
> noted you were planning on sending to John Halley Gotway:
>
> 1) Anytime that you subtract observed wind direction from model wind
> direction
> (e.g., for AGGR_WIND, ROW_MEAN_WDIR, and the new capability to
calculate
> wind
> direction error for individual obs) and the result is less than -180
or
> greater than +180 do you add 360 in the former case and subtract 360
in the
> latter case?
>
> 2) With the new capability to calculate wind direction errors at
individual
> stations, is it now possible to calculate mean wind direction errors
based
> on
> these individual stations wind direction errors? In other words, if
at 12
> UTC
> we have station A with a wind direction error of +50 and station B
with a
> wind
> direction error of -70, will MET tell us that the mean wind
direction error
> is -20 degrees and the mean absolute wind direction error is 60
degrees?
>
> Brian
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, October 02, 2014 1:45 PM
> To: Raby, John W CIV USARMY ARL (US)
> Cc: bgb at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind direction
> verification (UNCLASSIFIED)
>
> John,
>
> Here's the contents of the VL1L2 line: UFBAR, VFBAR, UOBAR, VOBAR,
UVFOBAR,
> UVFFBAR, UVOOBAR
>
> Yes, the forecast average U and V (UFBAR and VFBAR) are used in
computing
> the
> forecast wind speed and direction.  While UOBAR and VOBAR are used
in
> computing the observed wind speed and direction.  And you're
wondering how
> the
> UVFOBAR, UVFFBAR, and UVOOBAR data is used.  The answer is that
they're not
> currently being used in the computation any statistics!
>
> The MET software was derived from the NCEP verification tools which
include
> the VL1L2 output line.  I spoke with statisticians here, and we
believe
> that
> NCEP uses the data in those 3 columns to derive a combined U and V
> vector-based RMSE score.  FVS is the name of the plotting tools
associated
> with that package, and we'll check it's documentation to see if/how
NCEP is
> using those values to derive statistics.
>
> If so, we could potentially add that capability to MET, deriving
> vector-based
> scores.  And I'll forward whatever equations we find.
>
> Thanks for the question, and we'll let you know what we turn up.
>
> Thanks,
> John
>
> On Wed, Oct 1, 2014 at 10:13 AM, Julie Prestopnik via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> >
> > Hi John.  John is currently out of the office, but will hopefully
be
> > able to respond to your questions later this week.  I do not have
> > enough knowledge to answer your questions.
> >
> > Thanks,
> > Julie
> >
> > On Wed, Oct 1, 2014 at 10:09 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098 >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > John -
> > >
> > > Thanks for your detailed explanations for my questions. I think
it
> > > confirmed how the Stat-Analysis output is generated for me. Now
it
> > > raises some questions about how the vector partial sums in the
> > > Point-Stat VL1L2 line type are generated.
> > >
> > > I was able to trace how the partial sums UFBAR, VFBAR and UOBAR,
> > > VOBAR
> > were
> > > generated from the UGRD, VGRD matched pair data in the MPR file
in a
> > > spreadsheet. I simply took the mean of all 103 values of the
UGRD
> > > and
> > VGRD
> > > for
> > > the Z10 level and the results were identical. For example: for
> > > UFBAR, I took the average of all 103 forecast values of UGRD and
the
> > > result matched the value for UFBAR in the VL1L2 output. From
your
> > > answer below, I gather
> > that
> > > you
> > > use the code "convert_u_v_to_wdir" to convert UFBAR, VFBAR to
> > > generate
> > the
> > > forecast wind direction (FBAR in the Stat-Analysis output) and
the
> > > same code is used to convert UOBAR, VOBAR to generate the
observed
> > > wind direction (OBAR in the Stat-analysis output). Correct so
far?
> > >
> > > So, based on your answer and the above I can trace the data in
the
> > > MPR
> > file
> > > all the way through the Stat-Analysis output of FBAR, OBAR, ME
and MAE.
> > The
> > > only remaining question, I have is about the vector partial
sums:
> > UVFOBAR,
> > > UVFFBAR and UVOOBAR. I looked in the User's Guide where the
VL1L2
> > > line type is described and I see the formulas for how these are
> > > calculated, but I
> > don't
> > > see
> > > where these partial sums are used for generating the values in
the
> > > Stat-Analysis output. How are these partial sums used?
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, September 26, 2014 1:05 PM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind
direction
> > > verification (UNCLASSIFIED)
> > >
> > > John,
> > >
> > > I read through the document you sent about the output of wind
> > > direction errors and will respond to your questions below...
> > >
> > > I see that you've run a single VL1L2 line (computed over 103
matched
> > pairs)
> > > through STAT-Analysis to compute wind direction errors.  In the
> > > output
> > you
> > > see
> > > the same values for FBAR (94.25) and OBAR (89.09) repeated in
both
> > > lines (ROW_MEAN_WDIR and AGGR_WDIR) of the output.  You see that
> > > because this job is not actually aggregating multiple lines
> > > together.  It's just being run over a single VL1L2 line.  If
your
> > > job was run over multiple
> > > VL1L2 lines, it's likely that the numbers in the ROW_MEAN_WDIR
and
> > > AGGR_WDIR lines would differ.
> > >
> > > Your first question is - how is the MAE value calculated?  The
FBAR
> > > and OBAR columns contain the forecast and observed wind
directions,
> > > in degrees.
> > The
> > > error is the directed difference between those angles (f - o).
The
> > > mean error
> > > (ME) is the average of those angular differences. The range of
this
> > column
> > > is -180 to +180.  The MAE is the average of the *absolute value*
of
> > > those angular differences.  So the range of the MAE column is 0
to
> +180.
> > >
> > > You're wondering where in the MET code this logic is
implemented.
> > > In the file "src/basic/vx_math/angles.cc" the function named
> > > "convert_u_v_to_wdir"
> > > handles the conversion of the U and V components to wind
direction.
> > Also,
> > > the
> > > file "src/tools/core/stat_analysis/stat_analysis_job.cc"
contains
> > > the function "write_job_aggr_wind" which creates the output for
> > > STAT-Analysis.
> > >
> > > Your next question is why you the numbers are the same in
> > > ROW_MEAN_WDIR
> > And
> > > AGGR_WDIR - again, that's because you're running over a single
VL1L2
> > line.
> > > If you run on multiple, they will likely differ.  It's difficult
to
> > > describe the difference with words, but I'll give it a shot.
> > > Suppose you're
> > running
> > > this job over 10 different VL1L2 lines that represent output for
10
> > > different days over the same region.  And let's suppose 5 days
had
> > > 50 stations reporting and 5 days had 100 stations reporting...
> > >
> > > - For ROW_MEAN_WDIR, we do the following...
> > >   - For each VL1L2 line (meaning, for each day), compute the
average
> > > forecast and observation direction and compute the angular
> > > difference.
> > >   - Compute the average of those 10 angular differences and
write it
> > > in the ME column.  Also, put the average of the absolute value
of
> > > those differences in the MAE column.
> > >   - This is the daily average of the wind direction errors.
Even
> > > though some of the days were computed over 50 matched pairs and
> > > others were computed over 100.
> > >
> > > - For the AGGR_WDIR, aggregate together those 10 different VL1L2
> > > lines into a single VL1L2 line.  When we aggregate VL1L2 lines,
we
> > > do a weighted average, weighted by the "TOTAL" columns.
Relatively
> > > speaking, those 5 days with
> > a
> > > 100
> > > matched pairs will count twice as much as the other 5 days with
only
> > > 50 matched pairs.
> > >   - From that single aggregated VL1L2 line, compute forecast and
> > > observation directions.  The ME is the angular difference
between
> > > them.
> > >   - We could write out the MAE which would just be the absolute
> > > value of the the ME, but we're just writing NA.
> > >   - This is an aggregated wind direction difference.
> > >
> > > Both are reasonable ways of processing the data - but the
> > > computations
> > are
> > > slightly different.
> > >
> > > The real short-coming here is that we're only computing wind
> > > direction errors over points that have already been aggregated
> > > through space.  In
> > > METv4.1 STAT-Analysis, the only supported job type is "-job
> > > aggregate_stat -line_type VL1L2 -out_line_type MPR".
> > >
> > > To address this short-coming in met-5.0, we added a similar job
that
> > > operates over matched pair values instead.  Here's a selection
from
> > > the
> > > met-5.0 release notes:
> > >
> > > "Enhanced STAT-Analysis by adding support for an aggregate_stat
job
> > > type conversion of MPR to WDIR that reads UGRD and VGRD matched
pair
> > > lines and computes aggregated wind direction errors. Added
> > > -out_fcst_wind_thresh and -out_obs_wind_thresh options to filter
the
> > > U/V pairs by wind speed prior to computing wind direction
errors."
> > >
> > > For example, the following job would compute wind direction
errors
> > > for
> > each
> > > station individually...
> > >   -job aggregate_wdir -line_type MPR -out_line_type WDIR -by
OBS_SID
> > > -out_fcst_wind_thresh \>+2.0
> > >
> > > That'll compute wind direction errors for each unique station
name
> > > in the OBS_SID column of the MPR line, but only when the
forecast
> > > wind speed exceeds
> > > 2 m/s.
> > >
> > > This should make it easier to look at wind direction errors
> > > station-by-station, rather than have to aggregate over a much
larger
> > area.
> > >
> > > Please let me know if this answers your questions.  If not, I'll
> > > probably refer you to a scientist here to address the
interpretation
> > > of the
> > output.
> > > I can help you running the tools and describing the options, but
I
> > > think
> > a
> > > scientist would do a better job helping you interpret the
results.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Sep 25, 2014 at 9:24 AM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
>
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > John -
> > > >
> > > > Thanks for the update. I will be out Friday and Monday,
returning
> > > > on Tuesday.
> > > > Have a great weekend!
> > > >
> > > > R/
> > > > John
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > > Sent: Thursday, September 25, 2014 8:54 AM
> > > > To: Raby, John W CIV USARMY ARL (US)
> > > > Subject: Re: [rt.rap.ucar.edu #69098] Stat-Analysis wind
direction
> > > > verification (UNCLASSIFIED)
> > > >
> > > > John
> > > >
> > > > I'm out of the office today, but will take a close look at
this on
> > > Friday.
> > > >
> > > > Thanks
> > > > John
> > > >
> > > > On Wednesday, September 24, 2014, Raby, John W USA CIV via RT
<
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Sep 24 16:37:16 2014: Request 69098 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > <javascript:;>
> > > > >        Queue: met_help
> > > > >      Subject: Stat-Analysis wind direction verification
> > (UNCLASSIFIED)
> > > > >        Owner: Nobody
> > > > >   Requestors: john.w.raby2.civ at mail.mil <javascript:;>
> > > > >       Status: new
> > > > >  Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69098
> > > > > >
> > > > >
> > > > >
> > > > > Classification: UNCLASSIFIED
> > > > > Caveats: NONE
> > > > >
> > > > > I need to know more specifically how wind direction errors
are
> > > > calculated.
> > > > > I
> > > > > ran a case using METV4.1 using a forecast and observations
and I
> > > > > followed the guidance to assure that the Point-Stat VL1L2
lines
> > > > > were generated. When I ran Stat-Analysis I specified the job
as
> > > > > aggregate_stat, the -line_type as
> > > > > VL1L2 and the -out_line_type as WDIR. The attached document
> > > > > provides samples of the output and my attempts to calculate
the
> > > > > errors by hand so I could try to understand how MET
calculates
> > > > > them. I also highlighted the questions I have in the
document. I
> > > > > would appreciate having answers to my questions if possible.
> Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > >
> > > > > Classification: UNCLASSIFIED
> > > > > Caveats: NONE
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Julie Prestopnik
> > National Center for Atmospheric Research Research Applications
> > Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>

------------------------------------------------