[Met_help] [rt.rap.ucar.edu #91105] History for FW: Moore convection path forward (UNCLASSIFIED)

John Halley Gotway via RT met_help at ucar.edu
Thu Jul 18 09:45:38 MDT 2019


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

CLASSIFICATION: UNCLASSIFIED



-----Original Message-----
From: Reen, Brian P CIV USARMY CCDC ARL (USA) 
Sent: Tuesday, July 16, 2019 1:34 PM
To: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
<jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY FUTURES COMMAND
(USA) <robert.e.dumais.civ at mail.mil>; Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
Subject: RE: Moore convection path forward (UNCLASSIFIED)

CLASSIFICATION: UNCLASSIFIED

Huaqing,
Thank you for the additional details.  If I understand correctly, F_RATE is
the forecast rate, i.e., the number of neighborhoods where the forecast
exceeded the threshold, and similarly for O_RATE, but for observations.  

I am trying to think through what added value there is to reported bias for
a particular neighborhood size rather than the overall bias (which I think
would be equivalent to a neighborhood size of 1).  Do you see benefits to
reporting bias for a particular neighborhood size, rather than reporting the
overall bias?

The issue regarding the neighborhood size dependence of the bias was not
what I was trying to ask about in my previous email.  What I was asking can
be illustrated using a case with two times.

Time 1:
F_RATE(t=1) = 5
O_RATE(t=1) = 1
BIAS(t=1) = 5

Time 2:
F_RATE(t=2) = 50
O_RATE(t=2) = 49
BIAS(t=2) = ~1.02

Now, if we wanted the overall bias, my question is whether we would
calculate it via:
BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 
= [ 5 + 1.02 ] / 2 
= ~3.01

Or via:
BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [ O_RATE(t=1)+O_RATE(t=2) ] 
= [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1

Brian

-----Original Message-----
From: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
<jeffrey.a.smith1.civ at mail.mil> 
Sent: Tuesday, July 16, 2019 12:01 PM
To: Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
<robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL (USA)
<brian.p.reen.civ at mail.mil>; Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
Subject: RE: Moore convection path forward (UNCLASSIFIED)

All.

Couple of points relating to Brians summary below, and a few related things.

Once I am back from TDY I'll rebuild the data sets.  I need to get back to
that anyway, and I know the solution, just haven't had time.  What that will
do is give us the data in R package form that anyone can use for analysis.
Don't have to worry about all the various files, etc.  

Once I have that, I can take what I have already done in R and reproduce the
data sets using the packaged data as a starting point.  What that means is
then if we want to give this analysis to someone else, or reproduce it
later, the combination of the data package, and the code in package form,
should be sufficient to accomplish that.

My next step after rebuilding these data sets, is to make the code
executable on the HPC.  For most of the data sets I've seen, I think I know
enough to make the data package prep process automatable as a step after
what Leelinda has done.  The package will likely look like three sets of
data, one forecast, one obs, and one bias (because its easy to generate).
I'll right some extractors that will pull specific data variables from the
sets since from an abstract point of view, each will look exactly the same.

For the BIAS calculation that Huaqing provides below, I've already done that
in some test cases that John has provided.  Its actually easy to add things
like that in R with some of the tools R brings to the table since
calculations like that are operations on atomic data.  

Regards,

Jeffrey 


-----Original Message-----
From: Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil> 
Sent: Tuesday, July 16, 2019 9:34 AM
To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
<robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL (USA)
<brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
<jeffrey.a.smith1.civ at mail.mil>; Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
Subject: RE: Moore convection path forward (UNCLASSIFIED)

HI, Brian, Bob, John and Jeffrey,

Regarding to Brian's question below, although I am not 100% sure what he was
asking, here is my attempt trying to answer it. Let me know if it make any
sense to you.

BIAS is a standard parameter coming from calculations related to CSI (e.g.,
POD, FAR, BIAS, and CSI). Theoretically, BIAS has nothing to do with FSS and
its associated neighborhood sizes, but it is a function of threshold. When
we show FSS and BIAS, we basically are using two kinds of verification
methods, that is, the CSI and the FSS method. 

When I calculated BIAS from txt files containing FSS results, I used O_RATE
and F_RATE. Ideally BIAS=F_RATE/O_RATE, and BIAS should not change with
different neighborhood sizes. Unfortunately, because of the edge effects,
the number of matched pairs decreases with increased neighborhood sizes.
This edge effects caused BIAS changes with neighborhood size slightly for
small neighborhood sizes, but significantly for very large neighborhood
sizes. Fortunately, what we want for BIAS is the small neighborhood sizes,
which are very similar for many small neighborhood sizes (one example is
BIAS changed from 0.51 to 0.44 for neighborhood size from 1 to 11 grid
sizes).  So just keep in mind that the BIAS we shown was the BIAS for that
neighborhood size. We can change the MET setting and/or my calculation of
BIAS so that we always use the smallest neighborhood size for BIAS, if
necessary. 

Hope this helps.

Huaqing



-----Original Message-----
From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA) 
Sent: Monday, July 15, 2019 10:57 AM
To: Reen, Brian P CIV USARMY CCDC ARL (USA) <brian.p.reen.civ at mail.mil>;
Smith, Jeffrey A CIV USARMY CCDC ARL (USA) <jeffrey.a.smith1.civ at mail.mil>;
Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>; Raby,
John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: RE: Moore convection path forward (UNCLASSIFIED)

Hi again Brian,

  I was off Friday, so just finally getting to respond to all the messages
from that day. I believe you captured the details of our discussions very
well. For the purpose of the DANA document, the completion of the main body
of verification work for WRE-N ensemble DC and Moore hybrid DA will remain
as FY19 accomplishment targets (I don't think Ben wants us to modify the
FY19 accomplishments already listed in the document), but I will allow for
draft tech report and/or open lit publication efforts with these studies to
carry into FY20 (almost certain to be the case). As to one thing Brian
asked, and I am not sure he has received an answer:

"I am not sure if we discussed this but with bias do we know if the standard
practice is to weight each time-model run pair equally, or does one weight
by the number of neighborhoods where the obs exceeded the threshold?"


 
Bob



-----Original Message-----
From: Reen, Brian P CIV USARMY CCDC ARL (USA) 
Sent: Friday, July 12, 2019 11:47 AM
To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
<robert.e.dumais.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
<jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>; Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
Subject: Moore convection path forward (UNCLASSIFIED)

CLASSIFICATION: UNCLASSIFIED

All,
Based on our discussions yesterday, the following is my understanding of the
current status and the path forward.

Verification of radar data:
MET has been run to do the verification on column maximum reflectivity but
the output still needs to be aggregated and plotted.  Jeffrey has done work
towards this.  The goal is to aggregate per experiment by lead time and per
experiment by actual time for FSS and bias.  Since FSS requires specifying a
neighborhood size and a dBZ threshold, and there are many such possible
combinations we will need to figure out a very small number of these to
focus on.  Additionally, there are questions about how to aggregate FSS
properly.  It is not clear if this can be done from the MET output files we
already have or if this will need to be done via MET stat-analysis.  I am
not sure if we discussed this but with bias do we know if the standard
practice is to weight each time-model run pair equally, or does one weight
by the number of neighborhoods where the obs exceeded the threshold? 

Action: Huaqing Cai and John Raby will contact John Gotway to better
understand how to aggregate FSS.
Action: Huaqing Cai will create plots showing the variation in FSS by
neighborhood size and dBZ threshold to help determine which combinations of
these we wish to examine.

When it is understood how to aggregate FSS and we determine what
neighborhood size / dBZ threshold combinations we will be examining, someone
will need to do the aggregation and create the desired plots.  After the
plots are created, analysis of the results can proceed. 

Verification of surface data:
This has been completed and plots are available on the WSMR version of the
branch drive.  (This drive is not automatically mapped at ALC but can be
accessed and setup to automatically map via its name
[\\arlwa7001546h.arl.army.mil]). The location on that drive of the data is:
\hybrid vLAPS nudging project\plots_old
The data on which these plots are based are available at:
Z:\hybrid vLAPS nudging project\tables_old
My understanding is that the directories end with _old because Jeffrey has
been hoping to generate these via a new setup that is not quite ready, but
that the data in these plots and tables is correct.  The plots available
show data aggregated per experiment by lead time and actual time, as well as
of individual experiments.  There is a lot of information available in these
plots.  For now the plan is that the surface data verification will be used
to help interpret the verification of the radar data.  

Please let me know if there is something that I did not accurately convey or
if there are aspects that I missed.  Thank you John for your verification
efforts and for Jeffrey for all of your work in taking the output from MET
and converting that into very helpful graphics, and your work in creating R
code to assist with use of MET output.

Brian



-----Original Message-----
From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
<robert.e.dumais.civ at mail.mil> 
Sent: Thursday, July 11, 2019 10:21 AM
To: Reen, Brian P CIV USARMY CCDC ARL (USA) <brian.p.reen.civ at mail.mil>;
Smith, Jeffrey A CIV USARMY CCDC ARL (USA) <jeffrey.a.smith1.civ at mail.mil>;
Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>; Raby,
John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Confirmed- meet in small conference room at 1330 MDT for telecon
regarding Moore convection/DC ensemble studies

All,

   Gina confirmed that we can use the small conference room for our telecon
today at 1330 MDT (1530 EDT).  We don't really need the VTC- just a regular
dial in connection for Brian should be good enough. We can either dial into
Brian's office number or vica-versa. I say we just dial into his number. 

	
Bob


CLASSIFICATION: UNCLASSIFIED


CLASSIFICATION: UNCLASSIFIED
CLASSIFICATION: UNCLASSIFIED


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: FW: Moore convection path forward (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Jul 16 16:09:56 2019

John,

As we discussed in our telecon today, stat_analysis supports both
methods
of summarizing the results across multiple cases that Brian
referenced:

BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2
= [ 5 + 1.02 ] / 2
= ~3.01

In STAT-Analysis, this is done with a *summary* job type:
   -job summary -line_type CTS -column FBIAS -by
MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
The output summary columns include the mean (i.e. average) of the
input
column of data.

Or via:
BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
O_RATE(t=1)+O_RATE(t=2) ]
= [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1

In STAT-Analysis, this is done using an *aggregate_stat* job type:
   -job aggregate_stat -line_type CTC -out_line_type CTS -by
MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
The input CTC counts and summed up and the output CTS statistics are
computed from those counts.  Results are reported separately for each
unique combination of the values found in the columns listed for the
"-by"
option.

Thanks,
John

On Tue, Jul 16, 2019 at 3:28 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> Tue Jul 16 15:19:37 2019: Request 91105 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>        Queue: met_help
>      Subject: FW: Moore convection path forward (UNCLASSIFIED)
>        Owner: Nobody
>   Requestors: john.w.raby2.civ at mail.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91105 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
> -----Original Message-----
> From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> Sent: Tuesday, July 16, 2019 1:34 PM
> To: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC ARL
(USA)
> <huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY FUTURES
COMMAND
> (USA) <robert.e.dumais.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> (USA)
> <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> Huaqing,
> Thank you for the additional details.  If I understand correctly,
F_RATE is
> the forecast rate, i.e., the number of neighborhoods where the
forecast
> exceeded the threshold, and similarly for O_RATE, but for
observations.
>
> I am trying to think through what added value there is to reported
bias for
> a particular neighborhood size rather than the overall bias (which I
think
> would be equivalent to a neighborhood size of 1).  Do you see
benefits to
> reporting bias for a particular neighborhood size, rather than
reporting
> the
> overall bias?
>
> The issue regarding the neighborhood size dependence of the bias was
not
> what I was trying to ask about in my previous email.  What I was
asking can
> be illustrated using a case with two times.
>
> Time 1:
> F_RATE(t=1) = 5
> O_RATE(t=1) = 1
> BIAS(t=1) = 5
>
> Time 2:
> F_RATE(t=2) = 50
> O_RATE(t=2) = 49
> BIAS(t=2) = ~1.02
>
> Now, if we wanted the overall bias, my question is whether we would
> calculate it via:
> BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2
> = [ 5 + 1.02 ] / 2
> = ~3.01
>
> Or via:
> BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
O_RATE(t=1)+O_RATE(t=2) ]
> = [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
>
> Brian
>
> -----Original Message-----
> From: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> <jeffrey.a.smith1.civ at mail.mil>
> Sent: Tuesday, July 16, 2019 12:01 PM
> To: Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>;
> Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL
(USA)
> <brian.p.reen.civ at mail.mil>; Raby, John W CIV USARMY CCDC ARL (USA)
> <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> All.
>
> Couple of points relating to Brians summary below, and a few related
> things.
>
> Once I am back from TDY I'll rebuild the data sets.  I need to get
back to
> that anyway, and I know the solution, just haven't had time.  What
that
> will
> do is give us the data in R package form that anyone can use for
analysis.
> Don't have to worry about all the various files, etc.
>
> Once I have that, I can take what I have already done in R and
reproduce
> the
> data sets using the packaged data as a starting point.  What that
means is
> then if we want to give this analysis to someone else, or reproduce
it
> later, the combination of the data package, and the code in package
form,
> should be sufficient to accomplish that.
>
> My next step after rebuilding these data sets, is to make the code
> executable on the HPC.  For most of the data sets I've seen, I think
I know
> enough to make the data package prep process automatable as a step
after
> what Leelinda has done.  The package will likely look like three
sets of
> data, one forecast, one obs, and one bias (because its easy to
generate).
> I'll right some extractors that will pull specific data variables
from the
> sets since from an abstract point of view, each will look exactly
the same.
>
> For the BIAS calculation that Huaqing provides below, I've already
done
> that
> in some test cases that John has provided.  Its actually easy to add
things
> like that in R with some of the tools R brings to the table since
> calculations like that are operations on atomic data.
>
> Regards,
>
> Jeffrey
>
>
> -----Original Message-----
> From: Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>
> Sent: Tuesday, July 16, 2019 9:34 AM
> To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL
(USA)
> <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
(USA)
> <jeffrey.a.smith1.civ at mail.mil>; Raby, John W CIV USARMY CCDC ARL
(USA)
> <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> HI, Brian, Bob, John and Jeffrey,
>
> Regarding to Brian's question below, although I am not 100% sure
what he
> was
> asking, here is my attempt trying to answer it. Let me know if it
make any
> sense to you.
>
> BIAS is a standard parameter coming from calculations related to CSI
(e.g.,
> POD, FAR, BIAS, and CSI). Theoretically, BIAS has nothing to do with
FSS
> and
> its associated neighborhood sizes, but it is a function of
threshold. When
> we show FSS and BIAS, we basically are using two kinds of
verification
> methods, that is, the CSI and the FSS method.
>
> When I calculated BIAS from txt files containing FSS results, I used
O_RATE
> and F_RATE. Ideally BIAS=F_RATE/O_RATE, and BIAS should not change
with
> different neighborhood sizes. Unfortunately, because of the edge
effects,
> the number of matched pairs decreases with increased neighborhood
sizes.
> This edge effects caused BIAS changes with neighborhood size
slightly for
> small neighborhood sizes, but significantly for very large
neighborhood
> sizes. Fortunately, what we want for BIAS is the small neighborhood
sizes,
> which are very similar for many small neighborhood sizes (one
example is
> BIAS changed from 0.51 to 0.44 for neighborhood size from 1 to 11
grid
> sizes).  So just keep in mind that the BIAS we shown was the BIAS
for that
> neighborhood size. We can change the MET setting and/or my
calculation of
> BIAS so that we always use the smallest neighborhood size for BIAS,
if
> necessary.
>
> Hope this helps.
>
> Huaqing
>
>
>
> -----Original Message-----
> From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> Sent: Monday, July 15, 2019 10:57 AM
> To: Reen, Brian P CIV USARMY CCDC ARL (USA)
<brian.p.reen.civ at mail.mil>;
> Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
<jeffrey.a.smith1.civ at mail.mil
> >;
> Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
Raby,
> John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> Hi again Brian,
>
>   I was off Friday, so just finally getting to respond to all the
messages
> from that day. I believe you captured the details of our discussions
very
> well. For the purpose of the DANA document, the completion of the
main body
> of verification work for WRE-N ensemble DC and Moore hybrid DA will
remain
> as FY19 accomplishment targets (I don't think Ben wants us to modify
the
> FY19 accomplishments already listed in the document), but I will
allow for
> draft tech report and/or open lit publication efforts with these
studies to
> carry into FY20 (almost certain to be the case). As to one thing
Brian
> asked, and I am not sure he has received an answer:
>
> "I am not sure if we discussed this but with bias do we know if the
> standard
> practice is to weight each time-model run pair equally, or does one
weight
> by the number of neighborhoods where the obs exceeded the
threshold?"
>
>
>
> Bob
>
>
>
> -----Original Message-----
> From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> Sent: Friday, July 12, 2019 11:47 AM
> To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
(USA)
> <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC ARL
(USA)
> <huaqing.cai.civ at mail.mil>; Raby, John W CIV USARMY CCDC ARL (USA)
> <john.w.raby2.civ at mail.mil>
> Subject: Moore convection path forward (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> All,
> Based on our discussions yesterday, the following is my
understanding of
> the
> current status and the path forward.
>
> Verification of radar data:
> MET has been run to do the verification on column maximum
reflectivity but
> the output still needs to be aggregated and plotted.  Jeffrey has
done work
> towards this.  The goal is to aggregate per experiment by lead time
and per
> experiment by actual time for FSS and bias.  Since FSS requires
specifying
> a
> neighborhood size and a dBZ threshold, and there are many such
possible
> combinations we will need to figure out a very small number of these
to
> focus on.  Additionally, there are questions about how to aggregate
FSS
> properly.  It is not clear if this can be done from the MET output
files we
> already have or if this will need to be done via MET stat-analysis.
I am
> not sure if we discussed this but with bias do we know if the
standard
> practice is to weight each time-model run pair equally, or does one
weight
> by the number of neighborhoods where the obs exceeded the threshold?
>
> Action: Huaqing Cai and John Raby will contact John Gotway to better
> understand how to aggregate FSS.
> Action: Huaqing Cai will create plots showing the variation in FSS
by
> neighborhood size and dBZ threshold to help determine which
combinations of
> these we wish to examine.
>
> When it is understood how to aggregate FSS and we determine what
> neighborhood size / dBZ threshold combinations we will be examining,
> someone
> will need to do the aggregation and create the desired plots.  After
the
> plots are created, analysis of the results can proceed.
>
> Verification of surface data:
> This has been completed and plots are available on the WSMR version
of the
> branch drive.  (This drive is not automatically mapped at ALC but
can be
> accessed and setup to automatically map via its name
> [\\arlwa7001546h.arl.army.mil]). The location on that drive of the
data
> is:
> \hybrid vLAPS nudging project\plots_old
> The data on which these plots are based are available at:
> Z:\hybrid vLAPS nudging project\tables_old
> My understanding is that the directories end with _old because
Jeffrey has
> been hoping to generate these via a new setup that is not quite
ready, but
> that the data in these plots and tables is correct.  The plots
available
> show data aggregated per experiment by lead time and actual time, as
well
> as
> of individual experiments.  There is a lot of information available
in
> these
> plots.  For now the plan is that the surface data verification will
be used
> to help interpret the verification of the radar data.
>
> Please let me know if there is something that I did not accurately
convey
> or
> if there are aspects that I missed.  Thank you John for your
verification
> efforts and for Jeffrey for all of your work in taking the output
from MET
> and converting that into very helpful graphics, and your work in
creating R
> code to assist with use of MET output.
>
> Brian
>
>
>
> -----Original Message-----
> From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>
> Sent: Thursday, July 11, 2019 10:21 AM
> To: Reen, Brian P CIV USARMY CCDC ARL (USA)
<brian.p.reen.civ at mail.mil>;
> Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
<jeffrey.a.smith1.civ at mail.mil
> >;
> Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
Raby,
> John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> Subject: Confirmed- meet in small conference room at 1330 MDT for
telecon
> regarding Moore convection/DC ensemble studies
>
> All,
>
>    Gina confirmed that we can use the small conference room for our
telecon
> today at 1330 MDT (1530 EDT).  We don't really need the VTC- just a
regular
> dial in connection for Brian should be good enough. We can either
dial into
> Brian's office number or vica-versa. I say we just dial into his
number.
>
>
> Bob
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
> CLASSIFICATION: UNCLASSIFIED
> CLASSIFICATION: UNCLASSIFIED
>
>

------------------------------------------------
Subject: FW: Moore convection path forward (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Jul 17 07:35:58 2019

CLASSIFICATION: UNCLASSIFIED

John -

Thank you so much for taking the time yesterday to discuss the various
ways
that Stat-Analysis can aggregate the statistics for this project. Very
helpful!

You mentioned the hosting of METViewer as a public facing page on
Amazon Web
Services. Could you provide the URL for that?

I would also like to work with my local linux sysadm to see if I can
install
MetViewer on my local workstation. I know it has Java. I need to have
R
installed, but I used to have it and getting it back should be easy to
do.
I've never had MySQL. Do you have a URL to obtain it? Any other
dependencies?

R/
John

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, July 16, 2019 4:10 PM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore
convection
path forward (UNCLASSIFIED)

All active links contained in this email were disabled.  Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.




----

John,

As we discussed in our telecon today, stat_analysis supports both
methods of
summarizing the results across multiple cases that Brian referenced:

BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2 =
~3.01

In STAT-Analysis, this is done with a *summary* job type:
   -job summary -line_type CTS -column FBIAS -by
MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
The output summary columns include the mean (i.e. average) of the
input column
of data.

Or via:
BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
O_RATE(t=1)+O_RATE(t=2) ] =
[ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1

In STAT-Analysis, this is done using an *aggregate_stat* job type:
   -job aggregate_stat -line_type CTC -out_line_type CTS -by
MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
The input CTC counts and summed up and the output CTS statistics are
computed
from those counts.  Results are reported separately for each unique
combination of the values found in the columns listed for the "-by"
option.

Thanks,
John

On Tue, Jul 16, 2019 at 3:28 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> Tue Jul 16 15:19:37 2019: Request 91105 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>        Queue: met_help
>      Subject: FW: Moore convection path forward (UNCLASSIFIED)
>        Owner: Nobody
>   Requestors: john.w.raby2.civ at mail.mil
>       Status: new
>  Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91105 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
> -----Original Message-----
> From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> Sent: Tuesday, July 16, 2019 1:34 PM
> To: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC ARL
> (USA) <huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY
> FUTURES COMMAND
> (USA) <robert.e.dumais.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> (USA)
> <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> Huaqing,
> Thank you for the additional details.  If I understand correctly,
> F_RATE is the forecast rate, i.e., the number of neighborhoods where
> the forecast exceeded the threshold, and similarly for O_RATE, but
for
> observations.
>
> I am trying to think through what added value there is to reported
> bias for a particular neighborhood size rather than the overall bias
> (which I think would be equivalent to a neighborhood size of 1).  Do
> you see benefits to reporting bias for a particular neighborhood
size,
> rather than reporting the overall bias?
>
> The issue regarding the neighborhood size dependence of the bias was
> not what I was trying to ask about in my previous email.  What I was
> asking can be illustrated using a case with two times.
>
> Time 1:
> F_RATE(t=1) = 5
> O_RATE(t=1) = 1
> BIAS(t=1) = 5
>
> Time 2:
> F_RATE(t=2) = 50
> O_RATE(t=2) = 49
> BIAS(t=2) = ~1.02
>
> Now, if we wanted the overall bias, my question is whether we would
> calculate it via:
> BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2 =
> ~3.01
>
> Or via:
> BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
> O_RATE(t=1)+O_RATE(t=2) ] = [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
>
> Brian
>
> -----Original Message-----
> From: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> <jeffrey.a.smith1.civ at mail.mil>
> Sent: Tuesday, July 16, 2019 12:01 PM
> To: Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>;
> Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL
> (USA) <brian.p.reen.civ at mail.mil>; Raby, John W CIV USARMY CCDC ARL
> (USA) <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> All.
>
> Couple of points relating to Brians summary below, and a few related
> things.
>
> Once I am back from TDY I'll rebuild the data sets.  I need to get
> back to that anyway, and I know the solution, just haven't had time.
> What that will do is give us the data in R package form that anyone
> can use for analysis.
> Don't have to worry about all the various files, etc.
>
> Once I have that, I can take what I have already done in R and
> reproduce the data sets using the packaged data as a starting point.
> What that means is then if we want to give this analysis to someone
> else, or reproduce it later, the combination of the data package,
and
> the code in package form, should be sufficient to accomplish that.
>
> My next step after rebuilding these data sets, is to make the code
> executable on the HPC.  For most of the data sets I've seen, I think
I
> know enough to make the data package prep process automatable as a
> step after what Leelinda has done.  The package will likely look
like
> three sets of data, one forecast, one obs, and one bias (because its
easy to
> generate).
> I'll right some extractors that will pull specific data variables
from
> the sets since from an abstract point of view, each will look
exactly the
> same.
>
> For the BIAS calculation that Huaqing provides below, I've already
> done that in some test cases that John has provided.  Its actually
> easy to add things like that in R with some of the tools R brings to
> the table since calculations like that are operations on atomic
data.
>
> Regards,
>
> Jeffrey
>
>
> -----Original Message-----
> From: Cai, Huaqing CIV USARMY CCDC ARL (USA)
> <huaqing.cai.civ at mail.mil>
> Sent: Tuesday, July 16, 2019 9:34 AM
> To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL
> (USA) <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC
> ARL (USA) <jeffrey.a.smith1.civ at mail.mil>; Raby, John W CIV USARMY
> CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> HI, Brian, Bob, John and Jeffrey,
>
> Regarding to Brian's question below, although I am not 100% sure
what
> he was asking, here is my attempt trying to answer it. Let me know
if
> it make any sense to you.
>
> BIAS is a standard parameter coming from calculations related to CSI
> (e.g., POD, FAR, BIAS, and CSI). Theoretically, BIAS has nothing to
do
> with FSS and its associated neighborhood sizes, but it is a function
> of threshold. When we show FSS and BIAS, we basically are using two
> kinds of verification methods, that is, the CSI and the FSS method.
>
> When I calculated BIAS from txt files containing FSS results, I used
> O_RATE and F_RATE. Ideally BIAS=F_RATE/O_RATE, and BIAS should not
> change with different neighborhood sizes. Unfortunately, because of
> the edge effects, the number of matched pairs decreases with
increased
> neighborhood sizes.
> This edge effects caused BIAS changes with neighborhood size
slightly
> for small neighborhood sizes, but significantly for very large
> neighborhood sizes. Fortunately, what we want for BIAS is the small
> neighborhood sizes, which are very similar for many small
neighborhood
> sizes (one example is BIAS changed from 0.51 to 0.44 for
neighborhood
> size from 1 to 11 grid sizes).  So just keep in mind that the BIAS
we
> shown was the BIAS for that neighborhood size. We can change the MET
> setting and/or my calculation of BIAS so that we always use the
> smallest neighborhood size for BIAS, if necessary.
>
> Hope this helps.
>
> Huaqing
>
>
>
> -----Original Message-----
> From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> Sent: Monday, July 15, 2019 10:57 AM
> To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
> (USA) <jeffrey.a.smith1.civ at mail.mil
> >;
> Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
> Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> Hi again Brian,
>
>   I was off Friday, so just finally getting to respond to all the
> messages from that day. I believe you captured the details of our
> discussions very well. For the purpose of the DANA document, the
> completion of the main body of verification work for WRE-N ensemble
DC
> and Moore hybrid DA will remain as FY19 accomplishment targets (I
> don't think Ben wants us to modify the
> FY19 accomplishments already listed in the document), but I will
allow
> for draft tech report and/or open lit publication efforts with these
> studies to carry into FY20 (almost certain to be the case). As to
one
> thing Brian asked, and I am not sure he has received an answer:
>
> "I am not sure if we discussed this but with bias do we know if the
> standard practice is to weight each time-model run pair equally, or
> does one weight by the number of neighborhoods where the obs
exceeded
> the threshold?"
>
>
>
> Bob
>
>
>
> -----Original Message-----
> From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> Sent: Friday, July 12, 2019 11:47 AM
> To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
> (USA) <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC
> ARL (USA) <huaqing.cai.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> (USA) <john.w.raby2.civ at mail.mil>
> Subject: Moore convection path forward (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> All,
> Based on our discussions yesterday, the following is my
understanding
> of the current status and the path forward.
>
> Verification of radar data:
> MET has been run to do the verification on column maximum
reflectivity
> but the output still needs to be aggregated and plotted.  Jeffrey
has
> done work towards this.  The goal is to aggregate per experiment by
> lead time and per experiment by actual time for FSS and bias.  Since
> FSS requires specifying a neighborhood size and a dBZ threshold, and
> there are many such possible combinations we will need to figure out
a
> very small number of these to focus on.  Additionally, there are
> questions about how to aggregate FSS properly.  It is not clear if
> this can be done from the MET output files we already have or if
this
> will need to be done via MET stat-analysis.  I am not sure if we
> discussed this but with bias do we know if the standard practice is
to
> weight each time-model run pair equally, or does one weight by the
> number of neighborhoods where the obs exceeded the threshold?
>
> Action: Huaqing Cai and John Raby will contact John Gotway to better
> understand how to aggregate FSS.
> Action: Huaqing Cai will create plots showing the variation in FSS
by
> neighborhood size and dBZ threshold to help determine which
> combinations of these we wish to examine.
>
> When it is understood how to aggregate FSS and we determine what
> neighborhood size / dBZ threshold combinations we will be examining,
> someone will need to do the aggregation and create the desired
plots.
> After the plots are created, analysis of the results can proceed.
>
> Verification of surface data:
> This has been completed and plots are available on the WSMR version
of
> the branch drive.  (This drive is not automatically mapped at ALC
but
> can be accessed and setup to automatically map via its name
> [\\arlwa7001546h.arl.army.mil]). The location on that drive of the
> data
> is:
> \hybrid vLAPS nudging project\plots_old The data on which these
plots
> are based are available at:
> Z:\hybrid vLAPS nudging project\tables_old My understanding is that
> the directories end with _old because Jeffrey has been hoping to
> generate these via a new setup that is not quite ready, but that the
> data in these plots and tables is correct.  The plots available show
> data aggregated per experiment by lead time and actual time, as well
> as of individual experiments.  There is a lot of information
available
> in these plots.  For now the plan is that the surface data
> verification will be used to help interpret the verification of the
> radar data.
>
> Please let me know if there is something that I did not accurately
> convey or if there are aspects that I missed.  Thank you John for
your
> verification efforts and for Jeffrey for all of your work in taking
> the output from MET and converting that into very helpful graphics,
> and your work in creating R code to assist with use of MET output.
>
> Brian
>
>
>
> -----Original Message-----
> From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>
> Sent: Thursday, July 11, 2019 10:21 AM
> To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
> (USA) <jeffrey.a.smith1.civ at mail.mil
> >;
> Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
> Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> Subject: Confirmed- meet in small conference room at 1330 MDT for
> telecon regarding Moore convection/DC ensemble studies
>
> All,
>
>    Gina confirmed that we can use the small conference room for our
> telecon today at 1330 MDT (1530 EDT).  We don't really need the VTC-
> just a regular dial in connection for Brian should be good enough.
We
> can either dial into Brian's office number or vica-versa. I say we
just dial
> into his number.
>
>
> Bob
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
> CLASSIFICATION: UNCLASSIFIED
> CLASSIFICATION: UNCLASSIFIED
>
>

CLASSIFICATION: UNCLASSIFIED

------------------------------------------------
Subject: FW: Moore convection path forward (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Jul 17 09:24:54 2019

CLASSIFICATION: UNCLASSIFIED

John -

I have access to the METViewer at:
http://www.dtcenter.org/met/metviewer/metviewer1.jsp

Is this instance something which would work for me?

R/
John

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, July 16, 2019 4:10 PM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore
convection
path forward (UNCLASSIFIED)

All active links contained in this email were disabled.  Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.




----

John,

As we discussed in our telecon today, stat_analysis supports both
methods of
summarizing the results across multiple cases that Brian referenced:

BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2 =
~3.01

In STAT-Analysis, this is done with a *summary* job type:
   -job summary -line_type CTS -column FBIAS -by
MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
The output summary columns include the mean (i.e. average) of the
input column
of data.

Or via:
BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
O_RATE(t=1)+O_RATE(t=2) ] =
[ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1

In STAT-Analysis, this is done using an *aggregate_stat* job type:
   -job aggregate_stat -line_type CTC -out_line_type CTS -by
MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
The input CTC counts and summed up and the output CTS statistics are
computed
from those counts.  Results are reported separately for each unique
combination of the values found in the columns listed for the "-by"
option.

Thanks,
John

On Tue, Jul 16, 2019 at 3:28 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> Tue Jul 16 15:19:37 2019: Request 91105 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>        Queue: met_help
>      Subject: FW: Moore convection path forward (UNCLASSIFIED)
>        Owner: Nobody
>   Requestors: john.w.raby2.civ at mail.mil
>       Status: new
>  Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91105 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
> -----Original Message-----
> From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> Sent: Tuesday, July 16, 2019 1:34 PM
> To: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC ARL
> (USA) <huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY
> FUTURES COMMAND
> (USA) <robert.e.dumais.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> (USA)
> <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> Huaqing,
> Thank you for the additional details.  If I understand correctly,
> F_RATE is the forecast rate, i.e., the number of neighborhoods where
> the forecast exceeded the threshold, and similarly for O_RATE, but
for
> observations.
>
> I am trying to think through what added value there is to reported
> bias for a particular neighborhood size rather than the overall bias
> (which I think would be equivalent to a neighborhood size of 1).  Do
> you see benefits to reporting bias for a particular neighborhood
size,
> rather than reporting the overall bias?
>
> The issue regarding the neighborhood size dependence of the bias was
> not what I was trying to ask about in my previous email.  What I was
> asking can be illustrated using a case with two times.
>
> Time 1:
> F_RATE(t=1) = 5
> O_RATE(t=1) = 1
> BIAS(t=1) = 5
>
> Time 2:
> F_RATE(t=2) = 50
> O_RATE(t=2) = 49
> BIAS(t=2) = ~1.02
>
> Now, if we wanted the overall bias, my question is whether we would
> calculate it via:
> BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2 =
> ~3.01
>
> Or via:
> BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
> O_RATE(t=1)+O_RATE(t=2) ] = [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
>
> Brian
>
> -----Original Message-----
> From: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> <jeffrey.a.smith1.civ at mail.mil>
> Sent: Tuesday, July 16, 2019 12:01 PM
> To: Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>;
> Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL
> (USA) <brian.p.reen.civ at mail.mil>; Raby, John W CIV USARMY CCDC ARL
> (USA) <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> All.
>
> Couple of points relating to Brians summary below, and a few related
> things.
>
> Once I am back from TDY I'll rebuild the data sets.  I need to get
> back to that anyway, and I know the solution, just haven't had time.
> What that will do is give us the data in R package form that anyone
> can use for analysis.
> Don't have to worry about all the various files, etc.
>
> Once I have that, I can take what I have already done in R and
> reproduce the data sets using the packaged data as a starting point.
> What that means is then if we want to give this analysis to someone
> else, or reproduce it later, the combination of the data package,
and
> the code in package form, should be sufficient to accomplish that.
>
> My next step after rebuilding these data sets, is to make the code
> executable on the HPC.  For most of the data sets I've seen, I think
I
> know enough to make the data package prep process automatable as a
> step after what Leelinda has done.  The package will likely look
like
> three sets of data, one forecast, one obs, and one bias (because its
easy to
> generate).
> I'll right some extractors that will pull specific data variables
from
> the sets since from an abstract point of view, each will look
exactly the
> same.
>
> For the BIAS calculation that Huaqing provides below, I've already
> done that in some test cases that John has provided.  Its actually
> easy to add things like that in R with some of the tools R brings to
> the table since calculations like that are operations on atomic
data.
>
> Regards,
>
> Jeffrey
>
>
> -----Original Message-----
> From: Cai, Huaqing CIV USARMY CCDC ARL (USA)
> <huaqing.cai.civ at mail.mil>
> Sent: Tuesday, July 16, 2019 9:34 AM
> To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL
> (USA) <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC
> ARL (USA) <jeffrey.a.smith1.civ at mail.mil>; Raby, John W CIV USARMY
> CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> HI, Brian, Bob, John and Jeffrey,
>
> Regarding to Brian's question below, although I am not 100% sure
what
> he was asking, here is my attempt trying to answer it. Let me know
if
> it make any sense to you.
>
> BIAS is a standard parameter coming from calculations related to CSI
> (e.g., POD, FAR, BIAS, and CSI). Theoretically, BIAS has nothing to
do
> with FSS and its associated neighborhood sizes, but it is a function
> of threshold. When we show FSS and BIAS, we basically are using two
> kinds of verification methods, that is, the CSI and the FSS method.
>
> When I calculated BIAS from txt files containing FSS results, I used
> O_RATE and F_RATE. Ideally BIAS=F_RATE/O_RATE, and BIAS should not
> change with different neighborhood sizes. Unfortunately, because of
> the edge effects, the number of matched pairs decreases with
increased
> neighborhood sizes.
> This edge effects caused BIAS changes with neighborhood size
slightly
> for small neighborhood sizes, but significantly for very large
> neighborhood sizes. Fortunately, what we want for BIAS is the small
> neighborhood sizes, which are very similar for many small
neighborhood
> sizes (one example is BIAS changed from 0.51 to 0.44 for
neighborhood
> size from 1 to 11 grid sizes).  So just keep in mind that the BIAS
we
> shown was the BIAS for that neighborhood size. We can change the MET
> setting and/or my calculation of BIAS so that we always use the
> smallest neighborhood size for BIAS, if necessary.
>
> Hope this helps.
>
> Huaqing
>
>
>
> -----Original Message-----
> From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> Sent: Monday, July 15, 2019 10:57 AM
> To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
> (USA) <jeffrey.a.smith1.civ at mail.mil
> >;
> Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
> Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> Subject: RE: Moore convection path forward (UNCLASSIFIED)
>
> Hi again Brian,
>
>   I was off Friday, so just finally getting to respond to all the
> messages from that day. I believe you captured the details of our
> discussions very well. For the purpose of the DANA document, the
> completion of the main body of verification work for WRE-N ensemble
DC
> and Moore hybrid DA will remain as FY19 accomplishment targets (I
> don't think Ben wants us to modify the
> FY19 accomplishments already listed in the document), but I will
allow
> for draft tech report and/or open lit publication efforts with these
> studies to carry into FY20 (almost certain to be the case). As to
one
> thing Brian asked, and I am not sure he has received an answer:
>
> "I am not sure if we discussed this but with bias do we know if the
> standard practice is to weight each time-model run pair equally, or
> does one weight by the number of neighborhoods where the obs
exceeded
> the threshold?"
>
>
>
> Bob
>
>
>
> -----Original Message-----
> From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> Sent: Friday, July 12, 2019 11:47 AM
> To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
> (USA) <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC
> ARL (USA) <huaqing.cai.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> (USA) <john.w.raby2.civ at mail.mil>
> Subject: Moore convection path forward (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> All,
> Based on our discussions yesterday, the following is my
understanding
> of the current status and the path forward.
>
> Verification of radar data:
> MET has been run to do the verification on column maximum
reflectivity
> but the output still needs to be aggregated and plotted.  Jeffrey
has
> done work towards this.  The goal is to aggregate per experiment by
> lead time and per experiment by actual time for FSS and bias.  Since
> FSS requires specifying a neighborhood size and a dBZ threshold, and
> there are many such possible combinations we will need to figure out
a
> very small number of these to focus on.  Additionally, there are
> questions about how to aggregate FSS properly.  It is not clear if
> this can be done from the MET output files we already have or if
this
> will need to be done via MET stat-analysis.  I am not sure if we
> discussed this but with bias do we know if the standard practice is
to
> weight each time-model run pair equally, or does one weight by the
> number of neighborhoods where the obs exceeded the threshold?
>
> Action: Huaqing Cai and John Raby will contact John Gotway to better
> understand how to aggregate FSS.
> Action: Huaqing Cai will create plots showing the variation in FSS
by
> neighborhood size and dBZ threshold to help determine which
> combinations of these we wish to examine.
>
> When it is understood how to aggregate FSS and we determine what
> neighborhood size / dBZ threshold combinations we will be examining,
> someone will need to do the aggregation and create the desired
plots.
> After the plots are created, analysis of the results can proceed.
>
> Verification of surface data:
> This has been completed and plots are available on the WSMR version
of
> the branch drive.  (This drive is not automatically mapped at ALC
but
> can be accessed and setup to automatically map via its name
> [\\arlwa7001546h.arl.army.mil]). The location on that drive of the
> data
> is:
> \hybrid vLAPS nudging project\plots_old The data on which these
plots
> are based are available at:
> Z:\hybrid vLAPS nudging project\tables_old My understanding is that
> the directories end with _old because Jeffrey has been hoping to
> generate these via a new setup that is not quite ready, but that the
> data in these plots and tables is correct.  The plots available show
> data aggregated per experiment by lead time and actual time, as well
> as of individual experiments.  There is a lot of information
available
> in these plots.  For now the plan is that the surface data
> verification will be used to help interpret the verification of the
> radar data.
>
> Please let me know if there is something that I did not accurately
> convey or if there are aspects that I missed.  Thank you John for
your
> verification efforts and for Jeffrey for all of your work in taking
> the output from MET and converting that into very helpful graphics,
> and your work in creating R code to assist with use of MET output.
>
> Brian
>
>
>
> -----Original Message-----
> From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> <robert.e.dumais.civ at mail.mil>
> Sent: Thursday, July 11, 2019 10:21 AM
> To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
> (USA) <jeffrey.a.smith1.civ at mail.mil
> >;
> Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
> Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> Subject: Confirmed- meet in small conference room at 1330 MDT for
> telecon regarding Moore convection/DC ensemble studies
>
> All,
>
>    Gina confirmed that we can use the small conference room for our
> telecon today at 1330 MDT (1530 EDT).  We don't really need the VTC-
> just a regular dial in connection for Brian should be good enough.
We
> can either dial into Brian's office number or vica-versa. I say we
just dial
> into his number.
>
>
> Bob
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
> CLASSIFICATION: UNCLASSIFIED
> CLASSIFICATION: UNCLASSIFIED
>
>

CLASSIFICATION: UNCLASSIFIED

------------------------------------------------
Subject: FW: Moore convection path forward (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Jul 17 13:23:28 2019

John,

Yes, this is NCAR's public facing METviewer instance:
http://www.dtcenter.org/met/metviewer/metviewer1.jsp

If you send me your .stat output files, I could pretty easily load it
into
a new database there.  Then you could play around with your data and
decide
if METviewer is worth the setup effort to you.

NOAA's AWS instance of METviewer is not publicly accessible.  It sit
behind
a firewall.  If the Army is interested in using AWS, it should be
fairly
painless to clone that AWS setup for your use.  But then there's the
headache of figuring out how to work on AWS and pay them for your use
of it.

If you'd like to proceed with installing METviewer at ARL, let me
know, and
I'll reassign this ticket to Tatiana.

Thanks,
John

On Wed, Jul 17, 2019 at 9:25 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91105 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I have access to the METViewer at:
> http://www.dtcenter.org/met/metviewer/metviewer1.jsp
>
> Is this instance something which would work for me?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, July 16, 2019 4:10 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore
> convection
> path forward (UNCLASSIFIED)
>
> All active links contained in this email were disabled.  Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> As we discussed in our telecon today, stat_analysis supports both
methods
> of
> summarizing the results across multiple cases that Brian referenced:
>
> BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2 =
~3.01
>
> In STAT-Analysis, this is done with a *summary* job type:
>    -job summary -line_type CTS -column FBIAS -by
> MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
> The output summary columns include the mean (i.e. average) of the
input
> column
> of data.
>
> Or via:
> BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
O_RATE(t=1)+O_RATE(t=2) ]
> =
> [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
>
> In STAT-Analysis, this is done using an *aggregate_stat* job type:
>    -job aggregate_stat -line_type CTC -out_line_type CTS -by
> MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
> The input CTC counts and summed up and the output CTS statistics are
> computed
> from those counts.  Results are reported separately for each unique
> combination of the values found in the columns listed for the "-by"
> option.
>
> Thanks,
> John
>
> On Tue, Jul 16, 2019 at 3:28 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jul 16 15:19:37 2019: Request 91105 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> >        Queue: met_help
> >      Subject: FW: Moore convection path forward (UNCLASSIFIED)
> >        Owner: Nobody
> >   Requestors: john.w.raby2.civ at mail.mil
> >       Status: new
> >  Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91105 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
> > -----Original Message-----
> > From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > Sent: Tuesday, July 16, 2019 1:34 PM
> > To: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> > <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC ARL
> > (USA) <huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY
> > FUTURES COMMAND
> > (USA) <robert.e.dumais.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> > (USA)
> > <john.w.raby2.civ at mail.mil>
> > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Huaqing,
> > Thank you for the additional details.  If I understand correctly,
> > F_RATE is the forecast rate, i.e., the number of neighborhoods
where
> > the forecast exceeded the threshold, and similarly for O_RATE, but
for
> > observations.
> >
> > I am trying to think through what added value there is to reported
> > bias for a particular neighborhood size rather than the overall
bias
> > (which I think would be equivalent to a neighborhood size of 1).
Do
> > you see benefits to reporting bias for a particular neighborhood
size,
> > rather than reporting the overall bias?
> >
> > The issue regarding the neighborhood size dependence of the bias
was
> > not what I was trying to ask about in my previous email.  What I
was
> > asking can be illustrated using a case with two times.
> >
> > Time 1:
> > F_RATE(t=1) = 5
> > O_RATE(t=1) = 1
> > BIAS(t=1) = 5
> >
> > Time 2:
> > F_RATE(t=2) = 50
> > O_RATE(t=2) = 49
> > BIAS(t=2) = ~1.02
> >
> > Now, if we wanted the overall bias, my question is whether we
would
> > calculate it via:
> > BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2 =
> > ~3.01
> >
> > Or via:
> > BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
> > O_RATE(t=1)+O_RATE(t=2) ] = [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
> >
> > Brian
> >
> > -----Original Message-----
> > From: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> > <jeffrey.a.smith1.civ at mail.mil>
> > Sent: Tuesday, July 16, 2019 12:01 PM
> > To: Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>;
> > Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL
> > (USA) <brian.p.reen.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> > (USA) <john.w.raby2.civ at mail.mil>
> > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> >
> > All.
> >
> > Couple of points relating to Brians summary below, and a few
related
> > things.
> >
> > Once I am back from TDY I'll rebuild the data sets.  I need to get
> > back to that anyway, and I know the solution, just haven't had
time.
> > What that will do is give us the data in R package form that
anyone
> > can use for analysis.
> > Don't have to worry about all the various files, etc.
> >
> > Once I have that, I can take what I have already done in R and
> > reproduce the data sets using the packaged data as a starting
point.
> > What that means is then if we want to give this analysis to
someone
> > else, or reproduce it later, the combination of the data package,
and
> > the code in package form, should be sufficient to accomplish that.
> >
> > My next step after rebuilding these data sets, is to make the code
> > executable on the HPC.  For most of the data sets I've seen, I
think I
> > know enough to make the data package prep process automatable as a
> > step after what Leelinda has done.  The package will likely look
like
> > three sets of data, one forecast, one obs, and one bias (because
its
> easy to
> > generate).
> > I'll right some extractors that will pull specific data variables
from
> > the sets since from an abstract point of view, each will look
exactly
> the
> > same.
> >
> > For the BIAS calculation that Huaqing provides below, I've already
> > done that in some test cases that John has provided.  Its actually
> > easy to add things like that in R with some of the tools R brings
to
> > the table since calculations like that are operations on atomic
data.
> >
> > Regards,
> >
> > Jeffrey
> >
> >
> > -----Original Message-----
> > From: Cai, Huaqing CIV USARMY CCDC ARL (USA)
> > <huaqing.cai.civ at mail.mil>
> > Sent: Tuesday, July 16, 2019 9:34 AM
> > To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL
> > (USA) <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY
CCDC
> > ARL (USA) <jeffrey.a.smith1.civ at mail.mil>; Raby, John W CIV USARMY
> > CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> >
> > HI, Brian, Bob, John and Jeffrey,
> >
> > Regarding to Brian's question below, although I am not 100% sure
what
> > he was asking, here is my attempt trying to answer it. Let me know
if
> > it make any sense to you.
> >
> > BIAS is a standard parameter coming from calculations related to
CSI
> > (e.g., POD, FAR, BIAS, and CSI). Theoretically, BIAS has nothing
to do
> > with FSS and its associated neighborhood sizes, but it is a
function
> > of threshold. When we show FSS and BIAS, we basically are using
two
> > kinds of verification methods, that is, the CSI and the FSS
method.
> >
> > When I calculated BIAS from txt files containing FSS results, I
used
> > O_RATE and F_RATE. Ideally BIAS=F_RATE/O_RATE, and BIAS should not
> > change with different neighborhood sizes. Unfortunately, because
of
> > the edge effects, the number of matched pairs decreases with
increased
> > neighborhood sizes.
> > This edge effects caused BIAS changes with neighborhood size
slightly
> > for small neighborhood sizes, but significantly for very large
> > neighborhood sizes. Fortunately, what we want for BIAS is the
small
> > neighborhood sizes, which are very similar for many small
neighborhood
> > sizes (one example is BIAS changed from 0.51 to 0.44 for
neighborhood
> > size from 1 to 11 grid sizes).  So just keep in mind that the BIAS
we
> > shown was the BIAS for that neighborhood size. We can change the
MET
> > setting and/or my calculation of BIAS so that we always use the
> > smallest neighborhood size for BIAS, if necessary.
> >
> > Hope this helps.
> >
> > Huaqing
> >
> >
> >
> > -----Original Message-----
> > From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > Sent: Monday, July 15, 2019 10:57 AM
> > To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
> > (USA) <jeffrey.a.smith1.civ at mail.mil
> > >;
> > Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
> > Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> >
> > Hi again Brian,
> >
> >   I was off Friday, so just finally getting to respond to all the
> > messages from that day. I believe you captured the details of our
> > discussions very well. For the purpose of the DANA document, the
> > completion of the main body of verification work for WRE-N
ensemble DC
> > and Moore hybrid DA will remain as FY19 accomplishment targets (I
> > don't think Ben wants us to modify the
> > FY19 accomplishments already listed in the document), but I will
allow
> > for draft tech report and/or open lit publication efforts with
these
> > studies to carry into FY20 (almost certain to be the case). As to
one
> > thing Brian asked, and I am not sure he has received an answer:
> >
> > "I am not sure if we discussed this but with bias do we know if
the
> > standard practice is to weight each time-model run pair equally,
or
> > does one weight by the number of neighborhoods where the obs
exceeded
> > the threshold?"
> >
> >
> >
> > Bob
> >
> >
> >
> > -----Original Message-----
> > From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > Sent: Friday, July 12, 2019 11:47 AM
> > To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > <robert.e.dumais.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC
ARL
> > (USA) <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY
CCDC
> > ARL (USA) <huaqing.cai.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> > (USA) <john.w.raby2.civ at mail.mil>
> > Subject: Moore convection path forward (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > All,
> > Based on our discussions yesterday, the following is my
understanding
> > of the current status and the path forward.
> >
> > Verification of radar data:
> > MET has been run to do the verification on column maximum
reflectivity
> > but the output still needs to be aggregated and plotted.  Jeffrey
has
> > done work towards this.  The goal is to aggregate per experiment
by
> > lead time and per experiment by actual time for FSS and bias.
Since
> > FSS requires specifying a neighborhood size and a dBZ threshold,
and
> > there are many such possible combinations we will need to figure
out a
> > very small number of these to focus on.  Additionally, there are
> > questions about how to aggregate FSS properly.  It is not clear if
> > this can be done from the MET output files we already have or if
this
> > will need to be done via MET stat-analysis.  I am not sure if we
> > discussed this but with bias do we know if the standard practice
is to
> > weight each time-model run pair equally, or does one weight by the
> > number of neighborhoods where the obs exceeded the threshold?
> >
> > Action: Huaqing Cai and John Raby will contact John Gotway to
better
> > understand how to aggregate FSS.
> > Action: Huaqing Cai will create plots showing the variation in FSS
by
> > neighborhood size and dBZ threshold to help determine which
> > combinations of these we wish to examine.
> >
> > When it is understood how to aggregate FSS and we determine what
> > neighborhood size / dBZ threshold combinations we will be
examining,
> > someone will need to do the aggregation and create the desired
plots.
> > After the plots are created, analysis of the results can proceed.
> >
> > Verification of surface data:
> > This has been completed and plots are available on the WSMR
version of
> > the branch drive.  (This drive is not automatically mapped at ALC
but
> > can be accessed and setup to automatically map via its name
> > [\\arlwa7001546h.arl.army.mil]). The location on that drive of the
> > data
> > is:
> > \hybrid vLAPS nudging project\plots_old The data on which these
plots
> > are based are available at:
> > Z:\hybrid vLAPS nudging project\tables_old My understanding is
that
> > the directories end with _old because Jeffrey has been hoping to
> > generate these via a new setup that is not quite ready, but that
the
> > data in these plots and tables is correct.  The plots available
show
> > data aggregated per experiment by lead time and actual time, as
well
> > as of individual experiments.  There is a lot of information
available
> > in these plots.  For now the plan is that the surface data
> > verification will be used to help interpret the verification of
the
> > radar data.
> >
> > Please let me know if there is something that I did not accurately
> > convey or if there are aspects that I missed.  Thank you John for
your
> > verification efforts and for Jeffrey for all of your work in
taking
> > the output from MET and converting that into very helpful
graphics,
> > and your work in creating R code to assist with use of MET output.
> >
> > Brian
> >
> >
> >
> > -----Original Message-----
> > From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > <robert.e.dumais.civ at mail.mil>
> > Sent: Thursday, July 11, 2019 10:21 AM
> > To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
> > (USA) <jeffrey.a.smith1.civ at mail.mil
> > >;
> > Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
> > Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> > Subject: Confirmed- meet in small conference room at 1330 MDT for
> > telecon regarding Moore convection/DC ensemble studies
> >
> > All,
> >
> >    Gina confirmed that we can use the small conference room for
our
> > telecon today at 1330 MDT (1530 EDT).  We don't really need the
VTC-
> > just a regular dial in connection for Brian should be good enough.
We
> > can either dial into Brian's office number or vica-versa. I say we
just
> dial
> > into his number.
> >
> >
> > Bob
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>

------------------------------------------------
Subject: FW: Moore convection path forward (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Jul 17 14:14:53 2019

CLASSIFICATION: UNCLASSIFIED

John -

Thanks for your response and info on METViewer and for your offer to
set up
the new database. I'll send you the .stat files for each of 5 cycles
of runs
from a specific experimental model run. There are a total of 9
different
experimental model which were run. I am starting the preps to have
METViewer
installed on my local linux workstation. I already have Java and need
to have
R and MySQL installed. Anything else?

R/
John

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, July 17, 2019 1:23 PM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore
convection path forward (UNCLASSIFIED)

All active links contained in this email were disabled.  Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.




----

John,

Yes, this is NCAR's public facing METviewer instance:
Caution-http://www.dtcenter.org/met/metviewer/metviewer1.jsp

If you send me your .stat output files, I could pretty easily load it
into a
new database there.  Then you could play around with your data and
decide if
METviewer is worth the setup effort to you.

NOAA's AWS instance of METviewer is not publicly accessible.  It sit
behind a
firewall.  If the Army is interested in using AWS, it should be fairly
painless to clone that AWS setup for your use.  But then there's the
headache
of figuring out how to work on AWS and pay them for your use of it.

If you'd like to proceed with installing METviewer at ARL, let me
know, and
I'll reassign this ticket to Tatiana.

Thanks,
John

On Wed, Jul 17, 2019 at 9:25 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91105 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I have access to the METViewer at:
> Caution-http://www.dtcenter.org/met/metviewer/metviewer1.jsp
>
> Is this instance something which would work for me?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Tuesday, July 16, 2019 4:10 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore
> convection path forward (UNCLASSIFIED)
>
> All active links contained in this email were disabled.  Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> As we discussed in our telecon today, stat_analysis supports both
> methods of summarizing the results across multiple cases that Brian
> referenced:
>
> BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2 =
> ~3.01
>
> In STAT-Analysis, this is done with a *summary* job type:
>    -job summary -line_type CTS -column FBIAS -by
> MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
> The output summary columns include the mean (i.e. average) of the
> input column of data.
>
> Or via:
> BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
> O_RATE(t=1)+O_RATE(t=2) ] = [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
>
> In STAT-Analysis, this is done using an *aggregate_stat* job type:
>    -job aggregate_stat -line_type CTC -out_line_type CTS -by
> MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
> The input CTC counts and summed up and the output CTS statistics are
> computed from those counts.  Results are reported separately for
each
> unique combination of the values found in the columns listed for the
> "-by"
> option.
>
> Thanks,
> John
>
> On Tue, Jul 16, 2019 at 3:28 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jul 16 15:19:37 2019: Request 91105 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> >        Queue: met_help
> >      Subject: FW: Moore convection path forward (UNCLASSIFIED)
> >        Owner: Nobody
> >   Requestors: john.w.raby2.civ at mail.mil
> >       Status: new
> >  Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91
> > 105 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
> > -----Original Message-----
> > From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > Sent: Tuesday, July 16, 2019 1:34 PM
> > To: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> > <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC ARL
> > (USA) <huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY
> > FUTURES COMMAND
> > (USA) <robert.e.dumais.civ at mail.mil>; Raby, John W CIV USARMY CCDC
> > ARL
> > (USA)
> > <john.w.raby2.civ at mail.mil>
> > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Huaqing,
> > Thank you for the additional details.  If I understand correctly,
> > F_RATE is the forecast rate, i.e., the number of neighborhoods
where
> > the forecast exceeded the threshold, and similarly for O_RATE, but
> > for observations.
> >
> > I am trying to think through what added value there is to reported
> > bias for a particular neighborhood size rather than the overall
bias
> > (which I think would be equivalent to a neighborhood size of 1).
Do
> > you see benefits to reporting bias for a particular neighborhood
> > size, rather than reporting the overall bias?
> >
> > The issue regarding the neighborhood size dependence of the bias
was
> > not what I was trying to ask about in my previous email.  What I
was
> > asking can be illustrated using a case with two times.
> >
> > Time 1:
> > F_RATE(t=1) = 5
> > O_RATE(t=1) = 1
> > BIAS(t=1) = 5
> >
> > Time 2:
> > F_RATE(t=2) = 50
> > O_RATE(t=2) = 49
> > BIAS(t=2) = ~1.02
> >
> > Now, if we wanted the overall bias, my question is whether we
would
> > calculate it via:
> > BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2 =
> > ~3.01
> >
> > Or via:
> > BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
> > O_RATE(t=1)+O_RATE(t=2) ] = [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
> >
> > Brian
> >
> > -----Original Message-----
> > From: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> > <jeffrey.a.smith1.civ at mail.mil>
> > Sent: Tuesday, July 16, 2019 12:01 PM
> > To: Cai, Huaqing CIV USARMY CCDC ARL (USA)
> > <huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY FUTURES
> > COMMAND (USA) <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV
> > USARMY CCDC ARL
> > (USA) <brian.p.reen.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> > (USA) <john.w.raby2.civ at mail.mil>
> > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> >
> > All.
> >
> > Couple of points relating to Brians summary below, and a few
related
> > things.
> >
> > Once I am back from TDY I'll rebuild the data sets.  I need to get
> > back to that anyway, and I know the solution, just haven't had
time.
> > What that will do is give us the data in R package form that
anyone
> > can use for analysis.
> > Don't have to worry about all the various files, etc.
> >
> > Once I have that, I can take what I have already done in R and
> > reproduce the data sets using the packaged data as a starting
point.
> > What that means is then if we want to give this analysis to
someone
> > else, or reproduce it later, the combination of the data package,
> > and the code in package form, should be sufficient to accomplish
that.
> >
> > My next step after rebuilding these data sets, is to make the code
> > executable on the HPC.  For most of the data sets I've seen, I
think
> > I know enough to make the data package prep process automatable as
a
> > step after what Leelinda has done.  The package will likely look
> > like three sets of data, one forecast, one obs, and one bias
> > (because its
> easy to
> > generate).
> > I'll right some extractors that will pull specific data variables
> > from the sets since from an abstract point of view, each will look
> > exactly
> the
> > same.
> >
> > For the BIAS calculation that Huaqing provides below, I've already
> > done that in some test cases that John has provided.  Its actually
> > easy to add things like that in R with some of the tools R brings
to
> > the table since calculations like that are operations on atomic
data.
> >
> > Regards,
> >
> > Jeffrey
> >
> >
> > -----Original Message-----
> > From: Cai, Huaqing CIV USARMY CCDC ARL (USA)
> > <huaqing.cai.civ at mail.mil>
> > Sent: Tuesday, July 16, 2019 9:34 AM
> > To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC ARL
> > (USA) <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY
CCDC
> > ARL (USA) <jeffrey.a.smith1.civ at mail.mil>; Raby, John W CIV USARMY
> > CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> >
> > HI, Brian, Bob, John and Jeffrey,
> >
> > Regarding to Brian's question below, although I am not 100% sure
> > what he was asking, here is my attempt trying to answer it. Let me
> > know if it make any sense to you.
> >
> > BIAS is a standard parameter coming from calculations related to
CSI
> > (e.g., POD, FAR, BIAS, and CSI). Theoretically, BIAS has nothing
to
> > do with FSS and its associated neighborhood sizes, but it is a
> > function of threshold. When we show FSS and BIAS, we basically are
> > using two kinds of verification methods, that is, the CSI and the
FSS
> > method.
> >
> > When I calculated BIAS from txt files containing FSS results, I
used
> > O_RATE and F_RATE. Ideally BIAS=F_RATE/O_RATE, and BIAS should not
> > change with different neighborhood sizes. Unfortunately, because
of
> > the edge effects, the number of matched pairs decreases with
> > increased neighborhood sizes.
> > This edge effects caused BIAS changes with neighborhood size
> > slightly for small neighborhood sizes, but significantly for very
> > large neighborhood sizes. Fortunately, what we want for BIAS is
the
> > small neighborhood sizes, which are very similar for many small
> > neighborhood sizes (one example is BIAS changed from 0.51 to 0.44
> > for neighborhood size from 1 to 11 grid sizes).  So just keep in
> > mind that the BIAS we shown was the BIAS for that neighborhood
size.
> > We can change the MET setting and/or my calculation of BIAS so
that
> > we always use the smallest neighborhood size for BIAS, if
necessary.
> >
> > Hope this helps.
> >
> > Huaqing
> >
> >
> >
> > -----Original Message-----
> > From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > Sent: Monday, July 15, 2019 10:57 AM
> > To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
> > (USA) <jeffrey.a.smith1.civ at mail.mil
> > >;
> > Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
> > Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> >
> > Hi again Brian,
> >
> >   I was off Friday, so just finally getting to respond to all the
> > messages from that day. I believe you captured the details of our
> > discussions very well. For the purpose of the DANA document, the
> > completion of the main body of verification work for WRE-N
ensemble
> > DC and Moore hybrid DA will remain as FY19 accomplishment targets
(I
> > don't think Ben wants us to modify the
> > FY19 accomplishments already listed in the document), but I will
> > allow for draft tech report and/or open lit publication efforts
with
> > these studies to carry into FY20 (almost certain to be the case).
As
> > to one thing Brian asked, and I am not sure he has received an
answer:
> >
> > "I am not sure if we discussed this but with bias do we know if
the
> > standard practice is to weight each time-model run pair equally,
or
> > does one weight by the number of neighborhoods where the obs
> > exceeded the threshold?"
> >
> >
> >
> > Bob
> >
> >
> >
> > -----Original Message-----
> > From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > Sent: Friday, July 12, 2019 11:47 AM
> > To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > <robert.e.dumais.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC
ARL
> > (USA) <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY
CCDC
> > ARL (USA) <huaqing.cai.civ at mail.mil>; Raby, John W CIV USARMY CCDC
> > ARL
> > (USA) <john.w.raby2.civ at mail.mil>
> > Subject: Moore convection path forward (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > All,
> > Based on our discussions yesterday, the following is my
> > understanding of the current status and the path forward.
> >
> > Verification of radar data:
> > MET has been run to do the verification on column maximum
> > reflectivity but the output still needs to be aggregated and
> > plotted.  Jeffrey has done work towards this.  The goal is to
> > aggregate per experiment by lead time and per experiment by actual
> > time for FSS and bias.  Since FSS requires specifying a
neighborhood
> > size and a dBZ threshold, and there are many such possible
> > combinations we will need to figure out a very small number of
these
> > to focus on.  Additionally, there are questions about how to
> > aggregate FSS properly.  It is not clear if this can be done from
> > the MET output files we already have or if this will need to be
done
> > via MET stat-analysis.  I am not sure if we discussed this but
with
> > bias do we know if the standard practice is to weight each
> > time-model run pair equally, or does one weight by the number of
> > neighborhoods where the obs exceeded the threshold?
> >
> > Action: Huaqing Cai and John Raby will contact John Gotway to
better
> > understand how to aggregate FSS.
> > Action: Huaqing Cai will create plots showing the variation in FSS
> > by neighborhood size and dBZ threshold to help determine which
> > combinations of these we wish to examine.
> >
> > When it is understood how to aggregate FSS and we determine what
> > neighborhood size / dBZ threshold combinations we will be
examining,
> > someone will need to do the aggregation and create the desired
plots.
> > After the plots are created, analysis of the results can proceed.
> >
> > Verification of surface data:
> > This has been completed and plots are available on the WSMR
version
> > of the branch drive.  (This drive is not automatically mapped at
ALC
> > but can be accessed and setup to automatically map via its name
> > [\\arlwa7001546h.arl.army.mil]). The location on that drive of the
> > data
> > is:
> > \hybrid vLAPS nudging project\plots_old The data on which these
> > plots are based are available at:
> > Z:\hybrid vLAPS nudging project\tables_old My understanding is
that
> > the directories end with _old because Jeffrey has been hoping to
> > generate these via a new setup that is not quite ready, but that
the
> > data in these plots and tables is correct.  The plots available
show
> > data aggregated per experiment by lead time and actual time, as
well
> > as of individual experiments.  There is a lot of information
> > available in these plots.  For now the plan is that the surface
data
> > verification will be used to help interpret the verification of
the
> > radar data.
> >
> > Please let me know if there is something that I did not accurately
> > convey or if there are aspects that I missed.  Thank you John for
> > your verification efforts and for Jeffrey for all of your work in
> > taking the output from MET and converting that into very helpful
> > graphics, and your work in creating R code to assist with use of
MET
> > output.
> >
> > Brian
> >
> >
> >
> > -----Original Message-----
> > From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > <robert.e.dumais.civ at mail.mil>
> > Sent: Thursday, July 11, 2019 10:21 AM
> > To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC ARL
> > (USA) <jeffrey.a.smith1.civ at mail.mil
> > >;
> > Cai, Huaqing CIV USARMY CCDC ARL (USA) <huaqing.cai.civ at mail.mil>;
> > Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> > Subject: Confirmed- meet in small conference room at 1330 MDT for
> > telecon regarding Moore convection/DC ensemble studies
> >
> > All,
> >
> >    Gina confirmed that we can use the small conference room for
our
> > telecon today at 1330 MDT (1530 EDT).  We don't really need the
VTC-
> > just a regular dial in connection for Brian should be good enough.
> > We can either dial into Brian's office number or vica-versa. I say
> > we just
> dial
> > into his number.
> >
> >
> > Bob
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>

CLASSIFICATION: UNCLASSIFIED

------------------------------------------------
Subject: FW: Moore convection path forward (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Jul 18 09:34:24 2019

John,

With the goal of keeping things better organized, I'm going to resolve
this
current RT-ticket and create a new for your questions about installing
METviewer.  And I'll assign that new ticket to Tatiana.  She should be
able
to answer your questions about METviewer.

Thanks,
John

On Wed, Jul 17, 2019 at 2:15 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91105 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for your response and info on METViewer and for your offer to
set
> up
> the new database. I'll send you the .stat files for each of 5 cycles
of
> runs
> from a specific experimental model run. There are a total of 9
different
> experimental model which were run. I am starting the preps to have
> METViewer
> installed on my local linux workstation. I already have Java and
need to
> have
> R and MySQL installed. Anything else?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, July 17, 2019 1:23 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore
> convection path forward (UNCLASSIFIED)
>
> All active links contained in this email were disabled.  Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Yes, this is NCAR's public facing METviewer instance:
> Caution-http://www.dtcenter.org/met/metviewer/metviewer1.jsp
>
> If you send me your .stat output files, I could pretty easily load
it into
> a
> new database there.  Then you could play around with your data and
decide
> if
> METviewer is worth the setup effort to you.
>
> NOAA's AWS instance of METviewer is not publicly accessible.  It sit
> behind a
> firewall.  If the Army is interested in using AWS, it should be
fairly
> painless to clone that AWS setup for your use.  But then there's the
> headache
> of figuring out how to work on AWS and pay them for your use of it.
>
> If you'd like to proceed with installing METviewer at ARL, let me
know,
> and
> I'll reassign this ticket to Tatiana.
>
> Thanks,
> John
>
> On Wed, Jul 17, 2019 at 9:25 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91105 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I have access to the METViewer at:
> > Caution-http://www.dtcenter.org/met/metviewer/metviewer1.jsp
> >
> > Is this instance something which would work for me?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Tuesday, July 16, 2019 4:10 PM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore
> > convection path forward (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled.  Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > As we discussed in our telecon today, stat_analysis supports both
> > methods of summarizing the results across multiple cases that
Brian
> > referenced:
> >
> > BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2 =
> > ~3.01
> >
> > In STAT-Analysis, this is done with a *summary* job type:
> >    -job summary -line_type CTS -column FBIAS -by
> > MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
> > The output summary columns include the mean (i.e. average) of the
> > input column of data.
> >
> > Or via:
> > BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
> > O_RATE(t=1)+O_RATE(t=2) ] = [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
> >
> > In STAT-Analysis, this is done using an *aggregate_stat* job type:
> >    -job aggregate_stat -line_type CTC -out_line_type CTS -by
> > MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
> > The input CTC counts and summed up and the output CTS statistics
are
> > computed from those counts.  Results are reported separately for
each
> > unique combination of the values found in the columns listed for
the
> > "-by"
> > option.
> >
> > Thanks,
> > John
> >
> > On Tue, Jul 16, 2019 at 3:28 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jul 16 15:19:37 2019: Request 91105 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > >        Queue: met_help
> > >      Subject: FW: Moore convection path forward (UNCLASSIFIED)
> > >        Owner: Nobody
> > >   Requestors: john.w.raby2.civ at mail.mil
> > >       Status: new
> > >  Ticket <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91
> > > 105 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > > Sent: Tuesday, July 16, 2019 1:34 PM
> > > To: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> > > <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC
ARL
> > > (USA) <huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY
> > > FUTURES COMMAND
> > > (USA) <robert.e.dumais.civ at mail.mil>; Raby, John W CIV USARMY
CCDC
> > > ARL
> > > (USA)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Huaqing,
> > > Thank you for the additional details.  If I understand
correctly,
> > > F_RATE is the forecast rate, i.e., the number of neighborhoods
where
> > > the forecast exceeded the threshold, and similarly for O_RATE,
but
> > > for observations.
> > >
> > > I am trying to think through what added value there is to
reported
> > > bias for a particular neighborhood size rather than the overall
bias
> > > (which I think would be equivalent to a neighborhood size of 1).
Do
> > > you see benefits to reporting bias for a particular neighborhood
> > > size, rather than reporting the overall bias?
> > >
> > > The issue regarding the neighborhood size dependence of the bias
was
> > > not what I was trying to ask about in my previous email.  What I
was
> > > asking can be illustrated using a case with two times.
> > >
> > > Time 1:
> > > F_RATE(t=1) = 5
> > > O_RATE(t=1) = 1
> > > BIAS(t=1) = 5
> > >
> > > Time 2:
> > > F_RATE(t=2) = 50
> > > O_RATE(t=2) = 49
> > > BIAS(t=2) = ~1.02
> > >
> > > Now, if we wanted the overall bias, my question is whether we
would
> > > calculate it via:
> > > BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2
=
> > > ~3.01
> > >
> > > Or via:
> > > BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
> > > O_RATE(t=1)+O_RATE(t=2) ] = [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
> > >
> > > Brian
> > >
> > > -----Original Message-----
> > > From: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> > > <jeffrey.a.smith1.civ at mail.mil>
> > > Sent: Tuesday, July 16, 2019 12:01 PM
> > > To: Cai, Huaqing CIV USARMY CCDC ARL (USA)
> > > <huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY
FUTURES
> > > COMMAND (USA) <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV
> > > USARMY CCDC ARL
> > > (USA) <brian.p.reen.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> > > (USA) <john.w.raby2.civ at mail.mil>
> > > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> > >
> > > All.
> > >
> > > Couple of points relating to Brians summary below, and a few
related
> > > things.
> > >
> > > Once I am back from TDY I'll rebuild the data sets.  I need to
get
> > > back to that anyway, and I know the solution, just haven't had
time.
> > > What that will do is give us the data in R package form that
anyone
> > > can use for analysis.
> > > Don't have to worry about all the various files, etc.
> > >
> > > Once I have that, I can take what I have already done in R and
> > > reproduce the data sets using the packaged data as a starting
point.
> > > What that means is then if we want to give this analysis to
someone
> > > else, or reproduce it later, the combination of the data
package,
> > > and the code in package form, should be sufficient to accomplish
that.
> > >
> > > My next step after rebuilding these data sets, is to make the
code
> > > executable on the HPC.  For most of the data sets I've seen, I
think
> > > I know enough to make the data package prep process automatable
as a
> > > step after what Leelinda has done.  The package will likely look
> > > like three sets of data, one forecast, one obs, and one bias
> > > (because its
> > easy to
> > > generate).
> > > I'll right some extractors that will pull specific data
variables
> > > from the sets since from an abstract point of view, each will
look
> > > exactly
> > the
> > > same.
> > >
> > > For the BIAS calculation that Huaqing provides below, I've
already
> > > done that in some test cases that John has provided.  Its
actually
> > > easy to add things like that in R with some of the tools R
brings to
> > > the table since calculations like that are operations on atomic
data.
> > >
> > > Regards,
> > >
> > > Jeffrey
> > >
> > >
> > > -----Original Message-----
> > > From: Cai, Huaqing CIV USARMY CCDC ARL (USA)
> > > <huaqing.cai.civ at mail.mil>
> > > Sent: Tuesday, July 16, 2019 9:34 AM
> > > To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > > <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC
ARL
> > > (USA) <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY
CCDC
> > > ARL (USA) <jeffrey.a.smith1.civ at mail.mil>; Raby, John W CIV
USARMY
> > > CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> > > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> > >
> > > HI, Brian, Bob, John and Jeffrey,
> > >
> > > Regarding to Brian's question below, although I am not 100% sure
> > > what he was asking, here is my attempt trying to answer it. Let
me
> > > know if it make any sense to you.
> > >
> > > BIAS is a standard parameter coming from calculations related to
CSI
> > > (e.g., POD, FAR, BIAS, and CSI). Theoretically, BIAS has nothing
to
> > > do with FSS and its associated neighborhood sizes, but it is a
> > > function of threshold. When we show FSS and BIAS, we basically
are
> > > using two kinds of verification methods, that is, the CSI and
the FSS
> > > method.
> > >
> > > When I calculated BIAS from txt files containing FSS results, I
used
> > > O_RATE and F_RATE. Ideally BIAS=F_RATE/O_RATE, and BIAS should
not
> > > change with different neighborhood sizes. Unfortunately, because
of
> > > the edge effects, the number of matched pairs decreases with
> > > increased neighborhood sizes.
> > > This edge effects caused BIAS changes with neighborhood size
> > > slightly for small neighborhood sizes, but significantly for
very
> > > large neighborhood sizes. Fortunately, what we want for BIAS is
the
> > > small neighborhood sizes, which are very similar for many small
> > > neighborhood sizes (one example is BIAS changed from 0.51 to
0.44
> > > for neighborhood size from 1 to 11 grid sizes).  So just keep in
> > > mind that the BIAS we shown was the BIAS for that neighborhood
size.
> > > We can change the MET setting and/or my calculation of BIAS so
that
> > > we always use the smallest neighborhood size for BIAS, if
necessary.
> > >
> > > Hope this helps.
> > >
> > > Huaqing
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > > Sent: Monday, July 15, 2019 10:57 AM
> > > To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > > <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC
ARL
> > > (USA) <jeffrey.a.smith1.civ at mail.mil
> > > >;
> > > Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>;
> > > Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> > >
> > > Hi again Brian,
> > >
> > >   I was off Friday, so just finally getting to respond to all
the
> > > messages from that day. I believe you captured the details of
our
> > > discussions very well. For the purpose of the DANA document, the
> > > completion of the main body of verification work for WRE-N
ensemble
> > > DC and Moore hybrid DA will remain as FY19 accomplishment
targets (I
> > > don't think Ben wants us to modify the
> > > FY19 accomplishments already listed in the document), but I will
> > > allow for draft tech report and/or open lit publication efforts
with
> > > these studies to carry into FY20 (almost certain to be the
case). As
> > > to one thing Brian asked, and I am not sure he has received an
answer:
> > >
> > > "I am not sure if we discussed this but with bias do we know if
the
> > > standard practice is to weight each time-model run pair equally,
or
> > > does one weight by the number of neighborhoods where the obs
> > > exceeded the threshold?"
> > >
> > >
> > >
> > > Bob
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > > Sent: Friday, July 12, 2019 11:47 AM
> > > To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > > <robert.e.dumais.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC
ARL
> > > (USA) <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY
CCDC
> > > ARL (USA) <huaqing.cai.civ at mail.mil>; Raby, John W CIV USARMY
CCDC
> > > ARL
> > > (USA) <john.w.raby2.civ at mail.mil>
> > > Subject: Moore convection path forward (UNCLASSIFIED)
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > All,
> > > Based on our discussions yesterday, the following is my
> > > understanding of the current status and the path forward.
> > >
> > > Verification of radar data:
> > > MET has been run to do the verification on column maximum
> > > reflectivity but the output still needs to be aggregated and
> > > plotted.  Jeffrey has done work towards this.  The goal is to
> > > aggregate per experiment by lead time and per experiment by
actual
> > > time for FSS and bias.  Since FSS requires specifying a
neighborhood
> > > size and a dBZ threshold, and there are many such possible
> > > combinations we will need to figure out a very small number of
these
> > > to focus on.  Additionally, there are questions about how to
> > > aggregate FSS properly.  It is not clear if this can be done
from
> > > the MET output files we already have or if this will need to be
done
> > > via MET stat-analysis.  I am not sure if we discussed this but
with
> > > bias do we know if the standard practice is to weight each
> > > time-model run pair equally, or does one weight by the number of
> > > neighborhoods where the obs exceeded the threshold?
> > >
> > > Action: Huaqing Cai and John Raby will contact John Gotway to
better
> > > understand how to aggregate FSS.
> > > Action: Huaqing Cai will create plots showing the variation in
FSS
> > > by neighborhood size and dBZ threshold to help determine which
> > > combinations of these we wish to examine.
> > >
> > > When it is understood how to aggregate FSS and we determine what
> > > neighborhood size / dBZ threshold combinations we will be
examining,
> > > someone will need to do the aggregation and create the desired
plots.
> > > After the plots are created, analysis of the results can
proceed.
> > >
> > > Verification of surface data:
> > > This has been completed and plots are available on the WSMR
version
> > > of the branch drive.  (This drive is not automatically mapped at
ALC
> > > but can be accessed and setup to automatically map via its name
> > > [\\arlwa7001546h.arl.army.mil]). The location on that drive of
the
> > > data
> > > is:
> > > \hybrid vLAPS nudging project\plots_old The data on which these
> > > plots are based are available at:
> > > Z:\hybrid vLAPS nudging project\tables_old My understanding is
that
> > > the directories end with _old because Jeffrey has been hoping to
> > > generate these via a new setup that is not quite ready, but that
the
> > > data in these plots and tables is correct.  The plots available
show
> > > data aggregated per experiment by lead time and actual time, as
well
> > > as of individual experiments.  There is a lot of information
> > > available in these plots.  For now the plan is that the surface
data
> > > verification will be used to help interpret the verification of
the
> > > radar data.
> > >
> > > Please let me know if there is something that I did not
accurately
> > > convey or if there are aspects that I missed.  Thank you John
for
> > > your verification efforts and for Jeffrey for all of your work
in
> > > taking the output from MET and converting that into very helpful
> > > graphics, and your work in creating R code to assist with use of
MET
> > > output.
> > >
> > > Brian
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > > <robert.e.dumais.civ at mail.mil>
> > > Sent: Thursday, July 11, 2019 10:21 AM
> > > To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > > <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC
ARL
> > > (USA) <jeffrey.a.smith1.civ at mail.mil
> > > >;
> > > Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>;
> > > Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > > Subject: Confirmed- meet in small conference room at 1330 MDT
for
> > > telecon regarding Moore convection/DC ensemble studies
> > >
> > > All,
> > >
> > >    Gina confirmed that we can use the small conference room for
our
> > > telecon today at 1330 MDT (1530 EDT).  We don't really need the
VTC-
> > > just a regular dial in connection for Brian should be good
enough.
> > > We can either dial into Brian's office number or vica-versa. I
say
> > > we just
> > dial
> > > into his number.
> > >
> > >
> > > Bob
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore convection path forward (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Jul 18 09:39:09 2019

Sounds good.
Thanks, John.

R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, July 18, 2019 9:34 AM
To: Raby, John W CIV USARMY CCDC ARL (USA)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore
convection path forward (UNCLASSIFIED)

All active links contained in this email were disabled.  Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.




----

John,

With the goal of keeping things better organized, I'm going to resolve
this
current RT-ticket and create a new for your questions about installing
METviewer.  And I'll assign that new ticket to Tatiana.  She should be
able
to answer your questions about METviewer.

Thanks,
John

On Wed, Jul 17, 2019 at 2:15 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91105 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for your response and info on METViewer and for your offer to
set
> up
> the new database. I'll send you the .stat files for each of 5 cycles
of
> runs
> from a specific experimental model run. There are a total of 9
different
> experimental model which were run. I am starting the preps to have
> METViewer
> installed on my local linux workstation. I already have Java and
need to
> have
> R and MySQL installed. Anything else?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Wednesday, July 17, 2019 1:23 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore
> convection path forward (UNCLASSIFIED)
>
> All active links contained in this email were disabled.  Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Yes, this is NCAR's public facing METviewer instance:
> Caution-Caution-http://www.dtcenter.org/met/metviewer/metviewer1.jsp
>
> If you send me your .stat output files, I could pretty easily load
it into
> a
> new database there.  Then you could play around with your data and
decide
> if
> METviewer is worth the setup effort to you.
>
> NOAA's AWS instance of METviewer is not publicly accessible.  It sit
> behind a
> firewall.  If the Army is interested in using AWS, it should be
fairly
> painless to clone that AWS setup for your use.  But then there's the
> headache
> of figuring out how to work on AWS and pay them for your use of it.
>
> If you'd like to proceed with installing METviewer at ARL, let me
know,
> and
> I'll reassign this ticket to Tatiana.
>
> Thanks,
> John
>
> On Wed, Jul 17, 2019 at 9:25 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91105 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I have access to the METViewer at:
> > Caution-Caution-
http://www.dtcenter.org/met/metviewer/metviewer1.jsp
> >
> > Is this instance something which would work for me?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-Caution-
mailto:met_help at ucar.edu]
> > Sent: Tuesday, July 16, 2019 4:10 PM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91105] FW: Moore
> > convection path forward (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled.  Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > As we discussed in our telecon today, stat_analysis supports both
> > methods of summarizing the results across multiple cases that
Brian
> > referenced:
> >
> > BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2 =
> > ~3.01
> >
> > In STAT-Analysis, this is done with a *summary* job type:
> >    -job summary -line_type CTS -column FBIAS -by
> > MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
> > The output summary columns include the mean (i.e. average) of the
> > input column of data.
> >
> > Or via:
> > BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
> > O_RATE(t=1)+O_RATE(t=2) ] = [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
> >
> > In STAT-Analysis, this is done using an *aggregate_stat* job type:
> >    -job aggregate_stat -line_type CTC -out_line_type CTS -by
> > MODEL,FCST_LEAD,FCST_VAR,FCST_THRESH
> > The input CTC counts and summed up and the output CTS statistics
are
> > computed from those counts.  Results are reported separately for
each
> > unique combination of the values found in the columns listed for
the
> > "-by"
> > option.
> >
> > Thanks,
> > John
> >
> > On Tue, Jul 16, 2019 at 3:28 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jul 16 15:19:37 2019: Request 91105 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > >        Queue: met_help
> > >      Subject: FW: Moore convection path forward (UNCLASSIFIED)
> > >        Owner: Nobody
> > >   Requestors: john.w.raby2.civ at mail.mil
> > >       Status: new
> > >  Ticket <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91
> > > 105 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > > Sent: Tuesday, July 16, 2019 1:34 PM
> > > To: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> > > <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY CCDC
ARL
> > > (USA) <huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY
> > > FUTURES COMMAND
> > > (USA) <robert.e.dumais.civ at mail.mil>; Raby, John W CIV USARMY
CCDC
> > > ARL
> > > (USA)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Huaqing,
> > > Thank you for the additional details.  If I understand
correctly,
> > > F_RATE is the forecast rate, i.e., the number of neighborhoods
where
> > > the forecast exceeded the threshold, and similarly for O_RATE,
but
> > > for observations.
> > >
> > > I am trying to think through what added value there is to
reported
> > > bias for a particular neighborhood size rather than the overall
bias
> > > (which I think would be equivalent to a neighborhood size of 1).
Do
> > > you see benefits to reporting bias for a particular neighborhood
> > > size, rather than reporting the overall bias?
> > >
> > > The issue regarding the neighborhood size dependence of the bias
was
> > > not what I was trying to ask about in my previous email.  What I
was
> > > asking can be illustrated using a case with two times.
> > >
> > > Time 1:
> > > F_RATE(t=1) = 5
> > > O_RATE(t=1) = 1
> > > BIAS(t=1) = 5
> > >
> > > Time 2:
> > > F_RATE(t=2) = 50
> > > O_RATE(t=2) = 49
> > > BIAS(t=2) = ~1.02
> > >
> > > Now, if we wanted the overall bias, my question is whether we
would
> > > calculate it via:
> > > BIAS(average) = [ BIAS(t=1) + BIAS(t=2) ] / 2 = [ 5 + 1.02 ] / 2
=
> > > ~3.01
> > >
> > > Or via:
> > > BIAS (average) = [ F_RATE(t=1)+F_RATE(t=2) ] / [
> > > O_RATE(t=1)+O_RATE(t=2) ] = [ 5 + 50 ] / [ 1 + 49] = 55/50 = 1.1
> > >
> > > Brian
> > >
> > > -----Original Message-----
> > > From: Smith, Jeffrey A CIV USARMY CCDC ARL (USA)
> > > <jeffrey.a.smith1.civ at mail.mil>
> > > Sent: Tuesday, July 16, 2019 12:01 PM
> > > To: Cai, Huaqing CIV USARMY CCDC ARL (USA)
> > > <huaqing.cai.civ at mail.mil>; Dumais, Robert E Jr CIV USARMY
FUTURES
> > > COMMAND (USA) <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV
> > > USARMY CCDC ARL
> > > (USA) <brian.p.reen.civ at mail.mil>; Raby, John W CIV USARMY CCDC
ARL
> > > (USA) <john.w.raby2.civ at mail.mil>
> > > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> > >
> > > All.
> > >
> > > Couple of points relating to Brians summary below, and a few
related
> > > things.
> > >
> > > Once I am back from TDY I'll rebuild the data sets.  I need to
get
> > > back to that anyway, and I know the solution, just haven't had
time.
> > > What that will do is give us the data in R package form that
anyone
> > > can use for analysis.
> > > Don't have to worry about all the various files, etc.
> > >
> > > Once I have that, I can take what I have already done in R and
> > > reproduce the data sets using the packaged data as a starting
point.
> > > What that means is then if we want to give this analysis to
someone
> > > else, or reproduce it later, the combination of the data
package,
> > > and the code in package form, should be sufficient to accomplish
that.
> > >
> > > My next step after rebuilding these data sets, is to make the
code
> > > executable on the HPC.  For most of the data sets I've seen, I
think
> > > I know enough to make the data package prep process automatable
as a
> > > step after what Leelinda has done.  The package will likely look
> > > like three sets of data, one forecast, one obs, and one bias
> > > (because its
> > easy to
> > > generate).
> > > I'll right some extractors that will pull specific data
variables
> > > from the sets since from an abstract point of view, each will
look
> > > exactly
> > the
> > > same.
> > >
> > > For the BIAS calculation that Huaqing provides below, I've
already
> > > done that in some test cases that John has provided.  Its
actually
> > > easy to add things like that in R with some of the tools R
brings to
> > > the table since calculations like that are operations on atomic
data.
> > >
> > > Regards,
> > >
> > > Jeffrey
> > >
> > >
> > > -----Original Message-----
> > > From: Cai, Huaqing CIV USARMY CCDC ARL (USA)
> > > <huaqing.cai.civ at mail.mil>
> > > Sent: Tuesday, July 16, 2019 9:34 AM
> > > To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > > <robert.e.dumais.civ at mail.mil>; Reen, Brian P CIV USARMY CCDC
ARL
> > > (USA) <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY
CCDC
> > > ARL (USA) <jeffrey.a.smith1.civ at mail.mil>; Raby, John W CIV
USARMY
> > > CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
> > > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> > >
> > > HI, Brian, Bob, John and Jeffrey,
> > >
> > > Regarding to Brian's question below, although I am not 100% sure
> > > what he was asking, here is my attempt trying to answer it. Let
me
> > > know if it make any sense to you.
> > >
> > > BIAS is a standard parameter coming from calculations related to
CSI
> > > (e.g., POD, FAR, BIAS, and CSI). Theoretically, BIAS has nothing
to
> > > do with FSS and its associated neighborhood sizes, but it is a
> > > function of threshold. When we show FSS and BIAS, we basically
are
> > > using two kinds of verification methods, that is, the CSI and
the FSS
> > > method.
> > >
> > > When I calculated BIAS from txt files containing FSS results, I
used
> > > O_RATE and F_RATE. Ideally BIAS=F_RATE/O_RATE, and BIAS should
not
> > > change with different neighborhood sizes. Unfortunately, because
of
> > > the edge effects, the number of matched pairs decreases with
> > > increased neighborhood sizes.
> > > This edge effects caused BIAS changes with neighborhood size
> > > slightly for small neighborhood sizes, but significantly for
very
> > > large neighborhood sizes. Fortunately, what we want for BIAS is
the
> > > small neighborhood sizes, which are very similar for many small
> > > neighborhood sizes (one example is BIAS changed from 0.51 to
0.44
> > > for neighborhood size from 1 to 11 grid sizes).  So just keep in
> > > mind that the BIAS we shown was the BIAS for that neighborhood
size.
> > > We can change the MET setting and/or my calculation of BIAS so
that
> > > we always use the smallest neighborhood size for BIAS, if
necessary.
> > >
> > > Hope this helps.
> > >
> > > Huaqing
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > > Sent: Monday, July 15, 2019 10:57 AM
> > > To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > > <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC
ARL
> > > (USA) <jeffrey.a.smith1.civ at mail.mil
> > > >;
> > > Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>;
> > > Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > > Subject: RE: Moore convection path forward (UNCLASSIFIED)
> > >
> > > Hi again Brian,
> > >
> > >   I was off Friday, so just finally getting to respond to all
the
> > > messages from that day. I believe you captured the details of
our
> > > discussions very well. For the purpose of the DANA document, the
> > > completion of the main body of verification work for WRE-N
ensemble
> > > DC and Moore hybrid DA will remain as FY19 accomplishment
targets (I
> > > don't think Ben wants us to modify the
> > > FY19 accomplishments already listed in the document), but I will
> > > allow for draft tech report and/or open lit publication efforts
with
> > > these studies to carry into FY20 (almost certain to be the
case). As
> > > to one thing Brian asked, and I am not sure he has received an
answer:
> > >
> > > "I am not sure if we discussed this but with bias do we know if
the
> > > standard practice is to weight each time-model run pair equally,
or
> > > does one weight by the number of neighborhoods where the obs
> > > exceeded the threshold?"
> > >
> > >
> > >
> > > Bob
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > > Sent: Friday, July 12, 2019 11:47 AM
> > > To: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > > <robert.e.dumais.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC
ARL
> > > (USA) <jeffrey.a.smith1.civ at mail.mil>; Cai, Huaqing CIV USARMY
CCDC
> > > ARL (USA) <huaqing.cai.civ at mail.mil>; Raby, John W CIV USARMY
CCDC
> > > ARL
> > > (USA) <john.w.raby2.civ at mail.mil>
> > > Subject: Moore convection path forward (UNCLASSIFIED)
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > All,
> > > Based on our discussions yesterday, the following is my
> > > understanding of the current status and the path forward.
> > >
> > > Verification of radar data:
> > > MET has been run to do the verification on column maximum
> > > reflectivity but the output still needs to be aggregated and
> > > plotted.  Jeffrey has done work towards this.  The goal is to
> > > aggregate per experiment by lead time and per experiment by
actual
> > > time for FSS and bias.  Since FSS requires specifying a
neighborhood
> > > size and a dBZ threshold, and there are many such possible
> > > combinations we will need to figure out a very small number of
these
> > > to focus on.  Additionally, there are questions about how to
> > > aggregate FSS properly.  It is not clear if this can be done
from
> > > the MET output files we already have or if this will need to be
done
> > > via MET stat-analysis.  I am not sure if we discussed this but
with
> > > bias do we know if the standard practice is to weight each
> > > time-model run pair equally, or does one weight by the number of
> > > neighborhoods where the obs exceeded the threshold?
> > >
> > > Action: Huaqing Cai and John Raby will contact John Gotway to
better
> > > understand how to aggregate FSS.
> > > Action: Huaqing Cai will create plots showing the variation in
FSS
> > > by neighborhood size and dBZ threshold to help determine which
> > > combinations of these we wish to examine.
> > >
> > > When it is understood how to aggregate FSS and we determine what
> > > neighborhood size / dBZ threshold combinations we will be
examining,
> > > someone will need to do the aggregation and create the desired
plots.
> > > After the plots are created, analysis of the results can
proceed.
> > >
> > > Verification of surface data:
> > > This has been completed and plots are available on the WSMR
version
> > > of the branch drive.  (This drive is not automatically mapped at
ALC
> > > but can be accessed and setup to automatically map via its name
> > > [\\arlwa7001546h.arl.army.mil]). The location on that drive of
the
> > > data
> > > is:
> > > \hybrid vLAPS nudging project\plots_old The data on which these
> > > plots are based are available at:
> > > Z:\hybrid vLAPS nudging project\tables_old My understanding is
that
> > > the directories end with _old because Jeffrey has been hoping to
> > > generate these via a new setup that is not quite ready, but that
the
> > > data in these plots and tables is correct.  The plots available
show
> > > data aggregated per experiment by lead time and actual time, as
well
> > > as of individual experiments.  There is a lot of information
> > > available in these plots.  For now the plan is that the surface
data
> > > verification will be used to help interpret the verification of
the
> > > radar data.
> > >
> > > Please let me know if there is something that I did not
accurately
> > > convey or if there are aspects that I missed.  Thank you John
for
> > > your verification efforts and for Jeffrey for all of your work
in
> > > taking the output from MET and converting that into very helpful
> > > graphics, and your work in creating R code to assist with use of
MET
> > > output.
> > >
> > > Brian
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Dumais, Robert E Jr CIV USARMY FUTURES COMMAND (USA)
> > > <robert.e.dumais.civ at mail.mil>
> > > Sent: Thursday, July 11, 2019 10:21 AM
> > > To: Reen, Brian P CIV USARMY CCDC ARL (USA)
> > > <brian.p.reen.civ at mail.mil>; Smith, Jeffrey A CIV USARMY CCDC
ARL
> > > (USA) <jeffrey.a.smith1.civ at mail.mil
> > > >;
> > > Cai, Huaqing CIV USARMY CCDC ARL (USA)
<huaqing.cai.civ at mail.mil>;
> > > Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > > Subject: Confirmed- meet in small conference room at 1330 MDT
for
> > > telecon regarding Moore convection/DC ensemble studies
> > >
> > > All,
> > >
> > >    Gina confirmed that we can use the small conference room for
our
> > > telecon today at 1330 MDT (1530 EDT).  We don't really need the
VTC-
> > > just a regular dial in connection for Brian should be good
enough.
> > > We can either dial into Brian's office number or vica-versa. I
say
> > > we just
> > dial
> > > into his number.
> > >
> > >
> > > Bob
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>



------------------------------------------------


More information about the Met_help mailing list