[Met_help] [rt.rap.ucar.edu #96714] History for FW: Question about V2.11 METViewer (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Tue Sep 22 09:25:30 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Julie -
Thanks for letting me know about the misg text. This is a forwarded copy below.
R/
John
________________________________________
From: Raby, John W CIV USARMY CCDC ARL (USA)
Sent: Monday, September 14, 2020 4:10 PM
To: 'met_help at ucar.edu'
Subject: Question about V2.11 METViewer (UNCLASSIFIED)
CLASSIFICATION: UNCLASSIFIED
CLASSIFICATION: UNCLASSIFIED
I have a database populated with 2376 Grid-Stat STAT files and I generated a
plot of FSS vs lead time shown the attached PNG file. The settings used for
this plot as as follows:
Y1 series vrbl: TMP --> NBR_FSS for WRF with ind vrbl: FCST_LEAD.
The aggregated statistics plotted were "Nbrhood method count" (NBR_CNT).
The R-data (attached) shows that the Nstat value for each lead time is
19800. See the line for the first lead time below.
TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1 NA NA
19800
From <http://carson2.arl.army.mil:8080/metviewer/servlet?jsp=new>
The plot shows perfect FSS values (1.0) for all leads. I have also attached
the Rscript, xml, and log files.
Are the plotted aggregated FSS values at each lead averaged over all 8
thresholds and 24 neighborhood sizes (interp_pnts)? Is the NStat value
(19800) computed from the total number of FSS values available in the 2376
STAT files? (19800/2376=8.33). This seems to indicate that it was using the
values of FSS for each of the 8 thresholds, but why doesn't the math work
out to <8 since sometimes the value of FSS was "NA" at certain thresholds. I
guess I need to understand more about the computation of the FSS using thes
above settings.
I suppose that to break out the FSS for a specified combination of threshold
value and neighborhood size, I would add "fixed value" settings for each of
those plots to accomplish this, correct?
Thanks.
R/
John
Mr. John W. Raby
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
Office:(575) 678-2004 DSN 258-2004
Teleworking: Contact Admin Specialist at (410) 278-6974
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
CLASSIFICATION: UNCLASSIFIED
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Sep 15 07:50:22 2020
Hi John,
This is John Halley Gotway. Based on the data you sent, I can't really
tell
you very much about this plot.
I do see in the R-script file that you're plotting the "mean" (or
average)
value for each of the lead times:
strPlotStat = "mean";
Another option is median, but I see that you're using the mean
instead.
However, one detail about METviewer that can be frustrating at times
is
that it silently groups together multiple buckets of data, unless you
tell
it not to. In this case, you've stated that you're using 8 thresholds
and
24 neighborhood sizes. But there's nothing the METviewer that
indicates
that fact.
When making a new plot with METviewer, I always recommend that users
check
every single option in the "Fixed Values" section. If multiple values
are
listed for one, pick one of them. Of course, you won't select a single
value for whatever you plan to put on the X-axis (e.g. lead time). And
you
won't select ones that have been used to define the series (e.g. often
that's different model names).
So I'd recommend going back and picking a single threshold and
neighborhood
size. And once you understand how METviewer is plotting the data, you
can
add in more grouping and aggregation.
Hope that helps.
John
On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
>
>
> Julie -
>
> Thanks for letting me know about the misg text. This is a forwarded
copy
> below.
> R/
> John
> ________________________________________
> From: Raby, John W CIV USARMY CCDC ARL (USA)
> Sent: Monday, September 14, 2020 4:10 PM
> To: 'met_help at ucar.edu'
> Subject: Question about V2.11 METViewer (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> CLASSIFICATION: UNCLASSIFIED
>
> I have a database populated with 2376 Grid-Stat STAT files and I
generated
> a
> plot of FSS vs lead time shown the attached PNG file. The settings
used for
> this plot as as follows:
>
> Y1 series vrbl: TMP --> NBR_FSS for WRF with ind vrbl:
FCST_LEAD.
> The aggregated statistics plotted were "Nbrhood method count"
(NBR_CNT).
> The R-data (attached) shows that the Nstat value for each lead time
is
> 19800. See the line for the first lead time below.
> TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1 NA
NA
> 19800
>
> From
<http://carson2.arl.army.mil:8080/metviewer/servlet?jsp=new>
>
> The plot shows perfect FSS values (1.0) for all leads. I have also
attached
> the Rscript, xml, and log files.
>
> Are the plotted aggregated FSS values at each lead averaged over all
8
> thresholds and 24 neighborhood sizes (interp_pnts)? Is the NStat
value
> (19800) computed from the total number of FSS values available in
the 2376
> STAT files? (19800/2376=8.33). This seems to indicate that it was
using the
> values of FSS for each of the 8 thresholds, but why doesn't the math
work
> out to <8 since sometimes the value of FSS was "NA" at certain
thresholds.
> I
> guess I need to understand more about the computation of the FSS
using thes
> above settings.
>
> I suppose that to break out the FSS for a specified combination of
> threshold
> value and neighborhood size, I would add "fixed value" settings for
each of
> those plots to accomplish this, correct?
>
>
> Thanks.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> Office:(575) 678-2004 DSN 258-2004
> Teleworking: Contact Admin Specialist at (410) 278-6974
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> CLASSIFICATION: UNCLASSIFIED
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Sep 15 08:34:51 2020
CLASSIFICATION: UNCLASSIFIED
Hi John -
Thanks for taking a look at that output and providing some insight on
how to
approach using METViewer. I actually generated the attached plot along
the
lines of you suggestion below where I picked some specific values for
the
threshold and the neighborhood size and that plot. Notice that it also
shows
FSS = 1.0 for all leads except the first one which was 0.964. Notice
also,
that the NStats values are now reduced to 99 per lead vice the 19800
values in
the plot I sent yesterday. So, it's clear that picking a specific
threshold
and neighborhood size narrows down the number of values which were
aggregated
to produce the FSS values at each lead. Just so you can understand
more about
this plot, I captured the METViewer settings in the attached file with
"settings" in the filename. The settings I used for the plot I sent
yesterday
were identical except for specifying fixed values for threshold and
neighborhood size.
I find it surprising that, despite the reduction on the NStats values
from
19800 to 99, the FSS values are still perfect with that one exception.
I see now that I'll need to explore more values for threshold and
neighborhood
size to get a feel for where the performance drops from perfect to a
lesser
value to capture the dependence on scale size and threshold value.
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, September 15, 2020 7:50 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW: Question
about
V2.11 METViewer (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
Hi John,
This is John Halley Gotway. Based on the data you sent, I can't really
tell
you very much about this plot.
I do see in the R-script file that you're plotting the "mean" (or
average)
value for each of the lead times:
strPlotStat = "mean";
Another option is median, but I see that you're using the mean
instead.
However, one detail about METviewer that can be frustrating at times
is that
it silently groups together multiple buckets of data, unless you tell
it not
to. In this case, you've stated that you're using 8 thresholds and
24 neighborhood sizes. But there's nothing the METviewer that
indicates that
fact.
When making a new plot with METviewer, I always recommend that users
check
every single option in the "Fixed Values" section. If multiple values
are
listed for one, pick one of them. Of course, you won't select a single
value
for whatever you plan to put on the X-axis (e.g. lead time). And you
won't
select ones that have been used to define the series (e.g. often
that's
different model names).
So I'd recommend going back and picking a single threshold and
neighborhood
size. And once you understand how METviewer is plotting the data, you
can add
in more grouping and aggregation.
Hope that helps.
John
On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
>
>
> Julie -
>
> Thanks for letting me know about the misg text. This is a forwarded
> copy below.
> R/
> John
> ________________________________________
> From: Raby, John W CIV USARMY CCDC ARL (USA)
> Sent: Monday, September 14, 2020 4:10 PM
> To: 'met_help at ucar.edu'
> Subject: Question about V2.11 METViewer (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> CLASSIFICATION: UNCLASSIFIED
>
> I have a database populated with 2376 Grid-Stat STAT files and I
> generated a plot of FSS vs lead time shown the attached PNG file.
The
> settings used for this plot as as follows:
>
> Y1 series vrbl: TMP --> NBR_FSS for WRF with ind vrbl:
FCST_LEAD.
> The aggregated statistics plotted were "Nbrhood method count"
(NBR_CNT).
> The R-data (attached) shows that the Nstat value for each lead time
is
> 19800. See the line for the first lead time below.
> TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1 NA
NA
> 19800
>
> From
> <Caution-http://carson2.arl.army.mil:8080/metviewer/servlet?jsp=new>
>
> The plot shows perfect FSS values (1.0) for all leads. I have also
> attached the Rscript, xml, and log files.
>
> Are the plotted aggregated FSS values at each lead averaged over all
8
> thresholds and 24 neighborhood sizes (interp_pnts)? Is the NStat
value
> (19800) computed from the total number of FSS values available in
the
> 2376 STAT files? (19800/2376=8.33). This seems to indicate that it
was
> using the values of FSS for each of the 8 thresholds, but why
doesn't
> the math work out to <8 since sometimes the value of FSS was "NA" at
certain
> thresholds.
> I
> guess I need to understand more about the computation of the FSS
using
> thes above settings.
>
> I suppose that to break out the FSS for a specified combination of
> threshold value and neighborhood size, I would add "fixed value"
> settings for each of those plots to accomplish this, correct?
>
>
> Thanks.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> Office:(575) 678-2004 DSN 258-2004
> Teleworking: Contact Admin Specialist at (410) 278-6974 FAX (575)
> 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> CLASSIFICATION: UNCLASSIFIED
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Sep 15 09:11:28 2020
John,
Yes, agreed. I would guess the most likely explanation for this
behavior is
that your threshold is either too low (meaning no events) or too high
(meaning events almost everywhere).
If you've also written the CTC and/or CTS line types for these
thresholds
to METviewer, you could plot the BASE_RATE statistic to see the
frequency
at which that event is occurring.
Or rather than looking in METviewer, you could step back and look at
the
output for a single case directly from Grid-Stat. In fact, if you're
running a single example, you could configure Grid-Stat to write the
neighborhood fractional coverage field used to compute FSS:
nc_pairs_flag = {
...
nbrhd = TRUE;
...
}
Then, the NetCDF output created by Grid-Stat will include fields
showing
the fractional coverage fields. And those may be interesting to look
at
using the ncview utility.
Thanks,
John
On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Hi John -
>
> Thanks for taking a look at that output and providing some insight
on how
> to
> approach using METViewer. I actually generated the attached plot
along the
> lines of you suggestion below where I picked some specific values
for the
> threshold and the neighborhood size and that plot. Notice that it
also
> shows
> FSS = 1.0 for all leads except the first one which was 0.964.
Notice
> also,
> that the NStats values are now reduced to 99 per lead vice the 19800
> values in
> the plot I sent yesterday. So, it's clear that picking a specific
> threshold
> and neighborhood size narrows down the number of values which were
> aggregated
> to produce the FSS values at each lead. Just so you can understand
more
> about
> this plot, I captured the METViewer settings in the attached file
with
> "settings" in the filename. The settings I used for the plot I sent
> yesterday
> were identical except for specifying fixed values for threshold and
> neighborhood size.
>
> I find it surprising that, despite the reduction on the NStats
values from
> 19800 to 99, the FSS values are still perfect with that one
exception.
>
> I see now that I'll need to explore more values for threshold and
> neighborhood
> size to get a feel for where the performance drops from perfect to a
> lesser
> value to capture the dependence on scale size and threshold value.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, September 15, 2020 7:50 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW: Question
about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Hi John,
>
> This is John Halley Gotway. Based on the data you sent, I can't
really
> tell
> you very much about this plot.
>
> I do see in the R-script file that you're plotting the "mean" (or
average)
> value for each of the lead times:
>
> strPlotStat = "mean";
>
>
> Another option is median, but I see that you're using the mean
instead.
>
>
> However, one detail about METviewer that can be frustrating at times
is
> that
> it silently groups together multiple buckets of data, unless you
tell it
> not
> to. In this case, you've stated that you're using 8 thresholds and
> 24 neighborhood sizes. But there's nothing the METviewer that
indicates
> that
> fact.
>
>
> When making a new plot with METviewer, I always recommend that users
check
> every single option in the "Fixed Values" section. If multiple
values are
> listed for one, pick one of them. Of course, you won't select a
single
> value
> for whatever you plan to put on the X-axis (e.g. lead time). And you
won't
> select ones that have been used to define the series (e.g. often
that's
> different model names).
>
>
> So I'd recommend going back and picking a single threshold and
> neighborhood
> size. And once you understand how METviewer is plotting the data,
you can
> add
> in more grouping and aggregation.
>
>
> Hope that helps.
>
>
> John
>
> On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> >
> >
> >
> > Julie -
> >
> > Thanks for letting me know about the misg text. This is a
forwarded
> > copy below.
> > R/
> > John
> > ________________________________________
> > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > Sent: Monday, September 14, 2020 4:10 PM
> > To: 'met_help at ucar.edu'
> > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I have a database populated with 2376 Grid-Stat STAT files and I
> > generated a plot of FSS vs lead time shown the attached PNG file.
The
> > settings used for this plot as as follows:
> >
> > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind vrbl:
> FCST_LEAD.
> > The aggregated statistics plotted were "Nbrhood method count"
(NBR_CNT).
> > The R-data (attached) shows that the Nstat value for each lead
time is
> > 19800. See the line for the first lead time below.
> > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1 NA
> NA
> > 19800
> >
> > From
> > <Caution-
http://carson2.arl.army.mil:8080/metviewer/servlet?jsp=new>
> >
> > The plot shows perfect FSS values (1.0) for all leads. I have also
> > attached the Rscript, xml, and log files.
> >
> > Are the plotted aggregated FSS values at each lead averaged over
all 8
> > thresholds and 24 neighborhood sizes (interp_pnts)? Is the NStat
value
> > (19800) computed from the total number of FSS values available in
the
> > 2376 STAT files? (19800/2376=8.33). This seems to indicate that it
was
> > using the values of FSS for each of the 8 thresholds, but why
doesn't
> > the math work out to <8 since sometimes the value of FSS was "NA"
at
> certain
> > thresholds.
> > I
> > guess I need to understand more about the computation of the FSS
using
> > thes above settings.
> >
> > I suppose that to break out the FSS for a specified combination of
> > threshold value and neighborhood size, I would add "fixed value"
> > settings for each of those plots to accomplish this, correct?
> >
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > Office:(575) 678-2004 DSN 258-2004
> > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX (575)
> > 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Sep 15 09:55:26 2020
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for sharing your thoughts on other things to try. I think I
specified
the CTC and/or CTS line types. I'll check the STAT file to look for
them
there. I also specified the ncpairs flag for nbrhd for all thresholds
and
neighborhoods. Those plots should give you a sense of whether the
combination
of threshold and neighborhood size are appropriate (not too
restrictive or not
too unrestrictive). When you have operational thresholds however, I
guess you
have to take it as "it is what it is" I suppose and then try to
exploit the
neighborhood size to draw out the dependency on that along.
I have another question about NBR_FBIAS as the Y1 dep variable. For
aggregated
statistics (NBR_CNT), METviewer is responding "No image available".
Even if I
specify the fixed values for threshold and neighborhood size. If
NBR_FSS is
working fine then why isn't NBR_FBIAS?
Thanks.
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, September 15, 2020 9:11 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question about
V2.11 METViewer (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Yes, agreed. I would guess the most likely explanation for this
behavior is
that your threshold is either too low (meaning no events) or too high
(meaning
events almost everywhere).
If you've also written the CTC and/or CTS line types for these
thresholds to
METviewer, you could plot the BASE_RATE statistic to see the frequency
at
which that event is occurring.
Or rather than looking in METviewer, you could step back and look at
the
output for a single case directly from Grid-Stat. In fact, if you're
running a
single example, you could configure Grid-Stat to write the
neighborhood
fractional coverage field used to compute FSS:
nc_pairs_flag = {
...
nbrhd = TRUE;
...
}
Then, the NetCDF output created by Grid-Stat will include fields
showing the
fractional coverage fields. And those may be interesting to look at
using the
ncview utility.
Thanks,
John
On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Hi John -
>
> Thanks for taking a look at that output and providing some insight
on
> how to approach using METViewer. I actually generated the attached
> plot along the lines of you suggestion below where I picked some
> specific values for the threshold and the neighborhood size and that
> plot. Notice that it also shows FSS = 1.0 for all leads except the
> first one which was 0.964. Notice also, that the NStats values are
now
> reduced to 99 per lead vice the 19800 values in the plot I sent
> yesterday. So, it's clear that picking a specific threshold and
> neighborhood size narrows down the number of values which were
> aggregated to produce the FSS values at each lead. Just so you can
> understand more about this plot, I captured the METViewer settings
in
> the attached file with "settings" in the filename. The settings I
used
> for the plot I sent yesterday were identical except for specifying
> fixed values for threshold and neighborhood size.
>
> I find it surprising that, despite the reduction on the NStats
values
> from
> 19800 to 99, the FSS values are still perfect with that one
exception.
>
> I see now that I'll need to explore more values for threshold and
> neighborhood size to get a feel for where the performance drops from
> perfect to a lesser value to capture the dependence on scale size
and
> threshold value.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, September 15, 2020 7:50 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW: Question
> about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> Hi John,
>
> This is John Halley Gotway. Based on the data you sent, I can't
really
> tell you very much about this plot.
>
> I do see in the R-script file that you're plotting the "mean" (or
> average) value for each of the lead times:
>
> strPlotStat = "mean";
>
>
> Another option is median, but I see that you're using the mean
instead.
>
>
> However, one detail about METviewer that can be frustrating at times
> is that it silently groups together multiple buckets of data, unless
> you tell it not to. In this case, you've stated that you're using 8
> thresholds and
> 24 neighborhood sizes. But there's nothing the METviewer that
> indicates that fact.
>
>
> When making a new plot with METviewer, I always recommend that users
> check every single option in the "Fixed Values" section. If multiple
> values are listed for one, pick one of them. Of course, you won't
> select a single value for whatever you plan to put on the X-axis
(e.g.
> lead time). And you won't select ones that have been used to define
> the series (e.g. often that's different model names).
>
>
> So I'd recommend going back and picking a single threshold and
> neighborhood size. And once you understand how METviewer is plotting
> the data, you can add in more grouping and aggregation.
>
>
> Hope that helps.
>
>
> John
>
> On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> >
> >
> >
> > Julie -
> >
> > Thanks for letting me know about the misg text. This is a
forwarded
> > copy below.
> > R/
> > John
> > ________________________________________
> > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > Sent: Monday, September 14, 2020 4:10 PM
> > To: 'met_help at ucar.edu'
> > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I have a database populated with 2376 Grid-Stat STAT files and I
> > generated a plot of FSS vs lead time shown the attached PNG file.
The
> > settings used for this plot as as follows:
> >
> > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind vrbl:
> FCST_LEAD.
> > The aggregated statistics plotted were "Nbrhood method count"
(NBR_CNT).
> > The R-data (attached) shows that the Nstat value for each lead
time is
> > 19800. See the line for the first lead time below.
> > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1 NA
> NA
> > 19800
> >
> > From
> > <Caution-Caution-
http://carson2.arl.army.mil:8080/metviewer/servlet?jsp=new>
> >
> > The plot shows perfect FSS values (1.0) for all leads. I have also
> > attached the Rscript, xml, and log files.
> >
> > Are the plotted aggregated FSS values at each lead averaged over
all 8
> > thresholds and 24 neighborhood sizes (interp_pnts)? Is the NStat
value
> > (19800) computed from the total number of FSS values available in
the
> > 2376 STAT files? (19800/2376=8.33). This seems to indicate that it
was
> > using the values of FSS for each of the 8 thresholds, but why
doesn't
> > the math work out to <8 since sometimes the value of FSS was "NA"
at
> certain
> > thresholds.
> > I
> > guess I need to understand more about the computation of the FSS
using
> > thes above settings.
> >
> > I suppose that to break out the FSS for a specified combination of
> > threshold value and neighborhood size, I would add "fixed value"
> > settings for each of those plots to accomplish this, correct?
> >
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > Office:(575) 678-2004 DSN 258-2004
> > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX (575)
> > 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Sep 15 14:53:27 2020
CLASSIFICATION: UNCLASSIFIED
John -
I am able to get the Y1 dep vrbl "FBIAS" to plot but I had to remove
the fixed
values for neighborhood size and threshold. The statistics were
aggregation of
CTC. Just can't figure out why I can't plot the NBR_FBIAS. My STAT
file has
the line types: NBRCNT, NBRCTS and NBRCTC. I'll see if the log file
provides
any clues.
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, September 15, 2020 9:11 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question about
V2.11 METViewer (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Yes, agreed. I would guess the most likely explanation for this
behavior is
that your threshold is either too low (meaning no events) or too high
(meaning
events almost everywhere).
If you've also written the CTC and/or CTS line types for these
thresholds to
METviewer, you could plot the BASE_RATE statistic to see the frequency
at
which that event is occurring.
Or rather than looking in METviewer, you could step back and look at
the
output for a single case directly from Grid-Stat. In fact, if you're
running a
single example, you could configure Grid-Stat to write the
neighborhood
fractional coverage field used to compute FSS:
nc_pairs_flag = {
...
nbrhd = TRUE;
...
}
Then, the NetCDF output created by Grid-Stat will include fields
showing the
fractional coverage fields. And those may be interesting to look at
using the
ncview utility.
Thanks,
John
On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Hi John -
>
> Thanks for taking a look at that output and providing some insight
on
> how to approach using METViewer. I actually generated the attached
> plot along the lines of you suggestion below where I picked some
> specific values for the threshold and the neighborhood size and that
> plot. Notice that it also shows FSS = 1.0 for all leads except the
> first one which was 0.964. Notice also, that the NStats values are
now
> reduced to 99 per lead vice the 19800 values in the plot I sent
> yesterday. So, it's clear that picking a specific threshold and
> neighborhood size narrows down the number of values which were
> aggregated to produce the FSS values at each lead. Just so you can
> understand more about this plot, I captured the METViewer settings
in
> the attached file with "settings" in the filename. The settings I
used
> for the plot I sent yesterday were identical except for specifying
> fixed values for threshold and neighborhood size.
>
> I find it surprising that, despite the reduction on the NStats
values
> from
> 19800 to 99, the FSS values are still perfect with that one
exception.
>
> I see now that I'll need to explore more values for threshold and
> neighborhood size to get a feel for where the performance drops from
> perfect to a lesser value to capture the dependence on scale size
and
> threshold value.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, September 15, 2020 7:50 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW: Question
> about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> Hi John,
>
> This is John Halley Gotway. Based on the data you sent, I can't
really
> tell you very much about this plot.
>
> I do see in the R-script file that you're plotting the "mean" (or
> average) value for each of the lead times:
>
> strPlotStat = "mean";
>
>
> Another option is median, but I see that you're using the mean
instead.
>
>
> However, one detail about METviewer that can be frustrating at times
> is that it silently groups together multiple buckets of data, unless
> you tell it not to. In this case, you've stated that you're using 8
> thresholds and
> 24 neighborhood sizes. But there's nothing the METviewer that
> indicates that fact.
>
>
> When making a new plot with METviewer, I always recommend that users
> check every single option in the "Fixed Values" section. If multiple
> values are listed for one, pick one of them. Of course, you won't
> select a single value for whatever you plan to put on the X-axis
(e.g.
> lead time). And you won't select ones that have been used to define
> the series (e.g. often that's different model names).
>
>
> So I'd recommend going back and picking a single threshold and
> neighborhood size. And once you understand how METviewer is plotting
> the data, you can add in more grouping and aggregation.
>
>
> Hope that helps.
>
>
> John
>
> On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> >
> >
> >
> > Julie -
> >
> > Thanks for letting me know about the misg text. This is a
forwarded
> > copy below.
> > R/
> > John
> > ________________________________________
> > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > Sent: Monday, September 14, 2020 4:10 PM
> > To: 'met_help at ucar.edu'
> > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I have a database populated with 2376 Grid-Stat STAT files and I
> > generated a plot of FSS vs lead time shown the attached PNG file.
The
> > settings used for this plot as as follows:
> >
> > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind vrbl:
> FCST_LEAD.
> > The aggregated statistics plotted were "Nbrhood method count"
(NBR_CNT).
> > The R-data (attached) shows that the Nstat value for each lead
time is
> > 19800. See the line for the first lead time below.
> > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1 NA
> NA
> > 19800
> >
> > From
> > <Caution-Caution-
http://carson2.arl.army.mil:8080/metviewer/servlet?jsp=new>
> >
> > The plot shows perfect FSS values (1.0) for all leads. I have also
> > attached the Rscript, xml, and log files.
> >
> > Are the plotted aggregated FSS values at each lead averaged over
all 8
> > thresholds and 24 neighborhood sizes (interp_pnts)? Is the NStat
value
> > (19800) computed from the total number of FSS values available in
the
> > 2376 STAT files? (19800/2376=8.33). This seems to indicate that it
was
> > using the values of FSS for each of the 8 thresholds, but why
doesn't
> > the math work out to <8 since sometimes the value of FSS was "NA"
at
> certain
> > thresholds.
> > I
> > guess I need to understand more about the computation of the FSS
using
> > thes above settings.
> >
> > I suppose that to break out the FSS for a specified combination of
> > threshold value and neighborhood size, I would add "fixed value"
> > settings for each of those plots to accomplish this, correct?
> >
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > Office:(575) 678-2004 DSN 258-2004
> > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX (575)
> > 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Sep 15 15:08:05 2020
CLASSIFICATION: UNCLASSIFIED
To follow-up on trying to get NBR_FBIAS to plot, I used the settings
in the
attached, but still get "no image available". The log file states: "
Failed to
create a plot. aggregation type nbr_cnt isn't compatible with the
statistic
NBR_FBIAS". I'm guessing that METViewer would need to use the nbr_cts
aggregation type, but I didn't see that option in the drop-down menu.
See the
attached for the menu choices for aggregation types.
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, September 15, 2020 9:11 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question about
V2.11 METViewer (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Yes, agreed. I would guess the most likely explanation for this
behavior is
that your threshold is either too low (meaning no events) or too high
(meaning
events almost everywhere).
If you've also written the CTC and/or CTS line types for these
thresholds to
METviewer, you could plot the BASE_RATE statistic to see the frequency
at
which that event is occurring.
Or rather than looking in METviewer, you could step back and look at
the
output for a single case directly from Grid-Stat. In fact, if you're
running a
single example, you could configure Grid-Stat to write the
neighborhood
fractional coverage field used to compute FSS:
nc_pairs_flag = {
...
nbrhd = TRUE;
...
}
Then, the NetCDF output created by Grid-Stat will include fields
showing the
fractional coverage fields. And those may be interesting to look at
using the
ncview utility.
Thanks,
John
On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Hi John -
>
> Thanks for taking a look at that output and providing some insight
on
> how to approach using METViewer. I actually generated the attached
> plot along the lines of you suggestion below where I picked some
> specific values for the threshold and the neighborhood size and that
> plot. Notice that it also shows FSS = 1.0 for all leads except the
> first one which was 0.964. Notice also, that the NStats values are
now
> reduced to 99 per lead vice the 19800 values in the plot I sent
> yesterday. So, it's clear that picking a specific threshold and
> neighborhood size narrows down the number of values which were
> aggregated to produce the FSS values at each lead. Just so you can
> understand more about this plot, I captured the METViewer settings
in
> the attached file with "settings" in the filename. The settings I
used
> for the plot I sent yesterday were identical except for specifying
> fixed values for threshold and neighborhood size.
>
> I find it surprising that, despite the reduction on the NStats
values
> from
> 19800 to 99, the FSS values are still perfect with that one
exception.
>
> I see now that I'll need to explore more values for threshold and
> neighborhood size to get a feel for where the performance drops from
> perfect to a lesser value to capture the dependence on scale size
and
> threshold value.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, September 15, 2020 7:50 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW: Question
> about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> Hi John,
>
> This is John Halley Gotway. Based on the data you sent, I can't
really
> tell you very much about this plot.
>
> I do see in the R-script file that you're plotting the "mean" (or
> average) value for each of the lead times:
>
> strPlotStat = "mean";
>
>
> Another option is median, but I see that you're using the mean
instead.
>
>
> However, one detail about METviewer that can be frustrating at times
> is that it silently groups together multiple buckets of data, unless
> you tell it not to. In this case, you've stated that you're using 8
> thresholds and
> 24 neighborhood sizes. But there's nothing the METviewer that
> indicates that fact.
>
>
> When making a new plot with METviewer, I always recommend that users
> check every single option in the "Fixed Values" section. If multiple
> values are listed for one, pick one of them. Of course, you won't
> select a single value for whatever you plan to put on the X-axis
(e.g.
> lead time). And you won't select ones that have been used to define
> the series (e.g. often that's different model names).
>
>
> So I'd recommend going back and picking a single threshold and
> neighborhood size. And once you understand how METviewer is plotting
> the data, you can add in more grouping and aggregation.
>
>
> Hope that helps.
>
>
> John
>
> On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> >
> >
> >
> > Julie -
> >
> > Thanks for letting me know about the misg text. This is a
forwarded
> > copy below.
> > R/
> > John
> > ________________________________________
> > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > Sent: Monday, September 14, 2020 4:10 PM
> > To: 'met_help at ucar.edu'
> > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I have a database populated with 2376 Grid-Stat STAT files and I
> > generated a plot of FSS vs lead time shown the attached PNG file.
The
> > settings used for this plot as as follows:
> >
> > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind vrbl:
> FCST_LEAD.
> > The aggregated statistics plotted were "Nbrhood method count"
(NBR_CNT).
> > The R-data (attached) shows that the Nstat value for each lead
time is
> > 19800. See the line for the first lead time below.
> > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1 NA
> NA
> > 19800
> >
> > From
> > <Caution-Caution-
http://carson2.arl.army.mil:8080/metviewer/servlet?jsp=new>
> >
> > The plot shows perfect FSS values (1.0) for all leads. I have also
> > attached the Rscript, xml, and log files.
> >
> > Are the plotted aggregated FSS values at each lead averaged over
all 8
> > thresholds and 24 neighborhood sizes (interp_pnts)? Is the NStat
value
> > (19800) computed from the total number of FSS values available in
the
> > 2376 STAT files? (19800/2376=8.33). This seems to indicate that it
was
> > using the values of FSS for each of the 8 thresholds, but why
doesn't
> > the math work out to <8 since sometimes the value of FSS was "NA"
at
> certain
> > thresholds.
> > I
> > guess I need to understand more about the computation of the FSS
using
> > thes above settings.
> >
> > I suppose that to break out the FSS for a specified combination of
> > threshold value and neighborhood size, I would add "fixed value"
> > settings for each of those plots to accomplish this, correct?
> >
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > Office:(575) 678-2004 DSN 258-2004
> > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX (575)
> > 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Sep 15 16:02:23 2020
John,
I'd caution you against using the NBRCTC counts and the NBRCTS
statistics.
The Fractions Skill Score (NBR_FSS) is far and away the most commonly
used
neighborhood-based statistic. The NBRCTC and NBRCTS lines are the
counts
and statistics you'd get after applying a threshold (cov_thresh) to
the
fractional coverage fields used to compute FSS. And I'm not aware of
any
published results that use that method. So that's why I'm a little
wary of
it.
But to answer your question... NBR_FBIAS in METviewer is just the
FBIAS value from the NBRCTS line type. The contents of that line type
are
described in this table:
https://dtcenter.github.io/MET/Users_Guide/grid-stat.html#id9
In exactly the same way that you aggregate CTC lines in METviewer to
compute CTS statistics... you do exactly the same in aggregating
NBRCTC
lines to compute NBRCTS statistics. So in the "Aggregation statistics"
dropdown list, you should select "Neighborhood contingency table
(NBRCTC)".
Now I haven't actually tested this on a real example to confirm, but
that's
what *should* work.
Thanks,
John
On Tue, Sep 15, 2020 at 3:08 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> To follow-up on trying to get NBR_FBIAS to plot, I used the settings
in
> the
> attached, but still get "no image available". The log file states: "
> Failed to
> create a plot. aggregation type nbr_cnt isn't compatible with the
> statistic
> NBR_FBIAS". I'm guessing that METViewer would need to use the
nbr_cts
> aggregation type, but I didn't see that option in the drop-down
menu. See
> the
> attached for the menu choices for aggregation types.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, September 15, 2020 9:11 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question
> about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Yes, agreed. I would guess the most likely explanation for this
behavior
> is
> that your threshold is either too low (meaning no events) or too
high
> (meaning
> events almost everywhere).
>
> If you've also written the CTC and/or CTS line types for these
thresholds
> to
> METviewer, you could plot the BASE_RATE statistic to see the
frequency at
> which that event is occurring.
>
> Or rather than looking in METviewer, you could step back and look at
the
> output for a single case directly from Grid-Stat. In fact, if you're
> running a
> single example, you could configure Grid-Stat to write the
neighborhood
> fractional coverage field used to compute FSS:
>
>
> nc_pairs_flag = {
>
> ...
>
> nbrhd = TRUE;
>
> ...
>
> }
>
>
> Then, the NetCDF output created by Grid-Stat will include fields
showing
> the
> fractional coverage fields. And those may be interesting to look at
using
> the
> ncview utility.
>
>
> Thanks,
> John
>
> On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Hi John -
> >
> > Thanks for taking a look at that output and providing some insight
on
> > how to approach using METViewer. I actually generated the attached
> > plot along the lines of you suggestion below where I picked some
> > specific values for the threshold and the neighborhood size and
that
> > plot. Notice that it also shows FSS = 1.0 for all leads except
the
> > first one which was 0.964. Notice also, that the NStats values are
now
> > reduced to 99 per lead vice the 19800 values in the plot I sent
> > yesterday. So, it's clear that picking a specific threshold and
> > neighborhood size narrows down the number of values which were
> > aggregated to produce the FSS values at each lead. Just so you
can
> > understand more about this plot, I captured the METViewer settings
in
> > the attached file with "settings" in the filename. The settings I
used
> > for the plot I sent yesterday were identical except for specifying
> > fixed values for threshold and neighborhood size.
> >
> > I find it surprising that, despite the reduction on the NStats
values
> > from
> > 19800 to 99, the FSS values are still perfect with that one
exception.
> >
> > I see now that I'll need to explore more values for threshold and
> > neighborhood size to get a feel for where the performance drops
from
> > perfect to a lesser value to capture the dependence on scale size
and
> > threshold value.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Tuesday, September 15, 2020 7:50 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question
> > about
> > V2.11 METViewer (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > Hi John,
> >
> > This is John Halley Gotway. Based on the data you sent, I can't
really
> > tell you very much about this plot.
> >
> > I do see in the R-script file that you're plotting the "mean" (or
> > average) value for each of the lead times:
> >
> > strPlotStat = "mean";
> >
> >
> > Another option is median, but I see that you're using the mean
instead.
> >
> >
> > However, one detail about METviewer that can be frustrating at
times
> > is that it silently groups together multiple buckets of data,
unless
> > you tell it not to. In this case, you've stated that you're using
8
> > thresholds and
> > 24 neighborhood sizes. But there's nothing the METviewer that
> > indicates that fact.
> >
> >
> > When making a new plot with METviewer, I always recommend that
users
> > check every single option in the "Fixed Values" section. If
multiple
> > values are listed for one, pick one of them. Of course, you won't
> > select a single value for whatever you plan to put on the X-axis
(e.g.
> > lead time). And you won't select ones that have been used to
define
> > the series (e.g. often that's different model names).
> >
> >
> > So I'd recommend going back and picking a single threshold and
> > neighborhood size. And once you understand how METviewer is
plotting
> > the data, you can add in more grouping and aggregation.
> >
> >
> > Hope that helps.
> >
> >
> > John
> >
> > On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-url:
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> > >
> > >
> > >
> > > Julie -
> > >
> > > Thanks for letting me know about the misg text. This is a
forwarded
> > > copy below.
> > > R/
> > > John
> > > ________________________________________
> > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > Sent: Monday, September 14, 2020 4:10 PM
> > > To: 'met_help at ucar.edu'
> > > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I have a database populated with 2376 Grid-Stat STAT files and I
> > > generated a plot of FSS vs lead time shown the attached PNG
file. The
> > > settings used for this plot as as follows:
> > >
> > > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind vrbl:
> > FCST_LEAD.
> > > The aggregated statistics plotted were "Nbrhood method count"
> (NBR_CNT).
> > > The R-data (attached) shows that the Nstat value for each lead
time is
> > > 19800. See the line for the first lead time below.
> > > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1
NA
> > NA
> > > 19800
> > >
> > > From
> > > <Caution-Caution-
> http://carson2.arl.army.mil:8080/metviewer/servlet?jsp=new>
> > >
> > > The plot shows perfect FSS values (1.0) for all leads. I have
also
> > > attached the Rscript, xml, and log files.
> > >
> > > Are the plotted aggregated FSS values at each lead averaged over
all 8
> > > thresholds and 24 neighborhood sizes (interp_pnts)? Is the NStat
value
> > > (19800) computed from the total number of FSS values available
in the
> > > 2376 STAT files? (19800/2376=8.33). This seems to indicate that
it was
> > > using the values of FSS for each of the 8 thresholds, but why
doesn't
> > > the math work out to <8 since sometimes the value of FSS was
"NA" at
> > certain
> > > thresholds.
> > > I
> > > guess I need to understand more about the computation of the FSS
using
> > > thes above settings.
> > >
> > > I suppose that to break out the FSS for a specified combination
of
> > > threshold value and neighborhood size, I would add "fixed value"
> > > settings for each of those plots to accomplish this, correct?
> > >
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > Office:(575) 678-2004 DSN 258-2004
> > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
(575)
> > > 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Sep 15 16:20:35 2020
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for sharing that info about the community use of FSS vs FBIAS.
I wasn't
aware of that. I tried setting up the plot you describe below, but I
find that
the statistics menu doesn't have "Neighborhood contingency table
(NBRCTC)".
It only has something called "Nbrhood method count (NBR_CNT)" which,
if
selected fails to produce the plot due to the incompatibility between
the
NBRCNT and the statistic NBR_FBIAS because the FBIAS is only output in
the
NBRCTS line type and not in the NBRCNT line type. Is my METViewer
missing
something or is a because the version is too old?
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, September 15, 2020 4:02 PM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question about
V2.11 METViewer (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
I'd caution you against using the NBRCTC counts and the NBRCTS
statistics.
The Fractions Skill Score (NBR_FSS) is far and away the most commonly
used
neighborhood-based statistic. The NBRCTC and NBRCTS lines are the
counts and
statistics you'd get after applying a threshold (cov_thresh) to the
fractional
coverage fields used to compute FSS. And I'm not aware of any
published
results that use that method. So that's why I'm a little wary of it.
But to answer your question... NBR_FBIAS in METviewer is just the
FBIAS value
from the NBRCTS line type. The contents of that line type are
described in
this table:
Caution-https://dtcenter.github.io/MET/Users_Guide/grid-
stat.html#id9
In exactly the same way that you aggregate CTC lines in METviewer to
compute
CTS statistics... you do exactly the same in aggregating NBRCTC lines
to
compute NBRCTS statistics. So in the "Aggregation statistics"
dropdown list, you should select "Neighborhood contingency table
(NBRCTC)".
Now I haven't actually tested this on a real example to confirm, but
that's
what *should* work.
Thanks,
John
On Tue, Sep 15, 2020 at 3:08 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> To follow-up on trying to get NBR_FBIAS to plot, I used the settings
> in the attached, but still get "no image available". The log file
> states: "
> Failed to
> create a plot. aggregation type nbr_cnt isn't compatible with the
> statistic NBR_FBIAS". I'm guessing that METViewer would need to use
> the nbr_cts aggregation type, but I didn't see that option in the
> drop-down menu. See the attached for the menu choices for
aggregation
> types.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, September 15, 2020 9:11 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> Question about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Yes, agreed. I would guess the most likely explanation for this
> behavior is that your threshold is either too low (meaning no
events)
> or too high (meaning events almost everywhere).
>
> If you've also written the CTC and/or CTS line types for these
> thresholds to METviewer, you could plot the BASE_RATE statistic to
see
> the frequency at which that event is occurring.
>
> Or rather than looking in METviewer, you could step back and look at
> the output for a single case directly from Grid-Stat. In fact, if
> you're running a single example, you could configure Grid-Stat to
> write the neighborhood fractional coverage field used to compute
FSS:
>
>
> nc_pairs_flag = {
>
> ...
>
> nbrhd = TRUE;
>
> ...
>
> }
>
>
> Then, the NetCDF output created by Grid-Stat will include fields
> showing the fractional coverage fields. And those may be interesting
> to look at using the ncview utility.
>
>
> Thanks,
> John
>
> On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Hi John -
> >
> > Thanks for taking a look at that output and providing some insight
on
> > how to approach using METViewer. I actually generated the attached
> > plot along the lines of you suggestion below where I picked some
> > specific values for the threshold and the neighborhood size and
that
> > plot. Notice that it also shows FSS = 1.0 for all leads except
the
> > first one which was 0.964. Notice also, that the NStats values are
now
> > reduced to 99 per lead vice the 19800 values in the plot I sent
> > yesterday. So, it's clear that picking a specific threshold and
> > neighborhood size narrows down the number of values which were
> > aggregated to produce the FSS values at each lead. Just so you
can
> > understand more about this plot, I captured the METViewer settings
in
> > the attached file with "settings" in the filename. The settings I
used
> > for the plot I sent yesterday were identical except for specifying
> > fixed values for threshold and neighborhood size.
> >
> > I find it surprising that, despite the reduction on the NStats
values
> > from
> > 19800 to 99, the FSS values are still perfect with that one
exception.
> >
> > I see now that I'll need to explore more values for threshold and
> > neighborhood size to get a feel for where the performance drops
from
> > perfect to a lesser value to capture the dependence on scale size
and
> > threshold value.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Tuesday, September 15, 2020 7:50 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question
> > about
> > V2.11 METViewer (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > Hi John,
> >
> > This is John Halley Gotway. Based on the data you sent, I can't
really
> > tell you very much about this plot.
> >
> > I do see in the R-script file that you're plotting the "mean" (or
> > average) value for each of the lead times:
> >
> > strPlotStat = "mean";
> >
> >
> > Another option is median, but I see that you're using the mean
instead.
> >
> >
> > However, one detail about METviewer that can be frustrating at
times
> > is that it silently groups together multiple buckets of data,
unless
> > you tell it not to. In this case, you've stated that you're using
8
> > thresholds and
> > 24 neighborhood sizes. But there's nothing the METviewer that
> > indicates that fact.
> >
> >
> > When making a new plot with METviewer, I always recommend that
users
> > check every single option in the "Fixed Values" section. If
multiple
> > values are listed for one, pick one of them. Of course, you won't
> > select a single value for whatever you plan to put on the X-axis
(e.g.
> > lead time). And you won't select ones that have been used to
define
> > the series (e.g. often that's different model names).
> >
> >
> > So I'd recommend going back and picking a single threshold and
> > neighborhood size. And once you understand how METviewer is
plotting
> > the data, you can add in more grouping and aggregation.
> >
> >
> > Hope that helps.
> >
> >
> > John
> >
> > On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-Caution-url:
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> > >
> > >
> > >
> > > Julie -
> > >
> > > Thanks for letting me know about the misg text. This is a
forwarded
> > > copy below.
> > > R/
> > > John
> > > ________________________________________
> > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > Sent: Monday, September 14, 2020 4:10 PM
> > > To: 'met_help at ucar.edu'
> > > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I have a database populated with 2376 Grid-Stat STAT files and I
> > > generated a plot of FSS vs lead time shown the attached PNG
file. The
> > > settings used for this plot as as follows:
> > >
> > > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind vrbl:
> > FCST_LEAD.
> > > The aggregated statistics plotted were "Nbrhood method count"
> (NBR_CNT).
> > > The R-data (attached) shows that the Nstat value for each lead
time is
> > > 19800. See the line for the first lead time below.
> > > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1
NA
> > NA
> > > 19800
> > >
> > > From
> > > <Caution-Caution-
> Caution-http://carson2.arl.army.mil:8080/metviewer/servlet?jsp=new>
> > >
> > > The plot shows perfect FSS values (1.0) for all leads. I have
also
> > > attached the Rscript, xml, and log files.
> > >
> > > Are the plotted aggregated FSS values at each lead averaged over
all 8
> > > thresholds and 24 neighborhood sizes (interp_pnts)? Is the NStat
value
> > > (19800) computed from the total number of FSS values available
in the
> > > 2376 STAT files? (19800/2376=8.33). This seems to indicate that
it was
> > > using the values of FSS for each of the 8 thresholds, but why
doesn't
> > > the math work out to <8 since sometimes the value of FSS was
"NA" at
> > certain
> > > thresholds.
> > > I
> > > guess I need to understand more about the computation of the FSS
using
> > > thes above settings.
> > >
> > > I suppose that to break out the FSS for a specified combination
of
> > > threshold value and neighborhood size, I would add "fixed value"
> > > settings for each of those plots to accomplish this, correct?
> > >
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > Office:(575) 678-2004 DSN 258-2004
> > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
(575)
> > > 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: John Halley Gotway
Time: Mon Sep 21 17:01:54 2020
John,
Sorry, I lost track of this email from last week. We were talking
about
METviewer and in an earlier email I described plotting NBR_FBIAS but
did
not send an example. When you tried to make the plot I described you
didn't
see an option to aggregate the NBRCTC line type and the question is
whether
or not METviewer supports that.
I see from your email that you're running METviewer 2.11, and version
3.1
is available here:
http://dtcenter.ucar.edu/met/metviewer/metviewer1.jsp
I worked up the attached example using that DTC instance of METviewer.
This
plot compares 6-hour precip from two different data sources. It has 4
lines
on it: 2 categorical thresholds (>1/4" green and >1/2" blue) * 2
neighborhood sizes (3x3 - dashed and 5x5 - solid).
The statistics plotted is "NBR_FBIAS"... but that's defined by
applying the
coverage threshold (cov_thresh) to define events. I'm guessing that
the
default value of "cov_thresh = >0.5;" was used, but I'm really not
positive.
But we noticed in a telecon this morning that "COV_THRESH" DOES NOT
exist
as an option in METviewer. But Tatiana (cc'ed here) will write up a
GitHub
issue to add it.
Make sense?
Thanks,
John
On Tue, Sep 15, 2020 at 4:21 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for sharing that info about the community use of FSS vs
FBIAS. I
> wasn't
> aware of that. I tried setting up the plot you describe below, but I
find
> that
> the statistics menu doesn't have "Neighborhood contingency table
> (NBRCTC)".
> It only has something called "Nbrhood method count (NBR_CNT)" which,
if
> selected fails to produce the plot due to the incompatibility
between the
> NBRCNT and the statistic NBR_FBIAS because the FBIAS is only output
in the
> NBRCTS line type and not in the NBRCNT line type. Is my METViewer
missing
> something or is a because the version is too old?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, September 15, 2020 4:02 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question
> about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I'd caution you against using the NBRCTC counts and the NBRCTS
statistics.
> The Fractions Skill Score (NBR_FSS) is far and away the most
commonly used
> neighborhood-based statistic. The NBRCTC and NBRCTS lines are the
counts
> and
> statistics you'd get after applying a threshold (cov_thresh) to the
> fractional
> coverage fields used to compute FSS. And I'm not aware of any
published
> results that use that method. So that's why I'm a little wary of it.
>
> But to answer your question... NBR_FBIAS in METviewer is just the
FBIAS
> value
> from the NBRCTS line type. The contents of that line type are
described in
> this table:
> Caution-https://dtcenter.github.io/MET/Users_Guide/grid-
stat.html#id9
>
> In exactly the same way that you aggregate CTC lines in METviewer to
> compute
> CTS statistics... you do exactly the same in aggregating NBRCTC
lines to
> compute NBRCTS statistics. So in the "Aggregation statistics"
> dropdown list, you should select "Neighborhood contingency table
(NBRCTC)".
>
> Now I haven't actually tested this on a real example to confirm, but
> that's
> what *should* work.
>
> Thanks,
> John
>
> On Tue, Sep 15, 2020 at 3:08 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > To follow-up on trying to get NBR_FBIAS to plot, I used the
settings
> > in the attached, but still get "no image available". The log file
> > states: "
> > Failed to
> > create a plot. aggregation type nbr_cnt isn't compatible with the
> > statistic NBR_FBIAS". I'm guessing that METViewer would need to
use
> > the nbr_cts aggregation type, but I didn't see that option in the
> > drop-down menu. See the attached for the menu choices for
aggregation
> > types.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Tuesday, September 15, 2020 9:11 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > Question about
> > V2.11 METViewer (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Yes, agreed. I would guess the most likely explanation for this
> > behavior is that your threshold is either too low (meaning no
events)
> > or too high (meaning events almost everywhere).
> >
> > If you've also written the CTC and/or CTS line types for these
> > thresholds to METviewer, you could plot the BASE_RATE statistic to
see
> > the frequency at which that event is occurring.
> >
> > Or rather than looking in METviewer, you could step back and look
at
> > the output for a single case directly from Grid-Stat. In fact, if
> > you're running a single example, you could configure Grid-Stat to
> > write the neighborhood fractional coverage field used to compute
FSS:
> >
> >
> > nc_pairs_flag = {
> >
> > ...
> >
> > nbrhd = TRUE;
> >
> > ...
> >
> > }
> >
> >
> > Then, the NetCDF output created by Grid-Stat will include fields
> > showing the fractional coverage fields. And those may be
interesting
> > to look at using the ncview utility.
> >
> >
> > Thanks,
> > John
> >
> > On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Hi John -
> > >
> > > Thanks for taking a look at that output and providing some
insight on
> > > how to approach using METViewer. I actually generated the
attached
> > > plot along the lines of you suggestion below where I picked some
> > > specific values for the threshold and the neighborhood size and
that
> > > plot. Notice that it also shows FSS = 1.0 for all leads except
the
> > > first one which was 0.964. Notice also, that the NStats values
are now
> > > reduced to 99 per lead vice the 19800 values in the plot I sent
> > > yesterday. So, it's clear that picking a specific threshold and
> > > neighborhood size narrows down the number of values which were
> > > aggregated to produce the FSS values at each lead. Just so you
can
> > > understand more about this plot, I captured the METViewer
settings in
> > > the attached file with "settings" in the filename. The settings
I used
> > > for the plot I sent yesterday were identical except for
specifying
> > > fixed values for threshold and neighborhood size.
> > >
> > > I find it surprising that, despite the reduction on the NStats
values
> > > from
> > > 19800 to 99, the FSS values are still perfect with that one
exception.
> > >
> > > I see now that I'll need to explore more values for threshold
and
> > > neighborhood size to get a feel for where the performance drops
from
> > > perfect to a lesser value to capture the dependence on scale
size and
> > > threshold value.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Tuesday, September 15, 2020 7:50 AM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question
> > > about
> > > V2.11 METViewer (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
verify
> > > the identity of the sender, and confirm the authenticity of all
links
> > > contained within the message prior to copying and pasting the
address
> > > to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Hi John,
> > >
> > > This is John Halley Gotway. Based on the data you sent, I can't
really
> > > tell you very much about this plot.
> > >
> > > I do see in the R-script file that you're plotting the "mean"
(or
> > > average) value for each of the lead times:
> > >
> > > strPlotStat = "mean";
> > >
> > >
> > > Another option is median, but I see that you're using the mean
instead.
> > >
> > >
> > > However, one detail about METviewer that can be frustrating at
times
> > > is that it silently groups together multiple buckets of data,
unless
> > > you tell it not to. In this case, you've stated that you're
using 8
> > > thresholds and
> > > 24 neighborhood sizes. But there's nothing the METviewer that
> > > indicates that fact.
> > >
> > >
> > > When making a new plot with METviewer, I always recommend that
users
> > > check every single option in the "Fixed Values" section. If
multiple
> > > values are listed for one, pick one of them. Of course, you
won't
> > > select a single value for whatever you plan to put on the X-axis
(e.g.
> > > lead time). And you won't select ones that have been used to
define
> > > the series (e.g. often that's different model names).
> > >
> > >
> > > So I'd recommend going back and picking a single threshold and
> > > neighborhood size. And once you understand how METviewer is
plotting
> > > the data, you can add in more grouping and aggregation.
> > >
> > >
> > > Hope that helps.
> > >
> > >
> > > John
> > >
> > > On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: FW: Question about V2.11 METViewer
(UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-url:
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> > > >
> > > >
> > > >
> > > > Julie -
> > > >
> > > > Thanks for letting me know about the misg text. This is a
forwarded
> > > > copy below.
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > Sent: Monday, September 14, 2020 4:10 PM
> > > > To: 'met_help at ucar.edu'
> > > > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > I have a database populated with 2376 Grid-Stat STAT files and
I
> > > > generated a plot of FSS vs lead time shown the attached PNG
file. The
> > > > settings used for this plot as as follows:
> > > >
> > > > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind
vrbl:
> > > FCST_LEAD.
> > > > The aggregated statistics plotted were "Nbrhood method count"
> > (NBR_CNT).
> > > > The R-data (attached) shows that the Nstat value for each lead
time
> is
> > > > 19800. See the line for the first lead time below.
> > > > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1
NA
> > > NA
> > > > 19800
> > > >
> > > > From
> > > > <Caution-Caution-
> > Caution-
http://carson2.arl.army.mil:8080/metviewer/servlet?jsp=new>
> > > >
> > > > The plot shows perfect FSS values (1.0) for all leads. I have
also
> > > > attached the Rscript, xml, and log files.
> > > >
> > > > Are the plotted aggregated FSS values at each lead averaged
over all
> 8
> > > > thresholds and 24 neighborhood sizes (interp_pnts)? Is the
NStat
> value
> > > > (19800) computed from the total number of FSS values available
in the
> > > > 2376 STAT files? (19800/2376=8.33). This seems to indicate
that it
> was
> > > > using the values of FSS for each of the 8 thresholds, but why
doesn't
> > > > the math work out to <8 since sometimes the value of FSS was
"NA" at
> > > certain
> > > > thresholds.
> > > > I
> > > > guess I need to understand more about the computation of the
FSS
> using
> > > > thes above settings.
> > > >
> > > > I suppose that to break out the FSS for a specified
combination of
> > > > threshold value and neighborhood size, I would add "fixed
value"
> > > > settings for each of those plots to accomplish this, correct?
> > > >
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > Office:(575) 678-2004 DSN 258-2004
> > > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
(575)
> > > > 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Sep 22 07:47:56 2020
CLASSIFICATION: UNCLASSIFIED
Hi John -
Thanks for following up in this. So, if you want to plot statistics by
aggregating the NBRCTC line type, the effective threshold value is set
by
COV_THRESH rather than using the CAT_THRESH. Is this true? But in the
case of
FSS, the thresholds are set by CAT_THRESH, because the line type is
now
NBRCNT, correct? Can you specify an array of COV_THRESH values? It
looks like
you can. I'm thinking of how I would run Grid-Stat so I would be able
to
output both FSS and NBR_FBIAS for multiple thresholds and neighborhood
sizes.
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, September 21, 2020 5:02 PM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question about
V2.11 METViewer (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Sorry, I lost track of this email from last week. We were talking
about
METviewer and in an earlier email I described plotting NBR_FBIAS but
did not
send an example. When you tried to make the plot I described you
didn't see an
option to aggregate the NBRCTC line type and the question is whether
or not
METviewer supports that.
I see from your email that you're running METviewer 2.11, and version
3.1 is
available here:
Caution-http://dtcenter.ucar.edu/met/metviewer/metviewer1.jsp
I worked up the attached example using that DTC instance of METviewer.
This
plot compares 6-hour precip from two different data sources. It has 4
lines on
it: 2 categorical thresholds (>1/4" green and >1/2" blue) * 2
neighborhood
sizes (3x3 - dashed and 5x5 - solid).
The statistics plotted is "NBR_FBIAS"... but that's defined by
applying the
coverage threshold (cov_thresh) to define events. I'm guessing that
the
default value of "cov_thresh = >0.5;" was used, but I'm really not
positive.
But we noticed in a telecon this morning that "COV_THRESH" DOES NOT
exist as
an option in METviewer. But Tatiana (cc'ed here) will write up a
GitHub issue
to add it.
Make sense?
Thanks,
John
On Tue, Sep 15, 2020 at 4:21 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for sharing that info about the community use of FSS vs
FBIAS.
> I wasn't aware of that. I tried setting up the plot you describe
> below, but I find that the statistics menu doesn't have
"Neighborhood
> contingency table (NBRCTC)".
> It only has something called "Nbrhood method count (NBR_CNT)" which,
> if selected fails to produce the plot due to the incompatibility
> between the NBRCNT and the statistic NBR_FBIAS because the FBIAS is
> only output in the NBRCTS line type and not in the NBRCNT line type.
> Is my METViewer missing something or is a because the version is too
old?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, September 15, 2020 4:02 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> Question about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> I'd caution you against using the NBRCTC counts and the NBRCTS
statistics.
> The Fractions Skill Score (NBR_FSS) is far and away the most
commonly
> used neighborhood-based statistic. The NBRCTC and NBRCTS lines are
the
> counts and statistics you'd get after applying a threshold
> (cov_thresh) to the fractional coverage fields used to compute FSS.
> And I'm not aware of any published results that use that method. So
> that's why I'm a little wary of it.
>
> But to answer your question... NBR_FBIAS in METviewer is just the
> FBIAS value from the NBRCTS line type. The contents of that line
type
> are described in this table:
>
> Caution-Caution-https://dtcenter.github.io/MET/Users_Guide/grid-
stat.h
> tml#id9
>
> In exactly the same way that you aggregate CTC lines in METviewer to
> compute CTS statistics... you do exactly the same in aggregating
> NBRCTC lines to compute NBRCTS statistics. So in the "Aggregation
> statistics"
> dropdown list, you should select "Neighborhood contingency table
(NBRCTC)".
>
> Now I haven't actually tested this on a real example to confirm, but
> that's what *should* work.
>
> Thanks,
> John
>
> On Tue, Sep 15, 2020 at 3:08 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > 714 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > To follow-up on trying to get NBR_FBIAS to plot, I used the
settings
> > in the attached, but still get "no image available". The log file
> > states: "
> > Failed to
> > create a plot. aggregation type nbr_cnt isn't compatible with the
> > statistic NBR_FBIAS". I'm guessing that METViewer would need to
use
> > the nbr_cts aggregation type, but I didn't see that option in the
> > drop-down menu. See the attached for the menu choices for
> > aggregation types.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Tuesday, September 15, 2020 9:11 AM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > Question about
> > V2.11 METViewer (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Yes, agreed. I would guess the most likely explanation for this
> > behavior is that your threshold is either too low (meaning no
> > events) or too high (meaning events almost everywhere).
> >
> > If you've also written the CTC and/or CTS line types for these
> > thresholds to METviewer, you could plot the BASE_RATE statistic to
> > see the frequency at which that event is occurring.
> >
> > Or rather than looking in METviewer, you could step back and look
at
> > the output for a single case directly from Grid-Stat. In fact, if
> > you're running a single example, you could configure Grid-Stat to
> > write the neighborhood fractional coverage field used to compute
FSS:
> >
> >
> > nc_pairs_flag = {
> >
> > ...
> >
> > nbrhd = TRUE;
> >
> > ...
> >
> > }
> >
> >
> > Then, the NetCDF output created by Grid-Stat will include fields
> > showing the fractional coverage fields. And those may be
interesting
> > to look at using the ncview utility.
> >
> >
> > Thanks,
> > John
> >
> > On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Hi John -
> > >
> > > Thanks for taking a look at that output and providing some
insight
> > > on how to approach using METViewer. I actually generated the
> > > attached plot along the lines of you suggestion below where I
> > > picked some specific values for the threshold and the
neighborhood
> > > size and that plot. Notice that it also shows FSS = 1.0 for all
> > > leads except the first one which was 0.964. Notice also, that
the
> > > NStats values are now reduced to 99 per lead vice the 19800
values
> > > in the plot I sent yesterday. So, it's clear that picking a
> > > specific threshold and neighborhood size narrows down the number
> > > of values which were aggregated to produce the FSS values at
each
> > > lead. Just so you can understand more about this plot, I
captured
> > > the METViewer settings in the attached file with "settings" in
the
> > > filename. The settings I used for the plot I sent yesterday were
> > > identical except for specifying fixed values for threshold and
> > > neighborhood size.
> > >
> > > I find it surprising that, despite the reduction on the NStats
> > > values from
> > > 19800 to 99, the FSS values are still perfect with that one
exception.
> > >
> > > I see now that I'll need to explore more values for threshold
and
> > > neighborhood size to get a feel for where the performance drops
> > > from perfect to a lesser value to capture the dependence on
scale
> > > size and threshold value.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Tuesday, September 15, 2020 7:50 AM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > Question about
> > > V2.11 METViewer (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Hi John,
> > >
> > > This is John Halley Gotway. Based on the data you sent, I can't
> > > really tell you very much about this plot.
> > >
> > > I do see in the R-script file that you're plotting the "mean"
(or
> > > average) value for each of the lead times:
> > >
> > > strPlotStat = "mean";
> > >
> > >
> > > Another option is median, but I see that you're using the mean
instead.
> > >
> > >
> > > However, one detail about METviewer that can be frustrating at
> > > times is that it silently groups together multiple buckets of
> > > data, unless you tell it not to. In this case, you've stated
that
> > > you're using 8 thresholds and
> > > 24 neighborhood sizes. But there's nothing the METviewer that
> > > indicates that fact.
> > >
> > >
> > > When making a new plot with METviewer, I always recommend that
> > > users check every single option in the "Fixed Values" section.
If
> > > multiple values are listed for one, pick one of them. Of course,
> > > you won't select a single value for whatever you plan to put on
the
> > > X-axis (e.g.
> > > lead time). And you won't select ones that have been used to
> > > define the series (e.g. often that's different model names).
> > >
> > >
> > > So I'd recommend going back and picking a single threshold and
> > > neighborhood size. And once you understand how METviewer is
> > > plotting the data, you can add in more grouping and aggregation.
> > >
> > >
> > > Hope that helps.
> > >
> > >
> > > John
> > >
> > > On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: FW: Question about V2.11 METViewer
(UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > 714 >
> > > >
> > > >
> > > >
> > > > Julie -
> > > >
> > > > Thanks for letting me know about the misg text. This is a
> > > > forwarded copy below.
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > Sent: Monday, September 14, 2020 4:10 PM
> > > > To: 'met_help at ucar.edu'
> > > > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > I have a database populated with 2376 Grid-Stat STAT files and
I
> > > > generated a plot of FSS vs lead time shown the attached PNG
> > > > file. The settings used for this plot as as follows:
> > > >
> > > > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind
vrbl:
> > > FCST_LEAD.
> > > > The aggregated statistics plotted were "Nbrhood method count"
> > (NBR_CNT).
> > > > The R-data (attached) shows that the Nstat value for each lead
> > > > time
> is
> > > > 19800. See the line for the first lead time below.
> > > > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1
NA
> > > NA
> > > > 19800
> > > >
> > > > From
> > > > <Caution-Caution-
> > Caution-Caution-
http://carson2.arl.army.mil:8080/metviewer/servlet?j
> > sp=new>
> > > >
> > > > The plot shows perfect FSS values (1.0) for all leads. I have
> > > > also attached the Rscript, xml, and log files.
> > > >
> > > > Are the plotted aggregated FSS values at each lead averaged
over
> > > > all
> 8
> > > > thresholds and 24 neighborhood sizes (interp_pnts)? Is the
NStat
> value
> > > > (19800) computed from the total number of FSS values available
> > > > in the
> > > > 2376 STAT files? (19800/2376=8.33). This seems to indicate
that
> > > > it
> was
> > > > using the values of FSS for each of the 8 thresholds, but why
> > > > doesn't the math work out to <8 since sometimes the value of
FSS
> > > > was "NA" at
> > > certain
> > > > thresholds.
> > > > I
> > > > guess I need to understand more about the computation of the
FSS
> using
> > > > thes above settings.
> > > >
> > > > I suppose that to break out the FSS for a specified
combination
> > > > of threshold value and neighborhood size, I would add "fixed
value"
> > > > settings for each of those plots to accomplish this, correct?
> > > >
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > Office:(575) 678-2004 DSN 258-2004
> > > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
> > > > (575)
> > > > 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Sep 22 08:15:58 2020
John,
Yes, that is true.
For each combination of CAT_THRESH and neighborhood size, a fractional
coverage field is computed.
The fcst and obs fractional coverage values are compared directly to
compute the NBRCNT line.
With each COV_THRESH threshold, those fractional coverage fields are
thresholded and NBRCTC/NBRCTS line types are computed.
The shortcoming right now is that METviewer is not displaying
COV_THRESH in
the list of options. The setting in the default config file is
"cov_thresh
= >0.5;"
In a different telecon yesterday, we discussed how applying
"cov_thresh =
>0;" can be interpreted as meaning that the "event" (as defined by
cat_thresh) occurred at least once within the neighborhood.
Thanks,
John
On Tue, Sep 22, 2020 at 7:48 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Hi John -
>
> Thanks for following up in this. So, if you want to plot statistics
by
> aggregating the NBRCTC line type, the effective threshold value is
set by
> COV_THRESH rather than using the CAT_THRESH. Is this true? But in
the case
> of
> FSS, the thresholds are set by CAT_THRESH, because the line type is
now
> NBRCNT, correct? Can you specify an array of COV_THRESH values? It
looks
> like
> you can. I'm thinking of how I would run Grid-Stat so I would be
able to
> output both FSS and NBR_FBIAS for multiple thresholds and
neighborhood
> sizes.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Monday, September 21, 2020 5:02 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question
> about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Sorry, I lost track of this email from last week. We were talking
about
> METviewer and in an earlier email I described plotting NBR_FBIAS but
did
> not
> send an example. When you tried to make the plot I described you
didn't
> see an
> option to aggregate the NBRCTC line type and the question is whether
or
> not
> METviewer supports that.
>
> I see from your email that you're running METviewer 2.11, and
version 3.1
> is
> available here:
> Caution-http://dtcenter.ucar.edu/met/metviewer/metviewer1.jsp
>
> I worked up the attached example using that DTC instance of
METviewer.
> This
> plot compares 6-hour precip from two different data sources. It has
4
> lines on
> it: 2 categorical thresholds (>1/4" green and >1/2" blue) * 2
neighborhood
> sizes (3x3 - dashed and 5x5 - solid).
> The statistics plotted is "NBR_FBIAS"... but that's defined by
applying
> the
> coverage threshold (cov_thresh) to define events. I'm guessing that
the
> default value of "cov_thresh = >0.5;" was used, but I'm really not
> positive.
>
> But we noticed in a telecon this morning that "COV_THRESH" DOES NOT
exist
> as
> an option in METviewer. But Tatiana (cc'ed here) will write up a
GitHub
> issue
> to add it.
>
> Make sense?
>
> Thanks,
> John
>
>
> On Tue, Sep 15, 2020 at 4:21 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for sharing that info about the community use of FSS vs
FBIAS.
> > I wasn't aware of that. I tried setting up the plot you describe
> > below, but I find that the statistics menu doesn't have
"Neighborhood
> > contingency table (NBRCTC)".
> > It only has something called "Nbrhood method count (NBR_CNT)"
which,
> > if selected fails to produce the plot due to the incompatibility
> > between the NBRCNT and the statistic NBR_FBIAS because the FBIAS
is
> > only output in the NBRCTS line type and not in the NBRCNT line
type.
> > Is my METViewer missing something or is a because the version is
too old?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Tuesday, September 15, 2020 4:02 PM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > Question about
> > V2.11 METViewer (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I'd caution you against using the NBRCTC counts and the NBRCTS
> statistics.
> > The Fractions Skill Score (NBR_FSS) is far and away the most
commonly
> > used neighborhood-based statistic. The NBRCTC and NBRCTS lines are
the
> > counts and statistics you'd get after applying a threshold
> > (cov_thresh) to the fractional coverage fields used to compute
FSS.
> > And I'm not aware of any published results that use that method.
So
> > that's why I'm a little wary of it.
> >
> > But to answer your question... NBR_FBIAS in METviewer is just the
> > FBIAS value from the NBRCTS line type. The contents of that line
type
> > are described in this table:
> >
> > Caution-Caution-https://dtcenter.github.io/MET/Users_Guide/grid-
stat.h
> > tml#id9
> >
> > In exactly the same way that you aggregate CTC lines in METviewer
to
> > compute CTS statistics... you do exactly the same in aggregating
> > NBRCTC lines to compute NBRCTS statistics. So in the "Aggregation
> > statistics"
> > dropdown list, you should select "Neighborhood contingency table
> (NBRCTC)".
> >
> > Now I haven't actually tested this on a real example to confirm,
but
> > that's what *should* work.
> >
> > Thanks,
> > John
> >
> > On Tue, Sep 15, 2020 at 3:08 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > > 714 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > To follow-up on trying to get NBR_FBIAS to plot, I used the
settings
> > > in the attached, but still get "no image available". The log
file
> > > states: "
> > > Failed to
> > > create a plot. aggregation type nbr_cnt isn't compatible with
the
> > > statistic NBR_FBIAS". I'm guessing that METViewer would need to
use
> > > the nbr_cts aggregation type, but I didn't see that option in
the
> > > drop-down menu. See the attached for the menu choices for
> > > aggregation types.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Tuesday, September 15, 2020 9:11 AM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > Question about
> > > V2.11 METViewer (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Yes, agreed. I would guess the most likely explanation for this
> > > behavior is that your threshold is either too low (meaning no
> > > events) or too high (meaning events almost everywhere).
> > >
> > > If you've also written the CTC and/or CTS line types for these
> > > thresholds to METviewer, you could plot the BASE_RATE statistic
to
> > > see the frequency at which that event is occurring.
> > >
> > > Or rather than looking in METviewer, you could step back and
look at
> > > the output for a single case directly from Grid-Stat. In fact,
if
> > > you're running a single example, you could configure Grid-Stat
to
> > > write the neighborhood fractional coverage field used to compute
FSS:
> > >
> > >
> > > nc_pairs_flag = {
> > >
> > > ...
> > >
> > > nbrhd = TRUE;
> > >
> > > ...
> > >
> > > }
> > >
> > >
> > > Then, the NetCDF output created by Grid-Stat will include fields
> > > showing the fractional coverage fields. And those may be
interesting
> > > to look at using the ncview utility.
> > >
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-url:
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > Hi John -
> > > >
> > > > Thanks for taking a look at that output and providing some
insight
> > > > on how to approach using METViewer. I actually generated the
> > > > attached plot along the lines of you suggestion below where I
> > > > picked some specific values for the threshold and the
neighborhood
> > > > size and that plot. Notice that it also shows FSS = 1.0 for
all
> > > > leads except the first one which was 0.964. Notice also, that
the
> > > > NStats values are now reduced to 99 per lead vice the 19800
values
> > > > in the plot I sent yesterday. So, it's clear that picking a
> > > > specific threshold and neighborhood size narrows down the
number
> > > > of values which were aggregated to produce the FSS values at
each
> > > > lead. Just so you can understand more about this plot, I
captured
> > > > the METViewer settings in the attached file with "settings" in
the
> > > > filename. The settings I used for the plot I sent yesterday
were
> > > > identical except for specifying fixed values for threshold and
> > > > neighborhood size.
> > > >
> > > > I find it surprising that, despite the reduction on the NStats
> > > > values from
> > > > 19800 to 99, the FSS values are still perfect with that one
> exception.
> > > >
> > > > I see now that I'll need to explore more values for threshold
and
> > > > neighborhood size to get a feel for where the performance
drops
> > > > from perfect to a lesser value to capture the dependence on
scale
> > > > size and threshold value.
> > > >
> > > > R/
> > > > John
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: Tuesday, September 15, 2020 7:50 AM
> > > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > > Question about
> > > > V2.11 METViewer (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity of
> > > > all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > Hi John,
> > > >
> > > > This is John Halley Gotway. Based on the data you sent, I
can't
> > > > really tell you very much about this plot.
> > > >
> > > > I do see in the R-script file that you're plotting the "mean"
(or
> > > > average) value for each of the lead times:
> > > >
> > > > strPlotStat = "mean";
> > > >
> > > >
> > > > Another option is median, but I see that you're using the mean
> instead.
> > > >
> > > >
> > > > However, one detail about METviewer that can be frustrating at
> > > > times is that it silently groups together multiple buckets of
> > > > data, unless you tell it not to. In this case, you've stated
that
> > > > you're using 8 thresholds and
> > > > 24 neighborhood sizes. But there's nothing the METviewer that
> > > > indicates that fact.
> > > >
> > > >
> > > > When making a new plot with METviewer, I always recommend that
> > > > users check every single option in the "Fixed Values" section.
If
> > > > multiple values are listed for one, pick one of them. Of
course,
> > > > you won't select a single value for whatever you plan to put
on the
> > > > X-axis (e.g.
> > > > lead time). And you won't select ones that have been used to
> > > > define the series (e.g. often that's different model names).
> > > >
> > > >
> > > > So I'd recommend going back and picking a single threshold and
> > > > neighborhood size. And once you understand how METviewer is
> > > > plotting the data, you can add in more grouping and
aggregation.
> > > >
> > > >
> > > > Hope that helps.
> > > >
> > > >
> > > > John
> > > >
> > > > On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > Queue: met_help
> > > > > Subject: FW: Question about V2.11 METViewer
(UNCLASSIFIED)
> > > > > Owner: Nobody
> > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > Status: new
> > > > > Ticket <Caution-Caution-Caution-Caution-url:
> > > > > Caution-Caution-
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > > 714 >
> > > > >
> > > > >
> > > > >
> > > > > Julie -
> > > > >
> > > > > Thanks for letting me know about the misg text. This is a
> > > > > forwarded copy below.
> > > > > R/
> > > > > John
> > > > > ________________________________________
> > > > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > > Sent: Monday, September 14, 2020 4:10 PM
> > > > > To: 'met_help at ucar.edu'
> > > > > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > I have a database populated with 2376 Grid-Stat STAT files
and I
> > > > > generated a plot of FSS vs lead time shown the attached PNG
> > > > > file. The settings used for this plot as as follows:
> > > > >
> > > > > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind
vrbl:
> > > > FCST_LEAD.
> > > > > The aggregated statistics plotted were "Nbrhood method
count"
> > > (NBR_CNT).
> > > > > The R-data (attached) shows that the Nstat value for each
lead
> > > > > time
> > is
> > > > > 19800. See the line for the first lead time below.
> > > > > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1
NA
> > > > NA
> > > > > 19800
> > > > >
> > > > > From
> > > > > <Caution-Caution-
> > > Caution-Caution-
http://carson2.arl.army.mil:8080/metviewer/servlet?j
> > > sp=new>
> > > > >
> > > > > The plot shows perfect FSS values (1.0) for all leads. I
have
> > > > > also attached the Rscript, xml, and log files.
> > > > >
> > > > > Are the plotted aggregated FSS values at each lead averaged
over
> > > > > all
> > 8
> > > > > thresholds and 24 neighborhood sizes (interp_pnts)? Is the
NStat
> > value
> > > > > (19800) computed from the total number of FSS values
available
> > > > > in the
> > > > > 2376 STAT files? (19800/2376=8.33). This seems to indicate
that
> > > > > it
> > was
> > > > > using the values of FSS for each of the 8 thresholds, but
why
> > > > > doesn't the math work out to <8 since sometimes the value of
FSS
> > > > > was "NA" at
> > > > certain
> > > > > thresholds.
> > > > > I
> > > > > guess I need to understand more about the computation of the
FSS
> > using
> > > > > thes above settings.
> > > > >
> > > > > I suppose that to break out the FSS for a specified
combination
> > > > > of threshold value and neighborhood size, I would add "fixed
value"
> > > > > settings for each of those plots to accomplish this,
correct?
> > > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > Mr. John W. Raby
> > > > > U.S. Army Research Laboratory
> > > > > White Sands Missile Range, NM 88002
> > > > > Office:(575) 678-2004 DSN 258-2004
> > > > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
> > > > > (575)
> > > > > 678-1230 DSN 258-1230
> > > > > Email: john.w.raby2.civ at mail.mil
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Sep 22 08:25:31 2020
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for summarizing how those two line types apply thresholds and
use
neighborhoods. Just to be clear, the fractional fields computed by
COV_THRESH
are applied for each combination of COV_THRESH and neighborhood size,
correct?
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, September 22, 2020 8:16 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question about
V2.11 METViewer (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Yes, that is true.
For each combination of CAT_THRESH and neighborhood size, a fractional
coverage field is computed.
The fcst and obs fractional coverage values are compared directly to
compute
the NBRCNT line.
With each COV_THRESH threshold, those fractional coverage fields are
thresholded and NBRCTC/NBRCTS line types are computed.
The shortcoming right now is that METviewer is not displaying
COV_THRESH in
the list of options. The setting in the default config file is
"cov_thresh =
>0.5;"
In a different telecon yesterday, we discussed how applying
"cov_thresh =
>0;" can be interpreted as meaning that the "event" (as defined by
cat_thresh) occurred at least once within the neighborhood.
Thanks,
John
On Tue, Sep 22, 2020 at 7:48 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Hi John -
>
> Thanks for following up in this. So, if you want to plot statistics
by
> aggregating the NBRCTC line type, the effective threshold value is
set
> by COV_THRESH rather than using the CAT_THRESH. Is this true? But in
> the case of FSS, the thresholds are set by CAT_THRESH, because the
> line type is now NBRCNT, correct? Can you specify an array of
> COV_THRESH values? It looks like you can. I'm thinking of how I
would
> run Grid-Stat so I would be able to output both FSS and NBR_FBIAS
for
> multiple thresholds and neighborhood sizes.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Monday, September 21, 2020 5:02 PM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> Question about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Sorry, I lost track of this email from last week. We were talking
> about METviewer and in an earlier email I described plotting
NBR_FBIAS
> but did not send an example. When you tried to make the plot I
> described you didn't see an option to aggregate the NBRCTC line type
> and the question is whether or not METviewer supports that.
>
> I see from your email that you're running METviewer 2.11, and
version
> 3.1 is available here:
> Caution-Caution-
http://dtcenter.ucar.edu/met/metviewer/metviewer1.jsp
>
> I worked up the attached example using that DTC instance of
METviewer.
> This
> plot compares 6-hour precip from two different data sources. It has
4
> lines on
> it: 2 categorical thresholds (>1/4" green and >1/2" blue) * 2
> neighborhood sizes (3x3 - dashed and 5x5 - solid).
> The statistics plotted is "NBR_FBIAS"... but that's defined by
> applying the coverage threshold (cov_thresh) to define events. I'm
> guessing that the default value of "cov_thresh = >0.5;" was used,
but
> I'm really not positive.
>
> But we noticed in a telecon this morning that "COV_THRESH" DOES NOT
> exist as an option in METviewer. But Tatiana (cc'ed here) will write
> up a GitHub issue to add it.
>
> Make sense?
>
> Thanks,
> John
>
>
> On Tue, Sep 15, 2020 at 4:21 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > 714 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for sharing that info about the community use of FSS vs
FBIAS.
> > I wasn't aware of that. I tried setting up the plot you describe
> > below, but I find that the statistics menu doesn't have
> > "Neighborhood contingency table (NBRCTC)".
> > It only has something called "Nbrhood method count (NBR_CNT)"
which,
> > if selected fails to produce the plot due to the incompatibility
> > between the NBRCNT and the statistic NBR_FBIAS because the FBIAS
is
> > only output in the NBRCTS line type and not in the NBRCNT line
type.
> > Is my METViewer missing something or is a because the version is
too old?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Tuesday, September 15, 2020 4:02 PM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > Question about
> > V2.11 METViewer (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I'd caution you against using the NBRCTC counts and the NBRCTS
> statistics.
> > The Fractions Skill Score (NBR_FSS) is far and away the most
> > commonly used neighborhood-based statistic. The NBRCTC and NBRCTS
> > lines are the counts and statistics you'd get after applying a
> > threshold
> > (cov_thresh) to the fractional coverage fields used to compute
FSS.
> > And I'm not aware of any published results that use that method.
So
> > that's why I'm a little wary of it.
> >
> > But to answer your question... NBR_FBIAS in METviewer is just the
> > FBIAS value from the NBRCTS line type. The contents of that line
> > type are described in this table:
> >
> > Caution-Caution-Caution-
https://dtcenter.github.io/MET/Users_Guide/g
> > rid-stat.h
> > tml#id9
> >
> > In exactly the same way that you aggregate CTC lines in METviewer
to
> > compute CTS statistics... you do exactly the same in aggregating
> > NBRCTC lines to compute NBRCTS statistics. So in the "Aggregation
> > statistics"
> > dropdown list, you should select "Neighborhood contingency table
> (NBRCTC)".
> >
> > Now I haven't actually tested this on a real example to confirm,
but
> > that's what *should* work.
> >
> > Thanks,
> > John
> >
> > On Tue, Sep 15, 2020 at 3:08 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=96
> > > 714 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > To follow-up on trying to get NBR_FBIAS to plot, I used the
> > > settings in the attached, but still get "no image available".
The
> > > log file
> > > states: "
> > > Failed to
> > > create a plot. aggregation type nbr_cnt isn't compatible with
the
> > > statistic NBR_FBIAS". I'm guessing that METViewer would need to
> > > use the nbr_cts aggregation type, but I didn't see that option
in
> > > the drop-down menu. See the attached for the menu choices for
> > > aggregation types.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Tuesday, September 15, 2020 9:11 AM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > Question about
> > > V2.11 METViewer (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Yes, agreed. I would guess the most likely explanation for this
> > > behavior is that your threshold is either too low (meaning no
> > > events) or too high (meaning events almost everywhere).
> > >
> > > If you've also written the CTC and/or CTS line types for these
> > > thresholds to METviewer, you could plot the BASE_RATE statistic
to
> > > see the frequency at which that event is occurring.
> > >
> > > Or rather than looking in METviewer, you could step back and
look
> > > at the output for a single case directly from Grid-Stat. In
fact,
> > > if you're running a single example, you could configure Grid-
Stat
> > > to write the neighborhood fractional coverage field used to
compute FSS:
> > >
> > >
> > > nc_pairs_flag = {
> > >
> > > ...
> > >
> > > nbrhd = TRUE;
> > >
> > > ...
> > >
> > > }
> > >
> > >
> > > Then, the NetCDF output created by Grid-Stat will include fields
> > > showing the fractional coverage fields. And those may be
> > > interesting to look at using the ncview utility.
> > >
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > 714 >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > Hi John -
> > > >
> > > > Thanks for taking a look at that output and providing some
> > > > insight on how to approach using METViewer. I actually
generated
> > > > the attached plot along the lines of you suggestion below
where
> > > > I picked some specific values for the threshold and the
> > > > neighborhood size and that plot. Notice that it also shows FSS
=
> > > > 1.0 for all leads except the first one which was 0.964.
Notice
> > > > also, that the NStats values are now reduced to 99 per lead
vice
> > > > the 19800 values in the plot I sent yesterday. So, it's clear
> > > > that picking a specific threshold and neighborhood size
narrows
> > > > down the number of values which were aggregated to produce the
> > > > FSS values at each lead. Just so you can understand more
about
> > > > this plot, I captured the METViewer settings in the attached
> > > > file with "settings" in the filename. The settings I used for
> > > > the plot I sent yesterday were identical except for specifying
> > > > fixed values for threshold and neighborhood size.
> > > >
> > > > I find it surprising that, despite the reduction on the NStats
> > > > values from
> > > > 19800 to 99, the FSS values are still perfect with that one
> exception.
> > > >
> > > > I see now that I'll need to explore more values for threshold
> > > > and neighborhood size to get a feel for where the performance
> > > > drops from perfect to a lesser value to capture the dependence
> > > > on scale size and threshold value.
> > > >
> > > > R/
> > > > John
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: Tuesday, September 15, 2020 7:50 AM
> > > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > > Question about
> > > > V2.11 METViewer (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity
> > > > of all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > Hi John,
> > > >
> > > > This is John Halley Gotway. Based on the data you sent, I
can't
> > > > really tell you very much about this plot.
> > > >
> > > > I do see in the R-script file that you're plotting the "mean"
> > > > (or
> > > > average) value for each of the lead times:
> > > >
> > > > strPlotStat = "mean";
> > > >
> > > >
> > > > Another option is median, but I see that you're using the mean
> instead.
> > > >
> > > >
> > > > However, one detail about METviewer that can be frustrating at
> > > > times is that it silently groups together multiple buckets of
> > > > data, unless you tell it not to. In this case, you've stated
> > > > that you're using 8 thresholds and
> > > > 24 neighborhood sizes. But there's nothing the METviewer that
> > > > indicates that fact.
> > > >
> > > >
> > > > When making a new plot with METviewer, I always recommend that
> > > > users check every single option in the "Fixed Values" section.
> > > > If multiple values are listed for one, pick one of them. Of
> > > > course, you won't select a single value for whatever you plan
to
> > > > put on the X-axis (e.g.
> > > > lead time). And you won't select ones that have been used to
> > > > define the series (e.g. often that's different model names).
> > > >
> > > >
> > > > So I'd recommend going back and picking a single threshold and
> > > > neighborhood size. And once you understand how METviewer is
> > > > plotting the data, you can add in more grouping and
aggregation.
> > > >
> > > >
> > > > Hope that helps.
> > > >
> > > >
> > > > John
> > > >
> > > > On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > Queue: met_help
> > > > > Subject: FW: Question about V2.11 METViewer
(UNCLASSIFIED)
> > > > > Owner: Nobody
> > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > Status: new
> > > > > Ticket <Caution-Caution-Caution-Caution-Caution-url:
> > > > > Caution-Caution-
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=96
> > > 714 >
> > > > >
> > > > >
> > > > >
> > > > > Julie -
> > > > >
> > > > > Thanks for letting me know about the misg text. This is a
> > > > > forwarded copy below.
> > > > > R/
> > > > > John
> > > > > ________________________________________
> > > > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > > Sent: Monday, September 14, 2020 4:10 PM
> > > > > To: 'met_help at ucar.edu'
> > > > > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > I have a database populated with 2376 Grid-Stat STAT files
and
> > > > > I generated a plot of FSS vs lead time shown the attached
PNG
> > > > > file. The settings used for this plot as as follows:
> > > > >
> > > > > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind
vrbl:
> > > > FCST_LEAD.
> > > > > The aggregated statistics plotted were "Nbrhood method
count"
> > > (NBR_CNT).
> > > > > The R-data (attached) shows that the Nstat value for each
lead
> > > > > time
> > is
> > > > > 19800. See the line for the first lead time below.
> > > > > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1
NA
> > > > NA
> > > > > 19800
> > > > >
> > > > > From
> > > > > <Caution-Caution-
> > > Caution-Caution-Caution-
http://carson2.arl.army.mil:8080/metviewer
> > > /servlet?j
> > > sp=new>
> > > > >
> > > > > The plot shows perfect FSS values (1.0) for all leads. I
have
> > > > > also attached the Rscript, xml, and log files.
> > > > >
> > > > > Are the plotted aggregated FSS values at each lead averaged
> > > > > over all
> > 8
> > > > > thresholds and 24 neighborhood sizes (interp_pnts)? Is the
> > > > > NStat
> > value
> > > > > (19800) computed from the total number of FSS values
available
> > > > > in the
> > > > > 2376 STAT files? (19800/2376=8.33). This seems to indicate
> > > > > that it
> > was
> > > > > using the values of FSS for each of the 8 thresholds, but
why
> > > > > doesn't the math work out to <8 since sometimes the value of
> > > > > FSS was "NA" at
> > > > certain
> > > > > thresholds.
> > > > > I
> > > > > guess I need to understand more about the computation of the
> > > > > FSS
> > using
> > > > > thes above settings.
> > > > >
> > > > > I suppose that to break out the FSS for a specified
> > > > > combination of threshold value and neighborhood size, I
would add
> > > > > "fixed value"
> > > > > settings for each of those plots to accomplish this,
correct?
> > > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > Mr. John W. Raby
> > > > > U.S. Army Research Laboratory
> > > > > White Sands Missile Range, NM 88002
> > > > > Office:(575) 678-2004 DSN 258-2004
> > > > > Teleworking: Contact Admin Specialist at (410) 278-6974 FAX
> > > > > (575)
> > > > > 678-1230 DSN 258-1230
> > > > > Email: john.w.raby2.civ at mail.mil
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Sep 22 08:33:20 2020
Close... I think you understand it but just mis-typed it:
"Just to be clear, the fractional fields computed by COV_THRESH
are applied for each combination of *CAT_THRESH* and neighborhood
size,
correct?"
Yes that's correct.
For a cat_thresh of length 3 and nbrhd.width of length 5, you'd get 15
NBRCNT lines.
For a cov_thresh of length 2, you'd get 30 NBRCTC and NBRCTS lines.
You can see how the amount of output grows quickly!
John
On Tue, Sep 22, 2020 at 8:26 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for summarizing how those two line types apply thresholds and
use
> neighborhoods. Just to be clear, the fractional fields computed by
> COV_THRESH
> are applied for each combination of COV_THRESH and neighborhood
size,
> correct?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, September 22, 2020 8:16 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question
> about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
> contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Yes, that is true.
>
> For each combination of CAT_THRESH and neighborhood size, a
fractional
> coverage field is computed.
> The fcst and obs fractional coverage values are compared directly to
> compute
> the NBRCNT line.
>
> With each COV_THRESH threshold, those fractional coverage fields are
> thresholded and NBRCTC/NBRCTS line types are computed.
>
> The shortcoming right now is that METviewer is not displaying
COV_THRESH
> in
> the list of options. The setting in the default config file is
"cov_thresh
> =
> >0.5;"
>
> In a different telecon yesterday, we discussed how applying
"cov_thresh =
> >0;" can be interpreted as meaning that the "event" (as defined by
> cat_thresh) occurred at least once within the neighborhood.
>
> Thanks,
> John
>
> On Tue, Sep 22, 2020 at 7:48 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Hi John -
> >
> > Thanks for following up in this. So, if you want to plot
statistics by
> > aggregating the NBRCTC line type, the effective threshold value is
set
> > by COV_THRESH rather than using the CAT_THRESH. Is this true? But
in
> > the case of FSS, the thresholds are set by CAT_THRESH, because the
> > line type is now NBRCNT, correct? Can you specify an array of
> > COV_THRESH values? It looks like you can. I'm thinking of how I
would
> > run Grid-Stat so I would be able to output both FSS and NBR_FBIAS
for
> > multiple thresholds and neighborhood sizes.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Monday, September 21, 2020 5:02 PM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > Question about
> > V2.11 METViewer (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Sorry, I lost track of this email from last week. We were talking
> > about METviewer and in an earlier email I described plotting
NBR_FBIAS
> > but did not send an example. When you tried to make the plot I
> > described you didn't see an option to aggregate the NBRCTC line
type
> > and the question is whether or not METviewer supports that.
> >
> > I see from your email that you're running METviewer 2.11, and
version
> > 3.1 is available here:
> > Caution-Caution-
http://dtcenter.ucar.edu/met/metviewer/metviewer1.jsp
> >
> > I worked up the attached example using that DTC instance of
METviewer.
> > This
> > plot compares 6-hour precip from two different data sources. It
has 4
> > lines on
> > it: 2 categorical thresholds (>1/4" green and >1/2" blue) * 2
> > neighborhood sizes (3x3 - dashed and 5x5 - solid).
> > The statistics plotted is "NBR_FBIAS"... but that's defined by
> > applying the coverage threshold (cov_thresh) to define events. I'm
> > guessing that the default value of "cov_thresh = >0.5;" was used,
but
> > I'm really not positive.
> >
> > But we noticed in a telecon this morning that "COV_THRESH" DOES
NOT
> > exist as an option in METviewer. But Tatiana (cc'ed here) will
write
> > up a GitHub issue to add it.
> >
> > Make sense?
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Sep 15, 2020 at 4:21 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > > 714 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Thanks for sharing that info about the community use of FSS vs
FBIAS.
> > > I wasn't aware of that. I tried setting up the plot you describe
> > > below, but I find that the statistics menu doesn't have
> > > "Neighborhood contingency table (NBRCTC)".
> > > It only has something called "Nbrhood method count (NBR_CNT)"
which,
> > > if selected fails to produce the plot due to the incompatibility
> > > between the NBRCNT and the statistic NBR_FBIAS because the FBIAS
is
> > > only output in the NBRCTS line type and not in the NBRCNT line
type.
> > > Is my METViewer missing something or is a because the version is
too
> old?
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Tuesday, September 15, 2020 4:02 PM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > Question about
> > > V2.11 METViewer (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I'd caution you against using the NBRCTC counts and the NBRCTS
> > statistics.
> > > The Fractions Skill Score (NBR_FSS) is far and away the most
> > > commonly used neighborhood-based statistic. The NBRCTC and
NBRCTS
> > > lines are the counts and statistics you'd get after applying a
> > > threshold
> > > (cov_thresh) to the fractional coverage fields used to compute
FSS.
> > > And I'm not aware of any published results that use that method.
So
> > > that's why I'm a little wary of it.
> > >
> > > But to answer your question... NBR_FBIAS in METviewer is just
the
> > > FBIAS value from the NBRCTS line type. The contents of that line
> > > type are described in this table:
> > >
> > > Caution-Caution-Caution-
https://dtcenter.github.io/MET/Users_Guide/g
> > > rid-stat.h
> > > tml#id9
> > >
> > > In exactly the same way that you aggregate CTC lines in
METviewer to
> > > compute CTS statistics... you do exactly the same in aggregating
> > > NBRCTC lines to compute NBRCTS statistics. So in the
"Aggregation
> > > statistics"
> > > dropdown list, you should select "Neighborhood contingency table
> > (NBRCTC)".
> > >
> > > Now I haven't actually tested this on a real example to confirm,
but
> > > that's what *should* work.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Sep 15, 2020 at 3:08 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=96
> > > > 714 >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > To follow-up on trying to get NBR_FBIAS to plot, I used the
> > > > settings in the attached, but still get "no image available".
The
> > > > log file
> > > > states: "
> > > > Failed to
> > > > create a plot. aggregation type nbr_cnt isn't compatible with
the
> > > > statistic NBR_FBIAS". I'm guessing that METViewer would need
to
> > > > use the nbr_cts aggregation type, but I didn't see that option
in
> > > > the drop-down menu. See the attached for the menu choices for
> > > > aggregation types.
> > > >
> > > > R/
> > > > John
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: Tuesday, September 15, 2020 9:11 AM
> > > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > > Question about
> > > > V2.11 METViewer (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity of
> > > > all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > Yes, agreed. I would guess the most likely explanation for
this
> > > > behavior is that your threshold is either too low (meaning no
> > > > events) or too high (meaning events almost everywhere).
> > > >
> > > > If you've also written the CTC and/or CTS line types for these
> > > > thresholds to METviewer, you could plot the BASE_RATE
statistic to
> > > > see the frequency at which that event is occurring.
> > > >
> > > > Or rather than looking in METviewer, you could step back and
look
> > > > at the output for a single case directly from Grid-Stat. In
fact,
> > > > if you're running a single example, you could configure Grid-
Stat
> > > > to write the neighborhood fractional coverage field used to
compute
> FSS:
> > > >
> > > >
> > > > nc_pairs_flag = {
> > > >
> > > > ...
> > > >
> > > > nbrhd = TRUE;
> > > >
> > > > ...
> > > >
> > > > }
> > > >
> > > >
> > > > Then, the NetCDF output created by Grid-Stat will include
fields
> > > > showing the fractional coverage fields. And those may be
> > > > interesting to look at using the ncview utility.
> > > >
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <Caution-Caution-Caution-Caution-url:
> > > > > Caution-Caution-
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > > 714 >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > Hi John -
> > > > >
> > > > > Thanks for taking a look at that output and providing some
> > > > > insight on how to approach using METViewer. I actually
generated
> > > > > the attached plot along the lines of you suggestion below
where
> > > > > I picked some specific values for the threshold and the
> > > > > neighborhood size and that plot. Notice that it also shows
FSS =
> > > > > 1.0 for all leads except the first one which was 0.964.
Notice
> > > > > also, that the NStats values are now reduced to 99 per lead
vice
> > > > > the 19800 values in the plot I sent yesterday. So, it's
clear
> > > > > that picking a specific threshold and neighborhood size
narrows
> > > > > down the number of values which were aggregated to produce
the
> > > > > FSS values at each lead. Just so you can understand more
about
> > > > > this plot, I captured the METViewer settings in the attached
> > > > > file with "settings" in the filename. The settings I used
for
> > > > > the plot I sent yesterday were identical except for
specifying
> > > > > fixed values for threshold and neighborhood size.
> > > > >
> > > > > I find it surprising that, despite the reduction on the
NStats
> > > > > values from
> > > > > 19800 to 99, the FSS values are still perfect with that one
> > exception.
> > > > >
> > > > > I see now that I'll need to explore more values for
threshold
> > > > > and neighborhood size to get a feel for where the
performance
> > > > > drops from perfect to a lesser value to capture the
dependence
> > > > > on scale size and threshold value.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: Tuesday, September 15, 2020 7:50 AM
> > > > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > > <john.w.raby2.civ at mail.mil>
> > > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > > > Question about
> > > > > V2.11 METViewer (UNCLASSIFIED)
> > > > >
> > > > > All active links contained in this email were disabled.
Please
> > > > > verify the identity of the sender, and confirm the
authenticity
> > > > > of all links contained within the message prior to copying
and
> > > > > pasting the address to a Web browser.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----
> > > > >
> > > > > Hi John,
> > > > >
> > > > > This is John Halley Gotway. Based on the data you sent, I
can't
> > > > > really tell you very much about this plot.
> > > > >
> > > > > I do see in the R-script file that you're plotting the
"mean"
> > > > > (or
> > > > > average) value for each of the lead times:
> > > > >
> > > > > strPlotStat = "mean";
> > > > >
> > > > >
> > > > > Another option is median, but I see that you're using the
mean
> > instead.
> > > > >
> > > > >
> > > > > However, one detail about METviewer that can be frustrating
at
> > > > > times is that it silently groups together multiple buckets
of
> > > > > data, unless you tell it not to. In this case, you've stated
> > > > > that you're using 8 thresholds and
> > > > > 24 neighborhood sizes. But there's nothing the METviewer
that
> > > > > indicates that fact.
> > > > >
> > > > >
> > > > > When making a new plot with METviewer, I always recommend
that
> > > > > users check every single option in the "Fixed Values"
section.
> > > > > If multiple values are listed for one, pick one of them. Of
> > > > > course, you won't select a single value for whatever you
plan to
> > > > > put on the X-axis (e.g.
> > > > > lead time). And you won't select ones that have been used to
> > > > > define the series (e.g. often that's different model names).
> > > > >
> > > > >
> > > > > So I'd recommend going back and picking a single threshold
and
> > > > > neighborhood size. And once you understand how METviewer is
> > > > > plotting the data, you can add in more grouping and
aggregation.
> > > > >
> > > > >
> > > > > Hope that helps.
> > > > >
> > > > >
> > > > > John
> > > > >
> > > > > On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > > Queue: met_help
> > > > > > Subject: FW: Question about V2.11 METViewer
(UNCLASSIFIED)
> > > > > > Owner: Nobody
> > > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > > Status: new
> > > > > > Ticket <Caution-Caution-Caution-Caution-Caution-url:
> > > > > > Caution-Caution-
> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=96
> > > > 714 >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Julie -
> > > > > >
> > > > > > Thanks for letting me know about the misg text. This is a
> > > > > > forwarded copy below.
> > > > > > R/
> > > > > > John
> > > > > > ________________________________________
> > > > > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > > > Sent: Monday, September 14, 2020 4:10 PM
> > > > > > To: 'met_help at ucar.edu'
> > > > > > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> > > > > >
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > >
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > >
> > > > > > I have a database populated with 2376 Grid-Stat STAT files
and
> > > > > > I generated a plot of FSS vs lead time shown the attached
PNG
> > > > > > file. The settings used for this plot as as follows:
> > > > > >
> > > > > > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind
vrbl:
> > > > > FCST_LEAD.
> > > > > > The aggregated statistics plotted were "Nbrhood method
count"
> > > > (NBR_CNT).
> > > > > > The R-data (attached) shows that the Nstat value for each
lead
> > > > > > time
> > > is
> > > > > > 19800. See the line for the first lead time below.
> > > > > > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1
> NA
> > > > > NA
> > > > > > 19800
> > > > > >
> > > > > > From
> > > > > > <Caution-Caution-
> > > > Caution-Caution-Caution-
http://carson2.arl.army.mil:8080/metviewer
> > > > /servlet?j
> > > > sp=new>
> > > > > >
> > > > > > The plot shows perfect FSS values (1.0) for all leads. I
have
> > > > > > also attached the Rscript, xml, and log files.
> > > > > >
> > > > > > Are the plotted aggregated FSS values at each lead
averaged
> > > > > > over all
> > > 8
> > > > > > thresholds and 24 neighborhood sizes (interp_pnts)? Is the
> > > > > > NStat
> > > value
> > > > > > (19800) computed from the total number of FSS values
available
> > > > > > in the
> > > > > > 2376 STAT files? (19800/2376=8.33). This seems to indicate
> > > > > > that it
> > > was
> > > > > > using the values of FSS for each of the 8 thresholds, but
why
> > > > > > doesn't the math work out to <8 since sometimes the value
of
> > > > > > FSS was "NA" at
> > > > > certain
> > > > > > thresholds.
> > > > > > I
> > > > > > guess I need to understand more about the computation of
the
> > > > > > FSS
> > > using
> > > > > > thes above settings.
> > > > > >
> > > > > > I suppose that to break out the FSS for a specified
> > > > > > combination of threshold value and neighborhood size, I
would
> add
> > > > > > "fixed value"
> > > > > > settings for each of those plots to accomplish this,
correct?
> > > > > >
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > R/
> > > > > > John
> > > > > >
> > > > > > Mr. John W. Raby
> > > > > > U.S. Army Research Laboratory
> > > > > > White Sands Missile Range, NM 88002
> > > > > > Office:(575) 678-2004 DSN 258-2004
> > > > > > Teleworking: Contact Admin Specialist at (410) 278-6974
FAX
> > > > > > (575)
> > > > > > 678-1230 DSN 258-1230
> > > > > > Email: john.w.raby2.civ at mail.mil
> > > > > >
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > >
> > > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: FW: Question about V2.11 METViewer (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Sep 22 08:46:05 2020
CLASSIFICATION: UNCLASSIFIED
Thanks for correcting that typo. Appreciate all the info on what goes
into all
the neighborhood line types.
R/
John
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, September 22, 2020 8:33 AM
To: Raby, John W CIV USARMY CCDC ARL (USA) <john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
Question about
V2.11 METViewer (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
Close... I think you understand it but just mis-typed it:
"Just to be clear, the fractional fields computed by COV_THRESH are
applied
for each combination of *CAT_THRESH* and neighborhood size, correct?"
Yes that's correct.
For a cat_thresh of length 3 and nbrhd.width of length 5, you'd get 15
NBRCNT
lines.
For a cov_thresh of length 2, you'd get 30 NBRCTC and NBRCTS lines.
You can see how the amount of output grows quickly!
John
On Tue, Sep 22, 2020 at 8:26 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96714 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for summarizing how those two line types apply thresholds and
> use neighborhoods. Just to be clear, the fractional fields computed
by
> COV_THRESH are applied for each combination of COV_THRESH and
> neighborhood size, correct?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, September 22, 2020 8:16 AM
> To: Raby, John W CIV USARMY CCDC ARL (USA)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> Question about
> V2.11 METViewer (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Yes, that is true.
>
> For each combination of CAT_THRESH and neighborhood size, a
fractional
> coverage field is computed.
> The fcst and obs fractional coverage values are compared directly to
> compute the NBRCNT line.
>
> With each COV_THRESH threshold, those fractional coverage fields are
> thresholded and NBRCTC/NBRCTS line types are computed.
>
> The shortcoming right now is that METviewer is not displaying
> COV_THRESH in the list of options. The setting in the default config
> file is "cov_thresh = >0.5;"
>
> In a different telecon yesterday, we discussed how applying
> "cov_thresh =
> >0;" can be interpreted as meaning that the "event" (as defined by
> cat_thresh) occurred at least once within the neighborhood.
>
> Thanks,
> John
>
> On Tue, Sep 22, 2020 at 7:48 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96
> > 714 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Hi John -
> >
> > Thanks for following up in this. So, if you want to plot
statistics
> > by aggregating the NBRCTC line type, the effective threshold value
> > is set by COV_THRESH rather than using the CAT_THRESH. Is this
true?
> > But in the case of FSS, the thresholds are set by CAT_THRESH,
> > because the line type is now NBRCNT, correct? Can you specify an
> > array of COV_THRESH values? It looks like you can. I'm thinking of
> > how I would run Grid-Stat so I would be able to output both FSS
and
> > NBR_FBIAS for multiple thresholds and neighborhood sizes.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Monday, September 21, 2020 5:02 PM
> > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > Question about
> > V2.11 METViewer (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Sorry, I lost track of this email from last week. We were talking
> > about METviewer and in an earlier email I described plotting
> > NBR_FBIAS but did not send an example. When you tried to make the
> > plot I described you didn't see an option to aggregate the NBRCTC
> > line type and the question is whether or not METviewer supports
that.
> >
> > I see from your email that you're running METviewer 2.11, and
> > version
> > 3.1 is available here:
> > Caution-Caution-Caution-
http://dtcenter.ucar.edu/met/metviewer/metvi
> > ewer1.jsp
> >
> > I worked up the attached example using that DTC instance of
METviewer.
> > This
> > plot compares 6-hour precip from two different data sources. It
has
> > 4 lines on
> > it: 2 categorical thresholds (>1/4" green and >1/2" blue) * 2
> > neighborhood sizes (3x3 - dashed and 5x5 - solid).
> > The statistics plotted is "NBR_FBIAS"... but that's defined by
> > applying the coverage threshold (cov_thresh) to define events. I'm
> > guessing that the default value of "cov_thresh = >0.5;" was used,
> > but I'm really not positive.
> >
> > But we noticed in a telecon this morning that "COV_THRESH" DOES
NOT
> > exist as an option in METviewer. But Tatiana (cc'ed here) will
write
> > up a GitHub issue to add it.
> >
> > Make sense?
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Sep 15, 2020 at 4:21 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=96
> > > 714 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Thanks for sharing that info about the community use of FSS vs
FBIAS.
> > > I wasn't aware of that. I tried setting up the plot you describe
> > > below, but I find that the statistics menu doesn't have
> > > "Neighborhood contingency table (NBRCTC)".
> > > It only has something called "Nbrhood method count (NBR_CNT)"
> > > which, if selected fails to produce the plot due to the
> > > incompatibility between the NBRCNT and the statistic NBR_FBIAS
> > > because the FBIAS is only output in the NBRCTS line type and not
in the
> > > NBRCNT line type.
> > > Is my METViewer missing something or is a because the version is
> > > too
> old?
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Tuesday, September 15, 2020 4:02 PM
> > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > Question about
> > > V2.11 METViewer (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I'd caution you against using the NBRCTC counts and the NBRCTS
> > statistics.
> > > The Fractions Skill Score (NBR_FSS) is far and away the most
> > > commonly used neighborhood-based statistic. The NBRCTC and
NBRCTS
> > > lines are the counts and statistics you'd get after applying a
> > > threshold
> > > (cov_thresh) to the fractional coverage fields used to compute
FSS.
> > > And I'm not aware of any published results that use that method.
> > > So that's why I'm a little wary of it.
> > >
> > > But to answer your question... NBR_FBIAS in METviewer is just
the
> > > FBIAS value from the NBRCTS line type. The contents of that line
> > > type are described in this table:
> > >
> > > Caution-Caution-Caution-Caution-
https://dtcenter.github.io/MET/Use
> > > rs_Guide/g
> > > rid-stat.h
> > > tml#id9
> > >
> > > In exactly the same way that you aggregate CTC lines in
METviewer
> > > to compute CTS statistics... you do exactly the same in
> > > aggregating NBRCTC lines to compute NBRCTS statistics. So in the
> > > "Aggregation statistics"
> > > dropdown list, you should select "Neighborhood contingency table
> > (NBRCTC)".
> > >
> > > Now I haven't actually tested this on a real example to confirm,
> > > but that's what *should* work.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Sep 15, 2020 at 3:08 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=96
> > > > 714 >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > To follow-up on trying to get NBR_FBIAS to plot, I used the
> > > > settings in the attached, but still get "no image available".
> > > > The log file
> > > > states: "
> > > > Failed to
> > > > create a plot. aggregation type nbr_cnt isn't compatible with
> > > > the statistic NBR_FBIAS". I'm guessing that METViewer would
need
> > > > to use the nbr_cts aggregation type, but I didn't see that
> > > > option in the drop-down menu. See the attached for the menu
> > > > choices for aggregation types.
> > > >
> > > > R/
> > > > John
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: Tuesday, September 15, 2020 9:11 AM
> > > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > > Question about
> > > > V2.11 METViewer (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity
> > > > of all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > Yes, agreed. I would guess the most likely explanation for
this
> > > > behavior is that your threshold is either too low (meaning no
> > > > events) or too high (meaning events almost everywhere).
> > > >
> > > > If you've also written the CTC and/or CTS line types for these
> > > > thresholds to METviewer, you could plot the BASE_RATE
statistic
> > > > to see the frequency at which that event is occurring.
> > > >
> > > > Or rather than looking in METviewer, you could step back and
> > > > look at the output for a single case directly from Grid-Stat.
In
> > > > fact, if you're running a single example, you could configure
> > > > Grid-Stat to write the neighborhood fractional coverage field
> > > > used to compute
> FSS:
> > > >
> > > >
> > > > nc_pairs_flag = {
> > > >
> > > > ...
> > > >
> > > > nbrhd = TRUE;
> > > >
> > > > ...
> > > >
> > > > }
> > > >
> > > >
> > > > Then, the NetCDF output created by Grid-Stat will include
fields
> > > > showing the fractional coverage fields. And those may be
> > > > interesting to look at using the ncview utility.
> > > >
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Sep 15, 2020 at 8:35 AM Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <Caution-Caution-Caution-Caution-Caution-url:
> > > > > Caution-Caution-
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=96
> > > 714 >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > Hi John -
> > > > >
> > > > > Thanks for taking a look at that output and providing some
> > > > > insight on how to approach using METViewer. I actually
> > > > > generated the attached plot along the lines of you
suggestion
> > > > > below where I picked some specific values for the threshold
> > > > > and the neighborhood size and that plot. Notice that it also
> > > > > shows FSS =
> > > > > 1.0 for all leads except the first one which was 0.964.
> > > > > Notice also, that the NStats values are now reduced to 99
per
> > > > > lead vice the 19800 values in the plot I sent yesterday. So,
> > > > > it's clear that picking a specific threshold and
neighborhood
> > > > > size narrows down the number of values which were aggregated
> > > > > to produce the FSS values at each lead. Just so you can
> > > > > understand more about this plot, I captured the METViewer
> > > > > settings in the attached file with "settings" in the
filename.
> > > > > The settings I used for the plot I sent yesterday were
> > > > > identical except for specifying fixed values for threshold
and
> > > > > neighborhood size.
> > > > >
> > > > > I find it surprising that, despite the reduction on the
NStats
> > > > > values from
> > > > > 19800 to 99, the FSS values are still perfect with that one
> > exception.
> > > > >
> > > > > I see now that I'll need to explore more values for
threshold
> > > > > and neighborhood size to get a feel for where the
performance
> > > > > drops from perfect to a lesser value to capture the
dependence
> > > > > on scale size and threshold value.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: Tuesday, September 15, 2020 7:50 AM
> > > > > To: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > > <john.w.raby2.civ at mail.mil>
> > > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #96714] FW:
> > > > > Question about
> > > > > V2.11 METViewer (UNCLASSIFIED)
> > > > >
> > > > > All active links contained in this email were disabled.
> > > > > Please verify the identity of the sender, and confirm the
> > > > > authenticity of all links contained within the message prior
> > > > > to copying and pasting the address to a Web browser.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----
> > > > >
> > > > > Hi John,
> > > > >
> > > > > This is John Halley Gotway. Based on the data you sent, I
> > > > > can't really tell you very much about this plot.
> > > > >
> > > > > I do see in the R-script file that you're plotting the
"mean"
> > > > > (or
> > > > > average) value for each of the lead times:
> > > > >
> > > > > strPlotStat = "mean";
> > > > >
> > > > >
> > > > > Another option is median, but I see that you're using the
mean
> > instead.
> > > > >
> > > > >
> > > > > However, one detail about METviewer that can be frustrating
at
> > > > > times is that it silently groups together multiple buckets
of
> > > > > data, unless you tell it not to. In this case, you've stated
> > > > > that you're using 8 thresholds and
> > > > > 24 neighborhood sizes. But there's nothing the METviewer
that
> > > > > indicates that fact.
> > > > >
> > > > >
> > > > > When making a new plot with METviewer, I always recommend
that
> > > > > users check every single option in the "Fixed Values"
section.
> > > > > If multiple values are listed for one, pick one of them. Of
> > > > > course, you won't select a single value for whatever you
plan
> > > > > to put on the X-axis (e.g.
> > > > > lead time). And you won't select ones that have been used to
> > > > > define the series (e.g. often that's different model names).
> > > > >
> > > > >
> > > > > So I'd recommend going back and picking a single threshold
and
> > > > > neighborhood size. And once you understand how METviewer is
> > > > > plotting the data, you can add in more grouping and
aggregation.
> > > > >
> > > > >
> > > > > Hope that helps.
> > > > >
> > > > >
> > > > > John
> > > > >
> > > > > On Mon, Sep 14, 2020 at 7:26 PM Raby, John W USA CIV via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Mon Sep 14 19:25:37 2020: Request 96714 was acted upon.
> > > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > > Queue: met_help
> > > > > > Subject: FW: Question about V2.11 METViewer
(UNCLASSIFIED)
> > > > > > Owner: Nobody
> > > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > > Status: new
> > > > > > Ticket <Caution-Caution-Caution-Caution-Caution-Caution-
url:
> > > > > > Caution-Caution-
> > > > Caution-Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=96
> > > > 714 >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Julie -
> > > > > >
> > > > > > Thanks for letting me know about the misg text. This is a
> > > > > > forwarded copy below.
> > > > > > R/
> > > > > > John
> > > > > > ________________________________________
> > > > > > From: Raby, John W CIV USARMY CCDC ARL (USA)
> > > > > > Sent: Monday, September 14, 2020 4:10 PM
> > > > > > To: 'met_help at ucar.edu'
> > > > > > Subject: Question about V2.11 METViewer (UNCLASSIFIED)
> > > > > >
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > >
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > >
> > > > > > I have a database populated with 2376 Grid-Stat STAT files
> > > > > > and I generated a plot of FSS vs lead time shown the
> > > > > > attached PNG file. The settings used for this plot as as
follows:
> > > > > >
> > > > > > Y1 series vrbl: TMP --> NBR_FSS for WRF with ind
vrbl:
> > > > > FCST_LEAD.
> > > > > > The aggregated statistics plotted were "Nbrhood method
count"
> > > > (NBR_CNT).
> > > > > > The R-data (attached) shows that the Nstat value for each
> > > > > > lead time
> > > is
> > > > > > 19800. See the line for the first lead time below.
> > > > > > TMP WRE-N_d03_1km fcst_lead=3 NBR_FSS 1
> NA
> > > > > NA
> > > > > > 19800
> > > > > >
> > > > > > From
> > > > > > <Caution-Caution-
> > > > Caution-Caution-Caution-Caution-
http://carson2.arl.army.mil:8080
> > > > /metviewer
> > > > /servlet?j
> > > > sp=new>
> > > > > >
> > > > > > The plot shows perfect FSS values (1.0) for all leads. I
> > > > > > have also attached the Rscript, xml, and log files.
> > > > > >
> > > > > > Are the plotted aggregated FSS values at each lead
averaged
> > > > > > over all
> > > 8
> > > > > > thresholds and 24 neighborhood sizes (interp_pnts)? Is the
> > > > > > NStat
> > > value
> > > > > > (19800) computed from the total number of FSS values
> > > > > > available in the
> > > > > > 2376 STAT files? (19800/2376=8.33). This seems to indicate
> > > > > > that it
> > > was
> > > > > > using the values of FSS for each of the 8 thresholds, but
> > > > > > why doesn't the math work out to <8 since sometimes the
> > > > > > value of FSS was "NA" at
> > > > > certain
> > > > > > thresholds.
> > > > > > I
> > > > > > guess I need to understand more about the computation of
the
> > > > > > FSS
> > > using
> > > > > > thes above settings.
> > > > > >
> > > > > > I suppose that to break out the FSS for a specified
> > > > > > combination of threshold value and neighborhood size, I
> > > > > > would
> add
> > > > > > "fixed value"
> > > > > > settings for each of those plots to accomplish this,
correct?
> > > > > >
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > R/
> > > > > > John
> > > > > >
> > > > > > Mr. John W. Raby
> > > > > > U.S. Army Research Laboratory White Sands Missile Range,
NM
> > > > > > 88002
> > > > > > Office:(575) 678-2004 DSN 258-2004
> > > > > > Teleworking: Contact Admin Specialist at (410) 278-6974
FAX
> > > > > > (575)
> > > > > > 678-1230 DSN 258-1230
> > > > > > Email: john.w.raby2.civ at mail.mil
> > > > > >
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > >
> > > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
More information about the Met_help
mailing list