[Met_help] [rt.rap.ucar.edu #87349] History for question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Tue Oct 23 11:43:09 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
In the attached image from the postscript output file, is the blue-red color scale on the right border of the plot showing the values of En2 relative difference? Does the same apply to the plots for all scales?
Thanks
R/
John
Mr. John W. Raby
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Oct 11 09:58:15 2018
Hi John,
I see that you have a question about the PostScript output of the
Wavelet-Stat tool in MET. Wavelet-Stat implements the following
logic:
(1) Define a forecast and observation field to be compared.
(2) Apply a threshold to both to convert them to binary fields of 0's
and
1's.
(3) Compute the fcst - obs binary difference... and that's the plot
you
sent to me. If fcst and obs are both in range [0, 1], then their
difference has range [-1, 1]. And the default color table plots
negative
values in blue and positive values in red.
(4) Apply the haar wavelet (configurable) to decompose that difference
field by scale. And plot the difference field for each scale. The
decomposed differences also have range [-1, 1] and are plotted with
the
same color scale.
If you'd prefer a different color table you can easily change in the
Wavelet-Stat config file using the "color_table" entries.
Hope this helps clarify.
Thanks,
John Halley Gotway
On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> In the attached image from the postscript output file, is the blue-
red
> color scale on the right border of the plot showing the values of
En2
> relative difference? Does the same apply to the plots for all
scales?
>
> Thanks
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Oct 11 10:23:46 2018
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for clarifying how wavelet-stat works. I read the User's Guide
(p 48) and see the following which is why I thought that the scale
referred to En2:
"The En2 relative difference ranges between -1 and 1, positive values
indicate over-forecast and negative values indicate under-forecast."
Also see the attached images for the last two scales which wavelet-
stat produced. Why is there no color in the plot?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, October 11, 2018 9:58 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question about
MET V5.2 Wavelet-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Hi John,
I see that you have a question about the PostScript output of the
Wavelet-Stat tool in MET. Wavelet-Stat implements the following
logic:
(1) Define a forecast and observation field to be compared.
(2) Apply a threshold to both to convert them to binary fields of 0's
and 1's.
(3) Compute the fcst - obs binary difference... and that's the plot
you sent to me. If fcst and obs are both in range [0, 1], then their
difference has range [-1, 1]. And the default color table plots
negative values in blue and positive values in red.
(4) Apply the haar wavelet (configurable) to decompose that difference
field by scale. And plot the difference field for each scale. The
decomposed differences also have range [-1, 1] and are plotted with
the same color scale.
If you'd prefer a different color table you can easily change in the
Wavelet-Stat config file using the "color_table" entries.
Hope this helps clarify.
Thanks,
John Halley Gotway
On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> In the attached image from the postscript output file, is the blue-
red
> color scale on the right border of the plot showing the values of
En2
> relative difference? Does the same apply to the plots for all
scales?
>
> Thanks
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Oct 11 11:13:49 2018
John,
Looking at the color bar, I see that any differences between -0.1 and
+0.1
are plotted as white. The very small differences as the scale are
falling
into that category and being plotted as white.
Looking in the Wavelet-Stat config file, I see the following:
wvlt_plot = {
color_table = "MET_BASE/colortables/NCL_colortables/BlWhRe.ctable";
plot_min = -1.0;
plot_max = 1.0;
}
This defines the colortable used for these different plots. Setting
plot_min and plot_max to -1 and 1 tells Wavelet-Stat to rescale this
colortable using those min/max values. Let's switch to using the
"BlueRed.ctable" without the white in the middle, like this:
color_table =
"MET_BASE/colortables/NCL_colortables/BlueRed.ctable";
When you rerun, are those very small differences now showing up?
You can either use a different color table or manually make one
yourself to
plot it exactly how you'd like.
Thanks,
John
On Thu, Oct 11, 2018 at 10:24 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for clarifying how wavelet-stat works. I read the User's
Guide (p
> 48) and see the following which is why I thought that the scale
referred to
> En2:
> "The En2 relative difference ranges between -1 and 1, positive
values
> indicate over-forecast and negative values indicate under-forecast."
>
> Also see the attached images for the last two scales which wavelet-
stat
> produced. Why is there no color in the plot?
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, October 11, 2018 9:58 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about MET
> V5.2 Wavelet-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Hi John,
>
> I see that you have a question about the PostScript output of the
> Wavelet-Stat tool in MET. Wavelet-Stat implements the following
logic:
>
> (1) Define a forecast and observation field to be compared.
> (2) Apply a threshold to both to convert them to binary fields of
0's and
> 1's.
> (3) Compute the fcst - obs binary difference... and that's the plot
you
> sent to me. If fcst and obs are both in range [0, 1], then their
> difference has range [-1, 1]. And the default color table plots
negative
> values in blue and positive values in red.
> (4) Apply the haar wavelet (configurable) to decompose that
difference
> field by scale. And plot the difference field for each scale. The
> decomposed differences also have range [-1, 1] and are plotted with
the
> same color scale.
>
> If you'd prefer a different color table you can easily change in the
> Wavelet-Stat config file using the "color_table" entries.
>
> Hope this helps clarify.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > In the attached image from the postscript output file, is the
blue-red
> > color scale on the right border of the plot showing the values of
En2
> > relative difference? Does the same apply to the plots for all
scales?
> >
> > Thanks
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Oct 11 12:35:58 2018
CLASSIFICATION: UNCLASSIFIED
John -
I reran wavelet-stat after changing the ctable. See attached PDF which
contains the output for all thresholds including >=9.0 which is the
threshold I used for the plots I sent you before.
I see now how the difference in the color asssignments can make a
difference in how the results are visualized. I almost prefer the
BlueRed ctable, but I also like the impact of seeing white as an
indicator of small differences. Having the understanding that white is
a "real" result (very small differences) instead of thinking that
maybe there must have been some kind of error which prevented the
correct plotting helps now. The BlueRed ctable reduces the potential
for misunderstanding to my way of thinking.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, October 11, 2018 11:14 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Looking at the color bar, I see that any differences between -0.1 and
+0.1 are plotted as white. The very small differences as the scale
are falling into that category and being plotted as white.
Looking in the Wavelet-Stat config file, I see the following:
wvlt_plot = {
color_table = "MET_BASE/colortables/NCL_colortables/BlWhRe.ctable";
plot_min = -1.0;
plot_max = 1.0;
}
This defines the colortable used for these different plots. Setting
plot_min and plot_max to -1 and 1 tells Wavelet-Stat to rescale this
colortable using those min/max values. Let's switch to using the
"BlueRed.ctable" without the white in the middle, like this:
color_table =
"MET_BASE/colortables/NCL_colortables/BlueRed.ctable";
When you rerun, are those very small differences now showing up?
You can either use a different color table or manually make one
yourself to plot it exactly how you'd like.
Thanks,
John
On Thu, Oct 11, 2018 at 10:24 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for clarifying how wavelet-stat works. I read the User's
Guide
> (p
> 48) and see the following which is why I thought that the scale
> referred to
> En2:
> "The En2 relative difference ranges between -1 and 1, positive
values
> indicate over-forecast and negative values indicate under-forecast."
>
> Also see the attached images for the last two scales which
> wavelet-stat produced. Why is there no color in the plot?
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Thursday, October 11, 2018 9:58 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about
> MET
> V5.2 Wavelet-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> Hi John,
>
> I see that you have a question about the PostScript output of the
> Wavelet-Stat tool in MET. Wavelet-Stat implements the following
logic:
>
> (1) Define a forecast and observation field to be compared.
> (2) Apply a threshold to both to convert them to binary fields of
0's
> and 1's.
> (3) Compute the fcst - obs binary difference... and that's the plot
> you sent to me. If fcst and obs are both in range [0, 1], then
their
> difference has range [-1, 1]. And the default color table plots
> negative values in blue and positive values in red.
> (4) Apply the haar wavelet (configurable) to decompose that
difference
> field by scale. And plot the difference field for each scale. The
> decomposed differences also have range [-1, 1] and are plotted with
> the same color scale.
>
> If you'd prefer a different color table you can easily change in the
> Wavelet-Stat config file using the "color_table" entries.
>
> Hope this helps clarify.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 349 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > In the attached image from the postscript output file, is the
> > blue-red color scale on the right border of the plot showing the
> > values of En2 relative difference? Does the same apply to the
plots for all scales?
> >
> > Thanks
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Oct 11 12:48:09 2018
CLASSIFICATION: UNCLASSIFIED
John -
Sorry for the delayed response. I just got back from an hour long mtg.
Thanks for explaining the reason for the absence of any red of blue
color for small differences. I'll try your suggestion for a different
ctable and let you know how it works.
I noticed that the IS score associated with those two plots was the
highest of the scores for that threshold, so I think this makes sense
that the differences would fall into the smallest category with the
white color assignment. What is the expected range of the IS score?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, October 11, 2018 11:14 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Looking at the color bar, I see that any differences between -0.1 and
+0.1 are plotted as white. The very small differences as the scale
are falling into that category and being plotted as white.
Looking in the Wavelet-Stat config file, I see the following:
wvlt_plot = {
color_table = "MET_BASE/colortables/NCL_colortables/BlWhRe.ctable";
plot_min = -1.0;
plot_max = 1.0;
}
This defines the colortable used for these different plots. Setting
plot_min and plot_max to -1 and 1 tells Wavelet-Stat to rescale this
colortable using those min/max values. Let's switch to using the
"BlueRed.ctable" without the white in the middle, like this:
color_table =
"MET_BASE/colortables/NCL_colortables/BlueRed.ctable";
When you rerun, are those very small differences now showing up?
You can either use a different color table or manually make one
yourself to plot it exactly how you'd like.
Thanks,
John
On Thu, Oct 11, 2018 at 10:24 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for clarifying how wavelet-stat works. I read the User's
Guide
> (p
> 48) and see the following which is why I thought that the scale
> referred to
> En2:
> "The En2 relative difference ranges between -1 and 1, positive
values
> indicate over-forecast and negative values indicate under-forecast."
>
> Also see the attached images for the last two scales which
> wavelet-stat produced. Why is there no color in the plot?
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Thursday, October 11, 2018 9:58 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about
> MET
> V5.2 Wavelet-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> Hi John,
>
> I see that you have a question about the PostScript output of the
> Wavelet-Stat tool in MET. Wavelet-Stat implements the following
logic:
>
> (1) Define a forecast and observation field to be compared.
> (2) Apply a threshold to both to convert them to binary fields of
0's
> and 1's.
> (3) Compute the fcst - obs binary difference... and that's the plot
> you sent to me. If fcst and obs are both in range [0, 1], then
their
> difference has range [-1, 1]. And the default color table plots
> negative values in blue and positive values in red.
> (4) Apply the haar wavelet (configurable) to decompose that
difference
> field by scale. And plot the difference field for each scale. The
> decomposed differences also have range [-1, 1] and are plotted with
> the same color scale.
>
> If you'd prefer a different color table you can easily change in the
> Wavelet-Stat config file using the "color_table" entries.
>
> Hope this helps clarify.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 349 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > In the attached image from the postscript output file, is the
> > blue-red color scale on the right border of the plot showing the
> > values of En2 relative difference? Does the same apply to the
plots for all scales?
> >
> > Thanks
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Oct 11 15:03:32 2018
John,
Honestly, I haven't used Wavelet-Stat very much on testing and
evaluation
projects. So I'll refer you to Tressa Fowler (cc'ed here) for
question
about interpreting the results.
As for the range of the Intensity Scale Skill Score values, it looks
to be
(-inf, 1].
One nice feature of the MSE, FE2, and OE2 quantities is that they are
additive. When you sum their values across the scales you get the
value
for the binary difference field. And that's why and how the
PostScript
plot reports the percentage of the error for each scale.
Thanks,
John
On Thu, Oct 11, 2018 at 12:48 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Sorry for the delayed response. I just got back from an hour long
mtg.
>
> Thanks for explaining the reason for the absence of any red of blue
color
> for small differences. I'll try your suggestion for a different
ctable and
> let you know how it works.
>
> I noticed that the IS score associated with those two plots was the
> highest of the scores for that threshold, so I think this makes
sense that
> the differences would fall into the smallest category with the white
color
> assignment. What is the expected range of the IS score?
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, October 11, 2018 11:14 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about
> MET V5.2 Wavelet-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Looking at the color bar, I see that any differences between -0.1
and +0.1
> are plotted as white. The very small differences as the scale are
falling
> into that category and being plotted as white.
>
> Looking in the Wavelet-Stat config file, I see the following:
> wvlt_plot = {
> color_table =
"MET_BASE/colortables/NCL_colortables/BlWhRe.ctable";
> plot_min = -1.0;
> plot_max = 1.0;
> }
>
> This defines the colortable used for these different plots. Setting
> plot_min and plot_max to -1 and 1 tells Wavelet-Stat to rescale this
> colortable using those min/max values. Let's switch to using the
> "BlueRed.ctable" without the white in the middle, like this:
> color_table =
"MET_BASE/colortables/NCL_colortables/BlueRed.ctable";
>
> When you rerun, are those very small differences now showing up?
>
> You can either use a different color table or manually make one
yourself
> to plot it exactly how you'd like.
>
> Thanks,
> John
>
>
>
>
> On Thu, Oct 11, 2018 at 10:24 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for clarifying how wavelet-stat works. I read the User's
Guide
> > (p
> > 48) and see the following which is why I thought that the scale
> > referred to
> > En2:
> > "The En2 relative difference ranges between -1 and 1, positive
values
> > indicate over-forecast and negative values indicate under-
forecast."
> >
> > Also see the attached images for the last two scales which
> > wavelet-stat produced. Why is there no color in the plot?
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, October 11, 2018 9:58 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about
> > MET
> > V5.2 Wavelet-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > Hi John,
> >
> > I see that you have a question about the PostScript output of the
> > Wavelet-Stat tool in MET. Wavelet-Stat implements the following
logic:
> >
> > (1) Define a forecast and observation field to be compared.
> > (2) Apply a threshold to both to convert them to binary fields of
0's
> > and 1's.
> > (3) Compute the fcst - obs binary difference... and that's the
plot
> > you sent to me. If fcst and obs are both in range [0, 1], then
their
> > difference has range [-1, 1]. And the default color table plots
> > negative values in blue and positive values in red.
> > (4) Apply the haar wavelet (configurable) to decompose that
difference
> > field by scale. And plot the difference field for each scale.
The
> > decomposed differences also have range [-1, 1] and are plotted
with
> > the same color scale.
> >
> > If you'd prefer a different color table you can easily change in
the
> > Wavelet-Stat config file using the "color_table" entries.
> >
> > Hope this helps clarify.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: question about MET V5.2 Wavelet-Stat
(UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 349 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > In the attached image from the postscript output file, is the
> > > blue-red color scale on the right border of the plot showing the
> > > values of En2 relative difference? Does the same apply to the
plots
> for all scales?
> > >
> > > Thanks
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > (575) 678-2004 DSN 258-2004
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Oct 11 15:59:38 2018
CLASSIFICATION: UNCLASSIFIED
John or Tressa -
I used the "TILE" method to position a single tile to be centered over
the object I'm interested in and then ran wavelet-stat. See attached
output ps file.
I'm curious about the pattern seen in the graphic produced for
threshold >=3.0 for scales 1, 2, and maybe 3. I also see this for
threshold >=9.0 for scales 1 and 2, but not as pronounced. These show
horizontal banding which sometimes traverses the entire domain from
west to east. Is this a normal pattern?
I assume that the graphics for thresholds >0.0 and >=12.0 were without
results because these thresholds either have no events (>=12.0) or the
entire area was an event (>0.0) and thus the errors are undefined.
Let me know your thoughts on this.
Thanks.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, October 11, 2018 3:04 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Honestly, I haven't used Wavelet-Stat very much on testing and
evaluation projects. So I'll refer you to Tressa Fowler (cc'ed here)
for question about interpreting the results.
As for the range of the Intensity Scale Skill Score values, it looks
to be (-inf, 1].
One nice feature of the MSE, FE2, and OE2 quantities is that they are
additive. When you sum their values across the scales you get the
value for the binary difference field. And that's why and how the
PostScript plot reports the percentage of the error for each scale.
Thanks,
John
On Thu, Oct 11, 2018 at 12:48 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Sorry for the delayed response. I just got back from an hour long
mtg.
>
> Thanks for explaining the reason for the absence of any red of blue
> color for small differences. I'll try your suggestion for a
different
> ctable and let you know how it works.
>
> I noticed that the IS score associated with those two plots was the
> highest of the scores for that threshold, so I think this makes
sense
> that the differences would fall into the smallest category with the
> white color assignment. What is the expected range of the IS score?
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Thursday, October 11, 2018 11:14 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
> about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Looking at the color bar, I see that any differences between -0.1
and
> +0.1 are plotted as white. The very small differences as the scale
> are falling into that category and being plotted as white.
>
> Looking in the Wavelet-Stat config file, I see the following:
> wvlt_plot = {
> color_table =
"MET_BASE/colortables/NCL_colortables/BlWhRe.ctable";
> plot_min = -1.0;
> plot_max = 1.0;
> }
>
> This defines the colortable used for these different plots. Setting
> plot_min and plot_max to -1 and 1 tells Wavelet-Stat to rescale this
> colortable using those min/max values. Let's switch to using the
> "BlueRed.ctable" without the white in the middle, like this:
> color_table =
> "MET_BASE/colortables/NCL_colortables/BlueRed.ctable";
>
> When you rerun, are those very small differences now showing up?
>
> You can either use a different color table or manually make one
> yourself to plot it exactly how you'd like.
>
> Thanks,
> John
>
>
>
>
> On Thu, Oct 11, 2018 at 10:24 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 349 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for clarifying how wavelet-stat works. I read the User's
> > Guide (p
> > 48) and see the following which is why I thought that the scale
> > referred to
> > En2:
> > "The En2 relative difference ranges between -1 and 1, positive
> > values indicate over-forecast and negative values indicate under-
forecast."
> >
> > Also see the attached images for the last two scales which
> > wavelet-stat produced. Why is there no color in the plot?
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, October 11, 2018 9:58 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
> > about MET
> > V5.2 Wavelet-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > Hi John,
> >
> > I see that you have a question about the PostScript output of the
> > Wavelet-Stat tool in MET. Wavelet-Stat implements the following
logic:
> >
> > (1) Define a forecast and observation field to be compared.
> > (2) Apply a threshold to both to convert them to binary fields of
> > 0's and 1's.
> > (3) Compute the fcst - obs binary difference... and that's the
plot
> > you sent to me. If fcst and obs are both in range [0, 1], then
> > their difference has range [-1, 1]. And the default color table
> > plots negative values in blue and positive values in red.
> > (4) Apply the haar wavelet (configurable) to decompose that
> > difference field by scale. And plot the difference field for each
> > scale. The decomposed differences also have range [-1, 1] and are
> > plotted with the same color scale.
> >
> > If you'd prefer a different color table you can easily change in
the
> > Wavelet-Stat config file using the "color_table" entries.
> >
> > Hope this helps clarify.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: question about MET V5.2 Wavelet-Stat
(UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=87
> > > 349 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > In the attached image from the postscript output file, is the
> > > blue-red color scale on the right border of the plot showing the
> > > values of En2 relative difference? Does the same apply to the
> > > plots
> for all scales?
> > >
> > > Thanks
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > (575) 678-2004 DSN 258-2004
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Mon Oct 22 13:46:58 2018
John,
Sorry for the delay in responding. I think I lost track of this email
in
the shuffle.
So you've run with 2 thresholds. The lower one (>0) resulted in the
event
occurring everywhere in both the forecast and observation domains.
Since
there is no different in the binary fields, there isn't anything to
decompose.
But there is a meaningful difference for the higher threshold (>=3).
And
the results you're seeing are consistent with output I've seen in the
past. I don't have a good explanation for *why* you're seeing
horizontal
banding at the smallest scales, but I have seen it before. I suspect
they're small artifacts of the wavelet decomposition process.
There is an option in the Wavelet-Stat config file to switch what
wavelet
is used for the decomposition process. However, the only published
results
of this algorithm use the default "Haar" wavelet decomposition.
Thanks,
John
Thanks,
John
On Thu, Oct 11, 2018 at 4:00 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John or Tressa -
>
> I used the "TILE" method to position a single tile to be centered
over the
> object I'm interested in and then ran wavelet-stat. See attached
output ps
> file.
>
> I'm curious about the pattern seen in the graphic produced for
threshold
> >=3.0 for scales 1, 2, and maybe 3. I also see this for threshold
>=9.0 for
> scales 1 and 2, but not as pronounced. These show horizontal banding
which
> sometimes traverses the entire domain from west to east. Is this a
normal
> pattern?
>
> I assume that the graphics for thresholds >0.0 and >=12.0 were
without
> results because these thresholds either have no events (>=12.0) or
the
> entire area was an event (>0.0) and thus the errors are undefined.
Let me
> know your thoughts on this.
>
> Thanks.
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, October 11, 2018 3:04 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about
> MET V5.2 Wavelet-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Honestly, I haven't used Wavelet-Stat very much on testing and
evaluation
> projects. So I'll refer you to Tressa Fowler (cc'ed here) for
question
> about interpreting the results.
>
> As for the range of the Intensity Scale Skill Score values, it looks
to be
> (-inf, 1].
>
> One nice feature of the MSE, FE2, and OE2 quantities is that they
are
> additive. When you sum their values across the scales you get the
value
> for the binary difference field. And that's why and how the
PostScript
> plot reports the percentage of the error for each scale.
>
> Thanks,
> John
>
> On Thu, Oct 11, 2018 at 12:48 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Sorry for the delayed response. I just got back from an hour long
mtg.
> >
> > Thanks for explaining the reason for the absence of any red of
blue
> > color for small differences. I'll try your suggestion for a
different
> > ctable and let you know how it works.
> >
> > I noticed that the IS score associated with those two plots was
the
> > highest of the scores for that threshold, so I think this makes
sense
> > that the differences would fall into the smallest category with
the
> > white color assignment. What is the expected range of the IS
score?
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, October 11, 2018 11:14 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349]
question
> > about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Looking at the color bar, I see that any differences between -0.1
and
> > +0.1 are plotted as white. The very small differences as the
scale
> > are falling into that category and being plotted as white.
> >
> > Looking in the Wavelet-Stat config file, I see the following:
> > wvlt_plot = {
> > color_table =
"MET_BASE/colortables/NCL_colortables/BlWhRe.ctable";
> > plot_min = -1.0;
> > plot_max = 1.0;
> > }
> >
> > This defines the colortable used for these different plots.
Setting
> > plot_min and plot_max to -1 and 1 tells Wavelet-Stat to rescale
this
> > colortable using those min/max values. Let's switch to using the
> > "BlueRed.ctable" without the white in the middle, like this:
> > color_table =
> > "MET_BASE/colortables/NCL_colortables/BlueRed.ctable";
> >
> > When you rerun, are those very small differences now showing up?
> >
> > You can either use a different color table or manually make one
> > yourself to plot it exactly how you'd like.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> > On Thu, Oct 11, 2018 at 10:24 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 349 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Thanks for clarifying how wavelet-stat works. I read the User's
> > > Guide (p
> > > 48) and see the following which is why I thought that the scale
> > > referred to
> > > En2:
> > > "The En2 relative difference ranges between -1 and 1, positive
> > > values indicate over-forecast and negative values indicate
> under-forecast."
> > >
> > > Also see the attached images for the last two scales which
> > > wavelet-stat produced. Why is there no color in the plot?
> > >
> > > R/
> > > John
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Thursday, October 11, 2018 9:58 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
> > > about MET
> > > V5.2 Wavelet-Stat (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Hi John,
> > >
> > > I see that you have a question about the PostScript output of
the
> > > Wavelet-Stat tool in MET. Wavelet-Stat implements the following
logic:
> > >
> > > (1) Define a forecast and observation field to be compared.
> > > (2) Apply a threshold to both to convert them to binary fields
of
> > > 0's and 1's.
> > > (3) Compute the fcst - obs binary difference... and that's the
plot
> > > you sent to me. If fcst and obs are both in range [0, 1], then
> > > their difference has range [-1, 1]. And the default color table
> > > plots negative values in blue and positive values in red.
> > > (4) Apply the haar wavelet (configurable) to decompose that
> > > difference field by scale. And plot the difference field for
each
> > > scale. The decomposed differences also have range [-1, 1] and
are
> > > plotted with the same color scale.
> > >
> > > If you'd prefer a different color table you can easily change in
the
> > > Wavelet-Stat config file using the "color_table" entries.
> > >
> > > Hope this helps clarify.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: question about MET V5.2 Wavelet-Stat
(UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 349 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > In the attached image from the postscript output file, is the
> > > > blue-red color scale on the right border of the plot showing
the
> > > > values of En2 relative difference? Does the same apply to the
> > > > plots
> > for all scales?
> > > >
> > > > Thanks
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > (575) 678-2004 DSN 258-2004
> > > > FAX (575) 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Oct 23 07:53:28 2018
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for looking at the PDF containing my wavelet-stat output. I
think you had cc'd Tressa on the previous email in order to get her
take on the banding I was seeing for 2 of the 5 thresholds I used.
Your explanation for the results for the >0 threshold make sense to
me. I think the same situation applies to the results for >=12.0 (no
events). Do you concur?
Regards the banding artifact visible for the results for >=3.0 and
>=9.0, thanks for sharing your experience noting that this is not an
unexpected pattern. I think that I'll stick with the Haar wavelet,
based on the guidance from the journal articles which I've been
reading authored by Casati et al. which favor it when working with
precip verification.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, October 22, 2018 1:47 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Sorry for the delay in responding. I think I lost track of this email
in the shuffle.
So you've run with 2 thresholds. The lower one (>0) resulted in the
event occurring everywhere in both the forecast and observation
domains. Since there is no different in the binary fields, there
isn't anything to decompose.
But there is a meaningful difference for the higher threshold (>=3).
And the results you're seeing are consistent with output I've seen in
the past. I don't have a good explanation for *why* you're seeing
horizontal banding at the smallest scales, but I have seen it before.
I suspect they're small artifacts of the wavelet decomposition
process.
There is an option in the Wavelet-Stat config file to switch what
wavelet is used for the decomposition process. However, the only
published results of this algorithm use the default "Haar" wavelet
decomposition.
Thanks,
John
Thanks,
John
On Thu, Oct 11, 2018 at 4:00 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John or Tressa -
>
> I used the "TILE" method to position a single tile to be centered
over
> the object I'm interested in and then ran wavelet-stat. See attached
> output ps file.
>
> I'm curious about the pattern seen in the graphic produced for
> threshold
> >=3.0 for scales 1, 2, and maybe 3. I also see this for threshold
> >>=9.0 for
> scales 1 and 2, but not as pronounced. These show horizontal banding
> which sometimes traverses the entire domain from west to east. Is
this
> a normal pattern?
>
> I assume that the graphics for thresholds >0.0 and >=12.0 were
without
> results because these thresholds either have no events (>=12.0) or
the
> entire area was an event (>0.0) and thus the errors are undefined.
> Let me know your thoughts on this.
>
> Thanks.
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Thursday, October 11, 2018 3:04 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
> about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Honestly, I haven't used Wavelet-Stat very much on testing and
> evaluation projects. So I'll refer you to Tressa Fowler (cc'ed
here)
> for question about interpreting the results.
>
> As for the range of the Intensity Scale Skill Score values, it looks
> to be (-inf, 1].
>
> One nice feature of the MSE, FE2, and OE2 quantities is that they
are
> additive. When you sum their values across the scales you get the
> value for the binary difference field. And that's why and how the
> PostScript plot reports the percentage of the error for each scale.
>
> Thanks,
> John
>
> On Thu, Oct 11, 2018 at 12:48 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 349 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Sorry for the delayed response. I just got back from an hour long
mtg.
> >
> > Thanks for explaining the reason for the absence of any red of
blue
> > color for small differences. I'll try your suggestion for a
> > different ctable and let you know how it works.
> >
> > I noticed that the IS score associated with those two plots was
the
> > highest of the scores for that threshold, so I think this makes
> > sense that the differences would fall into the smallest category
> > with the white color assignment. What is the expected range of the
IS score?
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, October 11, 2018 11:14 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349]
question
> > about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Looking at the color bar, I see that any differences between -0.1
> > and
> > +0.1 are plotted as white. The very small differences as the
scale
> > are falling into that category and being plotted as white.
> >
> > Looking in the Wavelet-Stat config file, I see the following:
> > wvlt_plot = {
> > color_table =
"MET_BASE/colortables/NCL_colortables/BlWhRe.ctable";
> > plot_min = -1.0;
> > plot_max = 1.0;
> > }
> >
> > This defines the colortable used for these different plots.
Setting
> > plot_min and plot_max to -1 and 1 tells Wavelet-Stat to rescale
this
> > colortable using those min/max values. Let's switch to using the
> > "BlueRed.ctable" without the white in the middle, like this:
> > color_table =
> > "MET_BASE/colortables/NCL_colortables/BlueRed.ctable";
> >
> > When you rerun, are those very small differences now showing up?
> >
> > You can either use a different color table or manually make one
> > yourself to plot it exactly how you'd like.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> > On Thu, Oct 11, 2018 at 10:24 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=87
> > > 349 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Thanks for clarifying how wavelet-stat works. I read the User's
> > > Guide (p
> > > 48) and see the following which is why I thought that the scale
> > > referred to
> > > En2:
> > > "The En2 relative difference ranges between -1 and 1, positive
> > > values indicate over-forecast and negative values indicate
> under-forecast."
> > >
> > > Also see the attached images for the last two scales which
> > > wavelet-stat produced. Why is there no color in the plot?
> > >
> > > R/
> > > John
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Thursday, October 11, 2018 9:58 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
> > > about MET
> > > V5.2 Wavelet-Stat (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Hi John,
> > >
> > > I see that you have a question about the PostScript output of
the
> > > Wavelet-Stat tool in MET. Wavelet-Stat implements the following
logic:
> > >
> > > (1) Define a forecast and observation field to be compared.
> > > (2) Apply a threshold to both to convert them to binary fields
of
> > > 0's and 1's.
> > > (3) Compute the fcst - obs binary difference... and that's the
> > > plot you sent to me. If fcst and obs are both in range [0, 1],
> > > then their difference has range [-1, 1]. And the default color
> > > table plots negative values in blue and positive values in red.
> > > (4) Apply the haar wavelet (configurable) to decompose that
> > > difference field by scale. And plot the difference field for
each
> > > scale. The decomposed differences also have range [-1, 1] and
are
> > > plotted with the same color scale.
> > >
> > > If you'd prefer a different color table you can easily change in
> > > the Wavelet-Stat config file using the "color_table" entries.
> > >
> > > Hope this helps clarify.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: question about MET V5.2 Wavelet-Stat
(UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 349 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > In the attached image from the postscript output file, is the
> > > > blue-red color scale on the right border of the plot showing
the
> > > > values of En2 relative difference? Does the same apply to the
> > > > plots
> > for all scales?
> > > >
> > > > Thanks
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > (575) 678-2004 DSN 258-2004
> > > > FAX (575) 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Oct 23 09:47:07 2018
John,
Yes, I definitely agree with your conclusion about the lowest and
highest
thresholds. Wavelet-Stat decomposes the binary difference field after
the
threshold has been applied. If that difference field is constant
(either
all -1, all 0, or all 1), then it won't result in a very interesting
decomposition.
FYI, Eric Gilleland is a scientist at NCAR who's been applying some
interesting image analysis techniques to forecast verification. I've
worked with Eric for many years, but we finally have some time/funding
set
aside this year to put some of his image analysis work into MET. If
you're
interested in wavelet decompositions, you may find his work
interesting as
well. This is an outgrowth of the forecast inter-comparison project,
and
more info about it can be found here:
http://www.ral.ucar.edu/projects/icp/
Thanks,
John
On Tue, Oct 23, 2018 at 7:54 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for looking at the PDF containing my wavelet-stat output. I
think
> you had cc'd Tressa on the previous email in order to get her take
on the
> banding I was seeing for 2 of the 5 thresholds I used. Your
explanation for
> the results for the >0 threshold make sense to me. I think the same
> situation applies to the results for >=12.0 (no events). Do you
concur?
>
> Regards the banding artifact visible for the results for >=3.0 and
>=9.0,
> thanks for sharing your experience noting that this is not an
unexpected
> pattern. I think that I'll stick with the Haar wavelet, based on the
> guidance from the journal articles which I've been reading authored
by
> Casati et al. which favor it when working with precip verification.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, October 22, 2018 1:47 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about
> MET V5.2 Wavelet-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Sorry for the delay in responding. I think I lost track of this
email in
> the shuffle.
>
> So you've run with 2 thresholds. The lower one (>0) resulted in the
event
> occurring everywhere in both the forecast and observation domains.
Since
> there is no different in the binary fields, there isn't anything to
> decompose.
>
> But there is a meaningful difference for the higher threshold (>=3).
And
> the results you're seeing are consistent with output I've seen in
the
> past. I don't have a good explanation for *why* you're seeing
horizontal
> banding at the smallest scales, but I have seen it before. I
suspect
> they're small artifacts of the wavelet decomposition process.
>
> There is an option in the Wavelet-Stat config file to switch what
wavelet
> is used for the decomposition process. However, the only published
results
> of this algorithm use the default "Haar" wavelet decomposition.
>
> Thanks,
> John
>
>
>
> Thanks,
> John
>
>
>
> On Thu, Oct 11, 2018 at 4:00 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John or Tressa -
> >
> > I used the "TILE" method to position a single tile to be centered
over
> > the object I'm interested in and then ran wavelet-stat. See
attached
> > output ps file.
> >
> > I'm curious about the pattern seen in the graphic produced for
> > threshold
> > >=3.0 for scales 1, 2, and maybe 3. I also see this for threshold
> > >>=9.0 for
> > scales 1 and 2, but not as pronounced. These show horizontal
banding
> > which sometimes traverses the entire domain from west to east. Is
this
> > a normal pattern?
> >
> > I assume that the graphics for thresholds >0.0 and >=12.0 were
without
> > results because these thresholds either have no events (>=12.0) or
the
> > entire area was an event (>0.0) and thus the errors are
undefined.
> > Let me know your thoughts on this.
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, October 11, 2018 3:04 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349]
question
> > about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Honestly, I haven't used Wavelet-Stat very much on testing and
> > evaluation projects. So I'll refer you to Tressa Fowler (cc'ed
here)
> > for question about interpreting the results.
> >
> > As for the range of the Intensity Scale Skill Score values, it
looks
> > to be (-inf, 1].
> >
> > One nice feature of the MSE, FE2, and OE2 quantities is that they
are
> > additive. When you sum their values across the scales you get the
> > value for the binary difference field. And that's why and how the
> > PostScript plot reports the percentage of the error for each
scale.
> >
> > Thanks,
> > John
> >
> > On Thu, Oct 11, 2018 at 12:48 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 349 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Sorry for the delayed response. I just got back from an hour
long mtg.
> > >
> > > Thanks for explaining the reason for the absence of any red of
blue
> > > color for small differences. I'll try your suggestion for a
> > > different ctable and let you know how it works.
> > >
> > > I noticed that the IS score associated with those two plots was
the
> > > highest of the scores for that threshold, so I think this makes
> > > sense that the differences would fall into the smallest category
> > > with the white color assignment. What is the expected range of
the IS
> score?
> > >
> > > R/
> > > John
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Thursday, October 11, 2018 11:14 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349]
question
> > > about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Looking at the color bar, I see that any differences between
-0.1
> > > and
> > > +0.1 are plotted as white. The very small differences as the
scale
> > > are falling into that category and being plotted as white.
> > >
> > > Looking in the Wavelet-Stat config file, I see the following:
> > > wvlt_plot = {
> > > color_table =
"MET_BASE/colortables/NCL_colortables/BlWhRe.ctable";
> > > plot_min = -1.0;
> > > plot_max = 1.0;
> > > }
> > >
> > > This defines the colortable used for these different plots.
Setting
> > > plot_min and plot_max to -1 and 1 tells Wavelet-Stat to rescale
this
> > > colortable using those min/max values. Let's switch to using
the
> > > "BlueRed.ctable" without the white in the middle, like this:
> > > color_table =
> > > "MET_BASE/colortables/NCL_colortables/BlueRed.ctable";
> > >
> > > When you rerun, are those very small differences now showing up?
> > >
> > > You can either use a different color table or manually make one
> > > yourself to plot it exactly how you'd like.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > > On Thu, Oct 11, 2018 at 10:24 AM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 349 >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > John -
> > > >
> > > > Thanks for clarifying how wavelet-stat works. I read the
User's
> > > > Guide (p
> > > > 48) and see the following which is why I thought that the
scale
> > > > referred to
> > > > En2:
> > > > "The En2 relative difference ranges between -1 and 1, positive
> > > > values indicate over-forecast and negative values indicate
> > under-forecast."
> > > >
> > > > Also see the attached images for the last two scales which
> > > > wavelet-stat produced. Why is there no color in the plot?
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT
> > > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > > Sent: Thursday, October 11, 2018 9:58 AM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349]
question
> > > > about MET
> > > > V5.2 Wavelet-Stat (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity of
> > > > all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > Hi John,
> > > >
> > > > I see that you have a question about the PostScript output of
the
> > > > Wavelet-Stat tool in MET. Wavelet-Stat implements the
following
> logic:
> > > >
> > > > (1) Define a forecast and observation field to be compared.
> > > > (2) Apply a threshold to both to convert them to binary fields
of
> > > > 0's and 1's.
> > > > (3) Compute the fcst - obs binary difference... and that's the
> > > > plot you sent to me. If fcst and obs are both in range [0,
1],
> > > > then their difference has range [-1, 1]. And the default
color
> > > > table plots negative values in blue and positive values in
red.
> > > > (4) Apply the haar wavelet (configurable) to decompose that
> > > > difference field by scale. And plot the difference field for
each
> > > > scale. The decomposed differences also have range [-1, 1] and
are
> > > > plotted with the same color scale.
> > > >
> > > > If you'd prefer a different color table you can easily change
in
> > > > the Wavelet-Stat config file using the "color_table" entries.
> > > >
> > > > Hope this helps clarify.
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > > On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > Queue: met_help
> > > > > Subject: question about MET V5.2 Wavelet-Stat
(UNCLASSIFIED)
> > > > > Owner: Nobody
> > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > Status: new
> > > > > Ticket <Caution-Caution-Caution-Caution-url:
> > > > > Caution-Caution-Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > > html?id=87
> > > > > 349 >
> > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > In the attached image from the postscript output file, is
the
> > > > > blue-red color scale on the right border of the plot showing
the
> > > > > values of En2 relative difference? Does the same apply to
the
> > > > > plots
> > > for all scales?
> > > > >
> > > > > Thanks
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > Mr. John W. Raby
> > > > > U.S. Army Research Laboratory
> > > > > White Sands Missile Range, NM 88002
> > > > > (575) 678-2004 DSN 258-2004
> > > > > FAX (575) 678-1230 DSN 258-1230
> > > > > Email: john.w.raby2.civ at mail.mil
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Oct 23 10:41:20 2018
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for your response and the info about the image analysis work
being done by Eric as an additional reference to the lit search I've
been doing on the scale decomposition verification method. I'll check
out that link you provided.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, October 23, 2018 9:47 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Yes, I definitely agree with your conclusion about the lowest and
highest thresholds. Wavelet-Stat decomposes the binary difference
field after the threshold has been applied. If that difference field
is constant (either all -1, all 0, or all 1), then it won't result in
a very interesting decomposition.
FYI, Eric Gilleland is a scientist at NCAR who's been applying some
interesting image analysis techniques to forecast verification. I've
worked with Eric for many years, but we finally have some time/funding
set aside this year to put some of his image analysis work into MET.
If you're interested in wavelet decompositions, you may find his work
interesting as well. This is an outgrowth of the forecast inter-
comparison project, and more info about it can be found here:
Caution-http://www.ral.ucar.edu/projects/icp/
Thanks,
John
On Tue, Oct 23, 2018 at 7:54 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87349 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for looking at the PDF containing my wavelet-stat output. I
> think you had cc'd Tressa on the previous email in order to get her
> take on the banding I was seeing for 2 of the 5 thresholds I used.
> Your explanation for the results for the >0 threshold make sense to
> me. I think the same situation applies to the results for >=12.0 (no
events). Do you concur?
>
> Regards the banding artifact visible for the results for >=3.0 and
> >=9.0, thanks for sharing your experience noting that this is not an
> unexpected pattern. I think that I'll stick with the Haar wavelet,
> based on the guidance from the journal articles which I've been
> reading authored by Casati et al. which favor it when working with
precip verification.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Monday, October 22, 2018 1:47 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349] question
> about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Sorry for the delay in responding. I think I lost track of this
email
> in the shuffle.
>
> So you've run with 2 thresholds. The lower one (>0) resulted in the
> event occurring everywhere in both the forecast and observation
> domains. Since there is no different in the binary fields, there
> isn't anything to decompose.
>
> But there is a meaningful difference for the higher threshold (>=3).
> And the results you're seeing are consistent with output I've seen
in
> the past. I don't have a good explanation for *why* you're seeing
> horizontal banding at the smallest scales, but I have seen it
before.
> I suspect they're small artifacts of the wavelet decomposition
process.
>
> There is an option in the Wavelet-Stat config file to switch what
> wavelet is used for the decomposition process. However, the only
> published results of this algorithm use the default "Haar" wavelet
decomposition.
>
> Thanks,
> John
>
>
>
> Thanks,
> John
>
>
>
> On Thu, Oct 11, 2018 at 4:00 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 349 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John or Tressa -
> >
> > I used the "TILE" method to position a single tile to be centered
> > over the object I'm interested in and then ran wavelet-stat. See
> > attached output ps file.
> >
> > I'm curious about the pattern seen in the graphic produced for
> > threshold
> > >=3.0 for scales 1, 2, and maybe 3. I also see this for threshold
> > >>=9.0 for
> > scales 1 and 2, but not as pronounced. These show horizontal
banding
> > which sometimes traverses the entire domain from west to east. Is
> > this a normal pattern?
> >
> > I assume that the graphics for thresholds >0.0 and >=12.0 were
> > without results because these thresholds either have no events
> > (>=12.0) or the entire area was an event (>0.0) and thus the
errors are undefined.
> > Let me know your thoughts on this.
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, October 11, 2018 3:04 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349]
question
> > about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Honestly, I haven't used Wavelet-Stat very much on testing and
> > evaluation projects. So I'll refer you to Tressa Fowler (cc'ed
> > here) for question about interpreting the results.
> >
> > As for the range of the Intensity Scale Skill Score values, it
looks
> > to be (-inf, 1].
> >
> > One nice feature of the MSE, FE2, and OE2 quantities is that they
> > are additive. When you sum their values across the scales you get
> > the value for the binary difference field. And that's why and how
> > the PostScript plot reports the percentage of the error for each
scale.
> >
> > Thanks,
> > John
> >
> > On Thu, Oct 11, 2018 at 12:48 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=87
> > > 349 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Sorry for the delayed response. I just got back from an hour
long mtg.
> > >
> > > Thanks for explaining the reason for the absence of any red of
> > > blue color for small differences. I'll try your suggestion for a
> > > different ctable and let you know how it works.
> > >
> > > I noticed that the IS score associated with those two plots was
> > > the highest of the scores for that threshold, so I think this
> > > makes sense that the differences would fall into the smallest
> > > category with the white color assignment. What is the expected
> > > range of the IS
> score?
> > >
> > > R/
> > > John
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Thursday, October 11, 2018 11:14 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349]
> > > question about MET V5.2 Wavelet-Stat (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Looking at the color bar, I see that any differences between
-0.1
> > > and
> > > +0.1 are plotted as white. The very small differences as the
> > > +scale
> > > are falling into that category and being plotted as white.
> > >
> > > Looking in the Wavelet-Stat config file, I see the following:
> > > wvlt_plot = {
> > > color_table =
"MET_BASE/colortables/NCL_colortables/BlWhRe.ctable";
> > > plot_min = -1.0;
> > > plot_max = 1.0;
> > > }
> > >
> > > This defines the colortable used for these different plots.
> > > Setting plot_min and plot_max to -1 and 1 tells Wavelet-Stat to
> > > rescale this colortable using those min/max values. Let's
switch
> > > to using the "BlueRed.ctable" without the white in the middle,
like this:
> > > color_table =
> > > "MET_BASE/colortables/NCL_colortables/BlueRed.ctable";
> > >
> > > When you rerun, are those very small differences now showing up?
> > >
> > > You can either use a different color table or manually make one
> > > yourself to plot it exactly how you'd like.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > > On Thu, Oct 11, 2018 at 10:24 AM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 349 >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > John -
> > > >
> > > > Thanks for clarifying how wavelet-stat works. I read the
User's
> > > > Guide (p
> > > > 48) and see the following which is why I thought that the
scale
> > > > referred to
> > > > En2:
> > > > "The En2 relative difference ranges between -1 and 1, positive
> > > > values indicate over-forecast and negative values indicate
> > under-forecast."
> > > >
> > > > Also see the attached images for the last two scales which
> > > > wavelet-stat produced. Why is there no color in the plot?
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT
> > > > [Caution-Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > > Sent: Thursday, October 11, 2018 9:58 AM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87349]
question
> > > > about MET
> > > > V5.2 Wavelet-Stat (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity
> > > > of all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > Hi John,
> > > >
> > > > I see that you have a question about the PostScript output of
> > > > the Wavelet-Stat tool in MET. Wavelet-Stat implements the
> > > > following
> logic:
> > > >
> > > > (1) Define a forecast and observation field to be compared.
> > > > (2) Apply a threshold to both to convert them to binary fields
> > > > of 0's and 1's.
> > > > (3) Compute the fcst - obs binary difference... and that's the
> > > > plot you sent to me. If fcst and obs are both in range [0,
1],
> > > > then their difference has range [-1, 1]. And the default
color
> > > > table plots negative values in blue and positive values in
red.
> > > > (4) Apply the haar wavelet (configurable) to decompose that
> > > > difference field by scale. And plot the difference field for
> > > > each scale. The decomposed differences also have range [-1,
1]
> > > > and are plotted with the same color scale.
> > > >
> > > > If you'd prefer a different color table you can easily change
in
> > > > the Wavelet-Stat config file using the "color_table" entries.
> > > >
> > > > Hope this helps clarify.
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > > On Wed, Oct 10, 2018 at 3:57 PM Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Oct 10 15:50:23 2018: Request 87349 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > Queue: met_help
> > > > > Subject: question about MET V5.2 Wavelet-Stat
(UNCLASSIFIED)
> > > > > Owner: Nobody
> > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > Status: new
> > > > > Ticket <Caution-Caution-Caution-Caution-Caution-url:
> > > > > Caution-Caution-Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > > html?id=87
> > > > > 349 >
> > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > In the attached image from the postscript output file, is
the
> > > > > blue-red color scale on the right border of the plot showing
> > > > > the values of En2 relative difference? Does the same apply
to
> > > > > the plots
> > > for all scales?
> > > > >
> > > > > Thanks
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > Mr. John W. Raby
> > > > > U.S. Army Research Laboratory
> > > > > White Sands Missile Range, NM 88002
> > > > > (575) 678-2004 DSN 258-2004
> > > > > FAX (575) 678-1230 DSN 258-1230
> > > > > Email: john.w.raby2.civ at mail.mil
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
More information about the Met_help
mailing list