[Met_help] [rt.rap.ucar.edu #86995] History for MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
John Halley Gotway via RT
met_help at ucar.edu
Mon Oct 22 13:34:56 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi Tracey,
I haven't tried showing stats for each threshold, you're way ahead
of me. Let me forward your inquiry to met_help.
Dear met_help:
I think Tracey is referring to the numbers such as that on top of
this plot:
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/201809/intl_reg.201809.gif,
for each verifying threshold.
Thanks,
Ying
-------- Forwarded Message --------
Subject: FBIAS east vs. west US
Date: Mon, 17 Sep 2018 09:49:51 -0400
From: Tracey Dorian - NOAA Affiliate <tracey.dorian at noaa.gov>
To: Ying Lin <ying.lin at noaa.gov>
Hi Ying,
I've created plots of FBIAS for east regions vs. west regions and Geoff
was wondering if I can show the number of events for each threshold? I
know METViewer has the "display number of stats" option which I checked
off, but it seems to show the same number for each threshold. Do you
know if there is a way to show the different number of stats for each
threshold? We would probably expect the number of events to be high for
the lower thresholds and get progressively smaller as you move into the
higher thresholds, right?
Thanks,
Tracey
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: John Halley Gotway
Time: Mon Sep 17 11:02:59 2018
Ying and Tracey,
I see that you'd like to include in your METviewer plot information
about
the event frequency. You tried using the "Display Number of Stats"
option
but that that didn't do exactly what you want. Those numbers stay
constant
for all the thresholds.
The "NStats" values listed across the top indicate the sample size.
They
are computed as the sum of the number of cases that are included
across all
the series (i.e. lines on the plot) for each independent variable
value.
This is not the number of events meeting each threshold criteria...
it's a
count of the number of "cases" included. In the sample plots you
sent,
you have NStats = 33948. Since you didn't send the corresponding XML,
it's
difficult for me to exactly replicate them. Basically though, the
logic is:
- For a particular independent variable value, >1.0" for example,
you
have 2 lines on the plot.
- You have many model initializations included, perhaps 397.
- You have multiple lead times included. For APCP/24 probably 24,
48,
72, 96... perhaps about 8 of them.
- You're including multiple subregions in each plot... for example,
the
EAST consists of 6 subregions.
Multiplying all those numbers together... 2 * 397 * 8 * 6 = 38112 gets
you
somewhere near the actual number of 33948 we're seeing.
If you want to see the frequency at which the event is actually
occurring,
I'd suggest plotting the base rate statistic, called BASER in
METviewer.
That's exactly what you want... a number between 0 and 1 representing
the
event frequency in the observation data. The BASER could be shown
separately for the GFS and FV3GFS... but as long as you're using the
exact
same cases, they should be identical. We often plot the BASER on the
Y2
axis so that we can control the range of the Y2 axis, but I ran into
some
problems doing that.
Instead, I put it on the Y1 axis with FBIAS but plotted it like a
histogram
with bars.
I made the attached plot/xml using the mv_ylin_pcp database...
- It includes 4 lines where FV3GFS is blue and GFS is red.
- The solid lines are for the EAST region and dashed lines are for
the
WEST region. Take a close look at the Series definition section. I
defined the EAST as a "Group" of 6 subregions and the WEST as a
"Group" of
8 subregions.
- The gray bars show the base rate for the EAST and WEST regions.
Since
I only want to plot 2 of the 4, I chose "Yes" for the "Hide" column
for the
2 lines I wanted to ignore.
- In the "Series Formatting" I used the "Legend Text" column to
define
the legend.
Granted, this plot was pretty tedious to make manually because of the
masking region groups, but hopefully it includes all the info you
need.
And note that statistical significance is not currently included in
this
plot, but it could be added.
Thanks,
John
On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
> Transaction: Ticket created by ying.lin at noaa.gov
> Queue: met_help
> Subject: MV CTC plot: count of data points for each threshold
(Fwd:
> FBIAS east vs. west US
> Owner: Nobody
> Requestors: ying.lin at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>
>
> Hi Tracey,
>
> I haven't tried showing stats for each threshold, you're way
ahead
> of me. Let me forward your inquiry to met_help.
>
> Dear met_help:
>
> I think Tracey is referring to the numbers such as that on top
of
> this plot:
>
>
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/201809/intl_reg.201809.gif,
>
> for each verifying threshold.
>
> Thanks,
>
> Ying
>
>
> -------- Forwarded Message --------
> Subject: FBIAS east vs. west US
> Date: Mon, 17 Sep 2018 09:49:51 -0400
> From: Tracey Dorian - NOAA Affiliate <tracey.dorian at noaa.gov>
> To: Ying Lin <ying.lin at noaa.gov>
>
>
>
> Hi Ying,
>
> I've created plots of FBIAS for east regions vs. west regions and
Geoff
> was wondering if I can show the number of events for each threshold?
I
> know METViewer has the "display number of stats" option which I
checked
> off, but it seems to show the same number for each threshold. Do
you
> know if there is a way to show the different number of stats for
each
> threshold? We would probably expect the number of events to be high
for
> the lower thresholds and get progressively smaller as you move into
the
> higher thresholds, right?
>
> Thanks,
> Tracey
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #86995] MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: Ying Lin
Time: Mon Sep 17 11:32:36 2018
Thanks very much John!
Tracey, did this answer your question? I have an RFC/Dataflow crisis
today so haven't tested the answer. If you have further questions,
please follow-up on this met_help ticket (and cc to me - sure would
like
to know too). Thanks.
Ying
On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
> Ying and Tracey,
>
> I see that you'd like to include in your METviewer plot information
about
> the event frequency. You tried using the "Display Number of Stats"
option
> but that that didn't do exactly what you want. Those numbers stay
constant
> for all the thresholds.
>
> The "NStats" values listed across the top indicate the sample size.
They
> are computed as the sum of the number of cases that are included
across all
> the series (i.e. lines on the plot) for each independent variable
value.
> This is not the number of events meeting each threshold criteria...
it's a
> count of the number of "cases" included. In the sample plots you
sent,
> you have NStats = 33948. Since you didn't send the corresponding
XML, it's
> difficult for me to exactly replicate them. Basically though, the
logic is:
> - For a particular independent variable value, >1.0" for example,
you
> have 2 lines on the plot.
> - You have many model initializations included, perhaps 397.
> - You have multiple lead times included. For APCP/24 probably
24, 48,
> 72, 96... perhaps about 8 of them.
> - You're including multiple subregions in each plot... for
example, the
> EAST consists of 6 subregions.
>
> Multiplying all those numbers together... 2 * 397 * 8 * 6 = 38112
gets you
> somewhere near the actual number of 33948 we're seeing.
>
> If you want to see the frequency at which the event is actually
occurring,
> I'd suggest plotting the base rate statistic, called BASER in
METviewer.
> That's exactly what you want... a number between 0 and 1
representing the
> event frequency in the observation data. The BASER could be shown
> separately for the GFS and FV3GFS... but as long as you're using the
exact
> same cases, they should be identical. We often plot the BASER on
the Y2
> axis so that we can control the range of the Y2 axis, but I ran into
some
> problems doing that.
>
> Instead, I put it on the Y1 axis with FBIAS but plotted it like a
histogram
> with bars.
>
> I made the attached plot/xml using the mv_ylin_pcp database...
> - It includes 4 lines where FV3GFS is blue and GFS is red.
> - The solid lines are for the EAST region and dashed lines are
for the
> WEST region. Take a close look at the Series definition section. I
> defined the EAST as a "Group" of 6 subregions and the WEST as a
"Group" of
> 8 subregions.
> - The gray bars show the base rate for the EAST and WEST regions.
Since
> I only want to plot 2 of the 4, I chose "Yes" for the "Hide" column
for the
> 2 lines I wanted to ignore.
> - In the "Series Formatting" I used the "Legend Text" column to
define
> the legend.
>
> Granted, this plot was pretty tedious to make manually because of
the
> masking region groups, but hopefully it includes all the info you
need.
>
> And note that statistical significance is not currently included in
this
> plot, but it could be added.
>
> Thanks,
> John
>
>
>
> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
>> Transaction: Ticket created by ying.lin at noaa.gov
>> Queue: met_help
>> Subject: MV CTC plot: count of data points for each threshold
(Fwd:
>> FBIAS east vs. west US
>> Owner: Nobody
>> Requestors: ying.lin at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>>
>>
>> Hi Tracey,
>>
>> I haven't tried showing stats for each threshold, you're way
ahead
>> of me. Let me forward your inquiry to met_help.
>>
>> Dear met_help:
>>
>> I think Tracey is referring to the numbers such as that on
top of
>> this plot:
>>
>>
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/201809/intl_reg.201809.gif,
>>
>> for each verifying threshold.
>>
>> Thanks,
>>
>> Ying
>>
>>
>> -------- Forwarded Message --------
>> Subject: FBIAS east vs. west US
>> Date: Mon, 17 Sep 2018 09:49:51 -0400
>> From: Tracey Dorian - NOAA Affiliate <tracey.dorian at noaa.gov>
>> To: Ying Lin <ying.lin at noaa.gov>
>>
>>
>>
>> Hi Ying,
>>
>> I've created plots of FBIAS for east regions vs. west regions and
Geoff
>> was wondering if I can show the number of events for each
threshold? I
>> know METViewer has the "display number of stats" option which I
checked
>> off, but it seems to show the same number for each threshold. Do
you
>> know if there is a way to show the different number of stats for
each
>> threshold? We would probably expect the number of events to be
high for
>> the lower thresholds and get progressively smaller as you move into
the
>> higher thresholds, right?
>>
>> Thanks,
>> Tracey
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: tracey.dorian at noaa.gov
Time: Mon Sep 17 13:31:30 2018
Hi John,
Thanks for much for your quick reply! I am trying to add the BASER
field
to my plots, but I'm getting errors, specifically a proxy timeout. Do
you
know why this is happening? I've attached my XML file to this email.
Thanks,
Tracey
On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin <ying.lin at noaa.gov> wrote:
> Thanks very much John!
>
> Tracey, did this answer your question? I have an RFC/Dataflow
crisis
> today so haven't tested the answer. If you have further questions,
please
> follow-up on this met_help ticket (and cc to me - sure would like to
know
> too). Thanks.
>
> Ying
>
>
> On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
>
>> Ying and Tracey,
>>
>> I see that you'd like to include in your METviewer plot information
about
>> the event frequency. You tried using the "Display Number of Stats"
option
>> but that that didn't do exactly what you want. Those numbers stay
>> constant
>> for all the thresholds.
>>
>> The "NStats" values listed across the top indicate the sample size.
They
>> are computed as the sum of the number of cases that are included
across
>> all
>> the series (i.e. lines on the plot) for each independent variable
value.
>> This is not the number of events meeting each threshold criteria...
it's a
>> count of the number of "cases" included. In the sample plots you
sent,
>> you have NStats = 33948. Since you didn't send the corresponding
XML,
>> it's
>> difficult for me to exactly replicate them. Basically though, the
logic
>> is:
>> - For a particular independent variable value, >1.0" for
example, you
>> have 2 lines on the plot.
>> - You have many model initializations included, perhaps 397.
>> - You have multiple lead times included. For APCP/24 probably
24, 48,
>> 72, 96... perhaps about 8 of them.
>> - You're including multiple subregions in each plot... for
example, the
>> EAST consists of 6 subregions.
>>
>> Multiplying all those numbers together... 2 * 397 * 8 * 6 = 38112
gets you
>> somewhere near the actual number of 33948 we're seeing.
>>
>> If you want to see the frequency at which the event is actually
occurring,
>> I'd suggest plotting the base rate statistic, called BASER in
METviewer.
>> That's exactly what you want... a number between 0 and 1
representing the
>> event frequency in the observation data. The BASER could be shown
>> separately for the GFS and FV3GFS... but as long as you're using
the exact
>> same cases, they should be identical. We often plot the BASER on
the Y2
>> axis so that we can control the range of the Y2 axis, but I ran
into some
>> problems doing that.
>>
>> Instead, I put it on the Y1 axis with FBIAS but plotted it like a
>> histogram
>> with bars.
>>
>> I made the attached plot/xml using the mv_ylin_pcp database...
>> - It includes 4 lines where FV3GFS is blue and GFS is red.
>> - The solid lines are for the EAST region and dashed lines are
for the
>> WEST region. Take a close look at the Series definition section.
I
>> defined the EAST as a "Group" of 6 subregions and the WEST as a
"Group" of
>> 8 subregions.
>> - The gray bars show the base rate for the EAST and WEST
regions.
>> Since
>> I only want to plot 2 of the 4, I chose "Yes" for the "Hide" column
for
>> the
>> 2 lines I wanted to ignore.
>> - In the "Series Formatting" I used the "Legend Text" column to
define
>> the legend.
>>
>> Granted, this plot was pretty tedious to make manually because of
the
>> masking region groups, but hopefully it includes all the info you
need.
>>
>> And note that statistical significance is not currently included in
this
>> plot, but it could be added.
>>
>> Thanks,
>> John
>>
>>
>>
>> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <met_help at ucar.edu>
>> wrote:
>>
>> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
>>> Transaction: Ticket created by ying.lin at noaa.gov
>>> Queue: met_help
>>> Subject: MV CTC plot: count of data points for each
threshold (Fwd:
>>> FBIAS east vs. west US
>>> Owner: Nobody
>>> Requestors: ying.lin at noaa.gov
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>>>
>>>
>>> Hi Tracey,
>>>
>>> I haven't tried showing stats for each threshold, you're way
ahead
>>> of me. Let me forward your inquiry to met_help.
>>>
>>> Dear met_help:
>>>
>>> I think Tracey is referring to the numbers such as that on
top of
>>> this plot:
>>>
>>> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
>>> 01809/intl_reg.201809.gif,
>>>
>>> for each verifying threshold.
>>>
>>> Thanks,
>>>
>>> Ying
>>>
>>>
>>> -------- Forwarded Message --------
>>> Subject: FBIAS east vs. west US
>>> Date: Mon, 17 Sep 2018 09:49:51 -0400
>>> From: Tracey Dorian - NOAA Affiliate <tracey.dorian at noaa.gov>
>>> To: Ying Lin <ying.lin at noaa.gov>
>>>
>>>
>>>
>>> Hi Ying,
>>>
>>> I've created plots of FBIAS for east regions vs. west regions and
Geoff
>>> was wondering if I can show the number of events for each
threshold? I
>>> know METViewer has the "display number of stats" option which I
checked
>>> off, but it seems to show the same number for each threshold. Do
you
>>> know if there is a way to show the different number of stats for
each
>>> threshold? We would probably expect the number of events to be
high for
>>> the lower thresholds and get progressively smaller as you move
into the
>>> higher thresholds, right?
>>>
>>> Thanks,
>>> Tracey
>>>
>>>
>>>
>>>
>>>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
--
Tracey Dorian
NOAA/NCEP/EMC
IMSG contractor
5830 University Research Court
College Park, MD 20740
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: John Halley Gotway
Time: Mon Sep 17 14:02:41 2018
Tracey,
Thanks for sending your XML file. I didn't see any reference to
"baser" in
the XML you sent. So I added it as a second statistic to be plotted.
Since you didn't define a VX_MASK to be used, METviewer tried to use
all
available in the database, which likely made it run slowly.
For testing, I set VX_MASK = G218/SEC added a line for BASER and
attached
the result.
Is that what you're looking for?
Thanks,
John
On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>
> Hi John,
>
> Thanks for much for your quick reply! I am trying to add the BASER
field
> to my plots, but I'm getting errors, specifically a proxy timeout.
Do you
> know why this is happening? I've attached my XML file to this
email.
>
> Thanks,
> Tracey
>
>
>
> On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin <ying.lin at noaa.gov> wrote:
>
> > Thanks very much John!
> >
> > Tracey, did this answer your question? I have an RFC/Dataflow
crisis
> > today so haven't tested the answer. If you have further
questions,
> please
> > follow-up on this met_help ticket (and cc to me - sure would like
to know
> > too). Thanks.
> >
> > Ying
> >
> >
> > On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
> >
> >> Ying and Tracey,
> >>
> >> I see that you'd like to include in your METviewer plot
information
> about
> >> the event frequency. You tried using the "Display Number of
Stats"
> option
> >> but that that didn't do exactly what you want. Those numbers
stay
> >> constant
> >> for all the thresholds.
> >>
> >> The "NStats" values listed across the top indicate the sample
size.
> They
> >> are computed as the sum of the number of cases that are included
across
> >> all
> >> the series (i.e. lines on the plot) for each independent variable
value.
> >> This is not the number of events meeting each threshold
criteria...
> it's a
> >> count of the number of "cases" included. In the sample plots
you
> sent,
> >> you have NStats = 33948. Since you didn't send the corresponding
XML,
> >> it's
> >> difficult for me to exactly replicate them. Basically though,
the logic
> >> is:
> >> - For a particular independent variable value, >1.0" for
example, you
> >> have 2 lines on the plot.
> >> - You have many model initializations included, perhaps 397.
> >> - You have multiple lead times included. For APCP/24 probably
24,
> 48,
> >> 72, 96... perhaps about 8 of them.
> >> - You're including multiple subregions in each plot... for
example,
> the
> >> EAST consists of 6 subregions.
> >>
> >> Multiplying all those numbers together... 2 * 397 * 8 * 6 = 38112
gets
> you
> >> somewhere near the actual number of 33948 we're seeing.
> >>
> >> If you want to see the frequency at which the event is actually
> occurring,
> >> I'd suggest plotting the base rate statistic, called BASER in
METviewer.
> >> That's exactly what you want... a number between 0 and 1
representing
> the
> >> event frequency in the observation data. The BASER could be
shown
> >> separately for the GFS and FV3GFS... but as long as you're using
the
> exact
> >> same cases, they should be identical. We often plot the BASER on
the Y2
> >> axis so that we can control the range of the Y2 axis, but I ran
into
> some
> >> problems doing that.
> >>
> >> Instead, I put it on the Y1 axis with FBIAS but plotted it like a
> >> histogram
> >> with bars.
> >>
> >> I made the attached plot/xml using the mv_ylin_pcp database...
> >> - It includes 4 lines where FV3GFS is blue and GFS is red.
> >> - The solid lines are for the EAST region and dashed lines are
for
> the
> >> WEST region. Take a close look at the Series definition section.
I
> >> defined the EAST as a "Group" of 6 subregions and the WEST as a
"Group"
> of
> >> 8 subregions.
> >> - The gray bars show the base rate for the EAST and WEST
regions.
> >> Since
> >> I only want to plot 2 of the 4, I chose "Yes" for the "Hide"
column for
> >> the
> >> 2 lines I wanted to ignore.
> >> - In the "Series Formatting" I used the "Legend Text" column
to
> define
> >> the legend.
> >>
> >> Granted, this plot was pretty tedious to make manually because of
the
> >> masking region groups, but hopefully it includes all the info you
need.
> >>
> >> And note that statistical significance is not currently included
in this
> >> plot, but it could be added.
> >>
> >> Thanks,
> >> John
> >>
> >>
> >>
> >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>
> >> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
> >>> Transaction: Ticket created by ying.lin at noaa.gov
> >>> Queue: met_help
> >>> Subject: MV CTC plot: count of data points for each
threshold
> (Fwd:
> >>> FBIAS east vs. west US
> >>> Owner: Nobody
> >>> Requestors: ying.lin at noaa.gov
> >>> Status: new
> >>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
> >
> >>>
> >>>
> >>> Hi Tracey,
> >>>
> >>> I haven't tried showing stats for each threshold, you're
way
> ahead
> >>> of me. Let me forward your inquiry to met_help.
> >>>
> >>> Dear met_help:
> >>>
> >>> I think Tracey is referring to the numbers such as that on
top of
> >>> this plot:
> >>>
> >>> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> >>> 01809/intl_reg.201809.gif,
> >>>
> >>> for each verifying threshold.
> >>>
> >>> Thanks,
> >>>
> >>> Ying
> >>>
> >>>
> >>> -------- Forwarded Message --------
> >>> Subject: FBIAS east vs. west US
> >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> >>> From: Tracey Dorian - NOAA Affiliate <tracey.dorian at noaa.gov>
> >>> To: Ying Lin <ying.lin at noaa.gov>
> >>>
> >>>
> >>>
> >>> Hi Ying,
> >>>
> >>> I've created plots of FBIAS for east regions vs. west regions
and Geoff
> >>> was wondering if I can show the number of events for each
threshold? I
> >>> know METViewer has the "display number of stats" option which I
checked
> >>> off, but it seems to show the same number for each threshold.
Do you
> >>> know if there is a way to show the different number of stats for
each
> >>> threshold? We would probably expect the number of events to be
high
> for
> >>> the lower thresholds and get progressively smaller as you move
into the
> >>> higher thresholds, right?
> >>>
> >>> Thanks,
> >>> Tracey
> >>>
> >>>
> >>>
> >>>
> >>>
> > --
> > Ying Lin
> > NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> > NCWCP Cubicle No. 2015
> > Ying.Lin at noaa.gov
> >
> >
> >
>
>
> --
> Tracey Dorian
> NOAA/NCEP/EMC
> IMSG contractor
> 5830 University Research Court
> College Park, MD 20740
>
>
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: tracey.dorian at noaa.gov
Time: Mon Sep 17 14:26:37 2018
Hi John,
Yes, that's what I was looking for! How were you able to plot the
base
rate using bars instead of lines? Also, how would you suggest adding
statistical significance to the plot?
Thanks for sending your XML file!
-Tracey
On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Tracey,
>
> Thanks for sending your XML file. I didn't see any reference to
"baser" in
> the XML you sent. So I added it as a second statistic to be
plotted.
>
> Since you didn't define a VX_MASK to be used, METviewer tried to use
all
> available in the database, which likely made it run slowly.
>
> For testing, I set VX_MASK = G218/SEC added a line for BASER and
attached
> the result.
>
> Is that what you're looking for?
>
> Thanks,
> John
>
> On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> >
> > Hi John,
> >
> > Thanks for much for your quick reply! I am trying to add the
BASER field
> > to my plots, but I'm getting errors, specifically a proxy timeout.
Do
> you
> > know why this is happening? I've attached my XML file to this
email.
> >
> > Thanks,
> > Tracey
> >
> >
> >
> > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin <ying.lin at noaa.gov>
wrote:
> >
> > > Thanks very much John!
> > >
> > > Tracey, did this answer your question? I have an RFC/Dataflow
crisis
> > > today so haven't tested the answer. If you have further
questions,
> > please
> > > follow-up on this met_help ticket (and cc to me - sure would
like to
> know
> > > too). Thanks.
> > >
> > > Ying
> > >
> > >
> > > On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
> > >
> > >> Ying and Tracey,
> > >>
> > >> I see that you'd like to include in your METviewer plot
information
> > about
> > >> the event frequency. You tried using the "Display Number of
Stats"
> > option
> > >> but that that didn't do exactly what you want. Those numbers
stay
> > >> constant
> > >> for all the thresholds.
> > >>
> > >> The "NStats" values listed across the top indicate the sample
size.
> > They
> > >> are computed as the sum of the number of cases that are
included
> across
> > >> all
> > >> the series (i.e. lines on the plot) for each independent
variable
> value.
> > >> This is not the number of events meeting each threshold
criteria...
> > it's a
> > >> count of the number of "cases" included. In the sample plots
you
> > sent,
> > >> you have NStats = 33948. Since you didn't send the
corresponding XML,
> > >> it's
> > >> difficult for me to exactly replicate them. Basically though,
the
> logic
> > >> is:
> > >> - For a particular independent variable value, >1.0" for
example,
> you
> > >> have 2 lines on the plot.
> > >> - You have many model initializations included, perhaps 397.
> > >> - You have multiple lead times included. For APCP/24
probably 24,
> > 48,
> > >> 72, 96... perhaps about 8 of them.
> > >> - You're including multiple subregions in each plot... for
example,
> > the
> > >> EAST consists of 6 subregions.
> > >>
> > >> Multiplying all those numbers together... 2 * 397 * 8 * 6 =
38112 gets
> > you
> > >> somewhere near the actual number of 33948 we're seeing.
> > >>
> > >> If you want to see the frequency at which the event is actually
> > occurring,
> > >> I'd suggest plotting the base rate statistic, called BASER in
> METviewer.
> > >> That's exactly what you want... a number between 0 and 1
representing
> > the
> > >> event frequency in the observation data. The BASER could be
shown
> > >> separately for the GFS and FV3GFS... but as long as you're
using the
> > exact
> > >> same cases, they should be identical. We often plot the BASER
on the
> Y2
> > >> axis so that we can control the range of the Y2 axis, but I ran
into
> > some
> > >> problems doing that.
> > >>
> > >> Instead, I put it on the Y1 axis with FBIAS but plotted it like
a
> > >> histogram
> > >> with bars.
> > >>
> > >> I made the attached plot/xml using the mv_ylin_pcp database...
> > >> - It includes 4 lines where FV3GFS is blue and GFS is red.
> > >> - The solid lines are for the EAST region and dashed lines
are for
> > the
> > >> WEST region. Take a close look at the Series definition
section. I
> > >> defined the EAST as a "Group" of 6 subregions and the WEST as a
> "Group"
> > of
> > >> 8 subregions.
> > >> - The gray bars show the base rate for the EAST and WEST
regions.
> > >> Since
> > >> I only want to plot 2 of the 4, I chose "Yes" for the "Hide"
column
> for
> > >> the
> > >> 2 lines I wanted to ignore.
> > >> - In the "Series Formatting" I used the "Legend Text" column
to
> > define
> > >> the legend.
> > >>
> > >> Granted, this plot was pretty tedious to make manually because
of the
> > >> masking region groups, but hopefully it includes all the info
you
> need.
> > >>
> > >> And note that statistical significance is not currently
included in
> this
> > >> plot, but it could be added.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >>
> > >>
> > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT
<met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
> > >>> Transaction: Ticket created by ying.lin at noaa.gov
> > >>> Queue: met_help
> > >>> Subject: MV CTC plot: count of data points for each
threshold
> > (Fwd:
> > >>> FBIAS east vs. west US
> > >>> Owner: Nobody
> > >>> Requestors: ying.lin at noaa.gov
> > >>> Status: new
> > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=86995
> > >
> > >>>
> > >>>
> > >>> Hi Tracey,
> > >>>
> > >>> I haven't tried showing stats for each threshold, you're
way
> > ahead
> > >>> of me. Let me forward your inquiry to met_help.
> > >>>
> > >>> Dear met_help:
> > >>>
> > >>> I think Tracey is referring to the numbers such as that
on top
> of
> > >>> this plot:
> > >>>
> > >>> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> > >>> 01809/intl_reg.201809.gif,
> > >>>
> > >>> for each verifying threshold.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Ying
> > >>>
> > >>>
> > >>> -------- Forwarded Message --------
> > >>> Subject: FBIAS east vs. west US
> > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> > >>> From: Tracey Dorian - NOAA Affiliate
<tracey.dorian at noaa.gov>
> > >>> To: Ying Lin <ying.lin at noaa.gov>
> > >>>
> > >>>
> > >>>
> > >>> Hi Ying,
> > >>>
> > >>> I've created plots of FBIAS for east regions vs. west regions
and
> Geoff
> > >>> was wondering if I can show the number of events for each
> threshold? I
> > >>> know METViewer has the "display number of stats" option which
I
> checked
> > >>> off, but it seems to show the same number for each threshold.
Do you
> > >>> know if there is a way to show the different number of stats
for each
> > >>> threshold? We would probably expect the number of events to
be high
> > for
> > >>> the lower thresholds and get progressively smaller as you move
into
> the
> > >>> higher thresholds, right?
> > >>>
> > >>> Thanks,
> > >>> Tracey
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > > --
> > > Ying Lin
> > > NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> > > NCWCP Cubicle No. 2015
> > > Ying.Lin at noaa.gov
> > >
> > >
> > >
> >
> >
> > --
> > Tracey Dorian
> > NOAA/NCEP/EMC
> > IMSG contractor
> > 5830 University Research Court
> > College Park, MD 20740
> >
> >
>
>
--
Tracey Dorian
NOAA/NCEP/EMC
IMSG contractor
5830 University Research Court
College Park, MD 20740
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: John Halley Gotway
Time: Mon Sep 17 15:00:44 2018
Tracey,
I played around a bit. See attached. Here's the changes I made...
- Instead of plotting base rate on the "Y1" axis, I selected the "Y2"
tab
and defined it here.
- In the bottom-left corner of the GUI, I selected "+Add Derived
Curve" and
defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24 FBIAS".
- In the "Statistics" box, I selected "Aggregation Statistics" radio
button, "Contingency Table Count (CTC)" from the drop-down list, and
then
set "Bootstrapping replications" to 1000. (I used 100 for testing
since
it's much faster!)
- In the "Series Formatting" section, I set "GFS APCP/24 BASER" to
light
gray, "Series Line Type" = "histogram-like" and "Line Width" = 30.
- In the "Series Formatting" section for the "DIFF" line, I selected
"Show
Significance" to true.
Anywhere the confidence interval of the difference line is bold,
that
means the difference is statistically significant.
- In the "Legend Text" column I added some meaningful labels.
- In the "Formatting" tab I set "plot script commands" as
"abline(h=c(0,1),col="gray",lty=2,lwd=2)"
That'll draw horizontal gray dashed reference lines at 0 and 1.
Remember that this is currently only plotting results for "G218/SEC".
Be
sure to change that to the full set of regions you want to use.
John
On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>
> Hi John,
>
> Yes, that's what I was looking for! How were you able to plot the
base
> rate using bars instead of lines? Also, how would you suggest
adding
> statistical significance to the plot?
>
> Thanks for sending your XML file!
>
> -Tracey
>
> On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Tracey,
> >
> > Thanks for sending your XML file. I didn't see any reference to
"baser"
> in
> > the XML you sent. So I added it as a second statistic to be
plotted.
> >
> > Since you didn't define a VX_MASK to be used, METviewer tried to
use all
> > available in the database, which likely made it run slowly.
> >
> > For testing, I set VX_MASK = G218/SEC added a line for BASER and
attached
> > the result.
> >
> > Is that what you're looking for?
> >
> > Thanks,
> > John
> >
> > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> > >
> > > Hi John,
> > >
> > > Thanks for much for your quick reply! I am trying to add the
BASER
> field
> > > to my plots, but I'm getting errors, specifically a proxy
timeout. Do
> > you
> > > know why this is happening? I've attached my XML file to this
email.
> > >
> > > Thanks,
> > > Tracey
> > >
> > >
> > >
> > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin <ying.lin at noaa.gov>
wrote:
> > >
> > > > Thanks very much John!
> > > >
> > > > Tracey, did this answer your question? I have an RFC/Dataflow
crisis
> > > > today so haven't tested the answer. If you have further
questions,
> > > please
> > > > follow-up on this met_help ticket (and cc to me - sure would
like to
> > know
> > > > too). Thanks.
> > > >
> > > > Ying
> > > >
> > > >
> > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
> > > >
> > > >> Ying and Tracey,
> > > >>
> > > >> I see that you'd like to include in your METviewer plot
information
> > > about
> > > >> the event frequency. You tried using the "Display Number of
Stats"
> > > option
> > > >> but that that didn't do exactly what you want. Those numbers
stay
> > > >> constant
> > > >> for all the thresholds.
> > > >>
> > > >> The "NStats" values listed across the top indicate the sample
size.
> > > They
> > > >> are computed as the sum of the number of cases that are
included
> > across
> > > >> all
> > > >> the series (i.e. lines on the plot) for each independent
variable
> > value.
> > > >> This is not the number of events meeting each threshold
criteria...
> > > it's a
> > > >> count of the number of "cases" included. In the sample
plots you
> > > sent,
> > > >> you have NStats = 33948. Since you didn't send the
corresponding
> XML,
> > > >> it's
> > > >> difficult for me to exactly replicate them. Basically
though, the
> > logic
> > > >> is:
> > > >> - For a particular independent variable value, >1.0" for
example,
> > you
> > > >> have 2 lines on the plot.
> > > >> - You have many model initializations included, perhaps
397.
> > > >> - You have multiple lead times included. For APCP/24
probably
> 24,
> > > 48,
> > > >> 72, 96... perhaps about 8 of them.
> > > >> - You're including multiple subregions in each plot... for
> example,
> > > the
> > > >> EAST consists of 6 subregions.
> > > >>
> > > >> Multiplying all those numbers together... 2 * 397 * 8 * 6 =
38112
> gets
> > > you
> > > >> somewhere near the actual number of 33948 we're seeing.
> > > >>
> > > >> If you want to see the frequency at which the event is
actually
> > > occurring,
> > > >> I'd suggest plotting the base rate statistic, called BASER in
> > METviewer.
> > > >> That's exactly what you want... a number between 0 and 1
> representing
> > > the
> > > >> event frequency in the observation data. The BASER could be
shown
> > > >> separately for the GFS and FV3GFS... but as long as you're
using the
> > > exact
> > > >> same cases, they should be identical. We often plot the
BASER on
> the
> > Y2
> > > >> axis so that we can control the range of the Y2 axis, but I
ran into
> > > some
> > > >> problems doing that.
> > > >>
> > > >> Instead, I put it on the Y1 axis with FBIAS but plotted it
like a
> > > >> histogram
> > > >> with bars.
> > > >>
> > > >> I made the attached plot/xml using the mv_ylin_pcp
database...
> > > >> - It includes 4 lines where FV3GFS is blue and GFS is red.
> > > >> - The solid lines are for the EAST region and dashed lines
are
> for
> > > the
> > > >> WEST region. Take a close look at the Series definition
section. I
> > > >> defined the EAST as a "Group" of 6 subregions and the WEST as
a
> > "Group"
> > > of
> > > >> 8 subregions.
> > > >> - The gray bars show the base rate for the EAST and WEST
regions.
> > > >> Since
> > > >> I only want to plot 2 of the 4, I chose "Yes" for the "Hide"
column
> > for
> > > >> the
> > > >> 2 lines I wanted to ignore.
> > > >> - In the "Series Formatting" I used the "Legend Text"
column to
> > > define
> > > >> the legend.
> > > >>
> > > >> Granted, this plot was pretty tedious to make manually
because of
> the
> > > >> masking region groups, but hopefully it includes all the info
you
> > need.
> > > >>
> > > >> And note that statistical significance is not currently
included in
> > this
> > > >> plot, but it could be added.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >>
> > > >>
> > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT
<met_help at ucar.edu>
> > > >> wrote:
> > > >>
> > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
> > > >>> Transaction: Ticket created by ying.lin at noaa.gov
> > > >>> Queue: met_help
> > > >>> Subject: MV CTC plot: count of data points for each
threshold
> > > (Fwd:
> > > >>> FBIAS east vs. west US
> > > >>> Owner: Nobody
> > > >>> Requestors: ying.lin at noaa.gov
> > > >>> Status: new
> > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=86995
> > > >
> > > >>>
> > > >>>
> > > >>> Hi Tracey,
> > > >>>
> > > >>> I haven't tried showing stats for each threshold,
you're way
> > > ahead
> > > >>> of me. Let me forward your inquiry to met_help.
> > > >>>
> > > >>> Dear met_help:
> > > >>>
> > > >>> I think Tracey is referring to the numbers such as
that on
> top
> > of
> > > >>> this plot:
> > > >>>
> > > >>> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> > > >>> 01809/intl_reg.201809.gif,
> > > >>>
> > > >>> for each verifying threshold.
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Ying
> > > >>>
> > > >>>
> > > >>> -------- Forwarded Message --------
> > > >>> Subject: FBIAS east vs. west US
> > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> > > >>> From: Tracey Dorian - NOAA Affiliate
<tracey.dorian at noaa.gov>
> > > >>> To: Ying Lin <ying.lin at noaa.gov>
> > > >>>
> > > >>>
> > > >>>
> > > >>> Hi Ying,
> > > >>>
> > > >>> I've created plots of FBIAS for east regions vs. west
regions and
> > Geoff
> > > >>> was wondering if I can show the number of events for each
> > threshold? I
> > > >>> know METViewer has the "display number of stats" option
which I
> > checked
> > > >>> off, but it seems to show the same number for each
threshold. Do
> you
> > > >>> know if there is a way to show the different number of stats
for
> each
> > > >>> threshold? We would probably expect the number of events to
be
> high
> > > for
> > > >>> the lower thresholds and get progressively smaller as you
move into
> > the
> > > >>> higher thresholds, right?
> > > >>>
> > > >>> Thanks,
> > > >>> Tracey
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > > --
> > > > Ying Lin
> > > > NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> > > > NCWCP Cubicle No. 2015
> > > > Ying.Lin at noaa.gov
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Tracey Dorian
> > > NOAA/NCEP/EMC
> > > IMSG contractor
> > > 5830 University Research Court
> > > College Park, MD 20740
> > >
> > >
> >
> >
>
>
> --
> Tracey Dorian
> NOAA/NCEP/EMC
> IMSG contractor
> 5830 University Research Court
> College Park, MD 20740
>
>
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: tracey.dorian at noaa.gov
Time: Mon Sep 17 18:57:08 2018
Hi John,
Thanks very much for your help and for your detailed explanation!
Sincerely,
Tracey
On Mon, Sep 17, 2018 at 5:00 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Tracey,
>
> I played around a bit. See attached. Here's the changes I made...
>
> - Instead of plotting base rate on the "Y1" axis, I selected the
"Y2" tab
> and defined it here.
> - In the bottom-left corner of the GUI, I selected "+Add Derived
Curve" and
> defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24 FBIAS".
> - In the "Statistics" box, I selected "Aggregation Statistics" radio
> button, "Contingency Table Count (CTC)" from the drop-down list, and
then
> set "Bootstrapping replications" to 1000. (I used 100 for testing
since
> it's much faster!)
> - In the "Series Formatting" section, I set "GFS APCP/24 BASER" to
light
> gray, "Series Line Type" = "histogram-like" and "Line Width" = 30.
> - In the "Series Formatting" section for the "DIFF" line, I selected
"Show
> Significance" to true.
> Anywhere the confidence interval of the difference line is bold,
that
> means the difference is statistically significant.
> - In the "Legend Text" column I added some meaningful labels.
> - In the "Formatting" tab I set "plot script commands" as
> "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
> That'll draw horizontal gray dashed reference lines at 0 and 1.
>
> Remember that this is currently only plotting results for
"G218/SEC". Be
> sure to change that to the full set of regions you want to use.
>
> John
>
> On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> >
> > Hi John,
> >
> > Yes, that's what I was looking for! How were you able to plot the
base
> > rate using bars instead of lines? Also, how would you suggest
adding
> > statistical significance to the plot?
> >
> > Thanks for sending your XML file!
> >
> > -Tracey
> >
> > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Tracey,
> > >
> > > Thanks for sending your XML file. I didn't see any reference to
> "baser"
> > in
> > > the XML you sent. So I added it as a second statistic to be
plotted.
> > >
> > > Since you didn't define a VX_MASK to be used, METviewer tried to
use
> all
> > > available in the database, which likely made it run slowly.
> > >
> > > For testing, I set VX_MASK = G218/SEC added a line for BASER and
> attached
> > > the result.
> > >
> > > Is that what you're looking for?
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
>
> > > >
> > > > Hi John,
> > > >
> > > > Thanks for much for your quick reply! I am trying to add the
BASER
> > field
> > > > to my plots, but I'm getting errors, specifically a proxy
timeout.
> Do
> > > you
> > > > know why this is happening? I've attached my XML file to this
email.
> > > >
> > > > Thanks,
> > > > Tracey
> > > >
> > > >
> > > >
> > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin <ying.lin at noaa.gov>
wrote:
> > > >
> > > > > Thanks very much John!
> > > > >
> > > > > Tracey, did this answer your question? I have an
RFC/Dataflow
> crisis
> > > > > today so haven't tested the answer. If you have further
questions,
> > > > please
> > > > > follow-up on this met_help ticket (and cc to me - sure would
like
> to
> > > know
> > > > > too). Thanks.
> > > > >
> > > > > Ying
> > > > >
> > > > >
> > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
> > > > >
> > > > >> Ying and Tracey,
> > > > >>
> > > > >> I see that you'd like to include in your METviewer plot
> information
> > > > about
> > > > >> the event frequency. You tried using the "Display Number
of
> Stats"
> > > > option
> > > > >> but that that didn't do exactly what you want. Those
numbers stay
> > > > >> constant
> > > > >> for all the thresholds.
> > > > >>
> > > > >> The "NStats" values listed across the top indicate the
sample
> size.
> > > > They
> > > > >> are computed as the sum of the number of cases that are
included
> > > across
> > > > >> all
> > > > >> the series (i.e. lines on the plot) for each independent
variable
> > > value.
> > > > >> This is not the number of events meeting each threshold
> criteria...
> > > > it's a
> > > > >> count of the number of "cases" included. In the sample
plots
> you
> > > > sent,
> > > > >> you have NStats = 33948. Since you didn't send the
corresponding
> > XML,
> > > > >> it's
> > > > >> difficult for me to exactly replicate them. Basically
though, the
> > > logic
> > > > >> is:
> > > > >> - For a particular independent variable value, >1.0" for
> example,
> > > you
> > > > >> have 2 lines on the plot.
> > > > >> - You have many model initializations included, perhaps
397.
> > > > >> - You have multiple lead times included. For APCP/24
probably
> > 24,
> > > > 48,
> > > > >> 72, 96... perhaps about 8 of them.
> > > > >> - You're including multiple subregions in each plot...
for
> > example,
> > > > the
> > > > >> EAST consists of 6 subregions.
> > > > >>
> > > > >> Multiplying all those numbers together... 2 * 397 * 8 * 6 =
38112
> > gets
> > > > you
> > > > >> somewhere near the actual number of 33948 we're seeing.
> > > > >>
> > > > >> If you want to see the frequency at which the event is
actually
> > > > occurring,
> > > > >> I'd suggest plotting the base rate statistic, called BASER
in
> > > METviewer.
> > > > >> That's exactly what you want... a number between 0 and 1
> > representing
> > > > the
> > > > >> event frequency in the observation data. The BASER could
be shown
> > > > >> separately for the GFS and FV3GFS... but as long as you're
using
> the
> > > > exact
> > > > >> same cases, they should be identical. We often plot the
BASER on
> > the
> > > Y2
> > > > >> axis so that we can control the range of the Y2 axis, but I
ran
> into
> > > > some
> > > > >> problems doing that.
> > > > >>
> > > > >> Instead, I put it on the Y1 axis with FBIAS but plotted it
like a
> > > > >> histogram
> > > > >> with bars.
> > > > >>
> > > > >> I made the attached plot/xml using the mv_ylin_pcp
database...
> > > > >> - It includes 4 lines where FV3GFS is blue and GFS is
red.
> > > > >> - The solid lines are for the EAST region and dashed
lines are
> > for
> > > > the
> > > > >> WEST region. Take a close look at the Series definition
> section. I
> > > > >> defined the EAST as a "Group" of 6 subregions and the WEST
as a
> > > "Group"
> > > > of
> > > > >> 8 subregions.
> > > > >> - The gray bars show the base rate for the EAST and WEST
> regions.
> > > > >> Since
> > > > >> I only want to plot 2 of the 4, I chose "Yes" for the
"Hide"
> column
> > > for
> > > > >> the
> > > > >> 2 lines I wanted to ignore.
> > > > >> - In the "Series Formatting" I used the "Legend Text"
column to
> > > > define
> > > > >> the legend.
> > > > >>
> > > > >> Granted, this plot was pretty tedious to make manually
because of
> > the
> > > > >> masking region groups, but hopefully it includes all the
info you
> > > need.
> > > > >>
> > > > >> And note that statistical significance is not currently
included
> in
> > > this
> > > > >> plot, but it could be added.
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <
> met_help at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
> > > > >>> Transaction: Ticket created by ying.lin at noaa.gov
> > > > >>> Queue: met_help
> > > > >>> Subject: MV CTC plot: count of data points for each
> threshold
> > > > (Fwd:
> > > > >>> FBIAS east vs. west US
> > > > >>> Owner: Nobody
> > > > >>> Requestors: ying.lin at noaa.gov
> > > > >>> Status: new
> > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=86995
> > > > >
> > > > >>>
> > > > >>>
> > > > >>> Hi Tracey,
> > > > >>>
> > > > >>> I haven't tried showing stats for each threshold,
you're
> way
> > > > ahead
> > > > >>> of me. Let me forward your inquiry to met_help.
> > > > >>>
> > > > >>> Dear met_help:
> > > > >>>
> > > > >>> I think Tracey is referring to the numbers such as
that on
> > top
> > > of
> > > > >>> this plot:
> > > > >>>
> > > > >>>
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> > > > >>> 01809/intl_reg.201809.gif,
> > > > >>>
> > > > >>> for each verifying threshold.
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> Ying
> > > > >>>
> > > > >>>
> > > > >>> -------- Forwarded Message --------
> > > > >>> Subject: FBIAS east vs. west US
> > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> > > > >>> From: Tracey Dorian - NOAA Affiliate
<tracey.dorian at noaa.gov>
> > > > >>> To: Ying Lin <ying.lin at noaa.gov>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> Hi Ying,
> > > > >>>
> > > > >>> I've created plots of FBIAS for east regions vs. west
regions and
> > > Geoff
> > > > >>> was wondering if I can show the number of events for each
> > > threshold? I
> > > > >>> know METViewer has the "display number of stats" option
which I
> > > checked
> > > > >>> off, but it seems to show the same number for each
threshold. Do
> > you
> > > > >>> know if there is a way to show the different number of
stats for
> > each
> > > > >>> threshold? We would probably expect the number of events
to be
> > high
> > > > for
> > > > >>> the lower thresholds and get progressively smaller as you
move
> into
> > > the
> > > > >>> higher thresholds, right?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Tracey
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > > --
> > > > > Ying Lin
> > > > > NCEP/EMC/Verification, Post-processing and Product
Generation
> Branch
> > > > > NCWCP Cubicle No. 2015
> > > > > Ying.Lin at noaa.gov
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Tracey Dorian
> > > > NOAA/NCEP/EMC
> > > > IMSG contractor
> > > > 5830 University Research Court
> > > > College Park, MD 20740
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Tracey Dorian
> > NOAA/NCEP/EMC
> > IMSG contractor
> > 5830 University Research Court
> > College Park, MD 20740
> >
> >
>
>
--
Tracey Dorian
NOAA/NCEP/EMC
IMSG contractor
5830 University Research Court
College Park, MD 20740
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: tracey.dorian at noaa.gov
Time: Mon Sep 17 19:22:44 2018
Hi John,
One last question for you - just to make sure, is there no way to plot
what
you plotted but with several regions at once? I am still getting the
METViewer session timeout error when I try several regions (e.g. east
vs.
west).
Thanks,
Tracey
On Mon, Sep 17, 2018 at 8:56 PM, Tracey Dorian - NOAA Affiliate <
tracey.dorian at noaa.gov> wrote:
> Hi John,
>
> Thanks very much for your help and for your detailed explanation!
>
> Sincerely,
> Tracey
>
> On Mon, Sep 17, 2018 at 5:00 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Tracey,
>>
>> I played around a bit. See attached. Here's the changes I made...
>>
>> - Instead of plotting base rate on the "Y1" axis, I selected the
"Y2" tab
>> and defined it here.
>> - In the bottom-left corner of the GUI, I selected "+Add Derived
Curve"
>> and
>> defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24 FBIAS".
>> - In the "Statistics" box, I selected "Aggregation Statistics"
radio
>> button, "Contingency Table Count (CTC)" from the drop-down list,
and then
>> set "Bootstrapping replications" to 1000. (I used 100 for testing
since
>> it's much faster!)
>> - In the "Series Formatting" section, I set "GFS APCP/24 BASER" to
light
>> gray, "Series Line Type" = "histogram-like" and "Line Width" = 30.
>> - In the "Series Formatting" section for the "DIFF" line, I
selected "Show
>> Significance" to true.
>> Anywhere the confidence interval of the difference line is bold,
that
>> means the difference is statistically significant.
>> - In the "Legend Text" column I added some meaningful labels.
>> - In the "Formatting" tab I set "plot script commands" as
>> "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
>> That'll draw horizontal gray dashed reference lines at 0 and 1.
>>
>> Remember that this is currently only plotting results for
"G218/SEC". Be
>> sure to change that to the full set of regions you want to use.
>>
>> John
>>
>> On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>> >
>> > Hi John,
>> >
>> > Yes, that's what I was looking for! How were you able to plot
the base
>> > rate using bars instead of lines? Also, how would you suggest
adding
>> > statistical significance to the plot?
>> >
>> > Thanks for sending your XML file!
>> >
>> > -Tracey
>> >
>> > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > > Tracey,
>> > >
>> > > Thanks for sending your XML file. I didn't see any reference
to
>> "baser"
>> > in
>> > > the XML you sent. So I added it as a second statistic to be
plotted.
>> > >
>> > > Since you didn't define a VX_MASK to be used, METviewer tried
to use
>> all
>> > > available in the database, which likely made it run slowly.
>> > >
>> > > For testing, I set VX_MASK = G218/SEC added a line for BASER
and
>> attached
>> > > the result.
>> > >
>> > > Is that what you're looking for?
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
>
>> > > >
>> > > > Hi John,
>> > > >
>> > > > Thanks for much for your quick reply! I am trying to add the
BASER
>> > field
>> > > > to my plots, but I'm getting errors, specifically a proxy
timeout.
>> Do
>> > > you
>> > > > know why this is happening? I've attached my XML file to
this
>> email.
>> > > >
>> > > > Thanks,
>> > > > Tracey
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin <ying.lin at noaa.gov>
>> wrote:
>> > > >
>> > > > > Thanks very much John!
>> > > > >
>> > > > > Tracey, did this answer your question? I have an
RFC/Dataflow
>> crisis
>> > > > > today so haven't tested the answer. If you have further
>> questions,
>> > > > please
>> > > > > follow-up on this met_help ticket (and cc to me - sure
would like
>> to
>> > > know
>> > > > > too). Thanks.
>> > > > >
>> > > > > Ying
>> > > > >
>> > > > >
>> > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
>> > > > >
>> > > > >> Ying and Tracey,
>> > > > >>
>> > > > >> I see that you'd like to include in your METviewer plot
>> information
>> > > > about
>> > > > >> the event frequency. You tried using the "Display Number
of
>> Stats"
>> > > > option
>> > > > >> but that that didn't do exactly what you want. Those
numbers
>> stay
>> > > > >> constant
>> > > > >> for all the thresholds.
>> > > > >>
>> > > > >> The "NStats" values listed across the top indicate the
sample
>> size.
>> > > > They
>> > > > >> are computed as the sum of the number of cases that are
included
>> > > across
>> > > > >> all
>> > > > >> the series (i.e. lines on the plot) for each independent
variable
>> > > value.
>> > > > >> This is not the number of events meeting each threshold
>> criteria...
>> > > > it's a
>> > > > >> count of the number of "cases" included. In the sample
plots
>> you
>> > > > sent,
>> > > > >> you have NStats = 33948. Since you didn't send the
corresponding
>> > XML,
>> > > > >> it's
>> > > > >> difficult for me to exactly replicate them. Basically
though,
>> the
>> > > logic
>> > > > >> is:
>> > > > >> - For a particular independent variable value, >1.0"
for
>> example,
>> > > you
>> > > > >> have 2 lines on the plot.
>> > > > >> - You have many model initializations included, perhaps
397.
>> > > > >> - You have multiple lead times included. For APCP/24
probably
>> > 24,
>> > > > 48,
>> > > > >> 72, 96... perhaps about 8 of them.
>> > > > >> - You're including multiple subregions in each plot...
for
>> > example,
>> > > > the
>> > > > >> EAST consists of 6 subregions.
>> > > > >>
>> > > > >> Multiplying all those numbers together... 2 * 397 * 8 * 6
= 38112
>> > gets
>> > > > you
>> > > > >> somewhere near the actual number of 33948 we're seeing.
>> > > > >>
>> > > > >> If you want to see the frequency at which the event is
actually
>> > > > occurring,
>> > > > >> I'd suggest plotting the base rate statistic, called BASER
in
>> > > METviewer.
>> > > > >> That's exactly what you want... a number between 0 and 1
>> > representing
>> > > > the
>> > > > >> event frequency in the observation data. The BASER could
be
>> shown
>> > > > >> separately for the GFS and FV3GFS... but as long as you're
using
>> the
>> > > > exact
>> > > > >> same cases, they should be identical. We often plot the
BASER on
>> > the
>> > > Y2
>> > > > >> axis so that we can control the range of the Y2 axis, but
I ran
>> into
>> > > > some
>> > > > >> problems doing that.
>> > > > >>
>> > > > >> Instead, I put it on the Y1 axis with FBIAS but plotted it
like a
>> > > > >> histogram
>> > > > >> with bars.
>> > > > >>
>> > > > >> I made the attached plot/xml using the mv_ylin_pcp
database...
>> > > > >> - It includes 4 lines where FV3GFS is blue and GFS is
red.
>> > > > >> - The solid lines are for the EAST region and dashed
lines are
>> > for
>> > > > the
>> > > > >> WEST region. Take a close look at the Series definition
>> section. I
>> > > > >> defined the EAST as a "Group" of 6 subregions and the WEST
as a
>> > > "Group"
>> > > > of
>> > > > >> 8 subregions.
>> > > > >> - The gray bars show the base rate for the EAST and
WEST
>> regions.
>> > > > >> Since
>> > > > >> I only want to plot 2 of the 4, I chose "Yes" for the
"Hide"
>> column
>> > > for
>> > > > >> the
>> > > > >> 2 lines I wanted to ignore.
>> > > > >> - In the "Series Formatting" I used the "Legend Text"
column
>> to
>> > > > define
>> > > > >> the legend.
>> > > > >>
>> > > > >> Granted, this plot was pretty tedious to make manually
because of
>> > the
>> > > > >> masking region groups, but hopefully it includes all the
info you
>> > > need.
>> > > > >>
>> > > > >> And note that statistical significance is not currently
included
>> in
>> > > this
>> > > > >> plot, but it could be added.
>> > > > >>
>> > > > >> Thanks,
>> > > > >> John
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <
>> met_help at ucar.edu>
>> > > > >> wrote:
>> > > > >>
>> > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
>> > > > >>> Transaction: Ticket created by ying.lin at noaa.gov
>> > > > >>> Queue: met_help
>> > > > >>> Subject: MV CTC plot: count of data points for each
>> threshold
>> > > > (Fwd:
>> > > > >>> FBIAS east vs. west US
>> > > > >>> Owner: Nobody
>> > > > >>> Requestors: ying.lin at noaa.gov
>> > > > >>> Status: new
>> > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
>> > > Ticket/Display.html?id=86995
>> > > > >
>> > > > >>>
>> > > > >>>
>> > > > >>> Hi Tracey,
>> > > > >>>
>> > > > >>> I haven't tried showing stats for each threshold,
you're
>> way
>> > > > ahead
>> > > > >>> of me. Let me forward your inquiry to met_help.
>> > > > >>>
>> > > > >>> Dear met_help:
>> > > > >>>
>> > > > >>> I think Tracey is referring to the numbers such as
that on
>> > top
>> > > of
>> > > > >>> this plot:
>> > > > >>>
>> > > > >>>
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
>> > > > >>> 01809/intl_reg.201809.gif,
>> > > > >>>
>> > > > >>> for each verifying threshold.
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>>
>> > > > >>> Ying
>> > > > >>>
>> > > > >>>
>> > > > >>> -------- Forwarded Message --------
>> > > > >>> Subject: FBIAS east vs. west US
>> > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
>> > > > >>> From: Tracey Dorian - NOAA Affiliate
<tracey.dorian at noaa.gov>
>> > > > >>> To: Ying Lin <ying.lin at noaa.gov>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> Hi Ying,
>> > > > >>>
>> > > > >>> I've created plots of FBIAS for east regions vs. west
regions
>> and
>> > > Geoff
>> > > > >>> was wondering if I can show the number of events for each
>> > > threshold? I
>> > > > >>> know METViewer has the "display number of stats" option
which I
>> > > checked
>> > > > >>> off, but it seems to show the same number for each
threshold.
>> Do
>> > you
>> > > > >>> know if there is a way to show the different number of
stats for
>> > each
>> > > > >>> threshold? We would probably expect the number of events
to be
>> > high
>> > > > for
>> > > > >>> the lower thresholds and get progressively smaller as you
move
>> into
>> > > the
>> > > > >>> higher thresholds, right?
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Tracey
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > > --
>> > > > > Ying Lin
>> > > > > NCEP/EMC/Verification, Post-processing and Product
Generation
>> Branch
>> > > > > NCWCP Cubicle No. 2015
>> > > > > Ying.Lin at noaa.gov
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Tracey Dorian
>> > > > NOAA/NCEP/EMC
>> > > > IMSG contractor
>> > > > 5830 University Research Court
>> > > > College Park, MD 20740
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>> > --
>> > Tracey Dorian
>> > NOAA/NCEP/EMC
>> > IMSG contractor
>> > 5830 University Research Court
>> > College Park, MD 20740
>> >
>> >
>>
>>
>
>
> --
> Tracey Dorian
> NOAA/NCEP/EMC
> IMSG contractor
> 5830 University Research Court
> College Park, MD 20740
>
--
Tracey Dorian
NOAA/NCEP/EMC
IMSG contractor
5830 University Research Court
College Park, MD 20740
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: John Halley Gotway
Time: Tue Sep 18 09:24:48 2018
Tracey,
Plotting many regions *should* work just fine. The timeout is caused
by
the mysql query taking too long to process. I suspect that if we were
running METviewer on more robust hardware, we'd run into this sort of
issue
less often.
For testing, please try making your plot exactly how you'd like it,
but
restrict the dates to 1 week. Does that work?
John
On Mon, Sep 17, 2018 at 7:22 PM tracey.dorian at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>
> Hi John,
>
> One last question for you - just to make sure, is there no way to
plot what
> you plotted but with several regions at once? I am still getting
the
> METViewer session timeout error when I try several regions (e.g.
east vs.
> west).
>
> Thanks,
> Tracey
>
> On Mon, Sep 17, 2018 at 8:56 PM, Tracey Dorian - NOAA Affiliate <
> tracey.dorian at noaa.gov> wrote:
>
> > Hi John,
> >
> > Thanks very much for your help and for your detailed explanation!
> >
> > Sincerely,
> > Tracey
> >
> > On Mon, Sep 17, 2018 at 5:00 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Tracey,
> >>
> >> I played around a bit. See attached. Here's the changes I
made...
> >>
> >> - Instead of plotting base rate on the "Y1" axis, I selected the
"Y2"
> tab
> >> and defined it here.
> >> - In the bottom-left corner of the GUI, I selected "+Add Derived
Curve"
> >> and
> >> defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24 FBIAS".
> >> - In the "Statistics" box, I selected "Aggregation Statistics"
radio
> >> button, "Contingency Table Count (CTC)" from the drop-down list,
and
> then
> >> set "Bootstrapping replications" to 1000. (I used 100 for testing
since
> >> it's much faster!)
> >> - In the "Series Formatting" section, I set "GFS APCP/24 BASER"
to light
> >> gray, "Series Line Type" = "histogram-like" and "Line Width" =
30.
> >> - In the "Series Formatting" section for the "DIFF" line, I
selected
> "Show
> >> Significance" to true.
> >> Anywhere the confidence interval of the difference line is
bold, that
> >> means the difference is statistically significant.
> >> - In the "Legend Text" column I added some meaningful labels.
> >> - In the "Formatting" tab I set "plot script commands" as
> >> "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
> >> That'll draw horizontal gray dashed reference lines at 0 and
1.
> >>
> >> Remember that this is currently only plotting results for
"G218/SEC".
> Be
> >> sure to change that to the full set of regions you want to use.
> >>
> >> John
> >>
> >> On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> >> >
> >> > Hi John,
> >> >
> >> > Yes, that's what I was looking for! How were you able to plot
the
> base
> >> > rate using bars instead of lines? Also, how would you suggest
adding
> >> > statistical significance to the plot?
> >> >
> >> > Thanks for sending your XML file!
> >> >
> >> > -Tracey
> >> >
> >> > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > > Tracey,
> >> > >
> >> > > Thanks for sending your XML file. I didn't see any reference
to
> >> "baser"
> >> > in
> >> > > the XML you sent. So I added it as a second statistic to be
> plotted.
> >> > >
> >> > > Since you didn't define a VX_MASK to be used, METviewer tried
to use
> >> all
> >> > > available in the database, which likely made it run slowly.
> >> > >
> >> > > For testing, I set VX_MASK = G218/SEC added a line for BASER
and
> >> attached
> >> > > the result.
> >> > >
> >> > > Is that what you're looking for?
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov via RT
<
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> >> > > >
> >> > > > Hi John,
> >> > > >
> >> > > > Thanks for much for your quick reply! I am trying to add
the
> BASER
> >> > field
> >> > > > to my plots, but I'm getting errors, specifically a proxy
timeout.
> >> Do
> >> > > you
> >> > > > know why this is happening? I've attached my XML file to
this
> >> email.
> >> > > >
> >> > > > Thanks,
> >> > > > Tracey
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin
<ying.lin at noaa.gov>
> >> wrote:
> >> > > >
> >> > > > > Thanks very much John!
> >> > > > >
> >> > > > > Tracey, did this answer your question? I have an
RFC/Dataflow
> >> crisis
> >> > > > > today so haven't tested the answer. If you have further
> >> questions,
> >> > > > please
> >> > > > > follow-up on this met_help ticket (and cc to me - sure
would
> like
> >> to
> >> > > know
> >> > > > > too). Thanks.
> >> > > > >
> >> > > > > Ying
> >> > > > >
> >> > > > >
> >> > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
> >> > > > >
> >> > > > >> Ying and Tracey,
> >> > > > >>
> >> > > > >> I see that you'd like to include in your METviewer plot
> >> information
> >> > > > about
> >> > > > >> the event frequency. You tried using the "Display
Number of
> >> Stats"
> >> > > > option
> >> > > > >> but that that didn't do exactly what you want. Those
numbers
> >> stay
> >> > > > >> constant
> >> > > > >> for all the thresholds.
> >> > > > >>
> >> > > > >> The "NStats" values listed across the top indicate the
sample
> >> size.
> >> > > > They
> >> > > > >> are computed as the sum of the number of cases that are
> included
> >> > > across
> >> > > > >> all
> >> > > > >> the series (i.e. lines on the plot) for each independent
> variable
> >> > > value.
> >> > > > >> This is not the number of events meeting each threshold
> >> criteria...
> >> > > > it's a
> >> > > > >> count of the number of "cases" included. In the
sample plots
> >> you
> >> > > > sent,
> >> > > > >> you have NStats = 33948. Since you didn't send the
> corresponding
> >> > XML,
> >> > > > >> it's
> >> > > > >> difficult for me to exactly replicate them. Basically
though,
> >> the
> >> > > logic
> >> > > > >> is:
> >> > > > >> - For a particular independent variable value, >1.0"
for
> >> example,
> >> > > you
> >> > > > >> have 2 lines on the plot.
> >> > > > >> - You have many model initializations included,
perhaps 397.
> >> > > > >> - You have multiple lead times included. For APCP/24
> probably
> >> > 24,
> >> > > > 48,
> >> > > > >> 72, 96... perhaps about 8 of them.
> >> > > > >> - You're including multiple subregions in each
plot... for
> >> > example,
> >> > > > the
> >> > > > >> EAST consists of 6 subregions.
> >> > > > >>
> >> > > > >> Multiplying all those numbers together... 2 * 397 * 8 *
6 =
> 38112
> >> > gets
> >> > > > you
> >> > > > >> somewhere near the actual number of 33948 we're seeing.
> >> > > > >>
> >> > > > >> If you want to see the frequency at which the event is
actually
> >> > > > occurring,
> >> > > > >> I'd suggest plotting the base rate statistic, called
BASER in
> >> > > METviewer.
> >> > > > >> That's exactly what you want... a number between 0 and 1
> >> > representing
> >> > > > the
> >> > > > >> event frequency in the observation data. The BASER
could be
> >> shown
> >> > > > >> separately for the GFS and FV3GFS... but as long as
you're
> using
> >> the
> >> > > > exact
> >> > > > >> same cases, they should be identical. We often plot the
BASER
> on
> >> > the
> >> > > Y2
> >> > > > >> axis so that we can control the range of the Y2 axis,
but I ran
> >> into
> >> > > > some
> >> > > > >> problems doing that.
> >> > > > >>
> >> > > > >> Instead, I put it on the Y1 axis with FBIAS but plotted
it
> like a
> >> > > > >> histogram
> >> > > > >> with bars.
> >> > > > >>
> >> > > > >> I made the attached plot/xml using the mv_ylin_pcp
database...
> >> > > > >> - It includes 4 lines where FV3GFS is blue and GFS is
red.
> >> > > > >> - The solid lines are for the EAST region and dashed
lines
> are
> >> > for
> >> > > > the
> >> > > > >> WEST region. Take a close look at the Series definition
> >> section. I
> >> > > > >> defined the EAST as a "Group" of 6 subregions and the
WEST as a
> >> > > "Group"
> >> > > > of
> >> > > > >> 8 subregions.
> >> > > > >> - The gray bars show the base rate for the EAST and
WEST
> >> regions.
> >> > > > >> Since
> >> > > > >> I only want to plot 2 of the 4, I chose "Yes" for the
"Hide"
> >> column
> >> > > for
> >> > > > >> the
> >> > > > >> 2 lines I wanted to ignore.
> >> > > > >> - In the "Series Formatting" I used the "Legend Text"
column
> >> to
> >> > > > define
> >> > > > >> the legend.
> >> > > > >>
> >> > > > >> Granted, this plot was pretty tedious to make manually
because
> of
> >> > the
> >> > > > >> masking region groups, but hopefully it includes all the
info
> you
> >> > > need.
> >> > > > >>
> >> > > > >> And note that statistical significance is not currently
> included
> >> in
> >> > > this
> >> > > > >> plot, but it could be added.
> >> > > > >>
> >> > > > >> Thanks,
> >> > > > >> John
> >> > > > >>
> >> > > > >>
> >> > > > >>
> >> > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <
> >> met_help at ucar.edu>
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
> >> > > > >>> Transaction: Ticket created by ying.lin at noaa.gov
> >> > > > >>> Queue: met_help
> >> > > > >>> Subject: MV CTC plot: count of data points for
each
> >> threshold
> >> > > > (Fwd:
> >> > > > >>> FBIAS east vs. west US
> >> > > > >>> Owner: Nobody
> >> > > > >>> Requestors: ying.lin at noaa.gov
> >> > > > >>> Status: new
> >> > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> >> > > Ticket/Display.html?id=86995
> >> > > > >
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> Hi Tracey,
> >> > > > >>>
> >> > > > >>> I haven't tried showing stats for each threshold,
you're
> >> way
> >> > > > ahead
> >> > > > >>> of me. Let me forward your inquiry to met_help.
> >> > > > >>>
> >> > > > >>> Dear met_help:
> >> > > > >>>
> >> > > > >>> I think Tracey is referring to the numbers such
as that
> on
> >> > top
> >> > > of
> >> > > > >>> this plot:
> >> > > > >>>
> >> > > > >>>
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> >> > > > >>> 01809/intl_reg.201809.gif,
> >> > > > >>>
> >> > > > >>> for each verifying threshold.
> >> > > > >>>
> >> > > > >>> Thanks,
> >> > > > >>>
> >> > > > >>> Ying
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> -------- Forwarded Message --------
> >> > > > >>> Subject: FBIAS east vs. west US
> >> > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> >> > > > >>> From: Tracey Dorian - NOAA Affiliate <
> tracey.dorian at noaa.gov>
> >> > > > >>> To: Ying Lin <ying.lin at noaa.gov>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> Hi Ying,
> >> > > > >>>
> >> > > > >>> I've created plots of FBIAS for east regions vs. west
regions
> >> and
> >> > > Geoff
> >> > > > >>> was wondering if I can show the number of events for
each
> >> > > threshold? I
> >> > > > >>> know METViewer has the "display number of stats" option
which
> I
> >> > > checked
> >> > > > >>> off, but it seems to show the same number for each
threshold.
> >> Do
> >> > you
> >> > > > >>> know if there is a way to show the different number of
stats
> for
> >> > each
> >> > > > >>> threshold? We would probably expect the number of
events to
> be
> >> > high
> >> > > > for
> >> > > > >>> the lower thresholds and get progressively smaller as
you move
> >> into
> >> > > the
> >> > > > >>> higher thresholds, right?
> >> > > > >>>
> >> > > > >>> Thanks,
> >> > > > >>> Tracey
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > > --
> >> > > > > Ying Lin
> >> > > > > NCEP/EMC/Verification, Post-processing and Product
Generation
> >> Branch
> >> > > > > NCWCP Cubicle No. 2015
> >> > > > > Ying.Lin at noaa.gov
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Tracey Dorian
> >> > > > NOAA/NCEP/EMC
> >> > > > IMSG contractor
> >> > > > 5830 University Research Court
> >> > > > College Park, MD 20740
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Tracey Dorian
> >> > NOAA/NCEP/EMC
> >> > IMSG contractor
> >> > 5830 University Research Court
> >> > College Park, MD 20740
> >> >
> >> >
> >>
> >>
> >
> >
> > --
> > Tracey Dorian
> > NOAA/NCEP/EMC
> > IMSG contractor
> > 5830 University Research Court
> > College Park, MD 20740
> >
>
>
>
> --
> Tracey Dorian
> NOAA/NCEP/EMC
> IMSG contractor
> 5830 University Research Court
> College Park, MD 20740
>
>
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: tracey.dorian at noaa.gov
Time: Tue Sep 25 07:34:03 2018
Hi John,
Sorry for the late reply! Could you please resend the XML file that
produced this plot attached? For some reason it is not loading
correctly
into METViewer.
Thanks,
Tracey
On Mon, Sep 17, 2018 at 5:00 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Tracey,
>
> I played around a bit. See attached. Here's the changes I made...
>
> - Instead of plotting base rate on the "Y1" axis, I selected the
"Y2" tab
> and defined it here.
> - In the bottom-left corner of the GUI, I selected "+Add Derived
Curve" and
> defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24 FBIAS".
> - In the "Statistics" box, I selected "Aggregation Statistics" radio
> button, "Contingency Table Count (CTC)" from the drop-down list, and
then
> set "Bootstrapping replications" to 1000. (I used 100 for testing
since
> it's much faster!)
> - In the "Series Formatting" section, I set "GFS APCP/24 BASER" to
light
> gray, "Series Line Type" = "histogram-like" and "Line Width" = 30.
> - In the "Series Formatting" section for the "DIFF" line, I selected
"Show
> Significance" to true.
> Anywhere the confidence interval of the difference line is bold,
that
> means the difference is statistically significant.
> - In the "Legend Text" column I added some meaningful labels.
> - In the "Formatting" tab I set "plot script commands" as
> "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
> That'll draw horizontal gray dashed reference lines at 0 and 1.
>
> Remember that this is currently only plotting results for
"G218/SEC". Be
> sure to change that to the full set of regions you want to use.
>
> John
>
> On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> >
> > Hi John,
> >
> > Yes, that's what I was looking for! How were you able to plot the
base
> > rate using bars instead of lines? Also, how would you suggest
adding
> > statistical significance to the plot?
> >
> > Thanks for sending your XML file!
> >
> > -Tracey
> >
> > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Tracey,
> > >
> > > Thanks for sending your XML file. I didn't see any reference to
> "baser"
> > in
> > > the XML you sent. So I added it as a second statistic to be
plotted.
> > >
> > > Since you didn't define a VX_MASK to be used, METviewer tried to
use
> all
> > > available in the database, which likely made it run slowly.
> > >
> > > For testing, I set VX_MASK = G218/SEC added a line for BASER and
> attached
> > > the result.
> > >
> > > Is that what you're looking for?
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
>
> > > >
> > > > Hi John,
> > > >
> > > > Thanks for much for your quick reply! I am trying to add the
BASER
> > field
> > > > to my plots, but I'm getting errors, specifically a proxy
timeout.
> Do
> > > you
> > > > know why this is happening? I've attached my XML file to this
email.
> > > >
> > > > Thanks,
> > > > Tracey
> > > >
> > > >
> > > >
> > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin <ying.lin at noaa.gov>
wrote:
> > > >
> > > > > Thanks very much John!
> > > > >
> > > > > Tracey, did this answer your question? I have an
RFC/Dataflow
> crisis
> > > > > today so haven't tested the answer. If you have further
questions,
> > > > please
> > > > > follow-up on this met_help ticket (and cc to me - sure would
like
> to
> > > know
> > > > > too). Thanks.
> > > > >
> > > > > Ying
> > > > >
> > > > >
> > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
> > > > >
> > > > >> Ying and Tracey,
> > > > >>
> > > > >> I see that you'd like to include in your METviewer plot
> information
> > > > about
> > > > >> the event frequency. You tried using the "Display Number
of
> Stats"
> > > > option
> > > > >> but that that didn't do exactly what you want. Those
numbers stay
> > > > >> constant
> > > > >> for all the thresholds.
> > > > >>
> > > > >> The "NStats" values listed across the top indicate the
sample
> size.
> > > > They
> > > > >> are computed as the sum of the number of cases that are
included
> > > across
> > > > >> all
> > > > >> the series (i.e. lines on the plot) for each independent
variable
> > > value.
> > > > >> This is not the number of events meeting each threshold
> criteria...
> > > > it's a
> > > > >> count of the number of "cases" included. In the sample
plots
> you
> > > > sent,
> > > > >> you have NStats = 33948. Since you didn't send the
corresponding
> > XML,
> > > > >> it's
> > > > >> difficult for me to exactly replicate them. Basically
though, the
> > > logic
> > > > >> is:
> > > > >> - For a particular independent variable value, >1.0" for
> example,
> > > you
> > > > >> have 2 lines on the plot.
> > > > >> - You have many model initializations included, perhaps
397.
> > > > >> - You have multiple lead times included. For APCP/24
probably
> > 24,
> > > > 48,
> > > > >> 72, 96... perhaps about 8 of them.
> > > > >> - You're including multiple subregions in each plot...
for
> > example,
> > > > the
> > > > >> EAST consists of 6 subregions.
> > > > >>
> > > > >> Multiplying all those numbers together... 2 * 397 * 8 * 6 =
38112
> > gets
> > > > you
> > > > >> somewhere near the actual number of 33948 we're seeing.
> > > > >>
> > > > >> If you want to see the frequency at which the event is
actually
> > > > occurring,
> > > > >> I'd suggest plotting the base rate statistic, called BASER
in
> > > METviewer.
> > > > >> That's exactly what you want... a number between 0 and 1
> > representing
> > > > the
> > > > >> event frequency in the observation data. The BASER could
be shown
> > > > >> separately for the GFS and FV3GFS... but as long as you're
using
> the
> > > > exact
> > > > >> same cases, they should be identical. We often plot the
BASER on
> > the
> > > Y2
> > > > >> axis so that we can control the range of the Y2 axis, but I
ran
> into
> > > > some
> > > > >> problems doing that.
> > > > >>
> > > > >> Instead, I put it on the Y1 axis with FBIAS but plotted it
like a
> > > > >> histogram
> > > > >> with bars.
> > > > >>
> > > > >> I made the attached plot/xml using the mv_ylin_pcp
database...
> > > > >> - It includes 4 lines where FV3GFS is blue and GFS is
red.
> > > > >> - The solid lines are for the EAST region and dashed
lines are
> > for
> > > > the
> > > > >> WEST region. Take a close look at the Series definition
> section. I
> > > > >> defined the EAST as a "Group" of 6 subregions and the WEST
as a
> > > "Group"
> > > > of
> > > > >> 8 subregions.
> > > > >> - The gray bars show the base rate for the EAST and WEST
> regions.
> > > > >> Since
> > > > >> I only want to plot 2 of the 4, I chose "Yes" for the
"Hide"
> column
> > > for
> > > > >> the
> > > > >> 2 lines I wanted to ignore.
> > > > >> - In the "Series Formatting" I used the "Legend Text"
column to
> > > > define
> > > > >> the legend.
> > > > >>
> > > > >> Granted, this plot was pretty tedious to make manually
because of
> > the
> > > > >> masking region groups, but hopefully it includes all the
info you
> > > need.
> > > > >>
> > > > >> And note that statistical significance is not currently
included
> in
> > > this
> > > > >> plot, but it could be added.
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <
> met_help at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
> > > > >>> Transaction: Ticket created by ying.lin at noaa.gov
> > > > >>> Queue: met_help
> > > > >>> Subject: MV CTC plot: count of data points for each
> threshold
> > > > (Fwd:
> > > > >>> FBIAS east vs. west US
> > > > >>> Owner: Nobody
> > > > >>> Requestors: ying.lin at noaa.gov
> > > > >>> Status: new
> > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=86995
> > > > >
> > > > >>>
> > > > >>>
> > > > >>> Hi Tracey,
> > > > >>>
> > > > >>> I haven't tried showing stats for each threshold,
you're
> way
> > > > ahead
> > > > >>> of me. Let me forward your inquiry to met_help.
> > > > >>>
> > > > >>> Dear met_help:
> > > > >>>
> > > > >>> I think Tracey is referring to the numbers such as
that on
> > top
> > > of
> > > > >>> this plot:
> > > > >>>
> > > > >>>
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> > > > >>> 01809/intl_reg.201809.gif,
> > > > >>>
> > > > >>> for each verifying threshold.
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> Ying
> > > > >>>
> > > > >>>
> > > > >>> -------- Forwarded Message --------
> > > > >>> Subject: FBIAS east vs. west US
> > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> > > > >>> From: Tracey Dorian - NOAA Affiliate
<tracey.dorian at noaa.gov>
> > > > >>> To: Ying Lin <ying.lin at noaa.gov>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> Hi Ying,
> > > > >>>
> > > > >>> I've created plots of FBIAS for east regions vs. west
regions and
> > > Geoff
> > > > >>> was wondering if I can show the number of events for each
> > > threshold? I
> > > > >>> know METViewer has the "display number of stats" option
which I
> > > checked
> > > > >>> off, but it seems to show the same number for each
threshold. Do
> > you
> > > > >>> know if there is a way to show the different number of
stats for
> > each
> > > > >>> threshold? We would probably expect the number of events
to be
> > high
> > > > for
> > > > >>> the lower thresholds and get progressively smaller as you
move
> into
> > > the
> > > > >>> higher thresholds, right?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Tracey
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > > --
> > > > > Ying Lin
> > > > > NCEP/EMC/Verification, Post-processing and Product
Generation
> Branch
> > > > > NCWCP Cubicle No. 2015
> > > > > Ying.Lin at noaa.gov
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Tracey Dorian
> > > > NOAA/NCEP/EMC
> > > > IMSG contractor
> > > > 5830 University Research Court
> > > > College Park, MD 20740
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Tracey Dorian
> > NOAA/NCEP/EMC
> > IMSG contractor
> > 5830 University Research Court
> > College Park, MD 20740
> >
> >
>
>
--
Tracey Dorian
NOAA/NCEP/EMC
IMSG contractor
5830 University Research Court
College Park, MD 20740
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: John Halley Gotway
Time: Tue Sep 25 09:48:39 2018
I just tried to upload that XML myself. The upload appeared to be
fine but
when I clicked "Generate Plot", I got this error:
error creating a plot or Proxy timeout: [object Object]
So I tried to manually go remake that plot. I selected...
Database = mv_ylin_pcp_erly
Y1 Dependent Variable = APCP/24
But then the list of statistics is empty. I just says "N/A".
It appears there's a larger problem here... either with this
particular
database or with the whole METviewer installation.
I think we'll need to ask Tatiana to debug this further.
Thanks,
John
On Tue, Sep 25, 2018 at 7:34 AM tracey.dorian at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>
> Hi John,
>
> Sorry for the late reply! Could you please resend the XML file that
> produced this plot attached? For some reason it is not loading
correctly
> into METViewer.
>
> Thanks,
> Tracey
>
>
>
> On Mon, Sep 17, 2018 at 5:00 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Tracey,
> >
> > I played around a bit. See attached. Here's the changes I
made...
> >
> > - Instead of plotting base rate on the "Y1" axis, I selected the
"Y2" tab
> > and defined it here.
> > - In the bottom-left corner of the GUI, I selected "+Add Derived
Curve"
> and
> > defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24 FBIAS".
> > - In the "Statistics" box, I selected "Aggregation Statistics"
radio
> > button, "Contingency Table Count (CTC)" from the drop-down list,
and then
> > set "Bootstrapping replications" to 1000. (I used 100 for testing
since
> > it's much faster!)
> > - In the "Series Formatting" section, I set "GFS APCP/24 BASER" to
light
> > gray, "Series Line Type" = "histogram-like" and "Line Width" = 30.
> > - In the "Series Formatting" section for the "DIFF" line, I
selected
> "Show
> > Significance" to true.
> > Anywhere the confidence interval of the difference line is
bold, that
> > means the difference is statistically significant.
> > - In the "Legend Text" column I added some meaningful labels.
> > - In the "Formatting" tab I set "plot script commands" as
> > "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
> > That'll draw horizontal gray dashed reference lines at 0 and 1.
> >
> > Remember that this is currently only plotting results for
"G218/SEC". Be
> > sure to change that to the full set of regions you want to use.
> >
> > John
> >
> > On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> > >
> > > Hi John,
> > >
> > > Yes, that's what I was looking for! How were you able to plot
the base
> > > rate using bars instead of lines? Also, how would you suggest
adding
> > > statistical significance to the plot?
> > >
> > > Thanks for sending your XML file!
> > >
> > > -Tracey
> > >
> > > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Tracey,
> > > >
> > > > Thanks for sending your XML file. I didn't see any reference
to
> > "baser"
> > > in
> > > > the XML you sent. So I added it as a second statistic to be
plotted.
> > > >
> > > > Since you didn't define a VX_MASK to be used, METviewer tried
to use
> > all
> > > > available in the database, which likely made it run slowly.
> > > >
> > > > For testing, I set VX_MASK = G218/SEC added a line for BASER
and
> > attached
> > > > the result.
> > > >
> > > > Is that what you're looking for?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov via RT
<
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Thanks for much for your quick reply! I am trying to add
the BASER
> > > field
> > > > > to my plots, but I'm getting errors, specifically a proxy
timeout.
> > Do
> > > > you
> > > > > know why this is happening? I've attached my XML file to
this
> email.
> > > > >
> > > > > Thanks,
> > > > > Tracey
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin
<ying.lin at noaa.gov>
> wrote:
> > > > >
> > > > > > Thanks very much John!
> > > > > >
> > > > > > Tracey, did this answer your question? I have an
RFC/Dataflow
> > crisis
> > > > > > today so haven't tested the answer. If you have further
> questions,
> > > > > please
> > > > > > follow-up on this met_help ticket (and cc to me - sure
would like
> > to
> > > > know
> > > > > > too). Thanks.
> > > > > >
> > > > > > Ying
> > > > > >
> > > > > >
> > > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
> > > > > >
> > > > > >> Ying and Tracey,
> > > > > >>
> > > > > >> I see that you'd like to include in your METviewer plot
> > information
> > > > > about
> > > > > >> the event frequency. You tried using the "Display Number
of
> > Stats"
> > > > > option
> > > > > >> but that that didn't do exactly what you want. Those
numbers
> stay
> > > > > >> constant
> > > > > >> for all the thresholds.
> > > > > >>
> > > > > >> The "NStats" values listed across the top indicate the
sample
> > size.
> > > > > They
> > > > > >> are computed as the sum of the number of cases that are
included
> > > > across
> > > > > >> all
> > > > > >> the series (i.e. lines on the plot) for each independent
> variable
> > > > value.
> > > > > >> This is not the number of events meeting each threshold
> > criteria...
> > > > > it's a
> > > > > >> count of the number of "cases" included. In the sample
plots
> > you
> > > > > sent,
> > > > > >> you have NStats = 33948. Since you didn't send the
> corresponding
> > > XML,
> > > > > >> it's
> > > > > >> difficult for me to exactly replicate them. Basically
though,
> the
> > > > logic
> > > > > >> is:
> > > > > >> - For a particular independent variable value, >1.0"
for
> > example,
> > > > you
> > > > > >> have 2 lines on the plot.
> > > > > >> - You have many model initializations included,
perhaps 397.
> > > > > >> - You have multiple lead times included. For APCP/24
> probably
> > > 24,
> > > > > 48,
> > > > > >> 72, 96... perhaps about 8 of them.
> > > > > >> - You're including multiple subregions in each plot...
for
> > > example,
> > > > > the
> > > > > >> EAST consists of 6 subregions.
> > > > > >>
> > > > > >> Multiplying all those numbers together... 2 * 397 * 8 * 6
=
> 38112
> > > gets
> > > > > you
> > > > > >> somewhere near the actual number of 33948 we're seeing.
> > > > > >>
> > > > > >> If you want to see the frequency at which the event is
actually
> > > > > occurring,
> > > > > >> I'd suggest plotting the base rate statistic, called
BASER in
> > > > METviewer.
> > > > > >> That's exactly what you want... a number between 0 and 1
> > > representing
> > > > > the
> > > > > >> event frequency in the observation data. The BASER could
be
> shown
> > > > > >> separately for the GFS and FV3GFS... but as long as
you're using
> > the
> > > > > exact
> > > > > >> same cases, they should be identical. We often plot the
BASER
> on
> > > the
> > > > Y2
> > > > > >> axis so that we can control the range of the Y2 axis, but
I ran
> > into
> > > > > some
> > > > > >> problems doing that.
> > > > > >>
> > > > > >> Instead, I put it on the Y1 axis with FBIAS but plotted
it like
> a
> > > > > >> histogram
> > > > > >> with bars.
> > > > > >>
> > > > > >> I made the attached plot/xml using the mv_ylin_pcp
database...
> > > > > >> - It includes 4 lines where FV3GFS is blue and GFS is
red.
> > > > > >> - The solid lines are for the EAST region and dashed
lines
> are
> > > for
> > > > > the
> > > > > >> WEST region. Take a close look at the Series definition
> > section. I
> > > > > >> defined the EAST as a "Group" of 6 subregions and the
WEST as a
> > > > "Group"
> > > > > of
> > > > > >> 8 subregions.
> > > > > >> - The gray bars show the base rate for the EAST and
WEST
> > regions.
> > > > > >> Since
> > > > > >> I only want to plot 2 of the 4, I chose "Yes" for the
"Hide"
> > column
> > > > for
> > > > > >> the
> > > > > >> 2 lines I wanted to ignore.
> > > > > >> - In the "Series Formatting" I used the "Legend Text"
column
> to
> > > > > define
> > > > > >> the legend.
> > > > > >>
> > > > > >> Granted, this plot was pretty tedious to make manually
because
> of
> > > the
> > > > > >> masking region groups, but hopefully it includes all the
info
> you
> > > > need.
> > > > > >>
> > > > > >> And note that statistical significance is not currently
included
> > in
> > > > this
> > > > > >> plot, but it could be added.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <
> > met_help at ucar.edu>
> > > > > >> wrote:
> > > > > >>
> > > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
> > > > > >>> Transaction: Ticket created by ying.lin at noaa.gov
> > > > > >>> Queue: met_help
> > > > > >>> Subject: MV CTC plot: count of data points for
each
> > threshold
> > > > > (Fwd:
> > > > > >>> FBIAS east vs. west US
> > > > > >>> Owner: Nobody
> > > > > >>> Requestors: ying.lin at noaa.gov
> > > > > >>> Status: new
> > > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > Ticket/Display.html?id=86995
> > > > > >
> > > > > >>>
> > > > > >>>
> > > > > >>> Hi Tracey,
> > > > > >>>
> > > > > >>> I haven't tried showing stats for each threshold,
you're
> > way
> > > > > ahead
> > > > > >>> of me. Let me forward your inquiry to met_help.
> > > > > >>>
> > > > > >>> Dear met_help:
> > > > > >>>
> > > > > >>> I think Tracey is referring to the numbers such as
that
> on
> > > top
> > > > of
> > > > > >>> this plot:
> > > > > >>>
> > > > > >>>
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> > > > > >>> 01809/intl_reg.201809.gif,
> > > > > >>>
> > > > > >>> for each verifying threshold.
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>>
> > > > > >>> Ying
> > > > > >>>
> > > > > >>>
> > > > > >>> -------- Forwarded Message --------
> > > > > >>> Subject: FBIAS east vs. west US
> > > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> > > > > >>> From: Tracey Dorian - NOAA Affiliate
<tracey.dorian at noaa.gov
> >
> > > > > >>> To: Ying Lin <ying.lin at noaa.gov>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> Hi Ying,
> > > > > >>>
> > > > > >>> I've created plots of FBIAS for east regions vs. west
regions
> and
> > > > Geoff
> > > > > >>> was wondering if I can show the number of events for
each
> > > > threshold? I
> > > > > >>> know METViewer has the "display number of stats" option
which I
> > > > checked
> > > > > >>> off, but it seems to show the same number for each
threshold.
> Do
> > > you
> > > > > >>> know if there is a way to show the different number of
stats
> for
> > > each
> > > > > >>> threshold? We would probably expect the number of
events to be
> > > high
> > > > > for
> > > > > >>> the lower thresholds and get progressively smaller as
you move
> > into
> > > > the
> > > > > >>> higher thresholds, right?
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Tracey
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > > --
> > > > > > Ying Lin
> > > > > > NCEP/EMC/Verification, Post-processing and Product
Generation
> > Branch
> > > > > > NCWCP Cubicle No. 2015
> > > > > > Ying.Lin at noaa.gov
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Tracey Dorian
> > > > > NOAA/NCEP/EMC
> > > > > IMSG contractor
> > > > > 5830 University Research Court
> > > > > College Park, MD 20740
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Tracey Dorian
> > > NOAA/NCEP/EMC
> > > IMSG contractor
> > > 5830 University Research Court
> > > College Park, MD 20740
> > >
> > >
> >
> >
>
> --
> Tracey Dorian
> NOAA/NCEP/EMC
> IMSG contractor
> 5830 University Research Court
> College Park, MD 20740
>
>
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: tracey.dorian at noaa.gov
Time: Tue Sep 25 11:25:00 2018
Hi John,
Right, those were the same errors that I got when I tried to upload
the XML
file. Strange, thanks to both you and Tatiana for looking into this!
Thanks,
Tracey
On Tue, Sep 25, 2018 at 11:48 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> I just tried to upload that XML myself. The upload appeared to be
fine but
> when I clicked "Generate Plot", I got this error:
> error creating a plot or Proxy timeout: [object Object]
>
> So I tried to manually go remake that plot. I selected...
> Database = mv_ylin_pcp_erly
> Y1 Dependent Variable = APCP/24
> But then the list of statistics is empty. I just says "N/A".
>
> It appears there's a larger problem here... either with this
particular
> database or with the whole METviewer installation.
>
> I think we'll need to ask Tatiana to debug this further.
>
> Thanks,
> John
>
>
>
> On Tue, Sep 25, 2018 at 7:34 AM tracey.dorian at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> >
> > Hi John,
> >
> > Sorry for the late reply! Could you please resend the XML file
that
> > produced this plot attached? For some reason it is not loading
correctly
> > into METViewer.
> >
> > Thanks,
> > Tracey
> >
> >
> >
> > On Mon, Sep 17, 2018 at 5:00 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Tracey,
> > >
> > > I played around a bit. See attached. Here's the changes I
made...
> > >
> > > - Instead of plotting base rate on the "Y1" axis, I selected the
"Y2"
> tab
> > > and defined it here.
> > > - In the bottom-left corner of the GUI, I selected "+Add Derived
Curve"
> > and
> > > defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24
FBIAS".
> > > - In the "Statistics" box, I selected "Aggregation Statistics"
radio
> > > button, "Contingency Table Count (CTC)" from the drop-down list,
and
> then
> > > set "Bootstrapping replications" to 1000. (I used 100 for
testing since
> > > it's much faster!)
> > > - In the "Series Formatting" section, I set "GFS APCP/24 BASER"
to
> light
> > > gray, "Series Line Type" = "histogram-like" and "Line Width" =
30.
> > > - In the "Series Formatting" section for the "DIFF" line, I
selected
> > "Show
> > > Significance" to true.
> > > Anywhere the confidence interval of the difference line is
bold,
> that
> > > means the difference is statistically significant.
> > > - In the "Legend Text" column I added some meaningful labels.
> > > - In the "Formatting" tab I set "plot script commands" as
> > > "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
> > > That'll draw horizontal gray dashed reference lines at 0 and
1.
> > >
> > > Remember that this is currently only plotting results for
"G218/SEC".
> Be
> > > sure to change that to the full set of regions you want to use.
> > >
> > > John
> > >
> > > On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
>
> > > >
> > > > Hi John,
> > > >
> > > > Yes, that's what I was looking for! How were you able to plot
the
> base
> > > > rate using bars instead of lines? Also, how would you suggest
adding
> > > > statistical significance to the plot?
> > > >
> > > > Thanks for sending your XML file!
> > > >
> > > > -Tracey
> > > >
> > > > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Tracey,
> > > > >
> > > > > Thanks for sending your XML file. I didn't see any
reference to
> > > "baser"
> > > > in
> > > > > the XML you sent. So I added it as a second statistic to be
> plotted.
> > > > >
> > > > > Since you didn't define a VX_MASK to be used, METviewer
tried to
> use
> > > all
> > > > > available in the database, which likely made it run slowly.
> > > > >
> > > > > For testing, I set VX_MASK = G218/SEC added a line for BASER
and
> > > attached
> > > > > the result.
> > > > >
> > > > > Is that what you're looking for?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov via
RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Thanks for much for your quick reply! I am trying to add
the
> BASER
> > > > field
> > > > > > to my plots, but I'm getting errors, specifically a proxy
> timeout.
> > > Do
> > > > > you
> > > > > > know why this is happening? I've attached my XML file to
this
> > email.
> > > > > >
> > > > > > Thanks,
> > > > > > Tracey
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin
<ying.lin at noaa.gov>
> > wrote:
> > > > > >
> > > > > > > Thanks very much John!
> > > > > > >
> > > > > > > Tracey, did this answer your question? I have an
RFC/Dataflow
> > > crisis
> > > > > > > today so haven't tested the answer. If you have further
> > questions,
> > > > > > please
> > > > > > > follow-up on this met_help ticket (and cc to me - sure
would
> like
> > > to
> > > > > know
> > > > > > > too). Thanks.
> > > > > > >
> > > > > > > Ying
> > > > > > >
> > > > > > >
> > > > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT wrote:
> > > > > > >
> > > > > > >> Ying and Tracey,
> > > > > > >>
> > > > > > >> I see that you'd like to include in your METviewer plot
> > > information
> > > > > > about
> > > > > > >> the event frequency. You tried using the "Display
Number of
> > > Stats"
> > > > > > option
> > > > > > >> but that that didn't do exactly what you want. Those
numbers
> > stay
> > > > > > >> constant
> > > > > > >> for all the thresholds.
> > > > > > >>
> > > > > > >> The "NStats" values listed across the top indicate the
sample
> > > size.
> > > > > > They
> > > > > > >> are computed as the sum of the number of cases that are
> included
> > > > > across
> > > > > > >> all
> > > > > > >> the series (i.e. lines on the plot) for each
independent
> > variable
> > > > > value.
> > > > > > >> This is not the number of events meeting each threshold
> > > criteria...
> > > > > > it's a
> > > > > > >> count of the number of "cases" included. In the
sample
> plots
> > > you
> > > > > > sent,
> > > > > > >> you have NStats = 33948. Since you didn't send the
> > corresponding
> > > > XML,
> > > > > > >> it's
> > > > > > >> difficult for me to exactly replicate them. Basically
though,
> > the
> > > > > logic
> > > > > > >> is:
> > > > > > >> - For a particular independent variable value, >1.0"
for
> > > example,
> > > > > you
> > > > > > >> have 2 lines on the plot.
> > > > > > >> - You have many model initializations included,
perhaps
> 397.
> > > > > > >> - You have multiple lead times included. For
APCP/24
> > probably
> > > > 24,
> > > > > > 48,
> > > > > > >> 72, 96... perhaps about 8 of them.
> > > > > > >> - You're including multiple subregions in each
plot... for
> > > > example,
> > > > > > the
> > > > > > >> EAST consists of 6 subregions.
> > > > > > >>
> > > > > > >> Multiplying all those numbers together... 2 * 397 * 8 *
6 =
> > 38112
> > > > gets
> > > > > > you
> > > > > > >> somewhere near the actual number of 33948 we're seeing.
> > > > > > >>
> > > > > > >> If you want to see the frequency at which the event is
> actually
> > > > > > occurring,
> > > > > > >> I'd suggest plotting the base rate statistic, called
BASER in
> > > > > METviewer.
> > > > > > >> That's exactly what you want... a number between 0 and
1
> > > > representing
> > > > > > the
> > > > > > >> event frequency in the observation data. The BASER
could be
> > shown
> > > > > > >> separately for the GFS and FV3GFS... but as long as
you're
> using
> > > the
> > > > > > exact
> > > > > > >> same cases, they should be identical. We often plot
the BASER
> > on
> > > > the
> > > > > Y2
> > > > > > >> axis so that we can control the range of the Y2 axis,
but I
> ran
> > > into
> > > > > > some
> > > > > > >> problems doing that.
> > > > > > >>
> > > > > > >> Instead, I put it on the Y1 axis with FBIAS but plotted
it
> like
> > a
> > > > > > >> histogram
> > > > > > >> with bars.
> > > > > > >>
> > > > > > >> I made the attached plot/xml using the mv_ylin_pcp
database...
> > > > > > >> - It includes 4 lines where FV3GFS is blue and GFS
is red.
> > > > > > >> - The solid lines are for the EAST region and dashed
lines
> > are
> > > > for
> > > > > > the
> > > > > > >> WEST region. Take a close look at the Series
definition
> > > section. I
> > > > > > >> defined the EAST as a "Group" of 6 subregions and the
WEST as
> a
> > > > > "Group"
> > > > > > of
> > > > > > >> 8 subregions.
> > > > > > >> - The gray bars show the base rate for the EAST and
WEST
> > > regions.
> > > > > > >> Since
> > > > > > >> I only want to plot 2 of the 4, I chose "Yes" for the
"Hide"
> > > column
> > > > > for
> > > > > > >> the
> > > > > > >> 2 lines I wanted to ignore.
> > > > > > >> - In the "Series Formatting" I used the "Legend
Text"
> column
> > to
> > > > > > define
> > > > > > >> the legend.
> > > > > > >>
> > > > > > >> Granted, this plot was pretty tedious to make manually
because
> > of
> > > > the
> > > > > > >> masking region groups, but hopefully it includes all
the info
> > you
> > > > > need.
> > > > > > >>
> > > > > > >> And note that statistical significance is not currently
> included
> > > in
> > > > > this
> > > > > > >> plot, but it could be added.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> John
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <
> > > met_help at ucar.edu>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted upon.
> > > > > > >>> Transaction: Ticket created by ying.lin at noaa.gov
> > > > > > >>> Queue: met_help
> > > > > > >>> Subject: MV CTC plot: count of data points for
each
> > > threshold
> > > > > > (Fwd:
> > > > > > >>> FBIAS east vs. west US
> > > > > > >>> Owner: Nobody
> > > > > > >>> Requestors: ying.lin at noaa.gov
> > > > > > >>> Status: new
> > > > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > Ticket/Display.html?id=86995
> > > > > > >
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Hi Tracey,
> > > > > > >>>
> > > > > > >>> I haven't tried showing stats for each
threshold,
> you're
> > > way
> > > > > > ahead
> > > > > > >>> of me. Let me forward your inquiry to met_help.
> > > > > > >>>
> > > > > > >>> Dear met_help:
> > > > > > >>>
> > > > > > >>> I think Tracey is referring to the numbers such
as that
> > on
> > > > top
> > > > > of
> > > > > > >>> this plot:
> > > > > > >>>
> > > > > > >>>
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> > > > > > >>> 01809/intl_reg.201809.gif,
> > > > > > >>>
> > > > > > >>> for each verifying threshold.
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>>
> > > > > > >>> Ying
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> -------- Forwarded Message --------
> > > > > > >>> Subject: FBIAS east vs. west US
> > > > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> > > > > > >>> From: Tracey Dorian - NOAA Affiliate <
> tracey.dorian at noaa.gov
> > >
> > > > > > >>> To: Ying Lin <ying.lin at noaa.gov>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Hi Ying,
> > > > > > >>>
> > > > > > >>> I've created plots of FBIAS for east regions vs. west
regions
> > and
> > > > > Geoff
> > > > > > >>> was wondering if I can show the number of events for
each
> > > > > threshold? I
> > > > > > >>> know METViewer has the "display number of stats"
option
> which I
> > > > > checked
> > > > > > >>> off, but it seems to show the same number for each
threshold.
> > Do
> > > > you
> > > > > > >>> know if there is a way to show the different number of
stats
> > for
> > > > each
> > > > > > >>> threshold? We would probably expect the number of
events to
> be
> > > > high
> > > > > > for
> > > > > > >>> the lower thresholds and get progressively smaller as
you
> move
> > > into
> > > > > the
> > > > > > >>> higher thresholds, right?
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>> Tracey
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > > --
> > > > > > > Ying Lin
> > > > > > > NCEP/EMC/Verification, Post-processing and Product
Generation
> > > Branch
> > > > > > > NCWCP Cubicle No. 2015
> > > > > > > Ying.Lin at noaa.gov
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Tracey Dorian
> > > > > > NOAA/NCEP/EMC
> > > > > > IMSG contractor
> > > > > > 5830 University Research Court
> > > > > > College Park, MD 20740
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Tracey Dorian
> > > > NOAA/NCEP/EMC
> > > > IMSG contractor
> > > > 5830 University Research Court
> > > > College Park, MD 20740
> > > >
> > > >
> > >
> > >
> >
> > --
> > Tracey Dorian
> > NOAA/NCEP/EMC
> > IMSG contractor
> > 5830 University Research Court
> > College Park, MD 20740
> >
> >
>
>
--
Tracey Dorian
NOAA/NCEP/EMC
IMSG contractor
5830 University Research Court
College Park, MD 20740
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: John Halley Gotway
Time: Tue Sep 25 11:31:20 2018
Tracey,
I talked to Tatiana, and the METviewer instance at EMC is in a bad
state.
She's been coordinating with Perry (cc'ed here). The mysql db needs
to be
restarted before Perry can cleanup his databases.
The problem you're seeing is almost certainly a result of this bad
state.
I'm hopeful that once these METviewer system issues are resolved,
you'll be
able to re-generate your plot.
Sorry I don't have better news to report!
John
On Tue, Sep 25, 2018 at 11:25 AM tracey.dorian at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>
> Hi John,
>
> Right, those were the same errors that I got when I tried to upload
the XML
> file. Strange, thanks to both you and Tatiana for looking into
this!
>
> Thanks,
> Tracey
>
> On Tue, Sep 25, 2018 at 11:48 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > I just tried to upload that XML myself. The upload appeared to be
fine
> but
> > when I clicked "Generate Plot", I got this error:
> > error creating a plot or Proxy timeout: [object Object]
> >
> > So I tried to manually go remake that plot. I selected...
> > Database = mv_ylin_pcp_erly
> > Y1 Dependent Variable = APCP/24
> > But then the list of statistics is empty. I just says "N/A".
> >
> > It appears there's a larger problem here... either with this
particular
> > database or with the whole METviewer installation.
> >
> > I think we'll need to ask Tatiana to debug this further.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Tue, Sep 25, 2018 at 7:34 AM tracey.dorian at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> > >
> > > Hi John,
> > >
> > > Sorry for the late reply! Could you please resend the XML file
that
> > > produced this plot attached? For some reason it is not loading
> correctly
> > > into METViewer.
> > >
> > > Thanks,
> > > Tracey
> > >
> > >
> > >
> > > On Mon, Sep 17, 2018 at 5:00 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Tracey,
> > > >
> > > > I played around a bit. See attached. Here's the changes I
made...
> > > >
> > > > - Instead of plotting base rate on the "Y1" axis, I selected
the "Y2"
> > tab
> > > > and defined it here.
> > > > - In the bottom-left corner of the GUI, I selected "+Add
Derived
> Curve"
> > > and
> > > > defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24
FBIAS".
> > > > - In the "Statistics" box, I selected "Aggregation Statistics"
radio
> > > > button, "Contingency Table Count (CTC)" from the drop-down
list, and
> > then
> > > > set "Bootstrapping replications" to 1000. (I used 100 for
testing
> since
> > > > it's much faster!)
> > > > - In the "Series Formatting" section, I set "GFS APCP/24
BASER" to
> > light
> > > > gray, "Series Line Type" = "histogram-like" and "Line Width" =
30.
> > > > - In the "Series Formatting" section for the "DIFF" line, I
selected
> > > "Show
> > > > Significance" to true.
> > > > Anywhere the confidence interval of the difference line is
bold,
> > that
> > > > means the difference is statistically significant.
> > > > - In the "Legend Text" column I added some meaningful labels.
> > > > - In the "Formatting" tab I set "plot script commands" as
> > > > "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
> > > > That'll draw horizontal gray dashed reference lines at 0
and 1.
> > > >
> > > > Remember that this is currently only plotting results for
"G218/SEC".
> > Be
> > > > sure to change that to the full set of regions you want to
use.
> > > >
> > > > John
> > > >
> > > > On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via RT
<
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Yes, that's what I was looking for! How were you able to
plot the
> > base
> > > > > rate using bars instead of lines? Also, how would you
suggest
> adding
> > > > > statistical significance to the plot?
> > > > >
> > > > > Thanks for sending your XML file!
> > > > >
> > > > > -Tracey
> > > > >
> > > > > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Tracey,
> > > > > >
> > > > > > Thanks for sending your XML file. I didn't see any
reference to
> > > > "baser"
> > > > > in
> > > > > > the XML you sent. So I added it as a second statistic to
be
> > plotted.
> > > > > >
> > > > > > Since you didn't define a VX_MASK to be used, METviewer
tried to
> > use
> > > > all
> > > > > > available in the database, which likely made it run
slowly.
> > > > > >
> > > > > > For testing, I set VX_MASK = G218/SEC added a line for
BASER and
> > > > attached
> > > > > > the result.
> > > > > >
> > > > > > Is that what you're looking for?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov via
RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Thanks for much for your quick reply! I am trying to
add the
> > BASER
> > > > > field
> > > > > > > to my plots, but I'm getting errors, specifically a
proxy
> > timeout.
> > > > Do
> > > > > > you
> > > > > > > know why this is happening? I've attached my XML file
to this
> > > email.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Tracey
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin
<ying.lin at noaa.gov>
> > > wrote:
> > > > > > >
> > > > > > > > Thanks very much John!
> > > > > > > >
> > > > > > > > Tracey, did this answer your question? I have an
> RFC/Dataflow
> > > > crisis
> > > > > > > > today so haven't tested the answer. If you have
further
> > > questions,
> > > > > > > please
> > > > > > > > follow-up on this met_help ticket (and cc to me - sure
would
> > like
> > > > to
> > > > > > know
> > > > > > > > too). Thanks.
> > > > > > > >
> > > > > > > > Ying
> > > > > > > >
> > > > > > > >
> > > > > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT
wrote:
> > > > > > > >
> > > > > > > >> Ying and Tracey,
> > > > > > > >>
> > > > > > > >> I see that you'd like to include in your METviewer
plot
> > > > information
> > > > > > > about
> > > > > > > >> the event frequency. You tried using the "Display
Number of
> > > > Stats"
> > > > > > > option
> > > > > > > >> but that that didn't do exactly what you want. Those
> numbers
> > > stay
> > > > > > > >> constant
> > > > > > > >> for all the thresholds.
> > > > > > > >>
> > > > > > > >> The "NStats" values listed across the top indicate
the
> sample
> > > > size.
> > > > > > > They
> > > > > > > >> are computed as the sum of the number of cases that
are
> > included
> > > > > > across
> > > > > > > >> all
> > > > > > > >> the series (i.e. lines on the plot) for each
independent
> > > variable
> > > > > > value.
> > > > > > > >> This is not the number of events meeting each
threshold
> > > > criteria...
> > > > > > > it's a
> > > > > > > >> count of the number of "cases" included. In the
sample
> > plots
> > > > you
> > > > > > > sent,
> > > > > > > >> you have NStats = 33948. Since you didn't send the
> > > corresponding
> > > > > XML,
> > > > > > > >> it's
> > > > > > > >> difficult for me to exactly replicate them.
Basically
> though,
> > > the
> > > > > > logic
> > > > > > > >> is:
> > > > > > > >> - For a particular independent variable value,
>1.0" for
> > > > example,
> > > > > > you
> > > > > > > >> have 2 lines on the plot.
> > > > > > > >> - You have many model initializations included,
perhaps
> > 397.
> > > > > > > >> - You have multiple lead times included. For
APCP/24
> > > probably
> > > > > 24,
> > > > > > > 48,
> > > > > > > >> 72, 96... perhaps about 8 of them.
> > > > > > > >> - You're including multiple subregions in each
plot...
> for
> > > > > example,
> > > > > > > the
> > > > > > > >> EAST consists of 6 subregions.
> > > > > > > >>
> > > > > > > >> Multiplying all those numbers together... 2 * 397 * 8
* 6 =
> > > 38112
> > > > > gets
> > > > > > > you
> > > > > > > >> somewhere near the actual number of 33948 we're
seeing.
> > > > > > > >>
> > > > > > > >> If you want to see the frequency at which the event
is
> > actually
> > > > > > > occurring,
> > > > > > > >> I'd suggest plotting the base rate statistic, called
BASER
> in
> > > > > > METviewer.
> > > > > > > >> That's exactly what you want... a number between 0
and 1
> > > > > representing
> > > > > > > the
> > > > > > > >> event frequency in the observation data. The BASER
could be
> > > shown
> > > > > > > >> separately for the GFS and FV3GFS... but as long as
you're
> > using
> > > > the
> > > > > > > exact
> > > > > > > >> same cases, they should be identical. We often plot
the
> BASER
> > > on
> > > > > the
> > > > > > Y2
> > > > > > > >> axis so that we can control the range of the Y2 axis,
but I
> > ran
> > > > into
> > > > > > > some
> > > > > > > >> problems doing that.
> > > > > > > >>
> > > > > > > >> Instead, I put it on the Y1 axis with FBIAS but
plotted it
> > like
> > > a
> > > > > > > >> histogram
> > > > > > > >> with bars.
> > > > > > > >>
> > > > > > > >> I made the attached plot/xml using the mv_ylin_pcp
> database...
> > > > > > > >> - It includes 4 lines where FV3GFS is blue and GFS
is
> red.
> > > > > > > >> - The solid lines are for the EAST region and
dashed
> lines
> > > are
> > > > > for
> > > > > > > the
> > > > > > > >> WEST region. Take a close look at the Series
definition
> > > > section. I
> > > > > > > >> defined the EAST as a "Group" of 6 subregions and the
WEST
> as
> > a
> > > > > > "Group"
> > > > > > > of
> > > > > > > >> 8 subregions.
> > > > > > > >> - The gray bars show the base rate for the EAST
and WEST
> > > > regions.
> > > > > > > >> Since
> > > > > > > >> I only want to plot 2 of the 4, I chose "Yes" for the
"Hide"
> > > > column
> > > > > > for
> > > > > > > >> the
> > > > > > > >> 2 lines I wanted to ignore.
> > > > > > > >> - In the "Series Formatting" I used the "Legend
Text"
> > column
> > > to
> > > > > > > define
> > > > > > > >> the legend.
> > > > > > > >>
> > > > > > > >> Granted, this plot was pretty tedious to make
manually
> because
> > > of
> > > > > the
> > > > > > > >> masking region groups, but hopefully it includes all
the
> info
> > > you
> > > > > > need.
> > > > > > > >>
> > > > > > > >> And note that statistical significance is not
currently
> > included
> > > > in
> > > > > > this
> > > > > > > >> plot, but it could be added.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> John
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <
> > > > met_help at ucar.edu>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted
upon.
> > > > > > > >>> Transaction: Ticket created by ying.lin at noaa.gov
> > > > > > > >>> Queue: met_help
> > > > > > > >>> Subject: MV CTC plot: count of data points for
each
> > > > threshold
> > > > > > > (Fwd:
> > > > > > > >>> FBIAS east vs. west US
> > > > > > > >>> Owner: Nobody
> > > > > > > >>> Requestors: ying.lin at noaa.gov
> > > > > > > >>> Status: new
> > > > > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > > Ticket/Display.html?id=86995
> > > > > > > >
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Hi Tracey,
> > > > > > > >>>
> > > > > > > >>> I haven't tried showing stats for each
threshold,
> > you're
> > > > way
> > > > > > > ahead
> > > > > > > >>> of me. Let me forward your inquiry to met_help.
> > > > > > > >>>
> > > > > > > >>> Dear met_help:
> > > > > > > >>>
> > > > > > > >>> I think Tracey is referring to the numbers
such as
> that
> > > on
> > > > > top
> > > > > > of
> > > > > > > >>> this plot:
> > > > > > > >>>
> > > > > > > >>>
> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> > > > > > > >>> 01809/intl_reg.201809.gif,
> > > > > > > >>>
> > > > > > > >>> for each verifying threshold.
> > > > > > > >>>
> > > > > > > >>> Thanks,
> > > > > > > >>>
> > > > > > > >>> Ying
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> -------- Forwarded Message --------
> > > > > > > >>> Subject: FBIAS east vs. west US
> > > > > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> > > > > > > >>> From: Tracey Dorian - NOAA Affiliate <
> > tracey.dorian at noaa.gov
> > > >
> > > > > > > >>> To: Ying Lin <ying.lin at noaa.gov>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Hi Ying,
> > > > > > > >>>
> > > > > > > >>> I've created plots of FBIAS for east regions vs.
west
> regions
> > > and
> > > > > > Geoff
> > > > > > > >>> was wondering if I can show the number of events for
each
> > > > > > threshold? I
> > > > > > > >>> know METViewer has the "display number of stats"
option
> > which I
> > > > > > checked
> > > > > > > >>> off, but it seems to show the same number for each
> threshold.
> > > Do
> > > > > you
> > > > > > > >>> know if there is a way to show the different number
of
> stats
> > > for
> > > > > each
> > > > > > > >>> threshold? We would probably expect the number of
events
> to
> > be
> > > > > high
> > > > > > > for
> > > > > > > >>> the lower thresholds and get progressively smaller
as you
> > move
> > > > into
> > > > > > the
> > > > > > > >>> higher thresholds, right?
> > > > > > > >>>
> > > > > > > >>> Thanks,
> > > > > > > >>> Tracey
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > > --
> > > > > > > > Ying Lin
> > > > > > > > NCEP/EMC/Verification, Post-processing and Product
Generation
> > > > Branch
> > > > > > > > NCWCP Cubicle No. 2015
> > > > > > > > Ying.Lin at noaa.gov
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Tracey Dorian
> > > > > > > NOAA/NCEP/EMC
> > > > > > > IMSG contractor
> > > > > > > 5830 University Research Court
> > > > > > > College Park, MD 20740
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Tracey Dorian
> > > > > NOAA/NCEP/EMC
> > > > > IMSG contractor
> > > > > 5830 University Research Court
> > > > > College Park, MD 20740
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > Tracey Dorian
> > > NOAA/NCEP/EMC
> > > IMSG contractor
> > > 5830 University Research Court
> > > College Park, MD 20740
> > >
> > >
> >
> >
>
> --
> Tracey Dorian
> NOAA/NCEP/EMC
> IMSG contractor
> 5830 University Research Court
> College Park, MD 20740
>
>
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: tracey.dorian at noaa.gov
Time: Tue Sep 25 11:34:05 2018
Ok, thanks for the update!
On Tue, Sep 25, 2018 at 1:31 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Tracey,
>
> I talked to Tatiana, and the METviewer instance at EMC is in a bad
state.
> She's been coordinating with Perry (cc'ed here). The mysql db needs
to be
> restarted before Perry can cleanup his databases.
>
> The problem you're seeing is almost certainly a result of this bad
state.
> I'm hopeful that once these METviewer system issues are resolved,
you'll be
> able to re-generate your plot.
>
> Sorry I don't have better news to report!
>
> John
>
> On Tue, Sep 25, 2018 at 11:25 AM tracey.dorian at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> >
> > Hi John,
> >
> > Right, those were the same errors that I got when I tried to
upload the
> XML
> > file. Strange, thanks to both you and Tatiana for looking into
this!
> >
> > Thanks,
> > Tracey
> >
> > On Tue, Sep 25, 2018 at 11:48 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > I just tried to upload that XML myself. The upload appeared to
be fine
> > but
> > > when I clicked "Generate Plot", I got this error:
> > > error creating a plot or Proxy timeout: [object Object]
> > >
> > > So I tried to manually go remake that plot. I selected...
> > > Database = mv_ylin_pcp_erly
> > > Y1 Dependent Variable = APCP/24
> > > But then the list of statistics is empty. I just says "N/A".
> > >
> > > It appears there's a larger problem here... either with this
particular
> > > database or with the whole METviewer installation.
> > >
> > > I think we'll need to ask Tatiana to debug this further.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Tue, Sep 25, 2018 at 7:34 AM tracey.dorian at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
>
> > > >
> > > > Hi John,
> > > >
> > > > Sorry for the late reply! Could you please resend the XML
file that
> > > > produced this plot attached? For some reason it is not
loading
> > correctly
> > > > into METViewer.
> > > >
> > > > Thanks,
> > > > Tracey
> > > >
> > > >
> > > >
> > > > On Mon, Sep 17, 2018 at 5:00 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Tracey,
> > > > >
> > > > > I played around a bit. See attached. Here's the changes I
made...
> > > > >
> > > > > - Instead of plotting base rate on the "Y1" axis, I selected
the
> "Y2"
> > > tab
> > > > > and defined it here.
> > > > > - In the bottom-left corner of the GUI, I selected "+Add
Derived
> > Curve"
> > > > and
> > > > > defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24
FBIAS".
> > > > > - In the "Statistics" box, I selected "Aggregation
Statistics"
> radio
> > > > > button, "Contingency Table Count (CTC)" from the drop-down
list,
> and
> > > then
> > > > > set "Bootstrapping replications" to 1000. (I used 100 for
testing
> > since
> > > > > it's much faster!)
> > > > > - In the "Series Formatting" section, I set "GFS APCP/24
BASER" to
> > > light
> > > > > gray, "Series Line Type" = "histogram-like" and "Line Width"
= 30.
> > > > > - In the "Series Formatting" section for the "DIFF" line, I
> selected
> > > > "Show
> > > > > Significance" to true.
> > > > > Anywhere the confidence interval of the difference line
is bold,
> > > that
> > > > > means the difference is statistically significant.
> > > > > - In the "Legend Text" column I added some meaningful
labels.
> > > > > - In the "Formatting" tab I set "plot script commands" as
> > > > > "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
> > > > > That'll draw horizontal gray dashed reference lines at 0
and 1.
> > > > >
> > > > > Remember that this is currently only plotting results for
> "G218/SEC".
> > > Be
> > > > > sure to change that to the full set of regions you want to
use.
> > > > >
> > > > > John
> > > > >
> > > > > On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via
RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Yes, that's what I was looking for! How were you able to
plot
> the
> > > base
> > > > > > rate using bars instead of lines? Also, how would you
suggest
> > adding
> > > > > > statistical significance to the plot?
> > > > > >
> > > > > > Thanks for sending your XML file!
> > > > > >
> > > > > > -Tracey
> > > > > >
> > > > > > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Tracey,
> > > > > > >
> > > > > > > Thanks for sending your XML file. I didn't see any
reference
> to
> > > > > "baser"
> > > > > > in
> > > > > > > the XML you sent. So I added it as a second statistic
to be
> > > plotted.
> > > > > > >
> > > > > > > Since you didn't define a VX_MASK to be used, METviewer
tried
> to
> > > use
> > > > > all
> > > > > > > available in the database, which likely made it run
slowly.
> > > > > > >
> > > > > > > For testing, I set VX_MASK = G218/SEC added a line for
BASER
> and
> > > > > attached
> > > > > > > the result.
> > > > > > >
> > > > > > > Is that what you're looking for?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov
via RT
> <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
> > >
> > > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > Thanks for much for your quick reply! I am trying to
add the
> > > BASER
> > > > > > field
> > > > > > > > to my plots, but I'm getting errors, specifically a
proxy
> > > timeout.
> > > > > Do
> > > > > > > you
> > > > > > > > know why this is happening? I've attached my XML file
to
> this
> > > > email.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Tracey
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin
<ying.lin at noaa.gov
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks very much John!
> > > > > > > > >
> > > > > > > > > Tracey, did this answer your question? I have an
> > RFC/Dataflow
> > > > > crisis
> > > > > > > > > today so haven't tested the answer. If you have
further
> > > > questions,
> > > > > > > > please
> > > > > > > > > follow-up on this met_help ticket (and cc to me -
sure
> would
> > > like
> > > > > to
> > > > > > > know
> > > > > > > > > too). Thanks.
> > > > > > > > >
> > > > > > > > > Ying
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT
wrote:
> > > > > > > > >
> > > > > > > > >> Ying and Tracey,
> > > > > > > > >>
> > > > > > > > >> I see that you'd like to include in your METviewer
plot
> > > > > information
> > > > > > > > about
> > > > > > > > >> the event frequency. You tried using the "Display
Number
> of
> > > > > Stats"
> > > > > > > > option
> > > > > > > > >> but that that didn't do exactly what you want.
Those
> > numbers
> > > > stay
> > > > > > > > >> constant
> > > > > > > > >> for all the thresholds.
> > > > > > > > >>
> > > > > > > > >> The "NStats" values listed across the top indicate
the
> > sample
> > > > > size.
> > > > > > > > They
> > > > > > > > >> are computed as the sum of the number of cases that
are
> > > included
> > > > > > > across
> > > > > > > > >> all
> > > > > > > > >> the series (i.e. lines on the plot) for each
independent
> > > > variable
> > > > > > > value.
> > > > > > > > >> This is not the number of events meeting each
threshold
> > > > > criteria...
> > > > > > > > it's a
> > > > > > > > >> count of the number of "cases" included. In the
sample
> > > plots
> > > > > you
> > > > > > > > sent,
> > > > > > > > >> you have NStats = 33948. Since you didn't send the
> > > > corresponding
> > > > > > XML,
> > > > > > > > >> it's
> > > > > > > > >> difficult for me to exactly replicate them.
Basically
> > though,
> > > > the
> > > > > > > logic
> > > > > > > > >> is:
> > > > > > > > >> - For a particular independent variable value,
>1.0"
> for
> > > > > example,
> > > > > > > you
> > > > > > > > >> have 2 lines on the plot.
> > > > > > > > >> - You have many model initializations included,
perhaps
> > > 397.
> > > > > > > > >> - You have multiple lead times included. For
APCP/24
> > > > probably
> > > > > > 24,
> > > > > > > > 48,
> > > > > > > > >> 72, 96... perhaps about 8 of them.
> > > > > > > > >> - You're including multiple subregions in each
plot...
> > for
> > > > > > example,
> > > > > > > > the
> > > > > > > > >> EAST consists of 6 subregions.
> > > > > > > > >>
> > > > > > > > >> Multiplying all those numbers together... 2 * 397 *
8 * 6
> =
> > > > 38112
> > > > > > gets
> > > > > > > > you
> > > > > > > > >> somewhere near the actual number of 33948 we're
seeing.
> > > > > > > > >>
> > > > > > > > >> If you want to see the frequency at which the event
is
> > > actually
> > > > > > > > occurring,
> > > > > > > > >> I'd suggest plotting the base rate statistic,
called BASER
> > in
> > > > > > > METviewer.
> > > > > > > > >> That's exactly what you want... a number between 0
and 1
> > > > > > representing
> > > > > > > > the
> > > > > > > > >> event frequency in the observation data. The BASER
could
> be
> > > > shown
> > > > > > > > >> separately for the GFS and FV3GFS... but as long as
you're
> > > using
> > > > > the
> > > > > > > > exact
> > > > > > > > >> same cases, they should be identical. We often
plot the
> > BASER
> > > > on
> > > > > > the
> > > > > > > Y2
> > > > > > > > >> axis so that we can control the range of the Y2
axis, but
> I
> > > ran
> > > > > into
> > > > > > > > some
> > > > > > > > >> problems doing that.
> > > > > > > > >>
> > > > > > > > >> Instead, I put it on the Y1 axis with FBIAS but
plotted it
> > > like
> > > > a
> > > > > > > > >> histogram
> > > > > > > > >> with bars.
> > > > > > > > >>
> > > > > > > > >> I made the attached plot/xml using the mv_ylin_pcp
> > database...
> > > > > > > > >> - It includes 4 lines where FV3GFS is blue and
GFS is
> > red.
> > > > > > > > >> - The solid lines are for the EAST region and
dashed
> > lines
> > > > are
> > > > > > for
> > > > > > > > the
> > > > > > > > >> WEST region. Take a close look at the Series
definition
> > > > > section. I
> > > > > > > > >> defined the EAST as a "Group" of 6 subregions and
the WEST
> > as
> > > a
> > > > > > > "Group"
> > > > > > > > of
> > > > > > > > >> 8 subregions.
> > > > > > > > >> - The gray bars show the base rate for the EAST
and
> WEST
> > > > > regions.
> > > > > > > > >> Since
> > > > > > > > >> I only want to plot 2 of the 4, I chose "Yes" for
the
> "Hide"
> > > > > column
> > > > > > > for
> > > > > > > > >> the
> > > > > > > > >> 2 lines I wanted to ignore.
> > > > > > > > >> - In the "Series Formatting" I used the "Legend
Text"
> > > column
> > > > to
> > > > > > > > define
> > > > > > > > >> the legend.
> > > > > > > > >>
> > > > > > > > >> Granted, this plot was pretty tedious to make
manually
> > because
> > > > of
> > > > > > the
> > > > > > > > >> masking region groups, but hopefully it includes
all the
> > info
> > > > you
> > > > > > > need.
> > > > > > > > >>
> > > > > > > > >> And note that statistical significance is not
currently
> > > included
> > > > > in
> > > > > > > this
> > > > > > > > >> plot, but it could be added.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> John
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <
> > > > > met_help at ucar.edu>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted
upon.
> > > > > > > > >>> Transaction: Ticket created by ying.lin at noaa.gov
> > > > > > > > >>> Queue: met_help
> > > > > > > > >>> Subject: MV CTC plot: count of data points
for each
> > > > > threshold
> > > > > > > > (Fwd:
> > > > > > > > >>> FBIAS east vs. west US
> > > > > > > > >>> Owner: Nobody
> > > > > > > > >>> Requestors: ying.lin at noaa.gov
> > > > > > > > >>> Status: new
> > > > > > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > Ticket/Display.html?id=86995
> > > > > > > > >
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> Hi Tracey,
> > > > > > > > >>>
> > > > > > > > >>> I haven't tried showing stats for each
threshold,
> > > you're
> > > > > way
> > > > > > > > ahead
> > > > > > > > >>> of me. Let me forward your inquiry to met_help.
> > > > > > > > >>>
> > > > > > > > >>> Dear met_help:
> > > > > > > > >>>
> > > > > > > > >>> I think Tracey is referring to the numbers
such as
> > that
> > > > on
> > > > > > top
> > > > > > > of
> > > > > > > > >>> this plot:
> > > > > > > > >>>
> > > > > > > > >>>
> > http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> > > > > > > > >>> 01809/intl_reg.201809.gif,
> > > > > > > > >>>
> > > > > > > > >>> for each verifying threshold.
> > > > > > > > >>>
> > > > > > > > >>> Thanks,
> > > > > > > > >>>
> > > > > > > > >>> Ying
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> -------- Forwarded Message --------
> > > > > > > > >>> Subject: FBIAS east vs. west US
> > > > > > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> > > > > > > > >>> From: Tracey Dorian - NOAA Affiliate <
> > > tracey.dorian at noaa.gov
> > > > >
> > > > > > > > >>> To: Ying Lin <ying.lin at noaa.gov>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> Hi Ying,
> > > > > > > > >>>
> > > > > > > > >>> I've created plots of FBIAS for east regions vs.
west
> > regions
> > > > and
> > > > > > > Geoff
> > > > > > > > >>> was wondering if I can show the number of events
for each
> > > > > > > threshold? I
> > > > > > > > >>> know METViewer has the "display number of stats"
option
> > > which I
> > > > > > > checked
> > > > > > > > >>> off, but it seems to show the same number for each
> > threshold.
> > > > Do
> > > > > > you
> > > > > > > > >>> know if there is a way to show the different
number of
> > stats
> > > > for
> > > > > > each
> > > > > > > > >>> threshold? We would probably expect the number of
events
> > to
> > > be
> > > > > > high
> > > > > > > > for
> > > > > > > > >>> the lower thresholds and get progressively smaller
as you
> > > move
> > > > > into
> > > > > > > the
> > > > > > > > >>> higher thresholds, right?
> > > > > > > > >>>
> > > > > > > > >>> Thanks,
> > > > > > > > >>> Tracey
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > > --
> > > > > > > > > Ying Lin
> > > > > > > > > NCEP/EMC/Verification, Post-processing and Product
> Generation
> > > > > Branch
> > > > > > > > > NCWCP Cubicle No. 2015
> > > > > > > > > Ying.Lin at noaa.gov
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Tracey Dorian
> > > > > > > > NOAA/NCEP/EMC
> > > > > > > > IMSG contractor
> > > > > > > > 5830 University Research Court
> > > > > > > > College Park, MD 20740
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Tracey Dorian
> > > > > > NOAA/NCEP/EMC
> > > > > > IMSG contractor
> > > > > > 5830 University Research Court
> > > > > > College Park, MD 20740
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Tracey Dorian
> > > > NOAA/NCEP/EMC
> > > > IMSG contractor
> > > > 5830 University Research Court
> > > > College Park, MD 20740
> > > >
> > > >
> > >
> > >
> >
> > --
> > Tracey Dorian
> > NOAA/NCEP/EMC
> > IMSG contractor
> > 5830 University Research Court
> > College Park, MD 20740
> >
> >
>
>
--
Tracey Dorian
NOAA/NCEP/EMC
IMSG contractor
5830 University Research Court
College Park, MD 20740
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: perry.shafran at noaa.gov
Time: Tue Sep 25 11:37:29 2018
Hi, Tracey, etc,
I already have a helpdesk ticket into NCO as we need their assistance
to
help clean up the mysql database. I have cc'd Jason in the hopes that
we
can get this done quickly, as we might be dead in the water without
that.
Thanks!
Perry
On Tue, Sep 25, 2018 at 1:31 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Tracey,
>
> I talked to Tatiana, and the METviewer instance at EMC is in a bad
state.
> She's been coordinating with Perry (cc'ed here). The mysql db needs
to be
> restarted before Perry can cleanup his databases.
>
> The problem you're seeing is almost certainly a result of this bad
state.
> I'm hopeful that once these METviewer system issues are resolved,
you'll be
> able to re-generate your plot.
>
> Sorry I don't have better news to report!
>
> John
>
> On Tue, Sep 25, 2018 at 11:25 AM tracey.dorian at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>>
>> Hi John,
>>
>> Right, those were the same errors that I got when I tried to upload
the
>> XML
>> file. Strange, thanks to both you and Tatiana for looking into
this!
>>
>> Thanks,
>> Tracey
>>
>> On Tue, Sep 25, 2018 at 11:48 AM John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>> > I just tried to upload that XML myself. The upload appeared to
be fine
>> but
>> > when I clicked "Generate Plot", I got this error:
>> > error creating a plot or Proxy timeout: [object Object]
>> >
>> > So I tried to manually go remake that plot. I selected...
>> > Database = mv_ylin_pcp_erly
>> > Y1 Dependent Variable = APCP/24
>> > But then the list of statistics is empty. I just says "N/A".
>> >
>> > It appears there's a larger problem here... either with this
particular
>> > database or with the whole METviewer installation.
>> >
>> > I think we'll need to ask Tatiana to debug this further.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> >
>> > On Tue, Sep 25, 2018 at 7:34 AM tracey.dorian at noaa.gov via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>> > >
>> > > Hi John,
>> > >
>> > > Sorry for the late reply! Could you please resend the XML file
that
>> > > produced this plot attached? For some reason it is not loading
>> correctly
>> > > into METViewer.
>> > >
>> > > Thanks,
>> > > Tracey
>> > >
>> > >
>> > >
>> > > On Mon, Sep 17, 2018 at 5:00 PM John Halley Gotway via RT <
>> > > met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Tracey,
>> > > >
>> > > > I played around a bit. See attached. Here's the changes I
made...
>> > > >
>> > > > - Instead of plotting base rate on the "Y1" axis, I selected
the
>> "Y2"
>> > tab
>> > > > and defined it here.
>> > > > - In the bottom-left corner of the GUI, I selected "+Add
Derived
>> Curve"
>> > > and
>> > > > defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24
FBIAS".
>> > > > - In the "Statistics" box, I selected "Aggregation
Statistics" radio
>> > > > button, "Contingency Table Count (CTC)" from the drop-down
list, and
>> > then
>> > > > set "Bootstrapping replications" to 1000. (I used 100 for
testing
>> since
>> > > > it's much faster!)
>> > > > - In the "Series Formatting" section, I set "GFS APCP/24
BASER" to
>> > light
>> > > > gray, "Series Line Type" = "histogram-like" and "Line Width"
= 30.
>> > > > - In the "Series Formatting" section for the "DIFF" line, I
selected
>> > > "Show
>> > > > Significance" to true.
>> > > > Anywhere the confidence interval of the difference line is
bold,
>> > that
>> > > > means the difference is statistically significant.
>> > > > - In the "Legend Text" column I added some meaningful labels.
>> > > > - In the "Formatting" tab I set "plot script commands" as
>> > > > "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
>> > > > That'll draw horizontal gray dashed reference lines at 0
and 1.
>> > > >
>> > > > Remember that this is currently only plotting results for
>> "G218/SEC".
>> > Be
>> > > > sure to change that to the full set of regions you want to
use.
>> > > >
>> > > > John
>> > > >
>> > > > On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via RT
<
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
>> > > > >
>> > > > > Hi John,
>> > > > >
>> > > > > Yes, that's what I was looking for! How were you able to
plot the
>> > base
>> > > > > rate using bars instead of lines? Also, how would you
suggest
>> adding
>> > > > > statistical significance to the plot?
>> > > > >
>> > > > > Thanks for sending your XML file!
>> > > > >
>> > > > > -Tracey
>> > > > >
>> > > > > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via RT
<
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > > > Tracey,
>> > > > > >
>> > > > > > Thanks for sending your XML file. I didn't see any
reference to
>> > > > "baser"
>> > > > > in
>> > > > > > the XML you sent. So I added it as a second statistic to
be
>> > plotted.
>> > > > > >
>> > > > > > Since you didn't define a VX_MASK to be used, METviewer
tried to
>> > use
>> > > > all
>> > > > > > available in the database, which likely made it run
slowly.
>> > > > > >
>> > > > > > For testing, I set VX_MASK = G218/SEC added a line for
BASER and
>> > > > attached
>> > > > > > the result.
>> > > > > >
>> > > > > > Is that what you're looking for?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > John
>> > > > > >
>> > > > > > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov
via RT <
>> > > > > > met_help at ucar.edu> wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
>> >
>> > > > > > >
>> > > > > > > Hi John,
>> > > > > > >
>> > > > > > > Thanks for much for your quick reply! I am trying to
add the
>> > BASER
>> > > > > field
>> > > > > > > to my plots, but I'm getting errors, specifically a
proxy
>> > timeout.
>> > > > Do
>> > > > > > you
>> > > > > > > know why this is happening? I've attached my XML file
to this
>> > > email.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Tracey
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin
<ying.lin at noaa.gov>
>> > > wrote:
>> > > > > > >
>> > > > > > > > Thanks very much John!
>> > > > > > > >
>> > > > > > > > Tracey, did this answer your question? I have an
>> RFC/Dataflow
>> > > > crisis
>> > > > > > > > today so haven't tested the answer. If you have
further
>> > > questions,
>> > > > > > > please
>> > > > > > > > follow-up on this met_help ticket (and cc to me -
sure would
>> > like
>> > > > to
>> > > > > > know
>> > > > > > > > too). Thanks.
>> > > > > > > >
>> > > > > > > > Ying
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT
wrote:
>> > > > > > > >
>> > > > > > > >> Ying and Tracey,
>> > > > > > > >>
>> > > > > > > >> I see that you'd like to include in your METviewer
plot
>> > > > information
>> > > > > > > about
>> > > > > > > >> the event frequency. You tried using the "Display
Number
>> of
>> > > > Stats"
>> > > > > > > option
>> > > > > > > >> but that that didn't do exactly what you want.
Those
>> numbers
>> > > stay
>> > > > > > > >> constant
>> > > > > > > >> for all the thresholds.
>> > > > > > > >>
>> > > > > > > >> The "NStats" values listed across the top indicate
the
>> sample
>> > > > size.
>> > > > > > > They
>> > > > > > > >> are computed as the sum of the number of cases that
are
>> > included
>> > > > > > across
>> > > > > > > >> all
>> > > > > > > >> the series (i.e. lines on the plot) for each
independent
>> > > variable
>> > > > > > value.
>> > > > > > > >> This is not the number of events meeting each
threshold
>> > > > criteria...
>> > > > > > > it's a
>> > > > > > > >> count of the number of "cases" included. In the
sample
>> > plots
>> > > > you
>> > > > > > > sent,
>> > > > > > > >> you have NStats = 33948. Since you didn't send the
>> > > corresponding
>> > > > > XML,
>> > > > > > > >> it's
>> > > > > > > >> difficult for me to exactly replicate them.
Basically
>> though,
>> > > the
>> > > > > > logic
>> > > > > > > >> is:
>> > > > > > > >> - For a particular independent variable value,
>1.0" for
>> > > > example,
>> > > > > > you
>> > > > > > > >> have 2 lines on the plot.
>> > > > > > > >> - You have many model initializations included,
perhaps
>> > 397.
>> > > > > > > >> - You have multiple lead times included. For
APCP/24
>> > > probably
>> > > > > 24,
>> > > > > > > 48,
>> > > > > > > >> 72, 96... perhaps about 8 of them.
>> > > > > > > >> - You're including multiple subregions in each
plot...
>> for
>> > > > > example,
>> > > > > > > the
>> > > > > > > >> EAST consists of 6 subregions.
>> > > > > > > >>
>> > > > > > > >> Multiplying all those numbers together... 2 * 397 *
8 * 6 =
>> > > 38112
>> > > > > gets
>> > > > > > > you
>> > > > > > > >> somewhere near the actual number of 33948 we're
seeing.
>> > > > > > > >>
>> > > > > > > >> If you want to see the frequency at which the event
is
>> > actually
>> > > > > > > occurring,
>> > > > > > > >> I'd suggest plotting the base rate statistic, called
BASER
>> in
>> > > > > > METviewer.
>> > > > > > > >> That's exactly what you want... a number between 0
and 1
>> > > > > representing
>> > > > > > > the
>> > > > > > > >> event frequency in the observation data. The BASER
could
>> be
>> > > shown
>> > > > > > > >> separately for the GFS and FV3GFS... but as long as
you're
>> > using
>> > > > the
>> > > > > > > exact
>> > > > > > > >> same cases, they should be identical. We often plot
the
>> BASER
>> > > on
>> > > > > the
>> > > > > > Y2
>> > > > > > > >> axis so that we can control the range of the Y2
axis, but I
>> > ran
>> > > > into
>> > > > > > > some
>> > > > > > > >> problems doing that.
>> > > > > > > >>
>> > > > > > > >> Instead, I put it on the Y1 axis with FBIAS but
plotted it
>> > like
>> > > a
>> > > > > > > >> histogram
>> > > > > > > >> with bars.
>> > > > > > > >>
>> > > > > > > >> I made the attached plot/xml using the mv_ylin_pcp
>> database...
>> > > > > > > >> - It includes 4 lines where FV3GFS is blue and
GFS is
>> red.
>> > > > > > > >> - The solid lines are for the EAST region and
dashed
>> lines
>> > > are
>> > > > > for
>> > > > > > > the
>> > > > > > > >> WEST region. Take a close look at the Series
definition
>> > > > section. I
>> > > > > > > >> defined the EAST as a "Group" of 6 subregions and
the WEST
>> as
>> > a
>> > > > > > "Group"
>> > > > > > > of
>> > > > > > > >> 8 subregions.
>> > > > > > > >> - The gray bars show the base rate for the EAST
and WEST
>> > > > regions.
>> > > > > > > >> Since
>> > > > > > > >> I only want to plot 2 of the 4, I chose "Yes" for
the
>> "Hide"
>> > > > column
>> > > > > > for
>> > > > > > > >> the
>> > > > > > > >> 2 lines I wanted to ignore.
>> > > > > > > >> - In the "Series Formatting" I used the "Legend
Text"
>> > column
>> > > to
>> > > > > > > define
>> > > > > > > >> the legend.
>> > > > > > > >>
>> > > > > > > >> Granted, this plot was pretty tedious to make
manually
>> because
>> > > of
>> > > > > the
>> > > > > > > >> masking region groups, but hopefully it includes all
the
>> info
>> > > you
>> > > > > > need.
>> > > > > > > >>
>> > > > > > > >> And note that statistical significance is not
currently
>> > included
>> > > > in
>> > > > > > this
>> > > > > > > >> plot, but it could be added.
>> > > > > > > >>
>> > > > > > > >> Thanks,
>> > > > > > > >> John
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <
>> > > > met_help at ucar.edu>
>> > > > > > > >> wrote:
>> > > > > > > >>
>> > > > > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted
upon.
>> > > > > > > >>> Transaction: Ticket created by ying.lin at noaa.gov
>> > > > > > > >>> Queue: met_help
>> > > > > > > >>> Subject: MV CTC plot: count of data points
for each
>> > > > threshold
>> > > > > > > (Fwd:
>> > > > > > > >>> FBIAS east vs. west US
>> > > > > > > >>> Owner: Nobody
>> > > > > > > >>> Requestors: ying.lin at noaa.gov
>> > > > > > > >>> Status: new
>> > > > > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
>> > > > > > Ticket/Display.html?id=86995
>> > > > > > > >
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > >>> Hi Tracey,
>> > > > > > > >>>
>> > > > > > > >>> I haven't tried showing stats for each
threshold,
>> > you're
>> > > > way
>> > > > > > > ahead
>> > > > > > > >>> of me. Let me forward your inquiry to met_help.
>> > > > > > > >>>
>> > > > > > > >>> Dear met_help:
>> > > > > > > >>>
>> > > > > > > >>> I think Tracey is referring to the numbers
such as
>> that
>> > > on
>> > > > > top
>> > > > > > of
>> > > > > > > >>> this plot:
>> > > > > > > >>>
>> > > > > > > >>>
>> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
>> > > > > > > >>> 01809/intl_reg.201809.gif,
>> > > > > > > >>>
>> > > > > > > >>> for each verifying threshold.
>> > > > > > > >>>
>> > > > > > > >>> Thanks,
>> > > > > > > >>>
>> > > > > > > >>> Ying
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > >>> -------- Forwarded Message --------
>> > > > > > > >>> Subject: FBIAS east vs. west US
>> > > > > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
>> > > > > > > >>> From: Tracey Dorian - NOAA Affiliate <
>> > tracey.dorian at noaa.gov
>> > > >
>> > > > > > > >>> To: Ying Lin <ying.lin at noaa.gov>
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > >>> Hi Ying,
>> > > > > > > >>>
>> > > > > > > >>> I've created plots of FBIAS for east regions vs.
west
>> regions
>> > > and
>> > > > > > Geoff
>> > > > > > > >>> was wondering if I can show the number of events
for each
>> > > > > > threshold? I
>> > > > > > > >>> know METViewer has the "display number of stats"
option
>> > which I
>> > > > > > checked
>> > > > > > > >>> off, but it seems to show the same number for each
>> threshold.
>> > > Do
>> > > > > you
>> > > > > > > >>> know if there is a way to show the different number
of
>> stats
>> > > for
>> > > > > each
>> > > > > > > >>> threshold? We would probably expect the number of
events
>> to
>> > be
>> > > > > high
>> > > > > > > for
>> > > > > > > >>> the lower thresholds and get progressively smaller
as you
>> > move
>> > > > into
>> > > > > > the
>> > > > > > > >>> higher thresholds, right?
>> > > > > > > >>>
>> > > > > > > >>> Thanks,
>> > > > > > > >>> Tracey
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > >>>
>> > > > > > > > --
>> > > > > > > > Ying Lin
>> > > > > > > > NCEP/EMC/Verification, Post-processing and Product
>> Generation
>> > > > Branch
>> > > > > > > > NCWCP Cubicle No. 2015
>> > > > > > > > Ying.Lin at noaa.gov
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Tracey Dorian
>> > > > > > > NOAA/NCEP/EMC
>> > > > > > > IMSG contractor
>> > > > > > > 5830 University Research Court
>> > > > > > > College Park, MD 20740
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Tracey Dorian
>> > > > > NOAA/NCEP/EMC
>> > > > > IMSG contractor
>> > > > > 5830 University Research Court
>> > > > > College Park, MD 20740
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > > --
>> > > Tracey Dorian
>> > > NOAA/NCEP/EMC
>> > > IMSG contractor
>> > > 5830 University Research Court
>> > > College Park, MD 20740
>> > >
>> > >
>> >
>> >
>>
>> --
>> Tracey Dorian
>> NOAA/NCEP/EMC
>> IMSG contractor
>> 5830 University Research Court
>> College Park, MD 20740
>>
>>
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: Tatiana Burek
Time: Tue Sep 25 12:46:32 2018
Tracey,
You should be able to use 'mv_ylin_pcp_erly' database.
One of the tables was corrupted but it is fixed now.
Tatiana
On Tue Sep 25 11:37:29 2018, perry.shafran at noaa.gov wrote:
> Hi, Tracey, etc,
>
> I already have a helpdesk ticket into NCO as we need their
assistance
> to
> help clean up the mysql database. I have cc'd Jason in the hopes
that
> we
> can get this done quickly, as we might be dead in the water without
> that.
>
> Thanks!
>
> Perry
>
> On Tue, Sep 25, 2018 at 1:31 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Tracey,
> >
> > I talked to Tatiana, and the METviewer instance at EMC is in a bad
> > state.
> > She's been coordinating with Perry (cc'ed here). The mysql db
needs
> > to be
> > restarted before Perry can cleanup his databases.
> >
> > The problem you're seeing is almost certainly a result of this bad
> > state.
> > I'm hopeful that once these METviewer system issues are resolved,
> > you'll be
> > able to re-generate your plot.
> >
> > Sorry I don't have better news to report!
> >
> > John
> >
> > On Tue, Sep 25, 2018 at 11:25 AM tracey.dorian at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> >>
> >> Hi John,
> >>
> >> Right, those were the same errors that I got when I tried to
upload
> >> the
> >> XML
> >> file. Strange, thanks to both you and Tatiana for looking into
> >> this!
> >>
> >> Thanks,
> >> Tracey
> >>
> >> On Tue, Sep 25, 2018 at 11:48 AM John Halley Gotway via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> > I just tried to upload that XML myself. The upload appeared to
be
> >> > fine
> >> but
> >> > when I clicked "Generate Plot", I got this error:
> >> > error creating a plot or Proxy timeout: [object Object]
> >> >
> >> > So I tried to manually go remake that plot. I selected...
> >> > Database = mv_ylin_pcp_erly
> >> > Y1 Dependent Variable = APCP/24
> >> > But then the list of statistics is empty. I just says "N/A".
> >> >
> >> > It appears there's a larger problem here... either with this
> >> > particular
> >> > database or with the whole METviewer installation.
> >> >
> >> > I think we'll need to ask Tatiana to debug this further.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> >
> >> >
> >> > On Tue, Sep 25, 2018 at 7:34 AM tracey.dorian at noaa.gov via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
>
> >> > >
> >> > > Hi John,
> >> > >
> >> > > Sorry for the late reply! Could you please resend the XML
file
> >> > > that
> >> > > produced this plot attached? For some reason it is not
loading
> >> correctly
> >> > > into METViewer.
> >> > >
> >> > > Thanks,
> >> > > Tracey
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Sep 17, 2018 at 5:00 PM John Halley Gotway via RT <
> >> > > met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > > Tracey,
> >> > > >
> >> > > > I played around a bit. See attached. Here's the changes I
> >> > > > made...
> >> > > >
> >> > > > - Instead of plotting base rate on the "Y1" axis, I
selected
> >> > > > the
> >> "Y2"
> >> > tab
> >> > > > and defined it here.
> >> > > > - In the bottom-left corner of the GUI, I selected "+Add
> >> > > > Derived
> >> Curve"
> >> > > and
> >> > > > defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24
> >> > > > FBIAS".
> >> > > > - In the "Statistics" box, I selected "Aggregation
Statistics"
> >> > > > radio
> >> > > > button, "Contingency Table Count (CTC)" from the drop-down
> >> > > > list, and
> >> > then
> >> > > > set "Bootstrapping replications" to 1000. (I used 100 for
> >> > > > testing
> >> since
> >> > > > it's much faster!)
> >> > > > - In the "Series Formatting" section, I set "GFS APCP/24
> >> > > > BASER" to
> >> > light
> >> > > > gray, "Series Line Type" = "histogram-like" and "Line
Width" =
> >> > > > 30.
> >> > > > - In the "Series Formatting" section for the "DIFF" line, I
> >> > > > selected
> >> > > "Show
> >> > > > Significance" to true.
> >> > > > Anywhere the confidence interval of the difference line
is
> >> > > > bold,
> >> > that
> >> > > > means the difference is statistically significant.
> >> > > > - In the "Legend Text" column I added some meaningful
labels.
> >> > > > - In the "Formatting" tab I set "plot script commands" as
> >> > > > "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
> >> > > > That'll draw horizontal gray dashed reference lines at 0
> >> > > > and 1.
> >> > > >
> >> > > > Remember that this is currently only plotting results for
> >> "G218/SEC".
> >> > Be
> >> > > > sure to change that to the full set of regions you want to
> >> > > > use.
> >> > > >
> >> > > > John
> >> > > >
> >> > > > On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov via
RT
> >> > > > <
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > >
> >> > > > > <URL:
> >> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> >> > > > >
> >> > > > > Hi John,
> >> > > > >
> >> > > > > Yes, that's what I was looking for! How were you able to
> >> > > > > plot the
> >> > base
> >> > > > > rate using bars instead of lines? Also, how would you
> >> > > > > suggest
> >> adding
> >> > > > > statistical significance to the plot?
> >> > > > >
> >> > > > > Thanks for sending your XML file!
> >> > > > >
> >> > > > > -Tracey
> >> > > > >
> >> > > > > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via
RT <
> >> > > > > met_help at ucar.edu> wrote:
> >> > > > >
> >> > > > > > Tracey,
> >> > > > > >
> >> > > > > > Thanks for sending your XML file. I didn't see any
> >> > > > > > reference to
> >> > > > "baser"
> >> > > > > in
> >> > > > > > the XML you sent. So I added it as a second statistic
to
> >> > > > > > be
> >> > plotted.
> >> > > > > >
> >> > > > > > Since you didn't define a VX_MASK to be used, METviewer
> >> > > > > > tried to
> >> > use
> >> > > > all
> >> > > > > > available in the database, which likely made it run
> >> > > > > > slowly.
> >> > > > > >
> >> > > > > > For testing, I set VX_MASK = G218/SEC added a line for
> >> > > > > > BASER and
> >> > > > attached
> >> > > > > > the result.
> >> > > > > >
> >> > > > > > Is that what you're looking for?
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > John
> >> > > > > >
> >> > > > > > On Mon, Sep 17, 2018 at 1:32 PM tracey.dorian at noaa.gov
via
> >> > > > > > RT <
> >> > > > > > met_help at ucar.edu> wrote:
> >> > > > > >
> >> > > > > > >
> >> > > > > > > <URL:
> >> > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
> >> >
> >> > > > > > >
> >> > > > > > > Hi John,
> >> > > > > > >
> >> > > > > > > Thanks for much for your quick reply! I am trying to
> >> > > > > > > add the
> >> > BASER
> >> > > > > field
> >> > > > > > > to my plots, but I'm getting errors, specifically a
> >> > > > > > > proxy
> >> > timeout.
> >> > > > Do
> >> > > > > > you
> >> > > > > > > know why this is happening? I've attached my XML
file
> >> > > > > > > to this
> >> > > email.
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > Tracey
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin
> >> > > > > > > <ying.lin at noaa.gov>
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > Thanks very much John!
> >> > > > > > > >
> >> > > > > > > > Tracey, did this answer your question? I have an
> >> RFC/Dataflow
> >> > > > crisis
> >> > > > > > > > today so haven't tested the answer. If you have
> >> > > > > > > > further
> >> > > questions,
> >> > > > > > > please
> >> > > > > > > > follow-up on this met_help ticket (and cc to me -
sure
> >> > > > > > > > would
> >> > like
> >> > > > to
> >> > > > > > know
> >> > > > > > > > too). Thanks.
> >> > > > > > > >
> >> > > > > > > > Ying
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > >> Ying and Tracey,
> >> > > > > > > >>
> >> > > > > > > >> I see that you'd like to include in your METviewer
> >> > > > > > > >> plot
> >> > > > information
> >> > > > > > > about
> >> > > > > > > >> the event frequency. You tried using the "Display
> >> > > > > > > >> Number
> >> of
> >> > > > Stats"
> >> > > > > > > option
> >> > > > > > > >> but that that didn't do exactly what you want.
Those
> >> numbers
> >> > > stay
> >> > > > > > > >> constant
> >> > > > > > > >> for all the thresholds.
> >> > > > > > > >>
> >> > > > > > > >> The "NStats" values listed across the top indicate
> >> > > > > > > >> the
> >> sample
> >> > > > size.
> >> > > > > > > They
> >> > > > > > > >> are computed as the sum of the number of cases
that
> >> > > > > > > >> are
> >> > included
> >> > > > > > across
> >> > > > > > > >> all
> >> > > > > > > >> the series (i.e. lines on the plot) for each
> >> > > > > > > >> independent
> >> > > variable
> >> > > > > > value.
> >> > > > > > > >> This is not the number of events meeting each
> >> > > > > > > >> threshold
> >> > > > criteria...
> >> > > > > > > it's a
> >> > > > > > > >> count of the number of "cases" included. In the
> >> > > > > > > >> sample
> >> > plots
> >> > > > you
> >> > > > > > > sent,
> >> > > > > > > >> you have NStats = 33948. Since you didn't send
the
> >> > > corresponding
> >> > > > > XML,
> >> > > > > > > >> it's
> >> > > > > > > >> difficult for me to exactly replicate them.
> >> > > > > > > >> Basically
> >> though,
> >> > > the
> >> > > > > > logic
> >> > > > > > > >> is:
> >> > > > > > > >> - For a particular independent variable value,
> >> > > > > > > >> >1.0" for
> >> > > > example,
> >> > > > > > you
> >> > > > > > > >> have 2 lines on the plot.
> >> > > > > > > >> - You have many model initializations included,
> >> > > > > > > >> perhaps
> >> > 397.
> >> > > > > > > >> - You have multiple lead times included. For
APCP/24
> >> > > probably
> >> > > > > 24,
> >> > > > > > > 48,
> >> > > > > > > >> 72, 96... perhaps about 8 of them.
> >> > > > > > > >> - You're including multiple subregions in each
> >> > > > > > > >> plot...
> >> for
> >> > > > > example,
> >> > > > > > > the
> >> > > > > > > >> EAST consists of 6 subregions.
> >> > > > > > > >>
> >> > > > > > > >> Multiplying all those numbers together... 2 * 397
* 8
> >> > > > > > > >> * 6 =
> >> > > 38112
> >> > > > > gets
> >> > > > > > > you
> >> > > > > > > >> somewhere near the actual number of 33948 we're
> >> > > > > > > >> seeing.
> >> > > > > > > >>
> >> > > > > > > >> If you want to see the frequency at which the
event
> >> > > > > > > >> is
> >> > actually
> >> > > > > > > occurring,
> >> > > > > > > >> I'd suggest plotting the base rate statistic,
called
> >> > > > > > > >> BASER
> >> in
> >> > > > > > METviewer.
> >> > > > > > > >> That's exactly what you want... a number between 0
> >> > > > > > > >> and 1
> >> > > > > representing
> >> > > > > > > the
> >> > > > > > > >> event frequency in the observation data. The
BASER
> >> > > > > > > >> could
> >> be
> >> > > shown
> >> > > > > > > >> separately for the GFS and FV3GFS... but as long
as
> >> > > > > > > >> you're
> >> > using
> >> > > > the
> >> > > > > > > exact
> >> > > > > > > >> same cases, they should be identical. We often
plot
> >> > > > > > > >> the
> >> BASER
> >> > > on
> >> > > > > the
> >> > > > > > Y2
> >> > > > > > > >> axis so that we can control the range of the Y2
axis,
> >> > > > > > > >> but I
> >> > ran
> >> > > > into
> >> > > > > > > some
> >> > > > > > > >> problems doing that.
> >> > > > > > > >>
> >> > > > > > > >> Instead, I put it on the Y1 axis with FBIAS but
> >> > > > > > > >> plotted it
> >> > like
> >> > > a
> >> > > > > > > >> histogram
> >> > > > > > > >> with bars.
> >> > > > > > > >>
> >> > > > > > > >> I made the attached plot/xml using the mv_ylin_pcp
> >> database...
> >> > > > > > > >> - It includes 4 lines where FV3GFS is blue and GFS
is
> >> red.
> >> > > > > > > >> - The solid lines are for the EAST region and
dashed
> >> lines
> >> > > are
> >> > > > > for
> >> > > > > > > the
> >> > > > > > > >> WEST region. Take a close look at the Series
> >> > > > > > > >> definition
> >> > > > section. I
> >> > > > > > > >> defined the EAST as a "Group" of 6 subregions and
the
> >> > > > > > > >> WEST
> >> as
> >> > a
> >> > > > > > "Group"
> >> > > > > > > of
> >> > > > > > > >> 8 subregions.
> >> > > > > > > >> - The gray bars show the base rate for the EAST
> >> > > > > > > >> and WEST
> >> > > > regions.
> >> > > > > > > >> Since
> >> > > > > > > >> I only want to plot 2 of the 4, I chose "Yes" for
the
> >> "Hide"
> >> > > > column
> >> > > > > > for
> >> > > > > > > >> the
> >> > > > > > > >> 2 lines I wanted to ignore.
> >> > > > > > > >> - In the "Series Formatting" I used the "Legend
> >> > > > > > > >> Text"
> >> > column
> >> > > to
> >> > > > > > > define
> >> > > > > > > >> the legend.
> >> > > > > > > >>
> >> > > > > > > >> Granted, this plot was pretty tedious to make
> >> > > > > > > >> manually
> >> because
> >> > > of
> >> > > > > the
> >> > > > > > > >> masking region groups, but hopefully it includes
all
> >> > > > > > > >> the
> >> info
> >> > > you
> >> > > > > > need.
> >> > > > > > > >>
> >> > > > > > > >> And note that statistical significance is not
> >> > > > > > > >> currently
> >> > included
> >> > > > in
> >> > > > > > this
> >> > > > > > > >> plot, but it could be added.
> >> > > > > > > >>
> >> > > > > > > >> Thanks,
> >> > > > > > > >> John
> >> > > > > > > >>
> >> > > > > > > >>
> >> > > > > > > >>
> >> > > > > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT <
> >> > > > met_help at ucar.edu>
> >> > > > > > > >> wrote:
> >> > > > > > > >>
> >> > > > > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was acted
> >> > > > > > > >> upon.
> >> > > > > > > >>> Transaction: Ticket created by ying.lin at noaa.gov
> >> > > > > > > >>> Queue: met_help
> >> > > > > > > >>> Subject: MV CTC plot: count of data points
for
> >> > > > > > > >>> each
> >> > > > threshold
> >> > > > > > > (Fwd:
> >> > > > > > > >>> FBIAS east vs. west US
> >> > > > > > > >>> Owner: Nobody
> >> > > > > > > >>> Requestors: ying.lin at noaa.gov
> >> > > > > > > >>> Status: new
> >> > > > > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> >> > > > > > Ticket/Display.html?id=86995
> >> > > > > > > >
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>> Hi Tracey,
> >> > > > > > > >>>
> >> > > > > > > >>> I haven't tried showing stats for each threshold,
> >> > you're
> >> > > > way
> >> > > > > > > ahead
> >> > > > > > > >>> of me. Let me forward your inquiry to met_help.
> >> > > > > > > >>>
> >> > > > > > > >>> Dear met_help:
> >> > > > > > > >>>
> >> > > > > > > >>> I think Tracey is referring to the numbers such
as
> >> that
> >> > > on
> >> > > > > top
> >> > > > > > of
> >> > > > > > > >>> this plot:
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> >> > > > > > > >>> 01809/intl_reg.201809.gif,
> >> > > > > > > >>>
> >> > > > > > > >>> for each verifying threshold.
> >> > > > > > > >>>
> >> > > > > > > >>> Thanks,
> >> > > > > > > >>>
> >> > > > > > > >>> Ying
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>> -------- Forwarded Message --------
> >> > > > > > > >>> Subject: FBIAS east vs. west US
> >> > > > > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> >> > > > > > > >>> From: Tracey Dorian - NOAA Affiliate <
> >> > tracey.dorian at noaa.gov
> >> > > >
> >> > > > > > > >>> To: Ying Lin <ying.lin at noaa.gov>
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>> Hi Ying,
> >> > > > > > > >>>
> >> > > > > > > >>> I've created plots of FBIAS for east regions vs.
> >> > > > > > > >>> west
> >> regions
> >> > > and
> >> > > > > > Geoff
> >> > > > > > > >>> was wondering if I can show the number of events
for
> >> > > > > > > >>> each
> >> > > > > > threshold? I
> >> > > > > > > >>> know METViewer has the "display number of stats"
> >> > > > > > > >>> option
> >> > which I
> >> > > > > > checked
> >> > > > > > > >>> off, but it seems to show the same number for
each
> >> threshold.
> >> > > Do
> >> > > > > you
> >> > > > > > > >>> know if there is a way to show the different
number
> >> > > > > > > >>> of
> >> stats
> >> > > for
> >> > > > > each
> >> > > > > > > >>> threshold? We would probably expect the number
of
> >> > > > > > > >>> events
> >> to
> >> > be
> >> > > > > high
> >> > > > > > > for
> >> > > > > > > >>> the lower thresholds and get progressively
smaller
> >> > > > > > > >>> as you
> >> > move
> >> > > > into
> >> > > > > > the
> >> > > > > > > >>> higher thresholds, right?
> >> > > > > > > >>>
> >> > > > > > > >>> Thanks,
> >> > > > > > > >>> Tracey
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > >>>
> >> > > > > > > > --
> >> > > > > > > > Ying Lin
> >> > > > > > > > NCEP/EMC/Verification, Post-processing and Product
> >> Generation
> >> > > > Branch
> >> > > > > > > > NCWCP Cubicle No. 2015
> >> > > > > > > > Ying.Lin at noaa.gov
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > Tracey Dorian
> >> > > > > > > NOAA/NCEP/EMC
> >> > > > > > > IMSG contractor
> >> > > > > > > 5830 University Research Court
> >> > > > > > > College Park, MD 20740
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Tracey Dorian
> >> > > > > NOAA/NCEP/EMC
> >> > > > > IMSG contractor
> >> > > > > 5830 University Research Court
> >> > > > > College Park, MD 20740
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > > --
> >> > > Tracey Dorian
> >> > > NOAA/NCEP/EMC
> >> > > IMSG contractor
> >> > > 5830 University Research Court
> >> > > College Park, MD 20740
> >> > >
> >> > >
> >> >
> >> >
> >>
> >> --
> >> Tracey Dorian
> >> NOAA/NCEP/EMC
> >> IMSG contractor
> >> 5830 University Research Court
> >> College Park, MD 20740
> >>
> >>
------------------------------------------------
Subject: MV CTC plot: count of data points for each threshold (Fwd: FBIAS east vs. west US
From: tracey.dorian at noaa.gov
Time: Tue Sep 25 12:54:23 2018
Hi Tatiana,
It worked! Thanks for fixing the corrupted table so quickly!
Sincerely,
Tracey
On Tue, Sep 25, 2018 at 2:46 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Tracey,
> You should be able to use 'mv_ylin_pcp_erly' database.
> One of the tables was corrupted but it is fixed now.
>
> Tatiana
> On Tue Sep 25 11:37:29 2018, perry.shafran at noaa.gov wrote:
> > Hi, Tracey, etc,
> >
> > I already have a helpdesk ticket into NCO as we need their
assistance
> > to
> > help clean up the mysql database. I have cc'd Jason in the hopes
that
> > we
> > can get this done quickly, as we might be dead in the water
without
> > that.
> >
> > Thanks!
> >
> > Perry
> >
> > On Tue, Sep 25, 2018 at 1:31 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Tracey,
> > >
> > > I talked to Tatiana, and the METviewer instance at EMC is in a
bad
> > > state.
> > > She's been coordinating with Perry (cc'ed here). The mysql db
needs
> > > to be
> > > restarted before Perry can cleanup his databases.
> > >
> > > The problem you're seeing is almost certainly a result of this
bad
> > > state.
> > > I'm hopeful that once these METviewer system issues are
resolved,
> > > you'll be
> > > able to re-generate your plot.
> > >
> > > Sorry I don't have better news to report!
> > >
> > > John
> > >
> > > On Tue, Sep 25, 2018 at 11:25 AM tracey.dorian at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> > >>
> > >> Hi John,
> > >>
> > >> Right, those were the same errors that I got when I tried to
upload
> > >> the
> > >> XML
> > >> file. Strange, thanks to both you and Tatiana for looking into
> > >> this!
> > >>
> > >> Thanks,
> > >> Tracey
> > >>
> > >> On Tue, Sep 25, 2018 at 11:48 AM John Halley Gotway via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> > I just tried to upload that XML myself. The upload appeared
to be
> > >> > fine
> > >> but
> > >> > when I clicked "Generate Plot", I got this error:
> > >> > error creating a plot or Proxy timeout: [object Object]
> > >> >
> > >> > So I tried to manually go remake that plot. I selected...
> > >> > Database = mv_ylin_pcp_erly
> > >> > Y1 Dependent Variable = APCP/24
> > >> > But then the list of statistics is empty. I just says "N/A".
> > >> >
> > >> > It appears there's a larger problem here... either with this
> > >> > particular
> > >> > database or with the whole METviewer installation.
> > >> >
> > >> > I think we'll need to ask Tatiana to debug this further.
> > >> >
> > >> > Thanks,
> > >> > John
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Sep 25, 2018 at 7:34 AM tracey.dorian at noaa.gov via RT
<
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995 >
> > >> > >
> > >> > > Hi John,
> > >> > >
> > >> > > Sorry for the late reply! Could you please resend the XML
file
> > >> > > that
> > >> > > produced this plot attached? For some reason it is not
loading
> > >> correctly
> > >> > > into METViewer.
> > >> > >
> > >> > > Thanks,
> > >> > > Tracey
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Sep 17, 2018 at 5:00 PM John Halley Gotway via RT <
> > >> > > met_help at ucar.edu>
> > >> > > wrote:
> > >> > >
> > >> > > > Tracey,
> > >> > > >
> > >> > > > I played around a bit. See attached. Here's the changes
I
> > >> > > > made...
> > >> > > >
> > >> > > > - Instead of plotting base rate on the "Y1" axis, I
selected
> > >> > > > the
> > >> "Y2"
> > >> > tab
> > >> > > > and defined it here.
> > >> > > > - In the bottom-left corner of the GUI, I selected "+Add
> > >> > > > Derived
> > >> Curve"
> > >> > > and
> > >> > > > defined a "Diff" of "FV3GFS APCP/24 FBIAS" - "GFS APCP/24
> > >> > > > FBIAS".
> > >> > > > - In the "Statistics" box, I selected "Aggregation
Statistics"
> > >> > > > radio
> > >> > > > button, "Contingency Table Count (CTC)" from the drop-
down
> > >> > > > list, and
> > >> > then
> > >> > > > set "Bootstrapping replications" to 1000. (I used 100 for
> > >> > > > testing
> > >> since
> > >> > > > it's much faster!)
> > >> > > > - In the "Series Formatting" section, I set "GFS APCP/24
> > >> > > > BASER" to
> > >> > light
> > >> > > > gray, "Series Line Type" = "histogram-like" and "Line
Width" =
> > >> > > > 30.
> > >> > > > - In the "Series Formatting" section for the "DIFF" line,
I
> > >> > > > selected
> > >> > > "Show
> > >> > > > Significance" to true.
> > >> > > > Anywhere the confidence interval of the difference
line is
> > >> > > > bold,
> > >> > that
> > >> > > > means the difference is statistically significant.
> > >> > > > - In the "Legend Text" column I added some meaningful
labels.
> > >> > > > - In the "Formatting" tab I set "plot script commands" as
> > >> > > > "abline(h=c(0,1),col="gray",lty=2,lwd=2)"
> > >> > > > That'll draw horizontal gray dashed reference lines at
0
> > >> > > > and 1.
> > >> > > >
> > >> > > > Remember that this is currently only plotting results for
> > >> "G218/SEC".
> > >> > Be
> > >> > > > sure to change that to the full set of regions you want
to
> > >> > > > use.
> > >> > > >
> > >> > > > John
> > >> > > >
> > >> > > > On Mon, Sep 17, 2018 at 2:27 PM tracey.dorian at noaa.gov
via RT
> > >> > > > <
> > >> > > > met_help at ucar.edu> wrote:
> > >> > > >
> > >> > > > >
> > >> > > > > <URL:
> > >> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
>
> > >> > > > >
> > >> > > > > Hi John,
> > >> > > > >
> > >> > > > > Yes, that's what I was looking for! How were you able
to
> > >> > > > > plot the
> > >> > base
> > >> > > > > rate using bars instead of lines? Also, how would you
> > >> > > > > suggest
> > >> adding
> > >> > > > > statistical significance to the plot?
> > >> > > > >
> > >> > > > > Thanks for sending your XML file!
> > >> > > > >
> > >> > > > > -Tracey
> > >> > > > >
> > >> > > > > On Mon, Sep 17, 2018 at 4:02 PM, John Halley Gotway via
RT <
> > >> > > > > met_help at ucar.edu> wrote:
> > >> > > > >
> > >> > > > > > Tracey,
> > >> > > > > >
> > >> > > > > > Thanks for sending your XML file. I didn't see any
> > >> > > > > > reference to
> > >> > > > "baser"
> > >> > > > > in
> > >> > > > > > the XML you sent. So I added it as a second
statistic to
> > >> > > > > > be
> > >> > plotted.
> > >> > > > > >
> > >> > > > > > Since you didn't define a VX_MASK to be used,
METviewer
> > >> > > > > > tried to
> > >> > use
> > >> > > > all
> > >> > > > > > available in the database, which likely made it run
> > >> > > > > > slowly.
> > >> > > > > >
> > >> > > > > > For testing, I set VX_MASK = G218/SEC added a line
for
> > >> > > > > > BASER and
> > >> > > > attached
> > >> > > > > > the result.
> > >> > > > > >
> > >> > > > > > Is that what you're looking for?
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > John
> > >> > > > > >
> > >> > > > > > On Mon, Sep 17, 2018 at 1:32 PM
tracey.dorian at noaa.gov via
> > >> > > > > > RT <
> > >> > > > > > met_help at ucar.edu> wrote:
> > >> > > > > >
> > >> > > > > > >
> > >> > > > > > > <URL:
> > >> > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86995
> > >> >
> > >> > > > > > >
> > >> > > > > > > Hi John,
> > >> > > > > > >
> > >> > > > > > > Thanks for much for your quick reply! I am trying
to
> > >> > > > > > > add the
> > >> > BASER
> > >> > > > > field
> > >> > > > > > > to my plots, but I'm getting errors, specifically a
> > >> > > > > > > proxy
> > >> > timeout.
> > >> > > > Do
> > >> > > > > > you
> > >> > > > > > > know why this is happening? I've attached my XML
file
> > >> > > > > > > to this
> > >> > > email.
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Tracey
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Mon, Sep 17, 2018 at 1:32 PM, Ying Lin
> > >> > > > > > > <ying.lin at noaa.gov>
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Thanks very much John!
> > >> > > > > > > >
> > >> > > > > > > > Tracey, did this answer your question? I have an
> > >> RFC/Dataflow
> > >> > > > crisis
> > >> > > > > > > > today so haven't tested the answer. If you have
> > >> > > > > > > > further
> > >> > > questions,
> > >> > > > > > > please
> > >> > > > > > > > follow-up on this met_help ticket (and cc to me -
sure
> > >> > > > > > > > would
> > >> > like
> > >> > > > to
> > >> > > > > > know
> > >> > > > > > > > too). Thanks.
> > >> > > > > > > >
> > >> > > > > > > > Ying
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On 09/17/2018 01:02 PM, John Halley Gotway via RT
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > >> Ying and Tracey,
> > >> > > > > > > >>
> > >> > > > > > > >> I see that you'd like to include in your
METviewer
> > >> > > > > > > >> plot
> > >> > > > information
> > >> > > > > > > about
> > >> > > > > > > >> the event frequency. You tried using the
"Display
> > >> > > > > > > >> Number
> > >> of
> > >> > > > Stats"
> > >> > > > > > > option
> > >> > > > > > > >> but that that didn't do exactly what you want.
Those
> > >> numbers
> > >> > > stay
> > >> > > > > > > >> constant
> > >> > > > > > > >> for all the thresholds.
> > >> > > > > > > >>
> > >> > > > > > > >> The "NStats" values listed across the top
indicate
> > >> > > > > > > >> the
> > >> sample
> > >> > > > size.
> > >> > > > > > > They
> > >> > > > > > > >> are computed as the sum of the number of cases
that
> > >> > > > > > > >> are
> > >> > included
> > >> > > > > > across
> > >> > > > > > > >> all
> > >> > > > > > > >> the series (i.e. lines on the plot) for each
> > >> > > > > > > >> independent
> > >> > > variable
> > >> > > > > > value.
> > >> > > > > > > >> This is not the number of events meeting each
> > >> > > > > > > >> threshold
> > >> > > > criteria...
> > >> > > > > > > it's a
> > >> > > > > > > >> count of the number of "cases" included. In
the
> > >> > > > > > > >> sample
> > >> > plots
> > >> > > > you
> > >> > > > > > > sent,
> > >> > > > > > > >> you have NStats = 33948. Since you didn't send
the
> > >> > > corresponding
> > >> > > > > XML,
> > >> > > > > > > >> it's
> > >> > > > > > > >> difficult for me to exactly replicate them.
> > >> > > > > > > >> Basically
> > >> though,
> > >> > > the
> > >> > > > > > logic
> > >> > > > > > > >> is:
> > >> > > > > > > >> - For a particular independent variable
value,
> > >> > > > > > > >> >1.0" for
> > >> > > > example,
> > >> > > > > > you
> > >> > > > > > > >> have 2 lines on the plot.
> > >> > > > > > > >> - You have many model initializations
included,
> > >> > > > > > > >> perhaps
> > >> > 397.
> > >> > > > > > > >> - You have multiple lead times included. For
APCP/24
> > >> > > probably
> > >> > > > > 24,
> > >> > > > > > > 48,
> > >> > > > > > > >> 72, 96... perhaps about 8 of them.
> > >> > > > > > > >> - You're including multiple subregions in
each
> > >> > > > > > > >> plot...
> > >> for
> > >> > > > > example,
> > >> > > > > > > the
> > >> > > > > > > >> EAST consists of 6 subregions.
> > >> > > > > > > >>
> > >> > > > > > > >> Multiplying all those numbers together... 2 *
397 * 8
> > >> > > > > > > >> * 6 =
> > >> > > 38112
> > >> > > > > gets
> > >> > > > > > > you
> > >> > > > > > > >> somewhere near the actual number of 33948 we're
> > >> > > > > > > >> seeing.
> > >> > > > > > > >>
> > >> > > > > > > >> If you want to see the frequency at which the
event
> > >> > > > > > > >> is
> > >> > actually
> > >> > > > > > > occurring,
> > >> > > > > > > >> I'd suggest plotting the base rate statistic,
called
> > >> > > > > > > >> BASER
> > >> in
> > >> > > > > > METviewer.
> > >> > > > > > > >> That's exactly what you want... a number between
0
> > >> > > > > > > >> and 1
> > >> > > > > representing
> > >> > > > > > > the
> > >> > > > > > > >> event frequency in the observation data. The
BASER
> > >> > > > > > > >> could
> > >> be
> > >> > > shown
> > >> > > > > > > >> separately for the GFS and FV3GFS... but as long
as
> > >> > > > > > > >> you're
> > >> > using
> > >> > > > the
> > >> > > > > > > exact
> > >> > > > > > > >> same cases, they should be identical. We often
plot
> > >> > > > > > > >> the
> > >> BASER
> > >> > > on
> > >> > > > > the
> > >> > > > > > Y2
> > >> > > > > > > >> axis so that we can control the range of the Y2
axis,
> > >> > > > > > > >> but I
> > >> > ran
> > >> > > > into
> > >> > > > > > > some
> > >> > > > > > > >> problems doing that.
> > >> > > > > > > >>
> > >> > > > > > > >> Instead, I put it on the Y1 axis with FBIAS but
> > >> > > > > > > >> plotted it
> > >> > like
> > >> > > a
> > >> > > > > > > >> histogram
> > >> > > > > > > >> with bars.
> > >> > > > > > > >>
> > >> > > > > > > >> I made the attached plot/xml using the
mv_ylin_pcp
> > >> database...
> > >> > > > > > > >> - It includes 4 lines where FV3GFS is blue and
GFS is
> > >> red.
> > >> > > > > > > >> - The solid lines are for the EAST region and
dashed
> > >> lines
> > >> > > are
> > >> > > > > for
> > >> > > > > > > the
> > >> > > > > > > >> WEST region. Take a close look at the Series
> > >> > > > > > > >> definition
> > >> > > > section. I
> > >> > > > > > > >> defined the EAST as a "Group" of 6 subregions
and the
> > >> > > > > > > >> WEST
> > >> as
> > >> > a
> > >> > > > > > "Group"
> > >> > > > > > > of
> > >> > > > > > > >> 8 subregions.
> > >> > > > > > > >> - The gray bars show the base rate for the
EAST
> > >> > > > > > > >> and WEST
> > >> > > > regions.
> > >> > > > > > > >> Since
> > >> > > > > > > >> I only want to plot 2 of the 4, I chose "Yes"
for the
> > >> "Hide"
> > >> > > > column
> > >> > > > > > for
> > >> > > > > > > >> the
> > >> > > > > > > >> 2 lines I wanted to ignore.
> > >> > > > > > > >> - In the "Series Formatting" I used the
"Legend
> > >> > > > > > > >> Text"
> > >> > column
> > >> > > to
> > >> > > > > > > define
> > >> > > > > > > >> the legend.
> > >> > > > > > > >>
> > >> > > > > > > >> Granted, this plot was pretty tedious to make
> > >> > > > > > > >> manually
> > >> because
> > >> > > of
> > >> > > > > the
> > >> > > > > > > >> masking region groups, but hopefully it includes
all
> > >> > > > > > > >> the
> > >> info
> > >> > > you
> > >> > > > > > need.
> > >> > > > > > > >>
> > >> > > > > > > >> And note that statistical significance is not
> > >> > > > > > > >> currently
> > >> > included
> > >> > > > in
> > >> > > > > > this
> > >> > > > > > > >> plot, but it could be added.
> > >> > > > > > > >>
> > >> > > > > > > >> Thanks,
> > >> > > > > > > >> John
> > >> > > > > > > >>
> > >> > > > > > > >>
> > >> > > > > > > >>
> > >> > > > > > > >> On Mon, Sep 17, 2018 at 8:47 AM Ying Lin via RT
<
> > >> > > > met_help at ucar.edu>
> > >> > > > > > > >> wrote:
> > >> > > > > > > >>
> > >> > > > > > > >> Mon Sep 17 08:45:57 2018: Request 86995 was
acted
> > >> > > > > > > >> upon.
> > >> > > > > > > >>> Transaction: Ticket created by
ying.lin at noaa.gov
> > >> > > > > > > >>> Queue: met_help
> > >> > > > > > > >>> Subject: MV CTC plot: count of data
points for
> > >> > > > > > > >>> each
> > >> > > > threshold
> > >> > > > > > > (Fwd:
> > >> > > > > > > >>> FBIAS east vs. west US
> > >> > > > > > > >>> Owner: Nobody
> > >> > > > > > > >>> Requestors: ying.lin at noaa.gov
> > >> > > > > > > >>> Status: new
> > >> > > > > > > >>> Ticket <URL: https://rt.rap.ucar.edu/rt/
> > >> > > > > > Ticket/Display.html?id=86995
> > >> > > > > > > >
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> > > > > > > >>> Hi Tracey,
> > >> > > > > > > >>>
> > >> > > > > > > >>> I haven't tried showing stats for each
threshold,
> > >> > you're
> > >> > > > way
> > >> > > > > > > ahead
> > >> > > > > > > >>> of me. Let me forward your inquiry to
met_help.
> > >> > > > > > > >>>
> > >> > > > > > > >>> Dear met_help:
> > >> > > > > > > >>>
> > >> > > > > > > >>> I think Tracey is referring to the numbers such
as
> > >> that
> > >> > > on
> > >> > > > > top
> > >> > > > > > of
> > >> > > > > > > >>> this plot:
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2018/2
> > >> > > > > > > >>> 01809/intl_reg.201809.gif,
> > >> > > > > > > >>>
> > >> > > > > > > >>> for each verifying threshold.
> > >> > > > > > > >>>
> > >> > > > > > > >>> Thanks,
> > >> > > > > > > >>>
> > >> > > > > > > >>> Ying
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> > > > > > > >>> -------- Forwarded Message --------
> > >> > > > > > > >>> Subject: FBIAS east vs. west US
> > >> > > > > > > >>> Date: Mon, 17 Sep 2018 09:49:51 -0400
> > >> > > > > > > >>> From: Tracey Dorian - NOAA Affiliate <
> > >> > tracey.dorian at noaa.gov
> > >> > > >
> > >> > > > > > > >>> To: Ying Lin <ying.lin at noaa.gov>
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> > > > > > > >>> Hi Ying,
> > >> > > > > > > >>>
> > >> > > > > > > >>> I've created plots of FBIAS for east regions
vs.
> > >> > > > > > > >>> west
> > >> regions
> > >> > > and
> > >> > > > > > Geoff
> > >> > > > > > > >>> was wondering if I can show the number of
events for
> > >> > > > > > > >>> each
> > >> > > > > > threshold? I
> > >> > > > > > > >>> know METViewer has the "display number of
stats"
> > >> > > > > > > >>> option
> > >> > which I
> > >> > > > > > checked
> > >> > > > > > > >>> off, but it seems to show the same number for
each
> > >> threshold.
> > >> > > Do
> > >> > > > > you
> > >> > > > > > > >>> know if there is a way to show the different
number
> > >> > > > > > > >>> of
> > >> stats
> > >> > > for
> > >> > > > > each
> > >> > > > > > > >>> threshold? We would probably expect the number
of
> > >> > > > > > > >>> events
> > >> to
> > >> > be
> > >> > > > > high
> > >> > > > > > > for
> > >> > > > > > > >>> the lower thresholds and get progressively
smaller
> > >> > > > > > > >>> as you
> > >> > move
> > >> > > > into
> > >> > > > > > the
> > >> > > > > > > >>> higher thresholds, right?
> > >> > > > > > > >>>
> > >> > > > > > > >>> Thanks,
> > >> > > > > > > >>> Tracey
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> > > > > > > >>>
> > >> > > > > > > > --
> > >> > > > > > > > Ying Lin
> > >> > > > > > > > NCEP/EMC/Verification, Post-processing and
Product
> > >> Generation
> > >> > > > Branch
> > >> > > > > > > > NCWCP Cubicle No. 2015
> > >> > > > > > > > Ying.Lin at noaa.gov
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > Tracey Dorian
> > >> > > > > > > NOAA/NCEP/EMC
> > >> > > > > > > IMSG contractor
> > >> > > > > > > 5830 University Research Court
> > >> > > > > > > College Park, MD 20740
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > Tracey Dorian
> > >> > > > > NOAA/NCEP/EMC
> > >> > > > > IMSG contractor
> > >> > > > > 5830 University Research Court
> > >> > > > > College Park, MD 20740
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > > --
> > >> > > Tracey Dorian
> > >> > > NOAA/NCEP/EMC
> > >> > > IMSG contractor
> > >> > > 5830 University Research Court
> > >> > > College Park, MD 20740
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >> --
> > >> Tracey Dorian
> > >> NOAA/NCEP/EMC
> > >> IMSG contractor
> > >> 5830 University Research Court
> > >> College Park, MD 20740
> > >>
> > >>
>
>
>
>
--
Tracey Dorian
NOAA/NCEP/EMC
IMSG contractor
5830 University Research Court
College Park, MD 20740
------------------------------------------------
More information about the Met_help
mailing list