[Met_help] [rt.rap.ucar.edu #95555] History for stat_analysis FSS
John Halley Gotway via RT
met_help at ucar.edu
Wed Oct 21 11:30:10 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi MET help!
I am trying to aggregate the fractions skill score (FSS) for multiple days,
however MET runs into a problem when FSS=NA for one or more days. If
FSS=NA for one day, FSS=NA for the aggregate.
By definition, FSS = 1 - FBS/FBS_worst, where FBS is fractions Brier score
and FBS_worst is the worst-possible FBS.
For very high rainfall rates, you'll sometimes have no forecast and
observed events, and FBS_worst = 0 so FSS is undefined.
Even though FSS is undefined for this one day, it should not corrupt the
aggregate FSS. It's like adding a whole lot of correct nulls. Seems like
MET handles the aggregate FBS okay but not FSS.
The following Stat Analysis command and attached stat file illustrate the
issue.
stat_analysis -lookin t.stat -job aggregate -line_type NBRCNT
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: stat_analysis FSS
From: Julie Prestopnik
Time: Tue Jun 09 13:47:06 2020
Hi Dave.
I have assigned this ticket to John. Please allow a few business days
for
a response.
Julie
On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT
<met_help at ucar.edu>
wrote:
>
> Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> Transaction: Ticket created by ahijevyc
> Queue: met_help
> Subject: stat_analysis FSS
> Owner: Nobody
> Requestors: ahijevyc at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>
>
> Hi MET help!
> I am trying to aggregate the fractions skill score (FSS) for
multiple days,
> however MET runs into a problem when FSS=NA for one or more days.
If
> FSS=NA for one day, FSS=NA for the aggregate.
>
> By definition, FSS = 1 - FBS/FBS_worst, where FBS is fractions Brier
score
> and FBS_worst is the worst-possible FBS.
>
> For very high rainfall rates, you'll sometimes have no forecast and
> observed events, and FBS_worst = 0 so FSS is undefined.
>
> Even though FSS is undefined for this one day, it should not corrupt
the
> aggregate FSS. It's like adding a whole lot of correct nulls. Seems
like
> MET handles the aggregate FBS okay but not FSS.
>
> The following Stat Analysis command and attached stat file
illustrate the
> issue.
>
> stat_analysis -lookin t.stat -job aggregate -line_type NBRCNT
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: stat_analysis FSS
From: John Halley Gotway
Time: Tue Jun 09 15:27:28 2020
Dave,
Thanks for sending the sample data file to demonstrate this behavior.
Before testing with that, I tried using some other sample data, adding
FSS=NA in the last line of an input file. But I didn't see the
behavior you
described.
Then I found that this only happens when FSS=NA in the FIRST line
that's
processed. If you reverse the order of the 2 lines in the "t.stat"
file you
sent, then you do get an output value for FSS. So order matters... but
obviously it should not!
I suspect I'll need to update the handling of missing data in the
NBRCNT
aggregation logic.
Does that all make sense?
Thanks,
John
On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>
> Hi Dave.
>
> I have assigned this ticket to John. Please allow a few business
days for
> a response.
>
> Julie
>
> On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > Transaction: Ticket created by ahijevyc
> > Queue: met_help
> > Subject: stat_analysis FSS
> > Owner: Nobody
> > Requestors: ahijevyc at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> >
> >
> > Hi MET help!
> > I am trying to aggregate the fractions skill score (FSS) for
multiple
> days,
> > however MET runs into a problem when FSS=NA for one or more days.
If
> > FSS=NA for one day, FSS=NA for the aggregate.
> >
> > By definition, FSS = 1 - FBS/FBS_worst, where FBS is fractions
Brier
> score
> > and FBS_worst is the worst-possible FBS.
> >
> > For very high rainfall rates, you'll sometimes have no forecast
and
> > observed events, and FBS_worst = 0 so FSS is undefined.
> >
> > Even though FSS is undefined for this one day, it should not
corrupt the
> > aggregate FSS. It's like adding a whole lot of correct nulls.
Seems like
> > MET handles the aggregate FBS okay but not FSS.
> >
> > The following Stat Analysis command and attached stat file
illustrate the
> > issue.
> >
> > stat_analysis -lookin t.stat -job aggregate -line_type NBRCNT
> >
> >
>
> --
> Julie Prestopnik
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: David Ahijevych
Time: Tue Jun 09 15:37:15 2020
Yes, it does. And it explains the other strange behavior I was seeing,
with
some aggregations being handled correctly with FSS=NA.
Thanks for the quick answer!
Dave
On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Dave,
>
> Thanks for sending the sample data file to demonstrate this
behavior.
> Before testing with that, I tried using some other sample data,
adding
> FSS=NA in the last line of an input file. But I didn't see the
behavior you
> described.
>
> Then I found that this only happens when FSS=NA in the FIRST line
that's
> processed. If you reverse the order of the 2 lines in the "t.stat"
file you
> sent, then you do get an output value for FSS. So order matters...
but
> obviously it should not!
>
> I suspect I'll need to update the handling of missing data in the
NBRCNT
> aggregation logic.
>
> Does that all make sense?
>
> Thanks,
> John
>
> On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> >
> > Hi Dave.
> >
> > I have assigned this ticket to John. Please allow a few business
days
> for
> > a response.
> >
> > Julie
> >
> > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > > Transaction: Ticket created by ahijevyc
> > > Queue: met_help
> > > Subject: stat_analysis FSS
> > > Owner: Nobody
> > > Requestors: ahijevyc at ucar.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> >
> > >
> > >
> > > Hi MET help!
> > > I am trying to aggregate the fractions skill score (FSS) for
multiple
> > days,
> > > however MET runs into a problem when FSS=NA for one or more
days. If
> > > FSS=NA for one day, FSS=NA for the aggregate.
> > >
> > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is fractions
Brier
> > score
> > > and FBS_worst is the worst-possible FBS.
> > >
> > > For very high rainfall rates, you'll sometimes have no forecast
and
> > > observed events, and FBS_worst = 0 so FSS is undefined.
> > >
> > > Even though FSS is undefined for this one day, it should not
corrupt
> the
> > > aggregate FSS. It's like adding a whole lot of correct nulls.
Seems
> like
> > > MET handles the aggregate FBS okay but not FSS.
> > >
> > > The following Stat Analysis command and attached stat file
illustrate
> the
> > > issue.
> > >
> > > stat_analysis -lookin t.stat -job aggregate -line_type NBRCNT
> > >
> > >
> >
> > --
> > Julie Prestopnik
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: John Halley Gotway
Time: Tue Jun 09 16:58:36 2020
Dave,
Can you please take a look at the one line fix:
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
The existing logic said that if FSS can't be aggregated numerically,
just
keep its current value. But when FSS = NA is first, then NA is it's
"current value". The fix is checking for that condition, and using the
input FSS value instead.
Just want to confirm that's the behavior you're expecting. Once you
do,
I'll submit a pull request to merge this change into the master_v9.0
and
develop branches.
Thanks,
John
On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>
> Yes, it does. And it explains the other strange behavior I was
seeing, with
> some aggregations being handled correctly with FSS=NA.
> Thanks for the quick answer!
> Dave
>
>
>
> On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Dave,
> >
> > Thanks for sending the sample data file to demonstrate this
behavior.
> > Before testing with that, I tried using some other sample data,
adding
> > FSS=NA in the last line of an input file. But I didn't see the
behavior
> you
> > described.
> >
> > Then I found that this only happens when FSS=NA in the FIRST line
that's
> > processed. If you reverse the order of the 2 lines in the "t.stat"
file
> you
> > sent, then you do get an output value for FSS. So order matters...
but
> > obviously it should not!
> >
> > I suspect I'll need to update the handling of missing data in the
NBRCNT
> > aggregation logic.
> >
> > Does that all make sense?
> >
> > Thanks,
> > John
> >
> > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> > >
> > > Hi Dave.
> > >
> > > I have assigned this ticket to John. Please allow a few
business days
> > for
> > > a response.
> > >
> > > Julie
> > >
> > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > > > Transaction: Ticket created by ahijevyc
> > > > Queue: met_help
> > > > Subject: stat_analysis FSS
> > > > Owner: Nobody
> > > > Requestors: ahijevyc at ucar.edu
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> > >
> > > >
> > > >
> > > > Hi MET help!
> > > > I am trying to aggregate the fractions skill score (FSS) for
multiple
> > > days,
> > > > however MET runs into a problem when FSS=NA for one or more
days. If
> > > > FSS=NA for one day, FSS=NA for the aggregate.
> > > >
> > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is fractions
Brier
> > > score
> > > > and FBS_worst is the worst-possible FBS.
> > > >
> > > > For very high rainfall rates, you'll sometimes have no
forecast and
> > > > observed events, and FBS_worst = 0 so FSS is undefined.
> > > >
> > > > Even though FSS is undefined for this one day, it should not
corrupt
> > the
> > > > aggregate FSS. It's like adding a whole lot of correct nulls.
Seems
> > like
> > > > MET handles the aggregate FBS okay but not FSS.
> > > >
> > > > The following Stat Analysis command and attached stat file
illustrate
> > the
> > > > issue.
> > > >
> > > > stat_analysis -lookin t.stat -job aggregate -line_type NBRCNT
> > > >
> > > >
> > >
> > > --
> > > Julie Prestopnik
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Phone: 303.497.8399
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: David Ahijevych
Time: Wed Jun 10 10:44:51 2020
I assume if x=NA, is_bad_data(x) evaluates to True?
So this fix will allow a bad FSS propagated from the beginning to be
replaced by a good FSS?
I think I understand this logic. Looks good.
On Tue, Jun 9, 2020 at 4:59 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Dave,
>
> Can you please take a look at the one line fix:
>
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
>
> The existing logic said that if FSS can't be aggregated numerically,
just
> keep its current value. But when FSS = NA is first, then NA is it's
> "current value". The fix is checking for that condition, and using
the
> input FSS value instead.
>
> Just want to confirm that's the behavior you're expecting. Once you
do,
> I'll submit a pull request to merge this change into the master_v9.0
and
> develop branches.
>
> Thanks,
> John
>
>
> On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> >
> > Yes, it does. And it explains the other strange behavior I was
seeing,
> with
> > some aggregations being handled correctly with FSS=NA.
> > Thanks for the quick answer!
> > Dave
> >
> >
> >
> > On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Dave,
> > >
> > > Thanks for sending the sample data file to demonstrate this
behavior.
> > > Before testing with that, I tried using some other sample data,
adding
> > > FSS=NA in the last line of an input file. But I didn't see the
behavior
> > you
> > > described.
> > >
> > > Then I found that this only happens when FSS=NA in the FIRST
line
> that's
> > > processed. If you reverse the order of the 2 lines in the
"t.stat" file
> > you
> > > sent, then you do get an output value for FSS. So order
matters... but
> > > obviously it should not!
> > >
> > > I suspect I'll need to update the handling of missing data in
the
> NBRCNT
> > > aggregation logic.
> > >
> > > Does that all make sense?
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
>
> > > >
> > > > Hi Dave.
> > > >
> > > > I have assigned this ticket to John. Please allow a few
business
> days
> > > for
> > > > a response.
> > > >
> > > > Julie
> > > >
> > > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > > > > Transaction: Ticket created by ahijevyc
> > > > > Queue: met_help
> > > > > Subject: stat_analysis FSS
> > > > > Owner: Nobody
> > > > > Requestors: ahijevyc at ucar.edu
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> > > >
> > > > >
> > > > >
> > > > > Hi MET help!
> > > > > I am trying to aggregate the fractions skill score (FSS) for
> multiple
> > > > days,
> > > > > however MET runs into a problem when FSS=NA for one or more
days.
> If
> > > > > FSS=NA for one day, FSS=NA for the aggregate.
> > > > >
> > > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is
fractions
> Brier
> > > > score
> > > > > and FBS_worst is the worst-possible FBS.
> > > > >
> > > > > For very high rainfall rates, you'll sometimes have no
forecast and
> > > > > observed events, and FBS_worst = 0 so FSS is undefined.
> > > > >
> > > > > Even though FSS is undefined for this one day, it should not
> corrupt
> > > the
> > > > > aggregate FSS. It's like adding a whole lot of correct
nulls. Seems
> > > like
> > > > > MET handles the aggregate FBS okay but not FSS.
> > > > >
> > > > > The following Stat Analysis command and attached stat file
> illustrate
> > > the
> > > > > issue.
> > > > >
> > > > > stat_analysis -lookin t.stat -job aggregate -line_type
NBRCNT
> > > > >
> > > > >
> > > >
> > > > --
> > > > Julie Prestopnik
> > > > Software Engineer
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > Phone: 303.497.8399
> > > > Email: jpresto at ucar.edu
> > > >
> > > > My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: John Halley Gotway
Time: Wed Jun 10 13:15:19 2020
Yes, when x = NA, is_bad_data(x) is true.
And yes, this change makes it so that the order of FSS input doesn't
affect
the output.
I'll go ahead and incorporate this change. It'll be included in met-
9.1 at
the end of June. And I'm also committing it to the master_v9.0 bugfix
branch but I do not anticipate us actually doing another met-9.0.3
bugfix
release. But I wanted to show you the one-line change in case you need
to
modify a local version of MET.
Hope that works.
Thanks,
John
On Wed, Jun 10, 2020 at 10:45 AM David Ahijevych via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>
> I assume if x=NA, is_bad_data(x) evaluates to True?
>
> So this fix will allow a bad FSS propagated from the beginning to be
> replaced by a good FSS?
>
> I think I understand this logic. Looks good.
>
> On Tue, Jun 9, 2020 at 4:59 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Dave,
> >
> > Can you please take a look at the one line fix:
> >
>
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
> >
> > The existing logic said that if FSS can't be aggregated
numerically, just
> > keep its current value. But when FSS = NA is first, then NA is
it's
> > "current value". The fix is checking for that condition, and using
the
> > input FSS value instead.
> >
> > Just want to confirm that's the behavior you're expecting. Once
you do,
> > I'll submit a pull request to merge this change into the
master_v9.0 and
> > develop branches.
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> > >
> > > Yes, it does. And it explains the other strange behavior I was
seeing,
> > with
> > > some aggregations being handled correctly with FSS=NA.
> > > Thanks for the quick answer!
> > > Dave
> > >
> > >
> > >
> > > On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Dave,
> > > >
> > > > Thanks for sending the sample data file to demonstrate this
behavior.
> > > > Before testing with that, I tried using some other sample
data,
> adding
> > > > FSS=NA in the last line of an input file. But I didn't see the
> behavior
> > > you
> > > > described.
> > > >
> > > > Then I found that this only happens when FSS=NA in the FIRST
line
> > that's
> > > > processed. If you reverse the order of the 2 lines in the
"t.stat"
> file
> > > you
> > > > sent, then you do get an output value for FSS. So order
matters...
> but
> > > > obviously it should not!
> > > >
> > > > I suspect I'll need to update the handling of missing data in
the
> > NBRCNT
> > > > aggregation logic.
> > > >
> > > > Does that all make sense?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> > > > >
> > > > > Hi Dave.
> > > > >
> > > > > I have assigned this ticket to John. Please allow a few
business
> > days
> > > > for
> > > > > a response.
> > > > >
> > > > > Julie
> > > > >
> > > > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > > > > > Transaction: Ticket created by ahijevyc
> > > > > > Queue: met_help
> > > > > > Subject: stat_analysis FSS
> > > > > > Owner: Nobody
> > > > > > Requestors: ahijevyc at ucar.edu
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi MET help!
> > > > > > I am trying to aggregate the fractions skill score (FSS)
for
> > multiple
> > > > > days,
> > > > > > however MET runs into a problem when FSS=NA for one or
more days.
> > If
> > > > > > FSS=NA for one day, FSS=NA for the aggregate.
> > > > > >
> > > > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is
fractions
> > Brier
> > > > > score
> > > > > > and FBS_worst is the worst-possible FBS.
> > > > > >
> > > > > > For very high rainfall rates, you'll sometimes have no
forecast
> and
> > > > > > observed events, and FBS_worst = 0 so FSS is undefined.
> > > > > >
> > > > > > Even though FSS is undefined for this one day, it should
not
> > corrupt
> > > > the
> > > > > > aggregate FSS. It's like adding a whole lot of correct
nulls.
> Seems
> > > > like
> > > > > > MET handles the aggregate FBS okay but not FSS.
> > > > > >
> > > > > > The following Stat Analysis command and attached stat file
> > illustrate
> > > > the
> > > > > > issue.
> > > > > >
> > > > > > stat_analysis -lookin t.stat -job aggregate -line_type
NBRCNT
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Julie Prestopnik
> > > > > Software Engineer
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > Phone: 303.497.8399
> > > > > Email: jpresto at ucar.edu
> > > > >
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: David Ahijevych
Time: Tue Jul 28 14:07:46 2020
I have a question about the UFSS definition. In the MET User's Guide
it
references Roberts and Lean 2008. In Roberts and Lean 2008, UFSS is
defined
as FSSuniform 0.5 fo/2 where fo is *observed* frequency (fraction
of
observed points exceeding the threshold over the domain)--see Section
2d,
halfway down the right column of page 83. The critical word is
*observed.
*
But in the User's Guide v9.0.3 it says UFSS is equivalent to the
*forecast*
event proportion--a subtle but important difference if there is any
forecast frequency bias. UFSS will differ depending on if it is based
on
*observed* event proportion or *forecasted* event proportion.
Now, I do believe, behind the scenes, MET based the UFSS on the
observed
frequency, consistent with Roberts and Lean 2008. Only the User's
Guide
needs to be revised.
Dave
On Wed, Jun 10, 2020 at 1:15 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Yes, when x = NA, is_bad_data(x) is true.
>
> And yes, this change makes it so that the order of FSS input doesn't
affect
> the output.
>
> I'll go ahead and incorporate this change. It'll be included in met-
9.1 at
> the end of June. And I'm also committing it to the master_v9.0
bugfix
> branch but I do not anticipate us actually doing another met-9.0.3
bugfix
> release. But I wanted to show you the one-line change in case you
need to
> modify a local version of MET.
>
> Hope that works.
>
> Thanks,
> John
>
> On Wed, Jun 10, 2020 at 10:45 AM David Ahijevych via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> >
> > I assume if x=NA, is_bad_data(x) evaluates to True?
> >
> > So this fix will allow a bad FSS propagated from the beginning to
be
> > replaced by a good FSS?
> >
> > I think I understand this logic. Looks good.
> >
> > On Tue, Jun 9, 2020 at 4:59 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Dave,
> > >
> > > Can you please take a look at the one line fix:
> > >
> >
>
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
> > >
> > > The existing logic said that if FSS can't be aggregated
numerically,
> just
> > > keep its current value. But when FSS = NA is first, then NA is
it's
> > > "current value". The fix is checking for that condition, and
using the
> > > input FSS value instead.
> > >
> > > Just want to confirm that's the behavior you're expecting. Once
you do,
> > > I'll submit a pull request to merge this change into the
master_v9.0
> and
> > > develop branches.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
>
> > > >
> > > > Yes, it does. And it explains the other strange behavior I was
> seeing,
> > > with
> > > > some aggregations being handled correctly with FSS=NA.
> > > > Thanks for the quick answer!
> > > > Dave
> > > >
> > > >
> > > >
> > > > On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Dave,
> > > > >
> > > > > Thanks for sending the sample data file to demonstrate this
> behavior.
> > > > > Before testing with that, I tried using some other sample
data,
> > adding
> > > > > FSS=NA in the last line of an input file. But I didn't see
the
> > behavior
> > > > you
> > > > > described.
> > > > >
> > > > > Then I found that this only happens when FSS=NA in the FIRST
line
> > > that's
> > > > > processed. If you reverse the order of the 2 lines in the
"t.stat"
> > file
> > > > you
> > > > > sent, then you do get an output value for FSS. So order
matters...
> > but
> > > > > obviously it should not!
> > > > >
> > > > > I suspect I'll need to update the handling of missing data
in the
> > > NBRCNT
> > > > > aggregation logic.
> > > > >
> > > > > Does that all make sense?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> > > > > >
> > > > > > Hi Dave.
> > > > > >
> > > > > > I have assigned this ticket to John. Please allow a few
business
> > > days
> > > > > for
> > > > > > a response.
> > > > > >
> > > > > > Julie
> > > > > >
> > > > > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > > > > > > Transaction: Ticket created by ahijevyc
> > > > > > > Queue: met_help
> > > > > > > Subject: stat_analysis FSS
> > > > > > > Owner: Nobody
> > > > > > > Requestors: ahijevyc at ucar.edu
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi MET help!
> > > > > > > I am trying to aggregate the fractions skill score (FSS)
for
> > > multiple
> > > > > > days,
> > > > > > > however MET runs into a problem when FSS=NA for one or
more
> days.
> > > If
> > > > > > > FSS=NA for one day, FSS=NA for the aggregate.
> > > > > > >
> > > > > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is
fractions
> > > Brier
> > > > > > score
> > > > > > > and FBS_worst is the worst-possible FBS.
> > > > > > >
> > > > > > > For very high rainfall rates, you'll sometimes have no
forecast
> > and
> > > > > > > observed events, and FBS_worst = 0 so FSS is undefined.
> > > > > > >
> > > > > > > Even though FSS is undefined for this one day, it should
not
> > > corrupt
> > > > > the
> > > > > > > aggregate FSS. It's like adding a whole lot of correct
nulls.
> > Seems
> > > > > like
> > > > > > > MET handles the aggregate FBS okay but not FSS.
> > > > > > >
> > > > > > > The following Stat Analysis command and attached stat
file
> > > illustrate
> > > > > the
> > > > > > > issue.
> > > > > > >
> > > > > > > stat_analysis -lookin t.stat -job aggregate -line_type
NBRCNT
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Julie Prestopnik
> > > > > > Software Engineer
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > Phone: 303.497.8399
> > > > > > Email: jpresto at ucar.edu
> > > > > >
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: David Ahijevych
Time: Tue Jul 28 14:09:45 2020
Sorry, when I copied and pasted, I lost the equals sign and plus sign
in
FSSuniform = 0.5 + fo/2
On Tue, Jul 28, 2020 at 2:07 PM David Ahijevych <ahijevyc at ucar.edu>
wrote:
>
> I have a question about the UFSS definition. In the MET User's Guide
it
> references Roberts and Lean 2008. In Roberts and Lean 2008, UFSS is
defined
> as FSSuniform 0.5 fo/2 where fo is *observed* frequency
(fraction
> of observed points exceeding the threshold over the domain)--see
Section
> 2d, halfway down the right column of page 83. The critical word is
*observed.
> *
>
> But in the User's Guide v9.0.3 it says UFSS is equivalent to the
> *forecast* event proportion--a subtle but important difference if
there
> is any forecast frequency bias. UFSS will differ depending on if it
is
> based on *observed* event proportion or *forecasted* event
proportion.
>
> Now, I do believe, behind the scenes, MET based the UFSS on the
observed
> frequency, consistent with Roberts and Lean 2008. Only the User's
Guide
> needs to be revised.
>
> Dave
>
>
>
> On Wed, Jun 10, 2020 at 1:15 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Yes, when x = NA, is_bad_data(x) is true.
>>
>> And yes, this change makes it so that the order of FSS input
doesn't
>> affect
>> the output.
>>
>> I'll go ahead and incorporate this change. It'll be included in
met-9.1 at
>> the end of June. And I'm also committing it to the master_v9.0
bugfix
>> branch but I do not anticipate us actually doing another met-9.0.3
bugfix
>> release. But I wanted to show you the one-line change in case you
need to
>> modify a local version of MET.
>>
>> Hope that works.
>>
>> Thanks,
>> John
>>
>> On Wed, Jun 10, 2020 at 10:45 AM David Ahijevych via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>> >
>> > I assume if x=NA, is_bad_data(x) evaluates to True?
>> >
>> > So this fix will allow a bad FSS propagated from the beginning to
be
>> > replaced by a good FSS?
>> >
>> > I think I understand this logic. Looks good.
>> >
>> > On Tue, Jun 9, 2020 at 4:59 PM John Halley Gotway via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > Dave,
>> > >
>> > > Can you please take a look at the one line fix:
>> > >
>> >
>>
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
>> > >
>> > > The existing logic said that if FSS can't be aggregated
numerically,
>> just
>> > > keep its current value. But when FSS = NA is first, then NA is
it's
>> > > "current value". The fix is checking for that condition, and
using the
>> > > input FSS value instead.
>> > >
>> > > Just want to confirm that's the behavior you're expecting. Once
you
>> do,
>> > > I'll submit a pull request to merge this change into the
master_v9.0
>> and
>> > > develop branches.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > >
>> > > On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT <
>> met_help at ucar.edu
>> > >
>> > > wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
>
>> > > >
>> > > > Yes, it does. And it explains the other strange behavior I
was
>> seeing,
>> > > with
>> > > > some aggregations being handled correctly with FSS=NA.
>> > > > Thanks for the quick answer!
>> > > > Dave
>> > > >
>> > > >
>> > > >
>> > > > On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT <
>> > > > met_help at ucar.edu>
>> > > > wrote:
>> > > >
>> > > > > Dave,
>> > > > >
>> > > > > Thanks for sending the sample data file to demonstrate this
>> behavior.
>> > > > > Before testing with that, I tried using some other sample
data,
>> > adding
>> > > > > FSS=NA in the last line of an input file. But I didn't see
the
>> > behavior
>> > > > you
>> > > > > described.
>> > > > >
>> > > > > Then I found that this only happens when FSS=NA in the
FIRST line
>> > > that's
>> > > > > processed. If you reverse the order of the 2 lines in the
"t.stat"
>> > file
>> > > > you
>> > > > > sent, then you do get an output value for FSS. So order
matters...
>> > but
>> > > > > obviously it should not!
>> > > > >
>> > > > > I suspect I'll need to update the handling of missing data
in the
>> > > NBRCNT
>> > > > > aggregation logic.
>> > > > >
>> > > > > Does that all make sense?
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT <
>> > > > met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>> > > > > >
>> > > > > > Hi Dave.
>> > > > > >
>> > > > > > I have assigned this ticket to John. Please allow a few
>> business
>> > > days
>> > > > > for
>> > > > > > a response.
>> > > > > >
>> > > > > > Julie
>> > > > > >
>> > > > > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
>> > > > > met_help at ucar.edu>
>> > > > > > wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
>> > > > > > > Transaction: Ticket created by ahijevyc
>> > > > > > > Queue: met_help
>> > > > > > > Subject: stat_analysis FSS
>> > > > > > > Owner: Nobody
>> > > > > > > Requestors: ahijevyc at ucar.edu
>> > > > > > > Status: new
>> > > > > > > Ticket <URL:
>> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
>> > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > Hi MET help!
>> > > > > > > I am trying to aggregate the fractions skill score
(FSS) for
>> > > multiple
>> > > > > > days,
>> > > > > > > however MET runs into a problem when FSS=NA for one or
more
>> days.
>> > > If
>> > > > > > > FSS=NA for one day, FSS=NA for the aggregate.
>> > > > > > >
>> > > > > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is
fractions
>> > > Brier
>> > > > > > score
>> > > > > > > and FBS_worst is the worst-possible FBS.
>> > > > > > >
>> > > > > > > For very high rainfall rates, you'll sometimes have no
>> forecast
>> > and
>> > > > > > > observed events, and FBS_worst = 0 so FSS is undefined.
>> > > > > > >
>> > > > > > > Even though FSS is undefined for this one day, it
should not
>> > > corrupt
>> > > > > the
>> > > > > > > aggregate FSS. It's like adding a whole lot of correct
nulls.
>> > Seems
>> > > > > like
>> > > > > > > MET handles the aggregate FBS okay but not FSS.
>> > > > > > >
>> > > > > > > The following Stat Analysis command and attached stat
file
>> > > illustrate
>> > > > > the
>> > > > > > > issue.
>> > > > > > >
>> > > > > > > stat_analysis -lookin t.stat -job aggregate -line_type
NBRCNT
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Julie Prestopnik
>> > > > > > Software Engineer
>> > > > > > National Center for Atmospheric Research
>> > > > > > Research Applications Laboratory
>> > > > > > Phone: 303.497.8399
>> > > > > > Email: jpresto at ucar.edu
>> > > > > >
>> > > > > > My working day may not be your working day. Please do
not feel
>> > > obliged
>> > > > > to
>> > > > > > reply to this email outside of your normal working hours.
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
------------------------------------------------
Subject: stat_analysis FSS
From: David Ahijevych
Time: Tue Jul 28 14:48:14 2020
I made a pull request on github.
On Tue, Jul 28, 2020 at 2:09 PM David Ahijevych <ahijevyc at ucar.edu>
wrote:
> Sorry, when I copied and pasted, I lost the equals sign and plus
sign in
> FSSuniform = 0.5 + fo/2
>
> On Tue, Jul 28, 2020 at 2:07 PM David Ahijevych <ahijevyc at ucar.edu>
wrote:
>
>>
>> I have a question about the UFSS definition. In the MET User's
Guide it
>> references Roberts and Lean 2008. In Roberts and Lean 2008, UFSS is
defined
>> as FSSuniform 0.5 fo/2 where fo is *observed* frequency
(fraction
>> of observed points exceeding the threshold over the domain)--see
Section
>> 2d, halfway down the right column of page 83. The critical word is
*observed.
>> *
>>
>> But in the User's Guide v9.0.3 it says UFSS is equivalent to the
>> *forecast* event proportion--a subtle but important difference if
there
>> is any forecast frequency bias. UFSS will differ depending on if it
is
>> based on *observed* event proportion or *forecasted* event
proportion.
>>
>> Now, I do believe, behind the scenes, MET based the UFSS on the
observed
>> frequency, consistent with Roberts and Lean 2008. Only the User's
Guide
>> needs to be revised.
>>
>> Dave
>>
>>
>>
>> On Wed, Jun 10, 2020 at 1:15 PM John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>>> Yes, when x = NA, is_bad_data(x) is true.
>>>
>>> And yes, this change makes it so that the order of FSS input
doesn't
>>> affect
>>> the output.
>>>
>>> I'll go ahead and incorporate this change. It'll be included in
met-9.1
>>> at
>>> the end of June. And I'm also committing it to the master_v9.0
bugfix
>>> branch but I do not anticipate us actually doing another met-9.0.3
bugfix
>>> release. But I wanted to show you the one-line change in case you
need to
>>> modify a local version of MET.
>>>
>>> Hope that works.
>>>
>>> Thanks,
>>> John
>>>
>>> On Wed, Jun 10, 2020 at 10:45 AM David Ahijevych via RT <
>>> met_help at ucar.edu>
>>> wrote:
>>>
>>> >
>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>>> >
>>> > I assume if x=NA, is_bad_data(x) evaluates to True?
>>> >
>>> > So this fix will allow a bad FSS propagated from the beginning
to be
>>> > replaced by a good FSS?
>>> >
>>> > I think I understand this logic. Looks good.
>>> >
>>> > On Tue, Jun 9, 2020 at 4:59 PM John Halley Gotway via RT <
>>> > met_help at ucar.edu>
>>> > wrote:
>>> >
>>> > > Dave,
>>> > >
>>> > > Can you please take a look at the one line fix:
>>> > >
>>> >
>>>
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
>>> > >
>>> > > The existing logic said that if FSS can't be aggregated
numerically,
>>> just
>>> > > keep its current value. But when FSS = NA is first, then NA is
it's
>>> > > "current value". The fix is checking for that condition, and
using
>>> the
>>> > > input FSS value instead.
>>> > >
>>> > > Just want to confirm that's the behavior you're expecting.
Once you
>>> do,
>>> > > I'll submit a pull request to merge this change into the
master_v9.0
>>> and
>>> > > develop branches.
>>> > >
>>> > > Thanks,
>>> > > John
>>> > >
>>> > >
>>> > > On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT <
>>> met_help at ucar.edu
>>> > >
>>> > > wrote:
>>> > >
>>> > > >
>>> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>>> > > >
>>> > > > Yes, it does. And it explains the other strange behavior I
was
>>> seeing,
>>> > > with
>>> > > > some aggregations being handled correctly with FSS=NA.
>>> > > > Thanks for the quick answer!
>>> > > > Dave
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT <
>>> > > > met_help at ucar.edu>
>>> > > > wrote:
>>> > > >
>>> > > > > Dave,
>>> > > > >
>>> > > > > Thanks for sending the sample data file to demonstrate
this
>>> behavior.
>>> > > > > Before testing with that, I tried using some other sample
data,
>>> > adding
>>> > > > > FSS=NA in the last line of an input file. But I didn't see
the
>>> > behavior
>>> > > > you
>>> > > > > described.
>>> > > > >
>>> > > > > Then I found that this only happens when FSS=NA in the
FIRST line
>>> > > that's
>>> > > > > processed. If you reverse the order of the 2 lines in the
>>> "t.stat"
>>> > file
>>> > > > you
>>> > > > > sent, then you do get an output value for FSS. So order
>>> matters...
>>> > but
>>> > > > > obviously it should not!
>>> > > > >
>>> > > > > I suspect I'll need to update the handling of missing data
in the
>>> > > NBRCNT
>>> > > > > aggregation logic.
>>> > > > >
>>> > > > > Does that all make sense?
>>> > > > >
>>> > > > > Thanks,
>>> > > > > John
>>> > > > >
>>> > > > > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT <
>>> > > > met_help at ucar.edu>
>>> > > > > wrote:
>>> > > > >
>>> > > > > >
>>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
>>> >
>>> > > > > >
>>> > > > > > Hi Dave.
>>> > > > > >
>>> > > > > > I have assigned this ticket to John. Please allow a few
>>> business
>>> > > days
>>> > > > > for
>>> > > > > > a response.
>>> > > > > >
>>> > > > > > Julie
>>> > > > > >
>>> > > > > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
>>> > > > > met_help at ucar.edu>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > >
>>> > > > > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted
upon.
>>> > > > > > > Transaction: Ticket created by ahijevyc
>>> > > > > > > Queue: met_help
>>> > > > > > > Subject: stat_analysis FSS
>>> > > > > > > Owner: Nobody
>>> > > > > > > Requestors: ahijevyc at ucar.edu
>>> > > > > > > Status: new
>>> > > > > > > Ticket <URL:
>>> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
>>> > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > Hi MET help!
>>> > > > > > > I am trying to aggregate the fractions skill score
(FSS) for
>>> > > multiple
>>> > > > > > days,
>>> > > > > > > however MET runs into a problem when FSS=NA for one or
more
>>> days.
>>> > > If
>>> > > > > > > FSS=NA for one day, FSS=NA for the aggregate.
>>> > > > > > >
>>> > > > > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is
>>> fractions
>>> > > Brier
>>> > > > > > score
>>> > > > > > > and FBS_worst is the worst-possible FBS.
>>> > > > > > >
>>> > > > > > > For very high rainfall rates, you'll sometimes have no
>>> forecast
>>> > and
>>> > > > > > > observed events, and FBS_worst = 0 so FSS is
undefined.
>>> > > > > > >
>>> > > > > > > Even though FSS is undefined for this one day, it
should not
>>> > > corrupt
>>> > > > > the
>>> > > > > > > aggregate FSS. It's like adding a whole lot of correct
nulls.
>>> > Seems
>>> > > > > like
>>> > > > > > > MET handles the aggregate FBS okay but not FSS.
>>> > > > > > >
>>> > > > > > > The following Stat Analysis command and attached stat
file
>>> > > illustrate
>>> > > > > the
>>> > > > > > > issue.
>>> > > > > > >
>>> > > > > > > stat_analysis -lookin t.stat -job aggregate -line_type
NBRCNT
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > > > --
>>> > > > > > Julie Prestopnik
>>> > > > > > Software Engineer
>>> > > > > > National Center for Atmospheric Research
>>> > > > > > Research Applications Laboratory
>>> > > > > > Phone: 303.497.8399
>>> > > > > > Email: jpresto at ucar.edu
>>> > > > > >
>>> > > > > > My working day may not be your working day. Please do
not feel
>>> > > obliged
>>> > > > > to
>>> > > > > > reply to this email outside of your normal working
hours.
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> >
>>> >
>>>
>>>
------------------------------------------------
Subject: stat_analysis FSS
From: John Halley Gotway
Time: Wed Jul 29 16:18:27 2020
Thanks for submitting the pull request, David. I just reviewed and
approved
it.
I'll go ahead and resolve this issue.
Thanks again!
John Halley Gotway
On Tue, Jul 28, 2020 at 2:48 PM David Ahijevych via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>
> I made a pull request on github.
>
> On Tue, Jul 28, 2020 at 2:09 PM David Ahijevych <ahijevyc at ucar.edu>
wrote:
>
> > Sorry, when I copied and pasted, I lost the equals sign and plus
sign in
> > FSSuniform = 0.5 + fo/2
> >
> > On Tue, Jul 28, 2020 at 2:07 PM David Ahijevych
<ahijevyc at ucar.edu>
> wrote:
> >
> >>
> >> I have a question about the UFSS definition. In the MET User's
Guide it
> >> references Roberts and Lean 2008. In Roberts and Lean 2008, UFSS
is
> defined
> >> as FSSuniform 0.5 fo/2 where fo is *observed* frequency
(fraction
> >> of observed points exceeding the threshold over the domain)--see
Section
> >> 2d, halfway down the right column of page 83. The critical word
is
> *observed.
> >> *
> >>
> >> But in the User's Guide v9.0.3 it says UFSS is equivalent to the
> >> *forecast* event proportion--a subtle but important difference if
there
> >> is any forecast frequency bias. UFSS will differ depending on if
it is
> >> based on *observed* event proportion or *forecasted* event
proportion.
> >>
> >> Now, I do believe, behind the scenes, MET based the UFSS on the
observed
> >> frequency, consistent with Roberts and Lean 2008. Only the User's
Guide
> >> needs to be revised.
> >>
> >> Dave
> >>
> >>
> >>
> >> On Wed, Jun 10, 2020 at 1:15 PM John Halley Gotway via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >>> Yes, when x = NA, is_bad_data(x) is true.
> >>>
> >>> And yes, this change makes it so that the order of FSS input
doesn't
> >>> affect
> >>> the output.
> >>>
> >>> I'll go ahead and incorporate this change. It'll be included in
met-9.1
> >>> at
> >>> the end of June. And I'm also committing it to the master_v9.0
bugfix
> >>> branch but I do not anticipate us actually doing another met-
9.0.3
> bugfix
> >>> release. But I wanted to show you the one-line change in case
you need
> to
> >>> modify a local version of MET.
> >>>
> >>> Hope that works.
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>> On Wed, Jun 10, 2020 at 10:45 AM David Ahijevych via RT <
> >>> met_help at ucar.edu>
> >>> wrote:
> >>>
> >>> >
> >>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
>
> >>> >
> >>> > I assume if x=NA, is_bad_data(x) evaluates to True?
> >>> >
> >>> > So this fix will allow a bad FSS propagated from the beginning
to be
> >>> > replaced by a good FSS?
> >>> >
> >>> > I think I understand this logic. Looks good.
> >>> >
> >>> > On Tue, Jun 9, 2020 at 4:59 PM John Halley Gotway via RT <
> >>> > met_help at ucar.edu>
> >>> > wrote:
> >>> >
> >>> > > Dave,
> >>> > >
> >>> > > Can you please take a look at the one line fix:
> >>> > >
> >>> >
> >>>
>
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
> >>> > >
> >>> > > The existing logic said that if FSS can't be aggregated
> numerically,
> >>> just
> >>> > > keep its current value. But when FSS = NA is first, then NA
is it's
> >>> > > "current value". The fix is checking for that condition, and
using
> >>> the
> >>> > > input FSS value instead.
> >>> > >
> >>> > > Just want to confirm that's the behavior you're expecting.
Once you
> >>> do,
> >>> > > I'll submit a pull request to merge this change into the
> master_v9.0
> >>> and
> >>> > > develop branches.
> >>> > >
> >>> > > Thanks,
> >>> > > John
> >>> > >
> >>> > >
> >>> > > On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT <
> >>> met_help at ucar.edu
> >>> > >
> >>> > > wrote:
> >>> > >
> >>> > > >
> >>> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> >>> > > >
> >>> > > > Yes, it does. And it explains the other strange behavior I
was
> >>> seeing,
> >>> > > with
> >>> > > > some aggregations being handled correctly with FSS=NA.
> >>> > > > Thanks for the quick answer!
> >>> > > > Dave
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT <
> >>> > > > met_help at ucar.edu>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Dave,
> >>> > > > >
> >>> > > > > Thanks for sending the sample data file to demonstrate
this
> >>> behavior.
> >>> > > > > Before testing with that, I tried using some other
sample data,
> >>> > adding
> >>> > > > > FSS=NA in the last line of an input file. But I didn't
see the
> >>> > behavior
> >>> > > > you
> >>> > > > > described.
> >>> > > > >
> >>> > > > > Then I found that this only happens when FSS=NA in the
FIRST
> line
> >>> > > that's
> >>> > > > > processed. If you reverse the order of the 2 lines in
the
> >>> "t.stat"
> >>> > file
> >>> > > > you
> >>> > > > > sent, then you do get an output value for FSS. So order
> >>> matters...
> >>> > but
> >>> > > > > obviously it should not!
> >>> > > > >
> >>> > > > > I suspect I'll need to update the handling of missing
data in
> the
> >>> > > NBRCNT
> >>> > > > > aggregation logic.
> >>> > > > >
> >>> > > > > Does that all make sense?
> >>> > > > >
> >>> > > > > Thanks,
> >>> > > > > John
> >>> > > > >
> >>> > > > > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT <
> >>> > > > met_help at ucar.edu>
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > > >
> >>> > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> >>> >
> >>> > > > > >
> >>> > > > > > Hi Dave.
> >>> > > > > >
> >>> > > > > > I have assigned this ticket to John. Please allow a
few
> >>> business
> >>> > > days
> >>> > > > > for
> >>> > > > > > a response.
> >>> > > > > >
> >>> > > > > > Julie
> >>> > > > > >
> >>> > > > > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT
<
> >>> > > > > met_help at ucar.edu>
> >>> > > > > > wrote:
> >>> > > > > >
> >>> > > > > > >
> >>> > > > > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted
upon.
> >>> > > > > > > Transaction: Ticket created by ahijevyc
> >>> > > > > > > Queue: met_help
> >>> > > > > > > Subject: stat_analysis FSS
> >>> > > > > > > Owner: Nobody
> >>> > > > > > > Requestors: ahijevyc at ucar.edu
> >>> > > > > > > Status: new
> >>> > > > > > > Ticket <URL:
> >>> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> >>> > > > > >
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > Hi MET help!
> >>> > > > > > > I am trying to aggregate the fractions skill score
(FSS)
> for
> >>> > > multiple
> >>> > > > > > days,
> >>> > > > > > > however MET runs into a problem when FSS=NA for one
or more
> >>> days.
> >>> > > If
> >>> > > > > > > FSS=NA for one day, FSS=NA for the aggregate.
> >>> > > > > > >
> >>> > > > > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is
> >>> fractions
> >>> > > Brier
> >>> > > > > > score
> >>> > > > > > > and FBS_worst is the worst-possible FBS.
> >>> > > > > > >
> >>> > > > > > > For very high rainfall rates, you'll sometimes have
no
> >>> forecast
> >>> > and
> >>> > > > > > > observed events, and FBS_worst = 0 so FSS is
undefined.
> >>> > > > > > >
> >>> > > > > > > Even though FSS is undefined for this one day, it
should
> not
> >>> > > corrupt
> >>> > > > > the
> >>> > > > > > > aggregate FSS. It's like adding a whole lot of
correct
> nulls.
> >>> > Seems
> >>> > > > > like
> >>> > > > > > > MET handles the aggregate FBS okay but not FSS.
> >>> > > > > > >
> >>> > > > > > > The following Stat Analysis command and attached
stat file
> >>> > > illustrate
> >>> > > > > the
> >>> > > > > > > issue.
> >>> > > > > > >
> >>> > > > > > > stat_analysis -lookin t.stat -job aggregate
-line_type
> NBRCNT
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > > > --
> >>> > > > > > Julie Prestopnik
> >>> > > > > > Software Engineer
> >>> > > > > > National Center for Atmospheric Research
> >>> > > > > > Research Applications Laboratory
> >>> > > > > > Phone: 303.497.8399
> >>> > > > > > Email: jpresto at ucar.edu
> >>> > > > > >
> >>> > > > > > My working day may not be your working day. Please do
not
> feel
> >>> > > obliged
> >>> > > > > to
> >>> > > > > > reply to this email outside of your normal working
hours.
> >>> > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > > >
> >>> > > >
> >>> > > >
> >>> > >
> >>> > >
> >>> >
> >>> >
> >>>
> >>>
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: David Ahijevych
Time: Tue Oct 20 17:09:06 2020
Hello!
I would like to add bootstrap confidence intervals to my aggregate
Fractions Skill Scores (FSS) calculated with MET.
Although Stat-Analysis doesn't have this ability, MET Viewer
apparently
does, according to this page from MET-help archive:
https://mailman.ucar.edu/pipermail/met_help/2019-July/003502.html
Do you still recommend we use the MET Viewer? I briefly poked around
the
web application, but didn't find any database examples with FSS.
If yes, should I start with the containerized version of MET Viewer,
or is
there a dedicated module on NCAR HPCs like cheyenne or casper? I'm
sorry--I'm having trouble finding information on this via the usual
googling.
Dave
On Wed, Jul 29, 2020 at 4:18 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Thanks for submitting the pull request, David. I just reviewed and
approved
> it.
>
> I'll go ahead and resolve this issue.
>
> Thanks again!
>
> John Halley Gotway
>
> On Tue, Jul 28, 2020 at 2:48 PM David Ahijevych via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> >
> > I made a pull request on github.
> >
> > On Tue, Jul 28, 2020 at 2:09 PM David Ahijevych
<ahijevyc at ucar.edu>
> wrote:
> >
> > > Sorry, when I copied and pasted, I lost the equals sign and plus
sign
> in
> > > FSSuniform = 0.5 + fo/2
> > >
> > > On Tue, Jul 28, 2020 at 2:07 PM David Ahijevych
<ahijevyc at ucar.edu>
> > wrote:
> > >
> > >>
> > >> I have a question about the UFSS definition. In the MET User's
Guide
> it
> > >> references Roberts and Lean 2008. In Roberts and Lean 2008,
UFSS is
> > defined
> > >> as FSSuniform 0.5 fo/2 where fo is *observed* frequency
> (fraction
> > >> of observed points exceeding the threshold over the domain)--
see
> Section
> > >> 2d, halfway down the right column of page 83. The critical
word is
> > *observed.
> > >> *
> > >>
> > >> But in the User's Guide v9.0.3 it says UFSS is equivalent to
the
> > >> *forecast* event proportion--a subtle but important difference
if
> there
> > >> is any forecast frequency bias. UFSS will differ depending on
if it is
> > >> based on *observed* event proportion or *forecasted* event
proportion.
> > >>
> > >> Now, I do believe, behind the scenes, MET based the UFSS on the
> observed
> > >> frequency, consistent with Roberts and Lean 2008. Only the
User's
> Guide
> > >> needs to be revised.
> > >>
> > >> Dave
> > >>
> > >>
> > >>
> > >> On Wed, Jun 10, 2020 at 1:15 PM John Halley Gotway via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >>> Yes, when x = NA, is_bad_data(x) is true.
> > >>>
> > >>> And yes, this change makes it so that the order of FSS input
doesn't
> > >>> affect
> > >>> the output.
> > >>>
> > >>> I'll go ahead and incorporate this change. It'll be included
in
> met-9.1
> > >>> at
> > >>> the end of June. And I'm also committing it to the master_v9.0
bugfix
> > >>> branch but I do not anticipate us actually doing another met-
9.0.3
> > bugfix
> > >>> release. But I wanted to show you the one-line change in case
you
> need
> > to
> > >>> modify a local version of MET.
> > >>>
> > >>> Hope that works.
> > >>>
> > >>> Thanks,
> > >>> John
> > >>>
> > >>> On Wed, Jun 10, 2020 at 10:45 AM David Ahijevych via RT <
> > >>> met_help at ucar.edu>
> > >>> wrote:
> > >>>
> > >>> >
> > >>> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> > >>> >
> > >>> > I assume if x=NA, is_bad_data(x) evaluates to True?
> > >>> >
> > >>> > So this fix will allow a bad FSS propagated from the
beginning to
> be
> > >>> > replaced by a good FSS?
> > >>> >
> > >>> > I think I understand this logic. Looks good.
> > >>> >
> > >>> > On Tue, Jun 9, 2020 at 4:59 PM John Halley Gotway via RT <
> > >>> > met_help at ucar.edu>
> > >>> > wrote:
> > >>> >
> > >>> > > Dave,
> > >>> > >
> > >>> > > Can you please take a look at the one line fix:
> > >>> > >
> > >>> >
> > >>>
> >
>
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
> > >>> > >
> > >>> > > The existing logic said that if FSS can't be aggregated
> > numerically,
> > >>> just
> > >>> > > keep its current value. But when FSS = NA is first, then
NA is
> it's
> > >>> > > "current value". The fix is checking for that condition,
and
> using
> > >>> the
> > >>> > > input FSS value instead.
> > >>> > >
> > >>> > > Just want to confirm that's the behavior you're expecting.
Once
> you
> > >>> do,
> > >>> > > I'll submit a pull request to merge this change into the
> > master_v9.0
> > >>> and
> > >>> > > develop branches.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > John
> > >>> > >
> > >>> > >
> > >>> > > On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT <
> > >>> met_help at ucar.edu
> > >>> > >
> > >>> > > wrote:
> > >>> > >
> > >>> > > >
> > >>> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> >
> > >>> > > >
> > >>> > > > Yes, it does. And it explains the other strange behavior
I was
> > >>> seeing,
> > >>> > > with
> > >>> > > > some aggregations being handled correctly with FSS=NA.
> > >>> > > > Thanks for the quick answer!
> > >>> > > > Dave
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > > On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT
<
> > >>> > > > met_help at ucar.edu>
> > >>> > > > wrote:
> > >>> > > >
> > >>> > > > > Dave,
> > >>> > > > >
> > >>> > > > > Thanks for sending the sample data file to demonstrate
this
> > >>> behavior.
> > >>> > > > > Before testing with that, I tried using some other
sample
> data,
> > >>> > adding
> > >>> > > > > FSS=NA in the last line of an input file. But I didn't
see
> the
> > >>> > behavior
> > >>> > > > you
> > >>> > > > > described.
> > >>> > > > >
> > >>> > > > > Then I found that this only happens when FSS=NA in the
FIRST
> > line
> > >>> > > that's
> > >>> > > > > processed. If you reverse the order of the 2 lines in
the
> > >>> "t.stat"
> > >>> > file
> > >>> > > > you
> > >>> > > > > sent, then you do get an output value for FSS. So
order
> > >>> matters...
> > >>> > but
> > >>> > > > > obviously it should not!
> > >>> > > > >
> > >>> > > > > I suspect I'll need to update the handling of missing
data in
> > the
> > >>> > > NBRCNT
> > >>> > > > > aggregation logic.
> > >>> > > > >
> > >>> > > > > Does that all make sense?
> > >>> > > > >
> > >>> > > > > Thanks,
> > >>> > > > > John
> > >>> > > > >
> > >>> > > > > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT
<
> > >>> > > > met_help at ucar.edu>
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > > >
> > >>> > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> > >>> >
> > >>> > > > > >
> > >>> > > > > > Hi Dave.
> > >>> > > > > >
> > >>> > > > > > I have assigned this ticket to John. Please allow a
few
> > >>> business
> > >>> > > days
> > >>> > > > > for
> > >>> > > > > > a response.
> > >>> > > > > >
> > >>> > > > > > Julie
> > >>> > > > > >
> > >>> > > > > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via
RT <
> > >>> > > > > met_help at ucar.edu>
> > >>> > > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > >
> > >>> > > > > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted
upon.
> > >>> > > > > > > Transaction: Ticket created by ahijevyc
> > >>> > > > > > > Queue: met_help
> > >>> > > > > > > Subject: stat_analysis FSS
> > >>> > > > > > > Owner: Nobody
> > >>> > > > > > > Requestors: ahijevyc at ucar.edu
> > >>> > > > > > > Status: new
> > >>> > > > > > > Ticket <URL:
> > >>> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> > >>> > > > > >
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > Hi MET help!
> > >>> > > > > > > I am trying to aggregate the fractions skill score
(FSS)
> > for
> > >>> > > multiple
> > >>> > > > > > days,
> > >>> > > > > > > however MET runs into a problem when FSS=NA for
one or
> more
> > >>> days.
> > >>> > > If
> > >>> > > > > > > FSS=NA for one day, FSS=NA for the aggregate.
> > >>> > > > > > >
> > >>> > > > > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS
is
> > >>> fractions
> > >>> > > Brier
> > >>> > > > > > score
> > >>> > > > > > > and FBS_worst is the worst-possible FBS.
> > >>> > > > > > >
> > >>> > > > > > > For very high rainfall rates, you'll sometimes
have no
> > >>> forecast
> > >>> > and
> > >>> > > > > > > observed events, and FBS_worst = 0 so FSS is
undefined.
> > >>> > > > > > >
> > >>> > > > > > > Even though FSS is undefined for this one day, it
should
> > not
> > >>> > > corrupt
> > >>> > > > > the
> > >>> > > > > > > aggregate FSS. It's like adding a whole lot of
correct
> > nulls.
> > >>> > Seems
> > >>> > > > > like
> > >>> > > > > > > MET handles the aggregate FBS okay but not FSS.
> > >>> > > > > > >
> > >>> > > > > > > The following Stat Analysis command and attached
stat
> file
> > >>> > > illustrate
> > >>> > > > > the
> > >>> > > > > > > issue.
> > >>> > > > > > >
> > >>> > > > > > > stat_analysis -lookin t.stat -job aggregate
-line_type
> > NBRCNT
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > > > --
> > >>> > > > > > Julie Prestopnik
> > >>> > > > > > Software Engineer
> > >>> > > > > > National Center for Atmospheric Research
> > >>> > > > > > Research Applications Laboratory
> > >>> > > > > > Phone: 303.497.8399
> > >>> > > > > > Email: jpresto at ucar.edu
> > >>> > > > > >
> > >>> > > > > > My working day may not be your working day. Please
do not
> > feel
> > >>> > > obliged
> > >>> > > > > to
> > >>> > > > > > reply to this email outside of your normal working
hours.
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > > >
> > >>> > >
> > >>> > >
> > >>> >
> > >>> >
> > >>>
> > >>>
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list