[Met_help] [rt.rap.ucar.edu #95555] History for stat_analysis FSS
John Halley Gotway via RT
met_help at ucar.edu
Wed Jun 10 14:15:40 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi MET help!
I am trying to aggregate the fractions skill score (FSS) for multiple days,
however MET runs into a problem when FSS=NA for one or more days. If
FSS=NA for one day, FSS=NA for the aggregate.
By definition, FSS = 1 - FBS/FBS_worst, where FBS is fractions Brier score
and FBS_worst is the worst-possible FBS.
For very high rainfall rates, you'll sometimes have no forecast and
observed events, and FBS_worst = 0 so FSS is undefined.
Even though FSS is undefined for this one day, it should not corrupt the
aggregate FSS. It's like adding a whole lot of correct nulls. Seems like
MET handles the aggregate FBS okay but not FSS.
The following Stat Analysis command and attached stat file illustrate the
issue.
stat_analysis -lookin t.stat -job aggregate -line_type NBRCNT
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: stat_analysis FSS
From: Julie Prestopnik
Time: Tue Jun 09 13:47:06 2020
Hi Dave.
I have assigned this ticket to John. Please allow a few business days
for
a response.
Julie
On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT
<met_help at ucar.edu>
wrote:
>
> Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> Transaction: Ticket created by ahijevyc
> Queue: met_help
> Subject: stat_analysis FSS
> Owner: Nobody
> Requestors: ahijevyc at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>
>
> Hi MET help!
> I am trying to aggregate the fractions skill score (FSS) for
multiple days,
> however MET runs into a problem when FSS=NA for one or more days.
If
> FSS=NA for one day, FSS=NA for the aggregate.
>
> By definition, FSS = 1 - FBS/FBS_worst, where FBS is fractions Brier
score
> and FBS_worst is the worst-possible FBS.
>
> For very high rainfall rates, you'll sometimes have no forecast and
> observed events, and FBS_worst = 0 so FSS is undefined.
>
> Even though FSS is undefined for this one day, it should not corrupt
the
> aggregate FSS. It's like adding a whole lot of correct nulls. Seems
like
> MET handles the aggregate FBS okay but not FSS.
>
> The following Stat Analysis command and attached stat file
illustrate the
> issue.
>
> stat_analysis -lookin t.stat -job aggregate -line_type NBRCNT
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: stat_analysis FSS
From: John Halley Gotway
Time: Tue Jun 09 15:27:28 2020
Dave,
Thanks for sending the sample data file to demonstrate this behavior.
Before testing with that, I tried using some other sample data, adding
FSS=NA in the last line of an input file. But I didn't see the
behavior you
described.
Then I found that this only happens when FSS=NA in the FIRST line
that's
processed. If you reverse the order of the 2 lines in the "t.stat"
file you
sent, then you do get an output value for FSS. So order matters... but
obviously it should not!
I suspect I'll need to update the handling of missing data in the
NBRCNT
aggregation logic.
Does that all make sense?
Thanks,
John
On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>
> Hi Dave.
>
> I have assigned this ticket to John. Please allow a few business
days for
> a response.
>
> Julie
>
> On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > Transaction: Ticket created by ahijevyc
> > Queue: met_help
> > Subject: stat_analysis FSS
> > Owner: Nobody
> > Requestors: ahijevyc at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> >
> >
> > Hi MET help!
> > I am trying to aggregate the fractions skill score (FSS) for
multiple
> days,
> > however MET runs into a problem when FSS=NA for one or more days.
If
> > FSS=NA for one day, FSS=NA for the aggregate.
> >
> > By definition, FSS = 1 - FBS/FBS_worst, where FBS is fractions
Brier
> score
> > and FBS_worst is the worst-possible FBS.
> >
> > For very high rainfall rates, you'll sometimes have no forecast
and
> > observed events, and FBS_worst = 0 so FSS is undefined.
> >
> > Even though FSS is undefined for this one day, it should not
corrupt the
> > aggregate FSS. It's like adding a whole lot of correct nulls.
Seems like
> > MET handles the aggregate FBS okay but not FSS.
> >
> > The following Stat Analysis command and attached stat file
illustrate the
> > issue.
> >
> > stat_analysis -lookin t.stat -job aggregate -line_type NBRCNT
> >
> >
>
> --
> Julie Prestopnik
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: David Ahijevych
Time: Tue Jun 09 15:37:15 2020
Yes, it does. And it explains the other strange behavior I was seeing,
with
some aggregations being handled correctly with FSS=NA.
Thanks for the quick answer!
Dave
On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Dave,
>
> Thanks for sending the sample data file to demonstrate this
behavior.
> Before testing with that, I tried using some other sample data,
adding
> FSS=NA in the last line of an input file. But I didn't see the
behavior you
> described.
>
> Then I found that this only happens when FSS=NA in the FIRST line
that's
> processed. If you reverse the order of the 2 lines in the "t.stat"
file you
> sent, then you do get an output value for FSS. So order matters...
but
> obviously it should not!
>
> I suspect I'll need to update the handling of missing data in the
NBRCNT
> aggregation logic.
>
> Does that all make sense?
>
> Thanks,
> John
>
> On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> >
> > Hi Dave.
> >
> > I have assigned this ticket to John. Please allow a few business
days
> for
> > a response.
> >
> > Julie
> >
> > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > > Transaction: Ticket created by ahijevyc
> > > Queue: met_help
> > > Subject: stat_analysis FSS
> > > Owner: Nobody
> > > Requestors: ahijevyc at ucar.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> >
> > >
> > >
> > > Hi MET help!
> > > I am trying to aggregate the fractions skill score (FSS) for
multiple
> > days,
> > > however MET runs into a problem when FSS=NA for one or more
days. If
> > > FSS=NA for one day, FSS=NA for the aggregate.
> > >
> > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is fractions
Brier
> > score
> > > and FBS_worst is the worst-possible FBS.
> > >
> > > For very high rainfall rates, you'll sometimes have no forecast
and
> > > observed events, and FBS_worst = 0 so FSS is undefined.
> > >
> > > Even though FSS is undefined for this one day, it should not
corrupt
> the
> > > aggregate FSS. It's like adding a whole lot of correct nulls.
Seems
> like
> > > MET handles the aggregate FBS okay but not FSS.
> > >
> > > The following Stat Analysis command and attached stat file
illustrate
> the
> > > issue.
> > >
> > > stat_analysis -lookin t.stat -job aggregate -line_type NBRCNT
> > >
> > >
> >
> > --
> > Julie Prestopnik
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: John Halley Gotway
Time: Tue Jun 09 16:58:36 2020
Dave,
Can you please take a look at the one line fix:
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
The existing logic said that if FSS can't be aggregated numerically,
just
keep its current value. But when FSS = NA is first, then NA is it's
"current value". The fix is checking for that condition, and using the
input FSS value instead.
Just want to confirm that's the behavior you're expecting. Once you
do,
I'll submit a pull request to merge this change into the master_v9.0
and
develop branches.
Thanks,
John
On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>
> Yes, it does. And it explains the other strange behavior I was
seeing, with
> some aggregations being handled correctly with FSS=NA.
> Thanks for the quick answer!
> Dave
>
>
>
> On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Dave,
> >
> > Thanks for sending the sample data file to demonstrate this
behavior.
> > Before testing with that, I tried using some other sample data,
adding
> > FSS=NA in the last line of an input file. But I didn't see the
behavior
> you
> > described.
> >
> > Then I found that this only happens when FSS=NA in the FIRST line
that's
> > processed. If you reverse the order of the 2 lines in the "t.stat"
file
> you
> > sent, then you do get an output value for FSS. So order matters...
but
> > obviously it should not!
> >
> > I suspect I'll need to update the handling of missing data in the
NBRCNT
> > aggregation logic.
> >
> > Does that all make sense?
> >
> > Thanks,
> > John
> >
> > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> > >
> > > Hi Dave.
> > >
> > > I have assigned this ticket to John. Please allow a few
business days
> > for
> > > a response.
> > >
> > > Julie
> > >
> > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > > > Transaction: Ticket created by ahijevyc
> > > > Queue: met_help
> > > > Subject: stat_analysis FSS
> > > > Owner: Nobody
> > > > Requestors: ahijevyc at ucar.edu
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> > >
> > > >
> > > >
> > > > Hi MET help!
> > > > I am trying to aggregate the fractions skill score (FSS) for
multiple
> > > days,
> > > > however MET runs into a problem when FSS=NA for one or more
days. If
> > > > FSS=NA for one day, FSS=NA for the aggregate.
> > > >
> > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is fractions
Brier
> > > score
> > > > and FBS_worst is the worst-possible FBS.
> > > >
> > > > For very high rainfall rates, you'll sometimes have no
forecast and
> > > > observed events, and FBS_worst = 0 so FSS is undefined.
> > > >
> > > > Even though FSS is undefined for this one day, it should not
corrupt
> > the
> > > > aggregate FSS. It's like adding a whole lot of correct nulls.
Seems
> > like
> > > > MET handles the aggregate FBS okay but not FSS.
> > > >
> > > > The following Stat Analysis command and attached stat file
illustrate
> > the
> > > > issue.
> > > >
> > > > stat_analysis -lookin t.stat -job aggregate -line_type NBRCNT
> > > >
> > > >
> > >
> > > --
> > > Julie Prestopnik
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Phone: 303.497.8399
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: David Ahijevych
Time: Wed Jun 10 10:44:51 2020
I assume if x=NA, is_bad_data(x) evaluates to True?
So this fix will allow a bad FSS propagated from the beginning to be
replaced by a good FSS?
I think I understand this logic. Looks good.
On Tue, Jun 9, 2020 at 4:59 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Dave,
>
> Can you please take a look at the one line fix:
>
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
>
> The existing logic said that if FSS can't be aggregated numerically,
just
> keep its current value. But when FSS = NA is first, then NA is it's
> "current value". The fix is checking for that condition, and using
the
> input FSS value instead.
>
> Just want to confirm that's the behavior you're expecting. Once you
do,
> I'll submit a pull request to merge this change into the master_v9.0
and
> develop branches.
>
> Thanks,
> John
>
>
> On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> >
> > Yes, it does. And it explains the other strange behavior I was
seeing,
> with
> > some aggregations being handled correctly with FSS=NA.
> > Thanks for the quick answer!
> > Dave
> >
> >
> >
> > On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Dave,
> > >
> > > Thanks for sending the sample data file to demonstrate this
behavior.
> > > Before testing with that, I tried using some other sample data,
adding
> > > FSS=NA in the last line of an input file. But I didn't see the
behavior
> > you
> > > described.
> > >
> > > Then I found that this only happens when FSS=NA in the FIRST
line
> that's
> > > processed. If you reverse the order of the 2 lines in the
"t.stat" file
> > you
> > > sent, then you do get an output value for FSS. So order
matters... but
> > > obviously it should not!
> > >
> > > I suspect I'll need to update the handling of missing data in
the
> NBRCNT
> > > aggregation logic.
> > >
> > > Does that all make sense?
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
>
> > > >
> > > > Hi Dave.
> > > >
> > > > I have assigned this ticket to John. Please allow a few
business
> days
> > > for
> > > > a response.
> > > >
> > > > Julie
> > > >
> > > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > > > > Transaction: Ticket created by ahijevyc
> > > > > Queue: met_help
> > > > > Subject: stat_analysis FSS
> > > > > Owner: Nobody
> > > > > Requestors: ahijevyc at ucar.edu
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> > > >
> > > > >
> > > > >
> > > > > Hi MET help!
> > > > > I am trying to aggregate the fractions skill score (FSS) for
> multiple
> > > > days,
> > > > > however MET runs into a problem when FSS=NA for one or more
days.
> If
> > > > > FSS=NA for one day, FSS=NA for the aggregate.
> > > > >
> > > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is
fractions
> Brier
> > > > score
> > > > > and FBS_worst is the worst-possible FBS.
> > > > >
> > > > > For very high rainfall rates, you'll sometimes have no
forecast and
> > > > > observed events, and FBS_worst = 0 so FSS is undefined.
> > > > >
> > > > > Even though FSS is undefined for this one day, it should not
> corrupt
> > > the
> > > > > aggregate FSS. It's like adding a whole lot of correct
nulls. Seems
> > > like
> > > > > MET handles the aggregate FBS okay but not FSS.
> > > > >
> > > > > The following Stat Analysis command and attached stat file
> illustrate
> > > the
> > > > > issue.
> > > > >
> > > > > stat_analysis -lookin t.stat -job aggregate -line_type
NBRCNT
> > > > >
> > > > >
> > > >
> > > > --
> > > > Julie Prestopnik
> > > > Software Engineer
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > Phone: 303.497.8399
> > > > Email: jpresto at ucar.edu
> > > >
> > > > My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: stat_analysis FSS
From: John Halley Gotway
Time: Wed Jun 10 13:15:19 2020
Yes, when x = NA, is_bad_data(x) is true.
And yes, this change makes it so that the order of FSS input doesn't
affect
the output.
I'll go ahead and incorporate this change. It'll be included in met-
9.1 at
the end of June. And I'm also committing it to the master_v9.0 bugfix
branch but I do not anticipate us actually doing another met-9.0.3
bugfix
release. But I wanted to show you the one-line change in case you need
to
modify a local version of MET.
Hope that works.
Thanks,
John
On Wed, Jun 10, 2020 at 10:45 AM David Ahijevych via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
>
> I assume if x=NA, is_bad_data(x) evaluates to True?
>
> So this fix will allow a bad FSS propagated from the beginning to be
> replaced by a good FSS?
>
> I think I understand this logic. Looks good.
>
> On Tue, Jun 9, 2020 at 4:59 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Dave,
> >
> > Can you please take a look at the one line fix:
> >
>
https://github.com/NCAR/MET/commit/17ec2ab66bd49f3a605feaae011dccf6245bf524
> >
> > The existing logic said that if FSS can't be aggregated
numerically, just
> > keep its current value. But when FSS = NA is first, then NA is
it's
> > "current value". The fix is checking for that condition, and using
the
> > input FSS value instead.
> >
> > Just want to confirm that's the behavior you're expecting. Once
you do,
> > I'll submit a pull request to merge this change into the
master_v9.0 and
> > develop branches.
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Jun 9, 2020 at 3:37 PM David Ahijevych via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> > >
> > > Yes, it does. And it explains the other strange behavior I was
seeing,
> > with
> > > some aggregations being handled correctly with FSS=NA.
> > > Thanks for the quick answer!
> > > Dave
> > >
> > >
> > >
> > > On Tue, Jun 9, 2020 at 3:28 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Dave,
> > > >
> > > > Thanks for sending the sample data file to demonstrate this
behavior.
> > > > Before testing with that, I tried using some other sample
data,
> adding
> > > > FSS=NA in the last line of an input file. But I didn't see the
> behavior
> > > you
> > > > described.
> > > >
> > > > Then I found that this only happens when FSS=NA in the FIRST
line
> > that's
> > > > processed. If you reverse the order of the 2 lines in the
"t.stat"
> file
> > > you
> > > > sent, then you do get an output value for FSS. So order
matters...
> but
> > > > obviously it should not!
> > > >
> > > > I suspect I'll need to update the handling of missing data in
the
> > NBRCNT
> > > > aggregation logic.
> > > >
> > > > Does that all make sense?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Jun 9, 2020 at 1:47 PM Julie Prestopnik via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555 >
> > > > >
> > > > > Hi Dave.
> > > > >
> > > > > I have assigned this ticket to John. Please allow a few
business
> > days
> > > > for
> > > > > a response.
> > > > >
> > > > > Julie
> > > > >
> > > > > On Tue, Jun 9, 2020 at 10:58 AM David Ahijevych via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Tue Jun 09 10:58:46 2020: Request 95555 was acted upon.
> > > > > > Transaction: Ticket created by ahijevyc
> > > > > > Queue: met_help
> > > > > > Subject: stat_analysis FSS
> > > > > > Owner: Nobody
> > > > > > Requestors: ahijevyc at ucar.edu
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95555
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi MET help!
> > > > > > I am trying to aggregate the fractions skill score (FSS)
for
> > multiple
> > > > > days,
> > > > > > however MET runs into a problem when FSS=NA for one or
more days.
> > If
> > > > > > FSS=NA for one day, FSS=NA for the aggregate.
> > > > > >
> > > > > > By definition, FSS = 1 - FBS/FBS_worst, where FBS is
fractions
> > Brier
> > > > > score
> > > > > > and FBS_worst is the worst-possible FBS.
> > > > > >
> > > > > > For very high rainfall rates, you'll sometimes have no
forecast
> and
> > > > > > observed events, and FBS_worst = 0 so FSS is undefined.
> > > > > >
> > > > > > Even though FSS is undefined for this one day, it should
not
> > corrupt
> > > > the
> > > > > > aggregate FSS. It's like adding a whole lot of correct
nulls.
> Seems
> > > > like
> > > > > > MET handles the aggregate FBS okay but not FSS.
> > > > > >
> > > > > > The following Stat Analysis command and attached stat file
> > illustrate
> > > > the
> > > > > > issue.
> > > > > >
> > > > > > stat_analysis -lookin t.stat -job aggregate -line_type
NBRCNT
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Julie Prestopnik
> > > > > Software Engineer
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > Phone: 303.497.8399
> > > > > Email: jpresto at ucar.edu
> > > > >
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list