[Met_help] [rt.rap.ucar.edu #95843] History for CSI and CDF plot
Tatiana Burek via RT
met_help at ucar.edu
Tue Oct 13 12:16:11 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
This is Binyu from NOAA/EMC. I have 3 questions about using MET/METViewer
to for model verification :
1. Currently all the stats/plots (rely/ROC/Hist) obtained from
MET/METviewer are calculated based on the paired data ONLY: ie. both model
and obs. have valid data in the grid. But how about those grids that have
either ONLY forecast (no obs) or obs. (no forecast)?
2. For ensemble forecast, can we use MET/METViewer to make CSI plot?
3. Same as above, can we use MET/METViewer to make CDF plot?
Thank you very much.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: CSI and CDF plot
From: Julie Prestopnik
Time: Tue Jul 07 14:47:29 2020
Hi Binyu.
I have assigned this ticket to Tatiana. Please allow a few business
days
for a response.
Julie
On Tue, Jul 7, 2020 at 2:33 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> Tue Jul 07 14:33:01 2020: Request 95843 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
> Queue: met_help
> Subject: CSI and CDF plot
> Owner: Nobody
> Requestors: binyu.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95843 >
>
>
> Hello,
>
> This is Binyu from NOAA/EMC. I have 3 questions about using
MET/METViewer
> to for model verification :
>
> 1. Currently all the stats/plots (rely/ROC/Hist) obtained from
> MET/METviewer are calculated based on the paired data ONLY: ie. both
model
> and obs. have valid data in the grid. But how about those grids that
have
> either ONLY forecast (no obs) or obs. (no forecast)?
>
> 2. For ensemble forecast, can we use MET/METViewer to make CSI plot?
> 3. Same as above, can we use MET/METViewer to make CDF plot?
>
> Thank you very much.
> Binyu
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: CSI and CDF plot
From: Tara Jensen
Time: Tue Jul 07 15:56:20 2020
Binyu,
Here are some answers to your questions:
1. Reli (Reliability), ROC, and Hist diagrams have to be based on
paired
data. Reliability and ROC are event based scores that require an
observation or analysis value to define whether the event occurred or
not.
By definition, Rank and PIT Histograms are a measure of where the
observation lies within the ranked set of values from the ensemble
members.
I don't think I understand what you mean by "how about those grids
that
have either ONLY forecast (no obs) or obs. (no forecast)?". MET does
not
expect the forecast and obs to be in the same file. It actually
expects
two files to be passed in and it does the heavy lifting of pairing up
the
data for the statistics computation.
2. MET can compute categorical statistics, such as CSI, for each
ensemble
member and the ensemble mean field by passing the members and/or mean
field
into MET individually. Time series, box plots, and bar plots of CSI
can
then be plotted using METviewer.
3. MET has a new tool called Grid-Diag that computes the PDF of one or
two
fields from a given file. It could be used to do this for each
ensemble
member and/or ensemble mean fields if desired. The CDF could then be
derived from the PDF plotted using your own script if you'd like. We
do
not have plotting support available for the output of the Grid-Diag
tool
yet.
Hopefully this addresses your questions. Please let us know if it
doesn't.
Cheers, Tara
On Tue, Jul 7, 2020 at 2:33 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> Tue Jul 07 14:33:01 2020: Request 95843 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
> Queue: met_help
> Subject: CSI and CDF plot
> Owner: Nobody
> Requestors: binyu.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95843 >
>
>
> Hello,
>
> This is Binyu from NOAA/EMC. I have 3 questions about using
MET/METViewer
> to for model verification :
>
> 1. Currently all the stats/plots (rely/ROC/Hist) obtained from
> MET/METviewer are calculated based on the paired data ONLY: ie. both
model
> and obs. have valid data in the grid. But how about those grids that
have
> either ONLY forecast (no obs) or obs. (no forecast)?
>
> 2. For ensemble forecast, can we use MET/METViewer to make CSI plot?
> 3. Same as above, can we use MET/METViewer to make CDF plot?
>
> Thank you very much.
> Binyu
>
>
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tara Jensen
Project Manager II
NCAR RAL and DTC
PO Box 3000, Boulder, Colorado 80307 USA
+1 303-497-8479 jensen at ucar.edu
------------------------------------------------
Subject: CSI and CDF plot
From: binyu.wang at noaa.gov
Time: Tue Jul 07 20:20:49 2020
Helo Tara,
Thank you for the quick response.
Sorry I didn't explain clearly in my first question. Here is an
example for
volcano ash over some area:
Eg: there are 1000 grids that have valid value in the observed file,
950
grids that have valid value in the model. There were only 900 grids
that
were captured both by model and obs. Which means there were 100 grids
detected by obs. ONLY and 50 grids detected by model ONLY. In my
understanding, currently MET ONLY includes the paired 900 grids for
calculation like HIST, Rely, and ROC, it just ignores the 150 grids
which
are either captched by obs. or by model only, am I correct?
For CSI and Grid-Diag, is there a sample of configure file for that?
Thank you.
Binyu
On Tue, Jul 7, 2020 at 5:56 PM Tara Jensen via RT <met_help at ucar.edu>
wrote:
> Binyu,
>
> Here are some answers to your questions:
>
> 1. Reli (Reliability), ROC, and Hist diagrams have to be based on
paired
> data. Reliability and ROC are event based scores that require an
> observation or analysis value to define whether the event occurred
or not.
> By definition, Rank and PIT Histograms are a measure of where the
> observation lies within the ranked set of values from the ensemble
> members.
>
> I don't think I understand what you mean by "how about those grids
that
> have either ONLY forecast (no obs) or obs. (no forecast)?". MET
does not
> expect the forecast and obs to be in the same file. It actually
expects
> two files to be passed in and it does the heavy lifting of pairing
up the
> data for the statistics computation.
>
> 2. MET can compute categorical statistics, such as CSI, for each
ensemble
> member and the ensemble mean field by passing the members and/or
mean field
> into MET individually. Time series, box plots, and bar plots of CSI
can
> then be plotted using METviewer.
>
> 3. MET has a new tool called Grid-Diag that computes the PDF of one
or two
> fields from a given file. It could be used to do this for each
ensemble
> member and/or ensemble mean fields if desired. The CDF could then
be
> derived from the PDF plotted using your own script if you'd like.
We do
> not have plotting support available for the output of the Grid-Diag
tool
> yet.
>
> Hopefully this addresses your questions. Please let us know if it
doesn't.
> Cheers, Tara
>
> On Tue, Jul 7, 2020 at 2:33 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > Tue Jul 07 14:33:01 2020: Request 95843 was acted upon.
> > Transaction: Ticket created by binyu.wang at noaa.gov
> > Queue: met_help
> > Subject: CSI and CDF plot
> > Owner: Nobody
> > Requestors: binyu.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95843 >
> >
> >
> > Hello,
> >
> > This is Binyu from NOAA/EMC. I have 3 questions about using
MET/METViewer
> > to for model verification :
> >
> > 1. Currently all the stats/plots (rely/ROC/Hist) obtained from
> > MET/METviewer are calculated based on the paired data ONLY: ie.
both
> model
> > and obs. have valid data in the grid. But how about those grids
that have
> > either ONLY forecast (no obs) or obs. (no forecast)?
> >
> > 2. For ensemble forecast, can we use MET/METViewer to make CSI
plot?
> > 3. Same as above, can we use MET/METViewer to make CDF plot?
> >
> > Thank you very much.
> > Binyu
> >
> >
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Tara Jensen
> Project Manager II
> NCAR RAL and DTC
> PO Box 3000, Boulder, Colorado 80307 USA
> +1 303-497-8479 jensen at ucar.edu
>
>
------------------------------------------------
Subject: CSI and CDF plot
From: Tara Jensen
Time: Wed Jul 08 11:39:20 2020
Binyu,
You are correct that MET would only retain the 900 points for the
computation of statistics.
Default config files are included in the MET release in the
$MET_BASE/share/met/data/config directory.
You can also view them in our GitHub repository at:
https://github.com/NCAR/MET/tree/master_v9.0/met/data/config.
<https://github.com/NCAR/MET/tree/master_v9.0/met/data/config>
A description of config options can be found in Chapter 3 of the MET
User's
Guide.
Let me know if you have any further questions regarding these issues.
If
not, I'll close this ticket.
Cheers, Tara
On Tue, Jul 7, 2020 at 8:21 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95843 >
>
> Helo Tara,
>
> Thank you for the quick response.
>
> Sorry I didn't explain clearly in my first question. Here is an
example for
> volcano ash over some area:
> Eg: there are 1000 grids that have valid value in the observed file,
950
> grids that have valid value in the model. There were only 900 grids
that
> were captured both by model and obs. Which means there were 100
grids
> detected by obs. ONLY and 50 grids detected by model ONLY. In my
> understanding, currently MET ONLY includes the paired 900 grids for
> calculation like HIST, Rely, and ROC, it just ignores the 150 grids
which
> are either captched by obs. or by model only, am I correct?
>
> For CSI and Grid-Diag, is there a sample of configure file for that?
>
> Thank you.
> Binyu
>
>
> On Tue, Jul 7, 2020 at 5:56 PM Tara Jensen via RT
<met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > Here are some answers to your questions:
> >
> > 1. Reli (Reliability), ROC, and Hist diagrams have to be based on
paired
> > data. Reliability and ROC are event based scores that require an
> > observation or analysis value to define whether the event occurred
or
> not.
> > By definition, Rank and PIT Histograms are a measure of where the
> > observation lies within the ranked set of values from the ensemble
> > members.
> >
> > I don't think I understand what you mean by "how about those
grids that
> > have either ONLY forecast (no obs) or obs. (no forecast)?". MET
does not
> > expect the forecast and obs to be in the same file. It actually
expects
> > two files to be passed in and it does the heavy lifting of pairing
up the
> > data for the statistics computation.
> >
> > 2. MET can compute categorical statistics, such as CSI, for each
ensemble
> > member and the ensemble mean field by passing the members and/or
mean
> field
> > into MET individually. Time series, box plots, and bar plots of
CSI can
> > then be plotted using METviewer.
> >
> > 3. MET has a new tool called Grid-Diag that computes the PDF of
one or
> two
> > fields from a given file. It could be used to do this for each
ensemble
> > member and/or ensemble mean fields if desired. The CDF could then
be
> > derived from the PDF plotted using your own script if you'd like.
We do
> > not have plotting support available for the output of the Grid-
Diag tool
> > yet.
> >
> > Hopefully this addresses your questions. Please let us know if it
> doesn't.
> > Cheers, Tara
> >
> > On Tue, Jul 7, 2020 at 2:33 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Tue Jul 07 14:33:01 2020: Request 95843 was acted upon.
> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > Queue: met_help
> > > Subject: CSI and CDF plot
> > > Owner: Nobody
> > > Requestors: binyu.wang at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95843
> >
> > >
> > >
> > > Hello,
> > >
> > > This is Binyu from NOAA/EMC. I have 3 questions about using
> MET/METViewer
> > > to for model verification :
> > >
> > > 1. Currently all the stats/plots (rely/ROC/Hist) obtained from
> > > MET/METviewer are calculated based on the paired data ONLY: ie.
both
> > model
> > > and obs. have valid data in the grid. But how about those grids
that
> have
> > > either ONLY forecast (no obs) or obs. (no forecast)?
> > >
> > > 2. For ensemble forecast, can we use MET/METViewer to make CSI
plot?
> > > 3. Same as above, can we use MET/METViewer to make CDF plot?
> > >
> > > Thank you very much.
> > > Binyu
> > >
> > >
> >
> > --
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Tara Jensen
> > Project Manager II
> > NCAR RAL and DTC
> > PO Box 3000, Boulder, Colorado 80307 USA
> > +1 303-497-8479 jensen at ucar.edu
> >
> >
>
>
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tara Jensen
Project Manager II
NCAR RAL and DTC
PO Box 3000, Boulder, Colorado 80307 USA
+1 303-497-8479 jensen at ucar.edu
------------------------------------------------
Subject: CSI and CDF plot
From: binyu.wang at noaa.gov
Time: Wed Jul 08 12:36:15 2020
Thank you, Tara. You can close the ticket now.
Binyu
On Wed, Jul 8, 2020 at 1:39 PM Tara Jensen via RT <met_help at ucar.edu>
wrote:
> Binyu,
>
> You are correct that MET would only retain the 900 points for the
> computation of statistics.
>
> Default config files are included in the MET release in the
> $MET_BASE/share/met/data/config directory.
> You can also view them in our GitHub repository at:
> https://github.com/NCAR/MET/tree/master_v9.0/met/data/config.
> <https://github.com/NCAR/MET/tree/master_v9.0/met/data/config>
> A description of config options can be found in Chapter 3 of the MET
User's
> Guide.
>
> Let me know if you have any further questions regarding these
issues. If
> not, I'll close this ticket.
> Cheers, Tara
>
> On Tue, Jul 7, 2020 at 8:21 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95843 >
> >
> > Helo Tara,
> >
> > Thank you for the quick response.
> >
> > Sorry I didn't explain clearly in my first question. Here is an
example
> for
> > volcano ash over some area:
> > Eg: there are 1000 grids that have valid value in the observed
file, 950
> > grids that have valid value in the model. There were only 900
grids that
> > were captured both by model and obs. Which means there were 100
grids
> > detected by obs. ONLY and 50 grids detected by model ONLY. In my
> > understanding, currently MET ONLY includes the paired 900 grids
for
> > calculation like HIST, Rely, and ROC, it just ignores the 150
grids which
> > are either captched by obs. or by model only, am I correct?
> >
> > For CSI and Grid-Diag, is there a sample of configure file for
that?
> >
> > Thank you.
> > Binyu
> >
> >
> > On Tue, Jul 7, 2020 at 5:56 PM Tara Jensen via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > Here are some answers to your questions:
> > >
> > > 1. Reli (Reliability), ROC, and Hist diagrams have to be based
on
> paired
> > > data. Reliability and ROC are event based scores that require
an
> > > observation or analysis value to define whether the event
occurred or
> > not.
> > > By definition, Rank and PIT Histograms are a measure of where
the
> > > observation lies within the ranked set of values from the
ensemble
> > > members.
> > >
> > > I don't think I understand what you mean by "how about those
grids
> that
> > > have either ONLY forecast (no obs) or obs. (no forecast)?". MET
does
> not
> > > expect the forecast and obs to be in the same file. It actually
> expects
> > > two files to be passed in and it does the heavy lifting of
pairing up
> the
> > > data for the statistics computation.
> > >
> > > 2. MET can compute categorical statistics, such as CSI, for each
> ensemble
> > > member and the ensemble mean field by passing the members and/or
mean
> > field
> > > into MET individually. Time series, box plots, and bar plots of
CSI
> can
> > > then be plotted using METviewer.
> > >
> > > 3. MET has a new tool called Grid-Diag that computes the PDF of
one or
> > two
> > > fields from a given file. It could be used to do this for each
> ensemble
> > > member and/or ensemble mean fields if desired. The CDF could
then be
> > > derived from the PDF plotted using your own script if you'd
like. We
> do
> > > not have plotting support available for the output of the Grid-
Diag
> tool
> > > yet.
> > >
> > > Hopefully this addresses your questions. Please let us know if
it
> > doesn't.
> > > Cheers, Tara
> > >
> > > On Tue, Jul 7, 2020 at 2:33 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Tue Jul 07 14:33:01 2020: Request 95843 was acted upon.
> > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > Queue: met_help
> > > > Subject: CSI and CDF plot
> > > > Owner: Nobody
> > > > Requestors: binyu.wang at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95843
> > >
> > > >
> > > >
> > > > Hello,
> > > >
> > > > This is Binyu from NOAA/EMC. I have 3 questions about using
> > MET/METViewer
> > > > to for model verification :
> > > >
> > > > 1. Currently all the stats/plots (rely/ROC/Hist) obtained
from
> > > > MET/METviewer are calculated based on the paired data ONLY:
ie. both
> > > model
> > > > and obs. have valid data in the grid. But how about those
grids that
> > have
> > > > either ONLY forecast (no obs) or obs. (no forecast)?
> > > >
> > > > 2. For ensemble forecast, can we use MET/METViewer to make CSI
plot?
> > > > 3. Same as above, can we use MET/METViewer to make CDF plot?
> > > >
> > > > Thank you very much.
> > > > Binyu
> > > >
> > > >
> > >
> > > --
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > Tara Jensen
> > > Project Manager II
> > > NCAR RAL and DTC
> > > PO Box 3000, Boulder, Colorado 80307 USA
> > > +1 303-497-8479 jensen at ucar.edu
> > >
> > >
> >
> >
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Tara Jensen
> Project Manager II
> NCAR RAL and DTC
> PO Box 3000, Boulder, Colorado 80307 USA
> +1 303-497-8479 jensen at ucar.edu
>
>
------------------------------------------------
Subject: CSI and CDF plot
From: binyu.wang at noaa.gov
Time: Thu Jul 09 12:26:13 2020
Hello John,
Actually I did have two more questions about CSI:
1. I used "grid_stat" too to create CSI by setting "nbrcts = BOTH;" in
the
outptu_flag. When I tried to make plot using METViewer, I only find "
NBR_CSI" under "Y1 Dependent (Forecast) Variables:" is " NBR_CSI " the
SAME
thing as "CSI" mentioned under
"https://dtcenter.org/book/export/html/1105"?
I didn't find any definition about " NBR_CSI" in the documentation.
2. Assume " NBR_CSI " is the same as "CSI", however I can only get CSI
for
one forecast member. So how to put multiple members on one plot using
MET/METViewer?
Thank you.
Binyu
On Wed, Jul 8, 2020 at 11:04 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> According to our records, your request has been resolved. If you
have any
> further questions or concerns, please respond to this message.
>
------------------------------------------------
Subject: CSI and CDF plot
From: John Halley Gotway
Time: Tue Aug 11 16:25:02 2020
Binyu,
The NBRCTC line type contains contingency table counts created when
doing
categorical evaluation of the fractional coverage fields generated
during
the neighborhood verification process. Similarly, the NBRCTS line type
contains the corresponding statistics derived from those contingency
table
counts. The NBR_CSI statistic in METviewer is the CSI column from the
NBRCTS line type from Grid-Stat.
However, I really want to caution you that the NBRCTS statistics are
not
widely used, to my knowledge. Instead, the Fractions Skill Score (FSS)
statistic from the NBRCNT line type is by far the most commonly used
neighborhood verification statistic. I'd recommend using that
statistic,
unless you have a good reason to use the NBRCTS statistics instead.
Hope that helps.
Thanks,
John Halley Gotway
On Thu, Jul 9, 2020 at 12:26 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95843 >
>
> Hello John,
>
> Actually I did have two more questions about CSI:
>
> 1. I used "grid_stat" too to create CSI by setting "nbrcts = BOTH;"
in the
> outptu_flag. When I tried to make plot using METViewer, I only find
"
> NBR_CSI" under "Y1 Dependent (Forecast) Variables:" is " NBR_CSI "
the SAME
> thing as "CSI" mentioned under "
> https://dtcenter.org/book/export/html/1105"?
> I didn't find any definition about " NBR_CSI" in the documentation.
>
> 2. Assume " NBR_CSI " is the same as "CSI", however I can only get
CSI for
> one forecast member. So how to put multiple members on one plot
using
> MET/METViewer?
>
> Thank you.
> Binyu
>
> On Wed, Jul 8, 2020 at 11:04 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > According to our records, your request has been resolved. If you
have any
> > further questions or concerns, please respond to this message.
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list