[Met_help] [rt.rap.ucar.edu #95286] History for ROC curve points and probability contingency table files
John Halley Gotway via RT
met_help at ucar.edu
Tue May 19 12:12:33 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi everybody!
I hope you’re doing OK in these worrying times. I wish you, your teams and
your loved ones health and safety.
I want to generate roc curves, and probability contingency table counts,
using pointstat (Wind, temp ...) and gridstat (precipitation). In the
configuration files, I set:
*pct = BOTH;*
*pstd = BOTH;*
*prc = BOTH;*
to indicate that probability contingency table counts, probability
statistics and probabilistic ROC curve points should be created.
The created pct, pstd and prc text files are empty and they contain only
heading line. In the stat file the LINE_TYPE columns with PCT or PSTD or
PRC are missing.
I attached with this message examples of configuration files, stat file and
PRC output files. I don't know what I have to do to get these files?
Any help would be greatly appreciated.
Thanking you most sincerely for your time and consideration.
Cordially,
Rachid Moustabchir
Assistant Professor
Science Faculty, Agadir
Ibn Zohr Uiversity
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: ROC curve points and probability contingency table files
From: Moustabchir Rachid
Time: Mon May 18 05:15:52 2020
Hi again!
Please, excuse my ignorance. By passing through some old messages in
the
met help database, I find that I need to use pop probability of
precipitation forecast to be able to use probability tables for
precipitation.
So I have to create probability fields in some way and pass that to
the
Grid-Stat. Is it true ?
I really don't have any idea how to do that. Any suggestions are
welcome.
Thanks
Rachid Moustabchir
On Mon, 18 May 2020 at 09:22, met_help at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> Greetings,
>
> This message has been automatically generated in response to the
creation
> of a trouble ticket regarding:
> "ROC curve points and probability contingency table files",
> a summary of which appears below.
>
> There is no need to reply to this message right now. Your ticket
has been
> assigned an ID of [rt.rap.ucar.edu #95286].
>
> Please include the string:
>
> [rt.rap.ucar.edu #95286]
>
> in the subject line of all future correspondence about this issue.
To do
> so, you may reply to this message.
>
> For more information, please see:
>
> MET Online Tutorial:
>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>
> MET Users Guide:
> https://www.dtcenter.org/met/users/docs/overview.php
>
> MET FAQs:
> https://www.dtcenter.org/met/users/support/faqs/index.php
>
> MET-Help Email Archive:
> http://mailman.ucar.edu/pipermail/met_help
>
> Thank you,
> met_help at ucar.edu
>
>
-------------------------------------------------------------------------
> Hi everybody!
>
> I hope you’re doing OK in these worrying times. I wish you, your
teams and
> your loved ones health and safety.
>
> I want to generate roc curves, and probability contingency table
counts,
> using pointstat (Wind, temp ...) and gridstat (precipitation). In
the
> configuration files, I set:
>
> *pct = BOTH;*
>
> *pstd = BOTH;*
>
> *prc = BOTH;*
>
> to indicate that probability contingency table counts, probability
> statistics and probabilistic ROC curve points should be created.
>
> The created pct, pstd and prc text files are empty and they contain
only
> heading line. In the stat file the LINE_TYPE columns with PCT or
PSTD or
> PRC are missing.
>
> I attached with this message examples of configuration files, stat
file and
> PRC output files. I don't know what I have to do to get these files?
>
> Any help would be greatly appreciated.
>
> Thanking you most sincerely for your time and consideration.
>
> Cordially,
>
> Rachid Moustabchir
>
> Assistant Professor
>
> Science Faculty, Agadir
>
> Ibn Zohr Uiversity
>
>
------------------------------------------------
Subject: ROC curve points and probability contingency table files
From: John Halley Gotway
Time: Mon May 18 07:47:39 2020
Rachid,
I see that you’re interested in generating ROC curves but you don’t
have a
probability forecast data. Can you please describe your model data? Is
it a
single deterministic model model? Or is it an ensemble of models?
Thanks,
John
On Mon, May 18, 2020 at 5:16 AM Moustabchir Rachid via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
>
> Hi again!
>
> Please, excuse my ignorance. By passing through some old messages in
the
> met help database, I find that I need to use pop probability of
> precipitation forecast to be able to use probability tables for
> precipitation.
>
> So I have to create probability fields in some way and pass that to
the
> Grid-Stat. Is it true ?
>
> I really don't have any idea how to do that. Any suggestions are
welcome.
>
> Thanks
>
> Rachid Moustabchir
>
> On Mon, 18 May 2020 at 09:22, met_help at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Greetings,
> >
> > This message has been automatically generated in response to the
creation
> > of a trouble ticket regarding:
> > "ROC curve points and probability contingency table
files",
> > a summary of which appears below.
> >
> > There is no need to reply to this message right now. Your ticket
has
> been
> > assigned an ID of [rt.rap.ucar.edu #95286].
> >
> > Please include the string:
> >
> > [rt.rap.ucar.edu #95286]
> >
> > in the subject line of all future correspondence about this issue.
To do
> > so, you may reply to this message.
> >
> > For more information, please see:
> >
> > MET Online Tutorial:
> >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> >
> > MET Users Guide:
> > https://www.dtcenter.org/met/users/docs/overview.php
> >
> > MET FAQs:
> > https://www.dtcenter.org/met/users/support/faqs/index.php
> >
> > MET-Help Email Archive:
> > http://mailman.ucar.edu/pipermail/met_help
> >
> > Thank you,
> > met_help at ucar.edu
> >
> >
-------------------------------------------------------------------------
> > Hi everybody!
> >
> > I hope you’re doing OK in these worrying times. I wish you, your
teams
> and
> > your loved ones health and safety.
> >
> > I want to generate roc curves, and probability contingency table
counts,
> > using pointstat (Wind, temp ...) and gridstat (precipitation). In
the
> > configuration files, I set:
> >
> > *pct = BOTH;*
> >
> > *pstd = BOTH;*
> >
> > *prc = BOTH;*
> >
> > to indicate that probability contingency table counts, probability
> > statistics and probabilistic ROC curve points should be created.
> >
> > The created pct, pstd and prc text files are empty and they
contain only
> > heading line. In the stat file the LINE_TYPE columns with PCT or
PSTD or
> > PRC are missing.
> >
> > I attached with this message examples of configuration files, stat
file
> and
> > PRC output files. I don't know what I have to do to get these
files?
> >
> > Any help would be greatly appreciated.
> >
> > Thanking you most sincerely for your time and consideration.
> >
> > Cordially,
> >
> > Rachid Moustabchir
> >
> > Assistant Professor
> >
> > Science Faculty, Agadir
> >
> > Ibn Zohr Uiversity
> >
> >
>
>
------------------------------------------------
Subject: ROC curve points and probability contingency table files
From: Moustabchir Rachid
Time: Mon May 18 11:26:07 2020
Thanks John for your answer.
My data are a single deterministic model. I don’t know if this is
sufficient to create pop data. But I do have 4 different model
configuration output data. The first idea was to compare these
configurations and find a good one.
Rachid
On Mon, 18 May 2020 at 13:47, John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Rachid,
>
> I see that you’re interested in generating ROC curves but you don’t
have a
> probability forecast data. Can you please describe your model data?
Is it a
> single deterministic model model? Or is it an ensemble of models?
>
> Thanks,
> John
>
> On Mon, May 18, 2020 at 5:16 AM Moustabchir Rachid via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> >
> > Hi again!
> >
> > Please, excuse my ignorance. By passing through some old messages
in the
> > met help database, I find that I need to use pop probability of
> > precipitation forecast to be able to use probability tables for
> > precipitation.
> >
> > So I have to create probability fields in some way and pass that
to the
> > Grid-Stat. Is it true ?
> >
> > I really don't have any idea how to do that. Any suggestions are
welcome.
> >
> > Thanks
> >
> > Rachid Moustabchir
> >
> > On Mon, 18 May 2020 at 09:22, met_help at ucar.edu via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Greetings,
> > >
> > > This message has been automatically generated in response to the
> creation
> > > of a trouble ticket regarding:
> > > "ROC curve points and probability contingency table
files",
> > > a summary of which appears below.
> > >
> > > There is no need to reply to this message right now. Your
ticket has
> > been
> > > assigned an ID of [rt.rap.ucar.edu #95286].
> > >
> > > Please include the string:
> > >
> > > [rt.rap.ucar.edu #95286]
> > >
> > > in the subject line of all future correspondence about this
issue. To
> do
> > > so, you may reply to this message.
> > >
> > > For more information, please see:
> > >
> > > MET Online Tutorial:
> > >
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > >
> > > MET Users Guide:
> > > https://www.dtcenter.org/met/users/docs/overview.php
> > >
> > > MET FAQs:
> > > https://www.dtcenter.org/met/users/support/faqs/index.php
> > >
> > > MET-Help Email Archive:
> > > http://mailman.ucar.edu/pipermail/met_help
> > >
> > > Thank you,
> > > met_help at ucar.edu
> > >
> > >
>
-------------------------------------------------------------------------
> > > Hi everybody!
> > >
> > > I hope you’re doing OK in these worrying times. I wish you, your
teams
> > and
> > > your loved ones health and safety.
> > >
> > > I want to generate roc curves, and probability contingency table
> counts,
> > > using pointstat (Wind, temp ...) and gridstat (precipitation).
In the
> > > configuration files, I set:
> > >
> > > *pct = BOTH;*
> > >
> > > *pstd = BOTH;*
> > >
> > > *prc = BOTH;*
> > >
> > > to indicate that probability contingency table counts,
probability
> > > statistics and probabilistic ROC curve points should be created.
> > >
> > > The created pct, pstd and prc text files are empty and they
contain
> only
> > > heading line. In the stat file the LINE_TYPE columns with PCT or
PSTD
> or
> > > PRC are missing.
> > >
> > > I attached with this message examples of configuration files,
stat file
> > and
> > > PRC output files. I don't know what I have to do to get these
files?
> > >
> > > Any help would be greatly appreciated.
> > >
> > > Thanking you most sincerely for your time and consideration.
> > >
> > > Cordially,
> > >
> > > Rachid Moustabchir
> > >
> > > Assistant Professor
> > >
> > > Science Faculty, Agadir
> > >
> > > Ibn Zohr Uiversity
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ROC curve points and probability contingency table files
From: John Halley Gotway
Time: Mon May 18 12:07:18 2020
Rachid,
ROC curves are typically made in the evaluation of probabilistic data.
However, they can also be constructed from deterministic data as well,
using a series of 2x2 contingency tables stored in the CTC line type
from
MET. I believe METviewer does have the ability to generate ROC
diagrams
from CTC lines but you'd need to configure your precip thresholds in
an odd
way. And while this is supposed to work, I haven't actually run
METviewer
in this way myself.
You choose a single observation threshold, for each "precip > 10 mm".
And
then you choose many forecast thresholds (e.g. >0, >2, >4, >6, >8,
>10,
>12, >14, >16, >18, >20). You run Grid-Stat or Point-Stat to compute
CTC
lines for each of these comparisons (e.g. fcst>0 vs obs>10, fcst>2 vs
obs>10, fcst>4 vs obs>10, and so on).
Then you use METviewer to aggregate those results into a ROC diagram.
In my opinion, that's a lot of steps to produce a ROC diagram...
especially
if you don't have access to METviewer to facilitate it. You could
instead
consider choosing relevant thresholds, and comparing those multiple
runs
using contingency table stats, like CSI or ETS (called GSS in MET).
John
On Mon, May 18, 2020 at 11:26 AM Moustabchir Rachid via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
>
> Thanks John for your answer.
>
> My data are a single deterministic model. I don’t know if this is
> sufficient to create pop data. But I do have 4 different model
> configuration output data. The first idea was to compare these
> configurations and find a good one.
> Rachid
>
> On Mon, 18 May 2020 at 13:47, John Halley Gotway via RT
<met_help at ucar.edu
> >
> wrote:
>
> > Rachid,
> >
> > I see that you’re interested in generating ROC curves but you
don’t have
> a
> > probability forecast data. Can you please describe your model
data? Is
> it a
> > single deterministic model model? Or is it an ensemble of models?
> >
> > Thanks,
> > John
> >
> > On Mon, May 18, 2020 at 5:16 AM Moustabchir Rachid via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> > >
> > > Hi again!
> > >
> > > Please, excuse my ignorance. By passing through some old
messages in
> the
> > > met help database, I find that I need to use pop probability of
> > > precipitation forecast to be able to use probability tables for
> > > precipitation.
> > >
> > > So I have to create probability fields in some way and pass that
to the
> > > Grid-Stat. Is it true ?
> > >
> > > I really don't have any idea how to do that. Any suggestions are
> welcome.
> > >
> > > Thanks
> > >
> > > Rachid Moustabchir
> > >
> > > On Mon, 18 May 2020 at 09:22, met_help at ucar.edu via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Greetings,
> > > >
> > > > This message has been automatically generated in response to
the
> > creation
> > > > of a trouble ticket regarding:
> > > > "ROC curve points and probability contingency table
files",
> > > > a summary of which appears below.
> > > >
> > > > There is no need to reply to this message right now. Your
ticket has
> > > been
> > > > assigned an ID of [rt.rap.ucar.edu #95286].
> > > >
> > > > Please include the string:
> > > >
> > > > [rt.rap.ucar.edu #95286]
> > > >
> > > > in the subject line of all future correspondence about this
issue. To
> > do
> > > > so, you may reply to this message.
> > > >
> > > > For more information, please see:
> > > >
> > > > MET Online Tutorial:
> > > >
> >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > >
> > > > MET Users Guide:
> > > > https://www.dtcenter.org/met/users/docs/overview.php
> > > >
> > > > MET FAQs:
> > > > https://www.dtcenter.org/met/users/support/faqs/index.php
> > > >
> > > > MET-Help Email Archive:
> > > > http://mailman.ucar.edu/pipermail/met_help
> > > >
> > > > Thank you,
> > > > met_help at ucar.edu
> > > >
> > > >
> >
-------------------------------------------------------------------------
> > > > Hi everybody!
> > > >
> > > > I hope you’re doing OK in these worrying times. I wish you,
your
> teams
> > > and
> > > > your loved ones health and safety.
> > > >
> > > > I want to generate roc curves, and probability contingency
table
> > counts,
> > > > using pointstat (Wind, temp ...) and gridstat (precipitation).
In the
> > > > configuration files, I set:
> > > >
> > > > *pct = BOTH;*
> > > >
> > > > *pstd = BOTH;*
> > > >
> > > > *prc = BOTH;*
> > > >
> > > > to indicate that probability contingency table counts,
probability
> > > > statistics and probabilistic ROC curve points should be
created.
> > > >
> > > > The created pct, pstd and prc text files are empty and they
contain
> > only
> > > > heading line. In the stat file the LINE_TYPE columns with PCT
or PSTD
> > or
> > > > PRC are missing.
> > > >
> > > > I attached with this message examples of configuration files,
stat
> file
> > > and
> > > > PRC output files. I don't know what I have to do to get these
files?
> > > >
> > > > Any help would be greatly appreciated.
> > > >
> > > > Thanking you most sincerely for your time and consideration.
> > > >
> > > > Cordially,
> > > >
> > > > Rachid Moustabchir
> > > >
> > > > Assistant Professor
> > > >
> > > > Science Faculty, Agadir
> > > >
> > > > Ibn Zohr Uiversity
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ROC curve points and probability contingency table files
From: Moustabchir Rachid
Time: Mon May 18 13:06:47 2020
Thank you very much John for these explanations !
roc curve is harder than it sounds, but I will give it a try.
John, what you mean by "you'd need to configure your precip thresholds
in
an odd", because in the explanations you are using even values: >0,
>2, >4,
>6, >8, >10, >12, >14, >16, >18, >20.
Thanks
On Mon, 18 May 2020 at 18:07, John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Rachid,
>
> ROC curves are typically made in the evaluation of probabilistic
data.
> However, they can also be constructed from deterministic data as
well,
> using a series of 2x2 contingency tables stored in the CTC line type
from
> MET. I believe METviewer does have the ability to generate ROC
diagrams
> from CTC lines but you'd need to configure your precip thresholds in
an odd
> way. And while this is supposed to work, I haven't actually run
METviewer
> in this way myself.
>
> You choose a single observation threshold, for each "precip > 10
mm". And
> then you choose many forecast thresholds (e.g. >0, >2, >4, >6, >8,
>10,
> >12, >14, >16, >18, >20). You run Grid-Stat or Point-Stat to compute
CTC
> lines for each of these comparisons (e.g. fcst>0 vs obs>10, fcst>2
vs
> obs>10, fcst>4 vs obs>10, and so on).
>
> Then you use METviewer to aggregate those results into a ROC
diagram.
>
> In my opinion, that's a lot of steps to produce a ROC diagram...
especially
> if you don't have access to METviewer to facilitate it. You could
instead
> consider choosing relevant thresholds, and comparing those multiple
runs
> using contingency table stats, like CSI or ETS (called GSS in MET).
>
> John
>
> On Mon, May 18, 2020 at 11:26 AM Moustabchir Rachid via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> >
> > Thanks John for your answer.
> >
> > My data are a single deterministic model. I don’t know if this is
> > sufficient to create pop data. But I do have 4 different model
> > configuration output data. The first idea was to compare these
> > configurations and find a good one.
> > Rachid
> >
> > On Mon, 18 May 2020 at 13:47, John Halley Gotway via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > > Rachid,
> > >
> > > I see that you’re interested in generating ROC curves but you
don’t
> have
> > a
> > > probability forecast data. Can you please describe your model
data? Is
> > it a
> > > single deterministic model model? Or is it an ensemble of
models?
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, May 18, 2020 at 5:16 AM Moustabchir Rachid via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286
>
> > > >
> > > > Hi again!
> > > >
> > > > Please, excuse my ignorance. By passing through some old
messages in
> > the
> > > > met help database, I find that I need to use pop probability
of
> > > > precipitation forecast to be able to use probability tables
for
> > > > precipitation.
> > > >
> > > > So I have to create probability fields in some way and pass
that to
> the
> > > > Grid-Stat. Is it true ?
> > > >
> > > > I really don't have any idea how to do that. Any suggestions
are
> > welcome.
> > > >
> > > > Thanks
> > > >
> > > > Rachid Moustabchir
> > > >
> > > > On Mon, 18 May 2020 at 09:22, met_help at ucar.edu via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Greetings,
> > > > >
> > > > > This message has been automatically generated in response to
the
> > > creation
> > > > > of a trouble ticket regarding:
> > > > > "ROC curve points and probability contingency table
files",
> > > > > a summary of which appears below.
> > > > >
> > > > > There is no need to reply to this message right now. Your
ticket
> has
> > > > been
> > > > > assigned an ID of [rt.rap.ucar.edu #95286].
> > > > >
> > > > > Please include the string:
> > > > >
> > > > > [rt.rap.ucar.edu #95286]
> > > > >
> > > > > in the subject line of all future correspondence about this
issue.
> To
> > > do
> > > > > so, you may reply to this message.
> > > > >
> > > > > For more information, please see:
> > > > >
> > > > > MET Online Tutorial:
> > > > >
> > >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > >
> > > > > MET Users Guide:
> > > > > https://www.dtcenter.org/met/users/docs/overview.php
> > > > >
> > > > > MET FAQs:
> > > > > https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > >
> > > > > MET-Help Email Archive:
> > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > >
> > > > > Thank you,
> > > > > met_help at ucar.edu
> > > > >
> > > > >
> > >
>
-------------------------------------------------------------------------
> > > > > Hi everybody!
> > > > >
> > > > > I hope you’re doing OK in these worrying times. I wish you,
your
> > teams
> > > > and
> > > > > your loved ones health and safety.
> > > > >
> > > > > I want to generate roc curves, and probability contingency
table
> > > counts,
> > > > > using pointstat (Wind, temp ...) and gridstat
(precipitation). In
> the
> > > > > configuration files, I set:
> > > > >
> > > > > *pct = BOTH;*
> > > > >
> > > > > *pstd = BOTH;*
> > > > >
> > > > > *prc = BOTH;*
> > > > >
> > > > > to indicate that probability contingency table counts,
probability
> > > > > statistics and probabilistic ROC curve points should be
created.
> > > > >
> > > > > The created pct, pstd and prc text files are empty and they
contain
> > > only
> > > > > heading line. In the stat file the LINE_TYPE columns with
PCT or
> PSTD
> > > or
> > > > > PRC are missing.
> > > > >
> > > > > I attached with this message examples of configuration
files, stat
> > file
> > > > and
> > > > > PRC output files. I don't know what I have to do to get
these
> files?
> > > > >
> > > > > Any help would be greatly appreciated.
> > > > >
> > > > > Thanking you most sincerely for your time and consideration.
> > > > >
> > > > > Cordially,
> > > > >
> > > > > Rachid Moustabchir
> > > > >
> > > > > Assistant Professor
> > > > >
> > > > > Science Faculty, Agadir
> > > > >
> > > > > Ibn Zohr Uiversity
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ROC curve points and probability contingency table files
From: John Halley Gotway
Time: Mon May 18 13:12:34 2020
Rachid,
Typically when you verify precip, you apply the same threshold to both
the
forecast and observation data. But to make a ROC curve from
deterministic
data, I believe that you hold the obs thresholds constant and vary the
forecast threshold... and that's what I described as being "odd".
Here's
an example of config file settings which illustrate what I mean:
fcst = {
field = [
{
name = "APCP";
level = "A03";
cat_thresh = [ >0, >2, >4, >6, >8, >10, >12, >14, >16, >18,
>20 ];
}
];
}
obs = {
field = [
{
name = "APCP";
level = "A03";
cat_thresh = [ >10, >10, >10, >10, >10, >10, >10, >10, >10,
>10,
>10 ];
}
];
}
On Mon, May 18, 2020 at 1:06 PM Moustabchir Rachid via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
>
> Thank you very much John for these explanations !
> roc curve is harder than it sounds, but I will give it a try.
> John, what you mean by "you'd need to configure your precip
thresholds in
> an odd", because in the explanations you are using even values: >0,
>2, >4,
> >6, >8, >10, >12, >14, >16, >18, >20.
> Thanks
>
> On Mon, 18 May 2020 at 18:07, John Halley Gotway via RT
<met_help at ucar.edu
> >
> wrote:
>
> > Rachid,
> >
> > ROC curves are typically made in the evaluation of probabilistic
data.
> > However, they can also be constructed from deterministic data as
well,
> > using a series of 2x2 contingency tables stored in the CTC line
type from
> > MET. I believe METviewer does have the ability to generate ROC
diagrams
> > from CTC lines but you'd need to configure your precip thresholds
in an
> odd
> > way. And while this is supposed to work, I haven't actually run
METviewer
> > in this way myself.
> >
> > You choose a single observation threshold, for each "precip > 10
mm".
> And
> > then you choose many forecast thresholds (e.g. >0, >2, >4, >6, >8,
>10,
> > >12, >14, >16, >18, >20). You run Grid-Stat or Point-Stat to
compute CTC
> > lines for each of these comparisons (e.g. fcst>0 vs obs>10, fcst>2
vs
> > obs>10, fcst>4 vs obs>10, and so on).
> >
> > Then you use METviewer to aggregate those results into a ROC
diagram.
> >
> > In my opinion, that's a lot of steps to produce a ROC diagram...
> especially
> > if you don't have access to METviewer to facilitate it. You could
instead
> > consider choosing relevant thresholds, and comparing those
multiple runs
> > using contingency table stats, like CSI or ETS (called GSS in
MET).
> >
> > John
> >
> > On Mon, May 18, 2020 at 11:26 AM Moustabchir Rachid via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> > >
> > > Thanks John for your answer.
> > >
> > > My data are a single deterministic model. I don’t know if this
is
> > > sufficient to create pop data. But I do have 4 different model
> > > configuration output data. The first idea was to compare these
> > > configurations and find a good one.
> > > Rachid
> > >
> > > On Mon, 18 May 2020 at 13:47, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > > Rachid,
> > > >
> > > > I see that you’re interested in generating ROC curves but you
don’t
> > have
> > > a
> > > > probability forecast data. Can you please describe your model
data?
> Is
> > > it a
> > > > single deterministic model model? Or is it an ensemble of
models?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, May 18, 2020 at 5:16 AM Moustabchir Rachid via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> > > > >
> > > > > Hi again!
> > > > >
> > > > > Please, excuse my ignorance. By passing through some old
messages
> in
> > > the
> > > > > met help database, I find that I need to use pop probability
of
> > > > > precipitation forecast to be able to use probability tables
for
> > > > > precipitation.
> > > > >
> > > > > So I have to create probability fields in some way and pass
that to
> > the
> > > > > Grid-Stat. Is it true ?
> > > > >
> > > > > I really don't have any idea how to do that. Any suggestions
are
> > > welcome.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Rachid Moustabchir
> > > > >
> > > > > On Mon, 18 May 2020 at 09:22, met_help at ucar.edu via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Greetings,
> > > > > >
> > > > > > This message has been automatically generated in response
to the
> > > > creation
> > > > > > of a trouble ticket regarding:
> > > > > > "ROC curve points and probability contingency
table
> files",
> > > > > > a summary of which appears below.
> > > > > >
> > > > > > There is no need to reply to this message right now. Your
ticket
> > has
> > > > > been
> > > > > > assigned an ID of [rt.rap.ucar.edu #95286].
> > > > > >
> > > > > > Please include the string:
> > > > > >
> > > > > > [rt.rap.ucar.edu #95286]
> > > > > >
> > > > > > in the subject line of all future correspondence about
this
> issue.
> > To
> > > > do
> > > > > > so, you may reply to this message.
> > > > > >
> > > > > > For more information, please see:
> > > > > >
> > > > > > MET Online Tutorial:
> > > > > >
> > > >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > >
> > > > > > MET Users Guide:
> > > > > > https://www.dtcenter.org/met/users/docs/overview.php
> > > > > >
> > > > > > MET FAQs:
> > > > > >
https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > >
> > > > > > MET-Help Email Archive:
> > > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > > >
> > > > > > Thank you,
> > > > > > met_help at ucar.edu
> > > > > >
> > > > > >
> > > >
> >
-------------------------------------------------------------------------
> > > > > > Hi everybody!
> > > > > >
> > > > > > I hope you’re doing OK in these worrying times. I wish
you, your
> > > teams
> > > > > and
> > > > > > your loved ones health and safety.
> > > > > >
> > > > > > I want to generate roc curves, and probability contingency
table
> > > > counts,
> > > > > > using pointstat (Wind, temp ...) and gridstat
(precipitation). In
> > the
> > > > > > configuration files, I set:
> > > > > >
> > > > > > *pct = BOTH;*
> > > > > >
> > > > > > *pstd = BOTH;*
> > > > > >
> > > > > > *prc = BOTH;*
> > > > > >
> > > > > > to indicate that probability contingency table counts,
> probability
> > > > > > statistics and probabilistic ROC curve points should be
created.
> > > > > >
> > > > > > The created pct, pstd and prc text files are empty and
they
> contain
> > > > only
> > > > > > heading line. In the stat file the LINE_TYPE columns with
PCT or
> > PSTD
> > > > or
> > > > > > PRC are missing.
> > > > > >
> > > > > > I attached with this message examples of configuration
files,
> stat
> > > file
> > > > > and
> > > > > > PRC output files. I don't know what I have to do to get
these
> > files?
> > > > > >
> > > > > > Any help would be greatly appreciated.
> > > > > >
> > > > > > Thanking you most sincerely for your time and
consideration.
> > > > > >
> > > > > > Cordially,
> > > > > >
> > > > > > Rachid Moustabchir
> > > > > >
> > > > > > Assistant Professor
> > > > > >
> > > > > > Science Faculty, Agadir
> > > > > >
> > > > > > Ibn Zohr Uiversity
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ROC curve points and probability contingency table files
From: Moustabchir Rachid
Time: Mon May 18 13:16:01 2020
Got it.
Thank you very much John.
Rachid
On Mon, 18 May 2020 at 19:12, John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Rachid,
>
> Typically when you verify precip, you apply the same threshold to
both the
> forecast and observation data. But to make a ROC curve from
deterministic
> data, I believe that you hold the obs thresholds constant and vary
the
> forecast threshold... and that's what I described as being "odd".
Here's
> an example of config file settings which illustrate what I mean:
>
> fcst = {
> field = [
> {
> name = "APCP";
> level = "A03";
> cat_thresh = [ >0, >2, >4, >6, >8, >10, >12, >14, >16, >18,
>20 ];
> }
> ];
> }
> obs = {
> field = [
> {
> name = "APCP";
> level = "A03";
> cat_thresh = [ >10, >10, >10, >10, >10, >10, >10, >10, >10,
>10,
> >10 ];
> }
> ];
> }
>
>
>
> On Mon, May 18, 2020 at 1:06 PM Moustabchir Rachid via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> >
> > Thank you very much John for these explanations !
> > roc curve is harder than it sounds, but I will give it a try.
> > John, what you mean by "you'd need to configure your precip
thresholds in
> > an odd", because in the explanations you are using even values:
>0, >2,
> >4,
> > >6, >8, >10, >12, >14, >16, >18, >20.
> > Thanks
> >
> > On Mon, 18 May 2020 at 18:07, John Halley Gotway via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > > Rachid,
> > >
> > > ROC curves are typically made in the evaluation of probabilistic
data.
> > > However, they can also be constructed from deterministic data as
well,
> > > using a series of 2x2 contingency tables stored in the CTC line
type
> from
> > > MET. I believe METviewer does have the ability to generate ROC
diagrams
> > > from CTC lines but you'd need to configure your precip
thresholds in an
> > odd
> > > way. And while this is supposed to work, I haven't actually run
> METviewer
> > > in this way myself.
> > >
> > > You choose a single observation threshold, for each "precip > 10
mm".
> > And
> > > then you choose many forecast thresholds (e.g. >0, >2, >4, >6,
>8, >10,
> > > >12, >14, >16, >18, >20). You run Grid-Stat or Point-Stat to
compute
> CTC
> > > lines for each of these comparisons (e.g. fcst>0 vs obs>10,
fcst>2 vs
> > > obs>10, fcst>4 vs obs>10, and so on).
> > >
> > > Then you use METviewer to aggregate those results into a ROC
diagram.
> > >
> > > In my opinion, that's a lot of steps to produce a ROC diagram...
> > especially
> > > if you don't have access to METviewer to facilitate it. You
could
> instead
> > > consider choosing relevant thresholds, and comparing those
multiple
> runs
> > > using contingency table stats, like CSI or ETS (called GSS in
MET).
> > >
> > > John
> > >
> > > On Mon, May 18, 2020 at 11:26 AM Moustabchir Rachid via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286
>
> > > >
> > > > Thanks John for your answer.
> > > >
> > > > My data are a single deterministic model. I don’t know if this
is
> > > > sufficient to create pop data. But I do have 4 different model
> > > > configuration output data. The first idea was to compare these
> > > > configurations and find a good one.
> > > > Rachid
> > > >
> > > > On Mon, 18 May 2020 at 13:47, John Halley Gotway via RT <
> > > met_help at ucar.edu
> > > > >
> > > > wrote:
> > > >
> > > > > Rachid,
> > > > >
> > > > > I see that you’re interested in generating ROC curves but
you don’t
> > > have
> > > > a
> > > > > probability forecast data. Can you please describe your
model data?
> > Is
> > > > it a
> > > > > single deterministic model model? Or is it an ensemble of
models?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, May 18, 2020 at 5:16 AM Moustabchir Rachid via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> > > > > >
> > > > > > Hi again!
> > > > > >
> > > > > > Please, excuse my ignorance. By passing through some old
messages
> > in
> > > > the
> > > > > > met help database, I find that I need to use pop
probability of
> > > > > > precipitation forecast to be able to use probability
tables for
> > > > > > precipitation.
> > > > > >
> > > > > > So I have to create probability fields in some way and
pass that
> to
> > > the
> > > > > > Grid-Stat. Is it true ?
> > > > > >
> > > > > > I really don't have any idea how to do that. Any
suggestions are
> > > > welcome.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Rachid Moustabchir
> > > > > >
> > > > > > On Mon, 18 May 2020 at 09:22, met_help at ucar.edu via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Greetings,
> > > > > > >
> > > > > > > This message has been automatically generated in
response to
> the
> > > > > creation
> > > > > > > of a trouble ticket regarding:
> > > > > > > "ROC curve points and probability contingency
table
> > files",
> > > > > > > a summary of which appears below.
> > > > > > >
> > > > > > > There is no need to reply to this message right now.
Your
> ticket
> > > has
> > > > > > been
> > > > > > > assigned an ID of [rt.rap.ucar.edu #95286].
> > > > > > >
> > > > > > > Please include the string:
> > > > > > >
> > > > > > > [rt.rap.ucar.edu #95286]
> > > > > > >
> > > > > > > in the subject line of all future correspondence about
this
> > issue.
> > > To
> > > > > do
> > > > > > > so, you may reply to this message.
> > > > > > >
> > > > > > > For more information, please see:
> > > > > > >
> > > > > > > MET Online Tutorial:
> > > > > > >
> > > > >
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > >
> > > > > > > MET Users Guide:
> > > > > > > https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > >
> > > > > > > MET FAQs:
> > > > > > >
https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > >
> > > > > > > MET-Help Email Archive:
> > > > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > > > >
> > > > > > > Thank you,
> > > > > > > met_help at ucar.edu
> > > > > > >
> > > > > > >
> > > > >
> > >
>
-------------------------------------------------------------------------
> > > > > > > Hi everybody!
> > > > > > >
> > > > > > > I hope you’re doing OK in these worrying times. I wish
you,
> your
> > > > teams
> > > > > > and
> > > > > > > your loved ones health and safety.
> > > > > > >
> > > > > > > I want to generate roc curves, and probability
contingency
> table
> > > > > counts,
> > > > > > > using pointstat (Wind, temp ...) and gridstat
(precipitation).
> In
> > > the
> > > > > > > configuration files, I set:
> > > > > > >
> > > > > > > *pct = BOTH;*
> > > > > > >
> > > > > > > *pstd = BOTH;*
> > > > > > >
> > > > > > > *prc = BOTH;*
> > > > > > >
> > > > > > > to indicate that probability contingency table counts,
> > probability
> > > > > > > statistics and probabilistic ROC curve points should be
> created.
> > > > > > >
> > > > > > > The created pct, pstd and prc text files are empty and
they
> > contain
> > > > > only
> > > > > > > heading line. In the stat file the LINE_TYPE columns
with PCT
> or
> > > PSTD
> > > > > or
> > > > > > > PRC are missing.
> > > > > > >
> > > > > > > I attached with this message examples of configuration
files,
> > stat
> > > > file
> > > > > > and
> > > > > > > PRC output files. I don't know what I have to do to get
these
> > > files?
> > > > > > >
> > > > > > > Any help would be greatly appreciated.
> > > > > > >
> > > > > > > Thanking you most sincerely for your time and
consideration.
> > > > > > >
> > > > > > > Cordially,
> > > > > > >
> > > > > > > Rachid Moustabchir
> > > > > > >
> > > > > > > Assistant Professor
> > > > > > >
> > > > > > > Science Faculty, Agadir
> > > > > > >
> > > > > > > Ibn Zohr Uiversity
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ROC curve points and probability contingency table files
From: Moustabchir Rachid
Time: Tue May 19 02:51:42 2020
*D**ear **John !*
*With **the*
* clear explanations, I get stat files that contains CTC lines for
every
forecast and obs thresholds. This is an example of the output I
get:V9.0
.... >0 >20 NA NA CTC 254104 347
55379 5 198373V9.0 .... >2 >20 NA
NA CTC 254104 168 4679 184 249073V9.0
....
>4 >20 NA NA CTC 254104 53 2045
299 251707V9.0 .... >6 >20 NA NA
CTC
254104 36 1277 316 252475V9.0 ....
>8
>20 NA NA CTC 254104 26 843
326
252909V9.0 .... >10 >20 NA NA CTC
254104
21 615 331 253137V9.0 .... >12 >20
NA NA CTC 254104 16 444 336
253308V9.0
.... >14 >20 NA NA CTC 254104 10
345 342 253407V9.0 .... >16 >20 NA
NA
CTC 254104 8 283 344 253469V9.0 ....
>18 >20 NA NA CTC 254104 6 241
346 253511*
*It is not really **hard as to get roc curves. For each line we
compute
POFD **and PODY. By plotting PODY versus **POFD **we get roc curves
for **a
given observation threshold. *
*Thank you John for helping me to solve the problem. I have **more**
other
questions :*
1.
*Is it possible to compute the forecast probability for each ctc
data ?
What is the formula **a/(a+b), b/(a+b) … I could not find anything
about
it, since I want to plot reliability diagrams which show the
observed
relative frequencies as a function of the forecasted
probabilities.*
2.
*Can I use **the same procedure** for TMP, DPT, WDIR … with
pointstat
tool ?*
3.
*The third question perhaps will sound ridiculous. For cloud cover,
since values are fractions can I consider these values like
probabilities,
and create probability **files** with gridstat.*
*Thanks*
On Mon, 18 May 2020 at 19:15, Moustabchir Rachid <
rachid.moustabchir at gmail.com> wrote:
> Got it.
> Thank you very much John.
> Rachid
>
> On Mon, 18 May 2020 at 19:12, John Halley Gotway via RT
<met_help at ucar.edu>
> wrote:
>
>> Rachid,
>>
>> Typically when you verify precip, you apply the same threshold to
both the
>> forecast and observation data. But to make a ROC curve from
deterministic
>> data, I believe that you hold the obs thresholds constant and vary
the
>> forecast threshold... and that's what I described as being "odd".
Here's
>> an example of config file settings which illustrate what I mean:
>>
>> fcst = {
>> field = [
>> {
>> name = "APCP";
>> level = "A03";
>> cat_thresh = [ >0, >2, >4, >6, >8, >10, >12, >14, >16, >18,
>20 ];
>> }
>> ];
>> }
>> obs = {
>> field = [
>> {
>> name = "APCP";
>> level = "A03";
>> cat_thresh = [ >10, >10, >10, >10, >10, >10, >10, >10, >10,
>10,
>> >10 ];
>> }
>> ];
>> }
>>
>>
>>
>> On Mon, May 18, 2020 at 1:06 PM Moustabchir Rachid via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
>> >
>> > Thank you very much John for these explanations !
>> > roc curve is harder than it sounds, but I will give it a try.
>> > John, what you mean by "you'd need to configure your precip
thresholds
>> in
>> > an odd", because in the explanations you are using even values:
>0, >2,
>> >4,
>> > >6, >8, >10, >12, >14, >16, >18, >20.
>> > Thanks
>> >
>> > On Mon, 18 May 2020 at 18:07, John Halley Gotway via RT <
>> met_help at ucar.edu
>> > >
>> > wrote:
>> >
>> > > Rachid,
>> > >
>> > > ROC curves are typically made in the evaluation of
probabilistic data.
>> > > However, they can also be constructed from deterministic data
as well,
>> > > using a series of 2x2 contingency tables stored in the CTC line
type
>> from
>> > > MET. I believe METviewer does have the ability to generate ROC
>> diagrams
>> > > from CTC lines but you'd need to configure your precip
thresholds in
>> an
>> > odd
>> > > way. And while this is supposed to work, I haven't actually run
>> METviewer
>> > > in this way myself.
>> > >
>> > > You choose a single observation threshold, for each "precip >
10 mm".
>> > And
>> > > then you choose many forecast thresholds (e.g. >0, >2, >4, >6,
>8,
>> >10,
>> > > >12, >14, >16, >18, >20). You run Grid-Stat or Point-Stat to
compute
>> CTC
>> > > lines for each of these comparisons (e.g. fcst>0 vs obs>10,
fcst>2 vs
>> > > obs>10, fcst>4 vs obs>10, and so on).
>> > >
>> > > Then you use METviewer to aggregate those results into a ROC
diagram.
>> > >
>> > > In my opinion, that's a lot of steps to produce a ROC
diagram...
>> > especially
>> > > if you don't have access to METviewer to facilitate it. You
could
>> instead
>> > > consider choosing relevant thresholds, and comparing those
multiple
>> runs
>> > > using contingency table stats, like CSI or ETS (called GSS in
MET).
>> > >
>> > > John
>> > >
>> > > On Mon, May 18, 2020 at 11:26 AM Moustabchir Rachid via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286
>
>> > > >
>> > > > Thanks John for your answer.
>> > > >
>> > > > My data are a single deterministic model. I don’t know if
this is
>> > > > sufficient to create pop data. But I do have 4 different
model
>> > > > configuration output data. The first idea was to compare
these
>> > > > configurations and find a good one.
>> > > > Rachid
>> > > >
>> > > > On Mon, 18 May 2020 at 13:47, John Halley Gotway via RT <
>> > > met_help at ucar.edu
>> > > > >
>> > > > wrote:
>> > > >
>> > > > > Rachid,
>> > > > >
>> > > > > I see that you’re interested in generating ROC curves but
you
>> don’t
>> > > have
>> > > > a
>> > > > > probability forecast data. Can you please describe your
model
>> data?
>> > Is
>> > > > it a
>> > > > > single deterministic model model? Or is it an ensemble of
models?
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > On Mon, May 18, 2020 at 5:16 AM Moustabchir Rachid via RT <
>> > > > > met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
>> > > > > >
>> > > > > > Hi again!
>> > > > > >
>> > > > > > Please, excuse my ignorance. By passing through some old
>> messages
>> > in
>> > > > the
>> > > > > > met help database, I find that I need to use pop
probability of
>> > > > > > precipitation forecast to be able to use probability
tables for
>> > > > > > precipitation.
>> > > > > >
>> > > > > > So I have to create probability fields in some way and
pass
>> that to
>> > > the
>> > > > > > Grid-Stat. Is it true ?
>> > > > > >
>> > > > > > I really don't have any idea how to do that. Any
suggestions are
>> > > > welcome.
>> > > > > >
>> > > > > > Thanks
>> > > > > >
>> > > > > > Rachid Moustabchir
>> > > > > >
>> > > > > > On Mon, 18 May 2020 at 09:22, met_help at ucar.edu via RT <
>> > > > > met_help at ucar.edu>
>> > > > > > wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > Greetings,
>> > > > > > >
>> > > > > > > This message has been automatically generated in
response to
>> the
>> > > > > creation
>> > > > > > > of a trouble ticket regarding:
>> > > > > > > "ROC curve points and probability contingency
table
>> > files",
>> > > > > > > a summary of which appears below.
>> > > > > > >
>> > > > > > > There is no need to reply to this message right now.
Your
>> ticket
>> > > has
>> > > > > > been
>> > > > > > > assigned an ID of [rt.rap.ucar.edu #95286].
>> > > > > > >
>> > > > > > > Please include the string:
>> > > > > > >
>> > > > > > > [rt.rap.ucar.edu #95286]
>> > > > > > >
>> > > > > > > in the subject line of all future correspondence about
this
>> > issue.
>> > > To
>> > > > > do
>> > > > > > > so, you may reply to this message.
>> > > > > > >
>> > > > > > > For more information, please see:
>> > > > > > >
>> > > > > > > MET Online Tutorial:
>> > > > > > >
>> > > > >
>>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>> > > > > > >
>> > > > > > > MET Users Guide:
>> > > > > > > https://www.dtcenter.org/met/users/docs/overview.php
>> > > > > > >
>> > > > > > > MET FAQs:
>> > > > > > >
https://www.dtcenter.org/met/users/support/faqs/index.php
>> > > > > > >
>> > > > > > > MET-Help Email Archive:
>> > > > > > > http://mailman.ucar.edu/pipermail/met_help
>> > > > > > >
>> > > > > > > Thank you,
>> > > > > > > met_help at ucar.edu
>> > > > > > >
>> > > > > > >
>> > > > >
>> > >
>>
-------------------------------------------------------------------------
>> > > > > > > Hi everybody!
>> > > > > > >
>> > > > > > > I hope you’re doing OK in these worrying times. I wish
you,
>> your
>> > > > teams
>> > > > > > and
>> > > > > > > your loved ones health and safety.
>> > > > > > >
>> > > > > > > I want to generate roc curves, and probability
contingency
>> table
>> > > > > counts,
>> > > > > > > using pointstat (Wind, temp ...) and gridstat
>> (precipitation). In
>> > > the
>> > > > > > > configuration files, I set:
>> > > > > > >
>> > > > > > > *pct = BOTH;*
>> > > > > > >
>> > > > > > > *pstd = BOTH;*
>> > > > > > >
>> > > > > > > *prc = BOTH;*
>> > > > > > >
>> > > > > > > to indicate that probability contingency table counts,
>> > probability
>> > > > > > > statistics and probabilistic ROC curve points should be
>> created.
>> > > > > > >
>> > > > > > > The created pct, pstd and prc text files are empty and
they
>> > contain
>> > > > > only
>> > > > > > > heading line. In the stat file the LINE_TYPE columns
with PCT
>> or
>> > > PSTD
>> > > > > or
>> > > > > > > PRC are missing.
>> > > > > > >
>> > > > > > > I attached with this message examples of configuration
files,
>> > stat
>> > > > file
>> > > > > > and
>> > > > > > > PRC output files. I don't know what I have to do to get
these
>> > > files?
>> > > > > > >
>> > > > > > > Any help would be greatly appreciated.
>> > > > > > >
>> > > > > > > Thanking you most sincerely for your time and
consideration.
>> > > > > > >
>> > > > > > > Cordially,
>> > > > > > >
>> > > > > > > Rachid Moustabchir
>> > > > > > >
>> > > > > > > Assistant Professor
>> > > > > > >
>> > > > > > > Science Faculty, Agadir
>> > > > > > >
>> > > > > > > Ibn Zohr Uiversity
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
------------------------------------------------
Subject: ROC curve points and probability contingency table files
From: John Halley Gotway
Time: Tue May 19 09:37:59 2020
Rachid,
Glad you were able to figure out the ROC curve. Also, be aware that
if you
need to aggregate results across multiple runs, you can use the
stat-analysis tool to do so.
For example...
stat_analysis -lookin point_stat_output_dir -job aggregate_stat
-line_type
CTC -out_line_type CTS -by FCST_VAR,FCST_THRESH,OBS_THRESH -out_stat
agg_cts.stat
For each unique combination of forecast variable and thresholds, it'll
aggregate the CTC lines together and then derive CTS statistics from
the
aggregated counts.
For your other questions, yes, you could use the same process for TMP
and
DPT. However, applying thresholds to do data is a categorical
verification
method. TMP and DPT are more often verified as continuous variables
using
continuous statistics, like RMSE for example. However, if you have
meaningful thresholds for which you want to assess the model's ability
(like temperature > freezing), then using categorical stats is
appropriate.
You'll find that this won't work well for WDIR. Since it's a circular
variable (e.g. 0 to 360 degrees), then MET doesn't apply thresholds to
it
in the same way as scalar variables (like TMP and DPT). If you have
the U
and V components available, I'd recommend verifying both of those and
also
request the VL1L2 and VCNT output line types from Point-Stat.
Yes, you can interpret cloud cover amounts as being a probability
forecast. In the Grid-Stat config file, you just set "prob = TRUE;"
after
you define the name and level for that field. And Grid-Stat will
interpret
them as being probabilities. In fact, one project at NCAR did exactly
that
in the past year. I don't know how meaningful the results were though.
Probabilities are defined as a probability of some event. And
presumably
you'd interpret this data as being the probability of some cloud
amount in
the observations.
I'd like to defer answering your first question about deriving
probabilities. I'm a software engineer, not a statistician. And I
don't
want to lead you astray. I can say that I have never seen the approach
you
described used in my 15 years at NCAR. However, since you're verifying
against point observations, I'd definitely recommend considering the
HiRA
methodology.
Read section 7.2.2 of the MET User's Guide:
https://dtcenter.org/sites/default/files/community-code/met/docs/user-
guide/MET_Users_Guide_v9.0.1.pdf
This transforms a deterministic forecast into a probabilistic one
using the
nearby spatial points.
Thanks,
John
On Tue, May 19, 2020 at 2:52 AM Moustabchir Rachid via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
>
> *D**ear **John !*
>
> *With **the*
>
>
>
>
>
>
>
>
> * clear explanations, I get stat files that contains CTC lines for
every
> forecast and obs thresholds. This is an example of the output I
get:V9.0
> .... >0 >20 NA NA CTC 254104 347
> 55379 5 198373V9.0 .... >2 >20
NA
> NA CTC 254104 168 4679 184 249073V9.0
....
> >4 >20 NA NA CTC 254104 53
2045
> 299 251707V9.0 .... >6 >20 NA NA
CTC
> 254104 36 1277 316 252475V9.0 ....
>8
> >20 NA NA CTC 254104 26 843
326
> 252909V9.0 .... >10 >20 NA NA CTC
254104
> 21 615 331 253137V9.0 .... >12
>20
> NA NA CTC 254104 16 444 336
253308V9.0
> .... >14 >20 NA NA CTC 254104 10
> 345 342 253407V9.0 .... >16 >20 NA
NA
> CTC 254104 8 283 344 253469V9.0
....
> >18 >20 NA NA CTC 254104 6 241
> 346 253511*
>
> *It is not really **hard as to get roc curves. For each line we
compute
> POFD **and PODY. By plotting PODY versus **POFD **we get roc curves
for **a
> given observation threshold. *
>
> *Thank you John for helping me to solve the problem. I have **more**
other
> questions :*
>
> 1.
>
> *Is it possible to compute the forecast probability for each ctc
data ?
> What is the formula **a/(a+b), b/(a+b) … I could not find
anything about
> it, since I want to plot reliability diagrams which show the
observed
> relative frequencies as a function of the forecasted
probabilities.*
> 2.
>
> *Can I use **the same procedure** for TMP, DPT, WDIR … with
pointstat
> tool ?*
> 3.
>
> *The third question perhaps will sound ridiculous. For cloud
cover,
> since values are fractions can I consider these values like
> probabilities,
> and create probability **files** with gridstat.*
>
> *Thanks*
>
>
>
>
> On Mon, 18 May 2020 at 19:15, Moustabchir Rachid <
> rachid.moustabchir at gmail.com> wrote:
>
> > Got it.
> > Thank you very much John.
> > Rachid
> >
> > On Mon, 18 May 2020 at 19:12, John Halley Gotway via RT <
> met_help at ucar.edu>
> > wrote:
> >
> >> Rachid,
> >>
> >> Typically when you verify precip, you apply the same threshold to
both
> the
> >> forecast and observation data. But to make a ROC curve from
> deterministic
> >> data, I believe that you hold the obs thresholds constant and
vary the
> >> forecast threshold... and that's what I described as being "odd".
> Here's
> >> an example of config file settings which illustrate what I mean:
> >>
> >> fcst = {
> >> field = [
> >> {
> >> name = "APCP";
> >> level = "A03";
> >> cat_thresh = [ >0, >2, >4, >6, >8, >10, >12, >14, >16,
>18, >20
> ];
> >> }
> >> ];
> >> }
> >> obs = {
> >> field = [
> >> {
> >> name = "APCP";
> >> level = "A03";
> >> cat_thresh = [ >10, >10, >10, >10, >10, >10, >10, >10,
>10, >10,
> >> >10 ];
> >> }
> >> ];
> >> }
> >>
> >>
> >>
> >> On Mon, May 18, 2020 at 1:06 PM Moustabchir Rachid via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> >> >
> >> > Thank you very much John for these explanations !
> >> > roc curve is harder than it sounds, but I will give it a try.
> >> > John, what you mean by "you'd need to configure your precip
thresholds
> >> in
> >> > an odd", because in the explanations you are using even values:
>0,
> >2,
> >> >4,
> >> > >6, >8, >10, >12, >14, >16, >18, >20.
> >> > Thanks
> >> >
> >> > On Mon, 18 May 2020 at 18:07, John Halley Gotway via RT <
> >> met_help at ucar.edu
> >> > >
> >> > wrote:
> >> >
> >> > > Rachid,
> >> > >
> >> > > ROC curves are typically made in the evaluation of
probabilistic
> data.
> >> > > However, they can also be constructed from deterministic data
as
> well,
> >> > > using a series of 2x2 contingency tables stored in the CTC
line type
> >> from
> >> > > MET. I believe METviewer does have the ability to generate
ROC
> >> diagrams
> >> > > from CTC lines but you'd need to configure your precip
thresholds in
> >> an
> >> > odd
> >> > > way. And while this is supposed to work, I haven't actually
run
> >> METviewer
> >> > > in this way myself.
> >> > >
> >> > > You choose a single observation threshold, for each "precip >
10
> mm".
> >> > And
> >> > > then you choose many forecast thresholds (e.g. >0, >2, >4,
>6, >8,
> >> >10,
> >> > > >12, >14, >16, >18, >20). You run Grid-Stat or Point-Stat to
compute
> >> CTC
> >> > > lines for each of these comparisons (e.g. fcst>0 vs obs>10,
fcst>2
> vs
> >> > > obs>10, fcst>4 vs obs>10, and so on).
> >> > >
> >> > > Then you use METviewer to aggregate those results into a ROC
> diagram.
> >> > >
> >> > > In my opinion, that's a lot of steps to produce a ROC
diagram...
> >> > especially
> >> > > if you don't have access to METviewer to facilitate it. You
could
> >> instead
> >> > > consider choosing relevant thresholds, and comparing those
multiple
> >> runs
> >> > > using contingency table stats, like CSI or ETS (called GSS in
MET).
> >> > >
> >> > > John
> >> > >
> >> > > On Mon, May 18, 2020 at 11:26 AM Moustabchir Rachid via RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> >> > > >
> >> > > > Thanks John for your answer.
> >> > > >
> >> > > > My data are a single deterministic model. I don’t know if
this is
> >> > > > sufficient to create pop data. But I do have 4 different
model
> >> > > > configuration output data. The first idea was to compare
these
> >> > > > configurations and find a good one.
> >> > > > Rachid
> >> > > >
> >> > > > On Mon, 18 May 2020 at 13:47, John Halley Gotway via RT <
> >> > > met_help at ucar.edu
> >> > > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Rachid,
> >> > > > >
> >> > > > > I see that you’re interested in generating ROC curves but
you
> >> don’t
> >> > > have
> >> > > > a
> >> > > > > probability forecast data. Can you please describe your
model
> >> data?
> >> > Is
> >> > > > it a
> >> > > > > single deterministic model model? Or is it an ensemble of
> models?
> >> > > > >
> >> > > > > Thanks,
> >> > > > > John
> >> > > > >
> >> > > > > On Mon, May 18, 2020 at 5:16 AM Moustabchir Rachid via RT
<
> >> > > > > met_help at ucar.edu>
> >> > > > > wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286
> >
> >> > > > > >
> >> > > > > > Hi again!
> >> > > > > >
> >> > > > > > Please, excuse my ignorance. By passing through some
old
> >> messages
> >> > in
> >> > > > the
> >> > > > > > met help database, I find that I need to use pop
probability
> of
> >> > > > > > precipitation forecast to be able to use probability
tables
> for
> >> > > > > > precipitation.
> >> > > > > >
> >> > > > > > So I have to create probability fields in some way and
pass
> >> that to
> >> > > the
> >> > > > > > Grid-Stat. Is it true ?
> >> > > > > >
> >> > > > > > I really don't have any idea how to do that. Any
suggestions
> are
> >> > > > welcome.
> >> > > > > >
> >> > > > > > Thanks
> >> > > > > >
> >> > > > > > Rachid Moustabchir
> >> > > > > >
> >> > > > > > On Mon, 18 May 2020 at 09:22, met_help at ucar.edu via RT
<
> >> > > > > met_help at ucar.edu>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > >
> >> > > > > > > Greetings,
> >> > > > > > >
> >> > > > > > > This message has been automatically generated in
response to
> >> the
> >> > > > > creation
> >> > > > > > > of a trouble ticket regarding:
> >> > > > > > > "ROC curve points and probability contingency
table
> >> > files",
> >> > > > > > > a summary of which appears below.
> >> > > > > > >
> >> > > > > > > There is no need to reply to this message right now.
Your
> >> ticket
> >> > > has
> >> > > > > > been
> >> > > > > > > assigned an ID of [rt.rap.ucar.edu #95286].
> >> > > > > > >
> >> > > > > > > Please include the string:
> >> > > > > > >
> >> > > > > > > [rt.rap.ucar.edu #95286]
> >> > > > > > >
> >> > > > > > > in the subject line of all future correspondence
about this
> >> > issue.
> >> > > To
> >> > > > > do
> >> > > > > > > so, you may reply to this message.
> >> > > > > > >
> >> > > > > > > For more information, please see:
> >> > > > > > >
> >> > > > > > > MET Online Tutorial:
> >> > > > > > >
> >> > > > >
> >>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> >> > > > > > >
> >> > > > > > > MET Users Guide:
> >> > > > > > >
https://www.dtcenter.org/met/users/docs/overview.php
> >> > > > > > >
> >> > > > > > > MET FAQs:
> >> > > > > > >
> https://www.dtcenter.org/met/users/support/faqs/index.php
> >> > > > > > >
> >> > > > > > > MET-Help Email Archive:
> >> > > > > > > http://mailman.ucar.edu/pipermail/met_help
> >> > > > > > >
> >> > > > > > > Thank you,
> >> > > > > > > met_help at ucar.edu
> >> > > > > > >
> >> > > > > > >
> >> > > > >
> >> > >
> >>
>
-------------------------------------------------------------------------
> >> > > > > > > Hi everybody!
> >> > > > > > >
> >> > > > > > > I hope you’re doing OK in these worrying times. I
wish you,
> >> your
> >> > > > teams
> >> > > > > > and
> >> > > > > > > your loved ones health and safety.
> >> > > > > > >
> >> > > > > > > I want to generate roc curves, and probability
contingency
> >> table
> >> > > > > counts,
> >> > > > > > > using pointstat (Wind, temp ...) and gridstat
> >> (precipitation). In
> >> > > the
> >> > > > > > > configuration files, I set:
> >> > > > > > >
> >> > > > > > > *pct = BOTH;*
> >> > > > > > >
> >> > > > > > > *pstd = BOTH;*
> >> > > > > > >
> >> > > > > > > *prc = BOTH;*
> >> > > > > > >
> >> > > > > > > to indicate that probability contingency table
counts,
> >> > probability
> >> > > > > > > statistics and probabilistic ROC curve points should
be
> >> created.
> >> > > > > > >
> >> > > > > > > The created pct, pstd and prc text files are empty
and they
> >> > contain
> >> > > > > only
> >> > > > > > > heading line. In the stat file the LINE_TYPE columns
with
> PCT
> >> or
> >> > > PSTD
> >> > > > > or
> >> > > > > > > PRC are missing.
> >> > > > > > >
> >> > > > > > > I attached with this message examples of
configuration
> files,
> >> > stat
> >> > > > file
> >> > > > > > and
> >> > > > > > > PRC output files. I don't know what I have to do to
get
> these
> >> > > files?
> >> > > > > > >
> >> > > > > > > Any help would be greatly appreciated.
> >> > > > > > >
> >> > > > > > > Thanking you most sincerely for your time and
consideration.
> >> > > > > > >
> >> > > > > > > Cordially,
> >> > > > > > >
> >> > > > > > > Rachid Moustabchir
> >> > > > > > >
> >> > > > > > > Assistant Professor
> >> > > > > > >
> >> > > > > > > Science Faculty, Agadir
> >> > > > > > >
> >> > > > > > > Ibn Zohr Uiversity
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
>
>
------------------------------------------------
Subject: ROC curve points and probability contingency table files
From: Moustabchir Rachid
Time: Tue May 19 12:06:52 2020
Thanks John for making it clear. I appreciate your taking the time to
answer my questions.
Cordially,
Rachid
On Tue, 19 May 2020 at 15:38, John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Rachid,
>
> Glad you were able to figure out the ROC curve. Also, be aware that
if you
> need to aggregate results across multiple runs, you can use the
> stat-analysis tool to do so.
>
> For example...
> stat_analysis -lookin point_stat_output_dir -job aggregate_stat
-line_type
> CTC -out_line_type CTS -by FCST_VAR,FCST_THRESH,OBS_THRESH -out_stat
> agg_cts.stat
>
> For each unique combination of forecast variable and thresholds,
it'll
> aggregate the CTC lines together and then derive CTS statistics from
the
> aggregated counts.
>
> For your other questions, yes, you could use the same process for
TMP and
> DPT. However, applying thresholds to do data is a categorical
verification
> method. TMP and DPT are more often verified as continuous variables
using
> continuous statistics, like RMSE for example. However, if you have
> meaningful thresholds for which you want to assess the model's
ability
> (like temperature > freezing), then using categorical stats is
appropriate.
> You'll find that this won't work well for WDIR. Since it's a
circular
> variable (e.g. 0 to 360 degrees), then MET doesn't apply thresholds
to it
> in the same way as scalar variables (like TMP and DPT). If you have
the U
> and V components available, I'd recommend verifying both of those
and also
> request the VL1L2 and VCNT output line types from Point-Stat.
>
> Yes, you can interpret cloud cover amounts as being a probability
> forecast. In the Grid-Stat config file, you just set "prob = TRUE;"
after
> you define the name and level for that field. And Grid-Stat will
interpret
> them as being probabilities. In fact, one project at NCAR did
exactly that
> in the past year. I don't know how meaningful the results were
though.
> Probabilities are defined as a probability of some event. And
presumably
> you'd interpret this data as being the probability of some cloud
amount in
> the observations.
>
> I'd like to defer answering your first question about deriving
> probabilities. I'm a software engineer, not a statistician. And I
don't
> want to lead you astray. I can say that I have never seen the
approach you
> described used in my 15 years at NCAR. However, since you're
verifying
> against point observations, I'd definitely recommend considering the
HiRA
> methodology.
>
> Read section 7.2.2 of the MET User's Guide:
>
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.1.pdf
>
> This transforms a deterministic forecast into a probabilistic one
using the
> nearby spatial points.
>
> Thanks,
> John
>
> On Tue, May 19, 2020 at 2:52 AM Moustabchir Rachid via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> >
> > *D**ear **John !*
> >
> > *With **the*
> >
> >
> >
> >
> >
> >
> >
> >
> > * clear explanations, I get stat files that contains CTC lines for
every
> > forecast and obs thresholds. This is an example of the output I
> get:V9.0
> > .... >0 >20 NA NA CTC 254104
347
> > 55379 5 198373V9.0 .... >2 >20
NA
> > NA CTC 254104 168 4679 184 249073V9.0
....
> > >4 >20 NA NA CTC 254104 53
2045
> > 299 251707V9.0 .... >6 >20 NA
NA
> CTC
> > 254104 36 1277 316 252475V9.0 ....
>8
> > >20 NA NA CTC 254104 26 843
326
> > 252909V9.0 .... >10 >20 NA NA CTC
254104
> > 21 615 331 253137V9.0 .... >12
>20
> > NA NA CTC 254104 16 444 336
253308V9.0
> > .... >14 >20 NA NA CTC 254104 10
> > 345 342 253407V9.0 .... >16 >20
NA
> NA
> > CTC 254104 8 283 344 253469V9.0
....
> > >18 >20 NA NA CTC 254104 6
241
> > 346 253511*
> >
> > *It is not really **hard as to get roc curves. For each line we
compute
> > POFD **and PODY. By plotting PODY versus **POFD **we get roc
curves for
> **a
> > given observation threshold. *
> >
> > *Thank you John for helping me to solve the problem. I have
**more**
> other
> > questions :*
> >
> > 1.
> >
> > *Is it possible to compute the forecast probability for each
ctc data
> ?
> > What is the formula **a/(a+b), b/(a+b) … I could not find
anything
> about
> > it, since I want to plot reliability diagrams which show the
observed
> > relative frequencies as a function of the forecasted
probabilities.*
> > 2.
> >
> > *Can I use **the same procedure** for TMP, DPT, WDIR … with
pointstat
> > tool ?*
> > 3.
> >
> > *The third question perhaps will sound ridiculous. For cloud
cover,
> > since values are fractions can I consider these values like
> > probabilities,
> > and create probability **files** with gridstat.*
> >
> > *Thanks*
> >
> >
> >
> >
> > On Mon, 18 May 2020 at 19:15, Moustabchir Rachid <
> > rachid.moustabchir at gmail.com> wrote:
> >
> > > Got it.
> > > Thank you very much John.
> > > Rachid
> > >
> > > On Mon, 18 May 2020 at 19:12, John Halley Gotway via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > >> Rachid,
> > >>
> > >> Typically when you verify precip, you apply the same threshold
to both
> > the
> > >> forecast and observation data. But to make a ROC curve from
> > deterministic
> > >> data, I believe that you hold the obs thresholds constant and
vary the
> > >> forecast threshold... and that's what I described as being
"odd".
> > Here's
> > >> an example of config file settings which illustrate what I
mean:
> > >>
> > >> fcst = {
> > >> field = [
> > >> {
> > >> name = "APCP";
> > >> level = "A03";
> > >> cat_thresh = [ >0, >2, >4, >6, >8, >10, >12, >14, >16,
>18,
> >20
> > ];
> > >> }
> > >> ];
> > >> }
> > >> obs = {
> > >> field = [
> > >> {
> > >> name = "APCP";
> > >> level = "A03";
> > >> cat_thresh = [ >10, >10, >10, >10, >10, >10, >10, >10,
>10,
> >10,
> > >> >10 ];
> > >> }
> > >> ];
> > >> }
> > >>
> > >>
> > >>
> > >> On Mon, May 18, 2020 at 1:06 PM Moustabchir Rachid via RT <
> > >> met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286
>
> > >> >
> > >> > Thank you very much John for these explanations !
> > >> > roc curve is harder than it sounds, but I will give it a
try.
> > >> > John, what you mean by "you'd need to configure your precip
> thresholds
> > >> in
> > >> > an odd", because in the explanations you are using even
values: >0,
> > >2,
> > >> >4,
> > >> > >6, >8, >10, >12, >14, >16, >18, >20.
> > >> > Thanks
> > >> >
> > >> > On Mon, 18 May 2020 at 18:07, John Halley Gotway via RT <
> > >> met_help at ucar.edu
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > Rachid,
> > >> > >
> > >> > > ROC curves are typically made in the evaluation of
probabilistic
> > data.
> > >> > > However, they can also be constructed from deterministic
data as
> > well,
> > >> > > using a series of 2x2 contingency tables stored in the CTC
line
> type
> > >> from
> > >> > > MET. I believe METviewer does have the ability to generate
ROC
> > >> diagrams
> > >> > > from CTC lines but you'd need to configure your precip
thresholds
> in
> > >> an
> > >> > odd
> > >> > > way. And while this is supposed to work, I haven't actually
run
> > >> METviewer
> > >> > > in this way myself.
> > >> > >
> > >> > > You choose a single observation threshold, for each "precip
> 10
> > mm".
> > >> > And
> > >> > > then you choose many forecast thresholds (e.g. >0, >2, >4,
>6, >8,
> > >> >10,
> > >> > > >12, >14, >16, >18, >20). You run Grid-Stat or Point-Stat
to
> compute
> > >> CTC
> > >> > > lines for each of these comparisons (e.g. fcst>0 vs obs>10,
fcst>2
> > vs
> > >> > > obs>10, fcst>4 vs obs>10, and so on).
> > >> > >
> > >> > > Then you use METviewer to aggregate those results into a
ROC
> > diagram.
> > >> > >
> > >> > > In my opinion, that's a lot of steps to produce a ROC
diagram...
> > >> > especially
> > >> > > if you don't have access to METviewer to facilitate it. You
could
> > >> instead
> > >> > > consider choosing relevant thresholds, and comparing those
> multiple
> > >> runs
> > >> > > using contingency table stats, like CSI or ETS (called GSS
in
> MET).
> > >> > >
> > >> > > John
> > >> > >
> > >> > > On Mon, May 18, 2020 at 11:26 AM Moustabchir Rachid via RT
<
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > > >
> > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286 >
> > >> > > >
> > >> > > > Thanks John for your answer.
> > >> > > >
> > >> > > > My data are a single deterministic model. I don’t know if
this
> is
> > >> > > > sufficient to create pop data. But I do have 4 different
model
> > >> > > > configuration output data. The first idea was to compare
these
> > >> > > > configurations and find a good one.
> > >> > > > Rachid
> > >> > > >
> > >> > > > On Mon, 18 May 2020 at 13:47, John Halley Gotway via RT <
> > >> > > met_help at ucar.edu
> > >> > > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Rachid,
> > >> > > > >
> > >> > > > > I see that you’re interested in generating ROC curves
but you
> > >> don’t
> > >> > > have
> > >> > > > a
> > >> > > > > probability forecast data. Can you please describe your
model
> > >> data?
> > >> > Is
> > >> > > > it a
> > >> > > > > single deterministic model model? Or is it an ensemble
of
> > models?
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > John
> > >> > > > >
> > >> > > > > On Mon, May 18, 2020 at 5:16 AM Moustabchir Rachid via
RT <
> > >> > > > > met_help at ucar.edu>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > >
> > >> > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95286
> > >
> > >> > > > > >
> > >> > > > > > Hi again!
> > >> > > > > >
> > >> > > > > > Please, excuse my ignorance. By passing through some
old
> > >> messages
> > >> > in
> > >> > > > the
> > >> > > > > > met help database, I find that I need to use pop
probability
> > of
> > >> > > > > > precipitation forecast to be able to use probability
tables
> > for
> > >> > > > > > precipitation.
> > >> > > > > >
> > >> > > > > > So I have to create probability fields in some way
and pass
> > >> that to
> > >> > > the
> > >> > > > > > Grid-Stat. Is it true ?
> > >> > > > > >
> > >> > > > > > I really don't have any idea how to do that. Any
suggestions
> > are
> > >> > > > welcome.
> > >> > > > > >
> > >> > > > > > Thanks
> > >> > > > > >
> > >> > > > > > Rachid Moustabchir
> > >> > > > > >
> > >> > > > > > On Mon, 18 May 2020 at 09:22, met_help at ucar.edu via
RT <
> > >> > > > > met_help at ucar.edu>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > >
> > >> > > > > > > Greetings,
> > >> > > > > > >
> > >> > > > > > > This message has been automatically generated in
response
> to
> > >> the
> > >> > > > > creation
> > >> > > > > > > of a trouble ticket regarding:
> > >> > > > > > > "ROC curve points and probability
contingency
> table
> > >> > files",
> > >> > > > > > > a summary of which appears below.
> > >> > > > > > >
> > >> > > > > > > There is no need to reply to this message right
now. Your
> > >> ticket
> > >> > > has
> > >> > > > > > been
> > >> > > > > > > assigned an ID of [rt.rap.ucar.edu #95286].
> > >> > > > > > >
> > >> > > > > > > Please include the string:
> > >> > > > > > >
> > >> > > > > > > [rt.rap.ucar.edu #95286]
> > >> > > > > > >
> > >> > > > > > > in the subject line of all future correspondence
about
> this
> > >> > issue.
> > >> > > To
> > >> > > > > do
> > >> > > > > > > so, you may reply to this message.
> > >> > > > > > >
> > >> > > > > > > For more information, please see:
> > >> > > > > > >
> > >> > > > > > > MET Online Tutorial:
> > >> > > > > > >
> > >> > > > >
> > >>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > >> > > > > > >
> > >> > > > > > > MET Users Guide:
> > >> > > > > > >
https://www.dtcenter.org/met/users/docs/overview.php
> > >> > > > > > >
> > >> > > > > > > MET FAQs:
> > >> > > > > > >
> > https://www.dtcenter.org/met/users/support/faqs/index.php
> > >> > > > > > >
> > >> > > > > > > MET-Help Email Archive:
> > >> > > > > > > http://mailman.ucar.edu/pipermail/met_help
> > >> > > > > > >
> > >> > > > > > > Thank you,
> > >> > > > > > > met_help at ucar.edu
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > >
> > >> > >
> > >>
> >
-------------------------------------------------------------------------
> > >> > > > > > > Hi everybody!
> > >> > > > > > >
> > >> > > > > > > I hope you’re doing OK in these worrying times. I
wish
> you,
> > >> your
> > >> > > > teams
> > >> > > > > > and
> > >> > > > > > > your loved ones health and safety.
> > >> > > > > > >
> > >> > > > > > > I want to generate roc curves, and probability
contingency
> > >> table
> > >> > > > > counts,
> > >> > > > > > > using pointstat (Wind, temp ...) and gridstat
> > >> (precipitation). In
> > >> > > the
> > >> > > > > > > configuration files, I set:
> > >> > > > > > >
> > >> > > > > > > *pct = BOTH;*
> > >> > > > > > >
> > >> > > > > > > *pstd = BOTH;*
> > >> > > > > > >
> > >> > > > > > > *prc = BOTH;*
> > >> > > > > > >
> > >> > > > > > > to indicate that probability contingency table
counts,
> > >> > probability
> > >> > > > > > > statistics and probabilistic ROC curve points
should be
> > >> created.
> > >> > > > > > >
> > >> > > > > > > The created pct, pstd and prc text files are empty
and
> they
> > >> > contain
> > >> > > > > only
> > >> > > > > > > heading line. In the stat file the LINE_TYPE
columns with
> > PCT
> > >> or
> > >> > > PSTD
> > >> > > > > or
> > >> > > > > > > PRC are missing.
> > >> > > > > > >
> > >> > > > > > > I attached with this message examples of
configuration
> > files,
> > >> > stat
> > >> > > > file
> > >> > > > > > and
> > >> > > > > > > PRC output files. I don't know what I have to do to
get
> > these
> > >> > > files?
> > >> > > > > > >
> > >> > > > > > > Any help would be greatly appreciated.
> > >> > > > > > >
> > >> > > > > > > Thanking you most sincerely for your time and
> consideration.
> > >> > > > > > >
> > >> > > > > > > Cordially,
> > >> > > > > > >
> > >> > > > > > > Rachid Moustabchir
> > >> > > > > > >
> > >> > > > > > > Assistant Professor
> > >> > > > > > >
> > >> > > > > > > Science Faculty, Agadir
> > >> > > > > > >
> > >> > > > > > > Ibn Zohr Uiversity
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >>
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list