[Met_help] [rt.rap.ucar.edu #95347] History for Reliability diagram

John Halley Gotway via RT met_help at ucar.edu
Thu May 28 16:43:29 MDT 2020


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Folks - I get from MET tools OY_TP [observed YES total proportion] and ON_TP [observed NO total proportion] and I'm plotting OY, [.1, ..., .9] as the reliability diagrams. Is this correct?

Efren A. Serra (Contractor)
Physicist

DeVine Consulting, Inc.
Naval Research Laboratory
Marine Meteorology Division
7 Grace Hopper Ave., STOP 2
Monterey, CA 93943
Code 7542
Mobile: 408-425-5027



----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Reliability diagram
From: John Halley Gotway
Time: Sun May 24 15:06:47 2020

Efren,

Not quite.  Here's a useful link about reliability diagrams from the
WMO:
http://www.bom.gov.au/wmo/lrfvs/reliability.shtml

The points along the X-axis are the forecast probability thresholds..
Since you're using probability bins of 0 to 0.1, 0.1 to 0.2, and so
on, I'd
recommend plotting the center-point of each bin. So along the X-axis,
you'd
plot values at 0.5, 0.15, 0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85,
and
0.95.

The points along the Y-axis are the observed relative frequency
values. And
that's the OY_TP value from the MET output.

I also worked up an example using METviewer on some sample
probabilistic output data:
http://www.dtcenter.org/met/metviewer/metviewer1.jsp

See the attached XML and resulting plot. This shows the reliability
diagrams for probabilities from 3 different ensembles for 1-hour
precip >
0.10". I've also attached the corresponding data file and R-script
used to
generate this plot (also output from METviewer). The templatized R-
script
is rather difficult to parse through, but it might be useful.

Thanks,
John Halley Gotway

On Fri, May 22, 2020 at 6:17 PM efren.serra.ctr at nrlmry.navy.mil via RT
<
met_help at ucar.edu> wrote:

>
> Fri May 22 18:17:18 2020: Request 95347 was acted upon.
> Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
>        Queue: met_help
>      Subject: Reliability diagram
>        Owner: Nobody
>   Requestors: efren.serra.ctr at nrlmry.navy.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
>
>
> Folks - I get from MET tools OY_TP [observed YES total proportion]
and
> ON_TP [observed NO total proportion] and I'm plotting OY, [.1, ...,
.9] as
> the reliability diagrams. Is this correct?
>
> Efren A. Serra (Contractor)
> Physicist
>
> DeVine Consulting, Inc.
> Naval Research Laboratory
> Marine Meteorology Division
> 7 Grace Hopper Ave., STOP 2
> Monterey, CA 93943
> Code 7542
> Mobile: 408-425-5027
>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #95347] Reliability diagram
From: efren.serra.ctr at nrlmry.navy.mil
Time: Mon May 25 15:05:39 2020

John - Then, according to WMO link below, the plot of
OY_TP[i]/(OY_TP[i]+ON_TP[i]) versus the .5, .15, ..., .95 would be the
reliability diagram, isn't it?

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Sunday, May 24, 2020 2:07 PM
To: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram

Efren,

Not quite.  Here's a useful link about reliability diagrams from the
WMO:
http://www.bom.gov.au/wmo/lrfvs/reliability.shtml

The points along the X-axis are the forecast probability thresholds..
Since you're using probability bins of 0 to 0.1, 0.1 to 0.2, and so
on, I'd recommend plotting the center-point of each bin. So along the
X-axis, you'd plot values at 0.5, 0.15, 0.25, 0.35, 0.45, 0.55, 0.65,
0.75, 0.85, and 0.95.

The points along the Y-axis are the observed relative frequency
values. And that's the OY_TP value from the MET output.

I also worked up an example using METviewer on some sample
probabilistic output data:
http://www.dtcenter.org/met/metviewer/metviewer1.jsp

See the attached XML and resulting plot. This shows the reliability
diagrams for probabilities from 3 different ensembles for 1-hour
precip > 0.10". I've also attached the corresponding data file and R-
script used to generate this plot (also output from METviewer). The
templatized R-script is rather difficult to parse through, but it
might be useful.

Thanks,
John Halley Gotway

On Fri, May 22, 2020 at 6:17 PM efren.serra.ctr at nrlmry.navy.mil via RT
< met_help at ucar.edu> wrote:

>
> Fri May 22 18:17:18 2020: Request 95347 was acted upon.
> Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
>        Queue: met_help
>      Subject: Reliability diagram
>        Owner: Nobody
>   Requestors: efren.serra.ctr at nrlmry.navy.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347
> >
>
>
> Folks - I get from MET tools OY_TP [observed YES total proportion]
and
> ON_TP [observed NO total proportion] and I'm plotting OY, [.1, ...,
> .9] as the reliability diagrams. Is this correct?
>
> Efren A. Serra (Contractor)
> Physicist
>
> DeVine Consulting, Inc.
> Naval Research Laboratory
> Marine Meteorology Division
> 7 Grace Hopper Ave., STOP 2
> Monterey, CA 93943
> Code 7542
> Mobile: 408-425-5027
>
>
>



------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #95347] Reliability diagram
From: efren.serra.ctr at nrlmry.navy.mil
Time: Mon May 25 23:41:24 2020

John - I should point out that I'm using v8.1 of MET tools and that
I'm parsing *pjc.txt* output files.

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Sunday, May 24, 2020 2:07 PM
To: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram

Efren,

Not quite.  Here's a useful link about reliability diagrams from the
WMO:
http://www.bom.gov.au/wmo/lrfvs/reliability.shtml

The points along the X-axis are the forecast probability thresholds..
Since you're using probability bins of 0 to 0.1, 0.1 to 0.2, and so
on, I'd recommend plotting the center-point of each bin. So along the
X-axis, you'd plot values at 0.5, 0.15, 0.25, 0.35, 0.45, 0.55, 0.65,
0.75, 0.85, and 0.95.

The points along the Y-axis are the observed relative frequency
values. And that's the OY_TP value from the MET output.

I also worked up an example using METviewer on some sample
probabilistic output data:
http://www.dtcenter.org/met/metviewer/metviewer1.jsp

See the attached XML and resulting plot. This shows the reliability
diagrams for probabilities from 3 different ensembles for 1-hour
precip > 0.10". I've also attached the corresponding data file and R-
script used to generate this plot (also output from METviewer). The
templatized R-script is rather difficult to parse through, but it
might be useful.

Thanks,
John Halley Gotway

On Fri, May 22, 2020 at 6:17 PM efren.serra.ctr at nrlmry.navy.mil via RT
< met_help at ucar.edu> wrote:

>
> Fri May 22 18:17:18 2020: Request 95347 was acted upon.
> Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
>        Queue: met_help
>      Subject: Reliability diagram
>        Owner: Nobody
>   Requestors: efren.serra.ctr at nrlmry.navy.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347
> >
>
>
> Folks - I get from MET tools OY_TP [observed YES total proportion]
and
> ON_TP [observed NO total proportion] and I'm plotting OY, [.1, ...,
> .9] as the reliability diagrams. Is this correct?
>
> Efren A. Serra (Contractor)
> Physicist
>
> DeVine Consulting, Inc.
> Naval Research Laboratory
> Marine Meteorology Division
> 7 Grace Hopper Ave., STOP 2
> Monterey, CA 93943
> Code 7542
> Mobile: 408-425-5027
>
>
>



------------------------------------------------
Subject: Reliability diagram
From: Julie Prestopnik
Time: Tue May 26 13:09:44 2020

Hi Efren.  I just wanted to let you know that John is out of the
office
until later this week.  Unfortunately, I am unable to answer your
question.  I'm sure you will hear from John shortly after he returns
to
work.

Julie

On Mon, May 25, 2020 at 11:42 PM efren.serra.ctr at nrlmry.navy.mil via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
>
> John - I should point out that I'm using v8.1 of MET tools and that
I'm
> parsing *pjc.txt* output files.
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Sunday, May 24, 2020 2:07 PM
> To: Serra, Mr. Efren, Contractor, Code 7531 <
> efren.serra.ctr at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
>
> Efren,
>
> Not quite.  Here's a useful link about reliability diagrams from the
WMO:
> http://www.bom.gov.au/wmo/lrfvs/reliability.shtml
>
> The points along the X-axis are the forecast probability
thresholds..
> Since you're using probability bins of 0 to 0.1, 0.1 to 0.2, and so
on,
> I'd recommend plotting the center-point of each bin. So along the X-
axis,
> you'd plot values at 0.5, 0.15, 0.25, 0.35, 0.45, 0.55, 0.65, 0.75,
0.85,
> and 0.95.
>
> The points along the Y-axis are the observed relative frequency
values.
> And that's the OY_TP value from the MET output.
>
> I also worked up an example using METviewer on some sample
probabilistic
> output data:
> http://www.dtcenter.org/met/metviewer/metviewer1.jsp
>
> See the attached XML and resulting plot. This shows the reliability
> diagrams for probabilities from 3 different ensembles for 1-hour
precip >
> 0.10". I've also attached the corresponding data file and R-script
used to
> generate this plot (also output from METviewer). The templatized R-
script
> is rather difficult to parse through, but it might be useful.
>
> Thanks,
> John Halley Gotway
>
> On Fri, May 22, 2020 at 6:17 PM efren.serra.ctr at nrlmry.navy.mil via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri May 22 18:17:18 2020: Request 95347 was acted upon.
> > Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> >        Queue: met_help
> >      Subject: Reliability diagram
> >        Owner: Nobody
> >   Requestors: efren.serra.ctr at nrlmry.navy.mil
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347
> > >
> >
> >
> > Folks - I get from MET tools OY_TP [observed YES total proportion]
and
> > ON_TP [observed NO total proportion] and I'm plotting OY, [.1,
...,
> > .9] as the reliability diagrams. Is this correct?
> >
> > Efren A. Serra (Contractor)
> > Physicist
> >
> > DeVine Consulting, Inc.
> > Naval Research Laboratory
> > Marine Meteorology Division
> > 7 Grace Hopper Ave., STOP 2
> > Monterey, CA 93943
> > Code 7542
> > Mobile: 408-425-5027
> >
> >
> >
>
>
>
>

--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu

My working day may not be your working day.  Please do not feel
obliged to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #95347] Reliability diagram
From: efren.serra.ctr at nrlmry.navy.mil
Time: Tue May 26 13:22:27 2020

Thanks!

-----Original Message-----
From: Julie Prestopnik via RT <met_help at ucar.edu>
Sent: Tuesday, May 26, 2020 12:10 PM
To: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram

Hi Efren.  I just wanted to let you know that John is out of the
office until later this week.  Unfortunately, I am unable to answer
your question.  I'm sure you will hear from John shortly after he
returns to work.

Julie

On Mon, May 25, 2020 at 11:42 PM efren.serra.ctr at nrlmry.navy.mil via
RT < met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
>
> John - I should point out that I'm using v8.1 of MET tools and that
> I'm parsing *pjc.txt* output files.
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Sunday, May 24, 2020 2:07 PM
> To: Serra, Mr. Efren, Contractor, Code 7531 <
> efren.serra.ctr at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
>
> Efren,
>
> Not quite.  Here's a useful link about reliability diagrams from the
WMO:
> http://www.bom.gov.au/wmo/lrfvs/reliability.shtml
>
> The points along the X-axis are the forecast probability
thresholds..
> Since you're using probability bins of 0 to 0.1, 0.1 to 0.2, and so
> on, I'd recommend plotting the center-point of each bin. So along
the
> X-axis, you'd plot values at 0.5, 0.15, 0.25, 0.35, 0.45, 0.55,
0.65,
> 0.75, 0.85, and 0.95.
>
> The points along the Y-axis are the observed relative frequency
values.
> And that's the OY_TP value from the MET output.
>
> I also worked up an example using METviewer on some sample
> probabilistic output data:
> http://www.dtcenter.org/met/metviewer/metviewer1.jsp
>
> See the attached XML and resulting plot. This shows the reliability
> diagrams for probabilities from 3 different ensembles for 1-hour
> precip > 0.10". I've also attached the corresponding data file and
> R-script used to generate this plot (also output from METviewer).
The
> templatized R-script is rather difficult to parse through, but it
might be useful.
>
> Thanks,
> John Halley Gotway
>
> On Fri, May 22, 2020 at 6:17 PM efren.serra.ctr at nrlmry.navy.mil via
RT
> < met_help at ucar.edu> wrote:
>
> >
> > Fri May 22 18:17:18 2020: Request 95347 was acted upon.
> > Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> >        Queue: met_help
> >      Subject: Reliability diagram
> >        Owner: Nobody
> >   Requestors: efren.serra.ctr at nrlmry.navy.mil
> >       Status: new
> >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347
> > >
> >
> >
> > Folks - I get from MET tools OY_TP [observed YES total proportion]
> > and ON_TP [observed NO total proportion] and I'm plotting OY, [.1,
> > ..., .9] as the reliability diagrams. Is this correct?
> >
> > Efren A. Serra (Contractor)
> > Physicist
> >
> > DeVine Consulting, Inc.
> > Naval Research Laboratory
> > Marine Meteorology Division
> > 7 Grace Hopper Ave., STOP 2
> > Monterey, CA 93943
> > Code 7542
> > Mobile: 408-425-5027
> >
> >
> >
>
>
>
>

--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research Research Applications
Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu

My working day may not be your working day.  Please do not feel
obliged to reply to this email outside of your normal working hours.



------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #95347] Reliability diagram
From: efren.serra.ctr at nrlmry.navy.mil
Time: Wed May 27 17:35:44 2020

Julie - In some R code that John sent me, there's this file I need:

setwd("/opt/vxwww/tomcat/webapps/metviewer/R_work/");
source("include/util_plot.R"); *<<< This one. >>>*

I was wondering if you could send it to me. Thank you.


-----Original Message-----
From: Julie Prestopnik via RT <met_help at ucar.edu>
Sent: Tuesday, May 26, 2020 12:10 PM
To: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram

Hi Efren.  I just wanted to let you know that John is out of the
office until later this week.  Unfortunately, I am unable to answer
your question.  I'm sure you will hear from John shortly after he
returns to work.

Julie

On Mon, May 25, 2020 at 11:42 PM efren.serra.ctr at nrlmry.navy.mil via
RT < met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
>
> John - I should point out that I'm using v8.1 of MET tools and that
> I'm parsing *pjc.txt* output files.
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Sunday, May 24, 2020 2:07 PM
> To: Serra, Mr. Efren, Contractor, Code 7531 <
> efren.serra.ctr at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
>
> Efren,
>
> Not quite.  Here's a useful link about reliability diagrams from the
WMO:
> http://www.bom.gov.au/wmo/lrfvs/reliability.shtml
>
> The points along the X-axis are the forecast probability
thresholds..
> Since you're using probability bins of 0 to 0.1, 0.1 to 0.2, and so
> on, I'd recommend plotting the center-point of each bin. So along
the
> X-axis, you'd plot values at 0.5, 0.15, 0.25, 0.35, 0.45, 0.55,
0.65,
> 0.75, 0.85, and 0.95.
>
> The points along the Y-axis are the observed relative frequency
values.
> And that's the OY_TP value from the MET output.
>
> I also worked up an example using METviewer on some sample
> probabilistic output data:
> http://www.dtcenter.org/met/metviewer/metviewer1.jsp
>
> See the attached XML and resulting plot. This shows the reliability
> diagrams for probabilities from 3 different ensembles for 1-hour
> precip > 0.10". I've also attached the corresponding data file and
> R-script used to generate this plot (also output from METviewer).
The
> templatized R-script is rather difficult to parse through, but it
might be useful.
>
> Thanks,
> John Halley Gotway
>
> On Fri, May 22, 2020 at 6:17 PM efren.serra.ctr at nrlmry.navy.mil via
RT
> < met_help at ucar.edu> wrote:
>
> >
> > Fri May 22 18:17:18 2020: Request 95347 was acted upon.
> > Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> >        Queue: met_help
> >      Subject: Reliability diagram
> >        Owner: Nobody
> >   Requestors: efren.serra.ctr at nrlmry.navy.mil
> >       Status: new
> >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347
> > >
> >
> >
> > Folks - I get from MET tools OY_TP [observed YES total proportion]
> > and ON_TP [observed NO total proportion] and I'm plotting OY, [.1,
> > ..., .9] as the reliability diagrams. Is this correct?
> >
> > Efren A. Serra (Contractor)
> > Physicist
> >
> > DeVine Consulting, Inc.
> > Naval Research Laboratory
> > Marine Meteorology Division
> > 7 Grace Hopper Ave., STOP 2
> > Monterey, CA 93943
> > Code 7542
> > Mobile: 408-425-5027
> >
> >
> >
>
>
>
>

--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research Research Applications
Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu

My working day may not be your working day.  Please do not feel
obliged to reply to this email outside of your normal working hours.



------------------------------------------------
Subject: Reliability diagram
From: George McCabe
Time: Wed May 27 18:30:03 2020

Hi,

John and Julie are both out of the office today, but I can do my best
to
help.

The util_plot.R script can be found in the METViewer repository:

https://github.com/NCAR/METviewer/blob/master/R_work/include/util_plot.R

The command before the source line that sets the working directory to
/opt/vxwww/tomcat/webapps/metviewer/R_work/ should allow the script to
find
the file in the include directory. I am not sure why that file would
not be
in that directory. I may not be understanding the issue correctly. Let
me
know if that doesn't help. We may have to wait until John gets back.

Thanks,
George


On Wed, May 27, 2020 at 5:35 PM efren.serra.ctr at nrlmry.navy.mil via RT
<
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
>
> Julie - In some R code that John sent me, there's this file I need:
>
> setwd("/opt/vxwww/tomcat/webapps/metviewer/R_work/");
> source("include/util_plot.R"); *<<< This one. >>>*
>
> I was wondering if you could send it to me. Thank you.
>
>
> -----Original Message-----
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: Tuesday, May 26, 2020 12:10 PM
> To: Serra, Mr. Efren, Contractor, Code 7531 <
> efren.serra.ctr at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
>
> Hi Efren.  I just wanted to let you know that John is out of the
office
> until later this week.  Unfortunately, I am unable to answer your
> question.  I'm sure you will hear from John shortly after he returns
to
> work.
>
> Julie
>
> On Mon, May 25, 2020 at 11:42 PM efren.serra.ctr at nrlmry.navy.mil via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
> >
> > John - I should point out that I'm using v8.1 of MET tools and
that
> > I'm parsing *pjc.txt* output files.
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Sunday, May 24, 2020 2:07 PM
> > To: Serra, Mr. Efren, Contractor, Code 7531 <
> > efren.serra.ctr at nrlmry.navy.mil>
> > Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
> >
> > Efren,
> >
> > Not quite.  Here's a useful link about reliability diagrams from
the WMO:
> > http://www.bom.gov.au/wmo/lrfvs/reliability.shtml
> >
> > The points along the X-axis are the forecast probability
thresholds..
> > Since you're using probability bins of 0 to 0.1, 0.1 to 0.2, and
so
> > on, I'd recommend plotting the center-point of each bin. So along
the
> > X-axis, you'd plot values at 0.5, 0.15, 0.25, 0.35, 0.45, 0.55,
0.65,
> > 0.75, 0.85, and 0.95.
> >
> > The points along the Y-axis are the observed relative frequency
values.
> > And that's the OY_TP value from the MET output.
> >
> > I also worked up an example using METviewer on some sample
> > probabilistic output data:
> > http://www.dtcenter.org/met/metviewer/metviewer1.jsp
> >
> > See the attached XML and resulting plot. This shows the
reliability
> > diagrams for probabilities from 3 different ensembles for 1-hour
> > precip > 0.10". I've also attached the corresponding data file and
> > R-script used to generate this plot (also output from METviewer).
The
> > templatized R-script is rather difficult to parse through, but it
might
> be useful.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Fri, May 22, 2020 at 6:17 PM efren.serra.ctr at nrlmry.navy.mil
via RT
> > < met_help at ucar.edu> wrote:
> >
> > >
> > > Fri May 22 18:17:18 2020: Request 95347 was acted upon.
> > > Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> > >        Queue: met_help
> > >      Subject: Reliability diagram
> > >        Owner: Nobody
> > >   Requestors: efren.serra.ctr at nrlmry.navy.mil
> > >       Status: new
> > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347
> > > >
> > >
> > >
> > > Folks - I get from MET tools OY_TP [observed YES total
proportion]
> > > and ON_TP [observed NO total proportion] and I'm plotting OY,
[.1,
> > > ..., .9] as the reliability diagrams. Is this correct?
> > >
> > > Efren A. Serra (Contractor)
> > > Physicist
> > >
> > > DeVine Consulting, Inc.
> > > Naval Research Laboratory
> > > Marine Meteorology Division
> > > 7 Grace Hopper Ave., STOP 2
> > > Monterey, CA 93943
> > > Code 7542
> > > Mobile: 408-425-5027
> > >
> > >
> > >
> >
> >
> >
> >
>
> --
> Julie Prestopnik
> Software Engineer
> National Center for Atmospheric Research Research Applications
Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day.  Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #95347] Reliability diagram
From: efren.serra.ctr at nrlmry.navy.mil
Time: Wed May 27 18:31:59 2020

George - I'm running the code from my desktop using R studio in
Anaconda. Do you know how one can plot a reliability diagram from the
*pjc.txt, pstd.txt, prc.txt* files that MET v8 grid_stat tools
generate? Thanks.

-----Original Message-----
From: George McCabe via RT <met_help at ucar.edu>
Sent: Wednesday, May 27, 2020 5:30 PM
To: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram

Hi,

John and Julie are both out of the office today, but I can do my best
to help.

The util_plot.R script can be found in the METViewer repository:

https://github.com/NCAR/METviewer/blob/master/R_work/include/util_plot.R

The command before the source line that sets the working directory to
/opt/vxwww/tomcat/webapps/metviewer/R_work/ should allow the script to
find the file in the include directory. I am not sure why that file
would not be in that directory. I may not be understanding the issue
correctly. Let me know if that doesn't help. We may have to wait until
John gets back.

Thanks,
George


On Wed, May 27, 2020 at 5:35 PM efren.serra.ctr at nrlmry.navy.mil via RT
< met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
>
> Julie - In some R code that John sent me, there's this file I need:
>
> setwd("/opt/vxwww/tomcat/webapps/metviewer/R_work/");
> source("include/util_plot.R"); *<<< This one. >>>*
>
> I was wondering if you could send it to me. Thank you.
>
>
> -----Original Message-----
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: Tuesday, May 26, 2020 12:10 PM
> To: Serra, Mr. Efren, Contractor, Code 7531 <
> efren.serra.ctr at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
>
> Hi Efren.  I just wanted to let you know that John is out of the
> office until later this week.  Unfortunately, I am unable to answer
> your question.  I'm sure you will hear from John shortly after he
> returns to work.
>
> Julie
>
> On Mon, May 25, 2020 at 11:42 PM efren.serra.ctr at nrlmry.navy.mil via
> RT < met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
> >
> > John - I should point out that I'm using v8.1 of MET tools and
that
> > I'm parsing *pjc.txt* output files.
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Sunday, May 24, 2020 2:07 PM
> > To: Serra, Mr. Efren, Contractor, Code 7531 <
> > efren.serra.ctr at nrlmry.navy.mil>
> > Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
> >
> > Efren,
> >
> > Not quite.  Here's a useful link about reliability diagrams from
the WMO:
> > http://www.bom.gov.au/wmo/lrfvs/reliability.shtml
> >
> > The points along the X-axis are the forecast probability
thresholds..
> > Since you're using probability bins of 0 to 0.1, 0.1 to 0.2, and
so
> > on, I'd recommend plotting the center-point of each bin. So along
> > the X-axis, you'd plot values at 0.5, 0.15, 0.25, 0.35, 0.45,
0.55,
> > 0.65, 0.75, 0.85, and 0.95.
> >
> > The points along the Y-axis are the observed relative frequency
values.
> > And that's the OY_TP value from the MET output.
> >
> > I also worked up an example using METviewer on some sample
> > probabilistic output data:
> > http://www.dtcenter.org/met/metviewer/metviewer1.jsp
> >
> > See the attached XML and resulting plot. This shows the
reliability
> > diagrams for probabilities from 3 different ensembles for 1-hour
> > precip > 0.10". I've also attached the corresponding data file and
> > R-script used to generate this plot (also output from METviewer).
> > The templatized R-script is rather difficult to parse through, but
> > it might
> be useful.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Fri, May 22, 2020 at 6:17 PM efren.serra.ctr at nrlmry.navy.mil
via
> > RT < met_help at ucar.edu> wrote:
> >
> > >
> > > Fri May 22 18:17:18 2020: Request 95347 was acted upon.
> > > Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> > >        Queue: met_help
> > >      Subject: Reliability diagram
> > >        Owner: Nobody
> > >   Requestors: efren.serra.ctr at nrlmry.navy.mil
> > >       Status: new
> > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347
> > > >
> > >
> > >
> > > Folks - I get from MET tools OY_TP [observed YES total
proportion]
> > > and ON_TP [observed NO total proportion] and I'm plotting OY,
[.1,
> > > ..., .9] as the reliability diagrams. Is this correct?
> > >
> > > Efren A. Serra (Contractor)
> > > Physicist
> > >
> > > DeVine Consulting, Inc.
> > > Naval Research Laboratory
> > > Marine Meteorology Division
> > > 7 Grace Hopper Ave., STOP 2
> > > Monterey, CA 93943
> > > Code 7542
> > > Mobile: 408-425-5027
> > >
> > >
> > >
> >
> >
> >
> >
>
> --
> Julie Prestopnik
> Software Engineer
> National Center for Atmospheric Research Research Applications
> Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day.  Please do not feel
> obliged to reply to this email outside of your normal working hours.
>
>
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research Research Applications
Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to reply to this email outside of your normal working hours.



------------------------------------------------
Subject: Reliability diagram
From: John Halley Gotway
Time: Thu May 28 13:05:08 2020

Efren,

I see you've had some correspondence with George and Julie on this
issue. I
just wanted to check in to see what questions remain.

You had asked, "according to WMO link below, the plot of
OY_TP[i]/(OY_TP[i]+ON_TP[i]) versus the .5, .15, ..., .95 would be the
reliability diagram, isn't it?"

It's actually even easier than that. The x-value is midpoint of the
forecast probability bin and the y-value is the contents of the
BASER_i
column. I've attached sample PJC and PCT output files.

The following R commands make a reliability diagram for the 4-th line
of
the PJC output:












*pjc=read.table("grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pjc.txt",
header=TRUE)row=4x=c()y=c()ni=pjc[row,"N_THRESH"]for(i in 0:(ni-1))
{ x=c(x, (pjc[row,paste("THRESH_", i, sep='')] +
pjc[row,paste("THRESH_",
i+1, sep='')])/2) y=c(y, pjc[row,paste("BASER_", i,
sep='')])}png("reli_pjc.png")plot(x, y, type="b", xlim=c(0,1),
ylim=c(0,1),main=pjc[row,"FCST_VAR"])dev.off()*

Or you can process the raw PCT counts instead of the rates using this
sample R code:














*pct=read.table("grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pct.txt",
header=TRUE)row=4x=c()oy=c()oy_plus_on=c()ni=pct[row,"N_THRESH"]for(i
in
0:(ni-1)) { x=c(x, (pct[row,paste("THRESH_", i, sep='')] +
pct[row,paste("THRESH_", i+1, sep='')])/2) oy=c(oy,
pct[row,paste("OY_", i,
sep='')]) oy_plus_on=c(oy_plus_on, pct[row,paste("OY_", i, sep='')] +
pct[row,paste("ON_", i, sep='')])}png("reli.png)plot(x, oy/oy_plus_on,
type="b", xlim=c(0,1), ylim=c(0,1),main=pct[row,"FCST_VAR"])dev.off()*

Both produce the same reliability curve.

John

On Wed, May 27, 2020 at 6:32 PM efren.serra.ctr at nrlmry.navy.mil via RT
<
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
>
> George - I'm running the code from my desktop using R studio in
Anaconda.
> Do you know how one can plot a reliability diagram from the
*pjc.txt,
> pstd.txt, prc.txt* files that MET v8 grid_stat tools generate?
Thanks.
>
> -----Original Message-----
> From: George McCabe via RT <met_help at ucar.edu>
> Sent: Wednesday, May 27, 2020 5:30 PM
> To: Serra, Mr. Efren, Contractor, Code 7531 <
> efren.serra.ctr at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
>
> Hi,
>
> John and Julie are both out of the office today, but I can do my
best to
> help.
>
> The util_plot.R script can be found in the METViewer repository:
>
>
https://github.com/NCAR/METviewer/blob/master/R_work/include/util_plot.R
>
> The command before the source line that sets the working directory
to
> /opt/vxwww/tomcat/webapps/metviewer/R_work/ should allow the script
to find
> the file in the include directory. I am not sure why that file would
not be
> in that directory. I may not be understanding the issue correctly.
Let me
> know if that doesn't help. We may have to wait until John gets back.
>
> Thanks,
> George
>
>
> On Wed, May 27, 2020 at 5:35 PM efren.serra.ctr at nrlmry.navy.mil via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
> >
> > Julie - In some R code that John sent me, there's this file I
need:
> >
> > setwd("/opt/vxwww/tomcat/webapps/metviewer/R_work/");
> > source("include/util_plot.R"); *<<< This one. >>>*
> >
> > I was wondering if you could send it to me. Thank you.
> >
> >
> > -----Original Message-----
> > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > Sent: Tuesday, May 26, 2020 12:10 PM
> > To: Serra, Mr. Efren, Contractor, Code 7531 <
> > efren.serra.ctr at nrlmry.navy.mil>
> > Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
> >
> > Hi Efren.  I just wanted to let you know that John is out of the
> > office until later this week.  Unfortunately, I am unable to
answer
> > your question.  I'm sure you will hear from John shortly after he
> > returns to work.
> >
> > Julie
> >
> > On Mon, May 25, 2020 at 11:42 PM efren.serra.ctr at nrlmry.navy.mil
via
> > RT < met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
> > >
> > > John - I should point out that I'm using v8.1 of MET tools and
that
> > > I'm parsing *pjc.txt* output files.
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Sunday, May 24, 2020 2:07 PM
> > > To: Serra, Mr. Efren, Contractor, Code 7531 <
> > > efren.serra.ctr at nrlmry.navy.mil>
> > > Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
> > >
> > > Efren,
> > >
> > > Not quite.  Here's a useful link about reliability diagrams from
the
> WMO:
> > > http://www.bom.gov.au/wmo/lrfvs/reliability.shtml
> > >
> > > The points along the X-axis are the forecast probability
thresholds..
> > > Since you're using probability bins of 0 to 0.1, 0.1 to 0.2, and
so
> > > on, I'd recommend plotting the center-point of each bin. So
along
> > > the X-axis, you'd plot values at 0.5, 0.15, 0.25, 0.35, 0.45,
0.55,
> > > 0.65, 0.75, 0.85, and 0.95.
> > >
> > > The points along the Y-axis are the observed relative frequency
values.
> > > And that's the OY_TP value from the MET output.
> > >
> > > I also worked up an example using METviewer on some sample
> > > probabilistic output data:
> > > http://www.dtcenter.org/met/metviewer/metviewer1.jsp
> > >
> > > See the attached XML and resulting plot. This shows the
reliability
> > > diagrams for probabilities from 3 different ensembles for 1-hour
> > > precip > 0.10". I've also attached the corresponding data file
and
> > > R-script used to generate this plot (also output from
METviewer).
> > > The templatized R-script is rather difficult to parse through,
but
> > > it might
> > be useful.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Fri, May 22, 2020 at 6:17 PM efren.serra.ctr at nrlmry.navy.mil
via
> > > RT < met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Fri May 22 18:17:18 2020: Request 95347 was acted upon.
> > > > Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> > > >        Queue: met_help
> > > >      Subject: Reliability diagram
> > > >        Owner: Nobody
> > > >   Requestors: efren.serra.ctr at nrlmry.navy.mil
> > > >       Status: new
> > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347
> > > > >
> > > >
> > > >
> > > > Folks - I get from MET tools OY_TP [observed YES total
proportion]
> > > > and ON_TP [observed NO total proportion] and I'm plotting OY,
[.1,
> > > > ..., .9] as the reliability diagrams. Is this correct?
> > > >
> > > > Efren A. Serra (Contractor)
> > > > Physicist
> > > >
> > > > DeVine Consulting, Inc.
> > > > Naval Research Laboratory
> > > > Marine Meteorology Division
> > > > 7 Grace Hopper Ave., STOP 2
> > > > Monterey, CA 93943
> > > > Code 7542
> > > > Mobile: 408-425-5027
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> > --
> > Julie Prestopnik
> > Software Engineer
> > National Center for Atmospheric Research Research Applications
> > Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day.  Please do not feel
> > obliged to reply to this email outside of your normal working
hours.
> >
> >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research Research Applications
Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
>
>

------------------------------------------------
Subject: Reliability diagram
From: John Halley Gotway
Time: Thu May 28 13:05:08 2020

VERSION MODEL DESC       FCST_LEAD FCST_VALID_BEG  FCST_VALID_END
OBS_LEAD OBS_VALID_BEG   OBS_VALID_END   FCST_VAR          FCST_UNITS
FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK    INTERP_MTHD
INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL
N_THRESH THRESH_1 OY_TP_1    ON_TP_1 CALIBRATION_1 REFINEMENT_1
LIKELIHOOD_1 BASER_1    THRESH_2 OY_TP_2    ON_TP_2   CALIBRATION_2
REFINEMENT_2 LIKELIHOOD_2 BASER_2   THRESH_3 OY_TP_3    ON_TP_3
CALIBRATION_3 REFINEMENT_3 LIKELIHOOD_3 BASER_3   THRESH_4 OY_TP_4
ON_TP_4    CALIBRATION_4 REFINEMENT_4 LIKELIHOOD_4 BASER_4   THRESH_5
OY_TP_5   ON_TP_5    CALIBRATION_5 REFINEMENT_5 LIKELIHOOD_5 BASER_5
THRESH_6 OY_TP_6    ON_TP_6    CALIBRATION_6 REFINEMENT_6 LIKELIHOOD_6
BASER_6  THRESH_7 OY_TP_7    ON_TP_7    CALIBRATION_7 REFINEMENT_7
LIKELIHOOD_7 BASER_7  THRESH_8 OY_TP_8    ON_TP_8    CALIBRATION_8
REFINEMENT_8 LIKELIHOOD_8 BASER_8  THRESH_9 OY_TP_9    ON_TP_9
CALIBRATION_9 REFINEMENT_9 LIKELIHOOD_9 BASER_9  THRESH_10 OY_TP_10
ON_TP_10   CALIBRATION_10 REFINEMENT_10 LIKELIHOOD_10 BASER_10
THRESH_11
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_1 NEAREST     1
==0.10000   >12.89     NA         NA    PJC        2635       11
0 0          1          0               1         NA         0
0.1 0          0             NA           0           NA         NA
0.2 0          0              NA           0           NA         NA
0.3 0          0              NA           0           NA         NA
0.4 0         0               NA         0             NA        NA
0.5 0          0               NA         0            NA         NA
0.6 0          0               NA         0             NA        NA
0.7 0          0               NA          0            NA        NA
0.8 0          0               NA          0            NA        NA
0.9 0         0                NA           0              NA       NA
1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_2 NEAREST     1
==0.10000   >12.89     NA         NA    PJC        5907       11
0 0          1          0               1         NA         0
0.1 0          0             NA           0           NA         NA
0.2 0          0              NA           0           NA         NA
0.3 0          0              NA           0           NA         NA
0.4 0         0               NA         0             NA        NA
0.5 0          0               NA         0            NA         NA
0.6 0          0               NA         0             NA        NA
0.7 0          0               NA          0            NA        NA
0.8 0          0               NA          0            NA        NA
0.9 0         0                NA           0              NA       NA
1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_3 NEAREST     1
==0.10000   >12.89     NA         NA    PJC       15323       11
0 0.00065261 0.95647    0.00068185      0.95712    0.037453
0.00068185      0.1 0.00039157 0.010442       0.036145    0.010833
0.022472   0.036145      0.2 0.00071788 0.0088103       0.075342
0.0095282    0.041199   0.075342      0.3 0.00097892 0.0017621
0.35714     0.002741     0.05618    0.35714       0.4 0.0008484
0.0016968        0.33333   0.0025452      0.048689  0.33333      0.5
0.0011094  0.0013052        0.45946   0.0024147     0.06367    0.45946
0.6 0.001501   0.0015663        0.48936   0.0030673      0.086142
0.48936      0.7 0.0017621  0.00045683       0.79412    0.0022189
0.10112   0.79412      0.8 0.0021536  6.5261e-05       0.97059
0.0022189     0.1236    0.97059       0.9 0.0073093 0
1           0.0073093       0.41948  1               1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP<273.000) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_1 NEAREST     1
==0.10000   <273.0     NA         NA    PJC       14571       11
0 0.0025393  0.76673    0.0033009       0.76927    0.012724  0.0033009
0.1 0.00075492 0.0083042      0.083333    0.0090591    0.0037827
0.083333      0.2 0.003706   0.010157        0.26733     0.013863
0.018569   0.26733       0.3 0.0019216  0.0025393       0.43077
0.0044609    0.0096286  0.43077       0.4 0.0027452 0.0043237
0.38835   0.0070688      0.013755  0.38835      0.5 0.012628
0.0027452        0.82143   0.015373      0.063274   0.82143      0.6
0.0046668  0.0021275        0.68687   0.0067943      0.023384  0.68687
0.7 0.0099513  0.0017844        0.84795    0.011736      0.049862
0.84795      0.8 0.0040491  0.00075492       0.84286    0.0048041
0.020289  0.84286       0.9 0.15661   0.00096081        0.9939
0.15757         0.78473  0.9939          1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP<273.000) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_2 NEAREST     1
==0.10000   <273.0     NA         NA    PJC        4160       11
0 0.00024038 0.78293    0.00030694      0.78317    0.0013812
0.00030694      0.1 0.00096154 0.010577       0.083333    0.011538
0.0055249  0.083333      0.2 0.0038462  0.014183        0.21333
0.018029     0.022099   0.21333       0.3 0.0019231  0.0024038
0.44444     0.0043269    0.01105    0.44444       0.4 0.0021635
0.0028846        0.42857   0.0050481      0.012431  0.42857      0.5
0.011058   0.0057692        0.65714   0.016827      0.063536   0.65714
0.6 0.010096   0.0026442        0.79245   0.01274        0.058011
0.79245      0.7 0.016827   0.0040865        0.8046     0.020913
0.096685  0.8046       0.8 0.0064904  0.00024038       0.96429
0.0067308     0.037293  0.96429       0.9 0.12043   0.00024038
0.99801     0.12067         0.69199  0.99801         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP<273.000) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_3 NEAREST     1
==0.10000   <273.0     NA         NA    PJC        5134       11
0 0.00038956 0.5857     0.00066467      0.58609    0.001113
0.00066467      0.1 0.0013635  0.0070121      0.16279     0.0083755
0.0038954  0.16279       0.2 0.0021426  0.009739        0.18033
0.011882     0.0061213  0.18033       0.3 0.006233   0.0054538
0.53333     0.011687     0.017807   0.53333       0.4 0.0042852
0.0048695        0.46809   0.0091547      0.012243  0.46809      0.5
0.0070121  0.0083755        0.4557    0.015388      0.020033   0.4557
0.6 0.0068173  0.0058434        0.53846   0.012661       0.019477
0.53846      0.7 0.007986   0.006233         0.56164    0.014219
0.022816  0.56164      0.8 0.0083755  0.0044799        0.65152
0.012855      0.023929  0.65152       0.9 0.30541   0.012271
0.96137     0.31769         0.87257  0.96137         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP>298.800) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_1 NEAREST     1
==0.10000   >298.8     NA         NA    PJC       14571       11
0 0          0.97715    0               0.97715    0         0
0.1 0          0.0021275      0           0.0021275    0          0
0.2 0.00020589 0.0068629       0.029126    0.0070688    0.034884
0.029126      0.3 6.8629e-05 0.0015785       0.041667    0.0016471
0.011628   0.041667      0.4 0         0.0010981        0
0.0010981      0         0            0.5 0.00020589 0.00041178
0.33333   0.00061767    0.034884   0.33333      0.6 6.8629e-05
0.00061767       0.1       0.00068629     0.011628  0.1          0.7
0.00054904 0.0012353        0.30769    0.0017844     0.093023  0.30769
0.8 0.00054904 0.00089218       0.38095    0.0014412     0.093023
0.38095       0.9 0.004255  0.0021275         0.66667     0.0063825
0.72093  0.66667         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP>298.800) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_2 NEAREST     1
==0.10000   >298.8     NA         NA    PJC        4160       11
0 0          0.95144    0               0.95144    0         0
0.1 0          0.0033654      0           0.0033654    0          0
0.2 0          0.0043269       0           0.0043269    0          0
0.3 0          0.0012019       0           0.0012019    0          0
0.4 0         0.00096154       0         0.00096154     0         0
0.5 0.00096154 0.0012019        0.44444   0.0021635     0.029851
0.44444      0.6 0.0012019  0.00096154       0.55556   0.0021635
0.037313  0.55556      0.7 0.00072115 0.0012019        0.375
0.0019231     0.022388  0.375        0.8 0.00625    0.0019231
0.76471    0.0081731     0.19403   0.76471       0.9 0.023077
0.0012019         0.9505      0.024279        0.71642  0.9505
1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP>298.800) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_3 NEAREST     1
==0.10000   >298.8     NA         NA    PJC        5134       11
0 0.00077912 0.94079    0.00082747      0.94157    0.014035
0.00082747      0.1 0.00077912 0.0021426      0.26667     0.0029217
0.014035   0.26667       0.2 0.0019478  0.00077912      0.71429
0.0027269    0.035088   0.71429       0.3 0.0015582  0.00019478
0.88889     0.001753     0.02807    0.88889       0.4 0.0013635
0.00038956       0.77778   0.001753       0.024561  0.77778      0.5
0.00038956 0.00019478       0.66667   0.00058434    0.0070175  0.66667
0.6 0.001753   0                1         0.001753       0.031579  1
0.7 0.006233   0                1          0.006233      0.11228   1
0.8 0.011297   0                1          0.011297      0.20351   1
0.9 0.029412  0                 1           0.029412        0.52982  1
1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_1 NEAREST     1
==0.10000   >12.89     NA         NA    PJC        2635       11
0 0          1          0               1         NA         0
0.1 0          0             NA           0           NA         NA
0.2 0          0              NA           0           NA         NA
0.3 0          0              NA           0           NA         NA
0.4 0         0               NA         0             NA        NA
0.5 0          0               NA         0            NA         NA
0.6 0          0               NA         0             NA        NA
0.7 0          0               NA          0            NA        NA
0.8 0          0               NA          0            NA        NA
0.9 0         0                NA           0              NA       NA
1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_2 NEAREST     1
==0.10000   >12.89     NA         NA    PJC        5907       11
0 0          1          0               1         NA         0
0.1 0          0             NA           0           NA         NA
0.2 0          0              NA           0           NA         NA
0.3 0          0              NA           0           NA         NA
0.4 0         0               NA         0             NA        NA
0.5 0          0               NA         0            NA         NA
0.6 0          0               NA         0             NA        NA
0.7 0          0               NA          0            NA        NA
0.8 0          0               NA          0            NA        NA
0.9 0         0                NA           0              NA       NA
1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_3 NEAREST     1
==0.10000   >12.89     NA         NA    PJC       15323       11
0 0.00065261 0.95647    0.00068185      0.95712    0.037453
0.00068185      0.1 0.00039157 0.010442       0.036145    0.010833
0.022472   0.036145      0.2 0.00071788 0.0088103       0.075342
0.0095282    0.041199   0.075342      0.3 0.00097892 0.0017621
0.35714     0.002741     0.05618    0.35714       0.4 0.0008484
0.0016968        0.33333   0.0025452      0.048689  0.33333      0.5
0.0011094  0.0013052        0.45946   0.0024147     0.06367    0.45946
0.6 0.001501   0.0015663        0.48936   0.0030673      0.086142
0.48936      0.7 0.0017621  0.00045683       0.79412    0.0022189
0.10112   0.79412      0.8 0.0021536  6.5261e-05       0.97059
0.0022189     0.1236    0.97059       0.9 0.0073093 0
1           0.0073093       0.41948  1               1

------------------------------------------------
Subject: Reliability diagram
From: John Halley Gotway
Time: Thu May 28 13:05:08 2020

VERSION MODEL DESC       FCST_LEAD FCST_VALID_BEG  FCST_VALID_END
OBS_LEAD OBS_VALID_BEG   OBS_VALID_END   FCST_VAR          FCST_UNITS
FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK    INTERP_MTHD
INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL
N_THRESH THRESH_1 OY_1 ON_1  THRESH_2 OY_2 ON_2 THRESH_3 OY_3 ON_3
THRESH_4 OY_4 ON_4 THRESH_5 OY_5 ON_5 THRESH_6 OY_6 ON_6 THRESH_7 OY_7
ON_7 THRESH_8 OY_8 ON_8 THRESH_9 OY_9 ON_9 THRESH_10 OY_10 ON_10
THRESH_11
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_1 NEAREST     1
==0.10000   >12.89     NA         NA    PCT        2635       11
0    0  2635      0.1    0    0      0.2    0    0      0.3    0    0
0.4    0    0      0.5    0    0      0.6    0    0      0.7    0    0
0.8    0    0       0.9     0     0         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_2 NEAREST     1
==0.10000   >12.89     NA         NA    PCT        5907       11
0    0  5907      0.1    0    0      0.2    0    0      0.3    0    0
0.4    0    0      0.5    0    0      0.6    0    0      0.7    0    0
0.8    0    0       0.9     0     0         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_3 NEAREST     1
==0.10000   >12.89     NA         NA    PCT       15323       11
0   10 14656      0.1    6  160      0.2   11  135      0.3   15   27
0.4   13   26      0.5   17   20      0.6   23   24      0.7   27    7
0.8   33    1       0.9   112     0         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP<273.000) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_1 NEAREST     1
==0.10000   <273.0     NA         NA    PCT       14571       11
0   37 11172      0.1   11  121      0.2   54  148      0.3   28   37
0.4   40   63      0.5  184   40      0.6   68   31      0.7  145   26
0.8   59   11       0.9  2282    14         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP<273.000) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_2 NEAREST     1
==0.10000   <273.0     NA         NA    PCT        4160       11
0    1  3257      0.1    4   44      0.2   16   59      0.3    8   10
0.4    9   12      0.5   46   24      0.6   42   11      0.7   70   17
0.8   27    1       0.9   501     1         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP<273.000) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_3 NEAREST     1
==0.10000   <273.0     NA         NA    PCT        5134       11
0    2  3007      0.1    7   36      0.2   11   50      0.3   32   28
0.4   22   25      0.5   36   43      0.6   35   30      0.7   41   32
0.8   43   23       0.9  1568    63         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP>298.800) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_1 NEAREST     1
==0.10000   >298.8     NA         NA    PCT       14571       11
0    0 14238      0.1    0   31      0.2    3  100      0.3    1   23
0.4    0   16      0.5    3    6      0.6    1    9      0.7    8   18
0.8    8   13       0.9    62    31         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP>298.800) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_2 NEAREST     1
==0.10000   >298.8     NA         NA    PCT        4160       11
0    0  3958      0.1    0   14      0.2    0   18      0.3    0    5
0.4    0    4      0.5    4    5      0.6    5    4      0.7    3    5
0.8   26    8       0.9    96     5         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(TMP>298.800) %
Z2       TMP     K         Z2      GFSANL FULL_BIN_3 NEAREST     1
==0.10000   >298.8     NA         NA    PCT        5134       11
0    4  4830      0.1    4   11      0.2   10    4      0.3    8    1
0.4    7    2      0.5    2    1      0.6    9    0      0.7   32    0
0.8   58    0       0.9   151     0         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_1 NEAREST     1
==0.10000   >12.89     NA         NA    PCT        2635       11
0    0  2635      0.1    0    0      0.2    0    0      0.3    0    0
0.4    0    0      0.5    0    0      0.6    0    0      0.7    0    0
0.8    0    0       0.9     0     0         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_2 NEAREST     1
==0.10000   >12.89     NA         NA    PCT        5907       11
0    0  5907      0.1    0    0      0.2    0    0      0.3    0    0
0.4    0    0      0.5    0    0      0.6    0    0      0.7    0    0
0.8    0    0       0.9     0     0         1
V9.1    SREF  CLIMO_BINS 150000    20120409_120000 20120409_120000
000000   20120409_120000 20120409_120000 PROB(WIND>12.890) %
Z10      WIND    m/s       Z10     GFSANL FULL_BIN_3 NEAREST     1
==0.10000   >12.89     NA         NA    PCT       15323       11
0   10 14656      0.1    6  160      0.2   11  135      0.3   15   27
0.4   13   26      0.5   17   20      0.6   23   24      0.7   27    7
0.8   33    1       0.9   112     0         1

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #95347] Reliability diagram
From: efren.serra.ctr at nrlmry.navy.mil
Time: Thu May 28 16:42:14 2020

Beatiful! Thanks

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, May 28, 2020 12:05 PM
To: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram

Efren,

I see you've had some correspondence with George and Julie on this
issue. I just wanted to check in to see what questions remain.

You had asked, "according to WMO link below, the plot of
OY_TP[i]/(OY_TP[i]+ON_TP[i]) versus the .5, .15, ..., .95 would be the
reliability diagram, isn't it?"

It's actually even easier than that. The x-value is midpoint of the
forecast probability bin and the y-value is the contents of the
BASER_i column. I've attached sample PJC and PCT output files.

The following R commands make a reliability diagram for the 4-th line
of the PJC output:












*pjc=read.table("grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pjc.txt",
header=TRUE)row=4x=c()y=c()ni=pjc[row,"N_THRESH"]for(i in 0:(ni-1)) {
x=c(x, (pjc[row,paste("THRESH_", i, sep='')] +
pjc[row,paste("THRESH_",
i+1, sep='')])/2) y=c(y, pjc[row,paste("BASER_", i,
sep='')])}png("reli_pjc.png")plot(x, y, type="b", xlim=c(0,1),
ylim=c(0,1),main=pjc[row,"FCST_VAR"])dev.off()*

Or you can process the raw PCT counts instead of the rates using this
sample R code:














*pct=read.table("grid_stat_PROB_GFS_CLIMO_1.0DEG_150000L_20120409_120000V_pct.txt",
header=TRUE)row=4x=c()oy=c()oy_plus_on=c()ni=pct[row,"N_THRESH"]for(i
in
0:(ni-1)) { x=c(x, (pct[row,paste("THRESH_", i, sep='')] +
pct[row,paste("THRESH_", i+1, sep='')])/2) oy=c(oy,
pct[row,paste("OY_", i,
sep='')]) oy_plus_on=c(oy_plus_on, pct[row,paste("OY_", i, sep='')] +
pct[row,paste("ON_", i, sep='')])}png("reli.png)plot(x, oy/oy_plus_on,
type="b", xlim=c(0,1), ylim=c(0,1),main=pct[row,"FCST_VAR"])dev.off()*

Both produce the same reliability curve.

John

On Wed, May 27, 2020 at 6:32 PM efren.serra.ctr at nrlmry.navy.mil via RT
< met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
>
> George - I'm running the code from my desktop using R studio in
Anaconda.
> Do you know how one can plot a reliability diagram from the
*pjc.txt,
> pstd.txt, prc.txt* files that MET v8 grid_stat tools generate?
Thanks.
>
> -----Original Message-----
> From: George McCabe via RT <met_help at ucar.edu>
> Sent: Wednesday, May 27, 2020 5:30 PM
> To: Serra, Mr. Efren, Contractor, Code 7531 <
> efren.serra.ctr at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
>
> Hi,
>
> John and Julie are both out of the office today, but I can do my
best
> to help.
>
> The util_plot.R script can be found in the METViewer repository:
>
>
https://github.com/NCAR/METviewer/blob/master/R_work/include/util_plot
> .R
>
> The command before the source line that sets the working directory
to
> /opt/vxwww/tomcat/webapps/metviewer/R_work/ should allow the script
to
> find the file in the include directory. I am not sure why that file
> would not be in that directory. I may not be understanding the issue
> correctly. Let me know if that doesn't help. We may have to wait
until John gets back.
>
> Thanks,
> George
>
>
> On Wed, May 27, 2020 at 5:35 PM efren.serra.ctr at nrlmry.navy.mil via
RT
> < met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
> >
> > Julie - In some R code that John sent me, there's this file I
need:
> >
> > setwd("/opt/vxwww/tomcat/webapps/metviewer/R_work/");
> > source("include/util_plot.R"); *<<< This one. >>>*
> >
> > I was wondering if you could send it to me. Thank you.
> >
> >
> > -----Original Message-----
> > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > Sent: Tuesday, May 26, 2020 12:10 PM
> > To: Serra, Mr. Efren, Contractor, Code 7531 <
> > efren.serra.ctr at nrlmry.navy.mil>
> > Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
> >
> > Hi Efren.  I just wanted to let you know that John is out of the
> > office until later this week.  Unfortunately, I am unable to
answer
> > your question.  I'm sure you will hear from John shortly after he
> > returns to work.
> >
> > Julie
> >
> > On Mon, May 25, 2020 at 11:42 PM efren.serra.ctr at nrlmry.navy.mil
via
> > RT < met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347 >
> > >
> > > John - I should point out that I'm using v8.1 of MET tools and
> > > that I'm parsing *pjc.txt* output files.
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Sunday, May 24, 2020 2:07 PM
> > > To: Serra, Mr. Efren, Contractor, Code 7531 <
> > > efren.serra.ctr at nrlmry.navy.mil>
> > > Subject: Re: [rt.rap.ucar.edu #95347] Reliability diagram
> > >
> > > Efren,
> > >
> > > Not quite.  Here's a useful link about reliability diagrams from
> > > the
> WMO:
> > > http://www.bom.gov.au/wmo/lrfvs/reliability.shtml
> > >
> > > The points along the X-axis are the forecast probability
thresholds..
> > > Since you're using probability bins of 0 to 0.1, 0.1 to 0.2, and
> > > so on, I'd recommend plotting the center-point of each bin. So
> > > along the X-axis, you'd plot values at 0.5, 0.15, 0.25, 0.35,
> > > 0.45, 0.55, 0.65, 0.75, 0.85, and 0.95.
> > >
> > > The points along the Y-axis are the observed relative frequency
values.
> > > And that's the OY_TP value from the MET output.
> > >
> > > I also worked up an example using METviewer on some sample
> > > probabilistic output data:
> > > http://www.dtcenter.org/met/metviewer/metviewer1.jsp
> > >
> > > See the attached XML and resulting plot. This shows the
> > > reliability diagrams for probabilities from 3 different
ensembles
> > > for 1-hour precip > 0.10". I've also attached the corresponding
> > > data file and R-script used to generate this plot (also output
from METviewer).
> > > The templatized R-script is rather difficult to parse through,
but
> > > it might
> > be useful.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Fri, May 22, 2020 at 6:17 PM efren.serra.ctr at nrlmry.navy.mil
> > > via RT < met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Fri May 22 18:17:18 2020: Request 95347 was acted upon.
> > > > Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> > > >        Queue: met_help
> > > >      Subject: Reliability diagram
> > > >        Owner: Nobody
> > > >   Requestors: efren.serra.ctr at nrlmry.navy.mil
> > > >       Status: new
> > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95347
> > > > >
> > > >
> > > >
> > > > Folks - I get from MET tools OY_TP [observed YES total
> > > > proportion] and ON_TP [observed NO total proportion] and I'm
> > > > plotting OY, [.1, ..., .9] as the reliability diagrams. Is
this correct?
> > > >
> > > > Efren A. Serra (Contractor)
> > > > Physicist
> > > >
> > > > DeVine Consulting, Inc.
> > > > Naval Research Laboratory
> > > > Marine Meteorology Division
> > > > 7 Grace Hopper Ave., STOP 2
> > > > Monterey, CA 93943
> > > > Code 7542
> > > > Mobile: 408-425-5027
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> > --
> > Julie Prestopnik
> > Software Engineer
> > National Center for Atmospheric Research Research Applications
> > Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day.  Please do not feel
> > obliged to reply to this email outside of your normal working
hours.
> >
> >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research Research Applications
> Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged
> to reply to this email outside of your normal working hours.
>
>
>
>



------------------------------------------------


More information about the Met_help mailing list