[Met_help] [rt.rap.ucar.edu #92343] History for Probabilistic forecast error bars?
John Halley Gotway via RT
met_help at ucar.edu
Fri Oct 11 16:23:06 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi again,
So this question is more about following up regarding the output of grid_stat & stat_analysis and for my clarification. I now have the probabilistic output in the form of PCT, PJC, PSTD, and PRC ASCII files and am able to compute things like the reliability and ROC curves for each of my cases. The one thing that I am missing is the creation of the error bars to go with the points on these curves. Would I be able to extract the errors for each bin from any of the probabilistic output tables or is that something that has to be derived?
Thanks a lot for the continuing support!
-Brian
—————————————————————
Brian Matilla
Research Fellow— Warn-on-Forecast Team
Cooperative Institute for Mesoscale Meteorological Studies — The University of Oklahoma
NOAA National Severe Storms Laboratory
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Probabilistic forecast error bars?
From: John Halley Gotway
Time: Thu Sep 26 11:52:23 2019
Hello Brian,
There is not a way to add CI's to ROC and Reliability Diagrams in the
MET
tools themselves.
However, you can do so using the METviewer database and display
system.
I've attached a sample METviewer plot and corresponding plot XML spec
file
which demonstrates doing so using the publicly available instance of
METviewer.
I found some probabilistic vx output in METviewer and made a plot
showing
the reliability of the probability of 6-hour precip exceeding 1/2
inch.
The three lines on the plot are for forecast hours 12, 24, and 36.
I put CI's on these curves using the random resampling bootstrap
method
available in METviewer.
You can recreate this plot by...
(1) Save the attached XML file.
(2) Go to: http://www.dtcenter.org/met/metviewer/metviewer1.jsp
(3) In the upper-right corner, select "Load XML".
(4) Navigate to and select the XML file you saved and click OK.
(5) That populates the GUI with all the correct selections... and
click
"Generate Plot" at the top of the GUI.
This isn't instantaneous. Computing 1000 randomly resampled bootstrap
stats takes up to about a minute. If you were to turn off
bootstrapping by
setting the number of replicates to 0 ("0 Bootstrapping
replications"), it
runs in 5 seconds or so.
Do you have an instance of METviewer available to you?
Thanks,
John Halley Gotway
On Thu, Sep 26, 2019 at 9:29 AM Brian Matilla - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Thu Sep 26 09:28:32 2019: Request 92343 was acted upon.
> Transaction: Ticket created by brian.matilla at noaa.gov
> Queue: met_help
> Subject: Probabilistic forecast error bars?
> Owner: Nobody
> Requestors: brian.matilla at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92343 >
>
>
> Hi again,
>
> So this question is more about following up regarding the output of
> grid_stat & stat_analysis and for my clarification. I now have the
> probabilistic output in the form of PCT, PJC, PSTD, and PRC ASCII
files and
> am able to compute things like the reliability and ROC curves for
each of
> my cases. The one thing that I am missing is the creation of the
error bars
> to go with the points on these curves. Would I be able to extract
the
> errors for each bin from any of the probabilistic output tables or
is that
> something that has to be derived?
>
> Thanks a lot for the continuing support!
>
> -Brian
> —————————————————————
> Brian Matilla
> Research Fellow— Warn-on-Forecast Team
> Cooperative Institute for Mesoscale Meteorological Studies — The
> University of Oklahoma
> NOAA National Severe Storms Laboratory
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #92343] Probabilistic forecast error bars?
From: Brian Matilla - NOAA Affiliate
Time: Thu Sep 26 16:18:27 2019
Hi John,
I just got to explore a little bit with the XML and that was very
useful in visualizing the CIs and it’s exactly what I’m looking to do.
As far as METviewer is concerned, I currently do not have an instance
available to me. Is there any way that I could get set up to use
METviewer?
-Brian
> On Sep 26, 2019, at 12:52 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Hello Brian,
>
> There is not a way to add CI's to ROC and Reliability Diagrams in
the MET
> tools themselves.
>
> However, you can do so using the METviewer database and display
system.
> I've attached a sample METviewer plot and corresponding plot XML
spec file
> which demonstrates doing so using the publicly available instance of
> METviewer.
>
> I found some probabilistic vx output in METviewer and made a plot
showing
> the reliability of the probability of 6-hour precip exceeding 1/2
inch.
> The three lines on the plot are for forecast hours 12, 24, and 36.
>
> I put CI's on these curves using the random resampling bootstrap
method
> available in METviewer.
>
> You can recreate this plot by...
> (1) Save the attached XML file.
> (2) Go to: http://www.dtcenter.org/met/metviewer/metviewer1.jsp
> (3) In the upper-right corner, select "Load XML".
> (4) Navigate to and select the XML file you saved and click OK.
> (5) That populates the GUI with all the correct selections... and
click
> "Generate Plot" at the top of the GUI.
>
> This isn't instantaneous. Computing 1000 randomly resampled
bootstrap
> stats takes up to about a minute. If you were to turn off
bootstrapping by
> setting the number of replicates to 0 ("0 Bootstrapping
replications"), it
> runs in 5 seconds or so.
>
> Do you have an instance of METviewer available to you?
>
> Thanks,
> John Halley Gotway
>
>
>
> On Thu, Sep 26, 2019 at 9:29 AM Brian Matilla - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> Thu Sep 26 09:28:32 2019: Request 92343 was acted upon.
>> Transaction: Ticket created by brian.matilla at noaa.gov
>> Queue: met_help
>> Subject: Probabilistic forecast error bars?
>> Owner: Nobody
>> Requestors: brian.matilla at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92343 >
>>
>>
>> Hi again,
>>
>> So this question is more about following up regarding the output of
>> grid_stat & stat_analysis and for my clarification. I now have the
>> probabilistic output in the form of PCT, PJC, PSTD, and PRC ASCII
files and
>> am able to compute things like the reliability and ROC curves for
each of
>> my cases. The one thing that I am missing is the creation of the
error bars
>> to go with the points on these curves. Would I be able to extract
the
>> errors for each bin from any of the probabilistic output tables or
is that
>> something that has to be derived?
>>
>> Thanks a lot for the continuing support!
>>
>> -Brian
>> —————————————————————
>> Brian Matilla
>> Research Fellow— Warn-on-Forecast Team
>> Cooperative Institute for Mesoscale Meteorological Studies — The
>> University of Oklahoma
>> NOAA National Severe Storms Laboratory
>>
>>
>>
>
> <plot_20190926_174504.xml><plot_20190926_174504.png>
------------------------------------------------
Subject: Probabilistic forecast error bars?
From: John Halley Gotway
Time: Thu Sep 26 16:55:11 2019
Tara (and Brian),
Brian Matilla, from NOAA, would really benefit from being able to load
his
MET output into METviewer to put CI's on his ROC and Reliability
diagrams.
But I'm wondering where I should direct him. Here are the locations
of
METviewer instances:
NCAR METviewer instance:
http://www.dtcenter.org/met/metviewer/metviewer1.jsp
NOAA/ESRL instance:
https://www.esrl.noaa.gov/gsd/metvgsd/metviewer1.jsp
NOAA/EMC instance on AWS:
https://metviewer.nws.noaa.gov/metviewer1.jsp
(Log on with NOAA credentials)
METviewer as a Docker container:
https://github.com/NCAR/container-dtc-metviewer
Would any of these be a good fit?
Thanks,
John
On Thu, Sep 26, 2019 at 4:18 PM Brian Matilla - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92343 >
>
> Hi John,
>
> I just got to explore a little bit with the XML and that was very
useful
> in visualizing the CIs and it’s exactly what I’m looking to do.
>
> As far as METviewer is concerned, I currently do not have an
instance
> available to me. Is there any way that I could get set up to use
METviewer?
>
> -Brian
>
> > On Sep 26, 2019, at 12:52 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
> >
> > Hello Brian,
> >
> > There is not a way to add CI's to ROC and Reliability Diagrams in
the MET
> > tools themselves.
> >
> > However, you can do so using the METviewer database and display
system.
> > I've attached a sample METviewer plot and corresponding plot XML
spec
> file
> > which demonstrates doing so using the publicly available instance
of
> > METviewer.
> >
> > I found some probabilistic vx output in METviewer and made a plot
showing
> > the reliability of the probability of 6-hour precip exceeding 1/2
inch.
> > The three lines on the plot are for forecast hours 12, 24, and 36.
> >
> > I put CI's on these curves using the random resampling bootstrap
method
> > available in METviewer.
> >
> > You can recreate this plot by...
> > (1) Save the attached XML file.
> > (2) Go to: http://www.dtcenter.org/met/metviewer/metviewer1.jsp
> > (3) In the upper-right corner, select "Load XML".
> > (4) Navigate to and select the XML file you saved and click OK.
> > (5) That populates the GUI with all the correct selections... and
click
> > "Generate Plot" at the top of the GUI.
> >
> > This isn't instantaneous. Computing 1000 randomly resampled
bootstrap
> > stats takes up to about a minute. If you were to turn off
bootstrapping
> by
> > setting the number of replicates to 0 ("0 Bootstrapping
replications"),
> it
> > runs in 5 seconds or so.
> >
> > Do you have an instance of METviewer available to you?
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> >
> > On Thu, Sep 26, 2019 at 9:29 AM Brian Matilla - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> Thu Sep 26 09:28:32 2019: Request 92343 was acted upon.
> >> Transaction: Ticket created by brian.matilla at noaa.gov
> >> Queue: met_help
> >> Subject: Probabilistic forecast error bars?
> >> Owner: Nobody
> >> Requestors: brian.matilla at noaa.gov
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92343 >
> >>
> >>
> >> Hi again,
> >>
> >> So this question is more about following up regarding the output
of
> >> grid_stat & stat_analysis and for my clarification. I now have
the
> >> probabilistic output in the form of PCT, PJC, PSTD, and PRC ASCII
files
> and
> >> am able to compute things like the reliability and ROC curves for
each
> of
> >> my cases. The one thing that I am missing is the creation of the
error
> bars
> >> to go with the points on these curves. Would I be able to extract
the
> >> errors for each bin from any of the probabilistic output tables
or is
> that
> >> something that has to be derived?
> >>
> >> Thanks a lot for the continuing support!
> >>
> >> -Brian
> >> —————————————————————
> >> Brian Matilla
> >> Research Fellow— Warn-on-Forecast Team
> >> Cooperative Institute for Mesoscale Meteorological Studies — The
> >> University of Oklahoma
> >> NOAA National Severe Storms Laboratory
> >>
> >>
> >>
> >
> > <plot_20190926_174504.xml><plot_20190926_174504.png>
>
>
>
>
------------------------------------------------
Subject: Probabilistic forecast error bars?
From: Tara Jensen
Time: Fri Sep 27 08:01:27 2019
I believe Brian is at NSSL. In which case, maybe the most appropriate
instance is the one in a container on either Buxton2 or BigBang2. Not
sure
where it was installed. Hank Fisher would know. Brian could ask
Burkely
Gallo or Brett Roberts where it is also.
Cheers, Tara
On Thu, Sep 26, 2019 at 4:55 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Tara (and Brian),
>
> Brian Matilla, from NOAA, would really benefit from being able to
load his
> MET output into METviewer to put CI's on his ROC and Reliability
diagrams.
>
> But I'm wondering where I should direct him. Here are the locations
of
> METviewer instances:
>
> NCAR METviewer instance:
> http://www.dtcenter.org/met/metviewer/metviewer1.jsp
> NOAA/ESRL instance:
https://www.esrl.noaa.gov/gsd/metvgsd/metviewer1.jsp
> NOAA/EMC instance on AWS:
https://metviewer.nws.noaa.gov/metviewer1.jsp
> (Log on with NOAA credentials)
> METviewer as a Docker container:
> https://github.com/NCAR/container-dtc-metviewer
>
> Would any of these be a good fit?
>
> Thanks,
> John
>
>
> On Thu, Sep 26, 2019 at 4:18 PM Brian Matilla - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92343 >
>>
>> Hi John,
>>
>> I just got to explore a little bit with the XML and that was very
useful
>> in visualizing the CIs and it’s exactly what I’m looking to do.
>>
>> As far as METviewer is concerned, I currently do not have an
instance
>> available to me. Is there any way that I could get set up to use
METviewer?
>>
>> -Brian
>>
>> > On Sep 26, 2019, at 12:52 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>> >
>> > Hello Brian,
>> >
>> > There is not a way to add CI's to ROC and Reliability Diagrams in
the
>> MET
>> > tools themselves.
>> >
>> > However, you can do so using the METviewer database and display
system.
>> > I've attached a sample METviewer plot and corresponding plot XML
spec
>> file
>> > which demonstrates doing so using the publicly available instance
of
>> > METviewer.
>> >
>> > I found some probabilistic vx output in METviewer and made a plot
>> showing
>> > the reliability of the probability of 6-hour precip exceeding 1/2
inch.
>> > The three lines on the plot are for forecast hours 12, 24, and
36.
>> >
>> > I put CI's on these curves using the random resampling bootstrap
method
>> > available in METviewer.
>> >
>> > You can recreate this plot by...
>> > (1) Save the attached XML file.
>> > (2) Go to: http://www.dtcenter.org/met/metviewer/metviewer1.jsp
>> > (3) In the upper-right corner, select "Load XML".
>> > (4) Navigate to and select the XML file you saved and click OK.
>> > (5) That populates the GUI with all the correct selections... and
click
>> > "Generate Plot" at the top of the GUI.
>> >
>> > This isn't instantaneous. Computing 1000 randomly resampled
bootstrap
>> > stats takes up to about a minute. If you were to turn off
>> bootstrapping by
>> > setting the number of replicates to 0 ("0 Bootstrapping
replications"),
>> it
>> > runs in 5 seconds or so.
>> >
>> > Do you have an instance of METviewer available to you?
>> >
>> > Thanks,
>> > John Halley Gotway
>> >
>> >
>> >
>> > On Thu, Sep 26, 2019 at 9:29 AM Brian Matilla - NOAA Affiliate
via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >>
>> >> Thu Sep 26 09:28:32 2019: Request 92343 was acted upon.
>> >> Transaction: Ticket created by brian.matilla at noaa.gov
>> >> Queue: met_help
>> >> Subject: Probabilistic forecast error bars?
>> >> Owner: Nobody
>> >> Requestors: brian.matilla at noaa.gov
>> >> Status: new
>> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92343 >
>> >>
>> >>
>> >> Hi again,
>> >>
>> >> So this question is more about following up regarding the output
of
>> >> grid_stat & stat_analysis and for my clarification. I now have
the
>> >> probabilistic output in the form of PCT, PJC, PSTD, and PRC
ASCII
>> files and
>> >> am able to compute things like the reliability and ROC curves
for each
>> of
>> >> my cases. The one thing that I am missing is the creation of the
error
>> bars
>> >> to go with the points on these curves. Would I be able to
extract the
>> >> errors for each bin from any of the probabilistic output tables
or is
>> that
>> >> something that has to be derived?
>> >>
>> >> Thanks a lot for the continuing support!
>> >>
>> >> -Brian
>> >> —————————————————————
>> >> Brian Matilla
>> >> Research Fellow— Warn-on-Forecast Team
>> >> Cooperative Institute for Mesoscale Meteorological Studies — The
>> >> University of Oklahoma
>> >> NOAA National Severe Storms Laboratory
>> >>
>> >>
>> >>
>> >
>> > <plot_20190926_174504.xml><plot_20190926_174504.png>
>>
>>
>>
>>
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tara Jensen
Project Manager II
NCAR RAL and DTC
PO Box 3000, Boulder, Colorado 80307 USA
+1 303-497-8479 jensen at ucar.edu
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #92343] Probabilistic forecast error bars?
From: Brian Matilla - NOAA Affiliate
Time: Fri Sep 27 16:52:10 2019
Hi Tara (and John),
Apologies for the slower reply today… I’ll get in touch with either of
them to see where that instance of METviewer is.
Thanks!
-Brian
> On Sep 27, 2019, at 9:01 AM, Tara Jensen via RT <met_help at ucar.edu>
wrote:
>
> I believe Brian is at NSSL. In which case, maybe the most
appropriate
> instance is the one in a container on either Buxton2 or BigBang2.
Not sure
> where it was installed. Hank Fisher would know. Brian could ask
Burkely
> Gallo or Brett Roberts where it is also.
> Cheers, Tara
>
>
> On Thu, Sep 26, 2019 at 4:55 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
>> Tara (and Brian),
>>
>> Brian Matilla, from NOAA, would really benefit from being able to
load his
>> MET output into METviewer to put CI's on his ROC and Reliability
diagrams.
>>
>> But I'm wondering where I should direct him. Here are the
locations of
>> METviewer instances:
>>
>> NCAR METviewer instance:
>> http://www.dtcenter.org/met/metviewer/metviewer1.jsp
>> NOAA/ESRL instance:
https://www.esrl.noaa.gov/gsd/metvgsd/metviewer1.jsp
>> NOAA/EMC instance on AWS:
https://metviewer.nws.noaa.gov/metviewer1.jsp
>> (Log on with NOAA credentials)
>> METviewer as a Docker container:
>> https://github.com/NCAR/container-dtc-metviewer
>>
>> Would any of these be a good fit?
>>
>> Thanks,
>> John
>>
>>
>> On Thu, Sep 26, 2019 at 4:18 PM Brian Matilla - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92343 >
>>>
>>> Hi John,
>>>
>>> I just got to explore a little bit with the XML and that was very
useful
>>> in visualizing the CIs and it’s exactly what I’m looking to do.
>>>
>>> As far as METviewer is concerned, I currently do not have an
instance
>>> available to me. Is there any way that I could get set up to use
METviewer?
>>>
>>> -Brian
>>>
>>>> On Sep 26, 2019, at 12:52 PM, John Halley Gotway via RT <
>>> met_help at ucar.edu> wrote:
>>>>
>>>> Hello Brian,
>>>>
>>>> There is not a way to add CI's to ROC and Reliability Diagrams in
the
>>> MET
>>>> tools themselves.
>>>>
>>>> However, you can do so using the METviewer database and display
system.
>>>> I've attached a sample METviewer plot and corresponding plot XML
spec
>>> file
>>>> which demonstrates doing so using the publicly available instance
of
>>>> METviewer.
>>>>
>>>> I found some probabilistic vx output in METviewer and made a plot
>>> showing
>>>> the reliability of the probability of 6-hour precip exceeding 1/2
inch.
>>>> The three lines on the plot are for forecast hours 12, 24, and
36.
>>>>
>>>> I put CI's on these curves using the random resampling bootstrap
method
>>>> available in METviewer.
>>>>
>>>> You can recreate this plot by...
>>>> (1) Save the attached XML file.
>>>> (2) Go to: http://www.dtcenter.org/met/metviewer/metviewer1.jsp
>>>> (3) In the upper-right corner, select "Load XML".
>>>> (4) Navigate to and select the XML file you saved and click OK.
>>>> (5) That populates the GUI with all the correct selections... and
click
>>>> "Generate Plot" at the top of the GUI.
>>>>
>>>> This isn't instantaneous. Computing 1000 randomly resampled
bootstrap
>>>> stats takes up to about a minute. If you were to turn off
>>> bootstrapping by
>>>> setting the number of replicates to 0 ("0 Bootstrapping
replications"),
>>> it
>>>> runs in 5 seconds or so.
>>>>
>>>> Do you have an instance of METviewer available to you?
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>>
>>>>
>>>>
>>>> On Thu, Sep 26, 2019 at 9:29 AM Brian Matilla - NOAA Affiliate
via RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>>
>>>>> Thu Sep 26 09:28:32 2019: Request 92343 was acted upon.
>>>>> Transaction: Ticket created by brian.matilla at noaa.gov
>>>>> Queue: met_help
>>>>> Subject: Probabilistic forecast error bars?
>>>>> Owner: Nobody
>>>>> Requestors: brian.matilla at noaa.gov
>>>>> Status: new
>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92343 >
>>>>>
>>>>>
>>>>> Hi again,
>>>>>
>>>>> So this question is more about following up regarding the output
of
>>>>> grid_stat & stat_analysis and for my clarification. I now have
the
>>>>> probabilistic output in the form of PCT, PJC, PSTD, and PRC
ASCII
>>> files and
>>>>> am able to compute things like the reliability and ROC curves
for each
>>> of
>>>>> my cases. The one thing that I am missing is the creation of the
error
>>> bars
>>>>> to go with the points on these curves. Would I be able to
extract the
>>>>> errors for each bin from any of the probabilistic output tables
or is
>>> that
>>>>> something that has to be derived?
>>>>>
>>>>> Thanks a lot for the continuing support!
>>>>>
>>>>> -Brian
>>>>> —————————————————————
>>>>> Brian Matilla
>>>>> Research Fellow— Warn-on-Forecast Team
>>>>> Cooperative Institute for Mesoscale Meteorological Studies — The
>>>>> University of Oklahoma
>>>>> NOAA National Severe Storms Laboratory
>>>>>
>>>>>
>>>>>
>>>>
>>>> <plot_20190926_174504.xml><plot_20190926_174504.png>
>>>
>>>
>>>
>>>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Tara Jensen
> Project Manager II
> NCAR RAL and DTC
> PO Box 3000, Boulder, Colorado 80307 USA
> +1 303-497-8479 jensen at ucar.edu
>
------------------------------------------------
More information about the Met_help
mailing list