[Met_help] [rt.rap.ucar.edu #86767] History for ROC/Area under ROC in METviewer

John Halley Gotway via RT met_help at ucar.edu
Wed Jul 10 16:43:17 MDT 2019


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hello.  I'm trying to make a simple ROC plot.  The attached *.xml does 
not work - err msg

"Failed to run plot plot_20180827_184319 - reason: R error: ==== Start 
Rscript error ==== Error in .subset2(x, i, exact = exact) : subscript 
out of bounds In addition."

I verified that data for the forecast valid time/lead time (24h ending 
at 12Z 21 Aug) do exist and I could plot POD/FAR from them.  What should 
I be doing differently in the set up?  I asked Ben Blake about this 
(googling found his inquiries on ROC/AUR from early 2017) and he said to 
put OBS_THRESH under series variables instead of fixed values, that 
didn't work either.

Also, once I get ROC to work ... where is AUR?  Is it under Roc tab, or 
Series?  I can't figure out where it could be under Roc; under Series 
there doesn't seem to be something that might be AUR under "Select 
attribute stat".

Thanks!

Ying

-- 
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov




----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: ROC/Area under ROC in METviewer
From: Tatiana Burek
Time: Mon Aug 27 14:43:53 2018

Hi Ying,
ROC plop is fixed and I was able to create a plot using your XML.
What do you mean by AUR?

Tatiana

On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
> Hello.  I'm trying to make a simple ROC plot.  The attached *.xml
does
> not work - err msg
>
> "Failed to run plot plot_20180827_184319 - reason: R error: ====
Start
> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
> out of bounds In addition."
>
> I verified that data for the forecast valid time/lead time (24h
ending
> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.  What
should
> I be doing differently in the set up?  I asked Ben Blake about this
> (googling found his inquiries on ROC/AUR from early 2017) and he
said to
> put OBS_THRESH under series variables instead of fixed values, that
> didn't work either.
>
> Also, once I get ROC to work ... where is AUR?  Is it under Roc tab,
or
> Series?  I can't figure out where it could be under Roc; under
Series
> there doesn't seem to be something that might be AUR under "Select
> attribute stat".
>
> Thanks!
>
> Ying
>



------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #86767] ROC/Area under ROC in METviewer
From: Ying Lin
Time: Mon Aug 27 15:08:34 2018

Thanks Tatiana.

     Sorry - AUR "Area Under ROC".

     I see your ROC plot and it seems the difference between your xml
and my failed attempt is that you chose 'PCT' instead of 'CTC' for ROC
calculations.  I thought that PCT is for ROC score of probability
forecasts, while CTC is from the contingency table count of single
deterministic models' ROC?

Ying

On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
> Hi Ying,
> ROC plop is fixed and I was able to create a plot using your XML.
> What do you mean by AUR?
>
> Tatiana
>
> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>> Hello.  I'm trying to make a simple ROC plot.  The attached *.xml
does
>> not work - err msg
>>
>> "Failed to run plot plot_20180827_184319 - reason: R error: ====
Start
>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
>> out of bounds In addition."
>>
>> I verified that data for the forecast valid time/lead time (24h
ending
>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.  What
should
>> I be doing differently in the set up?  I asked Ben Blake about this
>> (googling found his inquiries on ROC/AUR from early 2017) and he
said to
>> put OBS_THRESH under series variables instead of fixed values, that
>> didn't work either.
>>
>> Also, once I get ROC to work ... where is AUR?  Is it under Roc
tab, or
>> Series?  I can't figure out where it could be under Roc; under
Series
>> there doesn't seem to be something that might be AUR under "Select
>> attribute stat".
>>
>> Thanks!
>>
>> Ying
>>
>
>

--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov



------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Ying Lin
Time: Mon Aug 27 15:34:21 2018

Hi Tatiana,

     I tried making a ROC plot from MET *.stat (the case of the
original
question was from VSDBs loaded to MV) and the CTC seems to work - see
attached xml.

     Is the CTC option not available for ROC from VSDBs?

     For the ROC from the MET *.stat, I played with the OBS_THRESH a
bit, it appears that the ROC plot can accept up to two different
OBS_THRESH (as in the attached xml), but more than two will result in
an
error.

Thanks again,

Ying

On 08/27/2018 05:08 PM, Ying Lin wrote:
> Thanks Tatiana.
>
>     Sorry - AUR "Area Under ROC".
>
>     I see your ROC plot and it seems the difference between your xml
> and my failed attempt is that you chose 'PCT' instead of 'CTC' for
ROC
> calculations.  I thought that PCT is for ROC score of probability
> forecasts, while CTC is from the contingency table count of single
> deterministic models' ROC?
>
> Ying
>
> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
>> Hi Ying,
>> ROC plop is fixed and I was able to create a plot using your XML.
>> What do you mean by AUR?
>>
>> Tatiana
>>
>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>>> Hello.  I'm trying to make a simple ROC plot.  The attached *.xml
does
>>> not work - err msg
>>>
>>> "Failed to run plot plot_20180827_184319 - reason: R error: ====
Start
>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
>>> out of bounds In addition."
>>>
>>> I verified that data for the forecast valid time/lead time (24h
ending
>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them. What
should
>>> I be doing differently in the set up?  I asked Ben Blake about
this
>>> (googling found his inquiries on ROC/AUR from early 2017) and he
>>> said to
>>> put OBS_THRESH under series variables instead of fixed values,
that
>>> didn't work either.
>>>
>>> Also, once I get ROC to work ... where is AUR?  Is it under Roc
tab, or
>>> Series?  I can't figure out where it could be under Roc; under
Series
>>> there doesn't seem to be something that might be AUR under "Select
>>> attribute stat".
>>>
>>> Thanks!
>>>
>>> Ying
>>>
>>
>>
>

--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov



------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #86767] ROC/Area under ROC in METviewer
From: Ying Lin
Time: Tue Aug 28 12:56:28 2018

Hi Tatiana,

     Follow up on this ... has the ROC plotting in METviewer been used
in the non-probabilistic sense before?  I'm thinking that maybe
obs_thresh shouldn't be "fix values".  The ROC plot seems to be
analogous to the performance diagram, in which data points are
produced
for forecast lead times - but FCST_LEAD in the performance diagrams
are
"independent variables" rather than fixed values.  There is no
"independent variables" option in ROC, the only way to make the
"CTC-type" ROC plot seems to be to set OBS_THRESH as fixed values, but
then it seems only two OBS_THRESH values can be accepted by METviewer.

Thanks -

Ying

On 08/27/2018 05:34 PM, Ying Lin wrote:
> Hi Tatiana,
>
>     I tried making a ROC plot from MET *.stat (the case of the
> original question was from VSDBs loaded to MV) and the CTC seems to
> work - see attached xml.
>
>     Is the CTC option not available for ROC from VSDBs?
>
>     For the ROC from the MET *.stat, I played with the OBS_THRESH a
> bit, it appears that the ROC plot can accept up to two different
> OBS_THRESH (as in the attached xml), but more than two will result
in
> an error.
>
> Thanks again,
>
> Ying
>
> On 08/27/2018 05:08 PM, Ying Lin wrote:
>> Thanks Tatiana.
>>
>>     Sorry - AUR "Area Under ROC".
>>
>>     I see your ROC plot and it seems the difference between your
xml
>> and my failed attempt is that you chose 'PCT' instead of 'CTC' for
>> ROC calculations.  I thought that PCT is for ROC score of
probability
>> forecasts, while CTC is from the contingency table count of single
>> deterministic models' ROC?
>>
>> Ying
>>
>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
>>> Hi Ying,
>>> ROC plop is fixed and I was able to create a plot using your XML.
>>> What do you mean by AUR?
>>>
>>> Tatiana
>>>
>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>>>> Hello.  I'm trying to make a simple ROC plot.  The attached *.xml
does
>>>> not work - err msg
>>>>
>>>> "Failed to run plot plot_20180827_184319 - reason: R error: ====
Start
>>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
>>>> out of bounds In addition."
>>>>
>>>> I verified that data for the forecast valid time/lead time (24h
ending
>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them. What
>>>> should
>>>> I be doing differently in the set up?  I asked Ben Blake about
this
>>>> (googling found his inquiries on ROC/AUR from early 2017) and he
>>>> said to
>>>> put OBS_THRESH under series variables instead of fixed values,
that
>>>> didn't work either.
>>>>
>>>> Also, once I get ROC to work ... where is AUR?  Is it under Roc
>>>> tab, or
>>>> Series?  I can't figure out where it could be under Roc; under
Series
>>>> there doesn't seem to be something that might be AUR under
"Select
>>>> attribute stat".
>>>>
>>>> Thanks!
>>>>
>>>> Ying
>>>>
>>>
>>>
>>
>

--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov



------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: John Halley Gotway
Time: Fri Sep 07 17:00:53 2018

Hi Tressa,

I was supposed to talk to you about this issue earlier this week but
forgot.  I think we need your guidance on this one.

This is a met-help ticket from Ying Lin at NCEP about ROC curves in
METviewer.  I believe there are 2 ways to create a ROC curve in
METviewer:

(1) The most common way I know is using the Nx2 Probabilistic
Contingency
Table counts from the PCT line in MET.  I understand this one.  MET
can
also write out the PSTD line type which includes the ROC_AUC column.
And
that's a number between 0 and 1 indicating the area under the ROC
curve,
with 1 being the best.  MET calculates this value and it gets loaded
into
METviewer.  METviewer can plot the ROC_AUC value but it doesn't
compute it.

(2) The less common way of generating a ROC curve is using several 2x2
contingency table lines.  This would be the FHO or CTC line types from
MET.  However, there's some requirement in how those CTC lines are
defined... but I can't remember exactly what.  Perhaps they need to be
multiple forecast thresholds but all for the same observation
threshold?
Does that sound correct to you?  If we get all the right, then
METviewer
could make a ROC plot.  But, since METviewer doesn't compute the area
under
the ROC curve, there's no way for METviewer to plot that statistic.

I think this would all be good info to clarify in the METviewer user's
guide:
- Show an example of using PCT output to make a ROC curve in
METviewer.
- Show an example of using CTC output to make a ROC curve in
METviewer...
and list the threshold requirements for doing so.

And lastly, we should identify what changes to METviewer are needed to
make
it more useful to Ying.  Perhaps we do need to enhance METviewer to
compute
the area under the ROC curve for case (2)?

Thanks,
John

On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>
> Hi Tatiana,
>
>      Follow up on this ... has the ROC plotting in METviewer been
used
> in the non-probabilistic sense before?  I'm thinking that maybe
> obs_thresh shouldn't be "fix values".  The ROC plot seems to be
> analogous to the performance diagram, in which data points are
produced
> for forecast lead times - but FCST_LEAD in the performance diagrams
are
> "independent variables" rather than fixed values.  There is no
> "independent variables" option in ROC, the only way to make the
> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed values,
but
> then it seems only two OBS_THRESH values can be accepted by
METviewer.
>
> Thanks -
>
> Ying
>
> On 08/27/2018 05:34 PM, Ying Lin wrote:
> > Hi Tatiana,
> >
> >     I tried making a ROC plot from MET *.stat (the case of the
> > original question was from VSDBs loaded to MV) and the CTC seems
to
> > work - see attached xml.
> >
> >     Is the CTC option not available for ROC from VSDBs?
> >
> >     For the ROC from the MET *.stat, I played with the OBS_THRESH
a
> > bit, it appears that the ROC plot can accept up to two different
> > OBS_THRESH (as in the attached xml), but more than two will result
in
> > an error.
> >
> > Thanks again,
> >
> > Ying
> >
> > On 08/27/2018 05:08 PM, Ying Lin wrote:
> >> Thanks Tatiana.
> >>
> >>     Sorry - AUR "Area Under ROC".
> >>
> >>     I see your ROC plot and it seems the difference between your
xml
> >> and my failed attempt is that you chose 'PCT' instead of 'CTC'
for
> >> ROC calculations.  I thought that PCT is for ROC score of
probability
> >> forecasts, while CTC is from the contingency table count of
single
> >> deterministic models' ROC?
> >>
> >> Ying
> >>
> >> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
> >>> Hi Ying,
> >>> ROC plop is fixed and I was able to create a plot using your
XML.
> >>> What do you mean by AUR?
> >>>
> >>> Tatiana
> >>>
> >>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
> >>>> Hello.  I'm trying to make a simple ROC plot.  The attached
*.xml does
> >>>> not work - err msg
> >>>>
> >>>> "Failed to run plot plot_20180827_184319 - reason: R error:
==== Start
> >>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
> >>>> out of bounds In addition."
> >>>>
> >>>> I verified that data for the forecast valid time/lead time (24h
ending
> >>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.
What
> >>>> should
> >>>> I be doing differently in the set up?  I asked Ben Blake about
this
> >>>> (googling found his inquiries on ROC/AUR from early 2017) and
he
> >>>> said to
> >>>> put OBS_THRESH under series variables instead of fixed values,
that
> >>>> didn't work either.
> >>>>
> >>>> Also, once I get ROC to work ... where is AUR?  Is it under Roc
> >>>> tab, or
> >>>> Series?  I can't figure out where it could be under Roc; under
Series
> >>>> there doesn't seem to be something that might be AUR under
"Select
> >>>> attribute stat".
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Ying
> >>>>
> >>>
> >>>
> >>
> >
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #86767] ROC/Area under ROC in METviewer
From: Ying Lin
Time: Sat Sep 08 16:37:02 2018

Hi John and Tressa,

	I was originally under the impression that the ROC/Area under ROC was
for probabilistic forecasts.  Do they make sense/are they worth
looking at in the non-probabilistic sense (e.g. ROC curve for varying
forecast/obs thresholds)?  I guess a case can be made that the CTC-ROC
makes some sense (similar to the performance diagram), but unlike the
probabilistic ROC curve that rises with decreasing PCT, the "CTC ROC"
based on the F/H/O contingency table counts for different thresholds
(and probably for different forecast lead time as well) can be all
over the place, making the concept of "area under ROC” problematic.

	If a non-prob ROC doesn't make enough sense/is not worth pursuing,
that's fine.  The AUC was listed as one of the measures in the CAM
metrics workshop, so I got curious when seeing the "CTC" option in the
METviewer ROC.  That's all.

Ying

> On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Hi Tressa,
>
> I was supposed to talk to you about this issue earlier this week but
> forgot.  I think we need your guidance on this one.
>
> This is a met-help ticket from Ying Lin at NCEP about ROC curves in
> METviewer.  I believe there are 2 ways to create a ROC curve in
METviewer:
>
> (1) The most common way I know is using the Nx2 Probabilistic
Contingency
> Table counts from the PCT line in MET.  I understand this one.  MET
can
> also write out the PSTD line type which includes the ROC_AUC column.
And
> that's a number between 0 and 1 indicating the area under the ROC
curve,
> with 1 being the best.  MET calculates this value and it gets loaded
into
> METviewer.  METviewer can plot the ROC_AUC value but it doesn't
compute it.
>
> (2) The less common way of generating a ROC curve is using several
2x2
> contingency table lines.  This would be the FHO or CTC line types
from
> MET.  However, there's some requirement in how those CTC lines are
> defined... but I can't remember exactly what.  Perhaps they need to
be
> multiple forecast thresholds but all for the same observation
threshold?
> Does that sound correct to you?  If we get all the right, then
METviewer
> could make a ROC plot.  But, since METviewer doesn't compute the
area under
> the ROC curve, there's no way for METviewer to plot that statistic.
>
> I think this would all be good info to clarify in the METviewer
user's
> guide:
> - Show an example of using PCT output to make a ROC curve in
METviewer.
> - Show an example of using CTC output to make a ROC curve in
METviewer...
> and list the threshold requirements for doing so.
>
> And lastly, we should identify what changes to METviewer are needed
to make
> it more useful to Ying.  Perhaps we do need to enhance METviewer to
compute
> the area under the ROC curve for case (2)?
>
> Thanks,
> John
>
> On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>>
>> Hi Tatiana,
>>
>>     Follow up on this ... has the ROC plotting in METviewer been
used
>> in the non-probabilistic sense before?  I'm thinking that maybe
>> obs_thresh shouldn't be "fix values".  The ROC plot seems to be
>> analogous to the performance diagram, in which data points are
produced
>> for forecast lead times - but FCST_LEAD in the performance diagrams
are
>> "independent variables" rather than fixed values.  There is no
>> "independent variables" option in ROC, the only way to make the
>> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed values,
but
>> then it seems only two OBS_THRESH values can be accepted by
METviewer.
>>
>> Thanks -
>>
>> Ying
>>
>> On 08/27/2018 05:34 PM, Ying Lin wrote:
>>> Hi Tatiana,
>>>
>>>    I tried making a ROC plot from MET *.stat (the case of the
>>> original question was from VSDBs loaded to MV) and the CTC seems
to
>>> work - see attached xml.
>>>
>>>    Is the CTC option not available for ROC from VSDBs?
>>>
>>>    For the ROC from the MET *.stat, I played with the OBS_THRESH a
>>> bit, it appears that the ROC plot can accept up to two different
>>> OBS_THRESH (as in the attached xml), but more than two will result
in
>>> an error.
>>>
>>> Thanks again,
>>>
>>> Ying
>>>
>>> On 08/27/2018 05:08 PM, Ying Lin wrote:
>>>> Thanks Tatiana.
>>>>
>>>>    Sorry - AUR "Area Under ROC".
>>>>
>>>>    I see your ROC plot and it seems the difference between your
xml
>>>> and my failed attempt is that you chose 'PCT' instead of 'CTC'
for
>>>> ROC calculations.  I thought that PCT is for ROC score of
probability
>>>> forecasts, while CTC is from the contingency table count of
single
>>>> deterministic models' ROC?
>>>>
>>>> Ying
>>>>
>>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
>>>>> Hi Ying,
>>>>> ROC plop is fixed and I was able to create a plot using your
XML.
>>>>> What do you mean by AUR?
>>>>>
>>>>> Tatiana
>>>>>
>>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>>>>>> Hello.  I'm trying to make a simple ROC plot.  The attached
*.xml does
>>>>>> not work - err msg
>>>>>>
>>>>>> "Failed to run plot plot_20180827_184319 - reason: R error:
==== Start
>>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
>>>>>> out of bounds In addition."
>>>>>>
>>>>>> I verified that data for the forecast valid time/lead time (24h
ending
>>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.
What
>>>>>> should
>>>>>> I be doing differently in the set up?  I asked Ben Blake about
this
>>>>>> (googling found his inquiries on ROC/AUR from early 2017) and
he
>>>>>> said to
>>>>>> put OBS_THRESH under series variables instead of fixed values,
that
>>>>>> didn't work either.
>>>>>>
>>>>>> Also, once I get ROC to work ... where is AUR?  Is it under Roc
>>>>>> tab, or
>>>>>> Series?  I can't figure out where it could be under Roc; under
Series
>>>>>> there doesn't seem to be something that might be AUR under
"Select
>>>>>> attribute stat".
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Ying
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
>



------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Tressa Fowler
Time: Mon Sep 10 13:55:21 2018

Hi Ying,

It definitely makes sense to calculate AUC for CTC-derived ROC curves,
so long as a single observation threshold is selected, and the
forecast thresholds fill in the curve pretty well. I think we can
create an issue to add this for future releases.

Tressa

On Sat Sep 08 16:37:02 2018, ying.lin at noaa.gov wrote:
> Hi John and Tressa,
>
> I was originally under the impression that the ROC/Area under ROC
was
> for probabilistic forecasts.  Do they make sense/are they worth
> looking at in the non-probabilistic sense (e.g. ROC curve for
varying
> forecast/obs thresholds)?  I guess a case can be made that the CTC-
ROC
> makes some sense (similar to the performance diagram), but unlike
the
> probabilistic ROC curve that rises with decreasing PCT, the "CTC
ROC"
> based on the F/H/O contingency table counts for different thresholds
> (and probably for different forecast lead time as well) can be all
> over the place, making the concept of "area under ROC” problematic.
>
> If a non-prob ROC doesn't make enough sense/is not worth pursuing,
> that's fine.  The AUC was listed as one of the measures in the CAM
> metrics workshop, so I got curious when seeing the "CTC" option in
the
> METviewer ROC.  That's all.
>
> Ying
>
> > On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
> > <met_help at ucar.edu> wrote:
> >
> > Hi Tressa,
> >
> > I was supposed to talk to you about this issue earlier this week
but
> > forgot.  I think we need your guidance on this one.
> >
> > This is a met-help ticket from Ying Lin at NCEP about ROC curves
in
> > METviewer.  I believe there are 2 ways to create a ROC curve in
> > METviewer:
> >
> > (1) The most common way I know is using the Nx2 Probabilistic
> > Contingency
> > Table counts from the PCT line in MET.  I understand this one.
MET
> > can
> > also write out the PSTD line type which includes the ROC_AUC
column.
> > And
> > that's a number between 0 and 1 indicating the area under the ROC
> > curve,
> > with 1 being the best.  MET calculates this value and it gets
loaded
> > into
> > METviewer.  METviewer can plot the ROC_AUC value but it doesn't
> > compute it.
> >
> > (2) The less common way of generating a ROC curve is using several
> > 2x2
> > contingency table lines.  This would be the FHO or CTC line types
> > from
> > MET.  However, there's some requirement in how those CTC lines are
> > defined... but I can't remember exactly what.  Perhaps they need
to
> > be
> > multiple forecast thresholds but all for the same observation
> > threshold?
> > Does that sound correct to you?  If we get all the right, then
> > METviewer
> > could make a ROC plot.  But, since METviewer doesn't compute the
area
> > under
> > the ROC curve, there's no way for METviewer to plot that
statistic.
> >
> > I think this would all be good info to clarify in the METviewer
> > user's
> > guide:
> > - Show an example of using PCT output to make a ROC curve in
> > METviewer.
> > - Show an example of using CTC output to make a ROC curve in
> > METviewer...
> > and list the threshold requirements for doing so.
> >
> > And lastly, we should identify what changes to METviewer are
needed
> > to make
> > it more useful to Ying.  Perhaps we do need to enhance METviewer
to
> > compute
> > the area under the ROC curve for case (2)?
> >
> > Thanks,
> > John
> >
> > On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT
<met_help at ucar.edu>
> > wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
> >>
> >> Hi Tatiana,
> >>
> >> Follow up on this ... has the ROC plotting in METviewer been used
> >> in the non-probabilistic sense before?  I'm thinking that maybe
> >> obs_thresh shouldn't be "fix values".  The ROC plot seems to be
> >> analogous to the performance diagram, in which data points are
> >> produced
> >> for forecast lead times - but FCST_LEAD in the performance
diagrams
> >> are
> >> "independent variables" rather than fixed values.  There is no
> >> "independent variables" option in ROC, the only way to make the
> >> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed
values,
> >> but
> >> then it seems only two OBS_THRESH values can be accepted by
> >> METviewer.
> >>
> >> Thanks -
> >>
> >> Ying
> >>
> >> On 08/27/2018 05:34 PM, Ying Lin wrote:
> >>> Hi Tatiana,
> >>>
> >>> I tried making a ROC plot from MET *.stat (the case of the
> >>> original question was from VSDBs loaded to MV) and the CTC seems
to
> >>> work - see attached xml.
> >>>
> >>> Is the CTC option not available for ROC from VSDBs?
> >>>
> >>> For the ROC from the MET *.stat, I played with the OBS_THRESH a
> >>> bit, it appears that the ROC plot can accept up to two different
> >>> OBS_THRESH (as in the attached xml), but more than two will
result
> >>> in
> >>> an error.
> >>>
> >>> Thanks again,
> >>>
> >>> Ying
> >>>
> >>> On 08/27/2018 05:08 PM, Ying Lin wrote:
> >>>> Thanks Tatiana.
> >>>>
> >>>> Sorry - AUR "Area Under ROC".
> >>>>
> >>>> I see your ROC plot and it seems the difference between your
xml
> >>>> and my failed attempt is that you chose 'PCT' instead of 'CTC'
for
> >>>> ROC calculations.  I thought that PCT is for ROC score of
> >>>> probability
> >>>> forecasts, while CTC is from the contingency table count of
single
> >>>> deterministic models' ROC?
> >>>>
> >>>> Ying
> >>>>
> >>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
> >>>>> Hi Ying,
> >>>>> ROC plop is fixed and I was able to create a plot using your
XML.
> >>>>> What do you mean by AUR?
> >>>>>
> >>>>> Tatiana
> >>>>>
> >>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
> >>>>>> Hello.  I'm trying to make a simple ROC plot.  The attached
> >>>>>> *.xml does
> >>>>>> not work - err msg
> >>>>>>
> >>>>>> "Failed to run plot plot_20180827_184319 - reason: R error:
====
> >>>>>> Start
> >>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
> >>>>>> subscript
> >>>>>> out of bounds In addition."
> >>>>>>
> >>>>>> I verified that data for the forecast valid time/lead time
(24h
> >>>>>> ending
> >>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.
What
> >>>>>> should
> >>>>>> I be doing differently in the set up?  I asked Ben Blake
about
> >>>>>> this
> >>>>>> (googling found his inquiries on ROC/AUR from early 2017) and
he
> >>>>>> said to
> >>>>>> put OBS_THRESH under series variables instead of fixed
values,
> >>>>>> that
> >>>>>> didn't work either.
> >>>>>>
> >>>>>> Also, once I get ROC to work ... where is AUR?  Is it under
Roc
> >>>>>> tab, or
> >>>>>> Series?  I can't figure out where it could be under Roc;
under
> >>>>>> Series
> >>>>>> there doesn't seem to be something that might be AUR under
> >>>>>> "Select
> >>>>>> attribute stat".
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>> Ying
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
> >



------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: John Halley Gotway
Time: Mon Sep 10 14:25:06 2018

Ying,

I believe that METviewer would be able to make this plot now (although
I
haven't confirmed that recently).  But you would have to configure MET
in
the right way to define the thresholds the way Tressa described.

For example, let's say you're interested in 1/2 inch of precip
accumulation
(i.e. 12.7 mm).  You define the thresholds in the MET config file like
the
following.  I've listed the same obs threshold 10 times (>= 1/2 inch).
But
I've listed 10 different forecast thresholds (e.g. >=0,
>=0.1",...,>=1.0").

fcst = {
...
  cat_thresh = [ ge0.00, ge2.54, ge5.08, ge7.62, ge10.16, ge12.7,
ge15.24,
ge17.78, ge20.32, ge22.86, ge25.4 ];
...
}
obs = {
...
  cat_thresh = [ ge12.7, ge12.7, ge12.7, ge12.7, ge12.7, ge12.7,
ge12.7,
ge12.7, ge12.7, ge12.7 ];
...
}

Setting up the config file in this way should enable METviewer to
generate
a ROC curve that consists of 10 points... once for threshold pair
listed
above.  I can tell you that we have not typically done this in the
past.

Does that make sense?

John

On Mon, Sep 10, 2018 at 1:55 PM Tressa Fowler via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>
> Hi Ying,
>
> It definitely makes sense to calculate AUC for CTC-derived ROC
curves, so
> long as a single observation threshold is selected, and the forecast
> thresholds fill in the curve pretty well. I think we can create an
issue to
> add this for future releases.
>
> Tressa
>
> On Sat Sep 08 16:37:02 2018, ying.lin at noaa.gov wrote:
> > Hi John and Tressa,
> >
> > I was originally under the impression that the ROC/Area under ROC
was
> > for probabilistic forecasts.  Do they make sense/are they worth
> > looking at in the non-probabilistic sense (e.g. ROC curve for
varying
> > forecast/obs thresholds)?  I guess a case can be made that the
CTC-ROC
> > makes some sense (similar to the performance diagram), but unlike
the
> > probabilistic ROC curve that rises with decreasing PCT, the "CTC
ROC"
> > based on the F/H/O contingency table counts for different
thresholds
> > (and probably for different forecast lead time as well) can be all
> > over the place, making the concept of "area under ROC”
problematic.
> >
> > If a non-prob ROC doesn't make enough sense/is not worth pursuing,
> > that's fine.  The AUC was listed as one of the measures in the CAM
> > metrics workshop, so I got curious when seeing the "CTC" option in
the
> > METviewer ROC.  That's all.
> >
> > Ying
> >
> > > On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
> > > <met_help at ucar.edu> wrote:
> > >
> > > Hi Tressa,
> > >
> > > I was supposed to talk to you about this issue earlier this week
but
> > > forgot.  I think we need your guidance on this one.
> > >
> > > This is a met-help ticket from Ying Lin at NCEP about ROC curves
in
> > > METviewer.  I believe there are 2 ways to create a ROC curve in
> > > METviewer:
> > >
> > > (1) The most common way I know is using the Nx2 Probabilistic
> > > Contingency
> > > Table counts from the PCT line in MET.  I understand this one.
MET
> > > can
> > > also write out the PSTD line type which includes the ROC_AUC
column.
> > > And
> > > that's a number between 0 and 1 indicating the area under the
ROC
> > > curve,
> > > with 1 being the best.  MET calculates this value and it gets
loaded
> > > into
> > > METviewer.  METviewer can plot the ROC_AUC value but it doesn't
> > > compute it.
> > >
> > > (2) The less common way of generating a ROC curve is using
several
> > > 2x2
> > > contingency table lines.  This would be the FHO or CTC line
types
> > > from
> > > MET.  However, there's some requirement in how those CTC lines
are
> > > defined... but I can't remember exactly what.  Perhaps they need
to
> > > be
> > > multiple forecast thresholds but all for the same observation
> > > threshold?
> > > Does that sound correct to you?  If we get all the right, then
> > > METviewer
> > > could make a ROC plot.  But, since METviewer doesn't compute the
area
> > > under
> > > the ROC curve, there's no way for METviewer to plot that
statistic.
> > >
> > > I think this would all be good info to clarify in the METviewer
> > > user's
> > > guide:
> > > - Show an example of using PCT output to make a ROC curve in
> > > METviewer.
> > > - Show an example of using CTC output to make a ROC curve in
> > > METviewer...
> > > and list the threshold requirements for doing so.
> > >
> > > And lastly, we should identify what changes to METviewer are
needed
> > > to make
> > > it more useful to Ying.  Perhaps we do need to enhance METviewer
to
> > > compute
> > > the area under the ROC curve for case (2)?
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
> > >>
> > >> Hi Tatiana,
> > >>
> > >> Follow up on this ... has the ROC plotting in METviewer been
used
> > >> in the non-probabilistic sense before?  I'm thinking that maybe
> > >> obs_thresh shouldn't be "fix values".  The ROC plot seems to be
> > >> analogous to the performance diagram, in which data points are
> > >> produced
> > >> for forecast lead times - but FCST_LEAD in the performance
diagrams
> > >> are
> > >> "independent variables" rather than fixed values.  There is no
> > >> "independent variables" option in ROC, the only way to make the
> > >> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed
values,
> > >> but
> > >> then it seems only two OBS_THRESH values can be accepted by
> > >> METviewer.
> > >>
> > >> Thanks -
> > >>
> > >> Ying
> > >>
> > >> On 08/27/2018 05:34 PM, Ying Lin wrote:
> > >>> Hi Tatiana,
> > >>>
> > >>> I tried making a ROC plot from MET *.stat (the case of the
> > >>> original question was from VSDBs loaded to MV) and the CTC
seems to
> > >>> work - see attached xml.
> > >>>
> > >>> Is the CTC option not available for ROC from VSDBs?
> > >>>
> > >>> For the ROC from the MET *.stat, I played with the OBS_THRESH
a
> > >>> bit, it appears that the ROC plot can accept up to two
different
> > >>> OBS_THRESH (as in the attached xml), but more than two will
result
> > >>> in
> > >>> an error.
> > >>>
> > >>> Thanks again,
> > >>>
> > >>> Ying
> > >>>
> > >>> On 08/27/2018 05:08 PM, Ying Lin wrote:
> > >>>> Thanks Tatiana.
> > >>>>
> > >>>> Sorry - AUR "Area Under ROC".
> > >>>>
> > >>>> I see your ROC plot and it seems the difference between your
xml
> > >>>> and my failed attempt is that you chose 'PCT' instead of
'CTC' for
> > >>>> ROC calculations.  I thought that PCT is for ROC score of
> > >>>> probability
> > >>>> forecasts, while CTC is from the contingency table count of
single
> > >>>> deterministic models' ROC?
> > >>>>
> > >>>> Ying
> > >>>>
> > >>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
> > >>>>> Hi Ying,
> > >>>>> ROC plop is fixed and I was able to create a plot using your
XML.
> > >>>>> What do you mean by AUR?
> > >>>>>
> > >>>>> Tatiana
> > >>>>>
> > >>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
> > >>>>>> Hello.  I'm trying to make a simple ROC plot.  The attached
> > >>>>>> *.xml does
> > >>>>>> not work - err msg
> > >>>>>>
> > >>>>>> "Failed to run plot plot_20180827_184319 - reason: R error:
====
> > >>>>>> Start
> > >>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
> > >>>>>> subscript
> > >>>>>> out of bounds In addition."
> > >>>>>>
> > >>>>>> I verified that data for the forecast valid time/lead time
(24h
> > >>>>>> ending
> > >>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.
What
> > >>>>>> should
> > >>>>>> I be doing differently in the set up?  I asked Ben Blake
about
> > >>>>>> this
> > >>>>>> (googling found his inquiries on ROC/AUR from early 2017)
and he
> > >>>>>> said to
> > >>>>>> put OBS_THRESH under series variables instead of fixed
values,
> > >>>>>> that
> > >>>>>> didn't work either.
> > >>>>>>
> > >>>>>> Also, once I get ROC to work ... where is AUR?  Is it under
Roc
> > >>>>>> tab, or
> > >>>>>> Series?  I can't figure out where it could be under Roc;
under
> > >>>>>> Series
> > >>>>>> there doesn't seem to be something that might be AUR under
> > >>>>>> "Select
> > >>>>>> attribute stat".
> > >>>>>>
> > >>>>>> Thanks!
> > >>>>>>
> > >>>>>> Ying
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >> --
> > >> Ying Lin
> > >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> > >> NCWCP Cubicle No. 2015
> > >> Ying.Lin at noaa.gov
> > >>
> > >>
> > >>
> > >>
> > >
>
>
>
>

------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Ying Lin
Time: Mon Sep 10 17:47:41 2018

Hi John (and Tressa),

     Thanks!  Yes, it's confirmed now that METviewer can make the
CTC-ROC plot.  I ran grid_stat with the 10 identical obs_thresh and 10
varying fcst_thresh and did get a ROC "curve" - see attached.  I'm
kind
of surprised that fcst_thresh numbers on the plot are in such perfect
order - expected some Brownian motion-like zig-zags rather than having
the fcst_thresh values in perfect descending order, just like in the
PCT-type ROC curves (is it possible that the connecting lines are
correct, but the point labels are wrong?  I don't quite know how to
interpret the R data).

     Anyway, interesting stuff (haven't thought of this way before)
but
probably of limited utility.  Please go ahead and close this ticket if
you wish.

Ying

On 09/10/2018 04:25 PM, John Halley Gotway via RT wrote:
> Ying,
>
> I believe that METviewer would be able to make this plot now
(although I
> haven't confirmed that recently).  But you would have to configure
MET in
> the right way to define the thresholds the way Tressa described.
>
> For example, let's say you're interested in 1/2 inch of precip
accumulation
> (i.e. 12.7 mm).  You define the thresholds in the MET config file
like the
> following.  I've listed the same obs threshold 10 times (>= 1/2
inch).  But
> I've listed 10 different forecast thresholds (e.g. >=0,
>=0.1",...,>=1.0").
>
> fcst = {
> ...
>    cat_thresh = [ ge0.00, ge2.54, ge5.08, ge7.62, ge10.16, ge12.7,
ge15.24,
> ge17.78, ge20.32, ge22.86, ge25.4 ];
> ...
> }
> obs = {
> ...
>    cat_thresh = [ ge12.7, ge12.7, ge12.7, ge12.7, ge12.7, ge12.7,
ge12.7,
> ge12.7, ge12.7, ge12.7 ];
> ...
> }
>
> Setting up the config file in this way should enable METviewer to
generate
> a ROC curve that consists of 10 points... once for threshold pair
listed
> above.  I can tell you that we have not typically done this in the
past.
>
> Does that make sense?
>
> John
>
> On Mon, Sep 10, 2018 at 1:55 PM Tressa Fowler via RT
<met_help at ucar.edu>
> wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>>
>> Hi Ying,
>>
>> It definitely makes sense to calculate AUC for CTC-derived ROC
curves, so
>> long as a single observation threshold is selected, and the
forecast
>> thresholds fill in the curve pretty well. I think we can create an
issue to
>> add this for future releases.
>>
>> Tressa
>>
>> On Sat Sep 08 16:37:02 2018, ying.lin at noaa.gov wrote:
>>> Hi John and Tressa,
>>>
>>> I was originally under the impression that the ROC/Area under ROC
was
>>> for probabilistic forecasts.  Do they make sense/are they worth
>>> looking at in the non-probabilistic sense (e.g. ROC curve for
varying
>>> forecast/obs thresholds)?  I guess a case can be made that the
CTC-ROC
>>> makes some sense (similar to the performance diagram), but unlike
the
>>> probabilistic ROC curve that rises with decreasing PCT, the "CTC
ROC"
>>> based on the F/H/O contingency table counts for different
thresholds
>>> (and probably for different forecast lead time as well) can be all
>>> over the place, making the concept of "area under ROC”
problematic.
>>>
>>> If a non-prob ROC doesn't make enough sense/is not worth pursuing,
>>> that's fine.  The AUC was listed as one of the measures in the CAM
>>> metrics workshop, so I got curious when seeing the "CTC" option in
the
>>> METviewer ROC.  That's all.
>>>
>>> Ying
>>>
>>>> On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
>>>> <met_help at ucar.edu> wrote:
>>>>
>>>> Hi Tressa,
>>>>
>>>> I was supposed to talk to you about this issue earlier this week
but
>>>> forgot.  I think we need your guidance on this one.
>>>>
>>>> This is a met-help ticket from Ying Lin at NCEP about ROC curves
in
>>>> METviewer.  I believe there are 2 ways to create a ROC curve in
>>>> METviewer:
>>>>
>>>> (1) The most common way I know is using the Nx2 Probabilistic
>>>> Contingency
>>>> Table counts from the PCT line in MET.  I understand this one.
MET
>>>> can
>>>> also write out the PSTD line type which includes the ROC_AUC
column.
>>>> And
>>>> that's a number between 0 and 1 indicating the area under the ROC
>>>> curve,
>>>> with 1 being the best.  MET calculates this value and it gets
loaded
>>>> into
>>>> METviewer.  METviewer can plot the ROC_AUC value but it doesn't
>>>> compute it.
>>>>
>>>> (2) The less common way of generating a ROC curve is using
several
>>>> 2x2
>>>> contingency table lines.  This would be the FHO or CTC line types
>>>> from
>>>> MET.  However, there's some requirement in how those CTC lines
are
>>>> defined... but I can't remember exactly what.  Perhaps they need
to
>>>> be
>>>> multiple forecast thresholds but all for the same observation
>>>> threshold?
>>>> Does that sound correct to you?  If we get all the right, then
>>>> METviewer
>>>> could make a ROC plot.  But, since METviewer doesn't compute the
area
>>>> under
>>>> the ROC curve, there's no way for METviewer to plot that
statistic.
>>>>
>>>> I think this would all be good info to clarify in the METviewer
>>>> user's
>>>> guide:
>>>> - Show an example of using PCT output to make a ROC curve in
>>>> METviewer.
>>>> - Show an example of using CTC output to make a ROC curve in
>>>> METviewer...
>>>> and list the threshold requirements for doing so.
>>>>
>>>> And lastly, we should identify what changes to METviewer are
needed
>>>> to make
>>>> it more useful to Ying.  Perhaps we do need to enhance METviewer
to
>>>> compute
>>>> the area under the ROC curve for case (2)?
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>>>>>
>>>>> Hi Tatiana,
>>>>>
>>>>> Follow up on this ... has the ROC plotting in METviewer been
used
>>>>> in the non-probabilistic sense before?  I'm thinking that maybe
>>>>> obs_thresh shouldn't be "fix values".  The ROC plot seems to be
>>>>> analogous to the performance diagram, in which data points are
>>>>> produced
>>>>> for forecast lead times - but FCST_LEAD in the performance
diagrams
>>>>> are
>>>>> "independent variables" rather than fixed values.  There is no
>>>>> "independent variables" option in ROC, the only way to make the
>>>>> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed
values,
>>>>> but
>>>>> then it seems only two OBS_THRESH values can be accepted by
>>>>> METviewer.
>>>>>
>>>>> Thanks -
>>>>>
>>>>> Ying
>>>>>
>>>>> On 08/27/2018 05:34 PM, Ying Lin wrote:
>>>>>> Hi Tatiana,
>>>>>>
>>>>>> I tried making a ROC plot from MET *.stat (the case of the
>>>>>> original question was from VSDBs loaded to MV) and the CTC
seems to
>>>>>> work - see attached xml.
>>>>>>
>>>>>> Is the CTC option not available for ROC from VSDBs?
>>>>>>
>>>>>> For the ROC from the MET *.stat, I played with the OBS_THRESH a
>>>>>> bit, it appears that the ROC plot can accept up to two
different
>>>>>> OBS_THRESH (as in the attached xml), but more than two will
result
>>>>>> in
>>>>>> an error.
>>>>>>
>>>>>> Thanks again,
>>>>>>
>>>>>> Ying
>>>>>>
>>>>>> On 08/27/2018 05:08 PM, Ying Lin wrote:
>>>>>>> Thanks Tatiana.
>>>>>>>
>>>>>>> Sorry - AUR "Area Under ROC".
>>>>>>>
>>>>>>> I see your ROC plot and it seems the difference between your
xml
>>>>>>> and my failed attempt is that you chose 'PCT' instead of 'CTC'
for
>>>>>>> ROC calculations.  I thought that PCT is for ROC score of
>>>>>>> probability
>>>>>>> forecasts, while CTC is from the contingency table count of
single
>>>>>>> deterministic models' ROC?
>>>>>>>
>>>>>>> Ying
>>>>>>>
>>>>>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
>>>>>>>> Hi Ying,
>>>>>>>> ROC plop is fixed and I was able to create a plot using your
XML.
>>>>>>>> What do you mean by AUR?
>>>>>>>>
>>>>>>>> Tatiana
>>>>>>>>
>>>>>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>>>>>>>>> Hello.  I'm trying to make a simple ROC plot.  The attached
>>>>>>>>> *.xml does
>>>>>>>>> not work - err msg
>>>>>>>>>
>>>>>>>>> "Failed to run plot plot_20180827_184319 - reason: R error:
====
>>>>>>>>> Start
>>>>>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
>>>>>>>>> subscript
>>>>>>>>> out of bounds In addition."
>>>>>>>>>
>>>>>>>>> I verified that data for the forecast valid time/lead time
(24h
>>>>>>>>> ending
>>>>>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.
What
>>>>>>>>> should
>>>>>>>>> I be doing differently in the set up?  I asked Ben Blake
about
>>>>>>>>> this
>>>>>>>>> (googling found his inquiries on ROC/AUR from early 2017)
and he
>>>>>>>>> said to
>>>>>>>>> put OBS_THRESH under series variables instead of fixed
values,
>>>>>>>>> that
>>>>>>>>> didn't work either.
>>>>>>>>>
>>>>>>>>> Also, once I get ROC to work ... where is AUR?  Is it under
Roc
>>>>>>>>> tab, or
>>>>>>>>> Series?  I can't figure out where it could be under Roc;
under
>>>>>>>>> Series
>>>>>>>>> there doesn't seem to be something that might be AUR under
>>>>>>>>> "Select
>>>>>>>>> attribute stat".
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Ying
>>>>>>>>>
>>>>>>>>
>>>>> --
>>>>> Ying Lin
>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>> NCWCP Cubicle No. 2015
>>>>> Ying.Lin at noaa.gov
>>>>>
>>>>>
>>>>>
>>>>>
>>
>>
>>

--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov



------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: John Halley Gotway
Time: Mon Sep 10 18:06:13 2018

Hmm, I do think there’s a problem in that plot in the order in which
the
dots are connected.  Probably need to ask Tatiana to generate that
plot in
the debugger and see what’s going on.

But Tatiana is out of town this week.  We’ll need to check with her
next
week.

Can you please send me the MET output file(s) you’ve generated for us
to
test?

Thanks
John

On Mon, Sep 10, 2018 at 5:48 PM Ying Lin via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>
> Hi John (and Tressa),
>
>      Thanks!  Yes, it's confirmed now that METviewer can make the
> CTC-ROC plot.  I ran grid_stat with the 10 identical obs_thresh and
10
> varying fcst_thresh and did get a ROC "curve" - see attached.  I'm
kind
> of surprised that fcst_thresh numbers on the plot are in such
perfect
> order - expected some Brownian motion-like zig-zags rather than
having
> the fcst_thresh values in perfect descending order, just like in the
> PCT-type ROC curves (is it possible that the connecting lines are
> correct, but the point labels are wrong?  I don't quite know how to
> interpret the R data).
>
>      Anyway, interesting stuff (haven't thought of this way before)
but
> probably of limited utility.  Please go ahead and close this ticket
if
> you wish.
>
> Ying
>
> On 09/10/2018 04:25 PM, John Halley Gotway via RT wrote:
> > Ying,
> >
> > I believe that METviewer would be able to make this plot now
(although I
> > haven't confirmed that recently).  But you would have to configure
MET in
> > the right way to define the thresholds the way Tressa described.
> >
> > For example, let's say you're interested in 1/2 inch of precip
> accumulation
> > (i.e. 12.7 mm).  You define the thresholds in the MET config file
like
> the
> > following.  I've listed the same obs threshold 10 times (>= 1/2
inch).
> But
> > I've listed 10 different forecast thresholds (e.g. >=0,
> >=0.1",...,>=1.0").
> >
> > fcst = {
> > ...
> >    cat_thresh = [ ge0.00, ge2.54, ge5.08, ge7.62, ge10.16, ge12.7,
> ge15.24,
> > ge17.78, ge20.32, ge22.86, ge25.4 ];
> > ...
> > }
> > obs = {
> > ...
> >    cat_thresh = [ ge12.7, ge12.7, ge12.7, ge12.7, ge12.7, ge12.7,
ge12.7,
> > ge12.7, ge12.7, ge12.7 ];
> > ...
> > }
> >
> > Setting up the config file in this way should enable METviewer to
> generate
> > a ROC curve that consists of 10 points... once for threshold pair
listed
> > above.  I can tell you that we have not typically done this in the
past.
> >
> > Does that make sense?
> >
> > John
> >
> > On Mon, Sep 10, 2018 at 1:55 PM Tressa Fowler via RT
<met_help at ucar.edu>
> > wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
> >>
> >> Hi Ying,
> >>
> >> It definitely makes sense to calculate AUC for CTC-derived ROC
curves,
> so
> >> long as a single observation threshold is selected, and the
forecast
> >> thresholds fill in the curve pretty well. I think we can create
an
> issue to
> >> add this for future releases.
> >>
> >> Tressa
> >>
> >> On Sat Sep 08 16:37:02 2018, ying.lin at noaa.gov wrote:
> >>> Hi John and Tressa,
> >>>
> >>> I was originally under the impression that the ROC/Area under
ROC was
> >>> for probabilistic forecasts.  Do they make sense/are they worth
> >>> looking at in the non-probabilistic sense (e.g. ROC curve for
varying
> >>> forecast/obs thresholds)?  I guess a case can be made that the
CTC-ROC
> >>> makes some sense (similar to the performance diagram), but
unlike the
> >>> probabilistic ROC curve that rises with decreasing PCT, the "CTC
ROC"
> >>> based on the F/H/O contingency table counts for different
thresholds
> >>> (and probably for different forecast lead time as well) can be
all
> >>> over the place, making the concept of "area under ROC”
problematic.
> >>>
> >>> If a non-prob ROC doesn't make enough sense/is not worth
pursuing,
> >>> that's fine.  The AUC was listed as one of the measures in the
CAM
> >>> metrics workshop, so I got curious when seeing the "CTC" option
in the
> >>> METviewer ROC.  That's all.
> >>>
> >>> Ying
> >>>
> >>>> On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
> >>>> <met_help at ucar.edu> wrote:
> >>>>
> >>>> Hi Tressa,
> >>>>
> >>>> I was supposed to talk to you about this issue earlier this
week but
> >>>> forgot.  I think we need your guidance on this one.
> >>>>
> >>>> This is a met-help ticket from Ying Lin at NCEP about ROC
curves in
> >>>> METviewer.  I believe there are 2 ways to create a ROC curve in
> >>>> METviewer:
> >>>>
> >>>> (1) The most common way I know is using the Nx2 Probabilistic
> >>>> Contingency
> >>>> Table counts from the PCT line in MET.  I understand this one.
MET
> >>>> can
> >>>> also write out the PSTD line type which includes the ROC_AUC
column.
> >>>> And
> >>>> that's a number between 0 and 1 indicating the area under the
ROC
> >>>> curve,
> >>>> with 1 being the best.  MET calculates this value and it gets
loaded
> >>>> into
> >>>> METviewer.  METviewer can plot the ROC_AUC value but it doesn't
> >>>> compute it.
> >>>>
> >>>> (2) The less common way of generating a ROC curve is using
several
> >>>> 2x2
> >>>> contingency table lines.  This would be the FHO or CTC line
types
> >>>> from
> >>>> MET.  However, there's some requirement in how those CTC lines
are
> >>>> defined... but I can't remember exactly what.  Perhaps they
need to
> >>>> be
> >>>> multiple forecast thresholds but all for the same observation
> >>>> threshold?
> >>>> Does that sound correct to you?  If we get all the right, then
> >>>> METviewer
> >>>> could make a ROC plot.  But, since METviewer doesn't compute
the area
> >>>> under
> >>>> the ROC curve, there's no way for METviewer to plot that
statistic.
> >>>>
> >>>> I think this would all be good info to clarify in the METviewer
> >>>> user's
> >>>> guide:
> >>>> - Show an example of using PCT output to make a ROC curve in
> >>>> METviewer.
> >>>> - Show an example of using CTC output to make a ROC curve in
> >>>> METviewer...
> >>>> and list the threshold requirements for doing so.
> >>>>
> >>>> And lastly, we should identify what changes to METviewer are
needed
> >>>> to make
> >>>> it more useful to Ying.  Perhaps we do need to enhance
METviewer to
> >>>> compute
> >>>> the area under the ROC curve for case (2)?
> >>>>
> >>>> Thanks,
> >>>> John
> >>>>
> >>>> On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT
<met_help at ucar.edu>
> >>>> wrote:
> >>>>
> >>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767
>
> >>>>>
> >>>>> Hi Tatiana,
> >>>>>
> >>>>> Follow up on this ... has the ROC plotting in METviewer been
used
> >>>>> in the non-probabilistic sense before?  I'm thinking that
maybe
> >>>>> obs_thresh shouldn't be "fix values".  The ROC plot seems to
be
> >>>>> analogous to the performance diagram, in which data points are
> >>>>> produced
> >>>>> for forecast lead times - but FCST_LEAD in the performance
diagrams
> >>>>> are
> >>>>> "independent variables" rather than fixed values.  There is no
> >>>>> "independent variables" option in ROC, the only way to make
the
> >>>>> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed
values,
> >>>>> but
> >>>>> then it seems only two OBS_THRESH values can be accepted by
> >>>>> METviewer.
> >>>>>
> >>>>> Thanks -
> >>>>>
> >>>>> Ying
> >>>>>
> >>>>> On 08/27/2018 05:34 PM, Ying Lin wrote:
> >>>>>> Hi Tatiana,
> >>>>>>
> >>>>>> I tried making a ROC plot from MET *.stat (the case of the
> >>>>>> original question was from VSDBs loaded to MV) and the CTC
seems to
> >>>>>> work - see attached xml.
> >>>>>>
> >>>>>> Is the CTC option not available for ROC from VSDBs?
> >>>>>>
> >>>>>> For the ROC from the MET *.stat, I played with the OBS_THRESH
a
> >>>>>> bit, it appears that the ROC plot can accept up to two
different
> >>>>>> OBS_THRESH (as in the attached xml), but more than two will
result
> >>>>>> in
> >>>>>> an error.
> >>>>>>
> >>>>>> Thanks again,
> >>>>>>
> >>>>>> Ying
> >>>>>>
> >>>>>> On 08/27/2018 05:08 PM, Ying Lin wrote:
> >>>>>>> Thanks Tatiana.
> >>>>>>>
> >>>>>>> Sorry - AUR "Area Under ROC".
> >>>>>>>
> >>>>>>> I see your ROC plot and it seems the difference between your
xml
> >>>>>>> and my failed attempt is that you chose 'PCT' instead of
'CTC' for
> >>>>>>> ROC calculations.  I thought that PCT is for ROC score of
> >>>>>>> probability
> >>>>>>> forecasts, while CTC is from the contingency table count of
single
> >>>>>>> deterministic models' ROC?
> >>>>>>>
> >>>>>>> Ying
> >>>>>>>
> >>>>>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
> >>>>>>>> Hi Ying,
> >>>>>>>> ROC plop is fixed and I was able to create a plot using
your XML.
> >>>>>>>> What do you mean by AUR?
> >>>>>>>>
> >>>>>>>> Tatiana
> >>>>>>>>
> >>>>>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
> >>>>>>>>> Hello.  I'm trying to make a simple ROC plot.  The
attached
> >>>>>>>>> *.xml does
> >>>>>>>>> not work - err msg
> >>>>>>>>>
> >>>>>>>>> "Failed to run plot plot_20180827_184319 - reason: R
error: ====
> >>>>>>>>> Start
> >>>>>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact)
:
> >>>>>>>>> subscript
> >>>>>>>>> out of bounds In addition."
> >>>>>>>>>
> >>>>>>>>> I verified that data for the forecast valid time/lead time
(24h
> >>>>>>>>> ending
> >>>>>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from
them. What
> >>>>>>>>> should
> >>>>>>>>> I be doing differently in the set up?  I asked Ben Blake
about
> >>>>>>>>> this
> >>>>>>>>> (googling found his inquiries on ROC/AUR from early 2017)
and he
> >>>>>>>>> said to
> >>>>>>>>> put OBS_THRESH under series variables instead of fixed
values,
> >>>>>>>>> that
> >>>>>>>>> didn't work either.
> >>>>>>>>>
> >>>>>>>>> Also, once I get ROC to work ... where is AUR?  Is it
under Roc
> >>>>>>>>> tab, or
> >>>>>>>>> Series?  I can't figure out where it could be under Roc;
under
> >>>>>>>>> Series
> >>>>>>>>> there doesn't seem to be something that might be AUR under
> >>>>>>>>> "Select
> >>>>>>>>> attribute stat".
> >>>>>>>>>
> >>>>>>>>> Thanks!
> >>>>>>>>>
> >>>>>>>>> Ying
> >>>>>>>>>
> >>>>>>>>
> >>>>> --
> >>>>> Ying Lin
> >>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>>> NCWCP Cubicle No. 2015
> >>>>> Ying.Lin at noaa.gov
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>

------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Ying Lin
Time: Mon Sep 10 18:53:14 2018

Hi John, attached is the *.stat file I loaded to METviewer to generate
the ROC 'curve'.  Thanks!
Ying

On 09/10/2018 08:07 PM, John Halley Gotway via RT wrote:
> Hmm, I do think there’s a problem in that plot in the order in which
the
> dots are connected.  Probably need to ask Tatiana to generate that
plot in
> the debugger and see what’s going on.
>
> But Tatiana is out of town this week.  We’ll need to check with her
next
> week.
>
> Can you please send me the MET output file(s) you’ve generated for
us to
> test?
>
> Thanks
> John
>
> On Mon, Sep 10, 2018 at 5:48 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>>
>> Hi John (and Tressa),
>>
>>       Thanks!  Yes, it's confirmed now that METviewer can make the
>> CTC-ROC plot.  I ran grid_stat with the 10 identical obs_thresh and
10
>> varying fcst_thresh and did get a ROC "curve" - see attached.  I'm
kind
>> of surprised that fcst_thresh numbers on the plot are in such
perfect
>> order - expected some Brownian motion-like zig-zags rather than
having
>> the fcst_thresh values in perfect descending order, just like in
the
>> PCT-type ROC curves (is it possible that the connecting lines are
>> correct, but the point labels are wrong?  I don't quite know how to
>> interpret the R data).
>>
>>       Anyway, interesting stuff (haven't thought of this way
before) but
>> probably of limited utility.  Please go ahead and close this ticket
if
>> you wish.
>>
>> Ying
>>
>> On 09/10/2018 04:25 PM, John Halley Gotway via RT wrote:
>>> Ying,
>>>
>>> I believe that METviewer would be able to make this plot now
(although I
>>> haven't confirmed that recently).  But you would have to configure
MET in
>>> the right way to define the thresholds the way Tressa described.
>>>
>>> For example, let's say you're interested in 1/2 inch of precip
>> accumulation
>>> (i.e. 12.7 mm).  You define the thresholds in the MET config file
like
>> the
>>> following.  I've listed the same obs threshold 10 times (>= 1/2
inch).
>> But
>>> I've listed 10 different forecast thresholds (e.g. >=0,
>>> =0.1",...,>=1.0").
>>>
>>> fcst = {
>>> ...
>>>     cat_thresh = [ ge0.00, ge2.54, ge5.08, ge7.62, ge10.16,
ge12.7,
>> ge15.24,
>>> ge17.78, ge20.32, ge22.86, ge25.4 ];
>>> ...
>>> }
>>> obs = {
>>> ...
>>>     cat_thresh = [ ge12.7, ge12.7, ge12.7, ge12.7, ge12.7, ge12.7,
ge12.7,
>>> ge12.7, ge12.7, ge12.7 ];
>>> ...
>>> }
>>>
>>> Setting up the config file in this way should enable METviewer to
>> generate
>>> a ROC curve that consists of 10 points... once for threshold pair
listed
>>> above.  I can tell you that we have not typically done this in the
past.
>>>
>>> Does that make sense?
>>>
>>> John
>>>
>>> On Mon, Sep 10, 2018 at 1:55 PM Tressa Fowler via RT
<met_help at ucar.edu>
>>> wrote:
>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>>>>
>>>> Hi Ying,
>>>>
>>>> It definitely makes sense to calculate AUC for CTC-derived ROC
curves,
>> so
>>>> long as a single observation threshold is selected, and the
forecast
>>>> thresholds fill in the curve pretty well. I think we can create
an
>> issue to
>>>> add this for future releases.
>>>>
>>>> Tressa
>>>>
>>>> On Sat Sep 08 16:37:02 2018, ying.lin at noaa.gov wrote:
>>>>> Hi John and Tressa,
>>>>>
>>>>> I was originally under the impression that the ROC/Area under
ROC was
>>>>> for probabilistic forecasts.  Do they make sense/are they worth
>>>>> looking at in the non-probabilistic sense (e.g. ROC curve for
varying
>>>>> forecast/obs thresholds)?  I guess a case can be made that the
CTC-ROC
>>>>> makes some sense (similar to the performance diagram), but
unlike the
>>>>> probabilistic ROC curve that rises with decreasing PCT, the "CTC
ROC"
>>>>> based on the F/H/O contingency table counts for different
thresholds
>>>>> (and probably for different forecast lead time as well) can be
all
>>>>> over the place, making the concept of "area under ROC”
problematic.
>>>>>
>>>>> If a non-prob ROC doesn't make enough sense/is not worth
pursuing,
>>>>> that's fine.  The AUC was listed as one of the measures in the
CAM
>>>>> metrics workshop, so I got curious when seeing the "CTC" option
in the
>>>>> METviewer ROC.  That's all.
>>>>>
>>>>> Ying
>>>>>
>>>>>> On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
>>>>>> <met_help at ucar.edu> wrote:
>>>>>>
>>>>>> Hi Tressa,
>>>>>>
>>>>>> I was supposed to talk to you about this issue earlier this
week but
>>>>>> forgot.  I think we need your guidance on this one.
>>>>>>
>>>>>> This is a met-help ticket from Ying Lin at NCEP about ROC
curves in
>>>>>> METviewer.  I believe there are 2 ways to create a ROC curve in
>>>>>> METviewer:
>>>>>>
>>>>>> (1) The most common way I know is using the Nx2 Probabilistic
>>>>>> Contingency
>>>>>> Table counts from the PCT line in MET.  I understand this one.
MET
>>>>>> can
>>>>>> also write out the PSTD line type which includes the ROC_AUC
column.
>>>>>> And
>>>>>> that's a number between 0 and 1 indicating the area under the
ROC
>>>>>> curve,
>>>>>> with 1 being the best.  MET calculates this value and it gets
loaded
>>>>>> into
>>>>>> METviewer.  METviewer can plot the ROC_AUC value but it doesn't
>>>>>> compute it.
>>>>>>
>>>>>> (2) The less common way of generating a ROC curve is using
several
>>>>>> 2x2
>>>>>> contingency table lines.  This would be the FHO or CTC line
types
>>>>>> from
>>>>>> MET.  However, there's some requirement in how those CTC lines
are
>>>>>> defined... but I can't remember exactly what.  Perhaps they
need to
>>>>>> be
>>>>>> multiple forecast thresholds but all for the same observation
>>>>>> threshold?
>>>>>> Does that sound correct to you?  If we get all the right, then
>>>>>> METviewer
>>>>>> could make a ROC plot.  But, since METviewer doesn't compute
the area
>>>>>> under
>>>>>> the ROC curve, there's no way for METviewer to plot that
statistic.
>>>>>>
>>>>>> I think this would all be good info to clarify in the METviewer
>>>>>> user's
>>>>>> guide:
>>>>>> - Show an example of using PCT output to make a ROC curve in
>>>>>> METviewer.
>>>>>> - Show an example of using CTC output to make a ROC curve in
>>>>>> METviewer...
>>>>>> and list the threshold requirements for doing so.
>>>>>>
>>>>>> And lastly, we should identify what changes to METviewer are
needed
>>>>>> to make
>>>>>> it more useful to Ying.  Perhaps we do need to enhance
METviewer to
>>>>>> compute
>>>>>> the area under the ROC curve for case (2)?
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>> On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT
<met_help at ucar.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767
>
>>>>>>>
>>>>>>> Hi Tatiana,
>>>>>>>
>>>>>>> Follow up on this ... has the ROC plotting in METviewer been
used
>>>>>>> in the non-probabilistic sense before?  I'm thinking that
maybe
>>>>>>> obs_thresh shouldn't be "fix values".  The ROC plot seems to
be
>>>>>>> analogous to the performance diagram, in which data points are
>>>>>>> produced
>>>>>>> for forecast lead times - but FCST_LEAD in the performance
diagrams
>>>>>>> are
>>>>>>> "independent variables" rather than fixed values.  There is no
>>>>>>> "independent variables" option in ROC, the only way to make
the
>>>>>>> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed
values,
>>>>>>> but
>>>>>>> then it seems only two OBS_THRESH values can be accepted by
>>>>>>> METviewer.
>>>>>>>
>>>>>>> Thanks -
>>>>>>>
>>>>>>> Ying
>>>>>>>
>>>>>>> On 08/27/2018 05:34 PM, Ying Lin wrote:
>>>>>>>> Hi Tatiana,
>>>>>>>>
>>>>>>>> I tried making a ROC plot from MET *.stat (the case of the
>>>>>>>> original question was from VSDBs loaded to MV) and the CTC
seems to
>>>>>>>> work - see attached xml.
>>>>>>>>
>>>>>>>> Is the CTC option not available for ROC from VSDBs?
>>>>>>>>
>>>>>>>> For the ROC from the MET *.stat, I played with the OBS_THRESH
a
>>>>>>>> bit, it appears that the ROC plot can accept up to two
different
>>>>>>>> OBS_THRESH (as in the attached xml), but more than two will
result
>>>>>>>> in
>>>>>>>> an error.
>>>>>>>>
>>>>>>>> Thanks again,
>>>>>>>>
>>>>>>>> Ying
>>>>>>>>
>>>>>>>> On 08/27/2018 05:08 PM, Ying Lin wrote:
>>>>>>>>> Thanks Tatiana.
>>>>>>>>>
>>>>>>>>> Sorry - AUR "Area Under ROC".
>>>>>>>>>
>>>>>>>>> I see your ROC plot and it seems the difference between your
xml
>>>>>>>>> and my failed attempt is that you chose 'PCT' instead of
'CTC' for
>>>>>>>>> ROC calculations.  I thought that PCT is for ROC score of
>>>>>>>>> probability
>>>>>>>>> forecasts, while CTC is from the contingency table count of
single
>>>>>>>>> deterministic models' ROC?
>>>>>>>>>
>>>>>>>>> Ying
>>>>>>>>>
>>>>>>>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
>>>>>>>>>> Hi Ying,
>>>>>>>>>> ROC plop is fixed and I was able to create a plot using
your XML.
>>>>>>>>>> What do you mean by AUR?
>>>>>>>>>>
>>>>>>>>>> Tatiana
>>>>>>>>>>
>>>>>>>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>>>>>>>>>>> Hello.  I'm trying to make a simple ROC plot.  The
attached
>>>>>>>>>>> *.xml does
>>>>>>>>>>> not work - err msg
>>>>>>>>>>>
>>>>>>>>>>> "Failed to run plot plot_20180827_184319 - reason: R
error: ====
>>>>>>>>>>> Start
>>>>>>>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact)
:
>>>>>>>>>>> subscript
>>>>>>>>>>> out of bounds In addition."
>>>>>>>>>>>
>>>>>>>>>>> I verified that data for the forecast valid time/lead time
(24h
>>>>>>>>>>> ending
>>>>>>>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from
them. What
>>>>>>>>>>> should
>>>>>>>>>>> I be doing differently in the set up?  I asked Ben Blake
about
>>>>>>>>>>> this
>>>>>>>>>>> (googling found his inquiries on ROC/AUR from early 2017)
and he
>>>>>>>>>>> said to
>>>>>>>>>>> put OBS_THRESH under series variables instead of fixed
values,
>>>>>>>>>>> that
>>>>>>>>>>> didn't work either.
>>>>>>>>>>>
>>>>>>>>>>> Also, once I get ROC to work ... where is AUR?  Is it
under Roc
>>>>>>>>>>> tab, or
>>>>>>>>>>> Series?  I can't figure out where it could be under Roc;
under
>>>>>>>>>>> Series
>>>>>>>>>>> there doesn't seem to be something that might be AUR under
>>>>>>>>>>> "Select
>>>>>>>>>>> attribute stat".
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> Ying
>>>>>>>>>>>
>>>>>>> --
>>>>>>> Ying Lin
>>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>>> NCWCP Cubicle No. 2015
>>>>>>> Ying.Lin at noaa.gov
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>

--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov



------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Ying Lin
Time: Mon Sep 10 18:53:14 2018

VERSION MODEL         DESC FCST_LEAD FCST_VALID_BEG  FCST_VALID_END
OBS_LEAD OBS_VALID_BEG   OBS_VALID_END   FCST_VAR FCST_LEV OBS_VAR
OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >0.254      >12.700    NA
NA    FHO       70390    0.32962        0.040958      0.052479
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >0.254      >12.700    NA
NA    CTC       70390 2883          20319           811        46377
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >0.254      >12.700    NA
0.01  CTS       70390    0.052479       0.050356      0.054686    NA
NA          0.32962    0.32507    0.3342    NA         NA    0.69982
0.69535  0.70425    NA   NA          6.281    NA         NA    0.78045
0.77641    0.78445   NA   NA         0.69535  0.69086  0.6998  NA
NA     0.30465    0.3002       0.30914   NA     NA        0.87574
0.87251  0.87891  NA        NA     0.12006   0.11694      0.12325   NA
NA 0.073058  NA NA 0.4758    0.45779    0.49382   NA NA 0.13617   NA
NA  8.1138  7.30759  9.00895 NA NA 2.09357 1.98891 2.19822 NA NA
0.78055 0.76011 0.801   NA NA  0.84484   0.83187   0.85782  NA NA
0.26976 0.26082  0.27869 NA NA 0.65488 0.60978  0.69997 NA NA 0.63135
0.60883  0.70093 NA NA 0.08347  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >0.254      >12.700    NA
NA    ECLV      70390    0.052479       0.4758       19
0.05     0.46432    0.1        0.16928    0.15      -0.19023    0.2
-0.59468   0.25    -1.05306     0.3 -1.57692    0.35     -2.18138
0.4 -2.88657   0.45      -3.71999    0.5 -4.72009   0.55    -5.94243
0.6     -7.47036   0.65 -9.43484    0.7        -12.05414    0.75
-15.72117  0.8     -21.22171  0.85    -30.38928   0.9 -48.72442   0.95
-103.72983
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >2.540      >12.700    NA
NA    FHO       70390    0.19784        0.031539      0.052479
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >2.540      >12.700    NA
NA    CTC       70390 2220          11706          1474        54990
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >2.540      >12.700    NA
0.01  CTS       70390    0.052479       0.050356      0.054686    NA
NA          0.19784    0.194      0.20174   NA         NA    0.81276
0.80894  0.81652    NA   NA          3.7699   NA         NA    0.60097
0.59621    0.60572   NA   NA         0.82449  0.82076  0.82815 NA
NA     0.17551    0.17185      0.17924   NA     NA        0.84059
0.837    0.84411  NA        NA     0.14416   0.14078      0.1476    NA
NA 0.10152   NA NA 0.42546   0.40416    0.44677   NA NA 0.18432   NA
NA  7.07507 6.46335  7.74468 NA NA 1.95658 1.86615 2.04701 NA NA
0.75232 0.7327  0.77195 NA NA  0.70537   0.68833   0.72241  NA NA
0.32145 0.30824  0.33465 NA NA 0.54722 0.513    0.58145 NA NA 0.58215
0.51282  0.58162 NA NA 0.088083 NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >2.540      >12.700    NA
NA    ECLV      70390    0.052479       0.42546      19
0.05     0.40458    0.1        0.24887    0.15       0.041753   0.2
-0.19126   0.25    -0.45533     0.3 -0.75714    0.35     -1.10537
0.4 -1.51164   0.45      -1.99178    0.5 -2.56795   0.55    -3.27215
0.6     -4.15241   0.65 -5.28417    0.7         -6.79318    0.75
-8.90579  0.8     -12.07472  0.85    -17.35625   0.9 -27.91933   0.95
-59.60855
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >3.810      >12.700    NA
NA    FHO       70390    0.17254        0.028782      0.052479
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >3.810      >12.700    NA
NA    CTC       70390 2026          10119          1668        56577
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >3.810      >12.700    NA
0.01  CTS       70390    0.052479       0.050356      0.054686    NA
NA          0.17254    0.1689     0.17624   NA         NA    0.83255
0.82889  0.83614    NA   NA          3.28776  NA         NA    0.54846
0.54362    0.55328   NA   NA         0.84828  0.84477  0.85173 NA
NA     0.15172    0.14827      0.15523   NA     NA        0.83318
0.82953  0.83677  NA        NA     0.14667   0.14327      0.15014   NA
NA 0.10539   NA NA 0.39674   0.37509    0.41839   NA NA 0.19069   NA
NA  6.79119 6.20925  7.42766 NA NA 1.91563 1.82604 2.00521 NA NA
0.7433  0.72325 0.76334 NA NA  0.66142   0.64341   0.67942  NA NA
0.32596 0.31158  0.34033 NA NA 0.51685 0.48431  0.54939 NA NA 0.5559
0.48368  0.55002 NA NA 0.088189 NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >3.810      >12.700    NA
NA    ECLV      70390    0.052479       0.39674      19
0.05     0.37311    0.1        0.24409    0.15       0.06505    0.2
-0.13637   0.25    -0.36465     0.3 -0.62553    0.35     -0.92655
0.4 -1.27775   0.45      -1.69279    0.5 -2.19085   0.55    -2.79958
0.6     -3.5605    0.65 -4.53883    0.7         -5.84326    0.75
-7.66946  0.8     -10.40877  0.85    -14.97428   0.9 -24.10531   0.95
-51.49838
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >6.350      >12.700    NA
NA    FHO       70390    0.13729        0.024648      0.052479
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >6.350      >12.700    NA
NA    CTC       70390 1735           7929          1959        58767
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >6.350      >12.700    NA
0.01  CTS       70390    0.052479       0.050356      0.054686    NA
NA          0.13729    0.13399    0.14067   NA         NA    0.85953
0.85612  0.86287    NA   NA          2.61613  NA         NA    0.46968
0.46484    0.47453   NA   NA         0.88112  0.87794  0.88422 NA
NA     0.11888    0.11578      0.12206   NA     NA        0.82047
0.81671  0.82416  NA        NA     0.14927   0.14585      0.15277   NA
NA 0.11046   NA NA 0.3508    0.32909    0.37251   NA NA 0.19894   NA
NA  6.56417 5.99718  7.18477 NA NA 1.88163 1.79129 1.97196 NA NA
0.7356  0.71487 0.75632 NA NA  0.59185   0.57249   0.61121  NA NA
0.33214 0.31594  0.34834 NA NA 0.47617 0.44589  0.50646 NA NA 0.51606
0.44462  0.50772 NA NA 0.089864 NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >6.350      >12.700    NA
NA    ECLV      70390    0.052479       0.3508       19
0.05     0.32305    0.1        0.23119    0.15       0.090895   0.2
-0.066933  0.25    -0.2458      0.3 -0.45023    0.35     -0.6861
0.4 -0.96129   0.45      -1.28651    0.5 -1.67677   0.55    -2.15376
0.6     -2.75      0.65 -3.51659    0.7         -4.53871    0.75
-5.96968  0.8      -8.11613  0.85    -11.69356   0.9 -18.8484    0.95
-40.31294
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >12.700     >12.700    NA
NA    FHO       70390    0.082952       0.017005      0.052479
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >12.700     >12.700    NA
NA    CTC       70390 1197           4642          2497        62054
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >12.700     >12.700    NA
0.01  CTS       70390    0.052479       0.050356      0.054686    NA
NA          0.082952   0.080314   0.085669  NA         NA    0.89858
0.89561  0.90147    NA   NA          1.58067  NA         NA    0.32404
0.31951    0.3286    NA   NA         0.9304   0.92789  0.93283 NA
NA     0.069599   0.067169     0.072111  NA     NA        0.795
0.79105  0.79889  NA        NA     0.14359   0.14022      0.14703   NA
NA 0.11091   NA NA 0.25444   0.23408    0.2748    NA NA 0.19968   NA
NA  6.40826 5.80614  7.07283 NA NA 1.85759 1.75892 1.95626 NA NA
0.73003 0.70699 0.75307 NA NA  0.44682   0.42508   0.46856  NA NA
0.33444 0.31439  0.35449 NA NA 0.40563 0.37834  0.43292 NA NA 0.4365
0.37646  0.4348  NA NA 0.09495  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >12.700     >12.700    NA
NA    ECLV      70390    0.052479       0.25444      19
0.05     0.21907    0.1        0.18441    0.15       0.10228    0.2
0.0098809 0.25    -0.094838    0.3 -0.21452    0.35     -0.35261
0.4 -0.51372   0.45      -0.70411    0.5 -0.93259   0.55    -1.21185
0.6     -1.56091   0.65 -2.00971    0.7         -2.6081     0.75
-3.44586  0.8      -4.70249  0.85     -6.79688   0.9 -10.98565   0.95
-23.55198
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >19.05      >12.700    NA
NA    FHO       70390    0.050646       0.011593      0.052479
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >19.05      >12.700    NA
NA    CTC       70390  816           2749          2878        63947
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >19.05      >12.700    NA
0.01  CTS       70390    0.052479       0.050356      0.054686    NA
NA          0.050646   0.04856    0.052818  NA         NA    0.92006
0.91739  0.92265    NA   NA          0.96508  NA         NA    0.2209
0.2169     0.22495   NA   NA         0.95878  0.95681  0.96067 NA
NA     0.041217   0.03933      0.043191  NA     NA        0.77111
0.767    0.77516  NA        NA     0.12665   0.12346      0.12991   NA
NA 0.10053   NA NA 0.17968   0.16163    0.19774   NA NA 0.1827    NA
NA  6.59546 5.88594  7.3905  NA NA 1.88638 1.77257 2.0002  NA NA
0.73668 0.71066 0.76271 NA NA  0.32245   0.29884   0.34606  NA NA
0.33043 0.30667  0.35418 NA NA 0.35728 0.33121  0.38336 NA NA 0.37798
0.32941  0.38515 NA NA 0.10227  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >19.05      >12.700    NA
NA    ECLV      70390    0.052479       0.17968      19
0.05     0.13891    0.1        0.13821    0.15       0.089573   0.2
0.034854  0.25    -0.027161    0.3 -0.098035   0.35     -0.17981
0.4 -0.27522   0.45      -0.38798    0.5 -0.52328   0.55    -0.68865
0.6     -0.89537   0.65 -1.16115    0.7         -1.51552    0.75
-2.01164  0.8      -2.75582  0.85     -3.99612   0.9  -6.47672   0.95
-13.91852
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >25.400     >12.700    NA
NA    FHO       70390    0.030913       0.0075153     0.052479
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >25.400     >12.700    NA
NA    CTC       70390  529           1647          3165        65049
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >25.400     >12.700    NA
0.01  CTS       70390    0.052479       0.050356      0.054686    NA
NA          0.030913   0.029277   0.032639  NA         NA    0.93164
0.92915  0.93405    NA   NA          0.58906  NA         NA    0.14321
0.13984    0.14664   NA   NA         0.97531  0.97375  0.97677 NA
NA     0.024694   0.023232     0.026246  NA     NA        0.75689
0.7527   0.76103  NA        NA     0.099045  0.096183     0.10198   NA
NA 0.079361  NA NA 0.11851   0.10326    0.13376   NA NA 0.14705   NA
NA  6.60129 5.75613  7.57055 NA NA 1.88727 1.75026 2.02427 NA NA
0.73689 0.70558 0.76819 NA NA  0.20525   0.17971   0.2308   NA NA
0.31346 0.28562  0.3413  NA NA 0.31139 0.2852   0.33759 NA NA 0.32404
0.28371  0.33908 NA NA 0.10565  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >25.400     >12.700    NA
NA    ECLV      70390    0.052479       0.11851      19
0.05     0.073678   0.1        0.093665   0.15       0.064524   0.2
0.031741  0.25    -0.0054142   0.3 -0.047877   0.35     -0.096872
0.4 -0.15403   0.45      -0.22159    0.5 -0.30265   0.55    -0.40173
0.6     -0.52558   0.65 -0.68482    0.7         -0.89713    0.75
-1.19437  0.8      -1.64023  0.85     -2.38332   0.9  -3.86952   0.95
-8.3281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >38.1       >12.700    NA
NA    FHO       70390    0.011607       0.0028413     0.052479
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >38.1       >12.700    NA
NA    CTC       70390  200            617          3494        66079
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >38.1       >12.700    NA
0.01  CTS       70390    0.052479       0.050356      0.054686    NA
NA          0.011607   0.010612   0.012694  NA         NA    0.9416
0.93928  0.94383    NA   NA          0.22117  NA         NA
0.054142  0.051987   0.056381  NA   NA         0.99075  0.98977
0.99163 NA        NA     0.0092509  0.0083666    0.010228  NA     NA
0.7552    0.751    0.75935  NA        NA     0.046393  0.044393
0.048478  NA NA 0.036814  NA NA 0.044891  0.035014   0.054768  NA NA
0.071013  NA NA  6.13035 4.94781  7.59552 NA NA 1.81325 1.59895
2.02756 NA NA 0.71951 0.66783 0.77119 NA NA  0.00532  -0.025051
0.035691 NA NA 0.26265 0.2245   0.30079 NA NA 0.23251 0.20276  0.26226
NA NA 0.23659 0.20189  0.26313 NA NA 0.10161  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >38.1       >12.700    NA
NA    ECLV      70390    0.052479       0.044891     19
0.05    -0.004603   0.1        0.035583   0.15       0.024666   0.2
0.012385  0.25    -0.001534    0.3 -0.017441   0.35     -0.035796
0.4 -0.05721   0.45      -0.082517   0.5 -0.11289   0.55    -0.15
0.6     -0.1964    0.65 -0.25605    0.7         -0.33559    0.75
-0.44694  0.8      -0.61397  0.85     -0.89235   0.9  -1.44911   0.95
-3.11938
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >50.8       >12.700    NA
NA    FHO       70390    0.0049013      0.0011223     0.052479
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >50.8       >12.700    NA
NA    CTC       70390   79            266          3615        66430
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >50.8       >12.700    NA
0.01  CTS       70390    0.052479       0.050356      0.054686    NA
NA          0.0049013  0.0042683  0.0056275 NA         NA    0.94486
0.94261  0.94704    NA   NA          0.093395 NA         NA
0.021386  0.020026   0.022836  NA   NA         0.99601  0.99535
0.99658 NA        NA     0.0039882  0.0034213    0.0046487 NA     NA
0.77101   0.76691  0.77507  NA        NA     0.019949  0.018637
0.021353  NA NA 0.015448  NA NA 0.017398  0.011046   0.023749  NA NA
0.030426  NA NA  5.4576  3.91199  7.61387 NA NA 1.69701 1.36405
2.02997 NA NA 0.69029 0.60313 0.77744 NA NA -0.13216  -0.16879
-0.09553  NA NA 0.2169  0.16554  0.26826 NA NA 0.17924 0.14261
0.21587 NA NA 0.18063 0.14212  0.21636 NA NA 0.09136  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >50.8       >12.700    NA
NA    ECLV      70390    0.052479       0.017398     19
0.05    -0.03381    0.1        0.013385   0.15       0.0086786  0.2
0.0033839 0.25    -0.0026169   0.3 -0.0094748  0.35     -0.017388
0.4 -0.02662   0.45      -0.03753    0.5 -0.050623  0.55    -0.066625
0.6     -0.086627  0.65 -0.11234    0.7         -0.14663    0.75
-0.19464  0.8      -0.26665  0.85     -0.38666   0.9  -0.62669   0.95
-1.34678
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >76.2       >12.700    NA
NA    FHO       70390    0.00098025     0.00026992    0.052479
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >76.2       >12.700    NA
NA    CTC       70390   19             50          3675        66646
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >76.2       >12.700    NA
0.01  CTS       70390    0.052479       0.050356      0.054686    NA
NA          0.00098025 0.00071986 0.0013347 NA         NA    0.94708
0.94486  0.94921    NA   NA          0.018679 NA         NA
0.0051435 0.0044941  0.0058861 NA   NA         0.99925  0.99893
0.99947 NA        NA     0.00074967 0.00052688   0.0010666 NA     NA
0.72464   0.72028  0.72895  NA        NA     0.0050748 0.00443
0.0058128 NA NA 0.0041116 NA NA 0.0043938 0.0011576  0.00763   NA NA
0.0081895 NA NA  6.89129 3.43733 13.81591 NA NA 1.93026 1.23469
2.62582 NA NA 0.74656 0.59261 0.9005  NA NA -0.28266  -0.33411
-0.2312   NA NA 0.20173 0.11553  0.28793 NA NA 0.15449 0.099696
0.20928 NA NA 0.15477 0.099502 0.20948 NA NA 0.11573  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           >76.2       >12.700    NA
NA    ECLV      70390    0.052479       0.0043938    19
0.05    -0.047664   0.1        0.0036395  0.15       0.0027549  0.2
0.0017596 0.25     0.00063165  0.3 -0.00065744 0.35     -0.0021449
0.4 -0.0038802 0.45      -0.005931   0.5 -0.008392  0.55    -0.0114
0.6     -0.01516   0.65 -0.019994   0.7         -0.026439   0.75
-0.035463 0.8      -0.048998 0.85     -0.071557  0.9  -0.11668   0.95
-0.25203
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >0.254      >12.700    NA
NA    FHO       59496    0.31395        0.045801      0.057281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >0.254      >12.700    NA
NA    CTC       59496 2725          15954           683        40134
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >0.254      >12.700    NA
0.01  CTS       59496    0.057281       0.054876      0.059785    NA
NA          0.31395    0.30907    0.31888   NA         NA    0.72037
0.7156   0.72508    NA   NA          5.48093  NA         NA    0.79959
0.79533    0.80378   NA   NA         0.71555  0.71077  0.72029 NA
NA     0.28445    0.27971      0.28923   NA     NA        0.85411
0.85035  0.8578   NA        NA     0.14074   0.13711      0.14445   NA
NA 0.090479  NA NA 0.51514   0.49696    0.53333   NA NA 0.16594   NA
NA 10.03665 8.96577 11.23543 NA NA 2.30624 2.19341 2.41907 NA NA
0.81879 0.80019 0.83738 NA NA  0.85493   0.84164   0.86822  NA NA
0.30318 0.29385  0.31252 NA NA 0.69794 0.65463  0.74124 NA NA 0.67376
0.65321  0.74267 NA NA 0.10206  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >0.254      >12.700    NA
NA    ECLV      59496    0.057281       0.51514      19
0.05     0.48419    0.1        0.27944    0.15      -0.026529   0.2
-0.37075   0.25    -0.76086     0.3 -1.2067     0.35     -1.72113
0.4 -2.3213    0.45      -3.0306     0.5 -3.88175   0.55    -4.92205
0.6     -6.22242   0.65 -7.89432    0.7        -10.12353    0.75
-13.24442  0.8     -17.92576  0.85    -25.72799   0.9 -41.33245   0.95
-88.14583
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >2.540      >12.700    NA
NA    FHO       59496    0.20169        0.035616      0.057281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >2.540      >12.700    NA
NA    CTC       59496 2119           9881          1289        46207
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >2.540      >12.700    NA
0.01  CTS       59496    0.057281       0.054876      0.059785    NA
NA          0.20169    0.19749    0.20596   NA         NA    0.81226
0.8081   0.81635    NA   NA          3.52113  NA         NA    0.62177
0.61664    0.62688   NA   NA         0.82383  0.81977  0.82782 NA
NA     0.17617    0.17218      0.18023   NA     NA        0.82342
0.81935  0.82741  NA        NA     0.15946   0.15563      0.16336   NA
NA 0.11361   NA NA 0.4456    0.42358    0.46762   NA NA 0.20403   NA
NA  7.6875  6.98828  8.45667 NA NA 2.0396  1.94424 2.13496 NA NA
0.76978 0.75036 0.78921 NA NA  0.71503   0.69733   0.73273  NA NA
0.33758 0.32378  0.35138 NA NA 0.57026 0.53483  0.60569 NA NA 0.60387
0.5349   0.60562 NA NA 0.099637 NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >2.540      >12.700    NA
NA    ECLV      59496    0.057281       0.4456       19
0.05     0.38718    0.1        0.29962    0.15       0.11012    0.2
-0.10307   0.25    -0.34468     0.3 -0.62081    0.35     -0.93942
0.4 -1.31113   0.45      -1.75043    0.5 -2.27758   0.55    -2.92188
0.6     -3.72726   0.65 -4.76274    0.7         -6.14339    0.75
-8.07629  0.8     -10.97565  0.85    -15.8079    0.9 -25.47242   0.95
-54.46596
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >3.810      >12.700    NA
NA    FHO       59496    0.177          0.032456      0.057281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >3.810      >12.700    NA
NA    CTC       59496 1931           8600          1477        47488
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >3.810      >12.700    NA
0.01  CTS       59496    0.057281       0.054876      0.059785    NA
NA          0.177      0.17301    0.18107   NA         NA    0.83063
0.82663  0.83455    NA   NA          3.09008  NA         NA    0.56661
0.56137    0.57183   NA   NA         0.84667  0.84283  0.85044 NA
NA     0.15333    0.14956      0.15717   NA     NA        0.81664
0.81251  0.82069  NA        NA     0.16081   0.15697      0.16473   NA
NA 0.11642   NA NA 0.41328   0.39078    0.43578   NA NA 0.20856   NA
NA  7.21917 6.57137  7.93083 NA NA 1.97674 1.88272 2.07076 NA NA
0.75667 0.73657 0.77676 NA NA  0.66855   0.64976   0.68733  NA NA
0.33942 0.32434  0.3545  NA NA 0.53497 0.50113  0.56882 NA NA 0.57367
0.50069  0.56925 NA NA 0.098419 NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >3.810      >12.700    NA
NA    ECLV      59496    0.057281       0.41328      19
0.05     0.34633    0.1        0.28622    0.15       0.12129    0.2
-0.064261  0.25    -0.27455     0.3 -0.51488    0.35     -0.79219
0.4 -1.11571   0.45      -1.49805    0.5 -1.95687   0.55    -2.51764
0.6     -3.2186    0.65 -4.11984    0.7         -5.3215     0.75
-7.00381  0.8      -9.52729  0.85    -13.73308   0.9 -22.14466   0.95
-47.3794
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >6.350      >12.700    NA
NA    FHO       59496    0.14216        0.027783      0.057281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >6.350      >12.700    NA
NA    CTC       59496 1653           6805          1755        49283
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >6.350      >12.700    NA
0.01  CTS       59496    0.057281       0.054876      0.059785    NA
NA          0.14216    0.13851    0.14589   NA         NA    0.85612
0.85238  0.85979    NA   NA          2.48181  NA         NA    0.48504
0.47976    0.49031   NA   NA         0.87867  0.87518  0.88208 NA
NA     0.12133    0.11792      0.12482   NA     NA        0.80456
0.80034  0.80872  NA        NA     0.16185   0.158        0.16578   NA
NA 0.12011   NA NA 0.36371   0.34102    0.3864    NA NA 0.21446   NA
NA  6.82126 6.20703  7.49627 NA NA 1.92004 1.82568 2.01441 NA NA
0.74429 0.72324 0.76533 NA NA  0.59617   0.57591   0.61642  NA NA
0.34249 0.32546  0.35953 NA NA 0.48917 0.45755  0.5208  NA NA 0.52955
0.45639  0.52196 NA NA 0.098733 NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >6.350      >12.700    NA
NA    ECLV      59496    0.057281       0.36371      19
0.05     0.28416    0.1        0.26317    0.15       0.13266    0.2
-0.014158  0.25    -0.18056     0.3 -0.37072    0.35     -0.59015
0.4 -0.84615   0.45      -1.14869    0.5 -1.51174   0.55    -1.95546
0.6     -2.51012   0.65 -3.22326    0.7         -4.1741     0.75
-5.50528  0.8      -7.50205  0.85    -10.83001   0.9 -17.48592   0.95
-37.45364
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >12.700     >12.700    NA
NA    FHO       59496    0.087199       0.019111      0.057281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >12.700     >12.700    NA
NA    CTC       59496 1137           4051          2271        52037
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >12.700     >12.700    NA
0.01  CTS       59496    0.057281       0.054876      0.059785    NA
NA          0.087199   0.084266   0.090225  NA         NA    0.89374
0.89044  0.89695    NA   NA          1.5223   NA         NA    0.33363
0.32867    0.33862   NA   NA         0.92777  0.92499  0.93046 NA
NA     0.072226   0.06954      0.075007  NA     NA        0.78084
0.77644  0.78518  NA        NA     0.15243   0.14868      0.15627   NA
NA 0.11726   NA NA 0.2614    0.23999    0.28281   NA NA 0.20991   NA
NA  6.43122 5.80422  7.12595 NA NA 1.86116 1.75859 1.96374 NA NA
0.73087 0.70697 0.75476 NA NA  0.44524   0.42247   0.46801  NA NA
0.33906 0.31796  0.36016 NA NA 0.41072 0.38217  0.43927 NA NA 0.44244
0.38023  0.44122 NA NA 0.10149  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >12.700     >12.700    NA
NA    ECLV      59496    0.057281       0.2614       19
0.05     0.15847    0.1        0.20155    0.15       0.12386    0.2
0.036458  0.25    -0.062598    0.3 -0.1758     0.35     -0.30643
0.4 -0.45882   0.45      -0.63892    0.5 -0.85505   0.55    -1.1192
0.6     -1.44938   0.65 -1.87391    0.7         -2.43995    0.75
-3.23239  0.8      -4.42107  0.85     -6.40219   0.9 -10.36444   0.95
-22.25117
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >19.05      >12.700    NA
NA    FHO       59496    0.053953       0.012875      0.057281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >19.05      >12.700    NA
NA    CTC       59496  766           2444          2642        53644
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >19.05      >12.700    NA
0.01  CTS       59496    0.057281       0.054876      0.059785    NA
NA          0.053953   0.051617   0.056389  NA         NA    0.91452
0.91152  0.91742    NA   NA          0.9419   NA         NA    0.22477
0.22039    0.2292    NA   NA         0.95643  0.95422  0.95853 NA
NA     0.043574   0.041469     0.045782  NA     NA        0.76137
0.75684  0.76584  NA        NA     0.1309    0.12737      0.1345    NA
NA 0.1027    NA NA 0.18119   0.16223    0.20015   NA NA 0.18627   NA
NA  6.36379 5.65338  7.16347 NA NA 1.85062 1.73225 1.96899 NA NA
0.7284  0.70062 0.75618 NA NA  0.31409   0.28935   0.33883  NA NA
0.32784 0.30285  0.35284 NA NA 0.35465 0.32734  0.38196 NA NA 0.37575
0.32543  0.38386 NA NA 0.10554  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >19.05      >12.700    NA
NA    ECLV      59496    0.057281       0.18119      19
0.05     0.061439   0.1        0.14508    0.15       0.098212   0.2
0.045481  0.25    -0.01428     0.3 -0.082579   0.35     -0.16138
0.4 -0.25333   0.45      -0.36198    0.5 -0.49237   0.55    -0.65173
0.6     -0.85094   0.65 -1.10706    0.7         -1.44855    0.75
-1.92664  0.8      -2.64378  0.85     -3.83901   0.9  -6.22946   0.95
-13.40082
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >25.400     >12.700    NA
NA    FHO       59496    0.033263       0.0083199     0.057281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >25.400     >12.700    NA
NA    CTC       59496  495           1484          2913        54604
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >25.400     >12.700    NA
0.01  CTS       59496    0.057281       0.054876      0.059785    NA
NA          0.033263   0.03142    0.035209  NA         NA    0.9261
0.92329  0.92881    NA   NA          0.58069  NA         NA    0.14525
0.14157    0.14901   NA   NA         0.97354  0.97179  0.97518 NA
NA     0.026458   0.024816     0.028207  NA     NA        0.74987
0.74527  0.75442  NA        NA     0.10119   0.098045     0.10441   NA
NA 0.079864  NA NA 0.11879   0.10278    0.1348    NA NA 0.14791   NA
NA  6.25252 5.42272  7.2093  NA NA 1.83299 1.6906  1.97537 NA NA
0.72423 0.69038 0.75809 NA NA  0.19429   0.16759   0.22098  NA NA
0.30778 0.27855  0.33701 NA NA 0.30619 0.27878  0.33359 NA NA 0.31904
0.27719  0.33519 NA NA 0.10714  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >25.400     >12.700    NA
NA    ECLV      59496    0.057281       0.11879      19
0.05    -0.013247   0.1        0.096864   0.15       0.068403   0.2
0.036385  0.25     9.7809e-05  0.3 -0.041373   0.35     -0.089224
0.4 -0.14505   0.45      -0.21103    0.5 -0.2902    0.55    -0.38697
0.6     -0.50792   0.65 -0.66344    0.7         -0.87079    0.75
-1.16109  0.8      -1.59654  0.85     -2.32228   0.9  -3.77377   0.95
-8.12823
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >38.1       >12.700    NA
NA    FHO       59496    0.012707       0.0030254     0.057281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >38.1       >12.700    NA
NA    CTC       59496  180            576          3228        55512
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >38.1       >12.700    NA
0.01  CTS       59496    0.057281       0.054876      0.059785    NA
NA          0.012707   0.011577   0.013945  NA         NA    0.93606
0.93343  0.9386     NA   NA          0.22183  NA         NA
0.052817  0.050504   0.055229  NA   NA         0.98973  0.98861
0.99074 NA        NA     0.01027    0.0092582    0.01139   NA     NA
0.7619    0.75738  0.76637  NA        NA     0.045181  0.043038
0.047425  NA NA 0.034688  NA NA 0.042547  0.032355   0.05274   NA NA
0.06705   NA NA  5.37407 4.29196  6.72901 NA NA 1.68159 1.45674
1.90643 NA NA 0.68623 0.62675 0.74571 NA NA -0.013988 -0.045749
0.017773 NA NA 0.24561 0.20548  0.28573 NA NA 0.21779 0.18644  0.24913
NA NA 0.22173 0.18553  0.25004 NA NA 0.095265 NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >38.1       >12.700    NA
NA    ECLV      59496    0.057281       0.042547     19
0.05    -0.10377    0.1        0.034038   0.15       0.022991   0.2
0.010563  0.25    -0.0035211   0.3 -0.019618   0.35     -0.038191
0.4 -0.059859  0.45      -0.085467   0.5 -0.1162    0.55    -0.15376
0.6     -0.2007    0.65 -0.26107    0.7         -0.34155    0.75
-0.45423  0.8      -0.62324  0.85     -0.90493   0.9  -1.46831   0.95
-3.15845
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >50.8       >12.700    NA
NA    FHO       59496    0.0053281      0.0011429     0.057281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >50.8       >12.700    NA
NA    CTC       59496   68            249          3340        55839
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >50.8       >12.700    NA
0.01  CTS       59496    0.057281       0.054876      0.059785    NA
NA          0.0053281  0.0046125  0.006154  NA         NA    0.93968
0.93711  0.94214    NA   NA          0.093016 NA         NA
0.019953  0.018529   0.021484  NA   NA         0.99556  0.9948
0.99621 NA        NA     0.0044395  0.0037905    0.0051989 NA     NA
0.78549   0.78112  0.78979  NA        NA     0.018594  0.017221
0.020076  NA NA 0.013697  NA NA 0.015514  0.0090962  0.021931  NA NA
0.027024  NA NA  4.56563 3.19995  6.51415 NA NA 1.51856 1.16314
1.87398 NA NA 0.64065 0.53588 0.74542 NA NA -0.15568  -0.19422
-0.11714  NA NA 0.19492 0.14037  0.24946 NA NA 0.16105 0.12201
0.20009 NA NA 0.1623  0.1215   0.2006  NA NA 0.080926 NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >50.8       >12.700    NA
NA    ECLV      59496    0.057281       0.015514     19
0.05    -0.13588    0.1        0.011835   0.15       0.0070595  0.2
0.0016872 0.25    -0.0044014   0.3 -0.01136    0.35     -0.019389
0.4 -0.028756  0.45      -0.039826   0.5 -0.05311   0.55    -0.069347
0.6     -0.089642  0.65 -0.11574    0.7         -0.15053    0.75
-0.19924  0.8      -0.2723   0.85     -0.39407   0.9  -0.63762   0.95
-1.36825
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >76.2       >12.700    NA
NA    FHO       59496    0.0010757      0.00028573    0.057281
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >76.2       >12.700    NA
NA    CTC       59496   17             47          3391        56041
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >76.2       >12.700    NA
0.01  CTS       59496    0.057281       0.054876      0.059785    NA
NA          0.0010757  0.00078075 0.0014819 NA         NA    0.94221
0.9397   0.94463    NA   NA          0.018779 NA         NA
0.0049883 0.0042975  0.0057894 NA   NA         0.99916  0.9988
0.99942 NA        NA     0.00083797 0.00058305   0.0012042 NA     NA
0.73438   0.72968  0.73901  NA        NA     0.0049204 0.0042347
0.0057166 NA NA 0.0038634 NA NA 0.0041503 0.00080965 0.0074909 NA NA
0.0076971 NA NA  5.97763 2.87943 12.40943 NA NA 1.78802 1.05759
2.51846 NA NA 0.71337 0.53401 0.89273 NA NA -0.29911  -0.35263
-0.24559  NA NA 0.18799 0.097273 0.27871 NA NA 0.14403 0.086253
0.20181 NA NA 0.1443  0.086051 0.20201 NA NA 0.10779  NA NA
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           >76.2       >12.700    NA
NA    ECLV      59496    0.057281       0.0041503    19
0.05    -0.14955    0.1        0.0034559  0.15       0.0025545  0.2
0.0015405 0.25     0.00039124  0.3 -0.0009222  0.35     -0.0024377
0.4 -0.0042058 0.45      -0.0062953  0.5 -0.0088028 0.55    -0.011868
0.6     -0.015698  0.65 -0.020624   0.7         -0.027191   0.75
-0.036385 0.8      -0.050176 0.85     -0.073161  0.9  -0.11913   0.95
-0.25704
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS FULL    NEAREST     1           NA          NA         NA
NA    GRAD      69480    3.40993        2.10619       4.62772
4.58341 99.04243  217.61602    1.619
V7.0    CONUSNEST4ROC NA   240000    20180809_120000 20180809_120000
240000   20180809_120000 20180809_120000 APCP_24  A24      APCP_24 A24
ANALYS CONUS   NEAREST     1           NA          NA         NA
NA    GRAD      59216    3.44714        2.18753       4.68524
4.631   98.84229  211.69954    1.57581

------------------------------------------------


More information about the Met_help mailing list