[Met_help] [rt.rap.ucar.edu #86767] History for ROC/Area under ROC in METviewer
John Halley Gotway via RT
met_help at ucar.edu
Wed Jul 10 16:43:17 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello. I'm trying to make a simple ROC plot. The attached *.xml does
not work - err msg
"Failed to run plot plot_20180827_184319 - reason: R error: ==== Start
Rscript error ==== Error in .subset2(x, i, exact = exact) : subscript
out of bounds In addition."
I verified that data for the forecast valid time/lead time (24h ending
at 12Z 21 Aug) do exist and I could plot POD/FAR from them. What should
I be doing differently in the set up? I asked Ben Blake about this
(googling found his inquiries on ROC/AUR from early 2017) and he said to
put OBS_THRESH under series variables instead of fixed values, that
didn't work either.
Also, once I get ROC to work ... where is AUR? Is it under Roc tab, or
Series? I can't figure out where it could be under Roc; under Series
there doesn't seem to be something that might be AUR under "Select
attribute stat".
Thanks!
Ying
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Tatiana Burek
Time: Mon Aug 27 14:43:53 2018
Hi Ying,
ROC plop is fixed and I was able to create a plot using your XML.
What do you mean by AUR?
Tatiana
On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
> Hello. I'm trying to make a simple ROC plot. The attached *.xml
does
> not work - err msg
>
> "Failed to run plot plot_20180827_184319 - reason: R error: ====
Start
> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
> out of bounds In addition."
>
> I verified that data for the forecast valid time/lead time (24h
ending
> at 12Z 21 Aug) do exist and I could plot POD/FAR from them. What
should
> I be doing differently in the set up? I asked Ben Blake about this
> (googling found his inquiries on ROC/AUR from early 2017) and he
said to
> put OBS_THRESH under series variables instead of fixed values, that
> didn't work either.
>
> Also, once I get ROC to work ... where is AUR? Is it under Roc tab,
or
> Series? I can't figure out where it could be under Roc; under
Series
> there doesn't seem to be something that might be AUR under "Select
> attribute stat".
>
> Thanks!
>
> Ying
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #86767] ROC/Area under ROC in METviewer
From: Ying Lin
Time: Mon Aug 27 15:08:34 2018
Thanks Tatiana.
Sorry - AUR "Area Under ROC".
I see your ROC plot and it seems the difference between your xml
and my failed attempt is that you chose 'PCT' instead of 'CTC' for ROC
calculations. I thought that PCT is for ROC score of probability
forecasts, while CTC is from the contingency table count of single
deterministic models' ROC?
Ying
On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
> Hi Ying,
> ROC plop is fixed and I was able to create a plot using your XML.
> What do you mean by AUR?
>
> Tatiana
>
> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>> Hello. I'm trying to make a simple ROC plot. The attached *.xml
does
>> not work - err msg
>>
>> "Failed to run plot plot_20180827_184319 - reason: R error: ====
Start
>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
>> out of bounds In addition."
>>
>> I verified that data for the forecast valid time/lead time (24h
ending
>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them. What
should
>> I be doing differently in the set up? I asked Ben Blake about this
>> (googling found his inquiries on ROC/AUR from early 2017) and he
said to
>> put OBS_THRESH under series variables instead of fixed values, that
>> didn't work either.
>>
>> Also, once I get ROC to work ... where is AUR? Is it under Roc
tab, or
>> Series? I can't figure out where it could be under Roc; under
Series
>> there doesn't seem to be something that might be AUR under "Select
>> attribute stat".
>>
>> Thanks!
>>
>> Ying
>>
>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Ying Lin
Time: Mon Aug 27 15:34:21 2018
Hi Tatiana,
I tried making a ROC plot from MET *.stat (the case of the
original
question was from VSDBs loaded to MV) and the CTC seems to work - see
attached xml.
Is the CTC option not available for ROC from VSDBs?
For the ROC from the MET *.stat, I played with the OBS_THRESH a
bit, it appears that the ROC plot can accept up to two different
OBS_THRESH (as in the attached xml), but more than two will result in
an
error.
Thanks again,
Ying
On 08/27/2018 05:08 PM, Ying Lin wrote:
> Thanks Tatiana.
>
> Sorry - AUR "Area Under ROC".
>
> I see your ROC plot and it seems the difference between your xml
> and my failed attempt is that you chose 'PCT' instead of 'CTC' for
ROC
> calculations. I thought that PCT is for ROC score of probability
> forecasts, while CTC is from the contingency table count of single
> deterministic models' ROC?
>
> Ying
>
> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
>> Hi Ying,
>> ROC plop is fixed and I was able to create a plot using your XML.
>> What do you mean by AUR?
>>
>> Tatiana
>>
>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>>> Hello. I'm trying to make a simple ROC plot. The attached *.xml
does
>>> not work - err msg
>>>
>>> "Failed to run plot plot_20180827_184319 - reason: R error: ====
Start
>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
>>> out of bounds In addition."
>>>
>>> I verified that data for the forecast valid time/lead time (24h
ending
>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them. What
should
>>> I be doing differently in the set up? I asked Ben Blake about
this
>>> (googling found his inquiries on ROC/AUR from early 2017) and he
>>> said to
>>> put OBS_THRESH under series variables instead of fixed values,
that
>>> didn't work either.
>>>
>>> Also, once I get ROC to work ... where is AUR? Is it under Roc
tab, or
>>> Series? I can't figure out where it could be under Roc; under
Series
>>> there doesn't seem to be something that might be AUR under "Select
>>> attribute stat".
>>>
>>> Thanks!
>>>
>>> Ying
>>>
>>
>>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #86767] ROC/Area under ROC in METviewer
From: Ying Lin
Time: Tue Aug 28 12:56:28 2018
Hi Tatiana,
Follow up on this ... has the ROC plotting in METviewer been used
in the non-probabilistic sense before? I'm thinking that maybe
obs_thresh shouldn't be "fix values". The ROC plot seems to be
analogous to the performance diagram, in which data points are
produced
for forecast lead times - but FCST_LEAD in the performance diagrams
are
"independent variables" rather than fixed values. There is no
"independent variables" option in ROC, the only way to make the
"CTC-type" ROC plot seems to be to set OBS_THRESH as fixed values, but
then it seems only two OBS_THRESH values can be accepted by METviewer.
Thanks -
Ying
On 08/27/2018 05:34 PM, Ying Lin wrote:
> Hi Tatiana,
>
> I tried making a ROC plot from MET *.stat (the case of the
> original question was from VSDBs loaded to MV) and the CTC seems to
> work - see attached xml.
>
> Is the CTC option not available for ROC from VSDBs?
>
> For the ROC from the MET *.stat, I played with the OBS_THRESH a
> bit, it appears that the ROC plot can accept up to two different
> OBS_THRESH (as in the attached xml), but more than two will result
in
> an error.
>
> Thanks again,
>
> Ying
>
> On 08/27/2018 05:08 PM, Ying Lin wrote:
>> Thanks Tatiana.
>>
>> Sorry - AUR "Area Under ROC".
>>
>> I see your ROC plot and it seems the difference between your
xml
>> and my failed attempt is that you chose 'PCT' instead of 'CTC' for
>> ROC calculations. I thought that PCT is for ROC score of
probability
>> forecasts, while CTC is from the contingency table count of single
>> deterministic models' ROC?
>>
>> Ying
>>
>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
>>> Hi Ying,
>>> ROC plop is fixed and I was able to create a plot using your XML.
>>> What do you mean by AUR?
>>>
>>> Tatiana
>>>
>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>>>> Hello. I'm trying to make a simple ROC plot. The attached *.xml
does
>>>> not work - err msg
>>>>
>>>> "Failed to run plot plot_20180827_184319 - reason: R error: ====
Start
>>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
>>>> out of bounds In addition."
>>>>
>>>> I verified that data for the forecast valid time/lead time (24h
ending
>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them. What
>>>> should
>>>> I be doing differently in the set up? I asked Ben Blake about
this
>>>> (googling found his inquiries on ROC/AUR from early 2017) and he
>>>> said to
>>>> put OBS_THRESH under series variables instead of fixed values,
that
>>>> didn't work either.
>>>>
>>>> Also, once I get ROC to work ... where is AUR? Is it under Roc
>>>> tab, or
>>>> Series? I can't figure out where it could be under Roc; under
Series
>>>> there doesn't seem to be something that might be AUR under
"Select
>>>> attribute stat".
>>>>
>>>> Thanks!
>>>>
>>>> Ying
>>>>
>>>
>>>
>>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: John Halley Gotway
Time: Fri Sep 07 17:00:53 2018
Hi Tressa,
I was supposed to talk to you about this issue earlier this week but
forgot. I think we need your guidance on this one.
This is a met-help ticket from Ying Lin at NCEP about ROC curves in
METviewer. I believe there are 2 ways to create a ROC curve in
METviewer:
(1) The most common way I know is using the Nx2 Probabilistic
Contingency
Table counts from the PCT line in MET. I understand this one. MET
can
also write out the PSTD line type which includes the ROC_AUC column.
And
that's a number between 0 and 1 indicating the area under the ROC
curve,
with 1 being the best. MET calculates this value and it gets loaded
into
METviewer. METviewer can plot the ROC_AUC value but it doesn't
compute it.
(2) The less common way of generating a ROC curve is using several 2x2
contingency table lines. This would be the FHO or CTC line types from
MET. However, there's some requirement in how those CTC lines are
defined... but I can't remember exactly what. Perhaps they need to be
multiple forecast thresholds but all for the same observation
threshold?
Does that sound correct to you? If we get all the right, then
METviewer
could make a ROC plot. But, since METviewer doesn't compute the area
under
the ROC curve, there's no way for METviewer to plot that statistic.
I think this would all be good info to clarify in the METviewer user's
guide:
- Show an example of using PCT output to make a ROC curve in
METviewer.
- Show an example of using CTC output to make a ROC curve in
METviewer...
and list the threshold requirements for doing so.
And lastly, we should identify what changes to METviewer are needed to
make
it more useful to Ying. Perhaps we do need to enhance METviewer to
compute
the area under the ROC curve for case (2)?
Thanks,
John
On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>
> Hi Tatiana,
>
> Follow up on this ... has the ROC plotting in METviewer been
used
> in the non-probabilistic sense before? I'm thinking that maybe
> obs_thresh shouldn't be "fix values". The ROC plot seems to be
> analogous to the performance diagram, in which data points are
produced
> for forecast lead times - but FCST_LEAD in the performance diagrams
are
> "independent variables" rather than fixed values. There is no
> "independent variables" option in ROC, the only way to make the
> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed values,
but
> then it seems only two OBS_THRESH values can be accepted by
METviewer.
>
> Thanks -
>
> Ying
>
> On 08/27/2018 05:34 PM, Ying Lin wrote:
> > Hi Tatiana,
> >
> > I tried making a ROC plot from MET *.stat (the case of the
> > original question was from VSDBs loaded to MV) and the CTC seems
to
> > work - see attached xml.
> >
> > Is the CTC option not available for ROC from VSDBs?
> >
> > For the ROC from the MET *.stat, I played with the OBS_THRESH
a
> > bit, it appears that the ROC plot can accept up to two different
> > OBS_THRESH (as in the attached xml), but more than two will result
in
> > an error.
> >
> > Thanks again,
> >
> > Ying
> >
> > On 08/27/2018 05:08 PM, Ying Lin wrote:
> >> Thanks Tatiana.
> >>
> >> Sorry - AUR "Area Under ROC".
> >>
> >> I see your ROC plot and it seems the difference between your
xml
> >> and my failed attempt is that you chose 'PCT' instead of 'CTC'
for
> >> ROC calculations. I thought that PCT is for ROC score of
probability
> >> forecasts, while CTC is from the contingency table count of
single
> >> deterministic models' ROC?
> >>
> >> Ying
> >>
> >> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
> >>> Hi Ying,
> >>> ROC plop is fixed and I was able to create a plot using your
XML.
> >>> What do you mean by AUR?
> >>>
> >>> Tatiana
> >>>
> >>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
> >>>> Hello. I'm trying to make a simple ROC plot. The attached
*.xml does
> >>>> not work - err msg
> >>>>
> >>>> "Failed to run plot plot_20180827_184319 - reason: R error:
==== Start
> >>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
> >>>> out of bounds In addition."
> >>>>
> >>>> I verified that data for the forecast valid time/lead time (24h
ending
> >>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.
What
> >>>> should
> >>>> I be doing differently in the set up? I asked Ben Blake about
this
> >>>> (googling found his inquiries on ROC/AUR from early 2017) and
he
> >>>> said to
> >>>> put OBS_THRESH under series variables instead of fixed values,
that
> >>>> didn't work either.
> >>>>
> >>>> Also, once I get ROC to work ... where is AUR? Is it under Roc
> >>>> tab, or
> >>>> Series? I can't figure out where it could be under Roc; under
Series
> >>>> there doesn't seem to be something that might be AUR under
"Select
> >>>> attribute stat".
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Ying
> >>>>
> >>>
> >>>
> >>
> >
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #86767] ROC/Area under ROC in METviewer
From: Ying Lin
Time: Sat Sep 08 16:37:02 2018
Hi John and Tressa,
I was originally under the impression that the ROC/Area under ROC was
for probabilistic forecasts. Do they make sense/are they worth
looking at in the non-probabilistic sense (e.g. ROC curve for varying
forecast/obs thresholds)? I guess a case can be made that the CTC-ROC
makes some sense (similar to the performance diagram), but unlike the
probabilistic ROC curve that rises with decreasing PCT, the "CTC ROC"
based on the F/H/O contingency table counts for different thresholds
(and probably for different forecast lead time as well) can be all
over the place, making the concept of "area under ROC” problematic.
If a non-prob ROC doesn't make enough sense/is not worth pursuing,
that's fine. The AUC was listed as one of the measures in the CAM
metrics workshop, so I got curious when seeing the "CTC" option in the
METviewer ROC. That's all.
Ying
> On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Hi Tressa,
>
> I was supposed to talk to you about this issue earlier this week but
> forgot. I think we need your guidance on this one.
>
> This is a met-help ticket from Ying Lin at NCEP about ROC curves in
> METviewer. I believe there are 2 ways to create a ROC curve in
METviewer:
>
> (1) The most common way I know is using the Nx2 Probabilistic
Contingency
> Table counts from the PCT line in MET. I understand this one. MET
can
> also write out the PSTD line type which includes the ROC_AUC column.
And
> that's a number between 0 and 1 indicating the area under the ROC
curve,
> with 1 being the best. MET calculates this value and it gets loaded
into
> METviewer. METviewer can plot the ROC_AUC value but it doesn't
compute it.
>
> (2) The less common way of generating a ROC curve is using several
2x2
> contingency table lines. This would be the FHO or CTC line types
from
> MET. However, there's some requirement in how those CTC lines are
> defined... but I can't remember exactly what. Perhaps they need to
be
> multiple forecast thresholds but all for the same observation
threshold?
> Does that sound correct to you? If we get all the right, then
METviewer
> could make a ROC plot. But, since METviewer doesn't compute the
area under
> the ROC curve, there's no way for METviewer to plot that statistic.
>
> I think this would all be good info to clarify in the METviewer
user's
> guide:
> - Show an example of using PCT output to make a ROC curve in
METviewer.
> - Show an example of using CTC output to make a ROC curve in
METviewer...
> and list the threshold requirements for doing so.
>
> And lastly, we should identify what changes to METviewer are needed
to make
> it more useful to Ying. Perhaps we do need to enhance METviewer to
compute
> the area under the ROC curve for case (2)?
>
> Thanks,
> John
>
> On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>>
>> Hi Tatiana,
>>
>> Follow up on this ... has the ROC plotting in METviewer been
used
>> in the non-probabilistic sense before? I'm thinking that maybe
>> obs_thresh shouldn't be "fix values". The ROC plot seems to be
>> analogous to the performance diagram, in which data points are
produced
>> for forecast lead times - but FCST_LEAD in the performance diagrams
are
>> "independent variables" rather than fixed values. There is no
>> "independent variables" option in ROC, the only way to make the
>> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed values,
but
>> then it seems only two OBS_THRESH values can be accepted by
METviewer.
>>
>> Thanks -
>>
>> Ying
>>
>> On 08/27/2018 05:34 PM, Ying Lin wrote:
>>> Hi Tatiana,
>>>
>>> I tried making a ROC plot from MET *.stat (the case of the
>>> original question was from VSDBs loaded to MV) and the CTC seems
to
>>> work - see attached xml.
>>>
>>> Is the CTC option not available for ROC from VSDBs?
>>>
>>> For the ROC from the MET *.stat, I played with the OBS_THRESH a
>>> bit, it appears that the ROC plot can accept up to two different
>>> OBS_THRESH (as in the attached xml), but more than two will result
in
>>> an error.
>>>
>>> Thanks again,
>>>
>>> Ying
>>>
>>> On 08/27/2018 05:08 PM, Ying Lin wrote:
>>>> Thanks Tatiana.
>>>>
>>>> Sorry - AUR "Area Under ROC".
>>>>
>>>> I see your ROC plot and it seems the difference between your
xml
>>>> and my failed attempt is that you chose 'PCT' instead of 'CTC'
for
>>>> ROC calculations. I thought that PCT is for ROC score of
probability
>>>> forecasts, while CTC is from the contingency table count of
single
>>>> deterministic models' ROC?
>>>>
>>>> Ying
>>>>
>>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
>>>>> Hi Ying,
>>>>> ROC plop is fixed and I was able to create a plot using your
XML.
>>>>> What do you mean by AUR?
>>>>>
>>>>> Tatiana
>>>>>
>>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>>>>>> Hello. I'm trying to make a simple ROC plot. The attached
*.xml does
>>>>>> not work - err msg
>>>>>>
>>>>>> "Failed to run plot plot_20180827_184319 - reason: R error:
==== Start
>>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
subscript
>>>>>> out of bounds In addition."
>>>>>>
>>>>>> I verified that data for the forecast valid time/lead time (24h
ending
>>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.
What
>>>>>> should
>>>>>> I be doing differently in the set up? I asked Ben Blake about
this
>>>>>> (googling found his inquiries on ROC/AUR from early 2017) and
he
>>>>>> said to
>>>>>> put OBS_THRESH under series variables instead of fixed values,
that
>>>>>> didn't work either.
>>>>>>
>>>>>> Also, once I get ROC to work ... where is AUR? Is it under Roc
>>>>>> tab, or
>>>>>> Series? I can't figure out where it could be under Roc; under
Series
>>>>>> there doesn't seem to be something that might be AUR under
"Select
>>>>>> attribute stat".
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Ying
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
>
------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Tressa Fowler
Time: Mon Sep 10 13:55:21 2018
Hi Ying,
It definitely makes sense to calculate AUC for CTC-derived ROC curves,
so long as a single observation threshold is selected, and the
forecast thresholds fill in the curve pretty well. I think we can
create an issue to add this for future releases.
Tressa
On Sat Sep 08 16:37:02 2018, ying.lin at noaa.gov wrote:
> Hi John and Tressa,
>
> I was originally under the impression that the ROC/Area under ROC
was
> for probabilistic forecasts. Do they make sense/are they worth
> looking at in the non-probabilistic sense (e.g. ROC curve for
varying
> forecast/obs thresholds)? I guess a case can be made that the CTC-
ROC
> makes some sense (similar to the performance diagram), but unlike
the
> probabilistic ROC curve that rises with decreasing PCT, the "CTC
ROC"
> based on the F/H/O contingency table counts for different thresholds
> (and probably for different forecast lead time as well) can be all
> over the place, making the concept of "area under ROC” problematic.
>
> If a non-prob ROC doesn't make enough sense/is not worth pursuing,
> that's fine. The AUC was listed as one of the measures in the CAM
> metrics workshop, so I got curious when seeing the "CTC" option in
the
> METviewer ROC. That's all.
>
> Ying
>
> > On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
> > <met_help at ucar.edu> wrote:
> >
> > Hi Tressa,
> >
> > I was supposed to talk to you about this issue earlier this week
but
> > forgot. I think we need your guidance on this one.
> >
> > This is a met-help ticket from Ying Lin at NCEP about ROC curves
in
> > METviewer. I believe there are 2 ways to create a ROC curve in
> > METviewer:
> >
> > (1) The most common way I know is using the Nx2 Probabilistic
> > Contingency
> > Table counts from the PCT line in MET. I understand this one.
MET
> > can
> > also write out the PSTD line type which includes the ROC_AUC
column.
> > And
> > that's a number between 0 and 1 indicating the area under the ROC
> > curve,
> > with 1 being the best. MET calculates this value and it gets
loaded
> > into
> > METviewer. METviewer can plot the ROC_AUC value but it doesn't
> > compute it.
> >
> > (2) The less common way of generating a ROC curve is using several
> > 2x2
> > contingency table lines. This would be the FHO or CTC line types
> > from
> > MET. However, there's some requirement in how those CTC lines are
> > defined... but I can't remember exactly what. Perhaps they need
to
> > be
> > multiple forecast thresholds but all for the same observation
> > threshold?
> > Does that sound correct to you? If we get all the right, then
> > METviewer
> > could make a ROC plot. But, since METviewer doesn't compute the
area
> > under
> > the ROC curve, there's no way for METviewer to plot that
statistic.
> >
> > I think this would all be good info to clarify in the METviewer
> > user's
> > guide:
> > - Show an example of using PCT output to make a ROC curve in
> > METviewer.
> > - Show an example of using CTC output to make a ROC curve in
> > METviewer...
> > and list the threshold requirements for doing so.
> >
> > And lastly, we should identify what changes to METviewer are
needed
> > to make
> > it more useful to Ying. Perhaps we do need to enhance METviewer
to
> > compute
> > the area under the ROC curve for case (2)?
> >
> > Thanks,
> > John
> >
> > On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT
<met_help at ucar.edu>
> > wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
> >>
> >> Hi Tatiana,
> >>
> >> Follow up on this ... has the ROC plotting in METviewer been used
> >> in the non-probabilistic sense before? I'm thinking that maybe
> >> obs_thresh shouldn't be "fix values". The ROC plot seems to be
> >> analogous to the performance diagram, in which data points are
> >> produced
> >> for forecast lead times - but FCST_LEAD in the performance
diagrams
> >> are
> >> "independent variables" rather than fixed values. There is no
> >> "independent variables" option in ROC, the only way to make the
> >> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed
values,
> >> but
> >> then it seems only two OBS_THRESH values can be accepted by
> >> METviewer.
> >>
> >> Thanks -
> >>
> >> Ying
> >>
> >> On 08/27/2018 05:34 PM, Ying Lin wrote:
> >>> Hi Tatiana,
> >>>
> >>> I tried making a ROC plot from MET *.stat (the case of the
> >>> original question was from VSDBs loaded to MV) and the CTC seems
to
> >>> work - see attached xml.
> >>>
> >>> Is the CTC option not available for ROC from VSDBs?
> >>>
> >>> For the ROC from the MET *.stat, I played with the OBS_THRESH a
> >>> bit, it appears that the ROC plot can accept up to two different
> >>> OBS_THRESH (as in the attached xml), but more than two will
result
> >>> in
> >>> an error.
> >>>
> >>> Thanks again,
> >>>
> >>> Ying
> >>>
> >>> On 08/27/2018 05:08 PM, Ying Lin wrote:
> >>>> Thanks Tatiana.
> >>>>
> >>>> Sorry - AUR "Area Under ROC".
> >>>>
> >>>> I see your ROC plot and it seems the difference between your
xml
> >>>> and my failed attempt is that you chose 'PCT' instead of 'CTC'
for
> >>>> ROC calculations. I thought that PCT is for ROC score of
> >>>> probability
> >>>> forecasts, while CTC is from the contingency table count of
single
> >>>> deterministic models' ROC?
> >>>>
> >>>> Ying
> >>>>
> >>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
> >>>>> Hi Ying,
> >>>>> ROC plop is fixed and I was able to create a plot using your
XML.
> >>>>> What do you mean by AUR?
> >>>>>
> >>>>> Tatiana
> >>>>>
> >>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
> >>>>>> Hello. I'm trying to make a simple ROC plot. The attached
> >>>>>> *.xml does
> >>>>>> not work - err msg
> >>>>>>
> >>>>>> "Failed to run plot plot_20180827_184319 - reason: R error:
====
> >>>>>> Start
> >>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
> >>>>>> subscript
> >>>>>> out of bounds In addition."
> >>>>>>
> >>>>>> I verified that data for the forecast valid time/lead time
(24h
> >>>>>> ending
> >>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.
What
> >>>>>> should
> >>>>>> I be doing differently in the set up? I asked Ben Blake
about
> >>>>>> this
> >>>>>> (googling found his inquiries on ROC/AUR from early 2017) and
he
> >>>>>> said to
> >>>>>> put OBS_THRESH under series variables instead of fixed
values,
> >>>>>> that
> >>>>>> didn't work either.
> >>>>>>
> >>>>>> Also, once I get ROC to work ... where is AUR? Is it under
Roc
> >>>>>> tab, or
> >>>>>> Series? I can't figure out where it could be under Roc;
under
> >>>>>> Series
> >>>>>> there doesn't seem to be something that might be AUR under
> >>>>>> "Select
> >>>>>> attribute stat".
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>> Ying
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
> >
------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: John Halley Gotway
Time: Mon Sep 10 14:25:06 2018
Ying,
I believe that METviewer would be able to make this plot now (although
I
haven't confirmed that recently). But you would have to configure MET
in
the right way to define the thresholds the way Tressa described.
For example, let's say you're interested in 1/2 inch of precip
accumulation
(i.e. 12.7 mm). You define the thresholds in the MET config file like
the
following. I've listed the same obs threshold 10 times (>= 1/2 inch).
But
I've listed 10 different forecast thresholds (e.g. >=0,
>=0.1",...,>=1.0").
fcst = {
...
cat_thresh = [ ge0.00, ge2.54, ge5.08, ge7.62, ge10.16, ge12.7,
ge15.24,
ge17.78, ge20.32, ge22.86, ge25.4 ];
...
}
obs = {
...
cat_thresh = [ ge12.7, ge12.7, ge12.7, ge12.7, ge12.7, ge12.7,
ge12.7,
ge12.7, ge12.7, ge12.7 ];
...
}
Setting up the config file in this way should enable METviewer to
generate
a ROC curve that consists of 10 points... once for threshold pair
listed
above. I can tell you that we have not typically done this in the
past.
Does that make sense?
John
On Mon, Sep 10, 2018 at 1:55 PM Tressa Fowler via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>
> Hi Ying,
>
> It definitely makes sense to calculate AUC for CTC-derived ROC
curves, so
> long as a single observation threshold is selected, and the forecast
> thresholds fill in the curve pretty well. I think we can create an
issue to
> add this for future releases.
>
> Tressa
>
> On Sat Sep 08 16:37:02 2018, ying.lin at noaa.gov wrote:
> > Hi John and Tressa,
> >
> > I was originally under the impression that the ROC/Area under ROC
was
> > for probabilistic forecasts. Do they make sense/are they worth
> > looking at in the non-probabilistic sense (e.g. ROC curve for
varying
> > forecast/obs thresholds)? I guess a case can be made that the
CTC-ROC
> > makes some sense (similar to the performance diagram), but unlike
the
> > probabilistic ROC curve that rises with decreasing PCT, the "CTC
ROC"
> > based on the F/H/O contingency table counts for different
thresholds
> > (and probably for different forecast lead time as well) can be all
> > over the place, making the concept of "area under ROC”
problematic.
> >
> > If a non-prob ROC doesn't make enough sense/is not worth pursuing,
> > that's fine. The AUC was listed as one of the measures in the CAM
> > metrics workshop, so I got curious when seeing the "CTC" option in
the
> > METviewer ROC. That's all.
> >
> > Ying
> >
> > > On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
> > > <met_help at ucar.edu> wrote:
> > >
> > > Hi Tressa,
> > >
> > > I was supposed to talk to you about this issue earlier this week
but
> > > forgot. I think we need your guidance on this one.
> > >
> > > This is a met-help ticket from Ying Lin at NCEP about ROC curves
in
> > > METviewer. I believe there are 2 ways to create a ROC curve in
> > > METviewer:
> > >
> > > (1) The most common way I know is using the Nx2 Probabilistic
> > > Contingency
> > > Table counts from the PCT line in MET. I understand this one.
MET
> > > can
> > > also write out the PSTD line type which includes the ROC_AUC
column.
> > > And
> > > that's a number between 0 and 1 indicating the area under the
ROC
> > > curve,
> > > with 1 being the best. MET calculates this value and it gets
loaded
> > > into
> > > METviewer. METviewer can plot the ROC_AUC value but it doesn't
> > > compute it.
> > >
> > > (2) The less common way of generating a ROC curve is using
several
> > > 2x2
> > > contingency table lines. This would be the FHO or CTC line
types
> > > from
> > > MET. However, there's some requirement in how those CTC lines
are
> > > defined... but I can't remember exactly what. Perhaps they need
to
> > > be
> > > multiple forecast thresholds but all for the same observation
> > > threshold?
> > > Does that sound correct to you? If we get all the right, then
> > > METviewer
> > > could make a ROC plot. But, since METviewer doesn't compute the
area
> > > under
> > > the ROC curve, there's no way for METviewer to plot that
statistic.
> > >
> > > I think this would all be good info to clarify in the METviewer
> > > user's
> > > guide:
> > > - Show an example of using PCT output to make a ROC curve in
> > > METviewer.
> > > - Show an example of using CTC output to make a ROC curve in
> > > METviewer...
> > > and list the threshold requirements for doing so.
> > >
> > > And lastly, we should identify what changes to METviewer are
needed
> > > to make
> > > it more useful to Ying. Perhaps we do need to enhance METviewer
to
> > > compute
> > > the area under the ROC curve for case (2)?
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
> > >>
> > >> Hi Tatiana,
> > >>
> > >> Follow up on this ... has the ROC plotting in METviewer been
used
> > >> in the non-probabilistic sense before? I'm thinking that maybe
> > >> obs_thresh shouldn't be "fix values". The ROC plot seems to be
> > >> analogous to the performance diagram, in which data points are
> > >> produced
> > >> for forecast lead times - but FCST_LEAD in the performance
diagrams
> > >> are
> > >> "independent variables" rather than fixed values. There is no
> > >> "independent variables" option in ROC, the only way to make the
> > >> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed
values,
> > >> but
> > >> then it seems only two OBS_THRESH values can be accepted by
> > >> METviewer.
> > >>
> > >> Thanks -
> > >>
> > >> Ying
> > >>
> > >> On 08/27/2018 05:34 PM, Ying Lin wrote:
> > >>> Hi Tatiana,
> > >>>
> > >>> I tried making a ROC plot from MET *.stat (the case of the
> > >>> original question was from VSDBs loaded to MV) and the CTC
seems to
> > >>> work - see attached xml.
> > >>>
> > >>> Is the CTC option not available for ROC from VSDBs?
> > >>>
> > >>> For the ROC from the MET *.stat, I played with the OBS_THRESH
a
> > >>> bit, it appears that the ROC plot can accept up to two
different
> > >>> OBS_THRESH (as in the attached xml), but more than two will
result
> > >>> in
> > >>> an error.
> > >>>
> > >>> Thanks again,
> > >>>
> > >>> Ying
> > >>>
> > >>> On 08/27/2018 05:08 PM, Ying Lin wrote:
> > >>>> Thanks Tatiana.
> > >>>>
> > >>>> Sorry - AUR "Area Under ROC".
> > >>>>
> > >>>> I see your ROC plot and it seems the difference between your
xml
> > >>>> and my failed attempt is that you chose 'PCT' instead of
'CTC' for
> > >>>> ROC calculations. I thought that PCT is for ROC score of
> > >>>> probability
> > >>>> forecasts, while CTC is from the contingency table count of
single
> > >>>> deterministic models' ROC?
> > >>>>
> > >>>> Ying
> > >>>>
> > >>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
> > >>>>> Hi Ying,
> > >>>>> ROC plop is fixed and I was able to create a plot using your
XML.
> > >>>>> What do you mean by AUR?
> > >>>>>
> > >>>>> Tatiana
> > >>>>>
> > >>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
> > >>>>>> Hello. I'm trying to make a simple ROC plot. The attached
> > >>>>>> *.xml does
> > >>>>>> not work - err msg
> > >>>>>>
> > >>>>>> "Failed to run plot plot_20180827_184319 - reason: R error:
====
> > >>>>>> Start
> > >>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
> > >>>>>> subscript
> > >>>>>> out of bounds In addition."
> > >>>>>>
> > >>>>>> I verified that data for the forecast valid time/lead time
(24h
> > >>>>>> ending
> > >>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.
What
> > >>>>>> should
> > >>>>>> I be doing differently in the set up? I asked Ben Blake
about
> > >>>>>> this
> > >>>>>> (googling found his inquiries on ROC/AUR from early 2017)
and he
> > >>>>>> said to
> > >>>>>> put OBS_THRESH under series variables instead of fixed
values,
> > >>>>>> that
> > >>>>>> didn't work either.
> > >>>>>>
> > >>>>>> Also, once I get ROC to work ... where is AUR? Is it under
Roc
> > >>>>>> tab, or
> > >>>>>> Series? I can't figure out where it could be under Roc;
under
> > >>>>>> Series
> > >>>>>> there doesn't seem to be something that might be AUR under
> > >>>>>> "Select
> > >>>>>> attribute stat".
> > >>>>>>
> > >>>>>> Thanks!
> > >>>>>>
> > >>>>>> Ying
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >> --
> > >> Ying Lin
> > >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> > >> NCWCP Cubicle No. 2015
> > >> Ying.Lin at noaa.gov
> > >>
> > >>
> > >>
> > >>
> > >
>
>
>
>
------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Ying Lin
Time: Mon Sep 10 17:47:41 2018
Hi John (and Tressa),
Thanks! Yes, it's confirmed now that METviewer can make the
CTC-ROC plot. I ran grid_stat with the 10 identical obs_thresh and 10
varying fcst_thresh and did get a ROC "curve" - see attached. I'm
kind
of surprised that fcst_thresh numbers on the plot are in such perfect
order - expected some Brownian motion-like zig-zags rather than having
the fcst_thresh values in perfect descending order, just like in the
PCT-type ROC curves (is it possible that the connecting lines are
correct, but the point labels are wrong? I don't quite know how to
interpret the R data).
Anyway, interesting stuff (haven't thought of this way before)
but
probably of limited utility. Please go ahead and close this ticket if
you wish.
Ying
On 09/10/2018 04:25 PM, John Halley Gotway via RT wrote:
> Ying,
>
> I believe that METviewer would be able to make this plot now
(although I
> haven't confirmed that recently). But you would have to configure
MET in
> the right way to define the thresholds the way Tressa described.
>
> For example, let's say you're interested in 1/2 inch of precip
accumulation
> (i.e. 12.7 mm). You define the thresholds in the MET config file
like the
> following. I've listed the same obs threshold 10 times (>= 1/2
inch). But
> I've listed 10 different forecast thresholds (e.g. >=0,
>=0.1",...,>=1.0").
>
> fcst = {
> ...
> cat_thresh = [ ge0.00, ge2.54, ge5.08, ge7.62, ge10.16, ge12.7,
ge15.24,
> ge17.78, ge20.32, ge22.86, ge25.4 ];
> ...
> }
> obs = {
> ...
> cat_thresh = [ ge12.7, ge12.7, ge12.7, ge12.7, ge12.7, ge12.7,
ge12.7,
> ge12.7, ge12.7, ge12.7 ];
> ...
> }
>
> Setting up the config file in this way should enable METviewer to
generate
> a ROC curve that consists of 10 points... once for threshold pair
listed
> above. I can tell you that we have not typically done this in the
past.
>
> Does that make sense?
>
> John
>
> On Mon, Sep 10, 2018 at 1:55 PM Tressa Fowler via RT
<met_help at ucar.edu>
> wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>>
>> Hi Ying,
>>
>> It definitely makes sense to calculate AUC for CTC-derived ROC
curves, so
>> long as a single observation threshold is selected, and the
forecast
>> thresholds fill in the curve pretty well. I think we can create an
issue to
>> add this for future releases.
>>
>> Tressa
>>
>> On Sat Sep 08 16:37:02 2018, ying.lin at noaa.gov wrote:
>>> Hi John and Tressa,
>>>
>>> I was originally under the impression that the ROC/Area under ROC
was
>>> for probabilistic forecasts. Do they make sense/are they worth
>>> looking at in the non-probabilistic sense (e.g. ROC curve for
varying
>>> forecast/obs thresholds)? I guess a case can be made that the
CTC-ROC
>>> makes some sense (similar to the performance diagram), but unlike
the
>>> probabilistic ROC curve that rises with decreasing PCT, the "CTC
ROC"
>>> based on the F/H/O contingency table counts for different
thresholds
>>> (and probably for different forecast lead time as well) can be all
>>> over the place, making the concept of "area under ROC”
problematic.
>>>
>>> If a non-prob ROC doesn't make enough sense/is not worth pursuing,
>>> that's fine. The AUC was listed as one of the measures in the CAM
>>> metrics workshop, so I got curious when seeing the "CTC" option in
the
>>> METviewer ROC. That's all.
>>>
>>> Ying
>>>
>>>> On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
>>>> <met_help at ucar.edu> wrote:
>>>>
>>>> Hi Tressa,
>>>>
>>>> I was supposed to talk to you about this issue earlier this week
but
>>>> forgot. I think we need your guidance on this one.
>>>>
>>>> This is a met-help ticket from Ying Lin at NCEP about ROC curves
in
>>>> METviewer. I believe there are 2 ways to create a ROC curve in
>>>> METviewer:
>>>>
>>>> (1) The most common way I know is using the Nx2 Probabilistic
>>>> Contingency
>>>> Table counts from the PCT line in MET. I understand this one.
MET
>>>> can
>>>> also write out the PSTD line type which includes the ROC_AUC
column.
>>>> And
>>>> that's a number between 0 and 1 indicating the area under the ROC
>>>> curve,
>>>> with 1 being the best. MET calculates this value and it gets
loaded
>>>> into
>>>> METviewer. METviewer can plot the ROC_AUC value but it doesn't
>>>> compute it.
>>>>
>>>> (2) The less common way of generating a ROC curve is using
several
>>>> 2x2
>>>> contingency table lines. This would be the FHO or CTC line types
>>>> from
>>>> MET. However, there's some requirement in how those CTC lines
are
>>>> defined... but I can't remember exactly what. Perhaps they need
to
>>>> be
>>>> multiple forecast thresholds but all for the same observation
>>>> threshold?
>>>> Does that sound correct to you? If we get all the right, then
>>>> METviewer
>>>> could make a ROC plot. But, since METviewer doesn't compute the
area
>>>> under
>>>> the ROC curve, there's no way for METviewer to plot that
statistic.
>>>>
>>>> I think this would all be good info to clarify in the METviewer
>>>> user's
>>>> guide:
>>>> - Show an example of using PCT output to make a ROC curve in
>>>> METviewer.
>>>> - Show an example of using CTC output to make a ROC curve in
>>>> METviewer...
>>>> and list the threshold requirements for doing so.
>>>>
>>>> And lastly, we should identify what changes to METviewer are
needed
>>>> to make
>>>> it more useful to Ying. Perhaps we do need to enhance METviewer
to
>>>> compute
>>>> the area under the ROC curve for case (2)?
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>>>>>
>>>>> Hi Tatiana,
>>>>>
>>>>> Follow up on this ... has the ROC plotting in METviewer been
used
>>>>> in the non-probabilistic sense before? I'm thinking that maybe
>>>>> obs_thresh shouldn't be "fix values". The ROC plot seems to be
>>>>> analogous to the performance diagram, in which data points are
>>>>> produced
>>>>> for forecast lead times - but FCST_LEAD in the performance
diagrams
>>>>> are
>>>>> "independent variables" rather than fixed values. There is no
>>>>> "independent variables" option in ROC, the only way to make the
>>>>> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed
values,
>>>>> but
>>>>> then it seems only two OBS_THRESH values can be accepted by
>>>>> METviewer.
>>>>>
>>>>> Thanks -
>>>>>
>>>>> Ying
>>>>>
>>>>> On 08/27/2018 05:34 PM, Ying Lin wrote:
>>>>>> Hi Tatiana,
>>>>>>
>>>>>> I tried making a ROC plot from MET *.stat (the case of the
>>>>>> original question was from VSDBs loaded to MV) and the CTC
seems to
>>>>>> work - see attached xml.
>>>>>>
>>>>>> Is the CTC option not available for ROC from VSDBs?
>>>>>>
>>>>>> For the ROC from the MET *.stat, I played with the OBS_THRESH a
>>>>>> bit, it appears that the ROC plot can accept up to two
different
>>>>>> OBS_THRESH (as in the attached xml), but more than two will
result
>>>>>> in
>>>>>> an error.
>>>>>>
>>>>>> Thanks again,
>>>>>>
>>>>>> Ying
>>>>>>
>>>>>> On 08/27/2018 05:08 PM, Ying Lin wrote:
>>>>>>> Thanks Tatiana.
>>>>>>>
>>>>>>> Sorry - AUR "Area Under ROC".
>>>>>>>
>>>>>>> I see your ROC plot and it seems the difference between your
xml
>>>>>>> and my failed attempt is that you chose 'PCT' instead of 'CTC'
for
>>>>>>> ROC calculations. I thought that PCT is for ROC score of
>>>>>>> probability
>>>>>>> forecasts, while CTC is from the contingency table count of
single
>>>>>>> deterministic models' ROC?
>>>>>>>
>>>>>>> Ying
>>>>>>>
>>>>>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
>>>>>>>> Hi Ying,
>>>>>>>> ROC plop is fixed and I was able to create a plot using your
XML.
>>>>>>>> What do you mean by AUR?
>>>>>>>>
>>>>>>>> Tatiana
>>>>>>>>
>>>>>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>>>>>>>>> Hello. I'm trying to make a simple ROC plot. The attached
>>>>>>>>> *.xml does
>>>>>>>>> not work - err msg
>>>>>>>>>
>>>>>>>>> "Failed to run plot plot_20180827_184319 - reason: R error:
====
>>>>>>>>> Start
>>>>>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact) :
>>>>>>>>> subscript
>>>>>>>>> out of bounds In addition."
>>>>>>>>>
>>>>>>>>> I verified that data for the forecast valid time/lead time
(24h
>>>>>>>>> ending
>>>>>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from them.
What
>>>>>>>>> should
>>>>>>>>> I be doing differently in the set up? I asked Ben Blake
about
>>>>>>>>> this
>>>>>>>>> (googling found his inquiries on ROC/AUR from early 2017)
and he
>>>>>>>>> said to
>>>>>>>>> put OBS_THRESH under series variables instead of fixed
values,
>>>>>>>>> that
>>>>>>>>> didn't work either.
>>>>>>>>>
>>>>>>>>> Also, once I get ROC to work ... where is AUR? Is it under
Roc
>>>>>>>>> tab, or
>>>>>>>>> Series? I can't figure out where it could be under Roc;
under
>>>>>>>>> Series
>>>>>>>>> there doesn't seem to be something that might be AUR under
>>>>>>>>> "Select
>>>>>>>>> attribute stat".
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Ying
>>>>>>>>>
>>>>>>>>
>>>>> --
>>>>> Ying Lin
>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>> NCWCP Cubicle No. 2015
>>>>> Ying.Lin at noaa.gov
>>>>>
>>>>>
>>>>>
>>>>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: John Halley Gotway
Time: Mon Sep 10 18:06:13 2018
Hmm, I do think there’s a problem in that plot in the order in which
the
dots are connected. Probably need to ask Tatiana to generate that
plot in
the debugger and see what’s going on.
But Tatiana is out of town this week. We’ll need to check with her
next
week.
Can you please send me the MET output file(s) you’ve generated for us
to
test?
Thanks
John
On Mon, Sep 10, 2018 at 5:48 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>
> Hi John (and Tressa),
>
> Thanks! Yes, it's confirmed now that METviewer can make the
> CTC-ROC plot. I ran grid_stat with the 10 identical obs_thresh and
10
> varying fcst_thresh and did get a ROC "curve" - see attached. I'm
kind
> of surprised that fcst_thresh numbers on the plot are in such
perfect
> order - expected some Brownian motion-like zig-zags rather than
having
> the fcst_thresh values in perfect descending order, just like in the
> PCT-type ROC curves (is it possible that the connecting lines are
> correct, but the point labels are wrong? I don't quite know how to
> interpret the R data).
>
> Anyway, interesting stuff (haven't thought of this way before)
but
> probably of limited utility. Please go ahead and close this ticket
if
> you wish.
>
> Ying
>
> On 09/10/2018 04:25 PM, John Halley Gotway via RT wrote:
> > Ying,
> >
> > I believe that METviewer would be able to make this plot now
(although I
> > haven't confirmed that recently). But you would have to configure
MET in
> > the right way to define the thresholds the way Tressa described.
> >
> > For example, let's say you're interested in 1/2 inch of precip
> accumulation
> > (i.e. 12.7 mm). You define the thresholds in the MET config file
like
> the
> > following. I've listed the same obs threshold 10 times (>= 1/2
inch).
> But
> > I've listed 10 different forecast thresholds (e.g. >=0,
> >=0.1",...,>=1.0").
> >
> > fcst = {
> > ...
> > cat_thresh = [ ge0.00, ge2.54, ge5.08, ge7.62, ge10.16, ge12.7,
> ge15.24,
> > ge17.78, ge20.32, ge22.86, ge25.4 ];
> > ...
> > }
> > obs = {
> > ...
> > cat_thresh = [ ge12.7, ge12.7, ge12.7, ge12.7, ge12.7, ge12.7,
ge12.7,
> > ge12.7, ge12.7, ge12.7 ];
> > ...
> > }
> >
> > Setting up the config file in this way should enable METviewer to
> generate
> > a ROC curve that consists of 10 points... once for threshold pair
listed
> > above. I can tell you that we have not typically done this in the
past.
> >
> > Does that make sense?
> >
> > John
> >
> > On Mon, Sep 10, 2018 at 1:55 PM Tressa Fowler via RT
<met_help at ucar.edu>
> > wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
> >>
> >> Hi Ying,
> >>
> >> It definitely makes sense to calculate AUC for CTC-derived ROC
curves,
> so
> >> long as a single observation threshold is selected, and the
forecast
> >> thresholds fill in the curve pretty well. I think we can create
an
> issue to
> >> add this for future releases.
> >>
> >> Tressa
> >>
> >> On Sat Sep 08 16:37:02 2018, ying.lin at noaa.gov wrote:
> >>> Hi John and Tressa,
> >>>
> >>> I was originally under the impression that the ROC/Area under
ROC was
> >>> for probabilistic forecasts. Do they make sense/are they worth
> >>> looking at in the non-probabilistic sense (e.g. ROC curve for
varying
> >>> forecast/obs thresholds)? I guess a case can be made that the
CTC-ROC
> >>> makes some sense (similar to the performance diagram), but
unlike the
> >>> probabilistic ROC curve that rises with decreasing PCT, the "CTC
ROC"
> >>> based on the F/H/O contingency table counts for different
thresholds
> >>> (and probably for different forecast lead time as well) can be
all
> >>> over the place, making the concept of "area under ROC”
problematic.
> >>>
> >>> If a non-prob ROC doesn't make enough sense/is not worth
pursuing,
> >>> that's fine. The AUC was listed as one of the measures in the
CAM
> >>> metrics workshop, so I got curious when seeing the "CTC" option
in the
> >>> METviewer ROC. That's all.
> >>>
> >>> Ying
> >>>
> >>>> On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
> >>>> <met_help at ucar.edu> wrote:
> >>>>
> >>>> Hi Tressa,
> >>>>
> >>>> I was supposed to talk to you about this issue earlier this
week but
> >>>> forgot. I think we need your guidance on this one.
> >>>>
> >>>> This is a met-help ticket from Ying Lin at NCEP about ROC
curves in
> >>>> METviewer. I believe there are 2 ways to create a ROC curve in
> >>>> METviewer:
> >>>>
> >>>> (1) The most common way I know is using the Nx2 Probabilistic
> >>>> Contingency
> >>>> Table counts from the PCT line in MET. I understand this one.
MET
> >>>> can
> >>>> also write out the PSTD line type which includes the ROC_AUC
column.
> >>>> And
> >>>> that's a number between 0 and 1 indicating the area under the
ROC
> >>>> curve,
> >>>> with 1 being the best. MET calculates this value and it gets
loaded
> >>>> into
> >>>> METviewer. METviewer can plot the ROC_AUC value but it doesn't
> >>>> compute it.
> >>>>
> >>>> (2) The less common way of generating a ROC curve is using
several
> >>>> 2x2
> >>>> contingency table lines. This would be the FHO or CTC line
types
> >>>> from
> >>>> MET. However, there's some requirement in how those CTC lines
are
> >>>> defined... but I can't remember exactly what. Perhaps they
need to
> >>>> be
> >>>> multiple forecast thresholds but all for the same observation
> >>>> threshold?
> >>>> Does that sound correct to you? If we get all the right, then
> >>>> METviewer
> >>>> could make a ROC plot. But, since METviewer doesn't compute
the area
> >>>> under
> >>>> the ROC curve, there's no way for METviewer to plot that
statistic.
> >>>>
> >>>> I think this would all be good info to clarify in the METviewer
> >>>> user's
> >>>> guide:
> >>>> - Show an example of using PCT output to make a ROC curve in
> >>>> METviewer.
> >>>> - Show an example of using CTC output to make a ROC curve in
> >>>> METviewer...
> >>>> and list the threshold requirements for doing so.
> >>>>
> >>>> And lastly, we should identify what changes to METviewer are
needed
> >>>> to make
> >>>> it more useful to Ying. Perhaps we do need to enhance
METviewer to
> >>>> compute
> >>>> the area under the ROC curve for case (2)?
> >>>>
> >>>> Thanks,
> >>>> John
> >>>>
> >>>> On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT
<met_help at ucar.edu>
> >>>> wrote:
> >>>>
> >>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767
>
> >>>>>
> >>>>> Hi Tatiana,
> >>>>>
> >>>>> Follow up on this ... has the ROC plotting in METviewer been
used
> >>>>> in the non-probabilistic sense before? I'm thinking that
maybe
> >>>>> obs_thresh shouldn't be "fix values". The ROC plot seems to
be
> >>>>> analogous to the performance diagram, in which data points are
> >>>>> produced
> >>>>> for forecast lead times - but FCST_LEAD in the performance
diagrams
> >>>>> are
> >>>>> "independent variables" rather than fixed values. There is no
> >>>>> "independent variables" option in ROC, the only way to make
the
> >>>>> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed
values,
> >>>>> but
> >>>>> then it seems only two OBS_THRESH values can be accepted by
> >>>>> METviewer.
> >>>>>
> >>>>> Thanks -
> >>>>>
> >>>>> Ying
> >>>>>
> >>>>> On 08/27/2018 05:34 PM, Ying Lin wrote:
> >>>>>> Hi Tatiana,
> >>>>>>
> >>>>>> I tried making a ROC plot from MET *.stat (the case of the
> >>>>>> original question was from VSDBs loaded to MV) and the CTC
seems to
> >>>>>> work - see attached xml.
> >>>>>>
> >>>>>> Is the CTC option not available for ROC from VSDBs?
> >>>>>>
> >>>>>> For the ROC from the MET *.stat, I played with the OBS_THRESH
a
> >>>>>> bit, it appears that the ROC plot can accept up to two
different
> >>>>>> OBS_THRESH (as in the attached xml), but more than two will
result
> >>>>>> in
> >>>>>> an error.
> >>>>>>
> >>>>>> Thanks again,
> >>>>>>
> >>>>>> Ying
> >>>>>>
> >>>>>> On 08/27/2018 05:08 PM, Ying Lin wrote:
> >>>>>>> Thanks Tatiana.
> >>>>>>>
> >>>>>>> Sorry - AUR "Area Under ROC".
> >>>>>>>
> >>>>>>> I see your ROC plot and it seems the difference between your
xml
> >>>>>>> and my failed attempt is that you chose 'PCT' instead of
'CTC' for
> >>>>>>> ROC calculations. I thought that PCT is for ROC score of
> >>>>>>> probability
> >>>>>>> forecasts, while CTC is from the contingency table count of
single
> >>>>>>> deterministic models' ROC?
> >>>>>>>
> >>>>>>> Ying
> >>>>>>>
> >>>>>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
> >>>>>>>> Hi Ying,
> >>>>>>>> ROC plop is fixed and I was able to create a plot using
your XML.
> >>>>>>>> What do you mean by AUR?
> >>>>>>>>
> >>>>>>>> Tatiana
> >>>>>>>>
> >>>>>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
> >>>>>>>>> Hello. I'm trying to make a simple ROC plot. The
attached
> >>>>>>>>> *.xml does
> >>>>>>>>> not work - err msg
> >>>>>>>>>
> >>>>>>>>> "Failed to run plot plot_20180827_184319 - reason: R
error: ====
> >>>>>>>>> Start
> >>>>>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact)
:
> >>>>>>>>> subscript
> >>>>>>>>> out of bounds In addition."
> >>>>>>>>>
> >>>>>>>>> I verified that data for the forecast valid time/lead time
(24h
> >>>>>>>>> ending
> >>>>>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from
them. What
> >>>>>>>>> should
> >>>>>>>>> I be doing differently in the set up? I asked Ben Blake
about
> >>>>>>>>> this
> >>>>>>>>> (googling found his inquiries on ROC/AUR from early 2017)
and he
> >>>>>>>>> said to
> >>>>>>>>> put OBS_THRESH under series variables instead of fixed
values,
> >>>>>>>>> that
> >>>>>>>>> didn't work either.
> >>>>>>>>>
> >>>>>>>>> Also, once I get ROC to work ... where is AUR? Is it
under Roc
> >>>>>>>>> tab, or
> >>>>>>>>> Series? I can't figure out where it could be under Roc;
under
> >>>>>>>>> Series
> >>>>>>>>> there doesn't seem to be something that might be AUR under
> >>>>>>>>> "Select
> >>>>>>>>> attribute stat".
> >>>>>>>>>
> >>>>>>>>> Thanks!
> >>>>>>>>>
> >>>>>>>>> Ying
> >>>>>>>>>
> >>>>>>>>
> >>>>> --
> >>>>> Ying Lin
> >>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>>> NCWCP Cubicle No. 2015
> >>>>> Ying.Lin at noaa.gov
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Ying Lin
Time: Mon Sep 10 18:53:14 2018
Hi John, attached is the *.stat file I loaded to METviewer to generate
the ROC 'curve'. Thanks!
Ying
On 09/10/2018 08:07 PM, John Halley Gotway via RT wrote:
> Hmm, I do think there’s a problem in that plot in the order in which
the
> dots are connected. Probably need to ask Tatiana to generate that
plot in
> the debugger and see what’s going on.
>
> But Tatiana is out of town this week. We’ll need to check with her
next
> week.
>
> Can you please send me the MET output file(s) you’ve generated for
us to
> test?
>
> Thanks
> John
>
> On Mon, Sep 10, 2018 at 5:48 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>>
>> Hi John (and Tressa),
>>
>> Thanks! Yes, it's confirmed now that METviewer can make the
>> CTC-ROC plot. I ran grid_stat with the 10 identical obs_thresh and
10
>> varying fcst_thresh and did get a ROC "curve" - see attached. I'm
kind
>> of surprised that fcst_thresh numbers on the plot are in such
perfect
>> order - expected some Brownian motion-like zig-zags rather than
having
>> the fcst_thresh values in perfect descending order, just like in
the
>> PCT-type ROC curves (is it possible that the connecting lines are
>> correct, but the point labels are wrong? I don't quite know how to
>> interpret the R data).
>>
>> Anyway, interesting stuff (haven't thought of this way
before) but
>> probably of limited utility. Please go ahead and close this ticket
if
>> you wish.
>>
>> Ying
>>
>> On 09/10/2018 04:25 PM, John Halley Gotway via RT wrote:
>>> Ying,
>>>
>>> I believe that METviewer would be able to make this plot now
(although I
>>> haven't confirmed that recently). But you would have to configure
MET in
>>> the right way to define the thresholds the way Tressa described.
>>>
>>> For example, let's say you're interested in 1/2 inch of precip
>> accumulation
>>> (i.e. 12.7 mm). You define the thresholds in the MET config file
like
>> the
>>> following. I've listed the same obs threshold 10 times (>= 1/2
inch).
>> But
>>> I've listed 10 different forecast thresholds (e.g. >=0,
>>> =0.1",...,>=1.0").
>>>
>>> fcst = {
>>> ...
>>> cat_thresh = [ ge0.00, ge2.54, ge5.08, ge7.62, ge10.16,
ge12.7,
>> ge15.24,
>>> ge17.78, ge20.32, ge22.86, ge25.4 ];
>>> ...
>>> }
>>> obs = {
>>> ...
>>> cat_thresh = [ ge12.7, ge12.7, ge12.7, ge12.7, ge12.7, ge12.7,
ge12.7,
>>> ge12.7, ge12.7, ge12.7 ];
>>> ...
>>> }
>>>
>>> Setting up the config file in this way should enable METviewer to
>> generate
>>> a ROC curve that consists of 10 points... once for threshold pair
listed
>>> above. I can tell you that we have not typically done this in the
past.
>>>
>>> Does that make sense?
>>>
>>> John
>>>
>>> On Mon, Sep 10, 2018 at 1:55 PM Tressa Fowler via RT
<met_help at ucar.edu>
>>> wrote:
>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767 >
>>>>
>>>> Hi Ying,
>>>>
>>>> It definitely makes sense to calculate AUC for CTC-derived ROC
curves,
>> so
>>>> long as a single observation threshold is selected, and the
forecast
>>>> thresholds fill in the curve pretty well. I think we can create
an
>> issue to
>>>> add this for future releases.
>>>>
>>>> Tressa
>>>>
>>>> On Sat Sep 08 16:37:02 2018, ying.lin at noaa.gov wrote:
>>>>> Hi John and Tressa,
>>>>>
>>>>> I was originally under the impression that the ROC/Area under
ROC was
>>>>> for probabilistic forecasts. Do they make sense/are they worth
>>>>> looking at in the non-probabilistic sense (e.g. ROC curve for
varying
>>>>> forecast/obs thresholds)? I guess a case can be made that the
CTC-ROC
>>>>> makes some sense (similar to the performance diagram), but
unlike the
>>>>> probabilistic ROC curve that rises with decreasing PCT, the "CTC
ROC"
>>>>> based on the F/H/O contingency table counts for different
thresholds
>>>>> (and probably for different forecast lead time as well) can be
all
>>>>> over the place, making the concept of "area under ROC”
problematic.
>>>>>
>>>>> If a non-prob ROC doesn't make enough sense/is not worth
pursuing,
>>>>> that's fine. The AUC was listed as one of the measures in the
CAM
>>>>> metrics workshop, so I got curious when seeing the "CTC" option
in the
>>>>> METviewer ROC. That's all.
>>>>>
>>>>> Ying
>>>>>
>>>>>> On Sep 7, 2018, at 7:00 PM, John Halley Gotway via RT
>>>>>> <met_help at ucar.edu> wrote:
>>>>>>
>>>>>> Hi Tressa,
>>>>>>
>>>>>> I was supposed to talk to you about this issue earlier this
week but
>>>>>> forgot. I think we need your guidance on this one.
>>>>>>
>>>>>> This is a met-help ticket from Ying Lin at NCEP about ROC
curves in
>>>>>> METviewer. I believe there are 2 ways to create a ROC curve in
>>>>>> METviewer:
>>>>>>
>>>>>> (1) The most common way I know is using the Nx2 Probabilistic
>>>>>> Contingency
>>>>>> Table counts from the PCT line in MET. I understand this one.
MET
>>>>>> can
>>>>>> also write out the PSTD line type which includes the ROC_AUC
column.
>>>>>> And
>>>>>> that's a number between 0 and 1 indicating the area under the
ROC
>>>>>> curve,
>>>>>> with 1 being the best. MET calculates this value and it gets
loaded
>>>>>> into
>>>>>> METviewer. METviewer can plot the ROC_AUC value but it doesn't
>>>>>> compute it.
>>>>>>
>>>>>> (2) The less common way of generating a ROC curve is using
several
>>>>>> 2x2
>>>>>> contingency table lines. This would be the FHO or CTC line
types
>>>>>> from
>>>>>> MET. However, there's some requirement in how those CTC lines
are
>>>>>> defined... but I can't remember exactly what. Perhaps they
need to
>>>>>> be
>>>>>> multiple forecast thresholds but all for the same observation
>>>>>> threshold?
>>>>>> Does that sound correct to you? If we get all the right, then
>>>>>> METviewer
>>>>>> could make a ROC plot. But, since METviewer doesn't compute
the area
>>>>>> under
>>>>>> the ROC curve, there's no way for METviewer to plot that
statistic.
>>>>>>
>>>>>> I think this would all be good info to clarify in the METviewer
>>>>>> user's
>>>>>> guide:
>>>>>> - Show an example of using PCT output to make a ROC curve in
>>>>>> METviewer.
>>>>>> - Show an example of using CTC output to make a ROC curve in
>>>>>> METviewer...
>>>>>> and list the threshold requirements for doing so.
>>>>>>
>>>>>> And lastly, we should identify what changes to METviewer are
needed
>>>>>> to make
>>>>>> it more useful to Ying. Perhaps we do need to enhance
METviewer to
>>>>>> compute
>>>>>> the area under the ROC curve for case (2)?
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>> On Tue, Aug 28, 2018 at 12:56 PM Ying Lin via RT
<met_help at ucar.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86767
>
>>>>>>>
>>>>>>> Hi Tatiana,
>>>>>>>
>>>>>>> Follow up on this ... has the ROC plotting in METviewer been
used
>>>>>>> in the non-probabilistic sense before? I'm thinking that
maybe
>>>>>>> obs_thresh shouldn't be "fix values". The ROC plot seems to
be
>>>>>>> analogous to the performance diagram, in which data points are
>>>>>>> produced
>>>>>>> for forecast lead times - but FCST_LEAD in the performance
diagrams
>>>>>>> are
>>>>>>> "independent variables" rather than fixed values. There is no
>>>>>>> "independent variables" option in ROC, the only way to make
the
>>>>>>> "CTC-type" ROC plot seems to be to set OBS_THRESH as fixed
values,
>>>>>>> but
>>>>>>> then it seems only two OBS_THRESH values can be accepted by
>>>>>>> METviewer.
>>>>>>>
>>>>>>> Thanks -
>>>>>>>
>>>>>>> Ying
>>>>>>>
>>>>>>> On 08/27/2018 05:34 PM, Ying Lin wrote:
>>>>>>>> Hi Tatiana,
>>>>>>>>
>>>>>>>> I tried making a ROC plot from MET *.stat (the case of the
>>>>>>>> original question was from VSDBs loaded to MV) and the CTC
seems to
>>>>>>>> work - see attached xml.
>>>>>>>>
>>>>>>>> Is the CTC option not available for ROC from VSDBs?
>>>>>>>>
>>>>>>>> For the ROC from the MET *.stat, I played with the OBS_THRESH
a
>>>>>>>> bit, it appears that the ROC plot can accept up to two
different
>>>>>>>> OBS_THRESH (as in the attached xml), but more than two will
result
>>>>>>>> in
>>>>>>>> an error.
>>>>>>>>
>>>>>>>> Thanks again,
>>>>>>>>
>>>>>>>> Ying
>>>>>>>>
>>>>>>>> On 08/27/2018 05:08 PM, Ying Lin wrote:
>>>>>>>>> Thanks Tatiana.
>>>>>>>>>
>>>>>>>>> Sorry - AUR "Area Under ROC".
>>>>>>>>>
>>>>>>>>> I see your ROC plot and it seems the difference between your
xml
>>>>>>>>> and my failed attempt is that you chose 'PCT' instead of
'CTC' for
>>>>>>>>> ROC calculations. I thought that PCT is for ROC score of
>>>>>>>>> probability
>>>>>>>>> forecasts, while CTC is from the contingency table count of
single
>>>>>>>>> deterministic models' ROC?
>>>>>>>>>
>>>>>>>>> Ying
>>>>>>>>>
>>>>>>>>> On 08/27/2018 04:43 PM, Tatiana Burek via RT wrote:
>>>>>>>>>> Hi Ying,
>>>>>>>>>> ROC plop is fixed and I was able to create a plot using
your XML.
>>>>>>>>>> What do you mean by AUR?
>>>>>>>>>>
>>>>>>>>>> Tatiana
>>>>>>>>>>
>>>>>>>>>> On Mon Aug 27 13:14:48 2018, ying.lin at noaa.gov wrote:
>>>>>>>>>>> Hello. I'm trying to make a simple ROC plot. The
attached
>>>>>>>>>>> *.xml does
>>>>>>>>>>> not work - err msg
>>>>>>>>>>>
>>>>>>>>>>> "Failed to run plot plot_20180827_184319 - reason: R
error: ====
>>>>>>>>>>> Start
>>>>>>>>>>> Rscript error ==== Error in .subset2(x, i, exact = exact)
:
>>>>>>>>>>> subscript
>>>>>>>>>>> out of bounds In addition."
>>>>>>>>>>>
>>>>>>>>>>> I verified that data for the forecast valid time/lead time
(24h
>>>>>>>>>>> ending
>>>>>>>>>>> at 12Z 21 Aug) do exist and I could plot POD/FAR from
them. What
>>>>>>>>>>> should
>>>>>>>>>>> I be doing differently in the set up? I asked Ben Blake
about
>>>>>>>>>>> this
>>>>>>>>>>> (googling found his inquiries on ROC/AUR from early 2017)
and he
>>>>>>>>>>> said to
>>>>>>>>>>> put OBS_THRESH under series variables instead of fixed
values,
>>>>>>>>>>> that
>>>>>>>>>>> didn't work either.
>>>>>>>>>>>
>>>>>>>>>>> Also, once I get ROC to work ... where is AUR? Is it
under Roc
>>>>>>>>>>> tab, or
>>>>>>>>>>> Series? I can't figure out where it could be under Roc;
under
>>>>>>>>>>> Series
>>>>>>>>>>> there doesn't seem to be something that might be AUR under
>>>>>>>>>>> "Select
>>>>>>>>>>> attribute stat".
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> Ying
>>>>>>>>>>>
>>>>>>> --
>>>>>>> Ying Lin
>>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>>> NCWCP Cubicle No. 2015
>>>>>>> Ying.Lin at noaa.gov
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: ROC/Area under ROC in METviewer
From: Ying Lin
Time: Mon Sep 10 18:53:14 2018
VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR
OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >0.254 >12.700 NA
NA FHO 70390 0.32962 0.040958 0.052479
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >0.254 >12.700 NA
NA CTC 70390 2883 20319 811 46377
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >0.254 >12.700 NA
0.01 CTS 70390 0.052479 0.050356 0.054686 NA
NA 0.32962 0.32507 0.3342 NA NA 0.69982
0.69535 0.70425 NA NA 6.281 NA NA 0.78045
0.77641 0.78445 NA NA 0.69535 0.69086 0.6998 NA
NA 0.30465 0.3002 0.30914 NA NA 0.87574
0.87251 0.87891 NA NA 0.12006 0.11694 0.12325 NA
NA 0.073058 NA NA 0.4758 0.45779 0.49382 NA NA 0.13617 NA
NA 8.1138 7.30759 9.00895 NA NA 2.09357 1.98891 2.19822 NA NA
0.78055 0.76011 0.801 NA NA 0.84484 0.83187 0.85782 NA NA
0.26976 0.26082 0.27869 NA NA 0.65488 0.60978 0.69997 NA NA 0.63135
0.60883 0.70093 NA NA 0.08347 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >0.254 >12.700 NA
NA ECLV 70390 0.052479 0.4758 19
0.05 0.46432 0.1 0.16928 0.15 -0.19023 0.2
-0.59468 0.25 -1.05306 0.3 -1.57692 0.35 -2.18138
0.4 -2.88657 0.45 -3.71999 0.5 -4.72009 0.55 -5.94243
0.6 -7.47036 0.65 -9.43484 0.7 -12.05414 0.75
-15.72117 0.8 -21.22171 0.85 -30.38928 0.9 -48.72442 0.95
-103.72983
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >2.540 >12.700 NA
NA FHO 70390 0.19784 0.031539 0.052479
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >2.540 >12.700 NA
NA CTC 70390 2220 11706 1474 54990
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >2.540 >12.700 NA
0.01 CTS 70390 0.052479 0.050356 0.054686 NA
NA 0.19784 0.194 0.20174 NA NA 0.81276
0.80894 0.81652 NA NA 3.7699 NA NA 0.60097
0.59621 0.60572 NA NA 0.82449 0.82076 0.82815 NA
NA 0.17551 0.17185 0.17924 NA NA 0.84059
0.837 0.84411 NA NA 0.14416 0.14078 0.1476 NA
NA 0.10152 NA NA 0.42546 0.40416 0.44677 NA NA 0.18432 NA
NA 7.07507 6.46335 7.74468 NA NA 1.95658 1.86615 2.04701 NA NA
0.75232 0.7327 0.77195 NA NA 0.70537 0.68833 0.72241 NA NA
0.32145 0.30824 0.33465 NA NA 0.54722 0.513 0.58145 NA NA 0.58215
0.51282 0.58162 NA NA 0.088083 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >2.540 >12.700 NA
NA ECLV 70390 0.052479 0.42546 19
0.05 0.40458 0.1 0.24887 0.15 0.041753 0.2
-0.19126 0.25 -0.45533 0.3 -0.75714 0.35 -1.10537
0.4 -1.51164 0.45 -1.99178 0.5 -2.56795 0.55 -3.27215
0.6 -4.15241 0.65 -5.28417 0.7 -6.79318 0.75
-8.90579 0.8 -12.07472 0.85 -17.35625 0.9 -27.91933 0.95
-59.60855
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >3.810 >12.700 NA
NA FHO 70390 0.17254 0.028782 0.052479
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >3.810 >12.700 NA
NA CTC 70390 2026 10119 1668 56577
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >3.810 >12.700 NA
0.01 CTS 70390 0.052479 0.050356 0.054686 NA
NA 0.17254 0.1689 0.17624 NA NA 0.83255
0.82889 0.83614 NA NA 3.28776 NA NA 0.54846
0.54362 0.55328 NA NA 0.84828 0.84477 0.85173 NA
NA 0.15172 0.14827 0.15523 NA NA 0.83318
0.82953 0.83677 NA NA 0.14667 0.14327 0.15014 NA
NA 0.10539 NA NA 0.39674 0.37509 0.41839 NA NA 0.19069 NA
NA 6.79119 6.20925 7.42766 NA NA 1.91563 1.82604 2.00521 NA NA
0.7433 0.72325 0.76334 NA NA 0.66142 0.64341 0.67942 NA NA
0.32596 0.31158 0.34033 NA NA 0.51685 0.48431 0.54939 NA NA 0.5559
0.48368 0.55002 NA NA 0.088189 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >3.810 >12.700 NA
NA ECLV 70390 0.052479 0.39674 19
0.05 0.37311 0.1 0.24409 0.15 0.06505 0.2
-0.13637 0.25 -0.36465 0.3 -0.62553 0.35 -0.92655
0.4 -1.27775 0.45 -1.69279 0.5 -2.19085 0.55 -2.79958
0.6 -3.5605 0.65 -4.53883 0.7 -5.84326 0.75
-7.66946 0.8 -10.40877 0.85 -14.97428 0.9 -24.10531 0.95
-51.49838
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >6.350 >12.700 NA
NA FHO 70390 0.13729 0.024648 0.052479
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >6.350 >12.700 NA
NA CTC 70390 1735 7929 1959 58767
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >6.350 >12.700 NA
0.01 CTS 70390 0.052479 0.050356 0.054686 NA
NA 0.13729 0.13399 0.14067 NA NA 0.85953
0.85612 0.86287 NA NA 2.61613 NA NA 0.46968
0.46484 0.47453 NA NA 0.88112 0.87794 0.88422 NA
NA 0.11888 0.11578 0.12206 NA NA 0.82047
0.81671 0.82416 NA NA 0.14927 0.14585 0.15277 NA
NA 0.11046 NA NA 0.3508 0.32909 0.37251 NA NA 0.19894 NA
NA 6.56417 5.99718 7.18477 NA NA 1.88163 1.79129 1.97196 NA NA
0.7356 0.71487 0.75632 NA NA 0.59185 0.57249 0.61121 NA NA
0.33214 0.31594 0.34834 NA NA 0.47617 0.44589 0.50646 NA NA 0.51606
0.44462 0.50772 NA NA 0.089864 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >6.350 >12.700 NA
NA ECLV 70390 0.052479 0.3508 19
0.05 0.32305 0.1 0.23119 0.15 0.090895 0.2
-0.066933 0.25 -0.2458 0.3 -0.45023 0.35 -0.6861
0.4 -0.96129 0.45 -1.28651 0.5 -1.67677 0.55 -2.15376
0.6 -2.75 0.65 -3.51659 0.7 -4.53871 0.75
-5.96968 0.8 -8.11613 0.85 -11.69356 0.9 -18.8484 0.95
-40.31294
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >12.700 >12.700 NA
NA FHO 70390 0.082952 0.017005 0.052479
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >12.700 >12.700 NA
NA CTC 70390 1197 4642 2497 62054
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >12.700 >12.700 NA
0.01 CTS 70390 0.052479 0.050356 0.054686 NA
NA 0.082952 0.080314 0.085669 NA NA 0.89858
0.89561 0.90147 NA NA 1.58067 NA NA 0.32404
0.31951 0.3286 NA NA 0.9304 0.92789 0.93283 NA
NA 0.069599 0.067169 0.072111 NA NA 0.795
0.79105 0.79889 NA NA 0.14359 0.14022 0.14703 NA
NA 0.11091 NA NA 0.25444 0.23408 0.2748 NA NA 0.19968 NA
NA 6.40826 5.80614 7.07283 NA NA 1.85759 1.75892 1.95626 NA NA
0.73003 0.70699 0.75307 NA NA 0.44682 0.42508 0.46856 NA NA
0.33444 0.31439 0.35449 NA NA 0.40563 0.37834 0.43292 NA NA 0.4365
0.37646 0.4348 NA NA 0.09495 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >12.700 >12.700 NA
NA ECLV 70390 0.052479 0.25444 19
0.05 0.21907 0.1 0.18441 0.15 0.10228 0.2
0.0098809 0.25 -0.094838 0.3 -0.21452 0.35 -0.35261
0.4 -0.51372 0.45 -0.70411 0.5 -0.93259 0.55 -1.21185
0.6 -1.56091 0.65 -2.00971 0.7 -2.6081 0.75
-3.44586 0.8 -4.70249 0.85 -6.79688 0.9 -10.98565 0.95
-23.55198
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >19.05 >12.700 NA
NA FHO 70390 0.050646 0.011593 0.052479
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >19.05 >12.700 NA
NA CTC 70390 816 2749 2878 63947
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >19.05 >12.700 NA
0.01 CTS 70390 0.052479 0.050356 0.054686 NA
NA 0.050646 0.04856 0.052818 NA NA 0.92006
0.91739 0.92265 NA NA 0.96508 NA NA 0.2209
0.2169 0.22495 NA NA 0.95878 0.95681 0.96067 NA
NA 0.041217 0.03933 0.043191 NA NA 0.77111
0.767 0.77516 NA NA 0.12665 0.12346 0.12991 NA
NA 0.10053 NA NA 0.17968 0.16163 0.19774 NA NA 0.1827 NA
NA 6.59546 5.88594 7.3905 NA NA 1.88638 1.77257 2.0002 NA NA
0.73668 0.71066 0.76271 NA NA 0.32245 0.29884 0.34606 NA NA
0.33043 0.30667 0.35418 NA NA 0.35728 0.33121 0.38336 NA NA 0.37798
0.32941 0.38515 NA NA 0.10227 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >19.05 >12.700 NA
NA ECLV 70390 0.052479 0.17968 19
0.05 0.13891 0.1 0.13821 0.15 0.089573 0.2
0.034854 0.25 -0.027161 0.3 -0.098035 0.35 -0.17981
0.4 -0.27522 0.45 -0.38798 0.5 -0.52328 0.55 -0.68865
0.6 -0.89537 0.65 -1.16115 0.7 -1.51552 0.75
-2.01164 0.8 -2.75582 0.85 -3.99612 0.9 -6.47672 0.95
-13.91852
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >25.400 >12.700 NA
NA FHO 70390 0.030913 0.0075153 0.052479
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >25.400 >12.700 NA
NA CTC 70390 529 1647 3165 65049
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >25.400 >12.700 NA
0.01 CTS 70390 0.052479 0.050356 0.054686 NA
NA 0.030913 0.029277 0.032639 NA NA 0.93164
0.92915 0.93405 NA NA 0.58906 NA NA 0.14321
0.13984 0.14664 NA NA 0.97531 0.97375 0.97677 NA
NA 0.024694 0.023232 0.026246 NA NA 0.75689
0.7527 0.76103 NA NA 0.099045 0.096183 0.10198 NA
NA 0.079361 NA NA 0.11851 0.10326 0.13376 NA NA 0.14705 NA
NA 6.60129 5.75613 7.57055 NA NA 1.88727 1.75026 2.02427 NA NA
0.73689 0.70558 0.76819 NA NA 0.20525 0.17971 0.2308 NA NA
0.31346 0.28562 0.3413 NA NA 0.31139 0.2852 0.33759 NA NA 0.32404
0.28371 0.33908 NA NA 0.10565 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >25.400 >12.700 NA
NA ECLV 70390 0.052479 0.11851 19
0.05 0.073678 0.1 0.093665 0.15 0.064524 0.2
0.031741 0.25 -0.0054142 0.3 -0.047877 0.35 -0.096872
0.4 -0.15403 0.45 -0.22159 0.5 -0.30265 0.55 -0.40173
0.6 -0.52558 0.65 -0.68482 0.7 -0.89713 0.75
-1.19437 0.8 -1.64023 0.85 -2.38332 0.9 -3.86952 0.95
-8.3281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >38.1 >12.700 NA
NA FHO 70390 0.011607 0.0028413 0.052479
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >38.1 >12.700 NA
NA CTC 70390 200 617 3494 66079
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >38.1 >12.700 NA
0.01 CTS 70390 0.052479 0.050356 0.054686 NA
NA 0.011607 0.010612 0.012694 NA NA 0.9416
0.93928 0.94383 NA NA 0.22117 NA NA
0.054142 0.051987 0.056381 NA NA 0.99075 0.98977
0.99163 NA NA 0.0092509 0.0083666 0.010228 NA NA
0.7552 0.751 0.75935 NA NA 0.046393 0.044393
0.048478 NA NA 0.036814 NA NA 0.044891 0.035014 0.054768 NA NA
0.071013 NA NA 6.13035 4.94781 7.59552 NA NA 1.81325 1.59895
2.02756 NA NA 0.71951 0.66783 0.77119 NA NA 0.00532 -0.025051
0.035691 NA NA 0.26265 0.2245 0.30079 NA NA 0.23251 0.20276 0.26226
NA NA 0.23659 0.20189 0.26313 NA NA 0.10161 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >38.1 >12.700 NA
NA ECLV 70390 0.052479 0.044891 19
0.05 -0.004603 0.1 0.035583 0.15 0.024666 0.2
0.012385 0.25 -0.001534 0.3 -0.017441 0.35 -0.035796
0.4 -0.05721 0.45 -0.082517 0.5 -0.11289 0.55 -0.15
0.6 -0.1964 0.65 -0.25605 0.7 -0.33559 0.75
-0.44694 0.8 -0.61397 0.85 -0.89235 0.9 -1.44911 0.95
-3.11938
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >50.8 >12.700 NA
NA FHO 70390 0.0049013 0.0011223 0.052479
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >50.8 >12.700 NA
NA CTC 70390 79 266 3615 66430
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >50.8 >12.700 NA
0.01 CTS 70390 0.052479 0.050356 0.054686 NA
NA 0.0049013 0.0042683 0.0056275 NA NA 0.94486
0.94261 0.94704 NA NA 0.093395 NA NA
0.021386 0.020026 0.022836 NA NA 0.99601 0.99535
0.99658 NA NA 0.0039882 0.0034213 0.0046487 NA NA
0.77101 0.76691 0.77507 NA NA 0.019949 0.018637
0.021353 NA NA 0.015448 NA NA 0.017398 0.011046 0.023749 NA NA
0.030426 NA NA 5.4576 3.91199 7.61387 NA NA 1.69701 1.36405
2.02997 NA NA 0.69029 0.60313 0.77744 NA NA -0.13216 -0.16879
-0.09553 NA NA 0.2169 0.16554 0.26826 NA NA 0.17924 0.14261
0.21587 NA NA 0.18063 0.14212 0.21636 NA NA 0.09136 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >50.8 >12.700 NA
NA ECLV 70390 0.052479 0.017398 19
0.05 -0.03381 0.1 0.013385 0.15 0.0086786 0.2
0.0033839 0.25 -0.0026169 0.3 -0.0094748 0.35 -0.017388
0.4 -0.02662 0.45 -0.03753 0.5 -0.050623 0.55 -0.066625
0.6 -0.086627 0.65 -0.11234 0.7 -0.14663 0.75
-0.19464 0.8 -0.26665 0.85 -0.38666 0.9 -0.62669 0.95
-1.34678
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >76.2 >12.700 NA
NA FHO 70390 0.00098025 0.00026992 0.052479
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >76.2 >12.700 NA
NA CTC 70390 19 50 3675 66646
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >76.2 >12.700 NA
0.01 CTS 70390 0.052479 0.050356 0.054686 NA
NA 0.00098025 0.00071986 0.0013347 NA NA 0.94708
0.94486 0.94921 NA NA 0.018679 NA NA
0.0051435 0.0044941 0.0058861 NA NA 0.99925 0.99893
0.99947 NA NA 0.00074967 0.00052688 0.0010666 NA NA
0.72464 0.72028 0.72895 NA NA 0.0050748 0.00443
0.0058128 NA NA 0.0041116 NA NA 0.0043938 0.0011576 0.00763 NA NA
0.0081895 NA NA 6.89129 3.43733 13.81591 NA NA 1.93026 1.23469
2.62582 NA NA 0.74656 0.59261 0.9005 NA NA -0.28266 -0.33411
-0.2312 NA NA 0.20173 0.11553 0.28793 NA NA 0.15449 0.099696
0.20928 NA NA 0.15477 0.099502 0.20948 NA NA 0.11573 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 >76.2 >12.700 NA
NA ECLV 70390 0.052479 0.0043938 19
0.05 -0.047664 0.1 0.0036395 0.15 0.0027549 0.2
0.0017596 0.25 0.00063165 0.3 -0.00065744 0.35 -0.0021449
0.4 -0.0038802 0.45 -0.005931 0.5 -0.008392 0.55 -0.0114
0.6 -0.01516 0.65 -0.019994 0.7 -0.026439 0.75
-0.035463 0.8 -0.048998 0.85 -0.071557 0.9 -0.11668 0.95
-0.25203
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >0.254 >12.700 NA
NA FHO 59496 0.31395 0.045801 0.057281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >0.254 >12.700 NA
NA CTC 59496 2725 15954 683 40134
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >0.254 >12.700 NA
0.01 CTS 59496 0.057281 0.054876 0.059785 NA
NA 0.31395 0.30907 0.31888 NA NA 0.72037
0.7156 0.72508 NA NA 5.48093 NA NA 0.79959
0.79533 0.80378 NA NA 0.71555 0.71077 0.72029 NA
NA 0.28445 0.27971 0.28923 NA NA 0.85411
0.85035 0.8578 NA NA 0.14074 0.13711 0.14445 NA
NA 0.090479 NA NA 0.51514 0.49696 0.53333 NA NA 0.16594 NA
NA 10.03665 8.96577 11.23543 NA NA 2.30624 2.19341 2.41907 NA NA
0.81879 0.80019 0.83738 NA NA 0.85493 0.84164 0.86822 NA NA
0.30318 0.29385 0.31252 NA NA 0.69794 0.65463 0.74124 NA NA 0.67376
0.65321 0.74267 NA NA 0.10206 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >0.254 >12.700 NA
NA ECLV 59496 0.057281 0.51514 19
0.05 0.48419 0.1 0.27944 0.15 -0.026529 0.2
-0.37075 0.25 -0.76086 0.3 -1.2067 0.35 -1.72113
0.4 -2.3213 0.45 -3.0306 0.5 -3.88175 0.55 -4.92205
0.6 -6.22242 0.65 -7.89432 0.7 -10.12353 0.75
-13.24442 0.8 -17.92576 0.85 -25.72799 0.9 -41.33245 0.95
-88.14583
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >2.540 >12.700 NA
NA FHO 59496 0.20169 0.035616 0.057281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >2.540 >12.700 NA
NA CTC 59496 2119 9881 1289 46207
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >2.540 >12.700 NA
0.01 CTS 59496 0.057281 0.054876 0.059785 NA
NA 0.20169 0.19749 0.20596 NA NA 0.81226
0.8081 0.81635 NA NA 3.52113 NA NA 0.62177
0.61664 0.62688 NA NA 0.82383 0.81977 0.82782 NA
NA 0.17617 0.17218 0.18023 NA NA 0.82342
0.81935 0.82741 NA NA 0.15946 0.15563 0.16336 NA
NA 0.11361 NA NA 0.4456 0.42358 0.46762 NA NA 0.20403 NA
NA 7.6875 6.98828 8.45667 NA NA 2.0396 1.94424 2.13496 NA NA
0.76978 0.75036 0.78921 NA NA 0.71503 0.69733 0.73273 NA NA
0.33758 0.32378 0.35138 NA NA 0.57026 0.53483 0.60569 NA NA 0.60387
0.5349 0.60562 NA NA 0.099637 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >2.540 >12.700 NA
NA ECLV 59496 0.057281 0.4456 19
0.05 0.38718 0.1 0.29962 0.15 0.11012 0.2
-0.10307 0.25 -0.34468 0.3 -0.62081 0.35 -0.93942
0.4 -1.31113 0.45 -1.75043 0.5 -2.27758 0.55 -2.92188
0.6 -3.72726 0.65 -4.76274 0.7 -6.14339 0.75
-8.07629 0.8 -10.97565 0.85 -15.8079 0.9 -25.47242 0.95
-54.46596
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >3.810 >12.700 NA
NA FHO 59496 0.177 0.032456 0.057281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >3.810 >12.700 NA
NA CTC 59496 1931 8600 1477 47488
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >3.810 >12.700 NA
0.01 CTS 59496 0.057281 0.054876 0.059785 NA
NA 0.177 0.17301 0.18107 NA NA 0.83063
0.82663 0.83455 NA NA 3.09008 NA NA 0.56661
0.56137 0.57183 NA NA 0.84667 0.84283 0.85044 NA
NA 0.15333 0.14956 0.15717 NA NA 0.81664
0.81251 0.82069 NA NA 0.16081 0.15697 0.16473 NA
NA 0.11642 NA NA 0.41328 0.39078 0.43578 NA NA 0.20856 NA
NA 7.21917 6.57137 7.93083 NA NA 1.97674 1.88272 2.07076 NA NA
0.75667 0.73657 0.77676 NA NA 0.66855 0.64976 0.68733 NA NA
0.33942 0.32434 0.3545 NA NA 0.53497 0.50113 0.56882 NA NA 0.57367
0.50069 0.56925 NA NA 0.098419 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >3.810 >12.700 NA
NA ECLV 59496 0.057281 0.41328 19
0.05 0.34633 0.1 0.28622 0.15 0.12129 0.2
-0.064261 0.25 -0.27455 0.3 -0.51488 0.35 -0.79219
0.4 -1.11571 0.45 -1.49805 0.5 -1.95687 0.55 -2.51764
0.6 -3.2186 0.65 -4.11984 0.7 -5.3215 0.75
-7.00381 0.8 -9.52729 0.85 -13.73308 0.9 -22.14466 0.95
-47.3794
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >6.350 >12.700 NA
NA FHO 59496 0.14216 0.027783 0.057281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >6.350 >12.700 NA
NA CTC 59496 1653 6805 1755 49283
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >6.350 >12.700 NA
0.01 CTS 59496 0.057281 0.054876 0.059785 NA
NA 0.14216 0.13851 0.14589 NA NA 0.85612
0.85238 0.85979 NA NA 2.48181 NA NA 0.48504
0.47976 0.49031 NA NA 0.87867 0.87518 0.88208 NA
NA 0.12133 0.11792 0.12482 NA NA 0.80456
0.80034 0.80872 NA NA 0.16185 0.158 0.16578 NA
NA 0.12011 NA NA 0.36371 0.34102 0.3864 NA NA 0.21446 NA
NA 6.82126 6.20703 7.49627 NA NA 1.92004 1.82568 2.01441 NA NA
0.74429 0.72324 0.76533 NA NA 0.59617 0.57591 0.61642 NA NA
0.34249 0.32546 0.35953 NA NA 0.48917 0.45755 0.5208 NA NA 0.52955
0.45639 0.52196 NA NA 0.098733 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >6.350 >12.700 NA
NA ECLV 59496 0.057281 0.36371 19
0.05 0.28416 0.1 0.26317 0.15 0.13266 0.2
-0.014158 0.25 -0.18056 0.3 -0.37072 0.35 -0.59015
0.4 -0.84615 0.45 -1.14869 0.5 -1.51174 0.55 -1.95546
0.6 -2.51012 0.65 -3.22326 0.7 -4.1741 0.75
-5.50528 0.8 -7.50205 0.85 -10.83001 0.9 -17.48592 0.95
-37.45364
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >12.700 >12.700 NA
NA FHO 59496 0.087199 0.019111 0.057281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >12.700 >12.700 NA
NA CTC 59496 1137 4051 2271 52037
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >12.700 >12.700 NA
0.01 CTS 59496 0.057281 0.054876 0.059785 NA
NA 0.087199 0.084266 0.090225 NA NA 0.89374
0.89044 0.89695 NA NA 1.5223 NA NA 0.33363
0.32867 0.33862 NA NA 0.92777 0.92499 0.93046 NA
NA 0.072226 0.06954 0.075007 NA NA 0.78084
0.77644 0.78518 NA NA 0.15243 0.14868 0.15627 NA
NA 0.11726 NA NA 0.2614 0.23999 0.28281 NA NA 0.20991 NA
NA 6.43122 5.80422 7.12595 NA NA 1.86116 1.75859 1.96374 NA NA
0.73087 0.70697 0.75476 NA NA 0.44524 0.42247 0.46801 NA NA
0.33906 0.31796 0.36016 NA NA 0.41072 0.38217 0.43927 NA NA 0.44244
0.38023 0.44122 NA NA 0.10149 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >12.700 >12.700 NA
NA ECLV 59496 0.057281 0.2614 19
0.05 0.15847 0.1 0.20155 0.15 0.12386 0.2
0.036458 0.25 -0.062598 0.3 -0.1758 0.35 -0.30643
0.4 -0.45882 0.45 -0.63892 0.5 -0.85505 0.55 -1.1192
0.6 -1.44938 0.65 -1.87391 0.7 -2.43995 0.75
-3.23239 0.8 -4.42107 0.85 -6.40219 0.9 -10.36444 0.95
-22.25117
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >19.05 >12.700 NA
NA FHO 59496 0.053953 0.012875 0.057281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >19.05 >12.700 NA
NA CTC 59496 766 2444 2642 53644
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >19.05 >12.700 NA
0.01 CTS 59496 0.057281 0.054876 0.059785 NA
NA 0.053953 0.051617 0.056389 NA NA 0.91452
0.91152 0.91742 NA NA 0.9419 NA NA 0.22477
0.22039 0.2292 NA NA 0.95643 0.95422 0.95853 NA
NA 0.043574 0.041469 0.045782 NA NA 0.76137
0.75684 0.76584 NA NA 0.1309 0.12737 0.1345 NA
NA 0.1027 NA NA 0.18119 0.16223 0.20015 NA NA 0.18627 NA
NA 6.36379 5.65338 7.16347 NA NA 1.85062 1.73225 1.96899 NA NA
0.7284 0.70062 0.75618 NA NA 0.31409 0.28935 0.33883 NA NA
0.32784 0.30285 0.35284 NA NA 0.35465 0.32734 0.38196 NA NA 0.37575
0.32543 0.38386 NA NA 0.10554 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >19.05 >12.700 NA
NA ECLV 59496 0.057281 0.18119 19
0.05 0.061439 0.1 0.14508 0.15 0.098212 0.2
0.045481 0.25 -0.01428 0.3 -0.082579 0.35 -0.16138
0.4 -0.25333 0.45 -0.36198 0.5 -0.49237 0.55 -0.65173
0.6 -0.85094 0.65 -1.10706 0.7 -1.44855 0.75
-1.92664 0.8 -2.64378 0.85 -3.83901 0.9 -6.22946 0.95
-13.40082
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >25.400 >12.700 NA
NA FHO 59496 0.033263 0.0083199 0.057281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >25.400 >12.700 NA
NA CTC 59496 495 1484 2913 54604
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >25.400 >12.700 NA
0.01 CTS 59496 0.057281 0.054876 0.059785 NA
NA 0.033263 0.03142 0.035209 NA NA 0.9261
0.92329 0.92881 NA NA 0.58069 NA NA 0.14525
0.14157 0.14901 NA NA 0.97354 0.97179 0.97518 NA
NA 0.026458 0.024816 0.028207 NA NA 0.74987
0.74527 0.75442 NA NA 0.10119 0.098045 0.10441 NA
NA 0.079864 NA NA 0.11879 0.10278 0.1348 NA NA 0.14791 NA
NA 6.25252 5.42272 7.2093 NA NA 1.83299 1.6906 1.97537 NA NA
0.72423 0.69038 0.75809 NA NA 0.19429 0.16759 0.22098 NA NA
0.30778 0.27855 0.33701 NA NA 0.30619 0.27878 0.33359 NA NA 0.31904
0.27719 0.33519 NA NA 0.10714 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >25.400 >12.700 NA
NA ECLV 59496 0.057281 0.11879 19
0.05 -0.013247 0.1 0.096864 0.15 0.068403 0.2
0.036385 0.25 9.7809e-05 0.3 -0.041373 0.35 -0.089224
0.4 -0.14505 0.45 -0.21103 0.5 -0.2902 0.55 -0.38697
0.6 -0.50792 0.65 -0.66344 0.7 -0.87079 0.75
-1.16109 0.8 -1.59654 0.85 -2.32228 0.9 -3.77377 0.95
-8.12823
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >38.1 >12.700 NA
NA FHO 59496 0.012707 0.0030254 0.057281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >38.1 >12.700 NA
NA CTC 59496 180 576 3228 55512
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >38.1 >12.700 NA
0.01 CTS 59496 0.057281 0.054876 0.059785 NA
NA 0.012707 0.011577 0.013945 NA NA 0.93606
0.93343 0.9386 NA NA 0.22183 NA NA
0.052817 0.050504 0.055229 NA NA 0.98973 0.98861
0.99074 NA NA 0.01027 0.0092582 0.01139 NA NA
0.7619 0.75738 0.76637 NA NA 0.045181 0.043038
0.047425 NA NA 0.034688 NA NA 0.042547 0.032355 0.05274 NA NA
0.06705 NA NA 5.37407 4.29196 6.72901 NA NA 1.68159 1.45674
1.90643 NA NA 0.68623 0.62675 0.74571 NA NA -0.013988 -0.045749
0.017773 NA NA 0.24561 0.20548 0.28573 NA NA 0.21779 0.18644 0.24913
NA NA 0.22173 0.18553 0.25004 NA NA 0.095265 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >38.1 >12.700 NA
NA ECLV 59496 0.057281 0.042547 19
0.05 -0.10377 0.1 0.034038 0.15 0.022991 0.2
0.010563 0.25 -0.0035211 0.3 -0.019618 0.35 -0.038191
0.4 -0.059859 0.45 -0.085467 0.5 -0.1162 0.55 -0.15376
0.6 -0.2007 0.65 -0.26107 0.7 -0.34155 0.75
-0.45423 0.8 -0.62324 0.85 -0.90493 0.9 -1.46831 0.95
-3.15845
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >50.8 >12.700 NA
NA FHO 59496 0.0053281 0.0011429 0.057281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >50.8 >12.700 NA
NA CTC 59496 68 249 3340 55839
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >50.8 >12.700 NA
0.01 CTS 59496 0.057281 0.054876 0.059785 NA
NA 0.0053281 0.0046125 0.006154 NA NA 0.93968
0.93711 0.94214 NA NA 0.093016 NA NA
0.019953 0.018529 0.021484 NA NA 0.99556 0.9948
0.99621 NA NA 0.0044395 0.0037905 0.0051989 NA NA
0.78549 0.78112 0.78979 NA NA 0.018594 0.017221
0.020076 NA NA 0.013697 NA NA 0.015514 0.0090962 0.021931 NA NA
0.027024 NA NA 4.56563 3.19995 6.51415 NA NA 1.51856 1.16314
1.87398 NA NA 0.64065 0.53588 0.74542 NA NA -0.15568 -0.19422
-0.11714 NA NA 0.19492 0.14037 0.24946 NA NA 0.16105 0.12201
0.20009 NA NA 0.1623 0.1215 0.2006 NA NA 0.080926 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >50.8 >12.700 NA
NA ECLV 59496 0.057281 0.015514 19
0.05 -0.13588 0.1 0.011835 0.15 0.0070595 0.2
0.0016872 0.25 -0.0044014 0.3 -0.01136 0.35 -0.019389
0.4 -0.028756 0.45 -0.039826 0.5 -0.05311 0.55 -0.069347
0.6 -0.089642 0.65 -0.11574 0.7 -0.15053 0.75
-0.19924 0.8 -0.2723 0.85 -0.39407 0.9 -0.63762 0.95
-1.36825
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >76.2 >12.700 NA
NA FHO 59496 0.0010757 0.00028573 0.057281
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >76.2 >12.700 NA
NA CTC 59496 17 47 3391 56041
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >76.2 >12.700 NA
0.01 CTS 59496 0.057281 0.054876 0.059785 NA
NA 0.0010757 0.00078075 0.0014819 NA NA 0.94221
0.9397 0.94463 NA NA 0.018779 NA NA
0.0049883 0.0042975 0.0057894 NA NA 0.99916 0.9988
0.99942 NA NA 0.00083797 0.00058305 0.0012042 NA NA
0.73438 0.72968 0.73901 NA NA 0.0049204 0.0042347
0.0057166 NA NA 0.0038634 NA NA 0.0041503 0.00080965 0.0074909 NA NA
0.0076971 NA NA 5.97763 2.87943 12.40943 NA NA 1.78802 1.05759
2.51846 NA NA 0.71337 0.53401 0.89273 NA NA -0.29911 -0.35263
-0.24559 NA NA 0.18799 0.097273 0.27871 NA NA 0.14403 0.086253
0.20181 NA NA 0.1443 0.086051 0.20201 NA NA 0.10779 NA NA
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 >76.2 >12.700 NA
NA ECLV 59496 0.057281 0.0041503 19
0.05 -0.14955 0.1 0.0034559 0.15 0.0025545 0.2
0.0015405 0.25 0.00039124 0.3 -0.0009222 0.35 -0.0024377
0.4 -0.0042058 0.45 -0.0062953 0.5 -0.0088028 0.55 -0.011868
0.6 -0.015698 0.65 -0.020624 0.7 -0.027191 0.75
-0.036385 0.8 -0.050176 0.85 -0.073161 0.9 -0.11913 0.95
-0.25704
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS FULL NEAREST 1 NA NA NA
NA GRAD 69480 3.40993 2.10619 4.62772
4.58341 99.04243 217.61602 1.619
V7.0 CONUSNEST4ROC NA 240000 20180809_120000 20180809_120000
240000 20180809_120000 20180809_120000 APCP_24 A24 APCP_24 A24
ANALYS CONUS NEAREST 1 NA NA NA
NA GRAD 59216 3.44714 2.18753 4.68524
4.631 98.84229 211.69954 1.57581
------------------------------------------------
More information about the Met_help
mailing list