[Met_help] [rt.rap.ucar.edu #95311] History for Surrogate severe use case help
George McCabe via RT
met_help at ucar.edu
Wed May 20 14:18:24 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I'm emailing to consult with the team on a slight problem I'm having with
getting surrogate severe verification to work in METplus.
I've been able to generate the surrogate severe forecast for various models
(using a mix of METplus tools and personal python code for the time being).
I have one forecast variable (based on one UH percentile threshold), and
I'm having difficulty generating probabilistic statistics. I am able to
successfully generate CTC and CTS line types when treating the surrogate
severe and practically perfect fields as scalars.
I've double checked my METplus conf and MET config files (2 of each)
against those in Tina's feature branch. Both METplus calls run
successfully, but the probabilistic stats call controlled by this conf
<https://github.com/NCAR/METplus/blob/feature_414_surrogate_severe_evaluation/parm/use_cases/model_applications/convection_allowing_models/GridStat_fcstHRRR_obsPracPerfect_SurrogateSevereProb.conf>
isn't producing any of the probabilistic statistic line types.
I've attached my METplus and MET configurations and example output file
showing a lack of PXXX stats.
I'm fairly new to running probabilistic verification with MET or METplus,
so I could have something slightly off. If it's easier to discuss my
problem over Slack or video chat, I'd be happy to do so.
Thanks,
Logan
--
Logan C. Dawson, Ph.D.
Physical Scientist, VPPPG/MEG
NOAA/NWS/NCEP/EMC
(301) 683-3944
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Surrogate severe use case help
From: Julie Prestopnik
Time: Tue May 19 16:24:03 2020
Hi Logan.
I have assigned this ticket to George, who is likely best able to
assist
you. Please allow a few business days for a response.
Thanks!
Julie
On Tue, May 19, 2020 at 1:53 PM Logan Dawson - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> Tue May 19 13:53:41 2020: Request 95311 was acted upon.
> Transaction: Ticket created by logan.dawson at noaa.gov
> Queue: met_help
> Subject: Surrogate severe use case help
> Owner: Nobody
> Requestors: logan.dawson at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
>
>
> Hi,
>
> I'm emailing to consult with the team on a slight problem I'm having
with
> getting surrogate severe verification to work in METplus.
>
> I've been able to generate the surrogate severe forecast for various
models
> (using a mix of METplus tools and personal python code for the time
being).
> I have one forecast variable (based on one UH percentile threshold),
and
> I'm having difficulty generating probabilistic statistics. I am able
to
> successfully generate CTC and CTS line types when treating the
surrogate
> severe and practically perfect fields as scalars.
>
> I've double checked my METplus conf and MET config files (2 of each)
> against those in Tina's feature branch. Both METplus calls run
> successfully, but the probabilistic stats call controlled by this
conf
> <
>
https://github.com/NCAR/METplus/blob/feature_414_surrogate_severe_evaluation/parm/use_cases/model_applications/convection_allowing_models/GridStat_fcstHRRR_obsPracPerfect_SurrogateSevereProb.conf
> >
> isn't producing any of the probabilistic statistic line types.
>
> I've attached my METplus and MET configurations and example output
file
> showing a lack of PXXX stats.
>
> I'm fairly new to running probabilistic verification with MET or
METplus,
> so I could have something slightly off. If it's easier to discuss my
> problem over Slack or video chat, I'd be happy to do so.
>
> Thanks,
> Logan
>
> --
> Logan C. Dawson, Ph.D.
> Physical Scientist, VPPPG/MEG
> NOAA/NWS/NCEP/EMC
> (301) 683-3944
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Surrogate severe use case help
From: George McCabe
Time: Wed May 20 11:00:45 2020
Hi Logan,
I forwarded this to Tina to get some advice. These use cases are still
in
progress. She said she has been updating them and they should be
working as
of yesterday, so I would pull the latest changes on that branch. She
made
changes to the GridStatConfig file that may turn on probabilistic
output.
Please let me know if you are still having issues and I can dig in
further.
Thanks,
George
On Tue, May 19, 2020 at 4:24 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
>
> Hi Logan.
>
> I have assigned this ticket to George, who is likely best able to
assist
> you. Please allow a few business days for a response.
>
> Thanks!
>
> Julie
>
> On Tue, May 19, 2020 at 1:53 PM Logan Dawson - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > Tue May 19 13:53:41 2020: Request 95311 was acted upon.
> > Transaction: Ticket created by logan.dawson at noaa.gov
> > Queue: met_help
> > Subject: Surrogate severe use case help
> > Owner: Nobody
> > Requestors: logan.dawson at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
> >
> >
> > Hi,
> >
> > I'm emailing to consult with the team on a slight problem I'm
having with
> > getting surrogate severe verification to work in METplus.
> >
> > I've been able to generate the surrogate severe forecast for
various
> models
> > (using a mix of METplus tools and personal python code for the
time
> being).
> > I have one forecast variable (based on one UH percentile
threshold), and
> > I'm having difficulty generating probabilistic statistics. I am
able to
> > successfully generate CTC and CTS line types when treating the
surrogate
> > severe and practically perfect fields as scalars.
> >
> > I've double checked my METplus conf and MET config files (2 of
each)
> > against those in Tina's feature branch. Both METplus calls run
> > successfully, but the probabilistic stats call controlled by this
conf
> > <
> >
>
https://github.com/NCAR/METplus/blob/feature_414_surrogate_severe_evaluation/parm/use_cases/model_applications/convection_allowing_models/GridStat_fcstHRRR_obsPracPerfect_SurrogateSevereProb.conf
> > >
> > isn't producing any of the probabilistic statistic line types.
> >
> > I've attached my METplus and MET configurations and example output
file
> > showing a lack of PXXX stats.
> >
> > I'm fairly new to running probabilistic verification with MET or
METplus,
> > so I could have something slightly off. If it's easier to discuss
my
> > problem over Slack or video chat, I'd be happy to do so.
> >
> > Thanks,
> > Logan
> >
> > --
> > Logan C. Dawson, Ph.D.
> > Physical Scientist, VPPPG/MEG
> > NOAA/NWS/NCEP/EMC
> > (301) 683-3944
> >
> >
>
> --
> Julie Prestopnik
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Surrogate severe use case help
From: George McCabe
Time: Wed May 20 11:21:19 2020
Hi Logan,
After pulling the latest changes from the feature branch, I was able
to run
the Surrogate Severe use cases and produce probabilistic stats, so
getting
those changes should help. This output is turned on with the
output_flag
dictionary in the GridStatConfig_ss_prob MET config file:
////////////////////////////////////////////////////////////////////////////////
//
// Statistical output types
//
output_flag = {
fho = NONE;
ctc = NONE;
cts = NONE;
mctc = NONE;
mcts = NONE;
cnt = NONE;
sl1l2 = NONE;
sal1l2 = NONE;
vl1l2 = NONE;
val1l2 = NONE;
vcnt = NONE;
* pct = BOTH; pstd = BOTH; pjc = BOTH; prc = BOTH;*
eclv = NONE;
nbrctc = NONE;
nbrcts = NONE;
nbrcnt = NONE;
grad = NONE;
dmap = NONE;
}
On Wed, May 20, 2020 at 11:00 AM George McCabe <mccabe at ucar.edu>
wrote:
> Hi Logan,
>
> I forwarded this to Tina to get some advice. These use cases are
still in
> progress. She said she has been updating them and they should be
working as
> of yesterday, so I would pull the latest changes on that branch. She
made
> changes to the GridStatConfig file that may turn on probabilistic
output.
> Please let me know if you are still having issues and I can dig in
further.
>
> Thanks,
> George
>
> On Tue, May 19, 2020 at 4:24 PM Julie Prestopnik via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
>>
>> Hi Logan.
>>
>> I have assigned this ticket to George, who is likely best able to
assist
>> you. Please allow a few business days for a response.
>>
>> Thanks!
>>
>> Julie
>>
>> On Tue, May 19, 2020 at 1:53 PM Logan Dawson - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Tue May 19 13:53:41 2020: Request 95311 was acted upon.
>> > Transaction: Ticket created by logan.dawson at noaa.gov
>> > Queue: met_help
>> > Subject: Surrogate severe use case help
>> > Owner: Nobody
>> > Requestors: logan.dawson at noaa.gov
>> > Status: new
>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
>> >
>> >
>> > Hi,
>> >
>> > I'm emailing to consult with the team on a slight problem I'm
having
>> with
>> > getting surrogate severe verification to work in METplus.
>> >
>> > I've been able to generate the surrogate severe forecast for
various
>> models
>> > (using a mix of METplus tools and personal python code for the
time
>> being).
>> > I have one forecast variable (based on one UH percentile
threshold), and
>> > I'm having difficulty generating probabilistic statistics. I am
able to
>> > successfully generate CTC and CTS line types when treating the
surrogate
>> > severe and practically perfect fields as scalars.
>> >
>> > I've double checked my METplus conf and MET config files (2 of
each)
>> > against those in Tina's feature branch. Both METplus calls run
>> > successfully, but the probabilistic stats call controlled by this
conf
>> > <
>> >
>>
https://github.com/NCAR/METplus/blob/feature_414_surrogate_severe_evaluation/parm/use_cases/model_applications/convection_allowing_models/GridStat_fcstHRRR_obsPracPerfect_SurrogateSevereProb.conf
>> > >
>> > isn't producing any of the probabilistic statistic line types.
>> >
>> > I've attached my METplus and MET configurations and example
output file
>> > showing a lack of PXXX stats.
>> >
>> > I'm fairly new to running probabilistic verification with MET or
>> METplus,
>> > so I could have something slightly off. If it's easier to discuss
my
>> > problem over Slack or video chat, I'd be happy to do so.
>> >
>> > Thanks,
>> > Logan
>> >
>> > --
>> > Logan C. Dawson, Ph.D.
>> > Physical Scientist, VPPPG/MEG
>> > NOAA/NWS/NCEP/EMC
>> > (301) 683-3944
>> >
>> >
>>
>> --
>> Julie Prestopnik
>> Software Engineer
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> Phone: 303.497.8399
>> Email: jpresto at ucar.edu
>>
>> My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Surrogate severe use case help
From: Logan Dawson - NOAA Affiliate
Time: Wed May 20 13:08:13 2020
Hi George,
Thanks for checking into this for me. I figured this use case was
still a
work in progress.
I updated my MET config to set the 4 probabilistic line types to BOTH
output files, but I'm still not getting probabilistic stats
unfortunately.
I get the ascii file for each line type, but only the MET output
header is
included in each file. It's odd to me because MET isn't failing. There
are
some differences between the MET/METplus config/conf files and Tina's
latest files, but I don't think they should be causing a lack of
output.
Would it be worthwhile for me to share a forecast file and an
observation
file from my end to see if my input data are causing the problem? For
this
particular configuration, the observation file isn't output by MET,
but I
know it's MET-compatible because I run it through RegridDataPlane to
generate the practically perfect field that's used for scalar
evaluation of
the probs. I'm wondering if the metadata or the actual data values are
slightly off or something.
Thanks,
Logan
On Wed, May 20, 2020 at 1:21 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Logan,
>
> After pulling the latest changes from the feature branch, I was able
to run
> the Surrogate Severe use cases and produce probabilistic stats, so
getting
> those changes should help. This output is turned on with the
output_flag
> dictionary in the GridStatConfig_ss_prob MET config file:
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Statistical output types
> //
> output_flag = {
> fho = NONE;
> ctc = NONE;
> cts = NONE;
> mctc = NONE;
> mcts = NONE;
> cnt = NONE;
> sl1l2 = NONE;
> sal1l2 = NONE;
> vl1l2 = NONE;
> val1l2 = NONE;
> vcnt = NONE;
>
>
>
> * pct = BOTH; pstd = BOTH; pjc = BOTH; prc =
BOTH;*
> eclv = NONE;
> nbrctc = NONE;
> nbrcts = NONE;
> nbrcnt = NONE;
> grad = NONE;
> dmap = NONE;
> }
>
> On Wed, May 20, 2020 at 11:00 AM George McCabe <mccabe at ucar.edu>
wrote:
>
> > Hi Logan,
> >
> > I forwarded this to Tina to get some advice. These use cases are
still in
> > progress. She said she has been updating them and they should be
working
> as
> > of yesterday, so I would pull the latest changes on that branch.
She made
> > changes to the GridStatConfig file that may turn on probabilistic
output.
> > Please let me know if you are still having issues and I can dig in
> further.
> >
> > Thanks,
> > George
> >
> > On Tue, May 19, 2020 at 4:24 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> > wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
> >>
> >> Hi Logan.
> >>
> >> I have assigned this ticket to George, who is likely best able to
assist
> >> you. Please allow a few business days for a response.
> >>
> >> Thanks!
> >>
> >> Julie
> >>
> >> On Tue, May 19, 2020 at 1:53 PM Logan Dawson - NOAA Affiliate via
RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > Tue May 19 13:53:41 2020: Request 95311 was acted upon.
> >> > Transaction: Ticket created by logan.dawson at noaa.gov
> >> > Queue: met_help
> >> > Subject: Surrogate severe use case help
> >> > Owner: Nobody
> >> > Requestors: logan.dawson at noaa.gov
> >> > Status: new
> >> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311
> >
> >> >
> >> >
> >> > Hi,
> >> >
> >> > I'm emailing to consult with the team on a slight problem I'm
having
> >> with
> >> > getting surrogate severe verification to work in METplus.
> >> >
> >> > I've been able to generate the surrogate severe forecast for
various
> >> models
> >> > (using a mix of METplus tools and personal python code for the
time
> >> being).
> >> > I have one forecast variable (based on one UH percentile
threshold),
> and
> >> > I'm having difficulty generating probabilistic statistics. I am
able
> to
> >> > successfully generate CTC and CTS line types when treating the
> surrogate
> >> > severe and practically perfect fields as scalars.
> >> >
> >> > I've double checked my METplus conf and MET config files (2 of
each)
> >> > against those in Tina's feature branch. Both METplus calls run
> >> > successfully, but the probabilistic stats call controlled by
this conf
> >> > <
> >> >
> >>
>
https://github.com/NCAR/METplus/blob/feature_414_surrogate_severe_evaluation/parm/use_cases/model_applications/convection_allowing_models/GridStat_fcstHRRR_obsPracPerfect_SurrogateSevereProb.conf
> >> > >
> >> > isn't producing any of the probabilistic statistic line types.
> >> >
> >> > I've attached my METplus and MET configurations and example
output
> file
> >> > showing a lack of PXXX stats.
> >> >
> >> > I'm fairly new to running probabilistic verification with MET
or
> >> METplus,
> >> > so I could have something slightly off. If it's easier to
discuss my
> >> > problem over Slack or video chat, I'd be happy to do so.
> >> >
> >> > Thanks,
> >> > Logan
> >> >
> >> > --
> >> > Logan C. Dawson, Ph.D.
> >> > Physical Scientist, VPPPG/MEG
> >> > NOAA/NWS/NCEP/EMC
> >> > (301) 683-3944
> >> >
> >> >
> >>
> >> --
> >> Julie Prestopnik
> >> Software Engineer
> >> National Center for Atmospheric Research
> >> Research Applications Laboratory
> >> Phone: 303.497.8399
> >> Email: jpresto at ucar.edu
> >>
> >> My working day may not be your working day. Please do not feel
obliged
> to
> >> reply to this email outside of your normal working hours.
> >>
> >>
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Logan C. Dawson, Ph.D.
Physical Scientist, VPPPG/MEG
NOAA/NWS/NCEP/EMC
(301) 683-3944
------------------------------------------------
Subject: Surrogate severe use case help
From: George McCabe
Time: Wed May 20 13:29:25 2020
My guess is your configuration is not set up correctly to produce
output. I
noticed there is a typo in one of the variables names:
FCST_GRID_STAT_PROB_TRESH
should be
FCST_GRID_STAT_PROB_THRESH
I believe the default value is ==0.1, which produces 10 equal bins
from 0
to 1. I'm not sure if fixing that would result in getting output
though.
If there is no error from MET then I would assume that it read the
data
correctly. You could boost the GridStat verbosity with [config]
LOG_GRID_STAT_VERBOSITY to see if anything in the log output gives you
a
hint. You could send me the log file and/or a matching set of forecast
and
observation data and I can take a look. I can access hera so I could
get
the data from there, but I don't have access to any of the other NOAA
machines yet, so I would have to ask someone who does to get the files
for
me.
On Wed, May 20, 2020 at 1:08 PM Logan Dawson - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
>
> Hi George,
>
> Thanks for checking into this for me. I figured this use case was
still a
> work in progress.
>
> I updated my MET config to set the 4 probabilistic line types to
BOTH
> output files, but I'm still not getting probabilistic stats
unfortunately.
> I get the ascii file for each line type, but only the MET output
header is
> included in each file. It's odd to me because MET isn't failing.
There are
> some differences between the MET/METplus config/conf files and
Tina's
> latest files, but I don't think they should be causing a lack of
output.
>
> Would it be worthwhile for me to share a forecast file and an
observation
> file from my end to see if my input data are causing the problem?
For this
> particular configuration, the observation file isn't output by MET,
but I
> know it's MET-compatible because I run it through RegridDataPlane to
> generate the practically perfect field that's used for scalar
evaluation of
> the probs. I'm wondering if the metadata or the actual data values
are
> slightly off or something.
>
> Thanks,
> Logan
>
>
>
> On Wed, May 20, 2020 at 1:21 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Logan,
> >
> > After pulling the latest changes from the feature branch, I was
able to
> run
> > the Surrogate Severe use cases and produce probabilistic stats, so
> getting
> > those changes should help. This output is turned on with the
output_flag
> > dictionary in the GridStatConfig_ss_prob MET config file:
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Statistical output types
> > //
> > output_flag = {
> > fho = NONE;
> > ctc = NONE;
> > cts = NONE;
> > mctc = NONE;
> > mcts = NONE;
> > cnt = NONE;
> > sl1l2 = NONE;
> > sal1l2 = NONE;
> > vl1l2 = NONE;
> > val1l2 = NONE;
> > vcnt = NONE;
> >
> >
> >
> > * pct = BOTH; pstd = BOTH; pjc = BOTH; prc =
BOTH;*
> > eclv = NONE;
> > nbrctc = NONE;
> > nbrcts = NONE;
> > nbrcnt = NONE;
> > grad = NONE;
> > dmap = NONE;
> > }
> >
> > On Wed, May 20, 2020 at 11:00 AM George McCabe <mccabe at ucar.edu>
wrote:
> >
> > > Hi Logan,
> > >
> > > I forwarded this to Tina to get some advice. These use cases are
still
> in
> > > progress. She said she has been updating them and they should be
> working
> > as
> > > of yesterday, so I would pull the latest changes on that branch.
She
> made
> > > changes to the GridStatConfig file that may turn on
probabilistic
> output.
> > > Please let me know if you are still having issues and I can dig
in
> > further.
> > >
> > > Thanks,
> > > George
> > >
> > > On Tue, May 19, 2020 at 4:24 PM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
> > >>
> > >> Hi Logan.
> > >>
> > >> I have assigned this ticket to George, who is likely best able
to
> assist
> > >> you. Please allow a few business days for a response.
> > >>
> > >> Thanks!
> > >>
> > >> Julie
> > >>
> > >> On Tue, May 19, 2020 at 1:53 PM Logan Dawson - NOAA Affiliate
via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > Tue May 19 13:53:41 2020: Request 95311 was acted upon.
> > >> > Transaction: Ticket created by logan.dawson at noaa.gov
> > >> > Queue: met_help
> > >> > Subject: Surrogate severe use case help
> > >> > Owner: Nobody
> > >> > Requestors: logan.dawson at noaa.gov
> > >> > Status: new
> > >> > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311
> > >
> > >> >
> > >> >
> > >> > Hi,
> > >> >
> > >> > I'm emailing to consult with the team on a slight problem I'm
having
> > >> with
> > >> > getting surrogate severe verification to work in METplus.
> > >> >
> > >> > I've been able to generate the surrogate severe forecast for
various
> > >> models
> > >> > (using a mix of METplus tools and personal python code for
the time
> > >> being).
> > >> > I have one forecast variable (based on one UH percentile
threshold),
> > and
> > >> > I'm having difficulty generating probabilistic statistics. I
am able
> > to
> > >> > successfully generate CTC and CTS line types when treating
the
> > surrogate
> > >> > severe and practically perfect fields as scalars.
> > >> >
> > >> > I've double checked my METplus conf and MET config files (2
of each)
> > >> > against those in Tina's feature branch. Both METplus calls
run
> > >> > successfully, but the probabilistic stats call controlled by
this
> conf
> > >> > <
> > >> >
> > >>
> >
>
https://github.com/NCAR/METplus/blob/feature_414_surrogate_severe_evaluation/parm/use_cases/model_applications/convection_allowing_models/GridStat_fcstHRRR_obsPracPerfect_SurrogateSevereProb.conf
> > >> > >
> > >> > isn't producing any of the probabilistic statistic line
types.
> > >> >
> > >> > I've attached my METplus and MET configurations and example
output
> > file
> > >> > showing a lack of PXXX stats.
> > >> >
> > >> > I'm fairly new to running probabilistic verification with MET
or
> > >> METplus,
> > >> > so I could have something slightly off. If it's easier to
discuss my
> > >> > problem over Slack or video chat, I'd be happy to do so.
> > >> >
> > >> > Thanks,
> > >> > Logan
> > >> >
> > >> > --
> > >> > Logan C. Dawson, Ph.D.
> > >> > Physical Scientist, VPPPG/MEG
> > >> > NOAA/NWS/NCEP/EMC
> > >> > (301) 683-3944
> > >> >
> > >> >
> > >>
> > >> --
> > >> Julie Prestopnik
> > >> Software Engineer
> > >> National Center for Atmospheric Research
> > >> Research Applications Laboratory
> > >> Phone: 303.497.8399
> > >> Email: jpresto at ucar.edu
> > >>
> > >> My working day may not be your working day. Please do not feel
> obliged
> > to
> > >> reply to this email outside of your normal working hours.
> > >>
> > >>
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Logan C. Dawson, Ph.D.
> Physical Scientist, VPPPG/MEG
> NOAA/NWS/NCEP/EMC
> (301) 683-3944
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Surrogate severe use case help
From: Logan Dawson - NOAA Affiliate
Time: Wed May 20 13:52:45 2020
Hi George,
I found my error and fixed it. I run my different verification use
cases
for 10+ models, so it's useful for me to have a single conf to
configure
the process, and then I use a separate conf for each model to define
the
forecast data path. Although I had FCST_IS_PROB = True in the GridStat
conf, I had FCST_IS_PROB set to False in the model conf. The model
conf is
the last one I pass in, so it was overwriting FCST_IS_PROB and
suppressing
the probabilistic output.
Thanks for the catch on my typo with FCST_GRID_STAT_PROB_THRESH. That
led
me to look at my metplus_final conf where I found my FCST_IS_PROB
error.
I removed the extra FCST_IS_PROB assignment, and I am getting
the probabilistic output now. You can close this ticket now.
Thanks very much for your help on this!
Logan
On Wed, May 20, 2020 at 3:29 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> My guess is your configuration is not set up correctly to produce
output. I
> noticed there is a typo in one of the variables names:
>
> FCST_GRID_STAT_PROB_TRESH
>
> should be
>
> FCST_GRID_STAT_PROB_THRESH
>
> I believe the default value is ==0.1, which produces 10 equal bins
from 0
> to 1. I'm not sure if fixing that would result in getting output
though.
>
> If there is no error from MET then I would assume that it read the
data
> correctly. You could boost the GridStat verbosity with [config]
> LOG_GRID_STAT_VERBOSITY to see if anything in the log output gives
you a
> hint. You could send me the log file and/or a matching set of
forecast and
> observation data and I can take a look. I can access hera so I could
get
> the data from there, but I don't have access to any of the other
NOAA
> machines yet, so I would have to ask someone who does to get the
files for
> me.
>
> On Wed, May 20, 2020 at 1:08 PM Logan Dawson - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
> >
> > Hi George,
> >
> > Thanks for checking into this for me. I figured this use case was
still a
> > work in progress.
> >
> > I updated my MET config to set the 4 probabilistic line types to
BOTH
> > output files, but I'm still not getting probabilistic stats
> unfortunately.
> > I get the ascii file for each line type, but only the MET output
header
> is
> > included in each file. It's odd to me because MET isn't failing.
There
> are
> > some differences between the MET/METplus config/conf files and
Tina's
> > latest files, but I don't think they should be causing a lack of
output.
> >
> > Would it be worthwhile for me to share a forecast file and an
observation
> > file from my end to see if my input data are causing the problem?
For
> this
> > particular configuration, the observation file isn't output by
MET, but I
> > know it's MET-compatible because I run it through RegridDataPlane
to
> > generate the practically perfect field that's used for scalar
evaluation
> of
> > the probs. I'm wondering if the metadata or the actual data values
are
> > slightly off or something.
> >
> > Thanks,
> > Logan
> >
> >
> >
> > On Wed, May 20, 2020 at 1:21 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Logan,
> > >
> > > After pulling the latest changes from the feature branch, I was
able to
> > run
> > > the Surrogate Severe use cases and produce probabilistic stats,
so
> > getting
> > > those changes should help. This output is turned on with the
> output_flag
> > > dictionary in the GridStatConfig_ss_prob MET config file:
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Statistical output types
> > > //
> > > output_flag = {
> > > fho = NONE;
> > > ctc = NONE;
> > > cts = NONE;
> > > mctc = NONE;
> > > mcts = NONE;
> > > cnt = NONE;
> > > sl1l2 = NONE;
> > > sal1l2 = NONE;
> > > vl1l2 = NONE;
> > > val1l2 = NONE;
> > > vcnt = NONE;
> > >
> > >
> > >
> > > * pct = BOTH; pstd = BOTH; pjc = BOTH; prc =
BOTH;*
> > > eclv = NONE;
> > > nbrctc = NONE;
> > > nbrcts = NONE;
> > > nbrcnt = NONE;
> > > grad = NONE;
> > > dmap = NONE;
> > > }
> > >
> > > On Wed, May 20, 2020 at 11:00 AM George McCabe <mccabe at ucar.edu>
> wrote:
> > >
> > > > Hi Logan,
> > > >
> > > > I forwarded this to Tina to get some advice. These use cases
are
> still
> > in
> > > > progress. She said she has been updating them and they should
be
> > working
> > > as
> > > > of yesterday, so I would pull the latest changes on that
branch. She
> > made
> > > > changes to the GridStatConfig file that may turn on
probabilistic
> > output.
> > > > Please let me know if you are still having issues and I can
dig in
> > > further.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Tue, May 19, 2020 at 4:24 PM Julie Prestopnik via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311
>
> > > >>
> > > >> Hi Logan.
> > > >>
> > > >> I have assigned this ticket to George, who is likely best
able to
> > assist
> > > >> you. Please allow a few business days for a response.
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Julie
> > > >>
> > > >> On Tue, May 19, 2020 at 1:53 PM Logan Dawson - NOAA Affiliate
via
> RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > Tue May 19 13:53:41 2020: Request 95311 was acted upon.
> > > >> > Transaction: Ticket created by logan.dawson at noaa.gov
> > > >> > Queue: met_help
> > > >> > Subject: Surrogate severe use case help
> > > >> > Owner: Nobody
> > > >> > Requestors: logan.dawson at noaa.gov
> > > >> > Status: new
> > > >> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311
> > > >
> > > >> >
> > > >> >
> > > >> > Hi,
> > > >> >
> > > >> > I'm emailing to consult with the team on a slight problem
I'm
> having
> > > >> with
> > > >> > getting surrogate severe verification to work in METplus.
> > > >> >
> > > >> > I've been able to generate the surrogate severe forecast
for
> various
> > > >> models
> > > >> > (using a mix of METplus tools and personal python code for
the
> time
> > > >> being).
> > > >> > I have one forecast variable (based on one UH percentile
> threshold),
> > > and
> > > >> > I'm having difficulty generating probabilistic statistics.
I am
> able
> > > to
> > > >> > successfully generate CTC and CTS line types when treating
the
> > > surrogate
> > > >> > severe and practically perfect fields as scalars.
> > > >> >
> > > >> > I've double checked my METplus conf and MET config files (2
of
> each)
> > > >> > against those in Tina's feature branch. Both METplus calls
run
> > > >> > successfully, but the probabilistic stats call controlled
by this
> > conf
> > > >> > <
> > > >> >
> > > >>
> > >
> >
>
https://github.com/NCAR/METplus/blob/feature_414_surrogate_severe_evaluation/parm/use_cases/model_applications/convection_allowing_models/GridStat_fcstHRRR_obsPracPerfect_SurrogateSevereProb.conf
> > > >> > >
> > > >> > isn't producing any of the probabilistic statistic line
types.
> > > >> >
> > > >> > I've attached my METplus and MET configurations and example
output
> > > file
> > > >> > showing a lack of PXXX stats.
> > > >> >
> > > >> > I'm fairly new to running probabilistic verification with
MET or
> > > >> METplus,
> > > >> > so I could have something slightly off. If it's easier to
discuss
> my
> > > >> > problem over Slack or video chat, I'd be happy to do so.
> > > >> >
> > > >> > Thanks,
> > > >> > Logan
> > > >> >
> > > >> > --
> > > >> > Logan C. Dawson, Ph.D.
> > > >> > Physical Scientist, VPPPG/MEG
> > > >> > NOAA/NWS/NCEP/EMC
> > > >> > (301) 683-3944
> > > >> >
> > > >> >
> > > >>
> > > >> --
> > > >> Julie Prestopnik
> > > >> Software Engineer
> > > >> National Center for Atmospheric Research
> > > >> Research Applications Laboratory
> > > >> Phone: 303.497.8399
> > > >> Email: jpresto at ucar.edu
> > > >>
> > > >> My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > >> reply to this email outside of your normal working hours.
> > > >>
> > > >>
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Logan C. Dawson, Ph.D.
> > Physical Scientist, VPPPG/MEG
> > NOAA/NWS/NCEP/EMC
> > (301) 683-3944
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Logan C. Dawson, Ph.D.
Physical Scientist, VPPPG/MEG
NOAA/NWS/NCEP/EMC
(301) 683-3944
------------------------------------------------
Subject: Surrogate severe use case help
From: George McCabe
Time: Wed May 20 13:59:42 2020
Good to hear!
On Wed, May 20, 2020 at 1:53 PM Logan Dawson - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
>
> Hi George,
>
> I found my error and fixed it. I run my different verification use
cases
> for 10+ models, so it's useful for me to have a single conf to
configure
> the process, and then I use a separate conf for each model to define
the
> forecast data path. Although I had FCST_IS_PROB = True in the
GridStat
> conf, I had FCST_IS_PROB set to False in the model conf. The model
conf is
> the last one I pass in, so it was overwriting FCST_IS_PROB and
suppressing
> the probabilistic output.
>
> Thanks for the catch on my typo with FCST_GRID_STAT_PROB_THRESH.
That led
> me to look at my metplus_final conf where I found my FCST_IS_PROB
error.
>
> I removed the extra FCST_IS_PROB assignment, and I am getting
> the probabilistic output now. You can close this ticket now.
>
> Thanks very much for your help on this!
> Logan
>
> On Wed, May 20, 2020 at 3:29 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > My guess is your configuration is not set up correctly to produce
> output. I
> > noticed there is a typo in one of the variables names:
> >
> > FCST_GRID_STAT_PROB_TRESH
> >
> > should be
> >
> > FCST_GRID_STAT_PROB_THRESH
> >
> > I believe the default value is ==0.1, which produces 10 equal bins
from 0
> > to 1. I'm not sure if fixing that would result in getting output
though.
> >
> > If there is no error from MET then I would assume that it read the
data
> > correctly. You could boost the GridStat verbosity with [config]
> > LOG_GRID_STAT_VERBOSITY to see if anything in the log output gives
you a
> > hint. You could send me the log file and/or a matching set of
forecast
> and
> > observation data and I can take a look. I can access hera so I
could get
> > the data from there, but I don't have access to any of the other
NOAA
> > machines yet, so I would have to ask someone who does to get the
files
> for
> > me.
> >
> > On Wed, May 20, 2020 at 1:08 PM Logan Dawson - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
> > >
> > > Hi George,
> > >
> > > Thanks for checking into this for me. I figured this use case
was
> still a
> > > work in progress.
> > >
> > > I updated my MET config to set the 4 probabilistic line types to
BOTH
> > > output files, but I'm still not getting probabilistic stats
> > unfortunately.
> > > I get the ascii file for each line type, but only the MET output
header
> > is
> > > included in each file. It's odd to me because MET isn't failing.
There
> > are
> > > some differences between the MET/METplus config/conf files and
Tina's
> > > latest files, but I don't think they should be causing a lack of
> output.
> > >
> > > Would it be worthwhile for me to share a forecast file and an
> observation
> > > file from my end to see if my input data are causing the
problem? For
> > this
> > > particular configuration, the observation file isn't output by
MET,
> but I
> > > know it's MET-compatible because I run it through
RegridDataPlane to
> > > generate the practically perfect field that's used for scalar
> evaluation
> > of
> > > the probs. I'm wondering if the metadata or the actual data
values are
> > > slightly off or something.
> > >
> > > Thanks,
> > > Logan
> > >
> > >
> > >
> > > On Wed, May 20, 2020 at 1:21 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Logan,
> > > >
> > > > After pulling the latest changes from the feature branch, I
was able
> to
> > > run
> > > > the Surrogate Severe use cases and produce probabilistic
stats, so
> > > getting
> > > > those changes should help. This output is turned on with the
> > output_flag
> > > > dictionary in the GridStatConfig_ss_prob MET config file:
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Statistical output types
> > > > //
> > > > output_flag = {
> > > > fho = NONE;
> > > > ctc = NONE;
> > > > cts = NONE;
> > > > mctc = NONE;
> > > > mcts = NONE;
> > > > cnt = NONE;
> > > > sl1l2 = NONE;
> > > > sal1l2 = NONE;
> > > > vl1l2 = NONE;
> > > > val1l2 = NONE;
> > > > vcnt = NONE;
> > > >
> > > >
> > > >
> > > > * pct = BOTH; pstd = BOTH; pjc = BOTH; prc
=
> BOTH;*
> > > > eclv = NONE;
> > > > nbrctc = NONE;
> > > > nbrcts = NONE;
> > > > nbrcnt = NONE;
> > > > grad = NONE;
> > > > dmap = NONE;
> > > > }
> > > >
> > > > On Wed, May 20, 2020 at 11:00 AM George McCabe
<mccabe at ucar.edu>
> > wrote:
> > > >
> > > > > Hi Logan,
> > > > >
> > > > > I forwarded this to Tina to get some advice. These use cases
are
> > still
> > > in
> > > > > progress. She said she has been updating them and they
should be
> > > working
> > > > as
> > > > > of yesterday, so I would pull the latest changes on that
branch.
> She
> > > made
> > > > > changes to the GridStatConfig file that may turn on
probabilistic
> > > output.
> > > > > Please let me know if you are still having issues and I can
dig in
> > > > further.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Tue, May 19, 2020 at 4:24 PM Julie Prestopnik via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311 >
> > > > >>
> > > > >> Hi Logan.
> > > > >>
> > > > >> I have assigned this ticket to George, who is likely best
able to
> > > assist
> > > > >> you. Please allow a few business days for a response.
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >> Julie
> > > > >>
> > > > >> On Tue, May 19, 2020 at 1:53 PM Logan Dawson - NOAA
Affiliate via
> > RT <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >> >
> > > > >> > Tue May 19 13:53:41 2020: Request 95311 was acted upon.
> > > > >> > Transaction: Ticket created by logan.dawson at noaa.gov
> > > > >> > Queue: met_help
> > > > >> > Subject: Surrogate severe use case help
> > > > >> > Owner: Nobody
> > > > >> > Requestors: logan.dawson at noaa.gov
> > > > >> > Status: new
> > > > >> > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95311
> > > > >
> > > > >> >
> > > > >> >
> > > > >> > Hi,
> > > > >> >
> > > > >> > I'm emailing to consult with the team on a slight problem
I'm
> > having
> > > > >> with
> > > > >> > getting surrogate severe verification to work in METplus.
> > > > >> >
> > > > >> > I've been able to generate the surrogate severe forecast
for
> > various
> > > > >> models
> > > > >> > (using a mix of METplus tools and personal python code
for the
> > time
> > > > >> being).
> > > > >> > I have one forecast variable (based on one UH percentile
> > threshold),
> > > > and
> > > > >> > I'm having difficulty generating probabilistic
statistics. I am
> > able
> > > > to
> > > > >> > successfully generate CTC and CTS line types when
treating the
> > > > surrogate
> > > > >> > severe and practically perfect fields as scalars.
> > > > >> >
> > > > >> > I've double checked my METplus conf and MET config files
(2 of
> > each)
> > > > >> > against those in Tina's feature branch. Both METplus
calls run
> > > > >> > successfully, but the probabilistic stats call controlled
by
> this
> > > conf
> > > > >> > <
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
https://github.com/NCAR/METplus/blob/feature_414_surrogate_severe_evaluation/parm/use_cases/model_applications/convection_allowing_models/GridStat_fcstHRRR_obsPracPerfect_SurrogateSevereProb.conf
> > > > >> > >
> > > > >> > isn't producing any of the probabilistic statistic line
types.
> > > > >> >
> > > > >> > I've attached my METplus and MET configurations and
example
> output
> > > > file
> > > > >> > showing a lack of PXXX stats.
> > > > >> >
> > > > >> > I'm fairly new to running probabilistic verification with
MET or
> > > > >> METplus,
> > > > >> > so I could have something slightly off. If it's easier to
> discuss
> > my
> > > > >> > problem over Slack or video chat, I'd be happy to do so.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Logan
> > > > >> >
> > > > >> > --
> > > > >> > Logan C. Dawson, Ph.D.
> > > > >> > Physical Scientist, VPPPG/MEG
> > > > >> > NOAA/NWS/NCEP/EMC
> > > > >> > (301) 683-3944
> > > > >> >
> > > > >> >
> > > > >>
> > > > >> --
> > > > >> Julie Prestopnik
> > > > >> Software Engineer
> > > > >> National Center for Atmospheric Research
> > > > >> Research Applications Laboratory
> > > > >> Phone: 303.497.8399
> > > > >> Email: jpresto at ucar.edu
> > > > >>
> > > > >> My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > >> reply to this email outside of your normal working hours.
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Logan C. Dawson, Ph.D.
> > > Physical Scientist, VPPPG/MEG
> > > NOAA/NWS/NCEP/EMC
> > > (301) 683-3944
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Logan C. Dawson, Ph.D.
> Physical Scientist, VPPPG/MEG
> NOAA/NWS/NCEP/EMC
> (301) 683-3944
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list