[Met_help] [rt.rap.ucar.edu #69953] History for Probability verification using unconventional grib data

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hello. I was sure to look through the MET guide and on-line tutorials
before I sent you this email. I am still having trouble getting grid-stat
to produce a grid_stat_000000L_20140914_000000V_pstd.txt file with values
in it. I currently only get the header to print out.

Background:
I have 2 grib1 files, forecasted and observed probabilities of 34-kt winds
in a common domain. I ran grid-stat and got the nice NetCDF file to be
outputted, and some of the other text files have values printed out, but
not the one with Brier Score and ROC.

I'm not sure if I'm using the right settings in my configuration file. Zeus
is down now, but I was wondering what you recommend I use in my
configuration file. Here is what I can remember I used:

   field = [
     {
       name       = "LCDC";
       level      = "Z0";
       cat_thresh = [ >=0 ];
     }
   ];
obs = fcst;

I'm not sure how to use "thresholds" because my grib data already accounts
for thresholds. I have the probabilities of 34-kt winds. So, is
using cat_thresh = [ >=0 ] the right way so that I include all the data
points in the domain?

What other configuration settings would be important to set to compare
probability grids?

Thanks so much for your help.
-Matt


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Probability verification using unconventional grib data
From: John Halley Gotway
Time: Tue Dec 02 09:41:09 2014

Matt,

I believe that zeus is down today, so I can't look at your files
directly.
But hopefully I can point you in the right direction.

There is an example of verifying probabilities in the test scripts
included
with the MET release.  Take a look in:
   met-5.0/scripts/config/GridStatConfig_POP_12

Your config file should look something like this...

fcst = {
   wind_thresh = [ NA ];

   field = [
      {
        name         = "LCDC";
        level          = [ "L0" ];
        prob          = TRUE;
        cat_thresh = [ >=0.0, >=0.1, >=0.2, >=0.3, >=0.4, >=0.5,
>=0.6,
>=0.7, >=0.8, >=0.9, >=1.0 ];
      }
   ];

};
obs = {

   wind_thresh = [ NA ];

   field = [
      {
        name         = "WIND";
        level          = [ "Z2" ];
        cat_thresh = [ >=34 ];
      }
   ];

};

I'm not sure about the "name" in the forecast section without seeing
how
it's encoded in your GRIB file.  The PROB flag is set to TRUE to tell
grid_stat to process this as probability data.  The cat_thresh is set
to
partition the probability values between 0 and 1.

Since you're evaluating a forecast probability of wind speed exceeding
34kts, you're likely comparing it against the wind speed values.  And
you'd
set the observed cat_thresh to >=34 to be consistent with with the
forecast
probability definition.

Give that a shot and just let us know if you have more questions.

Thanks,
John

On Tue, Dec 2, 2014 at 7:47 AM, Matthew Morin - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:

>
> Tue Dec 02 07:47:36 2014: Request 69953 was acted upon.
> Transaction: Ticket created by matthew.morin at noaa.gov
>        Queue: met_help
>      Subject: Probability verification using unconventional grib
data
>        Owner: Nobody
>   Requestors: matthew.morin at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69953 >
>
>
> Hello. I was sure to look through the MET guide and on-line
tutorials
> before I sent you this email. I am still having trouble getting
grid-stat
> to produce a grid_stat_000000L_20140914_000000V_pstd.txt file with
values
> in it. I currently only get the header to print out.
>
> Background:
> I have 2 grib1 files, forecasted and observed probabilities of 34-kt
winds
> in a common domain. I ran grid-stat and got the nice NetCDF file to
be
> outputted, and some of the other text files have values printed out,
but
> not the one with Brier Score and ROC.
>
> I'm not sure if I'm using the right settings in my configuration
file. Zeus
> is down now, but I was wondering what you recommend I use in my
> configuration file. Here is what I can remember I used:
>
>    field = [
>      {
>        name       = "LCDC";
>        level      = "Z0";
>        cat_thresh = [ >=0 ];
>      }
>    ];
> obs = fcst;
>
> I'm not sure how to use "thresholds" because my grib data already
accounts
> for thresholds. I have the probabilities of 34-kt winds. So, is
> using cat_thresh = [ >=0 ] the right way so that I include all the
data
> points in the domain?
>
> What other configuration settings would be important to set to
compare
> probability grids?
>
> Thanks so much for your help.
> -Matt
>
>

------------------------------------------------
Subject: Probability verification using unconventional grib data
From: Matthew Morin - NOAA Affiliate
Time: Wed Dec 03 08:47:38 2014

Hi John,

That seemed to work perfectly. Thanks! I'm amazed about how easy it is
to
get so much good verification information using MET.

Thanks for your help!
Matt

On Tue, Dec 2, 2014 at 11:41 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Matt,
>
> I believe that zeus is down today, so I can't look at your files
directly.
> But hopefully I can point you in the right direction.
>
> There is an example of verifying probabilities in the test scripts
included
> with the MET release.  Take a look in:
>    met-5.0/scripts/config/GridStatConfig_POP_12
>
> Your config file should look something like this...
>
> fcst = {
>    wind_thresh = [ NA ];
>
>    field = [
>       {
>         name         = "LCDC";
>         level          = [ "L0" ];
>         prob          = TRUE;
>         cat_thresh = [ >=0.0, >=0.1, >=0.2, >=0.3, >=0.4, >=0.5,
>=0.6,
> >=0.7, >=0.8, >=0.9, >=1.0 ];
>       }
>    ];
>
> };
> obs = {
>
>    wind_thresh = [ NA ];
>
>    field = [
>       {
>         name         = "WIND";
>         level          = [ "Z2" ];
>         cat_thresh = [ >=34 ];
>       }
>    ];
>
> };
>
> I'm not sure about the "name" in the forecast section without seeing
how
> it's encoded in your GRIB file.  The PROB flag is set to TRUE to
tell
> grid_stat to process this as probability data.  The cat_thresh is
set to
> partition the probability values between 0 and 1.
>
> Since you're evaluating a forecast probability of wind speed
exceeding
> 34kts, you're likely comparing it against the wind speed values.
And you'd
> set the observed cat_thresh to >=34 to be consistent with with the
forecast
> probability definition.
>
> Give that a shot and just let us know if you have more questions.
>
> Thanks,
> John
>
> On Tue, Dec 2, 2014 at 7:47 AM, Matthew Morin - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Dec 02 07:47:36 2014: Request 69953 was acted upon.
> > Transaction: Ticket created by matthew.morin at noaa.gov
> >        Queue: met_help
> >      Subject: Probability verification using unconventional grib
data
> >        Owner: Nobody
> >   Requestors: matthew.morin at noaa.gov
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=69953 >
> >
> >
> > Hello. I was sure to look through the MET guide and on-line
tutorials
> > before I sent you this email. I am still having trouble getting
grid-stat
> > to produce a grid_stat_000000L_20140914_000000V_pstd.txt file with
values
> > in it. I currently only get the header to print out.
> >
> > Background:
> > I have 2 grib1 files, forecasted and observed probabilities of 34-
kt
> winds
> > in a common domain. I ran grid-stat and got the nice NetCDF file
to be
> > outputted, and some of the other text files have values printed
out, but
> > not the one with Brier Score and ROC.
> >
> > I'm not sure if I'm using the right settings in my configuration
file.
> Zeus
> > is down now, but I was wondering what you recommend I use in my
> > configuration file. Here is what I can remember I used:
> >
> >    field = [
> >      {
> >        name       = "LCDC";
> >        level      = "Z0";
> >        cat_thresh = [ >=0 ];
> >      }
> >    ];
> > obs = fcst;
> >
> > I'm not sure how to use "thresholds" because my grib data already
> accounts
> > for thresholds. I have the probabilities of 34-kt winds. So, is
> > using cat_thresh = [ >=0 ] the right way so that I include all the
data
> > points in the domain?
> >
> > What other configuration settings would be important to set to
compare
> > probability grids?
> >
> > Thanks so much for your help.
> > -Matt
>

------------------------------------------------