[Met_help] [rt.rap.ucar.edu #70329] History for verification of a thunderstorm event.
John Halley Gotway via RT
met_help at ucar.edu
Tue Jan 27 09:35:27 MST 2015
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi John.
I am trying to verify the results of my thunderstorm simulations using the neighborhood methods. This event will have RF over an area say 10 -20 locations with varying intensities.
The question that I am trying to find out is "whether the same is reproduced by the model or not???. To be more detailed, I would like to know if these intensities are reproduced. They may be displaced as expected from the model FCST field.
So I read that Upscaling is the basic method. But using this and computig POD etc may not be so relevant.
My questions are------------
1. what other neighborhood method will be most relevant in this case???
2. so If I am using Multievent Contingency Table, which is Single Observation and Neighborhhod FCST, How will my configuration file look like???
Thanks
Geeta
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: verification of a thunderstorm event.
From: John Halley Gotway
Time: Mon Jan 12 14:48:40 2015
Geeta,
The neighborhood methods in grid_stat will give you a way to quantify
how
much spatial error your forecast has. You pick a single threshold to
define a binary (yes/no) field. Then you compute the neighborhood
stats
using multiple neighborhood sizes. And you can rerun with many
different
thresholds. That'll tell you about spatial error, but it won't give
you a
good description of the range of intensity values.
Instead, I'd suggest you do an object-based verification using the
MODE
tool. You'd configure and run MODE to define objects in your forecast
and
observation field. MODE will run the configurable logic you specify
in the
config file to decide which forecast objects should be matched with
which
observation objects. The ASCII MODE output includes percentile values
of
the raw values inside the MODE objects. This would help you ignore
spatial
error and look to see if your forecast is reproducing intensity values
similar to those found in the observations.
John
On Sun, Jan 11, 2015 at 10:14 AM, Geeta Geeta via RT
<met_help at ucar.edu>
wrote:
>
> Sun Jan 11 10:14:37 2015: Request 70329 was acted upon.
> Transaction: Ticket created by geeta124 at hotmail.com
> Queue: met_help
> Subject: verification of a thunderstorm event.
> Owner: Nobody
> Requestors: geeta124 at hotmail.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
>
>
> Hi John.
> I am trying to verify the results of my thunderstorm simulations
using the
> neighborhood methods. This event will have RF over an area say 10
-20
> locations with varying intensities.
>
> The question that I am trying to find out is "whether the same is
> reproduced by the model or not???. To be more detailed, I would like
to
> know if these intensities are reproduced. They may be displaced as
expected
> from the model FCST field.
>
> So I read that Upscaling is the basic method. But using this and
computig
> POD etc may not be so relevant.
> My questions are------------
> 1. what other neighborhood method will be most relevant in this
case???
> 2. so If I am using Multievent Contingency Table, which is Single
> Observation and Neighborhhod FCST, How will my configuration file
look
> like???
>
> Thanks
> Geeta
>
>
------------------------------------------------
Subject: verification of a thunderstorm event.
From: Geeta Geeta
Time: Mon Jan 12 22:51:51 2015
thanks John I will surely try that. but as of now, I have following
questions which I would like to be guided with.
The matching strategy in the MCTC is SINGLE OBSERVATION-NEIGHBOURHOOD
FCST. In my config file, I have selected 2 thresholds, say>=0.1,
>=5.0. So there will be 3x3 contingency TABLE and the OBS-FCST pairs
will be put in one of the categories. Now my questions are
1. If I am interested in applying this method for my station (in this
case it will be gridded RF nearest to the station, is this correct???)
, then how will choose/define MY grid point of interest? (See I have a
observation grid of say 53x53 points and same for FCST also).
2. Once the grid point of MY interest in CHOSEN, the neighbourhood
around this has to be DEFINED. There is a provision of defining the
neighbourhood in the CONFIG file (interp_width1 or 3 or 5 or 7 ....).
BUT This neighbourhood has to be AROUND MY STATION/GRID point of
interest. How do I do that in my config file???.
3. want to know if I understood this correctly-------- " SINGLE
OBSERVATION-NEIGHBOURHOOD FCST" means that I will consider only ONE
OBSERVATION (at MY GRID point). THEN HOW does MET generate the
MPR's????.
4. Also amongst many SINGLE OBSERVATION-NEIGHBOURHOOD FCST decision
models, WHY the scores are different??? say for eg. MCTC, One has to
look at ROC,V and for Pragmatic approach (BS, BSS)????? (Ebert 2008).
5. in case of thunderstorm event simulations, I will have RF at my
station and around 50 km say. then Do I have to choose each station
independently and then define a neighborhood around this????
Kindly clear these doubts for me so that I can get started.
Geeta
> Subject: Re: [rt.rap.ucar.edu #70329] verification of a thunderstorm
event.
> From: met_help at ucar.edu
> To: geeta124 at hotmail.com
> Date: Mon, 12 Jan 2015 14:48:40 -0700
>
> Geeta,
>
> The neighborhood methods in grid_stat will give you a way to
quantify how
> much spatial error your forecast has. You pick a single threshold
to
> define a binary (yes/no) field. Then you compute the neighborhood
stats
> using multiple neighborhood sizes. And you can rerun with many
different
> thresholds. That'll tell you about spatial error, but it won't give
you a
> good description of the range of intensity values.
>
> Instead, I'd suggest you do an object-based verification using the
MODE
> tool. You'd configure and run MODE to define objects in your
forecast and
> observation field. MODE will run the configurable logic you specify
in the
> config file to decide which forecast objects should be matched with
which
> observation objects. The ASCII MODE output includes percentile
values of
> the raw values inside the MODE objects. This would help you ignore
spatial
> error and look to see if your forecast is reproducing intensity
values
> similar to those found in the observations.
>
> John
>
> On Sun, Jan 11, 2015 at 10:14 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Sun Jan 11 10:14:37 2015: Request 70329 was acted upon.
> > Transaction: Ticket created by geeta124 at hotmail.com
> > Queue: met_help
> > Subject: verification of a thunderstorm event.
> > Owner: Nobody
> > Requestors: geeta124 at hotmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
> >
> >
> > Hi John.
> > I am trying to verify the results of my thunderstorm simulations
using the
> > neighborhood methods. This event will have RF over an area say 10
-20
> > locations with varying intensities.
> >
> > The question that I am trying to find out is "whether the same is
> > reproduced by the model or not???. To be more detailed, I would
like to
> > know if these intensities are reproduced. They may be displaced as
expected
> > from the model FCST field.
> >
> > So I read that Upscaling is the basic method. But using this and
computig
> > POD etc may not be so relevant.
> > My questions are------------
> > 1. what other neighborhood method will be most relevant in this
case???
> > 2. so If I am using Multievent Contingency Table, which is Single
> > Observation and Neighborhhod FCST, How will my configuration file
look
> > like???
> >
> > Thanks
> > Geeta
> >
> >
>
------------------------------------------------
Subject: verification of a thunderstorm event.
From: Geeta Geeta
Time: Tue Jan 13 10:08:02 2015
Hi John.
I have following entries in my config file.
// Neighborhood methods
//
nbrhd = {
vld_thresh = 1.0;
width = [ 1 ];
cov_thresh = [ >=0.5 ];
}
what does the last vld_thresh and cov_threh mean ?????
Kindly explain
Geeta
From: geeta124 at hotmail.com
To: met_help at ucar.edu
Subject: RE: [rt.rap.ucar.edu #70329] verification of a thunderstorm
event.
Date: Tue, 13 Jan 2015 11:21:46 +0530
thanks John I will surely try that. but as of now, I have following
questions which I would like to be guided with.
The matching strategy in the MCTC is SINGLE OBSERVATION-NEIGHBOURHOOD
FCST. In my config file, I have selected 2 thresholds, say>=0.1,
>=5.0. So there will be 3x3 contingency TABLE and the OBS-FCST pairs
will be put in one of the categories. Now my questions are
1. If I am interested in applying this method for my station (in this
case it will be gridded RF nearest to the station, is this correct???)
, then how will choose/define MY grid point of interest? (See I have a
observation grid of say 53x53 points and same for FCST also).
2. Once the grid point of MY interest in CHOSEN, the neighbourhood
around this has to be DEFINED. There is a provision of defining the
neighbourhood in the CONFIG file (interp_width1 or 3 or 5 or 7 ....).
BUT This neighbourhood has to be AROUND MY STATION/GRID point of
interest. How do I do that in my config file???.
3. want to know if I understood this correctly-------- " SINGLE
OBSERVATION-NEIGHBOURHOOD FCST" means that I will consider only ONE
OBSERVATION (at MY GRID point). THEN HOW does MET generate the
MPR's????.
4. Also amongst many SINGLE OBSERVATION-NEIGHBOURHOOD FCST decision
models, WHY the scores are different??? say for eg. MCTC, One has to
look at ROC,V and for Pragmatic approach (BS, BSS)????? (Ebert 2008).
5. in case of thunderstorm event simulations, I will have RF at my
station and around 50 km say. then Do I have to choose each station
independently and then define a neighborhood around this????
Kindly clear these doubts for me so that I can get started.
Geeta
> Subject: Re: [rt.rap.ucar.edu #70329] verification of a thunderstorm
event.
> From: met_help at ucar.edu
> To: geeta124 at hotmail.com
> Date: Mon, 12 Jan 2015 14:48:40 -0700
>
> Geeta,
>
> The neighborhood methods in grid_stat will give you a way to
quantify how
> much spatial error your forecast has. You pick a single threshold
to
> define a binary (yes/no) field. Then you compute the neighborhood
stats
> using multiple neighborhood sizes. And you can rerun with many
different
> thresholds. That'll tell you about spatial error, but it won't give
you a
> good description of the range of intensity values.
>
> Instead, I'd suggest you do an object-based verification using the
MODE
> tool. You'd configure and run MODE to define objects in your
forecast and
> observation field. MODE will run the configurable logic you specify
in the
> config file to decide which forecast objects should be matched with
which
> observation objects. The ASCII MODE output includes percentile
values of
> the raw values inside the MODE objects. This would help you ignore
spatial
> error and look to see if your forecast is reproducing intensity
values
> similar to those found in the observations.
>
> John
>
> On Sun, Jan 11, 2015 at 10:14 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Sun Jan 11 10:14:37 2015: Request 70329 was acted upon.
> > Transaction: Ticket created by geeta124 at hotmail.com
> > Queue: met_help
> > Subject: verification of a thunderstorm event.
> > Owner: Nobody
> > Requestors: geeta124 at hotmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
> >
> >
> > Hi John.
> > I am trying to verify the results of my thunderstorm simulations
using the
> > neighborhood methods. This event will have RF over an area say 10
-20
> > locations with varying intensities.
> >
> > The question that I am trying to find out is "whether the same is
> > reproduced by the model or not???. To be more detailed, I would
like to
> > know if these intensities are reproduced. They may be displaced as
expected
> > from the model FCST field.
> >
> > So I read that Upscaling is the basic method. But using this and
computig
> > POD etc may not be so relevant.
> > My questions are------------
> > 1. what other neighborhood method will be most relevant in this
case???
> > 2. so If I am using Multievent Contingency Table, which is Single
> > Observation and Neighborhhod FCST, How will my configuration file
look
> > like???
> >
> > Thanks
> > Geeta
> >
> >
>
------------------------------------------------
Subject: verification of a thunderstorm event.
From: John Halley Gotway
Time: Tue Jan 13 15:59:09 2015
Geeta,
Grid-Stat applies the following logic to compute neighborhood
statistics.
It computes neighborhood info for each combination of variable
(fcst.field), categorical threshold (cat_thresh), and neighborhood
width
(width).
Suppose you're evaluating accumulated precip using a threshold of
>25.4
(i.e. > 1 inch) using neighborhood width of 3, 5, 7, and 9. For width
3,
it'll process each grid point by looking at a 3x3 box centered on that
grid
point. It'll count up the number of those 9 values that have precip >
1
inch. Let's suppose 5 of them do. So the "fractional coverage" value
at
that point is 5/9 = 0.56.
The "vld_thresh" determines how many of those 9 grid points need to
contain
valid data for a fractional coverage value to be computed. Setting it
to 1
requires that they all must contain valid data. If you're at the edge
of
the grid or have sporadic missing data, you'll have to decide how much
missing data you want to allow. Setting vld_thresh = 0.5 would only
require that 1/2 of the grid point contain valid data.
I "cov_thresh" value is used to compute NBRCTC and NBRCTS output
lines, but
I recommend that you not use them. Instead, just use NBRCNT, which
contains fractions skill score, which is the most commonly used
neighborhood statistic.
I'd suggest setting width = [ 3, 5, 7, 9, 11 ]; and then look to see
how
FSS changes as you increase the neighborhood size.
Hope that helps.
Thanks,
John
(1) For each each neighborhood width, it computes a fraction
On Tue, Jan 13, 2015 at 10:08 AM, Geeta Geeta via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
>
> Hi John.
> I have following entries in my config file.
> // Neighborhood methods
> //
> nbrhd = {
> vld_thresh = 1.0;
> width = [ 1 ];
> cov_thresh = [ >=0.5 ];
> }
>
> what does the last vld_thresh and cov_threh mean ?????
> Kindly explain
> Geeta
> From: geeta124 at hotmail.com
> To: met_help at ucar.edu
> Subject: RE: [rt.rap.ucar.edu #70329] verification of a thunderstorm
> event.
> Date: Tue, 13 Jan 2015 11:21:46 +0530
>
>
>
>
> thanks John I will surely try that. but as of now, I have following
> questions which I would like to be guided with.
> The matching strategy in the MCTC is SINGLE OBSERVATION-
NEIGHBOURHOOD
> FCST. In my config file, I have selected 2 thresholds, say>=0.1,
>=5.0. So
> there will be 3x3 contingency TABLE and the OBS-FCST pairs will be
put in
> one of the categories. Now my questions are
> 1. If I am interested in applying this method for my station (in
this case
> it will be gridded RF nearest to the station, is this correct???) ,
then
> how will choose/define MY grid point of interest? (See I have a
observation
> grid of say 53x53 points and same for FCST also).
> 2. Once the grid point of MY interest in CHOSEN, the neighbourhood
around
> this has to be DEFINED. There is a provision of defining the
neighbourhood
> in the CONFIG file (interp_width1 or 3 or 5 or 7 ....). BUT This
> neighbourhood has to be AROUND MY STATION/GRID point of interest.
How do I
> do that in my config file???.
> 3. want to know if I understood this correctly-------- " SINGLE
> OBSERVATION-NEIGHBOURHOOD FCST" means that I will consider only ONE
> OBSERVATION (at MY GRID point). THEN HOW does MET generate the
MPR's????.
> 4. Also amongst many SINGLE OBSERVATION-NEIGHBOURHOOD FCST decision
> models, WHY the scores are different??? say for eg. MCTC, One has to
look
> at ROC,V and for Pragmatic approach (BS, BSS)????? (Ebert 2008).
> 5. in case of thunderstorm event simulations, I will have RF at my
station
> and around 50 km say. then Do I have to choose each station
independently
> and then define a neighborhood around this????
>
>
> Kindly clear these doubts for me so that I can get started.
> Geeta
>
> > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> event.
> > From: met_help at ucar.edu
> > To: geeta124 at hotmail.com
> > Date: Mon, 12 Jan 2015 14:48:40 -0700
> >
> > Geeta,
> >
> > The neighborhood methods in grid_stat will give you a way to
quantify how
> > much spatial error your forecast has. You pick a single threshold
to
> > define a binary (yes/no) field. Then you compute the neighborhood
stats
> > using multiple neighborhood sizes. And you can rerun with many
different
> > thresholds. That'll tell you about spatial error, but it won't
give you
> a
> > good description of the range of intensity values.
> >
> > Instead, I'd suggest you do an object-based verification using the
MODE
> > tool. You'd configure and run MODE to define objects in your
forecast
> and
> > observation field. MODE will run the configurable logic you
specify in
> the
> > config file to decide which forecast objects should be matched
with which
> > observation objects. The ASCII MODE output includes percentile
values of
> > the raw values inside the MODE objects. This would help you
ignore
> spatial
> > error and look to see if your forecast is reproducing intensity
values
> > similar to those found in the observations.
> >
> > John
> >
> > On Sun, Jan 11, 2015 at 10:14 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Sun Jan 11 10:14:37 2015: Request 70329 was acted upon.
> > > Transaction: Ticket created by geeta124 at hotmail.com
> > > Queue: met_help
> > > Subject: verification of a thunderstorm event.
> > > Owner: Nobody
> > > Requestors: geeta124 at hotmail.com
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329
> >
> > >
> > >
> > > Hi John.
> > > I am trying to verify the results of my thunderstorm simulations
using
> the
> > > neighborhood methods. This event will have RF over an area say
10 -20
> > > locations with varying intensities.
> > >
> > > The question that I am trying to find out is "whether the same
is
> > > reproduced by the model or not???. To be more detailed, I would
like to
> > > know if these intensities are reproduced. They may be displaced
as
> expected
> > > from the model FCST field.
> > >
> > > So I read that Upscaling is the basic method. But using this and
> computig
> > > POD etc may not be so relevant.
> > > My questions are------------
> > > 1. what other neighborhood method will be most relevant in this
case???
> > > 2. so If I am using Multievent Contingency Table, which is
Single
> > > Observation and Neighborhhod FCST, How will my configuration
file look
> > > like???
> > >
> > > Thanks
> > > Geeta
> > >
> > >
> >
>
>
------------------------------------------------
Subject: verification of a thunderstorm event.
From: Geeta Geeta
Time: Tue Jan 13 22:18:09 2015
thanks john. I would like some more details of the following ----
1. what would be the difference b/w NBRCTC and MCTC outputs or how
the NBRCTC and MCTC and MCTS outputs are calculated (methodology) ???
2. As I was discussing, I have done the some simulations of
Thunderstorm events. I want to verify the RF output. See I have model
output which has 177x177 points in the X and Y directions. The model
output is at 5 km resolution. MY point of interest does NOT lay in the
centre of the model domain. So If I wish to use the SINGLE
OBSERVATION-NEIGHBOURHOOD type of decision model, (or multi-event-
contingency count contingency table), then shall I extract (or make a
subset of the model data) the model data around my GRID POINT of
interest (say 10x10 grid points with MY GRID POINT at the centre).
and then compute the MCTC/ NBRCTC contingency Tables???
I had written a few points in my earlier mail. Kindly explain those
points to me. Unless I am not clear of what I am doing, I cannot
interpret my results well. another point that is not clear to me is
the use of different metrics for different type of Neighbourhood
methods.
Geeta
> Subject: Re: [rt.rap.ucar.edu #70329] verification of a thunderstorm
event.
> From: met_help at ucar.edu
> To: geeta124 at hotmail.com
> Date: Tue, 13 Jan 2015 15:59:09 -0700
>
> Geeta,
>
> Grid-Stat applies the following logic to compute neighborhood
statistics.
> It computes neighborhood info for each combination of variable
> (fcst.field), categorical threshold (cat_thresh), and neighborhood
width
> (width).
>
> Suppose you're evaluating accumulated precip using a threshold of
>25.4
> (i.e. > 1 inch) using neighborhood width of 3, 5, 7, and 9. For
width 3,
> it'll process each grid point by looking at a 3x3 box centered on
that grid
> point. It'll count up the number of those 9 values that have precip
> 1
> inch. Let's suppose 5 of them do. So the "fractional coverage"
value at
> that point is 5/9 = 0.56.
>
> The "vld_thresh" determines how many of those 9 grid points need to
contain
> valid data for a fractional coverage value to be computed. Setting
it to 1
> requires that they all must contain valid data. If you're at the
edge of
> the grid or have sporadic missing data, you'll have to decide how
much
> missing data you want to allow. Setting vld_thresh = 0.5 would only
> require that 1/2 of the grid point contain valid data.
>
> I "cov_thresh" value is used to compute NBRCTC and NBRCTS output
lines, but
> I recommend that you not use them. Instead, just use NBRCNT, which
> contains fractions skill score, which is the most commonly used
> neighborhood statistic.
>
> I'd suggest setting width = [ 3, 5, 7, 9, 11 ]; and then look to see
how
> FSS changes as you increase the neighborhood size.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> (1) For each each neighborhood width, it computes a fraction
>
> On Tue, Jan 13, 2015 at 10:08 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
> >
> > Hi John.
> > I have following entries in my config file.
> > // Neighborhood methods
> > //
> > nbrhd = {
> > vld_thresh = 1.0;
> > width = [ 1 ];
> > cov_thresh = [ >=0.5 ];
> > }
> >
> > what does the last vld_thresh and cov_threh mean ?????
> > Kindly explain
> > Geeta
> > From: geeta124 at hotmail.com
> > To: met_help at ucar.edu
> > Subject: RE: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > event.
> > Date: Tue, 13 Jan 2015 11:21:46 +0530
> >
> >
> >
> >
> > thanks John I will surely try that. but as of now, I have
following
> > questions which I would like to be guided with.
> > The matching strategy in the MCTC is SINGLE OBSERVATION-
NEIGHBOURHOOD
> > FCST. In my config file, I have selected 2 thresholds, say>=0.1,
>=5.0. So
> > there will be 3x3 contingency TABLE and the OBS-FCST pairs will
be put in
> > one of the categories. Now my questions are
> > 1. If I am interested in applying this method for my station (in
this case
> > it will be gridded RF nearest to the station, is this correct???)
, then
> > how will choose/define MY grid point of interest? (See I have a
observation
> > grid of say 53x53 points and same for FCST also).
> > 2. Once the grid point of MY interest in CHOSEN, the neighbourhood
around
> > this has to be DEFINED. There is a provision of defining the
neighbourhood
> > in the CONFIG file (interp_width1 or 3 or 5 or 7 ....). BUT This
> > neighbourhood has to be AROUND MY STATION/GRID point of interest.
How do I
> > do that in my config file???.
> > 3. want to know if I understood this correctly-------- " SINGLE
> > OBSERVATION-NEIGHBOURHOOD FCST" means that I will consider only
ONE
> > OBSERVATION (at MY GRID point). THEN HOW does MET generate the
MPR's????.
> > 4. Also amongst many SINGLE OBSERVATION-NEIGHBOURHOOD FCST
decision
> > models, WHY the scores are different??? say for eg. MCTC, One has
to look
> > at ROC,V and for Pragmatic approach (BS, BSS)????? (Ebert 2008).
> > 5. in case of thunderstorm event simulations, I will have RF at my
station
> > and around 50 km say. then Do I have to choose each station
independently
> > and then define a neighborhood around this????
> >
> >
> > Kindly clear these doubts for me so that I can get started.
> > Geeta
> >
> > > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > event.
> > > From: met_help at ucar.edu
> > > To: geeta124 at hotmail.com
> > > Date: Mon, 12 Jan 2015 14:48:40 -0700
> > >
> > > Geeta,
> > >
> > > The neighborhood methods in grid_stat will give you a way to
quantify how
> > > much spatial error your forecast has. You pick a single
threshold to
> > > define a binary (yes/no) field. Then you compute the
neighborhood stats
> > > using multiple neighborhood sizes. And you can rerun with many
different
> > > thresholds. That'll tell you about spatial error, but it won't
give you
> > a
> > > good description of the range of intensity values.
> > >
> > > Instead, I'd suggest you do an object-based verification using
the MODE
> > > tool. You'd configure and run MODE to define objects in your
forecast
> > and
> > > observation field. MODE will run the configurable logic you
specify in
> > the
> > > config file to decide which forecast objects should be matched
with which
> > > observation objects. The ASCII MODE output includes percentile
values of
> > > the raw values inside the MODE objects. This would help you
ignore
> > spatial
> > > error and look to see if your forecast is reproducing intensity
values
> > > similar to those found in the observations.
> > >
> > > John
> > >
> > > On Sun, Jan 11, 2015 at 10:14 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Sun Jan 11 10:14:37 2015: Request 70329 was acted upon.
> > > > Transaction: Ticket created by geeta124 at hotmail.com
> > > > Queue: met_help
> > > > Subject: verification of a thunderstorm event.
> > > > Owner: Nobody
> > > > Requestors: geeta124 at hotmail.com
> > > > Status: new
> > > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329
> > >
> > > >
> > > >
> > > > Hi John.
> > > > I am trying to verify the results of my thunderstorm
simulations using
> > the
> > > > neighborhood methods. This event will have RF over an area say
10 -20
> > > > locations with varying intensities.
> > > >
> > > > The question that I am trying to find out is "whether the same
is
> > > > reproduced by the model or not???. To be more detailed, I
would like to
> > > > know if these intensities are reproduced. They may be
displaced as
> > expected
> > > > from the model FCST field.
> > > >
> > > > So I read that Upscaling is the basic method. But using this
and
> > computig
> > > > POD etc may not be so relevant.
> > > > My questions are------------
> > > > 1. what other neighborhood method will be most relevant in
this case???
> > > > 2. so If I am using Multievent Contingency Table, which is
Single
> > > > Observation and Neighborhhod FCST, How will my configuration
file look
> > > > like???
> > > >
> > > > Thanks
> > > > Geeta
> > > >
> > > >
> > >
> >
> >
>
------------------------------------------------
Subject: verification of a thunderstorm event.
From: Geeta Geeta
Time: Thu Jan 15 07:16:54 2015
Hi John.
Kindly guide me further to my questions. I was waiting for your reply.
regards
geeta
From: geeta124 at hotmail.com
To: met_help at ucar.edu
Subject: RE: [rt.rap.ucar.edu #70329] verification of a thunderstorm
event.
Date: Wed, 14 Jan 2015 10:48:05 +0530
thanks john. I would like some more details of the following ----
1. what would be the difference b/w NBRCTC and MCTC outputs or how
the NBRCTC and MCTC and MCTS outputs are calculated (methodology) ???
2. As I was discussing, I have done the some simulations of
Thunderstorm events. I want to verify the RF output. See I have model
output which has 177x177 points in the X and Y directions. The model
output is at 5 km resolution. MY point of interest does NOT lay in the
centre of the model domain. So If I wish to use the SINGLE
OBSERVATION-NEIGHBOURHOOD type of decision model, (or multi-event-
contingency count contingency table), then shall I extract (or make a
subset of the model data) the model data around my GRID POINT of
interest (say 10x10 grid points with MY GRID POINT at the centre).
and then compute the MCTC/ NBRCTC contingency Tables???
I had written a few points in my earlier mail. Kindly explain those
points to me. Unless I am not clear of what I am doing, I cannot
interpret my results well. another point that is not clear to me is
the use of different metrics for different type of Neighbourhood
methods.
Geeta
> Subject: Re: [rt.rap.ucar.edu #70329] verification of a thunderstorm
event.
> From: met_help at ucar.edu
> To: geeta124 at hotmail.com
> Date: Tue, 13 Jan 2015 15:59:09 -0700
>
> Geeta,
>
> Grid-Stat applies the following logic to compute neighborhood
statistics.
> It computes neighborhood info for each combination of variable
> (fcst.field), categorical threshold (cat_thresh), and neighborhood
width
> (width).
>
> Suppose you're evaluating accumulated precip using a threshold of
>25.4
> (i.e. > 1 inch) using neighborhood width of 3, 5, 7, and 9. For
width 3,
> it'll process each grid point by looking at a 3x3 box centered on
that grid
> point. It'll count up the number of those 9 values that have precip
> 1
> inch. Let's suppose 5 of them do. So the "fractional coverage"
value at
> that point is 5/9 = 0.56.
>
> The "vld_thresh" determines how many of those 9 grid points need to
contain
> valid data for a fractional coverage value to be computed. Setting
it to 1
> requires that they all must contain valid data. If you're at the
edge of
> the grid or have sporadic missing data, you'll have to decide how
much
> missing data you want to allow. Setting vld_thresh = 0.5 would only
> require that 1/2 of the grid point contain valid data.
>
> I "cov_thresh" value is used to compute NBRCTC and NBRCTS output
lines, but
> I recommend that you not use them. Instead, just use NBRCNT, which
> contains fractions skill score, which is the most commonly used
> neighborhood statistic.
>
> I'd suggest setting width = [ 3, 5, 7, 9, 11 ]; and then look to see
how
> FSS changes as you increase the neighborhood size.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> (1) For each each neighborhood width, it computes a fraction
>
> On Tue, Jan 13, 2015 at 10:08 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
> >
> > Hi John.
> > I have following entries in my config file.
> > // Neighborhood methods
> > //
> > nbrhd = {
> > vld_thresh = 1.0;
> > width = [ 1 ];
> > cov_thresh = [ >=0.5 ];
> > }
> >
> > what does the last vld_thresh and cov_threh mean ?????
> > Kindly explain
> > Geeta
> > From: geeta124 at hotmail.com
> > To: met_help at ucar.edu
> > Subject: RE: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > event.
> > Date: Tue, 13 Jan 2015 11:21:46 +0530
> >
> >
> >
> >
> > thanks John I will surely try that. but as of now, I have
following
> > questions which I would like to be guided with.
> > The matching strategy in the MCTC is SINGLE OBSERVATION-
NEIGHBOURHOOD
> > FCST. In my config file, I have selected 2 thresholds, say>=0.1,
>=5.0. So
> > there will be 3x3 contingency TABLE and the OBS-FCST pairs will
be put in
> > one of the categories. Now my questions are
> > 1. If I am interested in applying this method for my station (in
this case
> > it will be gridded RF nearest to the station, is this correct???)
, then
> > how will choose/define MY grid point of interest? (See I have a
observation
> > grid of say 53x53 points and same for FCST also).
> > 2. Once the grid point of MY interest in CHOSEN, the neighbourhood
around
> > this has to be DEFINED. There is a provision of defining the
neighbourhood
> > in the CONFIG file (interp_width1 or 3 or 5 or 7 ....). BUT This
> > neighbourhood has to be AROUND MY STATION/GRID point of interest.
How do I
> > do that in my config file???.
> > 3. want to know if I understood this correctly-------- " SINGLE
> > OBSERVATION-NEIGHBOURHOOD FCST" means that I will consider only
ONE
> > OBSERVATION (at MY GRID point). THEN HOW does MET generate the
MPR's????.
> > 4. Also amongst many SINGLE OBSERVATION-NEIGHBOURHOOD FCST
decision
> > models, WHY the scores are different??? say for eg. MCTC, One has
to look
> > at ROC,V and for Pragmatic approach (BS, BSS)????? (Ebert 2008).
> > 5. in case of thunderstorm event simulations, I will have RF at my
station
> > and around 50 km say. then Do I have to choose each station
independently
> > and then define a neighborhood around this????
> >
> >
> > Kindly clear these doubts for me so that I can get started.
> > Geeta
> >
> > > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > event.
> > > From: met_help at ucar.edu
> > > To: geeta124 at hotmail.com
> > > Date: Mon, 12 Jan 2015 14:48:40 -0700
> > >
> > > Geeta,
> > >
> > > The neighborhood methods in grid_stat will give you a way to
quantify how
> > > much spatial error your forecast has. You pick a single
threshold to
> > > define a binary (yes/no) field. Then you compute the
neighborhood stats
> > > using multiple neighborhood sizes. And you can rerun with many
different
> > > thresholds. That'll tell you about spatial error, but it won't
give you
> > a
> > > good description of the range of intensity values.
> > >
> > > Instead, I'd suggest you do an object-based verification using
the MODE
> > > tool. You'd configure and run MODE to define objects in your
forecast
> > and
> > > observation field. MODE will run the configurable logic you
specify in
> > the
> > > config file to decide which forecast objects should be matched
with which
> > > observation objects. The ASCII MODE output includes percentile
values of
> > > the raw values inside the MODE objects. This would help you
ignore
> > spatial
> > > error and look to see if your forecast is reproducing intensity
values
> > > similar to those found in the observations.
> > >
> > > John
> > >
> > > On Sun, Jan 11, 2015 at 10:14 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Sun Jan 11 10:14:37 2015: Request 70329 was acted upon.
> > > > Transaction: Ticket created by geeta124 at hotmail.com
> > > > Queue: met_help
> > > > Subject: verification of a thunderstorm event.
> > > > Owner: Nobody
> > > > Requestors: geeta124 at hotmail.com
> > > > Status: new
> > > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329
> > >
> > > >
> > > >
> > > > Hi John.
> > > > I am trying to verify the results of my thunderstorm
simulations using
> > the
> > > > neighborhood methods. This event will have RF over an area say
10 -20
> > > > locations with varying intensities.
> > > >
> > > > The question that I am trying to find out is "whether the same
is
> > > > reproduced by the model or not???. To be more detailed, I
would like to
> > > > know if these intensities are reproduced. They may be
displaced as
> > expected
> > > > from the model FCST field.
> > > >
> > > > So I read that Upscaling is the basic method. But using this
and
> > computig
> > > > POD etc may not be so relevant.
> > > > My questions are------------
> > > > 1. what other neighborhood method will be most relevant in
this case???
> > > > 2. so If I am using Multievent Contingency Table, which is
Single
> > > > Observation and Neighborhhod FCST, How will my configuration
file look
> > > > like???
> > > >
> > > > Thanks
> > > > Geeta
> > > >
> > > >
> > >
> >
> >
>
------------------------------------------------
Subject: verification of a thunderstorm event.
From: John Halley Gotway
Time: Thu Jan 15 10:08:02 2015
Geeta,
The NBRCTC and MCTC output lines contain different information.
As I mentioned, I would recommend using the NBRCNT output line which
contains the fractions skill score (FSS). That's the most commonly
used
neighborhood method. I wouldn't recommend using NBRCTC and NBRCTS at
this
point.
The MCTC is the multi-category contingency table. As we've been
discussing, if you specify n categorical thresholds, MET will use
those to
define an (n+1) x (n+1) multi-category contingency table. The MCTC
output
line contains the counts for each of the cells in that table. And for
each
MCTC line, you'll see a corresponding MCTS line containing the
statistics
derived from that multi-category contingency table.
I am not familiar with the "the SINGLE OBSERVATION-NEIGHBOURHOOD type
of
decision mode". Based on your description, it sounds like you're
interested in your model performance over a very small subset of your
domain. I'm assuming you already have your forecast and observation
data
on the same grid. To do an evaluation over a small subset of your
domain,
just define a masking region for that area. The simplest would be to
define a polyline region of at least 3 lat/lon value. You could pass
that
polyline to grid_stat directly in the configuration file (vx_mask) or
run
it through the gen_poly_mask tool first to define a bitmap - and then
pass
it's output to grid_stat in the configuration file.
Is that the information you're looking for?
Thanks,
John
On Thu, Jan 15, 2015 at 7:16 AM, Geeta Geeta via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
>
> Hi John.
> Kindly guide me further to my questions. I was waiting for your
reply.
>
> regards
> geeta
>
> From: geeta124 at hotmail.com
> To: met_help at ucar.edu
> Subject: RE: [rt.rap.ucar.edu #70329] verification of a thunderstorm
> event.
> Date: Wed, 14 Jan 2015 10:48:05 +0530
>
>
>
>
> thanks john. I would like some more details of the following ----
> 1. what would be the difference b/w NBRCTC and MCTC outputs or how
the
> NBRCTC and MCTC and MCTS outputs are calculated (methodology) ???
> 2. As I was discussing, I have done the some simulations of
Thunderstorm
> events. I want to verify the RF output. See I have model output
which has
> 177x177 points in the X and Y directions. The model output is at 5
km
> resolution. MY point of interest does NOT lay in the centre of the
model
> domain. So If I wish to use the SINGLE OBSERVATION-NEIGHBOURHOOD
type of
> decision model, (or multi-event-contingency count contingency
table), then
> shall I extract (or make a subset of the model data) the model data
around
> my GRID POINT of interest (say 10x10 grid points with MY GRID POINT
at the
> centre).
> and then compute the MCTC/ NBRCTC contingency Tables???
> I had written a few points in my earlier mail. Kindly explain those
> points to me. Unless I am not clear of what I am doing, I cannot
interpret
> my results well. another point that is not clear to me is the use of
> different metrics for different type of Neighbourhood methods.
> Geeta
>
> > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> event.
> > From: met_help at ucar.edu
> > To: geeta124 at hotmail.com
> > Date: Tue, 13 Jan 2015 15:59:09 -0700
> >
> > Geeta,
> >
> > Grid-Stat applies the following logic to compute neighborhood
statistics.
> > It computes neighborhood info for each combination of variable
> > (fcst.field), categorical threshold (cat_thresh), and neighborhood
width
> > (width).
> >
> > Suppose you're evaluating accumulated precip using a threshold of
>25.4
> > (i.e. > 1 inch) using neighborhood width of 3, 5, 7, and 9. For
width 3,
> > it'll process each grid point by looking at a 3x3 box centered on
that
> grid
> > point. It'll count up the number of those 9 values that have
precip > 1
> > inch. Let's suppose 5 of them do. So the "fractional coverage"
value at
> > that point is 5/9 = 0.56.
> >
> > The "vld_thresh" determines how many of those 9 grid points need
to
> contain
> > valid data for a fractional coverage value to be computed.
Setting it
> to 1
> > requires that they all must contain valid data. If you're at the
edge of
> > the grid or have sporadic missing data, you'll have to decide how
much
> > missing data you want to allow. Setting vld_thresh = 0.5 would
only
> > require that 1/2 of the grid point contain valid data.
> >
> > I "cov_thresh" value is used to compute NBRCTC and NBRCTS output
lines,
> but
> > I recommend that you not use them. Instead, just use NBRCNT,
which
> > contains fractions skill score, which is the most commonly used
> > neighborhood statistic.
> >
> > I'd suggest setting width = [ 3, 5, 7, 9, 11 ]; and then look to
see how
> > FSS changes as you increase the neighborhood size.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> > (1) For each each neighborhood width, it computes a fraction
> >
> > On Tue, Jan 13, 2015 at 10:08 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
> > >
> > > Hi John.
> > > I have following entries in my config file.
> > > // Neighborhood methods
> > > //
> > > nbrhd = {
> > > vld_thresh = 1.0;
> > > width = [ 1 ];
> > > cov_thresh = [ >=0.5 ];
> > > }
> > >
> > > what does the last vld_thresh and cov_threh mean ?????
> > > Kindly explain
> > > Geeta
> > > From: geeta124 at hotmail.com
> > > To: met_help at ucar.edu
> > > Subject: RE: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > > event.
> > > Date: Tue, 13 Jan 2015 11:21:46 +0530
> > >
> > >
> > >
> > >
> > > thanks John I will surely try that. but as of now, I have
following
> > > questions which I would like to be guided with.
> > > The matching strategy in the MCTC is SINGLE OBSERVATION-
NEIGHBOURHOOD
> > > FCST. In my config file, I have selected 2 thresholds, say>=0.1,
> >=5.0. So
> > > there will be 3x3 contingency TABLE and the OBS-FCST pairs will
be
> put in
> > > one of the categories. Now my questions are
> > > 1. If I am interested in applying this method for my station (in
this
> case
> > > it will be gridded RF nearest to the station, is this
correct???) ,
> then
> > > how will choose/define MY grid point of interest? (See I have a
> observation
> > > grid of say 53x53 points and same for FCST also).
> > > 2. Once the grid point of MY interest in CHOSEN, the
neighbourhood
> around
> > > this has to be DEFINED. There is a provision of defining the
> neighbourhood
> > > in the CONFIG file (interp_width1 or 3 or 5 or 7 ....). BUT This
> > > neighbourhood has to be AROUND MY STATION/GRID point of
interest. How
> do I
> > > do that in my config file???.
> > > 3. want to know if I understood this correctly-------- " SINGLE
> > > OBSERVATION-NEIGHBOURHOOD FCST" means that I will consider only
ONE
> > > OBSERVATION (at MY GRID point). THEN HOW does MET generate the
> MPR's????.
> > > 4. Also amongst many SINGLE OBSERVATION-NEIGHBOURHOOD FCST
decision
> > > models, WHY the scores are different??? say for eg. MCTC, One
has to
> look
> > > at ROC,V and for Pragmatic approach (BS, BSS)????? (Ebert 2008).
> > > 5. in case of thunderstorm event simulations, I will have RF at
my
> station
> > > and around 50 km say. then Do I have to choose each station
> independently
> > > and then define a neighborhood around this????
> > >
> > >
> > > Kindly clear these doubts for me so that I can get started.
> > > Geeta
> > >
> > > > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > > event.
> > > > From: met_help at ucar.edu
> > > > To: geeta124 at hotmail.com
> > > > Date: Mon, 12 Jan 2015 14:48:40 -0700
> > > >
> > > > Geeta,
> > > >
> > > > The neighborhood methods in grid_stat will give you a way to
> quantify how
> > > > much spatial error your forecast has. You pick a single
threshold to
> > > > define a binary (yes/no) field. Then you compute the
neighborhood
> stats
> > > > using multiple neighborhood sizes. And you can rerun with
many
> different
> > > > thresholds. That'll tell you about spatial error, but it
won't give
> you
> > > a
> > > > good description of the range of intensity values.
> > > >
> > > > Instead, I'd suggest you do an object-based verification using
the
> MODE
> > > > tool. You'd configure and run MODE to define objects in your
> forecast
> > > and
> > > > observation field. MODE will run the configurable logic you
specify
> in
> > > the
> > > > config file to decide which forecast objects should be matched
with
> which
> > > > observation objects. The ASCII MODE output includes
percentile
> values of
> > > > the raw values inside the MODE objects. This would help you
ignore
> > > spatial
> > > > error and look to see if your forecast is reproducing
intensity
> values
> > > > similar to those found in the observations.
> > > >
> > > > John
> > > >
> > > > On Sun, Jan 11, 2015 at 10:14 AM, Geeta Geeta via RT <
> met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Sun Jan 11 10:14:37 2015: Request 70329 was acted upon.
> > > > > Transaction: Ticket created by geeta124 at hotmail.com
> > > > > Queue: met_help
> > > > > Subject: verification of a thunderstorm event.
> > > > > Owner: Nobody
> > > > > Requestors: geeta124 at hotmail.com
> > > > > Status: new
> > > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329
> > > >
> > > > >
> > > > >
> > > > > Hi John.
> > > > > I am trying to verify the results of my thunderstorm
simulations
> using
> > > the
> > > > > neighborhood methods. This event will have RF over an area
say 10
> -20
> > > > > locations with varying intensities.
> > > > >
> > > > > The question that I am trying to find out is "whether the
same is
> > > > > reproduced by the model or not???. To be more detailed, I
would
> like to
> > > > > know if these intensities are reproduced. They may be
displaced as
> > > expected
> > > > > from the model FCST field.
> > > > >
> > > > > So I read that Upscaling is the basic method. But using this
and
> > > computig
> > > > > POD etc may not be so relevant.
> > > > > My questions are------------
> > > > > 1. what other neighborhood method will be most relevant in
this
> case???
> > > > > 2. so If I am using Multievent Contingency Table, which is
Single
> > > > > Observation and Neighborhhod FCST, How will my configuration
file
> look
> > > > > like???
> > > > >
> > > > > Thanks
> > > > > Geeta
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>
>
------------------------------------------------
Subject: verification of a thunderstorm event.
From: Geeta Geeta
Time: Sat Jan 17 01:06:46 2015
thanks John.
It makes some sense. fine.
I need your suggestion reg this.
I have model simulations at 5km (NETCDF file). I donot have gridded
observations at the same resolution. But I have point observations.
For running grid stat, Can I use the format of FCST file (I mean copy
the FCST file in netcdf format as the Observation file) and edit this
OBS file (or Put my point observations at the nearest grid points) and
then Run Grid stat tool.
Is this approach OKay???. Because I wish to vary the neighbourhood
size in terms of 5km. Earlier I was using TRMM data which is at 25km
resolution so just setting neighbourhood size/points to 3 or 5 would
EXHAUST my domain.
Kindly suggest.
Regards
Geeta
> Subject: Re: FW: [rt.rap.ucar.edu #70329] verification of a
thunderstorm event.
> From: met_help at ucar.edu
> To: geeta124 at hotmail.com
> Date: Thu, 15 Jan 2015 10:08:02 -0700
>
> Geeta,
>
> The NBRCTC and MCTC output lines contain different information.
>
> As I mentioned, I would recommend using the NBRCNT output line which
> contains the fractions skill score (FSS). That's the most commonly
used
> neighborhood method. I wouldn't recommend using NBRCTC and NBRCTS
at this
> point.
>
> The MCTC is the multi-category contingency table. As we've been
> discussing, if you specify n categorical thresholds, MET will use
those to
> define an (n+1) x (n+1) multi-category contingency table. The MCTC
output
> line contains the counts for each of the cells in that table. And
for each
> MCTC line, you'll see a corresponding MCTS line containing the
statistics
> derived from that multi-category contingency table.
>
> I am not familiar with the "the SINGLE OBSERVATION-NEIGHBOURHOOD
type of
> decision mode". Based on your description, it sounds like you're
> interested in your model performance over a very small subset of
your
> domain. I'm assuming you already have your forecast and observation
data
> on the same grid. To do an evaluation over a small subset of your
domain,
> just define a masking region for that area. The simplest would be
to
> define a polyline region of at least 3 lat/lon value. You could
pass that
> polyline to grid_stat directly in the configuration file (vx_mask)
or run
> it through the gen_poly_mask tool first to define a bitmap - and
then pass
> it's output to grid_stat in the configuration file.
>
> Is that the information you're looking for?
>
> Thanks,
> John
>
> On Thu, Jan 15, 2015 at 7:16 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
> >
> > Hi John.
> > Kindly guide me further to my questions. I was waiting for your
reply.
> >
> > regards
> > geeta
> >
> > From: geeta124 at hotmail.com
> > To: met_help at ucar.edu
> > Subject: RE: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > event.
> > Date: Wed, 14 Jan 2015 10:48:05 +0530
> >
> >
> >
> >
> > thanks john. I would like some more details of the following ----
> > 1. what would be the difference b/w NBRCTC and MCTC outputs or
how the
> > NBRCTC and MCTC and MCTS outputs are calculated (methodology) ???
> > 2. As I was discussing, I have done the some simulations of
Thunderstorm
> > events. I want to verify the RF output. See I have model output
which has
> > 177x177 points in the X and Y directions. The model output is at 5
km
> > resolution. MY point of interest does NOT lay in the centre of the
model
> > domain. So If I wish to use the SINGLE OBSERVATION-NEIGHBOURHOOD
type of
> > decision model, (or multi-event-contingency count contingency
table), then
> > shall I extract (or make a subset of the model data) the model
data around
> > my GRID POINT of interest (say 10x10 grid points with MY GRID
POINT at the
> > centre).
> > and then compute the MCTC/ NBRCTC contingency Tables???
> > I had written a few points in my earlier mail. Kindly explain
those
> > points to me. Unless I am not clear of what I am doing, I cannot
interpret
> > my results well. another point that is not clear to me is the use
of
> > different metrics for different type of Neighbourhood methods.
> > Geeta
> >
> > > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > event.
> > > From: met_help at ucar.edu
> > > To: geeta124 at hotmail.com
> > > Date: Tue, 13 Jan 2015 15:59:09 -0700
> > >
> > > Geeta,
> > >
> > > Grid-Stat applies the following logic to compute neighborhood
statistics.
> > > It computes neighborhood info for each combination of variable
> > > (fcst.field), categorical threshold (cat_thresh), and
neighborhood width
> > > (width).
> > >
> > > Suppose you're evaluating accumulated precip using a threshold
of >25.4
> > > (i.e. > 1 inch) using neighborhood width of 3, 5, 7, and 9. For
width 3,
> > > it'll process each grid point by looking at a 3x3 box centered
on that
> > grid
> > > point. It'll count up the number of those 9 values that have
precip > 1
> > > inch. Let's suppose 5 of them do. So the "fractional coverage"
value at
> > > that point is 5/9 = 0.56.
> > >
> > > The "vld_thresh" determines how many of those 9 grid points need
to
> > contain
> > > valid data for a fractional coverage value to be computed.
Setting it
> > to 1
> > > requires that they all must contain valid data. If you're at
the edge of
> > > the grid or have sporadic missing data, you'll have to decide
how much
> > > missing data you want to allow. Setting vld_thresh = 0.5 would
only
> > > require that 1/2 of the grid point contain valid data.
> > >
> > > I "cov_thresh" value is used to compute NBRCTC and NBRCTS output
lines,
> > but
> > > I recommend that you not use them. Instead, just use NBRCNT,
which
> > > contains fractions skill score, which is the most commonly used
> > > neighborhood statistic.
> > >
> > > I'd suggest setting width = [ 3, 5, 7, 9, 11 ]; and then look to
see how
> > > FSS changes as you increase the neighborhood size.
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > (1) For each each neighborhood width, it computes a fraction
> > >
> > > On Tue, Jan 13, 2015 at 10:08 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329
>
> > > >
> > > > Hi John.
> > > > I have following entries in my config file.
> > > > // Neighborhood methods
> > > > //
> > > > nbrhd = {
> > > > vld_thresh = 1.0;
> > > > width = [ 1 ];
> > > > cov_thresh = [ >=0.5 ];
> > > > }
> > > >
> > > > what does the last vld_thresh and cov_threh mean ?????
> > > > Kindly explain
> > > > Geeta
> > > > From: geeta124 at hotmail.com
> > > > To: met_help at ucar.edu
> > > > Subject: RE: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > > > event.
> > > > Date: Tue, 13 Jan 2015 11:21:46 +0530
> > > >
> > > >
> > > >
> > > >
> > > > thanks John I will surely try that. but as of now, I have
following
> > > > questions which I would like to be guided with.
> > > > The matching strategy in the MCTC is SINGLE OBSERVATION-
NEIGHBOURHOOD
> > > > FCST. In my config file, I have selected 2 thresholds,
say>=0.1,
> > >=5.0. So
> > > > there will be 3x3 contingency TABLE and the OBS-FCST pairs
will be
> > put in
> > > > one of the categories. Now my questions are
> > > > 1. If I am interested in applying this method for my station
(in this
> > case
> > > > it will be gridded RF nearest to the station, is this
correct???) ,
> > then
> > > > how will choose/define MY grid point of interest? (See I have
a
> > observation
> > > > grid of say 53x53 points and same for FCST also).
> > > > 2. Once the grid point of MY interest in CHOSEN, the
neighbourhood
> > around
> > > > this has to be DEFINED. There is a provision of defining the
> > neighbourhood
> > > > in the CONFIG file (interp_width1 or 3 or 5 or 7 ....). BUT
This
> > > > neighbourhood has to be AROUND MY STATION/GRID point of
interest. How
> > do I
> > > > do that in my config file???.
> > > > 3. want to know if I understood this correctly-------- "
SINGLE
> > > > OBSERVATION-NEIGHBOURHOOD FCST" means that I will consider
only ONE
> > > > OBSERVATION (at MY GRID point). THEN HOW does MET generate the
> > MPR's????.
> > > > 4. Also amongst many SINGLE OBSERVATION-NEIGHBOURHOOD FCST
decision
> > > > models, WHY the scores are different??? say for eg. MCTC, One
has to
> > look
> > > > at ROC,V and for Pragmatic approach (BS, BSS)????? (Ebert
2008).
> > > > 5. in case of thunderstorm event simulations, I will have RF
at my
> > station
> > > > and around 50 km say. then Do I have to choose each station
> > independently
> > > > and then define a neighborhood around this????
> > > >
> > > >
> > > > Kindly clear these doubts for me so that I can get started.
> > > > Geeta
> > > >
> > > > > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > > > event.
> > > > > From: met_help at ucar.edu
> > > > > To: geeta124 at hotmail.com
> > > > > Date: Mon, 12 Jan 2015 14:48:40 -0700
> > > > >
> > > > > Geeta,
> > > > >
> > > > > The neighborhood methods in grid_stat will give you a way to
> > quantify how
> > > > > much spatial error your forecast has. You pick a single
threshold to
> > > > > define a binary (yes/no) field. Then you compute the
neighborhood
> > stats
> > > > > using multiple neighborhood sizes. And you can rerun with
many
> > different
> > > > > thresholds. That'll tell you about spatial error, but it
won't give
> > you
> > > > a
> > > > > good description of the range of intensity values.
> > > > >
> > > > > Instead, I'd suggest you do an object-based verification
using the
> > MODE
> > > > > tool. You'd configure and run MODE to define objects in
your
> > forecast
> > > > and
> > > > > observation field. MODE will run the configurable logic you
specify
> > in
> > > > the
> > > > > config file to decide which forecast objects should be
matched with
> > which
> > > > > observation objects. The ASCII MODE output includes
percentile
> > values of
> > > > > the raw values inside the MODE objects. This would help you
ignore
> > > > spatial
> > > > > error and look to see if your forecast is reproducing
intensity
> > values
> > > > > similar to those found in the observations.
> > > > >
> > > > > John
> > > > >
> > > > > On Sun, Jan 11, 2015 at 10:14 AM, Geeta Geeta via RT <
> > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Sun Jan 11 10:14:37 2015: Request 70329 was acted upon.
> > > > > > Transaction: Ticket created by geeta124 at hotmail.com
> > > > > > Queue: met_help
> > > > > > Subject: verification of a thunderstorm event.
> > > > > > Owner: Nobody
> > > > > > Requestors: geeta124 at hotmail.com
> > > > > > Status: new
> > > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi John.
> > > > > > I am trying to verify the results of my thunderstorm
simulations
> > using
> > > > the
> > > > > > neighborhood methods. This event will have RF over an area
say 10
> > -20
> > > > > > locations with varying intensities.
> > > > > >
> > > > > > The question that I am trying to find out is "whether the
same is
> > > > > > reproduced by the model or not???. To be more detailed, I
would
> > like to
> > > > > > know if these intensities are reproduced. They may be
displaced as
> > > > expected
> > > > > > from the model FCST field.
> > > > > >
> > > > > > So I read that Upscaling is the basic method. But using
this and
> > > > computig
> > > > > > POD etc may not be so relevant.
> > > > > > My questions are------------
> > > > > > 1. what other neighborhood method will be most relevant in
this
> > case???
> > > > > > 2. so If I am using Multievent Contingency Table, which is
Single
> > > > > > Observation and Neighborhhod FCST, How will my
configuration file
> > look
> > > > > > like???
> > > > > >
> > > > > > Thanks
> > > > > > Geeta
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> >
>
------------------------------------------------
Subject: verification of a thunderstorm event.
From: Geeta Geeta
Time: Sun Jan 18 09:54:36 2015
Hi John.
Can I generate the Observations at 5km resolution. I mean the TRMM
data at 25km resolution be transformed to 5km.
How this has to be done???
Can U suggest??
Geeta
From: geeta124 at hotmail.com
To: met_help at ucar.edu
Subject: RE: [rt.rap.ucar.edu #70329] verification of a thunderstorm
event.
Date: Sat, 17 Jan 2015 13:36:42 +0530
thanks John.
It makes some sense. fine.
I need your suggestion reg this.
I have model simulations at 5km (NETCDF file). I donot have gridded
observations at the same resolution. But I have point observations.
For running grid stat, Can I use the format of FCST file (I mean copy
the FCST file in netcdf format as the Observation file) and edit this
OBS file (or Put my point observations at the nearest grid points) and
then Run Grid stat tool.
Is this approach OKay???. Because I wish to vary the neighbourhood
size in terms of 5km. Earlier I was using TRMM data which is at 25km
resolution so just setting neighbourhood size/points to 3 or 5 would
EXHAUST my domain.
Kindly suggest.
Regards
Geeta
> Subject: Re: FW: [rt.rap.ucar.edu #70329] verification of a
thunderstorm event.
> From: met_help at ucar.edu
> To: geeta124 at hotmail.com
> Date: Thu, 15 Jan 2015 10:08:02 -0700
>
> Geeta,
>
> The NBRCTC and MCTC output lines contain different information.
>
> As I mentioned, I would recommend using the NBRCNT output line which
> contains the fractions skill score (FSS). That's the most commonly
used
> neighborhood method. I wouldn't recommend using NBRCTC and NBRCTS
at this
> point.
>
> The MCTC is the multi-category contingency table. As we've been
> discussing, if you specify n categorical thresholds, MET will use
those to
> define an (n+1) x (n+1) multi-category contingency table. The MCTC
output
> line contains the counts for each of the cells in that table. And
for each
> MCTC line, you'll see a corresponding MCTS line containing the
statistics
> derived from that multi-category contingency table.
>
> I am not familiar with the "the SINGLE OBSERVATION-NEIGHBOURHOOD
type of
> decision mode". Based on your description, it sounds like you're
> interested in your model performance over a very small subset of
your
> domain. I'm assuming you already have your forecast and observation
data
> on the same grid. To do an evaluation over a small subset of your
domain,
> just define a masking region for that area. The simplest would be
to
> define a polyline region of at least 3 lat/lon value. You could
pass that
> polyline to grid_stat directly in the configuration file (vx_mask)
or run
> it through the gen_poly_mask tool first to define a bitmap - and
then pass
> it's output to grid_stat in the configuration file.
>
> Is that the information you're looking for?
>
> Thanks,
> John
>
> On Thu, Jan 15, 2015 at 7:16 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
> >
> > Hi John.
> > Kindly guide me further to my questions. I was waiting for your
reply.
> >
> > regards
> > geeta
> >
> > From: geeta124 at hotmail.com
> > To: met_help at ucar.edu
> > Subject: RE: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > event.
> > Date: Wed, 14 Jan 2015 10:48:05 +0530
> >
> >
> >
> >
> > thanks john. I would like some more details of the following ----
> > 1. what would be the difference b/w NBRCTC and MCTC outputs or
how the
> > NBRCTC and MCTC and MCTS outputs are calculated (methodology) ???
> > 2. As I was discussing, I have done the some simulations of
Thunderstorm
> > events. I want to verify the RF output. See I have model output
which has
> > 177x177 points in the X and Y directions. The model output is at 5
km
> > resolution. MY point of interest does NOT lay in the centre of the
model
> > domain. So If I wish to use the SINGLE OBSERVATION-NEIGHBOURHOOD
type of
> > decision model, (or multi-event-contingency count contingency
table), then
> > shall I extract (or make a subset of the model data) the model
data around
> > my GRID POINT of interest (say 10x10 grid points with MY GRID
POINT at the
> > centre).
> > and then compute the MCTC/ NBRCTC contingency Tables???
> > I had written a few points in my earlier mail. Kindly explain
those
> > points to me. Unless I am not clear of what I am doing, I cannot
interpret
> > my results well. another point that is not clear to me is the use
of
> > different metrics for different type of Neighbourhood methods.
> > Geeta
> >
> > > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > event.
> > > From: met_help at ucar.edu
> > > To: geeta124 at hotmail.com
> > > Date: Tue, 13 Jan 2015 15:59:09 -0700
> > >
> > > Geeta,
> > >
> > > Grid-Stat applies the following logic to compute neighborhood
statistics.
> > > It computes neighborhood info for each combination of variable
> > > (fcst.field), categorical threshold (cat_thresh), and
neighborhood width
> > > (width).
> > >
> > > Suppose you're evaluating accumulated precip using a threshold
of >25.4
> > > (i.e. > 1 inch) using neighborhood width of 3, 5, 7, and 9. For
width 3,
> > > it'll process each grid point by looking at a 3x3 box centered
on that
> > grid
> > > point. It'll count up the number of those 9 values that have
precip > 1
> > > inch. Let's suppose 5 of them do. So the "fractional coverage"
value at
> > > that point is 5/9 = 0.56.
> > >
> > > The "vld_thresh" determines how many of those 9 grid points need
to
> > contain
> > > valid data for a fractional coverage value to be computed.
Setting it
> > to 1
> > > requires that they all must contain valid data. If you're at
the edge of
> > > the grid or have sporadic missing data, you'll have to decide
how much
> > > missing data you want to allow. Setting vld_thresh = 0.5 would
only
> > > require that 1/2 of the grid point contain valid data.
> > >
> > > I "cov_thresh" value is used to compute NBRCTC and NBRCTS output
lines,
> > but
> > > I recommend that you not use them. Instead, just use NBRCNT,
which
> > > contains fractions skill score, which is the most commonly used
> > > neighborhood statistic.
> > >
> > > I'd suggest setting width = [ 3, 5, 7, 9, 11 ]; and then look to
see how
> > > FSS changes as you increase the neighborhood size.
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > (1) For each each neighborhood width, it computes a fraction
> > >
> > > On Tue, Jan 13, 2015 at 10:08 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329
>
> > > >
> > > > Hi John.
> > > > I have following entries in my config file.
> > > > // Neighborhood methods
> > > > //
> > > > nbrhd = {
> > > > vld_thresh = 1.0;
> > > > width = [ 1 ];
> > > > cov_thresh = [ >=0.5 ];
> > > > }
> > > >
> > > > what does the last vld_thresh and cov_threh mean ?????
> > > > Kindly explain
> > > > Geeta
> > > > From: geeta124 at hotmail.com
> > > > To: met_help at ucar.edu
> > > > Subject: RE: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > > > event.
> > > > Date: Tue, 13 Jan 2015 11:21:46 +0530
> > > >
> > > >
> > > >
> > > >
> > > > thanks John I will surely try that. but as of now, I have
following
> > > > questions which I would like to be guided with.
> > > > The matching strategy in the MCTC is SINGLE OBSERVATION-
NEIGHBOURHOOD
> > > > FCST. In my config file, I have selected 2 thresholds,
say>=0.1,
> > >=5.0. So
> > > > there will be 3x3 contingency TABLE and the OBS-FCST pairs
will be
> > put in
> > > > one of the categories. Now my questions are
> > > > 1. If I am interested in applying this method for my station
(in this
> > case
> > > > it will be gridded RF nearest to the station, is this
correct???) ,
> > then
> > > > how will choose/define MY grid point of interest? (See I have
a
> > observation
> > > > grid of say 53x53 points and same for FCST also).
> > > > 2. Once the grid point of MY interest in CHOSEN, the
neighbourhood
> > around
> > > > this has to be DEFINED. There is a provision of defining the
> > neighbourhood
> > > > in the CONFIG file (interp_width1 or 3 or 5 or 7 ....). BUT
This
> > > > neighbourhood has to be AROUND MY STATION/GRID point of
interest. How
> > do I
> > > > do that in my config file???.
> > > > 3. want to know if I understood this correctly-------- "
SINGLE
> > > > OBSERVATION-NEIGHBOURHOOD FCST" means that I will consider
only ONE
> > > > OBSERVATION (at MY GRID point). THEN HOW does MET generate the
> > MPR's????.
> > > > 4. Also amongst many SINGLE OBSERVATION-NEIGHBOURHOOD FCST
decision
> > > > models, WHY the scores are different??? say for eg. MCTC, One
has to
> > look
> > > > at ROC,V and for Pragmatic approach (BS, BSS)????? (Ebert
2008).
> > > > 5. in case of thunderstorm event simulations, I will have RF
at my
> > station
> > > > and around 50 km say. then Do I have to choose each station
> > independently
> > > > and then define a neighborhood around this????
> > > >
> > > >
> > > > Kindly clear these doubts for me so that I can get started.
> > > > Geeta
> > > >
> > > > > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > > > event.
> > > > > From: met_help at ucar.edu
> > > > > To: geeta124 at hotmail.com
> > > > > Date: Mon, 12 Jan 2015 14:48:40 -0700
> > > > >
> > > > > Geeta,
> > > > >
> > > > > The neighborhood methods in grid_stat will give you a way to
> > quantify how
> > > > > much spatial error your forecast has. You pick a single
threshold to
> > > > > define a binary (yes/no) field. Then you compute the
neighborhood
> > stats
> > > > > using multiple neighborhood sizes. And you can rerun with
many
> > different
> > > > > thresholds. That'll tell you about spatial error, but it
won't give
> > you
> > > > a
> > > > > good description of the range of intensity values.
> > > > >
> > > > > Instead, I'd suggest you do an object-based verification
using the
> > MODE
> > > > > tool. You'd configure and run MODE to define objects in
your
> > forecast
> > > > and
> > > > > observation field. MODE will run the configurable logic you
specify
> > in
> > > > the
> > > > > config file to decide which forecast objects should be
matched with
> > which
> > > > > observation objects. The ASCII MODE output includes
percentile
> > values of
> > > > > the raw values inside the MODE objects. This would help you
ignore
> > > > spatial
> > > > > error and look to see if your forecast is reproducing
intensity
> > values
> > > > > similar to those found in the observations.
> > > > >
> > > > > John
> > > > >
> > > > > On Sun, Jan 11, 2015 at 10:14 AM, Geeta Geeta via RT <
> > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Sun Jan 11 10:14:37 2015: Request 70329 was acted upon.
> > > > > > Transaction: Ticket created by geeta124 at hotmail.com
> > > > > > Queue: met_help
> > > > > > Subject: verification of a thunderstorm event.
> > > > > > Owner: Nobody
> > > > > > Requestors: geeta124 at hotmail.com
> > > > > > Status: new
> > > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi John.
> > > > > > I am trying to verify the results of my thunderstorm
simulations
> > using
> > > > the
> > > > > > neighborhood methods. This event will have RF over an area
say 10
> > -20
> > > > > > locations with varying intensities.
> > > > > >
> > > > > > The question that I am trying to find out is "whether the
same is
> > > > > > reproduced by the model or not???. To be more detailed, I
would
> > like to
> > > > > > know if these intensities are reproduced. They may be
displaced as
> > > > expected
> > > > > > from the model FCST field.
> > > > > >
> > > > > > So I read that Upscaling is the basic method. But using
this and
> > > > computig
> > > > > > POD etc may not be so relevant.
> > > > > > My questions are------------
> > > > > > 1. what other neighborhood method will be most relevant in
this
> > case???
> > > > > > 2. so If I am using Multievent Contingency Table, which is
Single
> > > > > > Observation and Neighborhhod FCST, How will my
configuration file
> > look
> > > > > > like???
> > > > > >
> > > > > > Thanks
> > > > > > Geeta
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> >
>
------------------------------------------------
Subject: verification of a thunderstorm event.
From: John Halley Gotway
Time: Tue Jan 20 14:31:58 2015
Geeta,
If you have point observations, I would suggest running the Point-Stat
tool
to compare the gridded forecast to your point observations.
Of course, you're welcome to do whatever you'd like with your data.
But
the Point-Stat tool is meant to be used when you have point
observations.
Thanks,
John
On Sun, Jan 18, 2015 at 9:54 AM, Geeta Geeta via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
>
> Hi John.
> Can I generate the Observations at 5km resolution. I mean the TRMM
data at
> 25km resolution be transformed to 5km.
> How this has to be done???
> Can U suggest??
> Geeta
>
> From: geeta124 at hotmail.com
> To: met_help at ucar.edu
> Subject: RE: [rt.rap.ucar.edu #70329] verification of a thunderstorm
> event.
> Date: Sat, 17 Jan 2015 13:36:42 +0530
>
>
>
>
> thanks John.
> It makes some sense. fine.
> I need your suggestion reg this.
> I have model simulations at 5km (NETCDF file). I donot have gridded
> observations at the same resolution. But I have point observations.
>
> For running grid stat, Can I use the format of FCST file (I mean
copy the
> FCST file in netcdf format as the Observation file) and edit this
OBS file
> (or Put my point observations at the nearest grid points) and then
Run Grid
> stat tool.
>
> Is this approach OKay???. Because I wish to vary the neighbourhood
size in
> terms of 5km. Earlier I was using TRMM data which is at 25km
resolution so
> just setting neighbourhood size/points to 3 or 5 would EXHAUST my
domain.
> Kindly suggest.
> Regards
> Geeta
>
>
> > Subject: Re: FW: [rt.rap.ucar.edu #70329] verification of a
> thunderstorm event.
> > From: met_help at ucar.edu
> > To: geeta124 at hotmail.com
> > Date: Thu, 15 Jan 2015 10:08:02 -0700
> >
> > Geeta,
> >
> > The NBRCTC and MCTC output lines contain different information.
> >
> > As I mentioned, I would recommend using the NBRCNT output line
which
> > contains the fractions skill score (FSS). That's the most
commonly used
> > neighborhood method. I wouldn't recommend using NBRCTC and NBRCTS
at
> this
> > point.
> >
> > The MCTC is the multi-category contingency table. As we've been
> > discussing, if you specify n categorical thresholds, MET will use
those
> to
> > define an (n+1) x (n+1) multi-category contingency table. The
MCTC
> output
> > line contains the counts for each of the cells in that table. And
for
> each
> > MCTC line, you'll see a corresponding MCTS line containing the
statistics
> > derived from that multi-category contingency table.
> >
> > I am not familiar with the "the SINGLE OBSERVATION-NEIGHBOURHOOD
type of
> > decision mode". Based on your description, it sounds like you're
> > interested in your model performance over a very small subset of
your
> > domain. I'm assuming you already have your forecast and
observation data
> > on the same grid. To do an evaluation over a small subset of your
> domain,
> > just define a masking region for that area. The simplest would be
to
> > define a polyline region of at least 3 lat/lon value. You could
pass
> that
> > polyline to grid_stat directly in the configuration file (vx_mask)
or run
> > it through the gen_poly_mask tool first to define a bitmap - and
then
> pass
> > it's output to grid_stat in the configuration file.
> >
> > Is that the information you're looking for?
> >
> > Thanks,
> > John
> >
> > On Thu, Jan 15, 2015 at 7:16 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
> > >
> > > Hi John.
> > > Kindly guide me further to my questions. I was waiting for your
reply.
> > >
> > > regards
> > > geeta
> > >
> > > From: geeta124 at hotmail.com
> > > To: met_help at ucar.edu
> > > Subject: RE: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > > event.
> > > Date: Wed, 14 Jan 2015 10:48:05 +0530
> > >
> > >
> > >
> > >
> > > thanks john. I would like some more details of the following
----
> > > 1. what would be the difference b/w NBRCTC and MCTC outputs or
how the
> > > NBRCTC and MCTC and MCTS outputs are calculated (methodology)
???
> > > 2. As I was discussing, I have done the some simulations of
> Thunderstorm
> > > events. I want to verify the RF output. See I have model output
which
> has
> > > 177x177 points in the X and Y directions. The model output is at
5 km
> > > resolution. MY point of interest does NOT lay in the centre of
the
> model
> > > domain. So If I wish to use the SINGLE OBSERVATION-
NEIGHBOURHOOD type
> of
> > > decision model, (or multi-event-contingency count contingency
table),
> then
> > > shall I extract (or make a subset of the model data) the model
data
> around
> > > my GRID POINT of interest (say 10x10 grid points with MY GRID
POINT at
> the
> > > centre).
> > > and then compute the MCTC/ NBRCTC contingency Tables???
> > > I had written a few points in my earlier mail. Kindly explain
those
> > > points to me. Unless I am not clear of what I am doing, I cannot
> interpret
> > > my results well. another point that is not clear to me is the
use of
> > > different metrics for different type of Neighbourhood methods.
> > > Geeta
> > >
> > > > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
thunderstorm
> > > event.
> > > > From: met_help at ucar.edu
> > > > To: geeta124 at hotmail.com
> > > > Date: Tue, 13 Jan 2015 15:59:09 -0700
> > > >
> > > > Geeta,
> > > >
> > > > Grid-Stat applies the following logic to compute neighborhood
> statistics.
> > > > It computes neighborhood info for each combination of variable
> > > > (fcst.field), categorical threshold (cat_thresh), and
neighborhood
> width
> > > > (width).
> > > >
> > > > Suppose you're evaluating accumulated precip using a threshold
of
> >25.4
> > > > (i.e. > 1 inch) using neighborhood width of 3, 5, 7, and 9.
For
> width 3,
> > > > it'll process each grid point by looking at a 3x3 box centered
on
> that
> > > grid
> > > > point. It'll count up the number of those 9 values that have
precip
> > 1
> > > > inch. Let's suppose 5 of them do. So the "fractional
coverage"
> value at
> > > > that point is 5/9 = 0.56.
> > > >
> > > > The "vld_thresh" determines how many of those 9 grid points
need to
> > > contain
> > > > valid data for a fractional coverage value to be computed.
Setting
> it
> > > to 1
> > > > requires that they all must contain valid data. If you're at
the
> edge of
> > > > the grid or have sporadic missing data, you'll have to decide
how
> much
> > > > missing data you want to allow. Setting vld_thresh = 0.5
would only
> > > > require that 1/2 of the grid point contain valid data.
> > > >
> > > > I "cov_thresh" value is used to compute NBRCTC and NBRCTS
output
> lines,
> > > but
> > > > I recommend that you not use them. Instead, just use NBRCNT,
which
> > > > contains fractions skill score, which is the most commonly
used
> > > > neighborhood statistic.
> > > >
> > > > I'd suggest setting width = [ 3, 5, 7, 9, 11 ]; and then look
to see
> how
> > > > FSS changes as you increase the neighborhood size.
> > > >
> > > > Hope that helps.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > (1) For each each neighborhood width, it computes a fraction
> > > >
> > > > On Tue, Jan 13, 2015 at 10:08 AM, Geeta Geeta via RT <
> met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329 >
> > > > >
> > > > > Hi John.
> > > > > I have following entries in my config file.
> > > > > // Neighborhood methods
> > > > > //
> > > > > nbrhd = {
> > > > > vld_thresh = 1.0;
> > > > > width = [ 1 ];
> > > > > cov_thresh = [ >=0.5 ];
> > > > > }
> > > > >
> > > > > what does the last vld_thresh and cov_threh mean ?????
> > > > > Kindly explain
> > > > > Geeta
> > > > > From: geeta124 at hotmail.com
> > > > > To: met_help at ucar.edu
> > > > > Subject: RE: [rt.rap.ucar.edu #70329] verification of a
> thunderstorm
> > > > > event.
> > > > > Date: Tue, 13 Jan 2015 11:21:46 +0530
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > thanks John I will surely try that. but as of now, I have
following
> > > > > questions which I would like to be guided with.
> > > > > The matching strategy in the MCTC is SINGLE
> OBSERVATION-NEIGHBOURHOOD
> > > > > FCST. In my config file, I have selected 2 thresholds,
say>=0.1,
> > > >=5.0. So
> > > > > there will be 3x3 contingency TABLE and the OBS-FCST pairs
will be
> > > put in
> > > > > one of the categories. Now my questions are
> > > > > 1. If I am interested in applying this method for my station
(in
> this
> > > case
> > > > > it will be gridded RF nearest to the station, is this
correct???) ,
> > > then
> > > > > how will choose/define MY grid point of interest? (See I
have a
> > > observation
> > > > > grid of say 53x53 points and same for FCST also).
> > > > > 2. Once the grid point of MY interest in CHOSEN, the
neighbourhood
> > > around
> > > > > this has to be DEFINED. There is a provision of defining
the
> > > neighbourhood
> > > > > in the CONFIG file (interp_width1 or 3 or 5 or 7 ....). BUT
This
> > > > > neighbourhood has to be AROUND MY STATION/GRID point of
interest.
> How
> > > do I
> > > > > do that in my config file???.
> > > > > 3. want to know if I understood this correctly-------- "
SINGLE
> > > > > OBSERVATION-NEIGHBOURHOOD FCST" means that I will consider
only ONE
> > > > > OBSERVATION (at MY GRID point). THEN HOW does MET generate
the
> > > MPR's????.
> > > > > 4. Also amongst many SINGLE OBSERVATION-NEIGHBOURHOOD FCST
decision
> > > > > models, WHY the scores are different??? say for eg. MCTC,
One has
> to
> > > look
> > > > > at ROC,V and for Pragmatic approach (BS, BSS)????? (Ebert
2008).
> > > > > 5. in case of thunderstorm event simulations, I will have RF
at my
> > > station
> > > > > and around 50 km say. then Do I have to choose each station
> > > independently
> > > > > and then define a neighborhood around this????
> > > > >
> > > > >
> > > > > Kindly clear these doubts for me so that I can get started.
> > > > > Geeta
> > > > >
> > > > > > Subject: Re: [rt.rap.ucar.edu #70329] verification of a
> thunderstorm
> > > > > event.
> > > > > > From: met_help at ucar.edu
> > > > > > To: geeta124 at hotmail.com
> > > > > > Date: Mon, 12 Jan 2015 14:48:40 -0700
> > > > > >
> > > > > > Geeta,
> > > > > >
> > > > > > The neighborhood methods in grid_stat will give you a way
to
> > > quantify how
> > > > > > much spatial error your forecast has. You pick a single
> threshold to
> > > > > > define a binary (yes/no) field. Then you compute the
> neighborhood
> > > stats
> > > > > > using multiple neighborhood sizes. And you can rerun with
many
> > > different
> > > > > > thresholds. That'll tell you about spatial error, but it
won't
> give
> > > you
> > > > > a
> > > > > > good description of the range of intensity values.
> > > > > >
> > > > > > Instead, I'd suggest you do an object-based verification
using
> the
> > > MODE
> > > > > > tool. You'd configure and run MODE to define objects in
your
> > > forecast
> > > > > and
> > > > > > observation field. MODE will run the configurable logic
you
> specify
> > > in
> > > > > the
> > > > > > config file to decide which forecast objects should be
matched
> with
> > > which
> > > > > > observation objects. The ASCII MODE output includes
percentile
> > > values of
> > > > > > the raw values inside the MODE objects. This would help
you
> ignore
> > > > > spatial
> > > > > > error and look to see if your forecast is reproducing
intensity
> > > values
> > > > > > similar to those found in the observations.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Sun, Jan 11, 2015 at 10:14 AM, Geeta Geeta via RT <
> > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Sun Jan 11 10:14:37 2015: Request 70329 was acted upon.
> > > > > > > Transaction: Ticket created by geeta124 at hotmail.com
> > > > > > > Queue: met_help
> > > > > > > Subject: verification of a thunderstorm event.
> > > > > > > Owner: Nobody
> > > > > > > Requestors: geeta124 at hotmail.com
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70329
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi John.
> > > > > > > I am trying to verify the results of my thunderstorm
> simulations
> > > using
> > > > > the
> > > > > > > neighborhood methods. This event will have RF over an
area say
> 10
> > > -20
> > > > > > > locations with varying intensities.
> > > > > > >
> > > > > > > The question that I am trying to find out is "whether
the same
> is
> > > > > > > reproduced by the model or not???. To be more detailed,
I would
> > > like to
> > > > > > > know if these intensities are reproduced. They may be
> displaced as
> > > > > expected
> > > > > > > from the model FCST field.
> > > > > > >
> > > > > > > So I read that Upscaling is the basic method. But using
this
> and
> > > > > computig
> > > > > > > POD etc may not be so relevant.
> > > > > > > My questions are------------
> > > > > > > 1. what other neighborhood method will be most relevant
in this
> > > case???
> > > > > > > 2. so If I am using Multievent Contingency Table, which
is
> Single
> > > > > > > Observation and Neighborhhod FCST, How will my
configuration
> file
> > > look
> > > > > > > like???
> > > > > > >
> > > > > > > Thanks
> > > > > > > Geeta
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list