[Met_help] [rt.rap.ucar.edu #65427] History for Unable to visualize Fuzzy verf.
John Halley Gotway via RT
met_help at ucar.edu
Tue Feb 18 12:04:27 MST 2014
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi John.
Some basic things about the grid stat tool are not clear to me. The met=doc mentions that the FUZZy verification method is inbuilt in the MEt package.
IThe FUZZY verification mentions about the Radius that can be selected within which, if the OBS rf is found , then it is considered as a HIT.
I could not find that RADIUS (I mean how to select this radius and intensity )in the configuration file????
Can u help me to understand???????????????
geeta
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65427] Unable to visualize Fuzzy verf.
From: John Halley Gotway
Time: Thu Feb 13 10:33:46 2014
Geeta,
You are correct, the input forecast and observation files must be on
the same grid. In Grid-Stat, there are two ways you can perform
"fuzzy" verification.
(1) The first way is by applying an interpolation method to the data.
Since the data are already on the same grid, this is really a
"smoothing" operation instead. This is called "upscaling".
Smoother forecasts and observations tend to produce better traditional
verification scores. So you could see how your scores (like RMSE or
GSS) improve as you smooth the data more and more. In the
config file, you could try:
interp = {
field = BOTH;
vld_thresh = 1.0;
type = [
{ method = UW_MEAN; width = 1; },
{ method = UW_MEAN; width = 3; },
{ method = UW_MEAN; width = 6; },
{ method = UW_MEAN; width = 9; }
];
};
This tells Grid-Stat to compute its statistics 4 times, applying more
smoothing each time. Typically, the more the data has been smoothed,
the better the statistics will be.
(2) The second way is by applying neighborhood verification methods.
The most common are the Fractions Brier Score (FBS) and Fractions
Skill Score (FSS), both contained in the NBRCNT output line
type. Be sure to turn the NBRCNT output line on in the Grid-Stat
config file. For neighborhood verification, you pick multiple
neighborhood sizes and look to see how the FSS changes as you increase
the neighborhood size. As the neighborhood size increases, FSS
increases. And you look to see how large of a neighborhood size you
need to get a "useful" (FSS > 0.5) forecast.
Here's how this method works. You pick one or more thresholds
(cat_thresh) for your field. Grid-Stat applies the threshold to
produce a 0/1 binary field of your data. For each neighborhood size,
n,
it places an n x n box around each grid point and counts up the number
of events within that box. For a 3 x 3 box, if 4 of the 9 points
contained an event, the value for that point is 4/9. This is
done for every grid point in for forecast field and the observation
field. We call the result of this process the forecast and
observation "fractional coverage" fields. The FSS and FBS scores are
computed by comparing the forecast and observation fractional coverage
fields to each other.
If you're verifying a single field using 3 different thresholds and 6
different neighborhood sizes, you'd get 18 NBRCNT lines in the output
file.
Here's an example of how you might set this up in the Grid-Stat config
file:
nbrhd = {
vld_thresh = 1.0;
width = [ 3, 5, 9, 11, 13, 15 ];
cov_thresh = [ >=0.5 ];
}
For a given threshold, you should look to see how FSS changes as you
increase the neighborhood size.
Hopefully that helps get you going.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 02/12/2014 10:49 PM, Geeta Geeta via RT wrote:
>
> Wed Feb 12 22:49:14 2014: Request 65427 was acted upon.
> Transaction: Ticket created by geeta124 at hotmail.com
> Queue: met_help
> Subject: Unable to visualize Fuzzy verf.
> Owner: Nobody
> Requestors: geeta124 at hotmail.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65427 >
>
>
> Hi John/ met_help.
>
> I was reading MET doc that mentions about the FUZZY verification
methods. I am trying to visualise what grid stat does or how it
functions.
> After the copygb is run, the FCST and OBS are on the same grid. ie
1------------------2-----------------------3|
| | |
| || |
|| |
|4-------------------5-----------------------6
> ie at the Grid Points (GP) 1 to 6, U have Observations and the model
FCST.Now the MET doc (Pg 5-3) says that a SQUARE search window is
defined around each grid point, within which the obs and the FCST
events are counted. 1. I want to know HOW is this SQUARE WINDOW is
defined (I mean in the configuration file) of Grid stat. 2. How Can I
change the size of the SQUARE window. 3. If My model resolution in
10km and I am interested in the synoptic scale phenomenon, then what
should be the window size??????????????? your help is urgently
required.
>
> geeta
>
------------------------------------------------
Subject: Unable to visualize Fuzzy verf.
From: Geeta Geeta
Time: Fri Feb 14 01:38:20 2014
Thanks a lot John for your inputs and clarifications.
Still following doubts are there.
1. when I run copygb, what it does is to make the observation and
Model FC uniform ( I mean same GRID and RESOLUTION). These two
parameters are only important.
R u calling that as Upscaling???? So this process is not a part of
GRID-stat. So essentially copygb is doing the upscaling part.
2. There are interpolation methods in the grid-stat config file.
(analogous to that in point-stat. in point-stat, there are 3-4 like
nearest neigh, mean, distance weighted etc).
why should one have the interpolation ONCE again i.e (after copygb)
the grid fields are similar. ie. One GP has 2 values one OBS and one
FCST??? It is correct???
geeta
> Subject: Re: [rt.rap.ucar.edu #65427] Unable to visualize Fuzzy
verf.
> From: met_help at ucar.edu
> To: geeta124 at hotmail.com
> Date: Thu, 13 Feb 2014 10:33:47 -0700
>
> Geeta,
>
> You are correct, the input forecast and observation files must be on
the same grid. In Grid-Stat, there are two ways you can perform
"fuzzy" verification.
>
> (1) The first way is by applying an interpolation method to the
data. Since the data are already on the same grid, this is really a
"smoothing" operation instead. This is called "upscaling".
> Smoother forecasts and observations tend to produce better
traditional verification scores. So you could see how your scores
(like RMSE or GSS) improve as you smooth the data more and more. In
the
> config file, you could try:
>
> interp = {
> field = BOTH;
> vld_thresh = 1.0;
>
> type = [
> { method = UW_MEAN; width = 1; },
> { method = UW_MEAN; width = 3; },
> { method = UW_MEAN; width = 6; },
> { method = UW_MEAN; width = 9; }
> ];
> };
>
> This tells Grid-Stat to compute its statistics 4 times, applying
more smoothing each time. Typically, the more the data has been
smoothed, the better the statistics will be.
>
> (2) The second way is by applying neighborhood verification methods.
The most common are the Fractions Brier Score (FBS) and Fractions
Skill Score (FSS), both contained in the NBRCNT output line
> type. Be sure to turn the NBRCNT output line on in the Grid-Stat
config file. For neighborhood verification, you pick multiple
neighborhood sizes and look to see how the FSS changes as you increase
> the neighborhood size. As the neighborhood size increases, FSS
increases. And you look to see how large of a neighborhood size you
need to get a "useful" (FSS > 0.5) forecast.
>
> Here's how this method works. You pick one or more thresholds
(cat_thresh) for your field. Grid-Stat applies the threshold to
produce a 0/1 binary field of your data. For each neighborhood size,
n,
> it places an n x n box around each grid point and counts up the
number of events within that box. For a 3 x 3 box, if 4 of the 9
points contained an event, the value for that point is 4/9. This is
> done for every grid point in for forecast field and the observation
field. We call the result of this process the forecast and
observation "fractional coverage" fields. The FSS and FBS scores are
> computed by comparing the forecast and observation fractional
coverage fields to each other.
>
> If you're verifying a single field using 3 different thresholds and
6 different neighborhood sizes, you'd get 18 NBRCNT lines in the
output file.
>
> Here's an example of how you might set this up in the Grid-Stat
config file:
>
> nbrhd = {
> vld_thresh = 1.0;
> width = [ 3, 5, 9, 11, 13, 15 ];
> cov_thresh = [ >=0.5 ];
> }
>
> For a given threshold, you should look to see how FSS changes as you
increase the neighborhood size.
>
> Hopefully that helps get you going.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 02/12/2014 10:49 PM, Geeta Geeta via RT wrote:
> >
> > Wed Feb 12 22:49:14 2014: Request 65427 was acted upon.
> > Transaction: Ticket created by geeta124 at hotmail.com
> > Queue: met_help
> > Subject: Unable to visualize Fuzzy verf.
> > Owner: Nobody
> > Requestors: geeta124 at hotmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65427 >
> >
> >
> > Hi John/ met_help.
> >
> > I was reading MET doc that mentions about the FUZZY verification
methods. I am trying to visualise what grid stat does or how it
functions.
> > After the copygb is run, the FCST and OBS are on the same grid. ie
1------------------2-----------------------3|
| | |
| || |
|| |
|4-------------------5-----------------------6
> > ie at the Grid Points (GP) 1 to 6, U have Observations and the
model FCST.Now the MET doc (Pg 5-3) says that a SQUARE search window
is defined around each grid point, within which the obs and the FCST
events are counted. 1. I want to know HOW is this SQUARE WINDOW is
defined (I mean in the configuration file) of Grid stat. 2. How Can I
change the size of the SQUARE window. 3. If My model resolution in
10km and I am interested in the synoptic scale phenomenon, then what
should be the window size??????????????? your help is urgently
required.
> >
> > geeta
> >
>
------------------------------------------------
Subject: Unable to visualize Fuzzy verf.
From: Geeta Geeta
Time: Fri Feb 14 02:17:27 2014
Hi John,
I have run grid-stat. Following is the error.
bash-3.2$ ../bin/grid_stat ./fcst_nc/20110601*day1*
../trmm_nc_data/02june2011.nc GridStatConfig_APCP_24
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073321512274
Forecast File: ./fcst_nc/2011060100_WRFPRS_day1_003Z.nc
Observation File: ../trmm_nc_data/02june2011.nc
Configuration File: GridStatConfig_APCP_24
***WARNING***: process_scores() -> 61(*,*) not found in file:
./fcst_nc/2011060100_WRFPRS_day1_003Z.nc
--------------------------------------------------------------------------------
Pls suggest.
geeta
From: geeta124 at hotmail.com
To: met_help at ucar.edu
Subject: RE: [rt.rap.ucar.edu #65427] Unable to visualize Fuzzy verf.
Date: Fri, 14 Feb 2014 14:08:12 +0530
Thanks a lot John for your inputs and clarifications.
Still following doubts are there.
1. when I run copygb, what it does is to make the observation and
Model FC uniform ( I mean same GRID and RESOLUTION). These two
parameters are only important.
R u calling that as Upscaling???? So this process is not a part of
GRID-stat. So essentially copygb is doing the upscaling part.
2. There are interpolation methods in the grid-stat config file.
(analogous to that in point-stat. in point-stat, there are 3-4 like
nearest neigh, mean, distance weighted etc).
why should one have the interpolation ONCE again i.e (after copygb)
the grid fields are similar. ie. One GP has 2 values one OBS and one
FCST??? It is correct???
geeta
> Subject: Re: [rt.rap.ucar.edu #65427] Unable to visualize Fuzzy
verf.
> From: met_help at ucar.edu
> To: geeta124 at hotmail.com
> Date: Thu, 13 Feb 2014 10:33:47 -0700
>
> Geeta,
>
> You are correct, the input forecast and observation files must be on
the same grid. In Grid-Stat, there are two ways you can perform
"fuzzy" verification.
>
> (1) The first way is by applying an interpolation method to the
data. Since the data are already on the same grid, this is really a
"smoothing" operation instead. This is called "upscaling".
> Smoother forecasts and observations tend to produce better
traditional verification scores. So you could see how your scores
(like RMSE or GSS) improve as you smooth the data more and more. In
the
> config file, you could try:
>
> interp = {
> field = BOTH;
> vld_thresh = 1.0;
>
> type = [
> { method = UW_MEAN; width = 1; },
> { method = UW_MEAN; width = 3; },
> { method = UW_MEAN; width = 6; },
> { method = UW_MEAN; width = 9; }
> ];
> };
>
> This tells Grid-Stat to compute its statistics 4 times, applying
more smoothing each time. Typically, the more the data has been
smoothed, the better the statistics will be.
>
> (2) The second way is by applying neighborhood verification methods.
The most common are the Fractions Brier Score (FBS) and Fractions
Skill Score (FSS), both contained in the NBRCNT output line
> type. Be sure to turn the NBRCNT output line on in the Grid-Stat
config file. For neighborhood verification, you pick multiple
neighborhood sizes and look to see how the FSS changes as you increase
> the neighborhood size. As the neighborhood size increases, FSS
increases. And you look to see how large of a neighborhood size you
need to get a "useful" (FSS > 0.5) forecast.
>
> Here's how this method works. You pick one or more thresholds
(cat_thresh) for your field. Grid-Stat applies the threshold to
produce a 0/1 binary field of your data. For each neighborhood size,
n,
> it places an n x n box around each grid point and counts up the
number of events within that box. For a 3 x 3 box, if 4 of the 9
points contained an event, the value for that point is 4/9. This is
> done for every grid point in for forecast field and the observation
field. We call the result of this process the forecast and
observation "fractional coverage" fields. The FSS and FBS scores are
> computed by comparing the forecast and observation fractional
coverage fields to each other.
>
> If you're verifying a single field using 3 different thresholds and
6 different neighborhood sizes, you'd get 18 NBRCNT lines in the
output file.
>
> Here's an example of how you might set this up in the Grid-Stat
config file:
>
> nbrhd = {
> vld_thresh = 1.0;
> width = [ 3, 5, 9, 11, 13, 15 ];
> cov_thresh = [ >=0.5 ];
> }
>
> For a given threshold, you should look to see how FSS changes as you
increase the neighborhood size.
>
> Hopefully that helps get you going.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 02/12/2014 10:49 PM, Geeta Geeta via RT wrote:
> >
> > Wed Feb 12 22:49:14 2014: Request 65427 was acted upon.
> > Transaction: Ticket created by geeta124 at hotmail.com
> > Queue: met_help
> > Subject: Unable to visualize Fuzzy verf.
> > Owner: Nobody
> > Requestors: geeta124 at hotmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65427 >
> >
> >
> > Hi John/ met_help.
> >
> > I was reading MET doc that mentions about the FUZZY verification
methods. I am trying to visualise what grid stat does or how it
functions.
> > After the copygb is run, the FCST and OBS are on the same grid. ie
1------------------2-----------------------3|
| | |
| || |
|| |
|4-------------------5-----------------------6
> > ie at the Grid Points (GP) 1 to 6, U have Observations and the
model FCST.Now the MET doc (Pg 5-3) says that a SQUARE search window
is defined around each grid point, within which the obs and the FCST
events are counted. 1. I want to know HOW is this SQUARE WINDOW is
defined (I mean in the configuration file) of Grid stat. 2. How Can I
change the size of the SQUARE window. 3. If My model resolution in
10km and I am interested in the synoptic scale phenomenon, then what
should be the window size??????????????? your help is urgently
required.
> >
> > geeta
> >
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #65427] Unable to visualize Fuzzy verf.
From: John Halley Gotway
Time: Fri Feb 14 09:48:08 2014
Geeta,
You run copygb to put the forecast and observation fields on exactly
the same grid, meaning the exact same resolution and number of grid
points.
I was trying to make the point that the "interpolation methods" in the
grid_stat config file could be used as a form of "upscaling". You are
right, there is no *need* to interpolate the data since
you've already used copygb to put them on the same grid. In
grid_stat, the interpolation options provide a way of smoothing, or
upscaling, the data. For example, suppose you choose and
interpolation
option of UW_MEAN (for un-weighted mean) and width of 5. For each
grid point, grid_stat will replace the value at the grid point with
the average of the 25 points in a 5x5 box around that point.
Doing that for every point in the grid smooths the data and provides a
way of upscaling.
The default interpolation width is 1, meaning that no smoothing is
performed. However, you could use multiple smoothing widths and see
how your performance changes the more you smooth the data.
Does that make sense?
Regarding the runtime error you're getting, I see that you're using
input NetCDF files for the forecast and observation fields. In the
config file, you need to specify the name and dimensions of the
NetCDF variable to be used. Assuming the NetCDF variable is named
"APCP_24", it would look something like this:
fcst = {
wind_thresh = [ NA ];
field = [
{
name = "APCP_24";
level = [ "(*,*)" ];
cat_thresh = [ >0.0, >=5.0 ];
}
];
};
If you continue to experience problems, please send me sample forecast
and observation files along with the GridStatConfig file you're using.
You can post it to our anonymous ftp site following these
instructions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Thanks,
John
On 02/14/2014 02:17 AM, Geeta Geeta via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65427 >
>
> Hi John,
> I have run grid-stat. Following is the error.
>
> bash-3.2$ ../bin/grid_stat ./fcst_nc/20110601*day1*
../trmm_nc_data/02june2011.nc GridStatConfig_APCP_24
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073321512274
> Forecast File: ./fcst_nc/2011060100_WRFPRS_day1_003Z.nc
> Observation File: ../trmm_nc_data/02june2011.nc
> Configuration File: GridStatConfig_APCP_24
> ***WARNING***: process_scores() -> 61(*,*) not found in file:
./fcst_nc/2011060100_WRFPRS_day1_003Z.nc
>
>
--------------------------------------------------------------------------------
>
>
> Pls suggest.
>
> geeta
>
> From: geeta124 at hotmail.com
> To: met_help at ucar.edu
> Subject: RE: [rt.rap.ucar.edu #65427] Unable to visualize Fuzzy
verf.
> Date: Fri, 14 Feb 2014 14:08:12 +0530
>
>
>
>
> Thanks a lot John for your inputs and clarifications.
>
> Still following doubts are there.
>
> 1. when I run copygb, what it does is to make the observation and
Model FC uniform ( I mean same GRID and RESOLUTION). These two
parameters are only important.
> R u calling that as Upscaling???? So this process is not a part of
GRID-stat. So essentially copygb is doing the upscaling part.
>
> 2. There are interpolation methods in the grid-stat config file.
(analogous to that in point-stat. in point-stat, there are 3-4 like
nearest neigh, mean, distance weighted etc).
>
> why should one have the interpolation ONCE again i.e (after copygb)
the grid fields are similar. ie. One GP has 2 values one OBS and one
FCST??? It is correct???
>
> geeta
>
>> Subject: Re: [rt.rap.ucar.edu #65427] Unable to visualize Fuzzy
verf.
>> From: met_help at ucar.edu
>> To: geeta124 at hotmail.com
>> Date: Thu, 13 Feb 2014 10:33:47 -0700
>>
>> Geeta,
>>
>> You are correct, the input forecast and observation files must be
on the same grid. In Grid-Stat, there are two ways you can perform
"fuzzy" verification.
>>
>> (1) The first way is by applying an interpolation method to the
data. Since the data are already on the same grid, this is really a
"smoothing" operation instead. This is called "upscaling".
>> Smoother forecasts and observations tend to produce better
traditional verification scores. So you could see how your scores
(like RMSE or GSS) improve as you smooth the data more and more. In
the
>> config file, you could try:
>>
>> interp = {
>> field = BOTH;
>> vld_thresh = 1.0;
>>
>> type = [
>> { method = UW_MEAN; width = 1; },
>> { method = UW_MEAN; width = 3; },
>> { method = UW_MEAN; width = 6; },
>> { method = UW_MEAN; width = 9; }
>> ];
>> };
>>
>> This tells Grid-Stat to compute its statistics 4 times, applying
more smoothing each time. Typically, the more the data has been
smoothed, the better the statistics will be.
>>
>> (2) The second way is by applying neighborhood verification
methods. The most common are the Fractions Brier Score (FBS) and
Fractions Skill Score (FSS), both contained in the NBRCNT output line
>> type. Be sure to turn the NBRCNT output line on in the Grid-Stat
config file. For neighborhood verification, you pick multiple
neighborhood sizes and look to see how the FSS changes as you increase
>> the neighborhood size. As the neighborhood size increases, FSS
increases. And you look to see how large of a neighborhood size you
need to get a "useful" (FSS > 0.5) forecast.
>>
>> Here's how this method works. You pick one or more thresholds
(cat_thresh) for your field. Grid-Stat applies the threshold to
produce a 0/1 binary field of your data. For each neighborhood size,
n,
>> it places an n x n box around each grid point and counts up the
number of events within that box. For a 3 x 3 box, if 4 of the 9
points contained an event, the value for that point is 4/9. This is
>> done for every grid point in for forecast field and the observation
field. We call the result of this process the forecast and
observation "fractional coverage" fields. The FSS and FBS scores are
>> computed by comparing the forecast and observation fractional
coverage fields to each other.
>>
>> If you're verifying a single field using 3 different thresholds and
6 different neighborhood sizes, you'd get 18 NBRCNT lines in the
output file.
>>
>> Here's an example of how you might set this up in the Grid-Stat
config file:
>>
>> nbrhd = {
>> vld_thresh = 1.0;
>> width = [ 3, 5, 9, 11, 13, 15 ];
>> cov_thresh = [ >=0.5 ];
>> }
>>
>> For a given threshold, you should look to see how FSS changes as
you increase the neighborhood size.
>>
>> Hopefully that helps get you going.
>>
>> Thanks,
>> John Halley Gotway
>> met_help at ucar.edu
>>
>>
>> On 02/12/2014 10:49 PM, Geeta Geeta via RT wrote:
>>>
>>> Wed Feb 12 22:49:14 2014: Request 65427 was acted upon.
>>> Transaction: Ticket created by geeta124 at hotmail.com
>>> Queue: met_help
>>> Subject: Unable to visualize Fuzzy verf.
>>> Owner: Nobody
>>> Requestors: geeta124 at hotmail.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=65427 >
>>>
>>>
>>> Hi John/ met_help.
>>>
>>> I was reading MET doc that mentions about the FUZZY verification
methods. I am trying to visualise what grid stat does or how it
functions.
>>> After the copygb is run, the FCST and OBS are on the same grid. ie
1------------------2-----------------------3|
| | |
| || |
|| |
|4-------------------5-----------------------6
>>> ie at the Grid Points (GP) 1 to 6, U have Observations and the
model FCST.Now the MET doc (Pg 5-3) says that a SQUARE search window
is defined around each grid point, within which the obs and the FCST
events are counted. 1. I want to know HOW is this SQUARE WINDOW is
defined (I mean in the configuration file) of Grid stat. 2. How Can I
change the size of the SQUARE window. 3. If My model resolution in
10km and I am interested in the synoptic scale phenomenon, then what
should be the window size??????????????? your help is urgently
required.
>>>
>>> geeta
>>>
>>
>
>
------------------------------------------------
More information about the Met_help
mailing list