[Met_help] [rt.rap.ucar.edu #75271] History for Ensemble Fractions Skill Score Question
John Halley Gotway via RT
met_help at ucar.edu
Mon Mar 7 14:22:59 MST 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John, I have been running MET to generate fractions skill scores for different precipitation thresholds. We are verifying 4km ensemble forecasts using Stage IV for ground truth. What has me baffled is that I am getting the same Observation Rate not matter if I am verifying 6hr accumulations of precip > .01" or precip >1.00". I would expect the observation rate to be greater for precip > .01" than for precip > 1.0" (a much more rare event) but it stays the same for all thresholds. The model forecast frequencies to decrease as the precip threshold increases as I would export.
Below is a the configuration file I use for gridstat.
////////////////////////////////////////////////////////////////////////////////
//
// Grid-Stat configuration file.
//
// For additional information, see the MET_BASE/config/README file.
//
////////////////////////////////////////////////////////////////////////////////
//
// Output model name to be written
//
model = "WRF";
obtype = "ANALYS";
////////////////////////////////////////////////////////////////////////////////
//
// Verification grid
//
regrid = {
to_grid = FCST;
vld_thresh = 0.5;
method = BUDGET;
width = 2;
}
////////////////////////////////////////////////////////////////////////////////
cat_thresh = [ NA ];
cnt_thresh = [ NA ];
cnt_logic = UNION;
wind_thresh = [ NA ];
wind_logic = UNION;
prob=[ "True" ];
//
// Forecast and observation fields to be verified
//
fcst = {
wind_thresh = [ NA ];
field = [
{
name = "QP100";
level = [ "A06" ];
GRIB1_ptv = 3;
cat_thresh = [ > 0 ];
}
];
}
obs = {
wind_thresh = [ NA ];
field = [
{
name = "APCP_06";
level = [ "(*,*)" ];
cat_thresh = [ > 0 ];
}
];
}
////////////////////////////////////////////////////////////////////////////////
//
// Climatology mean data
//
climo_mean = {
file_name = [];
field = [];
regrid = {
vld_thresh = 0.5;
method = NEAREST;
width = 1;
}
time_interp_method = DW_MEAN;
match_day = FALSE;
time_step = 21600;
}
////////////////////////////////////////////////////////////////////////////////
//
// Verification masking regions
//
mask = {
grid = [ "FULL" ];
poly = [];
}
////////////////////////////////////////////////////////////////////////////////
//
// Confidence interval settings
//
ci_alpha = [ 0.05 ];
boot = {
interval = PCTILE;
rep_prop = 1.0;
n_rep = 0;
rng = "mt19937";
seed = "";
}
////////////////////////////////////////////////////////////////////////////////
//
// Data smoothing methods
//
interp = {
field = BOTH;
vld_thresh = 1.0;
type = [
{
method = NEAREST;
width = 1;
}
];
}
////////////////////////////////////////////////////////////////////////////////
//
// Neighborhood methods
//
nbrhd = {
vld_thresh = 1.0;
width = [1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
cov_thresh = [ >=0.5 ];
}
////////////////////////////////////////////////////////////////////////////////
//
// Statistical output types
//
output_flag = {
fho = NONE;
ctc = NONE;
cts = NONE;
mctc = NONE;
mcts = NONE;
cnt = NONE;
sl1l2 = NONE;
sal1l2 = NONE;
vl1l2 = NONE;
val1l2 = NONE;
pct = NONE;
pstd = NONE;
pjc = NONE;
prc = NONE;
nbrctc = NONE;
nbrcts = NONE;
nbrcnt = STAT;
}
//
// NetCDF matched pairs output file
//
nc_pairs_flag = {
latlon = TRUE;
raw = TRUE;
diff = TRUE;
climo = TRUE;
}
////////////////////////////////////////////////////////////////////////////////
rank_corr_flag = FALSE;
tmp_dir = "/tmp";
output_prefix = "100_APCP_6";
version = "V5.1";
////////////////////////////////////////////////////////////////////////////////
In this configuration file I change forecast field name and output prefix as vary the precipitation thresholds. I also vary the neighborhood size. Could I have something not set right in the config file?
Thanks
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Ensemble Fractions Skill Score Question
From: John Halley Gotway
Time: Thu Feb 25 14:11:28 2016
Bob,
I see you have some questions about the fractions skill score. And I
read
that you're varying the precipitation threshold in the configuration
file
for each run.
I have a couple of suggestions...
First, there's no need to run Grid-Stat multiple times with different
precipitation thresholds. Instead, you can list multiple thresholds,
like
this:
cat_thresh = [ >0, >0.254, >2.54, >25.4 ];
I'm assuming the precip is in mm, and I'm using 25.4 for 1 inch. You
should see a NBRCNT output line for each combination of precip
threshold
and neighborhood size. So that means you wouldn't need to include the
threshold in the output prefix.
Second, I find using environment variables in the MET config files to
very
convenient. For example, the script which calls Grid-Stat could set
an
environment variable named "FCST_NAME" and you could refer to that in
the
config file, like this:
name = "${FCST_NAME}";
...
And later on use it in the output prefix as well if you'd like:
output_prefix = "${FCST_NAME}";
Just wanted to mention it.
I just read your next email... and yes, I'd suggest defining the
thresholds
once and then referring to them in both the fcst and obs sections,
like
this:
precip_thresh = [ >0, >0.254, >2.54, >25.4 ];
And in the fcst and obs sections, set:
cat_thresh = precip_thresh;
Lastly, you mention that you're seeing the same value for Observation
Rate. Can you tell me specifically the name of the column and the
line
type, you're looking at? And better yet, send me some sample output
files
where that value is the same. That'd help me to better understand
what's
going on.
Thanks,
John
On Thu, Feb 25, 2016 at 1:41 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu Feb 25 13:41:02 2016: Request 75271 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Ensemble Fractions Skill Score Question
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
>
>
> John, I have been running MET to generate fractions skill scores for
> different precipitation thresholds. We are verifying 4km ensemble
> forecasts using Stage IV for ground truth. What has me baffled is
that I
> am getting the same Observation Rate not matter if I am verifying
6hr
> accumulations of precip > .01" or precip >1.00". I would expect the
> observation rate to be greater for precip > .01" than for precip >
1.0" (a
> much more rare event) but it stays the same for all thresholds. The
model
> forecast frequencies to decrease as the precip threshold increases
as I
> would export.
>
> Below is a the configuration file I use for gridstat.
>
>
>
////////////////////////////////////////////////////////////////////////////////
> //
> // Grid-Stat configuration file.
> //
> // For additional information, see the MET_BASE/config/README file.
> //
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Output model name to be written
> //
> model = "WRF";
>
> obtype = "ANALYS";
>
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Verification grid
> //
> regrid = {
> to_grid = FCST;
> vld_thresh = 0.5;
> method = BUDGET;
> width = 2;
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> cat_thresh = [ NA ];
> cnt_thresh = [ NA ];
> cnt_logic = UNION;
> wind_thresh = [ NA ];
> wind_logic = UNION;
> prob=[ "True" ];
> //
> // Forecast and observation fields to be verified
> //
> fcst = {
> wind_thresh = [ NA ];
>
> field = [
> {
> name = "QP100";
> level = [ "A06" ];
> GRIB1_ptv = 3;
> cat_thresh = [ > 0 ];
> }
> ];
>
> }
> obs = {
> wind_thresh = [ NA ];
>
> field = [
> {
> name = "APCP_06";
> level = [ "(*,*)" ];
> cat_thresh = [ > 0 ];
> }
> ];
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Climatology mean data
> //
> climo_mean = {
>
> file_name = [];
> field = [];
>
> regrid = {
> vld_thresh = 0.5;
> method = NEAREST;
> width = 1;
> }
>
> time_interp_method = DW_MEAN;
> match_day = FALSE;
> time_step = 21600;
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Verification masking regions
> //
> mask = {
> grid = [ "FULL" ];
> poly = [];
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Confidence interval settings
> //
> ci_alpha = [ 0.05 ];
>
> boot = {
> interval = PCTILE;
> rep_prop = 1.0;
> n_rep = 0;
> rng = "mt19937";
> seed = "";
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Data smoothing methods
> //
> interp = {
> field = BOTH;
> vld_thresh = 1.0;
>
> type = [
> {
> method = NEAREST;
> width = 1;
> }
> ];
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Neighborhood methods
> //
> nbrhd = {
> vld_thresh = 1.0;
> width =
>
[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
> cov_thresh = [ >=0.5 ];
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Statistical output types
> //
> output_flag = {
> fho = NONE;
> ctc = NONE;
> cts = NONE;
> mctc = NONE;
> mcts = NONE;
> cnt = NONE;
> sl1l2 = NONE;
> sal1l2 = NONE;
> vl1l2 = NONE;
> val1l2 = NONE;
> pct = NONE;
> pstd = NONE;
> pjc = NONE;
> prc = NONE;
> nbrctc = NONE;
> nbrcts = NONE;
> nbrcnt = STAT;
> }
>
> //
> // NetCDF matched pairs output file
> //
> nc_pairs_flag = {
> latlon = TRUE;
> raw = TRUE;
> diff = TRUE;
> climo = TRUE;
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> rank_corr_flag = FALSE;
> tmp_dir = "/tmp";
> output_prefix = "100_APCP_6";
> version = "V5.1";
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
>
> In this configuration file I change forecast field name and output
prefix
> as vary the precipitation thresholds. I also vary the neighborhood
size.
> Could I have something not set right in the config file?
>
> Thanks
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score Question
From: robert.craig.2 at us.af.mil
Time: Thu Feb 25 14:24:07 2016
John, the observation rate should be directly correlated with the
precipitation threshold (I believe). So by not setting the obs cat (
leaving it to the default value > 0) which I was doing, that should be
the reason for the observation rate not changing. I am testing that
theory now and will send the output to you if this doesn't solve the
problem.
More general question - I ran both cases ( where I regrid obs to model
space and model to ob space). For the fractions skill score where we
are using it to compare 4km with 20km ensembles, is which is the best
way to do it to get the most meaningful results?
Also, thank Tressa for her input on error bars. I am hoping I can
fund Matt Sittel to develop a method to generate meaningful error bars
for a roll up statistic.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, February 25, 2016 3:11 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
Question
Bob,
I see you have some questions about the fractions skill score. And I
read that you're varying the precipitation threshold in the
configuration file for each run.
I have a couple of suggestions...
First, there's no need to run Grid-Stat multiple times with different
precipitation thresholds. Instead, you can list multiple thresholds,
like
this:
cat_thresh = [ >0, >0.254, >2.54, >25.4 ];
I'm assuming the precip is in mm, and I'm using 25.4 for 1 inch. You
should see a NBRCNT output line for each combination of precip
threshold and neighborhood size. So that means you wouldn't need to
include the threshold in the output prefix.
Second, I find using environment variables in the MET config files to
very convenient. For example, the script which calls Grid-Stat could
set an environment variable named "FCST_NAME" and you could refer to
that in the config file, like this:
name = "${FCST_NAME}";
...
And later on use it in the output prefix as well if you'd like:
output_prefix = "${FCST_NAME}";
Just wanted to mention it.
I just read your next email... and yes, I'd suggest defining the
thresholds once and then referring to them in both the fcst and obs
sections, like
this:
precip_thresh = [ >0, >0.254, >2.54, >25.4 ];
And in the fcst and obs sections, set:
cat_thresh = precip_thresh;
Lastly, you mention that you're seeing the same value for Observation
Rate. Can you tell me specifically the name of the column and the
line type, you're looking at? And better yet, send me some sample
output files where that value is the same. That'd help me to better
understand what's going on.
Thanks,
John
On Thu, Feb 25, 2016 at 1:41 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu Feb 25 13:41:02 2016: Request 75271 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Ensemble Fractions Skill Score Question
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271
> >
>
>
> John, I have been running MET to generate fractions skill scores for
> different precipitation thresholds. We are verifying 4km ensemble
> forecasts using Stage IV for ground truth. What has me baffled is
that I
> am getting the same Observation Rate not matter if I am verifying
6hr
> accumulations of precip > .01" or precip >1.00". I would expect the
> observation rate to be greater for precip > .01" than for precip >
> 1.0" (a much more rare event) but it stays the same for all
> thresholds. The model forecast frequencies to decrease as the
precip
> threshold increases as I would export.
>
> Below is a the configuration file I use for gridstat.
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
> //
> // Grid-Stat configuration file.
> //
> // For additional information, see the MET_BASE/config/README file.
> //
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Output model name to be written
> //
> model = "WRF";
>
> obtype = "ANALYS";
>
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Verification grid
> //
> regrid = {
> to_grid = FCST;
> vld_thresh = 0.5;
> method = BUDGET;
> width = 2;
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> cat_thresh = [ NA ];
> cnt_thresh = [ NA ];
> cnt_logic = UNION;
> wind_thresh = [ NA ];
> wind_logic = UNION;
> prob=[ "True" ];
> //
> // Forecast and observation fields to be verified // fcst = {
> wind_thresh = [ NA ];
>
> field = [
> {
> name = "QP100";
> level = [ "A06" ];
> GRIB1_ptv = 3;
> cat_thresh = [ > 0 ];
> }
> ];
>
> }
> obs = {
> wind_thresh = [ NA ];
>
> field = [
> {
> name = "APCP_06";
> level = [ "(*,*)" ];
> cat_thresh = [ > 0 ];
> }
> ];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Climatology mean data
> //
> climo_mean = {
>
> file_name = [];
> field = [];
>
> regrid = {
> vld_thresh = 0.5;
> method = NEAREST;
> width = 1;
> }
>
> time_interp_method = DW_MEAN;
> match_day = FALSE;
> time_step = 21600;
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Verification masking regions
> //
> mask = {
> grid = [ "FULL" ];
> poly = [];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Confidence interval settings
> //
> ci_alpha = [ 0.05 ];
>
> boot = {
> interval = PCTILE;
> rep_prop = 1.0;
> n_rep = 0;
> rng = "mt19937";
> seed = "";
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Data smoothing methods
> //
> interp = {
> field = BOTH;
> vld_thresh = 1.0;
>
> type = [
> {
> method = NEAREST;
> width = 1;
> }
> ];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Neighborhood methods
> //
> nbrhd = {
> vld_thresh = 1.0;
> width =
>
[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
> cov_thresh = [ >=0.5 ];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Statistical output types
> //
> output_flag = {
> fho = NONE;
> ctc = NONE;
> cts = NONE;
> mctc = NONE;
> mcts = NONE;
> cnt = NONE;
> sl1l2 = NONE;
> sal1l2 = NONE;
> vl1l2 = NONE;
> val1l2 = NONE;
> pct = NONE;
> pstd = NONE;
> pjc = NONE;
> prc = NONE;
> nbrctc = NONE;
> nbrcts = NONE;
> nbrcnt = STAT;
> }
>
> //
> // NetCDF matched pairs output file
> //
> nc_pairs_flag = {
> latlon = TRUE;
> raw = TRUE;
> diff = TRUE;
> climo = TRUE;
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> rank_corr_flag = FALSE;
> tmp_dir = "/tmp";
> output_prefix = "100_APCP_6";
> version = "V5.1";
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
>
> In this configuration file I change forecast field name and output
> prefix as vary the precipitation thresholds. I also vary the
neighborhood size.
> Could I have something not set right in the config file?
>
> Thanks
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score Question
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Fri Feb 26 05:11:27 2016
You're going to fund me? I don't come cheap, just so you know... :)
-----Original Message-----
From: robert.craig.2 at us.af.mil via RT [mailto:met_help at ucar.edu]
Sent: Thursday, February 25, 2016 3:24 PM
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: RE: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
Question
John, the observation rate should be directly correlated with the
precipitation threshold (I believe). So by not setting the obs cat (
leaving it to the default value > 0) which I was doing, that should be
the reason for the observation rate not changing. I am testing that
theory now and will send the output to you if this doesn't solve the
problem.
More general question - I ran both cases ( where I regrid obs to model
space and model to ob space). For the fractions skill score where we
are using it to compare 4km with 20km ensembles, is which is the best
way to do it to get the most meaningful results?
Also, thank Tressa for her input on error bars. I am hoping I can
fund Matt Sittel to develop a method to generate meaningful error bars
for a roll up statistic.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, February 25, 2016 3:11 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
Question
Bob,
I see you have some questions about the fractions skill score. And I
read that you're varying the precipitation threshold in the
configuration file for each run.
I have a couple of suggestions...
First, there's no need to run Grid-Stat multiple times with different
precipitation thresholds. Instead, you can list multiple thresholds,
like
this:
cat_thresh = [ >0, >0.254, >2.54, >25.4 ];
I'm assuming the precip is in mm, and I'm using 25.4 for 1 inch. You
should see a NBRCNT output line for each combination of precip
threshold and neighborhood size. So that means you wouldn't need to
include the threshold in the output prefix.
Second, I find using environment variables in the MET config files to
very convenient. For example, the script which calls Grid-Stat could
set an environment variable named "FCST_NAME" and you could refer to
that in the config file, like this:
name = "${FCST_NAME}";
...
And later on use it in the output prefix as well if you'd like:
output_prefix = "${FCST_NAME}";
Just wanted to mention it.
I just read your next email... and yes, I'd suggest defining the
thresholds once and then referring to them in both the fcst and obs
sections, like
this:
precip_thresh = [ >0, >0.254, >2.54, >25.4 ];
And in the fcst and obs sections, set:
cat_thresh = precip_thresh;
Lastly, you mention that you're seeing the same value for Observation
Rate. Can you tell me specifically the name of the column and the
line type, you're looking at? And better yet, send me some sample
output files where that value is the same. That'd help me to better
understand what's going on.
Thanks,
John
On Thu, Feb 25, 2016 at 1:41 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu Feb 25 13:41:02 2016: Request 75271 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Ensemble Fractions Skill Score Question
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271
> >
>
>
> John, I have been running MET to generate fractions skill scores for
> different precipitation thresholds. We are verifying 4km ensemble
> forecasts using Stage IV for ground truth. What has me baffled is
that I
> am getting the same Observation Rate not matter if I am verifying
6hr
> accumulations of precip > .01" or precip >1.00". I would expect the
> observation rate to be greater for precip > .01" than for precip >
> 1.0" (a much more rare event) but it stays the same for all
> thresholds. The model forecast frequencies to decrease as the
precip
> threshold increases as I would export.
>
> Below is a the configuration file I use for gridstat.
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
> //
> // Grid-Stat configuration file.
> //
> // For additional information, see the MET_BASE/config/README file.
> //
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Output model name to be written
> //
> model = "WRF";
>
> obtype = "ANALYS";
>
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Verification grid
> //
> regrid = {
> to_grid = FCST;
> vld_thresh = 0.5;
> method = BUDGET;
> width = 2;
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> cat_thresh = [ NA ];
> cnt_thresh = [ NA ];
> cnt_logic = UNION;
> wind_thresh = [ NA ];
> wind_logic = UNION;
> prob=[ "True" ];
> //
> // Forecast and observation fields to be verified // fcst = {
> wind_thresh = [ NA ];
>
> field = [
> {
> name = "QP100";
> level = [ "A06" ];
> GRIB1_ptv = 3;
> cat_thresh = [ > 0 ];
> }
> ];
>
> }
> obs = {
> wind_thresh = [ NA ];
>
> field = [
> {
> name = "APCP_06";
> level = [ "(*,*)" ];
> cat_thresh = [ > 0 ];
> }
> ];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Climatology mean data
> //
> climo_mean = {
>
> file_name = [];
> field = [];
>
> regrid = {
> vld_thresh = 0.5;
> method = NEAREST;
> width = 1;
> }
>
> time_interp_method = DW_MEAN;
> match_day = FALSE;
> time_step = 21600;
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Verification masking regions
> //
> mask = {
> grid = [ "FULL" ];
> poly = [];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Confidence interval settings
> //
> ci_alpha = [ 0.05 ];
>
> boot = {
> interval = PCTILE;
> rep_prop = 1.0;
> n_rep = 0;
> rng = "mt19937";
> seed = "";
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Data smoothing methods
> //
> interp = {
> field = BOTH;
> vld_thresh = 1.0;
>
> type = [
> {
> method = NEAREST;
> width = 1;
> }
> ];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Neighborhood methods
> //
> nbrhd = {
> vld_thresh = 1.0;
> width =
>
[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
> cov_thresh = [ >=0.5 ];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Statistical output types
> //
> output_flag = {
> fho = NONE;
> ctc = NONE;
> cts = NONE;
> mctc = NONE;
> mcts = NONE;
> cnt = NONE;
> sl1l2 = NONE;
> sal1l2 = NONE;
> vl1l2 = NONE;
> val1l2 = NONE;
> pct = NONE;
> pstd = NONE;
> pjc = NONE;
> prc = NONE;
> nbrctc = NONE;
> nbrcts = NONE;
> nbrcnt = STAT;
> }
>
> //
> // NetCDF matched pairs output file
> //
> nc_pairs_flag = {
> latlon = TRUE;
> raw = TRUE;
> diff = TRUE;
> climo = TRUE;
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> rank_corr_flag = FALSE;
> tmp_dir = "/tmp";
> output_prefix = "100_APCP_6";
> version = "V5.1";
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
>
> In this configuration file I change forecast field name and output
> prefix as vary the precipitation thresholds. I also vary the
neighborhood size.
> Could I have something not set right in the config file?
>
> Thanks
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score Question
From: robert.craig.2 at us.af.mil
Time: Fri Feb 26 14:14:22 2016
John, I changed the ob variable category to > .01, >.10 , etc.. The
precip values are in inches in my input netCDF file. Since I am
dealing with ensemble probabilities, should I have the fcst categories
the same as precip or let it default to > 0? Now, when I do an ncdump
of the MET produced data file (ncdump -h
grid_stat_1_APCP_6_240000L_20160213_000000V_pairs.nc), I see that the
precip data units are in Kg/m^2, however looking at the data in the
netCDF file, the units are in inches. Will this cause any problems?
The listing is below:
dimensions:
lat = 750 ;
lon = 750 ;
variables:
float lat(lat, lon) ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
lat:standard_name = "latitude" ;
float lon(lat, lon) ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
lon:standard_name = "longitude" ;
float FCST_PQP1_A06_FULL(lat, lon) ;
FCST_PQP1_A06_FULL:name = "PQP1" ;
FCST_PQP1_A06_FULL:long_name = "PQP1 at A06" ;
FCST_PQP1_A06_FULL:level = "A06" ;
FCST_PQP1_A06_FULL:units = "%" ;
FCST_PQP1_A06_FULL:init_time = "20160212_000000" ;
FCST_PQP1_A06_FULL:init_time_ut = "1455235200" ;
FCST_PQP1_A06_FULL:valid_time = "20160213_000000" ;
FCST_PQP1_A06_FULL:valid_time_ut = "1455321600" ;
FCST_PQP1_A06_FULL:accum_time = "060000" ;
FCST_PQP1_A06_FULL:accum_time_sec = 21600 ;
FCST_PQP1_A06_FULL:_FillValue = -9999.f ;
FCST_PQP1_A06_FULL:masking_region = "FULL" ;
FCST_PQP1_A06_FULL:smoothing_method = "NEAREST" ;
FCST_PQP1_A06_FULL:smoothing_neighborhood = 1 ;
float OBS_APCP_06_A6_FULL(lat, lon) ;
OBS_APCP_06_A6_FULL:name = "APCP_06" ;
OBS_APCP_06_A6_FULL:long_name = "APCP_06 at A6" ;
OBS_APCP_06_A6_FULL:level = "A6" ;
OBS_APCP_06_A6_FULL:units = "kg/m^2" ;
OBS_APCP_06_A6_FULL:init_time = "20160213_000000" ;
OBS_APCP_06_A6_FULL:init_time_ut = "1455321600" ;
OBS_APCP_06_A6_FULL:valid_time = "20160213_000000" ;
OBS_APCP_06_A6_FULL:valid_time_ut = "1455321600" ;
OBS_APCP_06_A6_FULL:accum_time = "060000" ;
OBS_APCP_06_A6_FULL:accum_time_sec = 21600 ;
OBS_APCP_06_A6_FULL:_FillValue = -9999.f ;
OBS_APCP_06_A6_FULL:masking_region = "FULL" ;
OBS_APCP_06_A6_FULL:smoothing_method = "NEAREST" ;
OBS_APCP_06_A6_FULL:smoothing_neighborhood = 1 ;
float DIFF_PQP1_A06_APCP_06_A6_FULL(lat, lon) ;
DIFF_PQP1_A06_APCP_06_A6_FULL:name = "Forecast PQP1
minus Observed APCP_06" ;
DIFF_PQP1_A06_APCP_06_A6_FULL:long_name = "PQP1 at A06
and APCP_06 at A6" ;
DIFF_PQP1_A06_APCP_06_A6_FULL:level = "A06 and A6" ;
DIFF_PQP1_A06_APCP_06_A6_FULL:units = "% and kg/m^2" ;
DIFF_PQP1_A06_APCP_06_A6_FULL:_FillValue = -9999.f ;
DIFF_PQP1_A06_APCP_06_A6_FULL:init_time =
"20160212_000000" ;
DIFF_PQP1_A06_APCP_06_A6_FULL:init_time_ut =
"1455235200" ;
DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time =
"20160213_000000" ;
DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time_ut =
"1455321600" ;
DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time = "060000" ;
DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time_sec = 21600 ;
DIFF_PQP1_A06_APCP_06_A6_FULL:masking_region = "FULL"
;
DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_method =
"NEAREST" ;
DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_neighborhood =
1 ;
// global attributes:
:FileOrigins = "File
/h/data/global/WXQC/data/fss/metoutput_fss/none/2016021200/grid_stat_1_APCP_6_240000L_20160213_000000V_pairs.nc
generated 20160225_230147 UTC on host dev8dlsn1 by the MET grid_stat
tool" ;
:MET_version = "V5.1" ;
:MET_tool = "grid_stat" ;
:model = "WRF" ;
:obtype = "ANALYS" ;
:Difference = "Forecast Value - Observation Value" ;
:Projection = "Polar Stereographic" ;
:hemisphere = "N" ;
:scale_lat = "60.000000 degrees_north" ;
:lat_pin = "6.590000" ;
:lon_pin = "35.000000" ;
:x_pin = "0.000000" ;
:y_pin = "0.000000" ;
:lon_orient = "80.000000" ;
:d_km = "20.000000 km" ;
:r_km = "6371.200000 km" ;
:nx = "750" ;
:ny = "750" ;
}
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, February 25, 2016 3:11 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
Question
Bob,
I see you have some questions about the fractions skill score. And I
read that you're varying the precipitation threshold in the
configuration file for each run.
I have a couple of suggestions...
First, there's no need to run Grid-Stat multiple times with different
precipitation thresholds. Instead, you can list multiple thresholds,
like
this:
cat_thresh = [ >0, >0.254, >2.54, >25.4 ];
I'm assuming the precip is in mm, and I'm using 25.4 for 1 inch. You
should see a NBRCNT output line for each combination of precip
threshold and neighborhood size. So that means you wouldn't need to
include the threshold in the output prefix.
Second, I find using environment variables in the MET config files to
very convenient. For example, the script which calls Grid-Stat could
set an environment variable named "FCST_NAME" and you could refer to
that in the config file, like this:
name = "${FCST_NAME}";
...
And later on use it in the output prefix as well if you'd like:
output_prefix = "${FCST_NAME}";
Just wanted to mention it.
I just read your next email... and yes, I'd suggest defining the
thresholds once and then referring to them in both the fcst and obs
sections, like
this:
precip_thresh = [ >0, >0.254, >2.54, >25.4 ];
And in the fcst and obs sections, set:
cat_thresh = precip_thresh;
Lastly, you mention that you're seeing the same value for Observation
Rate. Can you tell me specifically the name of the column and the
line type, you're looking at? And better yet, send me some sample
output files where that value is the same. That'd help me to better
understand what's going on.
Thanks,
John
On Thu, Feb 25, 2016 at 1:41 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu Feb 25 13:41:02 2016: Request 75271 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Ensemble Fractions Skill Score Question
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271
> >
>
>
> John, I have been running MET to generate fractions skill scores for
> different precipitation thresholds. We are verifying 4km ensemble
> forecasts using Stage IV for ground truth. What has me baffled is
that I
> am getting the same Observation Rate not matter if I am verifying
6hr
> accumulations of precip > .01" or precip >1.00". I would expect the
> observation rate to be greater for precip > .01" than for precip >
> 1.0" (a much more rare event) but it stays the same for all
> thresholds. The model forecast frequencies to decrease as the
precip
> threshold increases as I would export.
>
> Below is a the configuration file I use for gridstat.
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
> //
> // Grid-Stat configuration file.
> //
> // For additional information, see the MET_BASE/config/README file.
> //
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Output model name to be written
> //
> model = "WRF";
>
> obtype = "ANALYS";
>
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Verification grid
> //
> regrid = {
> to_grid = FCST;
> vld_thresh = 0.5;
> method = BUDGET;
> width = 2;
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> cat_thresh = [ NA ];
> cnt_thresh = [ NA ];
> cnt_logic = UNION;
> wind_thresh = [ NA ];
> wind_logic = UNION;
> prob=[ "True" ];
> //
> // Forecast and observation fields to be verified // fcst = {
> wind_thresh = [ NA ];
>
> field = [
> {
> name = "QP100";
> level = [ "A06" ];
> GRIB1_ptv = 3;
> cat_thresh = [ > 0 ];
> }
> ];
>
> }
> obs = {
> wind_thresh = [ NA ];
>
> field = [
> {
> name = "APCP_06";
> level = [ "(*,*)" ];
> cat_thresh = [ > 0 ];
> }
> ];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Climatology mean data
> //
> climo_mean = {
>
> file_name = [];
> field = [];
>
> regrid = {
> vld_thresh = 0.5;
> method = NEAREST;
> width = 1;
> }
>
> time_interp_method = DW_MEAN;
> match_day = FALSE;
> time_step = 21600;
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Verification masking regions
> //
> mask = {
> grid = [ "FULL" ];
> poly = [];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Confidence interval settings
> //
> ci_alpha = [ 0.05 ];
>
> boot = {
> interval = PCTILE;
> rep_prop = 1.0;
> n_rep = 0;
> rng = "mt19937";
> seed = "";
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Data smoothing methods
> //
> interp = {
> field = BOTH;
> vld_thresh = 1.0;
>
> type = [
> {
> method = NEAREST;
> width = 1;
> }
> ];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Neighborhood methods
> //
> nbrhd = {
> vld_thresh = 1.0;
> width =
>
[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
> cov_thresh = [ >=0.5 ];
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> //
> // Statistical output types
> //
> output_flag = {
> fho = NONE;
> ctc = NONE;
> cts = NONE;
> mctc = NONE;
> mcts = NONE;
> cnt = NONE;
> sl1l2 = NONE;
> sal1l2 = NONE;
> vl1l2 = NONE;
> val1l2 = NONE;
> pct = NONE;
> pstd = NONE;
> pjc = NONE;
> prc = NONE;
> nbrctc = NONE;
> nbrcts = NONE;
> nbrcnt = STAT;
> }
>
> //
> // NetCDF matched pairs output file
> //
> nc_pairs_flag = {
> latlon = TRUE;
> raw = TRUE;
> diff = TRUE;
> climo = TRUE;
> }
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
> rank_corr_flag = FALSE;
> tmp_dir = "/tmp";
> output_prefix = "100_APCP_6";
> version = "V5.1";
>
>
>
//////////////////////////////////////////////////////////////////////
> //////////
>
>
> In this configuration file I change forecast field name and output
> prefix as vary the precipitation thresholds. I also vary the
neighborhood size.
> Could I have something not set right in the config file?
>
> Thanks
>
>
>
>
------------------------------------------------
Subject: Ensemble Fractions Skill Score Question
From: John Halley Gotway
Time: Wed Mar 02 11:38:15 2016
Bob,
Sorry I'm so slow in responding to this email. I was out of the
office a
couple of days in there.
Let me see if I understand what you're doing. You are running Grid-
Stat to
compare ensemble probabilities to observed precipitation. So you're
doing
probabilistic verification. And you're using StageIV as the
observation in
GRIB format.
The units for StageIV data in GRIB really are millimeters (kg/m^2).
Here are some practical considerations for doing probabilistic
verification
in MET.
(1) The only line types you want are PCT, PSTD, PJC, and PRC. Set all
the
other ones to "NONE".
(2) Probabilities are defined as a probability of some event. For
example,
probability of precip is probability of precip > 0. Use that event to
define the ***observation*** threshold.
However, you set the forecast threshold very differently. The
forecast
probability is a bunch of numbers between 0 and 1 (or 0 and 100). The
thresholds are used to partition that probability space, such as:
cat_thresh = [ >=0.00, >=0.25, >=0.50, >=0.75, >=1.00 ];
They must start with 0, end with 1, and all be the same probability
type.
Each probability value falls into exactly 1 of the 4 probability bins
listed above. MET uses these settings to defined a 4 x 2 contingency
table
(or N x 2, more generally). And all the probabilistic statistics are
derived from this table.
You can certainly use more bins than what's defined above. You might
define thresholds every 0.10 or 0.05. Just take a look at your
forecast
probabilities to get a good idea of the range of probability values
being
used.
I'm guessing you may have more questions. If you'd like, you could
send me
a sample forecast, observation, and config file you're passing to
Grid-Stat. I could take a close look, run it here, and offer some
suggestions.
Thanks,
John
On Fri, Feb 26, 2016 at 2:14 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
>
> John, I changed the ob variable category to > .01, >.10 , etc.. The
> precip values are in inches in my input netCDF file. Since I am
dealing
> with ensemble probabilities, should I have the fcst categories the
same as
> precip or let it default to > 0? Now, when I do an ncdump of the
MET
> produced data file (ncdump -h
> grid_stat_1_APCP_6_240000L_20160213_000000V_pairs.nc), I see that
the
> precip data units are in Kg/m^2, however looking at the data in the
netCDF
> file, the units are in inches. Will this cause any problems? The
listing
> is below:
>
> dimensions:
> lat = 750 ;
> lon = 750 ;
> variables:
> float lat(lat, lon) ;
> lat:long_name = "latitude" ;
> lat:units = "degrees_north" ;
> lat:standard_name = "latitude" ;
> float lon(lat, lon) ;
> lon:long_name = "longitude" ;
> lon:units = "degrees_east" ;
> lon:standard_name = "longitude" ;
> float FCST_PQP1_A06_FULL(lat, lon) ;
> FCST_PQP1_A06_FULL:name = "PQP1" ;
> FCST_PQP1_A06_FULL:long_name = "PQP1 at A06" ;
> FCST_PQP1_A06_FULL:level = "A06" ;
> FCST_PQP1_A06_FULL:units = "%" ;
> FCST_PQP1_A06_FULL:init_time = "20160212_000000" ;
> FCST_PQP1_A06_FULL:init_time_ut = "1455235200" ;
> FCST_PQP1_A06_FULL:valid_time = "20160213_000000" ;
> FCST_PQP1_A06_FULL:valid_time_ut = "1455321600" ;
> FCST_PQP1_A06_FULL:accum_time = "060000" ;
> FCST_PQP1_A06_FULL:accum_time_sec = 21600 ;
> FCST_PQP1_A06_FULL:_FillValue = -9999.f ;
> FCST_PQP1_A06_FULL:masking_region = "FULL" ;
> FCST_PQP1_A06_FULL:smoothing_method = "NEAREST" ;
> FCST_PQP1_A06_FULL:smoothing_neighborhood = 1 ;
> float OBS_APCP_06_A6_FULL(lat, lon) ;
> OBS_APCP_06_A6_FULL:name = "APCP_06" ;
> OBS_APCP_06_A6_FULL:long_name = "APCP_06 at A6" ;
> OBS_APCP_06_A6_FULL:level = "A6" ;
> OBS_APCP_06_A6_FULL:units = "kg/m^2" ;
> OBS_APCP_06_A6_FULL:init_time = "20160213_000000" ;
> OBS_APCP_06_A6_FULL:init_time_ut = "1455321600" ;
> OBS_APCP_06_A6_FULL:valid_time = "20160213_000000" ;
> OBS_APCP_06_A6_FULL:valid_time_ut = "1455321600" ;
> OBS_APCP_06_A6_FULL:accum_time = "060000" ;
> OBS_APCP_06_A6_FULL:accum_time_sec = 21600 ;
> OBS_APCP_06_A6_FULL:_FillValue = -9999.f ;
> OBS_APCP_06_A6_FULL:masking_region = "FULL" ;
> OBS_APCP_06_A6_FULL:smoothing_method = "NEAREST" ;
> OBS_APCP_06_A6_FULL:smoothing_neighborhood = 1 ;
> float DIFF_PQP1_A06_APCP_06_A6_FULL(lat, lon) ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:name = "Forecast PQP1
minus
> Observed APCP_06" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:long_name = "PQP1 at
A06 and
> APCP_06 at A6" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:level = "A06 and A6" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:units = "% and kg/m^2"
;
> DIFF_PQP1_A06_APCP_06_A6_FULL:_FillValue = -9999.f ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:init_time =
> "20160212_000000" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:init_time_ut =
"1455235200" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time =
> "20160213_000000" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time_ut =
"1455321600"
> ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time = "060000"
;
> DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time_sec = 21600
;
> DIFF_PQP1_A06_APCP_06_A6_FULL:masking_region =
"FULL" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_method =
"NEAREST"
> ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_neighborhood
= 1 ;
>
> // global attributes:
> :FileOrigins = "File
>
/h/data/global/WXQC/data/fss/metoutput_fss/none/2016021200/grid_stat_1_APCP_6_240000L_20160213_000000V_pairs.nc
> generated 20160225_230147 UTC on host dev8dlsn1 by the MET grid_stat
tool" ;
> :MET_version = "V5.1" ;
> :MET_tool = "grid_stat" ;
> :model = "WRF" ;
> :obtype = "ANALYS" ;
> :Difference = "Forecast Value - Observation Value" ;
> :Projection = "Polar Stereographic" ;
> :hemisphere = "N" ;
> :scale_lat = "60.000000 degrees_north" ;
> :lat_pin = "6.590000" ;
> :lon_pin = "35.000000" ;
> :x_pin = "0.000000" ;
> :y_pin = "0.000000" ;
> :lon_orient = "80.000000" ;
> :d_km = "20.000000 km" ;
> :r_km = "6371.200000 km" ;
> :nx = "750" ;
> :ny = "750" ;
> }
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, February 25, 2016 3:11 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
> Question
>
> Bob,
>
> I see you have some questions about the fractions skill score. And
I read
> that you're varying the precipitation threshold in the configuration
file
> for each run.
>
> I have a couple of suggestions...
>
> First, there's no need to run Grid-Stat multiple times with
different
> precipitation thresholds. Instead, you can list multiple
thresholds, like
> this:
> cat_thresh = [ >0, >0.254, >2.54, >25.4 ];
>
> I'm assuming the precip is in mm, and I'm using 25.4 for 1 inch.
You
> should see a NBRCNT output line for each combination of precip
threshold
> and neighborhood size. So that means you wouldn't need to include
the
> threshold in the output prefix.
>
> Second, I find using environment variables in the MET config files
to very
> convenient. For example, the script which calls Grid-Stat could set
an
> environment variable named "FCST_NAME" and you could refer to that
in the
> config file, like this:
> name = "${FCST_NAME}";
> ...
> And later on use it in the output prefix as well if you'd like:
> output_prefix = "${FCST_NAME}";
>
> Just wanted to mention it.
>
> I just read your next email... and yes, I'd suggest defining the
> thresholds once and then referring to them in both the fcst and obs
> sections, like
> this:
> precip_thresh = [ >0, >0.254, >2.54, >25.4 ];
>
> And in the fcst and obs sections, set:
> cat_thresh = precip_thresh;
>
> Lastly, you mention that you're seeing the same value for
Observation
> Rate. Can you tell me specifically the name of the column and the
line
> type, you're looking at? And better yet, send me some sample output
files
> where that value is the same. That'd help me to better understand
what's
> going on.
>
> Thanks,
> John
>
> On Thu, Feb 25, 2016 at 1:41 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Feb 25 13:41:02 2016: Request 75271 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Ensemble Fractions Skill Score Question
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271
> > >
> >
> >
> > John, I have been running MET to generate fractions skill scores
for
> > different precipitation thresholds. We are verifying 4km
ensemble
> > forecasts using Stage IV for ground truth. What has me baffled
is that
> I
> > am getting the same Observation Rate not matter if I am verifying
6hr
> > accumulations of precip > .01" or precip >1.00". I would expect
the
> > observation rate to be greater for precip > .01" than for precip >
> > 1.0" (a much more rare event) but it stays the same for all
> > thresholds. The model forecast frequencies to decrease as the
precip
> > threshold increases as I would export.
> >
> > Below is a the configuration file I use for gridstat.
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> > //
> > // Grid-Stat configuration file.
> > //
> > // For additional information, see the MET_BASE/config/README
file.
> > //
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> > //
> > // Output model name to be written
> > //
> > model = "WRF";
> >
> > obtype = "ANALYS";
> >
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> > //
> > // Verification grid
> > //
> > regrid = {
> > to_grid = FCST;
> > vld_thresh = 0.5;
> > method = BUDGET;
> > width = 2;
> > }
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> > cat_thresh = [ NA ];
> > cnt_thresh = [ NA ];
> > cnt_logic = UNION;
> > wind_thresh = [ NA ];
> > wind_logic = UNION;
> > prob=[ "True" ];
> > //
> > // Forecast and observation fields to be verified // fcst = {
> > wind_thresh = [ NA ];
> >
> > field = [
> > {
> > name = "QP100";
> > level = [ "A06" ];
> > GRIB1_ptv = 3;
> > cat_thresh = [ > 0 ];
> > }
> > ];
> >
> > }
> > obs = {
> > wind_thresh = [ NA ];
> >
> > field = [
> > {
> > name = "APCP_06";
> > level = [ "(*,*)" ];
> > cat_thresh = [ > 0 ];
> > }
> > ];
> > }
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> > //
> > // Climatology mean data
> > //
> > climo_mean = {
> >
> > file_name = [];
> > field = [];
> >
> > regrid = {
> > vld_thresh = 0.5;
> > method = NEAREST;
> > width = 1;
> > }
> >
> > time_interp_method = DW_MEAN;
> > match_day = FALSE;
> > time_step = 21600;
> > }
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> > //
> > // Verification masking regions
> > //
> > mask = {
> > grid = [ "FULL" ];
> > poly = [];
> > }
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> > //
> > // Confidence interval settings
> > //
> > ci_alpha = [ 0.05 ];
> >
> > boot = {
> > interval = PCTILE;
> > rep_prop = 1.0;
> > n_rep = 0;
> > rng = "mt19937";
> > seed = "";
> > }
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> > //
> > // Data smoothing methods
> > //
> > interp = {
> > field = BOTH;
> > vld_thresh = 1.0;
> >
> > type = [
> > {
> > method = NEAREST;
> > width = 1;
> > }
> > ];
> > }
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> > //
> > // Neighborhood methods
> > //
> > nbrhd = {
> > vld_thresh = 1.0;
> > width =
> >
[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
> > cov_thresh = [ >=0.5 ];
> > }
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> > //
> > // Statistical output types
> > //
> > output_flag = {
> > fho = NONE;
> > ctc = NONE;
> > cts = NONE;
> > mctc = NONE;
> > mcts = NONE;
> > cnt = NONE;
> > sl1l2 = NONE;
> > sal1l2 = NONE;
> > vl1l2 = NONE;
> > val1l2 = NONE;
> > pct = NONE;
> > pstd = NONE;
> > pjc = NONE;
> > prc = NONE;
> > nbrctc = NONE;
> > nbrcts = NONE;
> > nbrcnt = STAT;
> > }
> >
> > //
> > // NetCDF matched pairs output file
> > //
> > nc_pairs_flag = {
> > latlon = TRUE;
> > raw = TRUE;
> > diff = TRUE;
> > climo = TRUE;
> > }
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> > rank_corr_flag = FALSE;
> > tmp_dir = "/tmp";
> > output_prefix = "100_APCP_6";
> > version = "V5.1";
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //////////
> >
> >
> > In this configuration file I change forecast field name and output
> > prefix as vary the precipitation thresholds. I also vary the
> neighborhood size.
> > Could I have something not set right in the config file?
> >
> > Thanks
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Ensemble Fractions Skill Score Question
From: robert.craig.2 at us.af.mil
Time: Wed Mar 02 12:50:26 2016
Forwarded to you after I removed the NETcdf precip file. I will place
that
file on the FTP server.
Bob
-----Original Message-----
From: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Sent: Wednesday, March 02, 2016 1:48 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Subject: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
Question
-----Original Message-----
From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Sent: Wednesday, March 02, 2016 1:45 PM
To: met_help at ucar.edu
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: RE: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
Question
Hi, John,
I am not setting the fcst cat_threshold - I am letting it default to
> 0. I
am setting the obs threshold to thresholds you listed. I have
attached my
template I use for the case of precip > 1.0". I am reading a NETCDF
file
where the data appears to be in inches (looking at a dump of the
file).
Somebody else built the file for me but probably didn't set the units
correctly in the header. Would MET care if the units aren't correct?
By only setting pct,pstd,pjc,prc, aren't I eliminating neighborhood
methods
which is the whole reason I am running this to generate a fractions
skill
score. Can NBRCNT not be used for probability data? It seems to be
working.
I have attached the output file for each threshold.
I can send you the a model file where you can run the data but I will
wait
until you answer the questions above first.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, March 02, 2016 12:38 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
Question
Bob,
Sorry I'm so slow in responding to this email. I was out of the
office a
couple of days in there.
Let me see if I understand what you're doing. You are running Grid-
Stat to
compare ensemble probabilities to observed precipitation. So you're
doing
probabilistic verification. And you're using StageIV as the
observation in
GRIB format.
The units for StageIV data in GRIB really are millimeters (kg/m^2).
Here are some practical considerations for doing probabilistic
verification in
MET.
(1) The only line types you want are PCT, PSTD, PJC, and PRC. Set all
the
other ones to "NONE".
(2) Probabilities are defined as a probability of some event. For
example,
probability of precip is probability of precip > 0. Use that event to
define
the ***observation*** threshold.
However, you set the forecast threshold very differently. The
forecast
probability is a bunch of numbers between 0 and 1 (or 0 and 100). The
thresholds are used to partition that probability space, such as:
cat_thresh = [ >=0.00, >=0.25, >=0.50, >=0.75, >=1.00 ];
They must start with 0, end with 1, and all be the same probability
type.
Each probability value falls into exactly 1 of the 4 probability bins
listed
above. MET uses these settings to defined a 4 x 2 contingency table
(or N x
2, more generally). And all the probabilistic statistics are derived
from
this table.
You can certainly use more bins than what's defined above. You might
define
thresholds every 0.10 or 0.05. Just take a look at your forecast
probabilities to get a good idea of the range of probability values
being
used.
I'm guessing you may have more questions. If you'd like, you could
send me a
sample forecast, observation, and config file you're passing to Grid-
Stat. I
could take a close look, run it here, and offer some suggestions.
Thanks,
John
On Fri, Feb 26, 2016 at 2:14 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
>
> John, I changed the ob variable category to > .01, >.10 , etc.. The
> precip values are in inches in my input netCDF file. Since I am
> dealing with ensemble probabilities, should I have the fcst
categories
> the same as precip or let it default to > 0? Now, when I do an
ncdump
> of the MET produced data file (ncdump -h
> grid_stat_1_APCP_6_240000L_20160213_000000V_pairs.nc), I see that
the
> precip data units are in Kg/m^2, however looking at the data in the
netCDF
> file, the units are in inches. Will this cause any problems? The
listing
> is below:
>
> dimensions:
> lat = 750 ;
> lon = 750 ;
> variables:
> float lat(lat, lon) ;
> lat:long_name = "latitude" ;
> lat:units = "degrees_north" ;
> lat:standard_name = "latitude" ;
> float lon(lat, lon) ;
> lon:long_name = "longitude" ;
> lon:units = "degrees_east" ;
> lon:standard_name = "longitude" ;
> float FCST_PQP1_A06_FULL(lat, lon) ;
> FCST_PQP1_A06_FULL:name = "PQP1" ;
> FCST_PQP1_A06_FULL:long_name = "PQP1 at A06" ;
> FCST_PQP1_A06_FULL:level = "A06" ;
> FCST_PQP1_A06_FULL:units = "%" ;
> FCST_PQP1_A06_FULL:init_time = "20160212_000000" ;
> FCST_PQP1_A06_FULL:init_time_ut = "1455235200" ;
> FCST_PQP1_A06_FULL:valid_time = "20160213_000000" ;
> FCST_PQP1_A06_FULL:valid_time_ut = "1455321600" ;
> FCST_PQP1_A06_FULL:accum_time = "060000" ;
> FCST_PQP1_A06_FULL:accum_time_sec = 21600 ;
> FCST_PQP1_A06_FULL:_FillValue = -9999.f ;
> FCST_PQP1_A06_FULL:masking_region = "FULL" ;
> FCST_PQP1_A06_FULL:smoothing_method = "NEAREST" ;
> FCST_PQP1_A06_FULL:smoothing_neighborhood = 1 ;
> float OBS_APCP_06_A6_FULL(lat, lon) ;
> OBS_APCP_06_A6_FULL:name = "APCP_06" ;
> OBS_APCP_06_A6_FULL:long_name = "APCP_06 at A6" ;
> OBS_APCP_06_A6_FULL:level = "A6" ;
> OBS_APCP_06_A6_FULL:units = "kg/m^2" ;
> OBS_APCP_06_A6_FULL:init_time = "20160213_000000" ;
> OBS_APCP_06_A6_FULL:init_time_ut = "1455321600" ;
> OBS_APCP_06_A6_FULL:valid_time = "20160213_000000" ;
> OBS_APCP_06_A6_FULL:valid_time_ut = "1455321600" ;
> OBS_APCP_06_A6_FULL:accum_time = "060000" ;
> OBS_APCP_06_A6_FULL:accum_time_sec = 21600 ;
> OBS_APCP_06_A6_FULL:_FillValue = -9999.f ;
> OBS_APCP_06_A6_FULL:masking_region = "FULL" ;
> OBS_APCP_06_A6_FULL:smoothing_method = "NEAREST" ;
> OBS_APCP_06_A6_FULL:smoothing_neighborhood = 1 ;
> float DIFF_PQP1_A06_APCP_06_A6_FULL(lat, lon) ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:name = "Forecast PQP1
> minus Observed APCP_06" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:long_name = "PQP1 at
A06
> and
> APCP_06 at A6" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:level = "A06 and A6" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:units = "% and kg/m^2"
;
> DIFF_PQP1_A06_APCP_06_A6_FULL:_FillValue = -9999.f ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:init_time =
> "20160212_000000" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:init_time_ut =
"1455235200" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time =
> "20160213_000000" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time_ut =
"1455321600"
> ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time = "060000"
;
> DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time_sec = 21600
;
> DIFF_PQP1_A06_APCP_06_A6_FULL:masking_region =
"FULL" ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_method =
"NEAREST"
> ;
> DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_neighborhood
=
> 1 ;
>
> // global attributes:
> :FileOrigins = "File
>
/h/data/global/WXQC/data/fss/metoutput_fss/none/2016021200/grid_stat_1
> _APCP_6_240000L_20160213_000000V_pairs.nc
> generated 20160225_230147 UTC on host dev8dlsn1 by the MET grid_stat
tool" ;
> :MET_version = "V5.1" ;
> :MET_tool = "grid_stat" ;
> :model = "WRF" ;
> :obtype = "ANALYS" ;
> :Difference = "Forecast Value - Observation Value" ;
> :Projection = "Polar Stereographic" ;
> :hemisphere = "N" ;
> :scale_lat = "60.000000 degrees_north" ;
> :lat_pin = "6.590000" ;
> :lon_pin = "35.000000" ;
> :x_pin = "0.000000" ;
> :y_pin = "0.000000" ;
> :lon_orient = "80.000000" ;
> :d_km = "20.000000 km" ;
> :r_km = "6371.200000 km" ;
> :nx = "750" ;
> :ny = "750" ;
> }
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, February 25, 2016 3:11 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
> Question
>
> Bob,
>
> I see you have some questions about the fractions skill score. And
I
> read that you're varying the precipitation threshold in the
> configuration file for each run.
>
> I have a couple of suggestions...
>
> First, there's no need to run Grid-Stat multiple times with
different
> precipitation thresholds. Instead, you can list multiple
thresholds,
> like
> this:
> cat_thresh = [ >0, >0.254, >2.54, >25.4 ];
>
> I'm assuming the precip is in mm, and I'm using 25.4 for 1 inch.
You
> should see a NBRCNT output line for each combination of precip
> threshold and neighborhood size. So that means you wouldn't need to
> include the threshold in the output prefix.
>
> Second, I find using environment variables in the MET config files
to
> very convenient. For example, the script which calls Grid-Stat
could
> set an environment variable named "FCST_NAME" and you could refer to
> that in the config file, like this:
> name = "${FCST_NAME}";
> ...
> And later on use it in the output prefix as well if you'd like:
> output_prefix = "${FCST_NAME}";
>
> Just wanted to mention it.
>
> I just read your next email... and yes, I'd suggest defining the
> thresholds once and then referring to them in both the fcst and obs
> sections, like
> this:
> precip_thresh = [ >0, >0.254, >2.54, >25.4 ];
>
> And in the fcst and obs sections, set:
> cat_thresh = precip_thresh;
>
> Lastly, you mention that you're seeing the same value for
Observation
> Rate. Can you tell me specifically the name of the column and the
> line type, you're looking at? And better yet, send me some sample
> output files where that value is the same. That'd help me to better
> understand what's going on.
>
> Thanks,
> John
>
> On Thu, Feb 25, 2016 at 1:41 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Feb 25 13:41:02 2016: Request 75271 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Ensemble Fractions Skill Score Question
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271
> > >
> >
> >
> > John, I have been running MET to generate fractions skill scores
for
> > different precipitation thresholds. We are verifying 4km
ensemble
> > forecasts using Stage IV for ground truth. What has me baffled
is that
> I
> > am getting the same Observation Rate not matter if I am verifying
> > 6hr accumulations of precip > .01" or precip >1.00". I would
expect
> > the observation rate to be greater for precip > .01" than for
precip
> > > 1.0" (a much more rare event) but it stays the same for all
> > thresholds. The model forecast frequencies to decrease as the
> > precip threshold increases as I would export.
> >
> > Below is a the configuration file I use for gridstat.
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> > //
> > // Grid-Stat configuration file.
> > //
> > // For additional information, see the MET_BASE/config/README
file.
> > //
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> > //
> > // Output model name to be written
> > //
> > model = "WRF";
> >
> > obtype = "ANALYS";
> >
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> > //
> > // Verification grid
> > //
> > regrid = {
> > to_grid = FCST;
> > vld_thresh = 0.5;
> > method = BUDGET;
> > width = 2;
> > }
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> > cat_thresh = [ NA ];
> > cnt_thresh = [ NA ];
> > cnt_logic = UNION;
> > wind_thresh = [ NA ];
> > wind_logic = UNION;
> > prob=[ "True" ];
> > //
> > // Forecast and observation fields to be verified // fcst = {
> > wind_thresh = [ NA ];
> >
> > field = [
> > {
> > name = "QP100";
> > level = [ "A06" ];
> > GRIB1_ptv = 3;
> > cat_thresh = [ > 0 ];
> > }
> > ];
> >
> > }
> > obs = {
> > wind_thresh = [ NA ];
> >
> > field = [
> > {
> > name = "APCP_06";
> > level = [ "(*,*)" ];
> > cat_thresh = [ > 0 ];
> > }
> > ];
> > }
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> > //
> > // Climatology mean data
> > //
> > climo_mean = {
> >
> > file_name = [];
> > field = [];
> >
> > regrid = {
> > vld_thresh = 0.5;
> > method = NEAREST;
> > width = 1;
> > }
> >
> > time_interp_method = DW_MEAN;
> > match_day = FALSE;
> > time_step = 21600;
> > }
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> > //
> > // Verification masking regions
> > //
> > mask = {
> > grid = [ "FULL" ];
> > poly = [];
> > }
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> > //
> > // Confidence interval settings
> > //
> > ci_alpha = [ 0.05 ];
> >
> > boot = {
> > interval = PCTILE;
> > rep_prop = 1.0;
> > n_rep = 0;
> > rng = "mt19937";
> > seed = "";
> > }
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> > //
> > // Data smoothing methods
> > //
> > interp = {
> > field = BOTH;
> > vld_thresh = 1.0;
> >
> > type = [
> > {
> > method = NEAREST;
> > width = 1;
> > }
> > ];
> > }
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> > //
> > // Neighborhood methods
> > //
> > nbrhd = {
> > vld_thresh = 1.0;
> > width =
> >
[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
> > cov_thresh = [ >=0.5 ];
> > }
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> > //
> > // Statistical output types
> > //
> > output_flag = {
> > fho = NONE;
> > ctc = NONE;
> > cts = NONE;
> > mctc = NONE;
> > mcts = NONE;
> > cnt = NONE;
> > sl1l2 = NONE;
> > sal1l2 = NONE;
> > vl1l2 = NONE;
> > val1l2 = NONE;
> > pct = NONE;
> > pstd = NONE;
> > pjc = NONE;
> > prc = NONE;
> > nbrctc = NONE;
> > nbrcts = NONE;
> > nbrcnt = STAT;
> > }
> >
> > //
> > // NetCDF matched pairs output file
> > //
> > nc_pairs_flag = {
> > latlon = TRUE;
> > raw = TRUE;
> > diff = TRUE;
> > climo = TRUE;
> > }
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> > rank_corr_flag = FALSE;
> > tmp_dir = "/tmp";
> > output_prefix = "100_APCP_6";
> > version = "V5.1";
> >
> >
> >
////////////////////////////////////////////////////////////////////
> > //
> > //////////
> >
> >
> > In this configuration file I change forecast field name and output
> > prefix as vary the precipitation thresholds. I also vary the
> neighborhood size.
> > Could I have something not set right in the config file?
> >
> > Thanks
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Ensemble Fractions Skill Score Question
From: John Halley Gotway
Time: Fri Mar 04 15:59:10 2016
Hello Bob,
Sorry for the delay. I just talked to Tara and Tressa about this and
have
some feedback for you. Sorry for any confusion my previous email may
have
caused... it wasn't clear to me what you were trying to do.
First, you're correct that you can use Grid-Stat to compare a field of
probability values to precip amounts and compute fractions skill
score. We
haven't done that within the DTC, and Tressa isn't aware of any
published
examples, but there's no reason you can't do that. We have used MODE
to
compare this type of data, typically looking at probability objects >
10%,
25%, and 50% or so. That's somewhat similar to what you're doing.
I would say, if you have not yet applied the probabilistic methods
available in Grid-Stat, I'd suggest trying it. They implement
statistics
that are widely used and applied for probabilistic verification. To
that
end, I attached a sample Grid-Stat config file you could use for
probabilistic verification (GridStatConfig_PROB).
By looking at the .stat files you sent, I see that you're keeping the
forecast threshold fixed as >0 and then varying the observation
threshold
based on the probability field being evaluated (e.g. for QP100,
OBS_THRESH
>=1.0). That sounds good to me.
However, >0 is a very low probability threshold. You might also
consider
applying some higher probability thresholds. For very low
probabilities,
you'd hope that there isn't anything present in the observations most
of
the time. So try using >0, >10, >25, >50 or something like that.
This
should be very easy to do. In the "fcst" section set:
cat_thresh = [ >0, >10, >25, >50 ];
And in the "obs" section set:
cat_thresh = [ >=1.0, >=1.0, >=1.0, >=1.0 ];
You just repeat the observation threshold once for each forecast
threshold
to make them the same length.
Hopefully that helps. Just let us know what other questions come up.
Thanks,
John
On Wed, Mar 2, 2016 at 12:50 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
>
> Forwarded to you after I removed the NETcdf precip file. I will
place that
> file on the FTP server.
>
> Bob
>
> -----Original Message-----
> From: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Sent: Wednesday, March 02, 2016 1:48 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Subject: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
> Question
>
>
>
> -----Original Message-----
> From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Sent: Wednesday, March 02, 2016 1:45 PM
> To: met_help at ucar.edu
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: RE: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
> Question
>
> Hi, John,
>
> I am not setting the fcst cat_threshold - I am letting it default
to >
> 0. I
> am setting the obs threshold to thresholds you listed. I have
attached my
> template I use for the case of precip > 1.0". I am reading a NETCDF
file
> where the data appears to be in inches (looking at a dump of the
file).
> Somebody else built the file for me but probably didn't set the
units
> correctly in the header. Would MET care if the units aren't
correct?
>
> By only setting pct,pstd,pjc,prc, aren't I eliminating neighborhood
methods
> which is the whole reason I am running this to generate a fractions
skill
> score. Can NBRCNT not be used for probability data? It seems to be
> working.
> I have attached the output file for each threshold.
>
> I can send you the a model file where you can run the data but I
will wait
> until you answer the questions above first.
>
> Thanks
>
> Bob
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, March 02, 2016 12:38 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
> Question
>
> Bob,
>
> Sorry I'm so slow in responding to this email. I was out of the
office a
> couple of days in there.
>
> Let me see if I understand what you're doing. You are running Grid-
Stat to
> compare ensemble probabilities to observed precipitation. So you're
doing
> probabilistic verification. And you're using StageIV as the
observation in
> GRIB format.
>
> The units for StageIV data in GRIB really are millimeters (kg/m^2).
>
> Here are some practical considerations for doing probabilistic
> verification in
> MET.
>
> (1) The only line types you want are PCT, PSTD, PJC, and PRC. Set
all the
> other ones to "NONE".
> (2) Probabilities are defined as a probability of some event. For
example,
> probability of precip is probability of precip > 0. Use that event
to
> define
> the ***observation*** threshold.
>
> However, you set the forecast threshold very differently. The
forecast
> probability is a bunch of numbers between 0 and 1 (or 0 and 100).
The
> thresholds are used to partition that probability space, such as:
> cat_thresh = [ >=0.00, >=0.25, >=0.50, >=0.75, >=1.00 ];
>
> They must start with 0, end with 1, and all be the same probability
type.
> Each probability value falls into exactly 1 of the 4 probability
bins
> listed
> above. MET uses these settings to defined a 4 x 2 contingency table
(or N
> x
> 2, more generally). And all the probabilistic statistics are
derived from
> this table.
>
> You can certainly use more bins than what's defined above. You
might
> define
> thresholds every 0.10 or 0.05. Just take a look at your forecast
> probabilities to get a good idea of the range of probability values
being
> used.
>
> I'm guessing you may have more questions. If you'd like, you could
send
> me a
> sample forecast, observation, and config file you're passing to
> Grid-Stat. I
> could take a close look, run it here, and offer some suggestions.
>
> Thanks,
> John
>
>
> On Fri, Feb 26, 2016 at 2:14 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
> >
> > John, I changed the ob variable category to > .01, >.10 , etc..
The
> > precip values are in inches in my input netCDF file. Since I am
> > dealing with ensemble probabilities, should I have the fcst
categories
> > the same as precip or let it default to > 0? Now, when I do an
ncdump
> > of the MET produced data file (ncdump -h
> > grid_stat_1_APCP_6_240000L_20160213_000000V_pairs.nc), I see that
the
> > precip data units are in Kg/m^2, however looking at the data in
the
> netCDF
> > file, the units are in inches. Will this cause any problems?
The
> listing
> > is below:
> >
> > dimensions:
> > lat = 750 ;
> > lon = 750 ;
> > variables:
> > float lat(lat, lon) ;
> > lat:long_name = "latitude" ;
> > lat:units = "degrees_north" ;
> > lat:standard_name = "latitude" ;
> > float lon(lat, lon) ;
> > lon:long_name = "longitude" ;
> > lon:units = "degrees_east" ;
> > lon:standard_name = "longitude" ;
> > float FCST_PQP1_A06_FULL(lat, lon) ;
> > FCST_PQP1_A06_FULL:name = "PQP1" ;
> > FCST_PQP1_A06_FULL:long_name = "PQP1 at A06" ;
> > FCST_PQP1_A06_FULL:level = "A06" ;
> > FCST_PQP1_A06_FULL:units = "%" ;
> > FCST_PQP1_A06_FULL:init_time = "20160212_000000" ;
> > FCST_PQP1_A06_FULL:init_time_ut = "1455235200" ;
> > FCST_PQP1_A06_FULL:valid_time = "20160213_000000"
;
> > FCST_PQP1_A06_FULL:valid_time_ut = "1455321600" ;
> > FCST_PQP1_A06_FULL:accum_time = "060000" ;
> > FCST_PQP1_A06_FULL:accum_time_sec = 21600 ;
> > FCST_PQP1_A06_FULL:_FillValue = -9999.f ;
> > FCST_PQP1_A06_FULL:masking_region = "FULL" ;
> > FCST_PQP1_A06_FULL:smoothing_method = "NEAREST" ;
> > FCST_PQP1_A06_FULL:smoothing_neighborhood = 1 ;
> > float OBS_APCP_06_A6_FULL(lat, lon) ;
> > OBS_APCP_06_A6_FULL:name = "APCP_06" ;
> > OBS_APCP_06_A6_FULL:long_name = "APCP_06 at A6" ;
> > OBS_APCP_06_A6_FULL:level = "A6" ;
> > OBS_APCP_06_A6_FULL:units = "kg/m^2" ;
> > OBS_APCP_06_A6_FULL:init_time = "20160213_000000"
;
> > OBS_APCP_06_A6_FULL:init_time_ut = "1455321600" ;
> > OBS_APCP_06_A6_FULL:valid_time = "20160213_000000"
;
> > OBS_APCP_06_A6_FULL:valid_time_ut = "1455321600" ;
> > OBS_APCP_06_A6_FULL:accum_time = "060000" ;
> > OBS_APCP_06_A6_FULL:accum_time_sec = 21600 ;
> > OBS_APCP_06_A6_FULL:_FillValue = -9999.f ;
> > OBS_APCP_06_A6_FULL:masking_region = "FULL" ;
> > OBS_APCP_06_A6_FULL:smoothing_method = "NEAREST" ;
> > OBS_APCP_06_A6_FULL:smoothing_neighborhood = 1 ;
> > float DIFF_PQP1_A06_APCP_06_A6_FULL(lat, lon) ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:name = "Forecast
PQP1
> > minus Observed APCP_06" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:long_name = "PQP1 at
A06
> > and
> > APCP_06 at A6" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:level = "A06 and A6"
;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:units = "% and
kg/m^2" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:_FillValue = -9999.f
;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:init_time =
> > "20160212_000000" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:init_time_ut =
> "1455235200" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time =
> > "20160213_000000" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time_ut =
> "1455321600"
> > ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time =
"060000" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time_sec =
21600 ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:masking_region =
"FULL" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_method =
> "NEAREST"
> > ;
> >
DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_neighborhood =
> > 1 ;
> >
> > // global attributes:
> > :FileOrigins = "File
> >
/h/data/global/WXQC/data/fss/metoutput_fss/none/2016021200/grid_stat_1
> > _APCP_6_240000L_20160213_000000V_pairs.nc
> > generated 20160225_230147 UTC on host dev8dlsn1 by the MET
grid_stat
> tool" ;
> > :MET_version = "V5.1" ;
> > :MET_tool = "grid_stat" ;
> > :model = "WRF" ;
> > :obtype = "ANALYS" ;
> > :Difference = "Forecast Value - Observation Value"
;
> > :Projection = "Polar Stereographic" ;
> > :hemisphere = "N" ;
> > :scale_lat = "60.000000 degrees_north" ;
> > :lat_pin = "6.590000" ;
> > :lon_pin = "35.000000" ;
> > :x_pin = "0.000000" ;
> > :y_pin = "0.000000" ;
> > :lon_orient = "80.000000" ;
> > :d_km = "20.000000 km" ;
> > :r_km = "6371.200000 km" ;
> > :nx = "750" ;
> > :ny = "750" ;
> > }
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, February 25, 2016 3:11 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score
> > Question
> >
> > Bob,
> >
> > I see you have some questions about the fractions skill score.
And I
> > read that you're varying the precipitation threshold in the
> > configuration file for each run.
> >
> > I have a couple of suggestions...
> >
> > First, there's no need to run Grid-Stat multiple times with
different
> > precipitation thresholds. Instead, you can list multiple
thresholds,
> > like
> > this:
> > cat_thresh = [ >0, >0.254, >2.54, >25.4 ];
> >
> > I'm assuming the precip is in mm, and I'm using 25.4 for 1 inch.
You
> > should see a NBRCNT output line for each combination of precip
> > threshold and neighborhood size. So that means you wouldn't need
to
> > include the threshold in the output prefix.
> >
> > Second, I find using environment variables in the MET config files
to
> > very convenient. For example, the script which calls Grid-Stat
could
> > set an environment variable named "FCST_NAME" and you could refer
to
> > that in the config file, like this:
> > name = "${FCST_NAME}";
> > ...
> > And later on use it in the output prefix as well if you'd like:
> > output_prefix = "${FCST_NAME}";
> >
> > Just wanted to mention it.
> >
> > I just read your next email... and yes, I'd suggest defining the
> > thresholds once and then referring to them in both the fcst and
obs
> > sections, like
> > this:
> > precip_thresh = [ >0, >0.254, >2.54, >25.4 ];
> >
> > And in the fcst and obs sections, set:
> > cat_thresh = precip_thresh;
> >
> > Lastly, you mention that you're seeing the same value for
Observation
> > Rate. Can you tell me specifically the name of the column and the
> > line type, you're looking at? And better yet, send me some sample
> > output files where that value is the same. That'd help me to
better
> > understand what's going on.
> >
> > Thanks,
> > John
> >
> > On Thu, Feb 25, 2016 at 1:41 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Feb 25 13:41:02 2016: Request 75271 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: Ensemble Fractions Skill Score Question
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271
> > > >
> > >
> > >
> > > John, I have been running MET to generate fractions skill scores
for
> > > different precipitation thresholds. We are verifying 4km
ensemble
> > > forecasts using Stage IV for ground truth. What has me baffled
is
> that
> > I
> > > am getting the same Observation Rate not matter if I am
verifying
> > > 6hr accumulations of precip > .01" or precip >1.00". I would
expect
> > > the observation rate to be greater for precip > .01" than for
precip
> > > > 1.0" (a much more rare event) but it stays the same for all
> > > thresholds. The model forecast frequencies to decrease as the
> > > precip threshold increases as I would export.
> > >
> > > Below is a the configuration file I use for gridstat.
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > > //
> > > // Grid-Stat configuration file.
> > > //
> > > // For additional information, see the MET_BASE/config/README
file.
> > > //
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > //
> > > // Output model name to be written
> > > //
> > > model = "WRF";
> > >
> > > obtype = "ANALYS";
> > >
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > //
> > > // Verification grid
> > > //
> > > regrid = {
> > > to_grid = FCST;
> > > vld_thresh = 0.5;
> > > method = BUDGET;
> > > width = 2;
> > > }
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > cat_thresh = [ NA ];
> > > cnt_thresh = [ NA ];
> > > cnt_logic = UNION;
> > > wind_thresh = [ NA ];
> > > wind_logic = UNION;
> > > prob=[ "True" ];
> > > //
> > > // Forecast and observation fields to be verified // fcst = {
> > > wind_thresh = [ NA ];
> > >
> > > field = [
> > > {
> > > name = "QP100";
> > > level = [ "A06" ];
> > > GRIB1_ptv = 3;
> > > cat_thresh = [ > 0 ];
> > > }
> > > ];
> > >
> > > }
> > > obs = {
> > > wind_thresh = [ NA ];
> > >
> > > field = [
> > > {
> > > name = "APCP_06";
> > > level = [ "(*,*)" ];
> > > cat_thresh = [ > 0 ];
> > > }
> > > ];
> > > }
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > //
> > > // Climatology mean data
> > > //
> > > climo_mean = {
> > >
> > > file_name = [];
> > > field = [];
> > >
> > > regrid = {
> > > vld_thresh = 0.5;
> > > method = NEAREST;
> > > width = 1;
> > > }
> > >
> > > time_interp_method = DW_MEAN;
> > > match_day = FALSE;
> > > time_step = 21600;
> > > }
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > //
> > > // Verification masking regions
> > > //
> > > mask = {
> > > grid = [ "FULL" ];
> > > poly = [];
> > > }
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > //
> > > // Confidence interval settings
> > > //
> > > ci_alpha = [ 0.05 ];
> > >
> > > boot = {
> > > interval = PCTILE;
> > > rep_prop = 1.0;
> > > n_rep = 0;
> > > rng = "mt19937";
> > > seed = "";
> > > }
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > //
> > > // Data smoothing methods
> > > //
> > > interp = {
> > > field = BOTH;
> > > vld_thresh = 1.0;
> > >
> > > type = [
> > > {
> > > method = NEAREST;
> > > width = 1;
> > > }
> > > ];
> > > }
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > //
> > > // Neighborhood methods
> > > //
> > > nbrhd = {
> > > vld_thresh = 1.0;
> > > width =
> > >
[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
> > > cov_thresh = [ >=0.5 ];
> > > }
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > //
> > > // Statistical output types
> > > //
> > > output_flag = {
> > > fho = NONE;
> > > ctc = NONE;
> > > cts = NONE;
> > > mctc = NONE;
> > > mcts = NONE;
> > > cnt = NONE;
> > > sl1l2 = NONE;
> > > sal1l2 = NONE;
> > > vl1l2 = NONE;
> > > val1l2 = NONE;
> > > pct = NONE;
> > > pstd = NONE;
> > > pjc = NONE;
> > > prc = NONE;
> > > nbrctc = NONE;
> > > nbrcts = NONE;
> > > nbrcnt = STAT;
> > > }
> > >
> > > //
> > > // NetCDF matched pairs output file
> > > //
> > > nc_pairs_flag = {
> > > latlon = TRUE;
> > > raw = TRUE;
> > > diff = TRUE;
> > > climo = TRUE;
> > > }
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > rank_corr_flag = FALSE;
> > > tmp_dir = "/tmp";
> > > output_prefix = "100_APCP_6";
> > > version = "V5.1";
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > >
> > > In this configuration file I change forecast field name and
output
> > > prefix as vary the precipitation thresholds. I also vary the
> > neighborhood size.
> > > Could I have something not set right in the config file?
> > >
> > > Thanks
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: RE: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score Question
From: robert.craig.2 at us.af.mil
Time: Mon Mar 07 12:37:17 2016
Thanks John, that will help. Current management is hot on
neighborhood methods to verify ensemble forecasts (probably because
the MET Office is doing it), hence my interest. I starting the
process to convert our legacy verification code for ensembles to MET
(it predated MET), so I will be using the information you provided and
will most likely have future questions.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, March 04, 2016 4:59 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score Question
Hello Bob,
Sorry for the delay. I just talked to Tara and Tressa about this and
have some feedback for you. Sorry for any confusion my previous email
may have caused... it wasn't clear to me what you were trying to do.
First, you're correct that you can use Grid-Stat to compare a field of
probability values to precip amounts and compute fractions skill
score. We haven't done that within the DTC, and Tressa isn't aware of
any published examples, but there's no reason you can't do that. We
have used MODE to compare this type of data, typically looking at
probability objects > 10%, 25%, and 50% or so. That's somewhat
similar to what you're doing.
I would say, if you have not yet applied the probabilistic methods
available in Grid-Stat, I'd suggest trying it. They implement
statistics that are widely used and applied for probabilistic
verification. To that end, I attached a sample Grid-Stat config file
you could use for probabilistic verification (GridStatConfig_PROB).
By looking at the .stat files you sent, I see that you're keeping the
forecast threshold fixed as >0 and then varying the observation
threshold based on the probability field being evaluated (e.g. for
QP100, OBS_THRESH
>=1.0). That sounds good to me.
However, >0 is a very low probability threshold. You might also
consider applying some higher probability thresholds. For very low
probabilities, you'd hope that there isn't anything present in the
observations most of the time. So try using >0, >10, >25, >50 or
something like that. This should be very easy to do. In the "fcst"
section set:
cat_thresh = [ >0, >10, >25, >50 ];
And in the "obs" section set:
cat_thresh = [ >=1.0, >=1.0, >=1.0, >=1.0 ];
You just repeat the observation threshold once for each forecast
threshold to make them the same length.
Hopefully that helps. Just let us know what other questions come up.
Thanks,
John
On Wed, Mar 2, 2016 at 12:50 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
>
> Forwarded to you after I removed the NETcdf precip file. I will
place
> that file on the FTP server.
>
> Bob
>
> -----Original Message-----
> From: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Sent: Wednesday, March 02, 2016 1:48 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Subject: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
> Question
>
>
>
> -----Original Message-----
> From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Sent: Wednesday, March 02, 2016 1:45 PM
> To: met_help at ucar.edu
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: RE: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
> Question
>
> Hi, John,
>
> I am not setting the fcst cat_threshold - I am letting it default
to
> > 0. I
> am setting the obs threshold to thresholds you listed. I have
attached my
> template I use for the case of precip > 1.0". I am reading a NETCDF
> file where the data appears to be in inches (looking at a dump of
the file).
> Somebody else built the file for me but probably didn't set the
units
> correctly in the header. Would MET care if the units aren't
correct?
>
> By only setting pct,pstd,pjc,prc, aren't I eliminating neighborhood
> methods which is the whole reason I am running this to generate a
> fractions skill score. Can NBRCNT not be used for probability data?
> It seems to be working.
> I have attached the output file for each threshold.
>
> I can send you the a model file where you can run the data but I
will
> wait until you answer the questions above first.
>
> Thanks
>
> Bob
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, March 02, 2016 12:38 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score
> Question
>
> Bob,
>
> Sorry I'm so slow in responding to this email. I was out of the
> office a couple of days in there.
>
> Let me see if I understand what you're doing. You are running
> Grid-Stat to compare ensemble probabilities to observed
precipitation.
> So you're doing probabilistic verification. And you're using
StageIV
> as the observation in GRIB format.
>
> The units for StageIV data in GRIB really are millimeters (kg/m^2).
>
> Here are some practical considerations for doing probabilistic
> verification in MET.
>
> (1) The only line types you want are PCT, PSTD, PJC, and PRC. Set
all
> the other ones to "NONE".
> (2) Probabilities are defined as a probability of some event. For
> example, probability of precip is probability of precip > 0. Use
that
> event to define the ***observation*** threshold.
>
> However, you set the forecast threshold very differently. The
> forecast probability is a bunch of numbers between 0 and 1 (or 0 and
> 100). The thresholds are used to partition that probability space,
such as:
> cat_thresh = [ >=0.00, >=0.25, >=0.50, >=0.75, >=1.00 ];
>
> They must start with 0, end with 1, and all be the same probability
type.
> Each probability value falls into exactly 1 of the 4 probability
bins
> listed above. MET uses these settings to defined a 4 x 2
contingency
> table (or N x 2, more generally). And all the probabilistic
> statistics are derived from this table.
>
> You can certainly use more bins than what's defined above. You
might
> define thresholds every 0.10 or 0.05. Just take a look at your
> forecast probabilities to get a good idea of the range of
probability
> values being used.
>
> I'm guessing you may have more questions. If you'd like, you could
> send me a sample forecast, observation, and config file you're
passing
> to Grid-Stat. I could take a close look, run it here, and offer
some
> suggestions.
>
> Thanks,
> John
>
>
> On Fri, Feb 26, 2016 at 2:14 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
> >
> > John, I changed the ob variable category to > .01, >.10 , etc..
The
> > precip values are in inches in my input netCDF file. Since I am
> > dealing with ensemble probabilities, should I have the fcst
> > categories the same as precip or let it default to > 0? Now, when
I
> > do an ncdump of the MET produced data file (ncdump -h
> > grid_stat_1_APCP_6_240000L_20160213_000000V_pairs.nc), I see that
> > the precip data units are in Kg/m^2, however looking at the data
in
> > the
> netCDF
> > file, the units are in inches. Will this cause any problems?
The
> listing
> > is below:
> >
> > dimensions:
> > lat = 750 ;
> > lon = 750 ;
> > variables:
> > float lat(lat, lon) ;
> > lat:long_name = "latitude" ;
> > lat:units = "degrees_north" ;
> > lat:standard_name = "latitude" ;
> > float lon(lat, lon) ;
> > lon:long_name = "longitude" ;
> > lon:units = "degrees_east" ;
> > lon:standard_name = "longitude" ;
> > float FCST_PQP1_A06_FULL(lat, lon) ;
> > FCST_PQP1_A06_FULL:name = "PQP1" ;
> > FCST_PQP1_A06_FULL:long_name = "PQP1 at A06" ;
> > FCST_PQP1_A06_FULL:level = "A06" ;
> > FCST_PQP1_A06_FULL:units = "%" ;
> > FCST_PQP1_A06_FULL:init_time = "20160212_000000" ;
> > FCST_PQP1_A06_FULL:init_time_ut = "1455235200" ;
> > FCST_PQP1_A06_FULL:valid_time = "20160213_000000"
;
> > FCST_PQP1_A06_FULL:valid_time_ut = "1455321600" ;
> > FCST_PQP1_A06_FULL:accum_time = "060000" ;
> > FCST_PQP1_A06_FULL:accum_time_sec = 21600 ;
> > FCST_PQP1_A06_FULL:_FillValue = -9999.f ;
> > FCST_PQP1_A06_FULL:masking_region = "FULL" ;
> > FCST_PQP1_A06_FULL:smoothing_method = "NEAREST" ;
> > FCST_PQP1_A06_FULL:smoothing_neighborhood = 1 ;
> > float OBS_APCP_06_A6_FULL(lat, lon) ;
> > OBS_APCP_06_A6_FULL:name = "APCP_06" ;
> > OBS_APCP_06_A6_FULL:long_name = "APCP_06 at A6" ;
> > OBS_APCP_06_A6_FULL:level = "A6" ;
> > OBS_APCP_06_A6_FULL:units = "kg/m^2" ;
> > OBS_APCP_06_A6_FULL:init_time = "20160213_000000"
;
> > OBS_APCP_06_A6_FULL:init_time_ut = "1455321600" ;
> > OBS_APCP_06_A6_FULL:valid_time = "20160213_000000"
;
> > OBS_APCP_06_A6_FULL:valid_time_ut = "1455321600" ;
> > OBS_APCP_06_A6_FULL:accum_time = "060000" ;
> > OBS_APCP_06_A6_FULL:accum_time_sec = 21600 ;
> > OBS_APCP_06_A6_FULL:_FillValue = -9999.f ;
> > OBS_APCP_06_A6_FULL:masking_region = "FULL" ;
> > OBS_APCP_06_A6_FULL:smoothing_method = "NEAREST" ;
> > OBS_APCP_06_A6_FULL:smoothing_neighborhood = 1 ;
> > float DIFF_PQP1_A06_APCP_06_A6_FULL(lat, lon) ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:name = "Forecast
PQP1
> > minus Observed APCP_06" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:long_name = "PQP1 at
> > A06 and
> > APCP_06 at A6" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:level = "A06 and A6"
;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:units = "% and
kg/m^2" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:_FillValue = -9999.f
;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:init_time =
> > "20160212_000000" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:init_time_ut =
> "1455235200" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time =
> > "20160213_000000" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time_ut =
> "1455321600"
> > ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time =
"060000" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time_sec =
21600 ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:masking_region =
"FULL" ;
> > DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_method =
> "NEAREST"
> > ;
> >
DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_neighborhood
> > =
> > 1 ;
> >
> > // global attributes:
> > :FileOrigins = "File
> >
/h/data/global/WXQC/data/fss/metoutput_fss/none/2016021200/grid_stat
> > _1 _APCP_6_240000L_20160213_000000V_pairs.nc
> > generated 20160225_230147 UTC on host dev8dlsn1 by the MET
grid_stat
> tool" ;
> > :MET_version = "V5.1" ;
> > :MET_tool = "grid_stat" ;
> > :model = "WRF" ;
> > :obtype = "ANALYS" ;
> > :Difference = "Forecast Value - Observation Value"
;
> > :Projection = "Polar Stereographic" ;
> > :hemisphere = "N" ;
> > :scale_lat = "60.000000 degrees_north" ;
> > :lat_pin = "6.590000" ;
> > :lon_pin = "35.000000" ;
> > :x_pin = "0.000000" ;
> > :y_pin = "0.000000" ;
> > :lon_orient = "80.000000" ;
> > :d_km = "20.000000 km" ;
> > :r_km = "6371.200000 km" ;
> > :nx = "750" ;
> > :ny = "750" ;
> > }
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, February 25, 2016 3:11 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score
> > Question
> >
> > Bob,
> >
> > I see you have some questions about the fractions skill score.
And
> > I read that you're varying the precipitation threshold in the
> > configuration file for each run.
> >
> > I have a couple of suggestions...
> >
> > First, there's no need to run Grid-Stat multiple times with
> > different precipitation thresholds. Instead, you can list
multiple
> > thresholds, like
> > this:
> > cat_thresh = [ >0, >0.254, >2.54, >25.4 ];
> >
> > I'm assuming the precip is in mm, and I'm using 25.4 for 1 inch.
> > You should see a NBRCNT output line for each combination of precip
> > threshold and neighborhood size. So that means you wouldn't need
to
> > include the threshold in the output prefix.
> >
> > Second, I find using environment variables in the MET config files
> > to very convenient. For example, the script which calls Grid-Stat
> > could set an environment variable named "FCST_NAME" and you could
> > refer to that in the config file, like this:
> > name = "${FCST_NAME}";
> > ...
> > And later on use it in the output prefix as well if you'd like:
> > output_prefix = "${FCST_NAME}";
> >
> > Just wanted to mention it.
> >
> > I just read your next email... and yes, I'd suggest defining the
> > thresholds once and then referring to them in both the fcst and
obs
> > sections, like
> > this:
> > precip_thresh = [ >0, >0.254, >2.54, >25.4 ];
> >
> > And in the fcst and obs sections, set:
> > cat_thresh = precip_thresh;
> >
> > Lastly, you mention that you're seeing the same value for
> > Observation Rate. Can you tell me specifically the name of the
> > column and the line type, you're looking at? And better yet, send
> > me some sample output files where that value is the same. That'd
> > help me to better understand what's going on.
> >
> > Thanks,
> > John
> >
> > On Thu, Feb 25, 2016 at 1:41 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Feb 25 13:41:02 2016: Request 75271 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: Ensemble Fractions Skill Score Question
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271
> > > >
> > >
> > >
> > > John, I have been running MET to generate fractions skill scores
for
> > > different precipitation thresholds. We are verifying 4km
ensemble
> > > forecasts using Stage IV for ground truth. What has me baffled
is
> that
> > I
> > > am getting the same Observation Rate not matter if I am
verifying
> > > 6hr accumulations of precip > .01" or precip >1.00". I would
> > > expect the observation rate to be greater for precip > .01" than
> > > for precip
> > > > 1.0" (a much more rare event) but it stays the same for all
> > > thresholds. The model forecast frequencies to decrease as the
> > > precip threshold increases as I would export.
> > >
> > > Below is a the configuration file I use for gridstat.
> > >
> > >
> > >
//////////////////////////////////////////////////////////////////
> > > //
> > > //
> > > //////////
> > > //
> > > // Grid-Stat configuration file.
> > > //
> > > // For additional information, see the MET_BASE/config/README
file.
> > > //
> > >
> > >
//////////////////////////////////////////////////////////////////
> > > //
> > > //
> > > //////////
> > >
> > > //
> > > // Output model name to be written // model = "WRF";
> > >
> > > obtype = "ANALYS";
> > >
> > >
> > >
> > >
//////////////////////////////////////////////////////////////////
> > > //
> > > //
> > > //////////
> > >
> > > //
> > > // Verification grid
> > > //
> > > regrid = {
> > > to_grid = FCST;
> > > vld_thresh = 0.5;
> > > method = BUDGET;
> > > width = 2;
> > > }
> > >
> > >
> > >
//////////////////////////////////////////////////////////////////
> > > //
> > > //
> > > //////////
> > >
> > > cat_thresh = [ NA ];
> > > cnt_thresh = [ NA ];
> > > cnt_logic = UNION;
> > > wind_thresh = [ NA ];
> > > wind_logic = UNION;
> > > prob=[ "True" ];
> > > //
> > > // Forecast and observation fields to be verified // fcst = {
> > > wind_thresh = [ NA ];
> > >
> > > field = [
> > > {
> > > name = "QP100";
> > > level = [ "A06" ];
> > > GRIB1_ptv = 3;
> > > cat_thresh = [ > 0 ];
> > > }
> > > ];
> > >
> > > }
> > > obs = {
> > > wind_thresh = [ NA ];
> > >
> > > field = [
> > > {
> > > name = "APCP_06";
> > > level = [ "(*,*)" ];
> > > cat_thresh = [ > 0 ];
> > > }
> > > ];
> > > }
> > >
> > >
> > >
//////////////////////////////////////////////////////////////////
> > > //
> > > //
> > > //////////
> > >
> > > //
> > > // Climatology mean data
> > > //
> > > climo_mean = {
> > >
> > > file_name = [];
> > > field = [];
> > >
> > > regrid = {
> > > vld_thresh = 0.5;
> > > method = NEAREST;
> > > width = 1;
> > > }
> > >
> > > time_interp_method = DW_MEAN;
> > > match_day = FALSE;
> > > time_step = 21600;
> > > }
> > >
> > >
> > >
//////////////////////////////////////////////////////////////////
> > > //
> > > //
> > > //////////
> > >
> > > //
> > > // Verification masking regions
> > > //
> > > mask = {
> > > grid = [ "FULL" ];
> > > poly = [];
> > > }
> > >
> > >
> > >
//////////////////////////////////////////////////////////////////
> > > //
> > > //
> > > //////////
> > >
> > > //
> > > // Confidence interval settings
> > > //
> > > ci_alpha = [ 0.05 ];
> > >
> > > boot = {
> > > interval = PCTILE;
> > > rep_prop = 1.0;
> > > n_rep = 0;
> > > rng = "mt19937";
> > > seed = "";
> > > }
> > >
> > >
> > >
//////////////////////////////////////////////////////////////////
> > > //
> > > //
> > > //////////
> > >
> > > //
> > > // Data smoothing methods
> > > //
> > > interp = {
> > > field = BOTH;
> > > vld_thresh = 1.0;
> > >
> > > type = [
> > > {
> > > method = NEAREST;
> > > width = 1;
> > > }
> > > ];
> > > }
> > >
> > >
> > >
//////////////////////////////////////////////////////////////////
> > > //
> > > //
> > > //////////
> > >
> > > //
> > > // Neighborhood methods
> > > //
> > > nbrhd = {
> > > vld_thresh = 1.0;
> > > width =
> > >
[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
> > > cov_thresh = [ >=0.5 ];
> > > }
> > >
> > >
> > >
//////////////////////////////////////////////////////////////////
> > > //
> > > //
> > > //////////
> > >
> > > //
> > > // Statistical output types
> > > //
> > > output_flag = {
> > > fho = NONE;
> > > ctc = NONE;
> > > cts = NONE;
> > > mctc = NONE;
> > > mcts = NONE;
> > > cnt = NONE;
> > > sl1l2 = NONE;
> > > sal1l2 = NONE;
> > > vl1l2 = NONE;
> > > val1l2 = NONE;
> > > pct = NONE;
> > > pstd = NONE;
> > > pjc = NONE;
> > > prc = NONE;
> > > nbrctc = NONE;
> > > nbrcts = NONE;
> > > nbrcnt = STAT;
> > > }
> > >
> > > //
> > > // NetCDF matched pairs output file //
> > > nc_pairs_flag = {
> > > latlon = TRUE;
> > > raw = TRUE;
> > > diff = TRUE;
> > > climo = TRUE;
> > > }
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > > rank_corr_flag = FALSE;
> > > tmp_dir = "/tmp";
> > > output_prefix = "100_APCP_6";
> > > version = "V5.1";
> > >
> > >
> > >
////////////////////////////////////////////////////////////////////
> > > //
> > > //////////
> > >
> > >
> > > In this configuration file I change forecast field name and
output
> > > prefix as vary the precipitation thresholds. I also vary the
> > neighborhood size.
> > > Could I have something not set right in the config file?
> > >
> > > Thanks
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: Ensemble Fractions Skill Score Question
From: John Halley Gotway
Time: Mon Mar 07 13:12:38 2016
Bob,
Sounds good. Out of curiosity, what are you using to string together
the
MET tools for this? Shell scripts? PERL? Python?
I'm not sure what funding mechanism would be involved, but I'd really
like
for us in the DTC to be able to help you with this transition to using
MET. We could talk about it during the call on 3/14. I'm hopeful
that
you'll find that MET and METViewer will make your job easier. And you
can
provide feedback as to what additional capabilities you'd like to see
in
MET.
Thanks,
John
On Mon, Mar 7, 2016 at 12:37 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
>
> Thanks John, that will help. Current management is hot on
neighborhood
> methods to verify ensemble forecasts (probably because the MET
Office is
> doing it), hence my interest. I starting the process to convert our
legacy
> verification code for ensembles to MET (it predated MET), so I will
be
> using the information you provided and will most likely have future
> questions.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, March 04, 2016 4:59 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score
> Question
>
> Hello Bob,
>
> Sorry for the delay. I just talked to Tara and Tressa about this
and have
> some feedback for you. Sorry for any confusion my previous email
may have
> caused... it wasn't clear to me what you were trying to do.
>
> First, you're correct that you can use Grid-Stat to compare a field
of
> probability values to precip amounts and compute fractions skill
score. We
> haven't done that within the DTC, and Tressa isn't aware of any
published
> examples, but there's no reason you can't do that. We have used
MODE to
> compare this type of data, typically looking at probability objects
> 10%,
> 25%, and 50% or so. That's somewhat similar to what you're doing.
>
> I would say, if you have not yet applied the probabilistic methods
> available in Grid-Stat, I'd suggest trying it. They implement
statistics
> that are widely used and applied for probabilistic verification. To
that
> end, I attached a sample Grid-Stat config file you could use for
> probabilistic verification (GridStatConfig_PROB).
>
> By looking at the .stat files you sent, I see that you're keeping
the
> forecast threshold fixed as >0 and then varying the observation
threshold
> based on the probability field being evaluated (e.g. for QP100,
OBS_THRESH
> >=1.0). That sounds good to me.
>
> However, >0 is a very low probability threshold. You might also
consider
> applying some higher probability thresholds. For very low
probabilities,
> you'd hope that there isn't anything present in the observations
most of
> the time. So try using >0, >10, >25, >50 or something like that.
This
> should be very easy to do. In the "fcst" section set:
> cat_thresh = [ >0, >10, >25, >50 ];
> And in the "obs" section set:
> cat_thresh = [ >=1.0, >=1.0, >=1.0, >=1.0 ];
>
> You just repeat the observation threshold once for each forecast
threshold
> to make them the same length.
>
> Hopefully that helps. Just let us know what other questions come
up.
>
> Thanks,
> John
>
>
> On Wed, Mar 2, 2016 at 12:50 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
> >
> > Forwarded to you after I removed the NETcdf precip file. I will
place
> > that file on the FTP server.
> >
> > Bob
> >
> > -----Original Message-----
> > From: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Sent: Wednesday, March 02, 2016 1:48 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Subject: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score
> > Question
> >
> >
> >
> > -----Original Message-----
> > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Sent: Wednesday, March 02, 2016 1:45 PM
> > To: met_help at ucar.edu
> > Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: RE: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score
> > Question
> >
> > Hi, John,
> >
> > I am not setting the fcst cat_threshold - I am letting it default
to
> > > 0. I
> > am setting the obs threshold to thresholds you listed. I have
attached
> my
> > template I use for the case of precip > 1.0". I am reading a
NETCDF
> > file where the data appears to be in inches (looking at a dump of
the
> file).
> > Somebody else built the file for me but probably didn't set the
units
> > correctly in the header. Would MET care if the units aren't
correct?
> >
> > By only setting pct,pstd,pjc,prc, aren't I eliminating
neighborhood
> > methods which is the whole reason I am running this to generate a
> > fractions skill score. Can NBRCNT not be used for probability
data?
> > It seems to be working.
> > I have attached the output file for each threshold.
> >
> > I can send you the a model file where you can run the data but I
will
> > wait until you answer the questions above first.
> >
> > Thanks
> >
> > Bob
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, March 02, 2016 12:38 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score
> > Question
> >
> > Bob,
> >
> > Sorry I'm so slow in responding to this email. I was out of the
> > office a couple of days in there.
> >
> > Let me see if I understand what you're doing. You are running
> > Grid-Stat to compare ensemble probabilities to observed
precipitation.
> > So you're doing probabilistic verification. And you're using
StageIV
> > as the observation in GRIB format.
> >
> > The units for StageIV data in GRIB really are millimeters
(kg/m^2).
> >
> > Here are some practical considerations for doing probabilistic
> > verification in MET.
> >
> > (1) The only line types you want are PCT, PSTD, PJC, and PRC. Set
all
> > the other ones to "NONE".
> > (2) Probabilities are defined as a probability of some event. For
> > example, probability of precip is probability of precip > 0. Use
that
> > event to define the ***observation*** threshold.
> >
> > However, you set the forecast threshold very differently. The
> > forecast probability is a bunch of numbers between 0 and 1 (or 0
and
> > 100). The thresholds are used to partition that probability
space, such
> as:
> > cat_thresh = [ >=0.00, >=0.25, >=0.50, >=0.75, >=1.00 ];
> >
> > They must start with 0, end with 1, and all be the same
probability type.
> > Each probability value falls into exactly 1 of the 4 probability
bins
> > listed above. MET uses these settings to defined a 4 x 2
contingency
> > table (or N x 2, more generally). And all the probabilistic
> > statistics are derived from this table.
> >
> > You can certainly use more bins than what's defined above. You
might
> > define thresholds every 0.10 or 0.05. Just take a look at your
> > forecast probabilities to get a good idea of the range of
probability
> > values being used.
> >
> > I'm guessing you may have more questions. If you'd like, you
could
> > send me a sample forecast, observation, and config file you're
passing
> > to Grid-Stat. I could take a close look, run it here, and offer
some
> > suggestions.
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Feb 26, 2016 at 2:14 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
> > >
> > > John, I changed the ob variable category to > .01, >.10 , etc..
The
> > > precip values are in inches in my input netCDF file. Since I am
> > > dealing with ensemble probabilities, should I have the fcst
> > > categories the same as precip or let it default to > 0? Now,
when I
> > > do an ncdump of the MET produced data file (ncdump -h
> > > grid_stat_1_APCP_6_240000L_20160213_000000V_pairs.nc), I see
that
> > > the precip data units are in Kg/m^2, however looking at the data
in
> > > the
> > netCDF
> > > file, the units are in inches. Will this cause any problems?
The
> > listing
> > > is below:
> > >
> > > dimensions:
> > > lat = 750 ;
> > > lon = 750 ;
> > > variables:
> > > float lat(lat, lon) ;
> > > lat:long_name = "latitude" ;
> > > lat:units = "degrees_north" ;
> > > lat:standard_name = "latitude" ;
> > > float lon(lat, lon) ;
> > > lon:long_name = "longitude" ;
> > > lon:units = "degrees_east" ;
> > > lon:standard_name = "longitude" ;
> > > float FCST_PQP1_A06_FULL(lat, lon) ;
> > > FCST_PQP1_A06_FULL:name = "PQP1" ;
> > > FCST_PQP1_A06_FULL:long_name = "PQP1 at A06" ;
> > > FCST_PQP1_A06_FULL:level = "A06" ;
> > > FCST_PQP1_A06_FULL:units = "%" ;
> > > FCST_PQP1_A06_FULL:init_time = "20160212_000000"
;
> > > FCST_PQP1_A06_FULL:init_time_ut = "1455235200" ;
> > > FCST_PQP1_A06_FULL:valid_time =
"20160213_000000" ;
> > > FCST_PQP1_A06_FULL:valid_time_ut = "1455321600"
;
> > > FCST_PQP1_A06_FULL:accum_time = "060000" ;
> > > FCST_PQP1_A06_FULL:accum_time_sec = 21600 ;
> > > FCST_PQP1_A06_FULL:_FillValue = -9999.f ;
> > > FCST_PQP1_A06_FULL:masking_region = "FULL" ;
> > > FCST_PQP1_A06_FULL:smoothing_method = "NEAREST"
;
> > > FCST_PQP1_A06_FULL:smoothing_neighborhood = 1 ;
> > > float OBS_APCP_06_A6_FULL(lat, lon) ;
> > > OBS_APCP_06_A6_FULL:name = "APCP_06" ;
> > > OBS_APCP_06_A6_FULL:long_name = "APCP_06 at A6"
;
> > > OBS_APCP_06_A6_FULL:level = "A6" ;
> > > OBS_APCP_06_A6_FULL:units = "kg/m^2" ;
> > > OBS_APCP_06_A6_FULL:init_time =
"20160213_000000" ;
> > > OBS_APCP_06_A6_FULL:init_time_ut = "1455321600"
;
> > > OBS_APCP_06_A6_FULL:valid_time =
"20160213_000000" ;
> > > OBS_APCP_06_A6_FULL:valid_time_ut = "1455321600"
;
> > > OBS_APCP_06_A6_FULL:accum_time = "060000" ;
> > > OBS_APCP_06_A6_FULL:accum_time_sec = 21600 ;
> > > OBS_APCP_06_A6_FULL:_FillValue = -9999.f ;
> > > OBS_APCP_06_A6_FULL:masking_region = "FULL" ;
> > > OBS_APCP_06_A6_FULL:smoothing_method = "NEAREST"
;
> > > OBS_APCP_06_A6_FULL:smoothing_neighborhood = 1 ;
> > > float DIFF_PQP1_A06_APCP_06_A6_FULL(lat, lon) ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:name = "Forecast
PQP1
> > > minus Observed APCP_06" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:long_name = "PQP1
at
> > > A06 and
> > > APCP_06 at A6" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:level = "A06 and
A6" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:units = "% and
kg/m^2" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:_FillValue =
-9999.f ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:init_time =
> > > "20160212_000000" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:init_time_ut =
> > "1455235200" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time =
> > > "20160213_000000" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time_ut =
> > "1455321600"
> > > ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time =
"060000" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time_sec =
21600 ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:masking_region =
"FULL" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_method =
> > "NEAREST"
> > > ;
> > >
DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_neighborhood
> > > =
> > > 1 ;
> > >
> > > // global attributes:
> > > :FileOrigins = "File
> > >
/h/data/global/WXQC/data/fss/metoutput_fss/none/2016021200/grid_stat
> > > _1 _APCP_6_240000L_20160213_000000V_pairs.nc
> > > generated 20160225_230147 UTC on host dev8dlsn1 by the MET
grid_stat
> > tool" ;
> > > :MET_version = "V5.1" ;
> > > :MET_tool = "grid_stat" ;
> > > :model = "WRF" ;
> > > :obtype = "ANALYS" ;
> > > :Difference = "Forecast Value - Observation
Value" ;
> > > :Projection = "Polar Stereographic" ;
> > > :hemisphere = "N" ;
> > > :scale_lat = "60.000000 degrees_north" ;
> > > :lat_pin = "6.590000" ;
> > > :lon_pin = "35.000000" ;
> > > :x_pin = "0.000000" ;
> > > :y_pin = "0.000000" ;
> > > :lon_orient = "80.000000" ;
> > > :d_km = "20.000000 km" ;
> > > :r_km = "6371.200000 km" ;
> > > :nx = "750" ;
> > > :ny = "750" ;
> > > }
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, February 25, 2016 3:11 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score
> > > Question
> > >
> > > Bob,
> > >
> > > I see you have some questions about the fractions skill score.
And
> > > I read that you're varying the precipitation threshold in the
> > > configuration file for each run.
> > >
> > > I have a couple of suggestions...
> > >
> > > First, there's no need to run Grid-Stat multiple times with
> > > different precipitation thresholds. Instead, you can list
multiple
> > > thresholds, like
> > > this:
> > > cat_thresh = [ >0, >0.254, >2.54, >25.4 ];
> > >
> > > I'm assuming the precip is in mm, and I'm using 25.4 for 1 inch.
> > > You should see a NBRCNT output line for each combination of
precip
> > > threshold and neighborhood size. So that means you wouldn't
need to
> > > include the threshold in the output prefix.
> > >
> > > Second, I find using environment variables in the MET config
files
> > > to very convenient. For example, the script which calls Grid-
Stat
> > > could set an environment variable named "FCST_NAME" and you
could
> > > refer to that in the config file, like this:
> > > name = "${FCST_NAME}";
> > > ...
> > > And later on use it in the output prefix as well if you'd like:
> > > output_prefix = "${FCST_NAME}";
> > >
> > > Just wanted to mention it.
> > >
> > > I just read your next email... and yes, I'd suggest defining the
> > > thresholds once and then referring to them in both the fcst and
obs
> > > sections, like
> > > this:
> > > precip_thresh = [ >0, >0.254, >2.54, >25.4 ];
> > >
> > > And in the fcst and obs sections, set:
> > > cat_thresh = precip_thresh;
> > >
> > > Lastly, you mention that you're seeing the same value for
> > > Observation Rate. Can you tell me specifically the name of the
> > > column and the line type, you're looking at? And better yet,
send
> > > me some sample output files where that value is the same.
That'd
> > > help me to better understand what's going on.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Feb 25, 2016 at 1:41 PM, robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Feb 25 13:41:02 2016: Request 75271 was acted upon.
> > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > Queue: met_help
> > > > Subject: Ensemble Fractions Skill Score Question
> > > > Owner: Nobody
> > > > Requestors: robert.craig.2 at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271
> > > > >
> > > >
> > > >
> > > > John, I have been running MET to generate fractions skill
scores for
> > > > different precipitation thresholds. We are verifying 4km
ensemble
> > > > forecasts using Stage IV for ground truth. What has me
baffled is
> > that
> > > I
> > > > am getting the same Observation Rate not matter if I am
verifying
> > > > 6hr accumulations of precip > .01" or precip >1.00". I would
> > > > expect the observation rate to be greater for precip > .01"
than
> > > > for precip
> > > > > 1.0" (a much more rare event) but it stays the same for all
> > > > thresholds. The model forecast frequencies to decrease as the
> > > > precip threshold increases as I would export.
> > > >
> > > > Below is a the configuration file I use for gridstat.
> > > >
> > > >
> > > >
//////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //////////
> > > > //
> > > > // Grid-Stat configuration file.
> > > > //
> > > > // For additional information, see the MET_BASE/config/README
file.
> > > > //
> > > >
> > > >
//////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Output model name to be written // model = "WRF";
> > > >
> > > > obtype = "ANALYS";
> > > >
> > > >
> > > >
> > > >
//////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Verification grid
> > > > //
> > > > regrid = {
> > > > to_grid = FCST;
> > > > vld_thresh = 0.5;
> > > > method = BUDGET;
> > > > width = 2;
> > > > }
> > > >
> > > >
> > > >
//////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > cat_thresh = [ NA ];
> > > > cnt_thresh = [ NA ];
> > > > cnt_logic = UNION;
> > > > wind_thresh = [ NA ];
> > > > wind_logic = UNION;
> > > > prob=[ "True" ];
> > > > //
> > > > // Forecast and observation fields to be verified // fcst = {
> > > > wind_thresh = [ NA ];
> > > >
> > > > field = [
> > > > {
> > > > name = "QP100";
> > > > level = [ "A06" ];
> > > > GRIB1_ptv = 3;
> > > > cat_thresh = [ > 0 ];
> > > > }
> > > > ];
> > > >
> > > > }
> > > > obs = {
> > > > wind_thresh = [ NA ];
> > > >
> > > > field = [
> > > > {
> > > > name = "APCP_06";
> > > > level = [ "(*,*)" ];
> > > > cat_thresh = [ > 0 ];
> > > > }
> > > > ];
> > > > }
> > > >
> > > >
> > > >
//////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Climatology mean data
> > > > //
> > > > climo_mean = {
> > > >
> > > > file_name = [];
> > > > field = [];
> > > >
> > > > regrid = {
> > > > vld_thresh = 0.5;
> > > > method = NEAREST;
> > > > width = 1;
> > > > }
> > > >
> > > > time_interp_method = DW_MEAN;
> > > > match_day = FALSE;
> > > > time_step = 21600;
> > > > }
> > > >
> > > >
> > > >
//////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Verification masking regions
> > > > //
> > > > mask = {
> > > > grid = [ "FULL" ];
> > > > poly = [];
> > > > }
> > > >
> > > >
> > > >
//////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Confidence interval settings
> > > > //
> > > > ci_alpha = [ 0.05 ];
> > > >
> > > > boot = {
> > > > interval = PCTILE;
> > > > rep_prop = 1.0;
> > > > n_rep = 0;
> > > > rng = "mt19937";
> > > > seed = "";
> > > > }
> > > >
> > > >
> > > >
//////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Data smoothing methods
> > > > //
> > > > interp = {
> > > > field = BOTH;
> > > > vld_thresh = 1.0;
> > > >
> > > > type = [
> > > > {
> > > > method = NEAREST;
> > > > width = 1;
> > > > }
> > > > ];
> > > > }
> > > >
> > > >
> > > >
//////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Neighborhood methods
> > > > //
> > > > nbrhd = {
> > > > vld_thresh = 1.0;
> > > > width =
> > > >
[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
> > > > cov_thresh = [ >=0.5 ];
> > > > }
> > > >
> > > >
> > > >
//////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Statistical output types
> > > > //
> > > > output_flag = {
> > > > fho = NONE;
> > > > ctc = NONE;
> > > > cts = NONE;
> > > > mctc = NONE;
> > > > mcts = NONE;
> > > > cnt = NONE;
> > > > sl1l2 = NONE;
> > > > sal1l2 = NONE;
> > > > vl1l2 = NONE;
> > > > val1l2 = NONE;
> > > > pct = NONE;
> > > > pstd = NONE;
> > > > pjc = NONE;
> > > > prc = NONE;
> > > > nbrctc = NONE;
> > > > nbrcts = NONE;
> > > > nbrcnt = STAT;
> > > > }
> > > >
> > > > //
> > > > // NetCDF matched pairs output file //
> > > > nc_pairs_flag = {
> > > > latlon = TRUE;
> > > > raw = TRUE;
> > > > diff = TRUE;
> > > > climo = TRUE;
> > > > }
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////////
> > > > //
> > > > //////////
> > > >
> > > > rank_corr_flag = FALSE;
> > > > tmp_dir = "/tmp";
> > > > output_prefix = "100_APCP_6";
> > > > version = "V5.1";
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////////
> > > > //
> > > > //////////
> > > >
> > > >
> > > > In this configuration file I change forecast field name and
output
> > > > prefix as vary the precipitation thresholds. I also vary the
> > > neighborhood size.
> > > > Could I have something not set right in the config file?
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill Score Question
From: robert.craig.2 at us.af.mil
Time: Mon Mar 07 13:27:23 2016
We have been directed to write new code in Python, so that is what I
have been using. We are also trying to get Matt Sittel funded so he
could help with the transition of his global ensemble code to MET. In
the meantime, that will be my task. We will be generating brier
skill scores, spread skill, reliability diagrams, continuous ranked
probability score (CRPS) and CRPS skill score, and ensemble member
verification (ensemble mean). MET seems to be set up to handle most
of this. The one thing that might need work is CRPS skill score. I
obtained the set of NCEP climo data they were using as the reference
for this but could not duplicate their numbers. Look forward to the
call next week.
Bob
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, March 07, 2016 2:13 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
Subject: Re: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score Question
Bob,
Sounds good. Out of curiosity, what are you using to string together
the MET tools for this? Shell scripts? PERL? Python?
I'm not sure what funding mechanism would be involved, but I'd really
like for us in the DTC to be able to help you with this transition to
using MET. We could talk about it during the call on 3/14. I'm
hopeful that you'll find that MET and METViewer will make your job
easier. And you can provide feedback as to what additional
capabilities you'd like to see in MET.
Thanks,
John
On Mon, Mar 7, 2016 at 12:37 PM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
>
> Thanks John, that will help. Current management is hot on
> neighborhood methods to verify ensemble forecasts (probably because
> the MET Office is doing it), hence my interest. I starting the
> process to convert our legacy verification code for ensembles to MET
> (it predated MET), so I will be using the information you provided
and
> will most likely have future questions.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, March 04, 2016 4:59 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> Subject: Re: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
> Score Question
>
> Hello Bob,
>
> Sorry for the delay. I just talked to Tara and Tressa about this
and
> have some feedback for you. Sorry for any confusion my previous
email
> may have caused... it wasn't clear to me what you were trying to do.
>
> First, you're correct that you can use Grid-Stat to compare a field
of
> probability values to precip amounts and compute fractions skill
> score. We haven't done that within the DTC, and Tressa isn't aware
of
> any published examples, but there's no reason you can't do that. We
> have used MODE to compare this type of data, typically looking at
> probability objects > 10%, 25%, and 50% or so. That's somewhat
similar to what you're doing.
>
> I would say, if you have not yet applied the probabilistic methods
> available in Grid-Stat, I'd suggest trying it. They implement
> statistics that are widely used and applied for probabilistic
> verification. To that end, I attached a sample Grid-Stat config
file
> you could use for probabilistic verification (GridStatConfig_PROB).
>
> By looking at the .stat files you sent, I see that you're keeping
the
> forecast threshold fixed as >0 and then varying the observation
> threshold based on the probability field being evaluated (e.g. for
> QP100, OBS_THRESH
> >=1.0). That sounds good to me.
>
> However, >0 is a very low probability threshold. You might also
> consider applying some higher probability thresholds. For very low
> probabilities, you'd hope that there isn't anything present in the
> observations most of the time. So try using >0, >10, >25, >50 or
> something like that. This should be very easy to do. In the "fcst"
section set:
> cat_thresh = [ >0, >10, >25, >50 ]; And in the "obs" section set:
> cat_thresh = [ >=1.0, >=1.0, >=1.0, >=1.0 ];
>
> You just repeat the observation threshold once for each forecast
> threshold to make them the same length.
>
> Hopefully that helps. Just let us know what other questions come
up.
>
> Thanks,
> John
>
>
> On Wed, Mar 2, 2016 at 12:50 PM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
> >
> > Forwarded to you after I removed the NETcdf precip file. I will
> > place that file on the FTP server.
> >
> > Bob
> >
> > -----Original Message-----
> > From: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Sent: Wednesday, March 02, 2016 1:48 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Subject: FW: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score
> > Question
> >
> >
> >
> > -----Original Message-----
> > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Sent: Wednesday, March 02, 2016 1:45 PM
> > To: met_help at ucar.edu
> > Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: RE: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score
> > Question
> >
> > Hi, John,
> >
> > I am not setting the fcst cat_threshold - I am letting it default
> > to
> > > 0. I
> > am setting the obs threshold to thresholds you listed. I have
attached
> my
> > template I use for the case of precip > 1.0". I am reading a
NETCDF
> > file where the data appears to be in inches (looking at a dump of
> > the
> file).
> > Somebody else built the file for me but probably didn't set the
> > units correctly in the header. Would MET care if the units aren't
correct?
> >
> > By only setting pct,pstd,pjc,prc, aren't I eliminating
neighborhood
> > methods which is the whole reason I am running this to generate a
> > fractions skill score. Can NBRCNT not be used for probability
data?
> > It seems to be working.
> > I have attached the output file for each threshold.
> >
> > I can send you the a model file where you can run the data but I
> > will wait until you answer the questions above first.
> >
> > Thanks
> >
> > Bob
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, March 02, 2016 12:38 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
Score
> > Question
> >
> > Bob,
> >
> > Sorry I'm so slow in responding to this email. I was out of the
> > office a couple of days in there.
> >
> > Let me see if I understand what you're doing. You are running
> > Grid-Stat to compare ensemble probabilities to observed
precipitation.
> > So you're doing probabilistic verification. And you're using
> > StageIV as the observation in GRIB format.
> >
> > The units for StageIV data in GRIB really are millimeters
(kg/m^2).
> >
> > Here are some practical considerations for doing probabilistic
> > verification in MET.
> >
> > (1) The only line types you want are PCT, PSTD, PJC, and PRC. Set
> > all the other ones to "NONE".
> > (2) Probabilities are defined as a probability of some event. For
> > example, probability of precip is probability of precip > 0. Use
> > that event to define the ***observation*** threshold.
> >
> > However, you set the forecast threshold very differently. The
> > forecast probability is a bunch of numbers between 0 and 1 (or 0
and
> > 100). The thresholds are used to partition that probability
space,
> > such
> as:
> > cat_thresh = [ >=0.00, >=0.25, >=0.50, >=0.75, >=1.00 ];
> >
> > They must start with 0, end with 1, and all be the same
probability type.
> > Each probability value falls into exactly 1 of the 4 probability
> > bins listed above. MET uses these settings to defined a 4 x 2
> > contingency table (or N x 2, more generally). And all the
> > probabilistic statistics are derived from this table.
> >
> > You can certainly use more bins than what's defined above. You
> > might define thresholds every 0.10 or 0.05. Just take a look at
> > your forecast probabilities to get a good idea of the range of
> > probability values being used.
> >
> > I'm guessing you may have more questions. If you'd like, you
could
> > send me a sample forecast, observation, and config file you're
> > passing to Grid-Stat. I could take a close look, run it here, and
> > offer some suggestions.
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Feb 26, 2016 at 2:14 PM, robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271 >
> > >
> > > John, I changed the ob variable category to > .01, >.10 , etc..
> > > The precip values are in inches in my input netCDF file. Since
I
> > > am dealing with ensemble probabilities, should I have the fcst
> > > categories the same as precip or let it default to > 0? Now,
when
> > > I do an ncdump of the MET produced data file (ncdump -h
> > > grid_stat_1_APCP_6_240000L_20160213_000000V_pairs.nc), I see
that
> > > the precip data units are in Kg/m^2, however looking at the data
> > > in the
> > netCDF
> > > file, the units are in inches. Will this cause any problems?
The
> > listing
> > > is below:
> > >
> > > dimensions:
> > > lat = 750 ;
> > > lon = 750 ;
> > > variables:
> > > float lat(lat, lon) ;
> > > lat:long_name = "latitude" ;
> > > lat:units = "degrees_north" ;
> > > lat:standard_name = "latitude" ;
> > > float lon(lat, lon) ;
> > > lon:long_name = "longitude" ;
> > > lon:units = "degrees_east" ;
> > > lon:standard_name = "longitude" ;
> > > float FCST_PQP1_A06_FULL(lat, lon) ;
> > > FCST_PQP1_A06_FULL:name = "PQP1" ;
> > > FCST_PQP1_A06_FULL:long_name = "PQP1 at A06" ;
> > > FCST_PQP1_A06_FULL:level = "A06" ;
> > > FCST_PQP1_A06_FULL:units = "%" ;
> > > FCST_PQP1_A06_FULL:init_time = "20160212_000000"
;
> > > FCST_PQP1_A06_FULL:init_time_ut = "1455235200" ;
> > > FCST_PQP1_A06_FULL:valid_time =
"20160213_000000" ;
> > > FCST_PQP1_A06_FULL:valid_time_ut = "1455321600"
;
> > > FCST_PQP1_A06_FULL:accum_time = "060000" ;
> > > FCST_PQP1_A06_FULL:accum_time_sec = 21600 ;
> > > FCST_PQP1_A06_FULL:_FillValue = -9999.f ;
> > > FCST_PQP1_A06_FULL:masking_region = "FULL" ;
> > > FCST_PQP1_A06_FULL:smoothing_method = "NEAREST"
;
> > > FCST_PQP1_A06_FULL:smoothing_neighborhood = 1 ;
> > > float OBS_APCP_06_A6_FULL(lat, lon) ;
> > > OBS_APCP_06_A6_FULL:name = "APCP_06" ;
> > > OBS_APCP_06_A6_FULL:long_name = "APCP_06 at A6"
;
> > > OBS_APCP_06_A6_FULL:level = "A6" ;
> > > OBS_APCP_06_A6_FULL:units = "kg/m^2" ;
> > > OBS_APCP_06_A6_FULL:init_time =
"20160213_000000" ;
> > > OBS_APCP_06_A6_FULL:init_time_ut = "1455321600"
;
> > > OBS_APCP_06_A6_FULL:valid_time =
"20160213_000000" ;
> > > OBS_APCP_06_A6_FULL:valid_time_ut = "1455321600"
;
> > > OBS_APCP_06_A6_FULL:accum_time = "060000" ;
> > > OBS_APCP_06_A6_FULL:accum_time_sec = 21600 ;
> > > OBS_APCP_06_A6_FULL:_FillValue = -9999.f ;
> > > OBS_APCP_06_A6_FULL:masking_region = "FULL" ;
> > > OBS_APCP_06_A6_FULL:smoothing_method = "NEAREST"
;
> > > OBS_APCP_06_A6_FULL:smoothing_neighborhood = 1 ;
> > > float DIFF_PQP1_A06_APCP_06_A6_FULL(lat, lon) ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:name = "Forecast
> > > PQP1 minus Observed APCP_06" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:long_name = "PQP1
at
> > > A06 and
> > > APCP_06 at A6" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:level = "A06 and
A6" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:units = "% and
kg/m^2" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:_FillValue =
-9999.f ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:init_time =
> > > "20160212_000000" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:init_time_ut =
> > "1455235200" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time =
> > > "20160213_000000" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:valid_time_ut =
> > "1455321600"
> > > ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time =
"060000" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:accum_time_sec =
21600 ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:masking_region =
"FULL" ;
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_method =
> > "NEAREST"
> > > ;
> > >
> > > DIFF_PQP1_A06_APCP_06_A6_FULL:smoothing_neighborhood
> > > =
> > > 1 ;
> > >
> > > // global attributes:
> > > :FileOrigins = "File
> > >
/h/data/global/WXQC/data/fss/metoutput_fss/none/2016021200/grid_st
> > > at
> > > _1 _APCP_6_240000L_20160213_000000V_pairs.nc
> > > generated 20160225_230147 UTC on host dev8dlsn1 by the MET
> > > grid_stat
> > tool" ;
> > > :MET_version = "V5.1" ;
> > > :MET_tool = "grid_stat" ;
> > > :model = "WRF" ;
> > > :obtype = "ANALYS" ;
> > > :Difference = "Forecast Value - Observation
Value" ;
> > > :Projection = "Polar Stereographic" ;
> > > :hemisphere = "N" ;
> > > :scale_lat = "60.000000 degrees_north" ;
> > > :lat_pin = "6.590000" ;
> > > :lon_pin = "35.000000" ;
> > > :x_pin = "0.000000" ;
> > > :y_pin = "0.000000" ;
> > > :lon_orient = "80.000000" ;
> > > :d_km = "20.000000 km" ;
> > > :r_km = "6371.200000 km" ;
> > > :nx = "750" ;
> > > :ny = "750" ;
> > > }
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Thursday, February 25, 2016 3:11 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
> > > Subject: Re: [rt.rap.ucar.edu #75271] Ensemble Fractions Skill
> > > Score Question
> > >
> > > Bob,
> > >
> > > I see you have some questions about the fractions skill score.
> > > And I read that you're varying the precipitation threshold in
the
> > > configuration file for each run.
> > >
> > > I have a couple of suggestions...
> > >
> > > First, there's no need to run Grid-Stat multiple times with
> > > different precipitation thresholds. Instead, you can list
> > > multiple thresholds, like
> > > this:
> > > cat_thresh = [ >0, >0.254, >2.54, >25.4 ];
> > >
> > > I'm assuming the precip is in mm, and I'm using 25.4 for 1 inch.
> > > You should see a NBRCNT output line for each combination of
precip
> > > threshold and neighborhood size. So that means you wouldn't
need
> > > to include the threshold in the output prefix.
> > >
> > > Second, I find using environment variables in the MET config
files
> > > to very convenient. For example, the script which calls Grid-
Stat
> > > could set an environment variable named "FCST_NAME" and you
could
> > > refer to that in the config file, like this:
> > > name = "${FCST_NAME}";
> > > ...
> > > And later on use it in the output prefix as well if you'd like:
> > > output_prefix = "${FCST_NAME}";
> > >
> > > Just wanted to mention it.
> > >
> > > I just read your next email... and yes, I'd suggest defining the
> > > thresholds once and then referring to them in both the fcst and
> > > obs sections, like
> > > this:
> > > precip_thresh = [ >0, >0.254, >2.54, >25.4 ];
> > >
> > > And in the fcst and obs sections, set:
> > > cat_thresh = precip_thresh;
> > >
> > > Lastly, you mention that you're seeing the same value for
> > > Observation Rate. Can you tell me specifically the name of the
> > > column and the line type, you're looking at? And better yet,
send
> > > me some sample output files where that value is the same.
That'd
> > > help me to better understand what's going on.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Feb 25, 2016 at 1:41 PM, robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Feb 25 13:41:02 2016: Request 75271 was acted upon.
> > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > Queue: met_help
> > > > Subject: Ensemble Fractions Skill Score Question
> > > > Owner: Nobody
> > > > Requestors: robert.craig.2 at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75271
> > > > >
> > > >
> > > >
> > > > John, I have been running MET to generate fractions skill
scores for
> > > > different precipitation thresholds. We are verifying 4km
ensemble
> > > > forecasts using Stage IV for ground truth. What has me
baffled is
> > that
> > > I
> > > > am getting the same Observation Rate not matter if I am
> > > > verifying 6hr accumulations of precip > .01" or precip >1.00".
> > > > I would expect the observation rate to be greater for precip >
> > > > .01" than for precip
> > > > > 1.0" (a much more rare event) but it stays the same for all
> > > > thresholds. The model forecast frequencies to decrease as the
> > > > precip threshold increases as I would export.
> > > >
> > > > Below is a the configuration file I use for gridstat.
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //
> > > > //////////
> > > > //
> > > > // Grid-Stat configuration file.
> > > > //
> > > > // For additional information, see the MET_BASE/config/README
file.
> > > > //
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Output model name to be written // model = "WRF";
> > > >
> > > > obtype = "ANALYS";
> > > >
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Verification grid
> > > > //
> > > > regrid = {
> > > > to_grid = FCST;
> > > > vld_thresh = 0.5;
> > > > method = BUDGET;
> > > > width = 2;
> > > > }
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > cat_thresh = [ NA ];
> > > > cnt_thresh = [ NA ];
> > > > cnt_logic = UNION;
> > > > wind_thresh = [ NA ];
> > > > wind_logic = UNION;
> > > > prob=[ "True" ];
> > > > //
> > > > // Forecast and observation fields to be verified // fcst = {
> > > > wind_thresh = [ NA ];
> > > >
> > > > field = [
> > > > {
> > > > name = "QP100";
> > > > level = [ "A06" ];
> > > > GRIB1_ptv = 3;
> > > > cat_thresh = [ > 0 ];
> > > > }
> > > > ];
> > > >
> > > > }
> > > > obs = {
> > > > wind_thresh = [ NA ];
> > > >
> > > > field = [
> > > > {
> > > > name = "APCP_06";
> > > > level = [ "(*,*)" ];
> > > > cat_thresh = [ > 0 ];
> > > > }
> > > > ];
> > > > }
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Climatology mean data
> > > > //
> > > > climo_mean = {
> > > >
> > > > file_name = [];
> > > > field = [];
> > > >
> > > > regrid = {
> > > > vld_thresh = 0.5;
> > > > method = NEAREST;
> > > > width = 1;
> > > > }
> > > >
> > > > time_interp_method = DW_MEAN;
> > > > match_day = FALSE;
> > > > time_step = 21600;
> > > > }
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Verification masking regions
> > > > //
> > > > mask = {
> > > > grid = [ "FULL" ];
> > > > poly = [];
> > > > }
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Confidence interval settings
> > > > //
> > > > ci_alpha = [ 0.05 ];
> > > >
> > > > boot = {
> > > > interval = PCTILE;
> > > > rep_prop = 1.0;
> > > > n_rep = 0;
> > > > rng = "mt19937";
> > > > seed = "";
> > > > }
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Data smoothing methods
> > > > //
> > > > interp = {
> > > > field = BOTH;
> > > > vld_thresh = 1.0;
> > > >
> > > > type = [
> > > > {
> > > > method = NEAREST;
> > > > width = 1;
> > > > }
> > > > ];
> > > > }
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Neighborhood methods
> > > > //
> > > > nbrhd = {
> > > > vld_thresh = 1.0;
> > > > width =
> > > >
[1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,47,53];
> > > > cov_thresh = [ >=0.5 ];
> > > > }
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > //
> > > > //
> > > > //
> > > > //////////
> > > >
> > > > //
> > > > // Statistical output types
> > > > //
> > > > output_flag = {
> > > > fho = NONE;
> > > > ctc = NONE;
> > > > cts = NONE;
> > > > mctc = NONE;
> > > > mcts = NONE;
> > > > cnt = NONE;
> > > > sl1l2 = NONE;
> > > > sal1l2 = NONE;
> > > > vl1l2 = NONE;
> > > > val1l2 = NONE;
> > > > pct = NONE;
> > > > pstd = NONE;
> > > > pjc = NONE;
> > > > prc = NONE;
> > > > nbrctc = NONE;
> > > > nbrcts = NONE;
> > > > nbrcnt = STAT;
> > > > }
> > > >
> > > > //
> > > > // NetCDF matched pairs output file //
> > > > nc_pairs_flag = {
> > > > latlon = TRUE;
> > > > raw = TRUE;
> > > > diff = TRUE;
> > > > climo = TRUE;
> > > > }
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > ////
> > > > //
> > > > //////////
> > > >
> > > > rank_corr_flag = FALSE;
> > > > tmp_dir = "/tmp";
> > > > output_prefix = "100_APCP_6";
> > > > version = "V5.1";
> > > >
> > > >
> > > >
////////////////////////////////////////////////////////////////
> > > > ////
> > > > //
> > > > //////////
> > > >
> > > >
> > > > In this configuration file I change forecast field name and
> > > > output prefix as vary the precipitation thresholds. I also
vary
> > > > the
> > > neighborhood size.
> > > > Could I have something not set right in the config file?
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list