[Met_help] [rt.rap.ucar.edu #71629] History for problems with MODE not identifying objects in field

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Dear MET help,

I am using MODE in MET v4.1 to evaluate a custom hail forecast in comparison to a gridded observed hail field (MESH).  I've attached the first page of the .ps output from one of my dates.  For some reason no objects are being identified in the forecast field, but judging from the raw field, they should be.  My MODEConfig file is below.  Any idea what I'm doing wrong?

Thanks,

Becky Adams-Selin, Ph.D.
Staff Scientist II, AER


////////////////////////////////////////////////////////////////////////////////
//
// MODE configuration file.
//
// For additional information, see the MET_BASE/data/config/README file.
//
////////////////////////////////////////////////////////////////////////////////

//
// Output model name to be written
//
model = "WRF";

////////////////////////////////////////////////////////////////////////////////

//
// Approximate grid resolution (km)
//
grid_res = 4;

////////////////////////////////////////////////////////////////////////////////

//
// Forecast and observation fields to be verified
//
fcst = {
   field = {
      name  = "MAX_HAIL_24";
      level = "(*,*)";
   };

   raw_thresh        = >=1.0;
   conv_radius       = 4; // in grid squares
   conv_thresh       = >=0.5;
   vld_thresh        = 0.5;
   area_thresh       = >=4.0;
   inten_perc_value  = 100;
   inten_perc_thresh = >=0.0;
   merge_thresh      = >=0.5;
   merge_flag        = NONE;
};
obs = {
   field = {
      name = "MESH_hail";
      level = "(*,*)";
   };

   raw_thresh        = >=1.0;
   conv_radius       = 4; // in grid squares
   conv_thresh       = >=0.5;
   vld_thresh        = 0.5;
   area_thresh       = >=4.0;
   inten_perc_value  = 100;
   inten_perc_thresh = >=0.0;
   merge_thresh      = >=0.5;
   merge_flag        = NONE;
};

////////////////////////////////////////////////////////////////////////////////

//
// Handle missing data
//
mask_missing_flag = BOTH;

//
// Match objects between the forecast and observation fields
//
match_flag = MERGE_BOTH;

//
// Maximum centroid distance for objects to be compared
//
max_centroid_dist = 800.0/grid_res;

////////////////////////////////////////////////////////////////////////////////

//
// Verification masking regions
//
mask = {
   grid      = "";
   grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
   poly      = "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
   poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
};

//
// Fuzzy engine weights
//
weight = {
   centroid_dist    = 2.0;
   boundary_dist    = 4.0;
   convex_hull_dist = 0.0;
   angle_diff       = 1.0;
   area_ratio       = 1.0;
   int_area_ratio   = 2.0;
   complexity_ratio = 0.0;
   inten_perc_ratio = 0.0;
   inten_perc_value = 100;
}

////////////////////////////////////////////////////////////////////////////////

//
// Fuzzy engine interest functions
//
interest_function = {

   centroid_dist = (
      (            0.0, 1.0 )
      (  60.0/grid_res, 1.0 )
      ( 600.0/grid_res, 0.0 )
   );

   boundary_dist = (
      (            0.0, 1.0 )
      ( 400.0/grid_res, 0.0 )
   );

   convex_hull_dist = (
      (            0.0, 1.0 )
      ( 400.0/grid_res, 0.0 )
   );

   angle_diff = (
      (  0.0, 1.0 )
      ( 30.0, 1.0 )
      ( 90.0, 0.0 )
   corner   = 0.8;
   ratio_if = (
      (    0.0, 0.0 )
      ( corner, 1.0 )
      (    1.0, 1.0 )
   );

   area_ratio = ratio_if;

   int_area_ratio = (
      ( 0.00, 0.00 )
      ( 0.10, 0.50 )
      ( 0.25, 1.00 )
      ( 1.00, 1.00 )
   );

   complexity_ratio = ratio_if;

   inten_perc_ratio = ratio_if;
}

////////////////////////////////////////////////////////////////////////////////

//
// Total interest threshold for determining matches
//
total_interest_thresh = 0.5;

//
// Interest threshold for printing output pair information
//
print_interest_thresh = 0.0;

////////////////////////////////////////////////////////////////////////////////

//
// Plotting information
//
met_data_dir = "MET_BASE/data";

fcst_raw_plot = {
   color_table      = "MET_BASE/data/colortables/met_default.ctable";
   plot_min         = 0.0;
   plot_max         = 0.0;
   colorbar_spacing = 1;
};

obs_raw_plot = {
   color_table      = "MET_BASE/data/colortables/met_default.ctable";
   plot_min         = 0.0;
   plot_max         = 0.0;
   colorbar_spacing = 1;
};

object_plot = {
   color_table      = "MET_BASE/data/colortables/mode_obj.ctable";
};

//
// Number of grid boxes to fill with bad data values along the edge of the field
// to avoid edge effects.
//
zero_border_size = 1;

//
// Boolean for plotting on the region of valid data within the domain
//
plot_valid_flag = FALSE;

//
// Plot polyline edges using great circle arcs instead of straight lines
//
plot_gcarc_flag = FALSE;

////////////////////////////////////////////////////////////////////////////////

//
// NetCDF matched pairs, PostScript, and contingency table output files
//
ps_plot_flag    = TRUE;
nc_pairs_flag   = TRUE;
ct_stats_flag   = TRUE;

////////////////////////////////////////////////////////////////////////////////

output_prefix  = "";
version        = "V4.1";

////////////////////////////////////////////////////////////////////////////////


________________________________

This email is intended solely for the recipient. It may contain privileged, proprietary or confidential information or material. If you are not the intended recipient, please delete this email and any attachments and notify the sender of the error.


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: problems with MODE not identifying objects in field
From: John Halley Gotway
Time: Fri Apr 24 14:39:41 2015

Hello Becky,

I see that you're having trouble getting objects to show up in MODE.
Thanks for sending the sample plot and your configuration file.
Looking
closely at your choice of thresholds, I see a problem:
   raw_thresh        = >=1.0;
   conv_thresh       = >=0.5;

I would suggest changing it to this:
   raw_thresh        = >=0.0;
   conv_thresh       = >=0.5;

Try rerunning to see if you get objects.

The "raw_thresh" setting should generally not be used unless you have
a
good reason to do so.  It is applied to the raw data values and any
point
not meeting that criteria is set to a value of 0.  The resulting
values are
passed through to the object definition step.

Applying that logic doesn't make sense for your dataset.  You may
wonder
why you would you ever want to do this?  A while back we were
evaluating a
reflectivity forecast, but the system only forecast values of 35dBz or
higher.  So we had values of 35dBz surround by values of 0.  The
observations were continuous, with values of 35dBZ surrounded by 34dBZ
and
so on.  When defining objects in MODE, we do a smoothing step and
having
values of 0 vs values of 34 affected the output a lot.  So in order to
define observation and forecast objects in a consistent way, we zeroed
out
any observed values < 35.

Since this doesn't apply to your dataset, setting it as ">=0" should
pass
all of your data through to the object definition step.

Hope that helps clarify.

Thanks,
John Halley Gotway
met_help at ucar.edu


On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
met_help at ucar.edu> wrote:

>
> Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> Transaction: Ticket created by RSelin at aer.com
>        Queue: met_help
>      Subject: problems with MODE not identifying objects in field
>        Owner: Nobody
>   Requestors: RSelin at aer.com
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
>
> Dear MET help,
>
> I am using MODE in MET v4.1 to evaluate a custom hail forecast in
> comparison to a gridded observed hail field (MESH).  I've attached
the
> first page of the .ps output from one of my dates.  For some reason
no
> objects are being identified in the forecast field, but judging from
the
> raw field, they should be.  My MODEConfig file is below.  Any idea
what I'm
> doing wrong?
>
> Thanks,
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
>
>
>
////////////////////////////////////////////////////////////////////////////////
> //
> // MODE configuration file.
> //
> // For additional information, see the MET_BASE/data/config/README
file.
> //
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Output model name to be written
> //
> model = "WRF";
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Approximate grid resolution (km)
> //
> grid_res = 4;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Forecast and observation fields to be verified
> //
> fcst = {
>    field = {
>       name  = "MAX_HAIL_24";
>       level = "(*,*)";
>    };
>
>    raw_thresh        = >=1.0;
>    conv_radius       = 4; // in grid squares
>    conv_thresh       = >=0.5;
>    vld_thresh        = 0.5;
>    area_thresh       = >=4.0;
>    inten_perc_value  = 100;
>    inten_perc_thresh = >=0.0;
>    merge_thresh      = >=0.5;
>    merge_flag        = NONE;
> };
> obs = {
>    field = {
>       name = "MESH_hail";
>       level = "(*,*)";
>    };
>
>    raw_thresh        = >=1.0;
>    conv_radius       = 4; // in grid squares
>    conv_thresh       = >=0.5;
>    vld_thresh        = 0.5;
>    area_thresh       = >=4.0;
>    inten_perc_value  = 100;
>    inten_perc_thresh = >=0.0;
>    merge_thresh      = >=0.5;
>    merge_flag        = NONE;
> };
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Handle missing data
> //
> mask_missing_flag = BOTH;
>
> //
> // Match objects between the forecast and observation fields
> //
> match_flag = MERGE_BOTH;
>
> //
> // Maximum centroid distance for objects to be compared
> //
> max_centroid_dist = 800.0/grid_res;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Verification masking regions
> //
> mask = {
>    grid      = "";
>    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
>    poly      = "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
>    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> };
>
> //
> // Fuzzy engine weights
> //
> weight = {
>    centroid_dist    = 2.0;
>    boundary_dist    = 4.0;
>    convex_hull_dist = 0.0;
>    angle_diff       = 1.0;
>    area_ratio       = 1.0;
>    int_area_ratio   = 2.0;
>    complexity_ratio = 0.0;
>    inten_perc_ratio = 0.0;
>    inten_perc_value = 100;
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Fuzzy engine interest functions
> //
> interest_function = {
>
>    centroid_dist = (
>       (            0.0, 1.0 )
>       (  60.0/grid_res, 1.0 )
>       ( 600.0/grid_res, 0.0 )
>    );
>
>    boundary_dist = (
>       (            0.0, 1.0 )
>       ( 400.0/grid_res, 0.0 )
>    );
>
>    convex_hull_dist = (
>       (            0.0, 1.0 )
>       ( 400.0/grid_res, 0.0 )
>    );
>
>    angle_diff = (
>       (  0.0, 1.0 )
>       ( 30.0, 1.0 )
>       ( 90.0, 0.0 )
>    corner   = 0.8;
>    ratio_if = (
>       (    0.0, 0.0 )
>       ( corner, 1.0 )
>       (    1.0, 1.0 )
>    );
>
>    area_ratio = ratio_if;
>
>    int_area_ratio = (
>       ( 0.00, 0.00 )
>       ( 0.10, 0.50 )
>       ( 0.25, 1.00 )
>       ( 1.00, 1.00 )
>    );
>
>    complexity_ratio = ratio_if;
>
>    inten_perc_ratio = ratio_if;
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Total interest threshold for determining matches
> //
> total_interest_thresh = 0.5;
>
> //
> // Interest threshold for printing output pair information
> //
> print_interest_thresh = 0.0;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Plotting information
> //
> met_data_dir = "MET_BASE/data";
>
> fcst_raw_plot = {
>    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
>    plot_min         = 0.0;
>    plot_max         = 0.0;
>    colorbar_spacing = 1;
> };
>
> obs_raw_plot = {
>    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
>    plot_min         = 0.0;
>    plot_max         = 0.0;
>    colorbar_spacing = 1;
> };
>
> object_plot = {
>    color_table      = "MET_BASE/data/colortables/mode_obj.ctable";
> };
>
> //
> // Number of grid boxes to fill with bad data values along the edge
of the
> field
> // to avoid edge effects.
> //
> zero_border_size = 1;
>
> //
> // Boolean for plotting on the region of valid data within the
domain
> //
> plot_valid_flag = FALSE;
>
> //
> // Plot polyline edges using great circle arcs instead of straight
lines
> //
> plot_gcarc_flag = FALSE;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // NetCDF matched pairs, PostScript, and contingency table output
files
> //
> ps_plot_flag    = TRUE;
> nc_pairs_flag   = TRUE;
> ct_stats_flag   = TRUE;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> output_prefix  = "";
> version        = "V4.1";
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #71629] problems with MODE not identifying objects in field
From: Adams-Selin, Becky
Time: Sat Apr 25 08:10:03 2015

John,

I think that situation actually might apply to my dataset.  The
observational dataset only has hail values that are larger than 1.0
in.  The forecast dataset is continuous, from 0 inches upwards.  My
understanding was I should set the raw_thresh to 1.0 to make the
forecast and observational datasets consistent, like you said below.

Becky Adams-Selin, Ph.D.
Staff Scientist II, AER

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, April 24, 2015 3:39 PM
To: Adams-Selin, Becky
Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying objects in field

Hello Becky,

I see that you're having trouble getting objects to show up in MODE.
Thanks for sending the sample plot and your configuration file.
Looking
closely at your choice of thresholds, I see a problem:
   raw_thresh        = >=1.0;
   conv_thresh       = >=0.5;

I would suggest changing it to this:
   raw_thresh        = >=0.0;
   conv_thresh       = >=0.5;

Try rerunning to see if you get objects.

The "raw_thresh" setting should generally not be used unless you have
a
good reason to do so.  It is applied to the raw data values and any
point
not meeting that criteria is set to a value of 0.  The resulting
values are
passed through to the object definition step.

Applying that logic doesn't make sense for your dataset.  You may
wonder
why you would you ever want to do this?  A while back we were
evaluating a
reflectivity forecast, but the system only forecast values of 35dBz or
higher.  So we had values of 35dBz surround by values of 0.  The
observations were continuous, with values of 35dBZ surrounded by 34dBZ
and
so on.  When defining objects in MODE, we do a smoothing step and
having
values of 0 vs values of 34 affected the output a lot.  So in order to
define observation and forecast objects in a consistent way, we zeroed
out
any observed values < 35.

Since this doesn't apply to your dataset, setting it as ">=0" should
pass
all of your data through to the object definition step.

Hope that helps clarify.

Thanks,
John Halley Gotway
met_help at ucar.edu


On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
met_help at ucar.edu> wrote:

>
> Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> Transaction: Ticket created by RSelin at aer.com
>        Queue: met_help
>      Subject: problems with MODE not identifying objects in field
>        Owner: Nobody
>   Requestors: RSelin at aer.com
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
>
> Dear MET help,
>
> I am using MODE in MET v4.1 to evaluate a custom hail forecast in
> comparison to a gridded observed hail field (MESH).  I've attached
the
> first page of the .ps output from one of my dates.  For some reason
no
> objects are being identified in the forecast field, but judging from
the
> raw field, they should be.  My MODEConfig file is below.  Any idea
what I'm
> doing wrong?
>
> Thanks,
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
>
>
>
////////////////////////////////////////////////////////////////////////////////
> //
> // MODE configuration file.
> //
> // For additional information, see the MET_BASE/data/config/README
file.
> //
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Output model name to be written
> //
> model = "WRF";
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Approximate grid resolution (km)
> //
> grid_res = 4;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Forecast and observation fields to be verified
> //
> fcst = {
>    field = {
>       name  = "MAX_HAIL_24";
>       level = "(*,*)";
>    };
>
>    raw_thresh        = >=1.0;
>    conv_radius       = 4; // in grid squares
>    conv_thresh       = >=0.5;
>    vld_thresh        = 0.5;
>    area_thresh       = >=4.0;
>    inten_perc_value  = 100;
>    inten_perc_thresh = >=0.0;
>    merge_thresh      = >=0.5;
>    merge_flag        = NONE;
> };
> obs = {
>    field = {
>       name = "MESH_hail";
>       level = "(*,*)";
>    };
>
>    raw_thresh        = >=1.0;
>    conv_radius       = 4; // in grid squares
>    conv_thresh       = >=0.5;
>    vld_thresh        = 0.5;
>    area_thresh       = >=4.0;
>    inten_perc_value  = 100;
>    inten_perc_thresh = >=0.0;
>    merge_thresh      = >=0.5;
>    merge_flag        = NONE;
> };
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Handle missing data
> //
> mask_missing_flag = BOTH;
>
> //
> // Match objects between the forecast and observation fields
> //
> match_flag = MERGE_BOTH;
>
> //
> // Maximum centroid distance for objects to be compared
> //
> max_centroid_dist = 800.0/grid_res;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Verification masking regions
> //
> mask = {
>    grid      = "";
>    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
>    poly      = "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
>    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> };
>
> //
> // Fuzzy engine weights
> //
> weight = {
>    centroid_dist    = 2.0;
>    boundary_dist    = 4.0;
>    convex_hull_dist = 0.0;
>    angle_diff       = 1.0;
>    area_ratio       = 1.0;
>    int_area_ratio   = 2.0;
>    complexity_ratio = 0.0;
>    inten_perc_ratio = 0.0;
>    inten_perc_value = 100;
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Fuzzy engine interest functions
> //
> interest_function = {
>
>    centroid_dist = (
>       (            0.0, 1.0 )
>       (  60.0/grid_res, 1.0 )
>       ( 600.0/grid_res, 0.0 )
>    );
>
>    boundary_dist = (
>       (            0.0, 1.0 )
>       ( 400.0/grid_res, 0.0 )
>    );
>
>    convex_hull_dist = (
>       (            0.0, 1.0 )
>       ( 400.0/grid_res, 0.0 )
>    );
>
>    angle_diff = (
>       (  0.0, 1.0 )
>       ( 30.0, 1.0 )
>       ( 90.0, 0.0 )
>    corner   = 0.8;
>    ratio_if = (
>       (    0.0, 0.0 )
>       ( corner, 1.0 )
>       (    1.0, 1.0 )
>    );
>
>    area_ratio = ratio_if;
>
>    int_area_ratio = (
>       ( 0.00, 0.00 )
>       ( 0.10, 0.50 )
>       ( 0.25, 1.00 )
>       ( 1.00, 1.00 )
>    );
>
>    complexity_ratio = ratio_if;
>
>    inten_perc_ratio = ratio_if;
> }
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Total interest threshold for determining matches
> //
> total_interest_thresh = 0.5;
>
> //
> // Interest threshold for printing output pair information
> //
> print_interest_thresh = 0.0;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Plotting information
> //
> met_data_dir = "MET_BASE/data";
>
> fcst_raw_plot = {
>    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
>    plot_min         = 0.0;
>    plot_max         = 0.0;
>    colorbar_spacing = 1;
> };
>
> obs_raw_plot = {
>    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
>    plot_min         = 0.0;
>    plot_max         = 0.0;
>    colorbar_spacing = 1;
> };
>
> object_plot = {
>    color_table      = "MET_BASE/data/colortables/mode_obj.ctable";
> };
>
> //
> // Number of grid boxes to fill with bad data values along the edge
of the
> field
> // to avoid edge effects.
> //
> zero_border_size = 1;
>
> //
> // Boolean for plotting on the region of valid data within the
domain
> //
> plot_valid_flag = FALSE;
>
> //
> // Plot polyline edges using great circle arcs instead of straight
lines
> //
> plot_gcarc_flag = FALSE;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // NetCDF matched pairs, PostScript, and contingency table output
files
> //
> ps_plot_flag    = TRUE;
> nc_pairs_flag   = TRUE;
> ct_stats_flag   = TRUE;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> output_prefix  = "";
> version        = "V4.1";
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>


________________________________

This email is intended solely for the recipient. It may contain
privileged, proprietary or confidential information or material. If
you are not the intended recipient, please delete this email and any
attachments and notify the sender of the error.



------------------------------------------------
Subject: problems with MODE not identifying objects in field
From: John Halley Gotway
Time: Sat Apr 25 08:25:55 2015

Becky,

OK, sounds good.  You could try setting the convolution radius to 0
just to
see if you can get forecast objects.  Or try lowering the threshold.
Perhaps they're all getting smoothed out.

If you're still stuck I could take a look at some sample files.  I'd
need a
sample forecast, observation, and MODE config file.  You could post
them to
our anonymous for site following these instructions:

http://www.dtcenter.org/met/users/support/met_help.php#ftp

Thanks
John


On Saturday, April 25, 2015, Adams-Selin, Becky via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
> John,
>
> I think that situation actually might apply to my dataset.  The
> observational dataset only has hail values that are larger than 1.0
in.
> The forecast dataset is continuous, from 0 inches upwards.  My
> understanding was I should set the raw_thresh to 1.0 to make the
forecast
> and observational datasets consistent, like you said below.
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu <javascript:;>]
> Sent: Friday, April 24, 2015 3:39 PM
> To: Adams-Selin, Becky
> Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> objects in field
>
> Hello Becky,
>
> I see that you're having trouble getting objects to show up in MODE.
> Thanks for sending the sample plot and your configuration file.
Looking
> closely at your choice of thresholds, I see a problem:
>    raw_thresh        = >=1.0;
>    conv_thresh       = >=0.5;
>
> I would suggest changing it to this:
>    raw_thresh        = >=0.0;
>    conv_thresh       = >=0.5;
>
> Try rerunning to see if you get objects.
>
> The "raw_thresh" setting should generally not be used unless you
have a
> good reason to do so.  It is applied to the raw data values and any
point
> not meeting that criteria is set to a value of 0.  The resulting
values are
> passed through to the object definition step.
>
> Applying that logic doesn't make sense for your dataset.  You may
wonder
> why you would you ever want to do this?  A while back we were
evaluating a
> reflectivity forecast, but the system only forecast values of 35dBz
or
> higher.  So we had values of 35dBz surround by values of 0.  The
> observations were continuous, with values of 35dBZ surrounded by
34dBZ and
> so on.  When defining objects in MODE, we do a smoothing step and
having
> values of 0 vs values of 34 affected the output a lot.  So in order
to
> define observation and forecast objects in a consistent way, we
zeroed out
> any observed values < 35.
>
> Since this doesn't apply to your dataset, setting it as ">=0" should
pass
> all of your data through to the object definition step.
>
> Hope that helps clarify.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu <javascript:;>
>
>
> On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
> met_help at ucar.edu <javascript:;>> wrote:
>
> >
> > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > Transaction: Ticket created by RSelin at aer.com <javascript:;>
> >        Queue: met_help
> >      Subject: problems with MODE not identifying objects in field
> >        Owner: Nobody
> >   Requestors: RSelin at aer.com <javascript:;>
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> >
> >
> > Dear MET help,
> >
> > I am using MODE in MET v4.1 to evaluate a custom hail forecast in
> > comparison to a gridded observed hail field (MESH).  I've attached
the
> > first page of the .ps output from one of my dates.  For some
reason no
> > objects are being identified in the forecast field, but judging
from the
> > raw field, they should be.  My MODEConfig file is below.  Any idea
what
> I'm
> > doing wrong?
> >
> > Thanks,
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > //
> > // MODE configuration file.
> > //
> > // For additional information, see the MET_BASE/data/config/README
file.
> > //
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Output model name to be written
> > //
> > model = "WRF";
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Approximate grid resolution (km)
> > //
> > grid_res = 4;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Forecast and observation fields to be verified
> > //
> > fcst = {
> >    field = {
> >       name  = "MAX_HAIL_24";
> >       level = "(*,*)";
> >    };
> >
> >    raw_thresh        = >=1.0;
> >    conv_radius       = 4; // in grid squares
> >    conv_thresh       = >=0.5;
> >    vld_thresh        = 0.5;
> >    area_thresh       = >=4.0;
> >    inten_perc_value  = 100;
> >    inten_perc_thresh = >=0.0;
> >    merge_thresh      = >=0.5;
> >    merge_flag        = NONE;
> > };
> > obs = {
> >    field = {
> >       name = "MESH_hail";
> >       level = "(*,*)";
> >    };
> >
> >    raw_thresh        = >=1.0;
> >    conv_radius       = 4; // in grid squares
> >    conv_thresh       = >=0.5;
> >    vld_thresh        = 0.5;
> >    area_thresh       = >=4.0;
> >    inten_perc_value  = 100;
> >    inten_perc_thresh = >=0.0;
> >    merge_thresh      = >=0.5;
> >    merge_flag        = NONE;
> > };
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Handle missing data
> > //
> > mask_missing_flag = BOTH;
> >
> > //
> > // Match objects between the forecast and observation fields
> > //
> > match_flag = MERGE_BOTH;
> >
> > //
> > // Maximum centroid distance for objects to be compared
> > //
> > max_centroid_dist = 800.0/grid_res;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Verification masking regions
> > //
> > mask = {
> >    grid      = "";
> >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
> >    poly      = "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> > };
> >
> > //
> > // Fuzzy engine weights
> > //
> > weight = {
> >    centroid_dist    = 2.0;
> >    boundary_dist    = 4.0;
> >    convex_hull_dist = 0.0;
> >    angle_diff       = 1.0;
> >    area_ratio       = 1.0;
> >    int_area_ratio   = 2.0;
> >    complexity_ratio = 0.0;
> >    inten_perc_ratio = 0.0;
> >    inten_perc_value = 100;
> > }
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Fuzzy engine interest functions
> > //
> > interest_function = {
> >
> >    centroid_dist = (
> >       (            0.0, 1.0 )
> >       (  60.0/grid_res, 1.0 )
> >       ( 600.0/grid_res, 0.0 )
> >    );
> >
> >    boundary_dist = (
> >       (            0.0, 1.0 )
> >       ( 400.0/grid_res, 0.0 )
> >    );
> >
> >    convex_hull_dist = (
> >       (            0.0, 1.0 )
> >       ( 400.0/grid_res, 0.0 )
> >    );
> >
> >    angle_diff = (
> >       (  0.0, 1.0 )
> >       ( 30.0, 1.0 )
> >       ( 90.0, 0.0 )
> >    corner   = 0.8;
> >    ratio_if = (
> >       (    0.0, 0.0 )
> >       ( corner, 1.0 )
> >       (    1.0, 1.0 )
> >    );
> >
> >    area_ratio = ratio_if;
> >
> >    int_area_ratio = (
> >       ( 0.00, 0.00 )
> >       ( 0.10, 0.50 )
> >       ( 0.25, 1.00 )
> >       ( 1.00, 1.00 )
> >    );
> >
> >    complexity_ratio = ratio_if;
> >
> >    inten_perc_ratio = ratio_if;
> > }
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Total interest threshold for determining matches
> > //
> > total_interest_thresh = 0.5;
> >
> > //
> > // Interest threshold for printing output pair information
> > //
> > print_interest_thresh = 0.0;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Plotting information
> > //
> > met_data_dir = "MET_BASE/data";
> >
> > fcst_raw_plot = {
> >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> >    plot_min         = 0.0;
> >    plot_max         = 0.0;
> >    colorbar_spacing = 1;
> > };
> >
> > obs_raw_plot = {
> >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> >    plot_min         = 0.0;
> >    plot_max         = 0.0;
> >    colorbar_spacing = 1;
> > };
> >
> > object_plot = {
> >    color_table      = "MET_BASE/data/colortables/mode_obj.ctable";
> > };
> >
> > //
> > // Number of grid boxes to fill with bad data values along the
edge of
> the
> > field
> > // to avoid edge effects.
> > //
> > zero_border_size = 1;
> >
> > //
> > // Boolean for plotting on the region of valid data within the
domain
> > //
> > plot_valid_flag = FALSE;
> >
> > //
> > // Plot polyline edges using great circle arcs instead of straight
lines
> > //
> > plot_gcarc_flag = FALSE;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // NetCDF matched pairs, PostScript, and contingency table output
files
> > //
> > ps_plot_flag    = TRUE;
> > nc_pairs_flag   = TRUE;
> > ct_stats_flag   = TRUE;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > output_prefix  = "";
> > version        = "V4.1";
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material.
If you
> are
> > not the intended recipient, please delete this email and any
attachments
> > and notify the sender of the error.
> >
> >
>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>
>
>

------------------------------------------------
Subject: problems with MODE not identifying objects in field
From: Adams-Selin, Becky
Time: Sat Apr 25 09:19:41 2015

I tried lowering the conv_radius to 0, and the conv_threshold to 0.25.
I attached what I got.  I'm new to MODE, but just from visually
comparing the forecast and observation fields, I feel like I should be
getting roughly similar numbers of objects....I would even have
expected fewer objects from the observation field.  Something seems
wrong.

I went ahead and created a selin_data directory on the passive ftp
site.

The forecast file is s4m17_arw_140508_hailsum.nc
The observation file is HailSize20140508L1S_caps_maxnn_2.0km.nc
And the mode config file is MODEConfig_bigintense

If you could take a look I would really appreciate it.

Becky Adams-Selin, Ph.D.
Staff Scientist II, AER

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Saturday, April 25, 2015 9:25 AM
To: Adams-Selin, Becky
Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying objects in field

Becky,

OK, sounds good.  You could try setting the convolution radius to 0
just to
see if you can get forecast objects.  Or try lowering the threshold.
Perhaps they're all getting smoothed out.

If you're still stuck I could take a look at some sample files.  I'd
need a
sample forecast, observation, and MODE config file.  You could post
them to
our anonymous for site following these instructions:

http://www.dtcenter.org/met/users/support/met_help.php#ftp

Thanks
John


On Saturday, April 25, 2015, Adams-Selin, Becky via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
> John,
>
> I think that situation actually might apply to my dataset.  The
> observational dataset only has hail values that are larger than 1.0
in.
> The forecast dataset is continuous, from 0 inches upwards.  My
> understanding was I should set the raw_thresh to 1.0 to make the
forecast
> and observational datasets consistent, like you said below.
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu <javascript:;>]
> Sent: Friday, April 24, 2015 3:39 PM
> To: Adams-Selin, Becky
> Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> objects in field
>
> Hello Becky,
>
> I see that you're having trouble getting objects to show up in MODE.
> Thanks for sending the sample plot and your configuration file.
Looking
> closely at your choice of thresholds, I see a problem:
>    raw_thresh        = >=1.0;
>    conv_thresh       = >=0.5;
>
> I would suggest changing it to this:
>    raw_thresh        = >=0.0;
>    conv_thresh       = >=0.5;
>
> Try rerunning to see if you get objects.
>
> The "raw_thresh" setting should generally not be used unless you
have a
> good reason to do so.  It is applied to the raw data values and any
point
> not meeting that criteria is set to a value of 0.  The resulting
values are
> passed through to the object definition step.
>
> Applying that logic doesn't make sense for your dataset.  You may
wonder
> why you would you ever want to do this?  A while back we were
evaluating a
> reflectivity forecast, but the system only forecast values of 35dBz
or
> higher.  So we had values of 35dBz surround by values of 0.  The
> observations were continuous, with values of 35dBZ surrounded by
34dBZ and
> so on.  When defining objects in MODE, we do a smoothing step and
having
> values of 0 vs values of 34 affected the output a lot.  So in order
to
> define observation and forecast objects in a consistent way, we
zeroed out
> any observed values < 35.
>
> Since this doesn't apply to your dataset, setting it as ">=0" should
pass
> all of your data through to the object definition step.
>
> Hope that helps clarify.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu <javascript:;>
>
>
> On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
> met_help at ucar.edu <javascript:;>> wrote:
>
> >
> > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > Transaction: Ticket created by RSelin at aer.com <javascript:;>
> >        Queue: met_help
> >      Subject: problems with MODE not identifying objects in field
> >        Owner: Nobody
> >   Requestors: RSelin at aer.com <javascript:;>
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> >
> >
> > Dear MET help,
> >
> > I am using MODE in MET v4.1 to evaluate a custom hail forecast in
> > comparison to a gridded observed hail field (MESH).  I've attached
the
> > first page of the .ps output from one of my dates.  For some
reason no
> > objects are being identified in the forecast field, but judging
from the
> > raw field, they should be.  My MODEConfig file is below.  Any idea
what
> I'm
> > doing wrong?
> >
> > Thanks,
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > //
> > // MODE configuration file.
> > //
> > // For additional information, see the MET_BASE/data/config/README
file.
> > //
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Output model name to be written
> > //
> > model = "WRF";
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Approximate grid resolution (km)
> > //
> > grid_res = 4;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Forecast and observation fields to be verified
> > //
> > fcst = {
> >    field = {
> >       name  = "MAX_HAIL_24";
> >       level = "(*,*)";
> >    };
> >
> >    raw_thresh        = >=1.0;
> >    conv_radius       = 4; // in grid squares
> >    conv_thresh       = >=0.5;
> >    vld_thresh        = 0.5;
> >    area_thresh       = >=4.0;
> >    inten_perc_value  = 100;
> >    inten_perc_thresh = >=0.0;
> >    merge_thresh      = >=0.5;
> >    merge_flag        = NONE;
> > };
> > obs = {
> >    field = {
> >       name = "MESH_hail";
> >       level = "(*,*)";
> >    };
> >
> >    raw_thresh        = >=1.0;
> >    conv_radius       = 4; // in grid squares
> >    conv_thresh       = >=0.5;
> >    vld_thresh        = 0.5;
> >    area_thresh       = >=4.0;
> >    inten_perc_value  = 100;
> >    inten_perc_thresh = >=0.0;
> >    merge_thresh      = >=0.5;
> >    merge_flag        = NONE;
> > };
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Handle missing data
> > //
> > mask_missing_flag = BOTH;
> >
> > //
> > // Match objects between the forecast and observation fields
> > //
> > match_flag = MERGE_BOTH;
> >
> > //
> > // Maximum centroid distance for objects to be compared
> > //
> > max_centroid_dist = 800.0/grid_res;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Verification masking regions
> > //
> > mask = {
> >    grid      = "";
> >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
> >    poly      = "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> > };
> >
> > //
> > // Fuzzy engine weights
> > //
> > weight = {
> >    centroid_dist    = 2.0;
> >    boundary_dist    = 4.0;
> >    convex_hull_dist = 0.0;
> >    angle_diff       = 1.0;
> >    area_ratio       = 1.0;
> >    int_area_ratio   = 2.0;
> >    complexity_ratio = 0.0;
> >    inten_perc_ratio = 0.0;
> >    inten_perc_value = 100;
> > }
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Fuzzy engine interest functions
> > //
> > interest_function = {
> >
> >    centroid_dist = (
> >       (            0.0, 1.0 )
> >       (  60.0/grid_res, 1.0 )
> >       ( 600.0/grid_res, 0.0 )
> >    );
> >
> >    boundary_dist = (
> >       (            0.0, 1.0 )
> >       ( 400.0/grid_res, 0.0 )
> >    );
> >
> >    convex_hull_dist = (
> >       (            0.0, 1.0 )
> >       ( 400.0/grid_res, 0.0 )
> >    );
> >
> >    angle_diff = (
> >       (  0.0, 1.0 )
> >       ( 30.0, 1.0 )
> >       ( 90.0, 0.0 )
> >    corner   = 0.8;
> >    ratio_if = (
> >       (    0.0, 0.0 )
> >       ( corner, 1.0 )
> >       (    1.0, 1.0 )
> >    );
> >
> >    area_ratio = ratio_if;
> >
> >    int_area_ratio = (
> >       ( 0.00, 0.00 )
> >       ( 0.10, 0.50 )
> >       ( 0.25, 1.00 )
> >       ( 1.00, 1.00 )
> >    );
> >
> >    complexity_ratio = ratio_if;
> >
> >    inten_perc_ratio = ratio_if;
> > }
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Total interest threshold for determining matches
> > //
> > total_interest_thresh = 0.5;
> >
> > //
> > // Interest threshold for printing output pair information
> > //
> > print_interest_thresh = 0.0;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Plotting information
> > //
> > met_data_dir = "MET_BASE/data";
> >
> > fcst_raw_plot = {
> >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> >    plot_min         = 0.0;
> >    plot_max         = 0.0;
> >    colorbar_spacing = 1;
> > };
> >
> > obs_raw_plot = {
> >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> >    plot_min         = 0.0;
> >    plot_max         = 0.0;
> >    colorbar_spacing = 1;
> > };
> >
> > object_plot = {
> >    color_table      = "MET_BASE/data/colortables/mode_obj.ctable";
> > };
> >
> > //
> > // Number of grid boxes to fill with bad data values along the
edge of
> the
> > field
> > // to avoid edge effects.
> > //
> > zero_border_size = 1;
> >
> > //
> > // Boolean for plotting on the region of valid data within the
domain
> > //
> > plot_valid_flag = FALSE;
> >
> > //
> > // Plot polyline edges using great circle arcs instead of straight
lines
> > //
> > plot_gcarc_flag = FALSE;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // NetCDF matched pairs, PostScript, and contingency table output
files
> > //
> > ps_plot_flag    = TRUE;
> > nc_pairs_flag   = TRUE;
> > ct_stats_flag   = TRUE;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > output_prefix  = "";
> > version        = "V4.1";
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material.
If you
> are
> > not the intended recipient, please delete this email and any
attachments
> > and notify the sender of the error.
> >
> >
>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>
>
>

________________________________

This email is intended solely for the recipient. It may contain
privileged, proprietary or confidential information or material. If
you are not the intended recipient, please delete this email and any
attachments and notify the sender of the error.

------------------------------------------------
Subject: problems with MODE not identifying objects in field
From: John Halley Gotway
Time: Mon Apr 27 11:03:13 2015

Becky,

Thanks for sending the data.  I ran the following two plot_data_plane
(from
MET) commands to plot this data using the same scale of 0 to 1.25. The
resulting images are attached:

plot_data_plane \
   HailSize20140508L1S_caps_maxnn_2.0km.nc \
   HailSize20140508L1S_caps_maxnn_2.0km.ps \
   'name="MESH_hail"; level="(*,*)";' -plot_range 0 1.25

plot_data_plane \
   s4m17_arw_140508_hailsum.nc \
   s4m17_arw_140508_hailsum.ps \
   'name="MAX_HAIL_24"; level="(*,*)";' -plot_range 0 1.25

Looking at them, I notice that the observed values are much higher
than the
forecast values!  Very few forecast values are above 1.0 while most
observed values are above 1.0.  When you apply a raw thresh of >=1.0,
you're zeroing out almost all of the forecast points but relatively
few
observation points.  That's why you're getting so many more
observation
objects than forecast objects.

Inspecting the data more closely, I see...
 - the range on non-zero observed values is 0.50 to 3.74
 - the range of non-zero forecast values is 0.01 to 1.39

Based on that, I made the following changes to the config file...
 - switch the raw_thresh from >=1.0 to >=0.5
 - increase the smoothing radius (conv_radius) from 1 to 4
 - set the plot_max in the fcst_raw_plot and obs_raw_plot sections to
3.75
to use a consistent scale
 - set the forecast conv_thresh as >=0.25 and observed conv_thresh as
>=0.5.

This resulted in 45 forecast and 44 observed objects.  We typically
use the
same threshold for both forecast and observation, but your forecast
has a
very clear low bias, by about a factor of 2.  That's why I chose the
observed threshold to be 2 times the forecast threshold.

The config file and resulting output image is attached.
Unfortunately,
this takes a long time to run, about 5 min on my machine!

I'd suggest that you play around with the settings more.  You should
decide
what "scale" of objects you're looking for.  If you want a smaller
number
of smoother objects, try increasing the convolution radius and
lowering the
threshold.  The fewer the number of objects MODE has to keep track of,
the
faster it'll run.

Hope that helps.

Thanks,
John


On Sat, Apr 25, 2015 at 9:19 AM, Adams-Selin, Becky via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
> I tried lowering the conv_radius to 0, and the conv_threshold to
0.25.  I
> attached what I got.  I'm new to MODE, but just from visually
comparing the
> forecast and observation fields, I feel like I should be getting
roughly
> similar numbers of objects....I would even have expected fewer
objects from
> the observation field.  Something seems wrong.
>
> I went ahead and created a selin_data directory on the passive ftp
site.
>
> The forecast file is s4m17_arw_140508_hailsum.nc
> The observation file is HailSize20140508L1S_caps_maxnn_2.0km.nc
> And the mode config file is MODEConfig_bigintense
>
> If you could take a look I would really appreciate it.
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Saturday, April 25, 2015 9:25 AM
> To: Adams-Selin, Becky
> Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> objects in field
>
> Becky,
>
> OK, sounds good.  You could try setting the convolution radius to 0
just to
> see if you can get forecast objects.  Or try lowering the threshold.
> Perhaps they're all getting smoothed out.
>
> If you're still stuck I could take a look at some sample files.  I'd
need a
> sample forecast, observation, and MODE config file.  You could post
them to
> our anonymous for site following these instructions:
>
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks
> John
>
>
> On Saturday, April 25, 2015, Adams-Selin, Becky via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> >
> > John,
> >
> > I think that situation actually might apply to my dataset.  The
> > observational dataset only has hail values that are larger than
1.0 in.
> > The forecast dataset is continuous, from 0 inches upwards.  My
> > understanding was I should set the raw_thresh to 1.0 to make the
forecast
> > and observational datasets consistent, like you said below.
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu <javascript:;>]
> > Sent: Friday, April 24, 2015 3:39 PM
> > To: Adams-Selin, Becky
> > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> > objects in field
> >
> > Hello Becky,
> >
> > I see that you're having trouble getting objects to show up in
MODE.
> > Thanks for sending the sample plot and your configuration file.
Looking
> > closely at your choice of thresholds, I see a problem:
> >    raw_thresh        = >=1.0;
> >    conv_thresh       = >=0.5;
> >
> > I would suggest changing it to this:
> >    raw_thresh        = >=0.0;
> >    conv_thresh       = >=0.5;
> >
> > Try rerunning to see if you get objects.
> >
> > The "raw_thresh" setting should generally not be used unless you
have a
> > good reason to do so.  It is applied to the raw data values and
any point
> > not meeting that criteria is set to a value of 0.  The resulting
values
> are
> > passed through to the object definition step.
> >
> > Applying that logic doesn't make sense for your dataset.  You may
wonder
> > why you would you ever want to do this?  A while back we were
evaluating
> a
> > reflectivity forecast, but the system only forecast values of
35dBz or
> > higher.  So we had values of 35dBz surround by values of 0.  The
> > observations were continuous, with values of 35dBZ surrounded by
34dBZ
> and
> > so on.  When defining objects in MODE, we do a smoothing step and
having
> > values of 0 vs values of 34 affected the output a lot.  So in
order to
> > define observation and forecast objects in a consistent way, we
zeroed
> out
> > any observed values < 35.
> >
> > Since this doesn't apply to your dataset, setting it as ">=0"
should pass
> > all of your data through to the object definition step.
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John Halley Gotway
> > met_help at ucar.edu <javascript:;>
> >
> >
> > On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
> > met_help at ucar.edu <javascript:;>> wrote:
> >
> > >
> > > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > > Transaction: Ticket created by RSelin at aer.com <javascript:;>
> > >        Queue: met_help
> > >      Subject: problems with MODE not identifying objects in
field
> > >        Owner: Nobody
> > >   Requestors: RSelin at aer.com <javascript:;>
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> >
> > >
> > >
> > > Dear MET help,
> > >
> > > I am using MODE in MET v4.1 to evaluate a custom hail forecast
in
> > > comparison to a gridded observed hail field (MESH).  I've
attached the
> > > first page of the .ps output from one of my dates.  For some
reason no
> > > objects are being identified in the forecast field, but judging
from
> the
> > > raw field, they should be.  My MODEConfig file is below.  Any
idea what
> > I'm
> > > doing wrong?
> > >
> > > Thanks,
> > >
> > > Becky Adams-Selin, Ph.D.
> > > Staff Scientist II, AER
> > >
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > //
> > > // MODE configuration file.
> > > //
> > > // For additional information, see the
MET_BASE/data/config/README
> file.
> > > //
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Output model name to be written
> > > //
> > > model = "WRF";
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Approximate grid resolution (km)
> > > //
> > > grid_res = 4;
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Forecast and observation fields to be verified
> > > //
> > > fcst = {
> > >    field = {
> > >       name  = "MAX_HAIL_24";
> > >       level = "(*,*)";
> > >    };
> > >
> > >    raw_thresh        = >=1.0;
> > >    conv_radius       = 4; // in grid squares
> > >    conv_thresh       = >=0.5;
> > >    vld_thresh        = 0.5;
> > >    area_thresh       = >=4.0;
> > >    inten_perc_value  = 100;
> > >    inten_perc_thresh = >=0.0;
> > >    merge_thresh      = >=0.5;
> > >    merge_flag        = NONE;
> > > };
> > > obs = {
> > >    field = {
> > >       name = "MESH_hail";
> > >       level = "(*,*)";
> > >    };
> > >
> > >    raw_thresh        = >=1.0;
> > >    conv_radius       = 4; // in grid squares
> > >    conv_thresh       = >=0.5;
> > >    vld_thresh        = 0.5;
> > >    area_thresh       = >=4.0;
> > >    inten_perc_value  = 100;
> > >    inten_perc_thresh = >=0.0;
> > >    merge_thresh      = >=0.5;
> > >    merge_flag        = NONE;
> > > };
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Handle missing data
> > > //
> > > mask_missing_flag = BOTH;
> > >
> > > //
> > > // Match objects between the forecast and observation fields
> > > //
> > > match_flag = MERGE_BOTH;
> > >
> > > //
> > > // Maximum centroid distance for objects to be compared
> > > //
> > > max_centroid_dist = 800.0/grid_res;
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Verification masking regions
> > > //
> > > mask = {
> > >    grid      = "";
> > >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
> > >    poly      = "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> > >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> > > };
> > >
> > > //
> > > // Fuzzy engine weights
> > > //
> > > weight = {
> > >    centroid_dist    = 2.0;
> > >    boundary_dist    = 4.0;
> > >    convex_hull_dist = 0.0;
> > >    angle_diff       = 1.0;
> > >    area_ratio       = 1.0;
> > >    int_area_ratio   = 2.0;
> > >    complexity_ratio = 0.0;
> > >    inten_perc_ratio = 0.0;
> > >    inten_perc_value = 100;
> > > }
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Fuzzy engine interest functions
> > > //
> > > interest_function = {
> > >
> > >    centroid_dist = (
> > >       (            0.0, 1.0 )
> > >       (  60.0/grid_res, 1.0 )
> > >       ( 600.0/grid_res, 0.0 )
> > >    );
> > >
> > >    boundary_dist = (
> > >       (            0.0, 1.0 )
> > >       ( 400.0/grid_res, 0.0 )
> > >    );
> > >
> > >    convex_hull_dist = (
> > >       (            0.0, 1.0 )
> > >       ( 400.0/grid_res, 0.0 )
> > >    );
> > >
> > >    angle_diff = (
> > >       (  0.0, 1.0 )
> > >       ( 30.0, 1.0 )
> > >       ( 90.0, 0.0 )
> > >    corner   = 0.8;
> > >    ratio_if = (
> > >       (    0.0, 0.0 )
> > >       ( corner, 1.0 )
> > >       (    1.0, 1.0 )
> > >    );
> > >
> > >    area_ratio = ratio_if;
> > >
> > >    int_area_ratio = (
> > >       ( 0.00, 0.00 )
> > >       ( 0.10, 0.50 )
> > >       ( 0.25, 1.00 )
> > >       ( 1.00, 1.00 )
> > >    );
> > >
> > >    complexity_ratio = ratio_if;
> > >
> > >    inten_perc_ratio = ratio_if;
> > > }
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Total interest threshold for determining matches
> > > //
> > > total_interest_thresh = 0.5;
> > >
> > > //
> > > // Interest threshold for printing output pair information
> > > //
> > > print_interest_thresh = 0.0;
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Plotting information
> > > //
> > > met_data_dir = "MET_BASE/data";
> > >
> > > fcst_raw_plot = {
> > >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> > >    plot_min         = 0.0;
> > >    plot_max         = 0.0;
> > >    colorbar_spacing = 1;
> > > };
> > >
> > > obs_raw_plot = {
> > >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> > >    plot_min         = 0.0;
> > >    plot_max         = 0.0;
> > >    colorbar_spacing = 1;
> > > };
> > >
> > > object_plot = {
> > >    color_table      =
"MET_BASE/data/colortables/mode_obj.ctable";
> > > };
> > >
> > > //
> > > // Number of grid boxes to fill with bad data values along the
edge of
> > the
> > > field
> > > // to avoid edge effects.
> > > //
> > > zero_border_size = 1;
> > >
> > > //
> > > // Boolean for plotting on the region of valid data within the
domain
> > > //
> > > plot_valid_flag = FALSE;
> > >
> > > //
> > > // Plot polyline edges using great circle arcs instead of
straight
> lines
> > > //
> > > plot_gcarc_flag = FALSE;
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // NetCDF matched pairs, PostScript, and contingency table
output files
> > > //
> > > ps_plot_flag    = TRUE;
> > > nc_pairs_flag   = TRUE;
> > > ct_stats_flag   = TRUE;
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > output_prefix  = "";
> > > version        = "V4.1";
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > >
> > > ________________________________
> > >
> > > This email is intended solely for the recipient. It may contain
> > > privileged, proprietary or confidential information or material.
If you
> > are
> > > not the intended recipient, please delete this email and any
> attachments
> > > and notify the sender of the error.
> > >
> > >
> >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material.
If you
> are
> > not the intended recipient, please delete this email and any
attachments
> > and notify the sender of the error.
> >
> >
> >
> >
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #71629] problems with MODE not identifying objects in field
From: Adams-Selin, Becky
Time: Mon Apr 27 12:07:56 2015

Wow, yes, that would do it.  I didn't realize the forecasted/observed
plots on the first page of the output .ps file had different color
scales!  Thank you for the help and the suggested Mode config file.

If I were to report on these results obtained using a different
conv_threshold, would I just need to acknowledge a pre-existing bias?
The problem is complicated by the fact that I am verifying the
forecasts of multiple ensemble members, not all of whom are so biased.

Becky Adams-Selin, Ph.D.
Staff Scientist II, AER

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Monday, April 27, 2015 12:03 PM
To: Adams-Selin, Becky
Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying objects in field

Becky,

Thanks for sending the data.  I ran the following two plot_data_plane
(from
MET) commands to plot this data using the same scale of 0 to 1.25. The
resulting images are attached:

plot_data_plane \
   HailSize20140508L1S_caps_maxnn_2.0km.nc \
   HailSize20140508L1S_caps_maxnn_2.0km.ps \
   'name="MESH_hail"; level="(*,*)";' -plot_range 0 1.25

plot_data_plane \
   s4m17_arw_140508_hailsum.nc \
   s4m17_arw_140508_hailsum.ps \
   'name="MAX_HAIL_24"; level="(*,*)";' -plot_range 0 1.25

Looking at them, I notice that the observed values are much higher
than the
forecast values!  Very few forecast values are above 1.0 while most
observed values are above 1.0.  When you apply a raw thresh of >=1.0,
you're zeroing out almost all of the forecast points but relatively
few
observation points.  That's why you're getting so many more
observation
objects than forecast objects.

Inspecting the data more closely, I see...
 - the range on non-zero observed values is 0.50 to 3.74
 - the range of non-zero forecast values is 0.01 to 1.39

Based on that, I made the following changes to the config file...
 - switch the raw_thresh from >=1.0 to >=0.5
 - increase the smoothing radius (conv_radius) from 1 to 4
 - set the plot_max in the fcst_raw_plot and obs_raw_plot sections to
3.75
to use a consistent scale
 - set the forecast conv_thresh as >=0.25 and observed conv_thresh as
>=0.5.

This resulted in 45 forecast and 44 observed objects.  We typically
use the
same threshold for both forecast and observation, but your forecast
has a
very clear low bias, by about a factor of 2.  That's why I chose the
observed threshold to be 2 times the forecast threshold.

The config file and resulting output image is attached.
Unfortunately,
this takes a long time to run, about 5 min on my machine!

I'd suggest that you play around with the settings more.  You should
decide
what "scale" of objects you're looking for.  If you want a smaller
number
of smoother objects, try increasing the convolution radius and
lowering the
threshold.  The fewer the number of objects MODE has to keep track of,
the
faster it'll run.

Hope that helps.

Thanks,
John


On Sat, Apr 25, 2015 at 9:19 AM, Adams-Selin, Becky via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
> I tried lowering the conv_radius to 0, and the conv_threshold to
0.25.  I
> attached what I got.  I'm new to MODE, but just from visually
comparing the
> forecast and observation fields, I feel like I should be getting
roughly
> similar numbers of objects....I would even have expected fewer
objects from
> the observation field.  Something seems wrong.
>
> I went ahead and created a selin_data directory on the passive ftp
site.
>
> The forecast file is s4m17_arw_140508_hailsum.nc
> The observation file is HailSize20140508L1S_caps_maxnn_2.0km.nc
> And the mode config file is MODEConfig_bigintense
>
> If you could take a look I would really appreciate it.
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Saturday, April 25, 2015 9:25 AM
> To: Adams-Selin, Becky
> Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> objects in field
>
> Becky,
>
> OK, sounds good.  You could try setting the convolution radius to 0
just to
> see if you can get forecast objects.  Or try lowering the threshold.
> Perhaps they're all getting smoothed out.
>
> If you're still stuck I could take a look at some sample files.  I'd
need a
> sample forecast, observation, and MODE config file.  You could post
them to
> our anonymous for site following these instructions:
>
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks
> John
>
>
> On Saturday, April 25, 2015, Adams-Selin, Becky via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> >
> > John,
> >
> > I think that situation actually might apply to my dataset.  The
> > observational dataset only has hail values that are larger than
1.0 in.
> > The forecast dataset is continuous, from 0 inches upwards.  My
> > understanding was I should set the raw_thresh to 1.0 to make the
forecast
> > and observational datasets consistent, like you said below.
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu <javascript:;>]
> > Sent: Friday, April 24, 2015 3:39 PM
> > To: Adams-Selin, Becky
> > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> > objects in field
> >
> > Hello Becky,
> >
> > I see that you're having trouble getting objects to show up in
MODE.
> > Thanks for sending the sample plot and your configuration file.
Looking
> > closely at your choice of thresholds, I see a problem:
> >    raw_thresh        = >=1.0;
> >    conv_thresh       = >=0.5;
> >
> > I would suggest changing it to this:
> >    raw_thresh        = >=0.0;
> >    conv_thresh       = >=0.5;
> >
> > Try rerunning to see if you get objects.
> >
> > The "raw_thresh" setting should generally not be used unless you
have a
> > good reason to do so.  It is applied to the raw data values and
any point
> > not meeting that criteria is set to a value of 0.  The resulting
values
> are
> > passed through to the object definition step.
> >
> > Applying that logic doesn't make sense for your dataset.  You may
wonder
> > why you would you ever want to do this?  A while back we were
evaluating
> a
> > reflectivity forecast, but the system only forecast values of
35dBz or
> > higher.  So we had values of 35dBz surround by values of 0.  The
> > observations were continuous, with values of 35dBZ surrounded by
34dBZ
> and
> > so on.  When defining objects in MODE, we do a smoothing step and
having
> > values of 0 vs values of 34 affected the output a lot.  So in
order to
> > define observation and forecast objects in a consistent way, we
zeroed
> out
> > any observed values < 35.
> >
> > Since this doesn't apply to your dataset, setting it as ">=0"
should pass
> > all of your data through to the object definition step.
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John Halley Gotway
> > met_help at ucar.edu <javascript:;>
> >
> >
> > On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
> > met_help at ucar.edu <javascript:;>> wrote:
> >
> > >
> > > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > > Transaction: Ticket created by RSelin at aer.com <javascript:;>
> > >        Queue: met_help
> > >      Subject: problems with MODE not identifying objects in
field
> > >        Owner: Nobody
> > >   Requestors: RSelin at aer.com <javascript:;>
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> >
> > >
> > >
> > > Dear MET help,
> > >
> > > I am using MODE in MET v4.1 to evaluate a custom hail forecast
in
> > > comparison to a gridded observed hail field (MESH).  I've
attached the
> > > first page of the .ps output from one of my dates.  For some
reason no
> > > objects are being identified in the forecast field, but judging
from
> the
> > > raw field, they should be.  My MODEConfig file is below.  Any
idea what
> > I'm
> > > doing wrong?
> > >
> > > Thanks,
> > >
> > > Becky Adams-Selin, Ph.D.
> > > Staff Scientist II, AER
> > >
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > //
> > > // MODE configuration file.
> > > //
> > > // For additional information, see the
MET_BASE/data/config/README
> file.
> > > //
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Output model name to be written
> > > //
> > > model = "WRF";
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Approximate grid resolution (km)
> > > //
> > > grid_res = 4;
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Forecast and observation fields to be verified
> > > //
> > > fcst = {
> > >    field = {
> > >       name  = "MAX_HAIL_24";
> > >       level = "(*,*)";
> > >    };
> > >
> > >    raw_thresh        = >=1.0;
> > >    conv_radius       = 4; // in grid squares
> > >    conv_thresh       = >=0.5;
> > >    vld_thresh        = 0.5;
> > >    area_thresh       = >=4.0;
> > >    inten_perc_value  = 100;
> > >    inten_perc_thresh = >=0.0;
> > >    merge_thresh      = >=0.5;
> > >    merge_flag        = NONE;
> > > };
> > > obs = {
> > >    field = {
> > >       name = "MESH_hail";
> > >       level = "(*,*)";
> > >    };
> > >
> > >    raw_thresh        = >=1.0;
> > >    conv_radius       = 4; // in grid squares
> > >    conv_thresh       = >=0.5;
> > >    vld_thresh        = 0.5;
> > >    area_thresh       = >=4.0;
> > >    inten_perc_value  = 100;
> > >    inten_perc_thresh = >=0.0;
> > >    merge_thresh      = >=0.5;
> > >    merge_flag        = NONE;
> > > };
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Handle missing data
> > > //
> > > mask_missing_flag = BOTH;
> > >
> > > //
> > > // Match objects between the forecast and observation fields
> > > //
> > > match_flag = MERGE_BOTH;
> > >
> > > //
> > > // Maximum centroid distance for objects to be compared
> > > //
> > > max_centroid_dist = 800.0/grid_res;
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Verification masking regions
> > > //
> > > mask = {
> > >    grid      = "";
> > >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
> > >    poly      = "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> > >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> > > };
> > >
> > > //
> > > // Fuzzy engine weights
> > > //
> > > weight = {
> > >    centroid_dist    = 2.0;
> > >    boundary_dist    = 4.0;
> > >    convex_hull_dist = 0.0;
> > >    angle_diff       = 1.0;
> > >    area_ratio       = 1.0;
> > >    int_area_ratio   = 2.0;
> > >    complexity_ratio = 0.0;
> > >    inten_perc_ratio = 0.0;
> > >    inten_perc_value = 100;
> > > }
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Fuzzy engine interest functions
> > > //
> > > interest_function = {
> > >
> > >    centroid_dist = (
> > >       (            0.0, 1.0 )
> > >       (  60.0/grid_res, 1.0 )
> > >       ( 600.0/grid_res, 0.0 )
> > >    );
> > >
> > >    boundary_dist = (
> > >       (            0.0, 1.0 )
> > >       ( 400.0/grid_res, 0.0 )
> > >    );
> > >
> > >    convex_hull_dist = (
> > >       (            0.0, 1.0 )
> > >       ( 400.0/grid_res, 0.0 )
> > >    );
> > >
> > >    angle_diff = (
> > >       (  0.0, 1.0 )
> > >       ( 30.0, 1.0 )
> > >       ( 90.0, 0.0 )
> > >    corner   = 0.8;
> > >    ratio_if = (
> > >       (    0.0, 0.0 )
> > >       ( corner, 1.0 )
> > >       (    1.0, 1.0 )
> > >    );
> > >
> > >    area_ratio = ratio_if;
> > >
> > >    int_area_ratio = (
> > >       ( 0.00, 0.00 )
> > >       ( 0.10, 0.50 )
> > >       ( 0.25, 1.00 )
> > >       ( 1.00, 1.00 )
> > >    );
> > >
> > >    complexity_ratio = ratio_if;
> > >
> > >    inten_perc_ratio = ratio_if;
> > > }
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Total interest threshold for determining matches
> > > //
> > > total_interest_thresh = 0.5;
> > >
> > > //
> > > // Interest threshold for printing output pair information
> > > //
> > > print_interest_thresh = 0.0;
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // Plotting information
> > > //
> > > met_data_dir = "MET_BASE/data";
> > >
> > > fcst_raw_plot = {
> > >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> > >    plot_min         = 0.0;
> > >    plot_max         = 0.0;
> > >    colorbar_spacing = 1;
> > > };
> > >
> > > obs_raw_plot = {
> > >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> > >    plot_min         = 0.0;
> > >    plot_max         = 0.0;
> > >    colorbar_spacing = 1;
> > > };
> > >
> > > object_plot = {
> > >    color_table      =
"MET_BASE/data/colortables/mode_obj.ctable";
> > > };
> > >
> > > //
> > > // Number of grid boxes to fill with bad data values along the
edge of
> > the
> > > field
> > > // to avoid edge effects.
> > > //
> > > zero_border_size = 1;
> > >
> > > //
> > > // Boolean for plotting on the region of valid data within the
domain
> > > //
> > > plot_valid_flag = FALSE;
> > >
> > > //
> > > // Plot polyline edges using great circle arcs instead of
straight
> lines
> > > //
> > > plot_gcarc_flag = FALSE;
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > //
> > > // NetCDF matched pairs, PostScript, and contingency table
output files
> > > //
> > > ps_plot_flag    = TRUE;
> > > nc_pairs_flag   = TRUE;
> > > ct_stats_flag   = TRUE;
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > > output_prefix  = "";
> > > version        = "V4.1";
> > >
> > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > >
> > >
> > > ________________________________
> > >
> > > This email is intended solely for the recipient. It may contain
> > > privileged, proprietary or confidential information or material.
If you
> > are
> > > not the intended recipient, please delete this email and any
> attachments
> > > and notify the sender of the error.
> > >
> > >
> >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material.
If you
> are
> > not the intended recipient, please delete this email and any
attachments
> > and notify the sender of the error.
> >
> >
> >
> >
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>


________________________________

This email is intended solely for the recipient. It may contain
privileged, proprietary or confidential information or material. If
you are not the intended recipient, please delete this email and any
attachments and notify the sender of the error.



------------------------------------------------
Subject: problems with MODE not identifying objects in field
From: John Halley Gotway
Time: Mon Apr 27 12:33:23 2015

Becky,

MODE attempts to do things intelligently here.  If the forecast and
observation field names are identical, it chooses a consistent color
scale
using the range of both.  Your data is not named the same thing
(MAX_HAIL_24 vs MESH_hail) so it plots them using the range of the
data in
each.

As for interpreting the output, ultimately it's up to you.  You
mention
that you're dealing with an ensemble with varying biases.  I would
suggest
finding a configuration of MODE that identifies the objects of
interest and
helps you answer your scientific questions... and then apply the same
configuration to all members.  Changing the configuration for each
member
would be pretty confusing.

I can help you with the nuts and bolts of running MODE, but one of the
scientists here would be better at helping you interpret results.
Before
presenting results, we could probably to take a quick look to make
sure we
don't see any red flags if you'd like.

Thanks,
John



On Mon, Apr 27, 2015 at 12:07 PM, Adams-Selin, Becky via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
> Wow, yes, that would do it.  I didn't realize the
forecasted/observed
> plots on the first page of the output .ps file had different color
scales!
> Thank you for the help and the suggested Mode config file.
>
> If I were to report on these results obtained using a different
> conv_threshold, would I just need to acknowledge a pre-existing
bias?  The
> problem is complicated by the fact that I am verifying the forecasts
of
> multiple ensemble members, not all of whom are so biased.
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Monday, April 27, 2015 12:03 PM
> To: Adams-Selin, Becky
> Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> objects in field
>
> Becky,
>
> Thanks for sending the data.  I ran the following two
plot_data_plane (from
> MET) commands to plot this data using the same scale of 0 to 1.25.
The
> resulting images are attached:
>
> plot_data_plane \
>    HailSize20140508L1S_caps_maxnn_2.0km.nc \
>    HailSize20140508L1S_caps_maxnn_2.0km.ps \
>    'name="MESH_hail"; level="(*,*)";' -plot_range 0 1.25
>
> plot_data_plane \
>    s4m17_arw_140508_hailsum.nc \
>    s4m17_arw_140508_hailsum.ps \
>    'name="MAX_HAIL_24"; level="(*,*)";' -plot_range 0 1.25
>
> Looking at them, I notice that the observed values are much higher
than the
> forecast values!  Very few forecast values are above 1.0 while most
> observed values are above 1.0.  When you apply a raw thresh of
>=1.0,
> you're zeroing out almost all of the forecast points but relatively
few
> observation points.  That's why you're getting so many more
observation
> objects than forecast objects.
>
> Inspecting the data more closely, I see...
>  - the range on non-zero observed values is 0.50 to 3.74
>  - the range of non-zero forecast values is 0.01 to 1.39
>
> Based on that, I made the following changes to the config file...
>  - switch the raw_thresh from >=1.0 to >=0.5
>  - increase the smoothing radius (conv_radius) from 1 to 4
>  - set the plot_max in the fcst_raw_plot and obs_raw_plot sections
to 3.75
> to use a consistent scale
>  - set the forecast conv_thresh as >=0.25 and observed conv_thresh
as
> >=0.5.
>
> This resulted in 45 forecast and 44 observed objects.  We typically
use the
> same threshold for both forecast and observation, but your forecast
has a
> very clear low bias, by about a factor of 2.  That's why I chose the
> observed threshold to be 2 times the forecast threshold.
>
> The config file and resulting output image is attached.
Unfortunately,
> this takes a long time to run, about 5 min on my machine!
>
> I'd suggest that you play around with the settings more.  You should
decide
> what "scale" of objects you're looking for.  If you want a smaller
number
> of smoother objects, try increasing the convolution radius and
lowering the
> threshold.  The fewer the number of objects MODE has to keep track
of, the
> faster it'll run.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> On Sat, Apr 25, 2015 at 9:19 AM, Adams-Selin, Becky via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> >
> > I tried lowering the conv_radius to 0, and the conv_threshold to
0.25.  I
> > attached what I got.  I'm new to MODE, but just from visually
comparing
> the
> > forecast and observation fields, I feel like I should be getting
roughly
> > similar numbers of objects....I would even have expected fewer
objects
> from
> > the observation field.  Something seems wrong.
> >
> > I went ahead and created a selin_data directory on the passive ftp
site.
> >
> > The forecast file is s4m17_arw_140508_hailsum.nc
> > The observation file is HailSize20140508L1S_caps_maxnn_2.0km.nc
> > And the mode config file is MODEConfig_bigintense
> >
> > If you could take a look I would really appreciate it.
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Saturday, April 25, 2015 9:25 AM
> > To: Adams-Selin, Becky
> > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> > objects in field
> >
> > Becky,
> >
> > OK, sounds good.  You could try setting the convolution radius to
0 just
> to
> > see if you can get forecast objects.  Or try lowering the
threshold.
> > Perhaps they're all getting smoothed out.
> >
> > If you're still stuck I could take a look at some sample files.
I'd
> need a
> > sample forecast, observation, and MODE config file.  You could
post them
> to
> > our anonymous for site following these instructions:
> >
> > http://www.dtcenter.org/met/users/support/met_help.php#ftp
> >
> > Thanks
> > John
> >
> >
> > On Saturday, April 25, 2015, Adams-Selin, Becky via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> > >
> > > John,
> > >
> > > I think that situation actually might apply to my dataset.  The
> > > observational dataset only has hail values that are larger than
1.0 in.
> > > The forecast dataset is continuous, from 0 inches upwards.  My
> > > understanding was I should set the raw_thresh to 1.0 to make the
> forecast
> > > and observational datasets consistent, like you said below.
> > >
> > > Becky Adams-Selin, Ph.D.
> > > Staff Scientist II, AER
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu
<javascript:;>]
> > > Sent: Friday, April 24, 2015 3:39 PM
> > > To: Adams-Selin, Becky
> > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> identifying
> > > objects in field
> > >
> > > Hello Becky,
> > >
> > > I see that you're having trouble getting objects to show up in
MODE.
> > > Thanks for sending the sample plot and your configuration file.
> Looking
> > > closely at your choice of thresholds, I see a problem:
> > >    raw_thresh        = >=1.0;
> > >    conv_thresh       = >=0.5;
> > >
> > > I would suggest changing it to this:
> > >    raw_thresh        = >=0.0;
> > >    conv_thresh       = >=0.5;
> > >
> > > Try rerunning to see if you get objects.
> > >
> > > The "raw_thresh" setting should generally not be used unless you
have a
> > > good reason to do so.  It is applied to the raw data values and
any
> point
> > > not meeting that criteria is set to a value of 0.  The resulting
values
> > are
> > > passed through to the object definition step.
> > >
> > > Applying that logic doesn't make sense for your dataset.  You
may
> wonder
> > > why you would you ever want to do this?  A while back we were
> evaluating
> > a
> > > reflectivity forecast, but the system only forecast values of
35dBz or
> > > higher.  So we had values of 35dBz surround by values of 0.  The
> > > observations were continuous, with values of 35dBZ surrounded by
34dBZ
> > and
> > > so on.  When defining objects in MODE, we do a smoothing step
and
> having
> > > values of 0 vs values of 34 affected the output a lot.  So in
order to
> > > define observation and forecast objects in a consistent way, we
zeroed
> > out
> > > any observed values < 35.
> > >
> > > Since this doesn't apply to your dataset, setting it as ">=0"
should
> pass
> > > all of your data through to the object definition step.
> > >
> > > Hope that helps clarify.
> > >
> > > Thanks,
> > > John Halley Gotway
> > > met_help at ucar.edu <javascript:;>
> > >
> > >
> > > On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
> > > met_help at ucar.edu <javascript:;>> wrote:
> > >
> > > >
> > > > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > > > Transaction: Ticket created by RSelin at aer.com <javascript:;>
> > > >        Queue: met_help
> > > >      Subject: problems with MODE not identifying objects in
field
> > > >        Owner: Nobody
> > > >   Requestors: RSelin at aer.com <javascript:;>
> > > >       Status: new
> > > >  Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> > >
> > > >
> > > >
> > > > Dear MET help,
> > > >
> > > > I am using MODE in MET v4.1 to evaluate a custom hail forecast
in
> > > > comparison to a gridded observed hail field (MESH).  I've
attached
> the
> > > > first page of the .ps output from one of my dates.  For some
reason
> no
> > > > objects are being identified in the forecast field, but
judging from
> > the
> > > > raw field, they should be.  My MODEConfig file is below.  Any
idea
> what
> > > I'm
> > > > doing wrong?
> > > >
> > > > Thanks,
> > > >
> > > > Becky Adams-Selin, Ph.D.
> > > > Staff Scientist II, AER
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > //
> > > > // MODE configuration file.
> > > > //
> > > > // For additional information, see the
MET_BASE/data/config/README
> > file.
> > > > //
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Output model name to be written
> > > > //
> > > > model = "WRF";
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Approximate grid resolution (km)
> > > > //
> > > > grid_res = 4;
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Forecast and observation fields to be verified
> > > > //
> > > > fcst = {
> > > >    field = {
> > > >       name  = "MAX_HAIL_24";
> > > >       level = "(*,*)";
> > > >    };
> > > >
> > > >    raw_thresh        = >=1.0;
> > > >    conv_radius       = 4; // in grid squares
> > > >    conv_thresh       = >=0.5;
> > > >    vld_thresh        = 0.5;
> > > >    area_thresh       = >=4.0;
> > > >    inten_perc_value  = 100;
> > > >    inten_perc_thresh = >=0.0;
> > > >    merge_thresh      = >=0.5;
> > > >    merge_flag        = NONE;
> > > > };
> > > > obs = {
> > > >    field = {
> > > >       name = "MESH_hail";
> > > >       level = "(*,*)";
> > > >    };
> > > >
> > > >    raw_thresh        = >=1.0;
> > > >    conv_radius       = 4; // in grid squares
> > > >    conv_thresh       = >=0.5;
> > > >    vld_thresh        = 0.5;
> > > >    area_thresh       = >=4.0;
> > > >    inten_perc_value  = 100;
> > > >    inten_perc_thresh = >=0.0;
> > > >    merge_thresh      = >=0.5;
> > > >    merge_flag        = NONE;
> > > > };
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Handle missing data
> > > > //
> > > > mask_missing_flag = BOTH;
> > > >
> > > > //
> > > > // Match objects between the forecast and observation fields
> > > > //
> > > > match_flag = MERGE_BOTH;
> > > >
> > > > //
> > > > // Maximum centroid distance for objects to be compared
> > > > //
> > > > max_centroid_dist = 800.0/grid_res;
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Verification masking regions
> > > > //
> > > > mask = {
> > > >    grid      = "";
> > > >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
> > > >    poly      =
"/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> > > >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> > > > };
> > > >
> > > > //
> > > > // Fuzzy engine weights
> > > > //
> > > > weight = {
> > > >    centroid_dist    = 2.0;
> > > >    boundary_dist    = 4.0;
> > > >    convex_hull_dist = 0.0;
> > > >    angle_diff       = 1.0;
> > > >    area_ratio       = 1.0;
> > > >    int_area_ratio   = 2.0;
> > > >    complexity_ratio = 0.0;
> > > >    inten_perc_ratio = 0.0;
> > > >    inten_perc_value = 100;
> > > > }
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Fuzzy engine interest functions
> > > > //
> > > > interest_function = {
> > > >
> > > >    centroid_dist = (
> > > >       (            0.0, 1.0 )
> > > >       (  60.0/grid_res, 1.0 )
> > > >       ( 600.0/grid_res, 0.0 )
> > > >    );
> > > >
> > > >    boundary_dist = (
> > > >       (            0.0, 1.0 )
> > > >       ( 400.0/grid_res, 0.0 )
> > > >    );
> > > >
> > > >    convex_hull_dist = (
> > > >       (            0.0, 1.0 )
> > > >       ( 400.0/grid_res, 0.0 )
> > > >    );
> > > >
> > > >    angle_diff = (
> > > >       (  0.0, 1.0 )
> > > >       ( 30.0, 1.0 )
> > > >       ( 90.0, 0.0 )
> > > >    corner   = 0.8;
> > > >    ratio_if = (
> > > >       (    0.0, 0.0 )
> > > >       ( corner, 1.0 )
> > > >       (    1.0, 1.0 )
> > > >    );
> > > >
> > > >    area_ratio = ratio_if;
> > > >
> > > >    int_area_ratio = (
> > > >       ( 0.00, 0.00 )
> > > >       ( 0.10, 0.50 )
> > > >       ( 0.25, 1.00 )
> > > >       ( 1.00, 1.00 )
> > > >    );
> > > >
> > > >    complexity_ratio = ratio_if;
> > > >
> > > >    inten_perc_ratio = ratio_if;
> > > > }
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Total interest threshold for determining matches
> > > > //
> > > > total_interest_thresh = 0.5;
> > > >
> > > > //
> > > > // Interest threshold for printing output pair information
> > > > //
> > > > print_interest_thresh = 0.0;
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Plotting information
> > > > //
> > > > met_data_dir = "MET_BASE/data";
> > > >
> > > > fcst_raw_plot = {
> > > >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> > > >    plot_min         = 0.0;
> > > >    plot_max         = 0.0;
> > > >    colorbar_spacing = 1;
> > > > };
> > > >
> > > > obs_raw_plot = {
> > > >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> > > >    plot_min         = 0.0;
> > > >    plot_max         = 0.0;
> > > >    colorbar_spacing = 1;
> > > > };
> > > >
> > > > object_plot = {
> > > >    color_table      =
"MET_BASE/data/colortables/mode_obj.ctable";
> > > > };
> > > >
> > > > //
> > > > // Number of grid boxes to fill with bad data values along the
edge
> of
> > > the
> > > > field
> > > > // to avoid edge effects.
> > > > //
> > > > zero_border_size = 1;
> > > >
> > > > //
> > > > // Boolean for plotting on the region of valid data within the
domain
> > > > //
> > > > plot_valid_flag = FALSE;
> > > >
> > > > //
> > > > // Plot polyline edges using great circle arcs instead of
straight
> > lines
> > > > //
> > > > plot_gcarc_flag = FALSE;
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // NetCDF matched pairs, PostScript, and contingency table
output
> files
> > > > //
> > > > ps_plot_flag    = TRUE;
> > > > nc_pairs_flag   = TRUE;
> > > > ct_stats_flag   = TRUE;
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > output_prefix  = "";
> > > > version        = "V4.1";
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > >
> > > > ________________________________
> > > >
> > > > This email is intended solely for the recipient. It may
contain
> > > > privileged, proprietary or confidential information or
material. If
> you
> > > are
> > > > not the intended recipient, please delete this email and any
> > attachments
> > > > and notify the sender of the error.
> > > >
> > > >
> > >
> > >
> > > ________________________________
> > >
> > > This email is intended solely for the recipient. It may contain
> > > privileged, proprietary or confidential information or material.
If you
> > are
> > > not the intended recipient, please delete this email and any
> attachments
> > > and notify the sender of the error.
> > >
> > >
> > >
> > >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material.
If you
> are
> > not the intended recipient, please delete this email and any
attachments
> > and notify the sender of the error.
> >
> >
>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #71629] problems with MODE not identifying objects in field
From: Adams-Selin, Becky
Time: Fri May 01 10:03:24 2015

John,

OK, that makes sense, thank you.

Like you say, ideally I'd like my configuration file to be the same
for all my forecast members.  However, because each member is biased
differently, setting a conv_threshold value that makes sense for one
member, results in really large forecast clusters being identified in
another member.  Is there a good way to limit the size of the cluster
objects without changing the conv_threshold?

Becky Adams-Selin, Ph.D.
Staff Scientist II, AER

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Monday, April 27, 2015 1:33 PM
To: Adams-Selin, Becky
Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying objects in field

Becky,

MODE attempts to do things intelligently here.  If the forecast and
observation field names are identical, it chooses a consistent color
scale
using the range of both.  Your data is not named the same thing
(MAX_HAIL_24 vs MESH_hail) so it plots them using the range of the
data in
each.

As for interpreting the output, ultimately it's up to you.  You
mention
that you're dealing with an ensemble with varying biases.  I would
suggest
finding a configuration of MODE that identifies the objects of
interest and
helps you answer your scientific questions... and then apply the same
configuration to all members.  Changing the configuration for each
member
would be pretty confusing.

I can help you with the nuts and bolts of running MODE, but one of the
scientists here would be better at helping you interpret results.
Before
presenting results, we could probably to take a quick look to make
sure we
don't see any red flags if you'd like.

Thanks,
John



On Mon, Apr 27, 2015 at 12:07 PM, Adams-Selin, Becky via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
> Wow, yes, that would do it.  I didn't realize the
forecasted/observed
> plots on the first page of the output .ps file had different color
scales!
> Thank you for the help and the suggested Mode config file.
>
> If I were to report on these results obtained using a different
> conv_threshold, would I just need to acknowledge a pre-existing
bias?  The
> problem is complicated by the fact that I am verifying the forecasts
of
> multiple ensemble members, not all of whom are so biased.
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Monday, April 27, 2015 12:03 PM
> To: Adams-Selin, Becky
> Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> objects in field
>
> Becky,
>
> Thanks for sending the data.  I ran the following two
plot_data_plane (from
> MET) commands to plot this data using the same scale of 0 to 1.25.
The
> resulting images are attached:
>
> plot_data_plane \
>    HailSize20140508L1S_caps_maxnn_2.0km.nc \
>    HailSize20140508L1S_caps_maxnn_2.0km.ps \
>    'name="MESH_hail"; level="(*,*)";' -plot_range 0 1.25
>
> plot_data_plane \
>    s4m17_arw_140508_hailsum.nc \
>    s4m17_arw_140508_hailsum.ps \
>    'name="MAX_HAIL_24"; level="(*,*)";' -plot_range 0 1.25
>
> Looking at them, I notice that the observed values are much higher
than the
> forecast values!  Very few forecast values are above 1.0 while most
> observed values are above 1.0.  When you apply a raw thresh of
>=1.0,
> you're zeroing out almost all of the forecast points but relatively
few
> observation points.  That's why you're getting so many more
observation
> objects than forecast objects.
>
> Inspecting the data more closely, I see...
>  - the range on non-zero observed values is 0.50 to 3.74
>  - the range of non-zero forecast values is 0.01 to 1.39
>
> Based on that, I made the following changes to the config file...
>  - switch the raw_thresh from >=1.0 to >=0.5
>  - increase the smoothing radius (conv_radius) from 1 to 4
>  - set the plot_max in the fcst_raw_plot and obs_raw_plot sections
to 3.75
> to use a consistent scale
>  - set the forecast conv_thresh as >=0.25 and observed conv_thresh
as
> >=0.5.
>
> This resulted in 45 forecast and 44 observed objects.  We typically
use the
> same threshold for both forecast and observation, but your forecast
has a
> very clear low bias, by about a factor of 2.  That's why I chose the
> observed threshold to be 2 times the forecast threshold.
>
> The config file and resulting output image is attached.
Unfortunately,
> this takes a long time to run, about 5 min on my machine!
>
> I'd suggest that you play around with the settings more.  You should
decide
> what "scale" of objects you're looking for.  If you want a smaller
number
> of smoother objects, try increasing the convolution radius and
lowering the
> threshold.  The fewer the number of objects MODE has to keep track
of, the
> faster it'll run.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> On Sat, Apr 25, 2015 at 9:19 AM, Adams-Selin, Becky via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> >
> > I tried lowering the conv_radius to 0, and the conv_threshold to
0.25.  I
> > attached what I got.  I'm new to MODE, but just from visually
comparing
> the
> > forecast and observation fields, I feel like I should be getting
roughly
> > similar numbers of objects....I would even have expected fewer
objects
> from
> > the observation field.  Something seems wrong.
> >
> > I went ahead and created a selin_data directory on the passive ftp
site.
> >
> > The forecast file is s4m17_arw_140508_hailsum.nc
> > The observation file is HailSize20140508L1S_caps_maxnn_2.0km.nc
> > And the mode config file is MODEConfig_bigintense
> >
> > If you could take a look I would really appreciate it.
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Saturday, April 25, 2015 9:25 AM
> > To: Adams-Selin, Becky
> > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> > objects in field
> >
> > Becky,
> >
> > OK, sounds good.  You could try setting the convolution radius to
0 just
> to
> > see if you can get forecast objects.  Or try lowering the
threshold.
> > Perhaps they're all getting smoothed out.
> >
> > If you're still stuck I could take a look at some sample files.
I'd
> need a
> > sample forecast, observation, and MODE config file.  You could
post them
> to
> > our anonymous for site following these instructions:
> >
> > http://www.dtcenter.org/met/users/support/met_help.php#ftp
> >
> > Thanks
> > John
> >
> >
> > On Saturday, April 25, 2015, Adams-Selin, Becky via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> > >
> > > John,
> > >
> > > I think that situation actually might apply to my dataset.  The
> > > observational dataset only has hail values that are larger than
1.0 in.
> > > The forecast dataset is continuous, from 0 inches upwards.  My
> > > understanding was I should set the raw_thresh to 1.0 to make the
> forecast
> > > and observational datasets consistent, like you said below.
> > >
> > > Becky Adams-Selin, Ph.D.
> > > Staff Scientist II, AER
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu
<javascript:;>]
> > > Sent: Friday, April 24, 2015 3:39 PM
> > > To: Adams-Selin, Becky
> > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> identifying
> > > objects in field
> > >
> > > Hello Becky,
> > >
> > > I see that you're having trouble getting objects to show up in
MODE.
> > > Thanks for sending the sample plot and your configuration file.
> Looking
> > > closely at your choice of thresholds, I see a problem:
> > >    raw_thresh        = >=1.0;
> > >    conv_thresh       = >=0.5;
> > >
> > > I would suggest changing it to this:
> > >    raw_thresh        = >=0.0;
> > >    conv_thresh       = >=0.5;
> > >
> > > Try rerunning to see if you get objects.
> > >
> > > The "raw_thresh" setting should generally not be used unless you
have a
> > > good reason to do so.  It is applied to the raw data values and
any
> point
> > > not meeting that criteria is set to a value of 0.  The resulting
values
> > are
> > > passed through to the object definition step.
> > >
> > > Applying that logic doesn't make sense for your dataset.  You
may
> wonder
> > > why you would you ever want to do this?  A while back we were
> evaluating
> > a
> > > reflectivity forecast, but the system only forecast values of
35dBz or
> > > higher.  So we had values of 35dBz surround by values of 0.  The
> > > observations were continuous, with values of 35dBZ surrounded by
34dBZ
> > and
> > > so on.  When defining objects in MODE, we do a smoothing step
and
> having
> > > values of 0 vs values of 34 affected the output a lot.  So in
order to
> > > define observation and forecast objects in a consistent way, we
zeroed
> > out
> > > any observed values < 35.
> > >
> > > Since this doesn't apply to your dataset, setting it as ">=0"
should
> pass
> > > all of your data through to the object definition step.
> > >
> > > Hope that helps clarify.
> > >
> > > Thanks,
> > > John Halley Gotway
> > > met_help at ucar.edu <javascript:;>
> > >
> > >
> > > On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
> > > met_help at ucar.edu <javascript:;>> wrote:
> > >
> > > >
> > > > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > > > Transaction: Ticket created by RSelin at aer.com <javascript:;>
> > > >        Queue: met_help
> > > >      Subject: problems with MODE not identifying objects in
field
> > > >        Owner: Nobody
> > > >   Requestors: RSelin at aer.com <javascript:;>
> > > >       Status: new
> > > >  Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> > >
> > > >
> > > >
> > > > Dear MET help,
> > > >
> > > > I am using MODE in MET v4.1 to evaluate a custom hail forecast
in
> > > > comparison to a gridded observed hail field (MESH).  I've
attached
> the
> > > > first page of the .ps output from one of my dates.  For some
reason
> no
> > > > objects are being identified in the forecast field, but
judging from
> > the
> > > > raw field, they should be.  My MODEConfig file is below.  Any
idea
> what
> > > I'm
> > > > doing wrong?
> > > >
> > > > Thanks,
> > > >
> > > > Becky Adams-Selin, Ph.D.
> > > > Staff Scientist II, AER
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > //
> > > > // MODE configuration file.
> > > > //
> > > > // For additional information, see the
MET_BASE/data/config/README
> > file.
> > > > //
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Output model name to be written
> > > > //
> > > > model = "WRF";
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Approximate grid resolution (km)
> > > > //
> > > > grid_res = 4;
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Forecast and observation fields to be verified
> > > > //
> > > > fcst = {
> > > >    field = {
> > > >       name  = "MAX_HAIL_24";
> > > >       level = "(*,*)";
> > > >    };
> > > >
> > > >    raw_thresh        = >=1.0;
> > > >    conv_radius       = 4; // in grid squares
> > > >    conv_thresh       = >=0.5;
> > > >    vld_thresh        = 0.5;
> > > >    area_thresh       = >=4.0;
> > > >    inten_perc_value  = 100;
> > > >    inten_perc_thresh = >=0.0;
> > > >    merge_thresh      = >=0.5;
> > > >    merge_flag        = NONE;
> > > > };
> > > > obs = {
> > > >    field = {
> > > >       name = "MESH_hail";
> > > >       level = "(*,*)";
> > > >    };
> > > >
> > > >    raw_thresh        = >=1.0;
> > > >    conv_radius       = 4; // in grid squares
> > > >    conv_thresh       = >=0.5;
> > > >    vld_thresh        = 0.5;
> > > >    area_thresh       = >=4.0;
> > > >    inten_perc_value  = 100;
> > > >    inten_perc_thresh = >=0.0;
> > > >    merge_thresh      = >=0.5;
> > > >    merge_flag        = NONE;
> > > > };
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Handle missing data
> > > > //
> > > > mask_missing_flag = BOTH;
> > > >
> > > > //
> > > > // Match objects between the forecast and observation fields
> > > > //
> > > > match_flag = MERGE_BOTH;
> > > >
> > > > //
> > > > // Maximum centroid distance for objects to be compared
> > > > //
> > > > max_centroid_dist = 800.0/grid_res;
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Verification masking regions
> > > > //
> > > > mask = {
> > > >    grid      = "";
> > > >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
> > > >    poly      =
"/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> > > >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> > > > };
> > > >
> > > > //
> > > > // Fuzzy engine weights
> > > > //
> > > > weight = {
> > > >    centroid_dist    = 2.0;
> > > >    boundary_dist    = 4.0;
> > > >    convex_hull_dist = 0.0;
> > > >    angle_diff       = 1.0;
> > > >    area_ratio       = 1.0;
> > > >    int_area_ratio   = 2.0;
> > > >    complexity_ratio = 0.0;
> > > >    inten_perc_ratio = 0.0;
> > > >    inten_perc_value = 100;
> > > > }
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Fuzzy engine interest functions
> > > > //
> > > > interest_function = {
> > > >
> > > >    centroid_dist = (
> > > >       (            0.0, 1.0 )
> > > >       (  60.0/grid_res, 1.0 )
> > > >       ( 600.0/grid_res, 0.0 )
> > > >    );
> > > >
> > > >    boundary_dist = (
> > > >       (            0.0, 1.0 )
> > > >       ( 400.0/grid_res, 0.0 )
> > > >    );
> > > >
> > > >    convex_hull_dist = (
> > > >       (            0.0, 1.0 )
> > > >       ( 400.0/grid_res, 0.0 )
> > > >    );
> > > >
> > > >    angle_diff = (
> > > >       (  0.0, 1.0 )
> > > >       ( 30.0, 1.0 )
> > > >       ( 90.0, 0.0 )
> > > >    corner   = 0.8;
> > > >    ratio_if = (
> > > >       (    0.0, 0.0 )
> > > >       ( corner, 1.0 )
> > > >       (    1.0, 1.0 )
> > > >    );
> > > >
> > > >    area_ratio = ratio_if;
> > > >
> > > >    int_area_ratio = (
> > > >       ( 0.00, 0.00 )
> > > >       ( 0.10, 0.50 )
> > > >       ( 0.25, 1.00 )
> > > >       ( 1.00, 1.00 )
> > > >    );
> > > >
> > > >    complexity_ratio = ratio_if;
> > > >
> > > >    inten_perc_ratio = ratio_if;
> > > > }
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Total interest threshold for determining matches
> > > > //
> > > > total_interest_thresh = 0.5;
> > > >
> > > > //
> > > > // Interest threshold for printing output pair information
> > > > //
> > > > print_interest_thresh = 0.0;
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // Plotting information
> > > > //
> > > > met_data_dir = "MET_BASE/data";
> > > >
> > > > fcst_raw_plot = {
> > > >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> > > >    plot_min         = 0.0;
> > > >    plot_max         = 0.0;
> > > >    colorbar_spacing = 1;
> > > > };
> > > >
> > > > obs_raw_plot = {
> > > >    color_table      =
"MET_BASE/data/colortables/met_default.ctable";
> > > >    plot_min         = 0.0;
> > > >    plot_max         = 0.0;
> > > >    colorbar_spacing = 1;
> > > > };
> > > >
> > > > object_plot = {
> > > >    color_table      =
"MET_BASE/data/colortables/mode_obj.ctable";
> > > > };
> > > >
> > > > //
> > > > // Number of grid boxes to fill with bad data values along the
edge
> of
> > > the
> > > > field
> > > > // to avoid edge effects.
> > > > //
> > > > zero_border_size = 1;
> > > >
> > > > //
> > > > // Boolean for plotting on the region of valid data within the
domain
> > > > //
> > > > plot_valid_flag = FALSE;
> > > >
> > > > //
> > > > // Plot polyline edges using great circle arcs instead of
straight
> > lines
> > > > //
> > > > plot_gcarc_flag = FALSE;
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > //
> > > > // NetCDF matched pairs, PostScript, and contingency table
output
> files
> > > > //
> > > > ps_plot_flag    = TRUE;
> > > > nc_pairs_flag   = TRUE;
> > > > ct_stats_flag   = TRUE;
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > > output_prefix  = "";
> > > > version        = "V4.1";
> > > >
> > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > >
> > > >
> > > > ________________________________
> > > >
> > > > This email is intended solely for the recipient. It may
contain
> > > > privileged, proprietary or confidential information or
material. If
> you
> > > are
> > > > not the intended recipient, please delete this email and any
> > attachments
> > > > and notify the sender of the error.
> > > >
> > > >
> > >
> > >
> > > ________________________________
> > >
> > > This email is intended solely for the recipient. It may contain
> > > privileged, proprietary or confidential information or material.
If you
> > are
> > > not the intended recipient, please delete this email and any
> attachments
> > > and notify the sender of the error.
> > >
> > >
> > >
> > >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material.
If you
> are
> > not the intended recipient, please delete this email and any
attachments
> > and notify the sender of the error.
> >
> >
>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>
>
>


________________________________

This email is intended solely for the recipient. It may contain
privileged, proprietary or confidential information or material. If
you are not the intended recipient, please delete this email and any
attachments and notify the sender of the error.



------------------------------------------------
Subject: problems with MODE not identifying objects in field
From: John Halley Gotway
Time: Fri May 01 14:23:45 2015

Becky,

I talked to one of the scientists here, and we have a few thoughts...

(1) When working with an ensemble, one important step is ensemble
post-processing, which often includes a de-biasing step.  If you have
a
post-processed de-biased version of this data, I'd suggest evaluating
that.  But I'm assuming you don't.

(2) Next, I hear what you're saying that the varying biases makes it
impossible to find of configuration of MODE that works for all the
members.  In this case, using a different threshold for each member
would
be a reasonable approach.  But it'd be nice to do so in a systematic
way.
For example, you could define the threshold as a particular percentile
of
the raw data values.  And you could apply the same percentile
threshold to
all the members.  But for each member, the actual threshold value
would
change.

Using a percentile threshold is basically a simplistic bias removal.

We do have plans to add percentile thresholding directly to the MET
tools,
but that functionality won't be available for about a year.  In the
mean
time, here's what I'd suggest.  Write a wrapper script that...
   - Looks at the forecast values, ignoring the 0's, and computes the
percentile of interest.
   - Set that value as an environment variable, such as "setenv
FCST_PERC_THRESH 1.75"
   - Configure MODE to use that environment variable.  In the "fcst"
section, set "conv_thresh = >=${FCST_PERC_THRESH};"
   - Then run MODE for that case.

As for how to compute the percentile value of interest, you could use
a
variety of scripting tools.  I'd probably use R since I'm most
familiar
with that, but Matlab, IDL, or even a Python script could probably do
it.
Adding percentile thresholding directly to MET will make this much
easier
in the future.

Let me know if you get hung up at this step, and I might be able to
help
out.

(3) Lastly, you might also consider running this data through the
Grid-Stat
tool which can be used to compute neighborhood statistics, such as
Fractions Skill Score.  It really all depends on the questions you're
trying to answer about your dataset, but FSS is a nice simple
algorithm
that might give you some useful info about your data.

Hope that helps.

Thanks,
John

On Fri, May 1, 2015 at 10:03 AM, Adams-Selin, Becky via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
> John,
>
> OK, that makes sense, thank you.
>
> Like you say, ideally I'd like my configuration file to be the same
for
> all my forecast members.  However, because each member is biased
> differently, setting a conv_threshold value that makes sense for one
> member, results in really large forecast clusters being identified
in
> another member.  Is there a good way to limit the size of the
cluster
> objects without changing the conv_threshold?
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Monday, April 27, 2015 1:33 PM
> To: Adams-Selin, Becky
> Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> objects in field
>
> Becky,
>
> MODE attempts to do things intelligently here.  If the forecast and
> observation field names are identical, it chooses a consistent color
scale
> using the range of both.  Your data is not named the same thing
> (MAX_HAIL_24 vs MESH_hail) so it plots them using the range of the
data in
> each.
>
> As for interpreting the output, ultimately it's up to you.  You
mention
> that you're dealing with an ensemble with varying biases.  I would
suggest
> finding a configuration of MODE that identifies the objects of
interest and
> helps you answer your scientific questions... and then apply the
same
> configuration to all members.  Changing the configuration for each
member
> would be pretty confusing.
>
> I can help you with the nuts and bolts of running MODE, but one of
the
> scientists here would be better at helping you interpret results.
Before
> presenting results, we could probably to take a quick look to make
sure we
> don't see any red flags if you'd like.
>
> Thanks,
> John
>
>
>
> On Mon, Apr 27, 2015 at 12:07 PM, Adams-Selin, Becky via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> >
> > Wow, yes, that would do it.  I didn't realize the
forecasted/observed
> > plots on the first page of the output .ps file had different color
> scales!
> > Thank you for the help and the suggested Mode config file.
> >
> > If I were to report on these results obtained using a different
> > conv_threshold, would I just need to acknowledge a pre-existing
bias?
> The
> > problem is complicated by the fact that I am verifying the
forecasts of
> > multiple ensemble members, not all of whom are so biased.
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, April 27, 2015 12:03 PM
> > To: Adams-Selin, Becky
> > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> > objects in field
> >
> > Becky,
> >
> > Thanks for sending the data.  I ran the following two
plot_data_plane
> (from
> > MET) commands to plot this data using the same scale of 0 to 1.25.
The
> > resulting images are attached:
> >
> > plot_data_plane \
> >    HailSize20140508L1S_caps_maxnn_2.0km.nc \
> >    HailSize20140508L1S_caps_maxnn_2.0km.ps \
> >    'name="MESH_hail"; level="(*,*)";' -plot_range 0 1.25
> >
> > plot_data_plane \
> >    s4m17_arw_140508_hailsum.nc \
> >    s4m17_arw_140508_hailsum.ps \
> >    'name="MAX_HAIL_24"; level="(*,*)";' -plot_range 0 1.25
> >
> > Looking at them, I notice that the observed values are much higher
than
> the
> > forecast values!  Very few forecast values are above 1.0 while
most
> > observed values are above 1.0.  When you apply a raw thresh of
>=1.0,
> > you're zeroing out almost all of the forecast points but
relatively few
> > observation points.  That's why you're getting so many more
observation
> > objects than forecast objects.
> >
> > Inspecting the data more closely, I see...
> >  - the range on non-zero observed values is 0.50 to 3.74
> >  - the range of non-zero forecast values is 0.01 to 1.39
> >
> > Based on that, I made the following changes to the config file...
> >  - switch the raw_thresh from >=1.0 to >=0.5
> >  - increase the smoothing radius (conv_radius) from 1 to 4
> >  - set the plot_max in the fcst_raw_plot and obs_raw_plot sections
to
> 3.75
> > to use a consistent scale
> >  - set the forecast conv_thresh as >=0.25 and observed conv_thresh
as
> > >=0.5.
> >
> > This resulted in 45 forecast and 44 observed objects.  We
typically use
> the
> > same threshold for both forecast and observation, but your
forecast has a
> > very clear low bias, by about a factor of 2.  That's why I chose
the
> > observed threshold to be 2 times the forecast threshold.
> >
> > The config file and resulting output image is attached.
Unfortunately,
> > this takes a long time to run, about 5 min on my machine!
> >
> > I'd suggest that you play around with the settings more.  You
should
> decide
> > what "scale" of objects you're looking for.  If you want a smaller
number
> > of smoother objects, try increasing the convolution radius and
lowering
> the
> > threshold.  The fewer the number of objects MODE has to keep track
of,
> the
> > faster it'll run.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> > On Sat, Apr 25, 2015 at 9:19 AM, Adams-Selin, Becky via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> > >
> > > I tried lowering the conv_radius to 0, and the conv_threshold to
> 0.25.  I
> > > attached what I got.  I'm new to MODE, but just from visually
comparing
> > the
> > > forecast and observation fields, I feel like I should be getting
> roughly
> > > similar numbers of objects....I would even have expected fewer
objects
> > from
> > > the observation field.  Something seems wrong.
> > >
> > > I went ahead and created a selin_data directory on the passive
ftp
> site.
> > >
> > > The forecast file is s4m17_arw_140508_hailsum.nc
> > > The observation file is HailSize20140508L1S_caps_maxnn_2.0km.nc
> > > And the mode config file is MODEConfig_bigintense
> > >
> > > If you could take a look I would really appreciate it.
> > >
> > > Becky Adams-Selin, Ph.D.
> > > Staff Scientist II, AER
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Saturday, April 25, 2015 9:25 AM
> > > To: Adams-Selin, Becky
> > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> identifying
> > > objects in field
> > >
> > > Becky,
> > >
> > > OK, sounds good.  You could try setting the convolution radius
to 0
> just
> > to
> > > see if you can get forecast objects.  Or try lowering the
threshold.
> > > Perhaps they're all getting smoothed out.
> > >
> > > If you're still stuck I could take a look at some sample files.
I'd
> > need a
> > > sample forecast, observation, and MODE config file.  You could
post
> them
> > to
> > > our anonymous for site following these instructions:
> > >
> > > http://www.dtcenter.org/met/users/support/met_help.php#ftp
> > >
> > > Thanks
> > > John
> > >
> > >
> > > On Saturday, April 25, 2015, Adams-Selin, Becky via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
>
> > > >
> > > > John,
> > > >
> > > > I think that situation actually might apply to my dataset.
The
> > > > observational dataset only has hail values that are larger
than 1.0
> in.
> > > > The forecast dataset is continuous, from 0 inches upwards.  My
> > > > understanding was I should set the raw_thresh to 1.0 to make
the
> > forecast
> > > > and observational datasets consistent, like you said below.
> > > >
> > > > Becky Adams-Selin, Ph.D.
> > > > Staff Scientist II, AER
> > > >
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu
<javascript:;>]
> > > > Sent: Friday, April 24, 2015 3:39 PM
> > > > To: Adams-Selin, Becky
> > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> > identifying
> > > > objects in field
> > > >
> > > > Hello Becky,
> > > >
> > > > I see that you're having trouble getting objects to show up in
MODE.
> > > > Thanks for sending the sample plot and your configuration
file.
> > Looking
> > > > closely at your choice of thresholds, I see a problem:
> > > >    raw_thresh        = >=1.0;
> > > >    conv_thresh       = >=0.5;
> > > >
> > > > I would suggest changing it to this:
> > > >    raw_thresh        = >=0.0;
> > > >    conv_thresh       = >=0.5;
> > > >
> > > > Try rerunning to see if you get objects.
> > > >
> > > > The "raw_thresh" setting should generally not be used unless
you
> have a
> > > > good reason to do so.  It is applied to the raw data values
and any
> > point
> > > > not meeting that criteria is set to a value of 0.  The
resulting
> values
> > > are
> > > > passed through to the object definition step.
> > > >
> > > > Applying that logic doesn't make sense for your dataset.  You
may
> > wonder
> > > > why you would you ever want to do this?  A while back we were
> > evaluating
> > > a
> > > > reflectivity forecast, but the system only forecast values of
35dBz
> or
> > > > higher.  So we had values of 35dBz surround by values of 0.
The
> > > > observations were continuous, with values of 35dBZ surrounded
by
> 34dBZ
> > > and
> > > > so on.  When defining objects in MODE, we do a smoothing step
and
> > having
> > > > values of 0 vs values of 34 affected the output a lot.  So in
order
> to
> > > > define observation and forecast objects in a consistent way,
we
> zeroed
> > > out
> > > > any observed values < 35.
> > > >
> > > > Since this doesn't apply to your dataset, setting it as ">=0"
should
> > pass
> > > > all of your data through to the object definition step.
> > > >
> > > > Hope that helps clarify.
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > > met_help at ucar.edu <javascript:;>
> > > >
> > > >
> > > > On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
> > > > met_help at ucar.edu <javascript:;>> wrote:
> > > >
> > > > >
> > > > > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > > > > Transaction: Ticket created by RSelin at aer.com <javascript:;>
> > > > >        Queue: met_help
> > > > >      Subject: problems with MODE not identifying objects in
field
> > > > >        Owner: Nobody
> > > > >   Requestors: RSelin at aer.com <javascript:;>
> > > > >       Status: new
> > > > >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> > > >
> > > > >
> > > > >
> > > > > Dear MET help,
> > > > >
> > > > > I am using MODE in MET v4.1 to evaluate a custom hail
forecast in
> > > > > comparison to a gridded observed hail field (MESH).  I've
attached
> > the
> > > > > first page of the .ps output from one of my dates.  For some
reason
> > no
> > > > > objects are being identified in the forecast field, but
judging
> from
> > > the
> > > > > raw field, they should be.  My MODEConfig file is below.
Any idea
> > what
> > > > I'm
> > > > > doing wrong?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Becky Adams-Selin, Ph.D.
> > > > > Staff Scientist II, AER
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > //
> > > > > // MODE configuration file.
> > > > > //
> > > > > // For additional information, see the
MET_BASE/data/config/README
> > > file.
> > > > > //
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Output model name to be written
> > > > > //
> > > > > model = "WRF";
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Approximate grid resolution (km)
> > > > > //
> > > > > grid_res = 4;
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Forecast and observation fields to be verified
> > > > > //
> > > > > fcst = {
> > > > >    field = {
> > > > >       name  = "MAX_HAIL_24";
> > > > >       level = "(*,*)";
> > > > >    };
> > > > >
> > > > >    raw_thresh        = >=1.0;
> > > > >    conv_radius       = 4; // in grid squares
> > > > >    conv_thresh       = >=0.5;
> > > > >    vld_thresh        = 0.5;
> > > > >    area_thresh       = >=4.0;
> > > > >    inten_perc_value  = 100;
> > > > >    inten_perc_thresh = >=0.0;
> > > > >    merge_thresh      = >=0.5;
> > > > >    merge_flag        = NONE;
> > > > > };
> > > > > obs = {
> > > > >    field = {
> > > > >       name = "MESH_hail";
> > > > >       level = "(*,*)";
> > > > >    };
> > > > >
> > > > >    raw_thresh        = >=1.0;
> > > > >    conv_radius       = 4; // in grid squares
> > > > >    conv_thresh       = >=0.5;
> > > > >    vld_thresh        = 0.5;
> > > > >    area_thresh       = >=4.0;
> > > > >    inten_perc_value  = 100;
> > > > >    inten_perc_thresh = >=0.0;
> > > > >    merge_thresh      = >=0.5;
> > > > >    merge_flag        = NONE;
> > > > > };
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Handle missing data
> > > > > //
> > > > > mask_missing_flag = BOTH;
> > > > >
> > > > > //
> > > > > // Match objects between the forecast and observation fields
> > > > > //
> > > > > match_flag = MERGE_BOTH;
> > > > >
> > > > > //
> > > > > // Maximum centroid distance for objects to be compared
> > > > > //
> > > > > max_centroid_dist = 800.0/grid_res;
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Verification masking regions
> > > > > //
> > > > > mask = {
> > > > >    grid      = "";
> > > > >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
> > > > >    poly      =
"/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> > > > >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> > > > > };
> > > > >
> > > > > //
> > > > > // Fuzzy engine weights
> > > > > //
> > > > > weight = {
> > > > >    centroid_dist    = 2.0;
> > > > >    boundary_dist    = 4.0;
> > > > >    convex_hull_dist = 0.0;
> > > > >    angle_diff       = 1.0;
> > > > >    area_ratio       = 1.0;
> > > > >    int_area_ratio   = 2.0;
> > > > >    complexity_ratio = 0.0;
> > > > >    inten_perc_ratio = 0.0;
> > > > >    inten_perc_value = 100;
> > > > > }
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Fuzzy engine interest functions
> > > > > //
> > > > > interest_function = {
> > > > >
> > > > >    centroid_dist = (
> > > > >       (            0.0, 1.0 )
> > > > >       (  60.0/grid_res, 1.0 )
> > > > >       ( 600.0/grid_res, 0.0 )
> > > > >    );
> > > > >
> > > > >    boundary_dist = (
> > > > >       (            0.0, 1.0 )
> > > > >       ( 400.0/grid_res, 0.0 )
> > > > >    );
> > > > >
> > > > >    convex_hull_dist = (
> > > > >       (            0.0, 1.0 )
> > > > >       ( 400.0/grid_res, 0.0 )
> > > > >    );
> > > > >
> > > > >    angle_diff = (
> > > > >       (  0.0, 1.0 )
> > > > >       ( 30.0, 1.0 )
> > > > >       ( 90.0, 0.0 )
> > > > >    corner   = 0.8;
> > > > >    ratio_if = (
> > > > >       (    0.0, 0.0 )
> > > > >       ( corner, 1.0 )
> > > > >       (    1.0, 1.0 )
> > > > >    );
> > > > >
> > > > >    area_ratio = ratio_if;
> > > > >
> > > > >    int_area_ratio = (
> > > > >       ( 0.00, 0.00 )
> > > > >       ( 0.10, 0.50 )
> > > > >       ( 0.25, 1.00 )
> > > > >       ( 1.00, 1.00 )
> > > > >    );
> > > > >
> > > > >    complexity_ratio = ratio_if;
> > > > >
> > > > >    inten_perc_ratio = ratio_if;
> > > > > }
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Total interest threshold for determining matches
> > > > > //
> > > > > total_interest_thresh = 0.5;
> > > > >
> > > > > //
> > > > > // Interest threshold for printing output pair information
> > > > > //
> > > > > print_interest_thresh = 0.0;
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Plotting information
> > > > > //
> > > > > met_data_dir = "MET_BASE/data";
> > > > >
> > > > > fcst_raw_plot = {
> > > > >    color_table      =
> "MET_BASE/data/colortables/met_default.ctable";
> > > > >    plot_min         = 0.0;
> > > > >    plot_max         = 0.0;
> > > > >    colorbar_spacing = 1;
> > > > > };
> > > > >
> > > > > obs_raw_plot = {
> > > > >    color_table      =
> "MET_BASE/data/colortables/met_default.ctable";
> > > > >    plot_min         = 0.0;
> > > > >    plot_max         = 0.0;
> > > > >    colorbar_spacing = 1;
> > > > > };
> > > > >
> > > > > object_plot = {
> > > > >    color_table      =
"MET_BASE/data/colortables/mode_obj.ctable";
> > > > > };
> > > > >
> > > > > //
> > > > > // Number of grid boxes to fill with bad data values along
the edge
> > of
> > > > the
> > > > > field
> > > > > // to avoid edge effects.
> > > > > //
> > > > > zero_border_size = 1;
> > > > >
> > > > > //
> > > > > // Boolean for plotting on the region of valid data within
the
> domain
> > > > > //
> > > > > plot_valid_flag = FALSE;
> > > > >
> > > > > //
> > > > > // Plot polyline edges using great circle arcs instead of
straight
> > > lines
> > > > > //
> > > > > plot_gcarc_flag = FALSE;
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // NetCDF matched pairs, PostScript, and contingency table
output
> > files
> > > > > //
> > > > > ps_plot_flag    = TRUE;
> > > > > nc_pairs_flag   = TRUE;
> > > > > ct_stats_flag   = TRUE;
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > output_prefix  = "";
> > > > > version        = "V4.1";
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > >
> > > > > ________________________________
> > > > >
> > > > > This email is intended solely for the recipient. It may
contain
> > > > > privileged, proprietary or confidential information or
material. If
> > you
> > > > are
> > > > > not the intended recipient, please delete this email and any
> > > attachments
> > > > > and notify the sender of the error.
> > > > >
> > > > >
> > > >
> > > >
> > > > ________________________________
> > > >
> > > > This email is intended solely for the recipient. It may
contain
> > > > privileged, proprietary or confidential information or
material. If
> you
> > > are
> > > > not the intended recipient, please delete this email and any
> > attachments
> > > > and notify the sender of the error.
> > > >
> > > >
> > > >
> > > >
> > >
> > > ________________________________
> > >
> > > This email is intended solely for the recipient. It may contain
> > > privileged, proprietary or confidential information or material.
If you
> > are
> > > not the intended recipient, please delete this email and any
> attachments
> > > and notify the sender of the error.
> > >
> > >
> >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material.
If you
> are
> > not the intended recipient, please delete this email and any
attachments
> > and notify the sender of the error.
> >
> >
> >
> >
>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #71629] problems with MODE not identifying objects in field
From: Adams-Selin, Becky
Time: Fri May 01 15:11:36 2015

I agree de-biasing the ensemble members would be ideal.  However, I'm
actually comparing two different ensemble configurations, one where
the bias is largely mitigated by improvements to my hail algorithm, so
I want to be able to show that improvement in the statistics.

I like the idea of using a percentile of the raw data values!  That
would be simple to implement in python.  Would it be best to calculate
the percentile of all of the data over all of my dates, or re-
calculate the percentile for each date?

Becky Adams-Selin, Ph.D.
Staff Scientist II, AER

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, May 01, 2015 3:23 PM
To: Adams-Selin, Becky
Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying objects in field

Becky,

I talked to one of the scientists here, and we have a few thoughts...

(1) When working with an ensemble, one important step is ensemble
post-processing, which often includes a de-biasing step.  If you have
a
post-processed de-biased version of this data, I'd suggest evaluating
that.  But I'm assuming you don't.

(2) Next, I hear what you're saying that the varying biases makes it
impossible to find of configuration of MODE that works for all the
members.  In this case, using a different threshold for each member
would
be a reasonable approach.  But it'd be nice to do so in a systematic
way.
For example, you could define the threshold as a particular percentile
of
the raw data values.  And you could apply the same percentile
threshold to
all the members.  But for each member, the actual threshold value
would
change.

Using a percentile threshold is basically a simplistic bias removal.

We do have plans to add percentile thresholding directly to the MET
tools,
but that functionality won't be available for about a year.  In the
mean
time, here's what I'd suggest.  Write a wrapper script that...
   - Looks at the forecast values, ignoring the 0's, and computes the
percentile of interest.
   - Set that value as an environment variable, such as "setenv
FCST_PERC_THRESH 1.75"
   - Configure MODE to use that environment variable.  In the "fcst"
section, set "conv_thresh = >=${FCST_PERC_THRESH};"
   - Then run MODE for that case.

As for how to compute the percentile value of interest, you could use
a
variety of scripting tools.  I'd probably use R since I'm most
familiar
with that, but Matlab, IDL, or even a Python script could probably do
it.
Adding percentile thresholding directly to MET will make this much
easier
in the future.

Let me know if you get hung up at this step, and I might be able to
help
out.

(3) Lastly, you might also consider running this data through the
Grid-Stat
tool which can be used to compute neighborhood statistics, such as
Fractions Skill Score.  It really all depends on the questions you're
trying to answer about your dataset, but FSS is a nice simple
algorithm
that might give you some useful info about your data.

Hope that helps.

Thanks,
John

On Fri, May 1, 2015 at 10:03 AM, Adams-Selin, Becky via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
> John,
>
> OK, that makes sense, thank you.
>
> Like you say, ideally I'd like my configuration file to be the same
for
> all my forecast members.  However, because each member is biased
> differently, setting a conv_threshold value that makes sense for one
> member, results in really large forecast clusters being identified
in
> another member.  Is there a good way to limit the size of the
cluster
> objects without changing the conv_threshold?
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Monday, April 27, 2015 1:33 PM
> To: Adams-Selin, Becky
> Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> objects in field
>
> Becky,
>
> MODE attempts to do things intelligently here.  If the forecast and
> observation field names are identical, it chooses a consistent color
scale
> using the range of both.  Your data is not named the same thing
> (MAX_HAIL_24 vs MESH_hail) so it plots them using the range of the
data in
> each.
>
> As for interpreting the output, ultimately it's up to you.  You
mention
> that you're dealing with an ensemble with varying biases.  I would
suggest
> finding a configuration of MODE that identifies the objects of
interest and
> helps you answer your scientific questions... and then apply the
same
> configuration to all members.  Changing the configuration for each
member
> would be pretty confusing.
>
> I can help you with the nuts and bolts of running MODE, but one of
the
> scientists here would be better at helping you interpret results.
Before
> presenting results, we could probably to take a quick look to make
sure we
> don't see any red flags if you'd like.
>
> Thanks,
> John
>
>
>
> On Mon, Apr 27, 2015 at 12:07 PM, Adams-Selin, Becky via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> >
> > Wow, yes, that would do it.  I didn't realize the
forecasted/observed
> > plots on the first page of the output .ps file had different color
> scales!
> > Thank you for the help and the suggested Mode config file.
> >
> > If I were to report on these results obtained using a different
> > conv_threshold, would I just need to acknowledge a pre-existing
bias?
> The
> > problem is complicated by the fact that I am verifying the
forecasts of
> > multiple ensemble members, not all of whom are so biased.
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, April 27, 2015 12:03 PM
> > To: Adams-Selin, Becky
> > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying
> > objects in field
> >
> > Becky,
> >
> > Thanks for sending the data.  I ran the following two
plot_data_plane
> (from
> > MET) commands to plot this data using the same scale of 0 to 1.25.
The
> > resulting images are attached:
> >
> > plot_data_plane \
> >    HailSize20140508L1S_caps_maxnn_2.0km.nc \
> >    HailSize20140508L1S_caps_maxnn_2.0km.ps \
> >    'name="MESH_hail"; level="(*,*)";' -plot_range 0 1.25
> >
> > plot_data_plane \
> >    s4m17_arw_140508_hailsum.nc \
> >    s4m17_arw_140508_hailsum.ps \
> >    'name="MAX_HAIL_24"; level="(*,*)";' -plot_range 0 1.25
> >
> > Looking at them, I notice that the observed values are much higher
than
> the
> > forecast values!  Very few forecast values are above 1.0 while
most
> > observed values are above 1.0.  When you apply a raw thresh of
>=1.0,
> > you're zeroing out almost all of the forecast points but
relatively few
> > observation points.  That's why you're getting so many more
observation
> > objects than forecast objects.
> >
> > Inspecting the data more closely, I see...
> >  - the range on non-zero observed values is 0.50 to 3.74
> >  - the range of non-zero forecast values is 0.01 to 1.39
> >
> > Based on that, I made the following changes to the config file...
> >  - switch the raw_thresh from >=1.0 to >=0.5
> >  - increase the smoothing radius (conv_radius) from 1 to 4
> >  - set the plot_max in the fcst_raw_plot and obs_raw_plot sections
to
> 3.75
> > to use a consistent scale
> >  - set the forecast conv_thresh as >=0.25 and observed conv_thresh
as
> > >=0.5.
> >
> > This resulted in 45 forecast and 44 observed objects.  We
typically use
> the
> > same threshold for both forecast and observation, but your
forecast has a
> > very clear low bias, by about a factor of 2.  That's why I chose
the
> > observed threshold to be 2 times the forecast threshold.
> >
> > The config file and resulting output image is attached.
Unfortunately,
> > this takes a long time to run, about 5 min on my machine!
> >
> > I'd suggest that you play around with the settings more.  You
should
> decide
> > what "scale" of objects you're looking for.  If you want a smaller
number
> > of smoother objects, try increasing the convolution radius and
lowering
> the
> > threshold.  The fewer the number of objects MODE has to keep track
of,
> the
> > faster it'll run.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> > On Sat, Apr 25, 2015 at 9:19 AM, Adams-Selin, Becky via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> > >
> > > I tried lowering the conv_radius to 0, and the conv_threshold to
> 0.25.  I
> > > attached what I got.  I'm new to MODE, but just from visually
comparing
> > the
> > > forecast and observation fields, I feel like I should be getting
> roughly
> > > similar numbers of objects....I would even have expected fewer
objects
> > from
> > > the observation field.  Something seems wrong.
> > >
> > > I went ahead and created a selin_data directory on the passive
ftp
> site.
> > >
> > > The forecast file is s4m17_arw_140508_hailsum.nc
> > > The observation file is HailSize20140508L1S_caps_maxnn_2.0km.nc
> > > And the mode config file is MODEConfig_bigintense
> > >
> > > If you could take a look I would really appreciate it.
> > >
> > > Becky Adams-Selin, Ph.D.
> > > Staff Scientist II, AER
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Saturday, April 25, 2015 9:25 AM
> > > To: Adams-Selin, Becky
> > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> identifying
> > > objects in field
> > >
> > > Becky,
> > >
> > > OK, sounds good.  You could try setting the convolution radius
to 0
> just
> > to
> > > see if you can get forecast objects.  Or try lowering the
threshold.
> > > Perhaps they're all getting smoothed out.
> > >
> > > If you're still stuck I could take a look at some sample files.
I'd
> > need a
> > > sample forecast, observation, and MODE config file.  You could
post
> them
> > to
> > > our anonymous for site following these instructions:
> > >
> > > http://www.dtcenter.org/met/users/support/met_help.php#ftp
> > >
> > > Thanks
> > > John
> > >
> > >
> > > On Saturday, April 25, 2015, Adams-Selin, Becky via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
>
> > > >
> > > > John,
> > > >
> > > > I think that situation actually might apply to my dataset.
The
> > > > observational dataset only has hail values that are larger
than 1.0
> in.
> > > > The forecast dataset is continuous, from 0 inches upwards.  My
> > > > understanding was I should set the raw_thresh to 1.0 to make
the
> > forecast
> > > > and observational datasets consistent, like you said below.
> > > >
> > > > Becky Adams-Selin, Ph.D.
> > > > Staff Scientist II, AER
> > > >
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu
<javascript:;>]
> > > > Sent: Friday, April 24, 2015 3:39 PM
> > > > To: Adams-Selin, Becky
> > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> > identifying
> > > > objects in field
> > > >
> > > > Hello Becky,
> > > >
> > > > I see that you're having trouble getting objects to show up in
MODE.
> > > > Thanks for sending the sample plot and your configuration
file.
> > Looking
> > > > closely at your choice of thresholds, I see a problem:
> > > >    raw_thresh        = >=1.0;
> > > >    conv_thresh       = >=0.5;
> > > >
> > > > I would suggest changing it to this:
> > > >    raw_thresh        = >=0.0;
> > > >    conv_thresh       = >=0.5;
> > > >
> > > > Try rerunning to see if you get objects.
> > > >
> > > > The "raw_thresh" setting should generally not be used unless
you
> have a
> > > > good reason to do so.  It is applied to the raw data values
and any
> > point
> > > > not meeting that criteria is set to a value of 0.  The
resulting
> values
> > > are
> > > > passed through to the object definition step.
> > > >
> > > > Applying that logic doesn't make sense for your dataset.  You
may
> > wonder
> > > > why you would you ever want to do this?  A while back we were
> > evaluating
> > > a
> > > > reflectivity forecast, but the system only forecast values of
35dBz
> or
> > > > higher.  So we had values of 35dBz surround by values of 0.
The
> > > > observations were continuous, with values of 35dBZ surrounded
by
> 34dBZ
> > > and
> > > > so on.  When defining objects in MODE, we do a smoothing step
and
> > having
> > > > values of 0 vs values of 34 affected the output a lot.  So in
order
> to
> > > > define observation and forecast objects in a consistent way,
we
> zeroed
> > > out
> > > > any observed values < 35.
> > > >
> > > > Since this doesn't apply to your dataset, setting it as ">=0"
should
> > pass
> > > > all of your data through to the object definition step.
> > > >
> > > > Hope that helps clarify.
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > > met_help at ucar.edu <javascript:;>
> > > >
> > > >
> > > > On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
> > > > met_help at ucar.edu <javascript:;>> wrote:
> > > >
> > > > >
> > > > > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > > > > Transaction: Ticket created by RSelin at aer.com <javascript:;>
> > > > >        Queue: met_help
> > > > >      Subject: problems with MODE not identifying objects in
field
> > > > >        Owner: Nobody
> > > > >   Requestors: RSelin at aer.com <javascript:;>
> > > > >       Status: new
> > > > >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> > > >
> > > > >
> > > > >
> > > > > Dear MET help,
> > > > >
> > > > > I am using MODE in MET v4.1 to evaluate a custom hail
forecast in
> > > > > comparison to a gridded observed hail field (MESH).  I've
attached
> > the
> > > > > first page of the .ps output from one of my dates.  For some
reason
> > no
> > > > > objects are being identified in the forecast field, but
judging
> from
> > > the
> > > > > raw field, they should be.  My MODEConfig file is below.
Any idea
> > what
> > > > I'm
> > > > > doing wrong?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Becky Adams-Selin, Ph.D.
> > > > > Staff Scientist II, AER
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > //
> > > > > // MODE configuration file.
> > > > > //
> > > > > // For additional information, see the
MET_BASE/data/config/README
> > > file.
> > > > > //
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Output model name to be written
> > > > > //
> > > > > model = "WRF";
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Approximate grid resolution (km)
> > > > > //
> > > > > grid_res = 4;
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Forecast and observation fields to be verified
> > > > > //
> > > > > fcst = {
> > > > >    field = {
> > > > >       name  = "MAX_HAIL_24";
> > > > >       level = "(*,*)";
> > > > >    };
> > > > >
> > > > >    raw_thresh        = >=1.0;
> > > > >    conv_radius       = 4; // in grid squares
> > > > >    conv_thresh       = >=0.5;
> > > > >    vld_thresh        = 0.5;
> > > > >    area_thresh       = >=4.0;
> > > > >    inten_perc_value  = 100;
> > > > >    inten_perc_thresh = >=0.0;
> > > > >    merge_thresh      = >=0.5;
> > > > >    merge_flag        = NONE;
> > > > > };
> > > > > obs = {
> > > > >    field = {
> > > > >       name = "MESH_hail";
> > > > >       level = "(*,*)";
> > > > >    };
> > > > >
> > > > >    raw_thresh        = >=1.0;
> > > > >    conv_radius       = 4; // in grid squares
> > > > >    conv_thresh       = >=0.5;
> > > > >    vld_thresh        = 0.5;
> > > > >    area_thresh       = >=4.0;
> > > > >    inten_perc_value  = 100;
> > > > >    inten_perc_thresh = >=0.0;
> > > > >    merge_thresh      = >=0.5;
> > > > >    merge_flag        = NONE;
> > > > > };
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Handle missing data
> > > > > //
> > > > > mask_missing_flag = BOTH;
> > > > >
> > > > > //
> > > > > // Match objects between the forecast and observation fields
> > > > > //
> > > > > match_flag = MERGE_BOTH;
> > > > >
> > > > > //
> > > > > // Maximum centroid distance for objects to be compared
> > > > > //
> > > > > max_centroid_dist = 800.0/grid_res;
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Verification masking regions
> > > > > //
> > > > > mask = {
> > > > >    grid      = "";
> > > > >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
> > > > >    poly      =
"/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> > > > >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> > > > > };
> > > > >
> > > > > //
> > > > > // Fuzzy engine weights
> > > > > //
> > > > > weight = {
> > > > >    centroid_dist    = 2.0;
> > > > >    boundary_dist    = 4.0;
> > > > >    convex_hull_dist = 0.0;
> > > > >    angle_diff       = 1.0;
> > > > >    area_ratio       = 1.0;
> > > > >    int_area_ratio   = 2.0;
> > > > >    complexity_ratio = 0.0;
> > > > >    inten_perc_ratio = 0.0;
> > > > >    inten_perc_value = 100;
> > > > > }
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Fuzzy engine interest functions
> > > > > //
> > > > > interest_function = {
> > > > >
> > > > >    centroid_dist = (
> > > > >       (            0.0, 1.0 )
> > > > >       (  60.0/grid_res, 1.0 )
> > > > >       ( 600.0/grid_res, 0.0 )
> > > > >    );
> > > > >
> > > > >    boundary_dist = (
> > > > >       (            0.0, 1.0 )
> > > > >       ( 400.0/grid_res, 0.0 )
> > > > >    );
> > > > >
> > > > >    convex_hull_dist = (
> > > > >       (            0.0, 1.0 )
> > > > >       ( 400.0/grid_res, 0.0 )
> > > > >    );
> > > > >
> > > > >    angle_diff = (
> > > > >       (  0.0, 1.0 )
> > > > >       ( 30.0, 1.0 )
> > > > >       ( 90.0, 0.0 )
> > > > >    corner   = 0.8;
> > > > >    ratio_if = (
> > > > >       (    0.0, 0.0 )
> > > > >       ( corner, 1.0 )
> > > > >       (    1.0, 1.0 )
> > > > >    );
> > > > >
> > > > >    area_ratio = ratio_if;
> > > > >
> > > > >    int_area_ratio = (
> > > > >       ( 0.00, 0.00 )
> > > > >       ( 0.10, 0.50 )
> > > > >       ( 0.25, 1.00 )
> > > > >       ( 1.00, 1.00 )
> > > > >    );
> > > > >
> > > > >    complexity_ratio = ratio_if;
> > > > >
> > > > >    inten_perc_ratio = ratio_if;
> > > > > }
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Total interest threshold for determining matches
> > > > > //
> > > > > total_interest_thresh = 0.5;
> > > > >
> > > > > //
> > > > > // Interest threshold for printing output pair information
> > > > > //
> > > > > print_interest_thresh = 0.0;
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // Plotting information
> > > > > //
> > > > > met_data_dir = "MET_BASE/data";
> > > > >
> > > > > fcst_raw_plot = {
> > > > >    color_table      =
> "MET_BASE/data/colortables/met_default.ctable";
> > > > >    plot_min         = 0.0;
> > > > >    plot_max         = 0.0;
> > > > >    colorbar_spacing = 1;
> > > > > };
> > > > >
> > > > > obs_raw_plot = {
> > > > >    color_table      =
> "MET_BASE/data/colortables/met_default.ctable";
> > > > >    plot_min         = 0.0;
> > > > >    plot_max         = 0.0;
> > > > >    colorbar_spacing = 1;
> > > > > };
> > > > >
> > > > > object_plot = {
> > > > >    color_table      =
"MET_BASE/data/colortables/mode_obj.ctable";
> > > > > };
> > > > >
> > > > > //
> > > > > // Number of grid boxes to fill with bad data values along
the edge
> > of
> > > > the
> > > > > field
> > > > > // to avoid edge effects.
> > > > > //
> > > > > zero_border_size = 1;
> > > > >
> > > > > //
> > > > > // Boolean for plotting on the region of valid data within
the
> domain
> > > > > //
> > > > > plot_valid_flag = FALSE;
> > > > >
> > > > > //
> > > > > // Plot polyline edges using great circle arcs instead of
straight
> > > lines
> > > > > //
> > > > > plot_gcarc_flag = FALSE;
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > //
> > > > > // NetCDF matched pairs, PostScript, and contingency table
output
> > files
> > > > > //
> > > > > ps_plot_flag    = TRUE;
> > > > > nc_pairs_flag   = TRUE;
> > > > > ct_stats_flag   = TRUE;
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > > output_prefix  = "";
> > > > > version        = "V4.1";
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > >
> > > > >
> > > > > ________________________________
> > > > >
> > > > > This email is intended solely for the recipient. It may
contain
> > > > > privileged, proprietary or confidential information or
material. If
> > you
> > > > are
> > > > > not the intended recipient, please delete this email and any
> > > attachments
> > > > > and notify the sender of the error.
> > > > >
> > > > >
> > > >
> > > >
> > > > ________________________________
> > > >
> > > > This email is intended solely for the recipient. It may
contain
> > > > privileged, proprietary or confidential information or
material. If
> you
> > > are
> > > > not the intended recipient, please delete this email and any
> > attachments
> > > > and notify the sender of the error.
> > > >
> > > >
> > > >
> > > >
> > >
> > > ________________________________
> > >
> > > This email is intended solely for the recipient. It may contain
> > > privileged, proprietary or confidential information or material.
If you
> > are
> > > not the intended recipient, please delete this email and any
> attachments
> > > and notify the sender of the error.
> > >
> > >
> >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material.
If you
> are
> > not the intended recipient, please delete this email and any
attachments
> > and notify the sender of the error.
> >
> >
> >
> >
>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>
>
>

________________________________

This email is intended solely for the recipient. It may contain
privileged, proprietary or confidential information or material. If
you are not the intended recipient, please delete this email and any
attachments and notify the sender of the error.



------------------------------------------------
Subject: problems with MODE not identifying objects in field
From: Tara Jensen
Time: Fri May 01 15:50:50 2015

Hi Becky,

This is Tara Jensen.  I'm one of the scientists on the MET team.  The
method John suggested is exploratory in the sense that we have not
tried it much ourselves either.  While we do suggest it would probably
be best to look at your data as a whole, I wouldn't rule out the need
to maybe stratify your data by storm morphologies or forcing
mechanisms before determining your thresholds.  I would hope that you
wouldn't have to adjust the thresholds on a case by case basis because
that would likely be less scientifically defensible.  Hopefully that
helps you get started.  Please do not hesitate to let us know how your
analysis is progressing and if you have further questions.

Thanks, Tara Jensen

On Fri May 01 15:11:36 2015, RSelin at aer.com wrote:
> I agree de-biasing the ensemble members would be ideal.  However,
I'm
>    actually comparing two different ensemble configurations, one
where
>    the bias is largely mitigated by improvements to my hail
algorithm,
>    so I want to be able to show that improvement in the statistics.
>
> I like the idea of using a percentile of the raw data values!  That
>    would be simple to implement in python.  Would it be best to
>    calculate the percentile of all of the data over all of my dates,
>    or re-calculate the percentile for each date?
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Friday, May 01, 2015 3:23 PM
> To: Adams-Selin, Becky
> Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
>    identifying objects in field
>
> Becky,
>
> I talked to one of the scientists here, and we have a few
thoughts...
>
> (1) When working with an ensemble, one important step is ensemble
> post-processing, which often includes a de-biasing step.  If you
have
>    a
> post-processed de-biased version of this data, I'd suggest
evaluating
> that.  But I'm assuming you don't.
>
> (2) Next, I hear what you're saying that the varying biases makes it
> impossible to find of configuration of MODE that works for all the
> members.  In this case, using a different threshold for each member
>    would
> be a reasonable approach.  But it'd be nice to do so in a systematic
>    way.
> For example, you could define the threshold as a particular
percentile
>    of
> the raw data values.  And you could apply the same percentile
>    threshold to
> all the members.  But for each member, the actual threshold value
>    would
> change.
>
> Using a percentile threshold is basically a simplistic bias removal.
>
> We do have plans to add percentile thresholding directly to the MET
>    tools,
> but that functionality won't be available for about a year.  In the
>    mean
> time, here's what I'd suggest.  Write a wrapper script that...
>    - Looks at the forecast values, ignoring the 0's, and computes
the
> percentile of interest.
>    - Set that value as an environment variable, such as "setenv
> FCST_PERC_THRESH 1.75"
>    - Configure MODE to use that environment variable.  In the "fcst"
> section, set "conv_thresh = >=${FCST_PERC_THRESH};"
>    - Then run MODE for that case.
>
> As for how to compute the percentile value of interest, you could
use
>    a
> variety of scripting tools.  I'd probably use R since I'm most
>    familiar
> with that, but Matlab, IDL, or even a Python script could probably
do
>    it.
> Adding percentile thresholding directly to MET will make this much
>    easier
> in the future.
>
> Let me know if you get hung up at this step, and I might be able to
>    help
> out.
>
> (3) Lastly, you might also consider running this data through the
>    Grid-Stat
> tool which can be used to compute neighborhood statistics, such as
> Fractions Skill Score.  It really all depends on the questions
you're
> trying to answer about your dataset, but FSS is a nice simple
>    algorithm
> that might give you some useful info about your data.
>
> Hope that helps.
>
> Thanks,
> John
>
> On Fri, May 1, 2015 at 10:03 AM, Adams-Selin, Becky via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> >
> > John,
> >
> > OK, that makes sense, thank you.
> >
> > Like you say, ideally I'd like my configuration file to be the
same
>    for
> > all my forecast members.  However, because each member is biased
> > differently, setting a conv_threshold value that makes sense for
one
> > member, results in really large forecast clusters being identified
>    in
> > another member.  Is there a good way to limit the size of the
>    cluster
> > objects without changing the conv_threshold?
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, April 27, 2015 1:33 PM
> > To: Adams-Selin, Becky
> > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
>    identifying
> > objects in field
> >
> > Becky,
> >
> > MODE attempts to do things intelligently here.  If the forecast
and
> > observation field names are identical, it chooses a consistent
color
>    scale
> > using the range of both.  Your data is not named the same thing
> > (MAX_HAIL_24 vs MESH_hail) so it plots them using the range of the
>    data in
> > each.
> >
> > As for interpreting the output, ultimately it's up to you.  You
>    mention
> > that you're dealing with an ensemble with varying biases.  I would
>    suggest
> > finding a configuration of MODE that identifies the objects of
>    interest and
> > helps you answer your scientific questions... and then apply the
>    same
> > configuration to all members.  Changing the configuration for each
>    member
> > would be pretty confusing.
> >
> > I can help you with the nuts and bolts of running MODE, but one of
>    the
> > scientists here would be better at helping you interpret results.
>    Before
> > presenting results, we could probably to take a quick look to make
>    sure we
> > don't see any red flags if you'd like.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Mon, Apr 27, 2015 at 12:07 PM, Adams-Selin, Becky via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> > >
> > > Wow, yes, that would do it.  I didn't realize the
>    forecasted/observed
> > > plots on the first page of the output .ps file had different
color
> > scales!
> > > Thank you for the help and the suggested Mode config file.
> > >
> > > If I were to report on these results obtained using a different
> > > conv_threshold, would I just need to acknowledge a pre-existing
>    bias?
> > The
> > > problem is complicated by the fact that I am verifying the
>    forecasts of
> > > multiple ensemble members, not all of whom are so biased.
> > >
> > > Becky Adams-Selin, Ph.D.
> > > Staff Scientist II, AER
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Monday, April 27, 2015 12:03 PM
> > > To: Adams-Selin, Becky
> > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
>    identifying
> > > objects in field
> > >
> > > Becky,
> > >
> > > Thanks for sending the data.  I ran the following two
>    plot_data_plane
> > (from
> > > MET) commands to plot this data using the same scale of 0 to
1.25.
>    The
> > > resulting images are attached:
> > >
> > > plot_data_plane \
> > >    HailSize20140508L1S_caps_maxnn_2.0km.nc \
> > >    HailSize20140508L1S_caps_maxnn_2.0km.ps \
> > >    'name="MESH_hail"; level="(*,*)";' -plot_range 0 1.25
> > >
> > > plot_data_plane \
> > >    s4m17_arw_140508_hailsum.nc \
> > >    s4m17_arw_140508_hailsum.ps \
> > >    'name="MAX_HAIL_24"; level="(*,*)";' -plot_range 0 1.25
> > >
> > > Looking at them, I notice that the observed values are much
higher
>    than
> > the
> > > forecast values!  Very few forecast values are above 1.0 while
>    most
> > > observed values are above 1.0.  When you apply a raw thresh of
>    >=1.0,
> > > you're zeroing out almost all of the forecast points but
>    relatively few
> > > observation points.  That's why you're getting so many more
>    observation
> > > objects than forecast objects.
> > >
> > > Inspecting the data more closely, I see...
> > >  - the range on non-zero observed values is 0.50 to 3.74
> > >  - the range of non-zero forecast values is 0.01 to 1.39
> > >
> > > Based on that, I made the following changes to the config
file...
> > >  - switch the raw_thresh from >=1.0 to >=0.5
> > >  - increase the smoothing radius (conv_radius) from 1 to 4
> > >  - set the plot_max in the fcst_raw_plot and obs_raw_plot
sections
>    to
> > 3.75
> > > to use a consistent scale
> > >  - set the forecast conv_thresh as >=0.25 and observed
conv_thresh
>    as
> > > >=0.5.
> > >
> > > This resulted in 45 forecast and 44 observed objects.  We
>    typically use
> > the
> > > same threshold for both forecast and observation, but your
>    forecast has a
> > > very clear low bias, by about a factor of 2.  That's why I chose
>    the
> > > observed threshold to be 2 times the forecast threshold.
> > >
> > > The config file and resulting output image is attached.
>    Unfortunately,
> > > this takes a long time to run, about 5 min on my machine!
> > >
> > > I'd suggest that you play around with the settings more.  You
>    should
> > decide
> > > what "scale" of objects you're looking for.  If you want a
smaller
>    number
> > > of smoother objects, try increasing the convolution radius and
>    lowering
> > the
> > > threshold.  The fewer the number of objects MODE has to keep
track
>    of,
> > the
> > > faster it'll run.
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Sat, Apr 25, 2015 at 9:19 AM, Adams-Selin, Becky via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
>
> > > >
> > > > I tried lowering the conv_radius to 0, and the conv_threshold
to
> > 0.25.  I
> > > > attached what I got.  I'm new to MODE, but just from visually
>    comparing
> > > the
> > > > forecast and observation fields, I feel like I should be
getting
> > roughly
> > > > similar numbers of objects....I would even have expected fewer
>    objects
> > > from
> > > > the observation field.  Something seems wrong.
> > > >
> > > > I went ahead and created a selin_data directory on the passive
>    ftp
> > site.
> > > >
> > > > The forecast file is s4m17_arw_140508_hailsum.nc
> > > > The observation file is
HailSize20140508L1S_caps_maxnn_2.0km.nc
> > > > And the mode config file is MODEConfig_bigintense
> > > >
> > > > If you could take a look I would really appreciate it.
> > > >
> > > > Becky Adams-Selin, Ph.D.
> > > > Staff Scientist II, AER
> > > >
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Saturday, April 25, 2015 9:25 AM
> > > > To: Adams-Selin, Becky
> > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> > identifying
> > > > objects in field
> > > >
> > > > Becky,
> > > >
> > > > OK, sounds good.  You could try setting the convolution radius
>    to 0
> > just
> > > to
> > > > see if you can get forecast objects.  Or try lowering the
>    threshold.
> > > > Perhaps they're all getting smoothed out.
> > > >
> > > > If you're still stuck I could take a look at some sample
files.
>    I'd
> > > need a
> > > > sample forecast, observation, and MODE config file.  You could
>    post
> > them
> > > to
> > > > our anonymous for site following these instructions:
> > > >
> > > > http://www.dtcenter.org/met/users/support/met_help.php#ftp
> > > >
> > > > Thanks
> > > > John
> > > >
> > > >
> > > > On Saturday, April 25, 2015, Adams-Selin, Becky via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
>    >
> > > > >
> > > > > John,
> > > > >
> > > > > I think that situation actually might apply to my dataset.
>    The
> > > > > observational dataset only has hail values that are larger
>    than 1.0
> > in.
> > > > > The forecast dataset is continuous, from 0 inches upwards.
My
> > > > > understanding was I should set the raw_thresh to 1.0 to make
>    the
> > > forecast
> > > > > and observational datasets consistent, like you said below.
> > > > >
> > > > > Becky Adams-Selin, Ph.D.
> > > > > Staff Scientist II, AER
> > > > >
> > > > > ________________________________________
> > > > > From: John Halley Gotway via RT [met_help at ucar.edu
>    <javascript:;>]
> > > > > Sent: Friday, April 24, 2015 3:39 PM
> > > > > To: Adams-Selin, Becky
> > > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> > > identifying
> > > > > objects in field
> > > > >
> > > > > Hello Becky,
> > > > >
> > > > > I see that you're having trouble getting objects to show up
in
>    MODE.
> > > > > Thanks for sending the sample plot and your configuration
>    file.
> > > Looking
> > > > > closely at your choice of thresholds, I see a problem:
> > > > >    raw_thresh        = >=1.0;
> > > > >    conv_thresh       = >=0.5;
> > > > >
> > > > > I would suggest changing it to this:
> > > > >    raw_thresh        = >=0.0;
> > > > >    conv_thresh       = >=0.5;
> > > > >
> > > > > Try rerunning to see if you get objects.
> > > > >
> > > > > The "raw_thresh" setting should generally not be used unless
>    you
> > have a
> > > > > good reason to do so.  It is applied to the raw data values
>    and any
> > > point
> > > > > not meeting that criteria is set to a value of 0.  The
>    resulting
> > values
> > > > are
> > > > > passed through to the object definition step.
> > > > >
> > > > > Applying that logic doesn't make sense for your dataset.
You
>    may
> > > wonder
> > > > > why you would you ever want to do this?  A while back we
were
> > > evaluating
> > > > a
> > > > > reflectivity forecast, but the system only forecast values
of
>    35dBz
> > or
> > > > > higher.  So we had values of 35dBz surround by values of 0.
>    The
> > > > > observations were continuous, with values of 35dBZ
surrounded
>    by
> > 34dBZ
> > > > and
> > > > > so on.  When defining objects in MODE, we do a smoothing
step
>    and
> > > having
> > > > > values of 0 vs values of 34 affected the output a lot.  So
in
>    order
> > to
> > > > > define observation and forecast objects in a consistent way,
>    we
> > zeroed
> > > > out
> > > > > any observed values < 35.
> > > > >
> > > > > Since this doesn't apply to your dataset, setting it as
">=0"
>    should
> > > pass
> > > > > all of your data through to the object definition step.
> > > > >
> > > > > Hope that helps clarify.
> > > > >
> > > > > Thanks,
> > > > > John Halley Gotway
> > > > > met_help at ucar.edu <javascript:;>
> > > > >
> > > > >
> > > > > On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
> > > > > met_help at ucar.edu <javascript:;>> wrote:
> > > > >
> > > > > >
> > > > > > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > > > > > Transaction: Ticket created by RSelin at aer.com
<javascript:;>
> > > > > >        Queue: met_help
> > > > > >      Subject: problems with MODE not identifying objects
in
>    field
> > > > > >        Owner: Nobody
> > > > > >   Requestors: RSelin at aer.com <javascript:;>
> > > > > >       Status: new
> > > > > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> > > > >
> > > > > >
> > > > > >
> > > > > > Dear MET help,
> > > > > >
> > > > > > I am using MODE in MET v4.1 to evaluate a custom hail
>    forecast in
> > > > > > comparison to a gridded observed hail field (MESH).  I've
>    attached
> > > the
> > > > > > first page of the .ps output from one of my dates.  For
some
>    reason
> > > no
> > > > > > objects are being identified in the forecast field, but
>    judging
> > from
> > > > the
> > > > > > raw field, they should be.  My MODEConfig file is below.
>    Any idea
> > > what
> > > > > I'm
> > > > > > doing wrong?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Becky Adams-Selin, Ph.D.
> > > > > > Staff Scientist II, AER
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > //
> > > > > > // MODE configuration file.
> > > > > > //
> > > > > > // For additional information, see the
>    MET_BASE/data/config/README
> > > > file.
> > > > > > //
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Output model name to be written
> > > > > > //
> > > > > > model = "WRF";
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Approximate grid resolution (km)
> > > > > > //
> > > > > > grid_res = 4;
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Forecast and observation fields to be verified
> > > > > > //
> > > > > > fcst = {
> > > > > >    field = {
> > > > > >       name  = "MAX_HAIL_24";
> > > > > >       level = "(*,*)";
> > > > > >    };
> > > > > >
> > > > > >    raw_thresh        = >=1.0;
> > > > > >    conv_radius       = 4; // in grid squares
> > > > > >    conv_thresh       = >=0.5;
> > > > > >    vld_thresh        = 0.5;
> > > > > >    area_thresh       = >=4.0;
> > > > > >    inten_perc_value  = 100;
> > > > > >    inten_perc_thresh = >=0.0;
> > > > > >    merge_thresh      = >=0.5;
> > > > > >    merge_flag        = NONE;
> > > > > > };
> > > > > > obs = {
> > > > > >    field = {
> > > > > >       name = "MESH_hail";
> > > > > >       level = "(*,*)";
> > > > > >    };
> > > > > >
> > > > > >    raw_thresh        = >=1.0;
> > > > > >    conv_radius       = 4; // in grid squares
> > > > > >    conv_thresh       = >=0.5;
> > > > > >    vld_thresh        = 0.5;
> > > > > >    area_thresh       = >=4.0;
> > > > > >    inten_perc_value  = 100;
> > > > > >    inten_perc_thresh = >=0.0;
> > > > > >    merge_thresh      = >=0.5;
> > > > > >    merge_flag        = NONE;
> > > > > > };
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Handle missing data
> > > > > > //
> > > > > > mask_missing_flag = BOTH;
> > > > > >
> > > > > > //
> > > > > > // Match objects between the forecast and observation
fields
> > > > > > //
> > > > > > match_flag = MERGE_BOTH;
> > > > > >
> > > > > > //
> > > > > > // Maximum centroid distance for objects to be compared
> > > > > > //
> > > > > > max_centroid_dist = 800.0/grid_res;
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Verification masking regions
> > > > > > //
> > > > > > mask = {
> > > > > >    grid      = "";
> > > > > >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
> > > > > >    poly      =
>    "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> > > > > >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> > > > > > };
> > > > > >
> > > > > > //
> > > > > > // Fuzzy engine weights
> > > > > > //
> > > > > > weight = {
> > > > > >    centroid_dist    = 2.0;
> > > > > >    boundary_dist    = 4.0;
> > > > > >    convex_hull_dist = 0.0;
> > > > > >    angle_diff       = 1.0;
> > > > > >    area_ratio       = 1.0;
> > > > > >    int_area_ratio   = 2.0;
> > > > > >    complexity_ratio = 0.0;
> > > > > >    inten_perc_ratio = 0.0;
> > > > > >    inten_perc_value = 100;
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Fuzzy engine interest functions
> > > > > > //
> > > > > > interest_function = {
> > > > > >
> > > > > >    centroid_dist = (
> > > > > >       (            0.0, 1.0 )
> > > > > >       (  60.0/grid_res, 1.0 )
> > > > > >       ( 600.0/grid_res, 0.0 )
> > > > > >    );
> > > > > >
> > > > > >    boundary_dist = (
> > > > > >       (            0.0, 1.0 )
> > > > > >       ( 400.0/grid_res, 0.0 )
> > > > > >    );
> > > > > >
> > > > > >    convex_hull_dist = (
> > > > > >       (            0.0, 1.0 )
> > > > > >       ( 400.0/grid_res, 0.0 )
> > > > > >    );
> > > > > >
> > > > > >    angle_diff = (
> > > > > >       (  0.0, 1.0 )
> > > > > >       ( 30.0, 1.0 )
> > > > > >       ( 90.0, 0.0 )
> > > > > >    corner   = 0.8;
> > > > > >    ratio_if = (
> > > > > >       (    0.0, 0.0 )
> > > > > >       ( corner, 1.0 )
> > > > > >       (    1.0, 1.0 )
> > > > > >    );
> > > > > >
> > > > > >    area_ratio = ratio_if;
> > > > > >
> > > > > >    int_area_ratio = (
> > > > > >       ( 0.00, 0.00 )
> > > > > >       ( 0.10, 0.50 )
> > > > > >       ( 0.25, 1.00 )
> > > > > >       ( 1.00, 1.00 )
> > > > > >    );
> > > > > >
> > > > > >    complexity_ratio = ratio_if;
> > > > > >
> > > > > >    inten_perc_ratio = ratio_if;
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Total interest threshold for determining matches
> > > > > > //
> > > > > > total_interest_thresh = 0.5;
> > > > > >
> > > > > > //
> > > > > > // Interest threshold for printing output pair information
> > > > > > //
> > > > > > print_interest_thresh = 0.0;
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Plotting information
> > > > > > //
> > > > > > met_data_dir = "MET_BASE/data";
> > > > > >
> > > > > > fcst_raw_plot = {
> > > > > >    color_table      =
> > "MET_BASE/data/colortables/met_default.ctable";
> > > > > >    plot_min         = 0.0;
> > > > > >    plot_max         = 0.0;
> > > > > >    colorbar_spacing = 1;
> > > > > > };
> > > > > >
> > > > > > obs_raw_plot = {
> > > > > >    color_table      =
> > "MET_BASE/data/colortables/met_default.ctable";
> > > > > >    plot_min         = 0.0;
> > > > > >    plot_max         = 0.0;
> > > > > >    colorbar_spacing = 1;
> > > > > > };
> > > > > >
> > > > > > object_plot = {
> > > > > >    color_table      =
>    "MET_BASE/data/colortables/mode_obj.ctable";
> > > > > > };
> > > > > >
> > > > > > //
> > > > > > // Number of grid boxes to fill with bad data values along
>    the edge
> > > of
> > > > > the
> > > > > > field
> > > > > > // to avoid edge effects.
> > > > > > //
> > > > > > zero_border_size = 1;
> > > > > >
> > > > > > //
> > > > > > // Boolean for plotting on the region of valid data within
>    the
> > domain
> > > > > > //
> > > > > > plot_valid_flag = FALSE;
> > > > > >
> > > > > > //
> > > > > > // Plot polyline edges using great circle arcs instead of
>    straight
> > > > lines
> > > > > > //
> > > > > > plot_gcarc_flag = FALSE;
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // NetCDF matched pairs, PostScript, and contingency table
>    output
> > > files
> > > > > > //
> > > > > > ps_plot_flag    = TRUE;
> > > > > > nc_pairs_flag   = TRUE;
> > > > > > ct_stats_flag   = TRUE;
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > output_prefix  = "";
> > > > > > version        = "V4.1";
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > >
> > > > > > This email is intended solely for the recipient. It may
>    contain
> > > > > > privileged, proprietary or confidential information or
>    material. If
> > > you
> > > > > are
> > > > > > not the intended recipient, please delete this email and
any
> > > > attachments
> > > > > > and notify the sender of the error.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > ________________________________
> > > > >
> > > > > This email is intended solely for the recipient. It may
>    contain
> > > > > privileged, proprietary or confidential information or
>    material. If
> > you
> > > > are
> > > > > not the intended recipient, please delete this email and any
> > > attachments
> > > > > and notify the sender of the error.
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > ________________________________
> > > >
> > > > This email is intended solely for the recipient. It may
contain
> > > > privileged, proprietary or confidential information or
material.
>    If you
> > > are
> > > > not the intended recipient, please delete this email and any
> > attachments
> > > > and notify the sender of the error.
> > > >
> > > >
> > >
> > >
> > > ________________________________
> > >
> > > This email is intended solely for the recipient. It may contain
> > > privileged, proprietary or confidential information or material.
>    If you
> > are
> > > not the intended recipient, please delete this email and any
>    attachments
> > > and notify the sender of the error.
> > >
> > >
> > >
> > >
> >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material.
If
>    you are
> > not the intended recipient, please delete this email and any
>    attachments
> > and notify the sender of the error.
> >
> >
> >
> >
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
>    privileged, proprietary or confidential information or material.
If
>    you are not the intended recipient, please delete this email and
>    any attachments and notify the sender of the error.
>
>



------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #71629] problems with MODE not identifying objects in field
From: Adams-Selin, Becky
Time: Sun May 03 07:01:22 2015

Tara,

Nice to "meet" you.  I only have about 10 days worth of data, so I'm
not sure it's worth breaking things out by storm morphologies.  I was
able to get pretty good results with just using different
conv_thresholds for the different members as necessary.  Calculating a
percentile threshold for each member over the entire date range would
give me a defensible reason for using that specific threshold.  I'll
give that a try.

The data that I'm trying to verify is actually the HAILCAST forecasts
from the Spring Experiment last year.  I'm headed down to this year's
experiment this week - I see your name on the list of attendees!  I
look forward to meeting you in person.

Becky Adams-Selin, Ph.D.
Staff Scientist II, AER

________________________________________
From: Tara Jensen via RT [met_help at ucar.edu]
Sent: Friday, May 01, 2015 4:50 PM
To: Adams-Selin, Becky
Subject: [rt.rap.ucar.edu #71629] problems with MODE not identifying
objects in field

Hi Becky,

This is Tara Jensen.  I'm one of the scientists on the MET team.  The
method John suggested is exploratory in the sense that we have not
tried it much ourselves either.  While we do suggest it would probably
be best to look at your data as a whole, I wouldn't rule out the need
to maybe stratify your data by storm morphologies or forcing
mechanisms before determining your thresholds.  I would hope that you
wouldn't have to adjust the thresholds on a case by case basis because
that would likely be less scientifically defensible.  Hopefully that
helps you get started.  Please do not hesitate to let us know how your
analysis is progressing and if you have further questions.

Thanks, Tara Jensen

On Fri May 01 15:11:36 2015, RSelin at aer.com wrote:
> I agree de-biasing the ensemble members would be ideal.  However,
I'm
>    actually comparing two different ensemble configurations, one
where
>    the bias is largely mitigated by improvements to my hail
algorithm,
>    so I want to be able to show that improvement in the statistics.
>
> I like the idea of using a percentile of the raw data values!  That
>    would be simple to implement in python.  Would it be best to
>    calculate the percentile of all of the data over all of my dates,
>    or re-calculate the percentile for each date?
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Friday, May 01, 2015 3:23 PM
> To: Adams-Selin, Becky
> Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
>    identifying objects in field
>
> Becky,
>
> I talked to one of the scientists here, and we have a few
thoughts...
>
> (1) When working with an ensemble, one important step is ensemble
> post-processing, which often includes a de-biasing step.  If you
have
>    a
> post-processed de-biased version of this data, I'd suggest
evaluating
> that.  But I'm assuming you don't.
>
> (2) Next, I hear what you're saying that the varying biases makes it
> impossible to find of configuration of MODE that works for all the
> members.  In this case, using a different threshold for each member
>    would
> be a reasonable approach.  But it'd be nice to do so in a systematic
>    way.
> For example, you could define the threshold as a particular
percentile
>    of
> the raw data values.  And you could apply the same percentile
>    threshold to
> all the members.  But for each member, the actual threshold value
>    would
> change.
>
> Using a percentile threshold is basically a simplistic bias removal.
>
> We do have plans to add percentile thresholding directly to the MET
>    tools,
> but that functionality won't be available for about a year.  In the
>    mean
> time, here's what I'd suggest.  Write a wrapper script that...
>    - Looks at the forecast values, ignoring the 0's, and computes
the
> percentile of interest.
>    - Set that value as an environment variable, such as "setenv
> FCST_PERC_THRESH 1.75"
>    - Configure MODE to use that environment variable.  In the "fcst"
> section, set "conv_thresh = >=${FCST_PERC_THRESH};"
>    - Then run MODE for that case.
>
> As for how to compute the percentile value of interest, you could
use
>    a
> variety of scripting tools.  I'd probably use R since I'm most
>    familiar
> with that, but Matlab, IDL, or even a Python script could probably
do
>    it.
> Adding percentile thresholding directly to MET will make this much
>    easier
> in the future.
>
> Let me know if you get hung up at this step, and I might be able to
>    help
> out.
>
> (3) Lastly, you might also consider running this data through the
>    Grid-Stat
> tool which can be used to compute neighborhood statistics, such as
> Fractions Skill Score.  It really all depends on the questions
you're
> trying to answer about your dataset, but FSS is a nice simple
>    algorithm
> that might give you some useful info about your data.
>
> Hope that helps.
>
> Thanks,
> John
>
> On Fri, May 1, 2015 at 10:03 AM, Adams-Selin, Becky via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> >
> > John,
> >
> > OK, that makes sense, thank you.
> >
> > Like you say, ideally I'd like my configuration file to be the
same
>    for
> > all my forecast members.  However, because each member is biased
> > differently, setting a conv_threshold value that makes sense for
one
> > member, results in really large forecast clusters being identified
>    in
> > another member.  Is there a good way to limit the size of the
>    cluster
> > objects without changing the conv_threshold?
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, April 27, 2015 1:33 PM
> > To: Adams-Selin, Becky
> > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
>    identifying
> > objects in field
> >
> > Becky,
> >
> > MODE attempts to do things intelligently here.  If the forecast
and
> > observation field names are identical, it chooses a consistent
color
>    scale
> > using the range of both.  Your data is not named the same thing
> > (MAX_HAIL_24 vs MESH_hail) so it plots them using the range of the
>    data in
> > each.
> >
> > As for interpreting the output, ultimately it's up to you.  You
>    mention
> > that you're dealing with an ensemble with varying biases.  I would
>    suggest
> > finding a configuration of MODE that identifies the objects of
>    interest and
> > helps you answer your scientific questions... and then apply the
>    same
> > configuration to all members.  Changing the configuration for each
>    member
> > would be pretty confusing.
> >
> > I can help you with the nuts and bolts of running MODE, but one of
>    the
> > scientists here would be better at helping you interpret results.
>    Before
> > presenting results, we could probably to take a quick look to make
>    sure we
> > don't see any red flags if you'd like.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Mon, Apr 27, 2015 at 12:07 PM, Adams-Selin, Becky via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> > >
> > > Wow, yes, that would do it.  I didn't realize the
>    forecasted/observed
> > > plots on the first page of the output .ps file had different
color
> > scales!
> > > Thank you for the help and the suggested Mode config file.
> > >
> > > If I were to report on these results obtained using a different
> > > conv_threshold, would I just need to acknowledge a pre-existing
>    bias?
> > The
> > > problem is complicated by the fact that I am verifying the
>    forecasts of
> > > multiple ensemble members, not all of whom are so biased.
> > >
> > > Becky Adams-Selin, Ph.D.
> > > Staff Scientist II, AER
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Monday, April 27, 2015 12:03 PM
> > > To: Adams-Selin, Becky
> > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
>    identifying
> > > objects in field
> > >
> > > Becky,
> > >
> > > Thanks for sending the data.  I ran the following two
>    plot_data_plane
> > (from
> > > MET) commands to plot this data using the same scale of 0 to
1.25.
>    The
> > > resulting images are attached:
> > >
> > > plot_data_plane \
> > >    HailSize20140508L1S_caps_maxnn_2.0km.nc \
> > >    HailSize20140508L1S_caps_maxnn_2.0km.ps \
> > >    'name="MESH_hail"; level="(*,*)";' -plot_range 0 1.25
> > >
> > > plot_data_plane \
> > >    s4m17_arw_140508_hailsum.nc \
> > >    s4m17_arw_140508_hailsum.ps \
> > >    'name="MAX_HAIL_24"; level="(*,*)";' -plot_range 0 1.25
> > >
> > > Looking at them, I notice that the observed values are much
higher
>    than
> > the
> > > forecast values!  Very few forecast values are above 1.0 while
>    most
> > > observed values are above 1.0.  When you apply a raw thresh of
>    >=1.0,
> > > you're zeroing out almost all of the forecast points but
>    relatively few
> > > observation points.  That's why you're getting so many more
>    observation
> > > objects than forecast objects.
> > >
> > > Inspecting the data more closely, I see...
> > >  - the range on non-zero observed values is 0.50 to 3.74
> > >  - the range of non-zero forecast values is 0.01 to 1.39
> > >
> > > Based on that, I made the following changes to the config
file...
> > >  - switch the raw_thresh from >=1.0 to >=0.5
> > >  - increase the smoothing radius (conv_radius) from 1 to 4
> > >  - set the plot_max in the fcst_raw_plot and obs_raw_plot
sections
>    to
> > 3.75
> > > to use a consistent scale
> > >  - set the forecast conv_thresh as >=0.25 and observed
conv_thresh
>    as
> > > >=0.5.
> > >
> > > This resulted in 45 forecast and 44 observed objects.  We
>    typically use
> > the
> > > same threshold for both forecast and observation, but your
>    forecast has a
> > > very clear low bias, by about a factor of 2.  That's why I chose
>    the
> > > observed threshold to be 2 times the forecast threshold.
> > >
> > > The config file and resulting output image is attached.
>    Unfortunately,
> > > this takes a long time to run, about 5 min on my machine!
> > >
> > > I'd suggest that you play around with the settings more.  You
>    should
> > decide
> > > what "scale" of objects you're looking for.  If you want a
smaller
>    number
> > > of smoother objects, try increasing the convolution radius and
>    lowering
> > the
> > > threshold.  The fewer the number of objects MODE has to keep
track
>    of,
> > the
> > > faster it'll run.
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Sat, Apr 25, 2015 at 9:19 AM, Adams-Selin, Becky via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
>
> > > >
> > > > I tried lowering the conv_radius to 0, and the conv_threshold
to
> > 0.25.  I
> > > > attached what I got.  I'm new to MODE, but just from visually
>    comparing
> > > the
> > > > forecast and observation fields, I feel like I should be
getting
> > roughly
> > > > similar numbers of objects....I would even have expected fewer
>    objects
> > > from
> > > > the observation field.  Something seems wrong.
> > > >
> > > > I went ahead and created a selin_data directory on the passive
>    ftp
> > site.
> > > >
> > > > The forecast file is s4m17_arw_140508_hailsum.nc
> > > > The observation file is
HailSize20140508L1S_caps_maxnn_2.0km.nc
> > > > And the mode config file is MODEConfig_bigintense
> > > >
> > > > If you could take a look I would really appreciate it.
> > > >
> > > > Becky Adams-Selin, Ph.D.
> > > > Staff Scientist II, AER
> > > >
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Saturday, April 25, 2015 9:25 AM
> > > > To: Adams-Selin, Becky
> > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> > identifying
> > > > objects in field
> > > >
> > > > Becky,
> > > >
> > > > OK, sounds good.  You could try setting the convolution radius
>    to 0
> > just
> > > to
> > > > see if you can get forecast objects.  Or try lowering the
>    threshold.
> > > > Perhaps they're all getting smoothed out.
> > > >
> > > > If you're still stuck I could take a look at some sample
files.
>    I'd
> > > need a
> > > > sample forecast, observation, and MODE config file.  You could
>    post
> > them
> > > to
> > > > our anonymous for site following these instructions:
> > > >
> > > > http://www.dtcenter.org/met/users/support/met_help.php#ftp
> > > >
> > > > Thanks
> > > > John
> > > >
> > > >
> > > > On Saturday, April 25, 2015, Adams-Selin, Becky via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
>    >
> > > > >
> > > > > John,
> > > > >
> > > > > I think that situation actually might apply to my dataset.
>    The
> > > > > observational dataset only has hail values that are larger
>    than 1.0
> > in.
> > > > > The forecast dataset is continuous, from 0 inches upwards.
My
> > > > > understanding was I should set the raw_thresh to 1.0 to make
>    the
> > > forecast
> > > > > and observational datasets consistent, like you said below.
> > > > >
> > > > > Becky Adams-Selin, Ph.D.
> > > > > Staff Scientist II, AER
> > > > >
> > > > > ________________________________________
> > > > > From: John Halley Gotway via RT [met_help at ucar.edu
>    <javascript:;>]
> > > > > Sent: Friday, April 24, 2015 3:39 PM
> > > > > To: Adams-Selin, Becky
> > > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> > > identifying
> > > > > objects in field
> > > > >
> > > > > Hello Becky,
> > > > >
> > > > > I see that you're having trouble getting objects to show up
in
>    MODE.
> > > > > Thanks for sending the sample plot and your configuration
>    file.
> > > Looking
> > > > > closely at your choice of thresholds, I see a problem:
> > > > >    raw_thresh        = >=1.0;
> > > > >    conv_thresh       = >=0.5;
> > > > >
> > > > > I would suggest changing it to this:
> > > > >    raw_thresh        = >=0.0;
> > > > >    conv_thresh       = >=0.5;
> > > > >
> > > > > Try rerunning to see if you get objects.
> > > > >
> > > > > The "raw_thresh" setting should generally not be used unless
>    you
> > have a
> > > > > good reason to do so.  It is applied to the raw data values
>    and any
> > > point
> > > > > not meeting that criteria is set to a value of 0.  The
>    resulting
> > values
> > > > are
> > > > > passed through to the object definition step.
> > > > >
> > > > > Applying that logic doesn't make sense for your dataset.
You
>    may
> > > wonder
> > > > > why you would you ever want to do this?  A while back we
were
> > > evaluating
> > > > a
> > > > > reflectivity forecast, but the system only forecast values
of
>    35dBz
> > or
> > > > > higher.  So we had values of 35dBz surround by values of 0.
>    The
> > > > > observations were continuous, with values of 35dBZ
surrounded
>    by
> > 34dBZ
> > > > and
> > > > > so on.  When defining objects in MODE, we do a smoothing
step
>    and
> > > having
> > > > > values of 0 vs values of 34 affected the output a lot.  So
in
>    order
> > to
> > > > > define observation and forecast objects in a consistent way,
>    we
> > zeroed
> > > > out
> > > > > any observed values < 35.
> > > > >
> > > > > Since this doesn't apply to your dataset, setting it as
">=0"
>    should
> > > pass
> > > > > all of your data through to the object definition step.
> > > > >
> > > > > Hope that helps clarify.
> > > > >
> > > > > Thanks,
> > > > > John Halley Gotway
> > > > > met_help at ucar.edu <javascript:;>
> > > > >
> > > > >
> > > > > On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT <
> > > > > met_help at ucar.edu <javascript:;>> wrote:
> > > > >
> > > > > >
> > > > > > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > > > > > Transaction: Ticket created by RSelin at aer.com
<javascript:;>
> > > > > >        Queue: met_help
> > > > > >      Subject: problems with MODE not identifying objects
in
>    field
> > > > > >        Owner: Nobody
> > > > > >   Requestors: RSelin at aer.com <javascript:;>
> > > > > >       Status: new
> > > > > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> > > > >
> > > > > >
> > > > > >
> > > > > > Dear MET help,
> > > > > >
> > > > > > I am using MODE in MET v4.1 to evaluate a custom hail
>    forecast in
> > > > > > comparison to a gridded observed hail field (MESH).  I've
>    attached
> > > the
> > > > > > first page of the .ps output from one of my dates.  For
some
>    reason
> > > no
> > > > > > objects are being identified in the forecast field, but
>    judging
> > from
> > > > the
> > > > > > raw field, they should be.  My MODEConfig file is below.
>    Any idea
> > > what
> > > > > I'm
> > > > > > doing wrong?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Becky Adams-Selin, Ph.D.
> > > > > > Staff Scientist II, AER
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > //
> > > > > > // MODE configuration file.
> > > > > > //
> > > > > > // For additional information, see the
>    MET_BASE/data/config/README
> > > > file.
> > > > > > //
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Output model name to be written
> > > > > > //
> > > > > > model = "WRF";
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Approximate grid resolution (km)
> > > > > > //
> > > > > > grid_res = 4;
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Forecast and observation fields to be verified
> > > > > > //
> > > > > > fcst = {
> > > > > >    field = {
> > > > > >       name  = "MAX_HAIL_24";
> > > > > >       level = "(*,*)";
> > > > > >    };
> > > > > >
> > > > > >    raw_thresh        = >=1.0;
> > > > > >    conv_radius       = 4; // in grid squares
> > > > > >    conv_thresh       = >=0.5;
> > > > > >    vld_thresh        = 0.5;
> > > > > >    area_thresh       = >=4.0;
> > > > > >    inten_perc_value  = 100;
> > > > > >    inten_perc_thresh = >=0.0;
> > > > > >    merge_thresh      = >=0.5;
> > > > > >    merge_flag        = NONE;
> > > > > > };
> > > > > > obs = {
> > > > > >    field = {
> > > > > >       name = "MESH_hail";
> > > > > >       level = "(*,*)";
> > > > > >    };
> > > > > >
> > > > > >    raw_thresh        = >=1.0;
> > > > > >    conv_radius       = 4; // in grid squares
> > > > > >    conv_thresh       = >=0.5;
> > > > > >    vld_thresh        = 0.5;
> > > > > >    area_thresh       = >=4.0;
> > > > > >    inten_perc_value  = 100;
> > > > > >    inten_perc_thresh = >=0.0;
> > > > > >    merge_thresh      = >=0.5;
> > > > > >    merge_flag        = NONE;
> > > > > > };
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Handle missing data
> > > > > > //
> > > > > > mask_missing_flag = BOTH;
> > > > > >
> > > > > > //
> > > > > > // Match objects between the forecast and observation
fields
> > > > > > //
> > > > > > match_flag = MERGE_BOTH;
> > > > > >
> > > > > > //
> > > > > > // Maximum centroid distance for objects to be compared
> > > > > > //
> > > > > > max_centroid_dist = 800.0/grid_res;
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Verification masking regions
> > > > > > //
> > > > > > mask = {
> > > > > >    grid      = "";
> > > > > >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
> > > > > >    poly      =
>    "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> > > > > >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
> > > > > > };
> > > > > >
> > > > > > //
> > > > > > // Fuzzy engine weights
> > > > > > //
> > > > > > weight = {
> > > > > >    centroid_dist    = 2.0;
> > > > > >    boundary_dist    = 4.0;
> > > > > >    convex_hull_dist = 0.0;
> > > > > >    angle_diff       = 1.0;
> > > > > >    area_ratio       = 1.0;
> > > > > >    int_area_ratio   = 2.0;
> > > > > >    complexity_ratio = 0.0;
> > > > > >    inten_perc_ratio = 0.0;
> > > > > >    inten_perc_value = 100;
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Fuzzy engine interest functions
> > > > > > //
> > > > > > interest_function = {
> > > > > >
> > > > > >    centroid_dist = (
> > > > > >       (            0.0, 1.0 )
> > > > > >       (  60.0/grid_res, 1.0 )
> > > > > >       ( 600.0/grid_res, 0.0 )
> > > > > >    );
> > > > > >
> > > > > >    boundary_dist = (
> > > > > >       (            0.0, 1.0 )
> > > > > >       ( 400.0/grid_res, 0.0 )
> > > > > >    );
> > > > > >
> > > > > >    convex_hull_dist = (
> > > > > >       (            0.0, 1.0 )
> > > > > >       ( 400.0/grid_res, 0.0 )
> > > > > >    );
> > > > > >
> > > > > >    angle_diff = (
> > > > > >       (  0.0, 1.0 )
> > > > > >       ( 30.0, 1.0 )
> > > > > >       ( 90.0, 0.0 )
> > > > > >    corner   = 0.8;
> > > > > >    ratio_if = (
> > > > > >       (    0.0, 0.0 )
> > > > > >       ( corner, 1.0 )
> > > > > >       (    1.0, 1.0 )
> > > > > >    );
> > > > > >
> > > > > >    area_ratio = ratio_if;
> > > > > >
> > > > > >    int_area_ratio = (
> > > > > >       ( 0.00, 0.00 )
> > > > > >       ( 0.10, 0.50 )
> > > > > >       ( 0.25, 1.00 )
> > > > > >       ( 1.00, 1.00 )
> > > > > >    );
> > > > > >
> > > > > >    complexity_ratio = ratio_if;
> > > > > >
> > > > > >    inten_perc_ratio = ratio_if;
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Total interest threshold for determining matches
> > > > > > //
> > > > > > total_interest_thresh = 0.5;
> > > > > >
> > > > > > //
> > > > > > // Interest threshold for printing output pair information
> > > > > > //
> > > > > > print_interest_thresh = 0.0;
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // Plotting information
> > > > > > //
> > > > > > met_data_dir = "MET_BASE/data";
> > > > > >
> > > > > > fcst_raw_plot = {
> > > > > >    color_table      =
> > "MET_BASE/data/colortables/met_default.ctable";
> > > > > >    plot_min         = 0.0;
> > > > > >    plot_max         = 0.0;
> > > > > >    colorbar_spacing = 1;
> > > > > > };
> > > > > >
> > > > > > obs_raw_plot = {
> > > > > >    color_table      =
> > "MET_BASE/data/colortables/met_default.ctable";
> > > > > >    plot_min         = 0.0;
> > > > > >    plot_max         = 0.0;
> > > > > >    colorbar_spacing = 1;
> > > > > > };
> > > > > >
> > > > > > object_plot = {
> > > > > >    color_table      =
>    "MET_BASE/data/colortables/mode_obj.ctable";
> > > > > > };
> > > > > >
> > > > > > //
> > > > > > // Number of grid boxes to fill with bad data values along
>    the edge
> > > of
> > > > > the
> > > > > > field
> > > > > > // to avoid edge effects.
> > > > > > //
> > > > > > zero_border_size = 1;
> > > > > >
> > > > > > //
> > > > > > // Boolean for plotting on the region of valid data within
>    the
> > domain
> > > > > > //
> > > > > > plot_valid_flag = FALSE;
> > > > > >
> > > > > > //
> > > > > > // Plot polyline edges using great circle arcs instead of
>    straight
> > > > lines
> > > > > > //
> > > > > > plot_gcarc_flag = FALSE;
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > //
> > > > > > // NetCDF matched pairs, PostScript, and contingency table
>    output
> > > files
> > > > > > //
> > > > > > ps_plot_flag    = TRUE;
> > > > > > nc_pairs_flag   = TRUE;
> > > > > > ct_stats_flag   = TRUE;
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > > output_prefix  = "";
> > > > > > version        = "V4.1";
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > >
> > > > > > This email is intended solely for the recipient. It may
>    contain
> > > > > > privileged, proprietary or confidential information or
>    material. If
> > > you
> > > > > are
> > > > > > not the intended recipient, please delete this email and
any
> > > > attachments
> > > > > > and notify the sender of the error.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > ________________________________
> > > > >
> > > > > This email is intended solely for the recipient. It may
>    contain
> > > > > privileged, proprietary or confidential information or
>    material. If
> > you
> > > > are
> > > > > not the intended recipient, please delete this email and any
> > > attachments
> > > > > and notify the sender of the error.
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > ________________________________
> > > >
> > > > This email is intended solely for the recipient. It may
contain
> > > > privileged, proprietary or confidential information or
material.
>    If you
> > > are
> > > > not the intended recipient, please delete this email and any
> > attachments
> > > > and notify the sender of the error.
> > > >
> > > >
> > >
> > >
> > > ________________________________
> > >
> > > This email is intended solely for the recipient. It may contain
> > > privileged, proprietary or confidential information or material.
>    If you
> > are
> > > not the intended recipient, please delete this email and any
>    attachments
> > > and notify the sender of the error.
> > >
> > >
> > >
> > >
> >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material.
If
>    you are
> > not the intended recipient, please delete this email and any
>    attachments
> > and notify the sender of the error.
> >
> >
> >
> >
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
>    privileged, proprietary or confidential information or material.
If
>    you are not the intended recipient, please delete this email and
>    any attachments and notify the sender of the error.
>
>



________________________________

This email is intended solely for the recipient. It may contain
privileged, proprietary or confidential information or material. If
you are not the intended recipient, please delete this email and any
attachments and notify the sender of the error.



------------------------------------------------
Subject: problems with MODE not identifying objects in field
From: John Halley Gotway
Time: Mon May 04 09:36:50 2015

Becky,

I'll go ahead and resolve this ticket now.  Just let us know if more
issues
or questions come up in your use of MET and MODE.

Thanks,
John

On Sun, May 3, 2015 at 7:01 AM, Adams-Selin, Becky via RT
<met_help at ucar.edu
> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
> Tara,
>
> Nice to "meet" you.  I only have about 10 days worth of data, so I'm
not
> sure it's worth breaking things out by storm morphologies.  I was
able to
> get pretty good results with just using different conv_thresholds
for the
> different members as necessary.  Calculating a percentile threshold
for
> each member over the entire date range would give me a defensible
reason
> for using that specific threshold.  I'll give that a try.
>
> The data that I'm trying to verify is actually the HAILCAST
forecasts from
> the Spring Experiment last year.  I'm headed down to this year's
experiment
> this week - I see your name on the list of attendees!  I look
forward to
> meeting you in person.
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: Tara Jensen via RT [met_help at ucar.edu]
> Sent: Friday, May 01, 2015 4:50 PM
> To: Adams-Selin, Becky
> Subject: [rt.rap.ucar.edu #71629] problems with MODE not identifying
> objects in field
>
> Hi Becky,
>
> This is Tara Jensen.  I'm one of the scientists on the MET team.
The
> method John suggested is exploratory in the sense that we have not
tried it
> much ourselves either.  While we do suggest it would probably be
best to
> look at your data as a whole, I wouldn't rule out the need to maybe
> stratify your data by storm morphologies or forcing mechanisms
before
> determining your thresholds.  I would hope that you wouldn't have to
adjust
> the thresholds on a case by case basis because that would likely be
less
> scientifically defensible.  Hopefully that helps you get started.
Please
> do not hesitate to let us know how your analysis is progressing and
if you
> have further questions.
>
> Thanks, Tara Jensen
>
> On Fri May 01 15:11:36 2015, RSelin at aer.com wrote:
> > I agree de-biasing the ensemble members would be ideal.  However,
I'm
> >    actually comparing two different ensemble configurations, one
where
> >    the bias is largely mitigated by improvements to my hail
algorithm,
> >    so I want to be able to show that improvement in the
statistics.
> >
> > I like the idea of using a percentile of the raw data values!
That
> >    would be simple to implement in python.  Would it be best to
> >    calculate the percentile of all of the data over all of my
dates,
> >    or re-calculate the percentile for each date?
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Friday, May 01, 2015 3:23 PM
> > To: Adams-Selin, Becky
> > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> >    identifying objects in field
> >
> > Becky,
> >
> > I talked to one of the scientists here, and we have a few
thoughts...
> >
> > (1) When working with an ensemble, one important step is ensemble
> > post-processing, which often includes a de-biasing step.  If you
have
> >    a
> > post-processed de-biased version of this data, I'd suggest
evaluating
> > that.  But I'm assuming you don't.
> >
> > (2) Next, I hear what you're saying that the varying biases makes
it
> > impossible to find of configuration of MODE that works for all the
> > members.  In this case, using a different threshold for each
member
> >    would
> > be a reasonable approach.  But it'd be nice to do so in a
systematic
> >    way.
> > For example, you could define the threshold as a particular
percentile
> >    of
> > the raw data values.  And you could apply the same percentile
> >    threshold to
> > all the members.  But for each member, the actual threshold value
> >    would
> > change.
> >
> > Using a percentile threshold is basically a simplistic bias
removal.
> >
> > We do have plans to add percentile thresholding directly to the
MET
> >    tools,
> > but that functionality won't be available for about a year.  In
the
> >    mean
> > time, here's what I'd suggest.  Write a wrapper script that...
> >    - Looks at the forecast values, ignoring the 0's, and computes
the
> > percentile of interest.
> >    - Set that value as an environment variable, such as "setenv
> > FCST_PERC_THRESH 1.75"
> >    - Configure MODE to use that environment variable.  In the
"fcst"
> > section, set "conv_thresh = >=${FCST_PERC_THRESH};"
> >    - Then run MODE for that case.
> >
> > As for how to compute the percentile value of interest, you could
use
> >    a
> > variety of scripting tools.  I'd probably use R since I'm most
> >    familiar
> > with that, but Matlab, IDL, or even a Python script could probably
do
> >    it.
> > Adding percentile thresholding directly to MET will make this much
> >    easier
> > in the future.
> >
> > Let me know if you get hung up at this step, and I might be able
to
> >    help
> > out.
> >
> > (3) Lastly, you might also consider running this data through the
> >    Grid-Stat
> > tool which can be used to compute neighborhood statistics, such as
> > Fractions Skill Score.  It really all depends on the questions
you're
> > trying to answer about your dataset, but FSS is a nice simple
> >    algorithm
> > that might give you some useful info about your data.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> > On Fri, May 1, 2015 at 10:03 AM, Adams-Selin, Becky via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> > >
> > > John,
> > >
> > > OK, that makes sense, thank you.
> > >
> > > Like you say, ideally I'd like my configuration file to be the
same
> >    for
> > > all my forecast members.  However, because each member is biased
> > > differently, setting a conv_threshold value that makes sense for
one
> > > member, results in really large forecast clusters being
identified
> >    in
> > > another member.  Is there a good way to limit the size of the
> >    cluster
> > > objects without changing the conv_threshold?
> > >
> > > Becky Adams-Selin, Ph.D.
> > > Staff Scientist II, AER
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Monday, April 27, 2015 1:33 PM
> > > To: Adams-Selin, Becky
> > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> >    identifying
> > > objects in field
> > >
> > > Becky,
> > >
> > > MODE attempts to do things intelligently here.  If the forecast
and
> > > observation field names are identical, it chooses a consistent
color
> >    scale
> > > using the range of both.  Your data is not named the same thing
> > > (MAX_HAIL_24 vs MESH_hail) so it plots them using the range of
the
> >    data in
> > > each.
> > >
> > > As for interpreting the output, ultimately it's up to you.  You
> >    mention
> > > that you're dealing with an ensemble with varying biases.  I
would
> >    suggest
> > > finding a configuration of MODE that identifies the objects of
> >    interest and
> > > helps you answer your scientific questions... and then apply the
> >    same
> > > configuration to all members.  Changing the configuration for
each
> >    member
> > > would be pretty confusing.
> > >
> > > I can help you with the nuts and bolts of running MODE, but one
of
> >    the
> > > scientists here would be better at helping you interpret
results.
> >    Before
> > > presenting results, we could probably to take a quick look to
make
> >    sure we
> > > don't see any red flags if you'd like.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Mon, Apr 27, 2015 at 12:07 PM, Adams-Selin, Becky via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
>
> > > >
> > > > Wow, yes, that would do it.  I didn't realize the
> >    forecasted/observed
> > > > plots on the first page of the output .ps file had different
color
> > > scales!
> > > > Thank you for the help and the suggested Mode config file.
> > > >
> > > > If I were to report on these results obtained using a
different
> > > > conv_threshold, would I just need to acknowledge a pre-
existing
> >    bias?
> > > The
> > > > problem is complicated by the fact that I am verifying the
> >    forecasts of
> > > > multiple ensemble members, not all of whom are so biased.
> > > >
> > > > Becky Adams-Selin, Ph.D.
> > > > Staff Scientist II, AER
> > > >
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Monday, April 27, 2015 12:03 PM
> > > > To: Adams-Selin, Becky
> > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> >    identifying
> > > > objects in field
> > > >
> > > > Becky,
> > > >
> > > > Thanks for sending the data.  I ran the following two
> >    plot_data_plane
> > > (from
> > > > MET) commands to plot this data using the same scale of 0 to
1.25.
> >    The
> > > > resulting images are attached:
> > > >
> > > > plot_data_plane \
> > > >    HailSize20140508L1S_caps_maxnn_2.0km.nc \
> > > >    HailSize20140508L1S_caps_maxnn_2.0km.ps \
> > > >    'name="MESH_hail"; level="(*,*)";' -plot_range 0 1.25
> > > >
> > > > plot_data_plane \
> > > >    s4m17_arw_140508_hailsum.nc \
> > > >    s4m17_arw_140508_hailsum.ps \
> > > >    'name="MAX_HAIL_24"; level="(*,*)";' -plot_range 0 1.25
> > > >
> > > > Looking at them, I notice that the observed values are much
higher
> >    than
> > > the
> > > > forecast values!  Very few forecast values are above 1.0 while
> >    most
> > > > observed values are above 1.0.  When you apply a raw thresh of
> >    >=1.0,
> > > > you're zeroing out almost all of the forecast points but
> >    relatively few
> > > > observation points.  That's why you're getting so many more
> >    observation
> > > > objects than forecast objects.
> > > >
> > > > Inspecting the data more closely, I see...
> > > >  - the range on non-zero observed values is 0.50 to 3.74
> > > >  - the range of non-zero forecast values is 0.01 to 1.39
> > > >
> > > > Based on that, I made the following changes to the config
file...
> > > >  - switch the raw_thresh from >=1.0 to >=0.5
> > > >  - increase the smoothing radius (conv_radius) from 1 to 4
> > > >  - set the plot_max in the fcst_raw_plot and obs_raw_plot
sections
> >    to
> > > 3.75
> > > > to use a consistent scale
> > > >  - set the forecast conv_thresh as >=0.25 and observed
conv_thresh
> >    as
> > > > >=0.5.
> > > >
> > > > This resulted in 45 forecast and 44 observed objects.  We
> >    typically use
> > > the
> > > > same threshold for both forecast and observation, but your
> >    forecast has a
> > > > very clear low bias, by about a factor of 2.  That's why I
chose
> >    the
> > > > observed threshold to be 2 times the forecast threshold.
> > > >
> > > > The config file and resulting output image is attached.
> >    Unfortunately,
> > > > this takes a long time to run, about 5 min on my machine!
> > > >
> > > > I'd suggest that you play around with the settings more.  You
> >    should
> > > decide
> > > > what "scale" of objects you're looking for.  If you want a
smaller
> >    number
> > > > of smoother objects, try increasing the convolution radius and
> >    lowering
> > > the
> > > > threshold.  The fewer the number of objects MODE has to keep
track
> >    of,
> > > the
> > > > faster it'll run.
> > > >
> > > > Hope that helps.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Sat, Apr 25, 2015 at 9:19 AM, Adams-Selin, Becky via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> > > > >
> > > > > I tried lowering the conv_radius to 0, and the
conv_threshold to
> > > 0.25.  I
> > > > > attached what I got.  I'm new to MODE, but just from
visually
> >    comparing
> > > > the
> > > > > forecast and observation fields, I feel like I should be
getting
> > > roughly
> > > > > similar numbers of objects....I would even have expected
fewer
> >    objects
> > > > from
> > > > > the observation field.  Something seems wrong.
> > > > >
> > > > > I went ahead and created a selin_data directory on the
passive
> >    ftp
> > > site.
> > > > >
> > > > > The forecast file is s4m17_arw_140508_hailsum.nc
> > > > > The observation file is
HailSize20140508L1S_caps_maxnn_2.0km.nc
> > > > > And the mode config file is MODEConfig_bigintense
> > > > >
> > > > > If you could take a look I would really appreciate it.
> > > > >
> > > > > Becky Adams-Selin, Ph.D.
> > > > > Staff Scientist II, AER
> > > > >
> > > > > ________________________________________
> > > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > Sent: Saturday, April 25, 2015 9:25 AM
> > > > > To: Adams-Selin, Becky
> > > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> > > identifying
> > > > > objects in field
> > > > >
> > > > > Becky,
> > > > >
> > > > > OK, sounds good.  You could try setting the convolution
radius
> >    to 0
> > > just
> > > > to
> > > > > see if you can get forecast objects.  Or try lowering the
> >    threshold.
> > > > > Perhaps they're all getting smoothed out.
> > > > >
> > > > > If you're still stuck I could take a look at some sample
files.
> >    I'd
> > > > need a
> > > > > sample forecast, observation, and MODE config file.  You
could
> >    post
> > > them
> > > > to
> > > > > our anonymous for site following these instructions:
> > > > >
> > > > > http://www.dtcenter.org/met/users/support/met_help.php#ftp
> > > > >
> > > > > Thanks
> > > > > John
> > > > >
> > > > >
> > > > > On Saturday, April 25, 2015, Adams-Selin, Becky via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> >    >
> > > > > >
> > > > > > John,
> > > > > >
> > > > > > I think that situation actually might apply to my dataset.
> >    The
> > > > > > observational dataset only has hail values that are larger
> >    than 1.0
> > > in.
> > > > > > The forecast dataset is continuous, from 0 inches upwards.
My
> > > > > > understanding was I should set the raw_thresh to 1.0 to
make
> >    the
> > > > forecast
> > > > > > and observational datasets consistent, like you said
below.
> > > > > >
> > > > > > Becky Adams-Selin, Ph.D.
> > > > > > Staff Scientist II, AER
> > > > > >
> > > > > > ________________________________________
> > > > > > From: John Halley Gotway via RT [met_help at ucar.edu
> >    <javascript:;>]
> > > > > > Sent: Friday, April 24, 2015 3:39 PM
> > > > > > To: Adams-Selin, Becky
> > > > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE
not
> > > > identifying
> > > > > > objects in field
> > > > > >
> > > > > > Hello Becky,
> > > > > >
> > > > > > I see that you're having trouble getting objects to show
up in
> >    MODE.
> > > > > > Thanks for sending the sample plot and your configuration
> >    file.
> > > > Looking
> > > > > > closely at your choice of thresholds, I see a problem:
> > > > > >    raw_thresh        = >=1.0;
> > > > > >    conv_thresh       = >=0.5;
> > > > > >
> > > > > > I would suggest changing it to this:
> > > > > >    raw_thresh        = >=0.0;
> > > > > >    conv_thresh       = >=0.5;
> > > > > >
> > > > > > Try rerunning to see if you get objects.
> > > > > >
> > > > > > The "raw_thresh" setting should generally not be used
unless
> >    you
> > > have a
> > > > > > good reason to do so.  It is applied to the raw data
values
> >    and any
> > > > point
> > > > > > not meeting that criteria is set to a value of 0.  The
> >    resulting
> > > values
> > > > > are
> > > > > > passed through to the object definition step.
> > > > > >
> > > > > > Applying that logic doesn't make sense for your dataset.
You
> >    may
> > > > wonder
> > > > > > why you would you ever want to do this?  A while back we
were
> > > > evaluating
> > > > > a
> > > > > > reflectivity forecast, but the system only forecast values
of
> >    35dBz
> > > or
> > > > > > higher.  So we had values of 35dBz surround by values of
0.
> >    The
> > > > > > observations were continuous, with values of 35dBZ
surrounded
> >    by
> > > 34dBZ
> > > > > and
> > > > > > so on.  When defining objects in MODE, we do a smoothing
step
> >    and
> > > > having
> > > > > > values of 0 vs values of 34 affected the output a lot.  So
in
> >    order
> > > to
> > > > > > define observation and forecast objects in a consistent
way,
> >    we
> > > zeroed
> > > > > out
> > > > > > any observed values < 35.
> > > > > >
> > > > > > Since this doesn't apply to your dataset, setting it as
">=0"
> >    should
> > > > pass
> > > > > > all of your data through to the object definition step.
> > > > > >
> > > > > > Hope that helps clarify.
> > > > > >
> > > > > > Thanks,
> > > > > > John Halley Gotway
> > > > > > met_help at ucar.edu <javascript:;>
> > > > > >
> > > > > >
> > > > > > On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT
<
> > > > > > met_help at ucar.edu <javascript:;>> wrote:
> > > > > >
> > > > > > >
> > > > > > > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > > > > > > Transaction: Ticket created by RSelin at aer.com
<javascript:;>
> > > > > > >        Queue: met_help
> > > > > > >      Subject: problems with MODE not identifying objects
in
> >    field
> > > > > > >        Owner: Nobody
> > > > > > >   Requestors: RSelin at aer.com <javascript:;>
> > > > > > >       Status: new
> > > > > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Dear MET help,
> > > > > > >
> > > > > > > I am using MODE in MET v4.1 to evaluate a custom hail
> >    forecast in
> > > > > > > comparison to a gridded observed hail field (MESH).
I've
> >    attached
> > > > the
> > > > > > > first page of the .ps output from one of my dates.  For
some
> >    reason
> > > > no
> > > > > > > objects are being identified in the forecast field, but
> >    judging
> > > from
> > > > > the
> > > > > > > raw field, they should be.  My MODEConfig file is below.
> >    Any idea
> > > > what
> > > > > > I'm
> > > > > > > doing wrong?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Becky Adams-Selin, Ph.D.
> > > > > > > Staff Scientist II, AER
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > > //
> > > > > > > // MODE configuration file.
> > > > > > > //
> > > > > > > // For additional information, see the
> >    MET_BASE/data/config/README
> > > > > file.
> > > > > > > //
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Output model name to be written
> > > > > > > //
> > > > > > > model = "WRF";
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Approximate grid resolution (km)
> > > > > > > //
> > > > > > > grid_res = 4;
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Forecast and observation fields to be verified
> > > > > > > //
> > > > > > > fcst = {
> > > > > > >    field = {
> > > > > > >       name  = "MAX_HAIL_24";
> > > > > > >       level = "(*,*)";
> > > > > > >    };
> > > > > > >
> > > > > > >    raw_thresh        = >=1.0;
> > > > > > >    conv_radius       = 4; // in grid squares
> > > > > > >    conv_thresh       = >=0.5;
> > > > > > >    vld_thresh        = 0.5;
> > > > > > >    area_thresh       = >=4.0;
> > > > > > >    inten_perc_value  = 100;
> > > > > > >    inten_perc_thresh = >=0.0;
> > > > > > >    merge_thresh      = >=0.5;
> > > > > > >    merge_flag        = NONE;
> > > > > > > };
> > > > > > > obs = {
> > > > > > >    field = {
> > > > > > >       name = "MESH_hail";
> > > > > > >       level = "(*,*)";
> > > > > > >    };
> > > > > > >
> > > > > > >    raw_thresh        = >=1.0;
> > > > > > >    conv_radius       = 4; // in grid squares
> > > > > > >    conv_thresh       = >=0.5;
> > > > > > >    vld_thresh        = 0.5;
> > > > > > >    area_thresh       = >=4.0;
> > > > > > >    inten_perc_value  = 100;
> > > > > > >    inten_perc_thresh = >=0.0;
> > > > > > >    merge_thresh      = >=0.5;
> > > > > > >    merge_flag        = NONE;
> > > > > > > };
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Handle missing data
> > > > > > > //
> > > > > > > mask_missing_flag = BOTH;
> > > > > > >
> > > > > > > //
> > > > > > > // Match objects between the forecast and observation
fields
> > > > > > > //
> > > > > > > match_flag = MERGE_BOTH;
> > > > > > >
> > > > > > > //
> > > > > > > // Maximum centroid distance for objects to be compared
> > > > > > > //
> > > > > > > max_centroid_dist = 800.0/grid_res;
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Verification masking regions
> > > > > > > //
> > > > > > > mask = {
> > > > > > >    grid      = "";
> > > > > > >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or
BOTH
> > > > > > >    poly      =
> >    "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> > > > > > >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or
BOTH
> > > > > > > };
> > > > > > >
> > > > > > > //
> > > > > > > // Fuzzy engine weights
> > > > > > > //
> > > > > > > weight = {
> > > > > > >    centroid_dist    = 2.0;
> > > > > > >    boundary_dist    = 4.0;
> > > > > > >    convex_hull_dist = 0.0;
> > > > > > >    angle_diff       = 1.0;
> > > > > > >    area_ratio       = 1.0;
> > > > > > >    int_area_ratio   = 2.0;
> > > > > > >    complexity_ratio = 0.0;
> > > > > > >    inten_perc_ratio = 0.0;
> > > > > > >    inten_perc_value = 100;
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Fuzzy engine interest functions
> > > > > > > //
> > > > > > > interest_function = {
> > > > > > >
> > > > > > >    centroid_dist = (
> > > > > > >       (            0.0, 1.0 )
> > > > > > >       (  60.0/grid_res, 1.0 )
> > > > > > >       ( 600.0/grid_res, 0.0 )
> > > > > > >    );
> > > > > > >
> > > > > > >    boundary_dist = (
> > > > > > >       (            0.0, 1.0 )
> > > > > > >       ( 400.0/grid_res, 0.0 )
> > > > > > >    );
> > > > > > >
> > > > > > >    convex_hull_dist = (
> > > > > > >       (            0.0, 1.0 )
> > > > > > >       ( 400.0/grid_res, 0.0 )
> > > > > > >    );
> > > > > > >
> > > > > > >    angle_diff = (
> > > > > > >       (  0.0, 1.0 )
> > > > > > >       ( 30.0, 1.0 )
> > > > > > >       ( 90.0, 0.0 )
> > > > > > >    corner   = 0.8;
> > > > > > >    ratio_if = (
> > > > > > >       (    0.0, 0.0 )
> > > > > > >       ( corner, 1.0 )
> > > > > > >       (    1.0, 1.0 )
> > > > > > >    );
> > > > > > >
> > > > > > >    area_ratio = ratio_if;
> > > > > > >
> > > > > > >    int_area_ratio = (
> > > > > > >       ( 0.00, 0.00 )
> > > > > > >       ( 0.10, 0.50 )
> > > > > > >       ( 0.25, 1.00 )
> > > > > > >       ( 1.00, 1.00 )
> > > > > > >    );
> > > > > > >
> > > > > > >    complexity_ratio = ratio_if;
> > > > > > >
> > > > > > >    inten_perc_ratio = ratio_if;
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Total interest threshold for determining matches
> > > > > > > //
> > > > > > > total_interest_thresh = 0.5;
> > > > > > >
> > > > > > > //
> > > > > > > // Interest threshold for printing output pair
information
> > > > > > > //
> > > > > > > print_interest_thresh = 0.0;
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Plotting information
> > > > > > > //
> > > > > > > met_data_dir = "MET_BASE/data";
> > > > > > >
> > > > > > > fcst_raw_plot = {
> > > > > > >    color_table      =
> > > "MET_BASE/data/colortables/met_default.ctable";
> > > > > > >    plot_min         = 0.0;
> > > > > > >    plot_max         = 0.0;
> > > > > > >    colorbar_spacing = 1;
> > > > > > > };
> > > > > > >
> > > > > > > obs_raw_plot = {
> > > > > > >    color_table      =
> > > "MET_BASE/data/colortables/met_default.ctable";
> > > > > > >    plot_min         = 0.0;
> > > > > > >    plot_max         = 0.0;
> > > > > > >    colorbar_spacing = 1;
> > > > > > > };
> > > > > > >
> > > > > > > object_plot = {
> > > > > > >    color_table      =
> >    "MET_BASE/data/colortables/mode_obj.ctable";
> > > > > > > };
> > > > > > >
> > > > > > > //
> > > > > > > // Number of grid boxes to fill with bad data values
along
> >    the edge
> > > > of
> > > > > > the
> > > > > > > field
> > > > > > > // to avoid edge effects.
> > > > > > > //
> > > > > > > zero_border_size = 1;
> > > > > > >
> > > > > > > //
> > > > > > > // Boolean for plotting on the region of valid data
within
> >    the
> > > domain
> > > > > > > //
> > > > > > > plot_valid_flag = FALSE;
> > > > > > >
> > > > > > > //
> > > > > > > // Plot polyline edges using great circle arcs instead
of
> >    straight
> > > > > lines
> > > > > > > //
> > > > > > > plot_gcarc_flag = FALSE;
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // NetCDF matched pairs, PostScript, and contingency
table
> >    output
> > > > files
> > > > > > > //
> > > > > > > ps_plot_flag    = TRUE;
> > > > > > > nc_pairs_flag   = TRUE;
> > > > > > > ct_stats_flag   = TRUE;
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > output_prefix  = "";
> > > > > > > version        = "V4.1";
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > >
> > > > > > > ________________________________
> > > > > > >
> > > > > > > This email is intended solely for the recipient. It may
> >    contain
> > > > > > > privileged, proprietary or confidential information or
> >    material. If
> > > > you
> > > > > > are
> > > > > > > not the intended recipient, please delete this email and
any
> > > > > attachments
> > > > > > > and notify the sender of the error.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > >
> > > > > > This email is intended solely for the recipient. It may
> >    contain
> > > > > > privileged, proprietary or confidential information or
> >    material. If
> > > you
> > > > > are
> > > > > > not the intended recipient, please delete this email and
any
> > > > attachments
> > > > > > and notify the sender of the error.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > ________________________________
> > > > >
> > > > > This email is intended solely for the recipient. It may
contain
> > > > > privileged, proprietary or confidential information or
material.
> >    If you
> > > > are
> > > > > not the intended recipient, please delete this email and any
> > > attachments
> > > > > and notify the sender of the error.
> > > > >
> > > > >
> > > >
> > > >
> > > > ________________________________
> > > >
> > > > This email is intended solely for the recipient. It may
contain
> > > > privileged, proprietary or confidential information or
material.
> >    If you
> > > are
> > > > not the intended recipient, please delete this email and any
> >    attachments
> > > > and notify the sender of the error.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > ________________________________
> > >
> > > This email is intended solely for the recipient. It may contain
> > > privileged, proprietary or confidential information or material.
If
> >    you are
> > > not the intended recipient, please delete this email and any
> >    attachments
> > > and notify the sender of the error.
> > >
> > >
> > >
> > >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> >    privileged, proprietary or confidential information or
material. If
> >    you are not the intended recipient, please delete this email
and
> >    any attachments and notify the sender of the error.
> >
> >
>
>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #71629] problems with MODE not identifying objects in field
From: Adams-Selin, Becky
Time: Mon May 04 10:18:43 2015

Thanks John.

Becky Adams-Selin, Ph.D.
Staff Scientist II, AER

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Monday, May 04, 2015 10:36 AM
To: Adams-Selin, Becky
Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
identifying objects in field

Becky,

I'll go ahead and resolve this ticket now.  Just let us know if more
issues
or questions come up in your use of MET and MODE.

Thanks,
John

On Sun, May 3, 2015 at 7:01 AM, Adams-Selin, Becky via RT
<met_help at ucar.edu
> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
>
> Tara,
>
> Nice to "meet" you.  I only have about 10 days worth of data, so I'm
not
> sure it's worth breaking things out by storm morphologies.  I was
able to
> get pretty good results with just using different conv_thresholds
for the
> different members as necessary.  Calculating a percentile threshold
for
> each member over the entire date range would give me a defensible
reason
> for using that specific threshold.  I'll give that a try.
>
> The data that I'm trying to verify is actually the HAILCAST
forecasts from
> the Spring Experiment last year.  I'm headed down to this year's
experiment
> this week - I see your name on the list of attendees!  I look
forward to
> meeting you in person.
>
> Becky Adams-Selin, Ph.D.
> Staff Scientist II, AER
>
> ________________________________________
> From: Tara Jensen via RT [met_help at ucar.edu]
> Sent: Friday, May 01, 2015 4:50 PM
> To: Adams-Selin, Becky
> Subject: [rt.rap.ucar.edu #71629] problems with MODE not identifying
> objects in field
>
> Hi Becky,
>
> This is Tara Jensen.  I'm one of the scientists on the MET team.
The
> method John suggested is exploratory in the sense that we have not
tried it
> much ourselves either.  While we do suggest it would probably be
best to
> look at your data as a whole, I wouldn't rule out the need to maybe
> stratify your data by storm morphologies or forcing mechanisms
before
> determining your thresholds.  I would hope that you wouldn't have to
adjust
> the thresholds on a case by case basis because that would likely be
less
> scientifically defensible.  Hopefully that helps you get started.
Please
> do not hesitate to let us know how your analysis is progressing and
if you
> have further questions.
>
> Thanks, Tara Jensen
>
> On Fri May 01 15:11:36 2015, RSelin at aer.com wrote:
> > I agree de-biasing the ensemble members would be ideal.  However,
I'm
> >    actually comparing two different ensemble configurations, one
where
> >    the bias is largely mitigated by improvements to my hail
algorithm,
> >    so I want to be able to show that improvement in the
statistics.
> >
> > I like the idea of using a percentile of the raw data values!
That
> >    would be simple to implement in python.  Would it be best to
> >    calculate the percentile of all of the data over all of my
dates,
> >    or re-calculate the percentile for each date?
> >
> > Becky Adams-Selin, Ph.D.
> > Staff Scientist II, AER
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Friday, May 01, 2015 3:23 PM
> > To: Adams-Selin, Becky
> > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> >    identifying objects in field
> >
> > Becky,
> >
> > I talked to one of the scientists here, and we have a few
thoughts...
> >
> > (1) When working with an ensemble, one important step is ensemble
> > post-processing, which often includes a de-biasing step.  If you
have
> >    a
> > post-processed de-biased version of this data, I'd suggest
evaluating
> > that.  But I'm assuming you don't.
> >
> > (2) Next, I hear what you're saying that the varying biases makes
it
> > impossible to find of configuration of MODE that works for all the
> > members.  In this case, using a different threshold for each
member
> >    would
> > be a reasonable approach.  But it'd be nice to do so in a
systematic
> >    way.
> > For example, you could define the threshold as a particular
percentile
> >    of
> > the raw data values.  And you could apply the same percentile
> >    threshold to
> > all the members.  But for each member, the actual threshold value
> >    would
> > change.
> >
> > Using a percentile threshold is basically a simplistic bias
removal.
> >
> > We do have plans to add percentile thresholding directly to the
MET
> >    tools,
> > but that functionality won't be available for about a year.  In
the
> >    mean
> > time, here's what I'd suggest.  Write a wrapper script that...
> >    - Looks at the forecast values, ignoring the 0's, and computes
the
> > percentile of interest.
> >    - Set that value as an environment variable, such as "setenv
> > FCST_PERC_THRESH 1.75"
> >    - Configure MODE to use that environment variable.  In the
"fcst"
> > section, set "conv_thresh = >=${FCST_PERC_THRESH};"
> >    - Then run MODE for that case.
> >
> > As for how to compute the percentile value of interest, you could
use
> >    a
> > variety of scripting tools.  I'd probably use R since I'm most
> >    familiar
> > with that, but Matlab, IDL, or even a Python script could probably
do
> >    it.
> > Adding percentile thresholding directly to MET will make this much
> >    easier
> > in the future.
> >
> > Let me know if you get hung up at this step, and I might be able
to
> >    help
> > out.
> >
> > (3) Lastly, you might also consider running this data through the
> >    Grid-Stat
> > tool which can be used to compute neighborhood statistics, such as
> > Fractions Skill Score.  It really all depends on the questions
you're
> > trying to answer about your dataset, but FSS is a nice simple
> >    algorithm
> > that might give you some useful info about your data.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> > On Fri, May 1, 2015 at 10:03 AM, Adams-Selin, Becky via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> > >
> > > John,
> > >
> > > OK, that makes sense, thank you.
> > >
> > > Like you say, ideally I'd like my configuration file to be the
same
> >    for
> > > all my forecast members.  However, because each member is biased
> > > differently, setting a conv_threshold value that makes sense for
one
> > > member, results in really large forecast clusters being
identified
> >    in
> > > another member.  Is there a good way to limit the size of the
> >    cluster
> > > objects without changing the conv_threshold?
> > >
> > > Becky Adams-Selin, Ph.D.
> > > Staff Scientist II, AER
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Monday, April 27, 2015 1:33 PM
> > > To: Adams-Selin, Becky
> > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> >    identifying
> > > objects in field
> > >
> > > Becky,
> > >
> > > MODE attempts to do things intelligently here.  If the forecast
and
> > > observation field names are identical, it chooses a consistent
color
> >    scale
> > > using the range of both.  Your data is not named the same thing
> > > (MAX_HAIL_24 vs MESH_hail) so it plots them using the range of
the
> >    data in
> > > each.
> > >
> > > As for interpreting the output, ultimately it's up to you.  You
> >    mention
> > > that you're dealing with an ensemble with varying biases.  I
would
> >    suggest
> > > finding a configuration of MODE that identifies the objects of
> >    interest and
> > > helps you answer your scientific questions... and then apply the
> >    same
> > > configuration to all members.  Changing the configuration for
each
> >    member
> > > would be pretty confusing.
> > >
> > > I can help you with the nuts and bolts of running MODE, but one
of
> >    the
> > > scientists here would be better at helping you interpret
results.
> >    Before
> > > presenting results, we could probably to take a quick look to
make
> >    sure we
> > > don't see any red flags if you'd like.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Mon, Apr 27, 2015 at 12:07 PM, Adams-Selin, Becky via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
>
> > > >
> > > > Wow, yes, that would do it.  I didn't realize the
> >    forecasted/observed
> > > > plots on the first page of the output .ps file had different
color
> > > scales!
> > > > Thank you for the help and the suggested Mode config file.
> > > >
> > > > If I were to report on these results obtained using a
different
> > > > conv_threshold, would I just need to acknowledge a pre-
existing
> >    bias?
> > > The
> > > > problem is complicated by the fact that I am verifying the
> >    forecasts of
> > > > multiple ensemble members, not all of whom are so biased.
> > > >
> > > > Becky Adams-Selin, Ph.D.
> > > > Staff Scientist II, AER
> > > >
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Monday, April 27, 2015 12:03 PM
> > > > To: Adams-Selin, Becky
> > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> >    identifying
> > > > objects in field
> > > >
> > > > Becky,
> > > >
> > > > Thanks for sending the data.  I ran the following two
> >    plot_data_plane
> > > (from
> > > > MET) commands to plot this data using the same scale of 0 to
1.25.
> >    The
> > > > resulting images are attached:
> > > >
> > > > plot_data_plane \
> > > >    HailSize20140508L1S_caps_maxnn_2.0km.nc \
> > > >    HailSize20140508L1S_caps_maxnn_2.0km.ps \
> > > >    'name="MESH_hail"; level="(*,*)";' -plot_range 0 1.25
> > > >
> > > > plot_data_plane \
> > > >    s4m17_arw_140508_hailsum.nc \
> > > >    s4m17_arw_140508_hailsum.ps \
> > > >    'name="MAX_HAIL_24"; level="(*,*)";' -plot_range 0 1.25
> > > >
> > > > Looking at them, I notice that the observed values are much
higher
> >    than
> > > the
> > > > forecast values!  Very few forecast values are above 1.0 while
> >    most
> > > > observed values are above 1.0.  When you apply a raw thresh of
> >    >=1.0,
> > > > you're zeroing out almost all of the forecast points but
> >    relatively few
> > > > observation points.  That's why you're getting so many more
> >    observation
> > > > objects than forecast objects.
> > > >
> > > > Inspecting the data more closely, I see...
> > > >  - the range on non-zero observed values is 0.50 to 3.74
> > > >  - the range of non-zero forecast values is 0.01 to 1.39
> > > >
> > > > Based on that, I made the following changes to the config
file...
> > > >  - switch the raw_thresh from >=1.0 to >=0.5
> > > >  - increase the smoothing radius (conv_radius) from 1 to 4
> > > >  - set the plot_max in the fcst_raw_plot and obs_raw_plot
sections
> >    to
> > > 3.75
> > > > to use a consistent scale
> > > >  - set the forecast conv_thresh as >=0.25 and observed
conv_thresh
> >    as
> > > > >=0.5.
> > > >
> > > > This resulted in 45 forecast and 44 observed objects.  We
> >    typically use
> > > the
> > > > same threshold for both forecast and observation, but your
> >    forecast has a
> > > > very clear low bias, by about a factor of 2.  That's why I
chose
> >    the
> > > > observed threshold to be 2 times the forecast threshold.
> > > >
> > > > The config file and resulting output image is attached.
> >    Unfortunately,
> > > > this takes a long time to run, about 5 min on my machine!
> > > >
> > > > I'd suggest that you play around with the settings more.  You
> >    should
> > > decide
> > > > what "scale" of objects you're looking for.  If you want a
smaller
> >    number
> > > > of smoother objects, try increasing the convolution radius and
> >    lowering
> > > the
> > > > threshold.  The fewer the number of objects MODE has to keep
track
> >    of,
> > > the
> > > > faster it'll run.
> > > >
> > > > Hope that helps.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Sat, Apr 25, 2015 at 9:19 AM, Adams-Selin, Becky via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629 >
> > > > >
> > > > > I tried lowering the conv_radius to 0, and the
conv_threshold to
> > > 0.25.  I
> > > > > attached what I got.  I'm new to MODE, but just from
visually
> >    comparing
> > > > the
> > > > > forecast and observation fields, I feel like I should be
getting
> > > roughly
> > > > > similar numbers of objects....I would even have expected
fewer
> >    objects
> > > > from
> > > > > the observation field.  Something seems wrong.
> > > > >
> > > > > I went ahead and created a selin_data directory on the
passive
> >    ftp
> > > site.
> > > > >
> > > > > The forecast file is s4m17_arw_140508_hailsum.nc
> > > > > The observation file is
HailSize20140508L1S_caps_maxnn_2.0km.nc
> > > > > And the mode config file is MODEConfig_bigintense
> > > > >
> > > > > If you could take a look I would really appreciate it.
> > > > >
> > > > > Becky Adams-Selin, Ph.D.
> > > > > Staff Scientist II, AER
> > > > >
> > > > > ________________________________________
> > > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > Sent: Saturday, April 25, 2015 9:25 AM
> > > > > To: Adams-Selin, Becky
> > > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE not
> > > identifying
> > > > > objects in field
> > > > >
> > > > > Becky,
> > > > >
> > > > > OK, sounds good.  You could try setting the convolution
radius
> >    to 0
> > > just
> > > > to
> > > > > see if you can get forecast objects.  Or try lowering the
> >    threshold.
> > > > > Perhaps they're all getting smoothed out.
> > > > >
> > > > > If you're still stuck I could take a look at some sample
files.
> >    I'd
> > > > need a
> > > > > sample forecast, observation, and MODE config file.  You
could
> >    post
> > > them
> > > > to
> > > > > our anonymous for site following these instructions:
> > > > >
> > > > > http://www.dtcenter.org/met/users/support/met_help.php#ftp
> > > > >
> > > > > Thanks
> > > > > John
> > > > >
> > > > >
> > > > > On Saturday, April 25, 2015, Adams-Selin, Becky via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> >    >
> > > > > >
> > > > > > John,
> > > > > >
> > > > > > I think that situation actually might apply to my dataset.
> >    The
> > > > > > observational dataset only has hail values that are larger
> >    than 1.0
> > > in.
> > > > > > The forecast dataset is continuous, from 0 inches upwards.
My
> > > > > > understanding was I should set the raw_thresh to 1.0 to
make
> >    the
> > > > forecast
> > > > > > and observational datasets consistent, like you said
below.
> > > > > >
> > > > > > Becky Adams-Selin, Ph.D.
> > > > > > Staff Scientist II, AER
> > > > > >
> > > > > > ________________________________________
> > > > > > From: John Halley Gotway via RT [met_help at ucar.edu
> >    <javascript:;>]
> > > > > > Sent: Friday, April 24, 2015 3:39 PM
> > > > > > To: Adams-Selin, Becky
> > > > > > Subject: Re: [rt.rap.ucar.edu #71629] problems with MODE
not
> > > > identifying
> > > > > > objects in field
> > > > > >
> > > > > > Hello Becky,
> > > > > >
> > > > > > I see that you're having trouble getting objects to show
up in
> >    MODE.
> > > > > > Thanks for sending the sample plot and your configuration
> >    file.
> > > > Looking
> > > > > > closely at your choice of thresholds, I see a problem:
> > > > > >    raw_thresh        = >=1.0;
> > > > > >    conv_thresh       = >=0.5;
> > > > > >
> > > > > > I would suggest changing it to this:
> > > > > >    raw_thresh        = >=0.0;
> > > > > >    conv_thresh       = >=0.5;
> > > > > >
> > > > > > Try rerunning to see if you get objects.
> > > > > >
> > > > > > The "raw_thresh" setting should generally not be used
unless
> >    you
> > > have a
> > > > > > good reason to do so.  It is applied to the raw data
values
> >    and any
> > > > point
> > > > > > not meeting that criteria is set to a value of 0.  The
> >    resulting
> > > values
> > > > > are
> > > > > > passed through to the object definition step.
> > > > > >
> > > > > > Applying that logic doesn't make sense for your dataset.
You
> >    may
> > > > wonder
> > > > > > why you would you ever want to do this?  A while back we
were
> > > > evaluating
> > > > > a
> > > > > > reflectivity forecast, but the system only forecast values
of
> >    35dBz
> > > or
> > > > > > higher.  So we had values of 35dBz surround by values of
0.
> >    The
> > > > > > observations were continuous, with values of 35dBZ
surrounded
> >    by
> > > 34dBZ
> > > > > and
> > > > > > so on.  When defining objects in MODE, we do a smoothing
step
> >    and
> > > > having
> > > > > > values of 0 vs values of 34 affected the output a lot.  So
in
> >    order
> > > to
> > > > > > define observation and forecast objects in a consistent
way,
> >    we
> > > zeroed
> > > > > out
> > > > > > any observed values < 35.
> > > > > >
> > > > > > Since this doesn't apply to your dataset, setting it as
">=0"
> >    should
> > > > pass
> > > > > > all of your data through to the object definition step.
> > > > > >
> > > > > > Hope that helps clarify.
> > > > > >
> > > > > > Thanks,
> > > > > > John Halley Gotway
> > > > > > met_help at ucar.edu <javascript:;>
> > > > > >
> > > > > >
> > > > > > On Fri, Apr 24, 2015 at 2:08 PM, Adams-Selin, Becky via RT
<
> > > > > > met_help at ucar.edu <javascript:;>> wrote:
> > > > > >
> > > > > > >
> > > > > > > Fri Apr 24 14:08:13 2015: Request 71629 was acted upon.
> > > > > > > Transaction: Ticket created by RSelin at aer.com
<javascript:;>
> > > > > > >        Queue: met_help
> > > > > > >      Subject: problems with MODE not identifying objects
in
> >    field
> > > > > > >        Owner: Nobody
> > > > > > >   Requestors: RSelin at aer.com <javascript:;>
> > > > > > >       Status: new
> > > > > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71629
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Dear MET help,
> > > > > > >
> > > > > > > I am using MODE in MET v4.1 to evaluate a custom hail
> >    forecast in
> > > > > > > comparison to a gridded observed hail field (MESH).
I've
> >    attached
> > > > the
> > > > > > > first page of the .ps output from one of my dates.  For
some
> >    reason
> > > > no
> > > > > > > objects are being identified in the forecast field, but
> >    judging
> > > from
> > > > > the
> > > > > > > raw field, they should be.  My MODEConfig file is below.
> >    Any idea
> > > > what
> > > > > > I'm
> > > > > > > doing wrong?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Becky Adams-Selin, Ph.D.
> > > > > > > Staff Scientist II, AER
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > > //
> > > > > > > // MODE configuration file.
> > > > > > > //
> > > > > > > // For additional information, see the
> >    MET_BASE/data/config/README
> > > > > file.
> > > > > > > //
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Output model name to be written
> > > > > > > //
> > > > > > > model = "WRF";
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Approximate grid resolution (km)
> > > > > > > //
> > > > > > > grid_res = 4;
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Forecast and observation fields to be verified
> > > > > > > //
> > > > > > > fcst = {
> > > > > > >    field = {
> > > > > > >       name  = "MAX_HAIL_24";
> > > > > > >       level = "(*,*)";
> > > > > > >    };
> > > > > > >
> > > > > > >    raw_thresh        = >=1.0;
> > > > > > >    conv_radius       = 4; // in grid squares
> > > > > > >    conv_thresh       = >=0.5;
> > > > > > >    vld_thresh        = 0.5;
> > > > > > >    area_thresh       = >=4.0;
> > > > > > >    inten_perc_value  = 100;
> > > > > > >    inten_perc_thresh = >=0.0;
> > > > > > >    merge_thresh      = >=0.5;
> > > > > > >    merge_flag        = NONE;
> > > > > > > };
> > > > > > > obs = {
> > > > > > >    field = {
> > > > > > >       name = "MESH_hail";
> > > > > > >       level = "(*,*)";
> > > > > > >    };
> > > > > > >
> > > > > > >    raw_thresh        = >=1.0;
> > > > > > >    conv_radius       = 4; // in grid squares
> > > > > > >    conv_thresh       = >=0.5;
> > > > > > >    vld_thresh        = 0.5;
> > > > > > >    area_thresh       = >=4.0;
> > > > > > >    inten_perc_value  = 100;
> > > > > > >    inten_perc_thresh = >=0.0;
> > > > > > >    merge_thresh      = >=0.5;
> > > > > > >    merge_flag        = NONE;
> > > > > > > };
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Handle missing data
> > > > > > > //
> > > > > > > mask_missing_flag = BOTH;
> > > > > > >
> > > > > > > //
> > > > > > > // Match objects between the forecast and observation
fields
> > > > > > > //
> > > > > > > match_flag = MERGE_BOTH;
> > > > > > >
> > > > > > > //
> > > > > > > // Maximum centroid distance for objects to be compared
> > > > > > > //
> > > > > > > max_centroid_dist = 800.0/grid_res;
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Verification masking regions
> > > > > > > //
> > > > > > > mask = {
> > > > > > >    grid      = "";
> > > > > > >    grid_flag = NONE; // Apply to NONE, FCST, OBS, or
BOTH
> > > > > > >    poly      =
> >    "/home/adamsr/met/METv4.1/data/poly/CONUS.poly";
> > > > > > >    poly_flag = BOTH; // Apply to NONE, FCST, OBS, or
BOTH
> > > > > > > };
> > > > > > >
> > > > > > > //
> > > > > > > // Fuzzy engine weights
> > > > > > > //
> > > > > > > weight = {
> > > > > > >    centroid_dist    = 2.0;
> > > > > > >    boundary_dist    = 4.0;
> > > > > > >    convex_hull_dist = 0.0;
> > > > > > >    angle_diff       = 1.0;
> > > > > > >    area_ratio       = 1.0;
> > > > > > >    int_area_ratio   = 2.0;
> > > > > > >    complexity_ratio = 0.0;
> > > > > > >    inten_perc_ratio = 0.0;
> > > > > > >    inten_perc_value = 100;
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Fuzzy engine interest functions
> > > > > > > //
> > > > > > > interest_function = {
> > > > > > >
> > > > > > >    centroid_dist = (
> > > > > > >       (            0.0, 1.0 )
> > > > > > >       (  60.0/grid_res, 1.0 )
> > > > > > >       ( 600.0/grid_res, 0.0 )
> > > > > > >    );
> > > > > > >
> > > > > > >    boundary_dist = (
> > > > > > >       (            0.0, 1.0 )
> > > > > > >       ( 400.0/grid_res, 0.0 )
> > > > > > >    );
> > > > > > >
> > > > > > >    convex_hull_dist = (
> > > > > > >       (            0.0, 1.0 )
> > > > > > >       ( 400.0/grid_res, 0.0 )
> > > > > > >    );
> > > > > > >
> > > > > > >    angle_diff = (
> > > > > > >       (  0.0, 1.0 )
> > > > > > >       ( 30.0, 1.0 )
> > > > > > >       ( 90.0, 0.0 )
> > > > > > >    corner   = 0.8;
> > > > > > >    ratio_if = (
> > > > > > >       (    0.0, 0.0 )
> > > > > > >       ( corner, 1.0 )
> > > > > > >       (    1.0, 1.0 )
> > > > > > >    );
> > > > > > >
> > > > > > >    area_ratio = ratio_if;
> > > > > > >
> > > > > > >    int_area_ratio = (
> > > > > > >       ( 0.00, 0.00 )
> > > > > > >       ( 0.10, 0.50 )
> > > > > > >       ( 0.25, 1.00 )
> > > > > > >       ( 1.00, 1.00 )
> > > > > > >    );
> > > > > > >
> > > > > > >    complexity_ratio = ratio_if;
> > > > > > >
> > > > > > >    inten_perc_ratio = ratio_if;
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Total interest threshold for determining matches
> > > > > > > //
> > > > > > > total_interest_thresh = 0.5;
> > > > > > >
> > > > > > > //
> > > > > > > // Interest threshold for printing output pair
information
> > > > > > > //
> > > > > > > print_interest_thresh = 0.0;
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // Plotting information
> > > > > > > //
> > > > > > > met_data_dir = "MET_BASE/data";
> > > > > > >
> > > > > > > fcst_raw_plot = {
> > > > > > >    color_table      =
> > > "MET_BASE/data/colortables/met_default.ctable";
> > > > > > >    plot_min         = 0.0;
> > > > > > >    plot_max         = 0.0;
> > > > > > >    colorbar_spacing = 1;
> > > > > > > };
> > > > > > >
> > > > > > > obs_raw_plot = {
> > > > > > >    color_table      =
> > > "MET_BASE/data/colortables/met_default.ctable";
> > > > > > >    plot_min         = 0.0;
> > > > > > >    plot_max         = 0.0;
> > > > > > >    colorbar_spacing = 1;
> > > > > > > };
> > > > > > >
> > > > > > > object_plot = {
> > > > > > >    color_table      =
> >    "MET_BASE/data/colortables/mode_obj.ctable";
> > > > > > > };
> > > > > > >
> > > > > > > //
> > > > > > > // Number of grid boxes to fill with bad data values
along
> >    the edge
> > > > of
> > > > > > the
> > > > > > > field
> > > > > > > // to avoid edge effects.
> > > > > > > //
> > > > > > > zero_border_size = 1;
> > > > > > >
> > > > > > > //
> > > > > > > // Boolean for plotting on the region of valid data
within
> >    the
> > > domain
> > > > > > > //
> > > > > > > plot_valid_flag = FALSE;
> > > > > > >
> > > > > > > //
> > > > > > > // Plot polyline edges using great circle arcs instead
of
> >    straight
> > > > > lines
> > > > > > > //
> > > > > > > plot_gcarc_flag = FALSE;
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > //
> > > > > > > // NetCDF matched pairs, PostScript, and contingency
table
> >    output
> > > > files
> > > > > > > //
> > > > > > > ps_plot_flag    = TRUE;
> > > > > > > nc_pairs_flag   = TRUE;
> > > > > > > ct_stats_flag   = TRUE;
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > > output_prefix  = "";
> > > > > > > version        = "V4.1";
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
////////////////////////////////////////////////////////////////////////////////
> > > > > > >
> > > > > > >
> > > > > > > ________________________________
> > > > > > >
> > > > > > > This email is intended solely for the recipient. It may
> >    contain
> > > > > > > privileged, proprietary or confidential information or
> >    material. If
> > > > you
> > > > > > are
> > > > > > > not the intended recipient, please delete this email and
any
> > > > > attachments
> > > > > > > and notify the sender of the error.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > >
> > > > > > This email is intended solely for the recipient. It may
> >    contain
> > > > > > privileged, proprietary or confidential information or
> >    material. If
> > > you
> > > > > are
> > > > > > not the intended recipient, please delete this email and
any
> > > > attachments
> > > > > > and notify the sender of the error.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > ________________________________
> > > > >
> > > > > This email is intended solely for the recipient. It may
contain
> > > > > privileged, proprietary or confidential information or
material.
> >    If you
> > > > are
> > > > > not the intended recipient, please delete this email and any
> > > attachments
> > > > > and notify the sender of the error.
> > > > >
> > > > >
> > > >
> > > >
> > > > ________________________________
> > > >
> > > > This email is intended solely for the recipient. It may
contain
> > > > privileged, proprietary or confidential information or
material.
> >    If you
> > > are
> > > > not the intended recipient, please delete this email and any
> >    attachments
> > > > and notify the sender of the error.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > ________________________________
> > >
> > > This email is intended solely for the recipient. It may contain
> > > privileged, proprietary or confidential information or material.
If
> >    you are
> > > not the intended recipient, please delete this email and any
> >    attachments
> > > and notify the sender of the error.
> > >
> > >
> > >
> > >
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> >    privileged, proprietary or confidential information or
material. If
> >    you are not the intended recipient, please delete this email
and
> >    any attachments and notify the sender of the error.
> >
> >
>
>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If
you are
> not the intended recipient, please delete this email and any
attachments
> and notify the sender of the error.
>
>
>
>


________________________________

This email is intended solely for the recipient. It may contain
privileged, proprietary or confidential information or material. If
you are not the intended recipient, please delete this email and any
attachments and notify the sender of the error.



------------------------------------------------