[Met_help] [rt.rap.ucar.edu #91475] History for Guidance for MODE usage of merge_thresh option with ==FBIAS1
John Halley Gotway via RT
met_help at ucar.edu
Fri Aug 16 09:53:32 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
Following on from the attached issue, John suggested using the ==FBIAS1 value in the fcst section of the MODE configuration file.
We've chosen to use merge_flag option 'BOTH'.
For our particular application we have set the following values in the obs dictionary:
conv_thresh = [>=8.0];
merge_thresh = [>=6.5];
Then in the corresponding fcst configuration we set:
conv_thresh = [==FBIAS1];
I can't find in the documentation any information on how to set merge_thresh in this case (where conv_thresh is set to ==FBIAS1). I've just set it to ==FBIAS1 also, but I'm not sure if this the correct thing to do.
Could you provide some guidance, please?
Kind Regards,
Rachel
Dr Rachel North, Research Scientist
Met Office, FitzRoy Road, Exeter, EX1 3PB, United Kingdom
Tel: +44 (0)1392 884332 Fax: +44 (0)1392 885681
rachel.north at metoffice.gov.uk<mailto:rachel.north at metoffice.gov.uk> https://www.metoffice.gov.uk/
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Guidance for MODE usage of merge_thresh option with ==FBIAS1
From: John Halley Gotway
Time: Mon Aug 12 16:21:50 2019
Hi Rachel,
That’s a great question. I’m not in the office today and can’t verify
this
directly, but I did check the source code... this file in particular:
https://github.com/NCAR/MET/blob/master_v8.1/met/src/libcode/vx_shapedata/mode_conf_info.cc
The function set_perc_thresh() processes the fcst and obs merge
thresholds
together as a pair. So the fcst_merge_thresh = [ >=6.5 ]; threshold
corresponds to some percentile of the forecast data values. By
setting,
obs_merge_thresh = [ ==FBIAS1 ]; MODE chooses that same percentile of
the
observation data as the threshold value.
Try running MODE at something like verbosity 4 (-v 4) and look for log
messages telling you what percentile values were computed.
Please let me know if you find the actual behavior to be different.
Thanks,
John
On Mon, Aug 12, 2019 at 9:40 AM North, Rachel via RT
<met_help at ucar.edu>
wrote:
>
> Mon Aug 12 09:39:53 2019: Request 91475 was acted upon.
> Transaction: Ticket created by rachel.north at metoffice.gov.uk
> Queue: met_help
> Subject: Guidance for MODE usage of merge_thresh option with
==FBIAS1
> Owner: Nobody
> Requestors: rachel.north at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475 >
>
>
> Hi,
>
> Following on from the attached issue, John suggested using the
==FBIAS1
> value in the fcst section of the MODE configuration file.
> We've chosen to use merge_flag option 'BOTH'.
>
> For our particular application we have set the following values in
the obs
> dictionary:
> conv_thresh = [>=8.0];
> merge_thresh = [>=6.5];
>
> Then in the corresponding fcst configuration we set:
> conv_thresh = [==FBIAS1];
>
> I can't find in the documentation any information on how to set
> merge_thresh in this case (where conv_thresh is set to ==FBIAS1).
I've
> just set it to ==FBIAS1 also, but I'm not sure if this the correct
thing to
> do.
>
> Could you provide some guidance, please?
>
> Kind Regards,
> Rachel
>
> Dr Rachel North, Research Scientist
> Met Office, FitzRoy Road, Exeter, EX1 3PB, United Kingdom
> Tel: +44 (0)1392 884332 Fax: +44 (0)1392 885681
> rachel.north at metoffice.gov.uk<mailto:rachel.north at metoffice.gov.uk>
> https://www.metoffice.gov.uk/
>
>
> Marion,
>
> OK, so you're checking the PostScript output files. I understand
the
> problem you're describing, but I just don't see that behavior in the
> output.
>
> To clarify, I ran the data you sent through MODE using 2 radii and 2
> thresholds (different in the fcst and obs dictionaries) and quilt =
TRUE:
> quilt = TRUE;
> fcst = {
> ...
> conv_radius = [5, 10];
> conv_thresh = [>=2.5, >=1.25 ];
> ...
> }
> obs = {
> ...
> conv_radius = [6, 12];
> conv_thresh = [>=3, >=1.5 ];
> ...
> }
>
> This run produces 4 MODE PostScript files:
> mode_000000L_20180701_120000V_000000A_R1_T1.ps (5 >=4.7 5 >=2.5)
> mode_000000L_20180701_120000V_000000A_R1_T2.ps (5 >=5.6 5 >=3.2)
> mode_000000L_20180701_120000V_000000A_R2_T1.ps (6 >=4.7 6 >=2.5)
> mode_000000L_20180701_120000V_000000A_R2_T2.ps (6 >=5.6 6 >=3.2)
>
> After each I've listed the combination of FCST_RAD FCST_THR OBS_RAD
OBS_THR
> that were used for that run. And I double-checked the first page of
the
> PostScript output to confirm that the settings there are correctly
applied.
>
> So I can't replicate the problem you describe.
>
> If you're interested in debiasing the forecast, you could use the
new
> percentile threshold options in met-8.1. In the obs dictionary,
set:
> conv_thresh = [>=2.5, >=3.2];
> And in the fcst dictionary, set:
> conv_thresh = [==FBIAS1, ==FBIAS1];
>
> Thanks,
> John
>
> On Fri, Jun 21, 2019 at 1:55 AM marion.mittermaier at metoffice.gov.uk
via
> RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719 >
> >
> > Hi John,
> >
> > let me try your version and see whether it does what I am
expecting it to
> > do.
> >
> > Maybe I didn't explain it very well, but I want to create a quilt
plot
> > using different intensity thresholds for the forecast and
observations.
> > This is because there is a pretty big bias between the two and
using the
> > same thresholds for both really overwhelms all other signals. This
info
> is
> > useful up to a point. The cheap option, I thought was to get a set
of
> > thresholds from a quantile mapping which effectively removes the
bias.
> The
> > expensive option (in terms of storage) would be to pre-process
each
> > forecast field and then use the same thresholds again.
> >
> > When I supplied the two different threshold sequences and took
away the
> > obs= fcst, it ran but the postscript summary page (1) indicates
that the
> > SAME thresholds had been used, despite what I had specified. This
is NOT
> > what I had expected.
> >
> > It was only when I took the sequences away that I got the ps
output to
> > show that different thresholds had been used. I had turned
quilt=FALSE. I
> > hadn't tested it with quilt=TRUE and specifying only one intensity
> > threshold. I will do that, and consider your dictionary
definition
> below,
> > and get back to you.
> >
> > Regards
> > Marion
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 20 June 2019 19:52
> > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> > Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North,
Rachel <
> > rachel.north at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt and
> different
> > threshold sequences for forecast and observation
> >
> > Hi Marion,
> >
> > I see you have a question about the quilt option in MODE. I just
ran a
> > test using the met-8.1 release and see that it's behaving as
expected.
> Let
> > me explain what I did to see if that sheds some light on the the
issue.
> >
> > First, I defined separate convolution radii and threshold for the
fcst
> and
> > obs dictionaries, as shown below:
> >
> >
> >
> > *fcst = {... conv_radius = [5, 10]; conv_thresh
=
> [>=2.5,
> > >=1.25 ];*
> > *...*
> >
> >
> > *}obs = {...*
> >
> > * conv_radius = [6, 12]; conv_thresh = [>=3,
>=1.5 ];*
> > *...*
> >
> > *}*
> >
> > I ran this example twice, once with QUILT = FALSE and once with
QUILT =
> > TRUE.
> > Then I cat-ed the "*_obj.txt" output files together and looked at
the
> > unique entries in columns 12 - 15:
> >
> > cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}' |
> > sort -ru
> >
> >
> > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12 >=1.5*
> >
> > cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}' |
> > sort -ru
> >
> >
> >
> >
> > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6 >=1.510
>=2.5
> 12
> > >=310 >=1.25 12 >=1.5*
> >
> > This is exactly what the output is designed to be. With QUILT =
FALSE,
> we
> > get objects for 2 runs of MODE:
> > first radius/first threshold combo
> > second radius/second threshold combo
> >
> > With QUILT = TRUE, we get output for 4 runs of MODE:
> > first radius/first threshold combo
> > first radius/second threshold combo
> > second radius/first threshold combo
> > second radius/second threshold combo
> >
> > And I checked to make sure that the requested forecast radii (5
and 10)
> > and observation radii (6 and 12) show up in the right spots. So I
don't
> > see any obvious problems.
> >
> > But I can run the example case you sent to see if anything else
pops up.
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> >
> > On Thu, Jun 20, 2019 at 5:37 AM
marion.mittermaier at metoffice.gov.uk via
> > RT < met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
> > > Transaction: Ticket created by
marion.mittermaier at metoffice.gov.uk
> > > Queue: met_help
> > > Subject: Using mode with quilt and different threshold
sequences
> > > for forecast and observation
> > > Owner: Nobody
> > > Requestors: marion.mittermaier at metoffice.gov.uk
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
> > > >
> > >
> > >
> > > Hello,
> > >
> > > when quilt=TRUE I'm finding that mode is impervious to my taking
away
> > > the obs=fcst line in the config file and specifying different
> > > threshold sequences for both.
> > >
> > > It works only if quilt =FALSE, using a single threshold and
> > > convolution radius.
> > >
> > > Any ideas?
> > >
> > > Sample forecast, ob and config file attached.
> > >
> > > Regards
> > > Marion
> > >
> > > --
> > > Dr Marion Mittermaier Manager: Model diagnostics and novel
> > verification
> > >
> > > Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> > > Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
> > > 885681
> > > E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> > > marion.mittermaier at metoffice.gov.uk>
http://www.metoffice.gov.uk<
> > > http://www.metoffice.gov.uk/>
> > >
> > > http://www.metoffice.gov.uk/research/people/marion-mittermaier
> > >
> > > I am co-chair of the WMO WWRP Joint Working Group on Forecast
> > > Verification Research (JWGFVR)
> > >
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
> > >
> > >
> > >
> >
> >
> >
> >
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #91475] Guidance for MODE usage of merge_thresh option with ==FBIAS1
From: marion.mittermaier at metoffice.gov.uk
Time: Tue Aug 13 02:00:51 2019
Hi John,
thanks for the suggestion. Rachel will do that and get back to you.
I guess when you run it without FBIAS1 you set the merge threshold
around 10% lower (rule of thumb)... but we couldn't work out what to
put. Not specifying a threshold though meant that the code wouldn't
run. So, you have to specify something...
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 12 August 2019 23:22
To: North, Rachel <rachel.north at metoffice.gov.uk>
Cc: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
merge_thresh option with ==FBIAS1
Hi Rachel,
That’s a great question. I’m not in the office today and can’t verify
this directly, but I did check the source code... this file in
particular:
https://github.com/NCAR/MET/blob/master_v8.1/met/src/libcode/vx_shapedata/mode_conf_info.cc
The function set_perc_thresh() processes the fcst and obs merge
thresholds together as a pair. So the fcst_merge_thresh = [ >=6.5 ];
threshold corresponds to some percentile of the forecast data values.
By setting, obs_merge_thresh = [ ==FBIAS1 ]; MODE chooses that same
percentile of the observation data as the threshold value.
Try running MODE at something like verbosity 4 (-v 4) and look for log
messages telling you what percentile values were computed.
Please let me know if you find the actual behavior to be different.
Thanks,
John
On Mon, Aug 12, 2019 at 9:40 AM North, Rachel via RT
<met_help at ucar.edu>
wrote:
>
> Mon Aug 12 09:39:53 2019: Request 91475 was acted upon.
> Transaction: Ticket created by rachel.north at metoffice.gov.uk
> Queue: met_help
> Subject: Guidance for MODE usage of merge_thresh option with
==FBIAS1
> Owner: Nobody
> Requestors: rachel.north at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475
> >
>
>
> Hi,
>
> Following on from the attached issue, John suggested using the
> ==FBIAS1 value in the fcst section of the MODE configuration file.
> We've chosen to use merge_flag option 'BOTH'.
>
> For our particular application we have set the following values in
the
> obs
> dictionary:
> conv_thresh = [>=8.0];
> merge_thresh = [>=6.5];
>
> Then in the corresponding fcst configuration we set:
> conv_thresh = [==FBIAS1];
>
> I can't find in the documentation any information on how to set
> merge_thresh in this case (where conv_thresh is set to ==FBIAS1).
> I've just set it to ==FBIAS1 also, but I'm not sure if this the
> correct thing to do.
>
> Could you provide some guidance, please?
>
> Kind Regards,
> Rachel
>
> Dr Rachel North, Research Scientist
> Met Office, FitzRoy Road, Exeter, EX1 3PB, United Kingdom
> Tel: +44 (0)1392 884332 Fax: +44 (0)1392 885681
> rachel.north at metoffice.gov.uk<mailto:rachel.north at metoffice.gov.uk>
> https://www.metoffice.gov.uk/
>
>
> Marion,
>
> OK, so you're checking the PostScript output files. I understand
the
> problem you're describing, but I just don't see that behavior in the
> output.
>
> To clarify, I ran the data you sent through MODE using 2 radii and 2
> thresholds (different in the fcst and obs dictionaries) and quilt =
TRUE:
> quilt = TRUE;
> fcst = {
> ...
> conv_radius = [5, 10];
> conv_thresh = [>=2.5, >=1.25 ];
> ...
> }
> obs = {
> ...
> conv_radius = [6, 12];
> conv_thresh = [>=3, >=1.5 ];
> ...
> }
>
> This run produces 4 MODE PostScript files:
> mode_000000L_20180701_120000V_000000A_R1_T1.ps (5 >=4.7 5 >=2.5)
> mode_000000L_20180701_120000V_000000A_R1_T2.ps (5 >=5.6 5 >=3.2)
> mode_000000L_20180701_120000V_000000A_R2_T1.ps (6 >=4.7 6 >=2.5)
> mode_000000L_20180701_120000V_000000A_R2_T2.ps (6 >=5.6 6 >=3.2)
>
> After each I've listed the combination of FCST_RAD FCST_THR OBS_RAD
> OBS_THR that were used for that run. And I double-checked the first
> page of the PostScript output to confirm that the settings there are
correctly applied.
>
> So I can't replicate the problem you describe.
>
> If you're interested in debiasing the forecast, you could use the
new
> percentile threshold options in met-8.1. In the obs dictionary,
set:
> conv_thresh = [>=2.5, >=3.2];
> And in the fcst dictionary, set:
> conv_thresh = [==FBIAS1, ==FBIAS1];
>
> Thanks,
> John
>
> On Fri, Jun 21, 2019 at 1:55 AM marion.mittermaier at metoffice.gov.uk
> via RT < met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719 >
> >
> > Hi John,
> >
> > let me try your version and see whether it does what I am
expecting
> > it to do.
> >
> > Maybe I didn't explain it very well, but I want to create a quilt
> > plot using different intensity thresholds for the forecast and
observations.
> > This is because there is a pretty big bias between the two and
using
> > the same thresholds for both really overwhelms all other signals.
> > This info
> is
> > useful up to a point. The cheap option, I thought was to get a set
> > of thresholds from a quantile mapping which effectively removes
the bias.
> The
> > expensive option (in terms of storage) would be to pre-process
each
> > forecast field and then use the same thresholds again.
> >
> > When I supplied the two different threshold sequences and took
away
> > the obs= fcst, it ran but the postscript summary page (1)
indicates
> > that the SAME thresholds had been used, despite what I had
> > specified. This is NOT what I had expected.
> >
> > It was only when I took the sequences away that I got the ps
output
> > to show that different thresholds had been used. I had turned
> > quilt=FALSE. I hadn't tested it with quilt=TRUE and specifying
only
> > one intensity threshold. I will do that, and consider your
> > dictionary definition
> below,
> > and get back to you.
> >
> > Regards
> > Marion
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 20 June 2019 19:52
> > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> > Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North,
Rachel
> > < rachel.north at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt and
> different
> > threshold sequences for forecast and observation
> >
> > Hi Marion,
> >
> > I see you have a question about the quilt option in MODE. I just
> > ran a test using the met-8.1 release and see that it's behaving as
expected.
> Let
> > me explain what I did to see if that sheds some light on the the
issue.
> >
> > First, I defined separate convolution radii and threshold for the
> > fcst
> and
> > obs dictionaries, as shown below:
> >
> >
> >
> > *fcst = {... conv_radius = [5, 10]; conv_thresh
=
> [>=2.5,
> > >=1.25 ];*
> > *...*
> >
> >
> > *}obs = {...*
> >
> > * conv_radius = [6, 12]; conv_thresh = [>=3,
>=1.5 ];*
> > *...*
> >
> > *}*
> >
> > I ran this example twice, once with QUILT = FALSE and once with
> > QUILT = TRUE.
> > Then I cat-ed the "*_obj.txt" output files together and looked at
> > the unique entries in columns 12 - 15:
> >
> > cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13, $14,
> > $15}' | sort -ru
> >
> >
> > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12 >=1.5*
> >
> > cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14,
> > $15}' | sort -ru
> >
> >
> >
> >
> > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6 >=1.510
> > >=2.5
> 12
> > >=310 >=1.25 12 >=1.5*
> >
> > This is exactly what the output is designed to be. With QUILT =
> > FALSE,
> we
> > get objects for 2 runs of MODE:
> > first radius/first threshold combo
> > second radius/second threshold combo
> >
> > With QUILT = TRUE, we get output for 4 runs of MODE:
> > first radius/first threshold combo
> > first radius/second threshold combo
> > second radius/first threshold combo
> > second radius/second threshold combo
> >
> > And I checked to make sure that the requested forecast radii (5
and
> > 10) and observation radii (6 and 12) show up in the right spots.
So
> > I don't see any obvious problems.
> >
> > But I can run the example case you sent to see if anything else
pops up.
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> >
> > On Thu, Jun 20, 2019 at 5:37 AM
marion.mittermaier at metoffice.gov.uk
> > via RT < met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
> > > Transaction: Ticket created by
marion.mittermaier at metoffice.gov.uk
> > > Queue: met_help
> > > Subject: Using mode with quilt and different threshold
> > > sequences for forecast and observation
> > > Owner: Nobody
> > > Requestors: marion.mittermaier at metoffice.gov.uk
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
> > > >
> > >
> > >
> > > Hello,
> > >
> > > when quilt=TRUE I'm finding that mode is impervious to my taking
> > > away the obs=fcst line in the config file and specifying
different
> > > threshold sequences for both.
> > >
> > > It works only if quilt =FALSE, using a single threshold and
> > > convolution radius.
> > >
> > > Any ideas?
> > >
> > > Sample forecast, ob and config file attached.
> > >
> > > Regards
> > > Marion
> > >
> > > --
> > > Dr Marion Mittermaier Manager: Model diagnostics and novel
> > verification
> > >
> > > Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> > > Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
> > > 885681
> > > E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> > > marion.mittermaier at metoffice.gov.uk>
http://www.metoffice.gov.uk<
> > > http://www.metoffice.gov.uk/>
> > >
> > > http://www.metoffice.gov.uk/research/people/marion-mittermaier
> > >
> > > I am co-chair of the WMO WWRP Joint Working Group on Forecast
> > > Verification Research (JWGFVR)
> > >
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.
> > > html
> > >
> > >
> > >
> >
> >
> >
> >
>
>
------------------------------------------------
Subject: Guidance for MODE usage of merge_thresh option with ==FBIAS1
From: John Halley Gotway
Time: Tue Aug 13 11:13:15 2019
Marion,
Yes, I do think that using FBIAS1 for both the conv_thresh and
merge_thresh, as shown below, makes sense.
Honestly, the testing and evaluation projects in the DTC haven't made
a
whole lot of use of this percentile thresholding functionality yet.
So
I'll be interested to hear your feedback about how it's going for you.
Please let me know if you run across any problems... or see ways of
improving the logic to make it more useful.
Thanks,
John
obs = {...
conv_thresh = [>=8.0];
merge_thresh = [>=6.5];
...}
fcst = {...
conv_thresh = [==FBIAS1];
merge_thresh = [==FBIAS1];
...}
On Tue, Aug 13, 2019 at 2:01 AM marion.mittermaier at metoffice.gov.uk
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475 >
>
> Hi John,
>
> thanks for the suggestion. Rachel will do that and get back to you.
>
> I guess when you run it without FBIAS1 you set the merge threshold
around
> 10% lower (rule of thumb)... but we couldn't work out what to put.
Not
> specifying a threshold though meant that the code wouldn't run. So,
you
> have to specify something...
>
> Marion
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 12 August 2019 23:22
> To: North, Rachel <rachel.north at metoffice.gov.uk>
> Cc: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
> merge_thresh option with ==FBIAS1
>
> Hi Rachel,
>
> That’s a great question. I’m not in the office today and can’t
verify
> this directly, but I did check the source code... this file in
particular:
>
>
>
https://github.com/NCAR/MET/blob/master_v8.1/met/src/libcode/vx_shapedata/mode_conf_info.cc
>
> The function set_perc_thresh() processes the fcst and obs merge
thresholds
> together as a pair. So the fcst_merge_thresh = [ >=6.5 ]; threshold
> corresponds to some percentile of the forecast data values. By
setting,
> obs_merge_thresh = [ ==FBIAS1 ]; MODE chooses that same percentile
of the
> observation data as the threshold value.
>
> Try running MODE at something like verbosity 4 (-v 4) and look for
log
> messages telling you what percentile values were computed.
>
> Please let me know if you find the actual behavior to be different.
>
> Thanks,
> John
>
> On Mon, Aug 12, 2019 at 9:40 AM North, Rachel via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Mon Aug 12 09:39:53 2019: Request 91475 was acted upon.
> > Transaction: Ticket created by rachel.north at metoffice.gov.uk
> > Queue: met_help
> > Subject: Guidance for MODE usage of merge_thresh option with
> ==FBIAS1
> > Owner: Nobody
> > Requestors: rachel.north at metoffice.gov.uk
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475
> > >
> >
> >
> > Hi,
> >
> > Following on from the attached issue, John suggested using the
> > ==FBIAS1 value in the fcst section of the MODE configuration file.
> > We've chosen to use merge_flag option 'BOTH'.
> >
> > For our particular application we have set the following values in
the
> > obs
> > dictionary:
> > conv_thresh = [>=8.0];
> > merge_thresh = [>=6.5];
> >
> > Then in the corresponding fcst configuration we set:
> > conv_thresh = [==FBIAS1];
> >
> > I can't find in the documentation any information on how to set
> > merge_thresh in this case (where conv_thresh is set to ==FBIAS1).
> > I've just set it to ==FBIAS1 also, but I'm not sure if this the
> > correct thing to do.
> >
> > Could you provide some guidance, please?
> >
> > Kind Regards,
> > Rachel
> >
> > Dr Rachel North, Research Scientist
> > Met Office, FitzRoy Road, Exeter, EX1 3PB, United Kingdom
> > Tel: +44 (0)1392 884332 Fax: +44 (0)1392 885681
> >
rachel.north at metoffice.gov.uk<mailto:rachel.north at metoffice.gov.uk>
> > https://www.metoffice.gov.uk/
> >
> >
> > Marion,
> >
> > OK, so you're checking the PostScript output files. I understand
the
> > problem you're describing, but I just don't see that behavior in
the
> > output.
> >
> > To clarify, I ran the data you sent through MODE using 2 radii and
2
> > thresholds (different in the fcst and obs dictionaries) and quilt
= TRUE:
> > quilt = TRUE;
> > fcst = {
> > ...
> > conv_radius = [5, 10];
> > conv_thresh = [>=2.5, >=1.25 ];
> > ...
> > }
> > obs = {
> > ...
> > conv_radius = [6, 12];
> > conv_thresh = [>=3, >=1.5 ];
> > ...
> > }
> >
> > This run produces 4 MODE PostScript files:
> > mode_000000L_20180701_120000V_000000A_R1_T1.ps (5 >=4.7 5 >=2.5)
> > mode_000000L_20180701_120000V_000000A_R1_T2.ps (5 >=5.6 5 >=3.2)
> > mode_000000L_20180701_120000V_000000A_R2_T1.ps (6 >=4.7 6 >=2.5)
> > mode_000000L_20180701_120000V_000000A_R2_T2.ps (6 >=5.6 6 >=3.2)
> >
> > After each I've listed the combination of FCST_RAD FCST_THR
OBS_RAD
> > OBS_THR that were used for that run. And I double-checked the
first
> > page of the PostScript output to confirm that the settings there
are
> correctly applied.
> >
> > So I can't replicate the problem you describe.
> >
> > If you're interested in debiasing the forecast, you could use the
new
> > percentile threshold options in met-8.1. In the obs dictionary,
set:
> > conv_thresh = [>=2.5, >=3.2];
> > And in the fcst dictionary, set:
> > conv_thresh = [==FBIAS1, ==FBIAS1];
> >
> > Thanks,
> > John
> >
> > On Fri, Jun 21, 2019 at 1:55 AM
marion.mittermaier at metoffice.gov.uk
> > via RT < met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719 >
> > >
> > > Hi John,
> > >
> > > let me try your version and see whether it does what I am
expecting
> > > it to do.
> > >
> > > Maybe I didn't explain it very well, but I want to create a
quilt
> > > plot using different intensity thresholds for the forecast and
> observations.
> > > This is because there is a pretty big bias between the two and
using
> > > the same thresholds for both really overwhelms all other
signals.
> > > This info
> > is
> > > useful up to a point. The cheap option, I thought was to get a
set
> > > of thresholds from a quantile mapping which effectively removes
the
> bias.
> > The
> > > expensive option (in terms of storage) would be to pre-process
each
> > > forecast field and then use the same thresholds again.
> > >
> > > When I supplied the two different threshold sequences and took
away
> > > the obs= fcst, it ran but the postscript summary page (1)
indicates
> > > that the SAME thresholds had been used, despite what I had
> > > specified. This is NOT what I had expected.
> > >
> > > It was only when I took the sequences away that I got the ps
output
> > > to show that different thresholds had been used. I had turned
> > > quilt=FALSE. I hadn't tested it with quilt=TRUE and specifying
only
> > > one intensity threshold. I will do that, and consider your
> > > dictionary definition
> > below,
> > > and get back to you.
> > >
> > > Regards
> > > Marion
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 20 June 2019 19:52
> > > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> > > Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North,
Rachel
> > > < rachel.north at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt and
> > different
> > > threshold sequences for forecast and observation
> > >
> > > Hi Marion,
> > >
> > > I see you have a question about the quilt option in MODE. I
just
> > > ran a test using the met-8.1 release and see that it's behaving
as
> expected.
> > Let
> > > me explain what I did to see if that sheds some light on the the
issue.
> > >
> > > First, I defined separate convolution radii and threshold for
the
> > > fcst
> > and
> > > obs dictionaries, as shown below:
> > >
> > >
> > >
> > > *fcst = {... conv_radius = [5, 10]; conv_thresh
=
> > [>=2.5,
> > > >=1.25 ];*
> > > *...*
> > >
> > >
> > > *}obs = {...*
> > >
> > > * conv_radius = [6, 12]; conv_thresh = [>=3,
>=1.5 ];*
> > > *...*
> > >
> > > *}*
> > >
> > > I ran this example twice, once with QUILT = FALSE and once with
> > > QUILT = TRUE.
> > > Then I cat-ed the "*_obj.txt" output files together and looked
at
> > > the unique entries in columns 12 - 15:
> > >
> > > cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13,
$14,
> > > $15}' | sort -ru
> > >
> > >
> > > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12
>=1.5*
> > >
> > > cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14,
> > > $15}' | sort -ru
> > >
> > >
> > >
> > >
> > > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6
>=1.510
> > > >=2.5
> > 12
> > > >=310 >=1.25 12 >=1.5*
> > >
> > > This is exactly what the output is designed to be. With QUILT =
> > > FALSE,
> > we
> > > get objects for 2 runs of MODE:
> > > first radius/first threshold combo
> > > second radius/second threshold combo
> > >
> > > With QUILT = TRUE, we get output for 4 runs of MODE:
> > > first radius/first threshold combo
> > > first radius/second threshold combo
> > > second radius/first threshold combo
> > > second radius/second threshold combo
> > >
> > > And I checked to make sure that the requested forecast radii (5
and
> > > 10) and observation radii (6 and 12) show up in the right spots.
So
> > > I don't see any obvious problems.
> > >
> > > But I can run the example case you sent to see if anything else
pops
> up.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > >
> > >
> > > On Thu, Jun 20, 2019 at 5:37 AM
marion.mittermaier at metoffice.gov.uk
> > > via RT < met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
> > > > Transaction: Ticket created by
marion.mittermaier at metoffice.gov.uk
> > > > Queue: met_help
> > > > Subject: Using mode with quilt and different threshold
> > > > sequences for forecast and observation
> > > > Owner: Nobody
> > > > Requestors: marion.mittermaier at metoffice.gov.uk
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
> > > > >
> > > >
> > > >
> > > > Hello,
> > > >
> > > > when quilt=TRUE I'm finding that mode is impervious to my
taking
> > > > away the obs=fcst line in the config file and specifying
different
> > > > threshold sequences for both.
> > > >
> > > > It works only if quilt =FALSE, using a single threshold and
> > > > convolution radius.
> > > >
> > > > Any ideas?
> > > >
> > > > Sample forecast, ob and config file attached.
> > > >
> > > > Regards
> > > > Marion
> > > >
> > > > --
> > > > Dr Marion Mittermaier Manager: Model diagnostics and novel
> > > verification
> > > >
> > > > Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> > > > Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44
(0)1392
> > > > 885681
> > > > E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> > > > marion.mittermaier at metoffice.gov.uk>
http://www.metoffice.gov.uk<
> > > > http://www.metoffice.gov.uk/>
> > > >
> > > > http://www.metoffice.gov.uk/research/people/marion-mittermaier
> > > >
> > > > I am co-chair of the WMO WWRP Joint Working Group on Forecast
> > > > Verification Research (JWGFVR)
> > > >
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.
> > > > html
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Guidance for MODE usage of merge_thresh option with ==FBIAS1
From: marion.mittermaier at metoffice.gov.uk
Time: Wed Aug 14 06:02:40 2019
Hi John,
not sure whether Rachel has responded. Yes it works, and it seems to
have some reduction built in for the merge threshold. Only thing that
isn't very pretty is the way the text comes out in the postscript
files... see attached.
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 13 August 2019 18:13
To: North, Rachel <rachel.north at metoffice.gov.uk>
Cc: lblank at ucar.edu; Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
merge_thresh option with ==FBIAS1
Marion,
Yes, I do think that using FBIAS1 for both the conv_thresh and
merge_thresh, as shown below, makes sense.
Honestly, the testing and evaluation projects in the DTC haven't made
a whole lot of use of this percentile thresholding functionality yet.
So I'll be interested to hear your feedback about how it's going for
you.
Please let me know if you run across any problems... or see ways of
improving the logic to make it more useful.
Thanks,
John
obs = {...
conv_thresh = [>=8.0];
merge_thresh = [>=6.5];
...}
fcst = {...
conv_thresh = [==FBIAS1];
merge_thresh = [==FBIAS1];
...}
On Tue, Aug 13, 2019 at 2:01 AM marion.mittermaier at metoffice.gov.uk
via RT < met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475 >
>
> Hi John,
>
> thanks for the suggestion. Rachel will do that and get back to you.
>
> I guess when you run it without FBIAS1 you set the merge threshold
> around 10% lower (rule of thumb)... but we couldn't work out what to
> put. Not specifying a threshold though meant that the code wouldn't
> run. So, you have to specify something...
>
> Marion
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 12 August 2019 23:22
> To: North, Rachel <rachel.north at metoffice.gov.uk>
> Cc: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
> merge_thresh option with ==FBIAS1
>
> Hi Rachel,
>
> That’s a great question. I’m not in the office today and can’t
verify
> this directly, but I did check the source code... this file in
particular:
>
>
>
https://github.com/NCAR/MET/blob/master_v8.1/met/src/libcode/vx_shaped
> ata/mode_conf_info.cc
>
> The function set_perc_thresh() processes the fcst and obs merge
> thresholds together as a pair. So the fcst_merge_thresh = [ >=6.5
];
> threshold corresponds to some percentile of the forecast data
values.
> By setting, obs_merge_thresh = [ ==FBIAS1 ]; MODE chooses that same
> percentile of the observation data as the threshold value.
>
> Try running MODE at something like verbosity 4 (-v 4) and look for
log
> messages telling you what percentile values were computed.
>
> Please let me know if you find the actual behavior to be different.
>
> Thanks,
> John
>
> On Mon, Aug 12, 2019 at 9:40 AM North, Rachel via RT
> <met_help at ucar.edu>
> wrote:
>
> >
> > Mon Aug 12 09:39:53 2019: Request 91475 was acted upon.
> > Transaction: Ticket created by rachel.north at metoffice.gov.uk
> > Queue: met_help
> > Subject: Guidance for MODE usage of merge_thresh option with
> ==FBIAS1
> > Owner: Nobody
> > Requestors: rachel.north at metoffice.gov.uk
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475
> > >
> >
> >
> > Hi,
> >
> > Following on from the attached issue, John suggested using the
> > ==FBIAS1 value in the fcst section of the MODE configuration file.
> > We've chosen to use merge_flag option 'BOTH'.
> >
> > For our particular application we have set the following values in
> > the obs
> > dictionary:
> > conv_thresh = [>=8.0];
> > merge_thresh = [>=6.5];
> >
> > Then in the corresponding fcst configuration we set:
> > conv_thresh = [==FBIAS1];
> >
> > I can't find in the documentation any information on how to set
> > merge_thresh in this case (where conv_thresh is set to ==FBIAS1).
> > I've just set it to ==FBIAS1 also, but I'm not sure if this the
> > correct thing to do.
> >
> > Could you provide some guidance, please?
> >
> > Kind Regards,
> > Rachel
> >
> > Dr Rachel North, Research Scientist
> > Met Office, FitzRoy Road, Exeter, EX1 3PB, United Kingdom
> > Tel: +44 (0)1392 884332 Fax: +44 (0)1392 885681
> >
rachel.north at metoffice.gov.uk<mailto:rachel.north at metoffice.gov.uk>
> > https://www.metoffice.gov.uk/
> >
> >
> > Marion,
> >
> > OK, so you're checking the PostScript output files. I understand
> > the problem you're describing, but I just don't see that behavior
in
> > the output.
> >
> > To clarify, I ran the data you sent through MODE using 2 radii and
2
> > thresholds (different in the fcst and obs dictionaries) and quilt
= TRUE:
> > quilt = TRUE;
> > fcst = {
> > ...
> > conv_radius = [5, 10];
> > conv_thresh = [>=2.5, >=1.25 ];
> > ...
> > }
> > obs = {
> > ...
> > conv_radius = [6, 12];
> > conv_thresh = [>=3, >=1.5 ];
> > ...
> > }
> >
> > This run produces 4 MODE PostScript files:
> > mode_000000L_20180701_120000V_000000A_R1_T1.ps (5 >=4.7 5 >=2.5)
> > mode_000000L_20180701_120000V_000000A_R1_T2.ps (5 >=5.6 5 >=3.2)
> > mode_000000L_20180701_120000V_000000A_R2_T1.ps (6 >=4.7 6 >=2.5)
> > mode_000000L_20180701_120000V_000000A_R2_T2.ps (6 >=5.6 6 >=3.2)
> >
> > After each I've listed the combination of FCST_RAD FCST_THR
OBS_RAD
> > OBS_THR that were used for that run. And I double-checked the
first
> > page of the PostScript output to confirm that the settings there
are
> correctly applied.
> >
> > So I can't replicate the problem you describe.
> >
> > If you're interested in debiasing the forecast, you could use the
> > new percentile threshold options in met-8.1. In the obs
dictionary, set:
> > conv_thresh = [>=2.5, >=3.2];
> > And in the fcst dictionary, set:
> > conv_thresh = [==FBIAS1, ==FBIAS1];
> >
> > Thanks,
> > John
> >
> > On Fri, Jun 21, 2019 at 1:55 AM
marion.mittermaier at metoffice.gov.uk
> > via RT < met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719 >
> > >
> > > Hi John,
> > >
> > > let me try your version and see whether it does what I am
> > > expecting it to do.
> > >
> > > Maybe I didn't explain it very well, but I want to create a
quilt
> > > plot using different intensity thresholds for the forecast and
> observations.
> > > This is because there is a pretty big bias between the two and
> > > using the same thresholds for both really overwhelms all other
signals.
> > > This info
> > is
> > > useful up to a point. The cheap option, I thought was to get a
set
> > > of thresholds from a quantile mapping which effectively removes
> > > the
> bias.
> > The
> > > expensive option (in terms of storage) would be to pre-process
> > > each forecast field and then use the same thresholds again.
> > >
> > > When I supplied the two different threshold sequences and took
> > > away the obs= fcst, it ran but the postscript summary page (1)
> > > indicates that the SAME thresholds had been used, despite what I
> > > had specified. This is NOT what I had expected.
> > >
> > > It was only when I took the sequences away that I got the ps
> > > output to show that different thresholds had been used. I had
> > > turned quilt=FALSE. I hadn't tested it with quilt=TRUE and
> > > specifying only one intensity threshold. I will do that, and
> > > consider your dictionary definition
> > below,
> > > and get back to you.
> > >
> > > Regards
> > > Marion
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: 20 June 2019 19:52
> > > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> > > Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North,
> > > Rachel < rachel.north at metoffice.gov.uk>
> > > Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt and
> > different
> > > threshold sequences for forecast and observation
> > >
> > > Hi Marion,
> > >
> > > I see you have a question about the quilt option in MODE. I
just
> > > ran a test using the met-8.1 release and see that it's behaving
as
> expected.
> > Let
> > > me explain what I did to see if that sheds some light on the the
issue.
> > >
> > > First, I defined separate convolution radii and threshold for
the
> > > fcst
> > and
> > > obs dictionaries, as shown below:
> > >
> > >
> > >
> > > *fcst = {... conv_radius = [5, 10]; conv_thresh
=
> > [>=2.5,
> > > >=1.25 ];*
> > > *...*
> > >
> > >
> > > *}obs = {...*
> > >
> > > * conv_radius = [6, 12]; conv_thresh = [>=3,
>=1.5 ];*
> > > *...*
> > >
> > > *}*
> > >
> > > I ran this example twice, once with QUILT = FALSE and once with
> > > QUILT = TRUE.
> > > Then I cat-ed the "*_obj.txt" output files together and looked
at
> > > the unique entries in columns 12 - 15:
> > >
> > > cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13,
$14,
> > > $15}' | sort -ru
> > >
> > >
> > > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12
>=1.5*
> > >
> > > cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14,
> > > $15}' | sort -ru
> > >
> > >
> > >
> > >
> > > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6
>=1.510
> > > >=2.5
> > 12
> > > >=310 >=1.25 12 >=1.5*
> > >
> > > This is exactly what the output is designed to be. With QUILT =
> > > FALSE,
> > we
> > > get objects for 2 runs of MODE:
> > > first radius/first threshold combo
> > > second radius/second threshold combo
> > >
> > > With QUILT = TRUE, we get output for 4 runs of MODE:
> > > first radius/first threshold combo
> > > first radius/second threshold combo
> > > second radius/first threshold combo
> > > second radius/second threshold combo
> > >
> > > And I checked to make sure that the requested forecast radii (5
> > > and
> > > 10) and observation radii (6 and 12) show up in the right spots.
> > > So I don't see any obvious problems.
> > >
> > > But I can run the example case you sent to see if anything else
> > > pops
> up.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > >
> > >
> > > On Thu, Jun 20, 2019 at 5:37 AM
> > > marion.mittermaier at metoffice.gov.uk
> > > via RT < met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
> > > > Transaction: Ticket created by
marion.mittermaier at metoffice.gov.uk
> > > > Queue: met_help
> > > > Subject: Using mode with quilt and different threshold
> > > > sequences for forecast and observation
> > > > Owner: Nobody
> > > > Requestors: marion.mittermaier at metoffice.gov.uk
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
> > > > >
> > > >
> > > >
> > > > Hello,
> > > >
> > > > when quilt=TRUE I'm finding that mode is impervious to my
taking
> > > > away the obs=fcst line in the config file and specifying
> > > > different threshold sequences for both.
> > > >
> > > > It works only if quilt =FALSE, using a single threshold and
> > > > convolution radius.
> > > >
> > > > Any ideas?
> > > >
> > > > Sample forecast, ob and config file attached.
> > > >
> > > > Regards
> > > > Marion
> > > >
> > > > --
> > > > Dr Marion Mittermaier Manager: Model diagnostics and novel
> > > verification
> > > >
> > > > Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> > > > Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44
(0)1392
> > > > 885681
> > > > E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> > > > marion.mittermaier at metoffice.gov.uk>
> > > > http://www.metoffice.gov.uk< http://www.metoffice.gov.uk/>
> > > >
> > > > http://www.metoffice.gov.uk/research/people/marion-mittermaier
> > > >
> > > > I am co-chair of the WMO WWRP Joint Working Group on Forecast
> > > > Verification Research (JWGFVR)
> > > >
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.
> > > > html
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Guidance for MODE usage of merge_thresh option with ==FBIAS1
From: John Halley Gotway
Time: Wed Aug 14 10:15:02 2019
Marion,
Glad it's working. Thanks for letting me know about the PostScript
plot.
Yes, I've seen that in the past with long threshold strings. I'll
write up
a GitHub issue to (1) use the entire with of the page for the meta
data at
the bottom and (2) stop writing the fcst/obs interest pairs table rows
once
it reaches the bottom of the plots to avoid overwriting the table.
But I'll go ahead and resolve this ticket.
Thanks,
John
On Wed, Aug 14, 2019 at 6:03 AM marion.mittermaier at metoffice.gov.uk
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475 >
>
> Hi John,
>
> not sure whether Rachel has responded. Yes it works, and it seems to
have
> some reduction built in for the merge threshold. Only thing that
isn't very
> pretty is the way the text comes out in the postscript files... see
> attached.
>
> Marion
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 13 August 2019 18:13
> To: North, Rachel <rachel.north at metoffice.gov.uk>
> Cc: lblank at ucar.edu; Mittermaier, Marion <
> marion.mittermaier at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
> merge_thresh option with ==FBIAS1
>
> Marion,
>
> Yes, I do think that using FBIAS1 for both the conv_thresh and
> merge_thresh, as shown below, makes sense.
>
> Honestly, the testing and evaluation projects in the DTC haven't
made a
> whole lot of use of this percentile thresholding functionality yet.
So
> I'll be interested to hear your feedback about how it's going for
you.
> Please let me know if you run across any problems... or see ways of
> improving the logic to make it more useful.
>
> Thanks,
> John
>
> obs = {...
> conv_thresh = [>=8.0];
> merge_thresh = [>=6.5];
> ...}
> fcst = {...
> conv_thresh = [==FBIAS1];
> merge_thresh = [==FBIAS1];
> ...}
>
>
> On Tue, Aug 13, 2019 at 2:01 AM marion.mittermaier at metoffice.gov.uk
via
> RT < met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475 >
> >
> > Hi John,
> >
> > thanks for the suggestion. Rachel will do that and get back to
you.
> >
> > I guess when you run it without FBIAS1 you set the merge threshold
> > around 10% lower (rule of thumb)... but we couldn't work out what
to
> > put. Not specifying a threshold though meant that the code
wouldn't
> > run. So, you have to specify something...
> >
> > Marion
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: 12 August 2019 23:22
> > To: North, Rachel <rachel.north at metoffice.gov.uk>
> > Cc: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> > Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
> > merge_thresh option with ==FBIAS1
> >
> > Hi Rachel,
> >
> > That’s a great question. I’m not in the office today and can’t
verify
> > this directly, but I did check the source code... this file in
> particular:
> >
> >
> >
https://github.com/NCAR/MET/blob/master_v8.1/met/src/libcode/vx_shaped
> > ata/mode_conf_info.cc
> >
> > The function set_perc_thresh() processes the fcst and obs merge
> > thresholds together as a pair. So the fcst_merge_thresh = [ >=6.5
];
> > threshold corresponds to some percentile of the forecast data
values.
> > By setting, obs_merge_thresh = [ ==FBIAS1 ]; MODE chooses that
same
> > percentile of the observation data as the threshold value.
> >
> > Try running MODE at something like verbosity 4 (-v 4) and look for
log
> > messages telling you what percentile values were computed.
> >
> > Please let me know if you find the actual behavior to be
different.
> >
> > Thanks,
> > John
> >
> > On Mon, Aug 12, 2019 at 9:40 AM North, Rachel via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Mon Aug 12 09:39:53 2019: Request 91475 was acted upon.
> > > Transaction: Ticket created by rachel.north at metoffice.gov.uk
> > > Queue: met_help
> > > Subject: Guidance for MODE usage of merge_thresh option
with
> > ==FBIAS1
> > > Owner: Nobody
> > > Requestors: rachel.north at metoffice.gov.uk
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475
> > > >
> > >
> > >
> > > Hi,
> > >
> > > Following on from the attached issue, John suggested using the
> > > ==FBIAS1 value in the fcst section of the MODE configuration
file.
> > > We've chosen to use merge_flag option 'BOTH'.
> > >
> > > For our particular application we have set the following values
in
> > > the obs
> > > dictionary:
> > > conv_thresh = [>=8.0];
> > > merge_thresh = [>=6.5];
> > >
> > > Then in the corresponding fcst configuration we set:
> > > conv_thresh = [==FBIAS1];
> > >
> > > I can't find in the documentation any information on how to set
> > > merge_thresh in this case (where conv_thresh is set to
==FBIAS1).
> > > I've just set it to ==FBIAS1 also, but I'm not sure if this the
> > > correct thing to do.
> > >
> > > Could you provide some guidance, please?
> > >
> > > Kind Regards,
> > > Rachel
> > >
> > > Dr Rachel North, Research Scientist
> > > Met Office, FitzRoy Road, Exeter, EX1 3PB, United Kingdom
> > > Tel: +44 (0)1392 884332 Fax: +44 (0)1392 885681
> > >
rachel.north at metoffice.gov.uk<mailto:rachel.north at metoffice.gov.uk>
> > > https://www.metoffice.gov.uk/
> > >
> > >
> > > Marion,
> > >
> > > OK, so you're checking the PostScript output files. I
understand
> > > the problem you're describing, but I just don't see that
behavior in
> > > the output.
> > >
> > > To clarify, I ran the data you sent through MODE using 2 radii
and 2
> > > thresholds (different in the fcst and obs dictionaries) and
quilt =
> TRUE:
> > > quilt = TRUE;
> > > fcst = {
> > > ...
> > > conv_radius = [5, 10];
> > > conv_thresh = [>=2.5, >=1.25 ];
> > > ...
> > > }
> > > obs = {
> > > ...
> > > conv_radius = [6, 12];
> > > conv_thresh = [>=3, >=1.5 ];
> > > ...
> > > }
> > >
> > > This run produces 4 MODE PostScript files:
> > > mode_000000L_20180701_120000V_000000A_R1_T1.ps (5 >=4.7 5
>=2.5)
> > > mode_000000L_20180701_120000V_000000A_R1_T2.ps (5 >=5.6 5
>=3.2)
> > > mode_000000L_20180701_120000V_000000A_R2_T1.ps (6 >=4.7 6
>=2.5)
> > > mode_000000L_20180701_120000V_000000A_R2_T2.ps (6 >=5.6 6
>=3.2)
> > >
> > > After each I've listed the combination of FCST_RAD FCST_THR
OBS_RAD
> > > OBS_THR that were used for that run. And I double-checked the
first
> > > page of the PostScript output to confirm that the settings there
are
> > correctly applied.
> > >
> > > So I can't replicate the problem you describe.
> > >
> > > If you're interested in debiasing the forecast, you could use
the
> > > new percentile threshold options in met-8.1. In the obs
dictionary,
> set:
> > > conv_thresh = [>=2.5, >=3.2];
> > > And in the fcst dictionary, set:
> > > conv_thresh = [==FBIAS1, ==FBIAS1];
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Jun 21, 2019 at 1:55 AM
marion.mittermaier at metoffice.gov.uk
> > > via RT < met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
>
> > > >
> > > > Hi John,
> > > >
> > > > let me try your version and see whether it does what I am
> > > > expecting it to do.
> > > >
> > > > Maybe I didn't explain it very well, but I want to create a
quilt
> > > > plot using different intensity thresholds for the forecast and
> > observations.
> > > > This is because there is a pretty big bias between the two and
> > > > using the same thresholds for both really overwhelms all other
> signals.
> > > > This info
> > > is
> > > > useful up to a point. The cheap option, I thought was to get a
set
> > > > of thresholds from a quantile mapping which effectively
removes
> > > > the
> > bias.
> > > The
> > > > expensive option (in terms of storage) would be to pre-process
> > > > each forecast field and then use the same thresholds again.
> > > >
> > > > When I supplied the two different threshold sequences and took
> > > > away the obs= fcst, it ran but the postscript summary page (1)
> > > > indicates that the SAME thresholds had been used, despite what
I
> > > > had specified. This is NOT what I had expected.
> > > >
> > > > It was only when I took the sequences away that I got the ps
> > > > output to show that different thresholds had been used. I had
> > > > turned quilt=FALSE. I hadn't tested it with quilt=TRUE and
> > > > specifying only one intensity threshold. I will do that, and
> > > > consider your dictionary definition
> > > below,
> > > > and get back to you.
> > > >
> > > > Regards
> > > > Marion
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: 20 June 2019 19:52
> > > > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> > > > Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North,
> > > > Rachel < rachel.north at metoffice.gov.uk>
> > > > Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt
and
> > > different
> > > > threshold sequences for forecast and observation
> > > >
> > > > Hi Marion,
> > > >
> > > > I see you have a question about the quilt option in MODE. I
just
> > > > ran a test using the met-8.1 release and see that it's
behaving as
> > expected.
> > > Let
> > > > me explain what I did to see if that sheds some light on the
the
> issue.
> > > >
> > > > First, I defined separate convolution radii and threshold for
the
> > > > fcst
> > > and
> > > > obs dictionaries, as shown below:
> > > >
> > > >
> > > >
> > > > *fcst = {... conv_radius = [5, 10]; conv_thresh
=
> > > [>=2.5,
> > > > >=1.25 ];*
> > > > *...*
> > > >
> > > >
> > > > *}obs = {...*
> > > >
> > > > * conv_radius = [6, 12]; conv_thresh = [>=3,
>=1.5
> ];*
> > > > *...*
> > > >
> > > > *}*
> > > >
> > > > I ran this example twice, once with QUILT = FALSE and once
with
> > > > QUILT = TRUE.
> > > > Then I cat-ed the "*_obj.txt" output files together and looked
at
> > > > the unique entries in columns 12 - 15:
> > > >
> > > > cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13,
$14,
> > > > $15}' | sort -ru
> > > >
> > > >
> > > > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12
>=1.5*
> > > >
> > > > cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13,
$14,
> > > > $15}' | sort -ru
> > > >
> > > >
> > > >
> > > >
> > > > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6
>=1.510
> > > > >=2.5
> > > 12
> > > > >=310 >=1.25 12 >=1.5*
> > > >
> > > > This is exactly what the output is designed to be. With QUILT
=
> > > > FALSE,
> > > we
> > > > get objects for 2 runs of MODE:
> > > > first radius/first threshold combo
> > > > second radius/second threshold combo
> > > >
> > > > With QUILT = TRUE, we get output for 4 runs of MODE:
> > > > first radius/first threshold combo
> > > > first radius/second threshold combo
> > > > second radius/first threshold combo
> > > > second radius/second threshold combo
> > > >
> > > > And I checked to make sure that the requested forecast radii
(5
> > > > and
> > > > 10) and observation radii (6 and 12) show up in the right
spots.
> > > > So I don't see any obvious problems.
> > > >
> > > > But I can run the example case you sent to see if anything
else
> > > > pops
> > up.
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > >
> > > >
> > > > On Thu, Jun 20, 2019 at 5:37 AM
> > > > marion.mittermaier at metoffice.gov.uk
> > > > via RT < met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
> > > > > Transaction: Ticket created by
marion.mittermaier at metoffice.gov.uk
> > > > > Queue: met_help
> > > > > Subject: Using mode with quilt and different threshold
> > > > > sequences for forecast and observation
> > > > > Owner: Nobody
> > > > > Requestors: marion.mittermaier at metoffice.gov.uk
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
> > > > > >
> > > > >
> > > > >
> > > > > Hello,
> > > > >
> > > > > when quilt=TRUE I'm finding that mode is impervious to my
taking
> > > > > away the obs=fcst line in the config file and specifying
> > > > > different threshold sequences for both.
> > > > >
> > > > > It works only if quilt =FALSE, using a single threshold and
> > > > > convolution radius.
> > > > >
> > > > > Any ideas?
> > > > >
> > > > > Sample forecast, ob and config file attached.
> > > > >
> > > > > Regards
> > > > > Marion
> > > > >
> > > > > --
> > > > > Dr Marion Mittermaier Manager: Model diagnostics and
novel
> > > > verification
> > > > >
> > > > > Met Office FitzRoy Road Exeter EX1 3PB United
Kingdom
> > > > > Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44
(0)1392
> > > > > 885681
> > > > > E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> > > > > marion.mittermaier at metoffice.gov.uk>
> > > > > http://www.metoffice.gov.uk< http://www.metoffice.gov.uk/>
> > > > >
> > > > > http://www.metoffice.gov.uk/research/people/marion-
mittermaier
> > > > >
> > > > > I am co-chair of the WMO WWRP Joint Working Group on
Forecast
> > > > > Verification Research (JWGFVR)
> > > > >
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.
> > > > > html
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: Guidance for MODE usage of merge_thresh option with ==FBIAS1
From: John Halley Gotway
Time: Wed Aug 14 11:03:50 2019
FYI, here's the GitHub issue I added:
https://github.com/NCAR/MET/issues/1183
On Wed, Aug 14, 2019 at 10:14 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Marion,
>
> Glad it's working. Thanks for letting me know about the PostScript
plot.
> Yes, I've seen that in the past with long threshold strings. I'll
write up
> a GitHub issue to (1) use the entire with of the page for the meta
data at
> the bottom and (2) stop writing the fcst/obs interest pairs table
rows once
> it reaches the bottom of the plots to avoid overwriting the table.
>
> But I'll go ahead and resolve this ticket.
>
> Thanks,
> John
>
> On Wed, Aug 14, 2019 at 6:03 AM marion.mittermaier at metoffice.gov.uk
via
> RT <met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475 >
>>
>> Hi John,
>>
>> not sure whether Rachel has responded. Yes it works, and it seems
to have
>> some reduction built in for the merge threshold. Only thing that
isn't very
>> pretty is the way the text comes out in the postscript files... see
>> attached.
>>
>> Marion
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: 13 August 2019 18:13
>> To: North, Rachel <rachel.north at metoffice.gov.uk>
>> Cc: lblank at ucar.edu; Mittermaier, Marion <
>> marion.mittermaier at metoffice.gov.uk>
>> Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
>> merge_thresh option with ==FBIAS1
>>
>> Marion,
>>
>> Yes, I do think that using FBIAS1 for both the conv_thresh and
>> merge_thresh, as shown below, makes sense.
>>
>> Honestly, the testing and evaluation projects in the DTC haven't
made a
>> whole lot of use of this percentile thresholding functionality yet.
So
>> I'll be interested to hear your feedback about how it's going for
you.
>> Please let me know if you run across any problems... or see ways of
>> improving the logic to make it more useful.
>>
>> Thanks,
>> John
>>
>> obs = {...
>> conv_thresh = [>=8.0];
>> merge_thresh = [>=6.5];
>> ...}
>> fcst = {...
>> conv_thresh = [==FBIAS1];
>> merge_thresh = [==FBIAS1];
>> ...}
>>
>>
>> On Tue, Aug 13, 2019 at 2:01 AM marion.mittermaier at metoffice.gov.uk
via
>> RT < met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475 >
>> >
>> > Hi John,
>> >
>> > thanks for the suggestion. Rachel will do that and get back to
you.
>> >
>> > I guess when you run it without FBIAS1 you set the merge
threshold
>> > around 10% lower (rule of thumb)... but we couldn't work out what
to
>> > put. Not specifying a threshold though meant that the code
wouldn't
>> > run. So, you have to specify something...
>> >
>> > Marion
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > Sent: 12 August 2019 23:22
>> > To: North, Rachel <rachel.north at metoffice.gov.uk>
>> > Cc: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
>> > Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
>> > merge_thresh option with ==FBIAS1
>> >
>> > Hi Rachel,
>> >
>> > That’s a great question. I’m not in the office today and can’t
verify
>> > this directly, but I did check the source code... this file in
>> particular:
>> >
>> >
>> >
https://github.com/NCAR/MET/blob/master_v8.1/met/src/libcode/vx_shaped
>> > ata/mode_conf_info.cc
>> >
>> > The function set_perc_thresh() processes the fcst and obs merge
>> > thresholds together as a pair. So the fcst_merge_thresh = [
>=6.5 ];
>> > threshold corresponds to some percentile of the forecast data
values.
>> > By setting, obs_merge_thresh = [ ==FBIAS1 ]; MODE chooses that
same
>> > percentile of the observation data as the threshold value.
>> >
>> > Try running MODE at something like verbosity 4 (-v 4) and look
for log
>> > messages telling you what percentile values were computed.
>> >
>> > Please let me know if you find the actual behavior to be
different.
>> >
>> > Thanks,
>> > John
>> >
>> > On Mon, Aug 12, 2019 at 9:40 AM North, Rachel via RT
>> > <met_help at ucar.edu>
>> > wrote:
>> >
>> > >
>> > > Mon Aug 12 09:39:53 2019: Request 91475 was acted upon.
>> > > Transaction: Ticket created by rachel.north at metoffice.gov.uk
>> > > Queue: met_help
>> > > Subject: Guidance for MODE usage of merge_thresh option
with
>> > ==FBIAS1
>> > > Owner: Nobody
>> > > Requestors: rachel.north at metoffice.gov.uk
>> > > Status: new
>> > > Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475
>> > > >
>> > >
>> > >
>> > > Hi,
>> > >
>> > > Following on from the attached issue, John suggested using the
>> > > ==FBIAS1 value in the fcst section of the MODE configuration
file.
>> > > We've chosen to use merge_flag option 'BOTH'.
>> > >
>> > > For our particular application we have set the following values
in
>> > > the obs
>> > > dictionary:
>> > > conv_thresh = [>=8.0];
>> > > merge_thresh = [>=6.5];
>> > >
>> > > Then in the corresponding fcst configuration we set:
>> > > conv_thresh = [==FBIAS1];
>> > >
>> > > I can't find in the documentation any information on how to set
>> > > merge_thresh in this case (where conv_thresh is set to
==FBIAS1).
>> > > I've just set it to ==FBIAS1 also, but I'm not sure if this the
>> > > correct thing to do.
>> > >
>> > > Could you provide some guidance, please?
>> > >
>> > > Kind Regards,
>> > > Rachel
>> > >
>> > > Dr Rachel North, Research Scientist
>> > > Met Office, FitzRoy Road, Exeter, EX1 3PB, United Kingdom
>> > > Tel: +44 (0)1392 884332 Fax: +44 (0)1392 885681
>> > >
rachel.north at metoffice.gov.uk<mailto:rachel.north at metoffice.gov.uk>
>> > > https://www.metoffice.gov.uk/
>> > >
>> > >
>> > > Marion,
>> > >
>> > > OK, so you're checking the PostScript output files. I
understand
>> > > the problem you're describing, but I just don't see that
behavior in
>> > > the output.
>> > >
>> > > To clarify, I ran the data you sent through MODE using 2 radii
and 2
>> > > thresholds (different in the fcst and obs dictionaries) and
quilt =
>> TRUE:
>> > > quilt = TRUE;
>> > > fcst = {
>> > > ...
>> > > conv_radius = [5, 10];
>> > > conv_thresh = [>=2.5, >=1.25 ];
>> > > ...
>> > > }
>> > > obs = {
>> > > ...
>> > > conv_radius = [6, 12];
>> > > conv_thresh = [>=3, >=1.5 ];
>> > > ...
>> > > }
>> > >
>> > > This run produces 4 MODE PostScript files:
>> > > mode_000000L_20180701_120000V_000000A_R1_T1.ps (5 >=4.7 5
>=2.5)
>> > > mode_000000L_20180701_120000V_000000A_R1_T2.ps (5 >=5.6 5
>=3.2)
>> > > mode_000000L_20180701_120000V_000000A_R2_T1.ps (6 >=4.7 6
>=2.5)
>> > > mode_000000L_20180701_120000V_000000A_R2_T2.ps (6 >=5.6 6
>=3.2)
>> > >
>> > > After each I've listed the combination of FCST_RAD FCST_THR
OBS_RAD
>> > > OBS_THR that were used for that run. And I double-checked the
first
>> > > page of the PostScript output to confirm that the settings
there are
>> > correctly applied.
>> > >
>> > > So I can't replicate the problem you describe.
>> > >
>> > > If you're interested in debiasing the forecast, you could use
the
>> > > new percentile threshold options in met-8.1. In the obs
dictionary,
>> set:
>> > > conv_thresh = [>=2.5, >=3.2];
>> > > And in the fcst dictionary, set:
>> > > conv_thresh = [==FBIAS1, ==FBIAS1];
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Fri, Jun 21, 2019 at 1:55 AM
marion.mittermaier at metoffice.gov.uk
>> > > via RT < met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
>
>> > > >
>> > > > Hi John,
>> > > >
>> > > > let me try your version and see whether it does what I am
>> > > > expecting it to do.
>> > > >
>> > > > Maybe I didn't explain it very well, but I want to create a
quilt
>> > > > plot using different intensity thresholds for the forecast
and
>> > observations.
>> > > > This is because there is a pretty big bias between the two
and
>> > > > using the same thresholds for both really overwhelms all
other
>> signals.
>> > > > This info
>> > > is
>> > > > useful up to a point. The cheap option, I thought was to get
a set
>> > > > of thresholds from a quantile mapping which effectively
removes
>> > > > the
>> > bias.
>> > > The
>> > > > expensive option (in terms of storage) would be to pre-
process
>> > > > each forecast field and then use the same thresholds again.
>> > > >
>> > > > When I supplied the two different threshold sequences and
took
>> > > > away the obs= fcst, it ran but the postscript summary page
(1)
>> > > > indicates that the SAME thresholds had been used, despite
what I
>> > > > had specified. This is NOT what I had expected.
>> > > >
>> > > > It was only when I took the sequences away that I got the ps
>> > > > output to show that different thresholds had been used. I had
>> > > > turned quilt=FALSE. I hadn't tested it with quilt=TRUE and
>> > > > specifying only one intensity threshold. I will do that, and
>> > > > consider your dictionary definition
>> > > below,
>> > > > and get back to you.
>> > > >
>> > > > Regards
>> > > > Marion
>> > > >
>> > > > -----Original Message-----
>> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > > > Sent: 20 June 2019 19:52
>> > > > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
>> > > > Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North,
>> > > > Rachel < rachel.north at metoffice.gov.uk>
>> > > > Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt
and
>> > > different
>> > > > threshold sequences for forecast and observation
>> > > >
>> > > > Hi Marion,
>> > > >
>> > > > I see you have a question about the quilt option in MODE. I
just
>> > > > ran a test using the met-8.1 release and see that it's
behaving as
>> > expected.
>> > > Let
>> > > > me explain what I did to see if that sheds some light on the
the
>> issue.
>> > > >
>> > > > First, I defined separate convolution radii and threshold for
the
>> > > > fcst
>> > > and
>> > > > obs dictionaries, as shown below:
>> > > >
>> > > >
>> > > >
>> > > > *fcst = {... conv_radius = [5, 10]; conv_thresh
=
>> > > [>=2.5,
>> > > > >=1.25 ];*
>> > > > *...*
>> > > >
>> > > >
>> > > > *}obs = {...*
>> > > >
>> > > > * conv_radius = [6, 12]; conv_thresh = [>=3,
>=1.5
>> ];*
>> > > > *...*
>> > > >
>> > > > *}*
>> > > >
>> > > > I ran this example twice, once with QUILT = FALSE and once
with
>> > > > QUILT = TRUE.
>> > > > Then I cat-ed the "*_obj.txt" output files together and
looked at
>> > > > the unique entries in columns 12 - 15:
>> > > >
>> > > > cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13,
$14,
>> > > > $15}' | sort -ru
>> > > >
>> > > >
>> > > > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12
>=1.5*
>> > > >
>> > > > cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13,
$14,
>> > > > $15}' | sort -ru
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6
>=1.510
>> > > > >=2.5
>> > > 12
>> > > > >=310 >=1.25 12 >=1.5*
>> > > >
>> > > > This is exactly what the output is designed to be. With
QUILT =
>> > > > FALSE,
>> > > we
>> > > > get objects for 2 runs of MODE:
>> > > > first radius/first threshold combo
>> > > > second radius/second threshold combo
>> > > >
>> > > > With QUILT = TRUE, we get output for 4 runs of MODE:
>> > > > first radius/first threshold combo
>> > > > first radius/second threshold combo
>> > > > second radius/first threshold combo
>> > > > second radius/second threshold combo
>> > > >
>> > > > And I checked to make sure that the requested forecast radii
(5
>> > > > and
>> > > > 10) and observation radii (6 and 12) show up in the right
spots.
>> > > > So I don't see any obvious problems.
>> > > >
>> > > > But I can run the example case you sent to see if anything
else
>> > > > pops
>> > up.
>> > > >
>> > > > Thanks,
>> > > > John Halley Gotway
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Jun 20, 2019 at 5:37 AM
>> > > > marion.mittermaier at metoffice.gov.uk
>> > > > via RT < met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
>> > > > > Transaction: Ticket created by
>> marion.mittermaier at metoffice.gov.uk
>> > > > > Queue: met_help
>> > > > > Subject: Using mode with quilt and different threshold
>> > > > > sequences for forecast and observation
>> > > > > Owner: Nobody
>> > > > > Requestors: marion.mittermaier at metoffice.gov.uk
>> > > > > Status: new
>> > > > > Ticket <URL:
>> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
>> > > > > >
>> > > > >
>> > > > >
>> > > > > Hello,
>> > > > >
>> > > > > when quilt=TRUE I'm finding that mode is impervious to my
taking
>> > > > > away the obs=fcst line in the config file and specifying
>> > > > > different threshold sequences for both.
>> > > > >
>> > > > > It works only if quilt =FALSE, using a single threshold and
>> > > > > convolution radius.
>> > > > >
>> > > > > Any ideas?
>> > > > >
>> > > > > Sample forecast, ob and config file attached.
>> > > > >
>> > > > > Regards
>> > > > > Marion
>> > > > >
>> > > > > --
>> > > > > Dr Marion Mittermaier Manager: Model diagnostics and
novel
>> > > > verification
>> > > > >
>> > > > > Met Office FitzRoy Road Exeter EX1 3PB United
Kingdom
>> > > > > Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44
(0)1392
>> > > > > 885681
>> > > > > E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
>> > > > > marion.mittermaier at metoffice.gov.uk>
>> > > > > http://www.metoffice.gov.uk< http://www.metoffice.gov.uk/>
>> > > > >
>> > > > > http://www.metoffice.gov.uk/research/people/marion-
mittermaier
>> > > > >
>> > > > > I am co-chair of the WMO WWRP Joint Working Group on
Forecast
>> > > > > Verification Research (JWGFVR)
>> > > > >
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification
>> .
>> > > > > html
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #91475] Guidance for MODE usage of merge_thresh option with ==FBIAS1
From: marion.mittermaier at metoffice.gov.uk
Time: Fri Aug 16 05:59:50 2019
Thanks John.
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 14 August 2019 18:04
To: North, Rachel <rachel.north at metoffice.gov.uk>
Cc: lblank at ucar.edu; Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
merge_thresh option with ==FBIAS1
FYI, here's the GitHub issue I added:
https://github.com/NCAR/MET/issues/1183
On Wed, Aug 14, 2019 at 10:14 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Marion,
>
> Glad it's working. Thanks for letting me know about the PostScript
plot.
> Yes, I've seen that in the past with long threshold strings. I'll
> write up a GitHub issue to (1) use the entire with of the page for
the
> meta data at the bottom and (2) stop writing the fcst/obs interest
> pairs table rows once it reaches the bottom of the plots to avoid
overwriting the table.
>
> But I'll go ahead and resolve this ticket.
>
> Thanks,
> John
>
> On Wed, Aug 14, 2019 at 6:03 AM marion.mittermaier at metoffice.gov.uk
> via RT <met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475 >
>>
>> Hi John,
>>
>> not sure whether Rachel has responded. Yes it works, and it seems
to
>> have some reduction built in for the merge threshold. Only thing
that
>> isn't very pretty is the way the text comes out in the postscript
>> files... see attached.
>>
>> Marion
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: 13 August 2019 18:13
>> To: North, Rachel <rachel.north at metoffice.gov.uk>
>> Cc: lblank at ucar.edu; Mittermaier, Marion <
>> marion.mittermaier at metoffice.gov.uk>
>> Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
>> merge_thresh option with ==FBIAS1
>>
>> Marion,
>>
>> Yes, I do think that using FBIAS1 for both the conv_thresh and
>> merge_thresh, as shown below, makes sense.
>>
>> Honestly, the testing and evaluation projects in the DTC haven't
made
>> a whole lot of use of this percentile thresholding functionality
yet.
>> So I'll be interested to hear your feedback about how it's going
for you.
>> Please let me know if you run across any problems... or see ways of
>> improving the logic to make it more useful.
>>
>> Thanks,
>> John
>>
>> obs = {...
>> conv_thresh = [>=8.0];
>> merge_thresh = [>=6.5];
>> ...}
>> fcst = {...
>> conv_thresh = [==FBIAS1];
>> merge_thresh = [==FBIAS1];
>> ...}
>>
>>
>> On Tue, Aug 13, 2019 at 2:01 AM marion.mittermaier at metoffice.gov.uk
>> via RT < met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475 >
>> >
>> > Hi John,
>> >
>> > thanks for the suggestion. Rachel will do that and get back to
you.
>> >
>> > I guess when you run it without FBIAS1 you set the merge
threshold
>> > around 10% lower (rule of thumb)... but we couldn't work out what
>> > to put. Not specifying a threshold though meant that the code
>> > wouldn't run. So, you have to specify something...
>> >
>> > Marion
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > Sent: 12 August 2019 23:22
>> > To: North, Rachel <rachel.north at metoffice.gov.uk>
>> > Cc: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
>> > Subject: Re: [rt.rap.ucar.edu #91475] Guidance for MODE usage of
>> > merge_thresh option with ==FBIAS1
>> >
>> > Hi Rachel,
>> >
>> > That’s a great question. I’m not in the office today and can’t
>> > verify this directly, but I did check the source code... this
file
>> > in
>> particular:
>> >
>> >
>> >
https://github.com/NCAR/MET/blob/master_v8.1/met/src/libcode/vx_sha
>> > ped
>> > ata/mode_conf_info.cc
>> >
>> > The function set_perc_thresh() processes the fcst and obs merge
>> > thresholds together as a pair. So the fcst_merge_thresh = [
>=6.5
>> > ]; threshold corresponds to some percentile of the forecast data
values.
>> > By setting, obs_merge_thresh = [ ==FBIAS1 ]; MODE chooses that
same
>> > percentile of the observation data as the threshold value.
>> >
>> > Try running MODE at something like verbosity 4 (-v 4) and look
for
>> > log messages telling you what percentile values were computed.
>> >
>> > Please let me know if you find the actual behavior to be
different.
>> >
>> > Thanks,
>> > John
>> >
>> > On Mon, Aug 12, 2019 at 9:40 AM North, Rachel via RT
>> > <met_help at ucar.edu>
>> > wrote:
>> >
>> > >
>> > > Mon Aug 12 09:39:53 2019: Request 91475 was acted upon.
>> > > Transaction: Ticket created by rachel.north at metoffice.gov.uk
>> > > Queue: met_help
>> > > Subject: Guidance for MODE usage of merge_thresh option
with
>> > ==FBIAS1
>> > > Owner: Nobody
>> > > Requestors: rachel.north at metoffice.gov.uk
>> > > Status: new
>> > > Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91475
>> > > >
>> > >
>> > >
>> > > Hi,
>> > >
>> > > Following on from the attached issue, John suggested using the
>> > > ==FBIAS1 value in the fcst section of the MODE configuration
file.
>> > > We've chosen to use merge_flag option 'BOTH'.
>> > >
>> > > For our particular application we have set the following values
>> > > in the obs
>> > > dictionary:
>> > > conv_thresh = [>=8.0];
>> > > merge_thresh = [>=6.5];
>> > >
>> > > Then in the corresponding fcst configuration we set:
>> > > conv_thresh = [==FBIAS1];
>> > >
>> > > I can't find in the documentation any information on how to set
>> > > merge_thresh in this case (where conv_thresh is set to
==FBIAS1).
>> > > I've just set it to ==FBIAS1 also, but I'm not sure if this the
>> > > correct thing to do.
>> > >
>> > > Could you provide some guidance, please?
>> > >
>> > > Kind Regards,
>> > > Rachel
>> > >
>> > > Dr Rachel North, Research Scientist Met Office, FitzRoy Road,
>> > > Exeter, EX1 3PB, United Kingdom
>> > > Tel: +44 (0)1392 884332 Fax: +44 (0)1392 885681
>> > >
rachel.north at metoffice.gov.uk<mailto:rachel.north at metoffice.gov.u
>> > > k>
>> > > https://www.metoffice.gov.uk/
>> > >
>> > >
>> > > Marion,
>> > >
>> > > OK, so you're checking the PostScript output files. I
understand
>> > > the problem you're describing, but I just don't see that
behavior
>> > > in the output.
>> > >
>> > > To clarify, I ran the data you sent through MODE using 2 radii
>> > > and 2 thresholds (different in the fcst and obs dictionaries)
and
>> > > quilt =
>> TRUE:
>> > > quilt = TRUE;
>> > > fcst = {
>> > > ...
>> > > conv_radius = [5, 10];
>> > > conv_thresh = [>=2.5, >=1.25 ];
>> > > ...
>> > > }
>> > > obs = {
>> > > ...
>> > > conv_radius = [6, 12];
>> > > conv_thresh = [>=3, >=1.5 ];
>> > > ...
>> > > }
>> > >
>> > > This run produces 4 MODE PostScript files:
>> > > mode_000000L_20180701_120000V_000000A_R1_T1.ps (5 >=4.7 5
>=2.5)
>> > > mode_000000L_20180701_120000V_000000A_R1_T2.ps (5 >=5.6 5
>=3.2)
>> > > mode_000000L_20180701_120000V_000000A_R2_T1.ps (6 >=4.7 6
>=2.5)
>> > > mode_000000L_20180701_120000V_000000A_R2_T2.ps (6 >=5.6 6
>=3.2)
>> > >
>> > > After each I've listed the combination of FCST_RAD FCST_THR
>> > > OBS_RAD OBS_THR that were used for that run. And I
>> > > double-checked the first page of the PostScript output to
confirm
>> > > that the settings there are
>> > correctly applied.
>> > >
>> > > So I can't replicate the problem you describe.
>> > >
>> > > If you're interested in debiasing the forecast, you could use
the
>> > > new percentile threshold options in met-8.1. In the obs
>> > > dictionary,
>> set:
>> > > conv_thresh = [>=2.5, >=3.2];
>> > > And in the fcst dictionary, set:
>> > > conv_thresh = [==FBIAS1, ==FBIAS1];
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Fri, Jun 21, 2019 at 1:55 AM
>> > > marion.mittermaier at metoffice.gov.uk
>> > > via RT < met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
>
>> > > >
>> > > > Hi John,
>> > > >
>> > > > let me try your version and see whether it does what I am
>> > > > expecting it to do.
>> > > >
>> > > > Maybe I didn't explain it very well, but I want to create a
>> > > > quilt plot using different intensity thresholds for the
>> > > > forecast and
>> > observations.
>> > > > This is because there is a pretty big bias between the two
and
>> > > > using the same thresholds for both really overwhelms all
other
>> signals.
>> > > > This info
>> > > is
>> > > > useful up to a point. The cheap option, I thought was to get
a
>> > > > set of thresholds from a quantile mapping which effectively
>> > > > removes the
>> > bias.
>> > > The
>> > > > expensive option (in terms of storage) would be to pre-
process
>> > > > each forecast field and then use the same thresholds again.
>> > > >
>> > > > When I supplied the two different threshold sequences and
took
>> > > > away the obs= fcst, it ran but the postscript summary page
(1)
>> > > > indicates that the SAME thresholds had been used, despite
what
>> > > > I had specified. This is NOT what I had expected.
>> > > >
>> > > > It was only when I took the sequences away that I got the ps
>> > > > output to show that different thresholds had been used. I had
>> > > > turned quilt=FALSE. I hadn't tested it with quilt=TRUE and
>> > > > specifying only one intensity threshold. I will do that, and
>> > > > consider your dictionary definition
>> > > below,
>> > > > and get back to you.
>> > > >
>> > > > Regards
>> > > > Marion
>> > > >
>> > > > -----Original Message-----
>> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > > > Sent: 20 June 2019 19:52
>> > > > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
>> > > > Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North,
>> > > > Rachel < rachel.north at metoffice.gov.uk>
>> > > > Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt
and
>> > > different
>> > > > threshold sequences for forecast and observation
>> > > >
>> > > > Hi Marion,
>> > > >
>> > > > I see you have a question about the quilt option in MODE. I
>> > > > just ran a test using the met-8.1 release and see that it's
>> > > > behaving as
>> > expected.
>> > > Let
>> > > > me explain what I did to see if that sheds some light on the
>> > > > the
>> issue.
>> > > >
>> > > > First, I defined separate convolution radii and threshold for
>> > > > the fcst
>> > > and
>> > > > obs dictionaries, as shown below:
>> > > >
>> > > >
>> > > >
>> > > > *fcst = {... conv_radius = [5, 10]; conv_thresh
=
>> > > [>=2.5,
>> > > > >=1.25 ];*
>> > > > *...*
>> > > >
>> > > >
>> > > > *}obs = {...*
>> > > >
>> > > > * conv_radius = [6, 12]; conv_thresh = [>=3,
>=1.5
>> ];*
>> > > > *...*
>> > > >
>> > > > *}*
>> > > >
>> > > > I ran this example twice, once with QUILT = FALSE and once
with
>> > > > QUILT = TRUE.
>> > > > Then I cat-ed the "*_obj.txt" output files together and
looked
>> > > > at the unique entries in columns 12 - 15:
>> > > >
>> > > > cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13,
>> > > > $14, $15}' | sort -ru
>> > > >
>> > > >
>> > > > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12
>> > > > >=1.5*
>> > > >
>> > > > cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13,
>> > > > $14, $15}' | sort -ru
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6
>> > > > >=1.510
>> > > > >=2.5
>> > > 12
>> > > > >=310 >=1.25 12 >=1.5*
>> > > >
>> > > > This is exactly what the output is designed to be. With
QUILT
>> > > > = FALSE,
>> > > we
>> > > > get objects for 2 runs of MODE:
>> > > > first radius/first threshold combo
>> > > > second radius/second threshold combo
>> > > >
>> > > > With QUILT = TRUE, we get output for 4 runs of MODE:
>> > > > first radius/first threshold combo
>> > > > first radius/second threshold combo
>> > > > second radius/first threshold combo
>> > > > second radius/second threshold combo
>> > > >
>> > > > And I checked to make sure that the requested forecast radii
(5
>> > > > and
>> > > > 10) and observation radii (6 and 12) show up in the right
spots.
>> > > > So I don't see any obvious problems.
>> > > >
>> > > > But I can run the example case you sent to see if anything
else
>> > > > pops
>> > up.
>> > > >
>> > > > Thanks,
>> > > > John Halley Gotway
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Jun 20, 2019 at 5:37 AM
>> > > > marion.mittermaier at metoffice.gov.uk
>> > > > via RT < met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
>> > > > > Transaction: Ticket created by
>> marion.mittermaier at metoffice.gov.uk
>> > > > > Queue: met_help
>> > > > > Subject: Using mode with quilt and different threshold
>> > > > > sequences for forecast and observation
>> > > > > Owner: Nobody
>> > > > > Requestors: marion.mittermaier at metoffice.gov.uk
>> > > > > Status: new
>> > > > > Ticket <URL:
>> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
>> > > > > >
>> > > > >
>> > > > >
>> > > > > Hello,
>> > > > >
>> > > > > when quilt=TRUE I'm finding that mode is impervious to my
>> > > > > taking away the obs=fcst line in the config file and
>> > > > > specifying different threshold sequences for both.
>> > > > >
>> > > > > It works only if quilt =FALSE, using a single threshold and
>> > > > > convolution radius.
>> > > > >
>> > > > > Any ideas?
>> > > > >
>> > > > > Sample forecast, ob and config file attached.
>> > > > >
>> > > > > Regards
>> > > > > Marion
>> > > > >
>> > > > > --
>> > > > > Dr Marion Mittermaier Manager: Model diagnostics and
novel
>> > > > verification
>> > > > >
>> > > > > Met Office FitzRoy Road Exeter EX1 3PB United
Kingdom
>> > > > > Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44
>> > > > > (0)1392
>> > > > > 885681
>> > > > > E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
>> > > > > marion.mittermaier at metoffice.gov.uk>
>> > > > > http://www.metoffice.gov.uk< http://www.metoffice.gov.uk/>
>> > > > >
>> > > > > http://www.metoffice.gov.uk/research/people/marion-
mittermaie
>> > > > > r
>> > > > >
>> > > > > I am co-chair of the WMO WWRP Joint Working Group on
Forecast
>> > > > > Verification Research (JWGFVR)
>> > > > >
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verifica
>> > > > > tion
>> .
>> > > > > html
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
------------------------------------------------
More information about the Met_help
mailing list