[Met_help] [rt.rap.ucar.edu #90719] History for Using mode with quilt and different threshold sequences for forecast and observation
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:06:42 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
when quilt=TRUE I'm finding that mode is impervious to my taking away the obs=fcst line in the config file and specifying different threshold sequences for both.
It works only if quilt =FALSE, using a single threshold and convolution radius.
Any ideas?
Sample forecast, ob and config file attached.
Regards
Marion
--
Dr Marion Mittermaier Manager: Model diagnostics and novel verification
Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392 885681
E-mail: marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<http://www.metoffice.gov.uk/>
http://www.metoffice.gov.uk/research/people/marion-mittermaier
I am co-chair of the WMO WWRP Joint Working Group on Forecast Verification Research (JWGFVR) http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Using mode with quilt and different threshold sequences for forecast and observation
From: John Halley Gotway
Time: Thu Jun 20 12:51:40 2019
Hi Marion,
I see you have a question about the quilt option in MODE. I just ran
a
test using the met-8.1 release and see that it's behaving as expected.
Let
me explain what I did to see if that sheds some light on the the
issue.
First, I defined separate convolution radii and threshold for the fcst
and
obs dictionaries, as shown below:
*fcst = {... conv_radius = [5, 10]; conv_thresh =
[>=2.5,
>=1.25 ];*
*...*
*}obs = {...*
* conv_radius = [6, 12]; conv_thresh = [>=3, >=1.5
];*
*...*
*}*
I ran this example twice, once with QUILT = FALSE and once with QUILT
=
TRUE.
Then I cat-ed the "*_obj.txt" output files together and looked at the
unique entries in columns 12 - 15:
cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}' |
sort -ru
*FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12 >=1.5*
cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14, $15}'
|
sort -ru
*FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6 >=1.510
>=2.5 12
>=310 >=1.25 12 >=1.5*
This is exactly what the output is designed to be. With QUILT =
FALSE, we
get objects for 2 runs of MODE:
first radius/first threshold combo
second radius/second threshold combo
With QUILT = TRUE, we get output for 4 runs of MODE:
first radius/first threshold combo
first radius/second threshold combo
second radius/first threshold combo
second radius/second threshold combo
And I checked to make sure that the requested forecast radii (5 and
10) and
observation radii (6 and 12) show up in the right spots. So I don't
see
any obvious problems.
But I can run the example case you sent to see if anything else pops
up.
Thanks,
John Halley Gotway
On Thu, Jun 20, 2019 at 5:37 AM marion.mittermaier at metoffice.gov.uk
via RT <
met_help at ucar.edu> wrote:
>
> Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
> Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> Queue: met_help
> Subject: Using mode with quilt and different threshold
sequences for
> forecast and observation
> Owner: Nobody
> Requestors: marion.mittermaier at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719 >
>
>
> Hello,
>
> when quilt=TRUE I'm finding that mode is impervious to my taking
away the
> obs=fcst line in the config file and specifying different threshold
> sequences for both.
>
> It works only if quilt =FALSE, using a single threshold and
convolution
> radius.
>
> Any ideas?
>
> Sample forecast, ob and config file attached.
>
> Regards
> Marion
>
> --
> Dr Marion Mittermaier Manager: Model diagnostics and novel
verification
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
885681
> E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am co-chair of the WMO WWRP Joint Working Group on Forecast
Verification
> Research (JWGFVR)
>
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
>
>
>
------------------------------------------------
Subject: Using mode with quilt and different threshold sequences for forecast and observation
From: John Halley Gotway
Time: Thu Jun 20 15:01:10 2019
Marion,
I ran the test case you sent, using 2 radii and thresholds from your
lists,
quilt=true, and again see the expected behavior. See below. So where
is
the confusion coming in? Is this not the behavior you expected? Or
is
there some other problem I'm not seeing?
Thanks,
John
fcst = {
conv_radius = [5, 6];
conv_thresh = [>=4.7, >=5.6];
}
obs = {
conv_radius = [5, 6];
conv_thresh = [>=2.5, >=3.2];
}
cat out_mm/mode*_obj.txt | awk '{print $12, $13, $14, $15}' | sort -u
FCST_RAD FCST_THR OBS_RAD OBS_THR
5 >=4.7 5 >=2.5
5 >=5.6 5 >=3.2
6 >=4.7 6 >=2.5
6 >=5.6 6 >=3.2
There is output for the 4 expected permutation of radius/threshold.
On Thu, Jun 20, 2019 at 12:51 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Hi Marion,
>
> I see you have a question about the quilt option in MODE. I just
ran a
> test using the met-8.1 release and see that it's behaving as
expected. Let
> me explain what I did to see if that sheds some light on the the
issue.
>
> First, I defined separate convolution radii and threshold for the
fcst and
> obs dictionaries, as shown below:
>
>
>
> *fcst = {... conv_radius = [5, 10]; conv_thresh =
> [>=2.5, >=1.25 ];*
> *...*
>
>
> *}obs = {...*
>
> * conv_radius = [6, 12]; conv_thresh = [>=3, >=1.5
];*
> *...*
>
> *}*
>
> I ran this example twice, once with QUILT = FALSE and once with
QUILT =
> TRUE.
> Then I cat-ed the "*_obj.txt" output files together and looked at
the
> unique entries in columns 12 - 15:
>
> cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}' |
> sort -ru
>
>
> *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12 >=1.5*
>
> cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}' |
> sort -ru
>
>
>
>
> *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6 >=1.510
>=2.5 12
> >=310 >=1.25 12 >=1.5*
>
> This is exactly what the output is designed to be. With QUILT =
FALSE, we
> get objects for 2 runs of MODE:
> first radius/first threshold combo
> second radius/second threshold combo
>
> With QUILT = TRUE, we get output for 4 runs of MODE:
> first radius/first threshold combo
> first radius/second threshold combo
> second radius/first threshold combo
> second radius/second threshold combo
>
> And I checked to make sure that the requested forecast radii (5 and
10)
> and observation radii (6 and 12) show up in the right spots. So I
don't
> see any obvious problems.
>
> But I can run the example case you sent to see if anything else pops
up.
>
> Thanks,
> John Halley Gotway
>
>
>
> On Thu, Jun 20, 2019 at 5:37 AM marion.mittermaier at metoffice.gov.uk
via
> RT <met_help at ucar.edu> wrote:
>
>>
>> Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
>> Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
>> Queue: met_help
>> Subject: Using mode with quilt and different threshold
sequences for
>> forecast and observation
>> Owner: Nobody
>> Requestors: marion.mittermaier at metoffice.gov.uk
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719 >
>>
>>
>> Hello,
>>
>> when quilt=TRUE I'm finding that mode is impervious to my taking
away the
>> obs=fcst line in the config file and specifying different threshold
>> sequences for both.
>>
>> It works only if quilt =FALSE, using a single threshold and
convolution
>> radius.
>>
>> Any ideas?
>>
>> Sample forecast, ob and config file attached.
>>
>> Regards
>> Marion
>>
>> --
>> Dr Marion Mittermaier Manager: Model diagnostics and novel
>> verification
>>
>> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
>> Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
885681
>> E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
>> marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<
>> http://www.metoffice.gov.uk/>
>>
>> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>>
>> I am co-chair of the WMO WWRP Joint Working Group on Forecast
>> Verification Research (JWGFVR)
>>
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
>>
>>
>>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #90719] Using mode with quilt and different threshold sequences for forecast and observation
From: marion.mittermaier at metoffice.gov.uk
Time: Fri Jun 21 01:55:55 2019
Hi John,
let me try your version and see whether it does what I am expecting it
to do.
Maybe I didn't explain it very well, but I want to create a quilt plot
using different intensity thresholds for the forecast and
observations. This is because there is a pretty big bias between the
two and using the same thresholds for both really overwhelms all other
signals. This info is useful up to a point. The cheap option, I
thought was to get a set of thresholds from a quantile mapping which
effectively removes the bias. The expensive option (in terms of
storage) would be to pre-process each forecast field and then use the
same thresholds again.
When I supplied the two different threshold sequences and took away
the obs= fcst, it ran but the postscript summary page (1) indicates
that the SAME thresholds had been used, despite what I had specified.
This is NOT what I had expected.
It was only when I took the sequences away that I got the ps output to
show that different thresholds had been used. I had turned
quilt=FALSE. I hadn't tested it with quilt=TRUE and specifying only
one intensity threshold. I will do that, and consider your dictionary
definition below, and get back to you.
Regards
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 20 June 2019 19:52
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North, Rachel
<rachel.north at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt and
different threshold sequences for forecast and observation
Hi Marion,
I see you have a question about the quilt option in MODE. I just ran
a test using the met-8.1 release and see that it's behaving as
expected. Let me explain what I did to see if that sheds some light
on the the issue.
First, I defined separate convolution radii and threshold for the fcst
and obs dictionaries, as shown below:
*fcst = {... conv_radius = [5, 10]; conv_thresh =
[>=2.5,
>=1.25 ];*
*...*
*}obs = {...*
* conv_radius = [6, 12]; conv_thresh = [>=3, >=1.5
];*
*...*
*}*
I ran this example twice, once with QUILT = FALSE and once with QUILT
= TRUE.
Then I cat-ed the "*_obj.txt" output files together and looked at the
unique entries in columns 12 - 15:
cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}' | sort -ru
*FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12 >=1.5*
cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14, $15}'
| sort -ru
*FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6 >=1.510
>=2.5 12
>=310 >=1.25 12 >=1.5*
This is exactly what the output is designed to be. With QUILT =
FALSE, we get objects for 2 runs of MODE:
first radius/first threshold combo
second radius/second threshold combo
With QUILT = TRUE, we get output for 4 runs of MODE:
first radius/first threshold combo
first radius/second threshold combo
second radius/first threshold combo
second radius/second threshold combo
And I checked to make sure that the requested forecast radii (5 and
10) and observation radii (6 and 12) show up in the right spots. So I
don't see any obvious problems.
But I can run the example case you sent to see if anything else pops
up.
Thanks,
John Halley Gotway
On Thu, Jun 20, 2019 at 5:37 AM marion.mittermaier at metoffice.gov.uk
via RT < met_help at ucar.edu> wrote:
>
> Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
> Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> Queue: met_help
> Subject: Using mode with quilt and different threshold
sequences
> for forecast and observation
> Owner: Nobody
> Requestors: marion.mittermaier at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
> >
>
>
> Hello,
>
> when quilt=TRUE I'm finding that mode is impervious to my taking
away
> the obs=fcst line in the config file and specifying different
> threshold sequences for both.
>
> It works only if quilt =FALSE, using a single threshold and
> convolution radius.
>
> Any ideas?
>
> Sample forecast, ob and config file attached.
>
> Regards
> Marion
>
> --
> Dr Marion Mittermaier Manager: Model diagnostics and novel
verification
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
> 885681
> E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am co-chair of the WMO WWRP Joint Working Group on Forecast
> Verification Research (JWGFVR)
>
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
>
>
>
------------------------------------------------
Subject: Using mode with quilt and different threshold sequences for forecast and observation
From: marion.mittermaier at metoffice.gov.uk
Time: Fri Jun 21 03:45:31 2019
Hi John,
I've run the following setup with quilt=TRUE, different thresholds for
obs and forecast. I've attached sample output. This shows that it has
apparently applied two different thresholds to the fields.
I don't see any difference between what you did and I tried to do in
terms of syntax to expand the highlighted lines to include more than
one intensity threshold though… I will try that again.
Marion
fcst = {
conv_radius = [1,2,3,4,5,6,7,8,9,10,11]; // in grid squares
conv_thresh = [>=3.9]; --> sequence
vld_thresh = 0.5;
filter_attr_name = ["AREA"];
filter_attr_thresh = [>=10.0];
merge_thresh = [>=3.5]; --> sequence
merge_flag = BOTH; // options are none, thresh, engine, both
}
////////////////////obs = fcst;
obs = {
conv_radius = [1,2,3,4,5,6,7,8,9,10,11]; // in grid squares
conv_thresh = [>=2.]; --> sequence
vld_thresh = 0.5;
filter_attr_name = ["AREA"];
filter_attr_thresh = [>=10.0];
merge_thresh = [>=1.9]; --> sequence
merge_flag = BOTH; // options are none, thresh, engine, both
}
Regards
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 20 June 2019 19:52
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North, Rachel
<rachel.north at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt and
different threshold sequences for forecast and observation
Hi Marion,
I see you have a question about the quilt option in MODE. I just ran
a test using the met-8.1 release and see that it's behaving as
expected. Let me explain what I did to see if that sheds some light
on the the issue.
First, I defined separate convolution radii and threshold for the fcst
and obs dictionaries, as shown below:
*fcst = {... conv_radius = [5, 10]; conv_thresh =
[>=2.5,
>=1.25 ];*
*...*
*}obs = {...*
* conv_radius = [6, 12]; conv_thresh = [>=3, >=1.5
];*
*...*
*}*
I ran this example twice, once with QUILT = FALSE and once with QUILT
= TRUE.
Then I cat-ed the "*_obj.txt" output files together and looked at the
unique entries in columns 12 - 15:
cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}' | sort -ru
*FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12 >=1.5*
cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14, $15}'
| sort -ru
*FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6 >=1.510
>=2.5 12
>=310 >=1.25 12 >=1.5*
This is exactly what the output is designed to be. With QUILT =
FALSE, we get objects for 2 runs of MODE:
first radius/first threshold combo
second radius/second threshold combo
With QUILT = TRUE, we get output for 4 runs of MODE:
first radius/first threshold combo
first radius/second threshold combo
second radius/first threshold combo
second radius/second threshold combo
And I checked to make sure that the requested forecast radii (5 and
10) and observation radii (6 and 12) show up in the right spots. So I
don't see any obvious problems.
But I can run the example case you sent to see if anything else pops
up.
Thanks,
John Halley Gotway
On Thu, Jun 20, 2019 at 5:37 AM
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
via RT < met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
> Transaction: Ticket created by
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
> Queue: met_help
> Subject: Using mode with quilt and different threshold
sequences
> for forecast and observation
> Owner: Nobody
> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
> >
>
>
> Hello,
>
> when quilt=TRUE I'm finding that mode is impervious to my taking
away
> the obs=fcst line in the config file and specifying different
> threshold sequences for both.
>
> It works only if quilt =FALSE, using a single threshold and
> convolution radius.
>
> Any ideas?
>
> Sample forecast, ob and config file attached.
>
> Regards
> Marion
>
> --
> Dr Marion Mittermaier Manager: Model diagnostics and novel
verification
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
> 885681
> E-mail:
marion.mittermaier at metoffice.gov.uk<mailto<mailto:marion.mittermaier at metoffice.gov.uk%3cmailto>:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
http://www.metoffice.gov.uk<<http://www.metoffice.gov.uk%3c>
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am co-chair of the WMO WWRP Joint Working Group on Forecast
> Verification Research (JWGFVR)
>
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
>
>
>
------------------------------------------------
Subject: Using mode with quilt and different threshold sequences for forecast and observation
From: marion.mittermaier at metoffice.gov.uk
Time: Fri Jun 21 03:58:58 2019
Hi John,
ok, so this is VERY odd. I have now reinstated my thresholds in the
config file and it is now showing output using the different
thresholds... see attached.
However, I have now managed to replicate the memory fault with one of
the files... this did NOT happen yesterday.... I think I am going
slightly mad...!
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 20 June 2019 19:52
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North, Rachel
<rachel.north at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt and
different threshold sequences for forecast and observation
Hi Marion,
I see you have a question about the quilt option in MODE. I just ran
a test using the met-8.1 release and see that it's behaving as
expected. Let me explain what I did to see if that sheds some light
on the the issue.
First, I defined separate convolution radii and threshold for the fcst
and obs dictionaries, as shown below:
*fcst = {... conv_radius = [5, 10]; conv_thresh =
[>=2.5,
>=1.25 ];*
*...*
*}obs = {...*
* conv_radius = [6, 12]; conv_thresh = [>=3, >=1.5
];*
*...*
*}*
I ran this example twice, once with QUILT = FALSE and once with QUILT
= TRUE.
Then I cat-ed the "*_obj.txt" output files together and looked at the
unique entries in columns 12 - 15:
cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}' | sort -ru
*FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12 >=1.5*
cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14, $15}'
| sort -ru
*FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6 >=1.510
>=2.5 12
>=310 >=1.25 12 >=1.5*
This is exactly what the output is designed to be. With QUILT =
FALSE, we get objects for 2 runs of MODE:
first radius/first threshold combo
second radius/second threshold combo
With QUILT = TRUE, we get output for 4 runs of MODE:
first radius/first threshold combo
first radius/second threshold combo
second radius/first threshold combo
second radius/second threshold combo
And I checked to make sure that the requested forecast radii (5 and
10) and observation radii (6 and 12) show up in the right spots. So I
don't see any obvious problems.
But I can run the example case you sent to see if anything else pops
up.
Thanks,
John Halley Gotway
On Thu, Jun 20, 2019 at 5:37 AM marion.mittermaier at metoffice.gov.uk
via RT < met_help at ucar.edu> wrote:
>
> Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
> Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> Queue: met_help
> Subject: Using mode with quilt and different threshold
sequences
> for forecast and observation
> Owner: Nobody
> Requestors: marion.mittermaier at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
> >
>
>
> Hello,
>
> when quilt=TRUE I'm finding that mode is impervious to my taking
away
> the obs=fcst line in the config file and specifying different
> threshold sequences for both.
>
> It works only if quilt =FALSE, using a single threshold and
> convolution radius.
>
> Any ideas?
>
> Sample forecast, ob and config file attached.
>
> Regards
> Marion
>
> --
> Dr Marion Mittermaier Manager: Model diagnostics and novel
verification
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
> 885681
> E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am co-chair of the WMO WWRP Joint Working Group on Forecast
> Verification Research (JWGFVR)
>
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
>
>
>
------------------------------------------------
Subject: Using mode with quilt and different threshold sequences for forecast and observation
From: marion.mittermaier at metoffice.gov.uk
Time: Fri Jun 21 08:23:15 2019
John,
try using the attached observation file with the same forecast file
and see what you get...? I must be losing my marbles. The run script
uses this combination and did run before. I swear! We’ve got a whole
bunch of output to prove it (using obs=fcst), and I’ve just checked
that. Using the same thresholds does not result in a failure. There is
something about using different thresholds which seems to be causing
this failure.
Regards
Marion
====================================
vld497:/home/h02/frmm/MET/scripts >
LOUR_ATL_CHL_L4_NRT_OBSERVATIONS_009_037_2018-07-02.nc
$HOME/MET/config/MODEConfi*
DEBUG 1: Default Config File: /project/ukmo/meteval/met-
8.0/share/met/config/MODEConfig_default
DEBUG 1: Match Config File:
/home/h02/frmm/MET/config/MODEConfig_BIO_quilt_mergeboth
DEBUG 1: Merge Config File:
/home/h02/frmm/MET/config/MODEConfig_BIO_quilt_mergeboth
WARNING:
WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention
time unit "hours since 00:00:00"
WARNING:
DEBUG 1: Forecast File:
/scratch/fryl/HIVE/input/model/BIO/fake_forecasts/MetO-NWS-BIO-dm-
CPWC_Forecast_24_2018-07-02.nc
DEBUG 1: Observation File:
/project/hive/input/observations/BIO/AMM7REGRIDDED_OCEANCOLOUR_ATL_CHL_L4_NRT_OBSERVATIONS_009_037_2018-
07-02.nc
WARNING:
WARNING: setup_fcst_obs_data() -> Forecast and observation valid times
do not match 20180701_120000 != 20180702_000000 for CHL(0,0,*,*)
versus CHL(0,0,*,*).
WARNING:
DEBUG 1: Forecast Field: CHL at 0,0,*,*
DEBUG 1: Observation Field: CHL at 0,0,*,*
DEBUG 2: Processing masking regions.
DEBUG 2: Identifying objects in the forecast and observation fields...
DEBUG 2: Computing contingency table statistics...
DEBUG 2: Identified: 27 forecast objects and 37 observation objects.
DEBUG 2: Performing merging (BOTH) in the forecast field.
DEBUG 2: Performing merging (BOTH) in the observation field.
DEBUG 2: Remaining: 27 forecast objects and 37 observation objects.
DEBUG 2: Performing matching (MERGE_BOTH) between the forecast and
observation fields.
*** Error in `mode': double free or corruption (out):
0x000000000133ed10 ***
======= Backtrace: =========
/usr/lib64/libc.so.6(+0x81489)[0x7f726df5d489]
mode[0x4a3d0b]
mode[0x4a3b82]
mode[0x4a4614]
mode[0x4a468e]
mode[0x49531b]
mode[0x493df4]
mode[0x423026]
mode[0x411dc6]
mode[0x411c43]
/usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f726defe3d5]
mode[0x411b09]
======= Memory map: ========
00400000-007dc000 r-xp 00000000 00:33 13835060284413110624
/net/project/ukmo/rhel6/meteval/met-8.0/bin/mode
009db000-00abd000 rw-p 003db000 00:33 13835060284413110624
/net/project/ukmo/rhel6/meteval/met-8.0/bin/mode
00abd000-00ae5000 rw-p 00000000 00:00 0
01300000-071b6000 rw-p 00000000 00:00 0
[heap]
7f7260000000-7f7260021000 rw-p 00000000 00:00 0
7f7260021000-7f7264000000 ---p 00000000 00:00 0
7f7267979000-7f72682e4000 rw-p 00000000 00:00 0
7f72682e4000-7f72682e6000 r-xp 00000000 fd:00 8389978
/usr/lib64/libfreebl3.so
7f72682e6000-7f72684e5000 ---p 00002000 fd:00 8389978
/usr/lib64/libfreebl3.so
7f72684e5000-7f72684e6000 r--p 00001000 fd:00 8389978
/usr/lib64/libfreebl3.so
7f72684e6000-7f72684e7000 rw-p 00002000 fd:00 8389978
/usr/lib64/libfreebl3.so
7f72684e7000-7f7268547000 r-xp 00000000 fd:00 8405743
/usr/lib64/libpcre.so.1.2.0
7f7268547000-7f7268747000 ---p 00060000 fd:00 8405743
/usr/lib64/libpcre.so.1.2.0
7f7268747000-7f7268748000 r--p 00060000 fd:00 8405743
/usr/lib64/libpcre.so.1.2.0
7f7268748000-7f7268749000 rw-p 00061000 fd:00 8405743
/usr/lib64/libpcre.so.1.2.0
7f7268749000-7f7268751000 r-xp 00000000 fd:00 8784251
/usr/lib64/libcrypt-2.17.so
7f7268751000-7f7268950000 ---p 00008000 fd:00 8784251
/usr/lib64/libcrypt-2.17.so
7f7268950000-7f7268951000 r--p 00007000 fd:00 8784251
/usr/lib64/libcrypt-2.17.so
7f7268951000-7f7268952000 rw-p 00008000 fd:00 8784251
/usr/lib64/libcrypt-2.17.so
7f7268952000-7f7268980000 rw-p 00000000 00:00 0
7f7268980000-7f72689a4000 r-xp 00000000 fd:00 8405753
/usr/lib64/libselinux.so.1
7f72689a4000-7f7268ba3000 ---p 00024000 fd:00 8405753
/usr/lib64/libselinux.so.1
7f7268ba3000-7f7268ba4000 r--p 00023000 fd:00 8405753
/usr/lib64/libselinux.so.1
7f7268ba4000-7f7268ba5000 rw-p 00024000 fd:00 8405753
/usr/lib64/libselinux.so.1
7f7268ba5000-7f7268ba7000 rw-p 00000000 00:00 0
7f7268ba7000-7f7268be6000 r-xp 00000000 00:33 13835060284415658851
/net/project/ukmo/rhel7/fortran/opt/gfortran/packages/gcc/8.1.0/lib64/libquadmath.so.0.0.0
7f7268be6000-7f7268de5000 ---p 0003f000 00:33 13835060284415658851
/net/project/ukmo/rhel7/fortran/opt/gfortran/packages/gcc/8.1.0/lib64/libquadmath.so.0.0.0
7f7268de5000-7f7268de6000 r--p 0003e000 00:33 13835060284415658851
/net/project/ukmo/rhel7/fortran/opt/gfortran/packages/gcc/8.1.0/lib64/libquadmath.so.0.0.0
7f7268de6000-7f7268de7000 rw-p 0003f000 00:33 13835060284415658851
/net/project/ukmo/rhel7/fortran/opt/gfortran/packages/gcc/8.1.0/lib64/libquadmath.so.0.0.0
7f7268de7000-7f7268e03000 r-xp 00000000 fd:00 8484901
/usr/lib64/libsasl2.so.3.0.0
7f7268e03000-7f7269002000 ---p 0001c000 fd:00 8484901
/usr/lib64/libsasl2.so.3.0.0
7f7269002000-7f7269003000 r--p 0001b000 fd:00 8484901
/usr/lib64/libsasl2.so.3.0.0
7f7269003000-7f7269004000 rw-p 0001c000 fd:00 8484901
/usr/lib64/libsasl2.so.3.0.0
7f7269004000-7f726901a000 r-xp 00000000 fd:00 8828077
/usr/lib64/libresolv-2.17.so
7f726901a000-7f7269219000 ---p 00016000 fd:00 8828077
/usr/lib64/libresolv-2.17.so
7f7269219000-7f726921a000 r--p 00015000 fd:00 8828077
/usr/lib64/libresolv-2.17.so
7f726921a000-7f726921b000 rw-p 00016000 fd:00 8828077
/usr/lib64/libresolv-2.17.so
7f726921b000-7f726921d000 rw-p 00000000 00:00 0
7f726921d000-7f7269220000 r-xp 00000000 fd:00 8421134
/usr/lib64/libkeyutils.so.1.5
7f7269220000-7f726941f000 ---p 00003000 fd:00 8421134
/usr/lib64/libkeyutils.so.1.5
7f726941f000-7f7269420000 r--p 00002000 fd:00 8421134
/usr/lib64/libkeyutils.so.1.5
7f7269420000-7f7269421000 rw-p 00003000 fd:00 8421134
/usr/lib64/libkeyutils.so.1.5
7f7269421000-7f726942f000 r-xp 00000000 fd:00 9768408
/usr/lib64/libkrb5support.so.0.1
7f726942f000-7f726962f000 ---p 0000e000 fd:00 9768408
/usr/lib64/libkrb5support.so.0.1
7f726962f000-7f7269630000 r--p 0000e000 fd:00 9768408
/usr/lib64/libkrb5support.so.0.1
7f7269630000-7f7269631000 rw-p 0000f000 fd:00 9768408
/usr/lib64/libkrb5support.so.0.1
7f7269631000-7f7269866000 r-xp 00000000 fd:00 8464310
/usr/lib64/libcrypto.so.1.0.2k
7f7269866000-7f7269a66000 ---p 00235000 fd:00 8464310
/usr/lib64/libcrypto.so.1.0.2k
7f7269a66000-7f7269a82000 r--p 00235000 fd:00 8464310
/usr/lib64/libcrypto.so.1.0.2k
7f7269a82000-7f7269a8f000 rw-p 00251000 fd:00 8464310
/usr/lib64/libcrypto.so.1.0.2k
7f7269a8f000-7f7269a93000 rw-p 00000000 00:00 0
7f7269a93000-7f7269afa000 r-xp 00000000 fd:00 8663738
/usr/lib64/libssl.so.1.0.2k
7f7269afa000-7f7269cfa000 ---p 00067000 fd:00 8663738
/usr/lib64/libssl.so.1.0.2k
7f7269cfa000-7f7269cfe000 r--p 00067000 fd:00 8663738
/usr/lib64/libssl.so.1.0.2k
7f7269cfe000-7f7269d05000 rw-p 0006b000 fd:00 8663738
/usr/lib64/libssl.so.1.0.2k
7f7269d05000-7f7269e24000 r-xp 00000000 fd:00 8421092
/usr/lib64/libgfortran.so.3.0.0
7f7269e24000-7f726a024000 ---p 0011f000 fd:00 8421092
/usr/lib64/libgfortran.so.3.0.0
7f726a024000-7f726a025000 r--p 0011f000 fd:00 8421092
/usr/lib64/libgfortran.so.3.0.0
7f726a025000-7f726a027000 rw-p 00120000 fd:00 8421092
/usr/lib64/libgfortran.so.3.0.0
7f726a027000-7f726a079000 r-xp 00000000 fd:00 8389564
/usr/lib64/libldap-2.4.so.2.10.7Abort
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 20 June 2019 22:01
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North, Rachel
<rachel.north at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt and
different threshold sequences for forecast and observation
Marion,
I ran the test case you sent, using 2 radii and thresholds from your
lists, quilt=true, and again see the expected behavior. See below.
So where is the confusion coming in? Is this not the behavior you
expected? Or is there some other problem I'm not seeing?
Thanks,
John
fcst = {
conv_radius = [5, 6];
conv_thresh = [>=4.7, >=5.6];
}
obs = {
conv_radius = [5, 6];
conv_thresh = [>=2.5, >=3.2];
}
cat out_mm/mode*_obj.txt | awk '{print $12, $13, $14, $15}' | sort -u
FCST_RAD FCST_THR OBS_RAD OBS_THR
5 >=4.7 5 >=2.5
5 >=5.6 5 >=3.2
6 >=4.7 6 >=2.5
6 >=5.6 6 >=3.2
There is output for the 4 expected permutation of radius/threshold.
On Thu, Jun 20, 2019 at 12:51 PM John Halley Gotway
<johnhg at ucar.edu<mailto:johnhg at ucar.edu>> wrote:
> Hi Marion,
>
> I see you have a question about the quilt option in MODE. I just
ran
> a test using the met-8.1 release and see that it's behaving as
> expected. Let me explain what I did to see if that sheds some light
on the the issue.
>
> First, I defined separate convolution radii and threshold for the
fcst
> and obs dictionaries, as shown below:
>
>
>
> *fcst = {... conv_radius = [5, 10]; conv_thresh =
> [>=2.5, >=1.25 ];*
> *...*
>
>
> *}obs = {...*
>
> * conv_radius = [6, 12]; conv_thresh = [>=3, >=1.5
];*
> *...*
>
> *}*
>
> I ran this example twice, once with QUILT = FALSE and once with
QUILT
> = TRUE.
> Then I cat-ed the "*_obj.txt" output files together and looked at
the
> unique entries in columns 12 - 15:
>
> cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13, $14,
> $15}' | sort -ru
>
>
> *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12 >=1.5*
>
> cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}'
> | sort -ru
>
>
>
>
> *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6 >=1.510
> >=2.5 12
> >=310 >=1.25 12 >=1.5*
>
> This is exactly what the output is designed to be. With QUILT =
> FALSE, we get objects for 2 runs of MODE:
> first radius/first threshold combo
> second radius/second threshold combo
>
> With QUILT = TRUE, we get output for 4 runs of MODE:
> first radius/first threshold combo
> first radius/second threshold combo
> second radius/first threshold combo
> second radius/second threshold combo
>
> And I checked to make sure that the requested forecast radii (5 and
> 10) and observation radii (6 and 12) show up in the right spots. So
I
> don't see any obvious problems.
>
> But I can run the example case you sent to see if anything else pops
up.
>
> Thanks,
> John Halley Gotway
>
>
>
> On Thu, Jun 20, 2019 at 5:37 AM
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
> via RT <met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
>>
>> Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
>> Transaction: Ticket created by
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
>> Queue: met_help
>> Subject: Using mode with quilt and different threshold
sequences
>> for forecast and observation
>> Owner: Nobody
>> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
>> >
>>
>>
>> Hello,
>>
>> when quilt=TRUE I'm finding that mode is impervious to my taking
away
>> the obs=fcst line in the config file and specifying different
>> threshold sequences for both.
>>
>> It works only if quilt =FALSE, using a single threshold and
>> convolution radius.
>>
>> Any ideas?
>>
>> Sample forecast, ob and config file attached.
>>
>> Regards
>> Marion
>>
>> --
>> Dr Marion Mittermaier Manager: Model diagnostics and novel
>> verification
>>
>> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
>> Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
>> 885681
>> E-mail:
marion.mittermaier at metoffice.gov.uk<mailto<mailto:marion.mittermaier at metoffice.gov.uk%3cmailto>:
>>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
http://www.metoffice.gov.uk<<http://www.metoffice.gov.uk%3c>
>> http://www.metoffice.gov.uk/>
>>
>> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>>
>> I am co-chair of the WMO WWRP Joint Working Group on Forecast
>> Verification Research (JWGFVR)
>>
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.htm
>> l
>>
>>
>>
------------------------------------------------
Subject: Using mode with quilt and different threshold sequences for forecast and observation
From: John Halley Gotway
Time: Fri Jun 21 10:02:47 2019
Marion,
OK, so you're checking the PostScript output files. I understand the
problem you're describing, but I just don't see that behavior in the
output.
To clarify, I ran the data you sent through MODE using 2 radii and 2
thresholds (different in the fcst and obs dictionaries) and quilt =
TRUE:
quilt = TRUE;
fcst = {
...
conv_radius = [5, 10];
conv_thresh = [>=2.5, >=1.25 ];
...
}
obs = {
...
conv_radius = [6, 12];
conv_thresh = [>=3, >=1.5 ];
...
}
This run produces 4 MODE PostScript files:
mode_000000L_20180701_120000V_000000A_R1_T1.ps (5 >=4.7 5 >=2.5)
mode_000000L_20180701_120000V_000000A_R1_T2.ps (5 >=5.6 5 >=3.2)
mode_000000L_20180701_120000V_000000A_R2_T1.ps (6 >=4.7 6 >=2.5)
mode_000000L_20180701_120000V_000000A_R2_T2.ps (6 >=5.6 6 >=3.2)
After each I've listed the combination of FCST_RAD FCST_THR OBS_RAD
OBS_THR
that were used for that run. And I double-checked the first page of
the
PostScript output to confirm that the settings there are correctly
applied.
So I can't replicate the problem you describe.
If you're interested in debiasing the forecast, you could use the new
percentile threshold options in met-8.1. In the obs dictionary, set:
conv_thresh = [>=2.5, >=3.2];
And in the fcst dictionary, set:
conv_thresh = [==FBIAS1, ==FBIAS1];
Thanks,
John
On Fri, Jun 21, 2019 at 1:55 AM marion.mittermaier at metoffice.gov.uk
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719 >
>
> Hi John,
>
> let me try your version and see whether it does what I am expecting
it to
> do.
>
> Maybe I didn't explain it very well, but I want to create a quilt
plot
> using different intensity thresholds for the forecast and
observations.
> This is because there is a pretty big bias between the two and using
the
> same thresholds for both really overwhelms all other signals. This
info is
> useful up to a point. The cheap option, I thought was to get a set
of
> thresholds from a quantile mapping which effectively removes the
bias. The
> expensive option (in terms of storage) would be to pre-process each
> forecast field and then use the same thresholds again.
>
> When I supplied the two different threshold sequences and took away
the
> obs= fcst, it ran but the postscript summary page (1) indicates that
the
> SAME thresholds had been used, despite what I had specified. This is
NOT
> what I had expected.
>
> It was only when I took the sequences away that I got the ps output
to
> show that different thresholds had been used. I had turned
quilt=FALSE. I
> hadn't tested it with quilt=TRUE and specifying only one intensity
> threshold. I will do that, and consider your dictionary definition
below,
> and get back to you.
>
> Regards
> Marion
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 20 June 2019 19:52
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Cc: Maksymczuk, Jan <jan.maksymczuk at metoffice.gov.uk>; North, Rachel
<
> rachel.north at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #90719] Using mode with quilt and
different
> threshold sequences for forecast and observation
>
> Hi Marion,
>
> I see you have a question about the quilt option in MODE. I just
ran a
> test using the met-8.1 release and see that it's behaving as
expected. Let
> me explain what I did to see if that sheds some light on the the
issue.
>
> First, I defined separate convolution radii and threshold for the
fcst and
> obs dictionaries, as shown below:
>
>
>
> *fcst = {... conv_radius = [5, 10]; conv_thresh =
[>=2.5,
> >=1.25 ];*
> *...*
>
>
> *}obs = {...*
>
> * conv_radius = [6, 12]; conv_thresh = [>=3, >=1.5
];*
> *...*
>
> *}*
>
> I ran this example twice, once with QUILT = FALSE and once with
QUILT =
> TRUE.
> Then I cat-ed the "*_obj.txt" output files together and looked at
the
> unique entries in columns 12 - 15:
>
> cat out_QUILT_is_FALSE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}' |
> sort -ru
>
>
> *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=310 >=1.25 12 >=1.5*
>
> cat out_QUILT_is_TRUE/mode*_obj.txt | awk '{print $12, $13, $14,
$15}' |
> sort -ru
>
>
>
>
> *FCST_RAD FCST_THR OBS_RAD OBS_THR5 >=2.5 6 >=35 >=1.25 6 >=1.510
>=2.5 12
> >=310 >=1.25 12 >=1.5*
>
> This is exactly what the output is designed to be. With QUILT =
FALSE, we
> get objects for 2 runs of MODE:
> first radius/first threshold combo
> second radius/second threshold combo
>
> With QUILT = TRUE, we get output for 4 runs of MODE:
> first radius/first threshold combo
> first radius/second threshold combo
> second radius/first threshold combo
> second radius/second threshold combo
>
> And I checked to make sure that the requested forecast radii (5 and
10)
> and observation radii (6 and 12) show up in the right spots. So I
don't
> see any obvious problems.
>
> But I can run the example case you sent to see if anything else pops
up.
>
> Thanks,
> John Halley Gotway
>
>
>
> On Thu, Jun 20, 2019 at 5:37 AM marion.mittermaier at metoffice.gov.uk
via
> RT < met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 20 05:37:18 2019: Request 90719 was acted upon.
> > Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> > Queue: met_help
> > Subject: Using mode with quilt and different threshold
sequences
> > for forecast and observation
> > Owner: Nobody
> > Requestors: marion.mittermaier at metoffice.gov.uk
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90719
> > >
> >
> >
> > Hello,
> >
> > when quilt=TRUE I'm finding that mode is impervious to my taking
away
> > the obs=fcst line in the config file and specifying different
> > threshold sequences for both.
> >
> > It works only if quilt =FALSE, using a single threshold and
> > convolution radius.
> >
> > Any ideas?
> >
> > Sample forecast, ob and config file attached.
> >
> > Regards
> > Marion
> >
> > --
> > Dr Marion Mittermaier Manager: Model diagnostics and novel
> verification
> >
> > Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> > Tel: +44 (0)1392 884830 / +44 (0) 330 135 1604 Fax: +44 (0)1392
> > 885681
> > E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> > marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<
> > http://www.metoffice.gov.uk/>
> >
> > http://www.metoffice.gov.uk/research/people/marion-mittermaier
> >
> > I am co-chair of the WMO WWRP Joint Working Group on Forecast
> > Verification Research (JWGFVR)
> >
http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html
> >
> >
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list