[Met_help] [rt.rap.ucar.edu #67048] History for MET-TC paired tests

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

HI MET team,

I used plot_tcmpr.R to plot TK_ERR vs lead time with confidence 
intervals.  Plus, I plotted relative performance and rank.  Very nice, 
and easy to use! (coming from someone who doesn't even use R. :) )

But can I use MET- tc_stat to get pairwise differences between 2 models, 
such as described in 
http://www.ral.ucar.edu/projects/hfip/includes/h2013/2013-Stream-15-methodology-20May2013-final.pdf, 
so that I can address whether the track error differences are 
statistically significant?

It seems like this was done for the 2013 Stream 1.5 candidate 
evaluation, but I'm not sure if the pairwise differences can be done by 
MET-TC or if they have to be done separately.

dave






----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #67048] MET-TC paired tests
From: John Halley Gotway
Time: Mon May 12 12:09:15 2014

Dave,

Yep, that plotting script can handle pair-wise differences.  There's a
command line option named "-series" to specify what lines are drawn on
the plot.  Suppose your models are named GFSI and OFCL,
and you use "-series AMODEL GFSI,OFCL,GFSI-OFCL".

That'll plot 3 lines on the plot - one for GFSI, one for OFCL, and a
third for their pairwise difference.  If you only want to difference,
you'd use "-series AMODEL GFSL-OFCL".

Give that a shot and let me know if you run into any problems.

Thanks,
John

On 05/11/2014 12:35 PM, David Ahijevych via RT wrote:
>
> Sun May 11 12:35:30 2014: Request 67048 was acted upon.
> Transaction: Ticket created by ahijevyc
>         Queue: met_help
>       Subject: MET-TC paired tests
>         Owner: Nobody
>    Requestors: ahijevyc at ucar.edu
>        Status: new
>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67048 >
>
>
> HI MET team,
>
> I used plot_tcmpr.R to plot TK_ERR vs lead time with confidence
> intervals.  Plus, I plotted relative performance and rank.  Very
nice,
> and easy to use! (coming from someone who doesn't even use R. :) )
>
> But can I use MET- tc_stat to get pairwise differences between 2
models,
> such as described in
> http://www.ral.ucar.edu/projects/hfip/includes/h2013/2013-Stream-15-
methodology-20May2013-final.pdf,
> so that I can address whether the track error differences are
> statistically significant?
>
> It seems like this was done for the 2013 Stream 1.5 candidate
> evaluation, but I'm not sure if the pairwise differences can be done
by
> MET-TC or if they have to be done separately.
>
> dave
>
>
>
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #67048] MET-TC paired tests
From: David Ahijevych
Time: Mon May 12 15:01:06 2014

Thanks . .very powerful this script is.


On 5/12/14 12:09 PM, John Halley Gotway via RT wrote:
> Dave,
>
> Yep, that plotting script can handle pair-wise differences.  There's
a command line option named "-series" to specify what lines are drawn
on the plot.  Suppose your models are named GFSI and OFCL,
> and you use "-series AMODEL GFSI,OFCL,GFSI-OFCL".
>
> That'll plot 3 lines on the plot - one for GFSI, one for OFCL, and a
third for their pairwise difference.  If you only want to difference,
you'd use "-series AMODEL GFSL-OFCL".
>
> Give that a shot and let me know if you run into any problems.
>
> Thanks,
> John
>
> On 05/11/2014 12:35 PM, David Ahijevych via RT wrote:
>> Sun May 11 12:35:30 2014: Request 67048 was acted upon.
>> Transaction: Ticket created by ahijevyc
>>          Queue: met_help
>>        Subject: MET-TC paired tests
>>          Owner: Nobody
>>     Requestors: ahijevyc at ucar.edu
>>         Status: new
>>    Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67048 >
>>
>>
>> HI MET team,
>>
>> I used plot_tcmpr.R to plot TK_ERR vs lead time with confidence
>> intervals.  Plus, I plotted relative performance and rank.  Very
nice,
>> and easy to use! (coming from someone who doesn't even use R. :) )
>>
>> But can I use MET- tc_stat to get pairwise differences between 2
models,
>> such as described in
>> http://www.ral.ucar.edu/projects/hfip/includes/h2013/2013-Stream-
15-methodology-20May2013-final.pdf,
>> so that I can address whether the track error differences are
>> statistically significant?
>>
>> It seems like this was done for the 2013 Stream 1.5 candidate
>> evaluation, but I'm not sure if the pairwise differences can be
done by
>> MET-TC or if they have to be done separately.
>>
>> dave
>>
>>
>>
>>


------------------------------------------------