[Met_help] [rt.rap.ucar.edu #64285] History for question about aggregate_stat
John Halley Gotway via RT
met_help at ucar.edu
Mon Dec 2 15:05:54 MST 2013
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
thanks your detail and useful replay !
I think i have misunderstand the filtering operation.
by the way ask another several question about stat_analysis.
I want to statistics analysis all my interesting cases.
I have got all the scores and errors by point-stat tools.
first question:
I want to aggregate all my cases, the job setting in statanalysis_chongqing_time1 following :
"-job aggregate_stat -line_type CTC -out_line_type CTS -out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
"-job aggregate_stat -line_type SL1L2 -out_line_type CNT -out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"
the run csh script following :
@ numplot = 12
@ hour_forecast = 3
@ icycle = 3
@ lenfore = 24
while ( $icycle <= $lenfore )
setenv ihh $icycle
if ( $ihh < 10 ) then
set ii = 0$ihh
else
set ii = $ihh
endif
./stat_analysis \
-config ${inputdir}/statanalysis_chongqing_time1 \
-lookin ${datadir}/20130428/00Z_27km/metout/3km/time3 ${datadir}/20130513/00Z_27km/metout/3km/time3 \
-fcst_lead ${ii}0000 \
-out ${outdir}/3km_stat_analysis_${ii}_cts.out \
-v 2 \
@ icycle = $icycle + $hour_forecast
end
and theone stat_analysis output results following :
FILTER: -job filter -fcst_lead 060000 -fcst_var APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24 -fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH -obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24 -fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6 -obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10 -vx_mask chongqing -line_type CNT -dump_row ./outdata/27km_12h_job_filter_CNT.stat
JOB_LIST: -job aggregate_stat -fcst_lead 060000 -fcst_var APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24 -fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH -obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24 -fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6 -obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10 -vx_mask chongqing -line_type CTC -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat -out_line_type CTS -out_fcst_thresh >=5.000 -out_obs_thresh >=5.000 -out_alpha 0.05
COL_NAME: TOTAL BASER BASER_NCL BASER_NCU BASER_BCL BASER_BCU FMEAN FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU ACC ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS FBIAS_BCL FBIAS_BCU PODY PODY_NCL PODY_NCU PODY_BCL PODY_BCU PODN PODN_NCL PODN_NCU PODN_BCL PODN_BCU POFD POFD_NCL POFD_NCU POFD_BCL POFD_BCU FAR FAR_NCL FAR_NCU FAR_BCL FAR_BCU CSI CSI_NCL CSI_NCU CSI_BCL CSI_BCU GSS GSS_BCL GSS_BCU HK HK_NCL HK_NCU HK_BCL HK_BCU HSS HSS_BCL HSS_BCU ODDS ODDS_NCL ODDS_NCU ODDS_BCL ODDS_BCU
CTS: 9360 0.00887 0.00716 0.01098 NA NA 0.00278 0.00190 0.00407 NA NA 0.98857 0.98621 0.99053 NA NA 0.31325 NA NA 0.01205 0.01003 0.01447 NA NA 0.99731 0.99603 0.99817 NA NA 0.00269 0.00183 0.00397 NA NA 0.96154 0.95745 0.96525 NA NA 0.00926 0.00751 0.01141 NA NA 0.00714 NA NA 0.00935 -0.02223 0.04094 NA NA 0.01418 NA NA 4.51317 0.60437 33.70233 NA NA
JOB_LIST: -job aggregate_stat -fcst_lead 060000 -fcst_var APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24 -fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH -obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24 -fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6 -obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10 -vx_mask chongqing -line_type SL1L2 -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat -out_line_type CNT -out_fcst_thresh >=5.000 -out_obs_thresh >=5.000 -out_alpha 0.05 -rank_corr_flag 1
COL_NAME: TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL E90_BCU
CNT: 1872 0.01510 0.00624 0.02395 NA NA 0.19550 0.18943 0.20197 NA NA 0.01757 0.01059 0.02456 NA NA 0.15420 0.14942 0.15931 NA NA -0.00698 -0.05227 0.03834 NA NA NA NA 0 0 0 -0.00248 -0.01379 0.00884 NA NA 0.24984 0.24208 0.25811 NA NA 0.85904 NA NA NA NA NA 0.06239 NA NA 0.06238 NA NA 0.24978 NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
but when I add the job command in my run csh:
there some information(Ambiguous output redirect) and I can not get the correct output stat-analysis files.but I add the summary in my run csh and I can get the results. I do not know why.
second question:
the stat-analysis output results include many NA values. all the NA values involve the bootstrapping element. I think there are not enough samples for statistics. so I change the job setting in my statanalysis_chongqing_time1 :
"-job aggregate_stat -line_type MPR -out_line_type CTS -out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
"-job aggregate_stat -line_type MPR -out_line_type CNT -out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"
and then run my csh script, the output information show:
DEBUG 2: Processing Job 1: -job filter -fcst_lead 030000 -fcst_var APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24 -fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH -obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24 -fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6 -obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10 -vx_mask chongqing -line_type CNT -dump_row ./outdata/27km_12h_job_filter_CNT.stat
DEBUG 1: Creating STAT output file "./outdata/27km_12h_job_filter_CNT.stat"
DEBUG 2: Job 1 used 2 out of 1913 STAT lines.
Segmentation fault (core dumped)
I can not get my wanting results with core dumped errors. I do not know what the trouble is ?
Best wishes!
Qiang
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #64285] question about aggregate_stat
From: John Halley Gotway
Time: Wed Nov 13 10:18:41 2013
Qiang,
You listed two stat-analysis jobs:
(1) "-job aggregate_stat -line_type CTC -out_line_type CTS
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
(2) "-job aggregate_stat -line_type SL1L2 -out_line_type CNT
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"
These two jobs don't quite make sense. Instead, I think they should
look like this:
(1) "-job aggregate_stat -line_type CTC -out_line_type CTS
-fcst_thresh >=5.0 -obs_thresh >=5.0 -dump_row
./outdata1/27km_12h_job_aggregate_stat_CTC_CTS.stat",
(2) "-job aggregate_stat -line_type SL1L2 -out_line_type CNT
-dump_row ./outdata1/27km_12h_job_aggregate_stat_SL1L2_CNT.stat"
Notice that I've changed "out_fcst_thresh" and "out_obs_thresh" to
just "fcst_thresh" and "obs_thresh". The first job is just looking
for CTC lines for the >=5.0 threshold, aggregating them together,
and writing output contingency table stats.
For the second job, I removed the "out_fcst_thresh" and
"out_obs_thresh" options. There are no thresholds involved for
continuous statistics, so you don't need to filter by them.
If you have matched pair (MPR) output lines from Point-Stat, you could
also run jobs on them like the ones below. Perhaps that's what you
intended?
(1) "-job aggregate_stat -line_type MPR -out_line_type CTS
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
(2) "-job aggregate_stat -line_type MPR -out_line_type CNT
-dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"
In this case you do use the "out_fcst_thresh" and "out_obs_thresh"
because you're telling STAT-Analysis what thresholds to use for
defining a contingency table and corresponding statistics.
Hopefully I haven't confused you too much!
It also sounds like you're having some trouble putting these calls to
STAT-Analysis in a csh script. The only suspicious thing I see in the
script you sent occurs here:
-out ${outdir}/3km_stat_analysis_${ii}_cts.out \
-v 2 \
@ icycle = $icycle + $hour_forecast
I don't think you need that continuation marker "-v 2 \". Instead
that line should just end there "-v 2". Not sure if that's causing
the problem or not.
Regarding the NA's, yes those are in the bootstrapping column. And
that's fine. Bootstrapping is done over individual matched pair
values. When you run a STAT-Analysis job converting contingency
table counts (CTC) to contingency table statistics (CTS), you're not
operating on matched pairs. Therefore, there are no bootstrap CI's
available in the output. That's why they're all NA for that
type of job. But for MPR -> CTC or MPR -> CNT job types, there should
be output in the bootstrap CI columns.
Lastly, getting a segmentation fault from STAT-Analysis is never a
good thing. If you put together a sample input file and STAT-Analysis
job that illustrates this error, I'd be happy to debug it and
try to figure out what's going on.
Thanks,
John Halley Gotway
On 11/13/2013 09:14 AM, Li, Qiang via RT wrote:
>
> Wed Nov 13 09:14:58 2013: Request 64285 was acted upon.
> Transaction: Ticket created by liqiang at ou.edu
> Queue: met_help
> Subject: question about aggregate_stat
> Owner: Nobody
> Requestors: liqiang at ou.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=64285 >
>
>
> thanks your detail and useful replay !
> I think i have misunderstand the filtering operation.
>
> by the way ask another several question about stat_analysis.
> I want to statistics analysis all my interesting cases.
> I have got all the scores and errors by point-stat tools.
>
>
> first question:
>
> I want to aggregate all my cases, the job setting in
statanalysis_chongqing_time1 following :
> "-job aggregate_stat -line_type CTC -out_line_type CTS
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
> "-job aggregate_stat -line_type SL1L2 -out_line_type CNT
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"
>
> the run csh script following :
>
> @ numplot = 12
> @ hour_forecast = 3
> @ icycle = 3
> @ lenfore = 24
> while ( $icycle <= $lenfore )
> setenv ihh $icycle
> if ( $ihh < 10 ) then
> set ii = 0$ihh
> else
> set ii = $ihh
> endif
> ./stat_analysis \
> -config ${inputdir}/statanalysis_chongqing_time1 \
> -lookin ${datadir}/20130428/00Z_27km/metout/3km/time3
${datadir}/20130513/00Z_27km/metout/3km/time3 \
> -fcst_lead ${ii}0000 \
> -out ${outdir}/3km_stat_analysis_${ii}_cts.out \
> -v 2 \
> @ icycle = $icycle + $hour_forecast
> end
>
> and theone stat_analysis output results following :
>
> FILTER: -job filter -fcst_lead 060000 -fcst_var APCP_03
-fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24 -fcst_var TMP
-fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var APCP_06 -obs_var
APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH -obs_var WIND
-fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24 -fcst_lev Z2
-fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6 -obs_lev A12
-obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10 -vx_mask chongqing
-line_type CNT -dump_row ./outdata/27km_12h_job_filter_CNT.stat
> JOB_LIST: -job aggregate_stat -fcst_lead 060000 -fcst_var
APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24
-fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var
APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH
-obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24
-fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6
-obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10
-vx_mask chongqing -line_type CTC -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat -out_line_type CTS
-out_fcst_thresh >=5.000 -out_obs_thresh >=5.000 -out_alpha 0.05
> COL_NAME: TOTAL BASER BASER_NCL BASER_NCU BASER_BCL BASER_BCU
FMEAN FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU ACC ACC_NCL
ACC_NCU ACC_BCL ACC_BCU FBIAS FBIAS_BCL FBIAS_BCU PODY PODY_NCL
PODY_NCU PODY_BCL PODY_BCU PODN PODN_NCL PODN_NCU PODN_BCL PODN_BCU
POFD POFD_NCL POFD_NCU POFD_BCL POFD_BCU FAR FAR_NCL FAR_NCU
FAR_BCL FAR_BCU CSI CSI_NCL CSI_NCU CSI_BCL CSI_BCU GSS
GSS_BCL GSS_BCU HK HK_NCL HK_NCU HK_BCL HK_BCU HSS HSS_BCL
HSS_BCU ODDS ODDS_NCL ODDS_NCU ODDS_BCL ODDS_BCU
> CTS: 9360 0.00887 0.00716 0.01098 NA NA
0.00278 0.00190 0.00407 NA NA 0.98857 0.98621
0.99053 NA NA 0.31325 NA NA 0.01205 0.01003
0.01447 NA NA 0.99731 0.99603 0.99817 NA NA
0.00269 0.00183 0.00397 NA NA 0.96154 0.95745 0.96525 NA
NA 0.00926 0.00751 0.01141 NA NA 0.00714 NA NA
0.00935 -0.02223 0.04094 NA NA 0.01418 NA NA 4.51317
0.60437 33.70233 NA NA
>
> JOB_LIST: -job aggregate_stat -fcst_lead 060000 -fcst_var
APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24
-fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var
APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH
-obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24
-fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6
-obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10
-vx_mask chongqing -line_type SL1L2 -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat -out_line_type CNT
-out_fcst_thresh >=5.000 -out_obs_thresh >=5.000 -out_alpha 0.05
-rank_corr_flag 1
> COL_NAME: TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV
FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU
OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU
PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR
KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL ME_NCU ME_BCL
ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS
MBIAS_BCL MBIAS_BCU MAE MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE
BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU E25
E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL
E90_BCU
> CNT: 1872 0.01510 0.00624 0.02395 NA NA 0.19550
0.18943 0.20197 NA NA 0.01757 0.01059 0.02456
NA NA 0.15420 0.14942 0.15931 NA NA
-0.00698 -0.05227 0.03834 NA NA NA NA
0 0 0 -0.00248 -0.01379 0.00884 NA NA
0.24984 0.24208 0.25811 NA NA 0.85904 NA
NA NA NA NA 0.06239 NA NA 0.06238 NA
NA 0.24978 NA NA NA NA NA NA NA NA
NA NA NA NA NA NA NA NA NA
>
> but when I add the job command in my run csh:
>
> there some information(Ambiguous output redirect) and I can not
get the correct output stat-analysis files.but I add the summary in my
run csh and I can get the results. I do not know why.
>
>
> second question:
>
> the stat-analysis output results include many NA values. all the
NA values involve the bootstrapping element. I think there are not
enough samples for statistics. so I change the job setting in my
statanalysis_chongqing_time1 :
> "-job aggregate_stat -line_type MPR -out_line_type CTS
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
> "-job aggregate_stat -line_type MPR -out_line_type CNT
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"
> and then run my csh script, the output information show:
> DEBUG 2: Processing Job 1: -job filter -fcst_lead 030000 -fcst_var
APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24
-fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var
APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH
-obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24
-fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6
-obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10
-vx_mask chongqing -line_type CNT -dump_row
./outdata/27km_12h_job_filter_CNT.stat
> DEBUG 1: Creating STAT output file
"./outdata/27km_12h_job_filter_CNT.stat"
> DEBUG 2: Job 1 used 2 out of 1913 STAT lines.
> Segmentation fault (core dumped)
> I can not get my wanting results with core dumped errors. I do not
know what the trouble is ?
>
>
>
> Best wishes!
>
>
> Qiang
>
------------------------------------------------
More information about the Met_help
mailing list