[Met_help] [rt.rap.ucar.edu #64285] History for question about aggregate_stat

John Halley Gotway via RT met_help at ucar.edu
Mon Dec 2 15:05:54 MST 2013


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

thanks your detail and useful replay !
I think i have misunderstand the filtering operation.

by the way ask another several question about stat_analysis.
I want to statistics analysis all my interesting cases.
I have got all the scores and errors by point-stat tools.


first question:

I want to aggregate all my cases, the job setting in statanalysis_chongqing_time1  following :
   "-job aggregate_stat -line_type CTC   -out_line_type CTS  -out_fcst_thresh >=5.0 -out_obs_thresh >=5.0     -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
   "-job aggregate_stat -line_type SL1L2  -out_line_type CNT   -out_fcst_thresh >=5.0 -out_obs_thresh >=5.0    -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"

the run csh script following :

   @ numplot = 12
   @ hour_forecast = 3
   @ icycle = 3
   @ lenfore = 24
   while ( $icycle <= $lenfore )
         setenv ihh $icycle
         if ( $ihh < 10 ) then
            set ii = 0$ihh
         else
            set ii = $ihh
         endif
  ./stat_analysis \
   -config ${inputdir}/statanalysis_chongqing_time1 \
   -lookin ${datadir}/20130428/00Z_27km/metout/3km/time3 ${datadir}/20130513/00Z_27km/metout/3km/time3 \
  -fcst_lead ${ii}0000  \
   -out ${outdir}/3km_stat_analysis_${ii}_cts.out \
   -v 2 \
    @ icycle = $icycle + $hour_forecast
    end

and theone stat_analysis output results following :

FILTER:        -job filter -fcst_lead 060000 -fcst_var APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24 -fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH -obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24 -fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6 -obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10 -vx_mask chongqing -line_type CNT -dump_row ./outdata/27km_12h_job_filter_CNT.stat
JOB_LIST:      -job aggregate_stat -fcst_lead 060000 -fcst_var APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24 -fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH -obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24 -fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6 -obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10 -vx_mask chongqing -line_type CTC -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat -out_line_type CTS -out_fcst_thresh >=5.000 -out_obs_thresh >=5.000 -out_alpha 0.05
COL_NAME: TOTAL BASER   BASER_NCL BASER_NCU BASER_BCL BASER_BCU FMEAN   FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU ACC     ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS   FBIAS_BCL FBIAS_BCU PODY    PODY_NCL PODY_NCU PODY_BCL PODY_BCU PODN    PODN_NCL PODN_NCU PODN_BCL PODN_BCU POFD    POFD_NCL POFD_NCU POFD_BCL POFD_BCU FAR     FAR_NCL FAR_NCU FAR_BCL FAR_BCU CSI     CSI_NCL CSI_NCU CSI_BCL CSI_BCU GSS     GSS_BCL GSS_BCU HK      HK_NCL   HK_NCU  HK_BCL HK_BCU HSS     HSS_BCL HSS_BCU ODDS    ODDS_NCL ODDS_NCU ODDS_BCL ODDS_BCU
     CTS: 9360  0.00887 0.00716   0.01098   NA        NA        0.00278 0.00190   0.00407   NA        NA        0.98857 0.98621 0.99053 NA      NA      0.31325 NA        NA        0.01205 0.01003  0.01447  NA       NA       0.99731 0.99603  0.99817  NA       NA       0.00269 0.00183  0.00397  NA       NA       0.96154 0.95745 0.96525 NA      NA      0.00926 0.00751 0.01141 NA      NA      0.00714 NA      NA      0.00935 -0.02223 0.04094 NA     NA     0.01418 NA      NA      4.51317 0.60437  33.70233 NA       NA

JOB_LIST:      -job aggregate_stat -fcst_lead 060000 -fcst_var APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24 -fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH -obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24 -fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6 -obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10 -vx_mask chongqing -line_type SL1L2 -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat -out_line_type CNT -out_fcst_thresh >=5.000 -out_obs_thresh >=5.000 -out_alpha 0.05 -rank_corr_flag 1
COL_NAME: TOTAL FBAR    FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV  FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR    OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV  OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR  PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME       ME_NCL   ME_NCU  ME_BCL ME_BCU ESTDEV  ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS   MBIAS_BCL MBIAS_BCU MAE MAE_BCL MAE_BCU MSE     MSE_BCL MSE_BCU BCMSE   BCMSE_BCL BCMSE_BCU RMSE    RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL E90_BCU
     CNT: 1872  0.01510 0.00624  0.02395  NA       NA       0.19550 0.18943    0.20197    NA         NA         0.01757 0.01059  0.02456  NA       NA       0.15420 0.14942    0.15931    NA         NA         -0.00698 -0.05227    0.03834     NA          NA          NA      NA      0     0          0          -0.00248 -0.01379 0.00884 NA     NA     0.24984 0.24208    0.25811    NA         NA         0.85904 NA        NA        NA  NA      NA      0.06239 NA      NA      0.06238 NA        NA        0.24978 NA       NA       NA  NA      NA      NA  NA      NA      NA  NA      NA      NA  NA      NA      NA  NA      NA

but when I add the job command in my run csh:

 there some information(Ambiguous output redirect) and I can not get the correct output stat-analysis files.but I add the summary in my run csh and I can get the results. I do not know why.


second question:

the stat-analysis output results  include many NA  values. all the NA values involve the bootstrapping element. I think there are not enough samples for statistics. so I change the job setting in my  statanalysis_chongqing_time1 :
 "-job aggregate_stat -line_type MPR   -out_line_type CTS  -out_fcst_thresh >=5.0 -out_obs_thresh >=5.0     -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
   "-job aggregate_stat -line_type MPR  -out_line_type CNT   -out_fcst_thresh >=5.0 -out_obs_thresh >=5.0    -dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"
and then run my csh script, the output information show:
DEBUG 2: Processing Job 1: -job filter -fcst_lead 030000 -fcst_var APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24 -fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH -obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24 -fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6 -obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10 -vx_mask chongqing -line_type CNT -dump_row ./outdata/27km_12h_job_filter_CNT.stat
DEBUG 1: Creating STAT output file "./outdata/27km_12h_job_filter_CNT.stat"
DEBUG 2: Job 1 used 2 out of 1913 STAT lines.
Segmentation fault (core dumped)
I can not get my wanting results with core dumped errors. I do not know what the trouble is ?



Best wishes!


Qiang


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #64285] question about aggregate_stat
From: John Halley Gotway
Time: Wed Nov 13 10:18:41 2013

Qiang,

You listed two stat-analysis jobs:
(1) "-job aggregate_stat -line_type CTC   -out_line_type CTS
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0     -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
(2) "-job aggregate_stat -line_type SL1L2  -out_line_type CNT
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0    -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"

These two jobs don't quite make sense.  Instead, I think they should
look like this:
(1) "-job aggregate_stat -line_type CTC    -out_line_type CTS
-fcst_thresh >=5.0 -obs_thresh >=5.0 -dump_row
./outdata1/27km_12h_job_aggregate_stat_CTC_CTS.stat",
(2) "-job aggregate_stat -line_type SL1L2  -out_line_type CNT
-dump_row ./outdata1/27km_12h_job_aggregate_stat_SL1L2_CNT.stat"

Notice that I've changed "out_fcst_thresh" and "out_obs_thresh" to
just "fcst_thresh" and "obs_thresh".  The first job is just looking
for CTC lines for the >=5.0 threshold, aggregating them together,
and writing output contingency table stats.

For the second job, I removed the "out_fcst_thresh" and
"out_obs_thresh" options.  There are no thresholds involved for
continuous statistics, so you don't need to filter by them.

If you have matched pair (MPR) output lines from Point-Stat, you could
also run jobs on them like the ones below.  Perhaps that's what you
intended?
(1) "-job aggregate_stat -line_type MPR -out_line_type CTS
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0 -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
(2) "-job aggregate_stat -line_type MPR -out_line_type CNT
-dump_row ./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"

In this case you do use the "out_fcst_thresh" and "out_obs_thresh"
because you're telling STAT-Analysis what thresholds to use for
defining a contingency table and corresponding statistics.

Hopefully I haven't confused you too much!

It also sounds like you're having some trouble putting these calls to
STAT-Analysis in a csh script.  The only suspicious thing I see in the
script you sent occurs here:

     -out ${outdir}/3km_stat_analysis_${ii}_cts.out \
     -v 2 \
      @ icycle = $icycle + $hour_forecast

I don't think you need that continuation marker "-v 2 \".  Instead
that line should just end there "-v 2".  Not sure if that's causing
the problem or not.

Regarding the NA's, yes those are in the bootstrapping column.  And
that's fine.  Bootstrapping is done over individual matched pair
values.  When you run a STAT-Analysis job converting contingency
table counts (CTC) to contingency table statistics (CTS), you're not
operating on matched pairs.  Therefore, there are no bootstrap CI's
available in the output.  That's why they're all NA for that
type of job.  But for MPR -> CTC or MPR -> CNT job types, there should
be output in the bootstrap CI columns.

Lastly, getting a segmentation fault from STAT-Analysis is never a
good thing.  If you put together a sample input file and STAT-Analysis
job that illustrates this error, I'd be happy to debug it and
try to figure out what's going on.

Thanks,
John Halley Gotway


On 11/13/2013 09:14 AM, Li, Qiang via RT wrote:
>
> Wed Nov 13 09:14:58 2013: Request 64285 was acted upon.
> Transaction: Ticket created by liqiang at ou.edu
>         Queue: met_help
>       Subject: question about aggregate_stat
>         Owner: Nobody
>    Requestors: liqiang at ou.edu
>        Status: new
>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=64285 >
>
>
> thanks your detail and useful replay !
> I think i have misunderstand the filtering operation.
>
> by the way ask another several question about stat_analysis.
> I want to statistics analysis all my interesting cases.
> I have got all the scores and errors by point-stat tools.
>
>
> first question:
>
> I want to aggregate all my cases, the job setting in
statanalysis_chongqing_time1  following :
>     "-job aggregate_stat -line_type CTC   -out_line_type CTS
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0     -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
>     "-job aggregate_stat -line_type SL1L2  -out_line_type CNT
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0    -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"
>
> the run csh script following :
>
>     @ numplot = 12
>     @ hour_forecast = 3
>     @ icycle = 3
>     @ lenfore = 24
>     while ( $icycle <= $lenfore )
>           setenv ihh $icycle
>           if ( $ihh < 10 ) then
>              set ii = 0$ihh
>           else
>              set ii = $ihh
>           endif
>    ./stat_analysis \
>     -config ${inputdir}/statanalysis_chongqing_time1 \
>     -lookin ${datadir}/20130428/00Z_27km/metout/3km/time3
${datadir}/20130513/00Z_27km/metout/3km/time3 \
>    -fcst_lead ${ii}0000  \
>     -out ${outdir}/3km_stat_analysis_${ii}_cts.out \
>     -v 2 \
>      @ icycle = $icycle + $hour_forecast
>      end
>
> and theone stat_analysis output results following :
>
> FILTER:        -job filter -fcst_lead 060000 -fcst_var APCP_03
-fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24 -fcst_var TMP
-fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var APCP_06 -obs_var
APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH -obs_var WIND
-fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24 -fcst_lev Z2
-fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6 -obs_lev A12
-obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10 -vx_mask chongqing
-line_type CNT -dump_row ./outdata/27km_12h_job_filter_CNT.stat
> JOB_LIST:      -job aggregate_stat -fcst_lead 060000 -fcst_var
APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24
-fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var
APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH
-obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24
-fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6
-obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10
-vx_mask chongqing -line_type CTC -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat -out_line_type CTS
-out_fcst_thresh >=5.000 -out_obs_thresh >=5.000 -out_alpha 0.05
> COL_NAME: TOTAL BASER   BASER_NCL BASER_NCU BASER_BCL BASER_BCU
FMEAN   FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU ACC     ACC_NCL
ACC_NCU ACC_BCL ACC_BCU FBIAS   FBIAS_BCL FBIAS_BCU PODY    PODY_NCL
PODY_NCU PODY_BCL PODY_BCU PODN    PODN_NCL PODN_NCU PODN_BCL PODN_BCU
POFD    POFD_NCL POFD_NCU POFD_BCL POFD_BCU FAR     FAR_NCL FAR_NCU
FAR_BCL FAR_BCU CSI     CSI_NCL CSI_NCU CSI_BCL CSI_BCU GSS
GSS_BCL GSS_BCU HK      HK_NCL   HK_NCU  HK_BCL HK_BCU HSS     HSS_BCL
HSS_BCU ODDS    ODDS_NCL ODDS_NCU ODDS_BCL ODDS_BCU
>       CTS: 9360  0.00887 0.00716   0.01098   NA        NA
0.00278 0.00190   0.00407   NA        NA        0.98857 0.98621
0.99053 NA      NA      0.31325 NA        NA        0.01205 0.01003
0.01447  NA       NA       0.99731 0.99603  0.99817  NA       NA
0.00269 0.00183  0.00397  NA       NA       0.96154 0.95745 0.96525 NA
NA      0.00926 0.00751 0.01141 NA      NA      0.00714 NA      NA
0.00935 -0.02223 0.04094 NA     NA     0.01418 NA      NA      4.51317
0.60437  33.70233 NA       NA
>
> JOB_LIST:      -job aggregate_stat -fcst_lead 060000 -fcst_var
APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24
-fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var
APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH
-obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24
-fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6
-obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10
-vx_mask chongqing -line_type SL1L2 -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat -out_line_type CNT
-out_fcst_thresh >=5.000 -out_obs_thresh >=5.000 -out_alpha 0.05
-rank_corr_flag 1
> COL_NAME: TOTAL FBAR    FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV
FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR    OBAR_NCL OBAR_NCU
OBAR_BCL OBAR_BCU OSTDEV  OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU
PR_CORR  PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR
KT_CORR RANKS FRANK_TIES ORANK_TIES ME       ME_NCL   ME_NCU  ME_BCL
ME_BCU ESTDEV  ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS
MBIAS_BCL MBIAS_BCU MAE MAE_BCL MAE_BCU MSE     MSE_BCL MSE_BCU BCMSE
BCMSE_BCL BCMSE_BCU RMSE    RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU E25
E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL
E90_BCU
>       CNT: 1872  0.01510 0.00624  0.02395  NA       NA       0.19550
0.18943    0.20197    NA         NA         0.01757 0.01059  0.02456
NA       NA       0.15420 0.14942    0.15931    NA         NA
-0.00698 -0.05227    0.03834     NA          NA          NA      NA
0     0          0          -0.00248 -0.01379 0.00884 NA     NA
0.24984 0.24208    0.25811    NA         NA         0.85904 NA
NA        NA  NA      NA      0.06239 NA      NA      0.06238 NA
NA        0.24978 NA       NA       NA  NA      NA      NA  NA      NA
NA  NA      NA      NA  NA      NA      NA  NA      NA
>
> but when I add the job command in my run csh:
>
>   there some information(Ambiguous output redirect) and I can not
get the correct output stat-analysis files.but I add the summary in my
run csh and I can get the results. I do not know why.
>
>
> second question:
>
> the stat-analysis output results  include many NA  values. all the
NA values involve the bootstrapping element. I think there are not
enough samples for statistics. so I change the job setting in my
statanalysis_chongqing_time1 :
>   "-job aggregate_stat -line_type MPR   -out_line_type CTS
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0     -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CTS.stat",
>     "-job aggregate_stat -line_type MPR  -out_line_type CNT
-out_fcst_thresh >=5.0 -out_obs_thresh >=5.0    -dump_row
./outdata1/27km_12h_job_aggregate_stat_MPR_CNT.stat"
> and then run my csh script, the output information show:
> DEBUG 2: Processing Job 1: -job filter -fcst_lead 030000 -fcst_var
APCP_03 -fcst_var APCP_06 -fcst_var APCP_12 -fcst_var APCP_24
-fcst_var TMP -fcst_var RH -fcst_var WIND -obs_var APCP_03 -obs_var
APCP_06 -obs_var APCP_12 -obs_var APCP_24 -obs_var TMP -obs_var RH
-obs_var WIND -fcst_lev A3 -fcst_lev A6 -fcst_lev A12 -fcst_lev A24
-fcst_lev Z2 -fcst_lev Z2 -fcst_lev Z10 -obs_lev A3 -obs_lev A6
-obs_lev A12 -obs_lev A24 -obs_lev Z2 -obs_lev Z2 -obs_lev Z10
-vx_mask chongqing -line_type CNT -dump_row
./outdata/27km_12h_job_filter_CNT.stat
> DEBUG 1: Creating STAT output file
"./outdata/27km_12h_job_filter_CNT.stat"
> DEBUG 2: Job 1 used 2 out of 1913 STAT lines.
> Segmentation fault (core dumped)
> I can not get my wanting results with core dumped errors. I do not
know what the trouble is ?
>
>
>
> Best wishes!
>
>
> Qiang
>

------------------------------------------------


More information about the Met_help mailing list