[Met_help] [rt.rap.ucar.edu #64593] History for WDIR issue in Stat Analysis
John Halley Gotway via RT
met_help at ucar.edu
Mon Dec 16 13:38:38 MST 2013
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good afternoon,
I am attempting to verify wind direction from point stat output, using Stat Analysis. When I attempt to run Stat Analysis on the point stat files, it will tell me that there are no files in which it can work with (please see attached file). However, when I grep these files it shows data that meet my criteria, so I'm not sure what to make of this. I downloaded the latest patch file for METv4.1 earlier today, and it did not appear to help this issue. I have ftp'd 6 hours of point stat data as well as my config and executable scripts for your review. I appreciate any help you may be able to offer.
Thanks,
Elliot Tardif, Meteorologist II
NC DENR, Division of Air Quality
Planning Section, Attainment Planning Branch
1641 Mail Service Center
Raleigh, NC 27699-1641
Phone/Fax: 919-707-8483
Email: Elliot.Tardif at ncdenr.gov<mailto:Nick.Witcraft at ncdenr.gov>
Web : http://www.ncair.org<http://www.ncair.org/>
Email correspondence to and from this address is subject to the North Carolina Public Records Law and may be disclosed to third parties unless the content is exempt by statue or other regulation.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: WDIR issue in Stat Analysis
From: John Halley Gotway
Time: Wed Dec 11 16:31:53 2013
Elliot,
I ran the following command using the data you sent me:
METv4.1/bin/stat_analysis -config STATAnalysisConfig_emt -lookin
point_stat*.stat -out wdir.out
It appears to have run successfully and I've attached the output
"wdir.out" file.
Let me suggest that you simplify your job a bit using the "-by" case
option. Take a look at the following single job below that I ran on
the command line. I replaced the "-vx_mask" setting with "-by
vx_mask". So that runs the same job but for each unique value it
finds in the "vx_mask" column.
[johnhg at rambler]% METv4.1/bin/stat_analysis -job aggregate_stat
-line_type VL1L2 -out_line_type WDIR -fcst_lev Z10 -by vx_mask
-lookin *.stat
DEBUG 2: STAT Lines read = 40403
DEBUG 2: STAT Lines retained = 71
DEBUG 2:
DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_lev Z10
-line_type VL1L2 -by VX_MASK -out_line_type WDIR
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073121605714
DEBUG 2: Computing output for 11 case(s).
JOB_LIST: -job aggregate_stat -fcst_lev Z10 -line_type VL1L2 -by
VX_MASK -out_line_type WDIR
COL_NAME: VX_MASK TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: AL 6 61.85231 41.26332 20.55289 20.55289
AGGR_WDIR: AL 251 59.95983 41.34929 18.61054 NA
ROW_MEAN_WDIR: FL 6 235.03649 245.51103 -14.10477 30.83638
AGGR_WDIR: FL 761 239.55794 268.92570 -29.36776 NA
ROW_MEAN_WDIR: GA 6 29.30936 29.95539 -0.62353 4.48081
AGGR_WDIR: GA 324 29.98097 30.88385 -0.90288 NA
ROW_MEAN_WDIR: KY 6 83.68932 77.76918 5.93250 6.19367
AGGR_WDIR: KY 143 82.75768 75.90090 6.85677 NA
ROW_MEAN_WDIR: MS 6 21.17283 359.51207 19.46255 22.25670
AGGR_WDIR: MS 149 29.94464 357.81545 32.12920 NA
ROW_MEAN_WDIR: NC 6 46.34654 37.29122 9.05385 9.05385
AGGR_WDIR: NC 533 45.95594 37.21045 8.74549 NA
ROW_MEAN_WDIR: SC 6 28.29254 29.39961 -1.09695 2.05188
AGGR_WDIR: SC 259 28.41739 30.38964 -1.97225 NA
ROW_MEAN_WDIR: SEMAP 11 39.68436 34.02263 5.61859 5.61859
AGGR_WDIR: SEMAP 4279 40.32304 33.71078 6.61226 NA
ROW_MEAN_WDIR: TN 6 99.65929 89.42600 10.98451 10.98451
AGGR_WDIR: TN 148 91.38965 78.66731 12.72235 NA
ROW_MEAN_WDIR: VA 6 22.16236 17.73469 4.41535 4.41535
AGGR_WDIR: VA 461 22.41267 18.09502 4.31765 NA
ROW_MEAN_WDIR: WV 6 66.45464 61.35398 5.09928 5.60504
AGGR_WDIR: WV 120 67.10595 60.86002 6.24593 NA
DEBUG 2: Job 1 used 71 out of 71 STAT lines.
Now, why isn't this working on your machine? I really don't know.
Try turning up the verbosity level "-v 4" to see if STAT-Analysis give
you any more useful debugging information.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 12/11/2013 04:08 PM, Tardif, Elliot M via RT wrote:
>
> Wed Dec 11 16:07:59 2013: Request 64593 was acted upon.
> Transaction: Ticket created by elliot.tardif at ncdenr.gov
> Queue: met_help
> Subject: WDIR issue in Stat Analysis
> Owner: Nobody
> Requestors: elliot.tardif at ncdenr.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=64593 >
>
>
> Good afternoon,
>
> I am attempting to verify wind direction from point stat output,
using Stat Analysis. When I attempt to run Stat Analysis on the point
stat files, it will tell me that there are no files in which it can
work with (please see attached file). However, when I grep these
files it shows data that meet my criteria, so I'm not sure what to
make of this. I downloaded the latest patch file for METv4.1 earlier
today, and it did not appear to help this issue. I have ftp'd 6 hours
of point stat data as well as my config and executable scripts for
your review. I appreciate any help you may be able to offer.
>
> Thanks,
>
> Elliot Tardif, Meteorologist II
> NC DENR, Division of Air Quality
> Planning Section, Attainment Planning Branch
> 1641 Mail Service Center
> Raleigh, NC 27699-1641
> Phone/Fax: 919-707-8483
> Email: Elliot.Tardif at ncdenr.gov<mailto:Nick.Witcraft at ncdenr.gov>
> Web : http://www.ncair.org<http://www.ncair.org/>
>
> Email correspondence to and from this address is subject to the
North Carolina Public Records Law and may be disclosed to third
parties unless the content is exempt by statue or other regulation.
>
>
------------------------------------------------
Subject: WDIR issue in Stat Analysis
From: John Halley Gotway
Time: Wed Dec 11 16:31:53 2013
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask WV -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 70.36502 64.31167 6.06664 6.67356
AGGR_WDIR: 100 71.22780 64.36620 6.86160 NA
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask VA -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 22.35926 18.19993 4.14684 4.14684
AGGR_WDIR: 383 22.60838 18.63976 3.96862 NA
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask TN -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 107.74097 97.12025 11.09799 11.09799
AGGR_WDIR: 125 99.78104 85.34916 14.43188 NA
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask SC -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 26.65534 27.43737 -0.77243 1.91836
AGGR_WDIR: 217 26.72582 28.16280 -1.43698 NA
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask NC -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 45.68232 37.63574 8.04593 8.04593
AGGR_WDIR: 445 45.31386 37.56888 7.74498 NA
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask MS -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 17.15841 356.00292 18.93557 22.28855
AGGR_WDIR: 125 27.33562 352.86966 34.46596 NA
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask KY -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 85.10529 80.04670 5.07533 5.38873
AGGR_WDIR: 120 84.15017 78.55280 5.59737 NA
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask GA -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 28.18567 27.95231 0.24696 4.38178
AGGR_WDIR: 270 28.77845 28.78335 -0.00489 NA
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask FL -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 242.25750 264.55303 -21.13505 32.79432
AGGR_WDIR: 638 248.56904 292.97528 -44.40624 NA
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask AL -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 64.52465 41.94075 22.54915 22.54915
AGGR_WDIR: 214 62.69839 42.14491 20.55349 NA
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20110111_000000
-fcst_valid_end 20110111_230000 -obs_valid_beg 20110111_000000
-obs_valid_end 20110111_230000 -fcst_lev Z10 -vx_mask SEMAP -line_type
VL1L2 -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: 5 38.07599 32.49913 5.54401 5.54401
AGGR_WDIR: 1950 38.70878 32.03130 6.67749 NA
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #64593] WDIR issue in Stat Analysis
From: Tardif, Elliot M
Time: Mon Dec 16 12:26:03 2013
Good afternoon John,
Thank you for your reply. I utilized the "-by vx_mask" function in my
job list, and it appears to be helping because Stat Analysis is
generating WDIR data for me. However, I would like to get MAE values
for the AGGR_WDIR if possible. I've noticed in both your output and
mine that Stat Analysis is not generating this value for AGGR_WDIR
data. Additionally, when I run this for a monthly time period, I've
noticed that it won't even compute MAE for some ROW_MEAN_WDIR rows as
well (see output text below).
JOB_LIST: -job aggregate_stat -fcst_valid_beg 20111001_000000
-fcst_valid_end 20111031_230000 -obs_valid_beg 20111001_000000
-obs_valid_end 20111031_230000 -fcst_lev Z10 -vx_mask SEMAP -vx_mask
AL -vx_mask FL -vx_mask GA -vx_mask KY -vx_mask MS -vx_mask NC
-vx_mask SC -vx_mask TN -vx_mask VA -vx_mask WV -line_type VL1L2 -by
VX_MASK -dump_row ./aggregate_test.stat -out_line_type WDIR
COL_NAME: VX_MASK TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: AL 742 21.31569 8.31192 4.55038 16.51692
AGGR_WDIR: AL 29543 13.82994 3.77645 10.05349 NA
ROW_MEAN_WDIR: FL 742 32.49834 41.39321 -0.74336 12.64940
AGGR_WDIR: FL 82034 36.17281 42.39865 -6.22584 NA
ROW_MEAN_WDIR: GA 742 358.38956 355.42640 4.64258 11.93934
AGGR_WDIR: GA 41311 4.96111 5.84808 -0.88697 NA
ROW_MEAN_WDIR: KY 742 229.88268 226.50603 NA NA
AGGR_WDIR: KY 18636 258.00278 264.43371 -6.43093 NA
ROW_MEAN_WDIR: MS 742 51.94934 27.51373 3.66602 20.25508
AGGR_WDIR: MS 16708 35.56405 18.36394 17.20011 NA
ROW_MEAN_WDIR: NC 742 338.88680 333.54157 6.49033 9.67657
AGGR_WDIR: NC 60994 329.65060 318.05075 11.59985 NA
ROW_MEAN_WDIR: SC 742 353.55464 346.94238 7.69071 11.78999
AGGR_WDIR: SC 44090 357.74240 353.10774 4.63466 NA
ROW_MEAN_WDIR: SEMAP 742 12.42234 18.36407 1.08589 9.04197
AGGR_WDIR: SEMAP 279493 11.42224 15.85470 -4.43246 NA
ROW_MEAN_WDIR: TN 742 212.46821 209.62719 NA NA
AGGR_WDIR: TN 15666 279.77197 279.09069 0.68128 NA
ROW_MEAN_WDIR: VA 742 316.20936 302.85360 -1.69838 16.91692
AGGR_WDIR: VA 57346 294.82647 274.04371 20.78276 NA
ROW_MEAN_WDIR: WV 742 222.88275 238.23608 NA NA
AGGR_WDIR: WV 16886 254.47068 254.63629 -0.16561 NA
Can either of these issues be fixed?
Thanks,
Elliot Tardif, Meteorologist II
NC DENR, Division of Air Quality
Planning Section, Attainment Planning Branch
1641 Mail Service Center
Raleigh, NC 27699-1641
Phone/Fax: 919-707-8483
Email: Elliot.Tardif at ncdenr.gov
Web : http://www.ncair.org
Email correspondence to and from this address is subject to the North
Carolina Public Records Law and may be disclosed to third parties
unless the content is exempt by statue or other regulation.
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, December 11, 2013 6:32 PM
To: Tardif, Elliot M
Subject: Re: [rt.rap.ucar.edu #64593] WDIR issue in Stat Analysis
Elliot,
I ran the following command using the data you sent me:
METv4.1/bin/stat_analysis -config STATAnalysisConfig_emt -lookin
point_stat*.stat -out wdir.out
It appears to have run successfully and I've attached the output
"wdir.out" file.
Let me suggest that you simplify your job a bit using the "-by" case
option. Take a look at the following single job below that I ran on
the command line. I replaced the "-vx_mask" setting with "-by
vx_mask". So that runs the same job but for each unique value it
finds in the "vx_mask" column.
[johnhg at rambler]% METv4.1/bin/stat_analysis -job aggregate_stat
-line_type VL1L2 -out_line_type WDIR -fcst_lev Z10 -by vx_mask
-lookin *.stat
DEBUG 2: STAT Lines read = 40403
DEBUG 2: STAT Lines retained = 71
DEBUG 2:
DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_lev Z10
-line_type VL1L2 -by VX_MASK -out_line_type WDIR
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073121605714
DEBUG 2: Computing output for 11 case(s).
JOB_LIST: -job aggregate_stat -fcst_lev Z10 -line_type VL1L2 -by
VX_MASK -out_line_type WDIR
COL_NAME: VX_MASK TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: AL 6 61.85231 41.26332 20.55289 20.55289
AGGR_WDIR: AL 251 59.95983 41.34929 18.61054 NA
ROW_MEAN_WDIR: FL 6 235.03649 245.51103 -14.10477 30.83638
AGGR_WDIR: FL 761 239.55794 268.92570 -29.36776 NA
ROW_MEAN_WDIR: GA 6 29.30936 29.95539 -0.62353 4.48081
AGGR_WDIR: GA 324 29.98097 30.88385 -0.90288 NA
ROW_MEAN_WDIR: KY 6 83.68932 77.76918 5.93250 6.19367
AGGR_WDIR: KY 143 82.75768 75.90090 6.85677 NA
ROW_MEAN_WDIR: MS 6 21.17283 359.51207 19.46255 22.25670
AGGR_WDIR: MS 149 29.94464 357.81545 32.12920 NA
ROW_MEAN_WDIR: NC 6 46.34654 37.29122 9.05385 9.05385
AGGR_WDIR: NC 533 45.95594 37.21045 8.74549 NA
ROW_MEAN_WDIR: SC 6 28.29254 29.39961 -1.09695 2.05188
AGGR_WDIR: SC 259 28.41739 30.38964 -1.97225 NA
ROW_MEAN_WDIR: SEMAP 11 39.68436 34.02263 5.61859 5.61859
AGGR_WDIR: SEMAP 4279 40.32304 33.71078 6.61226 NA
ROW_MEAN_WDIR: TN 6 99.65929 89.42600 10.98451 10.98451
AGGR_WDIR: TN 148 91.38965 78.66731 12.72235 NA
ROW_MEAN_WDIR: VA 6 22.16236 17.73469 4.41535 4.41535
AGGR_WDIR: VA 461 22.41267 18.09502 4.31765 NA
ROW_MEAN_WDIR: WV 6 66.45464 61.35398 5.09928 5.60504
AGGR_WDIR: WV 120 67.10595 60.86002 6.24593 NA
DEBUG 2: Job 1 used 71 out of 71 STAT lines.
Now, why isn't this working on your machine? I really don't know.
Try turning up the verbosity level "-v 4" to see if STAT-Analysis give
you any more useful debugging information.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 12/11/2013 04:08 PM, Tardif, Elliot M via RT wrote:
>
> Wed Dec 11 16:07:59 2013: Request 64593 was acted upon.
> Transaction: Ticket created by elliot.tardif at ncdenr.gov
> Queue: met_help
> Subject: WDIR issue in Stat Analysis
> Owner: Nobody
> Requestors: elliot.tardif at ncdenr.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=64593
> >
>
>
> Good afternoon,
>
> I am attempting to verify wind direction from point stat output,
using Stat Analysis. When I attempt to run Stat Analysis on the point
stat files, it will tell me that there are no files in which it can
work with (please see attached file). However, when I grep these
files it shows data that meet my criteria, so I'm not sure what to
make of this. I downloaded the latest patch file for METv4.1 earlier
today, and it did not appear to help this issue. I have ftp'd 6 hours
of point stat data as well as my config and executable scripts for
your review. I appreciate any help you may be able to offer.
>
> Thanks,
>
> Elliot Tardif, Meteorologist II
> NC DENR, Division of Air Quality
> Planning Section, Attainment Planning Branch
> 1641 Mail Service Center
> Raleigh, NC 27699-1641
> Phone/Fax: 919-707-8483
> Email: Elliot.Tardif at ncdenr.gov<mailto:Nick.Witcraft at ncdenr.gov>
> Web : http://www.ncair.org<http://www.ncair.org/>
>
> Email correspondence to and from this address is subject to the
North Carolina Public Records Law and may be disclosed to third
parties unless the content is exempt by statue or other regulation.
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #64593] WDIR issue in Stat Analysis
From: John Halley Gotway
Time: Mon Dec 16 12:43:31 2013
Elliot,
MAE is not mathematically available for the "AGGR_WDIR" method of
computing errors. The WDIR stat-analysis job computes wind direction
errors in two different ways. Let's look at the first 3 lines
of the output you sent:
COL_NAME: VX_MASK TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: AL 742 21.31569 8.31192 4.55038 16.51692
AGGR_WDIR: AL 29543 13.82994 3.77645 10.05349 NA
Here, we see that the TOTAL in the "ROW_MEAN_WDIR" is 742. This means
that stat-analysis found and processed 742 VL1L2 lines for this case.
Here's how the logic works for the two different output
lines...
- For "ROW_MEAN_WDIR"...
(1) For each of the 742 VL1L2 lines, compute the forecast and
observation wind directions.
(2) Compute the wind direction error (fcst - obs) and also the
absolute value of the error.
(3) Compute the means of the 742 errors values and absolute error
values and write them out in the ME and MAE columns.
- For "AGGR_WDIR"...
(1) Aggregate together the partial sums found in those 742 VL1L2 to
create a single VL1L2 line representing all of the wind vectors.
(2) Compute forecast and observation wind directions for that single
VL1L2 line.
(3) Compute the wind direction error (fcst - obs) and write it out
in the ME column.
I suppose we could have also written out a single MAE value that's
just the abs(ME) for the row, but it seemed misleading.
Hopefully that helps clarify what the tool is doing.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 12/16/2013 12:26 PM, Tardif, Elliot M via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=64593 >
>
> Good afternoon John,
>
> Thank you for your reply. I utilized the "-by vx_mask" function in
my job list, and it appears to be helping because Stat Analysis is
generating WDIR data for me. However, I would like to get MAE values
for the AGGR_WDIR if possible. I've noticed in both your output and
mine that Stat Analysis is not generating this value for AGGR_WDIR
data. Additionally, when I run this for a monthly time period, I've
noticed that it won't even compute MAE for some ROW_MEAN_WDIR rows as
well (see output text below).
>
> JOB_LIST: -job aggregate_stat -fcst_valid_beg 20111001_000000
-fcst_valid_end 20111031_230000 -obs_valid_beg 20111001_000000
-obs_valid_end 20111031_230000 -fcst_lev Z10 -vx_mask SEMAP -vx_mask
AL -vx_mask FL -vx_mask GA -vx_mask KY -vx_mask MS -vx_mask NC
-vx_mask SC -vx_mask TN -vx_mask VA -vx_mask WV -line_type VL1L2 -by
VX_MASK -dump_row ./aggregate_test.stat -out_line_type WDIR
> COL_NAME: VX_MASK TOTAL FBAR OBAR ME MAE
> ROW_MEAN_WDIR: AL 742 21.31569 8.31192 4.55038 16.51692
> AGGR_WDIR: AL 29543 13.82994 3.77645 10.05349 NA
> ROW_MEAN_WDIR: FL 742 32.49834 41.39321 -0.74336 12.64940
> AGGR_WDIR: FL 82034 36.17281 42.39865 -6.22584 NA
> ROW_MEAN_WDIR: GA 742 358.38956 355.42640 4.64258 11.93934
> AGGR_WDIR: GA 41311 4.96111 5.84808 -0.88697 NA
> ROW_MEAN_WDIR: KY 742 229.88268 226.50603 NA NA
> AGGR_WDIR: KY 18636 258.00278 264.43371 -6.43093 NA
> ROW_MEAN_WDIR: MS 742 51.94934 27.51373 3.66602 20.25508
> AGGR_WDIR: MS 16708 35.56405 18.36394 17.20011 NA
> ROW_MEAN_WDIR: NC 742 338.88680 333.54157 6.49033 9.67657
> AGGR_WDIR: NC 60994 329.65060 318.05075 11.59985 NA
> ROW_MEAN_WDIR: SC 742 353.55464 346.94238 7.69071 11.78999
> AGGR_WDIR: SC 44090 357.74240 353.10774 4.63466 NA
> ROW_MEAN_WDIR: SEMAP 742 12.42234 18.36407 1.08589 9.04197
> AGGR_WDIR: SEMAP 279493 11.42224 15.85470 -4.43246 NA
> ROW_MEAN_WDIR: TN 742 212.46821 209.62719 NA NA
> AGGR_WDIR: TN 15666 279.77197 279.09069 0.68128 NA
> ROW_MEAN_WDIR: VA 742 316.20936 302.85360 -1.69838 16.91692
> AGGR_WDIR: VA 57346 294.82647 274.04371 20.78276 NA
> ROW_MEAN_WDIR: WV 742 222.88275 238.23608 NA NA
> AGGR_WDIR: WV 16886 254.47068 254.63629 -0.16561 NA
>
>
> Can either of these issues be fixed?
>
> Thanks,
>
> Elliot Tardif, Meteorologist II
> NC DENR, Division of Air Quality
> Planning Section, Attainment Planning Branch
> 1641 Mail Service Center
> Raleigh, NC 27699-1641
> Phone/Fax: 919-707-8483
> Email: Elliot.Tardif at ncdenr.gov
> Web : http://www.ncair.org
>
> Email correspondence to and from this address is subject to the
North Carolina Public Records Law and may be disclosed to third
parties unless the content is exempt by statue or other regulation.
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, December 11, 2013 6:32 PM
> To: Tardif, Elliot M
> Subject: Re: [rt.rap.ucar.edu #64593] WDIR issue in Stat Analysis
>
> Elliot,
>
> I ran the following command using the data you sent me:
> METv4.1/bin/stat_analysis -config STATAnalysisConfig_emt
-lookin point_stat*.stat -out wdir.out
>
> It appears to have run successfully and I've attached the output
"wdir.out" file.
>
> Let me suggest that you simplify your job a bit using the "-by" case
option. Take a look at the following single job below that I ran on
the command line. I replaced the "-vx_mask" setting with "-by
vx_mask". So that runs the same job but for each unique value it
finds in the "vx_mask" column.
>
> [johnhg at rambler]% METv4.1/bin/stat_analysis -job aggregate_stat
-line_type VL1L2 -out_line_type WDIR -fcst_lev Z10 -by vx_mask
-lookin *.stat
> DEBUG 2: STAT Lines read = 40403
> DEBUG 2: STAT Lines retained = 71
> DEBUG 2:
> DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_lev Z10
-line_type VL1L2 -by VX_MASK -out_line_type WDIR
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073121605714
> DEBUG 2: Computing output for 11 case(s).
> JOB_LIST: -job aggregate_stat -fcst_lev Z10 -line_type VL1L2
-by VX_MASK -out_line_type WDIR
> COL_NAME: VX_MASK TOTAL FBAR OBAR ME MAE
> ROW_MEAN_WDIR: AL 6 61.85231 41.26332 20.55289 20.55289
> AGGR_WDIR: AL 251 59.95983 41.34929 18.61054 NA
> ROW_MEAN_WDIR: FL 6 235.03649 245.51103 -14.10477 30.83638
> AGGR_WDIR: FL 761 239.55794 268.92570 -29.36776 NA
> ROW_MEAN_WDIR: GA 6 29.30936 29.95539 -0.62353 4.48081
> AGGR_WDIR: GA 324 29.98097 30.88385 -0.90288 NA
> ROW_MEAN_WDIR: KY 6 83.68932 77.76918 5.93250 6.19367
> AGGR_WDIR: KY 143 82.75768 75.90090 6.85677 NA
> ROW_MEAN_WDIR: MS 6 21.17283 359.51207 19.46255 22.25670
> AGGR_WDIR: MS 149 29.94464 357.81545 32.12920 NA
> ROW_MEAN_WDIR: NC 6 46.34654 37.29122 9.05385 9.05385
> AGGR_WDIR: NC 533 45.95594 37.21045 8.74549 NA
> ROW_MEAN_WDIR: SC 6 28.29254 29.39961 -1.09695 2.05188
> AGGR_WDIR: SC 259 28.41739 30.38964 -1.97225 NA
> ROW_MEAN_WDIR: SEMAP 11 39.68436 34.02263 5.61859 5.61859
> AGGR_WDIR: SEMAP 4279 40.32304 33.71078 6.61226 NA
> ROW_MEAN_WDIR: TN 6 99.65929 89.42600 10.98451 10.98451
> AGGR_WDIR: TN 148 91.38965 78.66731 12.72235 NA
> ROW_MEAN_WDIR: VA 6 22.16236 17.73469 4.41535 4.41535
> AGGR_WDIR: VA 461 22.41267 18.09502 4.31765 NA
> ROW_MEAN_WDIR: WV 6 66.45464 61.35398 5.09928 5.60504
> AGGR_WDIR: WV 120 67.10595 60.86002 6.24593 NA
>
> DEBUG 2: Job 1 used 71 out of 71 STAT lines.
>
> Now, why isn't this working on your machine? I really don't know.
Try turning up the verbosity level "-v 4" to see if STAT-Analysis give
you any more useful debugging information.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
>
> On 12/11/2013 04:08 PM, Tardif, Elliot M via RT wrote:
>>
>> Wed Dec 11 16:07:59 2013: Request 64593 was acted upon.
>> Transaction: Ticket created by elliot.tardif at ncdenr.gov
>> Queue: met_help
>> Subject: WDIR issue in Stat Analysis
>> Owner: Nobody
>> Requestors: elliot.tardif at ncdenr.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=64593
>>>
>>
>>
>> Good afternoon,
>>
>> I am attempting to verify wind direction from point stat output,
using Stat Analysis. When I attempt to run Stat Analysis on the point
stat files, it will tell me that there are no files in which it can
work with (please see attached file). However, when I grep these
files it shows data that meet my criteria, so I'm not sure what to
make of this. I downloaded the latest patch file for METv4.1 earlier
today, and it did not appear to help this issue. I have ftp'd 6 hours
of point stat data as well as my config and executable scripts for
your review. I appreciate any help you may be able to offer.
>>
>> Thanks,
>>
>> Elliot Tardif, Meteorologist II
>> NC DENR, Division of Air Quality
>> Planning Section, Attainment Planning Branch
>> 1641 Mail Service Center
>> Raleigh, NC 27699-1641
>> Phone/Fax: 919-707-8483
>> Email: Elliot.Tardif at ncdenr.gov<mailto:Nick.Witcraft at ncdenr.gov>
>> Web : http://www.ncair.org<http://www.ncair.org/>
>>
>> Email correspondence to and from this address is subject to the
North Carolina Public Records Law and may be disclosed to third
parties unless the content is exempt by statue or other regulation.
>>
>>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #64593] WDIR issue in Stat Analysis
From: Tardif, Elliot M
Time: Mon Dec 16 13:26:30 2013
Good afternoon John, I appreciate your clarification on the MAE value
with respect to the WDIR process, as well as what "AGGR_WDIR" and
"ROW_MEAN_WDIR" represent. Thanks very much for your time.
Best Regards,
Elliot Tardif, Meteorologist II
NC DENR, Division of Air Quality
Planning Section, Attainment Planning Branch
1641 Mail Service Center
Raleigh, NC 27699-1641
Phone/Fax: 919-707-8483
Email: Elliot.Tardif at ncdenr.gov
Web : http://www.ncair.org
Email correspondence to and from this address is subject to the North
Carolina Public Records Law and may be disclosed to third parties
unless the content is exempt by statue or other regulation.
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, December 16, 2013 2:44 PM
To: Tardif, Elliot M
Subject: Re: [rt.rap.ucar.edu #64593] WDIR issue in Stat Analysis
Elliot,
MAE is not mathematically available for the "AGGR_WDIR" method of
computing errors. The WDIR stat-analysis job computes wind direction
errors in two different ways. Let's look at the first 3 lines of the
output you sent:
COL_NAME: VX_MASK TOTAL FBAR OBAR ME MAE
ROW_MEAN_WDIR: AL 742 21.31569 8.31192 4.55038 16.51692
AGGR_WDIR: AL 29543 13.82994 3.77645 10.05349 NA
Here, we see that the TOTAL in the "ROW_MEAN_WDIR" is 742. This means
that stat-analysis found and processed 742 VL1L2 lines for this case.
Here's how the logic works for the two different output lines...
- For "ROW_MEAN_WDIR"...
(1) For each of the 742 VL1L2 lines, compute the forecast and
observation wind directions.
(2) Compute the wind direction error (fcst - obs) and also the
absolute value of the error.
(3) Compute the means of the 742 errors values and absolute error
values and write them out in the ME and MAE columns.
- For "AGGR_WDIR"...
(1) Aggregate together the partial sums found in those 742 VL1L2 to
create a single VL1L2 line representing all of the wind vectors.
(2) Compute forecast and observation wind directions for that single
VL1L2 line.
(3) Compute the wind direction error (fcst - obs) and write it out
in the ME column.
I suppose we could have also written out a single MAE value that's
just the abs(ME) for the row, but it seemed misleading.
Hopefully that helps clarify what the tool is doing.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 12/16/2013 12:26 PM, Tardif, Elliot M via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=64593 >
>
> Good afternoon John,
>
> Thank you for your reply. I utilized the "-by vx_mask" function in
my job list, and it appears to be helping because Stat Analysis is
generating WDIR data for me. However, I would like to get MAE values
for the AGGR_WDIR if possible. I've noticed in both your output and
mine that Stat Analysis is not generating this value for AGGR_WDIR
data. Additionally, when I run this for a monthly time period, I've
noticed that it won't even compute MAE for some ROW_MEAN_WDIR rows as
well (see output text below).
>
> JOB_LIST: -job aggregate_stat -fcst_valid_beg 20111001_000000
-fcst_valid_end 20111031_230000 -obs_valid_beg 20111001_000000
-obs_valid_end 20111031_230000 -fcst_lev Z10 -vx_mask SEMAP -vx_mask
AL -vx_mask FL -vx_mask GA -vx_mask KY -vx_mask MS -vx_mask NC
-vx_mask SC -vx_mask TN -vx_mask VA -vx_mask WV -line_type VL1L2 -by
VX_MASK -dump_row ./aggregate_test.stat -out_line_type WDIR
> COL_NAME: VX_MASK TOTAL FBAR OBAR ME MAE
> ROW_MEAN_WDIR: AL 742 21.31569 8.31192 4.55038 16.51692
> AGGR_WDIR: AL 29543 13.82994 3.77645 10.05349 NA
> ROW_MEAN_WDIR: FL 742 32.49834 41.39321 -0.74336 12.64940
> AGGR_WDIR: FL 82034 36.17281 42.39865 -6.22584 NA
> ROW_MEAN_WDIR: GA 742 358.38956 355.42640 4.64258 11.93934
> AGGR_WDIR: GA 41311 4.96111 5.84808 -0.88697 NA
> ROW_MEAN_WDIR: KY 742 229.88268 226.50603 NA NA
> AGGR_WDIR: KY 18636 258.00278 264.43371 -6.43093 NA
> ROW_MEAN_WDIR: MS 742 51.94934 27.51373 3.66602 20.25508
> AGGR_WDIR: MS 16708 35.56405 18.36394 17.20011 NA
> ROW_MEAN_WDIR: NC 742 338.88680 333.54157 6.49033 9.67657
> AGGR_WDIR: NC 60994 329.65060 318.05075 11.59985 NA
> ROW_MEAN_WDIR: SC 742 353.55464 346.94238 7.69071 11.78999
> AGGR_WDIR: SC 44090 357.74240 353.10774 4.63466 NA
> ROW_MEAN_WDIR: SEMAP 742 12.42234 18.36407 1.08589 9.04197
> AGGR_WDIR: SEMAP 279493 11.42224 15.85470 -4.43246 NA
> ROW_MEAN_WDIR: TN 742 212.46821 209.62719 NA NA
> AGGR_WDIR: TN 15666 279.77197 279.09069 0.68128 NA
> ROW_MEAN_WDIR: VA 742 316.20936 302.85360 -1.69838 16.91692
> AGGR_WDIR: VA 57346 294.82647 274.04371 20.78276 NA
> ROW_MEAN_WDIR: WV 742 222.88275 238.23608 NA NA
> AGGR_WDIR: WV 16886 254.47068 254.63629 -0.16561 NA
>
>
> Can either of these issues be fixed?
>
> Thanks,
>
> Elliot Tardif, Meteorologist II
> NC DENR, Division of Air Quality
> Planning Section, Attainment Planning Branch
> 1641 Mail Service Center
> Raleigh, NC 27699-1641
> Phone/Fax: 919-707-8483
> Email: Elliot.Tardif at ncdenr.gov
> Web : http://www.ncair.org
>
> Email correspondence to and from this address is subject to the
North Carolina Public Records Law and may be disclosed to third
parties unless the content is exempt by statue or other regulation.
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, December 11, 2013 6:32 PM
> To: Tardif, Elliot M
> Subject: Re: [rt.rap.ucar.edu #64593] WDIR issue in Stat Analysis
>
> Elliot,
>
> I ran the following command using the data you sent me:
> METv4.1/bin/stat_analysis -config STATAnalysisConfig_emt
-lookin
> point_stat*.stat -out wdir.out
>
> It appears to have run successfully and I've attached the output
"wdir.out" file.
>
> Let me suggest that you simplify your job a bit using the "-by" case
option. Take a look at the following single job below that I ran on
the command line. I replaced the "-vx_mask" setting with "-by
vx_mask". So that runs the same job but for each unique value it
finds in the "vx_mask" column.
>
> [johnhg at rambler]% METv4.1/bin/stat_analysis -job aggregate_stat
-line_type VL1L2 -out_line_type WDIR -fcst_lev Z10 -by vx_mask
-lookin *.stat
> DEBUG 2: STAT Lines read = 40403
> DEBUG 2: STAT Lines retained = 71
> DEBUG 2:
> DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_lev Z10
> -line_type VL1L2 -by VX_MASK -out_line_type WDIR
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073121605714
> DEBUG 2: Computing output for 11 case(s).
> JOB_LIST: -job aggregate_stat -fcst_lev Z10 -line_type VL1L2
-by VX_MASK -out_line_type WDIR
> COL_NAME: VX_MASK TOTAL FBAR OBAR ME MAE
> ROW_MEAN_WDIR: AL 6 61.85231 41.26332 20.55289 20.55289
> AGGR_WDIR: AL 251 59.95983 41.34929 18.61054 NA
> ROW_MEAN_WDIR: FL 6 235.03649 245.51103 -14.10477 30.83638
> AGGR_WDIR: FL 761 239.55794 268.92570 -29.36776 NA
> ROW_MEAN_WDIR: GA 6 29.30936 29.95539 -0.62353 4.48081
> AGGR_WDIR: GA 324 29.98097 30.88385 -0.90288 NA
> ROW_MEAN_WDIR: KY 6 83.68932 77.76918 5.93250 6.19367
> AGGR_WDIR: KY 143 82.75768 75.90090 6.85677 NA
> ROW_MEAN_WDIR: MS 6 21.17283 359.51207 19.46255 22.25670
> AGGR_WDIR: MS 149 29.94464 357.81545 32.12920 NA
> ROW_MEAN_WDIR: NC 6 46.34654 37.29122 9.05385 9.05385
> AGGR_WDIR: NC 533 45.95594 37.21045 8.74549 NA
> ROW_MEAN_WDIR: SC 6 28.29254 29.39961 -1.09695 2.05188
> AGGR_WDIR: SC 259 28.41739 30.38964 -1.97225 NA
> ROW_MEAN_WDIR: SEMAP 11 39.68436 34.02263 5.61859 5.61859
> AGGR_WDIR: SEMAP 4279 40.32304 33.71078 6.61226 NA
> ROW_MEAN_WDIR: TN 6 99.65929 89.42600 10.98451 10.98451
> AGGR_WDIR: TN 148 91.38965 78.66731 12.72235 NA
> ROW_MEAN_WDIR: VA 6 22.16236 17.73469 4.41535 4.41535
> AGGR_WDIR: VA 461 22.41267 18.09502 4.31765 NA
> ROW_MEAN_WDIR: WV 6 66.45464 61.35398 5.09928 5.60504
> AGGR_WDIR: WV 120 67.10595 60.86002 6.24593 NA
>
> DEBUG 2: Job 1 used 71 out of 71 STAT lines.
>
> Now, why isn't this working on your machine? I really don't know.
Try turning up the verbosity level "-v 4" to see if STAT-Analysis give
you any more useful debugging information.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
>
> On 12/11/2013 04:08 PM, Tardif, Elliot M via RT wrote:
>>
>> Wed Dec 11 16:07:59 2013: Request 64593 was acted upon.
>> Transaction: Ticket created by elliot.tardif at ncdenr.gov
>> Queue: met_help
>> Subject: WDIR issue in Stat Analysis
>> Owner: Nobody
>> Requestors: elliot.tardif at ncdenr.gov
>> Status: new
>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=64593
>>>
>>
>>
>> Good afternoon,
>>
>> I am attempting to verify wind direction from point stat output,
using Stat Analysis. When I attempt to run Stat Analysis on the point
stat files, it will tell me that there are no files in which it can
work with (please see attached file). However, when I grep these
files it shows data that meet my criteria, so I'm not sure what to
make of this. I downloaded the latest patch file for METv4.1 earlier
today, and it did not appear to help this issue. I have ftp'd 6 hours
of point stat data as well as my config and executable scripts for
your review. I appreciate any help you may be able to offer.
>>
>> Thanks,
>>
>> Elliot Tardif, Meteorologist II
>> NC DENR, Division of Air Quality
>> Planning Section, Attainment Planning Branch
>> 1641 Mail Service Center
>> Raleigh, NC 27699-1641
>> Phone/Fax: 919-707-8483
>> Email: Elliot.Tardif at ncdenr.gov<mailto:Nick.Witcraft at ncdenr.gov>
>> Web : http://www.ncair.org<http://www.ncair.org/>
>>
>> Email correspondence to and from this address is subject to the
North Carolina Public Records Law and may be disclosed to third
parties unless the content is exempt by statue or other regulation.
>>
>>
>
>
------------------------------------------------
More information about the Met_help
mailing list