[Met_help] [rt.rap.ucar.edu #86480] History for Errors using -column_thresh
John Halley Gotway via RT
met_help at ucar.edu
Wed Aug 15 12:14:09 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I'm trying to use -column_thresh with stat_analysis to mask out a lat/lon
box. I've added this command in my StatAnalysis config file:
"-job aggregate_stat -out_line_type CTC -column_thresh OBS gt1.0
-out_thresh ge5.5689 \
-column_thresh OBS_LAT 'ge15.0&&le70.0' \
-column_thresh OBS_LON 'ge-100.0&&le-10.0' \
-by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
-out_stat /opc_test/home/opc_test/data/met_verif/regen_mar2018_data/
new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
But, when it runs, I get these errors:
Output:
DEBUG 2: Processing Job 2: -job aggregate_stat -fcst_var WIND -fcst_lev Z10
-obtype ASCAT -line_type MPR -column_thresh OBS >1.0 -column_thresh OBS_LAT
>=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-10.0 -by FCST_VALID_BEG
-by FCST_LEAD -by VX_MASK -out_stat /opc_test/home/opc_test/data/
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat
-out_line_type CTC -out_thresh >=5.5689
DEBUG 1: Creating STAT output file "/opc_test/home/opc_test/data/
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
MPR_to_CTC_ge5.5689.stat"
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073401622619
DEBUG 2: Computing output for 0 case(s).
WARNING:
WARNING: do_job_aggr_stat() -> no matching STAT lines found for job: -job
aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype ASCAT -line_type MPR
-column_thresh OBS >1.0 -column_thresh OBS_LAT >=15.0&&<=70.0
-column_thresh OBS_LON >=-100.0&&<=-10.0 -by FCST_VALID_BEG -by FCST_LEAD
-by VX_MASK -out_stat /opc_test/home/opc_test/data/
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat
-out_line_type CTC -out_thresh >=5.5689
WARNING:
DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
DEBUG 2:
Am I not using the command properly?
Thanks!
Roz
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Errors using -column_thresh
From: John Halley Gotway
Time: Tue Aug 07 16:07:37 2018
Hi Roz,
I see you're having trouble getting with STAT-Analysis, getting MPR
lines
to be included in the column thresholds you've defined. I don't see
anything obviously wrong with what you've defined.
The warning message indicates that the filters you defined have
discarded
all of the input MPR lines.
We'd just need to go through your filtering options one-by-one to find
out
why.
- Are you passing STAT-Analysis files which contain MPR lines?
- Does the FCST_VAR column say "WIND"?
- Does the FCST_LEV column say "Z10"?
- Does the OBTYPE column say "ASCAT"?
- Does the OBS column have values > 1.0?
- What are the ranges of the OBS_LAT and OBS_LON columns?
If you'd like to send me a sample .stat file, I could probably track
it
down.
Thanks,
John
On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Mon Aug 06 07:43:46 2018: Request 86480 was acted upon.
> Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> Queue: met_help
> Subject: Errors using -column_thresh
> Owner: Nobody
> Requestors: rosalyn.maccracken at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
>
>
> Hi,
>
> I'm trying to use -column_thresh with stat_analysis to mask out a
lat/lon
> box. I've added this command in my StatAnalysis config file:
>
> "-job aggregate_stat -out_line_type CTC -column_thresh OBS gt1.0
> -out_thresh ge5.5689 \
> -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
> -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> -out_stat
/opc_test/home/opc_test/data/met_verif/regen_mar2018_data/
> new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
>
> But, when it runs, I get these errors:
>
> Output:
> DEBUG 2: Processing Job 2: -job aggregate_stat -fcst_var WIND
-fcst_lev Z10
> -obtype ASCAT -line_type MPR -column_thresh OBS >1.0 -column_thresh
OBS_LAT
> >=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
FCST_VALID_BEG
> -by FCST_LEAD -by VX_MASK -out_stat /opc_test/home/opc_test/data/
>
>
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat
> -out_line_type CTC -out_thresh >=5.5689
> DEBUG 1: Creating STAT output file "/opc_test/home/opc_test/data/
> met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> MPR_to_CTC_ge5.5689.stat"
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073401622619
> DEBUG 2: Computing output for 0 case(s).
> WARNING:
> WARNING: do_job_aggr_stat() -> no matching STAT lines found for job:
-job
> aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype ASCAT -line_type
MPR
> -column_thresh OBS >1.0 -column_thresh OBS_LAT >=15.0&&<=70.0
> -column_thresh OBS_LON >=-100.0&&<=-10.0 -by FCST_VALID_BEG -by
FCST_LEAD
> -by VX_MASK -out_stat /opc_test/home/opc_test/data/
>
>
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat
> -out_line_type CTC -out_thresh >=5.5689
> WARNING:
> DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> DEBUG 2:
>
> Am I not using the command properly?
>
> Thanks!
>
> Roz
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Fri Aug 10 11:41:43 2018
Hi John,
I'm sorry I didn't get back to you sooner, but, I was tied up with
some
other things. Let me double check with all of those items in my file
and
make sure that I don't have some sort of obvious error. I'll be back
in
touch after I check on this.
Roz
On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Hi Roz,
>
> I see you're having trouble getting with STAT-Analysis, getting MPR
lines
> to be included in the column thresholds you've defined. I don't see
> anything obviously wrong with what you've defined.
>
> The warning message indicates that the filters you defined have
discarded
> all of the input MPR lines.
>
> We'd just need to go through your filtering options one-by-one to
find out
> why.
>
> - Are you passing STAT-Analysis files which contain MPR lines?
> - Does the FCST_VAR column say "WIND"?
> - Does the FCST_LEV column say "Z10"?
> - Does the OBTYPE column say "ASCAT"?
> - Does the OBS column have values > 1.0?
> - What are the ranges of the OBS_LAT and OBS_LON columns?
>
> If you'd like to send me a sample .stat file, I could probably track
it
> down.
>
> Thanks,
> John
>
>
> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Aug 06 07:43:46 2018: Request 86480 was acted upon.
> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > Queue: met_help
> > Subject: Errors using -column_thresh
> > Owner: Nobody
> > Requestors: rosalyn.maccracken at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> >
> >
> > Hi,
> >
> > I'm trying to use -column_thresh with stat_analysis to mask out a
lat/lon
> > box. I've added this command in my StatAnalysis config file:
> >
> > "-job aggregate_stat -out_line_type CTC -column_thresh OBS gt1.0
> > -out_thresh ge5.5689 \
> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > -out_stat
/opc_test/home/opc_test/data/met_verif/regen_mar2018_data/
> > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> >
> > But, when it runs, I get these errors:
> >
> > Output:
> > DEBUG 2: Processing Job 2: -job aggregate_stat -fcst_var WIND
-fcst_lev
> Z10
> > -obtype ASCAT -line_type MPR -column_thresh OBS >1.0
-column_thresh
> OBS_LAT
> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
> FCST_VALID_BEG
> > -by FCST_LEAD -by VX_MASK -out_stat /opc_test/home/opc_test/data/
> >
> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> MPR_to_CTC_ge5.5689.stat
> > -out_line_type CTC -out_thresh >=5.5689
> > DEBUG 1: Creating STAT output file "/opc_test/home/opc_test/data/
> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> > MPR_to_CTC_ge5.5689.stat"
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=18446744073401622619
> > DEBUG 2: Computing output for 0 case(s).
> > WARNING:
> > WARNING: do_job_aggr_stat() -> no matching STAT lines found for
job: -job
> > aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype ASCAT
-line_type MPR
> > -column_thresh OBS >1.0 -column_thresh OBS_LAT >=15.0&&<=70.0
> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by FCST_VALID_BEG -by
FCST_LEAD
> > -by VX_MASK -out_stat /opc_test/home/opc_test/data/
> >
> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> MPR_to_CTC_ge5.5689.stat
> > -out_line_type CTC -out_thresh >=5.5689
> > WARNING:
> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> > DEBUG 2:
> >
> > Am I not using the command properly?
> >
> > Thanks!
> >
> > Roz
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Fri Aug 10 13:24:53 2018
Hi John,
I double checked all those things you listed, and they are all
correct.
So, I've put a file in the /maccracken directory on your ftp site.
Let me
know what you see.
Thanks for your help!
Roz
On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn MacCracken - NOAA Affiliate <
rosalyn.maccracken at noaa.gov> wrote:
> Hi John,
>
> I'm sorry I didn't get back to you sooner, but, I was tied up with
some
> other things. Let me double check with all of those items in my
file and
> make sure that I don't have some sort of obvious error. I'll be
back in
> touch after I check on this.
>
> Roz
>
> On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Hi Roz,
>>
>> I see you're having trouble getting with STAT-Analysis, getting MPR
lines
>> to be included in the column thresholds you've defined. I don't
see
>> anything obviously wrong with what you've defined.
>>
>> The warning message indicates that the filters you defined have
discarded
>> all of the input MPR lines.
>>
>> We'd just need to go through your filtering options one-by-one to
find out
>> why.
>>
>> - Are you passing STAT-Analysis files which contain MPR lines?
>> - Does the FCST_VAR column say "WIND"?
>> - Does the FCST_LEV column say "Z10"?
>> - Does the OBTYPE column say "ASCAT"?
>> - Does the OBS column have values > 1.0?
>> - What are the ranges of the OBS_LAT and OBS_LON columns?
>>
>> If you'd like to send me a sample .stat file, I could probably
track it
>> down.
>>
>> Thanks,
>> John
>>
>>
>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken - NOAA Affiliate
via RT
>> <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Mon Aug 06 07:43:46 2018: Request 86480 was acted upon.
>> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
>> > Queue: met_help
>> > Subject: Errors using -column_thresh
>> > Owner: Nobody
>> > Requestors: rosalyn.maccracken at noaa.gov
>> > Status: new
>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
>> >
>> >
>> > Hi,
>> >
>> > I'm trying to use -column_thresh with stat_analysis to mask out a
>> lat/lon
>> > box. I've added this command in my StatAnalysis config file:
>> >
>> > "-job aggregate_stat -out_line_type CTC -column_thresh OBS gt1.0
>> > -out_thresh ge5.5689 \
>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
>> > -out_stat /opc_test/home/opc_test/data/m
>> et_verif/regen_mar2018_data/
>> > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
>> >
>> > But, when it runs, I get these errors:
>> >
>> > Output:
>> > DEBUG 2: Processing Job 2: -job aggregate_stat -fcst_var WIND
-fcst_lev
>> Z10
>> > -obtype ASCAT -line_type MPR -column_thresh OBS >1.0
-column_thresh
>> OBS_LAT
>> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
>> FCST_VALID_BEG
>> > -by FCST_LEAD -by VX_MASK -out_stat /opc_test/home/opc_test/data/
>> >
>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
>> _to_CTC_ge5.5689.stat
>> > -out_line_type CTC -out_thresh >=5.5689
>> > DEBUG 1: Creating STAT output file "/opc_test/home/opc_test/data/
>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
>> > MPR_to_CTC_ge5.5689.stat"
>> > GSL_RNG_TYPE=mt19937
>> > GSL_RNG_SEED=18446744073401622619
>> > DEBUG 2: Computing output for 0 case(s).
>> > WARNING:
>> > WARNING: do_job_aggr_stat() -> no matching STAT lines found for
job:
>> -job
>> > aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype ASCAT
-line_type MPR
>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT >=15.0&&<=70.0
>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by FCST_VALID_BEG -by
>> FCST_LEAD
>> > -by VX_MASK -out_stat /opc_test/home/opc_test/data/
>> >
>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
>> _to_CTC_ge5.5689.stat
>> > -out_line_type CTC -out_thresh >=5.5689
>> > WARNING:
>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
>> > DEBUG 2:
>> >
>> > Am I not using the command properly?
>> >
>> > Thanks!
>> >
>> > Roz
>> >
>> > --
>> > Rosalyn MacCracken
>> > Support Scientist
>> >
>> > Ocean Applications Branch
>> > NOAA/NWS Ocean Prediction Center
>> > NCWCP
>> > 5830 University Research Ct
>> > College Park, MD 20740-3818
>> >
>> > (p) 301-683-1551
>> > rosalyn.maccracken at noaa.gov
>> >
>> >
>>
>>
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Mon Aug 13 08:33:37 2018
Hi John,
I have another question related to my reprocessing of my data.
Ok, so, I realized that when I switched ASCAT data sources back in
March,
from prepbufr to MGDRlite files, I should have been using quality
flags to
filter out my data. Because I wasn't checking for the quality flags,
my
statistics were not so good, since it was letting in bad data, and
stats
were being calculated from this bad data.
So, I need to find a way to correct this, but, the old story is I
don't
have the original GFS data, and it's too massive to grab from HPSS. I
do
have the original ASCAT files, which I can run through my program to
filter
out the bad observations, as well as matched points from the *mpr and
*stat
files.
Originally, I was just going to filter out using that -column_thresh
command, and filter out over land. But, I had this idea that I wanted
to
run by you, to see if this makes sense to do.
I have the time, lat, lon and GFS observation and the corresponding
ASCAT
observation point. Can I do this:
1) read the time, lat, lon and GFS observation out of the mpr file,
and
make a new input file with just those variables in the ASCII format
that
MET can use
2) reprocess my ASCAT data and write it into an ASCII file in MET
format
3) run ASCII2NC on these files, to create the two input files for
MET,
then, run point_stat on the netcdf files to regenerate the new
corrected
stat files
Does that make sense to do? I know it could be alot of work, but, if
I can
get corrected statistics, it might be worth it. What do you think?
Roz
On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn MacCracken - NOAA Affiliate <
rosalyn.maccracken at noaa.gov> wrote:
> Hi John,
>
> I double checked all those things you listed, and they are all
correct.
> So, I've put a file in the /maccracken directory on your ftp site.
Let me
> know what you see.
>
> Thanks for your help!
>
> Roz
>
> On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn MacCracken - NOAA Affiliate
<
> rosalyn.maccracken at noaa.gov> wrote:
>
>> Hi John,
>>
>> I'm sorry I didn't get back to you sooner, but, I was tied up with
some
>> other things. Let me double check with all of those items in my
file and
>> make sure that I don't have some sort of obvious error. I'll be
back in
>> touch after I check on this.
>>
>> Roz
>>
>> On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>>> Hi Roz,
>>>
>>> I see you're having trouble getting with STAT-Analysis, getting
MPR lines
>>> to be included in the column thresholds you've defined. I don't
see
>>> anything obviously wrong with what you've defined.
>>>
>>> The warning message indicates that the filters you defined have
discarded
>>> all of the input MPR lines.
>>>
>>> We'd just need to go through your filtering options one-by-one to
find
>>> out
>>> why.
>>>
>>> - Are you passing STAT-Analysis files which contain MPR lines?
>>> - Does the FCST_VAR column say "WIND"?
>>> - Does the FCST_LEV column say "Z10"?
>>> - Does the OBTYPE column say "ASCAT"?
>>> - Does the OBS column have values > 1.0?
>>> - What are the ranges of the OBS_LAT and OBS_LON columns?
>>>
>>> If you'd like to send me a sample .stat file, I could probably
track it
>>> down.
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken - NOAA Affiliate
via
>>> RT <
>>> met_help at ucar.edu> wrote:
>>>
>>> >
>>> > Mon Aug 06 07:43:46 2018: Request 86480 was acted upon.
>>> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
>>> > Queue: met_help
>>> > Subject: Errors using -column_thresh
>>> > Owner: Nobody
>>> > Requestors: rosalyn.maccracken at noaa.gov
>>> > Status: new
>>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
>>> >
>>> >
>>> >
>>> > Hi,
>>> >
>>> > I'm trying to use -column_thresh with stat_analysis to mask out
a
>>> lat/lon
>>> > box. I've added this command in my StatAnalysis config file:
>>> >
>>> > "-job aggregate_stat -out_line_type CTC -column_thresh OBS gt1.0
>>> > -out_thresh ge5.5689 \
>>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
>>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
>>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
>>> > -out_stat /opc_test/home/opc_test/data/m
>>> et_verif/regen_mar2018_data/
>>> > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
>>> >
>>> > But, when it runs, I get these errors:
>>> >
>>> > Output:
>>> > DEBUG 2: Processing Job 2: -job aggregate_stat -fcst_var WIND
>>> -fcst_lev Z10
>>> > -obtype ASCAT -line_type MPR -column_thresh OBS >1.0
-column_thresh
>>> OBS_LAT
>>> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
>>> FCST_VALID_BEG
>>> > -by FCST_LEAD -by VX_MASK -out_stat
/opc_test/home/opc_test/data/
>>> >
>>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
>>> _to_CTC_ge5.5689.stat
>>> > -out_line_type CTC -out_thresh >=5.5689
>>> > DEBUG 1: Creating STAT output file
"/opc_test/home/opc_test/data/
>>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
>>> > MPR_to_CTC_ge5.5689.stat"
>>> > GSL_RNG_TYPE=mt19937
>>> > GSL_RNG_SEED=18446744073401622619
>>> > DEBUG 2: Computing output for 0 case(s).
>>> > WARNING:
>>> > WARNING: do_job_aggr_stat() -> no matching STAT lines found for
job:
>>> -job
>>> > aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype ASCAT
-line_type
>>> MPR
>>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT >=15.0&&<=70.0
>>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by FCST_VALID_BEG -by
>>> FCST_LEAD
>>> > -by VX_MASK -out_stat /opc_test/home/opc_test/data/
>>> >
>>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
>>> _to_CTC_ge5.5689.stat
>>> > -out_line_type CTC -out_thresh >=5.5689
>>> > WARNING:
>>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
>>> > DEBUG 2:
>>> >
>>> > Am I not using the command properly?
>>> >
>>> > Thanks!
>>> >
>>> > Roz
>>> >
>>> > --
>>> > Rosalyn MacCracken
>>> > Support Scientist
>>> >
>>> > Ocean Applications Branch
>>> > NOAA/NWS Ocean Prediction Center
>>> > NCWCP
>>> > 5830 University Research Ct
>>> > College Park, MD 20740-3818
>>> >
>>> > (p) 301-683-1551
>>> > rosalyn.maccracken at noaa.gov
>>> >
>>> >
>>>
>>>
>>
>>
>> --
>> Rosalyn MacCracken
>> Support Scientist
>>
>> Ocean Applications Branch
>> NOAA/NWS Ocean Prediction Center
>> NCWCP
>> 5830 University Research Ct
>> College Park, MD 20740-3818
>>
>> (p) 301-683-1551
>> rosalyn.maccracken at noaa.gov
>>
>
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Errors using -column_thresh
From: John Halley Gotway
Time: Mon Aug 13 09:42:42 2018
Roz,
Point-Stat takes as input a gridded NWP forecast file and the NetCDF
output
of a point pre-processing tool (like ascii2nc or pb2nc). Based on
your
description, it doesn't sound like you have gridded NWP forecast files
available. So no, I don't think that approach would work.
The MPR output line type does include a column named "OBS_QC" which is
a
string containing the observation quality control value. I'd suggest
checking the contents of that column in your MPR output lines to see
if
those values could be filtered to do the quality control filtering
you're
after. Check to see if the values in that column are strings or
numbers
(in PREPBUFR they're integers... in MADIS they're strings).
In STAT-Analysis, you can use the "-column_thresh" option to threshold
a
numeric column or you can use the "-column_str" option to list strings
to
retained. For example:
-column_thresh OBS_QC 'ge0&&le3' # this keeps all values between 0
and 3
-column_str OBS_QC aa,ab,ac # this keeps strings "aa",
"ab", or
"ac"
Hope that helps.
Thanks,
John
On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
>
> Hi John,
>
> I have another question related to my reprocessing of my data.
>
> Ok, so, I realized that when I switched ASCAT data sources back in
March,
> from prepbufr to MGDRlite files, I should have been using quality
flags to
> filter out my data. Because I wasn't checking for the quality
flags, my
> statistics were not so good, since it was letting in bad data, and
stats
> were being calculated from this bad data.
>
> So, I need to find a way to correct this, but, the old story is I
don't
> have the original GFS data, and it's too massive to grab from HPSS.
I do
> have the original ASCAT files, which I can run through my program to
filter
> out the bad observations, as well as matched points from the *mpr
and *stat
> files.
>
> Originally, I was just going to filter out using that -column_thresh
> command, and filter out over land. But, I had this idea that I
wanted to
> run by you, to see if this makes sense to do.
>
> I have the time, lat, lon and GFS observation and the corresponding
ASCAT
> observation point. Can I do this:
> 1) read the time, lat, lon and GFS observation out of the mpr file,
and
> make a new input file with just those variables in the ASCII format
that
> MET can use
> 2) reprocess my ASCAT data and write it into an ASCII file in MET
format
> 3) run ASCII2NC on these files, to create the two input files for
MET,
> then, run point_stat on the netcdf files to regenerate the new
corrected
> stat files
>
> Does that make sense to do? I know it could be alot of work, but,
if I can
> get corrected statistics, it might be worth it. What do you think?
>
> Roz
>
> On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn MacCracken - NOAA Affiliate
<
> rosalyn.maccracken at noaa.gov> wrote:
>
> > Hi John,
> >
> > I double checked all those things you listed, and they are all
correct.
> > So, I've put a file in the /maccracken directory on your ftp site.
Let
> me
> > know what you see.
> >
> > Thanks for your help!
> >
> > Roz
> >
> > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn MacCracken - NOAA
Affiliate <
> > rosalyn.maccracken at noaa.gov> wrote:
> >
> >> Hi John,
> >>
> >> I'm sorry I didn't get back to you sooner, but, I was tied up
with some
> >> other things. Let me double check with all of those items in my
file
> and
> >> make sure that I don't have some sort of obvious error. I'll be
back in
> >> touch after I check on this.
> >>
> >> Roz
> >>
> >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >>> Hi Roz,
> >>>
> >>> I see you're having trouble getting with STAT-Analysis, getting
MPR
> lines
> >>> to be included in the column thresholds you've defined. I don't
see
> >>> anything obviously wrong with what you've defined.
> >>>
> >>> The warning message indicates that the filters you defined have
> discarded
> >>> all of the input MPR lines.
> >>>
> >>> We'd just need to go through your filtering options one-by-one
to find
> >>> out
> >>> why.
> >>>
> >>> - Are you passing STAT-Analysis files which contain MPR lines?
> >>> - Does the FCST_VAR column say "WIND"?
> >>> - Does the FCST_LEV column say "Z10"?
> >>> - Does the OBTYPE column say "ASCAT"?
> >>> - Does the OBS column have values > 1.0?
> >>> - What are the ranges of the OBS_LAT and OBS_LON columns?
> >>>
> >>> If you'd like to send me a sample .stat file, I could probably
track it
> >>> down.
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>>
> >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken - NOAA
Affiliate via
> >>> RT <
> >>> met_help at ucar.edu> wrote:
> >>>
> >>> >
> >>> > Mon Aug 06 07:43:46 2018: Request 86480 was acted upon.
> >>> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> >>> > Queue: met_help
> >>> > Subject: Errors using -column_thresh
> >>> > Owner: Nobody
> >>> > Requestors: rosalyn.maccracken at noaa.gov
> >>> > Status: new
> >>> > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> >>> >
> >>> >
> >>> >
> >>> > Hi,
> >>> >
> >>> > I'm trying to use -column_thresh with stat_analysis to mask
out a
> >>> lat/lon
> >>> > box. I've added this command in my StatAnalysis config file:
> >>> >
> >>> > "-job aggregate_stat -out_line_type CTC -column_thresh OBS
gt1.0
> >>> > -out_thresh ge5.5689 \
> >>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> >>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
> >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> >>> > -out_stat /opc_test/home/opc_test/data/m
> >>> et_verif/regen_mar2018_data/
> >>> > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> >>> >
> >>> > But, when it runs, I get these errors:
> >>> >
> >>> > Output:
> >>> > DEBUG 2: Processing Job 2: -job aggregate_stat -fcst_var WIND
> >>> -fcst_lev Z10
> >>> > -obtype ASCAT -line_type MPR -column_thresh OBS >1.0
-column_thresh
> >>> OBS_LAT
> >>> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
> >>> FCST_VALID_BEG
> >>> > -by FCST_LEAD -by VX_MASK -out_stat
/opc_test/home/opc_test/data/
> >>> >
> >>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
> >>> _to_CTC_ge5.5689.stat
> >>> > -out_line_type CTC -out_thresh >=5.5689
> >>> > DEBUG 1: Creating STAT output file
"/opc_test/home/opc_test/data/
> >>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> >>> > MPR_to_CTC_ge5.5689.stat"
> >>> > GSL_RNG_TYPE=mt19937
> >>> > GSL_RNG_SEED=18446744073401622619
> >>> > DEBUG 2: Computing output for 0 case(s).
> >>> > WARNING:
> >>> > WARNING: do_job_aggr_stat() -> no matching STAT lines found
for job:
> >>> -job
> >>> > aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype ASCAT
-line_type
> >>> MPR
> >>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT >=15.0&&<=70.0
> >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by FCST_VALID_BEG
-by
> >>> FCST_LEAD
> >>> > -by VX_MASK -out_stat /opc_test/home/opc_test/data/
> >>> >
> >>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
> >>> _to_CTC_ge5.5689.stat
> >>> > -out_line_type CTC -out_thresh >=5.5689
> >>> > WARNING:
> >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> >>> > DEBUG 2:
> >>> >
> >>> > Am I not using the command properly?
> >>> >
> >>> > Thanks!
> >>> >
> >>> > Roz
> >>> >
> >>> > --
> >>> > Rosalyn MacCracken
> >>> > Support Scientist
> >>> >
> >>> > Ocean Applications Branch
> >>> > NOAA/NWS Ocean Prediction Center
> >>> > NCWCP
> >>> > 5830 University Research Ct
> >>> > College Park, MD 20740-3818
> >>> >
> >>> > (p) 301-683-1551
> >>> > rosalyn.maccracken at noaa.gov
> >>> >
> >>> >
> >>>
> >>>
> >>
> >>
> >> --
> >> Rosalyn MacCracken
> >> Support Scientist
> >>
> >> Ocean Applications Branch
> >> NOAA/NWS Ocean Prediction Center
> >> NCWCP
> >> 5830 University Research Ct
> >> College Park, MD 20740-3818
> >>
> >> (p) 301-683-1551
> >> rosalyn.maccracken at noaa.gov
> >>
> >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
>
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Mon Aug 13 10:56:42 2018
So, you can't compare two point files, even if both of them are NetCDF
format? Is there a way to write point observations out to a grid? In
other words, take the point GFS observations, and write them to a
global
grid (using some other tool, like wgrib or something else), and fill
in
missing values with NaN, or something? Then, use that gridded file to
compare with the ASCAT NetCDF file?
I don't have anything in the OBS_QC column because when I wrote it
into the
ASCII file, I didn't have any flags from my MGDRlite files. This is
part
of the problem...
Now, your answer for the STAT-Analysis, is that what I did wrong in my
use
of stat-analysis from the previous email? Or, did you not get a
chance to
look at my *stat file that I uploaded to the ftp site?
Roz
On Mon, Aug 13, 2018 at 3:42 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Roz,
>
> Point-Stat takes as input a gridded NWP forecast file and the NetCDF
output
> of a point pre-processing tool (like ascii2nc or pb2nc). Based on
your
> description, it doesn't sound like you have gridded NWP forecast
files
> available. So no, I don't think that approach would work.
>
> The MPR output line type does include a column named "OBS_QC" which
is a
> string containing the observation quality control value. I'd
suggest
> checking the contents of that column in your MPR output lines to see
if
> those values could be filtered to do the quality control filtering
you're
> after. Check to see if the values in that column are strings or
numbers
> (in PREPBUFR they're integers... in MADIS they're strings).
>
> In STAT-Analysis, you can use the "-column_thresh" option to
threshold a
> numeric column or you can use the "-column_str" option to list
strings to
> retained. For example:
> -column_thresh OBS_QC 'ge0&&le3' # this keeps all values between
0 and
> 3
> -column_str OBS_QC aa,ab,ac # this keeps strings "aa",
"ab", or
> "ac"
>
> Hope that helps.
>
> Thanks,
> John
>
> On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> >
> > Hi John,
> >
> > I have another question related to my reprocessing of my data.
> >
> > Ok, so, I realized that when I switched ASCAT data sources back in
March,
> > from prepbufr to MGDRlite files, I should have been using quality
flags
> to
> > filter out my data. Because I wasn't checking for the quality
flags, my
> > statistics were not so good, since it was letting in bad data, and
stats
> > were being calculated from this bad data.
> >
> > So, I need to find a way to correct this, but, the old story is I
don't
> > have the original GFS data, and it's too massive to grab from
HPSS. I do
> > have the original ASCAT files, which I can run through my program
to
> filter
> > out the bad observations, as well as matched points from the *mpr
and
> *stat
> > files.
> >
> > Originally, I was just going to filter out using that
-column_thresh
> > command, and filter out over land. But, I had this idea that I
wanted to
> > run by you, to see if this makes sense to do.
> >
> > I have the time, lat, lon and GFS observation and the
corresponding ASCAT
> > observation point. Can I do this:
> > 1) read the time, lat, lon and GFS observation out of the mpr
file, and
> > make a new input file with just those variables in the ASCII
format that
> > MET can use
> > 2) reprocess my ASCAT data and write it into an ASCII file in MET
format
> > 3) run ASCII2NC on these files, to create the two input files for
MET,
> > then, run point_stat on the netcdf files to regenerate the new
corrected
> > stat files
> >
> > Does that make sense to do? I know it could be alot of work, but,
if I
> can
> > get corrected statistics, it might be worth it. What do you
think?
> >
> > Roz
> >
> > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn MacCracken - NOAA
Affiliate <
> > rosalyn.maccracken at noaa.gov> wrote:
> >
> > > Hi John,
> > >
> > > I double checked all those things you listed, and they are all
correct.
> > > So, I've put a file in the /maccracken directory on your ftp
site. Let
> > me
> > > know what you see.
> > >
> > > Thanks for your help!
> > >
> > > Roz
> > >
> > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn MacCracken - NOAA
Affiliate <
> > > rosalyn.maccracken at noaa.gov> wrote:
> > >
> > >> Hi John,
> > >>
> > >> I'm sorry I didn't get back to you sooner, but, I was tied up
with
> some
> > >> other things. Let me double check with all of those items in
my file
> > and
> > >> make sure that I don't have some sort of obvious error. I'll
be back
> in
> > >> touch after I check on this.
> > >>
> > >> Roz
> > >>
> > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >>> Hi Roz,
> > >>>
> > >>> I see you're having trouble getting with STAT-Analysis,
getting MPR
> > lines
> > >>> to be included in the column thresholds you've defined. I
don't see
> > >>> anything obviously wrong with what you've defined.
> > >>>
> > >>> The warning message indicates that the filters you defined
have
> > discarded
> > >>> all of the input MPR lines.
> > >>>
> > >>> We'd just need to go through your filtering options one-by-one
to
> find
> > >>> out
> > >>> why.
> > >>>
> > >>> - Are you passing STAT-Analysis files which contain MPR lines?
> > >>> - Does the FCST_VAR column say "WIND"?
> > >>> - Does the FCST_LEV column say "Z10"?
> > >>> - Does the OBTYPE column say "ASCAT"?
> > >>> - Does the OBS column have values > 1.0?
> > >>> - What are the ranges of the OBS_LAT and OBS_LON columns?
> > >>>
> > >>> If you'd like to send me a sample .stat file, I could probably
track
> it
> > >>> down.
> > >>>
> > >>> Thanks,
> > >>> John
> > >>>
> > >>>
> > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken - NOAA
Affiliate
> via
> > >>> RT <
> > >>> met_help at ucar.edu> wrote:
> > >>>
> > >>> >
> > >>> > Mon Aug 06 07:43:46 2018: Request 86480 was acted upon.
> > >>> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > >>> > Queue: met_help
> > >>> > Subject: Errors using -column_thresh
> > >>> > Owner: Nobody
> > >>> > Requestors: rosalyn.maccracken at noaa.gov
> > >>> > Status: new
> > >>> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > >>> >
> > >>> >
> > >>> >
> > >>> > Hi,
> > >>> >
> > >>> > I'm trying to use -column_thresh with stat_analysis to mask
out a
> > >>> lat/lon
> > >>> > box. I've added this command in my StatAnalysis config
file:
> > >>> >
> > >>> > "-job aggregate_stat -out_line_type CTC -column_thresh OBS
gt1.0
> > >>> > -out_thresh ge5.5689 \
> > >>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> > >>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
> > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > >>> > -out_stat /opc_test/home/opc_test/data/m
> > >>> et_verif/regen_mar2018_data/
> > >>> > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > >>> >
> > >>> > But, when it runs, I get these errors:
> > >>> >
> > >>> > Output:
> > >>> > DEBUG 2: Processing Job 2: -job aggregate_stat -fcst_var
WIND
> > >>> -fcst_lev Z10
> > >>> > -obtype ASCAT -line_type MPR -column_thresh OBS >1.0
-column_thresh
> > >>> OBS_LAT
> > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
> > >>> FCST_VALID_BEG
> > >>> > -by FCST_LEAD -by VX_MASK -out_stat
/opc_test/home/opc_test/data/
> > >>> >
> > >>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
> > >>> _to_CTC_ge5.5689.stat
> > >>> > -out_line_type CTC -out_thresh >=5.5689
> > >>> > DEBUG 1: Creating STAT output file
"/opc_test/home/opc_test/data/
> > >>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> > >>> > MPR_to_CTC_ge5.5689.stat"
> > >>> > GSL_RNG_TYPE=mt19937
> > >>> > GSL_RNG_SEED=18446744073401622619
> > >>> > DEBUG 2: Computing output for 0 case(s).
> > >>> > WARNING:
> > >>> > WARNING: do_job_aggr_stat() -> no matching STAT lines found
for
> job:
> > >>> -job
> > >>> > aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype ASCAT
> -line_type
> > >>> MPR
> > >>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT
>=15.0&&<=70.0
> > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by FCST_VALID_BEG
-by
> > >>> FCST_LEAD
> > >>> > -by VX_MASK -out_stat /opc_test/home/opc_test/data/
> > >>> >
> > >>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
> > >>> _to_CTC_ge5.5689.stat
> > >>> > -out_line_type CTC -out_thresh >=5.5689
> > >>> > WARNING:
> > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> > >>> > DEBUG 2:
> > >>> >
> > >>> > Am I not using the command properly?
> > >>> >
> > >>> > Thanks!
> > >>> >
> > >>> > Roz
> > >>> >
> > >>> > --
> > >>> > Rosalyn MacCracken
> > >>> > Support Scientist
> > >>> >
> > >>> > Ocean Applications Branch
> > >>> > NOAA/NWS Ocean Prediction Center
> > >>> > NCWCP
> > >>> > 5830 University Research Ct
> > >>> > College Park, MD 20740-3818
> > >>> >
> > >>> > (p) 301-683-1551
> > >>> > rosalyn.maccracken at noaa.gov
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Rosalyn MacCracken
> > >> Support Scientist
> > >>
> > >> Ocean Applications Branch
> > >> NOAA/NWS Ocean Prediction Center
> > >> NCWCP
> > >> 5830 University Research Ct
> > >> College Park, MD 20740-3818
> > >>
> > >> (p) 301-683-1551
> > >> rosalyn.maccracken at noaa.gov
> > >>
> > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Errors using -column_thresh
From: John Halley Gotway
Time: Mon Aug 13 11:32:16 2018
Roz,
That's correct, there is no tool in MET for comparing one set of point
observations to another set of point observations. Presumably you
could
write point observations to a grid and then reconfigure and rerun
Point-Stat, but that'd be pretty difficult, time consuming, and
fraught
with problems.
You currently have many, many MPR lines and the issue is the the
OBS_QC
column is empty. If that OBS_QC column were *not* empty then it'd be
pretty easy to process this data... correct? One option would be
writing a
script to "patch" that column. For each MPR line, do some processing
to
figure out what that OBS_QC column should be and then update it's
value.
Then the question is this... is there a way of getting quality control
values from the MGDRlite files?
I'm sorry, I hadn't had a chance to look at that data file yet. But I
just
checked on the ftp site and don't see a maccracken sub-directory
there:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
Could you please repost?
Thanks,
John
On Mon, Aug 13, 2018 at 10:57 AM Rosalyn MacCracken - NOAA Affiliate
via RT
<met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
>
> So, you can't compare two point files, even if both of them are
NetCDF
> format? Is there a way to write point observations out to a grid?
In
> other words, take the point GFS observations, and write them to a
global
> grid (using some other tool, like wgrib or something else), and fill
in
> missing values with NaN, or something? Then, use that gridded file
to
> compare with the ASCAT NetCDF file?
>
> I don't have anything in the OBS_QC column because when I wrote it
into the
> ASCII file, I didn't have any flags from my MGDRlite files. This is
part
> of the problem...
>
> Now, your answer for the STAT-Analysis, is that what I did wrong in
my use
> of stat-analysis from the previous email? Or, did you not get a
chance to
> look at my *stat file that I uploaded to the ftp site?
>
> Roz
>
> On Mon, Aug 13, 2018 at 3:42 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Roz,
> >
> > Point-Stat takes as input a gridded NWP forecast file and the
NetCDF
> output
> > of a point pre-processing tool (like ascii2nc or pb2nc). Based on
your
> > description, it doesn't sound like you have gridded NWP forecast
files
> > available. So no, I don't think that approach would work.
> >
> > The MPR output line type does include a column named "OBS_QC"
which is a
> > string containing the observation quality control value. I'd
suggest
> > checking the contents of that column in your MPR output lines to
see if
> > those values could be filtered to do the quality control filtering
you're
> > after. Check to see if the values in that column are strings or
numbers
> > (in PREPBUFR they're integers... in MADIS they're strings).
> >
> > In STAT-Analysis, you can use the "-column_thresh" option to
threshold a
> > numeric column or you can use the "-column_str" option to list
strings to
> > retained. For example:
> > -column_thresh OBS_QC 'ge0&&le3' # this keeps all values
between 0
> and
> > 3
> > -column_str OBS_QC aa,ab,ac # this keeps strings "aa",
"ab",
> or
> > "ac"
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > >
> > > Hi John,
> > >
> > > I have another question related to my reprocessing of my data.
> > >
> > > Ok, so, I realized that when I switched ASCAT data sources back
in
> March,
> > > from prepbufr to MGDRlite files, I should have been using
quality flags
> > to
> > > filter out my data. Because I wasn't checking for the quality
flags,
> my
> > > statistics were not so good, since it was letting in bad data,
and
> stats
> > > were being calculated from this bad data.
> > >
> > > So, I need to find a way to correct this, but, the old story is
I don't
> > > have the original GFS data, and it's too massive to grab from
HPSS. I
> do
> > > have the original ASCAT files, which I can run through my
program to
> > filter
> > > out the bad observations, as well as matched points from the
*mpr and
> > *stat
> > > files.
> > >
> > > Originally, I was just going to filter out using that
-column_thresh
> > > command, and filter out over land. But, I had this idea that I
wanted
> to
> > > run by you, to see if this makes sense to do.
> > >
> > > I have the time, lat, lon and GFS observation and the
corresponding
> ASCAT
> > > observation point. Can I do this:
> > > 1) read the time, lat, lon and GFS observation out of the mpr
file,
> and
> > > make a new input file with just those variables in the ASCII
format
> that
> > > MET can use
> > > 2) reprocess my ASCAT data and write it into an ASCII file in
MET
> format
> > > 3) run ASCII2NC on these files, to create the two input files
for MET,
> > > then, run point_stat on the netcdf files to regenerate the new
> corrected
> > > stat files
> > >
> > > Does that make sense to do? I know it could be alot of work,
but, if I
> > can
> > > get corrected statistics, it might be worth it. What do you
think?
> > >
> > > Roz
> > >
> > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn MacCracken - NOAA
Affiliate <
> > > rosalyn.maccracken at noaa.gov> wrote:
> > >
> > > > Hi John,
> > > >
> > > > I double checked all those things you listed, and they are all
> correct.
> > > > So, I've put a file in the /maccracken directory on your ftp
site.
> Let
> > > me
> > > > know what you see.
> > > >
> > > > Thanks for your help!
> > > >
> > > > Roz
> > > >
> > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn MacCracken - NOAA
Affiliate
> <
> > > > rosalyn.maccracken at noaa.gov> wrote:
> > > >
> > > >> Hi John,
> > > >>
> > > >> I'm sorry I didn't get back to you sooner, but, I was tied up
with
> > some
> > > >> other things. Let me double check with all of those items in
my
> file
> > > and
> > > >> make sure that I don't have some sort of obvious error. I'll
be
> back
> > in
> > > >> touch after I check on this.
> > > >>
> > > >> Roz
> > > >>
> > > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway via RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >>> Hi Roz,
> > > >>>
> > > >>> I see you're having trouble getting with STAT-Analysis,
getting MPR
> > > lines
> > > >>> to be included in the column thresholds you've defined. I
don't
> see
> > > >>> anything obviously wrong with what you've defined.
> > > >>>
> > > >>> The warning message indicates that the filters you defined
have
> > > discarded
> > > >>> all of the input MPR lines.
> > > >>>
> > > >>> We'd just need to go through your filtering options one-by-
one to
> > find
> > > >>> out
> > > >>> why.
> > > >>>
> > > >>> - Are you passing STAT-Analysis files which contain MPR
lines?
> > > >>> - Does the FCST_VAR column say "WIND"?
> > > >>> - Does the FCST_LEV column say "Z10"?
> > > >>> - Does the OBTYPE column say "ASCAT"?
> > > >>> - Does the OBS column have values > 1.0?
> > > >>> - What are the ranges of the OBS_LAT and OBS_LON columns?
> > > >>>
> > > >>> If you'd like to send me a sample .stat file, I could
probably
> track
> > it
> > > >>> down.
> > > >>>
> > > >>> Thanks,
> > > >>> John
> > > >>>
> > > >>>
> > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > >>> RT <
> > > >>> met_help at ucar.edu> wrote:
> > > >>>
> > > >>> >
> > > >>> > Mon Aug 06 07:43:46 2018: Request 86480 was acted upon.
> > > >>> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > > >>> > Queue: met_help
> > > >>> > Subject: Errors using -column_thresh
> > > >>> > Owner: Nobody
> > > >>> > Requestors: rosalyn.maccracken at noaa.gov
> > > >>> > Status: new
> > > >>> > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > Hi,
> > > >>> >
> > > >>> > I'm trying to use -column_thresh with stat_analysis to
mask out a
> > > >>> lat/lon
> > > >>> > box. I've added this command in my StatAnalysis config
file:
> > > >>> >
> > > >>> > "-job aggregate_stat -out_line_type CTC -column_thresh OBS
gt1.0
> > > >>> > -out_thresh ge5.5689 \
> > > >>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> > > >>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
> > > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > >>> > -out_stat /opc_test/home/opc_test/data/m
> > > >>> et_verif/regen_mar2018_data/
> > > >>> > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > >>> >
> > > >>> > But, when it runs, I get these errors:
> > > >>> >
> > > >>> > Output:
> > > >>> > DEBUG 2: Processing Job 2: -job aggregate_stat -fcst_var
WIND
> > > >>> -fcst_lev Z10
> > > >>> > -obtype ASCAT -line_type MPR -column_thresh OBS >1.0
> -column_thresh
> > > >>> OBS_LAT
> > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-10.0
-by
> > > >>> FCST_VALID_BEG
> > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
/opc_test/home/opc_test/data/
> > > >>> >
> > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
> > > >>> _to_CTC_ge5.5689.stat
> > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > >>> > DEBUG 1: Creating STAT output file
"/opc_test/home/opc_test/data/
> > > >>> > met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > >>> > GSL_RNG_TYPE=mt19937
> > > >>> > GSL_RNG_SEED=18446744073401622619
> > > >>> > DEBUG 2: Computing output for 0 case(s).
> > > >>> > WARNING:
> > > >>> > WARNING: do_job_aggr_stat() -> no matching STAT lines
found for
> > job:
> > > >>> -job
> > > >>> > aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype ASCAT
> > -line_type
> > > >>> MPR
> > > >>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT
>=15.0&&<=70.0
> > > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
FCST_VALID_BEG -by
> > > >>> FCST_LEAD
> > > >>> > -by VX_MASK -out_stat /opc_test/home/opc_test/data/
> > > >>> >
> > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
> > > >>> _to_CTC_ge5.5689.stat
> > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > >>> > WARNING:
> > > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> > > >>> > DEBUG 2:
> > > >>> >
> > > >>> > Am I not using the command properly?
> > > >>> >
> > > >>> > Thanks!
> > > >>> >
> > > >>> > Roz
> > > >>> >
> > > >>> > --
> > > >>> > Rosalyn MacCracken
> > > >>> > Support Scientist
> > > >>> >
> > > >>> > Ocean Applications Branch
> > > >>> > NOAA/NWS Ocean Prediction Center
> > > >>> > NCWCP
> > > >>> > 5830 University Research Ct
> > > >>> > College Park, MD 20740-3818
> > > >>> >
> > > >>> > (p) 301-683-1551
> > > >>> > rosalyn.maccracken at noaa.gov
> > > >>> >
> > > >>> >
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Rosalyn MacCracken
> > > >> Support Scientist
> > > >>
> > > >> Ocean Applications Branch
> > > >> NOAA/NWS Ocean Prediction Center
> > > >> NCWCP
> > > >> 5830 University Research Ct
> > > >> College Park, MD 20740-3818
> > > >>
> > > >> (p) 301-683-1551
> > > >> rosalyn.maccracken at noaa.gov
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Mon Aug 13 11:55:26 2018
Hi John,
Yeah, I can repost the file. I'll do that in a bit. Actually, if I
can
get this issue solved, you don't need to look at the file, because
I'll
just reprocess that data with my new method. So, I'll hold off on
ftping
that file to you for right now.
So, the OBS_QC column has N/A values. That's easy enough to replace.
Now,
I have the original ASCAT data, which looks for the quality flag as a
bit
(i.e., ibits[5]) and then doesn't use that. It doesn't write it to
the
output file. Maybe what I should do is find the first bad observation
(time, lat, lon, value), then, match it exactly to the line in the mpr
file, and replace the observation.
So, if my observations are in dataframes, and the mpr file has been
read in
as a dataframe, then, I could do a df.query('Lat = ?? and Lon =??), or
however the syntax is...no wait, I need an if statement to compare
first...I'll work on those detail. Then, once I get the OBS_QC value
changed, then what? How do you get the other stat files from this
point?
Roz
On Mon, Aug 13, 2018 at 5:32 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Roz,
>
> That's correct, there is no tool in MET for comparing one set of
point
> observations to another set of point observations. Presumably you
could
> write point observations to a grid and then reconfigure and rerun
> Point-Stat, but that'd be pretty difficult, time consuming, and
fraught
> with problems.
>
> You currently have many, many MPR lines and the issue is the the
OBS_QC
> column is empty. If that OBS_QC column were *not* empty then it'd
be
> pretty easy to process this data... correct? One option would be
writing a
> script to "patch" that column. For each MPR line, do some
processing to
> figure out what that OBS_QC column should be and then update it's
value.
>
> Then the question is this... is there a way of getting quality
control
> values from the MGDRlite files?
>
> I'm sorry, I hadn't had a chance to look at that data file yet. But
I just
> checked on the ftp site and don't see a maccracken sub-directory
there:
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
>
> Could you please repost?
>
> Thanks,
> John
>
> On Mon, Aug 13, 2018 at 10:57 AM Rosalyn MacCracken - NOAA Affiliate
via RT
> <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> >
> > So, you can't compare two point files, even if both of them are
NetCDF
> > format? Is there a way to write point observations out to a grid?
In
> > other words, take the point GFS observations, and write them to a
global
> > grid (using some other tool, like wgrib or something else), and
fill in
> > missing values with NaN, or something? Then, use that gridded
file to
> > compare with the ASCAT NetCDF file?
> >
> > I don't have anything in the OBS_QC column because when I wrote it
into
> the
> > ASCII file, I didn't have any flags from my MGDRlite files. This
is part
> > of the problem...
> >
> > Now, your answer for the STAT-Analysis, is that what I did wrong
in my
> use
> > of stat-analysis from the previous email? Or, did you not get a
chance
> to
> > look at my *stat file that I uploaded to the ftp site?
> >
> > Roz
> >
> > On Mon, Aug 13, 2018 at 3:42 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Roz,
> > >
> > > Point-Stat takes as input a gridded NWP forecast file and the
NetCDF
> > output
> > > of a point pre-processing tool (like ascii2nc or pb2nc). Based
on your
> > > description, it doesn't sound like you have gridded NWP forecast
files
> > > available. So no, I don't think that approach would work.
> > >
> > > The MPR output line type does include a column named "OBS_QC"
which is
> a
> > > string containing the observation quality control value. I'd
suggest
> > > checking the contents of that column in your MPR output lines to
see if
> > > those values could be filtered to do the quality control
filtering
> you're
> > > after. Check to see if the values in that column are strings or
> numbers
> > > (in PREPBUFR they're integers... in MADIS they're strings).
> > >
> > > In STAT-Analysis, you can use the "-column_thresh" option to
threshold
> a
> > > numeric column or you can use the "-column_str" option to list
strings
> to
> > > retained. For example:
> > > -column_thresh OBS_QC 'ge0&&le3' # this keeps all values
between 0
> > and
> > > 3
> > > -column_str OBS_QC aa,ab,ac # this keeps strings
"aa", "ab",
> > or
> > > "ac"
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken - NOAA
Affiliate via
> > RT
> > > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
>
> > > >
> > > > Hi John,
> > > >
> > > > I have another question related to my reprocessing of my data.
> > > >
> > > > Ok, so, I realized that when I switched ASCAT data sources
back in
> > March,
> > > > from prepbufr to MGDRlite files, I should have been using
quality
> flags
> > > to
> > > > filter out my data. Because I wasn't checking for the quality
flags,
> > my
> > > > statistics were not so good, since it was letting in bad data,
and
> > stats
> > > > were being calculated from this bad data.
> > > >
> > > > So, I need to find a way to correct this, but, the old story
is I
> don't
> > > > have the original GFS data, and it's too massive to grab from
HPSS.
> I
> > do
> > > > have the original ASCAT files, which I can run through my
program to
> > > filter
> > > > out the bad observations, as well as matched points from the
*mpr and
> > > *stat
> > > > files.
> > > >
> > > > Originally, I was just going to filter out using that
-column_thresh
> > > > command, and filter out over land. But, I had this idea that
I
> wanted
> > to
> > > > run by you, to see if this makes sense to do.
> > > >
> > > > I have the time, lat, lon and GFS observation and the
corresponding
> > ASCAT
> > > > observation point. Can I do this:
> > > > 1) read the time, lat, lon and GFS observation out of the mpr
file,
> > and
> > > > make a new input file with just those variables in the ASCII
format
> > that
> > > > MET can use
> > > > 2) reprocess my ASCAT data and write it into an ASCII file in
MET
> > format
> > > > 3) run ASCII2NC on these files, to create the two input files
for
> MET,
> > > > then, run point_stat on the netcdf files to regenerate the new
> > corrected
> > > > stat files
> > > >
> > > > Does that make sense to do? I know it could be alot of work,
but,
> if I
> > > can
> > > > get corrected statistics, it might be worth it. What do you
think?
> > > >
> > > > Roz
> > > >
> > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn MacCracken - NOAA
Affiliate
> <
> > > > rosalyn.maccracken at noaa.gov> wrote:
> > > >
> > > > > Hi John,
> > > > >
> > > > > I double checked all those things you listed, and they are
all
> > correct.
> > > > > So, I've put a file in the /maccracken directory on your ftp
site.
> > Let
> > > > me
> > > > > know what you see.
> > > > >
> > > > > Thanks for your help!
> > > > >
> > > > > Roz
> > > > >
> > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn MacCracken - NOAA
> Affiliate
> > <
> > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > >
> > > > >> Hi John,
> > > > >>
> > > > >> I'm sorry I didn't get back to you sooner, but, I was tied
up with
> > > some
> > > > >> other things. Let me double check with all of those items
in my
> > file
> > > > and
> > > > >> make sure that I don't have some sort of obvious error.
I'll be
> > back
> > > in
> > > > >> touch after I check on this.
> > > > >>
> > > > >> Roz
> > > > >>
> > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway via RT <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >>> Hi Roz,
> > > > >>>
> > > > >>> I see you're having trouble getting with STAT-Analysis,
getting
> MPR
> > > > lines
> > > > >>> to be included in the column thresholds you've defined. I
don't
> > see
> > > > >>> anything obviously wrong with what you've defined.
> > > > >>>
> > > > >>> The warning message indicates that the filters you defined
have
> > > > discarded
> > > > >>> all of the input MPR lines.
> > > > >>>
> > > > >>> We'd just need to go through your filtering options one-
by-one to
> > > find
> > > > >>> out
> > > > >>> why.
> > > > >>>
> > > > >>> - Are you passing STAT-Analysis files which contain MPR
lines?
> > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > >>> - Does the OBS column have values > 1.0?
> > > > >>> - What are the ranges of the OBS_LAT and OBS_LON columns?
> > > > >>>
> > > > >>> If you'd like to send me a sample .stat file, I could
probably
> > track
> > > it
> > > > >>> down.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> John
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken - NOAA
> Affiliate
> > > via
> > > > >>> RT <
> > > > >>> met_help at ucar.edu> wrote:
> > > > >>>
> > > > >>> >
> > > > >>> > Mon Aug 06 07:43:46 2018: Request 86480 was acted upon.
> > > > >>> > Transaction: Ticket created by
rosalyn.maccracken at noaa.gov
> > > > >>> > Queue: met_help
> > > > >>> > Subject: Errors using -column_thresh
> > > > >>> > Owner: Nobody
> > > > >>> > Requestors: rosalyn.maccracken at noaa.gov
> > > > >>> > Status: new
> > > > >>> > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > >>> >
> > > > >>> >
> > > > >>> >
> > > > >>> > Hi,
> > > > >>> >
> > > > >>> > I'm trying to use -column_thresh with stat_analysis to
mask
> out a
> > > > >>> lat/lon
> > > > >>> > box. I've added this command in my StatAnalysis config
file:
> > > > >>> >
> > > > >>> > "-job aggregate_stat -out_line_type CTC -column_thresh
OBS
> gt1.0
> > > > >>> > -out_thresh ge5.5689 \
> > > > >>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> > > > >>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
> > > > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > > >>> > -out_stat /opc_test/home/opc_test/data/m
> > > > >>> et_verif/regen_mar2018_data/
> > > > >>> > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > >>> >
> > > > >>> > But, when it runs, I get these errors:
> > > > >>> >
> > > > >>> > Output:
> > > > >>> > DEBUG 2: Processing Job 2: -job aggregate_stat -fcst_var
WIND
> > > > >>> -fcst_lev Z10
> > > > >>> > -obtype ASCAT -line_type MPR -column_thresh OBS >1.0
> > -column_thresh
> > > > >>> OBS_LAT
> > > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-10.0
-by
> > > > >>> FCST_VALID_BEG
> > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> /opc_test/home/opc_test/data/
> > > > >>> >
> > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
> > > > >>> _to_CTC_ge5.5689.stat
> > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > >>> > DEBUG 1: Creating STAT output file
> "/opc_test/home/opc_test/data/
> > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > >>> > GSL_RNG_TYPE=mt19937
> > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > >>> > DEBUG 2: Computing output for 0 case(s).
> > > > >>> > WARNING:
> > > > >>> > WARNING: do_job_aggr_stat() -> no matching STAT lines
found for
> > > job:
> > > > >>> -job
> > > > >>> > aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype
ASCAT
> > > -line_type
> > > > >>> MPR
> > > > >>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT
>=15.0&&<=70.0
> > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
FCST_VALID_BEG -by
> > > > >>> FCST_LEAD
> > > > >>> > -by VX_MASK -out_stat /opc_test/home/opc_test/data/
> > > > >>> >
> > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
> > > > >>> _to_CTC_ge5.5689.stat
> > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > >>> > WARNING:
> > > > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> > > > >>> > DEBUG 2:
> > > > >>> >
> > > > >>> > Am I not using the command properly?
> > > > >>> >
> > > > >>> > Thanks!
> > > > >>> >
> > > > >>> > Roz
> > > > >>> >
> > > > >>> > --
> > > > >>> > Rosalyn MacCracken
> > > > >>> > Support Scientist
> > > > >>> >
> > > > >>> > Ocean Applications Branch
> > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > >>> > NCWCP
> > > > >>> > 5830 University Research Ct
> > > > >>> > College Park, MD 20740-3818
> > > > >>> >
> > > > >>> > (p) 301-683-1551
> > > > >>> > rosalyn.maccracken at noaa.gov
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Rosalyn MacCracken
> > > > >> Support Scientist
> > > > >>
> > > > >> Ocean Applications Branch
> > > > >> NOAA/NWS Ocean Prediction Center
> > > > >> NCWCP
> > > > >> 5830 University Research Ct
> > > > >> College Park, MD 20740-3818
> > > > >>
> > > > >> (p) 301-683-1551
> > > > >> rosalyn.maccracken at noaa.gov
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applications Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Errors using -column_thresh
From: John Halley Gotway
Time: Mon Aug 13 12:30:34 2018
Roz,
Once the OBS_QC values are set in the MPR lines, you can use STAT-
Analysis
to reprocess the data and recompute whatever other STAT lines you'd
like to
compute:
stat_analysis -job aggregate_stat -line_type MPR -out_line_type CNT
-column_str OBS_QC a,b,c
This job would be run only on MPR lines that have strings of "a", "b",
or
"c" in the OBS_QC column. Make sense?
John
On Mon, Aug 13, 2018 at 11:55 AM Rosalyn MacCracken - NOAA Affiliate
via RT
<met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
>
> Hi John,
>
> Yeah, I can repost the file. I'll do that in a bit. Actually, if I
can
> get this issue solved, you don't need to look at the file, because
I'll
> just reprocess that data with my new method. So, I'll hold off on
ftping
> that file to you for right now.
>
> So, the OBS_QC column has N/A values. That's easy enough to
replace. Now,
> I have the original ASCAT data, which looks for the quality flag as
a bit
> (i.e., ibits[5]) and then doesn't use that. It doesn't write it to
the
> output file. Maybe what I should do is find the first bad
observation
> (time, lat, lon, value), then, match it exactly to the line in the
mpr
> file, and replace the observation.
>
> So, if my observations are in dataframes, and the mpr file has been
read in
> as a dataframe, then, I could do a df.query('Lat = ?? and Lon =??),
or
> however the syntax is...no wait, I need an if statement to compare
> first...I'll work on those detail. Then, once I get the OBS_QC
value
> changed, then what? How do you get the other stat files from this
point?
>
> Roz
>
>
> On Mon, Aug 13, 2018 at 5:32 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Roz,
> >
> > That's correct, there is no tool in MET for comparing one set of
point
> > observations to another set of point observations. Presumably you
could
> > write point observations to a grid and then reconfigure and rerun
> > Point-Stat, but that'd be pretty difficult, time consuming, and
fraught
> > with problems.
> >
> > You currently have many, many MPR lines and the issue is the the
OBS_QC
> > column is empty. If that OBS_QC column were *not* empty then it'd
be
> > pretty easy to process this data... correct? One option would be
> writing a
> > script to "patch" that column. For each MPR line, do some
processing to
> > figure out what that OBS_QC column should be and then update it's
value.
> >
> > Then the question is this... is there a way of getting quality
control
> > values from the MGDRlite files?
> >
> > I'm sorry, I hadn't had a chance to look at that data file yet.
But I
> just
> > checked on the ftp site and don't see a maccracken sub-directory
there:
> > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> >
> > Could you please repost?
> >
> > Thanks,
> > John
> >
> > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > >
> > > So, you can't compare two point files, even if both of them are
NetCDF
> > > format? Is there a way to write point observations out to a
grid? In
> > > other words, take the point GFS observations, and write them to
a
> global
> > > grid (using some other tool, like wgrib or something else), and
fill in
> > > missing values with NaN, or something? Then, use that gridded
file to
> > > compare with the ASCAT NetCDF file?
> > >
> > > I don't have anything in the OBS_QC column because when I wrote
it into
> > the
> > > ASCII file, I didn't have any flags from my MGDRlite files.
This is
> part
> > > of the problem...
> > >
> > > Now, your answer for the STAT-Analysis, is that what I did wrong
in my
> > use
> > > of stat-analysis from the previous email? Or, did you not get a
chance
> > to
> > > look at my *stat file that I uploaded to the ftp site?
> > >
> > > Roz
> > >
> > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Roz,
> > > >
> > > > Point-Stat takes as input a gridded NWP forecast file and the
NetCDF
> > > output
> > > > of a point pre-processing tool (like ascii2nc or pb2nc).
Based on
> your
> > > > description, it doesn't sound like you have gridded NWP
forecast
> files
> > > > available. So no, I don't think that approach would work.
> > > >
> > > > The MPR output line type does include a column named "OBS_QC"
which
> is
> > a
> > > > string containing the observation quality control value. I'd
suggest
> > > > checking the contents of that column in your MPR output lines
to see
> if
> > > > those values could be filtered to do the quality control
filtering
> > you're
> > > > after. Check to see if the values in that column are strings
or
> > numbers
> > > > (in PREPBUFR they're integers... in MADIS they're strings).
> > > >
> > > > In STAT-Analysis, you can use the "-column_thresh" option to
> threshold
> > a
> > > > numeric column or you can use the "-column_str" option to list
> strings
> > to
> > > > retained. For example:
> > > > -column_thresh OBS_QC 'ge0&&le3' # this keeps all values
between
> 0
> > > and
> > > > 3
> > > > -column_str OBS_QC aa,ab,ac # this keeps strings
"aa",
> "ab",
> > > or
> > > > "ac"
> > > >
> > > > Hope that helps.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > I have another question related to my reprocessing of my
data.
> > > > >
> > > > > Ok, so, I realized that when I switched ASCAT data sources
back in
> > > March,
> > > > > from prepbufr to MGDRlite files, I should have been using
quality
> > flags
> > > > to
> > > > > filter out my data. Because I wasn't checking for the
quality
> flags,
> > > my
> > > > > statistics were not so good, since it was letting in bad
data, and
> > > stats
> > > > > were being calculated from this bad data.
> > > > >
> > > > > So, I need to find a way to correct this, but, the old story
is I
> > don't
> > > > > have the original GFS data, and it's too massive to grab
from HPSS.
> > I
> > > do
> > > > > have the original ASCAT files, which I can run through my
program
> to
> > > > filter
> > > > > out the bad observations, as well as matched points from the
*mpr
> and
> > > > *stat
> > > > > files.
> > > > >
> > > > > Originally, I was just going to filter out using that
> -column_thresh
> > > > > command, and filter out over land. But, I had this idea
that I
> > wanted
> > > to
> > > > > run by you, to see if this makes sense to do.
> > > > >
> > > > > I have the time, lat, lon and GFS observation and the
corresponding
> > > ASCAT
> > > > > observation point. Can I do this:
> > > > > 1) read the time, lat, lon and GFS observation out of the
mpr
> file,
> > > and
> > > > > make a new input file with just those variables in the ASCII
format
> > > that
> > > > > MET can use
> > > > > 2) reprocess my ASCAT data and write it into an ASCII file
in MET
> > > format
> > > > > 3) run ASCII2NC on these files, to create the two input
files for
> > MET,
> > > > > then, run point_stat on the netcdf files to regenerate the
new
> > > corrected
> > > > > stat files
> > > > >
> > > > > Does that make sense to do? I know it could be alot of
work, but,
> > if I
> > > > can
> > > > > get corrected statistics, it might be worth it. What do you
think?
> > > > >
> > > > > Roz
> > > > >
> > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn MacCracken - NOAA
> Affiliate
> > <
> > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > I double checked all those things you listed, and they are
all
> > > correct.
> > > > > > So, I've put a file in the /maccracken directory on your
ftp
> site.
> > > Let
> > > > > me
> > > > > > know what you see.
> > > > > >
> > > > > > Thanks for your help!
> > > > > >
> > > > > > Roz
> > > > > >
> > > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn MacCracken - NOAA
> > Affiliate
> > > <
> > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > >
> > > > > >> Hi John,
> > > > > >>
> > > > > >> I'm sorry I didn't get back to you sooner, but, I was
tied up
> with
> > > > some
> > > > > >> other things. Let me double check with all of those
items in my
> > > file
> > > > > and
> > > > > >> make sure that I don't have some sort of obvious error.
I'll be
> > > back
> > > > in
> > > > > >> touch after I check on this.
> > > > > >>
> > > > > >> Roz
> > > > > >>
> > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway via RT
<
> > > > > >> met_help at ucar.edu> wrote:
> > > > > >>
> > > > > >>> Hi Roz,
> > > > > >>>
> > > > > >>> I see you're having trouble getting with STAT-Analysis,
getting
> > MPR
> > > > > lines
> > > > > >>> to be included in the column thresholds you've defined.
I
> don't
> > > see
> > > > > >>> anything obviously wrong with what you've defined.
> > > > > >>>
> > > > > >>> The warning message indicates that the filters you
defined have
> > > > > discarded
> > > > > >>> all of the input MPR lines.
> > > > > >>>
> > > > > >>> We'd just need to go through your filtering options one-
by-one
> to
> > > > find
> > > > > >>> out
> > > > > >>> why.
> > > > > >>>
> > > > > >>> - Are you passing STAT-Analysis files which contain MPR
lines?
> > > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > > >>> - Does the OBS column have values > 1.0?
> > > > > >>> - What are the ranges of the OBS_LAT and OBS_LON
columns?
> > > > > >>>
> > > > > >>> If you'd like to send me a sample .stat file, I could
probably
> > > track
> > > > it
> > > > > >>> down.
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> John
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken - NOAA
> > Affiliate
> > > > via
> > > > > >>> RT <
> > > > > >>> met_help at ucar.edu> wrote:
> > > > > >>>
> > > > > >>> >
> > > > > >>> > Mon Aug 06 07:43:46 2018: Request 86480 was acted
upon.
> > > > > >>> > Transaction: Ticket created by
rosalyn.maccracken at noaa.gov
> > > > > >>> > Queue: met_help
> > > > > >>> > Subject: Errors using -column_thresh
> > > > > >>> > Owner: Nobody
> > > > > >>> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > >>> > Status: new
> > > > > >>> > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > >>> >
> > > > > >>> >
> > > > > >>> >
> > > > > >>> > Hi,
> > > > > >>> >
> > > > > >>> > I'm trying to use -column_thresh with stat_analysis to
mask
> > out a
> > > > > >>> lat/lon
> > > > > >>> > box. I've added this command in my StatAnalysis
config file:
> > > > > >>> >
> > > > > >>> > "-job aggregate_stat -out_line_type CTC -column_thresh
OBS
> > gt1.0
> > > > > >>> > -out_thresh ge5.5689 \
> > > > > >>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> > > > > >>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
> > > > > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > > > >>> > -out_stat /opc_test/home/opc_test/data/m
> > > > > >>> et_verif/regen_mar2018_data/
> > > > > >>> > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > >>> >
> > > > > >>> > But, when it runs, I get these errors:
> > > > > >>> >
> > > > > >>> > Output:
> > > > > >>> > DEBUG 2: Processing Job 2: -job aggregate_stat
-fcst_var WIND
> > > > > >>> -fcst_lev Z10
> > > > > >>> > -obtype ASCAT -line_type MPR -column_thresh OBS >1.0
> > > -column_thresh
> > > > > >>> OBS_LAT
> > > > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-
10.0 -by
> > > > > >>> FCST_VALID_BEG
> > > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> > /opc_test/home/opc_test/data/
> > > > > >>> >
> > > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
> > > > > >>> _to_CTC_ge5.5689.stat
> > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > >>> > DEBUG 1: Creating STAT output file
> > "/opc_test/home/opc_test/data/
> > > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > >>> > DEBUG 2: Computing output for 0 case(s).
> > > > > >>> > WARNING:
> > > > > >>> > WARNING: do_job_aggr_stat() -> no matching STAT lines
found
> for
> > > > job:
> > > > > >>> -job
> > > > > >>> > aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype
ASCAT
> > > > -line_type
> > > > > >>> MPR
> > > > > >>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT
>=15.0&&<=70.0
> > > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
FCST_VALID_BEG
> -by
> > > > > >>> FCST_LEAD
> > > > > >>> > -by VX_MASK -out_stat /opc_test/home/opc_test/data/
> > > > > >>> >
> > > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_MPR
> > > > > >>> _to_CTC_ge5.5689.stat
> > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > >>> > WARNING:
> > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> > > > > >>> > DEBUG 2:
> > > > > >>> >
> > > > > >>> > Am I not using the command properly?
> > > > > >>> >
> > > > > >>> > Thanks!
> > > > > >>> >
> > > > > >>> > Roz
> > > > > >>> >
> > > > > >>> > --
> > > > > >>> > Rosalyn MacCracken
> > > > > >>> > Support Scientist
> > > > > >>> >
> > > > > >>> > Ocean Applications Branch
> > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > >>> > NCWCP
> > > > > >>> > 5830 University Research Ct
> > > > > >>> > College Park, MD 20740-3818
> > > > > >>> >
> > > > > >>> > (p) 301-683-1551
> > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > >>> >
> > > > > >>> >
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Rosalyn MacCracken
> > > > > >> Support Scientist
> > > > > >>
> > > > > >> Ocean Applications Branch
> > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > >> NCWCP
> > > > > >> 5830 University Research Ct
> > > > > >> College Park, MD 20740-3818
> > > > > >>
> > > > > >> (p) 301-683-1551
> > > > > >> rosalyn.maccracken at noaa.gov
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applications Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applications Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Mon Aug 13 12:37:16 2018
Yes, that makes sense. So, the MPR line would be "N/A" that it would
process the data with. Ok. And, I could do something similar to what
I
did before with copying my MPR files to a temp directory, and use
-lookin.
Ok, so, the hard part will be to write this script that replaces my
bad
values. I guess I like a challenge, right?
Roz
On Mon, Aug 13, 2018 at 6:30 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Roz,
>
> Once the OBS_QC values are set in the MPR lines, you can use STAT-
Analysis
> to reprocess the data and recompute whatever other STAT lines you'd
like to
> compute:
>
> stat_analysis -job aggregate_stat -line_type MPR -out_line_type CNT
> -column_str OBS_QC a,b,c
>
> This job would be run only on MPR lines that have strings of "a",
"b", or
> "c" in the OBS_QC column. Make sense?
>
> John
>
> On Mon, Aug 13, 2018 at 11:55 AM Rosalyn MacCracken - NOAA Affiliate
via RT
> <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> >
> > Hi John,
> >
> > Yeah, I can repost the file. I'll do that in a bit. Actually, if
I can
> > get this issue solved, you don't need to look at the file, because
I'll
> > just reprocess that data with my new method. So, I'll hold off on
ftping
> > that file to you for right now.
> >
> > So, the OBS_QC column has N/A values. That's easy enough to
replace.
> Now,
> > I have the original ASCAT data, which looks for the quality flag
as a bit
> > (i.e., ibits[5]) and then doesn't use that. It doesn't write it
to the
> > output file. Maybe what I should do is find the first bad
observation
> > (time, lat, lon, value), then, match it exactly to the line in the
mpr
> > file, and replace the observation.
> >
> > So, if my observations are in dataframes, and the mpr file has
been read
> in
> > as a dataframe, then, I could do a df.query('Lat = ?? and Lon
=??), or
> > however the syntax is...no wait, I need an if statement to compare
> > first...I'll work on those detail. Then, once I get the OBS_QC
value
> > changed, then what? How do you get the other stat files from this
point?
> >
> > Roz
> >
> >
> > On Mon, Aug 13, 2018 at 5:32 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Roz,
> > >
> > > That's correct, there is no tool in MET for comparing one set of
point
> > > observations to another set of point observations. Presumably
you
> could
> > > write point observations to a grid and then reconfigure and
rerun
> > > Point-Stat, but that'd be pretty difficult, time consuming, and
fraught
> > > with problems.
> > >
> > > You currently have many, many MPR lines and the issue is the the
OBS_QC
> > > column is empty. If that OBS_QC column were *not* empty then
it'd be
> > > pretty easy to process this data... correct? One option would
be
> > writing a
> > > script to "patch" that column. For each MPR line, do some
processing
> to
> > > figure out what that OBS_QC column should be and then update
it's
> value.
> > >
> > > Then the question is this... is there a way of getting quality
control
> > > values from the MGDRlite files?
> > >
> > > I'm sorry, I hadn't had a chance to look at that data file yet.
But I
> > just
> > > checked on the ftp site and don't see a maccracken sub-directory
there:
> > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > >
> > > Could you please repost?
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn MacCracken - NOAA
Affiliate
> via
> > RT
> > > <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
>
> > > >
> > > > So, you can't compare two point files, even if both of them
are
> NetCDF
> > > > format? Is there a way to write point observations out to a
grid?
> In
> > > > other words, take the point GFS observations, and write them
to a
> > global
> > > > grid (using some other tool, like wgrib or something else),
and fill
> in
> > > > missing values with NaN, or something? Then, use that gridded
file
> to
> > > > compare with the ASCAT NetCDF file?
> > > >
> > > > I don't have anything in the OBS_QC column because when I
wrote it
> into
> > > the
> > > > ASCII file, I didn't have any flags from my MGDRlite files.
This is
> > part
> > > > of the problem...
> > > >
> > > > Now, your answer for the STAT-Analysis, is that what I did
wrong in
> my
> > > use
> > > > of stat-analysis from the previous email? Or, did you not get
a
> chance
> > > to
> > > > look at my *stat file that I uploaded to the ftp site?
> > > >
> > > > Roz
> > > >
> > > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Roz,
> > > > >
> > > > > Point-Stat takes as input a gridded NWP forecast file and
the
> NetCDF
> > > > output
> > > > > of a point pre-processing tool (like ascii2nc or pb2nc).
Based on
> > your
> > > > > description, it doesn't sound like you have gridded NWP
forecast
> > files
> > > > > available. So no, I don't think that approach would work.
> > > > >
> > > > > The MPR output line type does include a column named
"OBS_QC" which
> > is
> > > a
> > > > > string containing the observation quality control value.
I'd
> suggest
> > > > > checking the contents of that column in your MPR output
lines to
> see
> > if
> > > > > those values could be filtered to do the quality control
filtering
> > > you're
> > > > > after. Check to see if the values in that column are
strings or
> > > numbers
> > > > > (in PREPBUFR they're integers... in MADIS they're strings).
> > > > >
> > > > > In STAT-Analysis, you can use the "-column_thresh" option to
> > threshold
> > > a
> > > > > numeric column or you can use the "-column_str" option to
list
> > strings
> > > to
> > > > > retained. For example:
> > > > > -column_thresh OBS_QC 'ge0&&le3' # this keeps all values
> between
> > 0
> > > > and
> > > > > 3
> > > > > -column_str OBS_QC aa,ab,ac # this keeps strings
"aa",
> > "ab",
> > > > or
> > > > > "ac"
> > > > >
> > > > > Hope that helps.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > I have another question related to my reprocessing of my
data.
> > > > > >
> > > > > > Ok, so, I realized that when I switched ASCAT data sources
back
> in
> > > > March,
> > > > > > from prepbufr to MGDRlite files, I should have been using
quality
> > > flags
> > > > > to
> > > > > > filter out my data. Because I wasn't checking for the
quality
> > flags,
> > > > my
> > > > > > statistics were not so good, since it was letting in bad
data,
> and
> > > > stats
> > > > > > were being calculated from this bad data.
> > > > > >
> > > > > > So, I need to find a way to correct this, but, the old
story is I
> > > don't
> > > > > > have the original GFS data, and it's too massive to grab
from
> HPSS.
> > > I
> > > > do
> > > > > > have the original ASCAT files, which I can run through my
program
> > to
> > > > > filter
> > > > > > out the bad observations, as well as matched points from
the *mpr
> > and
> > > > > *stat
> > > > > > files.
> > > > > >
> > > > > > Originally, I was just going to filter out using that
> > -column_thresh
> > > > > > command, and filter out over land. But, I had this idea
that I
> > > wanted
> > > > to
> > > > > > run by you, to see if this makes sense to do.
> > > > > >
> > > > > > I have the time, lat, lon and GFS observation and the
> corresponding
> > > > ASCAT
> > > > > > observation point. Can I do this:
> > > > > > 1) read the time, lat, lon and GFS observation out of the
mpr
> > file,
> > > > and
> > > > > > make a new input file with just those variables in the
ASCII
> format
> > > > that
> > > > > > MET can use
> > > > > > 2) reprocess my ASCAT data and write it into an ASCII
file in
> MET
> > > > format
> > > > > > 3) run ASCII2NC on these files, to create the two input
files
> for
> > > MET,
> > > > > > then, run point_stat on the netcdf files to regenerate the
new
> > > > corrected
> > > > > > stat files
> > > > > >
> > > > > > Does that make sense to do? I know it could be alot of
work,
> but,
> > > if I
> > > > > can
> > > > > > get corrected statistics, it might be worth it. What do
you
> think?
> > > > > >
> > > > > > Roz
> > > > > >
> > > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn MacCracken - NOAA
> > Affiliate
> > > <
> > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > I double checked all those things you listed, and they
are all
> > > > correct.
> > > > > > > So, I've put a file in the /maccracken directory on your
ftp
> > site.
> > > > Let
> > > > > > me
> > > > > > > know what you see.
> > > > > > >
> > > > > > > Thanks for your help!
> > > > > > >
> > > > > > > Roz
> > > > > > >
> > > > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn MacCracken -
NOAA
> > > Affiliate
> > > > <
> > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > >
> > > > > > >> Hi John,
> > > > > > >>
> > > > > > >> I'm sorry I didn't get back to you sooner, but, I was
tied up
> > with
> > > > > some
> > > > > > >> other things. Let me double check with all of those
items in
> my
> > > > file
> > > > > > and
> > > > > > >> make sure that I don't have some sort of obvious error.
I'll
> be
> > > > back
> > > > > in
> > > > > > >> touch after I check on this.
> > > > > > >>
> > > > > > >> Roz
> > > > > > >>
> > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway via
RT <
> > > > > > >> met_help at ucar.edu> wrote:
> > > > > > >>
> > > > > > >>> Hi Roz,
> > > > > > >>>
> > > > > > >>> I see you're having trouble getting with STAT-
Analysis,
> getting
> > > MPR
> > > > > > lines
> > > > > > >>> to be included in the column thresholds you've
defined. I
> > don't
> > > > see
> > > > > > >>> anything obviously wrong with what you've defined.
> > > > > > >>>
> > > > > > >>> The warning message indicates that the filters you
defined
> have
> > > > > > discarded
> > > > > > >>> all of the input MPR lines.
> > > > > > >>>
> > > > > > >>> We'd just need to go through your filtering options
> one-by-one
> > to
> > > > > find
> > > > > > >>> out
> > > > > > >>> why.
> > > > > > >>>
> > > > > > >>> - Are you passing STAT-Analysis files which contain
MPR
> lines?
> > > > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > > > >>> - Does the OBS column have values > 1.0?
> > > > > > >>> - What are the ranges of the OBS_LAT and OBS_LON
columns?
> > > > > > >>>
> > > > > > >>> If you'd like to send me a sample .stat file, I could
> probably
> > > > track
> > > > > it
> > > > > > >>> down.
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>> John
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken -
NOAA
> > > Affiliate
> > > > > via
> > > > > > >>> RT <
> > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > >>>
> > > > > > >>> >
> > > > > > >>> > Mon Aug 06 07:43:46 2018: Request 86480 was acted
upon.
> > > > > > >>> > Transaction: Ticket created by
rosalyn.maccracken at noaa.gov
> > > > > > >>> > Queue: met_help
> > > > > > >>> > Subject: Errors using -column_thresh
> > > > > > >>> > Owner: Nobody
> > > > > > >>> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > > >>> > Status: new
> > > > > > >>> > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>> > Hi,
> > > > > > >>> >
> > > > > > >>> > I'm trying to use -column_thresh with stat_analysis
to mask
> > > out a
> > > > > > >>> lat/lon
> > > > > > >>> > box. I've added this command in my StatAnalysis
config
> file:
> > > > > > >>> >
> > > > > > >>> > "-job aggregate_stat -out_line_type CTC
-column_thresh OBS
> > > gt1.0
> > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > >>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> > > > > > >>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
> > > > > > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > > > > >>> > -out_stat /opc_test/home/opc_test/data/m
> > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > >>> >
new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > >>> >
> > > > > > >>> > But, when it runs, I get these errors:
> > > > > > >>> >
> > > > > > >>> > Output:
> > > > > > >>> > DEBUG 2: Processing Job 2: -job aggregate_stat
-fcst_var
> WIND
> > > > > > >>> -fcst_lev Z10
> > > > > > >>> > -obtype ASCAT -line_type MPR -column_thresh OBS >1.0
> > > > -column_thresh
> > > > > > >>> OBS_LAT
> > > > > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-100.0&&<=-
10.0 -by
> > > > > > >>> FCST_VALID_BEG
> > > > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> > > /opc_test/home/opc_test/data/
> > > > > > >>> >
> > > > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> MPR
> > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > >>> > DEBUG 1: Creating STAT output file
> > > "/opc_test/home/opc_test/data/
> > > > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > > >>> > DEBUG 2: Computing output for 0 case(s).
> > > > > > >>> > WARNING:
> > > > > > >>> > WARNING: do_job_aggr_stat() -> no matching STAT
lines found
> > for
> > > > > job:
> > > > > > >>> -job
> > > > > > >>> > aggregate_stat -fcst_var WIND -fcst_lev Z10 -obtype
ASCAT
> > > > > -line_type
> > > > > > >>> MPR
> > > > > > >>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT
> >=15.0&&<=70.0
> > > > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
FCST_VALID_BEG
> > -by
> > > > > > >>> FCST_LEAD
> > > > > > >>> > -by VX_MASK -out_stat /opc_test/home/opc_test/data/
> > > > > > >>> >
> > > > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> MPR
> > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > >>> > WARNING:
> > > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> > > > > > >>> > DEBUG 2:
> > > > > > >>> >
> > > > > > >>> > Am I not using the command properly?
> > > > > > >>> >
> > > > > > >>> > Thanks!
> > > > > > >>> >
> > > > > > >>> > Roz
> > > > > > >>> >
> > > > > > >>> > --
> > > > > > >>> > Rosalyn MacCracken
> > > > > > >>> > Support Scientist
> > > > > > >>> >
> > > > > > >>> > Ocean Applications Branch
> > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > >>> > NCWCP
> > > > > > >>> > 5830 University Research Ct
> > > > > > >>> > College Park, MD 20740-3818
> > > > > > >>> >
> > > > > > >>> > (p) 301-683-1551
> > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Rosalyn MacCracken
> > > > > > >> Support Scientist
> > > > > > >>
> > > > > > >> Ocean Applications Branch
> > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > >> NCWCP
> > > > > > >> 5830 University Research Ct
> > > > > > >> College Park, MD 20740-3818
> > > > > > >>
> > > > > > >> (p) 301-683-1551
> > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rosalyn MacCracken
> > > > > > > Support Scientist
> > > > > > >
> > > > > > > Ocean Applications Branch
> > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > NCWCP
> > > > > > > 5830 University Research Ct
> > > > > > > College Park, MD 20740-3818
> > > > > > >
> > > > > > > (p) 301-683-1551
> > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applications Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Errors using -column_thresh
From: John Halley Gotway
Time: Mon Aug 13 12:41:42 2018
Right, updating the OBS_QC values from NA to the real QC value is the
difficult part, but I think that's probably easier that "gridding" the
point data and going back through Point-Stat.
So do you know how to get the QBS_QC value from the MGDRlite data?
And,
going forward, can you set it up so that the OBS_QC column is
populated in
your Point-Stat runs going forward?
Perhaps that's the place to start. Make sure your new Point-Stat
output
includes meaningful OBS_QC output. Once that's working, go through
your
*old* output and backfill the OBS_QC column with what it should
contain.
John
On Mon, Aug 13, 2018 at 12:37 PM Rosalyn MacCracken - NOAA Affiliate
via RT
<met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
>
> Yes, that makes sense. So, the MPR line would be "N/A" that it
would
> process the data with. Ok. And, I could do something similar to
what I
> did before with copying my MPR files to a temp directory, and use
-lookin.
>
> Ok, so, the hard part will be to write this script that replaces my
bad
> values. I guess I like a challenge, right?
>
> Roz
>
> On Mon, Aug 13, 2018 at 6:30 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Roz,
> >
> > Once the OBS_QC values are set in the MPR lines, you can use
> STAT-Analysis
> > to reprocess the data and recompute whatever other STAT lines
you'd like
> to
> > compute:
> >
> > stat_analysis -job aggregate_stat -line_type MPR -out_line_type
CNT
> > -column_str OBS_QC a,b,c
> >
> > This job would be run only on MPR lines that have strings of "a",
"b", or
> > "c" in the OBS_QC column. Make sense?
> >
> > John
> >
> > On Mon, Aug 13, 2018 at 11:55 AM Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > >
> > > Hi John,
> > >
> > > Yeah, I can repost the file. I'll do that in a bit. Actually,
if I
> can
> > > get this issue solved, you don't need to look at the file,
because I'll
> > > just reprocess that data with my new method. So, I'll hold off
on
> ftping
> > > that file to you for right now.
> > >
> > > So, the OBS_QC column has N/A values. That's easy enough to
replace.
> > Now,
> > > I have the original ASCAT data, which looks for the quality flag
as a
> bit
> > > (i.e., ibits[5]) and then doesn't use that. It doesn't write it
to the
> > > output file. Maybe what I should do is find the first bad
observation
> > > (time, lat, lon, value), then, match it exactly to the line in
the mpr
> > > file, and replace the observation.
> > >
> > > So, if my observations are in dataframes, and the mpr file has
been
> read
> > in
> > > as a dataframe, then, I could do a df.query('Lat = ?? and Lon
=??), or
> > > however the syntax is...no wait, I need an if statement to
compare
> > > first...I'll work on those detail. Then, once I get the OBS_QC
value
> > > changed, then what? How do you get the other stat files from
this
> point?
> > >
> > > Roz
> > >
> > >
> > > On Mon, Aug 13, 2018 at 5:32 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Roz,
> > > >
> > > > That's correct, there is no tool in MET for comparing one set
of
> point
> > > > observations to another set of point observations. Presumably
you
> > could
> > > > write point observations to a grid and then reconfigure and
rerun
> > > > Point-Stat, but that'd be pretty difficult, time consuming,
and
> fraught
> > > > with problems.
> > > >
> > > > You currently have many, many MPR lines and the issue is the
the
> OBS_QC
> > > > column is empty. If that OBS_QC column were *not* empty then
it'd be
> > > > pretty easy to process this data... correct? One option would
be
> > > writing a
> > > > script to "patch" that column. For each MPR line, do some
processing
> > to
> > > > figure out what that OBS_QC column should be and then update
it's
> > value.
> > > >
> > > > Then the question is this... is there a way of getting quality
> control
> > > > values from the MGDRlite files?
> > > >
> > > > I'm sorry, I hadn't had a chance to look at that data file
yet. But
> I
> > > just
> > > > checked on the ftp site and don't see a maccracken sub-
directory
> there:
> > > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > > >
> > > > Could you please repost?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > RT
> > > > <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > >
> > > > > So, you can't compare two point files, even if both of them
are
> > NetCDF
> > > > > format? Is there a way to write point observations out to a
grid?
> > In
> > > > > other words, take the point GFS observations, and write them
to a
> > > global
> > > > > grid (using some other tool, like wgrib or something else),
and
> fill
> > in
> > > > > missing values with NaN, or something? Then, use that
gridded file
> > to
> > > > > compare with the ASCAT NetCDF file?
> > > > >
> > > > > I don't have anything in the OBS_QC column because when I
wrote it
> > into
> > > > the
> > > > > ASCII file, I didn't have any flags from my MGDRlite files.
This
> is
> > > part
> > > > > of the problem...
> > > > >
> > > > > Now, your answer for the STAT-Analysis, is that what I did
wrong in
> > my
> > > > use
> > > > > of stat-analysis from the previous email? Or, did you not
get a
> > chance
> > > > to
> > > > > look at my *stat file that I uploaded to the ftp site?
> > > > >
> > > > > Roz
> > > > >
> > > > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Roz,
> > > > > >
> > > > > > Point-Stat takes as input a gridded NWP forecast file and
the
> > NetCDF
> > > > > output
> > > > > > of a point pre-processing tool (like ascii2nc or pb2nc).
Based
> on
> > > your
> > > > > > description, it doesn't sound like you have gridded NWP
forecast
> > > files
> > > > > > available. So no, I don't think that approach would work.
> > > > > >
> > > > > > The MPR output line type does include a column named
"OBS_QC"
> which
> > > is
> > > > a
> > > > > > string containing the observation quality control value.
I'd
> > suggest
> > > > > > checking the contents of that column in your MPR output
lines to
> > see
> > > if
> > > > > > those values could be filtered to do the quality control
> filtering
> > > > you're
> > > > > > after. Check to see if the values in that column are
strings or
> > > > numbers
> > > > > > (in PREPBUFR they're integers... in MADIS they're
strings).
> > > > > >
> > > > > > In STAT-Analysis, you can use the "-column_thresh" option
to
> > > threshold
> > > > a
> > > > > > numeric column or you can use the "-column_str" option to
list
> > > strings
> > > > to
> > > > > > retained. For example:
> > > > > > -column_thresh OBS_QC 'ge0&&le3' # this keeps all
values
> > between
> > > 0
> > > > > and
> > > > > > 3
> > > > > > -column_str OBS_QC aa,ab,ac # this keeps
strings "aa",
> > > "ab",
> > > > > or
> > > > > > "ac"
> > > > > >
> > > > > > Hope that helps.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken - NOAA
> Affiliate
> > > via
> > > > > RT
> > > > > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > I have another question related to my reprocessing of my
data.
> > > > > > >
> > > > > > > Ok, so, I realized that when I switched ASCAT data
sources back
> > in
> > > > > March,
> > > > > > > from prepbufr to MGDRlite files, I should have been
using
> quality
> > > > flags
> > > > > > to
> > > > > > > filter out my data. Because I wasn't checking for the
quality
> > > flags,
> > > > > my
> > > > > > > statistics were not so good, since it was letting in bad
data,
> > and
> > > > > stats
> > > > > > > were being calculated from this bad data.
> > > > > > >
> > > > > > > So, I need to find a way to correct this, but, the old
story
> is I
> > > > don't
> > > > > > > have the original GFS data, and it's too massive to grab
from
> > HPSS.
> > > > I
> > > > > do
> > > > > > > have the original ASCAT files, which I can run through
my
> program
> > > to
> > > > > > filter
> > > > > > > out the bad observations, as well as matched points from
the
> *mpr
> > > and
> > > > > > *stat
> > > > > > > files.
> > > > > > >
> > > > > > > Originally, I was just going to filter out using that
> > > -column_thresh
> > > > > > > command, and filter out over land. But, I had this idea
that I
> > > > wanted
> > > > > to
> > > > > > > run by you, to see if this makes sense to do.
> > > > > > >
> > > > > > > I have the time, lat, lon and GFS observation and the
> > corresponding
> > > > > ASCAT
> > > > > > > observation point. Can I do this:
> > > > > > > 1) read the time, lat, lon and GFS observation out of
the mpr
> > > file,
> > > > > and
> > > > > > > make a new input file with just those variables in the
ASCII
> > format
> > > > > that
> > > > > > > MET can use
> > > > > > > 2) reprocess my ASCAT data and write it into an ASCII
file in
> > MET
> > > > > format
> > > > > > > 3) run ASCII2NC on these files, to create the two input
files
> > for
> > > > MET,
> > > > > > > then, run point_stat on the netcdf files to regenerate
the new
> > > > > corrected
> > > > > > > stat files
> > > > > > >
> > > > > > > Does that make sense to do? I know it could be alot of
work,
> > but,
> > > > if I
> > > > > > can
> > > > > > > get corrected statistics, it might be worth it. What do
you
> > think?
> > > > > > >
> > > > > > > Roz
> > > > > > >
> > > > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn MacCracken -
NOAA
> > > Affiliate
> > > > <
> > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > I double checked all those things you listed, and they
are
> all
> > > > > correct.
> > > > > > > > So, I've put a file in the /maccracken directory on
your ftp
> > > site.
> > > > > Let
> > > > > > > me
> > > > > > > > know what you see.
> > > > > > > >
> > > > > > > > Thanks for your help!
> > > > > > > >
> > > > > > > > Roz
> > > > > > > >
> > > > > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn MacCracken -
NOAA
> > > > Affiliate
> > > > > <
> > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > >
> > > > > > > >> Hi John,
> > > > > > > >>
> > > > > > > >> I'm sorry I didn't get back to you sooner, but, I was
tied
> up
> > > with
> > > > > > some
> > > > > > > >> other things. Let me double check with all of those
items
> in
> > my
> > > > > file
> > > > > > > and
> > > > > > > >> make sure that I don't have some sort of obvious
error.
> I'll
> > be
> > > > > back
> > > > > > in
> > > > > > > >> touch after I check on this.
> > > > > > > >>
> > > > > > > >> Roz
> > > > > > > >>
> > > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway
via RT <
> > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > >>
> > > > > > > >>> Hi Roz,
> > > > > > > >>>
> > > > > > > >>> I see you're having trouble getting with STAT-
Analysis,
> > getting
> > > > MPR
> > > > > > > lines
> > > > > > > >>> to be included in the column thresholds you've
defined. I
> > > don't
> > > > > see
> > > > > > > >>> anything obviously wrong with what you've defined.
> > > > > > > >>>
> > > > > > > >>> The warning message indicates that the filters you
defined
> > have
> > > > > > > discarded
> > > > > > > >>> all of the input MPR lines.
> > > > > > > >>>
> > > > > > > >>> We'd just need to go through your filtering options
> > one-by-one
> > > to
> > > > > > find
> > > > > > > >>> out
> > > > > > > >>> why.
> > > > > > > >>>
> > > > > > > >>> - Are you passing STAT-Analysis files which contain
MPR
> > lines?
> > > > > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > > > > >>> - Does the OBS column have values > 1.0?
> > > > > > > >>> - What are the ranges of the OBS_LAT and OBS_LON
columns?
> > > > > > > >>>
> > > > > > > >>> If you'd like to send me a sample .stat file, I
could
> > probably
> > > > > track
> > > > > > it
> > > > > > > >>> down.
> > > > > > > >>>
> > > > > > > >>> Thanks,
> > > > > > > >>> John
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken -
NOAA
> > > > Affiliate
> > > > > > via
> > > > > > > >>> RT <
> > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > >>>
> > > > > > > >>> >
> > > > > > > >>> > Mon Aug 06 07:43:46 2018: Request 86480 was acted
upon.
> > > > > > > >>> > Transaction: Ticket created by
> rosalyn.maccracken at noaa.gov
> > > > > > > >>> > Queue: met_help
> > > > > > > >>> > Subject: Errors using -column_thresh
> > > > > > > >>> > Owner: Nobody
> > > > > > > >>> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > > > >>> > Status: new
> > > > > > > >>> > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > >>> > Hi,
> > > > > > > >>> >
> > > > > > > >>> > I'm trying to use -column_thresh with
stat_analysis to
> mask
> > > > out a
> > > > > > > >>> lat/lon
> > > > > > > >>> > box. I've added this command in my StatAnalysis
config
> > file:
> > > > > > > >>> >
> > > > > > > >>> > "-job aggregate_stat -out_line_type CTC
-column_thresh
> OBS
> > > > gt1.0
> > > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > > >>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> > > > > > > >>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
> > > > > > > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > > > > > >>> > -out_stat /opc_test/home/opc_test/data/m
> > > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > > >>> >
new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > > >>> >
> > > > > > > >>> > But, when it runs, I get these errors:
> > > > > > > >>> >
> > > > > > > >>> > Output:
> > > > > > > >>> > DEBUG 2: Processing Job 2: -job aggregate_stat
-fcst_var
> > WIND
> > > > > > > >>> -fcst_lev Z10
> > > > > > > >>> > -obtype ASCAT -line_type MPR -column_thresh OBS
>1.0
> > > > > -column_thresh
> > > > > > > >>> OBS_LAT
> > > > > > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-
100.0&&<=-10.0
> -by
> > > > > > > >>> FCST_VALID_BEG
> > > > > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> > > > /opc_test/home/opc_test/data/
> > > > > > > >>> >
> > > > > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> > MPR
> > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > >>> > DEBUG 1: Creating STAT output file
> > > > "/opc_test/home/opc_test/data/
> > > > > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> > > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > > > >>> > DEBUG 2: Computing output for 0 case(s).
> > > > > > > >>> > WARNING:
> > > > > > > >>> > WARNING: do_job_aggr_stat() -> no matching STAT
lines
> found
> > > for
> > > > > > job:
> > > > > > > >>> -job
> > > > > > > >>> > aggregate_stat -fcst_var WIND -fcst_lev Z10
-obtype ASCAT
> > > > > > -line_type
> > > > > > > >>> MPR
> > > > > > > >>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT
> > >=15.0&&<=70.0
> > > > > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
> FCST_VALID_BEG
> > > -by
> > > > > > > >>> FCST_LEAD
> > > > > > > >>> > -by VX_MASK -out_stat
/opc_test/home/opc_test/data/
> > > > > > > >>> >
> > > > > > > >>> >
met_verif/regen_mar2018_data/new_out_stat_test2/agg_stat_
> > MPR
> > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > >>> > WARNING:
> > > > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> > > > > > > >>> > DEBUG 2:
> > > > > > > >>> >
> > > > > > > >>> > Am I not using the command properly?
> > > > > > > >>> >
> > > > > > > >>> > Thanks!
> > > > > > > >>> >
> > > > > > > >>> > Roz
> > > > > > > >>> >
> > > > > > > >>> > --
> > > > > > > >>> > Rosalyn MacCracken
> > > > > > > >>> > Support Scientist
> > > > > > > >>> >
> > > > > > > >>> > Ocean Applications Branch
> > > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > > >>> > NCWCP
> > > > > > > >>> > 5830 University Research Ct
> > > > > > > >>> > College Park, MD 20740-3818
> > > > > > > >>> >
> > > > > > > >>> > (p) 301-683-1551
> > > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Rosalyn MacCracken
> > > > > > > >> Support Scientist
> > > > > > > >>
> > > > > > > >> Ocean Applications Branch
> > > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > > >> NCWCP
> > > > > > > >> 5830 University Research Ct
> > > > > > > >> College Park, MD 20740-3818
> > > > > > > >>
> > > > > > > >> (p) 301-683-1551
> > > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rosalyn MacCracken
> > > > > > > > Support Scientist
> > > > > > > >
> > > > > > > > Ocean Applications Branch
> > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > NCWCP
> > > > > > > > 5830 University Research Ct
> > > > > > > > College Park, MD 20740-3818
> > > > > > > >
> > > > > > > > (p) 301-683-1551
> > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rosalyn MacCracken
> > > > > > > Support Scientist
> > > > > > >
> > > > > > > Ocean Applications Branch
> > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > NCWCP
> > > > > > > 5830 University Research Ct
> > > > > > > College Park, MD 20740-3818
> > > > > > >
> > > > > > > (p) 301-683-1551
> > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applications Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Mon Aug 13 12:47:31 2018
No, I don't know the correct OBS_QC value. Can I just make something
up,
like use "2", or some arbitrary number? Or does MET expect some
value,
given a certain file type?
Actually, I currently don't write these values into my point file,
which I
use to make the netcdf file. So, it's just for the back data.
Roz
On Mon, Aug 13, 2018 at 6:41 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Right, updating the OBS_QC values from NA to the real QC value is
the
> difficult part, but I think that's probably easier that "gridding"
the
> point data and going back through Point-Stat.
>
> So do you know how to get the QBS_QC value from the MGDRlite data?
And,
> going forward, can you set it up so that the OBS_QC column is
populated in
> your Point-Stat runs going forward?
>
> Perhaps that's the place to start. Make sure your new Point-Stat
output
> includes meaningful OBS_QC output. Once that's working, go through
your
> *old* output and backfill the OBS_QC column with what it should
contain.
>
> John
>
> On Mon, Aug 13, 2018 at 12:37 PM Rosalyn MacCracken - NOAA Affiliate
via RT
> <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> >
> > Yes, that makes sense. So, the MPR line would be "N/A" that it
would
> > process the data with. Ok. And, I could do something similar to
what I
> > did before with copying my MPR files to a temp directory, and use
> -lookin.
> >
> > Ok, so, the hard part will be to write this script that replaces
my bad
> > values. I guess I like a challenge, right?
> >
> > Roz
> >
> > On Mon, Aug 13, 2018 at 6:30 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Roz,
> > >
> > > Once the OBS_QC values are set in the MPR lines, you can use
> > STAT-Analysis
> > > to reprocess the data and recompute whatever other STAT lines
you'd
> like
> > to
> > > compute:
> > >
> > > stat_analysis -job aggregate_stat -line_type MPR -out_line_type
CNT
> > > -column_str OBS_QC a,b,c
> > >
> > > This job would be run only on MPR lines that have strings of
"a", "b",
> or
> > > "c" in the OBS_QC column. Make sense?
> > >
> > > John
> > >
> > > On Mon, Aug 13, 2018 at 11:55 AM Rosalyn MacCracken - NOAA
Affiliate
> via
> > RT
> > > <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
>
> > > >
> > > > Hi John,
> > > >
> > > > Yeah, I can repost the file. I'll do that in a bit.
Actually, if I
> > can
> > > > get this issue solved, you don't need to look at the file,
because
> I'll
> > > > just reprocess that data with my new method. So, I'll hold
off on
> > ftping
> > > > that file to you for right now.
> > > >
> > > > So, the OBS_QC column has N/A values. That's easy enough to
replace.
> > > Now,
> > > > I have the original ASCAT data, which looks for the quality
flag as a
> > bit
> > > > (i.e., ibits[5]) and then doesn't use that. It doesn't write
it to
> the
> > > > output file. Maybe what I should do is find the first bad
> observation
> > > > (time, lat, lon, value), then, match it exactly to the line in
the
> mpr
> > > > file, and replace the observation.
> > > >
> > > > So, if my observations are in dataframes, and the mpr file has
been
> > read
> > > in
> > > > as a dataframe, then, I could do a df.query('Lat = ?? and Lon
=??),
> or
> > > > however the syntax is...no wait, I need an if statement to
compare
> > > > first...I'll work on those detail. Then, once I get the
OBS_QC value
> > > > changed, then what? How do you get the other stat files from
this
> > point?
> > > >
> > > > Roz
> > > >
> > > >
> > > > On Mon, Aug 13, 2018 at 5:32 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Roz,
> > > > >
> > > > > That's correct, there is no tool in MET for comparing one
set of
> > point
> > > > > observations to another set of point observations.
Presumably you
> > > could
> > > > > write point observations to a grid and then reconfigure and
rerun
> > > > > Point-Stat, but that'd be pretty difficult, time consuming,
and
> > fraught
> > > > > with problems.
> > > > >
> > > > > You currently have many, many MPR lines and the issue is the
the
> > OBS_QC
> > > > > column is empty. If that OBS_QC column were *not* empty
then it'd
> be
> > > > > pretty easy to process this data... correct? One option
would be
> > > > writing a
> > > > > script to "patch" that column. For each MPR line, do some
> processing
> > > to
> > > > > figure out what that OBS_QC column should be and then update
it's
> > > value.
> > > > >
> > > > > Then the question is this... is there a way of getting
quality
> > control
> > > > > values from the MGDRlite files?
> > > > >
> > > > > I'm sorry, I hadn't had a chance to look at that data file
yet.
> But
> > I
> > > > just
> > > > > checked on the ftp site and don't see a maccracken sub-
directory
> > there:
> > > > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > > > >
> > > > > Could you please repost?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn MacCracken - NOAA
> Affiliate
> > > via
> > > > RT
> > > > > <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > > >
> > > > > > So, you can't compare two point files, even if both of
them are
> > > NetCDF
> > > > > > format? Is there a way to write point observations out to
a
> grid?
> > > In
> > > > > > other words, take the point GFS observations, and write
them to a
> > > > global
> > > > > > grid (using some other tool, like wgrib or something
else), and
> > fill
> > > in
> > > > > > missing values with NaN, or something? Then, use that
gridded
> file
> > > to
> > > > > > compare with the ASCAT NetCDF file?
> > > > > >
> > > > > > I don't have anything in the OBS_QC column because when I
wrote
> it
> > > into
> > > > > the
> > > > > > ASCII file, I didn't have any flags from my MGDRlite
files. This
> > is
> > > > part
> > > > > > of the problem...
> > > > > >
> > > > > > Now, your answer for the STAT-Analysis, is that what I did
wrong
> in
> > > my
> > > > > use
> > > > > > of stat-analysis from the previous email? Or, did you not
get a
> > > chance
> > > > > to
> > > > > > look at my *stat file that I uploaded to the ftp site?
> > > > > >
> > > > > > Roz
> > > > > >
> > > > > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Roz,
> > > > > > >
> > > > > > > Point-Stat takes as input a gridded NWP forecast file
and the
> > > NetCDF
> > > > > > output
> > > > > > > of a point pre-processing tool (like ascii2nc or pb2nc).
Based
> > on
> > > > your
> > > > > > > description, it doesn't sound like you have gridded NWP
> forecast
> > > > files
> > > > > > > available. So no, I don't think that approach would
work.
> > > > > > >
> > > > > > > The MPR output line type does include a column named
"OBS_QC"
> > which
> > > > is
> > > > > a
> > > > > > > string containing the observation quality control value.
I'd
> > > suggest
> > > > > > > checking the contents of that column in your MPR output
lines
> to
> > > see
> > > > if
> > > > > > > those values could be filtered to do the quality control
> > filtering
> > > > > you're
> > > > > > > after. Check to see if the values in that column are
strings
> or
> > > > > numbers
> > > > > > > (in PREPBUFR they're integers... in MADIS they're
strings).
> > > > > > >
> > > > > > > In STAT-Analysis, you can use the "-column_thresh"
option to
> > > > threshold
> > > > > a
> > > > > > > numeric column or you can use the "-column_str" option
to list
> > > > strings
> > > > > to
> > > > > > > retained. For example:
> > > > > > > -column_thresh OBS_QC 'ge0&&le3' # this keeps all
values
> > > between
> > > > 0
> > > > > > and
> > > > > > > 3
> > > > > > > -column_str OBS_QC aa,ab,ac # this keeps
strings
> "aa",
> > > > "ab",
> > > > > > or
> > > > > > > "ac"
> > > > > > >
> > > > > > > Hope that helps.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken -
NOAA
> > Affiliate
> > > > via
> > > > > > RT
> > > > > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=86480
> > >
> > > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > I have another question related to my reprocessing of
my
> data.
> > > > > > > >
> > > > > > > > Ok, so, I realized that when I switched ASCAT data
sources
> back
> > > in
> > > > > > March,
> > > > > > > > from prepbufr to MGDRlite files, I should have been
using
> > quality
> > > > > flags
> > > > > > > to
> > > > > > > > filter out my data. Because I wasn't checking for the
> quality
> > > > flags,
> > > > > > my
> > > > > > > > statistics were not so good, since it was letting in
bad
> data,
> > > and
> > > > > > stats
> > > > > > > > were being calculated from this bad data.
> > > > > > > >
> > > > > > > > So, I need to find a way to correct this, but, the old
story
> > is I
> > > > > don't
> > > > > > > > have the original GFS data, and it's too massive to
grab from
> > > HPSS.
> > > > > I
> > > > > > do
> > > > > > > > have the original ASCAT files, which I can run through
my
> > program
> > > > to
> > > > > > > filter
> > > > > > > > out the bad observations, as well as matched points
from the
> > *mpr
> > > > and
> > > > > > > *stat
> > > > > > > > files.
> > > > > > > >
> > > > > > > > Originally, I was just going to filter out using that
> > > > -column_thresh
> > > > > > > > command, and filter out over land. But, I had this
idea
> that I
> > > > > wanted
> > > > > > to
> > > > > > > > run by you, to see if this makes sense to do.
> > > > > > > >
> > > > > > > > I have the time, lat, lon and GFS observation and the
> > > corresponding
> > > > > > ASCAT
> > > > > > > > observation point. Can I do this:
> > > > > > > > 1) read the time, lat, lon and GFS observation out of
the
> mpr
> > > > file,
> > > > > > and
> > > > > > > > make a new input file with just those variables in the
ASCII
> > > format
> > > > > > that
> > > > > > > > MET can use
> > > > > > > > 2) reprocess my ASCAT data and write it into an ASCII
file
> in
> > > MET
> > > > > > format
> > > > > > > > 3) run ASCII2NC on these files, to create the two
input
> files
> > > for
> > > > > MET,
> > > > > > > > then, run point_stat on the netcdf files to regenerate
the
> new
> > > > > > corrected
> > > > > > > > stat files
> > > > > > > >
> > > > > > > > Does that make sense to do? I know it could be alot
of work,
> > > but,
> > > > > if I
> > > > > > > can
> > > > > > > > get corrected statistics, it might be worth it. What
do you
> > > think?
> > > > > > > >
> > > > > > > > Roz
> > > > > > > >
> > > > > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn MacCracken -
NOAA
> > > > Affiliate
> > > > > <
> > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > >
> > > > > > > > > Hi John,
> > > > > > > > >
> > > > > > > > > I double checked all those things you listed, and
they are
> > all
> > > > > > correct.
> > > > > > > > > So, I've put a file in the /maccracken directory on
your
> ftp
> > > > site.
> > > > > > Let
> > > > > > > > me
> > > > > > > > > know what you see.
> > > > > > > > >
> > > > > > > > > Thanks for your help!
> > > > > > > > >
> > > > > > > > > Roz
> > > > > > > > >
> > > > > > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn MacCracken
- NOAA
> > > > > Affiliate
> > > > > > <
> > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > >
> > > > > > > > >> Hi John,
> > > > > > > > >>
> > > > > > > > >> I'm sorry I didn't get back to you sooner, but, I
was tied
> > up
> > > > with
> > > > > > > some
> > > > > > > > >> other things. Let me double check with all of
those items
> > in
> > > my
> > > > > > file
> > > > > > > > and
> > > > > > > > >> make sure that I don't have some sort of obvious
error.
> > I'll
> > > be
> > > > > > back
> > > > > > > in
> > > > > > > > >> touch after I check on this.
> > > > > > > > >>
> > > > > > > > >> Roz
> > > > > > > > >>
> > > > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley Gotway
via RT
> <
> > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi Roz,
> > > > > > > > >>>
> > > > > > > > >>> I see you're having trouble getting with STAT-
Analysis,
> > > getting
> > > > > MPR
> > > > > > > > lines
> > > > > > > > >>> to be included in the column thresholds you've
defined.
> I
> > > > don't
> > > > > > see
> > > > > > > > >>> anything obviously wrong with what you've defined.
> > > > > > > > >>>
> > > > > > > > >>> The warning message indicates that the filters you
> defined
> > > have
> > > > > > > > discarded
> > > > > > > > >>> all of the input MPR lines.
> > > > > > > > >>>
> > > > > > > > >>> We'd just need to go through your filtering
options
> > > one-by-one
> > > > to
> > > > > > > find
> > > > > > > > >>> out
> > > > > > > > >>> why.
> > > > > > > > >>>
> > > > > > > > >>> - Are you passing STAT-Analysis files which
contain MPR
> > > lines?
> > > > > > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > > > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > > > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > > > > > >>> - Does the OBS column have values > 1.0?
> > > > > > > > >>> - What are the ranges of the OBS_LAT and OBS_LON
columns?
> > > > > > > > >>>
> > > > > > > > >>> If you'd like to send me a sample .stat file, I
could
> > > probably
> > > > > > track
> > > > > > > it
> > > > > > > > >>> down.
> > > > > > > > >>>
> > > > > > > > >>> Thanks,
> > > > > > > > >>> John
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn MacCracken
- NOAA
> > > > > Affiliate
> > > > > > > via
> > > > > > > > >>> RT <
> > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > >>>
> > > > > > > > >>> >
> > > > > > > > >>> > Mon Aug 06 07:43:46 2018: Request 86480 was
acted upon.
> > > > > > > > >>> > Transaction: Ticket created by
> > rosalyn.maccracken at noaa.gov
> > > > > > > > >>> > Queue: met_help
> > > > > > > > >>> > Subject: Errors using -column_thresh
> > > > > > > > >>> > Owner: Nobody
> > > > > > > > >>> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > > > > >>> > Status: new
> > > > > > > > >>> > Ticket <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > > > >>> >
> > > > > > > > >>> >
> > > > > > > > >>> >
> > > > > > > > >>> > Hi,
> > > > > > > > >>> >
> > > > > > > > >>> > I'm trying to use -column_thresh with
stat_analysis to
> > mask
> > > > > out a
> > > > > > > > >>> lat/lon
> > > > > > > > >>> > box. I've added this command in my StatAnalysis
config
> > > file:
> > > > > > > > >>> >
> > > > > > > > >>> > "-job aggregate_stat -out_line_type CTC
-column_thresh
> > OBS
> > > > > gt1.0
> > > > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > > > >>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> > > > > > > > >>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0' \
> > > > > > > > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > > > > > > >>> > -out_stat /opc_test/home/opc_test/data/m
> > > > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > > > >>> >
new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > > > >>> >
> > > > > > > > >>> > But, when it runs, I get these errors:
> > > > > > > > >>> >
> > > > > > > > >>> > Output:
> > > > > > > > >>> > DEBUG 2: Processing Job 2: -job aggregate_stat
> -fcst_var
> > > WIND
> > > > > > > > >>> -fcst_lev Z10
> > > > > > > > >>> > -obtype ASCAT -line_type MPR -column_thresh OBS
>1.0
> > > > > > -column_thresh
> > > > > > > > >>> OBS_LAT
> > > > > > > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON >=-
100.0&&<=-10.0
> > -by
> > > > > > > > >>> FCST_VALID_BEG
> > > > > > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> > > > > /opc_test/home/opc_test/data/
> > > > > > > > >>> >
> > > > > > > > >>> > met_verif/regen_mar2018_data/
> new_out_stat_test2/agg_stat_
> > > MPR
> > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > > >>> > DEBUG 1: Creating STAT output file
> > > > > "/opc_test/home/opc_test/data/
> > > > > > > > >>> > met_verif/regen_mar2018_data/
> new_out_stat_test2/agg_stat_
> > > > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > > > > >>> > DEBUG 2: Computing output for 0 case(s).
> > > > > > > > >>> > WARNING:
> > > > > > > > >>> > WARNING: do_job_aggr_stat() -> no matching STAT
lines
> > found
> > > > for
> > > > > > > job:
> > > > > > > > >>> -job
> > > > > > > > >>> > aggregate_stat -fcst_var WIND -fcst_lev Z10
-obtype
> ASCAT
> > > > > > > -line_type
> > > > > > > > >>> MPR
> > > > > > > > >>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT
> > > >=15.0&&<=70.0
> > > > > > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
> > FCST_VALID_BEG
> > > > -by
> > > > > > > > >>> FCST_LEAD
> > > > > > > > >>> > -by VX_MASK -out_stat
/opc_test/home/opc_test/data/
> > > > > > > > >>> >
> > > > > > > > >>> > met_verif/regen_mar2018_data/
> new_out_stat_test2/agg_stat_
> > > MPR
> > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > > >>> > WARNING:
> > > > > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> > > > > > > > >>> > DEBUG 2:
> > > > > > > > >>> >
> > > > > > > > >>> > Am I not using the command properly?
> > > > > > > > >>> >
> > > > > > > > >>> > Thanks!
> > > > > > > > >>> >
> > > > > > > > >>> > Roz
> > > > > > > > >>> >
> > > > > > > > >>> > --
> > > > > > > > >>> > Rosalyn MacCracken
> > > > > > > > >>> > Support Scientist
> > > > > > > > >>> >
> > > > > > > > >>> > Ocean Applications Branch
> > > > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > > > >>> > NCWCP
> > > > > > > > >>> > 5830 University Research Ct
> > > > > > > > >>> > College Park, MD 20740-3818
> > > > > > > > >>> >
> > > > > > > > >>> > (p) 301-683-1551
> > > > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > > > >>> >
> > > > > > > > >>> >
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> Rosalyn MacCracken
> > > > > > > > >> Support Scientist
> > > > > > > > >>
> > > > > > > > >> Ocean Applications Branch
> > > > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > > > >> NCWCP
> > > > > > > > >> 5830 University Research Ct
> > > > > > > > >> College Park, MD 20740-3818
> > > > > > > > >>
> > > > > > > > >> (p) 301-683-1551
> > > > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rosalyn MacCracken
> > > > > > > > > Support Scientist
> > > > > > > > >
> > > > > > > > > Ocean Applications Branch
> > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > NCWCP
> > > > > > > > > 5830 University Research Ct
> > > > > > > > > College Park, MD 20740-3818
> > > > > > > > >
> > > > > > > > > (p) 301-683-1551
> > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rosalyn MacCracken
> > > > > > > > Support Scientist
> > > > > > > >
> > > > > > > > Ocean Applications Branch
> > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > NCWCP
> > > > > > > > 5830 University Research Ct
> > > > > > > > College Park, MD 20740-3818
> > > > > > > >
> > > > > > > > (p) 301-683-1551
> > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applications Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Errors using -column_thresh
From: John Halley Gotway
Time: Mon Aug 13 13:20:14 2018
Roz,
I'm confused. Here's is what I understood the situation to be...
(1) In March, you switched from pulling ASCAT data from PREPBUFR to
pulling
it from MGDRlite.
(2) When you made that switch, you forgot to set the system up to
filter
the MGDRlite data by quality control flags.
(3) As a result of that, many "bad" observations were included in your
evaluation which led to bad statistics.
(4) You now want to go back and correct the bad evaluation by
reprocessing
using the actual quality control flags.
I suggested that you modify the MET output MPR lines by replacing the
NA
string with the *actual* quality control flag. This logic only works
if
the MGDRlite data source includes quality control information... and
you
know how to extract it.
Just putting a constant flag value of 2 in the OBS_QC column won't
really
fix anything. I'm suggesting that you get the actual flag values and
"patch" the existing MPR lines by inserting them.
Am I misunderstanding the situation?
Thanks,
John
On Mon, Aug 13, 2018 at 12:47 PM Rosalyn MacCracken - NOAA Affiliate
via RT
<met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
>
> No, I don't know the correct OBS_QC value. Can I just make
something up,
> like use "2", or some arbitrary number? Or does MET expect some
value,
> given a certain file type?
>
> Actually, I currently don't write these values into my point file,
which I
> use to make the netcdf file. So, it's just for the back data.
>
> Roz
>
> On Mon, Aug 13, 2018 at 6:41 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Right, updating the OBS_QC values from NA to the real QC value is
the
> > difficult part, but I think that's probably easier that "gridding"
the
> > point data and going back through Point-Stat.
> >
> > So do you know how to get the QBS_QC value from the MGDRlite data?
And,
> > going forward, can you set it up so that the OBS_QC column is
populated
> in
> > your Point-Stat runs going forward?
> >
> > Perhaps that's the place to start. Make sure your new Point-Stat
output
> > includes meaningful OBS_QC output. Once that's working, go
through your
> > *old* output and backfill the OBS_QC column with what it should
contain.
> >
> > John
> >
> > On Mon, Aug 13, 2018 at 12:37 PM Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > >
> > > Yes, that makes sense. So, the MPR line would be "N/A" that it
would
> > > process the data with. Ok. And, I could do something similar
to what
> I
> > > did before with copying my MPR files to a temp directory, and
use
> > -lookin.
> > >
> > > Ok, so, the hard part will be to write this script that replaces
my bad
> > > values. I guess I like a challenge, right?
> > >
> > > Roz
> > >
> > > On Mon, Aug 13, 2018 at 6:30 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Roz,
> > > >
> > > > Once the OBS_QC values are set in the MPR lines, you can use
> > > STAT-Analysis
> > > > to reprocess the data and recompute whatever other STAT lines
you'd
> > like
> > > to
> > > > compute:
> > > >
> > > > stat_analysis -job aggregate_stat -line_type MPR
-out_line_type CNT
> > > > -column_str OBS_QC a,b,c
> > > >
> > > > This job would be run only on MPR lines that have strings of
"a",
> "b",
> > or
> > > > "c" in the OBS_QC column. Make sense?
> > > >
> > > > John
> > > >
> > > > On Mon, Aug 13, 2018 at 11:55 AM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > RT
> > > > <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Yeah, I can repost the file. I'll do that in a bit.
Actually, if
> I
> > > can
> > > > > get this issue solved, you don't need to look at the file,
because
> > I'll
> > > > > just reprocess that data with my new method. So, I'll hold
off on
> > > ftping
> > > > > that file to you for right now.
> > > > >
> > > > > So, the OBS_QC column has N/A values. That's easy enough to
> replace.
> > > > Now,
> > > > > I have the original ASCAT data, which looks for the quality
flag
> as a
> > > bit
> > > > > (i.e., ibits[5]) and then doesn't use that. It doesn't
write it to
> > the
> > > > > output file. Maybe what I should do is find the first bad
> > observation
> > > > > (time, lat, lon, value), then, match it exactly to the line
in the
> > mpr
> > > > > file, and replace the observation.
> > > > >
> > > > > So, if my observations are in dataframes, and the mpr file
has been
> > > read
> > > > in
> > > > > as a dataframe, then, I could do a df.query('Lat = ?? and
Lon =??),
> > or
> > > > > however the syntax is...no wait, I need an if statement to
compare
> > > > > first...I'll work on those detail. Then, once I get the
OBS_QC
> value
> > > > > changed, then what? How do you get the other stat files
from this
> > > point?
> > > > >
> > > > > Roz
> > > > >
> > > > >
> > > > > On Mon, Aug 13, 2018 at 5:32 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Roz,
> > > > > >
> > > > > > That's correct, there is no tool in MET for comparing one
set of
> > > point
> > > > > > observations to another set of point observations.
Presumably
> you
> > > > could
> > > > > > write point observations to a grid and then reconfigure
and rerun
> > > > > > Point-Stat, but that'd be pretty difficult, time
consuming, and
> > > fraught
> > > > > > with problems.
> > > > > >
> > > > > > You currently have many, many MPR lines and the issue is
the the
> > > OBS_QC
> > > > > > column is empty. If that OBS_QC column were *not* empty
then
> it'd
> > be
> > > > > > pretty easy to process this data... correct? One option
would be
> > > > > writing a
> > > > > > script to "patch" that column. For each MPR line, do some
> > processing
> > > > to
> > > > > > figure out what that OBS_QC column should be and then
update it's
> > > > value.
> > > > > >
> > > > > > Then the question is this... is there a way of getting
quality
> > > control
> > > > > > values from the MGDRlite files?
> > > > > >
> > > > > > I'm sorry, I hadn't had a chance to look at that data file
yet.
> > But
> > > I
> > > > > just
> > > > > > checked on the ftp site and don't see a maccracken sub-
directory
> > > there:
> > > > > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > > > > >
> > > > > > Could you please repost?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn MacCracken - NOAA
> > Affiliate
> > > > via
> > > > > RT
> > > > > > <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> >
> > > > > > >
> > > > > > > So, you can't compare two point files, even if both of
them are
> > > > NetCDF
> > > > > > > format? Is there a way to write point observations out
to a
> > grid?
> > > > In
> > > > > > > other words, take the point GFS observations, and write
them
> to a
> > > > > global
> > > > > > > grid (using some other tool, like wgrib or something
else), and
> > > fill
> > > > in
> > > > > > > missing values with NaN, or something? Then, use that
gridded
> > file
> > > > to
> > > > > > > compare with the ASCAT NetCDF file?
> > > > > > >
> > > > > > > I don't have anything in the OBS_QC column because when
I wrote
> > it
> > > > into
> > > > > > the
> > > > > > > ASCII file, I didn't have any flags from my MGDRlite
files.
> This
> > > is
> > > > > part
> > > > > > > of the problem...
> > > > > > >
> > > > > > > Now, your answer for the STAT-Analysis, is that what I
did
> wrong
> > in
> > > > my
> > > > > > use
> > > > > > > of stat-analysis from the previous email? Or, did you
not get
> a
> > > > chance
> > > > > > to
> > > > > > > look at my *stat file that I uploaded to the ftp site?
> > > > > > >
> > > > > > > Roz
> > > > > > >
> > > > > > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > > Roz,
> > > > > > > >
> > > > > > > > Point-Stat takes as input a gridded NWP forecast file
and the
> > > > NetCDF
> > > > > > > output
> > > > > > > > of a point pre-processing tool (like ascii2nc or
pb2nc).
> Based
> > > on
> > > > > your
> > > > > > > > description, it doesn't sound like you have gridded
NWP
> > forecast
> > > > > files
> > > > > > > > available. So no, I don't think that approach would
work.
> > > > > > > >
> > > > > > > > The MPR output line type does include a column named
"OBS_QC"
> > > which
> > > > > is
> > > > > > a
> > > > > > > > string containing the observation quality control
value. I'd
> > > > suggest
> > > > > > > > checking the contents of that column in your MPR
output lines
> > to
> > > > see
> > > > > if
> > > > > > > > those values could be filtered to do the quality
control
> > > filtering
> > > > > > you're
> > > > > > > > after. Check to see if the values in that column are
strings
> > or
> > > > > > numbers
> > > > > > > > (in PREPBUFR they're integers... in MADIS they're
strings).
> > > > > > > >
> > > > > > > > In STAT-Analysis, you can use the "-column_thresh"
option to
> > > > > threshold
> > > > > > a
> > > > > > > > numeric column or you can use the "-column_str" option
to
> list
> > > > > strings
> > > > > > to
> > > > > > > > retained. For example:
> > > > > > > > -column_thresh OBS_QC 'ge0&&le3' # this keeps all
values
> > > > between
> > > > > 0
> > > > > > > and
> > > > > > > > 3
> > > > > > > > -column_str OBS_QC aa,ab,ac # this keeps
strings
> > "aa",
> > > > > "ab",
> > > > > > > or
> > > > > > > > "ac"
> > > > > > > >
> > > > > > > > Hope that helps.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken -
NOAA
> > > Affiliate
> > > > > via
> > > > > > > RT
> > > > > > > > <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=86480
> > > >
> > > > > > > > >
> > > > > > > > > Hi John,
> > > > > > > > >
> > > > > > > > > I have another question related to my reprocessing
of my
> > data.
> > > > > > > > >
> > > > > > > > > Ok, so, I realized that when I switched ASCAT data
sources
> > back
> > > > in
> > > > > > > March,
> > > > > > > > > from prepbufr to MGDRlite files, I should have been
using
> > > quality
> > > > > > flags
> > > > > > > > to
> > > > > > > > > filter out my data. Because I wasn't checking for
the
> > quality
> > > > > flags,
> > > > > > > my
> > > > > > > > > statistics were not so good, since it was letting in
bad
> > data,
> > > > and
> > > > > > > stats
> > > > > > > > > were being calculated from this bad data.
> > > > > > > > >
> > > > > > > > > So, I need to find a way to correct this, but, the
old
> story
> > > is I
> > > > > > don't
> > > > > > > > > have the original GFS data, and it's too massive to
grab
> from
> > > > HPSS.
> > > > > > I
> > > > > > > do
> > > > > > > > > have the original ASCAT files, which I can run
through my
> > > program
> > > > > to
> > > > > > > > filter
> > > > > > > > > out the bad observations, as well as matched points
from
> the
> > > *mpr
> > > > > and
> > > > > > > > *stat
> > > > > > > > > files.
> > > > > > > > >
> > > > > > > > > Originally, I was just going to filter out using
that
> > > > > -column_thresh
> > > > > > > > > command, and filter out over land. But, I had this
idea
> > that I
> > > > > > wanted
> > > > > > > to
> > > > > > > > > run by you, to see if this makes sense to do.
> > > > > > > > >
> > > > > > > > > I have the time, lat, lon and GFS observation and
the
> > > > corresponding
> > > > > > > ASCAT
> > > > > > > > > observation point. Can I do this:
> > > > > > > > > 1) read the time, lat, lon and GFS observation out
of the
> > mpr
> > > > > file,
> > > > > > > and
> > > > > > > > > make a new input file with just those variables in
the
> ASCII
> > > > format
> > > > > > > that
> > > > > > > > > MET can use
> > > > > > > > > 2) reprocess my ASCAT data and write it into an
ASCII file
> > in
> > > > MET
> > > > > > > format
> > > > > > > > > 3) run ASCII2NC on these files, to create the two
input
> > files
> > > > for
> > > > > > MET,
> > > > > > > > > then, run point_stat on the netcdf files to
regenerate the
> > new
> > > > > > > corrected
> > > > > > > > > stat files
> > > > > > > > >
> > > > > > > > > Does that make sense to do? I know it could be alot
of
> work,
> > > > but,
> > > > > > if I
> > > > > > > > can
> > > > > > > > > get corrected statistics, it might be worth it.
What do
> you
> > > > think?
> > > > > > > > >
> > > > > > > > > Roz
> > > > > > > > >
> > > > > > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn MacCracken
- NOAA
> > > > > Affiliate
> > > > > > <
> > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > >
> > > > > > > > > > Hi John,
> > > > > > > > > >
> > > > > > > > > > I double checked all those things you listed, and
they
> are
> > > all
> > > > > > > correct.
> > > > > > > > > > So, I've put a file in the /maccracken directory
on your
> > ftp
> > > > > site.
> > > > > > > Let
> > > > > > > > > me
> > > > > > > > > > know what you see.
> > > > > > > > > >
> > > > > > > > > > Thanks for your help!
> > > > > > > > > >
> > > > > > > > > > Roz
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn
MacCracken -
> NOAA
> > > > > > Affiliate
> > > > > > > <
> > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi John,
> > > > > > > > > >>
> > > > > > > > > >> I'm sorry I didn't get back to you sooner, but, I
was
> tied
> > > up
> > > > > with
> > > > > > > > some
> > > > > > > > > >> other things. Let me double check with all of
those
> items
> > > in
> > > > my
> > > > > > > file
> > > > > > > > > and
> > > > > > > > > >> make sure that I don't have some sort of obvious
error.
> > > I'll
> > > > be
> > > > > > > back
> > > > > > > > in
> > > > > > > > > >> touch after I check on this.
> > > > > > > > > >>
> > > > > > > > > >> Roz
> > > > > > > > > >>
> > > > > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley
Gotway via
> RT
> > <
> > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Hi Roz,
> > > > > > > > > >>>
> > > > > > > > > >>> I see you're having trouble getting with STAT-
Analysis,
> > > > getting
> > > > > > MPR
> > > > > > > > > lines
> > > > > > > > > >>> to be included in the column thresholds you've
defined.
> > I
> > > > > don't
> > > > > > > see
> > > > > > > > > >>> anything obviously wrong with what you've
defined.
> > > > > > > > > >>>
> > > > > > > > > >>> The warning message indicates that the filters
you
> > defined
> > > > have
> > > > > > > > > discarded
> > > > > > > > > >>> all of the input MPR lines.
> > > > > > > > > >>>
> > > > > > > > > >>> We'd just need to go through your filtering
options
> > > > one-by-one
> > > > > to
> > > > > > > > find
> > > > > > > > > >>> out
> > > > > > > > > >>> why.
> > > > > > > > > >>>
> > > > > > > > > >>> - Are you passing STAT-Analysis files which
contain MPR
> > > > lines?
> > > > > > > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > > > > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > > > > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > > > > > > >>> - Does the OBS column have values > 1.0?
> > > > > > > > > >>> - What are the ranges of the OBS_LAT and OBS_LON
> columns?
> > > > > > > > > >>>
> > > > > > > > > >>> If you'd like to send me a sample .stat file, I
could
> > > > probably
> > > > > > > track
> > > > > > > > it
> > > > > > > > > >>> down.
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks,
> > > > > > > > > >>> John
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn
MacCracken -
> NOAA
> > > > > > Affiliate
> > > > > > > > via
> > > > > > > > > >>> RT <
> > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> >
> > > > > > > > > >>> > Mon Aug 06 07:43:46 2018: Request 86480 was
acted
> upon.
> > > > > > > > > >>> > Transaction: Ticket created by
> > > rosalyn.maccracken at noaa.gov
> > > > > > > > > >>> > Queue: met_help
> > > > > > > > > >>> > Subject: Errors using -column_thresh
> > > > > > > > > >>> > Owner: Nobody
> > > > > > > > > >>> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > > > > > >>> > Status: new
> > > > > > > > > >>> > Ticket <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > > > > >>> >
> > > > > > > > > >>> >
> > > > > > > > > >>> >
> > > > > > > > > >>> > Hi,
> > > > > > > > > >>> >
> > > > > > > > > >>> > I'm trying to use -column_thresh with
stat_analysis
> to
> > > mask
> > > > > > out a
> > > > > > > > > >>> lat/lon
> > > > > > > > > >>> > box. I've added this command in my
StatAnalysis
> config
> > > > file:
> > > > > > > > > >>> >
> > > > > > > > > >>> > "-job aggregate_stat -out_line_type CTC
> -column_thresh
> > > OBS
> > > > > > gt1.0
> > > > > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > > > > >>> > -column_thresh OBS_LAT 'ge15.0&&le70.0' \
> > > > > > > > > >>> > -column_thresh OBS_LON 'ge-100.0&&le-10.0'
\
> > > > > > > > > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > > > > > > > >>> > -out_stat /opc_test/home/opc_test/data/m
> > > > > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > > > > >>> >
> new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > > > > >>> >
> > > > > > > > > >>> > But, when it runs, I get these errors:
> > > > > > > > > >>> >
> > > > > > > > > >>> > Output:
> > > > > > > > > >>> > DEBUG 2: Processing Job 2: -job aggregate_stat
> > -fcst_var
> > > > WIND
> > > > > > > > > >>> -fcst_lev Z10
> > > > > > > > > >>> > -obtype ASCAT -line_type MPR -column_thresh
OBS >1.0
> > > > > > > -column_thresh
> > > > > > > > > >>> OBS_LAT
> > > > > > > > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON
> >=-100.0&&<=-10.0
> > > -by
> > > > > > > > > >>> FCST_VALID_BEG
> > > > > > > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> > > > > > /opc_test/home/opc_test/data/
> > > > > > > > > >>> >
> > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > new_out_stat_test2/agg_stat_
> > > > MPR
> > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > > > >>> > DEBUG 1: Creating STAT output file
> > > > > > "/opc_test/home/opc_test/data/
> > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > new_out_stat_test2/agg_stat_
> > > > > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > > > > > >>> > DEBUG 2: Computing output for 0 case(s).
> > > > > > > > > >>> > WARNING:
> > > > > > > > > >>> > WARNING: do_job_aggr_stat() -> no matching
STAT lines
> > > found
> > > > > for
> > > > > > > > job:
> > > > > > > > > >>> -job
> > > > > > > > > >>> > aggregate_stat -fcst_var WIND -fcst_lev Z10
-obtype
> > ASCAT
> > > > > > > > -line_type
> > > > > > > > > >>> MPR
> > > > > > > > > >>> > -column_thresh OBS >1.0 -column_thresh OBS_LAT
> > > > >=15.0&&<=70.0
> > > > > > > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
> > > FCST_VALID_BEG
> > > > > -by
> > > > > > > > > >>> FCST_LEAD
> > > > > > > > > >>> > -by VX_MASK -out_stat
/opc_test/home/opc_test/data/
> > > > > > > > > >>> >
> > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > new_out_stat_test2/agg_stat_
> > > > MPR
> > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > > > >>> > WARNING:
> > > > > > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT lines.
> > > > > > > > > >>> > DEBUG 2:
> > > > > > > > > >>> >
> > > > > > > > > >>> > Am I not using the command properly?
> > > > > > > > > >>> >
> > > > > > > > > >>> > Thanks!
> > > > > > > > > >>> >
> > > > > > > > > >>> > Roz
> > > > > > > > > >>> >
> > > > > > > > > >>> > --
> > > > > > > > > >>> > Rosalyn MacCracken
> > > > > > > > > >>> > Support Scientist
> > > > > > > > > >>> >
> > > > > > > > > >>> > Ocean Applications Branch
> > > > > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > > > > >>> > NCWCP
> > > > > > > > > >>> > 5830 University Research Ct
> > > > > > > > > >>> > College Park, MD 20740-3818
> > > > > > > > > >>> >
> > > > > > > > > >>> > (p) 301-683-1551
> > > > > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > > > > >>> >
> > > > > > > > > >>> >
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> --
> > > > > > > > > >> Rosalyn MacCracken
> > > > > > > > > >> Support Scientist
> > > > > > > > > >>
> > > > > > > > > >> Ocean Applications Branch
> > > > > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > > > > >> NCWCP
> > > > > > > > > >> 5830 University Research Ct
> > > > > > > > > >> College Park, MD 20740-3818
> > > > > > > > > >>
> > > > > > > > > >> (p) 301-683-1551
> > > > > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > Support Scientist
> > > > > > > > > >
> > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > NCWCP
> > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > >
> > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rosalyn MacCracken
> > > > > > > > > Support Scientist
> > > > > > > > >
> > > > > > > > > Ocean Applications Branch
> > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > NCWCP
> > > > > > > > > 5830 University Research Ct
> > > > > > > > > College Park, MD 20740-3818
> > > > > > > > >
> > > > > > > > > (p) 301-683-1551
> > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rosalyn MacCracken
> > > > > > > Support Scientist
> > > > > > >
> > > > > > > Ocean Applications Branch
> > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > NCWCP
> > > > > > > 5830 University Research Ct
> > > > > > > College Park, MD 20740-3818
> > > > > > >
> > > > > > > (p) 301-683-1551
> > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applications Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Mon Aug 13 13:31:46 2018
No, that's correct. Originally, we weren't checking for the flag, and
just
taking the observation from the MEDRlite file, and writing it directly
to a
MET ASCII format file. So, as a result, I have MPR files that have
N/A
where a QC flag number should be.
Now, we are using bit position in place of the quality control flag.
So, I
take the binary MGDRlite data, read it into a python program, evaluate
if
the bit position is a "1". If it's a "0", then it writes a line to an
ASCII file that has the MET format to be used with ASCII2NC. If it's
a
"1", it skips writing that value to the MET formatted file. This way,
we
are only keeping good observations and tossing the others. That's why
I
asked about just using some constant value, because it's bit position
and
not a QC flag. I was thinking it it's something other than N/A, then
it
won't be used. And, it's only for the back files. I can correct it
going
forward, and just not write that observation to my ASCII file.
Roz
On Mon, Aug 13, 2018 at 7:20 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Roz,
>
> I'm confused. Here's is what I understood the situation to be...
>
> (1) In March, you switched from pulling ASCAT data from PREPBUFR to
pulling
> it from MGDRlite.
> (2) When you made that switch, you forgot to set the system up to
filter
> the MGDRlite data by quality control flags.
> (3) As a result of that, many "bad" observations were included in
your
> evaluation which led to bad statistics.
> (4) You now want to go back and correct the bad evaluation by
reprocessing
> using the actual quality control flags.
>
> I suggested that you modify the MET output MPR lines by replacing
the NA
> string with the *actual* quality control flag. This logic only
works if
> the MGDRlite data source includes quality control information... and
you
> know how to extract it.
>
> Just putting a constant flag value of 2 in the OBS_QC column won't
really
> fix anything. I'm suggesting that you get the actual flag values
and
> "patch" the existing MPR lines by inserting them.
>
> Am I misunderstanding the situation?
>
> Thanks,
> John
>
> On Mon, Aug 13, 2018 at 12:47 PM Rosalyn MacCracken - NOAA Affiliate
via RT
> <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> >
> > No, I don't know the correct OBS_QC value. Can I just make
something up,
> > like use "2", or some arbitrary number? Or does MET expect some
value,
> > given a certain file type?
> >
> > Actually, I currently don't write these values into my point file,
which
> I
> > use to make the netcdf file. So, it's just for the back data.
> >
> > Roz
> >
> > On Mon, Aug 13, 2018 at 6:41 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Right, updating the OBS_QC values from NA to the real QC value
is the
> > > difficult part, but I think that's probably easier that
"gridding" the
> > > point data and going back through Point-Stat.
> > >
> > > So do you know how to get the QBS_QC value from the MGDRlite
data?
> And,
> > > going forward, can you set it up so that the OBS_QC column is
populated
> > in
> > > your Point-Stat runs going forward?
> > >
> > > Perhaps that's the place to start. Make sure your new Point-
Stat
> output
> > > includes meaningful OBS_QC output. Once that's working, go
through
> your
> > > *old* output and backfill the OBS_QC column with what it should
> contain.
> > >
> > > John
> > >
> > > On Mon, Aug 13, 2018 at 12:37 PM Rosalyn MacCracken - NOAA
Affiliate
> via
> > RT
> > > <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
>
> > > >
> > > > Yes, that makes sense. So, the MPR line would be "N/A" that
it would
> > > > process the data with. Ok. And, I could do something similar
to
> what
> > I
> > > > did before with copying my MPR files to a temp directory, and
use
> > > -lookin.
> > > >
> > > > Ok, so, the hard part will be to write this script that
replaces my
> bad
> > > > values. I guess I like a challenge, right?
> > > >
> > > > Roz
> > > >
> > > > On Mon, Aug 13, 2018 at 6:30 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Roz,
> > > > >
> > > > > Once the OBS_QC values are set in the MPR lines, you can use
> > > > STAT-Analysis
> > > > > to reprocess the data and recompute whatever other STAT
lines you'd
> > > like
> > > > to
> > > > > compute:
> > > > >
> > > > > stat_analysis -job aggregate_stat -line_type MPR
-out_line_type CNT
> > > > > -column_str OBS_QC a,b,c
> > > > >
> > > > > This job would be run only on MPR lines that have strings of
"a",
> > "b",
> > > or
> > > > > "c" in the OBS_QC column. Make sense?
> > > > >
> > > > > John
> > > > >
> > > > > On Mon, Aug 13, 2018 at 11:55 AM Rosalyn MacCracken - NOAA
> Affiliate
> > > via
> > > > RT
> > > > > <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Yeah, I can repost the file. I'll do that in a bit.
Actually,
> if
> > I
> > > > can
> > > > > > get this issue solved, you don't need to look at the file,
> because
> > > I'll
> > > > > > just reprocess that data with my new method. So, I'll
hold off
> on
> > > > ftping
> > > > > > that file to you for right now.
> > > > > >
> > > > > > So, the OBS_QC column has N/A values. That's easy enough
to
> > replace.
> > > > > Now,
> > > > > > I have the original ASCAT data, which looks for the
quality flag
> > as a
> > > > bit
> > > > > > (i.e., ibits[5]) and then doesn't use that. It doesn't
write it
> to
> > > the
> > > > > > output file. Maybe what I should do is find the first bad
> > > observation
> > > > > > (time, lat, lon, value), then, match it exactly to the
line in
> the
> > > mpr
> > > > > > file, and replace the observation.
> > > > > >
> > > > > > So, if my observations are in dataframes, and the mpr file
has
> been
> > > > read
> > > > > in
> > > > > > as a dataframe, then, I could do a df.query('Lat = ?? and
Lon
> =??),
> > > or
> > > > > > however the syntax is...no wait, I need an if statement to
> compare
> > > > > > first...I'll work on those detail. Then, once I get the
OBS_QC
> > value
> > > > > > changed, then what? How do you get the other stat files
from
> this
> > > > point?
> > > > > >
> > > > > > Roz
> > > > > >
> > > > > >
> > > > > > On Mon, Aug 13, 2018 at 5:32 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Roz,
> > > > > > >
> > > > > > > That's correct, there is no tool in MET for comparing
one set
> of
> > > > point
> > > > > > > observations to another set of point observations.
Presumably
> > you
> > > > > could
> > > > > > > write point observations to a grid and then reconfigure
and
> rerun
> > > > > > > Point-Stat, but that'd be pretty difficult, time
consuming, and
> > > > fraught
> > > > > > > with problems.
> > > > > > >
> > > > > > > You currently have many, many MPR lines and the issue is
the
> the
> > > > OBS_QC
> > > > > > > column is empty. If that OBS_QC column were *not* empty
then
> > it'd
> > > be
> > > > > > > pretty easy to process this data... correct? One option
would
> be
> > > > > > writing a
> > > > > > > script to "patch" that column. For each MPR line, do
some
> > > processing
> > > > > to
> > > > > > > figure out what that OBS_QC column should be and then
update
> it's
> > > > > value.
> > > > > > >
> > > > > > > Then the question is this... is there a way of getting
quality
> > > > control
> > > > > > > values from the MGDRlite files?
> > > > > > >
> > > > > > > I'm sorry, I hadn't had a chance to look at that data
file yet.
> > > But
> > > > I
> > > > > > just
> > > > > > > checked on the ftp site and don't see a maccracken
> sub-directory
> > > > there:
> > > > > > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > > > > > >
> > > > > > > Could you please repost?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn MacCracken -
NOAA
> > > Affiliate
> > > > > via
> > > > > > RT
> > > > > > > <met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=86480
> > >
> > > > > > > >
> > > > > > > > So, you can't compare two point files, even if both of
them
> are
> > > > > NetCDF
> > > > > > > > format? Is there a way to write point observations
out to a
> > > grid?
> > > > > In
> > > > > > > > other words, take the point GFS observations, and
write them
> > to a
> > > > > > global
> > > > > > > > grid (using some other tool, like wgrib or something
else),
> and
> > > > fill
> > > > > in
> > > > > > > > missing values with NaN, or something? Then, use that
> gridded
> > > file
> > > > > to
> > > > > > > > compare with the ASCAT NetCDF file?
> > > > > > > >
> > > > > > > > I don't have anything in the OBS_QC column because
when I
> wrote
> > > it
> > > > > into
> > > > > > > the
> > > > > > > > ASCII file, I didn't have any flags from my MGDRlite
files.
> > This
> > > > is
> > > > > > part
> > > > > > > > of the problem...
> > > > > > > >
> > > > > > > > Now, your answer for the STAT-Analysis, is that what I
did
> > wrong
> > > in
> > > > > my
> > > > > > > use
> > > > > > > > of stat-analysis from the previous email? Or, did you
not
> get
> > a
> > > > > chance
> > > > > > > to
> > > > > > > > look at my *stat file that I uploaded to the ftp site?
> > > > > > > >
> > > > > > > > Roz
> > > > > > > >
> > > > > > > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley Gotway
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > > Roz,
> > > > > > > > >
> > > > > > > > > Point-Stat takes as input a gridded NWP forecast
file and
> the
> > > > > NetCDF
> > > > > > > > output
> > > > > > > > > of a point pre-processing tool (like ascii2nc or
pb2nc).
> > Based
> > > > on
> > > > > > your
> > > > > > > > > description, it doesn't sound like you have gridded
NWP
> > > forecast
> > > > > > files
> > > > > > > > > available. So no, I don't think that approach would
work.
> > > > > > > > >
> > > > > > > > > The MPR output line type does include a column named
> "OBS_QC"
> > > > which
> > > > > > is
> > > > > > > a
> > > > > > > > > string containing the observation quality control
value.
> I'd
> > > > > suggest
> > > > > > > > > checking the contents of that column in your MPR
output
> lines
> > > to
> > > > > see
> > > > > > if
> > > > > > > > > those values could be filtered to do the quality
control
> > > > filtering
> > > > > > > you're
> > > > > > > > > after. Check to see if the values in that column
are
> strings
> > > or
> > > > > > > numbers
> > > > > > > > > (in PREPBUFR they're integers... in MADIS they're
strings).
> > > > > > > > >
> > > > > > > > > In STAT-Analysis, you can use the "-column_thresh"
option
> to
> > > > > > threshold
> > > > > > > a
> > > > > > > > > numeric column or you can use the "-column_str"
option to
> > list
> > > > > > strings
> > > > > > > to
> > > > > > > > > retained. For example:
> > > > > > > > > -column_thresh OBS_QC 'ge0&&le3' # this keeps
all
> values
> > > > > between
> > > > > > 0
> > > > > > > > and
> > > > > > > > > 3
> > > > > > > > > -column_str OBS_QC aa,ab,ac # this keeps
strings
> > > "aa",
> > > > > > "ab",
> > > > > > > > or
> > > > > > > > > "ac"
> > > > > > > > >
> > > > > > > > > Hope that helps.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken -
NOAA
> > > > Affiliate
> > > > > > via
> > > > > > > > RT
> > > > > > > > > <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=86480
> > > > >
> > > > > > > > > >
> > > > > > > > > > Hi John,
> > > > > > > > > >
> > > > > > > > > > I have another question related to my reprocessing
of my
> > > data.
> > > > > > > > > >
> > > > > > > > > > Ok, so, I realized that when I switched ASCAT data
> sources
> > > back
> > > > > in
> > > > > > > > March,
> > > > > > > > > > from prepbufr to MGDRlite files, I should have
been using
> > > > quality
> > > > > > > flags
> > > > > > > > > to
> > > > > > > > > > filter out my data. Because I wasn't checking for
the
> > > quality
> > > > > > flags,
> > > > > > > > my
> > > > > > > > > > statistics were not so good, since it was letting
in bad
> > > data,
> > > > > and
> > > > > > > > stats
> > > > > > > > > > were being calculated from this bad data.
> > > > > > > > > >
> > > > > > > > > > So, I need to find a way to correct this, but, the
old
> > story
> > > > is I
> > > > > > > don't
> > > > > > > > > > have the original GFS data, and it's too massive
to grab
> > from
> > > > > HPSS.
> > > > > > > I
> > > > > > > > do
> > > > > > > > > > have the original ASCAT files, which I can run
through my
> > > > program
> > > > > > to
> > > > > > > > > filter
> > > > > > > > > > out the bad observations, as well as matched
points from
> > the
> > > > *mpr
> > > > > > and
> > > > > > > > > *stat
> > > > > > > > > > files.
> > > > > > > > > >
> > > > > > > > > > Originally, I was just going to filter out using
that
> > > > > > -column_thresh
> > > > > > > > > > command, and filter out over land. But, I had
this idea
> > > that I
> > > > > > > wanted
> > > > > > > > to
> > > > > > > > > > run by you, to see if this makes sense to do.
> > > > > > > > > >
> > > > > > > > > > I have the time, lat, lon and GFS observation and
the
> > > > > corresponding
> > > > > > > > ASCAT
> > > > > > > > > > observation point. Can I do this:
> > > > > > > > > > 1) read the time, lat, lon and GFS observation
out of
> the
> > > mpr
> > > > > > file,
> > > > > > > > and
> > > > > > > > > > make a new input file with just those variables in
the
> > ASCII
> > > > > format
> > > > > > > > that
> > > > > > > > > > MET can use
> > > > > > > > > > 2) reprocess my ASCAT data and write it into an
ASCII
> file
> > > in
> > > > > MET
> > > > > > > > format
> > > > > > > > > > 3) run ASCII2NC on these files, to create the two
input
> > > files
> > > > > for
> > > > > > > MET,
> > > > > > > > > > then, run point_stat on the netcdf files to
regenerate
> the
> > > new
> > > > > > > > corrected
> > > > > > > > > > stat files
> > > > > > > > > >
> > > > > > > > > > Does that make sense to do? I know it could be
alot of
> > work,
> > > > > but,
> > > > > > > if I
> > > > > > > > > can
> > > > > > > > > > get corrected statistics, it might be worth it.
What do
> > you
> > > > > think?
> > > > > > > > > >
> > > > > > > > > > Roz
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn
MacCracken -
> NOAA
> > > > > > Affiliate
> > > > > > > <
> > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi John,
> > > > > > > > > > >
> > > > > > > > > > > I double checked all those things you listed,
and they
> > are
> > > > all
> > > > > > > > correct.
> > > > > > > > > > > So, I've put a file in the /maccracken directory
on
> your
> > > ftp
> > > > > > site.
> > > > > > > > Let
> > > > > > > > > > me
> > > > > > > > > > > know what you see.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your help!
> > > > > > > > > > >
> > > > > > > > > > > Roz
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn
MacCracken -
> > NOAA
> > > > > > > Affiliate
> > > > > > > > <
> > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi John,
> > > > > > > > > > >>
> > > > > > > > > > >> I'm sorry I didn't get back to you sooner, but,
I was
> > tied
> > > > up
> > > > > > with
> > > > > > > > > some
> > > > > > > > > > >> other things. Let me double check with all of
those
> > items
> > > > in
> > > > > my
> > > > > > > > file
> > > > > > > > > > and
> > > > > > > > > > >> make sure that I don't have some sort of
obvious
> error.
> > > > I'll
> > > > > be
> > > > > > > > back
> > > > > > > > > in
> > > > > > > > > > >> touch after I check on this.
> > > > > > > > > > >>
> > > > > > > > > > >> Roz
> > > > > > > > > > >>
> > > > > > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley
Gotway via
> > RT
> > > <
> > > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> Hi Roz,
> > > > > > > > > > >>>
> > > > > > > > > > >>> I see you're having trouble getting with
> STAT-Analysis,
> > > > > getting
> > > > > > > MPR
> > > > > > > > > > lines
> > > > > > > > > > >>> to be included in the column thresholds you've
> defined.
> > > I
> > > > > > don't
> > > > > > > > see
> > > > > > > > > > >>> anything obviously wrong with what you've
defined.
> > > > > > > > > > >>>
> > > > > > > > > > >>> The warning message indicates that the filters
you
> > > defined
> > > > > have
> > > > > > > > > > discarded
> > > > > > > > > > >>> all of the input MPR lines.
> > > > > > > > > > >>>
> > > > > > > > > > >>> We'd just need to go through your filtering
options
> > > > > one-by-one
> > > > > > to
> > > > > > > > > find
> > > > > > > > > > >>> out
> > > > > > > > > > >>> why.
> > > > > > > > > > >>>
> > > > > > > > > > >>> - Are you passing STAT-Analysis files which
contain
> MPR
> > > > > lines?
> > > > > > > > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > > > > > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > > > > > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > > > > > > > >>> - Does the OBS column have values > 1.0?
> > > > > > > > > > >>> - What are the ranges of the OBS_LAT and
OBS_LON
> > columns?
> > > > > > > > > > >>>
> > > > > > > > > > >>> If you'd like to send me a sample .stat file,
I could
> > > > > probably
> > > > > > > > track
> > > > > > > > > it
> > > > > > > > > > >>> down.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Thanks,
> > > > > > > > > > >>> John
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn
MacCracken -
> > NOAA
> > > > > > > Affiliate
> > > > > > > > > via
> > > > > > > > > > >>> RT <
> > > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Mon Aug 06 07:43:46 2018: Request 86480 was
acted
> > upon.
> > > > > > > > > > >>> > Transaction: Ticket created by
> > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > >>> > Queue: met_help
> > > > > > > > > > >>> > Subject: Errors using -column_thresh
> > > > > > > > > > >>> > Owner: Nobody
> > > > > > > > > > >>> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > > > > > > >>> > Status: new
> > > > > > > > > > >>> > Ticket <URL:
> > > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > > > > > >>> >
> > > > > > > > > > >>> >
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Hi,
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > I'm trying to use -column_thresh with
stat_analysis
> > to
> > > > mask
> > > > > > > out a
> > > > > > > > > > >>> lat/lon
> > > > > > > > > > >>> > box. I've added this command in my
StatAnalysis
> > config
> > > > > file:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > "-job aggregate_stat -out_line_type CTC
> > -column_thresh
> > > > OBS
> > > > > > > gt1.0
> > > > > > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > > > > > >>> > -column_thresh OBS_LAT 'ge15.0&&le70.0'
\
> > > > > > > > > > >>> > -column_thresh OBS_LON 'ge-100.0&&le-
10.0' \
> > > > > > > > > > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > > > > > > > > >>> > -out_stat /opc_test/home/opc_test/data/m
> > > > > > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > > > > > >>> >
> > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > But, when it runs, I get these errors:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Output:
> > > > > > > > > > >>> > DEBUG 2: Processing Job 2: -job
aggregate_stat
> > > -fcst_var
> > > > > WIND
> > > > > > > > > > >>> -fcst_lev Z10
> > > > > > > > > > >>> > -obtype ASCAT -line_type MPR -column_thresh
OBS
> >1.0
> > > > > > > > -column_thresh
> > > > > > > > > > >>> OBS_LAT
> > > > > > > > > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON
> > >=-100.0&&<=-10.0
> > > > -by
> > > > > > > > > > >>> FCST_VALID_BEG
> > > > > > > > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> > > > > > > /opc_test/home/opc_test/data/
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > new_out_stat_test2/agg_stat_
> > > > > MPR
> > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > > > > >>> > DEBUG 1: Creating STAT output file
> > > > > > > "/opc_test/home/opc_test/data/
> > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > new_out_stat_test2/agg_stat_
> > > > > > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > > > > > > >>> > DEBUG 2: Computing output for 0 case(s).
> > > > > > > > > > >>> > WARNING:
> > > > > > > > > > >>> > WARNING: do_job_aggr_stat() -> no matching
STAT
> lines
> > > > found
> > > > > > for
> > > > > > > > > job:
> > > > > > > > > > >>> -job
> > > > > > > > > > >>> > aggregate_stat -fcst_var WIND -fcst_lev Z10
-obtype
> > > ASCAT
> > > > > > > > > -line_type
> > > > > > > > > > >>> MPR
> > > > > > > > > > >>> > -column_thresh OBS >1.0 -column_thresh
OBS_LAT
> > > > > >=15.0&&<=70.0
> > > > > > > > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0 -by
> > > > FCST_VALID_BEG
> > > > > > -by
> > > > > > > > > > >>> FCST_LEAD
> > > > > > > > > > >>> > -by VX_MASK -out_stat
/opc_test/home/opc_test/data/
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > new_out_stat_test2/agg_stat_
> > > > > MPR
> > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > > > > >>> > WARNING:
> > > > > > > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT
lines.
> > > > > > > > > > >>> > DEBUG 2:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Am I not using the command properly?
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Thanks!
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Roz
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > --
> > > > > > > > > > >>> > Rosalyn MacCracken
> > > > > > > > > > >>> > Support Scientist
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Ocean Applications Branch
> > > > > > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > >>> > NCWCP
> > > > > > > > > > >>> > 5830 University Research Ct
> > > > > > > > > > >>> > College Park, MD 20740-3818
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > (p) 301-683-1551
> > > > > > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > > > > > >>> >
> > > > > > > > > > >>> >
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >> Rosalyn MacCracken
> > > > > > > > > > >> Support Scientist
> > > > > > > > > > >>
> > > > > > > > > > >> Ocean Applications Branch
> > > > > > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > > > > > >> NCWCP
> > > > > > > > > > >> 5830 University Research Ct
> > > > > > > > > > >> College Park, MD 20740-3818
> > > > > > > > > > >>
> > > > > > > > > > >> (p) 301-683-1551
> > > > > > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > Support Scientist
> > > > > > > > > > >
> > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > NCWCP
> > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > >
> > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > Support Scientist
> > > > > > > > > >
> > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > NCWCP
> > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > >
> > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rosalyn MacCracken
> > > > > > > > Support Scientist
> > > > > > > >
> > > > > > > > Ocean Applications Branch
> > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > NCWCP
> > > > > > > > 5830 University Research Ct
> > > > > > > > College Park, MD 20740-3818
> > > > > > > >
> > > > > > > > (p) 301-683-1551
> > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applications Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Errors using -column_thresh
From: John Halley Gotway
Time: Tue Aug 14 13:33:25 2018
Roz,
Sure, you can write a constant value to the OBS_QC column. And using
0
versus a 1 is just fine.
John
On Mon, Aug 13, 2018 at 1:32 PM Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
>
> No, that's correct. Originally, we weren't checking for the flag,
and just
> taking the observation from the MEDRlite file, and writing it
directly to a
> MET ASCII format file. So, as a result, I have MPR files that have
N/A
> where a QC flag number should be.
>
> Now, we are using bit position in place of the quality control flag.
So, I
> take the binary MGDRlite data, read it into a python program,
evaluate if
> the bit position is a "1". If it's a "0", then it writes a line to
an
> ASCII file that has the MET format to be used with ASCII2NC. If
it's a
> "1", it skips writing that value to the MET formatted file. This
way, we
> are only keeping good observations and tossing the others. That's
why I
> asked about just using some constant value, because it's bit
position and
> not a QC flag. I was thinking it it's something other than N/A,
then it
> won't be used. And, it's only for the back files. I can correct it
going
> forward, and just not write that observation to my ASCII file.
>
> Roz
>
>
>
> On Mon, Aug 13, 2018 at 7:20 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Roz,
> >
> > I'm confused. Here's is what I understood the situation to be...
> >
> > (1) In March, you switched from pulling ASCAT data from PREPBUFR
to
> pulling
> > it from MGDRlite.
> > (2) When you made that switch, you forgot to set the system up to
filter
> > the MGDRlite data by quality control flags.
> > (3) As a result of that, many "bad" observations were included in
your
> > evaluation which led to bad statistics.
> > (4) You now want to go back and correct the bad evaluation by
> reprocessing
> > using the actual quality control flags.
> >
> > I suggested that you modify the MET output MPR lines by replacing
the NA
> > string with the *actual* quality control flag. This logic only
works if
> > the MGDRlite data source includes quality control information...
and you
> > know how to extract it.
> >
> > Just putting a constant flag value of 2 in the OBS_QC column won't
really
> > fix anything. I'm suggesting that you get the actual flag values
and
> > "patch" the existing MPR lines by inserting them.
> >
> > Am I misunderstanding the situation?
> >
> > Thanks,
> > John
> >
> > On Mon, Aug 13, 2018 at 12:47 PM Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > >
> > > No, I don't know the correct OBS_QC value. Can I just make
something
> up,
> > > like use "2", or some arbitrary number? Or does MET expect some
value,
> > > given a certain file type?
> > >
> > > Actually, I currently don't write these values into my point
file,
> which
> > I
> > > use to make the netcdf file. So, it's just for the back data.
> > >
> > > Roz
> > >
> > > On Mon, Aug 13, 2018 at 6:41 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Right, updating the OBS_QC values from NA to the real QC value
is the
> > > > difficult part, but I think that's probably easier that
"gridding"
> the
> > > > point data and going back through Point-Stat.
> > > >
> > > > So do you know how to get the QBS_QC value from the MGDRlite
data?
> > And,
> > > > going forward, can you set it up so that the OBS_QC column is
> populated
> > > in
> > > > your Point-Stat runs going forward?
> > > >
> > > > Perhaps that's the place to start. Make sure your new Point-
Stat
> > output
> > > > includes meaningful OBS_QC output. Once that's working, go
through
> > your
> > > > *old* output and backfill the OBS_QC column with what it
should
> > contain.
> > > >
> > > > John
> > > >
> > > > On Mon, Aug 13, 2018 at 12:37 PM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > RT
> > > > <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > >
> > > > > Yes, that makes sense. So, the MPR line would be "N/A" that
it
> would
> > > > > process the data with. Ok. And, I could do something
similar to
> > what
> > > I
> > > > > did before with copying my MPR files to a temp directory,
and use
> > > > -lookin.
> > > > >
> > > > > Ok, so, the hard part will be to write this script that
replaces my
> > bad
> > > > > values. I guess I like a challenge, right?
> > > > >
> > > > > Roz
> > > > >
> > > > > On Mon, Aug 13, 2018 at 6:30 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Roz,
> > > > > >
> > > > > > Once the OBS_QC values are set in the MPR lines, you can
use
> > > > > STAT-Analysis
> > > > > > to reprocess the data and recompute whatever other STAT
lines
> you'd
> > > > like
> > > > > to
> > > > > > compute:
> > > > > >
> > > > > > stat_analysis -job aggregate_stat -line_type MPR
-out_line_type
> CNT
> > > > > > -column_str OBS_QC a,b,c
> > > > > >
> > > > > > This job would be run only on MPR lines that have strings
of "a",
> > > "b",
> > > > or
> > > > > > "c" in the OBS_QC column. Make sense?
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Mon, Aug 13, 2018 at 11:55 AM Rosalyn MacCracken - NOAA
> > Affiliate
> > > > via
> > > > > RT
> > > > > > <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Yeah, I can repost the file. I'll do that in a bit.
Actually,
> > if
> > > I
> > > > > can
> > > > > > > get this issue solved, you don't need to look at the
file,
> > because
> > > > I'll
> > > > > > > just reprocess that data with my new method. So, I'll
hold off
> > on
> > > > > ftping
> > > > > > > that file to you for right now.
> > > > > > >
> > > > > > > So, the OBS_QC column has N/A values. That's easy
enough to
> > > replace.
> > > > > > Now,
> > > > > > > I have the original ASCAT data, which looks for the
quality
> flag
> > > as a
> > > > > bit
> > > > > > > (i.e., ibits[5]) and then doesn't use that. It doesn't
write
> it
> > to
> > > > the
> > > > > > > output file. Maybe what I should do is find the first
bad
> > > > observation
> > > > > > > (time, lat, lon, value), then, match it exactly to the
line in
> > the
> > > > mpr
> > > > > > > file, and replace the observation.
> > > > > > >
> > > > > > > So, if my observations are in dataframes, and the mpr
file has
> > been
> > > > > read
> > > > > > in
> > > > > > > as a dataframe, then, I could do a df.query('Lat = ??
and Lon
> > =??),
> > > > or
> > > > > > > however the syntax is...no wait, I need an if statement
to
> > compare
> > > > > > > first...I'll work on those detail. Then, once I get the
OBS_QC
> > > value
> > > > > > > changed, then what? How do you get the other stat files
from
> > this
> > > > > point?
> > > > > > >
> > > > > > > Roz
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Aug 13, 2018 at 5:32 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > > Roz,
> > > > > > > >
> > > > > > > > That's correct, there is no tool in MET for comparing
one set
> > of
> > > > > point
> > > > > > > > observations to another set of point observations.
> Presumably
> > > you
> > > > > > could
> > > > > > > > write point observations to a grid and then
reconfigure and
> > rerun
> > > > > > > > Point-Stat, but that'd be pretty difficult, time
consuming,
> and
> > > > > fraught
> > > > > > > > with problems.
> > > > > > > >
> > > > > > > > You currently have many, many MPR lines and the issue
is the
> > the
> > > > > OBS_QC
> > > > > > > > column is empty. If that OBS_QC column were *not*
empty then
> > > it'd
> > > > be
> > > > > > > > pretty easy to process this data... correct? One
option
> would
> > be
> > > > > > > writing a
> > > > > > > > script to "patch" that column. For each MPR line, do
some
> > > > processing
> > > > > > to
> > > > > > > > figure out what that OBS_QC column should be and then
update
> > it's
> > > > > > value.
> > > > > > > >
> > > > > > > > Then the question is this... is there a way of getting
> quality
> > > > > control
> > > > > > > > values from the MGDRlite files?
> > > > > > > >
> > > > > > > > I'm sorry, I hadn't had a chance to look at that data
file
> yet.
> > > > But
> > > > > I
> > > > > > > just
> > > > > > > > checked on the ftp site and don't see a maccracken
> > sub-directory
> > > > > there:
> > > > > > > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > > > > > > >
> > > > > > > > Could you please repost?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn MacCracken -
NOAA
> > > > Affiliate
> > > > > > via
> > > > > > > RT
> > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=86480
> > > >
> > > > > > > > >
> > > > > > > > > So, you can't compare two point files, even if both
of them
> > are
> > > > > > NetCDF
> > > > > > > > > format? Is there a way to write point observations
out to
> a
> > > > grid?
> > > > > > In
> > > > > > > > > other words, take the point GFS observations, and
write
> them
> > > to a
> > > > > > > global
> > > > > > > > > grid (using some other tool, like wgrib or something
else),
> > and
> > > > > fill
> > > > > > in
> > > > > > > > > missing values with NaN, or something? Then, use
that
> > gridded
> > > > file
> > > > > > to
> > > > > > > > > compare with the ASCAT NetCDF file?
> > > > > > > > >
> > > > > > > > > I don't have anything in the OBS_QC column because
when I
> > wrote
> > > > it
> > > > > > into
> > > > > > > > the
> > > > > > > > > ASCII file, I didn't have any flags from my MGDRlite
files.
> > > This
> > > > > is
> > > > > > > part
> > > > > > > > > of the problem...
> > > > > > > > >
> > > > > > > > > Now, your answer for the STAT-Analysis, is that what
I did
> > > wrong
> > > > in
> > > > > > my
> > > > > > > > use
> > > > > > > > > of stat-analysis from the previous email? Or, did
you not
> > get
> > > a
> > > > > > chance
> > > > > > > > to
> > > > > > > > > look at my *stat file that I uploaded to the ftp
site?
> > > > > > > > >
> > > > > > > > > Roz
> > > > > > > > >
> > > > > > > > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley Gotway
via RT
> <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > > Roz,
> > > > > > > > > >
> > > > > > > > > > Point-Stat takes as input a gridded NWP forecast
file and
> > the
> > > > > > NetCDF
> > > > > > > > > output
> > > > > > > > > > of a point pre-processing tool (like ascii2nc or
pb2nc).
> > > Based
> > > > > on
> > > > > > > your
> > > > > > > > > > description, it doesn't sound like you have
gridded NWP
> > > > forecast
> > > > > > > files
> > > > > > > > > > available. So no, I don't think that approach
would
> work.
> > > > > > > > > >
> > > > > > > > > > The MPR output line type does include a column
named
> > "OBS_QC"
> > > > > which
> > > > > > > is
> > > > > > > > a
> > > > > > > > > > string containing the observation quality control
value.
> > I'd
> > > > > > suggest
> > > > > > > > > > checking the contents of that column in your MPR
output
> > lines
> > > > to
> > > > > > see
> > > > > > > if
> > > > > > > > > > those values could be filtered to do the quality
control
> > > > > filtering
> > > > > > > > you're
> > > > > > > > > > after. Check to see if the values in that column
are
> > strings
> > > > or
> > > > > > > > numbers
> > > > > > > > > > (in PREPBUFR they're integers... in MADIS they're
> strings).
> > > > > > > > > >
> > > > > > > > > > In STAT-Analysis, you can use the "-column_thresh"
option
> > to
> > > > > > > threshold
> > > > > > > > a
> > > > > > > > > > numeric column or you can use the "-column_str"
option to
> > > list
> > > > > > > strings
> > > > > > > > to
> > > > > > > > > > retained. For example:
> > > > > > > > > > -column_thresh OBS_QC 'ge0&&le3' # this keeps
all
> > values
> > > > > > between
> > > > > > > 0
> > > > > > > > > and
> > > > > > > > > > 3
> > > > > > > > > > -column_str OBS_QC aa,ab,ac # this
keeps
> strings
> > > > "aa",
> > > > > > > "ab",
> > > > > > > > > or
> > > > > > > > > > "ac"
> > > > > > > > > >
> > > > > > > > > > Hope that helps.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn MacCracken
- NOAA
> > > > > Affiliate
> > > > > > > via
> > > > > > > > > RT
> > > > > > > > > > <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > Ticket/Display.html?id=86480
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi John,
> > > > > > > > > > >
> > > > > > > > > > > I have another question related to my
reprocessing of
> my
> > > > data.
> > > > > > > > > > >
> > > > > > > > > > > Ok, so, I realized that when I switched ASCAT
data
> > sources
> > > > back
> > > > > > in
> > > > > > > > > March,
> > > > > > > > > > > from prepbufr to MGDRlite files, I should have
been
> using
> > > > > quality
> > > > > > > > flags
> > > > > > > > > > to
> > > > > > > > > > > filter out my data. Because I wasn't checking
for the
> > > > quality
> > > > > > > flags,
> > > > > > > > > my
> > > > > > > > > > > statistics were not so good, since it was
letting in
> bad
> > > > data,
> > > > > > and
> > > > > > > > > stats
> > > > > > > > > > > were being calculated from this bad data.
> > > > > > > > > > >
> > > > > > > > > > > So, I need to find a way to correct this, but,
the old
> > > story
> > > > > is I
> > > > > > > > don't
> > > > > > > > > > > have the original GFS data, and it's too massive
to
> grab
> > > from
> > > > > > HPSS.
> > > > > > > > I
> > > > > > > > > do
> > > > > > > > > > > have the original ASCAT files, which I can run
through
> my
> > > > > program
> > > > > > > to
> > > > > > > > > > filter
> > > > > > > > > > > out the bad observations, as well as matched
points
> from
> > > the
> > > > > *mpr
> > > > > > > and
> > > > > > > > > > *stat
> > > > > > > > > > > files.
> > > > > > > > > > >
> > > > > > > > > > > Originally, I was just going to filter out using
that
> > > > > > > -column_thresh
> > > > > > > > > > > command, and filter out over land. But, I had
this
> idea
> > > > that I
> > > > > > > > wanted
> > > > > > > > > to
> > > > > > > > > > > run by you, to see if this makes sense to do.
> > > > > > > > > > >
> > > > > > > > > > > I have the time, lat, lon and GFS observation
and the
> > > > > > corresponding
> > > > > > > > > ASCAT
> > > > > > > > > > > observation point. Can I do this:
> > > > > > > > > > > 1) read the time, lat, lon and GFS observation
out of
> > the
> > > > mpr
> > > > > > > file,
> > > > > > > > > and
> > > > > > > > > > > make a new input file with just those variables
in the
> > > ASCII
> > > > > > format
> > > > > > > > > that
> > > > > > > > > > > MET can use
> > > > > > > > > > > 2) reprocess my ASCAT data and write it into an
ASCII
> > file
> > > > in
> > > > > > MET
> > > > > > > > > format
> > > > > > > > > > > 3) run ASCII2NC on these files, to create the
two
> input
> > > > files
> > > > > > for
> > > > > > > > MET,
> > > > > > > > > > > then, run point_stat on the netcdf files to
regenerate
> > the
> > > > new
> > > > > > > > > corrected
> > > > > > > > > > > stat files
> > > > > > > > > > >
> > > > > > > > > > > Does that make sense to do? I know it could be
alot of
> > > work,
> > > > > > but,
> > > > > > > > if I
> > > > > > > > > > can
> > > > > > > > > > > get corrected statistics, it might be worth it.
What
> do
> > > you
> > > > > > think?
> > > > > > > > > > >
> > > > > > > > > > > Roz
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn
MacCracken -
> > NOAA
> > > > > > > Affiliate
> > > > > > > > <
> > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi John,
> > > > > > > > > > > >
> > > > > > > > > > > > I double checked all those things you listed,
and
> they
> > > are
> > > > > all
> > > > > > > > > correct.
> > > > > > > > > > > > So, I've put a file in the /maccracken
directory on
> > your
> > > > ftp
> > > > > > > site.
> > > > > > > > > Let
> > > > > > > > > > > me
> > > > > > > > > > > > know what you see.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your help!
> > > > > > > > > > > >
> > > > > > > > > > > > Roz
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn
MacCracken -
> > > NOAA
> > > > > > > > Affiliate
> > > > > > > > > <
> > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi John,
> > > > > > > > > > > >>
> > > > > > > > > > > >> I'm sorry I didn't get back to you sooner,
but, I
> was
> > > tied
> > > > > up
> > > > > > > with
> > > > > > > > > > some
> > > > > > > > > > > >> other things. Let me double check with all
of those
> > > items
> > > > > in
> > > > > > my
> > > > > > > > > file
> > > > > > > > > > > and
> > > > > > > > > > > >> make sure that I don't have some sort of
obvious
> > error.
> > > > > I'll
> > > > > > be
> > > > > > > > > back
> > > > > > > > > > in
> > > > > > > > > > > >> touch after I check on this.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Roz
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley
Gotway
> via
> > > RT
> > > > <
> > > > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >>> Hi Roz,
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> I see you're having trouble getting with
> > STAT-Analysis,
> > > > > > getting
> > > > > > > > MPR
> > > > > > > > > > > lines
> > > > > > > > > > > >>> to be included in the column thresholds
you've
> > defined.
> > > > I
> > > > > > > don't
> > > > > > > > > see
> > > > > > > > > > > >>> anything obviously wrong with what you've
defined.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> The warning message indicates that the
filters you
> > > > defined
> > > > > > have
> > > > > > > > > > > discarded
> > > > > > > > > > > >>> all of the input MPR lines.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> We'd just need to go through your filtering
options
> > > > > > one-by-one
> > > > > > > to
> > > > > > > > > > find
> > > > > > > > > > > >>> out
> > > > > > > > > > > >>> why.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> - Are you passing STAT-Analysis files which
contain
> > MPR
> > > > > > lines?
> > > > > > > > > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > > > > > > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > > > > > > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > > > > > > > > >>> - Does the OBS column have values > 1.0?
> > > > > > > > > > > >>> - What are the ranges of the OBS_LAT and
OBS_LON
> > > columns?
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> If you'd like to send me a sample .stat
file, I
> could
> > > > > > probably
> > > > > > > > > track
> > > > > > > > > > it
> > > > > > > > > > > >>> down.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Thanks,
> > > > > > > > > > > >>> John
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn
MacCracken -
> > > NOAA
> > > > > > > > Affiliate
> > > > > > > > > > via
> > > > > > > > > > > >>> RT <
> > > > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Mon Aug 06 07:43:46 2018: Request 86480
was acted
> > > upon.
> > > > > > > > > > > >>> > Transaction: Ticket created by
> > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > >>> > Queue: met_help
> > > > > > > > > > > >>> > Subject: Errors using -column_thresh
> > > > > > > > > > > >>> > Owner: Nobody
> > > > > > > > > > > >>> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > > > > > > > >>> > Status: new
> > > > > > > > > > > >>> > Ticket <URL:
> > > > > > > > > > >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Hi,
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > I'm trying to use -column_thresh with
> stat_analysis
> > > to
> > > > > mask
> > > > > > > > out a
> > > > > > > > > > > >>> lat/lon
> > > > > > > > > > > >>> > box. I've added this command in my
StatAnalysis
> > > config
> > > > > > file:
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > "-job aggregate_stat -out_line_type CTC
> > > -column_thresh
> > > > > OBS
> > > > > > > > gt1.0
> > > > > > > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > > > > > > >>> > -column_thresh OBS_LAT
'ge15.0&&le70.0' \
> > > > > > > > > > > >>> > -column_thresh OBS_LON 'ge-100.0&&le-
10.0' \
> > > > > > > > > > > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > > > > > > > > > >>> > -out_stat
/opc_test/home/opc_test/data/m
> > > > > > > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > > > > > > >>> >
> > > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > But, when it runs, I get these errors:
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Output:
> > > > > > > > > > > >>> > DEBUG 2: Processing Job 2: -job
aggregate_stat
> > > > -fcst_var
> > > > > > WIND
> > > > > > > > > > > >>> -fcst_lev Z10
> > > > > > > > > > > >>> > -obtype ASCAT -line_type MPR
-column_thresh OBS
> > >1.0
> > > > > > > > > -column_thresh
> > > > > > > > > > > >>> OBS_LAT
> > > > > > > > > > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON
> > > >=-100.0&&<=-10.0
> > > > > -by
> > > > > > > > > > > >>> FCST_VALID_BEG
> > > > > > > > > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> > > > > > > > /opc_test/home/opc_test/data/
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > new_out_stat_test2/agg_stat_
> > > > > > MPR
> > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > > > > > >>> > DEBUG 1: Creating STAT output file
> > > > > > > > "/opc_test/home/opc_test/data/
> > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > new_out_stat_test2/agg_stat_
> > > > > > > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > > > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > > > > > > > >>> > DEBUG 2: Computing output for 0 case(s).
> > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > >>> > WARNING: do_job_aggr_stat() -> no matching
STAT
> > lines
> > > > > found
> > > > > > > for
> > > > > > > > > > job:
> > > > > > > > > > > >>> -job
> > > > > > > > > > > >>> > aggregate_stat -fcst_var WIND -fcst_lev
Z10
> -obtype
> > > > ASCAT
> > > > > > > > > > -line_type
> > > > > > > > > > > >>> MPR
> > > > > > > > > > > >>> > -column_thresh OBS >1.0 -column_thresh
OBS_LAT
> > > > > > >=15.0&&<=70.0
> > > > > > > > > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0
-by
> > > > > FCST_VALID_BEG
> > > > > > > -by
> > > > > > > > > > > >>> FCST_LEAD
> > > > > > > > > > > >>> > -by VX_MASK -out_stat
> /opc_test/home/opc_test/data/
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > new_out_stat_test2/agg_stat_
> > > > > > MPR
> > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT
lines.
> > > > > > > > > > > >>> > DEBUG 2:
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Am I not using the command properly?
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Thanks!
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Roz
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > --
> > > > > > > > > > > >>> > Rosalyn MacCracken
> > > > > > > > > > > >>> > Support Scientist
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Ocean Applications Branch
> > > > > > > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > >>> > NCWCP
> > > > > > > > > > > >>> > 5830 University Research Ct
> > > > > > > > > > > >>> > College Park, MD 20740-3818
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > (p) 301-683-1551
> > > > > > > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> --
> > > > > > > > > > > >> Rosalyn MacCracken
> > > > > > > > > > > >> Support Scientist
> > > > > > > > > > > >>
> > > > > > > > > > > >> Ocean Applications Branch
> > > > > > > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > >> NCWCP
> > > > > > > > > > > >> 5830 University Research Ct
> > > > > > > > > > > >> College Park, MD 20740-3818
> > > > > > > > > > > >>
> > > > > > > > > > > >> (p) 301-683-1551
> > > > > > > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > Support Scientist
> > > > > > > > > > > >
> > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > NCWCP
> > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > >
> > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > Support Scientist
> > > > > > > > > > >
> > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > NCWCP
> > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > >
> > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rosalyn MacCracken
> > > > > > > > > Support Scientist
> > > > > > > > >
> > > > > > > > > Ocean Applications Branch
> > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > NCWCP
> > > > > > > > > 5830 University Research Ct
> > > > > > > > > College Park, MD 20740-3818
> > > > > > > > >
> > > > > > > > > (p) 301-683-1551
> > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rosalyn MacCracken
> > > > > > > Support Scientist
> > > > > > >
> > > > > > > Ocean Applications Branch
> > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > NCWCP
> > > > > > > 5830 University Research Ct
> > > > > > > College Park, MD 20740-3818
> > > > > > >
> > > > > > > (p) 301-683-1551
> > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applications Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Wed Aug 15 08:04:26 2018
Ok, I think that I could just set the bad observations in that column
to 1.
So, if I do that, how will they be ignored in further processing? Do
I
need to set something so those observations will be skipped?
Roz
On Tue, Aug 14, 2018 at 7:33 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Roz,
>
> Sure, you can write a constant value to the OBS_QC column. And
using 0
> versus a 1 is just fine.
>
> John
>
> On Mon, Aug 13, 2018 at 1:32 PM Rosalyn MacCracken - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> >
> > No, that's correct. Originally, we weren't checking for the flag,
and
> just
> > taking the observation from the MEDRlite file, and writing it
directly
> to a
> > MET ASCII format file. So, as a result, I have MPR files that
have N/A
> > where a QC flag number should be.
> >
> > Now, we are using bit position in place of the quality control
flag.
> So, I
> > take the binary MGDRlite data, read it into a python program,
evaluate if
> > the bit position is a "1". If it's a "0", then it writes a line
to an
> > ASCII file that has the MET format to be used with ASCII2NC. If
it's a
> > "1", it skips writing that value to the MET formatted file. This
way, we
> > are only keeping good observations and tossing the others. That's
why I
> > asked about just using some constant value, because it's bit
position and
> > not a QC flag. I was thinking it it's something other than N/A,
then it
> > won't be used. And, it's only for the back files. I can correct
it
> going
> > forward, and just not write that observation to my ASCII file.
> >
> > Roz
> >
> >
> >
> > On Mon, Aug 13, 2018 at 7:20 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Roz,
> > >
> > > I'm confused. Here's is what I understood the situation to
be...
> > >
> > > (1) In March, you switched from pulling ASCAT data from PREPBUFR
to
> > pulling
> > > it from MGDRlite.
> > > (2) When you made that switch, you forgot to set the system up
to
> filter
> > > the MGDRlite data by quality control flags.
> > > (3) As a result of that, many "bad" observations were included
in your
> > > evaluation which led to bad statistics.
> > > (4) You now want to go back and correct the bad evaluation by
> > reprocessing
> > > using the actual quality control flags.
> > >
> > > I suggested that you modify the MET output MPR lines by
replacing the
> NA
> > > string with the *actual* quality control flag. This logic only
works
> if
> > > the MGDRlite data source includes quality control information...
and
> you
> > > know how to extract it.
> > >
> > > Just putting a constant flag value of 2 in the OBS_QC column
won't
> really
> > > fix anything. I'm suggesting that you get the actual flag
values and
> > > "patch" the existing MPR lines by inserting them.
> > >
> > > Am I misunderstanding the situation?
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Aug 13, 2018 at 12:47 PM Rosalyn MacCracken - NOAA
Affiliate
> via
> > RT
> > > <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
>
> > > >
> > > > No, I don't know the correct OBS_QC value. Can I just make
something
> > up,
> > > > like use "2", or some arbitrary number? Or does MET expect
some
> value,
> > > > given a certain file type?
> > > >
> > > > Actually, I currently don't write these values into my point
file,
> > which
> > > I
> > > > use to make the netcdf file. So, it's just for the back data.
> > > >
> > > > Roz
> > > >
> > > > On Mon, Aug 13, 2018 at 6:41 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Right, updating the OBS_QC values from NA to the real QC
value is
> the
> > > > > difficult part, but I think that's probably easier that
"gridding"
> > the
> > > > > point data and going back through Point-Stat.
> > > > >
> > > > > So do you know how to get the QBS_QC value from the MGDRlite
data?
> > > And,
> > > > > going forward, can you set it up so that the OBS_QC column
is
> > populated
> > > > in
> > > > > your Point-Stat runs going forward?
> > > > >
> > > > > Perhaps that's the place to start. Make sure your new
Point-Stat
> > > output
> > > > > includes meaningful OBS_QC output. Once that's working, go
through
> > > your
> > > > > *old* output and backfill the OBS_QC column with what it
should
> > > contain.
> > > > >
> > > > > John
> > > > >
> > > > > On Mon, Aug 13, 2018 at 12:37 PM Rosalyn MacCracken - NOAA
> Affiliate
> > > via
> > > > RT
> > > > > <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > > >
> > > > > > Yes, that makes sense. So, the MPR line would be "N/A"
that it
> > would
> > > > > > process the data with. Ok. And, I could do something
similar to
> > > what
> > > > I
> > > > > > did before with copying my MPR files to a temp directory,
and use
> > > > > -lookin.
> > > > > >
> > > > > > Ok, so, the hard part will be to write this script that
replaces
> my
> > > bad
> > > > > > values. I guess I like a challenge, right?
> > > > > >
> > > > > > Roz
> > > > > >
> > > > > > On Mon, Aug 13, 2018 at 6:30 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Roz,
> > > > > > >
> > > > > > > Once the OBS_QC values are set in the MPR lines, you can
use
> > > > > > STAT-Analysis
> > > > > > > to reprocess the data and recompute whatever other STAT
lines
> > you'd
> > > > > like
> > > > > > to
> > > > > > > compute:
> > > > > > >
> > > > > > > stat_analysis -job aggregate_stat -line_type MPR
-out_line_type
> > CNT
> > > > > > > -column_str OBS_QC a,b,c
> > > > > > >
> > > > > > > This job would be run only on MPR lines that have
strings of
> "a",
> > > > "b",
> > > > > or
> > > > > > > "c" in the OBS_QC column. Make sense?
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Aug 13, 2018 at 11:55 AM Rosalyn MacCracken -
NOAA
> > > Affiliate
> > > > > via
> > > > > > RT
> > > > > > > <met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=86480
> > >
> > > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > Yeah, I can repost the file. I'll do that in a bit.
> Actually,
> > > if
> > > > I
> > > > > > can
> > > > > > > > get this issue solved, you don't need to look at the
file,
> > > because
> > > > > I'll
> > > > > > > > just reprocess that data with my new method. So, I'll
hold
> off
> > > on
> > > > > > ftping
> > > > > > > > that file to you for right now.
> > > > > > > >
> > > > > > > > So, the OBS_QC column has N/A values. That's easy
enough to
> > > > replace.
> > > > > > > Now,
> > > > > > > > I have the original ASCAT data, which looks for the
quality
> > flag
> > > > as a
> > > > > > bit
> > > > > > > > (i.e., ibits[5]) and then doesn't use that. It
doesn't write
> > it
> > > to
> > > > > the
> > > > > > > > output file. Maybe what I should do is find the first
bad
> > > > > observation
> > > > > > > > (time, lat, lon, value), then, match it exactly to the
line
> in
> > > the
> > > > > mpr
> > > > > > > > file, and replace the observation.
> > > > > > > >
> > > > > > > > So, if my observations are in dataframes, and the mpr
file
> has
> > > been
> > > > > > read
> > > > > > > in
> > > > > > > > as a dataframe, then, I could do a df.query('Lat = ??
and Lon
> > > =??),
> > > > > or
> > > > > > > > however the syntax is...no wait, I need an if
statement to
> > > compare
> > > > > > > > first...I'll work on those detail. Then, once I get
the
> OBS_QC
> > > > value
> > > > > > > > changed, then what? How do you get the other stat
files from
> > > this
> > > > > > point?
> > > > > > > >
> > > > > > > > Roz
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Aug 13, 2018 at 5:32 PM, John Halley Gotway
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > > Roz,
> > > > > > > > >
> > > > > > > > > That's correct, there is no tool in MET for
comparing one
> set
> > > of
> > > > > > point
> > > > > > > > > observations to another set of point observations.
> > Presumably
> > > > you
> > > > > > > could
> > > > > > > > > write point observations to a grid and then
reconfigure and
> > > rerun
> > > > > > > > > Point-Stat, but that'd be pretty difficult, time
consuming,
> > and
> > > > > > fraught
> > > > > > > > > with problems.
> > > > > > > > >
> > > > > > > > > You currently have many, many MPR lines and the
issue is
> the
> > > the
> > > > > > OBS_QC
> > > > > > > > > column is empty. If that OBS_QC column were *not*
empty
> then
> > > > it'd
> > > > > be
> > > > > > > > > pretty easy to process this data... correct? One
option
> > would
> > > be
> > > > > > > > writing a
> > > > > > > > > script to "patch" that column. For each MPR line,
do some
> > > > > processing
> > > > > > > to
> > > > > > > > > figure out what that OBS_QC column should be and
then
> update
> > > it's
> > > > > > > value.
> > > > > > > > >
> > > > > > > > > Then the question is this... is there a way of
getting
> > quality
> > > > > > control
> > > > > > > > > values from the MGDRlite files?
> > > > > > > > >
> > > > > > > > > I'm sorry, I hadn't had a chance to look at that
data file
> > yet.
> > > > > But
> > > > > > I
> > > > > > > > just
> > > > > > > > > checked on the ftp site and don't see a maccracken
> > > sub-directory
> > > > > > there:
> > > > > > > > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > > > > > > > >
> > > > > > > > > Could you please repost?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn MacCracken
- NOAA
> > > > > Affiliate
> > > > > > > via
> > > > > > > > RT
> > > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=86480
> > > > >
> > > > > > > > > >
> > > > > > > > > > So, you can't compare two point files, even if
both of
> them
> > > are
> > > > > > > NetCDF
> > > > > > > > > > format? Is there a way to write point
observations out
> to
> > a
> > > > > grid?
> > > > > > > In
> > > > > > > > > > other words, take the point GFS observations, and
write
> > them
> > > > to a
> > > > > > > > global
> > > > > > > > > > grid (using some other tool, like wgrib or
something
> else),
> > > and
> > > > > > fill
> > > > > > > in
> > > > > > > > > > missing values with NaN, or something? Then, use
that
> > > gridded
> > > > > file
> > > > > > > to
> > > > > > > > > > compare with the ASCAT NetCDF file?
> > > > > > > > > >
> > > > > > > > > > I don't have anything in the OBS_QC column because
when I
> > > wrote
> > > > > it
> > > > > > > into
> > > > > > > > > the
> > > > > > > > > > ASCII file, I didn't have any flags from my
MGDRlite
> files.
> > > > This
> > > > > > is
> > > > > > > > part
> > > > > > > > > > of the problem...
> > > > > > > > > >
> > > > > > > > > > Now, your answer for the STAT-Analysis, is that
what I
> did
> > > > wrong
> > > > > in
> > > > > > > my
> > > > > > > > > use
> > > > > > > > > > of stat-analysis from the previous email? Or, did
you
> not
> > > get
> > > > a
> > > > > > > chance
> > > > > > > > > to
> > > > > > > > > > look at my *stat file that I uploaded to the ftp
site?
> > > > > > > > > >
> > > > > > > > > > Roz
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley
Gotway via
> RT
> > <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > > Roz,
> > > > > > > > > > >
> > > > > > > > > > > Point-Stat takes as input a gridded NWP forecast
file
> and
> > > the
> > > > > > > NetCDF
> > > > > > > > > > output
> > > > > > > > > > > of a point pre-processing tool (like ascii2nc or
> pb2nc).
> > > > Based
> > > > > > on
> > > > > > > > your
> > > > > > > > > > > description, it doesn't sound like you have
gridded NWP
> > > > > forecast
> > > > > > > > files
> > > > > > > > > > > available. So no, I don't think that approach
would
> > work.
> > > > > > > > > > >
> > > > > > > > > > > The MPR output line type does include a column
named
> > > "OBS_QC"
> > > > > > which
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > > string containing the observation quality
control
> value.
> > > I'd
> > > > > > > suggest
> > > > > > > > > > > checking the contents of that column in your MPR
output
> > > lines
> > > > > to
> > > > > > > see
> > > > > > > > if
> > > > > > > > > > > those values could be filtered to do the quality
> control
> > > > > > filtering
> > > > > > > > > you're
> > > > > > > > > > > after. Check to see if the values in that
column are
> > > strings
> > > > > or
> > > > > > > > > numbers
> > > > > > > > > > > (in PREPBUFR they're integers... in MADIS
they're
> > strings).
> > > > > > > > > > >
> > > > > > > > > > > In STAT-Analysis, you can use the "-
column_thresh"
> option
> > > to
> > > > > > > > threshold
> > > > > > > > > a
> > > > > > > > > > > numeric column or you can use the "-column_str"
option
> to
> > > > list
> > > > > > > > strings
> > > > > > > > > to
> > > > > > > > > > > retained. For example:
> > > > > > > > > > > -column_thresh OBS_QC 'ge0&&le3' # this
keeps all
> > > values
> > > > > > > between
> > > > > > > > 0
> > > > > > > > > > and
> > > > > > > > > > > 3
> > > > > > > > > > > -column_str OBS_QC aa,ab,ac # this
keeps
> > strings
> > > > > "aa",
> > > > > > > > "ab",
> > > > > > > > > > or
> > > > > > > > > > > "ac"
> > > > > > > > > > >
> > > > > > > > > > > Hope that helps.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn
MacCracken -
> NOAA
> > > > > > Affiliate
> > > > > > > > via
> > > > > > > > > > RT
> > > > > > > > > > > <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > Ticket/Display.html?id=86480
> > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi John,
> > > > > > > > > > > >
> > > > > > > > > > > > I have another question related to my
reprocessing of
> > my
> > > > > data.
> > > > > > > > > > > >
> > > > > > > > > > > > Ok, so, I realized that when I switched ASCAT
data
> > > sources
> > > > > back
> > > > > > > in
> > > > > > > > > > March,
> > > > > > > > > > > > from prepbufr to MGDRlite files, I should have
been
> > using
> > > > > > quality
> > > > > > > > > flags
> > > > > > > > > > > to
> > > > > > > > > > > > filter out my data. Because I wasn't checking
for
> the
> > > > > quality
> > > > > > > > flags,
> > > > > > > > > > my
> > > > > > > > > > > > statistics were not so good, since it was
letting in
> > bad
> > > > > data,
> > > > > > > and
> > > > > > > > > > stats
> > > > > > > > > > > > were being calculated from this bad data.
> > > > > > > > > > > >
> > > > > > > > > > > > So, I need to find a way to correct this, but,
the
> old
> > > > story
> > > > > > is I
> > > > > > > > > don't
> > > > > > > > > > > > have the original GFS data, and it's too
massive to
> > grab
> > > > from
> > > > > > > HPSS.
> > > > > > > > > I
> > > > > > > > > > do
> > > > > > > > > > > > have the original ASCAT files, which I can run
> through
> > my
> > > > > > program
> > > > > > > > to
> > > > > > > > > > > filter
> > > > > > > > > > > > out the bad observations, as well as matched
points
> > from
> > > > the
> > > > > > *mpr
> > > > > > > > and
> > > > > > > > > > > *stat
> > > > > > > > > > > > files.
> > > > > > > > > > > >
> > > > > > > > > > > > Originally, I was just going to filter out
using that
> > > > > > > > -column_thresh
> > > > > > > > > > > > command, and filter out over land. But, I had
this
> > idea
> > > > > that I
> > > > > > > > > wanted
> > > > > > > > > > to
> > > > > > > > > > > > run by you, to see if this makes sense to do.
> > > > > > > > > > > >
> > > > > > > > > > > > I have the time, lat, lon and GFS observation
and the
> > > > > > > corresponding
> > > > > > > > > > ASCAT
> > > > > > > > > > > > observation point. Can I do this:
> > > > > > > > > > > > 1) read the time, lat, lon and GFS
observation out
> of
> > > the
> > > > > mpr
> > > > > > > > file,
> > > > > > > > > > and
> > > > > > > > > > > > make a new input file with just those
variables in
> the
> > > > ASCII
> > > > > > > format
> > > > > > > > > > that
> > > > > > > > > > > > MET can use
> > > > > > > > > > > > 2) reprocess my ASCAT data and write it into
an
> ASCII
> > > file
> > > > > in
> > > > > > > MET
> > > > > > > > > > format
> > > > > > > > > > > > 3) run ASCII2NC on these files, to create the
two
> > input
> > > > > files
> > > > > > > for
> > > > > > > > > MET,
> > > > > > > > > > > > then, run point_stat on the netcdf files to
> regenerate
> > > the
> > > > > new
> > > > > > > > > > corrected
> > > > > > > > > > > > stat files
> > > > > > > > > > > >
> > > > > > > > > > > > Does that make sense to do? I know it could
be alot
> of
> > > > work,
> > > > > > > but,
> > > > > > > > > if I
> > > > > > > > > > > can
> > > > > > > > > > > > get corrected statistics, it might be worth
it. What
> > do
> > > > you
> > > > > > > think?
> > > > > > > > > > > >
> > > > > > > > > > > > Roz
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn
MacCracken -
> > > NOAA
> > > > > > > > Affiliate
> > > > > > > > > <
> > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi John,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I double checked all those things you
listed, and
> > they
> > > > are
> > > > > > all
> > > > > > > > > > correct.
> > > > > > > > > > > > > So, I've put a file in the /maccracken
directory on
> > > your
> > > > > ftp
> > > > > > > > site.
> > > > > > > > > > Let
> > > > > > > > > > > > me
> > > > > > > > > > > > > know what you see.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your help!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Roz
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn
> MacCracken -
> > > > NOAA
> > > > > > > > > Affiliate
> > > > > > > > > > <
> > > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi John,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> I'm sorry I didn't get back to you sooner,
but, I
> > was
> > > > tied
> > > > > > up
> > > > > > > > with
> > > > > > > > > > > some
> > > > > > > > > > > > >> other things. Let me double check with all
of
> those
> > > > items
> > > > > > in
> > > > > > > my
> > > > > > > > > > file
> > > > > > > > > > > > and
> > > > > > > > > > > > >> make sure that I don't have some sort of
obvious
> > > error.
> > > > > > I'll
> > > > > > > be
> > > > > > > > > > back
> > > > > > > > > > > in
> > > > > > > > > > > > >> touch after I check on this.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Roz
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John Halley
Gotway
> > via
> > > > RT
> > > > > <
> > > > > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>> Hi Roz,
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> I see you're having trouble getting with
> > > STAT-Analysis,
> > > > > > > getting
> > > > > > > > > MPR
> > > > > > > > > > > > lines
> > > > > > > > > > > > >>> to be included in the column thresholds
you've
> > > defined.
> > > > > I
> > > > > > > > don't
> > > > > > > > > > see
> > > > > > > > > > > > >>> anything obviously wrong with what you've
> defined.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> The warning message indicates that the
filters
> you
> > > > > defined
> > > > > > > have
> > > > > > > > > > > > discarded
> > > > > > > > > > > > >>> all of the input MPR lines.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> We'd just need to go through your
filtering
> options
> > > > > > > one-by-one
> > > > > > > > to
> > > > > > > > > > > find
> > > > > > > > > > > > >>> out
> > > > > > > > > > > > >>> why.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> - Are you passing STAT-Analysis files
which
> contain
> > > MPR
> > > > > > > lines?
> > > > > > > > > > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > > > > > > > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > > > > > > > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > > > > > > > > > >>> - Does the OBS column have values > 1.0?
> > > > > > > > > > > > >>> - What are the ranges of the OBS_LAT and
OBS_LON
> > > > columns?
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> If you'd like to send me a sample .stat
file, I
> > could
> > > > > > > probably
> > > > > > > > > > track
> > > > > > > > > > > it
> > > > > > > > > > > > >>> down.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Thanks,
> > > > > > > > > > > > >>> John
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn
> MacCracken -
> > > > NOAA
> > > > > > > > > Affiliate
> > > > > > > > > > > via
> > > > > > > > > > > > >>> RT <
> > > > > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Mon Aug 06 07:43:46 2018: Request 86480
was
> acted
> > > > upon.
> > > > > > > > > > > > >>> > Transaction: Ticket created by
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > >>> > Queue: met_help
> > > > > > > > > > > > >>> > Subject: Errors using
-column_thresh
> > > > > > > > > > > > >>> > Owner: Nobody
> > > > > > > > > > > > >>> > Requestors:
rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > >>> > Status: new
> > > > > > > > > > > > >>> > Ticket <URL:
> > > > > > > > > > > >
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Hi,
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > I'm trying to use -column_thresh with
> > stat_analysis
> > > > to
> > > > > > mask
> > > > > > > > > out a
> > > > > > > > > > > > >>> lat/lon
> > > > > > > > > > > > >>> > box. I've added this command in my
> StatAnalysis
> > > > config
> > > > > > > file:
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > "-job aggregate_stat -out_line_type CTC
> > > > -column_thresh
> > > > > > OBS
> > > > > > > > > gt1.0
> > > > > > > > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > > > > > > > >>> > -column_thresh OBS_LAT
'ge15.0&&le70.0' \
> > > > > > > > > > > > >>> > -column_thresh OBS_LON 'ge-
100.0&&le-10.0'
> \
> > > > > > > > > > > > >>> > -by FCST_VALID_BEG,FCST_LEAD,VX_MASK
\
> > > > > > > > > > > > >>> > -out_stat
/opc_test/home/opc_test/data/m
> > > > > > > > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > > > > > > > >>> >
> > > > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > But, when it runs, I get these errors:
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Output:
> > > > > > > > > > > > >>> > DEBUG 2: Processing Job 2: -job
aggregate_stat
> > > > > -fcst_var
> > > > > > > WIND
> > > > > > > > > > > > >>> -fcst_lev Z10
> > > > > > > > > > > > >>> > -obtype ASCAT -line_type MPR
-column_thresh OBS
> > > >1.0
> > > > > > > > > > -column_thresh
> > > > > > > > > > > > >>> OBS_LAT
> > > > > > > > > > > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON
> > > > >=-100.0&&<=-10.0
> > > > > > -by
> > > > > > > > > > > > >>> FCST_VALID_BEG
> > > > > > > > > > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> > > > > > > > > /opc_test/home/opc_test/data/
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > new_out_stat_test2/agg_stat_
> > > > > > > MPR
> > > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > > > > > > >>> > DEBUG 1: Creating STAT output file
> > > > > > > > > "/opc_test/home/opc_test/data/
> > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > > > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > > > > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > > > > > > > > >>> > DEBUG 2: Computing output for 0 case(s).
> > > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > > >>> > WARNING: do_job_aggr_stat() -> no
matching STAT
> > > lines
> > > > > > found
> > > > > > > > for
> > > > > > > > > > > job:
> > > > > > > > > > > > >>> -job
> > > > > > > > > > > > >>> > aggregate_stat -fcst_var WIND -fcst_lev
Z10
> > -obtype
> > > > > ASCAT
> > > > > > > > > > > -line_type
> > > > > > > > > > > > >>> MPR
> > > > > > > > > > > > >>> > -column_thresh OBS >1.0 -column_thresh
OBS_LAT
> > > > > > > >=15.0&&<=70.0
> > > > > > > > > > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-10.0
-by
> > > > > > FCST_VALID_BEG
> > > > > > > > -by
> > > > > > > > > > > > >>> FCST_LEAD
> > > > > > > > > > > > >>> > -by VX_MASK -out_stat
> > /opc_test/home/opc_test/data/
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > new_out_stat_test2/agg_stat_
> > > > > > > MPR
> > > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > > >>> > -out_line_type CTC -out_thresh >=5.5689
> > > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137 STAT
lines.
> > > > > > > > > > > > >>> > DEBUG 2:
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Am I not using the command properly?
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Thanks!
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Roz
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > --
> > > > > > > > > > > > >>> > Rosalyn MacCracken
> > > > > > > > > > > > >>> > Support Scientist
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Ocean Applications Branch
> > > > > > > > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > >>> > NCWCP
> > > > > > > > > > > > >>> > 5830 University Research Ct
> > > > > > > > > > > > >>> > College Park, MD 20740-3818
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > (p) 301-683-1551
> > > > > > > > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> --
> > > > > > > > > > > > >> Rosalyn MacCracken
> > > > > > > > > > > > >> Support Scientist
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Ocean Applications Branch
> > > > > > > > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > >> NCWCP
> > > > > > > > > > > > >> 5830 University Research Ct
> > > > > > > > > > > > >> College Park, MD 20740-3818
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> (p) 301-683-1551
> > > > > > > > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > >
> > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > Support Scientist
> > > > > > > > > > > >
> > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > NCWCP
> > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > >
> > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > Support Scientist
> > > > > > > > > >
> > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > NCWCP
> > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > >
> > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rosalyn MacCracken
> > > > > > > > Support Scientist
> > > > > > > >
> > > > > > > > Ocean Applications Branch
> > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > NCWCP
> > > > > > > > 5830 University Research Ct
> > > > > > > > College Park, MD 20740-3818
> > > > > > > >
> > > > > > > > (p) 301-683-1551
> > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applications Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Errors using -column_thresh
From: John Halley Gotway
Time: Wed Aug 15 09:25:23 2018
Roz,
Here's how the filtering by quality control flag logic would work:
(1) The ascii data now has a meaningful quality control flag: 0 for
good
and 1 for bad.
(2) When you run it through ascii2nc, that quality control info will
be
passed along to the output file.
(3) Set up your Point-Stat configuration file to specify which quality
control flag you want to include. In your case, you only want to use
0's:
obs_quality = [ "0" ];
Note that this is specified as a string, since some obs use
integers
(PREPBUFR) and other use strings (MADIS).
In this way Point-Stat will ignore any observations that have a
quality
control flag that is not "0".
Whether you ignore those obs when you pre-process them into ascii
format or
ignore them when you run point-stat, the effect should be the same.
So
it's up to you to setup the logic how you'd like.
John
On Wed, Aug 15, 2018 at 8:04 AM Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
>
> Ok, I think that I could just set the bad observations in that
column to 1.
>
> So, if I do that, how will they be ignored in further processing?
Do I
> need to set something so those observations will be skipped?
>
> Roz
>
> On Tue, Aug 14, 2018 at 7:33 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Roz,
> >
> > Sure, you can write a constant value to the OBS_QC column. And
using 0
> > versus a 1 is just fine.
> >
> > John
> >
> > On Mon, Aug 13, 2018 at 1:32 PM Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > >
> > > No, that's correct. Originally, we weren't checking for the
flag, and
> > just
> > > taking the observation from the MEDRlite file, and writing it
directly
> > to a
> > > MET ASCII format file. So, as a result, I have MPR files that
have N/A
> > > where a QC flag number should be.
> > >
> > > Now, we are using bit position in place of the quality control
flag.
> > So, I
> > > take the binary MGDRlite data, read it into a python program,
evaluate
> if
> > > the bit position is a "1". If it's a "0", then it writes a line
to an
> > > ASCII file that has the MET format to be used with ASCII2NC. If
it's a
> > > "1", it skips writing that value to the MET formatted file.
This way,
> we
> > > are only keeping good observations and tossing the others.
That's why
> I
> > > asked about just using some constant value, because it's bit
position
> and
> > > not a QC flag. I was thinking it it's something other than N/A,
then
> it
> > > won't be used. And, it's only for the back files. I can
correct it
> > going
> > > forward, and just not write that observation to my ASCII file.
> > >
> > > Roz
> > >
> > >
> > >
> > > On Mon, Aug 13, 2018 at 7:20 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Roz,
> > > >
> > > > I'm confused. Here's is what I understood the situation to
be...
> > > >
> > > > (1) In March, you switched from pulling ASCAT data from
PREPBUFR to
> > > pulling
> > > > it from MGDRlite.
> > > > (2) When you made that switch, you forgot to set the system up
to
> > filter
> > > > the MGDRlite data by quality control flags.
> > > > (3) As a result of that, many "bad" observations were included
in
> your
> > > > evaluation which led to bad statistics.
> > > > (4) You now want to go back and correct the bad evaluation by
> > > reprocessing
> > > > using the actual quality control flags.
> > > >
> > > > I suggested that you modify the MET output MPR lines by
replacing the
> > NA
> > > > string with the *actual* quality control flag. This logic
only works
> > if
> > > > the MGDRlite data source includes quality control
information... and
> > you
> > > > know how to extract it.
> > > >
> > > > Just putting a constant flag value of 2 in the OBS_QC column
won't
> > really
> > > > fix anything. I'm suggesting that you get the actual flag
values and
> > > > "patch" the existing MPR lines by inserting them.
> > > >
> > > > Am I misunderstanding the situation?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Aug 13, 2018 at 12:47 PM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > RT
> > > > <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > >
> > > > > No, I don't know the correct OBS_QC value. Can I just make
> something
> > > up,
> > > > > like use "2", or some arbitrary number? Or does MET expect
some
> > value,
> > > > > given a certain file type?
> > > > >
> > > > > Actually, I currently don't write these values into my point
file,
> > > which
> > > > I
> > > > > use to make the netcdf file. So, it's just for the back
data.
> > > > >
> > > > > Roz
> > > > >
> > > > > On Mon, Aug 13, 2018 at 6:41 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Right, updating the OBS_QC values from NA to the real QC
value is
> > the
> > > > > > difficult part, but I think that's probably easier that
> "gridding"
> > > the
> > > > > > point data and going back through Point-Stat.
> > > > > >
> > > > > > So do you know how to get the QBS_QC value from the
MGDRlite
> data?
> > > > And,
> > > > > > going forward, can you set it up so that the OBS_QC column
is
> > > populated
> > > > > in
> > > > > > your Point-Stat runs going forward?
> > > > > >
> > > > > > Perhaps that's the place to start. Make sure your new
Point-Stat
> > > > output
> > > > > > includes meaningful OBS_QC output. Once that's working,
go
> through
> > > > your
> > > > > > *old* output and backfill the OBS_QC column with what it
should
> > > > contain.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Mon, Aug 13, 2018 at 12:37 PM Rosalyn MacCracken - NOAA
> > Affiliate
> > > > via
> > > > > RT
> > > > > > <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> >
> > > > > > >
> > > > > > > Yes, that makes sense. So, the MPR line would be "N/A"
that it
> > > would
> > > > > > > process the data with. Ok. And, I could do something
similar
> to
> > > > what
> > > > > I
> > > > > > > did before with copying my MPR files to a temp
directory, and
> use
> > > > > > -lookin.
> > > > > > >
> > > > > > > Ok, so, the hard part will be to write this script that
> replaces
> > my
> > > > bad
> > > > > > > values. I guess I like a challenge, right?
> > > > > > >
> > > > > > > Roz
> > > > > > >
> > > > > > > On Mon, Aug 13, 2018 at 6:30 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > > Roz,
> > > > > > > >
> > > > > > > > Once the OBS_QC values are set in the MPR lines, you
can use
> > > > > > > STAT-Analysis
> > > > > > > > to reprocess the data and recompute whatever other
STAT lines
> > > you'd
> > > > > > like
> > > > > > > to
> > > > > > > > compute:
> > > > > > > >
> > > > > > > > stat_analysis -job aggregate_stat -line_type MPR
> -out_line_type
> > > CNT
> > > > > > > > -column_str OBS_QC a,b,c
> > > > > > > >
> > > > > > > > This job would be run only on MPR lines that have
strings of
> > "a",
> > > > > "b",
> > > > > > or
> > > > > > > > "c" in the OBS_QC column. Make sense?
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Mon, Aug 13, 2018 at 11:55 AM Rosalyn MacCracken -
NOAA
> > > > Affiliate
> > > > > > via
> > > > > > > RT
> > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=86480
> > > >
> > > > > > > > >
> > > > > > > > > Hi John,
> > > > > > > > >
> > > > > > > > > Yeah, I can repost the file. I'll do that in a bit.
> > Actually,
> > > > if
> > > > > I
> > > > > > > can
> > > > > > > > > get this issue solved, you don't need to look at the
file,
> > > > because
> > > > > > I'll
> > > > > > > > > just reprocess that data with my new method. So,
I'll hold
> > off
> > > > on
> > > > > > > ftping
> > > > > > > > > that file to you for right now.
> > > > > > > > >
> > > > > > > > > So, the OBS_QC column has N/A values. That's easy
enough
> to
> > > > > replace.
> > > > > > > > Now,
> > > > > > > > > I have the original ASCAT data, which looks for the
quality
> > > flag
> > > > > as a
> > > > > > > bit
> > > > > > > > > (i.e., ibits[5]) and then doesn't use that. It
doesn't
> write
> > > it
> > > > to
> > > > > > the
> > > > > > > > > output file. Maybe what I should do is find the
first bad
> > > > > > observation
> > > > > > > > > (time, lat, lon, value), then, match it exactly to
the line
> > in
> > > > the
> > > > > > mpr
> > > > > > > > > file, and replace the observation.
> > > > > > > > >
> > > > > > > > > So, if my observations are in dataframes, and the
mpr file
> > has
> > > > been
> > > > > > > read
> > > > > > > > in
> > > > > > > > > as a dataframe, then, I could do a df.query('Lat =
?? and
> Lon
> > > > =??),
> > > > > > or
> > > > > > > > > however the syntax is...no wait, I need an if
statement to
> > > > compare
> > > > > > > > > first...I'll work on those detail. Then, once I get
the
> > OBS_QC
> > > > > value
> > > > > > > > > changed, then what? How do you get the other stat
files
> from
> > > > this
> > > > > > > point?
> > > > > > > > >
> > > > > > > > > Roz
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Aug 13, 2018 at 5:32 PM, John Halley Gotway
via RT
> <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > > Roz,
> > > > > > > > > >
> > > > > > > > > > That's correct, there is no tool in MET for
comparing one
> > set
> > > > of
> > > > > > > point
> > > > > > > > > > observations to another set of point observations.
> > > Presumably
> > > > > you
> > > > > > > > could
> > > > > > > > > > write point observations to a grid and then
reconfigure
> and
> > > > rerun
> > > > > > > > > > Point-Stat, but that'd be pretty difficult, time
> consuming,
> > > and
> > > > > > > fraught
> > > > > > > > > > with problems.
> > > > > > > > > >
> > > > > > > > > > You currently have many, many MPR lines and the
issue is
> > the
> > > > the
> > > > > > > OBS_QC
> > > > > > > > > > column is empty. If that OBS_QC column were *not*
empty
> > then
> > > > > it'd
> > > > > > be
> > > > > > > > > > pretty easy to process this data... correct? One
option
> > > would
> > > > be
> > > > > > > > > writing a
> > > > > > > > > > script to "patch" that column. For each MPR line,
do
> some
> > > > > > processing
> > > > > > > > to
> > > > > > > > > > figure out what that OBS_QC column should be and
then
> > update
> > > > it's
> > > > > > > > value.
> > > > > > > > > >
> > > > > > > > > > Then the question is this... is there a way of
getting
> > > quality
> > > > > > > control
> > > > > > > > > > values from the MGDRlite files?
> > > > > > > > > >
> > > > > > > > > > I'm sorry, I hadn't had a chance to look at that
data
> file
> > > yet.
> > > > > > But
> > > > > > > I
> > > > > > > > > just
> > > > > > > > > > checked on the ftp site and don't see a maccracken
> > > > sub-directory
> > > > > > > there:
> > > > > > > > > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > > > > > > > > >
> > > > > > > > > > Could you please repost?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn
MacCracken -
> NOAA
> > > > > > Affiliate
> > > > > > > > via
> > > > > > > > > RT
> > > > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > Ticket/Display.html?id=86480
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > So, you can't compare two point files, even if
both of
> > them
> > > > are
> > > > > > > > NetCDF
> > > > > > > > > > > format? Is there a way to write point
observations out
> > to
> > > a
> > > > > > grid?
> > > > > > > > In
> > > > > > > > > > > other words, take the point GFS observations,
and write
> > > them
> > > > > to a
> > > > > > > > > global
> > > > > > > > > > > grid (using some other tool, like wgrib or
something
> > else),
> > > > and
> > > > > > > fill
> > > > > > > > in
> > > > > > > > > > > missing values with NaN, or something? Then,
use that
> > > > gridded
> > > > > > file
> > > > > > > > to
> > > > > > > > > > > compare with the ASCAT NetCDF file?
> > > > > > > > > > >
> > > > > > > > > > > I don't have anything in the OBS_QC column
because
> when I
> > > > wrote
> > > > > > it
> > > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > ASCII file, I didn't have any flags from my
MGDRlite
> > files.
> > > > > This
> > > > > > > is
> > > > > > > > > part
> > > > > > > > > > > of the problem...
> > > > > > > > > > >
> > > > > > > > > > > Now, your answer for the STAT-Analysis, is that
what I
> > did
> > > > > wrong
> > > > > > in
> > > > > > > > my
> > > > > > > > > > use
> > > > > > > > > > > of stat-analysis from the previous email? Or,
did you
> > not
> > > > get
> > > > > a
> > > > > > > > chance
> > > > > > > > > > to
> > > > > > > > > > > look at my *stat file that I uploaded to the ftp
site?
> > > > > > > > > > >
> > > > > > > > > > > Roz
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley
Gotway via
> > RT
> > > <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Roz,
> > > > > > > > > > > >
> > > > > > > > > > > > Point-Stat takes as input a gridded NWP
forecast file
> > and
> > > > the
> > > > > > > > NetCDF
> > > > > > > > > > > output
> > > > > > > > > > > > of a point pre-processing tool (like ascii2nc
or
> > pb2nc).
> > > > > Based
> > > > > > > on
> > > > > > > > > your
> > > > > > > > > > > > description, it doesn't sound like you have
gridded
> NWP
> > > > > > forecast
> > > > > > > > > files
> > > > > > > > > > > > available. So no, I don't think that approach
would
> > > work.
> > > > > > > > > > > >
> > > > > > > > > > > > The MPR output line type does include a column
named
> > > > "OBS_QC"
> > > > > > > which
> > > > > > > > > is
> > > > > > > > > > a
> > > > > > > > > > > > string containing the observation quality
control
> > value.
> > > > I'd
> > > > > > > > suggest
> > > > > > > > > > > > checking the contents of that column in your
MPR
> output
> > > > lines
> > > > > > to
> > > > > > > > see
> > > > > > > > > if
> > > > > > > > > > > > those values could be filtered to do the
quality
> > control
> > > > > > > filtering
> > > > > > > > > > you're
> > > > > > > > > > > > after. Check to see if the values in that
column are
> > > > strings
> > > > > > or
> > > > > > > > > > numbers
> > > > > > > > > > > > (in PREPBUFR they're integers... in MADIS
they're
> > > strings).
> > > > > > > > > > > >
> > > > > > > > > > > > In STAT-Analysis, you can use the "-
column_thresh"
> > option
> > > > to
> > > > > > > > > threshold
> > > > > > > > > > a
> > > > > > > > > > > > numeric column or you can use the "-
column_str"
> option
> > to
> > > > > list
> > > > > > > > > strings
> > > > > > > > > > to
> > > > > > > > > > > > retained. For example:
> > > > > > > > > > > > -column_thresh OBS_QC 'ge0&&le3' # this
keeps all
> > > > values
> > > > > > > > between
> > > > > > > > > 0
> > > > > > > > > > > and
> > > > > > > > > > > > 3
> > > > > > > > > > > > -column_str OBS_QC aa,ab,ac # this
keeps
> > > strings
> > > > > > "aa",
> > > > > > > > > "ab",
> > > > > > > > > > > or
> > > > > > > > > > > > "ac"
> > > > > > > > > > > >
> > > > > > > > > > > > Hope that helps.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > John
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn
MacCracken -
> > NOAA
> > > > > > > Affiliate
> > > > > > > > > via
> > > > > > > > > > > RT
> > > > > > > > > > > > <
> > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > > Ticket/Display.html?id=86480
> > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi John,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have another question related to my
reprocessing
> of
> > > my
> > > > > > data.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ok, so, I realized that when I switched
ASCAT data
> > > > sources
> > > > > > back
> > > > > > > > in
> > > > > > > > > > > March,
> > > > > > > > > > > > > from prepbufr to MGDRlite files, I should
have been
> > > using
> > > > > > > quality
> > > > > > > > > > flags
> > > > > > > > > > > > to
> > > > > > > > > > > > > filter out my data. Because I wasn't
checking for
> > the
> > > > > > quality
> > > > > > > > > flags,
> > > > > > > > > > > my
> > > > > > > > > > > > > statistics were not so good, since it was
letting
> in
> > > bad
> > > > > > data,
> > > > > > > > and
> > > > > > > > > > > stats
> > > > > > > > > > > > > were being calculated from this bad data.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So, I need to find a way to correct this,
but, the
> > old
> > > > > story
> > > > > > > is I
> > > > > > > > > > don't
> > > > > > > > > > > > > have the original GFS data, and it's too
massive to
> > > grab
> > > > > from
> > > > > > > > HPSS.
> > > > > > > > > > I
> > > > > > > > > > > do
> > > > > > > > > > > > > have the original ASCAT files, which I can
run
> > through
> > > my
> > > > > > > program
> > > > > > > > > to
> > > > > > > > > > > > filter
> > > > > > > > > > > > > out the bad observations, as well as matched
points
> > > from
> > > > > the
> > > > > > > *mpr
> > > > > > > > > and
> > > > > > > > > > > > *stat
> > > > > > > > > > > > > files.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Originally, I was just going to filter out
using
> that
> > > > > > > > > -column_thresh
> > > > > > > > > > > > > command, and filter out over land. But, I
had this
> > > idea
> > > > > > that I
> > > > > > > > > > wanted
> > > > > > > > > > > to
> > > > > > > > > > > > > run by you, to see if this makes sense to
do.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have the time, lat, lon and GFS
observation and
> the
> > > > > > > > corresponding
> > > > > > > > > > > ASCAT
> > > > > > > > > > > > > observation point. Can I do this:
> > > > > > > > > > > > > 1) read the time, lat, lon and GFS
observation out
> > of
> > > > the
> > > > > > mpr
> > > > > > > > > file,
> > > > > > > > > > > and
> > > > > > > > > > > > > make a new input file with just those
variables in
> > the
> > > > > ASCII
> > > > > > > > format
> > > > > > > > > > > that
> > > > > > > > > > > > > MET can use
> > > > > > > > > > > > > 2) reprocess my ASCAT data and write it
into an
> > ASCII
> > > > file
> > > > > > in
> > > > > > > > MET
> > > > > > > > > > > format
> > > > > > > > > > > > > 3) run ASCII2NC on these files, to create
the two
> > > input
> > > > > > files
> > > > > > > > for
> > > > > > > > > > MET,
> > > > > > > > > > > > > then, run point_stat on the netcdf files to
> > regenerate
> > > > the
> > > > > > new
> > > > > > > > > > > corrected
> > > > > > > > > > > > > stat files
> > > > > > > > > > > > >
> > > > > > > > > > > > > Does that make sense to do? I know it could
be
> alot
> > of
> > > > > work,
> > > > > > > > but,
> > > > > > > > > > if I
> > > > > > > > > > > > can
> > > > > > > > > > > > > get corrected statistics, it might be worth
it.
> What
> > > do
> > > > > you
> > > > > > > > think?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Roz
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn
> MacCracken -
> > > > NOAA
> > > > > > > > > Affiliate
> > > > > > > > > > <
> > > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi John,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I double checked all those things you
listed, and
> > > they
> > > > > are
> > > > > > > all
> > > > > > > > > > > correct.
> > > > > > > > > > > > > > So, I've put a file in the /maccracken
directory
> on
> > > > your
> > > > > > ftp
> > > > > > > > > site.
> > > > > > > > > > > Let
> > > > > > > > > > > > > me
> > > > > > > > > > > > > > know what you see.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your help!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Roz
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn
> > MacCracken -
> > > > > NOAA
> > > > > > > > > > Affiliate
> > > > > > > > > > > <
> > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Hi John,
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> I'm sorry I didn't get back to you
sooner, but,
> I
> > > was
> > > > > tied
> > > > > > > up
> > > > > > > > > with
> > > > > > > > > > > > some
> > > > > > > > > > > > > >> other things. Let me double check with
all of
> > those
> > > > > items
> > > > > > > in
> > > > > > > > my
> > > > > > > > > > > file
> > > > > > > > > > > > > and
> > > > > > > > > > > > > >> make sure that I don't have some sort of
obvious
> > > > error.
> > > > > > > I'll
> > > > > > > > be
> > > > > > > > > > > back
> > > > > > > > > > > > in
> > > > > > > > > > > > > >> touch after I check on this.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Roz
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John
Halley
> Gotway
> > > via
> > > > > RT
> > > > > > <
> > > > > > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>> Hi Roz,
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> I see you're having trouble getting with
> > > > STAT-Analysis,
> > > > > > > > getting
> > > > > > > > > > MPR
> > > > > > > > > > > > > lines
> > > > > > > > > > > > > >>> to be included in the column thresholds
you've
> > > > defined.
> > > > > > I
> > > > > > > > > don't
> > > > > > > > > > > see
> > > > > > > > > > > > > >>> anything obviously wrong with what
you've
> > defined.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> The warning message indicates that the
filters
> > you
> > > > > > defined
> > > > > > > > have
> > > > > > > > > > > > > discarded
> > > > > > > > > > > > > >>> all of the input MPR lines.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> We'd just need to go through your
filtering
> > options
> > > > > > > > one-by-one
> > > > > > > > > to
> > > > > > > > > > > > find
> > > > > > > > > > > > > >>> out
> > > > > > > > > > > > > >>> why.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> - Are you passing STAT-Analysis files
which
> > contain
> > > > MPR
> > > > > > > > lines?
> > > > > > > > > > > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > > > > > > > > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > > > > > > > > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > > > > > > > > > > >>> - Does the OBS column have values > 1.0?
> > > > > > > > > > > > > >>> - What are the ranges of the OBS_LAT and
> OBS_LON
> > > > > columns?
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> If you'd like to send me a sample .stat
file, I
> > > could
> > > > > > > > probably
> > > > > > > > > > > track
> > > > > > > > > > > > it
> > > > > > > > > > > > > >>> down.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> Thanks,
> > > > > > > > > > > > > >>> John
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn
> > MacCracken -
> > > > > NOAA
> > > > > > > > > > Affiliate
> > > > > > > > > > > > via
> > > > > > > > > > > > > >>> RT <
> > > > > > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > Mon Aug 06 07:43:46 2018: Request
86480 was
> > acted
> > > > > upon.
> > > > > > > > > > > > > >>> > Transaction: Ticket created by
> > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > >>> > Queue: met_help
> > > > > > > > > > > > > >>> > Subject: Errors using
-column_thresh
> > > > > > > > > > > > > >>> > Owner: Nobody
> > > > > > > > > > > > > >>> > Requestors:
rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > >>> > Status: new
> > > > > > > > > > > > > >>> > Ticket <URL:
> > > > > > > > > > > > >
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > Hi,
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > I'm trying to use -column_thresh with
> > > stat_analysis
> > > > > to
> > > > > > > mask
> > > > > > > > > > out a
> > > > > > > > > > > > > >>> lat/lon
> > > > > > > > > > > > > >>> > box. I've added this command in my
> > StatAnalysis
> > > > > config
> > > > > > > > file:
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > "-job aggregate_stat -out_line_type
CTC
> > > > > -column_thresh
> > > > > > > OBS
> > > > > > > > > > gt1.0
> > > > > > > > > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > > > > > > > > >>> > -column_thresh OBS_LAT
'ge15.0&&le70.0' \
> > > > > > > > > > > > > >>> > -column_thresh OBS_LON
> 'ge-100.0&&le-10.0'
> > \
> > > > > > > > > > > > > >>> > -by
FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > > > > > > > > > > > >>> > -out_stat
/opc_test/home/opc_test/data/m
> > > > > > > > > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > > > > > > > > >>> >
> > > > > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > But, when it runs, I get these errors:
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > Output:
> > > > > > > > > > > > > >>> > DEBUG 2: Processing Job 2: -job
> aggregate_stat
> > > > > > -fcst_var
> > > > > > > > WIND
> > > > > > > > > > > > > >>> -fcst_lev Z10
> > > > > > > > > > > > > >>> > -obtype ASCAT -line_type MPR
-column_thresh
> OBS
> > > > >1.0
> > > > > > > > > > > -column_thresh
> > > > > > > > > > > > > >>> OBS_LAT
> > > > > > > > > > > > > >>> > >=15.0&&<=70.0 -column_thresh OBS_LON
> > > > > >=-100.0&&<=-10.0
> > > > > > > -by
> > > > > > > > > > > > > >>> FCST_VALID_BEG
> > > > > > > > > > > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> > > > > > > > > > /opc_test/home/opc_test/data/
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > MPR
> > > > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > > > >>> > -out_line_type CTC -out_thresh
>=5.5689
> > > > > > > > > > > > > >>> > DEBUG 1: Creating STAT output file
> > > > > > > > > > "/opc_test/home/opc_test/data/
> > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > > > > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > > > > > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > > > > > > > > > >>> > DEBUG 2: Computing output for 0
case(s).
> > > > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > > > >>> > WARNING: do_job_aggr_stat() -> no
matching
> STAT
> > > > lines
> > > > > > > found
> > > > > > > > > for
> > > > > > > > > > > > job:
> > > > > > > > > > > > > >>> -job
> > > > > > > > > > > > > >>> > aggregate_stat -fcst_var WIND
-fcst_lev Z10
> > > -obtype
> > > > > > ASCAT
> > > > > > > > > > > > -line_type
> > > > > > > > > > > > > >>> MPR
> > > > > > > > > > > > > >>> > -column_thresh OBS >1.0 -column_thresh
> OBS_LAT
> > > > > > > > >=15.0&&<=70.0
> > > > > > > > > > > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-
10.0 -by
> > > > > > > FCST_VALID_BEG
> > > > > > > > > -by
> > > > > > > > > > > > > >>> FCST_LEAD
> > > > > > > > > > > > > >>> > -by VX_MASK -out_stat
> > > /opc_test/home/opc_test/data/
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > MPR
> > > > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > > > >>> > -out_line_type CTC -out_thresh
>=5.5689
> > > > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137
STAT
> lines.
> > > > > > > > > > > > > >>> > DEBUG 2:
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > Am I not using the command properly?
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > Thanks!
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > Roz
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > --
> > > > > > > > > > > > > >>> > Rosalyn MacCracken
> > > > > > > > > > > > > >>> > Support Scientist
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > Ocean Applications Branch
> > > > > > > > > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > >>> > NCWCP
> > > > > > > > > > > > > >>> > 5830 University Research Ct
> > > > > > > > > > > > > >>> > College Park, MD 20740-3818
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > (p) 301-683-1551
> > > > > > > > > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> --
> > > > > > > > > > > > > >> Rosalyn MacCracken
> > > > > > > > > > > > > >> Support Scientist
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Ocean Applications Branch
> > > > > > > > > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > >> NCWCP
> > > > > > > > > > > > > >> 5830 University Research Ct
> > > > > > > > > > > > > >> College Park, MD 20740-3818
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> (p) 301-683-1551
> > > > > > > > > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > >
> > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > Support Scientist
> > > > > > > > > > >
> > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > NCWCP
> > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > >
> > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rosalyn MacCracken
> > > > > > > > > Support Scientist
> > > > > > > > >
> > > > > > > > > Ocean Applications Branch
> > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > NCWCP
> > > > > > > > > 5830 University Research Ct
> > > > > > > > > College Park, MD 20740-3818
> > > > > > > > >
> > > > > > > > > (p) 301-683-1551
> > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rosalyn MacCracken
> > > > > > > Support Scientist
> > > > > > >
> > > > > > > Ocean Applications Branch
> > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > NCWCP
> > > > > > > 5830 University Research Ct
> > > > > > > College Park, MD 20740-3818
> > > > > > >
> > > > > > > (p) 301-683-1551
> > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applications Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Wed Aug 15 09:56:15 2018
Ok, so, that would be for going forward. But you were saying I could
add
it to the MPR files, and use it to filter in Stat-Analysis? That's
what
you were saying about -column_str. If the value of "1" is there,
stat-analysis will see it as a bad ob and ignore it? Is that right?
Roz
On Wed, Aug 15, 2018 at 3:25 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Roz,
>
> Here's how the filtering by quality control flag logic would work:
> (1) The ascii data now has a meaningful quality control flag: 0 for
good
> and 1 for bad.
> (2) When you run it through ascii2nc, that quality control info will
be
> passed along to the output file.
> (3) Set up your Point-Stat configuration file to specify which
quality
> control flag you want to include. In your case, you only want to
use 0's:
> obs_quality = [ "0" ];
> Note that this is specified as a string, since some obs use
integers
> (PREPBUFR) and other use strings (MADIS).
>
> In this way Point-Stat will ignore any observations that have a
quality
> control flag that is not "0".
>
> Whether you ignore those obs when you pre-process them into ascii
format or
> ignore them when you run point-stat, the effect should be the same.
So
> it's up to you to setup the logic how you'd like.
>
> John
>
>
> On Wed, Aug 15, 2018 at 8:04 AM Rosalyn MacCracken - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> >
> > Ok, I think that I could just set the bad observations in that
column to
> 1.
> >
> > So, if I do that, how will they be ignored in further processing?
Do I
> > need to set something so those observations will be skipped?
> >
> > Roz
> >
> > On Tue, Aug 14, 2018 at 7:33 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Roz,
> > >
> > > Sure, you can write a constant value to the OBS_QC column. And
using 0
> > > versus a 1 is just fine.
> > >
> > > John
> > >
> > > On Mon, Aug 13, 2018 at 1:32 PM Rosalyn MacCracken - NOAA
Affiliate via
> > RT
> > > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
>
> > > >
> > > > No, that's correct. Originally, we weren't checking for the
flag,
> and
> > > just
> > > > taking the observation from the MEDRlite file, and writing it
> directly
> > > to a
> > > > MET ASCII format file. So, as a result, I have MPR files that
have
> N/A
> > > > where a QC flag number should be.
> > > >
> > > > Now, we are using bit position in place of the quality control
flag.
> > > So, I
> > > > take the binary MGDRlite data, read it into a python program,
> evaluate
> > if
> > > > the bit position is a "1". If it's a "0", then it writes a
line to
> an
> > > > ASCII file that has the MET format to be used with ASCII2NC.
If
> it's a
> > > > "1", it skips writing that value to the MET formatted file.
This
> way,
> > we
> > > > are only keeping good observations and tossing the others.
That's
> why
> > I
> > > > asked about just using some constant value, because it's bit
position
> > and
> > > > not a QC flag. I was thinking it it's something other than
N/A, then
> > it
> > > > won't be used. And, it's only for the back files. I can
correct it
> > > going
> > > > forward, and just not write that observation to my ASCII file.
> > > >
> > > > Roz
> > > >
> > > >
> > > >
> > > > On Mon, Aug 13, 2018 at 7:20 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Roz,
> > > > >
> > > > > I'm confused. Here's is what I understood the situation to
be...
> > > > >
> > > > > (1) In March, you switched from pulling ASCAT data from
PREPBUFR to
> > > > pulling
> > > > > it from MGDRlite.
> > > > > (2) When you made that switch, you forgot to set the system
up to
> > > filter
> > > > > the MGDRlite data by quality control flags.
> > > > > (3) As a result of that, many "bad" observations were
included in
> > your
> > > > > evaluation which led to bad statistics.
> > > > > (4) You now want to go back and correct the bad evaluation
by
> > > > reprocessing
> > > > > using the actual quality control flags.
> > > > >
> > > > > I suggested that you modify the MET output MPR lines by
replacing
> the
> > > NA
> > > > > string with the *actual* quality control flag. This logic
only
> works
> > > if
> > > > > the MGDRlite data source includes quality control
information...
> and
> > > you
> > > > > know how to extract it.
> > > > >
> > > > > Just putting a constant flag value of 2 in the OBS_QC column
won't
> > > really
> > > > > fix anything. I'm suggesting that you get the actual flag
values
> and
> > > > > "patch" the existing MPR lines by inserting them.
> > > > >
> > > > > Am I misunderstanding the situation?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, Aug 13, 2018 at 12:47 PM Rosalyn MacCracken - NOAA
> Affiliate
> > > via
> > > > RT
> > > > > <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > > >
> > > > > > No, I don't know the correct OBS_QC value. Can I just
make
> > something
> > > > up,
> > > > > > like use "2", or some arbitrary number? Or does MET
expect some
> > > value,
> > > > > > given a certain file type?
> > > > > >
> > > > > > Actually, I currently don't write these values into my
point
> file,
> > > > which
> > > > > I
> > > > > > use to make the netcdf file. So, it's just for the back
data.
> > > > > >
> > > > > > Roz
> > > > > >
> > > > > > On Mon, Aug 13, 2018 at 6:41 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Right, updating the OBS_QC values from NA to the real QC
value
> is
> > > the
> > > > > > > difficult part, but I think that's probably easier that
> > "gridding"
> > > > the
> > > > > > > point data and going back through Point-Stat.
> > > > > > >
> > > > > > > So do you know how to get the QBS_QC value from the
MGDRlite
> > data?
> > > > > And,
> > > > > > > going forward, can you set it up so that the OBS_QC
column is
> > > > populated
> > > > > > in
> > > > > > > your Point-Stat runs going forward?
> > > > > > >
> > > > > > > Perhaps that's the place to start. Make sure your new
> Point-Stat
> > > > > output
> > > > > > > includes meaningful OBS_QC output. Once that's working,
go
> > through
> > > > > your
> > > > > > > *old* output and backfill the OBS_QC column with what it
should
> > > > > contain.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Aug 13, 2018 at 12:37 PM Rosalyn MacCracken -
NOAA
> > > Affiliate
> > > > > via
> > > > > > RT
> > > > > > > <met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=86480
> > >
> > > > > > > >
> > > > > > > > Yes, that makes sense. So, the MPR line would be
"N/A" that
> it
> > > > would
> > > > > > > > process the data with. Ok. And, I could do something
> similar
> > to
> > > > > what
> > > > > > I
> > > > > > > > did before with copying my MPR files to a temp
directory, and
> > use
> > > > > > > -lookin.
> > > > > > > >
> > > > > > > > Ok, so, the hard part will be to write this script
that
> > replaces
> > > my
> > > > > bad
> > > > > > > > values. I guess I like a challenge, right?
> > > > > > > >
> > > > > > > > Roz
> > > > > > > >
> > > > > > > > On Mon, Aug 13, 2018 at 6:30 PM, John Halley Gotway
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > > Roz,
> > > > > > > > >
> > > > > > > > > Once the OBS_QC values are set in the MPR lines, you
can
> use
> > > > > > > > STAT-Analysis
> > > > > > > > > to reprocess the data and recompute whatever other
STAT
> lines
> > > > you'd
> > > > > > > like
> > > > > > > > to
> > > > > > > > > compute:
> > > > > > > > >
> > > > > > > > > stat_analysis -job aggregate_stat -line_type MPR
> > -out_line_type
> > > > CNT
> > > > > > > > > -column_str OBS_QC a,b,c
> > > > > > > > >
> > > > > > > > > This job would be run only on MPR lines that have
strings
> of
> > > "a",
> > > > > > "b",
> > > > > > > or
> > > > > > > > > "c" in the OBS_QC column. Make sense?
> > > > > > > > >
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Mon, Aug 13, 2018 at 11:55 AM Rosalyn MacCracken
- NOAA
> > > > > Affiliate
> > > > > > > via
> > > > > > > > RT
> > > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=86480
> > > > >
> > > > > > > > > >
> > > > > > > > > > Hi John,
> > > > > > > > > >
> > > > > > > > > > Yeah, I can repost the file. I'll do that in a
bit.
> > > Actually,
> > > > > if
> > > > > > I
> > > > > > > > can
> > > > > > > > > > get this issue solved, you don't need to look at
the
> file,
> > > > > because
> > > > > > > I'll
> > > > > > > > > > just reprocess that data with my new method. So,
I'll
> hold
> > > off
> > > > > on
> > > > > > > > ftping
> > > > > > > > > > that file to you for right now.
> > > > > > > > > >
> > > > > > > > > > So, the OBS_QC column has N/A values. That's easy
enough
> > to
> > > > > > replace.
> > > > > > > > > Now,
> > > > > > > > > > I have the original ASCAT data, which looks for
the
> quality
> > > > flag
> > > > > > as a
> > > > > > > > bit
> > > > > > > > > > (i.e., ibits[5]) and then doesn't use that. It
doesn't
> > write
> > > > it
> > > > > to
> > > > > > > the
> > > > > > > > > > output file. Maybe what I should do is find the
first
> bad
> > > > > > > observation
> > > > > > > > > > (time, lat, lon, value), then, match it exactly to
the
> line
> > > in
> > > > > the
> > > > > > > mpr
> > > > > > > > > > file, and replace the observation.
> > > > > > > > > >
> > > > > > > > > > So, if my observations are in dataframes, and the
mpr
> file
> > > has
> > > > > been
> > > > > > > > read
> > > > > > > > > in
> > > > > > > > > > as a dataframe, then, I could do a df.query('Lat =
?? and
> > Lon
> > > > > =??),
> > > > > > > or
> > > > > > > > > > however the syntax is...no wait, I need an if
statement
> to
> > > > > compare
> > > > > > > > > > first...I'll work on those detail. Then, once I
get the
> > > OBS_QC
> > > > > > value
> > > > > > > > > > changed, then what? How do you get the other stat
files
> > from
> > > > > this
> > > > > > > > point?
> > > > > > > > > >
> > > > > > > > > > Roz
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 13, 2018 at 5:32 PM, John Halley
Gotway via
> RT
> > <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > > Roz,
> > > > > > > > > > >
> > > > > > > > > > > That's correct, there is no tool in MET for
comparing
> one
> > > set
> > > > > of
> > > > > > > > point
> > > > > > > > > > > observations to another set of point
observations.
> > > > Presumably
> > > > > > you
> > > > > > > > > could
> > > > > > > > > > > write point observations to a grid and then
reconfigure
> > and
> > > > > rerun
> > > > > > > > > > > Point-Stat, but that'd be pretty difficult, time
> > consuming,
> > > > and
> > > > > > > > fraught
> > > > > > > > > > > with problems.
> > > > > > > > > > >
> > > > > > > > > > > You currently have many, many MPR lines and the
issue
> is
> > > the
> > > > > the
> > > > > > > > OBS_QC
> > > > > > > > > > > column is empty. If that OBS_QC column were
*not*
> empty
> > > then
> > > > > > it'd
> > > > > > > be
> > > > > > > > > > > pretty easy to process this data... correct?
One
> option
> > > > would
> > > > > be
> > > > > > > > > > writing a
> > > > > > > > > > > script to "patch" that column. For each MPR
line, do
> > some
> > > > > > > processing
> > > > > > > > > to
> > > > > > > > > > > figure out what that OBS_QC column should be and
then
> > > update
> > > > > it's
> > > > > > > > > value.
> > > > > > > > > > >
> > > > > > > > > > > Then the question is this... is there a way of
getting
> > > > quality
> > > > > > > > control
> > > > > > > > > > > values from the MGDRlite files?
> > > > > > > > > > >
> > > > > > > > > > > I'm sorry, I hadn't had a chance to look at that
data
> > file
> > > > yet.
> > > > > > > But
> > > > > > > > I
> > > > > > > > > > just
> > > > > > > > > > > checked on the ftp site and don't see a
maccracken
> > > > > sub-directory
> > > > > > > > there:
> > > > > > > > > > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > > > > > > > > > >
> > > > > > > > > > > Could you please repost?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn
MacCracken -
> > NOAA
> > > > > > > Affiliate
> > > > > > > > > via
> > > > > > > > > > RT
> > > > > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > Ticket/Display.html?id=86480
> > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > So, you can't compare two point files, even if
both
> of
> > > them
> > > > > are
> > > > > > > > > NetCDF
> > > > > > > > > > > > format? Is there a way to write point
observations
> out
> > > to
> > > > a
> > > > > > > grid?
> > > > > > > > > In
> > > > > > > > > > > > other words, take the point GFS observations,
and
> write
> > > > them
> > > > > > to a
> > > > > > > > > > global
> > > > > > > > > > > > grid (using some other tool, like wgrib or
something
> > > else),
> > > > > and
> > > > > > > > fill
> > > > > > > > > in
> > > > > > > > > > > > missing values with NaN, or something? Then,
use
> that
> > > > > gridded
> > > > > > > file
> > > > > > > > > to
> > > > > > > > > > > > compare with the ASCAT NetCDF file?
> > > > > > > > > > > >
> > > > > > > > > > > > I don't have anything in the OBS_QC column
because
> > when I
> > > > > wrote
> > > > > > > it
> > > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > > ASCII file, I didn't have any flags from my
MGDRlite
> > > files.
> > > > > > This
> > > > > > > > is
> > > > > > > > > > part
> > > > > > > > > > > > of the problem...
> > > > > > > > > > > >
> > > > > > > > > > > > Now, your answer for the STAT-Analysis, is
that what
> I
> > > did
> > > > > > wrong
> > > > > > > in
> > > > > > > > > my
> > > > > > > > > > > use
> > > > > > > > > > > > of stat-analysis from the previous email? Or,
did
> you
> > > not
> > > > > get
> > > > > > a
> > > > > > > > > chance
> > > > > > > > > > > to
> > > > > > > > > > > > look at my *stat file that I uploaded to the
ftp
> site?
> > > > > > > > > > > >
> > > > > > > > > > > > Roz
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley
Gotway
> via
> > > RT
> > > > <
> > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Roz,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Point-Stat takes as input a gridded NWP
forecast
> file
> > > and
> > > > > the
> > > > > > > > > NetCDF
> > > > > > > > > > > > output
> > > > > > > > > > > > > of a point pre-processing tool (like
ascii2nc or
> > > pb2nc).
> > > > > > Based
> > > > > > > > on
> > > > > > > > > > your
> > > > > > > > > > > > > description, it doesn't sound like you have
gridded
> > NWP
> > > > > > > forecast
> > > > > > > > > > files
> > > > > > > > > > > > > available. So no, I don't think that
approach
> would
> > > > work.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The MPR output line type does include a
column
> named
> > > > > "OBS_QC"
> > > > > > > > which
> > > > > > > > > > is
> > > > > > > > > > > a
> > > > > > > > > > > > > string containing the observation quality
control
> > > value.
> > > > > I'd
> > > > > > > > > suggest
> > > > > > > > > > > > > checking the contents of that column in your
MPR
> > output
> > > > > lines
> > > > > > > to
> > > > > > > > > see
> > > > > > > > > > if
> > > > > > > > > > > > > those values could be filtered to do the
quality
> > > control
> > > > > > > > filtering
> > > > > > > > > > > you're
> > > > > > > > > > > > > after. Check to see if the values in that
column
> are
> > > > > strings
> > > > > > > or
> > > > > > > > > > > numbers
> > > > > > > > > > > > > (in PREPBUFR they're integers... in MADIS
they're
> > > > strings).
> > > > > > > > > > > > >
> > > > > > > > > > > > > In STAT-Analysis, you can use the "-
column_thresh"
> > > option
> > > > > to
> > > > > > > > > > threshold
> > > > > > > > > > > a
> > > > > > > > > > > > > numeric column or you can use the "-
column_str"
> > option
> > > to
> > > > > > list
> > > > > > > > > > strings
> > > > > > > > > > > to
> > > > > > > > > > > > > retained. For example:
> > > > > > > > > > > > > -column_thresh OBS_QC 'ge0&&le3' # this
keeps
> all
> > > > > values
> > > > > > > > > between
> > > > > > > > > > 0
> > > > > > > > > > > > and
> > > > > > > > > > > > > 3
> > > > > > > > > > > > > -column_str OBS_QC aa,ab,ac #
this keeps
> > > > strings
> > > > > > > "aa",
> > > > > > > > > > "ab",
> > > > > > > > > > > > or
> > > > > > > > > > > > > "ac"
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hope that helps.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > John
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn
MacCracken
> -
> > > NOAA
> > > > > > > > Affiliate
> > > > > > > > > > via
> > > > > > > > > > > > RT
> > > > > > > > > > > > > <
> > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > Ticket/Display.html?id=86480
> > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi John,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have another question related to my
> reprocessing
> > of
> > > > my
> > > > > > > data.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Ok, so, I realized that when I switched
ASCAT
> data
> > > > > sources
> > > > > > > back
> > > > > > > > > in
> > > > > > > > > > > > March,
> > > > > > > > > > > > > > from prepbufr to MGDRlite files, I should
have
> been
> > > > using
> > > > > > > > quality
> > > > > > > > > > > flags
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > filter out my data. Because I wasn't
checking
> for
> > > the
> > > > > > > quality
> > > > > > > > > > flags,
> > > > > > > > > > > > my
> > > > > > > > > > > > > > statistics were not so good, since it was
letting
> > in
> > > > bad
> > > > > > > data,
> > > > > > > > > and
> > > > > > > > > > > > stats
> > > > > > > > > > > > > > were being calculated from this bad data.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So, I need to find a way to correct this,
but,
> the
> > > old
> > > > > > story
> > > > > > > > is I
> > > > > > > > > > > don't
> > > > > > > > > > > > > > have the original GFS data, and it's too
massive
> to
> > > > grab
> > > > > > from
> > > > > > > > > HPSS.
> > > > > > > > > > > I
> > > > > > > > > > > > do
> > > > > > > > > > > > > > have the original ASCAT files, which I can
run
> > > through
> > > > my
> > > > > > > > program
> > > > > > > > > > to
> > > > > > > > > > > > > filter
> > > > > > > > > > > > > > out the bad observations, as well as
matched
> points
> > > > from
> > > > > > the
> > > > > > > > *mpr
> > > > > > > > > > and
> > > > > > > > > > > > > *stat
> > > > > > > > > > > > > > files.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Originally, I was just going to filter out
using
> > that
> > > > > > > > > > -column_thresh
> > > > > > > > > > > > > > command, and filter out over land. But, I
had
> this
> > > > idea
> > > > > > > that I
> > > > > > > > > > > wanted
> > > > > > > > > > > > to
> > > > > > > > > > > > > > run by you, to see if this makes sense to
do.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have the time, lat, lon and GFS
observation and
> > the
> > > > > > > > > corresponding
> > > > > > > > > > > > ASCAT
> > > > > > > > > > > > > > observation point. Can I do this:
> > > > > > > > > > > > > > 1) read the time, lat, lon and GFS
observation
> out
> > > of
> > > > > the
> > > > > > > mpr
> > > > > > > > > > file,
> > > > > > > > > > > > and
> > > > > > > > > > > > > > make a new input file with just those
variables
> in
> > > the
> > > > > > ASCII
> > > > > > > > > format
> > > > > > > > > > > > that
> > > > > > > > > > > > > > MET can use
> > > > > > > > > > > > > > 2) reprocess my ASCAT data and write it
into an
> > > ASCII
> > > > > file
> > > > > > > in
> > > > > > > > > MET
> > > > > > > > > > > > format
> > > > > > > > > > > > > > 3) run ASCII2NC on these files, to create
the
> two
> > > > input
> > > > > > > files
> > > > > > > > > for
> > > > > > > > > > > MET,
> > > > > > > > > > > > > > then, run point_stat on the netcdf files
to
> > > regenerate
> > > > > the
> > > > > > > new
> > > > > > > > > > > > corrected
> > > > > > > > > > > > > > stat files
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Does that make sense to do? I know it
could be
> > alot
> > > of
> > > > > > work,
> > > > > > > > > but,
> > > > > > > > > > > if I
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > get corrected statistics, it might be
worth it.
> > What
> > > > do
> > > > > > you
> > > > > > > > > think?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Roz
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn
> > MacCracken -
> > > > > NOAA
> > > > > > > > > > Affiliate
> > > > > > > > > > > <
> > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi John,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I double checked all those things you
listed,
> and
> > > > they
> > > > > > are
> > > > > > > > all
> > > > > > > > > > > > correct.
> > > > > > > > > > > > > > > So, I've put a file in the /maccracken
> directory
> > on
> > > > > your
> > > > > > > ftp
> > > > > > > > > > site.
> > > > > > > > > > > > Let
> > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > know what you see.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your help!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Roz
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 10, 2018 at 1:41 PM, Rosalyn
> > > MacCracken -
> > > > > > NOAA
> > > > > > > > > > > Affiliate
> > > > > > > > > > > > <
> > > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Hi John,
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> I'm sorry I didn't get back to you
sooner,
> but,
> > I
> > > > was
> > > > > > tied
> > > > > > > > up
> > > > > > > > > > with
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > >> other things. Let me double check with
all of
> > > those
> > > > > > items
> > > > > > > > in
> > > > > > > > > my
> > > > > > > > > > > > file
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > >> make sure that I don't have some sort
of
> obvious
> > > > > error.
> > > > > > > > I'll
> > > > > > > > > be
> > > > > > > > > > > > back
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > >> touch after I check on this.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Roz
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John
Halley
> > Gotway
> > > > via
> > > > > > RT
> > > > > > > <
> > > > > > > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>> Hi Roz,
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> I see you're having trouble getting
with
> > > > > STAT-Analysis,
> > > > > > > > > getting
> > > > > > > > > > > MPR
> > > > > > > > > > > > > > lines
> > > > > > > > > > > > > > >>> to be included in the column
thresholds
> you've
> > > > > defined.
> > > > > > > I
> > > > > > > > > > don't
> > > > > > > > > > > > see
> > > > > > > > > > > > > > >>> anything obviously wrong with what
you've
> > > defined.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> The warning message indicates that the
> filters
> > > you
> > > > > > > defined
> > > > > > > > > have
> > > > > > > > > > > > > > discarded
> > > > > > > > > > > > > > >>> all of the input MPR lines.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> We'd just need to go through your
filtering
> > > options
> > > > > > > > > one-by-one
> > > > > > > > > > to
> > > > > > > > > > > > > find
> > > > > > > > > > > > > > >>> out
> > > > > > > > > > > > > > >>> why.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> - Are you passing STAT-Analysis files
which
> > > contain
> > > > > MPR
> > > > > > > > > lines?
> > > > > > > > > > > > > > >>> - Does the FCST_VAR column say "WIND"?
> > > > > > > > > > > > > > >>> - Does the FCST_LEV column say "Z10"?
> > > > > > > > > > > > > > >>> - Does the OBTYPE column say "ASCAT"?
> > > > > > > > > > > > > > >>> - Does the OBS column have values >
1.0?
> > > > > > > > > > > > > > >>> - What are the ranges of the OBS_LAT
and
> > OBS_LON
> > > > > > columns?
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> If you'd like to send me a sample
.stat
> file, I
> > > > could
> > > > > > > > > probably
> > > > > > > > > > > > track
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > >>> down.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Thanks,
> > > > > > > > > > > > > > >>> John
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM Rosalyn
> > > MacCracken -
> > > > > > NOAA
> > > > > > > > > > > Affiliate
> > > > > > > > > > > > > via
> > > > > > > > > > > > > > >>> RT <
> > > > > > > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > Mon Aug 06 07:43:46 2018: Request
86480 was
> > > acted
> > > > > > upon.
> > > > > > > > > > > > > > >>> > Transaction: Ticket created by
> > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > >>> > Queue: met_help
> > > > > > > > > > > > > > >>> > Subject: Errors using
-column_thresh
> > > > > > > > > > > > > > >>> > Owner: Nobody
> > > > > > > > > > > > > > >>> > Requestors:
rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > >>> > Status: new
> > > > > > > > > > > > > > >>> > Ticket <URL:
> > > > > > > > > > > > > >
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > Hi,
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > I'm trying to use -column_thresh
with
> > > > stat_analysis
> > > > > > to
> > > > > > > > mask
> > > > > > > > > > > out a
> > > > > > > > > > > > > > >>> lat/lon
> > > > > > > > > > > > > > >>> > box. I've added this command in my
> > > StatAnalysis
> > > > > > config
> > > > > > > > > file:
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > "-job aggregate_stat -out_line_type
CTC
> > > > > > -column_thresh
> > > > > > > > OBS
> > > > > > > > > > > gt1.0
> > > > > > > > > > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > > > > > > > > > >>> > -column_thresh OBS_LAT
> 'ge15.0&&le70.0' \
> > > > > > > > > > > > > > >>> > -column_thresh OBS_LON
> > 'ge-100.0&&le-10.0'
> > > \
> > > > > > > > > > > > > > >>> > -by
FCST_VALID_BEG,FCST_LEAD,VX_MASK \
> > > > > > > > > > > > > > >>> > -out_stat
> /opc_test/home/opc_test/data/m
> > > > > > > > > > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > > > > > > > > > >>> >
> > > > > > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > But, when it runs, I get these
errors:
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > Output:
> > > > > > > > > > > > > > >>> > DEBUG 2: Processing Job 2: -job
> > aggregate_stat
> > > > > > > -fcst_var
> > > > > > > > > WIND
> > > > > > > > > > > > > > >>> -fcst_lev Z10
> > > > > > > > > > > > > > >>> > -obtype ASCAT -line_type MPR
-column_thresh
> > OBS
> > > > > >1.0
> > > > > > > > > > > > -column_thresh
> > > > > > > > > > > > > > >>> OBS_LAT
> > > > > > > > > > > > > > >>> > >=15.0&&<=70.0 -column_thresh
OBS_LON
> > > > > > >=-100.0&&<=-10.0
> > > > > > > > -by
> > > > > > > > > > > > > > >>> FCST_VALID_BEG
> > > > > > > > > > > > > > >>> > -by FCST_LEAD -by VX_MASK -out_stat
> > > > > > > > > > > /opc_test/home/opc_test/data/
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > MPR
> > > > > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > > > > >>> > -out_line_type CTC -out_thresh
>=5.5689
> > > > > > > > > > > > > > >>> > DEBUG 1: Creating STAT output file
> > > > > > > > > > > "/opc_test/home/opc_test/data/
> > > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > > > > > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > > > > > > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > > > > > > > > > > >>> > DEBUG 2: Computing output for 0
case(s).
> > > > > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > > > > >>> > WARNING: do_job_aggr_stat() -> no
matching
> > STAT
> > > > > lines
> > > > > > > > found
> > > > > > > > > > for
> > > > > > > > > > > > > job:
> > > > > > > > > > > > > > >>> -job
> > > > > > > > > > > > > > >>> > aggregate_stat -fcst_var WIND
-fcst_lev Z10
> > > > -obtype
> > > > > > > ASCAT
> > > > > > > > > > > > > -line_type
> > > > > > > > > > > > > > >>> MPR
> > > > > > > > > > > > > > >>> > -column_thresh OBS >1.0
-column_thresh
> > OBS_LAT
> > > > > > > > > >=15.0&&<=70.0
> > > > > > > > > > > > > > >>> > -column_thresh OBS_LON >=-100.0&&<=-
10.0
> -by
> > > > > > > > FCST_VALID_BEG
> > > > > > > > > > -by
> > > > > > > > > > > > > > >>> FCST_LEAD
> > > > > > > > > > > > > > >>> > -by VX_MASK -out_stat
> > > > /opc_test/home/opc_test/data/
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > MPR
> > > > > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > > > > >>> > -out_line_type CTC -out_thresh
>=5.5689
> > > > > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137
STAT
> > lines.
> > > > > > > > > > > > > > >>> > DEBUG 2:
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > Am I not using the command properly?
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > Thanks!
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > Roz
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > --
> > > > > > > > > > > > > > >>> > Rosalyn MacCracken
> > > > > > > > > > > > > > >>> > Support Scientist
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > Ocean Applications Branch
> > > > > > > > > > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > >>> > NCWCP
> > > > > > > > > > > > > > >>> > 5830 University Research Ct
> > > > > > > > > > > > > > >>> > College Park, MD 20740-3818
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > (p) 301-683-1551
> > > > > > > > > > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> --
> > > > > > > > > > > > > > >> Rosalyn MacCracken
> > > > > > > > > > > > > > >> Support Scientist
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Ocean Applications Branch
> > > > > > > > > > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > >> NCWCP
> > > > > > > > > > > > > > >> 5830 University Research Ct
> > > > > > > > > > > > > > >> College Park, MD 20740-3818
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> (p) 301-683-1551
> > > > > > > > > > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > Support Scientist
> > > > > > > > > > > >
> > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > NCWCP
> > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > >
> > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > Support Scientist
> > > > > > > > > >
> > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > NCWCP
> > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > >
> > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rosalyn MacCracken
> > > > > > > > Support Scientist
> > > > > > > >
> > > > > > > > Ocean Applications Branch
> > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > NCWCP
> > > > > > > > 5830 University Research Ct
> > > > > > > > College Park, MD 20740-3818
> > > > > > > >
> > > > > > > > (p) 301-683-1551
> > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applications Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Errors using -column_thresh
From: John Halley Gotway
Time: Wed Aug 15 11:22:15 2018
Yes, that's right. If "0" is a good obs and "1" is a bad obs, you can
add
this to your stat-analysis jobs:
-column_str OBS_QC 0 ... to only use the ones that have 0 in the
OBS_QC column.
FYI, since your QC flag actually is numeric, you could also use
"-column_thresh OBS_QC eq0" to do the same thing.
The -column_str option does string matching and the -column_thresh
option
applies a threshold to numeric values.
John
On Wed, Aug 15, 2018 at 9:56 AM Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
>
> Ok, so, that would be for going forward. But you were saying I
could add
> it to the MPR files, and use it to filter in Stat-Analysis? That's
what
> you were saying about -column_str. If the value of "1" is there,
> stat-analysis will see it as a bad ob and ignore it? Is that right?
>
> Roz
>
> On Wed, Aug 15, 2018 at 3:25 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Roz,
> >
> > Here's how the filtering by quality control flag logic would work:
> > (1) The ascii data now has a meaningful quality control flag: 0
for good
> > and 1 for bad.
> > (2) When you run it through ascii2nc, that quality control info
will be
> > passed along to the output file.
> > (3) Set up your Point-Stat configuration file to specify which
quality
> > control flag you want to include. In your case, you only want to
use
> 0's:
> > obs_quality = [ "0" ];
> > Note that this is specified as a string, since some obs use
integers
> > (PREPBUFR) and other use strings (MADIS).
> >
> > In this way Point-Stat will ignore any observations that have a
quality
> > control flag that is not "0".
> >
> > Whether you ignore those obs when you pre-process them into ascii
format
> or
> > ignore them when you run point-stat, the effect should be the
same. So
> > it's up to you to setup the logic how you'd like.
> >
> > John
> >
> >
> > On Wed, Aug 15, 2018 at 8:04 AM Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > >
> > > Ok, I think that I could just set the bad observations in that
column
> to
> > 1.
> > >
> > > So, if I do that, how will they be ignored in further
processing? Do I
> > > need to set something so those observations will be skipped?
> > >
> > > Roz
> > >
> > > On Tue, Aug 14, 2018 at 7:33 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Roz,
> > > >
> > > > Sure, you can write a constant value to the OBS_QC column.
And
> using 0
> > > > versus a 1 is just fine.
> > > >
> > > > John
> > > >
> > > > On Mon, Aug 13, 2018 at 1:32 PM Rosalyn MacCracken - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > >
> > > > > No, that's correct. Originally, we weren't checking for the
flag,
> > and
> > > > just
> > > > > taking the observation from the MEDRlite file, and writing
it
> > directly
> > > > to a
> > > > > MET ASCII format file. So, as a result, I have MPR files
that have
> > N/A
> > > > > where a QC flag number should be.
> > > > >
> > > > > Now, we are using bit position in place of the quality
control
> flag.
> > > > So, I
> > > > > take the binary MGDRlite data, read it into a python
program,
> > evaluate
> > > if
> > > > > the bit position is a "1". If it's a "0", then it writes a
line to
> > an
> > > > > ASCII file that has the MET format to be used with ASCII2NC.
If
> > it's a
> > > > > "1", it skips writing that value to the MET formatted file.
This
> > way,
> > > we
> > > > > are only keeping good observations and tossing the others.
That's
> > why
> > > I
> > > > > asked about just using some constant value, because it's bit
> position
> > > and
> > > > > not a QC flag. I was thinking it it's something other than
N/A,
> then
> > > it
> > > > > won't be used. And, it's only for the back files. I can
correct
> it
> > > > going
> > > > > forward, and just not write that observation to my ASCII
file.
> > > > >
> > > > > Roz
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Aug 13, 2018 at 7:20 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Roz,
> > > > > >
> > > > > > I'm confused. Here's is what I understood the situation
to be...
> > > > > >
> > > > > > (1) In March, you switched from pulling ASCAT data from
PREPBUFR
> to
> > > > > pulling
> > > > > > it from MGDRlite.
> > > > > > (2) When you made that switch, you forgot to set the
system up to
> > > > filter
> > > > > > the MGDRlite data by quality control flags.
> > > > > > (3) As a result of that, many "bad" observations were
included in
> > > your
> > > > > > evaluation which led to bad statistics.
> > > > > > (4) You now want to go back and correct the bad evaluation
by
> > > > > reprocessing
> > > > > > using the actual quality control flags.
> > > > > >
> > > > > > I suggested that you modify the MET output MPR lines by
replacing
> > the
> > > > NA
> > > > > > string with the *actual* quality control flag. This logic
only
> > works
> > > > if
> > > > > > the MGDRlite data source includes quality control
information...
> > and
> > > > you
> > > > > > know how to extract it.
> > > > > >
> > > > > > Just putting a constant flag value of 2 in the OBS_QC
column
> won't
> > > > really
> > > > > > fix anything. I'm suggesting that you get the actual flag
values
> > and
> > > > > > "patch" the existing MPR lines by inserting them.
> > > > > >
> > > > > > Am I misunderstanding the situation?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Mon, Aug 13, 2018 at 12:47 PM Rosalyn MacCracken - NOAA
> > Affiliate
> > > > via
> > > > > RT
> > > > > > <met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> >
> > > > > > >
> > > > > > > No, I don't know the correct OBS_QC value. Can I just
make
> > > something
> > > > > up,
> > > > > > > like use "2", or some arbitrary number? Or does MET
expect
> some
> > > > value,
> > > > > > > given a certain file type?
> > > > > > >
> > > > > > > Actually, I currently don't write these values into my
point
> > file,
> > > > > which
> > > > > > I
> > > > > > > use to make the netcdf file. So, it's just for the back
data.
> > > > > > >
> > > > > > > Roz
> > > > > > >
> > > > > > > On Mon, Aug 13, 2018 at 6:41 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > > Right, updating the OBS_QC values from NA to the real
QC
> value
> > is
> > > > the
> > > > > > > > difficult part, but I think that's probably easier
that
> > > "gridding"
> > > > > the
> > > > > > > > point data and going back through Point-Stat.
> > > > > > > >
> > > > > > > > So do you know how to get the QBS_QC value from the
MGDRlite
> > > data?
> > > > > > And,
> > > > > > > > going forward, can you set it up so that the OBS_QC
column is
> > > > > populated
> > > > > > > in
> > > > > > > > your Point-Stat runs going forward?
> > > > > > > >
> > > > > > > > Perhaps that's the place to start. Make sure your new
> > Point-Stat
> > > > > > output
> > > > > > > > includes meaningful OBS_QC output. Once that's
working, go
> > > through
> > > > > > your
> > > > > > > > *old* output and backfill the OBS_QC column with what
it
> should
> > > > > > contain.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Mon, Aug 13, 2018 at 12:37 PM Rosalyn MacCracken -
NOAA
> > > > Affiliate
> > > > > > via
> > > > > > > RT
> > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=86480
> > > >
> > > > > > > > >
> > > > > > > > > Yes, that makes sense. So, the MPR line would be
"N/A"
> that
> > it
> > > > > would
> > > > > > > > > process the data with. Ok. And, I could do
something
> > similar
> > > to
> > > > > > what
> > > > > > > I
> > > > > > > > > did before with copying my MPR files to a temp
directory,
> and
> > > use
> > > > > > > > -lookin.
> > > > > > > > >
> > > > > > > > > Ok, so, the hard part will be to write this script
that
> > > replaces
> > > > my
> > > > > > bad
> > > > > > > > > values. I guess I like a challenge, right?
> > > > > > > > >
> > > > > > > > > Roz
> > > > > > > > >
> > > > > > > > > On Mon, Aug 13, 2018 at 6:30 PM, John Halley Gotway
via RT
> <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > > Roz,
> > > > > > > > > >
> > > > > > > > > > Once the OBS_QC values are set in the MPR lines,
you can
> > use
> > > > > > > > > STAT-Analysis
> > > > > > > > > > to reprocess the data and recompute whatever other
STAT
> > lines
> > > > > you'd
> > > > > > > > like
> > > > > > > > > to
> > > > > > > > > > compute:
> > > > > > > > > >
> > > > > > > > > > stat_analysis -job aggregate_stat -line_type MPR
> > > -out_line_type
> > > > > CNT
> > > > > > > > > > -column_str OBS_QC a,b,c
> > > > > > > > > >
> > > > > > > > > > This job would be run only on MPR lines that have
strings
> > of
> > > > "a",
> > > > > > > "b",
> > > > > > > > or
> > > > > > > > > > "c" in the OBS_QC column. Make sense?
> > > > > > > > > >
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 13, 2018 at 11:55 AM Rosalyn
MacCracken -
> NOAA
> > > > > > Affiliate
> > > > > > > > via
> > > > > > > > > RT
> > > > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > Ticket/Display.html?id=86480
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi John,
> > > > > > > > > > >
> > > > > > > > > > > Yeah, I can repost the file. I'll do that in a
bit.
> > > > Actually,
> > > > > > if
> > > > > > > I
> > > > > > > > > can
> > > > > > > > > > > get this issue solved, you don't need to look at
the
> > file,
> > > > > > because
> > > > > > > > I'll
> > > > > > > > > > > just reprocess that data with my new method.
So, I'll
> > hold
> > > > off
> > > > > > on
> > > > > > > > > ftping
> > > > > > > > > > > that file to you for right now.
> > > > > > > > > > >
> > > > > > > > > > > So, the OBS_QC column has N/A values. That's
easy
> enough
> > > to
> > > > > > > replace.
> > > > > > > > > > Now,
> > > > > > > > > > > I have the original ASCAT data, which looks for
the
> > quality
> > > > > flag
> > > > > > > as a
> > > > > > > > > bit
> > > > > > > > > > > (i.e., ibits[5]) and then doesn't use that. It
doesn't
> > > write
> > > > > it
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > output file. Maybe what I should do is find the
first
> > bad
> > > > > > > > observation
> > > > > > > > > > > (time, lat, lon, value), then, match it exactly
to the
> > line
> > > > in
> > > > > > the
> > > > > > > > mpr
> > > > > > > > > > > file, and replace the observation.
> > > > > > > > > > >
> > > > > > > > > > > So, if my observations are in dataframes, and
the mpr
> > file
> > > > has
> > > > > > been
> > > > > > > > > read
> > > > > > > > > > in
> > > > > > > > > > > as a dataframe, then, I could do a df.query('Lat
= ??
> and
> > > Lon
> > > > > > =??),
> > > > > > > > or
> > > > > > > > > > > however the syntax is...no wait, I need an if
statement
> > to
> > > > > > compare
> > > > > > > > > > > first...I'll work on those detail. Then, once I
get
> the
> > > > OBS_QC
> > > > > > > value
> > > > > > > > > > > changed, then what? How do you get the other
stat
> files
> > > from
> > > > > > this
> > > > > > > > > point?
> > > > > > > > > > >
> > > > > > > > > > > Roz
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Aug 13, 2018 at 5:32 PM, John Halley
Gotway via
> > RT
> > > <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Roz,
> > > > > > > > > > > >
> > > > > > > > > > > > That's correct, there is no tool in MET for
comparing
> > one
> > > > set
> > > > > > of
> > > > > > > > > point
> > > > > > > > > > > > observations to another set of point
observations.
> > > > > Presumably
> > > > > > > you
> > > > > > > > > > could
> > > > > > > > > > > > write point observations to a grid and then
> reconfigure
> > > and
> > > > > > rerun
> > > > > > > > > > > > Point-Stat, but that'd be pretty difficult,
time
> > > consuming,
> > > > > and
> > > > > > > > > fraught
> > > > > > > > > > > > with problems.
> > > > > > > > > > > >
> > > > > > > > > > > > You currently have many, many MPR lines and
the issue
> > is
> > > > the
> > > > > > the
> > > > > > > > > OBS_QC
> > > > > > > > > > > > column is empty. If that OBS_QC column were
*not*
> > empty
> > > > then
> > > > > > > it'd
> > > > > > > > be
> > > > > > > > > > > > pretty easy to process this data... correct?
One
> > option
> > > > > would
> > > > > > be
> > > > > > > > > > > writing a
> > > > > > > > > > > > script to "patch" that column. For each MPR
line, do
> > > some
> > > > > > > > processing
> > > > > > > > > > to
> > > > > > > > > > > > figure out what that OBS_QC column should be
and then
> > > > update
> > > > > > it's
> > > > > > > > > > value.
> > > > > > > > > > > >
> > > > > > > > > > > > Then the question is this... is there a way of
> getting
> > > > > quality
> > > > > > > > > control
> > > > > > > > > > > > values from the MGDRlite files?
> > > > > > > > > > > >
> > > > > > > > > > > > I'm sorry, I hadn't had a chance to look at
that data
> > > file
> > > > > yet.
> > > > > > > > But
> > > > > > > > > I
> > > > > > > > > > > just
> > > > > > > > > > > > checked on the ftp site and don't see a
maccracken
> > > > > > sub-directory
> > > > > > > > > there:
> > > > > > > > > > > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > > > > > > > > > > >
> > > > > > > > > > > > Could you please repost?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > John
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn
MacCracken -
> > > NOAA
> > > > > > > > Affiliate
> > > > > > > > > > via
> > > > > > > > > > > RT
> > > > > > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > > Ticket/Display.html?id=86480
> > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > So, you can't compare two point files, even
if both
> > of
> > > > them
> > > > > > are
> > > > > > > > > > NetCDF
> > > > > > > > > > > > > format? Is there a way to write point
observations
> > out
> > > > to
> > > > > a
> > > > > > > > grid?
> > > > > > > > > > In
> > > > > > > > > > > > > other words, take the point GFS
observations, and
> > write
> > > > > them
> > > > > > > to a
> > > > > > > > > > > global
> > > > > > > > > > > > > grid (using some other tool, like wgrib or
> something
> > > > else),
> > > > > > and
> > > > > > > > > fill
> > > > > > > > > > in
> > > > > > > > > > > > > missing values with NaN, or something?
Then, use
> > that
> > > > > > gridded
> > > > > > > > file
> > > > > > > > > > to
> > > > > > > > > > > > > compare with the ASCAT NetCDF file?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't have anything in the OBS_QC column
because
> > > when I
> > > > > > wrote
> > > > > > > > it
> > > > > > > > > > into
> > > > > > > > > > > > the
> > > > > > > > > > > > > ASCII file, I didn't have any flags from my
> MGDRlite
> > > > files.
> > > > > > > This
> > > > > > > > > is
> > > > > > > > > > > part
> > > > > > > > > > > > > of the problem...
> > > > > > > > > > > > >
> > > > > > > > > > > > > Now, your answer for the STAT-Analysis, is
that
> what
> > I
> > > > did
> > > > > > > wrong
> > > > > > > > in
> > > > > > > > > > my
> > > > > > > > > > > > use
> > > > > > > > > > > > > of stat-analysis from the previous email?
Or, did
> > you
> > > > not
> > > > > > get
> > > > > > > a
> > > > > > > > > > chance
> > > > > > > > > > > > to
> > > > > > > > > > > > > look at my *stat file that I uploaded to the
ftp
> > site?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Roz
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Aug 13, 2018 at 3:42 PM, John Halley
Gotway
> > via
> > > > RT
> > > > > <
> > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Roz,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Point-Stat takes as input a gridded NWP
forecast
> > file
> > > > and
> > > > > > the
> > > > > > > > > > NetCDF
> > > > > > > > > > > > > output
> > > > > > > > > > > > > > of a point pre-processing tool (like
ascii2nc or
> > > > pb2nc).
> > > > > > > Based
> > > > > > > > > on
> > > > > > > > > > > your
> > > > > > > > > > > > > > description, it doesn't sound like you
have
> gridded
> > > NWP
> > > > > > > > forecast
> > > > > > > > > > > files
> > > > > > > > > > > > > > available. So no, I don't think that
approach
> > would
> > > > > work.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The MPR output line type does include a
column
> > named
> > > > > > "OBS_QC"
> > > > > > > > > which
> > > > > > > > > > > is
> > > > > > > > > > > > a
> > > > > > > > > > > > > > string containing the observation quality
control
> > > > value.
> > > > > > I'd
> > > > > > > > > > suggest
> > > > > > > > > > > > > > checking the contents of that column in
your MPR
> > > output
> > > > > > lines
> > > > > > > > to
> > > > > > > > > > see
> > > > > > > > > > > if
> > > > > > > > > > > > > > those values could be filtered to do the
quality
> > > > control
> > > > > > > > > filtering
> > > > > > > > > > > > you're
> > > > > > > > > > > > > > after. Check to see if the values in that
column
> > are
> > > > > > strings
> > > > > > > > or
> > > > > > > > > > > > numbers
> > > > > > > > > > > > > > (in PREPBUFR they're integers... in MADIS
they're
> > > > > strings).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In STAT-Analysis, you can use the
> "-column_thresh"
> > > > option
> > > > > > to
> > > > > > > > > > > threshold
> > > > > > > > > > > > a
> > > > > > > > > > > > > > numeric column or you can use the "-
column_str"
> > > option
> > > > to
> > > > > > > list
> > > > > > > > > > > strings
> > > > > > > > > > > > to
> > > > > > > > > > > > > > retained. For example:
> > > > > > > > > > > > > > -column_thresh OBS_QC 'ge0&&le3' #
this keeps
> > all
> > > > > > values
> > > > > > > > > > between
> > > > > > > > > > > 0
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > 3
> > > > > > > > > > > > > > -column_str OBS_QC aa,ab,ac #
this
> keeps
> > > > > strings
> > > > > > > > "aa",
> > > > > > > > > > > "ab",
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > "ac"
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hope that helps.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > John
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn
> MacCracken
> > -
> > > > NOAA
> > > > > > > > > Affiliate
> > > > > > > > > > > via
> > > > > > > > > > > > > RT
> > > > > > > > > > > > > > <
> > > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > > Ticket/Display.html?id=86480
> > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi John,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have another question related to my
> > reprocessing
> > > of
> > > > > my
> > > > > > > > data.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ok, so, I realized that when I switched
ASCAT
> > data
> > > > > > sources
> > > > > > > > back
> > > > > > > > > > in
> > > > > > > > > > > > > March,
> > > > > > > > > > > > > > > from prepbufr to MGDRlite files, I
should have
> > been
> > > > > using
> > > > > > > > > quality
> > > > > > > > > > > > flags
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > filter out my data. Because I wasn't
checking
> > for
> > > > the
> > > > > > > > quality
> > > > > > > > > > > flags,
> > > > > > > > > > > > > my
> > > > > > > > > > > > > > > statistics were not so good, since it
was
> letting
> > > in
> > > > > bad
> > > > > > > > data,
> > > > > > > > > > and
> > > > > > > > > > > > > stats
> > > > > > > > > > > > > > > were being calculated from this bad
data.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > So, I need to find a way to correct
this, but,
> > the
> > > > old
> > > > > > > story
> > > > > > > > > is I
> > > > > > > > > > > > don't
> > > > > > > > > > > > > > > have the original GFS data, and it's too
> massive
> > to
> > > > > grab
> > > > > > > from
> > > > > > > > > > HPSS.
> > > > > > > > > > > > I
> > > > > > > > > > > > > do
> > > > > > > > > > > > > > > have the original ASCAT files, which I
can run
> > > > through
> > > > > my
> > > > > > > > > program
> > > > > > > > > > > to
> > > > > > > > > > > > > > filter
> > > > > > > > > > > > > > > out the bad observations, as well as
matched
> > points
> > > > > from
> > > > > > > the
> > > > > > > > > *mpr
> > > > > > > > > > > and
> > > > > > > > > > > > > > *stat
> > > > > > > > > > > > > > > files.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Originally, I was just going to filter
out
> using
> > > that
> > > > > > > > > > > -column_thresh
> > > > > > > > > > > > > > > command, and filter out over land. But,
I had
> > this
> > > > > idea
> > > > > > > > that I
> > > > > > > > > > > > wanted
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > run by you, to see if this makes sense
to do.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have the time, lat, lon and GFS
observation
> and
> > > the
> > > > > > > > > > corresponding
> > > > > > > > > > > > > ASCAT
> > > > > > > > > > > > > > > observation point. Can I do this:
> > > > > > > > > > > > > > > 1) read the time, lat, lon and GFS
observation
> > out
> > > > of
> > > > > > the
> > > > > > > > mpr
> > > > > > > > > > > file,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > make a new input file with just those
variables
> > in
> > > > the
> > > > > > > ASCII
> > > > > > > > > > format
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > MET can use
> > > > > > > > > > > > > > > 2) reprocess my ASCAT data and write it
into
> an
> > > > ASCII
> > > > > > file
> > > > > > > > in
> > > > > > > > > > MET
> > > > > > > > > > > > > format
> > > > > > > > > > > > > > > 3) run ASCII2NC on these files, to
create the
> > two
> > > > > input
> > > > > > > > files
> > > > > > > > > > for
> > > > > > > > > > > > MET,
> > > > > > > > > > > > > > > then, run point_stat on the netcdf files
to
> > > > regenerate
> > > > > > the
> > > > > > > > new
> > > > > > > > > > > > > corrected
> > > > > > > > > > > > > > > stat files
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Does that make sense to do? I know it
could be
> > > alot
> > > > of
> > > > > > > work,
> > > > > > > > > > but,
> > > > > > > > > > > > if I
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > get corrected statistics, it might be
worth it.
> > > What
> > > > > do
> > > > > > > you
> > > > > > > > > > think?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Roz
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 10, 2018 at 7:24 PM, Rosalyn
> > > MacCracken -
> > > > > > NOAA
> > > > > > > > > > > Affiliate
> > > > > > > > > > > > <
> > > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi John,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I double checked all those things you
listed,
> > and
> > > > > they
> > > > > > > are
> > > > > > > > > all
> > > > > > > > > > > > > correct.
> > > > > > > > > > > > > > > > So, I've put a file in the /maccracken
> > directory
> > > on
> > > > > > your
> > > > > > > > ftp
> > > > > > > > > > > site.
> > > > > > > > > > > > > Let
> > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > know what you see.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for your help!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Roz
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Aug 10, 2018 at 1:41 PM,
Rosalyn
> > > > MacCracken -
> > > > > > > NOAA
> > > > > > > > > > > > Affiliate
> > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Hi John,
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> I'm sorry I didn't get back to you
sooner,
> > but,
> > > I
> > > > > was
> > > > > > > tied
> > > > > > > > > up
> > > > > > > > > > > with
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > >> other things. Let me double check
with all
> of
> > > > those
> > > > > > > items
> > > > > > > > > in
> > > > > > > > > > my
> > > > > > > > > > > > > file
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> make sure that I don't have some sort
of
> > obvious
> > > > > > error.
> > > > > > > > > I'll
> > > > > > > > > > be
> > > > > > > > > > > > > back
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > >> touch after I check on this.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Roz
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM, John
Halley
> > > Gotway
> > > > > via
> > > > > > > RT
> > > > > > > > <
> > > > > > > > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>> Hi Roz,
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> I see you're having trouble getting
with
> > > > > > STAT-Analysis,
> > > > > > > > > > getting
> > > > > > > > > > > > MPR
> > > > > > > > > > > > > > > lines
> > > > > > > > > > > > > > > >>> to be included in the column
thresholds
> > you've
> > > > > > defined.
> > > > > > > > I
> > > > > > > > > > > don't
> > > > > > > > > > > > > see
> > > > > > > > > > > > > > > >>> anything obviously wrong with what
you've
> > > > defined.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> The warning message indicates that
the
> > filters
> > > > you
> > > > > > > > defined
> > > > > > > > > > have
> > > > > > > > > > > > > > > discarded
> > > > > > > > > > > > > > > >>> all of the input MPR lines.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> We'd just need to go through your
filtering
> > > > options
> > > > > > > > > > one-by-one
> > > > > > > > > > > to
> > > > > > > > > > > > > > find
> > > > > > > > > > > > > > > >>> out
> > > > > > > > > > > > > > > >>> why.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> - Are you passing STAT-Analysis
files which
> > > > contain
> > > > > > MPR
> > > > > > > > > > lines?
> > > > > > > > > > > > > > > >>> - Does the FCST_VAR column say
"WIND"?
> > > > > > > > > > > > > > > >>> - Does the FCST_LEV column say
"Z10"?
> > > > > > > > > > > > > > > >>> - Does the OBTYPE column say
"ASCAT"?
> > > > > > > > > > > > > > > >>> - Does the OBS column have values >
1.0?
> > > > > > > > > > > > > > > >>> - What are the ranges of the OBS_LAT
and
> > > OBS_LON
> > > > > > > columns?
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> If you'd like to send me a sample
.stat
> > file, I
> > > > > could
> > > > > > > > > > probably
> > > > > > > > > > > > > track
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > >>> down.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> Thanks,
> > > > > > > > > > > > > > > >>> John
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM
Rosalyn
> > > > MacCracken -
> > > > > > > NOAA
> > > > > > > > > > > > Affiliate
> > > > > > > > > > > > > > via
> > > > > > > > > > > > > > > >>> RT <
> > > > > > > > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > Mon Aug 06 07:43:46 2018: Request
86480
> was
> > > > acted
> > > > > > > upon.
> > > > > > > > > > > > > > > >>> > Transaction: Ticket created by
> > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > >>> > Queue: met_help
> > > > > > > > > > > > > > > >>> > Subject: Errors using
-column_thresh
> > > > > > > > > > > > > > > >>> > Owner: Nobody
> > > > > > > > > > > > > > > >>> > Requestors:
> rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > >>> > Status: new
> > > > > > > > > > > > > > > >>> > Ticket <URL:
> > > > > > > > > > > > > > >
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > Hi,
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > I'm trying to use -column_thresh
with
> > > > > stat_analysis
> > > > > > > to
> > > > > > > > > mask
> > > > > > > > > > > > out a
> > > > > > > > > > > > > > > >>> lat/lon
> > > > > > > > > > > > > > > >>> > box. I've added this command in
my
> > > > StatAnalysis
> > > > > > > config
> > > > > > > > > > file:
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > "-job aggregate_stat
-out_line_type CTC
> > > > > > > -column_thresh
> > > > > > > > > OBS
> > > > > > > > > > > > gt1.0
> > > > > > > > > > > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > > > > > > > > > > >>> > -column_thresh OBS_LAT
> > 'ge15.0&&le70.0' \
> > > > > > > > > > > > > > > >>> > -column_thresh OBS_LON
> > > 'ge-100.0&&le-10.0'
> > > > \
> > > > > > > > > > > > > > > >>> > -by
FCST_VALID_BEG,FCST_LEAD,VX_MASK
> \
> > > > > > > > > > > > > > > >>> > -out_stat
> > /opc_test/home/opc_test/data/m
> > > > > > > > > > > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > > > > > > > > > > >>> >
> > > > > > > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > But, when it runs, I get these
errors:
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > Output:
> > > > > > > > > > > > > > > >>> > DEBUG 2: Processing Job 2: -job
> > > aggregate_stat
> > > > > > > > -fcst_var
> > > > > > > > > > WIND
> > > > > > > > > > > > > > > >>> -fcst_lev Z10
> > > > > > > > > > > > > > > >>> > -obtype ASCAT -line_type MPR
> -column_thresh
> > > OBS
> > > > > > >1.0
> > > > > > > > > > > > > -column_thresh
> > > > > > > > > > > > > > > >>> OBS_LAT
> > > > > > > > > > > > > > > >>> > >=15.0&&<=70.0 -column_thresh
OBS_LON
> > > > > > > >=-100.0&&<=-10.0
> > > > > > > > > -by
> > > > > > > > > > > > > > > >>> FCST_VALID_BEG
> > > > > > > > > > > > > > > >>> > -by FCST_LEAD -by VX_MASK
-out_stat
> > > > > > > > > > > > /opc_test/home/opc_test/data/
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > > MPR
> > > > > > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > > > > > >>> > -out_line_type CTC -out_thresh
>=5.5689
> > > > > > > > > > > > > > > >>> > DEBUG 1: Creating STAT output file
> > > > > > > > > > > > "/opc_test/home/opc_test/data/
> > > > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > > > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > > > > > > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > > > > > > > > > > >>> > GSL_RNG_SEED=18446744073401622619
> > > > > > > > > > > > > > > >>> > DEBUG 2: Computing output for 0
case(s).
> > > > > > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > > > > > >>> > WARNING: do_job_aggr_stat() -> no
> matching
> > > STAT
> > > > > > lines
> > > > > > > > > found
> > > > > > > > > > > for
> > > > > > > > > > > > > > job:
> > > > > > > > > > > > > > > >>> -job
> > > > > > > > > > > > > > > >>> > aggregate_stat -fcst_var WIND
-fcst_lev
> Z10
> > > > > -obtype
> > > > > > > > ASCAT
> > > > > > > > > > > > > > -line_type
> > > > > > > > > > > > > > > >>> MPR
> > > > > > > > > > > > > > > >>> > -column_thresh OBS >1.0
-column_thresh
> > > OBS_LAT
> > > > > > > > > > >=15.0&&<=70.0
> > > > > > > > > > > > > > > >>> > -column_thresh OBS_LON >=-
100.0&&<=-10.0
> > -by
> > > > > > > > > FCST_VALID_BEG
> > > > > > > > > > > -by
> > > > > > > > > > > > > > > >>> FCST_LEAD
> > > > > > > > > > > > > > > >>> > -by VX_MASK -out_stat
> > > > > /opc_test/home/opc_test/data/
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > > MPR
> > > > > > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > > > > > >>> > -out_line_type CTC -out_thresh
>=5.5689
> > > > > > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > > > > > >>> > DEBUG 2: Job 2 used 0 out of 69137
STAT
> > > lines.
> > > > > > > > > > > > > > > >>> > DEBUG 2:
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > Am I not using the command
properly?
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > Thanks!
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > Roz
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > --
> > > > > > > > > > > > > > > >>> > Rosalyn MacCracken
> > > > > > > > > > > > > > > >>> > Support Scientist
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > Ocean Applications Branch
> > > > > > > > > > > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > > >>> > NCWCP
> > > > > > > > > > > > > > > >>> > 5830 University Research Ct
> > > > > > > > > > > > > > > >>> > College Park, MD 20740-3818
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > (p) 301-683-1551
> > > > > > > > > > > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> --
> > > > > > > > > > > > > > > >> Rosalyn MacCracken
> > > > > > > > > > > > > > > >> Support Scientist
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Ocean Applications Branch
> > > > > > > > > > > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > > >> NCWCP
> > > > > > > > > > > > > > > >> 5830 University Research Ct
> > > > > > > > > > > > > > > >> College Park, MD 20740-3818
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> (p) 301-683-1551
> > > > > > > > > > > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > >
> > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > Support Scientist
> > > > > > > > > > >
> > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > NCWCP
> > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > >
> > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rosalyn MacCracken
> > > > > > > > > Support Scientist
> > > > > > > > >
> > > > > > > > > Ocean Applications Branch
> > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > NCWCP
> > > > > > > > > 5830 University Research Ct
> > > > > > > > > College Park, MD 20740-3818
> > > > > > > > >
> > > > > > > > > (p) 301-683-1551
> > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rosalyn MacCracken
> > > > > > > Support Scientist
> > > > > > >
> > > > > > > Ocean Applications Branch
> > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > NCWCP
> > > > > > > 5830 University Research Ct
> > > > > > > College Park, MD 20740-3818
> > > > > > >
> > > > > > > (p) 301-683-1551
> > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applications Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Errors using -column_thresh
From: Rosalyn MacCracken - NOAA Affiliate
Time: Wed Aug 15 12:05:56 2018
Ok, that sounds like what I'm looking for. I have some more checking
on my
end before I actually start working on regenerating data. But, I'm
guessing, in the meantime, we could close the ticket.
thanks for your help!
Roz
On Wed, Aug 15, 2018 at 5:22 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Yes, that's right. If "0" is a good obs and "1" is a bad obs, you
can add
> this to your stat-analysis jobs:
> -column_str OBS_QC 0 ... to only use the ones that have 0 in
the
> OBS_QC column.
> FYI, since your QC flag actually is numeric, you could also use
> "-column_thresh OBS_QC eq0" to do the same thing.
>
> The -column_str option does string matching and the -column_thresh
option
> applies a threshold to numeric values.
>
> John
>
> On Wed, Aug 15, 2018 at 9:56 AM Rosalyn MacCracken - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> >
> > Ok, so, that would be for going forward. But you were saying I
could add
> > it to the MPR files, and use it to filter in Stat-Analysis?
That's what
> > you were saying about -column_str. If the value of "1" is there,
> > stat-analysis will see it as a bad ob and ignore it? Is that
right?
> >
> > Roz
> >
> > On Wed, Aug 15, 2018 at 3:25 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Roz,
> > >
> > > Here's how the filtering by quality control flag logic would
work:
> > > (1) The ascii data now has a meaningful quality control flag: 0
for
> good
> > > and 1 for bad.
> > > (2) When you run it through ascii2nc, that quality control info
will be
> > > passed along to the output file.
> > > (3) Set up your Point-Stat configuration file to specify which
quality
> > > control flag you want to include. In your case, you only want
to use
> > 0's:
> > > obs_quality = [ "0" ];
> > > Note that this is specified as a string, since some obs use
> integers
> > > (PREPBUFR) and other use strings (MADIS).
> > >
> > > In this way Point-Stat will ignore any observations that have a
quality
> > > control flag that is not "0".
> > >
> > > Whether you ignore those obs when you pre-process them into
ascii
> format
> > or
> > > ignore them when you run point-stat, the effect should be the
same. So
> > > it's up to you to setup the logic how you'd like.
> > >
> > > John
> > >
> > >
> > > On Wed, Aug 15, 2018 at 8:04 AM Rosalyn MacCracken - NOAA
Affiliate via
> > RT
> > > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
>
> > > >
> > > > Ok, I think that I could just set the bad observations in that
column
> > to
> > > 1.
> > > >
> > > > So, if I do that, how will they be ignored in further
processing?
> Do I
> > > > need to set something so those observations will be skipped?
> > > >
> > > > Roz
> > > >
> > > > On Tue, Aug 14, 2018 at 7:33 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Roz,
> > > > >
> > > > > Sure, you can write a constant value to the OBS_QC column.
And
> > using 0
> > > > > versus a 1 is just fine.
> > > > >
> > > > > John
> > > > >
> > > > > On Mon, Aug 13, 2018 at 1:32 PM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480 >
> > > > > >
> > > > > > No, that's correct. Originally, we weren't checking for
the
> flag,
> > > and
> > > > > just
> > > > > > taking the observation from the MEDRlite file, and writing
it
> > > directly
> > > > > to a
> > > > > > MET ASCII format file. So, as a result, I have MPR files
that
> have
> > > N/A
> > > > > > where a QC flag number should be.
> > > > > >
> > > > > > Now, we are using bit position in place of the quality
control
> > flag.
> > > > > So, I
> > > > > > take the binary MGDRlite data, read it into a python
program,
> > > evaluate
> > > > if
> > > > > > the bit position is a "1". If it's a "0", then it writes
a line
> to
> > > an
> > > > > > ASCII file that has the MET format to be used with
ASCII2NC. If
> > > it's a
> > > > > > "1", it skips writing that value to the MET formatted
file. This
> > > way,
> > > > we
> > > > > > are only keeping good observations and tossing the others.
> That's
> > > why
> > > > I
> > > > > > asked about just using some constant value, because it's
bit
> > position
> > > > and
> > > > > > not a QC flag. I was thinking it it's something other
than N/A,
> > then
> > > > it
> > > > > > won't be used. And, it's only for the back files. I can
correct
> > it
> > > > > going
> > > > > > forward, and just not write that observation to my ASCII
file.
> > > > > >
> > > > > > Roz
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Aug 13, 2018 at 7:20 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Roz,
> > > > > > >
> > > > > > > I'm confused. Here's is what I understood the situation
to
> be...
> > > > > > >
> > > > > > > (1) In March, you switched from pulling ASCAT data from
> PREPBUFR
> > to
> > > > > > pulling
> > > > > > > it from MGDRlite.
> > > > > > > (2) When you made that switch, you forgot to set the
system up
> to
> > > > > filter
> > > > > > > the MGDRlite data by quality control flags.
> > > > > > > (3) As a result of that, many "bad" observations were
included
> in
> > > > your
> > > > > > > evaluation which led to bad statistics.
> > > > > > > (4) You now want to go back and correct the bad
evaluation by
> > > > > > reprocessing
> > > > > > > using the actual quality control flags.
> > > > > > >
> > > > > > > I suggested that you modify the MET output MPR lines by
> replacing
> > > the
> > > > > NA
> > > > > > > string with the *actual* quality control flag. This
logic only
> > > works
> > > > > if
> > > > > > > the MGDRlite data source includes quality control
> information...
> > > and
> > > > > you
> > > > > > > know how to extract it.
> > > > > > >
> > > > > > > Just putting a constant flag value of 2 in the OBS_QC
column
> > won't
> > > > > really
> > > > > > > fix anything. I'm suggesting that you get the actual
flag
> values
> > > and
> > > > > > > "patch" the existing MPR lines by inserting them.
> > > > > > >
> > > > > > > Am I misunderstanding the situation?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Aug 13, 2018 at 12:47 PM Rosalyn MacCracken -
NOAA
> > > Affiliate
> > > > > via
> > > > > > RT
> > > > > > > <met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=86480
> > >
> > > > > > > >
> > > > > > > > No, I don't know the correct OBS_QC value. Can I just
make
> > > > something
> > > > > > up,
> > > > > > > > like use "2", or some arbitrary number? Or does MET
expect
> > some
> > > > > value,
> > > > > > > > given a certain file type?
> > > > > > > >
> > > > > > > > Actually, I currently don't write these values into my
point
> > > file,
> > > > > > which
> > > > > > > I
> > > > > > > > use to make the netcdf file. So, it's just for the
back
> data.
> > > > > > > >
> > > > > > > > Roz
> > > > > > > >
> > > > > > > > On Mon, Aug 13, 2018 at 6:41 PM, John Halley Gotway
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > > Right, updating the OBS_QC values from NA to the
real QC
> > value
> > > is
> > > > > the
> > > > > > > > > difficult part, but I think that's probably easier
that
> > > > "gridding"
> > > > > > the
> > > > > > > > > point data and going back through Point-Stat.
> > > > > > > > >
> > > > > > > > > So do you know how to get the QBS_QC value from the
> MGDRlite
> > > > data?
> > > > > > > And,
> > > > > > > > > going forward, can you set it up so that the OBS_QC
column
> is
> > > > > > populated
> > > > > > > > in
> > > > > > > > > your Point-Stat runs going forward?
> > > > > > > > >
> > > > > > > > > Perhaps that's the place to start. Make sure your
new
> > > Point-Stat
> > > > > > > output
> > > > > > > > > includes meaningful OBS_QC output. Once that's
working, go
> > > > through
> > > > > > > your
> > > > > > > > > *old* output and backfill the OBS_QC column with
what it
> > should
> > > > > > > contain.
> > > > > > > > >
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Mon, Aug 13, 2018 at 12:37 PM Rosalyn MacCracken
- NOAA
> > > > > Affiliate
> > > > > > > via
> > > > > > > > RT
> > > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=86480
> > > > >
> > > > > > > > > >
> > > > > > > > > > Yes, that makes sense. So, the MPR line would be
"N/A"
> > that
> > > it
> > > > > > would
> > > > > > > > > > process the data with. Ok. And, I could do
something
> > > similar
> > > > to
> > > > > > > what
> > > > > > > > I
> > > > > > > > > > did before with copying my MPR files to a temp
directory,
> > and
> > > > use
> > > > > > > > > -lookin.
> > > > > > > > > >
> > > > > > > > > > Ok, so, the hard part will be to write this script
that
> > > > replaces
> > > > > my
> > > > > > > bad
> > > > > > > > > > values. I guess I like a challenge, right?
> > > > > > > > > >
> > > > > > > > > > Roz
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 13, 2018 at 6:30 PM, John Halley
Gotway via
> RT
> > <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > > Roz,
> > > > > > > > > > >
> > > > > > > > > > > Once the OBS_QC values are set in the MPR lines,
you
> can
> > > use
> > > > > > > > > > STAT-Analysis
> > > > > > > > > > > to reprocess the data and recompute whatever
other STAT
> > > lines
> > > > > > you'd
> > > > > > > > > like
> > > > > > > > > > to
> > > > > > > > > > > compute:
> > > > > > > > > > >
> > > > > > > > > > > stat_analysis -job aggregate_stat -line_type MPR
> > > > -out_line_type
> > > > > > CNT
> > > > > > > > > > > -column_str OBS_QC a,b,c
> > > > > > > > > > >
> > > > > > > > > > > This job would be run only on MPR lines that
have
> strings
> > > of
> > > > > "a",
> > > > > > > > "b",
> > > > > > > > > or
> > > > > > > > > > > "c" in the OBS_QC column. Make sense?
> > > > > > > > > > >
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Aug 13, 2018 at 11:55 AM Rosalyn
MacCracken -
> > NOAA
> > > > > > > Affiliate
> > > > > > > > > via
> > > > > > > > > > RT
> > > > > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > Ticket/Display.html?id=86480
> > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi John,
> > > > > > > > > > > >
> > > > > > > > > > > > Yeah, I can repost the file. I'll do that in
a bit.
> > > > > Actually,
> > > > > > > if
> > > > > > > > I
> > > > > > > > > > can
> > > > > > > > > > > > get this issue solved, you don't need to look
at the
> > > file,
> > > > > > > because
> > > > > > > > > I'll
> > > > > > > > > > > > just reprocess that data with my new method.
So,
> I'll
> > > hold
> > > > > off
> > > > > > > on
> > > > > > > > > > ftping
> > > > > > > > > > > > that file to you for right now.
> > > > > > > > > > > >
> > > > > > > > > > > > So, the OBS_QC column has N/A values. That's
easy
> > enough
> > > > to
> > > > > > > > replace.
> > > > > > > > > > > Now,
> > > > > > > > > > > > I have the original ASCAT data, which looks
for the
> > > quality
> > > > > > flag
> > > > > > > > as a
> > > > > > > > > > bit
> > > > > > > > > > > > (i.e., ibits[5]) and then doesn't use that.
It
> doesn't
> > > > write
> > > > > > it
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > output file. Maybe what I should do is find
the
> first
> > > bad
> > > > > > > > > observation
> > > > > > > > > > > > (time, lat, lon, value), then, match it
exactly to
> the
> > > line
> > > > > in
> > > > > > > the
> > > > > > > > > mpr
> > > > > > > > > > > > file, and replace the observation.
> > > > > > > > > > > >
> > > > > > > > > > > > So, if my observations are in dataframes, and
the mpr
> > > file
> > > > > has
> > > > > > > been
> > > > > > > > > > read
> > > > > > > > > > > in
> > > > > > > > > > > > as a dataframe, then, I could do a
df.query('Lat = ??
> > and
> > > > Lon
> > > > > > > =??),
> > > > > > > > > or
> > > > > > > > > > > > however the syntax is...no wait, I need an if
> statement
> > > to
> > > > > > > compare
> > > > > > > > > > > > first...I'll work on those detail. Then, once
I get
> > the
> > > > > OBS_QC
> > > > > > > > value
> > > > > > > > > > > > changed, then what? How do you get the other
stat
> > files
> > > > from
> > > > > > > this
> > > > > > > > > > point?
> > > > > > > > > > > >
> > > > > > > > > > > > Roz
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Aug 13, 2018 at 5:32 PM, John Halley
Gotway
> via
> > > RT
> > > > <
> > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Roz,
> > > > > > > > > > > > >
> > > > > > > > > > > > > That's correct, there is no tool in MET for
> comparing
> > > one
> > > > > set
> > > > > > > of
> > > > > > > > > > point
> > > > > > > > > > > > > observations to another set of point
observations.
> > > > > > Presumably
> > > > > > > > you
> > > > > > > > > > > could
> > > > > > > > > > > > > write point observations to a grid and then
> > reconfigure
> > > > and
> > > > > > > rerun
> > > > > > > > > > > > > Point-Stat, but that'd be pretty difficult,
time
> > > > consuming,
> > > > > > and
> > > > > > > > > > fraught
> > > > > > > > > > > > > with problems.
> > > > > > > > > > > > >
> > > > > > > > > > > > > You currently have many, many MPR lines and
the
> issue
> > > is
> > > > > the
> > > > > > > the
> > > > > > > > > > OBS_QC
> > > > > > > > > > > > > column is empty. If that OBS_QC column were
*not*
> > > empty
> > > > > then
> > > > > > > > it'd
> > > > > > > > > be
> > > > > > > > > > > > > pretty easy to process this data... correct?
One
> > > option
> > > > > > would
> > > > > > > be
> > > > > > > > > > > > writing a
> > > > > > > > > > > > > script to "patch" that column. For each MPR
line,
> do
> > > > some
> > > > > > > > > processing
> > > > > > > > > > > to
> > > > > > > > > > > > > figure out what that OBS_QC column should be
and
> then
> > > > > update
> > > > > > > it's
> > > > > > > > > > > value.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Then the question is this... is there a way
of
> > getting
> > > > > > quality
> > > > > > > > > > control
> > > > > > > > > > > > > values from the MGDRlite files?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm sorry, I hadn't had a chance to look at
that
> data
> > > > file
> > > > > > yet.
> > > > > > > > > But
> > > > > > > > > > I
> > > > > > > > > > > > just
> > > > > > > > > > > > > checked on the ftp site and don't see a
maccracken
> > > > > > > sub-directory
> > > > > > > > > > there:
> > > > > > > > > > > > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/
> > > > > > > > > > > > >
> > > > > > > > > > > > > Could you please repost?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > John
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Aug 13, 2018 at 10:57 AM Rosalyn
> MacCracken -
> > > > NOAA
> > > > > > > > > Affiliate
> > > > > > > > > > > via
> > > > > > > > > > > > RT
> > > > > > > > > > > > > <met_help at ucar.edu> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > Ticket/Display.html?id=86480
> > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So, you can't compare two point files,
even if
> both
> > > of
> > > > > them
> > > > > > > are
> > > > > > > > > > > NetCDF
> > > > > > > > > > > > > > format? Is there a way to write point
> observations
> > > out
> > > > > to
> > > > > > a
> > > > > > > > > grid?
> > > > > > > > > > > In
> > > > > > > > > > > > > > other words, take the point GFS
observations, and
> > > write
> > > > > > them
> > > > > > > > to a
> > > > > > > > > > > > global
> > > > > > > > > > > > > > grid (using some other tool, like wgrib or
> > something
> > > > > else),
> > > > > > > and
> > > > > > > > > > fill
> > > > > > > > > > > in
> > > > > > > > > > > > > > missing values with NaN, or something?
Then, use
> > > that
> > > > > > > gridded
> > > > > > > > > file
> > > > > > > > > > > to
> > > > > > > > > > > > > > compare with the ASCAT NetCDF file?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I don't have anything in the OBS_QC column
> because
> > > > when I
> > > > > > > wrote
> > > > > > > > > it
> > > > > > > > > > > into
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > ASCII file, I didn't have any flags from
my
> > MGDRlite
> > > > > files.
> > > > > > > > This
> > > > > > > > > > is
> > > > > > > > > > > > part
> > > > > > > > > > > > > > of the problem...
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Now, your answer for the STAT-Analysis, is
that
> > what
> > > I
> > > > > did
> > > > > > > > wrong
> > > > > > > > > in
> > > > > > > > > > > my
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > of stat-analysis from the previous email?
Or,
> did
> > > you
> > > > > not
> > > > > > > get
> > > > > > > > a
> > > > > > > > > > > chance
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > look at my *stat file that I uploaded to
the ftp
> > > site?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Roz
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Aug 13, 2018 at 3:42 PM, John
Halley
> Gotway
> > > via
> > > > > RT
> > > > > > <
> > > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Roz,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Point-Stat takes as input a gridded NWP
> forecast
> > > file
> > > > > and
> > > > > > > the
> > > > > > > > > > > NetCDF
> > > > > > > > > > > > > > output
> > > > > > > > > > > > > > > of a point pre-processing tool (like
ascii2nc
> or
> > > > > pb2nc).
> > > > > > > > Based
> > > > > > > > > > on
> > > > > > > > > > > > your
> > > > > > > > > > > > > > > description, it doesn't sound like you
have
> > gridded
> > > > NWP
> > > > > > > > > forecast
> > > > > > > > > > > > files
> > > > > > > > > > > > > > > available. So no, I don't think that
approach
> > > would
> > > > > > work.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The MPR output line type does include a
column
> > > named
> > > > > > > "OBS_QC"
> > > > > > > > > > which
> > > > > > > > > > > > is
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > string containing the observation
quality
> control
> > > > > value.
> > > > > > > I'd
> > > > > > > > > > > suggest
> > > > > > > > > > > > > > > checking the contents of that column in
your
> MPR
> > > > output
> > > > > > > lines
> > > > > > > > > to
> > > > > > > > > > > see
> > > > > > > > > > > > if
> > > > > > > > > > > > > > > those values could be filtered to do the
> quality
> > > > > control
> > > > > > > > > > filtering
> > > > > > > > > > > > > you're
> > > > > > > > > > > > > > > after. Check to see if the values in
that
> column
> > > are
> > > > > > > strings
> > > > > > > > > or
> > > > > > > > > > > > > numbers
> > > > > > > > > > > > > > > (in PREPBUFR they're integers... in
MADIS
> they're
> > > > > > strings).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In STAT-Analysis, you can use the
> > "-column_thresh"
> > > > > option
> > > > > > > to
> > > > > > > > > > > > threshold
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > numeric column or you can use the "-
column_str"
> > > > option
> > > > > to
> > > > > > > > list
> > > > > > > > > > > > strings
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > retained. For example:
> > > > > > > > > > > > > > > -column_thresh OBS_QC 'ge0&&le3' #
this
> keeps
> > > all
> > > > > > > values
> > > > > > > > > > > between
> > > > > > > > > > > > 0
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > 3
> > > > > > > > > > > > > > > -column_str OBS_QC aa,ab,ac #
this
> > keeps
> > > > > > strings
> > > > > > > > > "aa",
> > > > > > > > > > > > "ab",
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > "ac"
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hope that helps.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > John
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Aug 13, 2018 at 8:34 AM Rosalyn
> > MacCracken
> > > -
> > > > > NOAA
> > > > > > > > > > Affiliate
> > > > > > > > > > > > via
> > > > > > > > > > > > > > RT
> > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > > > Ticket/Display.html?id=86480
> > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi John,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have another question related to my
> > > reprocessing
> > > > of
> > > > > > my
> > > > > > > > > data.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Ok, so, I realized that when I
switched ASCAT
> > > data
> > > > > > > sources
> > > > > > > > > back
> > > > > > > > > > > in
> > > > > > > > > > > > > > March,
> > > > > > > > > > > > > > > > from prepbufr to MGDRlite files, I
should
> have
> > > been
> > > > > > using
> > > > > > > > > > quality
> > > > > > > > > > > > > flags
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > filter out my data. Because I wasn't
> checking
> > > for
> > > > > the
> > > > > > > > > quality
> > > > > > > > > > > > flags,
> > > > > > > > > > > > > > my
> > > > > > > > > > > > > > > > statistics were not so good, since it
was
> > letting
> > > > in
> > > > > > bad
> > > > > > > > > data,
> > > > > > > > > > > and
> > > > > > > > > > > > > > stats
> > > > > > > > > > > > > > > > were being calculated from this bad
data.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > So, I need to find a way to correct
this,
> but,
> > > the
> > > > > old
> > > > > > > > story
> > > > > > > > > > is I
> > > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > have the original GFS data, and it's
too
> > massive
> > > to
> > > > > > grab
> > > > > > > > from
> > > > > > > > > > > HPSS.
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > have the original ASCAT files, which I
can
> run
> > > > > through
> > > > > > my
> > > > > > > > > > program
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > filter
> > > > > > > > > > > > > > > > out the bad observations, as well as
matched
> > > points
> > > > > > from
> > > > > > > > the
> > > > > > > > > > *mpr
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > *stat
> > > > > > > > > > > > > > > > files.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Originally, I was just going to filter
out
> > using
> > > > that
> > > > > > > > > > > > -column_thresh
> > > > > > > > > > > > > > > > command, and filter out over land.
But, I
> had
> > > this
> > > > > > idea
> > > > > > > > > that I
> > > > > > > > > > > > > wanted
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > run by you, to see if this makes sense
to do.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have the time, lat, lon and GFS
observation
> > and
> > > > the
> > > > > > > > > > > corresponding
> > > > > > > > > > > > > > ASCAT
> > > > > > > > > > > > > > > > observation point. Can I do this:
> > > > > > > > > > > > > > > > 1) read the time, lat, lon and GFS
> observation
> > > out
> > > > > of
> > > > > > > the
> > > > > > > > > mpr
> > > > > > > > > > > > file,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > make a new input file with just those
> variables
> > > in
> > > > > the
> > > > > > > > ASCII
> > > > > > > > > > > format
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > MET can use
> > > > > > > > > > > > > > > > 2) reprocess my ASCAT data and write
it into
> > an
> > > > > ASCII
> > > > > > > file
> > > > > > > > > in
> > > > > > > > > > > MET
> > > > > > > > > > > > > > format
> > > > > > > > > > > > > > > > 3) run ASCII2NC on these files, to
create
> the
> > > two
> > > > > > input
> > > > > > > > > files
> > > > > > > > > > > for
> > > > > > > > > > > > > MET,
> > > > > > > > > > > > > > > > then, run point_stat on the netcdf
files to
> > > > > regenerate
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > > > > corrected
> > > > > > > > > > > > > > > > stat files
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Does that make sense to do? I know it
could
> be
> > > > alot
> > > > > of
> > > > > > > > work,
> > > > > > > > > > > but,
> > > > > > > > > > > > > if I
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > get corrected statistics, it might be
worth
> it.
> > > > What
> > > > > > do
> > > > > > > > you
> > > > > > > > > > > think?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Roz
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Aug 10, 2018 at 7:24 PM,
Rosalyn
> > > > MacCracken -
> > > > > > > NOAA
> > > > > > > > > > > > Affiliate
> > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi John,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I double checked all those things
you
> listed,
> > > and
> > > > > > they
> > > > > > > > are
> > > > > > > > > > all
> > > > > > > > > > > > > > correct.
> > > > > > > > > > > > > > > > > So, I've put a file in the
/maccracken
> > > directory
> > > > on
> > > > > > > your
> > > > > > > > > ftp
> > > > > > > > > > > > site.
> > > > > > > > > > > > > > Let
> > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > know what you see.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for your help!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Roz
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Aug 10, 2018 at 1:41 PM,
Rosalyn
> > > > > MacCracken -
> > > > > > > > NOAA
> > > > > > > > > > > > > Affiliate
> > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> Hi John,
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> I'm sorry I didn't get back to you
sooner,
> > > but,
> > > > I
> > > > > > was
> > > > > > > > tied
> > > > > > > > > > up
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > >> other things. Let me double check
with
> all
> > of
> > > > > those
> > > > > > > > items
> > > > > > > > > > in
> > > > > > > > > > > my
> > > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >> make sure that I don't have some
sort of
> > > obvious
> > > > > > > error.
> > > > > > > > > > I'll
> > > > > > > > > > > be
> > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > >> touch after I check on this.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Roz
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> On Tue, Aug 7, 2018 at 6:07 PM,
John
> Halley
> > > > Gotway
> > > > > > via
> > > > > > > > RT
> > > > > > > > > <
> > > > > > > > > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>> Hi Roz,
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> I see you're having trouble
getting with
> > > > > > > STAT-Analysis,
> > > > > > > > > > > getting
> > > > > > > > > > > > > MPR
> > > > > > > > > > > > > > > > lines
> > > > > > > > > > > > > > > > >>> to be included in the column
thresholds
> > > you've
> > > > > > > defined.
> > > > > > > > > I
> > > > > > > > > > > > don't
> > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > >>> anything obviously wrong with what
you've
> > > > > defined.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> The warning message indicates that
the
> > > filters
> > > > > you
> > > > > > > > > defined
> > > > > > > > > > > have
> > > > > > > > > > > > > > > > discarded
> > > > > > > > > > > > > > > > >>> all of the input MPR lines.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> We'd just need to go through your
> filtering
> > > > > options
> > > > > > > > > > > one-by-one
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > find
> > > > > > > > > > > > > > > > >>> out
> > > > > > > > > > > > > > > > >>> why.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> - Are you passing STAT-Analysis
files
> which
> > > > > contain
> > > > > > > MPR
> > > > > > > > > > > lines?
> > > > > > > > > > > > > > > > >>> - Does the FCST_VAR column say
"WIND"?
> > > > > > > > > > > > > > > > >>> - Does the FCST_LEV column say
"Z10"?
> > > > > > > > > > > > > > > > >>> - Does the OBTYPE column say
"ASCAT"?
> > > > > > > > > > > > > > > > >>> - Does the OBS column have values
> 1.0?
> > > > > > > > > > > > > > > > >>> - What are the ranges of the
OBS_LAT and
> > > > OBS_LON
> > > > > > > > columns?
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> If you'd like to send me a sample
.stat
> > > file, I
> > > > > > could
> > > > > > > > > > > probably
> > > > > > > > > > > > > > track
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > >>> down.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> Thanks,
> > > > > > > > > > > > > > > > >>> John
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> On Mon, Aug 6, 2018 at 7:43 AM
Rosalyn
> > > > > MacCracken -
> > > > > > > > NOAA
> > > > > > > > > > > > > Affiliate
> > > > > > > > > > > > > > > via
> > > > > > > > > > > > > > > > >>> RT <
> > > > > > > > > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > Mon Aug 06 07:43:46 2018:
Request 86480
> > was
> > > > > acted
> > > > > > > > upon.
> > > > > > > > > > > > > > > > >>> > Transaction: Ticket created by
> > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > > >>> > Queue: met_help
> > > > > > > > > > > > > > > > >>> > Subject: Errors using
> -column_thresh
> > > > > > > > > > > > > > > > >>> > Owner: Nobody
> > > > > > > > > > > > > > > > >>> > Requestors:
> > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > > >>> > Status: new
> > > > > > > > > > > > > > > > >>> > Ticket <URL:
> > > > > > > > > > > > > > > >
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86480
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > Hi,
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > I'm trying to use -column_thresh
with
> > > > > > stat_analysis
> > > > > > > > to
> > > > > > > > > > mask
> > > > > > > > > > > > > out a
> > > > > > > > > > > > > > > > >>> lat/lon
> > > > > > > > > > > > > > > > >>> > box. I've added this command in
my
> > > > > StatAnalysis
> > > > > > > > config
> > > > > > > > > > > file:
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > "-job aggregate_stat
-out_line_type CTC
> > > > > > > > -column_thresh
> > > > > > > > > > OBS
> > > > > > > > > > > > > gt1.0
> > > > > > > > > > > > > > > > >>> > -out_thresh ge5.5689 \
> > > > > > > > > > > > > > > > >>> > -column_thresh OBS_LAT
> > > 'ge15.0&&le70.0' \
> > > > > > > > > > > > > > > > >>> > -column_thresh OBS_LON
> > > > 'ge-100.0&&le-10.0'
> > > > > \
> > > > > > > > > > > > > > > > >>> > -by
FCST_VALID_BEG,FCST_LEAD,VX_
> MASK
> > \
> > > > > > > > > > > > > > > > >>> > -out_stat
> > > /opc_test/home/opc_test/data/m
> > > > > > > > > > > > > > > > >>> et_verif/regen_mar2018_data/
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > new_out_stat_test2/agg_stat_MPR_to_CTC_ge5.5689.stat",
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > But, when it runs, I get these
errors:
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > Output:
> > > > > > > > > > > > > > > > >>> > DEBUG 2: Processing Job 2: -job
> > > > aggregate_stat
> > > > > > > > > -fcst_var
> > > > > > > > > > > WIND
> > > > > > > > > > > > > > > > >>> -fcst_lev Z10
> > > > > > > > > > > > > > > > >>> > -obtype ASCAT -line_type MPR
> > -column_thresh
> > > > OBS
> > > > > > > >1.0
> > > > > > > > > > > > > > -column_thresh
> > > > > > > > > > > > > > > > >>> OBS_LAT
> > > > > > > > > > > > > > > > >>> > >=15.0&&<=70.0 -column_thresh
OBS_LON
> > > > > > > > >=-100.0&&<=-10.0
> > > > > > > > > > -by
> > > > > > > > > > > > > > > > >>> FCST_VALID_BEG
> > > > > > > > > > > > > > > > >>> > -by FCST_LEAD -by VX_MASK
-out_stat
> > > > > > > > > > > > > /opc_test/home/opc_test/data/
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > > > MPR
> > > > > > > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > > > > > > >>> > -out_line_type CTC -out_thresh
>=5.5689
> > > > > > > > > > > > > > > > >>> > DEBUG 1: Creating STAT output
file
> > > > > > > > > > > > > "/opc_test/home/opc_test/data/
> > > > > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > > > > > > > > >>> > MPR_to_CTC_ge5.5689.stat"
> > > > > > > > > > > > > > > > >>> > GSL_RNG_TYPE=mt19937
> > > > > > > > > > > > > > > > >>> >
GSL_RNG_SEED=18446744073401622619
> > > > > > > > > > > > > > > > >>> > DEBUG 2: Computing output for 0
> case(s).
> > > > > > > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > > > > > > >>> > WARNING: do_job_aggr_stat() ->
no
> > matching
> > > > STAT
> > > > > > > lines
> > > > > > > > > > found
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > job:
> > > > > > > > > > > > > > > > >>> -job
> > > > > > > > > > > > > > > > >>> > aggregate_stat -fcst_var WIND
-fcst_lev
> > Z10
> > > > > > -obtype
> > > > > > > > > ASCAT
> > > > > > > > > > > > > > > -line_type
> > > > > > > > > > > > > > > > >>> MPR
> > > > > > > > > > > > > > > > >>> > -column_thresh OBS >1.0
-column_thresh
> > > > OBS_LAT
> > > > > > > > > > > >=15.0&&<=70.0
> > > > > > > > > > > > > > > > >>> > -column_thresh OBS_LON
> >=-100.0&&<=-10.0
> > > -by
> > > > > > > > > > FCST_VALID_BEG
> > > > > > > > > > > > -by
> > > > > > > > > > > > > > > > >>> FCST_LEAD
> > > > > > > > > > > > > > > > >>> > -by VX_MASK -out_stat
> > > > > > /opc_test/home/opc_test/data/
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > met_verif/regen_mar2018_data/
> > > > > > > > > new_out_stat_test2/agg_stat_
> > > > > > > > > > > MPR
> > > > > > > > > > > > > > > > >>> _to_CTC_ge5.5689.stat
> > > > > > > > > > > > > > > > >>> > -out_line_type CTC -out_thresh
>=5.5689
> > > > > > > > > > > > > > > > >>> > WARNING:
> > > > > > > > > > > > > > > > >>> > DEBUG 2: Job 2 used 0 out of
69137 STAT
> > > > lines.
> > > > > > > > > > > > > > > > >>> > DEBUG 2:
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > Am I not using the command
properly?
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > Thanks!
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > Roz
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > --
> > > > > > > > > > > > > > > > >>> > Rosalyn MacCracken
> > > > > > > > > > > > > > > > >>> > Support Scientist
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > Ocean Applications Branch
> > > > > > > > > > > > > > > > >>> > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > > > >>> > NCWCP
> > > > > > > > > > > > > > > > >>> > 5830 University Research Ct
> > > > > > > > > > > > > > > > >>> > College Park, MD 20740-3818
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > (p) 301-683-1551
> > > > > > > > > > > > > > > > >>> > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> --
> > > > > > > > > > > > > > > > >> Rosalyn MacCracken
> > > > > > > > > > > > > > > > >> Support Scientist
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Ocean Applications Branch
> > > > > > > > > > > > > > > > >> NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > > > >> NCWCP
> > > > > > > > > > > > > > > > >> 5830 University Research Ct
> > > > > > > > > > > > > > > > >> College Park, MD 20740-3818
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> (p) 301-683-1551
> > > > > > > > > > > > > > > > >> rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > > > Support Scientist
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > > > NCWCP
> > > > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > > > Support Scientist
> > > > > > > > > > > >
> > > > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > > > NCWCP
> > > > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > > > >
> > > > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Rosalyn MacCracken
> > > > > > > > > > Support Scientist
> > > > > > > > > >
> > > > > > > > > > Ocean Applications Branch
> > > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > > NCWCP
> > > > > > > > > > 5830 University Research Ct
> > > > > > > > > > College Park, MD 20740-3818
> > > > > > > > > >
> > > > > > > > > > (p) 301-683-1551
> > > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rosalyn MacCracken
> > > > > > > > Support Scientist
> > > > > > > >
> > > > > > > > Ocean Applications Branch
> > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > NCWCP
> > > > > > > > 5830 University Research Ct
> > > > > > > > College Park, MD 20740-3818
> > > > > > > >
> > > > > > > > (p) 301-683-1551
> > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applications Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list