[Met_help] [rt.rap.ucar.edu #92027] History for python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
George McCabe via RT
met_help at ucar.edu
Wed Oct 2 08:32:33 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi George,
I found the python v3.6.3 module on Dell. Is the
bugfix_v2.2_pcp_combine branch in the METplus github the beta version
you're referring to?
In re LEAD_SEQ and INIT_SEQ: for my purposes INIT_SEQ combined with
LEAD_SEQ_MIN/LEAD_SEQ_MAX and along with VALID_BEG and VALID_END,
VALID_INCREMENT provide the right amount of specification on finding the
valid time/forecast hour pairs. At the moment I can't think of
instances when both LEAD_SEQ and INIT_SEQ are needed (yeah, that's what
I was asking Tuesday, but after yesterday's discussion/test run I've
come around to think the current set up works fine, once a user
understands the rationale for the set up).
Thanks,
Ying
https://github.com/NCAR/METplus/tree/bugfix_v2.2_pcp_combine
On 9/11/19 10:43 PM, George McCabe wrote:
> Glad to help! Thanks for letting me know LEAD_SEQ_MIN is actually
> required.
>
> It seems like it may be useful to allow both LEAD_SEQ and INIT_SEQ
> instead of requiring you to set min/max and not set LEAD_SEQ. LEAD_SEQ
> would define the leads you want to use and INIT_SEQ would weed out the
> leads that shouldn't be included based on the current run time.
>
> Please don't hesitate to mention situations that comes up in your
> workflow that prohibits you from doing what you need to do via
> METplus. We need real use cases to help us develop the software to be
> as user-friendly as possible and you feedback greatly helps us with that.
>
> On another note, after speaking with Tara it sounds like you should
> have access to python 3.6.3 (or higher) at EMC. If you could find out
> where it is located, I could provide you with a beta version of
> METplus that would have many improvements from the version you are
> using, including better error messages to determine why files couldn't
> be found.
>
> On Wed, Sep 11, 2019 at 4:07 PM Ying Lin <ying.lin at noaa.gov
> <mailto:ying.lin at noaa.gov>> wrote:
>
> Hi George, thank you again for sitting down with me today to resolve
> various issues, and for taking the time to go over the concept of
> LEAD_SEQ vs. INIT_SEQ. I replaced the LEAD_SEQ in my conf with
> INIT_SEQ
> and LEAD_SEQ_MIN/LEAD_SEQ_MAX and it worked fine, eliminated a lot of
> "ERROR" messages in the log from trying all possible fcst lead vs.
> valid
> time combinations.
>
> Initially I only defined LEAD_SEQ_MAX and the scripts complained
> about
> something like "initial time not defined". Setting LEAD_SEQ_MIN
> fixed
> that.
>
> Ying
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov <mailto:Ying.Lin at noaa.gov>
>
>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: George McCabe
Time: Mon Sep 16 12:06:57 2019
Hi Ying,
I determined the cause of your error. It is a combination of
mis-configuration and a bug. You have set:
OBS_WINDOW_BEGIN = -2700
OBS_WINDOW_END = 2700
to specify the range of data to be allowed to be read in. However,
thse
values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if those are
not
set. The _FILE_WINDOW_ variables determine the window of times to find
the
files based on their filename in case the filename times don't exactly
line
up with the valid time. You are supposed to be able to set
OBS_FILE_WINDOW_BEGIN = 0
OBS_FILE_WINDOW_END = 0
to override the values. HOWEVER, there is a typo in the code that
handles
these variables and PointStat actually uses OBS_WINDOW_[BEGIN/END]
instead.
I will fix this bug, but in the meantime, you should be able to set
OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
OBS_POINT_STAT_FILE_WINDOW_END = 0
and it should work as intended. Let me know if that does not work.
On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
>
> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
> Transaction: Given to mccabe (George McCabe) by johnhg
> Queue: met_help
> Subject: python/v3.6.3 and METplus beta version; Re: init_seq
works -
> thanks!
> Owner: mccabe
> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: Ying Lin
Time: Mon Sep 16 15:36:27 2019
Hi George,
I made a number of runs, here are three(to be brief) at
20190916195402 (run1), 20190916203610 (run2) and 20190916210024
(run3):
Various configs/logs in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
I've also placed the pcprtma.2019091512_A24h Run3 log complained
about:
In run1, I changed
OBS_WINDOW_BEGIN = -2700
OBS_WINDOW_END = 2700
to
OBS_WINDOW_BEGIN = 0
OBS_WINDOW_END = 0
w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts found the
current gauge .nc file, but while running PointStat and reading in
/parm/PointStatConfig_gauges it complains of
unexpected "$" in line 132 of config file
Below is the offending line:
//
// Verification masking regions
//
mask = {
grid = ${POINT_STAT_GRID};
poly = ${POINT_STAT_POLY}; <=== line 132
sid = [];
llpnt = [];
}
Log file shows that the "poly" files for the mask was found, not sure
why line 132 was objectionable.
In run2, in pcprtma_24h.conf I have
OBS_WINDOW_BEGIN = -2700
OBS_WINDOW_END = 2700
OBS_FILE_WINDOW_BEGIN = 0
OBS_FILE_WINDOW_END = 0
and I defined POINT_STAT_POLY in PointStatConfig_gauges instead of in
pcprtma_24h.conf to get past the "error in line 132 in above run1.
Now I'm getting the same old error as last week:
09/16 20:36:27.051 metplus.PointStat (point_stat_wrapper.py:194)
ERROR:
Could not find observation file in
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc matching
template
good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
Run 3: I went back to setting
OBS_WINDOW_BEGIN = 0
OBS_WINDOW_END = 0
w/o setting the OBS_FILE_WINDOW_BEGIN[END]
as in Run1, and left the mask setting as in Run2. This time the
script
can find the gauge file, no complains about the mask poly files, but
does not seem to be able to find the gridded 24h precip:
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Reading data for APCP_24(*,*).
ERROR :
ERROR : grd_file_type() -> file does not exist "NONE"
ERROR :
So in brief:
* PointStat cannot read the 24h accumulation in
pcprtma.2019091512_A24h
* Script can only find the gauges file when
OBS_WINDOW_BEGIN[_END]are
set to zero. This is undesirable since it narrows the valid time
for each 24h gauge too much
Thanks
Ying
On 9/16/19 2:06 PM, George McCabe via RT wrote:
> Hi Ying,
>
> I determined the cause of your error. It is a combination of
> mis-configuration and a bug. You have set:
>
> OBS_WINDOW_BEGIN = -2700
> OBS_WINDOW_END = 2700
>
> to specify the range of data to be allowed to be read in. However,
thse
> values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if those
are not
> set. The _FILE_WINDOW_ variables determine the window of times to
find the
> files based on their filename in case the filename times don't
exactly line
> up with the valid time. You are supposed to be able to set
>
> OBS_FILE_WINDOW_BEGIN = 0
> OBS_FILE_WINDOW_END = 0
>
> to override the values. HOWEVER, there is a typo in the code that
handles
> these variables and PointStat actually uses OBS_WINDOW_[BEGIN/END]
instead.
> I will fix this bug, but in the meantime, you should be able to set
>
> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> OBS_POINT_STAT_FILE_WINDOW_END = 0
>
> and it should work as intended. Let me know if that does not work.
>
> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
>> Transaction: Given to mccabe (George McCabe) by johnhg
>> Queue: met_help
>> Subject: python/v3.6.3 and METplus beta version; Re: init_seq
works -
>> thanks!
>> Owner: mccabe
>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>
>>
>> This transaction appears to have no content
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: George McCabe
Time: Mon Sep 16 15:41:50 2019
Hi Ying,
You should actually set:
OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
OBS_POINT_STAT_FILE_WINDOW_END = 0
instead of
OBS_FILE_WINDOW_BEGIN and END
You SHOULD be able to set the latter, but there is a bug that prevents
that
from working. I think you will need OBS_WINDOW_BEGIN and END to be
-2700 to
2700 so that PointStat can find the correct data inside the files.
Does
that make sense?
On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>
> Hi George,
>
> I made a number of runs, here are three(to be brief) at
> 20190916195402 (run1), 20190916203610 (run2) and 20190916210024
(run3):
> Various configs/logs in
> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
> I've also placed the pcprtma.2019091512_A24h Run3 log complained
about:
>
>
> In run1, I changed
> OBS_WINDOW_BEGIN = -2700
> OBS_WINDOW_END = 2700
> to
> OBS_WINDOW_BEGIN = 0
> OBS_WINDOW_END = 0
>
> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts found the
> current gauge .nc file, but while running PointStat and reading in
> /parm/PointStatConfig_gauges it complains of
> unexpected "$" in line 132 of config file
>
> Below is the offending line:
> //
> // Verification masking regions
> //
> mask = {
> grid = ${POINT_STAT_GRID};
> poly = ${POINT_STAT_POLY}; <=== line 132
> sid = [];
> llpnt = [];
> }
>
> Log file shows that the "poly" files for the mask was found, not
sure
> why line 132 was objectionable.
>
> In run2, in pcprtma_24h.conf I have
>
> OBS_WINDOW_BEGIN = -2700
> OBS_WINDOW_END = 2700
>
> OBS_FILE_WINDOW_BEGIN = 0
> OBS_FILE_WINDOW_END = 0
>
> and I defined POINT_STAT_POLY in PointStatConfig_gauges instead of
in
> pcprtma_24h.conf to get past the "error in line 132 in above run1.
>
> Now I'm getting the same old error as last week:
> 09/16 20:36:27.051 metplus.PointStat (point_stat_wrapper.py:194)
ERROR:
> Could not find observation file in
> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc matching
template
> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
>
> Run 3: I went back to setting
> OBS_WINDOW_BEGIN = 0
> OBS_WINDOW_END = 0
> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
> as in Run1, and left the mask setting as in Run2. This time the
script
> can find the gauge file, no complains about the mask poly files, but
> does not seem to be able to find the gridded 24h precip:
> DEBUG 2:
> DEBUG 2:
>
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for APCP_24(*,*).
> ERROR :
> ERROR : grd_file_type() -> file does not exist "NONE"
> ERROR :
>
> So in brief:
>
> * PointStat cannot read the 24h accumulation in
pcprtma.2019091512_A24h
> * Script can only find the gauges file when
OBS_WINDOW_BEGIN[_END]are
> set to zero. This is undesirable since it narrows the valid
time
> for each 24h gauge too much
>
> Thanks
> Ying
>
> On 9/16/19 2:06 PM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > I determined the cause of your error. It is a combination of
> > mis-configuration and a bug. You have set:
> >
> > OBS_WINDOW_BEGIN = -2700
> > OBS_WINDOW_END = 2700
> >
> > to specify the range of data to be allowed to be read in. However,
thse
> > values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if those
are not
> > set. The _FILE_WINDOW_ variables determine the window of times to
find
> the
> > files based on their filename in case the filename times don't
exactly
> line
> > up with the valid time. You are supposed to be able to set
> >
> > OBS_FILE_WINDOW_BEGIN = 0
> > OBS_FILE_WINDOW_END = 0
> >
> > to override the values. HOWEVER, there is a typo in the code that
handles
> > these variables and PointStat actually uses OBS_WINDOW_[BEGIN/END]
> instead.
> > I will fix this bug, but in the meantime, you should be able to
set
> >
> > OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> > OBS_POINT_STAT_FILE_WINDOW_END = 0
> >
> > and it should work as intended. Let me know if that does not work.
> >
> > On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
> >> Transaction: Given to mccabe (George McCabe) by johnhg
> >> Queue: met_help
> >> Subject: python/v3.6.3 and METplus beta version; Re:
init_seq
> works -
> >> thanks!
> >> Owner: mccabe
> >> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
> >
> >>
> >>
> >> This transaction appears to have no content
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: George McCabe
Time: Mon Sep 16 15:45:53 2019
I will take a look at the POLY issue. My guess is you need to set the
poly
variable using {DMASK} instead of ${DMASK} since it is referencing the
METplus config variable. I will let you know what I find.
On Mon, Sep 16, 2019 at 3:41 PM George McCabe <mccabe at ucar.edu> wrote:
> Hi Ying,
>
> You should actually set:
>
> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> OBS_POINT_STAT_FILE_WINDOW_END = 0
> instead of
> OBS_FILE_WINDOW_BEGIN and END
>
> You SHOULD be able to set the latter, but there is a bug that
prevents
> that from working. I think you will need OBS_WINDOW_BEGIN and END to
be
> -2700 to 2700 so that PointStat can find the correct data inside the
files.
> Does that make sense?
>
> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>
>> Hi George,
>>
>> I made a number of runs, here are three(to be brief) at
>> 20190916195402 (run1), 20190916203610 (run2) and 20190916210024
(run3):
>> Various configs/logs in
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
>> I've also placed the pcprtma.2019091512_A24h Run3 log complained
about:
>>
>>
>> In run1, I changed
>> OBS_WINDOW_BEGIN = -2700
>> OBS_WINDOW_END = 2700
>> to
>> OBS_WINDOW_BEGIN = 0
>> OBS_WINDOW_END = 0
>>
>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts found the
>> current gauge .nc file, but while running PointStat and reading in
>> /parm/PointStatConfig_gauges it complains of
>> unexpected "$" in line 132 of config file
>>
>> Below is the offending line:
>> //
>> // Verification masking regions
>> //
>> mask = {
>> grid = ${POINT_STAT_GRID};
>> poly = ${POINT_STAT_POLY}; <=== line 132
>> sid = [];
>> llpnt = [];
>> }
>>
>> Log file shows that the "poly" files for the mask was found, not
sure
>> why line 132 was objectionable.
>>
>> In run2, in pcprtma_24h.conf I have
>>
>> OBS_WINDOW_BEGIN = -2700
>> OBS_WINDOW_END = 2700
>>
>> OBS_FILE_WINDOW_BEGIN = 0
>> OBS_FILE_WINDOW_END = 0
>>
>> and I defined POINT_STAT_POLY in PointStatConfig_gauges instead of
in
>> pcprtma_24h.conf to get past the "error in line 132 in above run1.
>>
>> Now I'm getting the same old error as last week:
>> 09/16 20:36:27.051 metplus.PointStat (point_stat_wrapper.py:194)
ERROR:
>> Could not find observation file in
>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc matching
template
>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
>>
>> Run 3: I went back to setting
>> OBS_WINDOW_BEGIN = 0
>> OBS_WINDOW_END = 0
>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
>> as in Run1, and left the mask setting as in Run2. This time the
script
>> can find the gauge file, no complains about the mask poly files,
but
>> does not seem to be able to find the gridded 24h precip:
>> DEBUG 2:
>> DEBUG 2:
>>
>>
--------------------------------------------------------------------------------
>> DEBUG 2:
>> DEBUG 2: Reading data for APCP_24(*,*).
>> ERROR :
>> ERROR : grd_file_type() -> file does not exist "NONE"
>> ERROR :
>>
>> So in brief:
>>
>> * PointStat cannot read the 24h accumulation in
pcprtma.2019091512_A24h
>> * Script can only find the gauges file when
OBS_WINDOW_BEGIN[_END]are
>> set to zero. This is undesirable since it narrows the valid
time
>> for each 24h gauge too much
>>
>> Thanks
>> Ying
>>
>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
>> > Hi Ying,
>> >
>> > I determined the cause of your error. It is a combination of
>> > mis-configuration and a bug. You have set:
>> >
>> > OBS_WINDOW_BEGIN = -2700
>> > OBS_WINDOW_END = 2700
>> >
>> > to specify the range of data to be allowed to be read in.
However, thse
>> > values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if those
are
>> not
>> > set. The _FILE_WINDOW_ variables determine the window of times to
find
>> the
>> > files based on their filename in case the filename times don't
exactly
>> line
>> > up with the valid time. You are supposed to be able to set
>> >
>> > OBS_FILE_WINDOW_BEGIN = 0
>> > OBS_FILE_WINDOW_END = 0
>> >
>> > to override the values. HOWEVER, there is a typo in the code that
>> handles
>> > these variables and PointStat actually uses
OBS_WINDOW_[BEGIN/END]
>> instead.
>> > I will fix this bug, but in the meantime, you should be able to
set
>> >
>> > OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>> > OBS_POINT_STAT_FILE_WINDOW_END = 0
>> >
>> > and it should work as intended. Let me know if that does not
work.
>> >
>> > On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
>> >> Transaction: Given to mccabe (George McCabe) by johnhg
>> >> Queue: met_help
>> >> Subject: python/v3.6.3 and METplus beta version; Re:
init_seq
>> works -
>> >> thanks!
>> >> Owner: mccabe
>> >> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
>> >> Status: new
>> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>> >
>> >>
>> >>
>> >> This transaction appears to have no content
>> >>
>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #92027] python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: Ying Lin
Time: Mon Sep 16 18:12:51 2019
Hi George,
Looks like
OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
OBS_POINT_STAT_FILE_WINDOW_END = 0
fixed the issue with finding the gauge .nc file. Great!
Setting the
Now the issue remains is the PointStat err msg:
DEBUG 1: Forecast File:
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
DEBUG 1: Observation File:
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/good-usa-
dlyprcp-20190915.nc
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Reading data for APCP_24(*,*).
ERROR :
ERROR : grd_file_type() -> file does not exist "NONE"
ERROR :
I placed the latest configs, log etc in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
(older ones now under old/).
Thanks again -
Ying
On 9/16/19 5:45 PM, George McCabe via RT wrote:
> I will take a look at the POLY issue. My guess is you need to set
the poly
> variable using {DMASK} instead of ${DMASK} since it is referencing
the
> METplus config variable. I will let you know what I find.
>
> On Mon, Sep 16, 2019 at 3:41 PM George McCabe <mccabe at ucar.edu>
wrote:
>
>> Hi Ying,
>>
>> You should actually set:
>>
>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>> instead of
>> OBS_FILE_WINDOW_BEGIN and END
>>
>> You SHOULD be able to set the latter, but there is a bug that
prevents
>> that from working. I think you will need OBS_WINDOW_BEGIN and END
to be
>> -2700 to 2700 so that PointStat can find the correct data inside
the files.
>> Does that make sense?
>>
>> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>
>>> Hi George,
>>>
>>> I made a number of runs, here are three(to be brief) at
>>> 20190916195402 (run1), 20190916203610 (run2) and 20190916210024
(run3):
>>> Various configs/logs in
>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
>>> I've also placed the pcprtma.2019091512_A24h Run3 log complained
about:
>>>
>>>
>>> In run1, I changed
>>> OBS_WINDOW_BEGIN = -2700
>>> OBS_WINDOW_END = 2700
>>> to
>>> OBS_WINDOW_BEGIN = 0
>>> OBS_WINDOW_END = 0
>>>
>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts found the
>>> current gauge .nc file, but while running PointStat and reading in
>>> /parm/PointStatConfig_gauges it complains of
>>> unexpected "$" in line 132 of config file
>>>
>>> Below is the offending line:
>>> //
>>> // Verification masking regions
>>> //
>>> mask = {
>>> grid = ${POINT_STAT_GRID};
>>> poly = ${POINT_STAT_POLY}; <=== line 132
>>> sid = [];
>>> llpnt = [];
>>> }
>>>
>>> Log file shows that the "poly" files for the mask was found, not
sure
>>> why line 132 was objectionable.
>>>
>>> In run2, in pcprtma_24h.conf I have
>>>
>>> OBS_WINDOW_BEGIN = -2700
>>> OBS_WINDOW_END = 2700
>>>
>>> OBS_FILE_WINDOW_BEGIN = 0
>>> OBS_FILE_WINDOW_END = 0
>>>
>>> and I defined POINT_STAT_POLY in PointStatConfig_gauges instead of
in
>>> pcprtma_24h.conf to get past the "error in line 132 in above run1.
>>>
>>> Now I'm getting the same old error as last week:
>>> 09/16 20:36:27.051 metplus.PointStat (point_stat_wrapper.py:194)
ERROR:
>>> Could not find observation file in
>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc matching
template
>>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
>>>
>>> Run 3: I went back to setting
>>> OBS_WINDOW_BEGIN = 0
>>> OBS_WINDOW_END = 0
>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
>>> as in Run1, and left the mask setting as in Run2. This time the
script
>>> can find the gauge file, no complains about the mask poly files,
but
>>> does not seem to be able to find the gridded 24h precip:
>>> DEBUG 2:
>>> DEBUG 2:
>>>
>>>
--------------------------------------------------------------------------------
>>> DEBUG 2:
>>> DEBUG 2: Reading data for APCP_24(*,*).
>>> ERROR :
>>> ERROR : grd_file_type() -> file does not exist "NONE"
>>> ERROR :
>>>
>>> So in brief:
>>>
>>> * PointStat cannot read the 24h accumulation in
pcprtma.2019091512_A24h
>>> * Script can only find the gauges file when
OBS_WINDOW_BEGIN[_END]are
>>> set to zero. This is undesirable since it narrows the valid
time
>>> for each 24h gauge too much
>>>
>>> Thanks
>>> Ying
>>>
>>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
>>>> Hi Ying,
>>>>
>>>> I determined the cause of your error. It is a combination of
>>>> mis-configuration and a bug. You have set:
>>>>
>>>> OBS_WINDOW_BEGIN = -2700
>>>> OBS_WINDOW_END = 2700
>>>>
>>>> to specify the range of data to be allowed to be read in.
However, thse
>>>> values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if those
are
>>> not
>>>> set. The _FILE_WINDOW_ variables determine the window of times to
find
>>> the
>>>> files based on their filename in case the filename times don't
exactly
>>> line
>>>> up with the valid time. You are supposed to be able to set
>>>>
>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>> OBS_FILE_WINDOW_END = 0
>>>>
>>>> to override the values. HOWEVER, there is a typo in the code that
>>> handles
>>>> these variables and PointStat actually uses
OBS_WINDOW_[BEGIN/END]
>>> instead.
>>>> I will fix this bug, but in the meantime, you should be able to
set
>>>>
>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>
>>>> and it should work as intended. Let me know if that does not
work.
>>>>
>>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
>>>>> Queue: met_help
>>>>> Subject: python/v3.6.3 and METplus beta version; Re:
init_seq
>>> works -
>>>>> thanks!
>>>>> Owner: mccabe
>>>>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
>>>>> Status: new
>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>>>>>
>>>>> This transaction appears to have no content
>>>>>
>>> --
>>> Ying Lin
>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>> NCWCP Cubicle No. 2015
>>> Ying.Lin at noaa.gov
>>>
>>>
>>>
>>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: George McCabe
Time: Mon Sep 16 18:53:46 2019
Hi Ying,
In your PointStatConfig_gauges file, you need to remove the quotes
around
regrid = {
to_grid = "${REGRID_TO_GRID}";
it should be
regrid = {
to_grid = ${REGRID_TO_GRID};
METplus checks if the value is NONE and adds quotes around it if it is
not
set to NONE.
On Mon, Sep 16, 2019 at 6:13 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>
> Hi George,
>
> Looks like
>
> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> OBS_POINT_STAT_FILE_WINDOW_END = 0
>
> fixed the issue with finding the gauge .nc file. Great!
>
> Setting the
>
> Now the issue remains is the PointStat err msg:
>
> DEBUG 1: Forecast File:
>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
> DEBUG 1: Observation File:
> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/
> good-usa-dlyprcp-20190915.nc
> DEBUG 2:
> DEBUG 2:
>
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for APCP_24(*,*).
> ERROR :
> ERROR : grd_file_type() -> file does not exist "NONE"
> ERROR :
>
> I placed the latest configs, log etc in
> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
> (older ones now under old/).
>
> Thanks again -
>
> Ying
>
> On 9/16/19 5:45 PM, George McCabe via RT wrote:
> > I will take a look at the POLY issue. My guess is you need to set
the
> poly
> > variable using {DMASK} instead of ${DMASK} since it is referencing
the
> > METplus config variable. I will let you know what I find.
> >
> > On Mon, Sep 16, 2019 at 3:41 PM George McCabe <mccabe at ucar.edu>
wrote:
> >
> >> Hi Ying,
> >>
> >> You should actually set:
> >>
> >> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >> instead of
> >> OBS_FILE_WINDOW_BEGIN and END
> >>
> >> You SHOULD be able to set the latter, but there is a bug that
prevents
> >> that from working. I think you will need OBS_WINDOW_BEGIN and END
to be
> >> -2700 to 2700 so that PointStat can find the correct data inside
the
> files.
> >> Does that make sense?
> >>
> >> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>>
> >>> Hi George,
> >>>
> >>> I made a number of runs, here are three(to be brief) at
> >>> 20190916195402 (run1), 20190916203610 (run2) and 20190916210024
(run3):
> >>> Various configs/logs in
> >>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
> >>> I've also placed the pcprtma.2019091512_A24h Run3 log complained
about:
> >>>
> >>>
> >>> In run1, I changed
> >>> OBS_WINDOW_BEGIN = -2700
> >>> OBS_WINDOW_END = 2700
> >>> to
> >>> OBS_WINDOW_BEGIN = 0
> >>> OBS_WINDOW_END = 0
> >>>
> >>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts found
the
> >>> current gauge .nc file, but while running PointStat and reading
in
> >>> /parm/PointStatConfig_gauges it complains of
> >>> unexpected "$" in line 132 of config file
> >>>
> >>> Below is the offending line:
> >>> //
> >>> // Verification masking regions
> >>> //
> >>> mask = {
> >>> grid = ${POINT_STAT_GRID};
> >>> poly = ${POINT_STAT_POLY}; <=== line 132
> >>> sid = [];
> >>> llpnt = [];
> >>> }
> >>>
> >>> Log file shows that the "poly" files for the mask was found, not
sure
> >>> why line 132 was objectionable.
> >>>
> >>> In run2, in pcprtma_24h.conf I have
> >>>
> >>> OBS_WINDOW_BEGIN = -2700
> >>> OBS_WINDOW_END = 2700
> >>>
> >>> OBS_FILE_WINDOW_BEGIN = 0
> >>> OBS_FILE_WINDOW_END = 0
> >>>
> >>> and I defined POINT_STAT_POLY in PointStatConfig_gauges instead
of in
> >>> pcprtma_24h.conf to get past the "error in line 132 in above
run1.
> >>>
> >>> Now I'm getting the same old error as last week:
> >>> 09/16 20:36:27.051 metplus.PointStat (point_stat_wrapper.py:194)
ERROR:
> >>> Could not find observation file in
> >>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc matching
> template
> >>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
> >>>
> >>> Run 3: I went back to setting
> >>> OBS_WINDOW_BEGIN = 0
> >>> OBS_WINDOW_END = 0
> >>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
> >>> as in Run1, and left the mask setting as in Run2. This time the
script
> >>> can find the gauge file, no complains about the mask poly files,
but
> >>> does not seem to be able to find the gridded 24h precip:
> >>> DEBUG 2:
> >>> DEBUG 2:
> >>>
> >>>
>
--------------------------------------------------------------------------------
> >>> DEBUG 2:
> >>> DEBUG 2: Reading data for APCP_24(*,*).
> >>> ERROR :
> >>> ERROR : grd_file_type() -> file does not exist "NONE"
> >>> ERROR :
> >>>
> >>> So in brief:
> >>>
> >>> * PointStat cannot read the 24h accumulation in
> pcprtma.2019091512_A24h
> >>> * Script can only find the gauges file when
> OBS_WINDOW_BEGIN[_END]are
> >>> set to zero. This is undesirable since it narrows the
valid time
> >>> for each 24h gauge too much
> >>>
> >>> Thanks
> >>> Ying
> >>>
> >>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
> >>>> Hi Ying,
> >>>>
> >>>> I determined the cause of your error. It is a combination of
> >>>> mis-configuration and a bug. You have set:
> >>>>
> >>>> OBS_WINDOW_BEGIN = -2700
> >>>> OBS_WINDOW_END = 2700
> >>>>
> >>>> to specify the range of data to be allowed to be read in.
However,
> thse
> >>>> values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if
those are
> >>> not
> >>>> set. The _FILE_WINDOW_ variables determine the window of times
to find
> >>> the
> >>>> files based on their filename in case the filename times don't
exactly
> >>> line
> >>>> up with the valid time. You are supposed to be able to set
> >>>>
> >>>> OBS_FILE_WINDOW_BEGIN = 0
> >>>> OBS_FILE_WINDOW_END = 0
> >>>>
> >>>> to override the values. HOWEVER, there is a typo in the code
that
> >>> handles
> >>>> these variables and PointStat actually uses
OBS_WINDOW_[BEGIN/END]
> >>> instead.
> >>>> I will fix this bug, but in the meantime, you should be able to
set
> >>>>
> >>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>
> >>>> and it should work as intended. Let me know if that does not
work.
> >>>>
> >>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT <
> >>>> met_help at ucar.edu> wrote:
> >>>>
> >>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
> >>>>> Transaction: Given to mccabe (George McCabe) by johnhg
> >>>>> Queue: met_help
> >>>>> Subject: python/v3.6.3 and METplus beta version; Re:
init_seq
> >>> works -
> >>>>> thanks!
> >>>>> Owner: mccabe
> >>>>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
> >>>>> Status: new
> >>>>> Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
> >>>>>
> >>>>> This transaction appears to have no content
> >>>>>
> >>> --
> >>> Ying Lin
> >>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>> NCWCP Cubicle No. 2015
> >>> Ying.Lin at noaa.gov
> >>>
> >>>
> >>>
> >>>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #92027] python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: Ying Lin
Time: Mon Sep 16 19:19:42 2019
Hi George,
OK, thanks, now we got pass that point. New err msg:
ERROR :
ERROR : PointStatVxOpt::set_vx_pd() -> "message_type_group_map" must
contain an entry for "LANDSF".
ERROR :
Logs etc again in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
Let's get back to this tomorrow.
Ying
On 9/16/19 8:53 PM, George McCabe via RT wrote:
> Hi Ying,
>
> In your PointStatConfig_gauges file, you need to remove the quotes
around
>
> regrid = {
> to_grid = "${REGRID_TO_GRID}";
>
> it should be
>
> regrid = {
> to_grid = ${REGRID_TO_GRID};
>
> METplus checks if the value is NONE and adds quotes around it if it
is not
> set to NONE.
>
> On Mon, Sep 16, 2019 at 6:13 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>
>> Hi George,
>>
>> Looks like
>>
>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>
>> fixed the issue with finding the gauge .nc file. Great!
>>
>> Setting the
>>
>> Now the issue remains is the PointStat err msg:
>>
>> DEBUG 1: Forecast File:
>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
>> DEBUG 1: Observation File:
>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/
>> good-usa-dlyprcp-20190915.nc
>> DEBUG 2:
>> DEBUG 2:
>>
>>
--------------------------------------------------------------------------------
>> DEBUG 2:
>> DEBUG 2: Reading data for APCP_24(*,*).
>> ERROR :
>> ERROR : grd_file_type() -> file does not exist "NONE"
>> ERROR :
>>
>> I placed the latest configs, log etc in
>> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
>> (older ones now under old/).
>>
>> Thanks again -
>>
>> Ying
>>
>> On 9/16/19 5:45 PM, George McCabe via RT wrote:
>>> I will take a look at the POLY issue. My guess is you need to set
the
>> poly
>>> variable using {DMASK} instead of ${DMASK} since it is referencing
the
>>> METplus config variable. I will let you know what I find.
>>>
>>> On Mon, Sep 16, 2019 at 3:41 PM George McCabe <mccabe at ucar.edu>
wrote:
>>>
>>>> Hi Ying,
>>>>
>>>> You should actually set:
>>>>
>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>> instead of
>>>> OBS_FILE_WINDOW_BEGIN and END
>>>>
>>>> You SHOULD be able to set the latter, but there is a bug that
prevents
>>>> that from working. I think you will need OBS_WINDOW_BEGIN and END
to be
>>>> -2700 to 2700 so that PointStat can find the correct data inside
the
>> files.
>>>> Does that make sense?
>>>>
>>>> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>>
>>>>> Hi George,
>>>>>
>>>>> I made a number of runs, here are three(to be brief) at
>>>>> 20190916195402 (run1), 20190916203610 (run2) and 20190916210024
(run3):
>>>>> Various configs/logs in
>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
>>>>> I've also placed the pcprtma.2019091512_A24h Run3 log complained
about:
>>>>>
>>>>>
>>>>> In run1, I changed
>>>>> OBS_WINDOW_BEGIN = -2700
>>>>> OBS_WINDOW_END = 2700
>>>>> to
>>>>> OBS_WINDOW_BEGIN = 0
>>>>> OBS_WINDOW_END = 0
>>>>>
>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts found
the
>>>>> current gauge .nc file, but while running PointStat and reading
in
>>>>> /parm/PointStatConfig_gauges it complains of
>>>>> unexpected "$" in line 132 of config file
>>>>>
>>>>> Below is the offending line:
>>>>> //
>>>>> // Verification masking regions
>>>>> //
>>>>> mask = {
>>>>> grid = ${POINT_STAT_GRID};
>>>>> poly = ${POINT_STAT_POLY}; <=== line 132
>>>>> sid = [];
>>>>> llpnt = [];
>>>>> }
>>>>>
>>>>> Log file shows that the "poly" files for the mask was found, not
sure
>>>>> why line 132 was objectionable.
>>>>>
>>>>> In run2, in pcprtma_24h.conf I have
>>>>>
>>>>> OBS_WINDOW_BEGIN = -2700
>>>>> OBS_WINDOW_END = 2700
>>>>>
>>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>>> OBS_FILE_WINDOW_END = 0
>>>>>
>>>>> and I defined POINT_STAT_POLY in PointStatConfig_gauges instead
of in
>>>>> pcprtma_24h.conf to get past the "error in line 132 in above
run1.
>>>>>
>>>>> Now I'm getting the same old error as last week:
>>>>> 09/16 20:36:27.051 metplus.PointStat (point_stat_wrapper.py:194)
ERROR:
>>>>> Could not find observation file in
>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc matching
>> template
>>>>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
>>>>>
>>>>> Run 3: I went back to setting
>>>>> OBS_WINDOW_BEGIN = 0
>>>>> OBS_WINDOW_END = 0
>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
>>>>> as in Run1, and left the mask setting as in Run2. This time the
script
>>>>> can find the gauge file, no complains about the mask poly files,
but
>>>>> does not seem to be able to find the gridded 24h precip:
>>>>> DEBUG 2:
>>>>> DEBUG 2:
>>>>>
>>>>>
>>
--------------------------------------------------------------------------------
>>>>> DEBUG 2:
>>>>> DEBUG 2: Reading data for APCP_24(*,*).
>>>>> ERROR :
>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
>>>>> ERROR :
>>>>>
>>>>> So in brief:
>>>>>
>>>>> * PointStat cannot read the 24h accumulation in
>> pcprtma.2019091512_A24h
>>>>> * Script can only find the gauges file when
>> OBS_WINDOW_BEGIN[_END]are
>>>>> set to zero. This is undesirable since it narrows the
valid time
>>>>> for each 24h gauge too much
>>>>>
>>>>> Thanks
>>>>> Ying
>>>>>
>>>>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
>>>>>> Hi Ying,
>>>>>>
>>>>>> I determined the cause of your error. It is a combination of
>>>>>> mis-configuration and a bug. You have set:
>>>>>>
>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>> OBS_WINDOW_END = 2700
>>>>>>
>>>>>> to specify the range of data to be allowed to be read in.
However,
>> thse
>>>>>> values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if
those are
>>>>> not
>>>>>> set. The _FILE_WINDOW_ variables determine the window of times
to find
>>>>> the
>>>>>> files based on their filename in case the filename times don't
exactly
>>>>> line
>>>>>> up with the valid time. You are supposed to be able to set
>>>>>>
>>>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>>>> OBS_FILE_WINDOW_END = 0
>>>>>>
>>>>>> to override the values. HOWEVER, there is a typo in the code
that
>>>>> handles
>>>>>> these variables and PointStat actually uses
OBS_WINDOW_[BEGIN/END]
>>>>> instead.
>>>>>> I will fix this bug, but in the meantime, you should be able to
set
>>>>>>
>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>
>>>>>> and it should work as intended. Let me know if that does not
work.
>>>>>>
>>>>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT <
>>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
>>>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
>>>>>>> Queue: met_help
>>>>>>> Subject: python/v3.6.3 and METplus beta version; Re:
init_seq
>>>>> works -
>>>>>>> thanks!
>>>>>>> Owner: mccabe
>>>>>>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
>>>>>>> Status: new
>>>>>>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>>>>>>> This transaction appears to have no content
>>>>>>>
>>>>> --
>>>>> Ying Lin
>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>> NCWCP Cubicle No. 2015
>>>>> Ying.Lin at noaa.gov
>>>>>
>>>>>
>>>>>
>>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: George McCabe
Time: Tue Sep 17 09:42:32 2019
Hi Ying,
There are new default values that need to be set in your
PointStatConfig_gauges file that are required by the latest version of
MET.
You should have an entry that looks like this:
message_type_group_map = [
{ key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET"; },
{ key = "ANYAIR"; val = "AIRCAR,AIRCFT"; },
{ key = "ANYSFC"; val = "ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; },
{ key = "ONLYSF"; val = "ADPSFC,SFCSHP"; },
{ key = "LANDSF"; val = "ADPSFC,MSONET"; },
{ key = "WATERSF"; val = "SFCSHP"; }
];
You can refer to the default config files (located in share/met/config
under the location MET is installed) to see what all of the defaults
are.
You should add all of the default values to your config if they are
not
already set.
On Tue, Sep 17, 2019 at 1:19 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>
> Hi George,
>
> OK, thanks, now we got pass that point. New err msg:
>
> ERROR :
> ERROR : PointStatVxOpt::set_vx_pd() -> "message_type_group_map"
must
> contain an entry for "LANDSF".
> ERROR :
>
> Logs etc again in
> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
>
> Let's get back to this tomorrow.
>
> Ying
>
> On 9/16/19 8:53 PM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > In your PointStatConfig_gauges file, you need to remove the quotes
around
> >
> > regrid = {
> > to_grid = "${REGRID_TO_GRID}";
> >
> > it should be
> >
> > regrid = {
> > to_grid = ${REGRID_TO_GRID};
> >
> > METplus checks if the value is NONE and adds quotes around it if
it is
> not
> > set to NONE.
> >
> > On Mon, Sep 16, 2019 at 6:13 PM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>
> >> Hi George,
> >>
> >> Looks like
> >>
> >> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>
> >> fixed the issue with finding the gauge .nc file. Great!
> >>
> >> Setting the
> >>
> >> Now the issue remains is the PointStat err msg:
> >>
> >> DEBUG 1: Forecast File:
> >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
> >> DEBUG 1: Observation File:
> >> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/
> >> good-usa-dlyprcp-20190915.nc
> >> DEBUG 2:
> >> DEBUG 2:
> >>
> >>
>
--------------------------------------------------------------------------------
> >> DEBUG 2:
> >> DEBUG 2: Reading data for APCP_24(*,*).
> >> ERROR :
> >> ERROR : grd_file_type() -> file does not exist "NONE"
> >> ERROR :
> >>
> >> I placed the latest configs, log etc in
> >>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
> >> (older ones now under old/).
> >>
> >> Thanks again -
> >>
> >> Ying
> >>
> >> On 9/16/19 5:45 PM, George McCabe via RT wrote:
> >>> I will take a look at the POLY issue. My guess is you need to
set the
> >> poly
> >>> variable using {DMASK} instead of ${DMASK} since it is
referencing the
> >>> METplus config variable. I will let you know what I find.
> >>>
> >>> On Mon, Sep 16, 2019 at 3:41 PM George McCabe <mccabe at ucar.edu>
wrote:
> >>>
> >>>> Hi Ying,
> >>>>
> >>>> You should actually set:
> >>>>
> >>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>> instead of
> >>>> OBS_FILE_WINDOW_BEGIN and END
> >>>>
> >>>> You SHOULD be able to set the latter, but there is a bug that
prevents
> >>>> that from working. I think you will need OBS_WINDOW_BEGIN and
END to
> be
> >>>> -2700 to 2700 so that PointStat can find the correct data
inside the
> >> files.
> >>>> Does that make sense?
> >>>>
> >>>> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>
> >>>>>
> >>>>> Hi George,
> >>>>>
> >>>>> I made a number of runs, here are three(to be brief) at
> >>>>> 20190916195402 (run1), 20190916203610 (run2) and
20190916210024
> (run3):
> >>>>> Various configs/logs in
> >>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
> >>>>> I've also placed the pcprtma.2019091512_A24h Run3 log
complained
> about:
> >>>>>
> >>>>>
> >>>>> In run1, I changed
> >>>>> OBS_WINDOW_BEGIN = -2700
> >>>>> OBS_WINDOW_END = 2700
> >>>>> to
> >>>>> OBS_WINDOW_BEGIN = 0
> >>>>> OBS_WINDOW_END = 0
> >>>>>
> >>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts found
the
> >>>>> current gauge .nc file, but while running PointStat and
reading in
> >>>>> /parm/PointStatConfig_gauges it complains of
> >>>>> unexpected "$" in line 132 of config file
> >>>>>
> >>>>> Below is the offending line:
> >>>>> //
> >>>>> // Verification masking regions
> >>>>> //
> >>>>> mask = {
> >>>>> grid = ${POINT_STAT_GRID};
> >>>>> poly = ${POINT_STAT_POLY}; <=== line 132
> >>>>> sid = [];
> >>>>> llpnt = [];
> >>>>> }
> >>>>>
> >>>>> Log file shows that the "poly" files for the mask was found,
not sure
> >>>>> why line 132 was objectionable.
> >>>>>
> >>>>> In run2, in pcprtma_24h.conf I have
> >>>>>
> >>>>> OBS_WINDOW_BEGIN = -2700
> >>>>> OBS_WINDOW_END = 2700
> >>>>>
> >>>>> OBS_FILE_WINDOW_BEGIN = 0
> >>>>> OBS_FILE_WINDOW_END = 0
> >>>>>
> >>>>> and I defined POINT_STAT_POLY in PointStatConfig_gauges
instead of in
> >>>>> pcprtma_24h.conf to get past the "error in line 132 in above
run1.
> >>>>>
> >>>>> Now I'm getting the same old error as last week:
> >>>>> 09/16 20:36:27.051 metplus.PointStat
(point_stat_wrapper.py:194)
> ERROR:
> >>>>> Could not find observation file in
> >>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc
matching
> >> template
> >>>>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
> >>>>>
> >>>>> Run 3: I went back to setting
> >>>>> OBS_WINDOW_BEGIN = 0
> >>>>> OBS_WINDOW_END = 0
> >>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
> >>>>> as in Run1, and left the mask setting as in Run2. This time
the
> script
> >>>>> can find the gauge file, no complains about the mask poly
files, but
> >>>>> does not seem to be able to find the gridded 24h precip:
> >>>>> DEBUG 2:
> >>>>> DEBUG 2:
> >>>>>
> >>>>>
> >>
>
--------------------------------------------------------------------------------
> >>>>> DEBUG 2:
> >>>>> DEBUG 2: Reading data for APCP_24(*,*).
> >>>>> ERROR :
> >>>>> ERROR : grd_file_type() -> file does not exist "NONE"
> >>>>> ERROR :
> >>>>>
> >>>>> So in brief:
> >>>>>
> >>>>> * PointStat cannot read the 24h accumulation in
> >> pcprtma.2019091512_A24h
> >>>>> * Script can only find the gauges file when
> >> OBS_WINDOW_BEGIN[_END]are
> >>>>> set to zero. This is undesirable since it narrows the
valid
> time
> >>>>> for each 24h gauge too much
> >>>>>
> >>>>> Thanks
> >>>>> Ying
> >>>>>
> >>>>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
> >>>>>> Hi Ying,
> >>>>>>
> >>>>>> I determined the cause of your error. It is a combination of
> >>>>>> mis-configuration and a bug. You have set:
> >>>>>>
> >>>>>> OBS_WINDOW_BEGIN = -2700
> >>>>>> OBS_WINDOW_END = 2700
> >>>>>>
> >>>>>> to specify the range of data to be allowed to be read in.
However,
> >> thse
> >>>>>> values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if
those
> are
> >>>>> not
> >>>>>> set. The _FILE_WINDOW_ variables determine the window of
times to
> find
> >>>>> the
> >>>>>> files based on their filename in case the filename times
don't
> exactly
> >>>>> line
> >>>>>> up with the valid time. You are supposed to be able to set
> >>>>>>
> >>>>>> OBS_FILE_WINDOW_BEGIN = 0
> >>>>>> OBS_FILE_WINDOW_END = 0
> >>>>>>
> >>>>>> to override the values. HOWEVER, there is a typo in the code
that
> >>>>> handles
> >>>>>> these variables and PointStat actually uses
OBS_WINDOW_[BEGIN/END]
> >>>>> instead.
> >>>>>> I will fix this bug, but in the meantime, you should be able
to set
> >>>>>>
> >>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>>>
> >>>>>> and it should work as intended. Let me know if that does not
work.
> >>>>>>
> >>>>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT <
> >>>>>> met_help at ucar.edu> wrote:
> >>>>>>
> >>>>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
> >>>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
> >>>>>>> Queue: met_help
> >>>>>>> Subject: python/v3.6.3 and METplus beta version; Re:
> init_seq
> >>>>> works -
> >>>>>>> thanks!
> >>>>>>> Owner: mccabe
> >>>>>>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
> >>>>>>> Status: new
> >>>>>>> Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
> >>>>>>> This transaction appears to have no content
> >>>>>>>
> >>>>> --
> >>>>> Ying Lin
> >>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>>> NCWCP Cubicle No. 2015
> >>>>> Ying.Lin at noaa.gov
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #92027] python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: Ying Lin
Time: Wed Sep 18 09:40:46 2019
Thanks George, I see that my PointStatConfig was missing the settings
for "LANDSF" and "WATERSF" in the message_type_group_map. Now the
test
no longer has the err msgs.
However, the point_stat step did not seem to have found any matches -
the point_stat_*.txt and *.stat files all had only the header line:
see
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/
A few weeks ago I ran the pcpRTMA vs. gauges in MET7.0 (couldn't get
it
to run in MET8.1 at the time, probably due to the undefined default
values as you mentioned in the email below), and that ran fine and
produced reasonable results. Output: point_stat_met7.0.out
Thanks again for looking into this.
Ying
On 9/17/19 11:42 AM, George McCabe via RT wrote:
> Hi Ying,
>
> There are new default values that need to be set in your
> PointStatConfig_gauges file that are required by the latest version
of MET.
> You should have an entry that looks like this:
>
> message_type_group_map = [
> { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";
},
> { key = "ANYAIR"; val = "AIRCAR,AIRCFT";
},
> { key = "ANYSFC"; val = "ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET";
},
> { key = "ONLYSF"; val = "ADPSFC,SFCSHP";
},
> { key = "LANDSF"; val = "ADPSFC,MSONET";
},
> { key = "WATERSF"; val = "SFCSHP"; }
> ];
>
> You can refer to the default config files (located in
share/met/config
> under the location MET is installed) to see what all of the defaults
are.
> You should add all of the default values to your config if they are
not
> already set.
>
> On Tue, Sep 17, 2019 at 1:19 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>
>> Hi George,
>>
>> OK, thanks, now we got pass that point. New err msg:
>>
>> ERROR :
>> ERROR : PointStatVxOpt::set_vx_pd() -> "message_type_group_map"
must
>> contain an entry for "LANDSF".
>> ERROR :
>>
>> Logs etc again in
>> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
>>
>> Let's get back to this tomorrow.
>>
>> Ying
>>
>> On 9/16/19 8:53 PM, George McCabe via RT wrote:
>>> Hi Ying,
>>>
>>> In your PointStatConfig_gauges file, you need to remove the quotes
around
>>>
>>> regrid = {
>>> to_grid = "${REGRID_TO_GRID}";
>>>
>>> it should be
>>>
>>> regrid = {
>>> to_grid = ${REGRID_TO_GRID};
>>>
>>> METplus checks if the value is NONE and adds quotes around it if
it is
>> not
>>> set to NONE.
>>>
>>> On Mon, Sep 16, 2019 at 6:13 PM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>
>>>> Hi George,
>>>>
>>>> Looks like
>>>>
>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>
>>>> fixed the issue with finding the gauge .nc file. Great!
>>>>
>>>> Setting the
>>>>
>>>> Now the issue remains is the PointStat err msg:
>>>>
>>>> DEBUG 1: Forecast File:
>>>>
>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
>>>> DEBUG 1: Observation File:
>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/
>>>> good-usa-dlyprcp-20190915.nc
>>>> DEBUG 2:
>>>> DEBUG 2:
>>>>
>>>>
>>
--------------------------------------------------------------------------------
>>>> DEBUG 2:
>>>> DEBUG 2: Reading data for APCP_24(*,*).
>>>> ERROR :
>>>> ERROR : grd_file_type() -> file does not exist "NONE"
>>>> ERROR :
>>>>
>>>> I placed the latest configs, log etc in
>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
>>>> (older ones now under old/).
>>>>
>>>> Thanks again -
>>>>
>>>> Ying
>>>>
>>>> On 9/16/19 5:45 PM, George McCabe via RT wrote:
>>>>> I will take a look at the POLY issue. My guess is you need to
set the
>>>> poly
>>>>> variable using {DMASK} instead of ${DMASK} since it is
referencing the
>>>>> METplus config variable. I will let you know what I find.
>>>>>
>>>>> On Mon, Sep 16, 2019 at 3:41 PM George McCabe <mccabe at ucar.edu>
wrote:
>>>>>
>>>>>> Hi Ying,
>>>>>>
>>>>>> You should actually set:
>>>>>>
>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>> instead of
>>>>>> OBS_FILE_WINDOW_BEGIN and END
>>>>>>
>>>>>> You SHOULD be able to set the latter, but there is a bug that
prevents
>>>>>> that from working. I think you will need OBS_WINDOW_BEGIN and
END to
>> be
>>>>>> -2700 to 2700 so that PointStat can find the correct data
inside the
>>>> files.
>>>>>> Does that make sense?
>>>>>>
>>>>>> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>
>>>>>>>
>>>>>>> Hi George,
>>>>>>>
>>>>>>> I made a number of runs, here are three(to be brief)
at
>>>>>>> 20190916195402 (run1), 20190916203610 (run2) and
20190916210024
>> (run3):
>>>>>>> Various configs/logs in
>>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
>>>>>>> I've also placed the pcprtma.2019091512_A24h Run3 log
complained
>> about:
>>>>>>>
>>>>>>> In run1, I changed
>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>> OBS_WINDOW_END = 2700
>>>>>>> to
>>>>>>> OBS_WINDOW_BEGIN = 0
>>>>>>> OBS_WINDOW_END = 0
>>>>>>>
>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts found
the
>>>>>>> current gauge .nc file, but while running PointStat and
reading in
>>>>>>> /parm/PointStatConfig_gauges it complains of
>>>>>>> unexpected "$" in line 132 of config file
>>>>>>>
>>>>>>> Below is the offending line:
>>>>>>> //
>>>>>>> // Verification masking regions
>>>>>>> //
>>>>>>> mask = {
>>>>>>> grid = ${POINT_STAT_GRID};
>>>>>>> poly = ${POINT_STAT_POLY}; <=== line 132
>>>>>>> sid = [];
>>>>>>> llpnt = [];
>>>>>>> }
>>>>>>>
>>>>>>> Log file shows that the "poly" files for the mask was found,
not sure
>>>>>>> why line 132 was objectionable.
>>>>>>>
>>>>>>> In run2, in pcprtma_24h.conf I have
>>>>>>>
>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>
>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>>>>> OBS_FILE_WINDOW_END = 0
>>>>>>>
>>>>>>> and I defined POINT_STAT_POLY in PointStatConfig_gauges
instead of in
>>>>>>> pcprtma_24h.conf to get past the "error in line 132 in above
run1.
>>>>>>>
>>>>>>> Now I'm getting the same old error as last week:
>>>>>>> 09/16 20:36:27.051 metplus.PointStat
(point_stat_wrapper.py:194)
>> ERROR:
>>>>>>> Could not find observation file in
>>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc
matching
>>>> template
>>>>>>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
>>>>>>>
>>>>>>> Run 3: I went back to setting
>>>>>>> OBS_WINDOW_BEGIN = 0
>>>>>>> OBS_WINDOW_END = 0
>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
>>>>>>> as in Run1, and left the mask setting as in Run2. This time
the
>> script
>>>>>>> can find the gauge file, no complains about the mask poly
files, but
>>>>>>> does not seem to be able to find the gridded 24h precip:
>>>>>>> DEBUG 2:
>>>>>>> DEBUG 2:
>>>>>>>
>>>>>>>
>>
--------------------------------------------------------------------------------
>>>>>>> DEBUG 2:
>>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
>>>>>>> ERROR :
>>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
>>>>>>> ERROR :
>>>>>>>
>>>>>>> So in brief:
>>>>>>>
>>>>>>> * PointStat cannot read the 24h accumulation in
>>>> pcprtma.2019091512_A24h
>>>>>>> * Script can only find the gauges file when
>>>> OBS_WINDOW_BEGIN[_END]are
>>>>>>> set to zero. This is undesirable since it narrows the
valid
>> time
>>>>>>> for each 24h gauge too much
>>>>>>>
>>>>>>> Thanks
>>>>>>> Ying
>>>>>>>
>>>>>>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
>>>>>>>> Hi Ying,
>>>>>>>>
>>>>>>>> I determined the cause of your error. It is a combination of
>>>>>>>> mis-configuration and a bug. You have set:
>>>>>>>>
>>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>>
>>>>>>>> to specify the range of data to be allowed to be read in.
However,
>>>> thse
>>>>>>>> values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if
those
>> are
>>>>>>> not
>>>>>>>> set. The _FILE_WINDOW_ variables determine the window of
times to
>> find
>>>>>>> the
>>>>>>>> files based on their filename in case the filename times
don't
>> exactly
>>>>>>> line
>>>>>>>> up with the valid time. You are supposed to be able to set
>>>>>>>>
>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>>>>>> OBS_FILE_WINDOW_END = 0
>>>>>>>>
>>>>>>>> to override the values. HOWEVER, there is a typo in the code
that
>>>>>>> handles
>>>>>>>> these variables and PointStat actually uses
OBS_WINDOW_[BEGIN/END]
>>>>>>> instead.
>>>>>>>> I will fix this bug, but in the meantime, you should be able
to set
>>>>>>>>
>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>>>
>>>>>>>> and it should work as intended. Let me know if that does not
work.
>>>>>>>>
>>>>>>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT <
>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>
>>>>>>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
>>>>>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
>>>>>>>>> Queue: met_help
>>>>>>>>> Subject: python/v3.6.3 and METplus beta version;
Re:
>> init_seq
>>>>>>> works -
>>>>>>>>> thanks!
>>>>>>>>> Owner: mccabe
>>>>>>>>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
>>>>>>>>> Status: new
>>>>>>>>> Ticket <URL:
>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>>>>>>>>> This transaction appears to have no content
>>>>>>>>>
>>>>>>> --
>>>>>>> Ying Lin
>>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>>> NCWCP Cubicle No. 2015
>>>>>>> Ying.Lin at noaa.gov
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: George McCabe
Time: Wed Sep 18 10:05:18 2019
Hi Ying,
It is hard to say why this run didn't work without comparing the
PointStat
config file you used last time to what is set here. I would recommend
rerunning point_stat with verbosity increased to 3 so you can see the
reasons that the matched pairs were rejected. To do this, copy all of
the
export commands from the line after COPYABLE ENVIRONMENT FOR NEXT
COMMAND:
and paste that into a command prompt. Then copy the point_stat command
but
add '-v 3' to it. The output will now show how the matches pairs were
rejected, which is already shown in your old output. Compare the logs
to
see why they are being rejected. This will give you a clue as to what
is
different in your configuration between runs.
On Wed, Sep 18, 2019 at 9:41 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>
> Thanks George, I see that my PointStatConfig was missing the
settings
> for "LANDSF" and "WATERSF" in the message_type_group_map. Now the
test
> no longer has the err msgs.
>
> However, the point_stat step did not seem to have found any matches
-
> the point_stat_*.txt and *.stat files all had only the header line:
see
> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/
>
> A few weeks ago I ran the pcpRTMA vs. gauges in MET7.0 (couldn't
get it
> to run in MET8.1 at the time, probably due to the undefined default
> values as you mentioned in the email below), and that ran fine and
> produced reasonable results. Output: point_stat_met7.0.out
>
> Thanks again for looking into this.
>
> Ying
>
>
>
> On 9/17/19 11:42 AM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > There are new default values that need to be set in your
> > PointStatConfig_gauges file that are required by the latest
version of
> MET.
> > You should have an entry that looks like this:
> >
> > message_type_group_map = [
> > { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";
},
> > { key = "ANYAIR"; val = "AIRCAR,AIRCFT";
},
> > { key = "ANYSFC"; val = "ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET";
},
> > { key = "ONLYSF"; val = "ADPSFC,SFCSHP";
},
> > { key = "LANDSF"; val = "ADPSFC,MSONET";
},
> > { key = "WATERSF"; val = "SFCSHP";
}
> > ];
> >
> > You can refer to the default config files (located in
share/met/config
> > under the location MET is installed) to see what all of the
defaults are.
> > You should add all of the default values to your config if they
are not
> > already set.
> >
> > On Tue, Sep 17, 2019 at 1:19 AM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>
> >> Hi George,
> >>
> >> OK, thanks, now we got pass that point. New err msg:
> >>
> >> ERROR :
> >> ERROR : PointStatVxOpt::set_vx_pd() -> "message_type_group_map"
must
> >> contain an entry for "LANDSF".
> >> ERROR :
> >>
> >> Logs etc again in
> >>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
> >>
> >> Let's get back to this tomorrow.
> >>
> >> Ying
> >>
> >> On 9/16/19 8:53 PM, George McCabe via RT wrote:
> >>> Hi Ying,
> >>>
> >>> In your PointStatConfig_gauges file, you need to remove the
quotes
> around
> >>>
> >>> regrid = {
> >>> to_grid = "${REGRID_TO_GRID}";
> >>>
> >>> it should be
> >>>
> >>> regrid = {
> >>> to_grid = ${REGRID_TO_GRID};
> >>>
> >>> METplus checks if the value is NONE and adds quotes around it if
it is
> >> not
> >>> set to NONE.
> >>>
> >>> On Mon, Sep 16, 2019 at 6:13 PM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>>>
> >>>> Hi George,
> >>>>
> >>>> Looks like
> >>>>
> >>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>
> >>>> fixed the issue with finding the gauge .nc file. Great!
> >>>>
> >>>> Setting the
> >>>>
> >>>> Now the issue remains is the PointStat err msg:
> >>>>
> >>>> DEBUG 1: Forecast File:
> >>>>
> >>>>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
> >>>> DEBUG 1: Observation File:
> >>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/
> >>>> good-usa-dlyprcp-20190915.nc
> >>>> DEBUG 2:
> >>>> DEBUG 2:
> >>>>
> >>>>
> >>
>
--------------------------------------------------------------------------------
> >>>> DEBUG 2:
> >>>> DEBUG 2: Reading data for APCP_24(*,*).
> >>>> ERROR :
> >>>> ERROR : grd_file_type() -> file does not exist "NONE"
> >>>> ERROR :
> >>>>
> >>>> I placed the latest configs, log etc in
> >>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
> >>>> (older ones now under old/).
> >>>>
> >>>> Thanks again -
> >>>>
> >>>> Ying
> >>>>
> >>>> On 9/16/19 5:45 PM, George McCabe via RT wrote:
> >>>>> I will take a look at the POLY issue. My guess is you need to
set the
> >>>> poly
> >>>>> variable using {DMASK} instead of ${DMASK} since it is
referencing
> the
> >>>>> METplus config variable. I will let you know what I find.
> >>>>>
> >>>>> On Mon, Sep 16, 2019 at 3:41 PM George McCabe
<mccabe at ucar.edu>
> wrote:
> >>>>>
> >>>>>> Hi Ying,
> >>>>>>
> >>>>>> You should actually set:
> >>>>>>
> >>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>>> instead of
> >>>>>> OBS_FILE_WINDOW_BEGIN and END
> >>>>>>
> >>>>>> You SHOULD be able to set the latter, but there is a bug that
> prevents
> >>>>>> that from working. I think you will need OBS_WINDOW_BEGIN and
END to
> >> be
> >>>>>> -2700 to 2700 so that PointStat can find the correct data
inside the
> >>>> files.
> >>>>>> Does that make sense?
> >>>>>>
> >>>>>> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT
<met_help at ucar.edu>
> >>>> wrote:
> >>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>>>>>>
> >>>>>>> Hi George,
> >>>>>>>
> >>>>>>> I made a number of runs, here are three(to be brief)
at
> >>>>>>> 20190916195402 (run1), 20190916203610 (run2) and
20190916210024
> >> (run3):
> >>>>>>> Various configs/logs in
> >>>>>>>
> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
> >>>>>>> I've also placed the pcprtma.2019091512_A24h Run3 log
complained
> >> about:
> >>>>>>>
> >>>>>>> In run1, I changed
> >>>>>>> OBS_WINDOW_BEGIN = -2700
> >>>>>>> OBS_WINDOW_END = 2700
> >>>>>>> to
> >>>>>>> OBS_WINDOW_BEGIN = 0
> >>>>>>> OBS_WINDOW_END = 0
> >>>>>>>
> >>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts
found the
> >>>>>>> current gauge .nc file, but while running PointStat and
reading in
> >>>>>>> /parm/PointStatConfig_gauges it complains of
> >>>>>>> unexpected "$" in line 132 of config file
> >>>>>>>
> >>>>>>> Below is the offending line:
> >>>>>>> //
> >>>>>>> // Verification masking regions
> >>>>>>> //
> >>>>>>> mask = {
> >>>>>>> grid = ${POINT_STAT_GRID};
> >>>>>>> poly = ${POINT_STAT_POLY}; <=== line 132
> >>>>>>> sid = [];
> >>>>>>> llpnt = [];
> >>>>>>> }
> >>>>>>>
> >>>>>>> Log file shows that the "poly" files for the mask was found,
not
> sure
> >>>>>>> why line 132 was objectionable.
> >>>>>>>
> >>>>>>> In run2, in pcprtma_24h.conf I have
> >>>>>>>
> >>>>>>> OBS_WINDOW_BEGIN = -2700
> >>>>>>> OBS_WINDOW_END = 2700
> >>>>>>>
> >>>>>>> OBS_FILE_WINDOW_BEGIN = 0
> >>>>>>> OBS_FILE_WINDOW_END = 0
> >>>>>>>
> >>>>>>> and I defined POINT_STAT_POLY in PointStatConfig_gauges
instead of
> in
> >>>>>>> pcprtma_24h.conf to get past the "error in line 132 in above
run1.
> >>>>>>>
> >>>>>>> Now I'm getting the same old error as last week:
> >>>>>>> 09/16 20:36:27.051 metplus.PointStat
(point_stat_wrapper.py:194)
> >> ERROR:
> >>>>>>> Could not find observation file in
> >>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc
matching
> >>>> template
> >>>>>>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
> >>>>>>>
> >>>>>>> Run 3: I went back to setting
> >>>>>>> OBS_WINDOW_BEGIN = 0
> >>>>>>> OBS_WINDOW_END = 0
> >>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
> >>>>>>> as in Run1, and left the mask setting as in Run2. This time
the
> >> script
> >>>>>>> can find the gauge file, no complains about the mask poly
files,
> but
> >>>>>>> does not seem to be able to find the gridded 24h precip:
> >>>>>>> DEBUG 2:
> >>>>>>> DEBUG 2:
> >>>>>>>
> >>>>>>>
> >>
>
--------------------------------------------------------------------------------
> >>>>>>> DEBUG 2:
> >>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
> >>>>>>> ERROR :
> >>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
> >>>>>>> ERROR :
> >>>>>>>
> >>>>>>> So in brief:
> >>>>>>>
> >>>>>>> * PointStat cannot read the 24h accumulation in
> >>>> pcprtma.2019091512_A24h
> >>>>>>> * Script can only find the gauges file when
> >>>> OBS_WINDOW_BEGIN[_END]are
> >>>>>>> set to zero. This is undesirable since it narrows
the valid
> >> time
> >>>>>>> for each 24h gauge too much
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>> Ying
> >>>>>>>
> >>>>>>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
> >>>>>>>> Hi Ying,
> >>>>>>>>
> >>>>>>>> I determined the cause of your error. It is a combination
of
> >>>>>>>> mis-configuration and a bug. You have set:
> >>>>>>>>
> >>>>>>>> OBS_WINDOW_BEGIN = -2700
> >>>>>>>> OBS_WINDOW_END = 2700
> >>>>>>>>
> >>>>>>>> to specify the range of data to be allowed to be read in.
However,
> >>>> thse
> >>>>>>>> values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if
those
> >> are
> >>>>>>> not
> >>>>>>>> set. The _FILE_WINDOW_ variables determine the window of
times to
> >> find
> >>>>>>> the
> >>>>>>>> files based on their filename in case the filename times
don't
> >> exactly
> >>>>>>> line
> >>>>>>>> up with the valid time. You are supposed to be able to set
> >>>>>>>>
> >>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
> >>>>>>>> OBS_FILE_WINDOW_END = 0
> >>>>>>>>
> >>>>>>>> to override the values. HOWEVER, there is a typo in the
code that
> >>>>>>> handles
> >>>>>>>> these variables and PointStat actually uses
OBS_WINDOW_[BEGIN/END]
> >>>>>>> instead.
> >>>>>>>> I will fix this bug, but in the meantime, you should be
able to
> set
> >>>>>>>>
> >>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>>>>>
> >>>>>>>> and it should work as intended. Let me know if that does
not work.
> >>>>>>>>
> >>>>>>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT
<
> >>>>>>>> met_help at ucar.edu> wrote:
> >>>>>>>>
> >>>>>>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
> >>>>>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
> >>>>>>>>> Queue: met_help
> >>>>>>>>> Subject: python/v3.6.3 and METplus beta version;
Re:
> >> init_seq
> >>>>>>> works -
> >>>>>>>>> thanks!
> >>>>>>>>> Owner: mccabe
> >>>>>>>>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
> >>>>>>>>> Status: new
> >>>>>>>>> Ticket <URL:
> >>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
> >>>>>>>>> This transaction appears to have no content
> >>>>>>>>>
> >>>>>>> --
> >>>>>>> Ying Lin
> >>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
> Branch
> >>>>>>> NCWCP Cubicle No. 2015
> >>>>>>> Ying.Lin at noaa.gov
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>> --
> >>>> Ying Lin
> >>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>> NCWCP Cubicle No. 2015
> >>>> Ying.Lin at noaa.gov
> >>>>
> >>>>
> >>>>
> >>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #92027] python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: Ying Lin
Time: Wed Sep 18 15:43:16 2019
Hi George,
Good stuff, the COPYABLE ENVIRONMENT FOR NEXT COMMAND. So I did
as
you said, and the reason for not having any matching pairs is that all
observations were rejected due to quality marker
(https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/master_metplus.log.test).
Initially I thought it had something to do with all entries having the
elev_now and height_now (in precip_ascii2nc.py, same ftp dir) set to
'-9999.' (as in
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/good-
usa-dlyprcp-20190915_formatted)
so I went and changed both of these columns to '0.0', since they're
not
used in the verification anyway; and I also decided to have any
negative
precip values (-9999., or an unintentional negative precip amount
report) in the input to be encoded as '-9999.' in the
good-usa-dlyprcp-20190915.nc, and there are 69 such reports.
It didn't make a difference though. master_metplus.log.test
shows
that
DEBUG 3: Observations processed = 7646
....
DEBUG 3: Rejected: quality marker = 7646
Can you tell by looking at good-usa-dlyprcp-20190915_formatted what
makes every entry fail the quality marker? Maybe MET8.1 requires a
different format for the point output. I originally got the
precip_ascii2nc.py from Mallory more than a year ago. Output from it
(even with height and elev both set to '-9999.') works for MET7.0
Thanks again -
Ying
On 9/18/19 12:05 PM, George McCabe via RT wrote:
> Hi Ying,
>
> It is hard to say why this run didn't work without comparing the
PointStat
> config file you used last time to what is set here. I would
recommend
> rerunning point_stat with verbosity increased to 3 so you can see
the
> reasons that the matched pairs were rejected. To do this, copy all
of the
> export commands from the line after COPYABLE ENVIRONMENT FOR NEXT
COMMAND:
> and paste that into a command prompt. Then copy the point_stat
command but
> add '-v 3' to it. The output will now show how the matches pairs
were
> rejected, which is already shown in your old output. Compare the
logs to
> see why they are being rejected. This will give you a clue as to
what is
> different in your configuration between runs.
>
> On Wed, Sep 18, 2019 at 9:41 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>
>> Thanks George, I see that my PointStatConfig was missing the
settings
>> for "LANDSF" and "WATERSF" in the message_type_group_map. Now the
test
>> no longer has the err msgs.
>>
>> However, the point_stat step did not seem to have found any matches
-
>> the point_stat_*.txt and *.stat files all had only the header line:
see
>> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/
>>
>> A few weeks ago I ran the pcpRTMA vs. gauges in MET7.0 (couldn't
get it
>> to run in MET8.1 at the time, probably due to the undefined default
>> values as you mentioned in the email below), and that ran fine and
>> produced reasonable results. Output: point_stat_met7.0.out
>>
>> Thanks again for looking into this.
>>
>> Ying
>>
>>
>>
>> On 9/17/19 11:42 AM, George McCabe via RT wrote:
>>> Hi Ying,
>>>
>>> There are new default values that need to be set in your
>>> PointStatConfig_gauges file that are required by the latest
version of
>> MET.
>>> You should have an entry that looks like this:
>>>
>>> message_type_group_map = [
>>> { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";
},
>>> { key = "ANYAIR"; val = "AIRCAR,AIRCFT";
},
>>> { key = "ANYSFC"; val =
"ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; },
>>> { key = "ONLYSF"; val = "ADPSFC,SFCSHP";
},
>>> { key = "LANDSF"; val = "ADPSFC,MSONET";
},
>>> { key = "WATERSF"; val = "SFCSHP";
}
>>> ];
>>>
>>> You can refer to the default config files (located in
share/met/config
>>> under the location MET is installed) to see what all of the
defaults are.
>>> You should add all of the default values to your config if they
are not
>>> already set.
>>>
>>> On Tue, Sep 17, 2019 at 1:19 AM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>
>>>> Hi George,
>>>>
>>>> OK, thanks, now we got pass that point. New err msg:
>>>>
>>>> ERROR :
>>>> ERROR : PointStatVxOpt::set_vx_pd() -> "message_type_group_map"
must
>>>> contain an entry for "LANDSF".
>>>> ERROR :
>>>>
>>>> Logs etc again in
>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
>>>>
>>>> Let's get back to this tomorrow.
>>>>
>>>> Ying
>>>>
>>>> On 9/16/19 8:53 PM, George McCabe via RT wrote:
>>>>> Hi Ying,
>>>>>
>>>>> In your PointStatConfig_gauges file, you need to remove the
quotes
>> around
>>>>> regrid = {
>>>>> to_grid = "${REGRID_TO_GRID}";
>>>>>
>>>>> it should be
>>>>>
>>>>> regrid = {
>>>>> to_grid = ${REGRID_TO_GRID};
>>>>>
>>>>> METplus checks if the value is NONE and adds quotes around it if
it is
>>>> not
>>>>> set to NONE.
>>>>>
>>>>> On Mon, Sep 16, 2019 at 6:13 PM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>>>
>>>>>> Hi George,
>>>>>>
>>>>>> Looks like
>>>>>>
>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>
>>>>>> fixed the issue with finding the gauge .nc file. Great!
>>>>>>
>>>>>> Setting the
>>>>>>
>>>>>> Now the issue remains is the PointStat err msg:
>>>>>>
>>>>>> DEBUG 1: Forecast File:
>>>>>>
>>>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
>>>>>> DEBUG 1: Observation File:
>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/
>>>>>> good-usa-dlyprcp-20190915.nc
>>>>>> DEBUG 2:
>>>>>> DEBUG 2:
>>>>>>
>>>>>>
>>
--------------------------------------------------------------------------------
>>>>>> DEBUG 2:
>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
>>>>>> ERROR :
>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
>>>>>> ERROR :
>>>>>>
>>>>>> I placed the latest configs, log etc in
>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
>>>>>> (older ones now under old/).
>>>>>>
>>>>>> Thanks again -
>>>>>>
>>>>>> Ying
>>>>>>
>>>>>> On 9/16/19 5:45 PM, George McCabe via RT wrote:
>>>>>>> I will take a look at the POLY issue. My guess is you need to
set the
>>>>>> poly
>>>>>>> variable using {DMASK} instead of ${DMASK} since it is
referencing
>> the
>>>>>>> METplus config variable. I will let you know what I find.
>>>>>>>
>>>>>>> On Mon, Sep 16, 2019 at 3:41 PM George McCabe
<mccabe at ucar.edu>
>> wrote:
>>>>>>>> Hi Ying,
>>>>>>>>
>>>>>>>> You should actually set:
>>>>>>>>
>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>>> instead of
>>>>>>>> OBS_FILE_WINDOW_BEGIN and END
>>>>>>>>
>>>>>>>> You SHOULD be able to set the latter, but there is a bug that
>> prevents
>>>>>>>> that from working. I think you will need OBS_WINDOW_BEGIN and
END to
>>>> be
>>>>>>>> -2700 to 2700 so that PointStat can find the correct data
inside the
>>>>>> files.
>>>>>>>> Does that make sense?
>>>>>>>>
>>>>>>>> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT
<met_help at ucar.edu>
>>>>>> wrote:
>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>>>>>>
>>>>>>>>> Hi George,
>>>>>>>>>
>>>>>>>>> I made a number of runs, here are three(to be
brief) at
>>>>>>>>> 20190916195402 (run1), 20190916203610 (run2) and
20190916210024
>>>> (run3):
>>>>>>>>> Various configs/logs in
>>>>>>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
>>>>>>>>> I've also placed the pcprtma.2019091512_A24h Run3 log
complained
>>>> about:
>>>>>>>>> In run1, I changed
>>>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>>> to
>>>>>>>>> OBS_WINDOW_BEGIN = 0
>>>>>>>>> OBS_WINDOW_END = 0
>>>>>>>>>
>>>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts
found the
>>>>>>>>> current gauge .nc file, but while running PointStat and
reading in
>>>>>>>>> /parm/PointStatConfig_gauges it complains of
>>>>>>>>> unexpected "$" in line 132 of config file
>>>>>>>>>
>>>>>>>>> Below is the offending line:
>>>>>>>>> //
>>>>>>>>> // Verification masking regions
>>>>>>>>> //
>>>>>>>>> mask = {
>>>>>>>>> grid = ${POINT_STAT_GRID};
>>>>>>>>> poly = ${POINT_STAT_POLY}; <=== line 132
>>>>>>>>> sid = [];
>>>>>>>>> llpnt = [];
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Log file shows that the "poly" files for the mask was found,
not
>> sure
>>>>>>>>> why line 132 was objectionable.
>>>>>>>>>
>>>>>>>>> In run2, in pcprtma_24h.conf I have
>>>>>>>>>
>>>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>>>
>>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>>>>>>> OBS_FILE_WINDOW_END = 0
>>>>>>>>>
>>>>>>>>> and I defined POINT_STAT_POLY in PointStatConfig_gauges
instead of
>> in
>>>>>>>>> pcprtma_24h.conf to get past the "error in line 132 in above
run1.
>>>>>>>>>
>>>>>>>>> Now I'm getting the same old error as last week:
>>>>>>>>> 09/16 20:36:27.051 metplus.PointStat
(point_stat_wrapper.py:194)
>>>> ERROR:
>>>>>>>>> Could not find observation file in
>>>>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc
matching
>>>>>> template
>>>>>>>>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
>>>>>>>>>
>>>>>>>>> Run 3: I went back to setting
>>>>>>>>> OBS_WINDOW_BEGIN = 0
>>>>>>>>> OBS_WINDOW_END = 0
>>>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
>>>>>>>>> as in Run1, and left the mask setting as in Run2. This time
the
>>>> script
>>>>>>>>> can find the gauge file, no complains about the mask poly
files,
>> but
>>>>>>>>> does not seem to be able to find the gridded 24h precip:
>>>>>>>>> DEBUG 2:
>>>>>>>>> DEBUG 2:
>>>>>>>>>
>>>>>>>>>
>>
--------------------------------------------------------------------------------
>>>>>>>>> DEBUG 2:
>>>>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
>>>>>>>>> ERROR :
>>>>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
>>>>>>>>> ERROR :
>>>>>>>>>
>>>>>>>>> So in brief:
>>>>>>>>>
>>>>>>>>> * PointStat cannot read the 24h accumulation in
>>>>>> pcprtma.2019091512_A24h
>>>>>>>>> * Script can only find the gauges file when
>>>>>> OBS_WINDOW_BEGIN[_END]are
>>>>>>>>> set to zero. This is undesirable since it narrows
the valid
>>>> time
>>>>>>>>> for each 24h gauge too much
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Ying
>>>>>>>>>
>>>>>>>>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
>>>>>>>>>> Hi Ying,
>>>>>>>>>>
>>>>>>>>>> I determined the cause of your error. It is a combination
of
>>>>>>>>>> mis-configuration and a bug. You have set:
>>>>>>>>>>
>>>>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>>>>
>>>>>>>>>> to specify the range of data to be allowed to be read in.
However,
>>>>>> thse
>>>>>>>>>> values also set the value of OBS_FILE_WINDOW_[BEGIN/END] if
those
>>>> are
>>>>>>>>> not
>>>>>>>>>> set. The _FILE_WINDOW_ variables determine the window of
times to
>>>> find
>>>>>>>>> the
>>>>>>>>>> files based on their filename in case the filename times
don't
>>>> exactly
>>>>>>>>> line
>>>>>>>>>> up with the valid time. You are supposed to be able to set
>>>>>>>>>>
>>>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>>>>>>>> OBS_FILE_WINDOW_END = 0
>>>>>>>>>>
>>>>>>>>>> to override the values. HOWEVER, there is a typo in the
code that
>>>>>>>>> handles
>>>>>>>>>> these variables and PointStat actually uses
OBS_WINDOW_[BEGIN/END]
>>>>>>>>> instead.
>>>>>>>>>> I will fix this bug, but in the meantime, you should be
able to
>> set
>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>>>>>
>>>>>>>>>> and it should work as intended. Let me know if that does
not work.
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via RT
<
>>>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
>>>>>>>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
>>>>>>>>>>> Queue: met_help
>>>>>>>>>>> Subject: python/v3.6.3 and METplus beta version;
Re:
>>>> init_seq
>>>>>>>>> works -
>>>>>>>>>>> thanks!
>>>>>>>>>>> Owner: mccabe
>>>>>>>>>>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
>>>>>>>>>>> Status: new
>>>>>>>>>>> Ticket <URL:
>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>>>>>>>>>>> This transaction appears to have no content
>>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ying Lin
>>>>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
>> Branch
>>>>>>>>> NCWCP Cubicle No. 2015
>>>>>>>>> Ying.Lin at noaa.gov
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> --
>>>>>> Ying Lin
>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>> NCWCP Cubicle No. 2015
>>>>>> Ying.Lin at noaa.gov
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: John Halley Gotway
Time: Wed Sep 18 16:38:47 2019
Ying and George,
So here's the ASCII point observations which are presumably being run
through ascii2nc:
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/good-
usa-dlyprcp-20190915_formatted
The QC_String is the second to last column and is set to a constant
value
of "NA". And running ncdump on the output of ascii2nc confirms that
to be
the case:
* ncdump -v obs_qty_table good-usa-dlyprcp-20190915.nc
<http://good-usa-dlyprcp-20190915.nc>*
...
obs_qty_table =
"NA" ;
...
Here's the Point-Stat config file you're using:
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/PointStatConfig_gauges
And in there you'll find that we're requesting obs with a quality
string of
1, 2, or 3:
* obs_quality = [ "1", "2", "3" ];*
You can fix this behavior either by...
(1) Including "NA" in the obs_quality list: obs_quality = [ "1",
"2",
"3", "NA" ];
(2) Or by changing the QC value in the ASCII obs from "NA" to a value
of 1,
2, or 3.
Hope that does it.
Thanks,
John
On Wed, Sep 18, 2019 at 3:43 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>
> Hi George,
>
> Good stuff, the COPYABLE ENVIRONMENT FOR NEXT COMMAND. So I
did as
> you said, and the reason for not having any matching pairs is that
all
> observations were rejected due to quality marker
> (
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/master_metplus.log.test).
>
> Initially I thought it had something to do with all entries having
the
> elev_now and height_now (in precip_ascii2nc.py, same ftp dir) set to
> '-9999.' (as in
>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/good-
usa-dlyprcp-20190915_formatted)
>
> so I went and changed both of these columns to '0.0', since they're
not
> used in the verification anyway; and I also decided to have any
negative
> precip values (-9999., or an unintentional negative precip amount
> report) in the input to be encoded as '-9999.' in the
> good-usa-dlyprcp-20190915.nc, and there are 69 such reports.
>
> It didn't make a difference though. master_metplus.log.test
shows
> that
> DEBUG 3: Observations processed = 7646
> ....
> DEBUG 3: Rejected: quality marker = 7646
>
> Can you tell by looking at good-usa-dlyprcp-20190915_formatted what
> makes every entry fail the quality marker? Maybe MET8.1 requires a
> different format for the point output. I originally got the
> precip_ascii2nc.py from Mallory more than a year ago. Output from
it
> (even with height and elev both set to '-9999.') works for MET7.0
>
> Thanks again -
>
> Ying
>
> On 9/18/19 12:05 PM, George McCabe via RT wrote:
> > Hi Ying,
> >
> > It is hard to say why this run didn't work without comparing the
> PointStat
> > config file you used last time to what is set here. I would
recommend
> > rerunning point_stat with verbosity increased to 3 so you can see
the
> > reasons that the matched pairs were rejected. To do this, copy all
of the
> > export commands from the line after COPYABLE ENVIRONMENT FOR NEXT
> COMMAND:
> > and paste that into a command prompt. Then copy the point_stat
command
> but
> > add '-v 3' to it. The output will now show how the matches pairs
were
> > rejected, which is already shown in your old output. Compare the
logs to
> > see why they are being rejected. This will give you a clue as to
what is
> > different in your configuration between runs.
> >
> > On Wed, Sep 18, 2019 at 9:41 AM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>
> >> Thanks George, I see that my PointStatConfig was missing the
settings
> >> for "LANDSF" and "WATERSF" in the message_type_group_map. Now
the test
> >> no longer has the err msgs.
> >>
> >> However, the point_stat step did not seem to have found any
matches -
> >> the point_stat_*.txt and *.stat files all had only the header
line: see
> >>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/
> >>
> >> A few weeks ago I ran the pcpRTMA vs. gauges in MET7.0 (couldn't
get it
> >> to run in MET8.1 at the time, probably due to the undefined
default
> >> values as you mentioned in the email below), and that ran fine
and
> >> produced reasonable results. Output: point_stat_met7.0.out
> >>
> >> Thanks again for looking into this.
> >>
> >> Ying
> >>
> >>
> >>
> >> On 9/17/19 11:42 AM, George McCabe via RT wrote:
> >>> Hi Ying,
> >>>
> >>> There are new default values that need to be set in your
> >>> PointStatConfig_gauges file that are required by the latest
version of
> >> MET.
> >>> You should have an entry that looks like this:
> >>>
> >>> message_type_group_map = [
> >>> { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";
},
> >>> { key = "ANYAIR"; val = "AIRCAR,AIRCFT";
},
> >>> { key = "ANYSFC"; val =
"ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; },
> >>> { key = "ONLYSF"; val = "ADPSFC,SFCSHP";
},
> >>> { key = "LANDSF"; val = "ADPSFC,MSONET";
},
> >>> { key = "WATERSF"; val = "SFCSHP";
}
> >>> ];
> >>>
> >>> You can refer to the default config files (located in
share/met/config
> >>> under the location MET is installed) to see what all of the
defaults
> are.
> >>> You should add all of the default values to your config if they
are not
> >>> already set.
> >>>
> >>> On Tue, Sep 17, 2019 at 1:19 AM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>>>
> >>>> Hi George,
> >>>>
> >>>> OK, thanks, now we got pass that point. New err msg:
> >>>>
> >>>> ERROR :
> >>>> ERROR : PointStatVxOpt::set_vx_pd() ->
"message_type_group_map" must
> >>>> contain an entry for "LANDSF".
> >>>> ERROR :
> >>>>
> >>>> Logs etc again in
> >>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
> >>>>
> >>>> Let's get back to this tomorrow.
> >>>>
> >>>> Ying
> >>>>
> >>>> On 9/16/19 8:53 PM, George McCabe via RT wrote:
> >>>>> Hi Ying,
> >>>>>
> >>>>> In your PointStatConfig_gauges file, you need to remove the
quotes
> >> around
> >>>>> regrid = {
> >>>>> to_grid = "${REGRID_TO_GRID}";
> >>>>>
> >>>>> it should be
> >>>>>
> >>>>> regrid = {
> >>>>> to_grid = ${REGRID_TO_GRID};
> >>>>>
> >>>>> METplus checks if the value is NONE and adds quotes around it
if it
> is
> >>>> not
> >>>>> set to NONE.
> >>>>>
> >>>>> On Mon, Sep 16, 2019 at 6:13 PM Ying Lin via RT
<met_help at ucar.edu>
> >>>> wrote:
> >>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>
> >>>>>>
> >>>>>> Hi George,
> >>>>>>
> >>>>>> Looks like
> >>>>>>
> >>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>>>
> >>>>>> fixed the issue with finding the gauge .nc file. Great!
> >>>>>>
> >>>>>> Setting the
> >>>>>>
> >>>>>> Now the issue remains is the PointStat err msg:
> >>>>>>
> >>>>>> DEBUG 1: Forecast File:
> >>>>>>
> >>>>>>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
> >>>>>> DEBUG 1: Observation File:
> >>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/
> >>>>>> good-usa-dlyprcp-20190915.nc
> >>>>>> DEBUG 2:
> >>>>>> DEBUG 2:
> >>>>>>
> >>>>>>
> >>
>
--------------------------------------------------------------------------------
> >>>>>> DEBUG 2:
> >>>>>> DEBUG 2: Reading data for APCP_24(*,*).
> >>>>>> ERROR :
> >>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
> >>>>>> ERROR :
> >>>>>>
> >>>>>> I placed the latest configs, log etc in
> >>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
> >>>>>> (older ones now under old/).
> >>>>>>
> >>>>>> Thanks again -
> >>>>>>
> >>>>>> Ying
> >>>>>>
> >>>>>> On 9/16/19 5:45 PM, George McCabe via RT wrote:
> >>>>>>> I will take a look at the POLY issue. My guess is you need
to set
> the
> >>>>>> poly
> >>>>>>> variable using {DMASK} instead of ${DMASK} since it is
referencing
> >> the
> >>>>>>> METplus config variable. I will let you know what I find.
> >>>>>>>
> >>>>>>> On Mon, Sep 16, 2019 at 3:41 PM George McCabe
<mccabe at ucar.edu>
> >> wrote:
> >>>>>>>> Hi Ying,
> >>>>>>>>
> >>>>>>>> You should actually set:
> >>>>>>>>
> >>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>>>>> instead of
> >>>>>>>> OBS_FILE_WINDOW_BEGIN and END
> >>>>>>>>
> >>>>>>>> You SHOULD be able to set the latter, but there is a bug
that
> >> prevents
> >>>>>>>> that from working. I think you will need OBS_WINDOW_BEGIN
and END
> to
> >>>> be
> >>>>>>>> -2700 to 2700 so that PointStat can find the correct data
inside
> the
> >>>>>> files.
> >>>>>>>> Does that make sense?
> >>>>>>>>
> >>>>>>>> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT <
> met_help at ucar.edu>
> >>>>>> wrote:
> >>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>>>>>>>>
> >>>>>>>>> Hi George,
> >>>>>>>>>
> >>>>>>>>> I made a number of runs, here are three(to be
brief) at
> >>>>>>>>> 20190916195402 (run1), 20190916203610 (run2) and
20190916210024
> >>>> (run3):
> >>>>>>>>> Various configs/logs in
> >>>>>>>>>
> >>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
> >>>>>>>>> I've also placed the pcprtma.2019091512_A24h Run3 log
complained
> >>>> about:
> >>>>>>>>> In run1, I changed
> >>>>>>>>> OBS_WINDOW_BEGIN = -2700
> >>>>>>>>> OBS_WINDOW_END = 2700
> >>>>>>>>> to
> >>>>>>>>> OBS_WINDOW_BEGIN = 0
> >>>>>>>>> OBS_WINDOW_END = 0
> >>>>>>>>>
> >>>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts
found
> the
> >>>>>>>>> current gauge .nc file, but while running PointStat and
reading
> in
> >>>>>>>>> /parm/PointStatConfig_gauges it complains of
> >>>>>>>>> unexpected "$" in line 132 of config file
> >>>>>>>>>
> >>>>>>>>> Below is the offending line:
> >>>>>>>>> //
> >>>>>>>>> // Verification masking regions
> >>>>>>>>> //
> >>>>>>>>> mask = {
> >>>>>>>>> grid = ${POINT_STAT_GRID};
> >>>>>>>>> poly = ${POINT_STAT_POLY}; <=== line 132
> >>>>>>>>> sid = [];
> >>>>>>>>> llpnt = [];
> >>>>>>>>> }
> >>>>>>>>>
> >>>>>>>>> Log file shows that the "poly" files for the mask was
found, not
> >> sure
> >>>>>>>>> why line 132 was objectionable.
> >>>>>>>>>
> >>>>>>>>> In run2, in pcprtma_24h.conf I have
> >>>>>>>>>
> >>>>>>>>> OBS_WINDOW_BEGIN = -2700
> >>>>>>>>> OBS_WINDOW_END = 2700
> >>>>>>>>>
> >>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
> >>>>>>>>> OBS_FILE_WINDOW_END = 0
> >>>>>>>>>
> >>>>>>>>> and I defined POINT_STAT_POLY in PointStatConfig_gauges
instead
> of
> >> in
> >>>>>>>>> pcprtma_24h.conf to get past the "error in line 132 in
above
> run1.
> >>>>>>>>>
> >>>>>>>>> Now I'm getting the same old error as last week:
> >>>>>>>>> 09/16 20:36:27.051 metplus.PointStat
(point_stat_wrapper.py:194)
> >>>> ERROR:
> >>>>>>>>> Could not find observation file in
> >>>>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc
matching
> >>>>>> template
> >>>>>>>>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
> >>>>>>>>>
> >>>>>>>>> Run 3: I went back to setting
> >>>>>>>>> OBS_WINDOW_BEGIN = 0
> >>>>>>>>> OBS_WINDOW_END = 0
> >>>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
> >>>>>>>>> as in Run1, and left the mask setting as in Run2. This
time the
> >>>> script
> >>>>>>>>> can find the gauge file, no complains about the mask poly
files,
> >> but
> >>>>>>>>> does not seem to be able to find the gridded 24h precip:
> >>>>>>>>> DEBUG 2:
> >>>>>>>>> DEBUG 2:
> >>>>>>>>>
> >>>>>>>>>
> >>
>
--------------------------------------------------------------------------------
> >>>>>>>>> DEBUG 2:
> >>>>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
> >>>>>>>>> ERROR :
> >>>>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
> >>>>>>>>> ERROR :
> >>>>>>>>>
> >>>>>>>>> So in brief:
> >>>>>>>>>
> >>>>>>>>> * PointStat cannot read the 24h accumulation in
> >>>>>> pcprtma.2019091512_A24h
> >>>>>>>>> * Script can only find the gauges file when
> >>>>>> OBS_WINDOW_BEGIN[_END]are
> >>>>>>>>> set to zero. This is undesirable since it narrows
the
> valid
> >>>> time
> >>>>>>>>> for each 24h gauge too much
> >>>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>> Ying
> >>>>>>>>>
> >>>>>>>>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
> >>>>>>>>>> Hi Ying,
> >>>>>>>>>>
> >>>>>>>>>> I determined the cause of your error. It is a combination
of
> >>>>>>>>>> mis-configuration and a bug. You have set:
> >>>>>>>>>>
> >>>>>>>>>> OBS_WINDOW_BEGIN = -2700
> >>>>>>>>>> OBS_WINDOW_END = 2700
> >>>>>>>>>>
> >>>>>>>>>> to specify the range of data to be allowed to be read in.
> However,
> >>>>>> thse
> >>>>>>>>>> values also set the value of OBS_FILE_WINDOW_[BEGIN/END]
if
> those
> >>>> are
> >>>>>>>>> not
> >>>>>>>>>> set. The _FILE_WINDOW_ variables determine the window of
times
> to
> >>>> find
> >>>>>>>>> the
> >>>>>>>>>> files based on their filename in case the filename times
don't
> >>>> exactly
> >>>>>>>>> line
> >>>>>>>>>> up with the valid time. You are supposed to be able to
set
> >>>>>>>>>>
> >>>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
> >>>>>>>>>> OBS_FILE_WINDOW_END = 0
> >>>>>>>>>>
> >>>>>>>>>> to override the values. HOWEVER, there is a typo in the
code
> that
> >>>>>>>>> handles
> >>>>>>>>>> these variables and PointStat actually uses
> OBS_WINDOW_[BEGIN/END]
> >>>>>>>>> instead.
> >>>>>>>>>> I will fix this bug, but in the meantime, you should be
able to
> >> set
> >>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>>>>>>>
> >>>>>>>>>> and it should work as intended. Let me know if that does
not
> work.
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via
RT <
> >>>>>>>>>> met_help at ucar.edu> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
> >>>>>>>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
> >>>>>>>>>>> Queue: met_help
> >>>>>>>>>>> Subject: python/v3.6.3 and METplus beta
version; Re:
> >>>> init_seq
> >>>>>>>>> works -
> >>>>>>>>>>> thanks!
> >>>>>>>>>>> Owner: mccabe
> >>>>>>>>>>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
> >>>>>>>>>>> Status: new
> >>>>>>>>>>> Ticket <URL:
> >>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
> >>>>>>>>>>> This transaction appears to have no content
> >>>>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Ying Lin
> >>>>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
> >> Branch
> >>>>>>>>> NCWCP Cubicle No. 2015
> >>>>>>>>> Ying.Lin at noaa.gov
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> --
> >>>>>> Ying Lin
> >>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>>>> NCWCP Cubicle No. 2015
> >>>>>> Ying.Lin at noaa.gov
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>> --
> >>>> Ying Lin
> >>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>> NCWCP Cubicle No. 2015
> >>>> Ying.Lin at noaa.gov
> >>>>
> >>>>
> >>>>
> >>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: Ying Lin
Time: Wed Sep 18 19:06:56 2019
Thank you John, setting
obs_quality = [ "1", "2","3", "NA" ];
seems to have worked.
Question: in the PointStat input (*.nc; last coloum in *_formatted),
does a value of '-9999.' denote "missing/bad obs value"?
In my gauge data, there are 69 entries that has "-9999." for the
precip
amount and I meant for it to be "missing" (the raw data have mostly
"-9999." for these 69, and one of them has a small negative amount, I
set all the negative amounts to '-9999.'). In the job output, there
are
DEBUG 3: Rejected: valid time = 202
DEBUG 3: Rejected: bad obs value = 57
Looks like some of the 69 entries with "-9999." also had valid time
more
than an hour (+/-3600 seconds) away from 12:00Z, so the above looks
reasonable and I'm guessing "-9999." does mean "missing" to PointStat.
Ying
On 9/18/19 6:38 PM, John Halley Gotway via RT wrote:
> Ying and George,
>
> So here's the ASCII point observations which are presumably being
run
> through ascii2nc:
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/good-
usa-dlyprcp-20190915_formatted
>
> The QC_String is the second to last column and is set to a constant
value
> of "NA". And running ncdump on the output of ascii2nc confirms that
to be
> the case:
>
> * ncdump -v obs_qty_table good-usa-dlyprcp-20190915.nc
> <http://good-usa-dlyprcp-20190915.nc>*
> ...
> obs_qty_table =
> "NA" ;
> ...
>
> Here's the Point-Stat config file you're using:
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/PointStatConfig_gauges
>
> And in there you'll find that we're requesting obs with a quality
string of
> 1, 2, or 3:
> * obs_quality = [ "1", "2", "3" ];*
>
> You can fix this behavior either by...
> (1) Including "NA" in the obs_quality list: obs_quality = [
"1", "2",
> "3", "NA" ];
> (2) Or by changing the QC value in the ASCII obs from "NA" to a
value of 1,
> 2, or 3.
>
> Hope that does it.
>
> Thanks,
> John
>
>
> On Wed, Sep 18, 2019 at 3:43 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>
>> Hi George,
>>
>> Good stuff, the COPYABLE ENVIRONMENT FOR NEXT COMMAND. So I
did as
>> you said, and the reason for not having any matching pairs is that
all
>> observations were rejected due to quality marker
>> (
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/master_metplus.log.test).
>>
>> Initially I thought it had something to do with all entries having
the
>> elev_now and height_now (in precip_ascii2nc.py, same ftp dir) set
to
>> '-9999.' (as in
>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/good-
usa-dlyprcp-20190915_formatted)
>>
>> so I went and changed both of these columns to '0.0', since they're
not
>> used in the verification anyway; and I also decided to have any
negative
>> precip values (-9999., or an unintentional negative precip amount
>> report) in the input to be encoded as '-9999.' in the
>> good-usa-dlyprcp-20190915.nc, and there are 69 such reports.
>>
>> It didn't make a difference though. master_metplus.log.test
shows
>> that
>> DEBUG 3: Observations processed = 7646
>> ....
>> DEBUG 3: Rejected: quality marker = 7646
>>
>> Can you tell by looking at good-usa-dlyprcp-20190915_formatted what
>> makes every entry fail the quality marker? Maybe MET8.1 requires a
>> different format for the point output. I originally got the
>> precip_ascii2nc.py from Mallory more than a year ago. Output from
it
>> (even with height and elev both set to '-9999.') works for MET7.0
>>
>> Thanks again -
>>
>> Ying
>>
>> On 9/18/19 12:05 PM, George McCabe via RT wrote:
>>> Hi Ying,
>>>
>>> It is hard to say why this run didn't work without comparing the
>> PointStat
>>> config file you used last time to what is set here. I would
recommend
>>> rerunning point_stat with verbosity increased to 3 so you can see
the
>>> reasons that the matched pairs were rejected. To do this, copy all
of the
>>> export commands from the line after COPYABLE ENVIRONMENT FOR NEXT
>> COMMAND:
>>> and paste that into a command prompt. Then copy the point_stat
command
>> but
>>> add '-v 3' to it. The output will now show how the matches pairs
were
>>> rejected, which is already shown in your old output. Compare the
logs to
>>> see why they are being rejected. This will give you a clue as to
what is
>>> different in your configuration between runs.
>>>
>>> On Wed, Sep 18, 2019 at 9:41 AM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>
>>>> Thanks George, I see that my PointStatConfig was missing the
settings
>>>> for "LANDSF" and "WATERSF" in the message_type_group_map. Now
the test
>>>> no longer has the err msgs.
>>>>
>>>> However, the point_stat step did not seem to have found any
matches -
>>>> the point_stat_*.txt and *.stat files all had only the header
line: see
>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/
>>>>
>>>> A few weeks ago I ran the pcpRTMA vs. gauges in MET7.0 (couldn't
get it
>>>> to run in MET8.1 at the time, probably due to the undefined
default
>>>> values as you mentioned in the email below), and that ran fine
and
>>>> produced reasonable results. Output: point_stat_met7.0.out
>>>>
>>>> Thanks again for looking into this.
>>>>
>>>> Ying
>>>>
>>>>
>>>>
>>>> On 9/17/19 11:42 AM, George McCabe via RT wrote:
>>>>> Hi Ying,
>>>>>
>>>>> There are new default values that need to be set in your
>>>>> PointStatConfig_gauges file that are required by the latest
version of
>>>> MET.
>>>>> You should have an entry that looks like this:
>>>>>
>>>>> message_type_group_map = [
>>>>> { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";
},
>>>>> { key = "ANYAIR"; val = "AIRCAR,AIRCFT";
},
>>>>> { key = "ANYSFC"; val =
"ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; },
>>>>> { key = "ONLYSF"; val = "ADPSFC,SFCSHP";
},
>>>>> { key = "LANDSF"; val = "ADPSFC,MSONET";
},
>>>>> { key = "WATERSF"; val = "SFCSHP";
}
>>>>> ];
>>>>>
>>>>> You can refer to the default config files (located in
share/met/config
>>>>> under the location MET is installed) to see what all of the
defaults
>> are.
>>>>> You should add all of the default values to your config if they
are not
>>>>> already set.
>>>>>
>>>>> On Tue, Sep 17, 2019 at 1:19 AM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>>>
>>>>>> Hi George,
>>>>>>
>>>>>> OK, thanks, now we got pass that point. New err msg:
>>>>>>
>>>>>> ERROR :
>>>>>> ERROR : PointStatVxOpt::set_vx_pd() ->
"message_type_group_map" must
>>>>>> contain an entry for "LANDSF".
>>>>>> ERROR :
>>>>>>
>>>>>> Logs etc again in
>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
>>>>>>
>>>>>> Let's get back to this tomorrow.
>>>>>>
>>>>>> Ying
>>>>>>
>>>>>> On 9/16/19 8:53 PM, George McCabe via RT wrote:
>>>>>>> Hi Ying,
>>>>>>>
>>>>>>> In your PointStatConfig_gauges file, you need to remove the
quotes
>>>> around
>>>>>>> regrid = {
>>>>>>> to_grid = "${REGRID_TO_GRID}";
>>>>>>>
>>>>>>> it should be
>>>>>>>
>>>>>>> regrid = {
>>>>>>> to_grid = ${REGRID_TO_GRID};
>>>>>>>
>>>>>>> METplus checks if the value is NONE and adds quotes around it
if it
>> is
>>>>>> not
>>>>>>> set to NONE.
>>>>>>>
>>>>>>> On Mon, Sep 16, 2019 at 6:13 PM Ying Lin via RT
<met_help at ucar.edu>
>>>>>> wrote:
>>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>
>>>>>>>>
>>>>>>>> Hi George,
>>>>>>>>
>>>>>>>> Looks like
>>>>>>>>
>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>>>
>>>>>>>> fixed the issue with finding the gauge .nc file. Great!
>>>>>>>>
>>>>>>>> Setting the
>>>>>>>>
>>>>>>>> Now the issue remains is the PointStat err msg:
>>>>>>>>
>>>>>>>> DEBUG 1: Forecast File:
>>>>>>>>
>>>>>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
>>>>>>>> DEBUG 1: Observation File:
>>>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/
>>>>>>>> good-usa-dlyprcp-20190915.nc
>>>>>>>> DEBUG 2:
>>>>>>>> DEBUG 2:
>>>>>>>>
>>>>>>>>
>>
--------------------------------------------------------------------------------
>>>>>>>> DEBUG 2:
>>>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
>>>>>>>> ERROR :
>>>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
>>>>>>>> ERROR :
>>>>>>>>
>>>>>>>> I placed the latest configs, log etc in
>>>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
>>>>>>>> (older ones now under old/).
>>>>>>>>
>>>>>>>> Thanks again -
>>>>>>>>
>>>>>>>> Ying
>>>>>>>>
>>>>>>>> On 9/16/19 5:45 PM, George McCabe via RT wrote:
>>>>>>>>> I will take a look at the POLY issue. My guess is you need
to set
>> the
>>>>>>>> poly
>>>>>>>>> variable using {DMASK} instead of ${DMASK} since it is
referencing
>>>> the
>>>>>>>>> METplus config variable. I will let you know what I find.
>>>>>>>>>
>>>>>>>>> On Mon, Sep 16, 2019 at 3:41 PM George McCabe
<mccabe at ucar.edu>
>>>> wrote:
>>>>>>>>>> Hi Ying,
>>>>>>>>>>
>>>>>>>>>> You should actually set:
>>>>>>>>>>
>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>>>>> instead of
>>>>>>>>>> OBS_FILE_WINDOW_BEGIN and END
>>>>>>>>>>
>>>>>>>>>> You SHOULD be able to set the latter, but there is a bug
that
>>>> prevents
>>>>>>>>>> that from working. I think you will need OBS_WINDOW_BEGIN
and END
>> to
>>>>>> be
>>>>>>>>>> -2700 to 2700 so that PointStat can find the correct data
inside
>> the
>>>>>>>> files.
>>>>>>>>>> Does that make sense?
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT <
>> met_help at ucar.edu>
>>>>>>>> wrote:
>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>>>>>>>>
>>>>>>>>>>> Hi George,
>>>>>>>>>>>
>>>>>>>>>>> I made a number of runs, here are three(to be
brief) at
>>>>>>>>>>> 20190916195402 (run1), 20190916203610 (run2) and
20190916210024
>>>>>> (run3):
>>>>>>>>>>> Various configs/logs in
>>>>>>>>>>>
>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
>>>>>>>>>>> I've also placed the pcprtma.2019091512_A24h Run3 log
complained
>>>>>> about:
>>>>>>>>>>> In run1, I changed
>>>>>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>>>>> to
>>>>>>>>>>> OBS_WINDOW_BEGIN = 0
>>>>>>>>>>> OBS_WINDOW_END = 0
>>>>>>>>>>>
>>>>>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts
found
>> the
>>>>>>>>>>> current gauge .nc file, but while running PointStat and
reading
>> in
>>>>>>>>>>> /parm/PointStatConfig_gauges it complains of
>>>>>>>>>>> unexpected "$" in line 132 of config file
>>>>>>>>>>>
>>>>>>>>>>> Below is the offending line:
>>>>>>>>>>> //
>>>>>>>>>>> // Verification masking regions
>>>>>>>>>>> //
>>>>>>>>>>> mask = {
>>>>>>>>>>> grid = ${POINT_STAT_GRID};
>>>>>>>>>>> poly = ${POINT_STAT_POLY}; <=== line 132
>>>>>>>>>>> sid = [];
>>>>>>>>>>> llpnt = [];
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> Log file shows that the "poly" files for the mask was
found, not
>>>> sure
>>>>>>>>>>> why line 132 was objectionable.
>>>>>>>>>>>
>>>>>>>>>>> In run2, in pcprtma_24h.conf I have
>>>>>>>>>>>
>>>>>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>>>>>
>>>>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>>>>>>>>> OBS_FILE_WINDOW_END = 0
>>>>>>>>>>>
>>>>>>>>>>> and I defined POINT_STAT_POLY in PointStatConfig_gauges
instead
>> of
>>>> in
>>>>>>>>>>> pcprtma_24h.conf to get past the "error in line 132 in
above
>> run1.
>>>>>>>>>>> Now I'm getting the same old error as last week:
>>>>>>>>>>> 09/16 20:36:27.051 metplus.PointStat
(point_stat_wrapper.py:194)
>>>>>> ERROR:
>>>>>>>>>>> Could not find observation file in
>>>>>>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc
matching
>>>>>>>> template
>>>>>>>>>>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
>>>>>>>>>>>
>>>>>>>>>>> Run 3: I went back to setting
>>>>>>>>>>> OBS_WINDOW_BEGIN = 0
>>>>>>>>>>> OBS_WINDOW_END = 0
>>>>>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
>>>>>>>>>>> as in Run1, and left the mask setting as in Run2. This
time the
>>>>>> script
>>>>>>>>>>> can find the gauge file, no complains about the mask poly
files,
>>>> but
>>>>>>>>>>> does not seem to be able to find the gridded 24h precip:
>>>>>>>>>>> DEBUG 2:
>>>>>>>>>>> DEBUG 2:
>>>>>>>>>>>
>>>>>>>>>>>
>>
--------------------------------------------------------------------------------
>>>>>>>>>>> DEBUG 2:
>>>>>>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
>>>>>>>>>>> ERROR :
>>>>>>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
>>>>>>>>>>> ERROR :
>>>>>>>>>>>
>>>>>>>>>>> So in brief:
>>>>>>>>>>>
>>>>>>>>>>> * PointStat cannot read the 24h accumulation in
>>>>>>>> pcprtma.2019091512_A24h
>>>>>>>>>>> * Script can only find the gauges file when
>>>>>>>> OBS_WINDOW_BEGIN[_END]are
>>>>>>>>>>> set to zero. This is undesirable since it
narrows the
>> valid
>>>>>> time
>>>>>>>>>>> for each 24h gauge too much
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Ying
>>>>>>>>>>>
>>>>>>>>>>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
>>>>>>>>>>>> Hi Ying,
>>>>>>>>>>>>
>>>>>>>>>>>> I determined the cause of your error. It is a combination
of
>>>>>>>>>>>> mis-configuration and a bug. You have set:
>>>>>>>>>>>>
>>>>>>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>>>>>>
>>>>>>>>>>>> to specify the range of data to be allowed to be read in.
>> However,
>>>>>>>> thse
>>>>>>>>>>>> values also set the value of OBS_FILE_WINDOW_[BEGIN/END]
if
>> those
>>>>>> are
>>>>>>>>>>> not
>>>>>>>>>>>> set. The _FILE_WINDOW_ variables determine the window of
times
>> to
>>>>>> find
>>>>>>>>>>> the
>>>>>>>>>>>> files based on their filename in case the filename times
don't
>>>>>> exactly
>>>>>>>>>>> line
>>>>>>>>>>>> up with the valid time. You are supposed to be able to
set
>>>>>>>>>>>>
>>>>>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>>>>>>>>>> OBS_FILE_WINDOW_END = 0
>>>>>>>>>>>>
>>>>>>>>>>>> to override the values. HOWEVER, there is a typo in the
code
>> that
>>>>>>>>>>> handles
>>>>>>>>>>>> these variables and PointStat actually uses
>> OBS_WINDOW_[BEGIN/END]
>>>>>>>>>>> instead.
>>>>>>>>>>>> I will fix this bug, but in the meantime, you should be
able to
>>>> set
>>>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>>>>>>>
>>>>>>>>>>>> and it should work as intended. Let me know if that does
not
>> work.
>>>>>>>>>>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via
RT <
>>>>>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted upon.
>>>>>>>>>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
>>>>>>>>>>>>> Queue: met_help
>>>>>>>>>>>>> Subject: python/v3.6.3 and METplus beta
version; Re:
>>>>>> init_seq
>>>>>>>>>>> works -
>>>>>>>>>>>>> thanks!
>>>>>>>>>>>>> Owner: mccabe
>>>>>>>>>>>>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
>>>>>>>>>>>>> Status: new
>>>>>>>>>>>>> Ticket <URL:
>>>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>>>>>>>>>>>>> This transaction appears to have no content
>>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ying Lin
>>>>>>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
>>>> Branch
>>>>>>>>>>> NCWCP Cubicle No. 2015
>>>>>>>>>>> Ying.Lin at noaa.gov
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> --
>>>>>>>> Ying Lin
>>>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>>>> NCWCP Cubicle No. 2015
>>>>>>>> Ying.Lin at noaa.gov
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> --
>>>>>> Ying Lin
>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>> NCWCP Cubicle No. 2015
>>>>>> Ying.Lin at noaa.gov
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: John Halley Gotway
Time: Thu Sep 19 09:41:08 2019
Ying,
Yes, that's correct. -9999 is the missing data value used in MET, and
your
explanation makes sense to me.
But let's say at some point in the future you have point observations
of
precip with slightly negative values that you do want to ignore. You
can
do this in the Point-Stat config file using the censor_thresh and
censor_val options:
obs = {
field = [ { name="APCP"; level="A24"; censor_thresh = [ <0 ];
censor_val
= [ -9999 ]; } ];
}
That tells Point-Stat to replace any negative observations with the
bad
data value. So there's multiple ways of addressing this issue.
Thanks,
John
On Wed, Sep 18, 2019 at 7:07 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>
> Thank you John, setting
>
> obs_quality = [ "1", "2","3", "NA" ];
>
> seems to have worked.
>
> Question: in the PointStat input (*.nc; last coloum in
*_formatted),
> does a value of '-9999.' denote "missing/bad obs value"?
>
> In my gauge data, there are 69 entries that has "-9999." for the
precip
> amount and I meant for it to be "missing" (the raw data have mostly
> "-9999." for these 69, and one of them has a small negative amount,
I
> set all the negative amounts to '-9999.'). In the job output, there
are
>
> DEBUG 3: Rejected: valid time = 202
> DEBUG 3: Rejected: bad obs value = 57
>
> Looks like some of the 69 entries with "-9999." also had valid time
more
> than an hour (+/-3600 seconds) away from 12:00Z, so the above looks
> reasonable and I'm guessing "-9999." does mean "missing" to
PointStat.
>
> Ying
>
> On 9/18/19 6:38 PM, John Halley Gotway via RT wrote:
> > Ying and George,
> >
> > So here's the ASCII point observations which are presumably being
run
> > through ascii2nc:
> >
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/good-
usa-dlyprcp-20190915_formatted
> >
> > The QC_String is the second to last column and is set to a
constant value
> > of "NA". And running ncdump on the output of ascii2nc confirms
that to
> be
> > the case:
> >
> > * ncdump -v obs_qty_table good-usa-dlyprcp-20190915.nc
> > <http://good-usa-dlyprcp-20190915.nc>*
> > ...
> > obs_qty_table =
> > "NA" ;
> > ...
> >
> > Here's the Point-Stat config file you're using:
> >
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/PointStatConfig_gauges
> >
> > And in there you'll find that we're requesting obs with a quality
string
> of
> > 1, 2, or 3:
> > * obs_quality = [ "1", "2", "3" ];*
> >
> > You can fix this behavior either by...
> > (1) Including "NA" in the obs_quality list: obs_quality = [
"1",
> "2",
> > "3", "NA" ];
> > (2) Or by changing the QC value in the ASCII obs from "NA" to a
value of
> 1,
> > 2, or 3.
> >
> > Hope that does it.
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Sep 18, 2019 at 3:43 PM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>
> >> Hi George,
> >>
> >> Good stuff, the COPYABLE ENVIRONMENT FOR NEXT COMMAND. So
I did
> as
> >> you said, and the reason for not having any matching pairs is
that all
> >> observations were rejected due to quality marker
> >> (
> >>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/master_metplus.log.test
> ).
> >>
> >> Initially I thought it had something to do with all entries
having the
> >> elev_now and height_now (in precip_ascii2nc.py, same ftp dir) set
to
> >> '-9999.' (as in
> >>
> >>
>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/good-
usa-dlyprcp-20190915_formatted
> )
> >>
> >> so I went and changed both of these columns to '0.0', since
they're not
> >> used in the verification anyway; and I also decided to have any
negative
> >> precip values (-9999., or an unintentional negative precip amount
> >> report) in the input to be encoded as '-9999.' in the
> >> good-usa-dlyprcp-20190915.nc, and there are 69 such reports.
> >>
> >> It didn't make a difference though.
master_metplus.log.test shows
> >> that
> >> DEBUG 3: Observations processed = 7646
> >> ....
> >> DEBUG 3: Rejected: quality marker = 7646
> >>
> >> Can you tell by looking at good-usa-dlyprcp-20190915_formatted
what
> >> makes every entry fail the quality marker? Maybe MET8.1 requires
a
> >> different format for the point output. I originally got the
> >> precip_ascii2nc.py from Mallory more than a year ago. Output
from it
> >> (even with height and elev both set to '-9999.') works for MET7.0
> >>
> >> Thanks again -
> >>
> >> Ying
> >>
> >> On 9/18/19 12:05 PM, George McCabe via RT wrote:
> >>> Hi Ying,
> >>>
> >>> It is hard to say why this run didn't work without comparing the
> >> PointStat
> >>> config file you used last time to what is set here. I would
recommend
> >>> rerunning point_stat with verbosity increased to 3 so you can
see the
> >>> reasons that the matched pairs were rejected. To do this, copy
all of
> the
> >>> export commands from the line after COPYABLE ENVIRONMENT FOR
NEXT
> >> COMMAND:
> >>> and paste that into a command prompt. Then copy the point_stat
command
> >> but
> >>> add '-v 3' to it. The output will now show how the matches pairs
were
> >>> rejected, which is already shown in your old output. Compare the
logs
> to
> >>> see why they are being rejected. This will give you a clue as to
what
> is
> >>> different in your configuration between runs.
> >>>
> >>> On Wed, Sep 18, 2019 at 9:41 AM Ying Lin via RT
<met_help at ucar.edu>
> >> wrote:
> >>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>>>
> >>>> Thanks George, I see that my PointStatConfig was missing the
settings
> >>>> for "LANDSF" and "WATERSF" in the message_type_group_map. Now
the
> test
> >>>> no longer has the err msgs.
> >>>>
> >>>> However, the point_stat step did not seem to have found any
matches -
> >>>> the point_stat_*.txt and *.stat files all had only the header
line:
> see
> >>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/
> >>>>
> >>>> A few weeks ago I ran the pcpRTMA vs. gauges in MET7.0
(couldn't get
> it
> >>>> to run in MET8.1 at the time, probably due to the undefined
default
> >>>> values as you mentioned in the email below), and that ran fine
and
> >>>> produced reasonable results. Output: point_stat_met7.0.out
> >>>>
> >>>> Thanks again for looking into this.
> >>>>
> >>>> Ying
> >>>>
> >>>>
> >>>>
> >>>> On 9/17/19 11:42 AM, George McCabe via RT wrote:
> >>>>> Hi Ying,
> >>>>>
> >>>>> There are new default values that need to be set in your
> >>>>> PointStatConfig_gauges file that are required by the latest
version
> of
> >>>> MET.
> >>>>> You should have an entry that looks like this:
> >>>>>
> >>>>> message_type_group_map = [
> >>>>> { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";
> },
> >>>>> { key = "ANYAIR"; val = "AIRCAR,AIRCFT";
> },
> >>>>> { key = "ANYSFC"; val =
"ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET";
> },
> >>>>> { key = "ONLYSF"; val = "ADPSFC,SFCSHP";
> },
> >>>>> { key = "LANDSF"; val = "ADPSFC,MSONET";
> },
> >>>>> { key = "WATERSF"; val = "SFCSHP";
> }
> >>>>> ];
> >>>>>
> >>>>> You can refer to the default config files (located in
> share/met/config
> >>>>> under the location MET is installed) to see what all of the
defaults
> >> are.
> >>>>> You should add all of the default values to your config if
they are
> not
> >>>>> already set.
> >>>>>
> >>>>> On Tue, Sep 17, 2019 at 1:19 AM Ying Lin via RT
<met_help at ucar.edu>
> >>>> wrote:
> >>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>
> >>>>>>
> >>>>>> Hi George,
> >>>>>>
> >>>>>> OK, thanks, now we got pass that point. New err msg:
> >>>>>>
> >>>>>> ERROR :
> >>>>>> ERROR : PointStatVxOpt::set_vx_pd() ->
"message_type_group_map"
> must
> >>>>>> contain an entry for "LANDSF".
> >>>>>> ERROR :
> >>>>>>
> >>>>>> Logs etc again in
> >>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
> >>>>>>
> >>>>>> Let's get back to this tomorrow.
> >>>>>>
> >>>>>> Ying
> >>>>>>
> >>>>>> On 9/16/19 8:53 PM, George McCabe via RT wrote:
> >>>>>>> Hi Ying,
> >>>>>>>
> >>>>>>> In your PointStatConfig_gauges file, you need to remove the
quotes
> >>>> around
> >>>>>>> regrid = {
> >>>>>>> to_grid = "${REGRID_TO_GRID}";
> >>>>>>>
> >>>>>>> it should be
> >>>>>>>
> >>>>>>> regrid = {
> >>>>>>> to_grid = ${REGRID_TO_GRID};
> >>>>>>>
> >>>>>>> METplus checks if the value is NONE and adds quotes around
it if it
> >> is
> >>>>>> not
> >>>>>>> set to NONE.
> >>>>>>>
> >>>>>>> On Mon, Sep 16, 2019 at 6:13 PM Ying Lin via RT
<met_help at ucar.edu
> >
> >>>>>> wrote:
> >>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
> >>>>>>>>
> >>>>>>>> Hi George,
> >>>>>>>>
> >>>>>>>> Looks like
> >>>>>>>>
> >>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>>>>>
> >>>>>>>> fixed the issue with finding the gauge .nc file. Great!
> >>>>>>>>
> >>>>>>>> Setting the
> >>>>>>>>
> >>>>>>>> Now the issue remains is the PointStat err msg:
> >>>>>>>>
> >>>>>>>> DEBUG 1: Forecast File:
> >>>>>>>>
> >>>>>>>>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
> >>>>>>>> DEBUG 1: Observation File:
> >>>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/
> >>>>>>>> good-usa-dlyprcp-20190915.nc
> >>>>>>>> DEBUG 2:
> >>>>>>>> DEBUG 2:
> >>>>>>>>
> >>>>>>>>
> >>
>
--------------------------------------------------------------------------------
> >>>>>>>> DEBUG 2:
> >>>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
> >>>>>>>> ERROR :
> >>>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
> >>>>>>>> ERROR :
> >>>>>>>>
> >>>>>>>> I placed the latest configs, log etc in
> >>>>>>>>
> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
> >>>>>>>> (older ones now under old/).
> >>>>>>>>
> >>>>>>>> Thanks again -
> >>>>>>>>
> >>>>>>>> Ying
> >>>>>>>>
> >>>>>>>> On 9/16/19 5:45 PM, George McCabe via RT wrote:
> >>>>>>>>> I will take a look at the POLY issue. My guess is you need
to set
> >> the
> >>>>>>>> poly
> >>>>>>>>> variable using {DMASK} instead of ${DMASK} since it is
> referencing
> >>>> the
> >>>>>>>>> METplus config variable. I will let you know what I find.
> >>>>>>>>>
> >>>>>>>>> On Mon, Sep 16, 2019 at 3:41 PM George McCabe
<mccabe at ucar.edu>
> >>>> wrote:
> >>>>>>>>>> Hi Ying,
> >>>>>>>>>>
> >>>>>>>>>> You should actually set:
> >>>>>>>>>>
> >>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>>>>>>> instead of
> >>>>>>>>>> OBS_FILE_WINDOW_BEGIN and END
> >>>>>>>>>>
> >>>>>>>>>> You SHOULD be able to set the latter, but there is a bug
that
> >>>> prevents
> >>>>>>>>>> that from working. I think you will need OBS_WINDOW_BEGIN
and
> END
> >> to
> >>>>>> be
> >>>>>>>>>> -2700 to 2700 so that PointStat can find the correct data
inside
> >> the
> >>>>>>>> files.
> >>>>>>>>>> Does that make sense?
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT <
> >> met_help at ucar.edu>
> >>>>>>>> wrote:
> >>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
> >
> >>>>>>>>>>>
> >>>>>>>>>>> Hi George,
> >>>>>>>>>>>
> >>>>>>>>>>> I made a number of runs, here are three(to be
brief)
> at
> >>>>>>>>>>> 20190916195402 (run1), 20190916203610 (run2) and
20190916210024
> >>>>>> (run3):
> >>>>>>>>>>> Various configs/logs in
> >>>>>>>>>>>
> >>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
> >>>>>>>>>>> I've also placed the pcprtma.2019091512_A24h Run3 log
> complained
> >>>>>> about:
> >>>>>>>>>>> In run1, I changed
> >>>>>>>>>>> OBS_WINDOW_BEGIN = -2700
> >>>>>>>>>>> OBS_WINDOW_END = 2700
> >>>>>>>>>>> to
> >>>>>>>>>>> OBS_WINDOW_BEGIN = 0
> >>>>>>>>>>> OBS_WINDOW_END = 0
> >>>>>>>>>>>
> >>>>>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts
found
> >> the
> >>>>>>>>>>> current gauge .nc file, but while running PointStat and
reading
> >> in
> >>>>>>>>>>> /parm/PointStatConfig_gauges it complains of
> >>>>>>>>>>> unexpected "$" in line 132 of config file
> >>>>>>>>>>>
> >>>>>>>>>>> Below is the offending line:
> >>>>>>>>>>> //
> >>>>>>>>>>> // Verification masking regions
> >>>>>>>>>>> //
> >>>>>>>>>>> mask = {
> >>>>>>>>>>> grid = ${POINT_STAT_GRID};
> >>>>>>>>>>> poly = ${POINT_STAT_POLY}; <=== line 132
> >>>>>>>>>>> sid = [];
> >>>>>>>>>>> llpnt = [];
> >>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>> Log file shows that the "poly" files for the mask was
found,
> not
> >>>> sure
> >>>>>>>>>>> why line 132 was objectionable.
> >>>>>>>>>>>
> >>>>>>>>>>> In run2, in pcprtma_24h.conf I have
> >>>>>>>>>>>
> >>>>>>>>>>> OBS_WINDOW_BEGIN = -2700
> >>>>>>>>>>> OBS_WINDOW_END = 2700
> >>>>>>>>>>>
> >>>>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
> >>>>>>>>>>> OBS_FILE_WINDOW_END = 0
> >>>>>>>>>>>
> >>>>>>>>>>> and I defined POINT_STAT_POLY in PointStatConfig_gauges
instead
> >> of
> >>>> in
> >>>>>>>>>>> pcprtma_24h.conf to get past the "error in line 132 in
above
> >> run1.
> >>>>>>>>>>> Now I'm getting the same old error as last week:
> >>>>>>>>>>> 09/16 20:36:27.051 metplus.PointStat
> (point_stat_wrapper.py:194)
> >>>>>> ERROR:
> >>>>>>>>>>> Could not find observation file in
> >>>>>>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc
matching
> >>>>>>>> template
> >>>>>>>>>>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
> >>>>>>>>>>>
> >>>>>>>>>>> Run 3: I went back to setting
> >>>>>>>>>>> OBS_WINDOW_BEGIN = 0
> >>>>>>>>>>> OBS_WINDOW_END = 0
> >>>>>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
> >>>>>>>>>>> as in Run1, and left the mask setting as in Run2. This
time
> the
> >>>>>> script
> >>>>>>>>>>> can find the gauge file, no complains about the mask
poly
> files,
> >>>> but
> >>>>>>>>>>> does not seem to be able to find the gridded 24h precip:
> >>>>>>>>>>> DEBUG 2:
> >>>>>>>>>>> DEBUG 2:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>
>
--------------------------------------------------------------------------------
> >>>>>>>>>>> DEBUG 2:
> >>>>>>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
> >>>>>>>>>>> ERROR :
> >>>>>>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
> >>>>>>>>>>> ERROR :
> >>>>>>>>>>>
> >>>>>>>>>>> So in brief:
> >>>>>>>>>>>
> >>>>>>>>>>> * PointStat cannot read the 24h accumulation in
> >>>>>>>> pcprtma.2019091512_A24h
> >>>>>>>>>>> * Script can only find the gauges file when
> >>>>>>>> OBS_WINDOW_BEGIN[_END]are
> >>>>>>>>>>> set to zero. This is undesirable since it
narrows the
> >> valid
> >>>>>> time
> >>>>>>>>>>> for each 24h gauge too much
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>> Ying
> >>>>>>>>>>>
> >>>>>>>>>>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
> >>>>>>>>>>>> Hi Ying,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I determined the cause of your error. It is a
combination of
> >>>>>>>>>>>> mis-configuration and a bug. You have set:
> >>>>>>>>>>>>
> >>>>>>>>>>>> OBS_WINDOW_BEGIN = -2700
> >>>>>>>>>>>> OBS_WINDOW_END = 2700
> >>>>>>>>>>>>
> >>>>>>>>>>>> to specify the range of data to be allowed to be read
in.
> >> However,
> >>>>>>>> thse
> >>>>>>>>>>>> values also set the value of
OBS_FILE_WINDOW_[BEGIN/END] if
> >> those
> >>>>>> are
> >>>>>>>>>>> not
> >>>>>>>>>>>> set. The _FILE_WINDOW_ variables determine the window
of times
> >> to
> >>>>>> find
> >>>>>>>>>>> the
> >>>>>>>>>>>> files based on their filename in case the filename
times don't
> >>>>>> exactly
> >>>>>>>>>>> line
> >>>>>>>>>>>> up with the valid time. You are supposed to be able to
set
> >>>>>>>>>>>>
> >>>>>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
> >>>>>>>>>>>> OBS_FILE_WINDOW_END = 0
> >>>>>>>>>>>>
> >>>>>>>>>>>> to override the values. HOWEVER, there is a typo in the
code
> >> that
> >>>>>>>>>>> handles
> >>>>>>>>>>>> these variables and PointStat actually uses
> >> OBS_WINDOW_[BEGIN/END]
> >>>>>>>>>>> instead.
> >>>>>>>>>>>> I will fix this bug, but in the meantime, you should be
able
> to
> >>>> set
> >>>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
> >>>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
> >>>>>>>>>>>>
> >>>>>>>>>>>> and it should work as intended. Let me know if that
does not
> >> work.
> >>>>>>>>>>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via
RT <
> >>>>>>>>>>>> met_help at ucar.edu> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted
upon.
> >>>>>>>>>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
> >>>>>>>>>>>>> Queue: met_help
> >>>>>>>>>>>>> Subject: python/v3.6.3 and METplus beta
version;
> Re:
> >>>>>> init_seq
> >>>>>>>>>>> works -
> >>>>>>>>>>>>> thanks!
> >>>>>>>>>>>>> Owner: mccabe
> >>>>>>>>>>>>> Requestors: mccabe at ucar.edu, ying.lin at noaa.gov
> >>>>>>>>>>>>> Status: new
> >>>>>>>>>>>>> Ticket <URL:
> >>>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
> >>>>>>>>>>>>> This transaction appears to have no content
> >>>>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Ying Lin
> >>>>>>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
> >>>> Branch
> >>>>>>>>>>> NCWCP Cubicle No. 2015
> >>>>>>>>>>> Ying.Lin at noaa.gov
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>> --
> >>>>>>>> Ying Lin
> >>>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
> Branch
> >>>>>>>> NCWCP Cubicle No. 2015
> >>>>>>>> Ying.Lin at noaa.gov
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>> --
> >>>>>> Ying Lin
> >>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>>>> NCWCP Cubicle No. 2015
> >>>>>> Ying.Lin at noaa.gov
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>> --
> >>>> Ying Lin
> >>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >>>> NCWCP Cubicle No. 2015
> >>>> Ying.Lin at noaa.gov
> >>>>
> >>>>
> >>>>
> >>>>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #92027] python/v3.6.3 and METplus beta version; Re: init_seq works - thanks!
From: Ying Lin
Time: Fri Sep 20 22:38:29 2019
Hi John,
Thanks for the great info! If you like, please close this ticket at
your convenience.
Ying
On 9/19/19 11:41 AM, John Halley Gotway via RT wrote:
> Ying,
>
> Yes, that's correct. -9999 is the missing data value used in MET,
and your
> explanation makes sense to me.
>
> But let's say at some point in the future you have point
observations of
> precip with slightly negative values that you do want to ignore.
You can
> do this in the Point-Stat config file using the censor_thresh and
> censor_val options:
>
> obs = {
> field = [ { name="APCP"; level="A24"; censor_thresh = [ <0 ];
censor_val
> = [ -9999 ]; } ];
> }
>
> That tells Point-Stat to replace any negative observations with the
bad
> data value. So there's multiple ways of addressing this issue.
>
> Thanks,
> John
>
>
>
> On Wed, Sep 18, 2019 at 7:07 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>
>> Thank you John, setting
>>
>> obs_quality = [ "1", "2","3", "NA" ];
>>
>> seems to have worked.
>>
>> Question: in the PointStat input (*.nc; last coloum in
*_formatted),
>> does a value of '-9999.' denote "missing/bad obs value"?
>>
>> In my gauge data, there are 69 entries that has "-9999." for the
precip
>> amount and I meant for it to be "missing" (the raw data have mostly
>> "-9999." for these 69, and one of them has a small negative amount,
I
>> set all the negative amounts to '-9999.'). In the job output,
there are
>>
>> DEBUG 3: Rejected: valid time = 202
>> DEBUG 3: Rejected: bad obs value = 57
>>
>> Looks like some of the 69 entries with "-9999." also had valid time
more
>> than an hour (+/-3600 seconds) away from 12:00Z, so the above looks
>> reasonable and I'm guessing "-9999." does mean "missing" to
PointStat.
>>
>> Ying
>>
>> On 9/18/19 6:38 PM, John Halley Gotway via RT wrote:
>>> Ying and George,
>>>
>>> So here's the ASCII point observations which are presumably being
run
>>> through ascii2nc:
>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/good-
usa-dlyprcp-20190915_formatted
>>> The QC_String is the second to last column and is set to a
constant value
>>> of "NA". And running ncdump on the output of ascii2nc confirms
that to
>> be
>>> the case:
>>>
>>> * ncdump -v obs_qty_table good-usa-dlyprcp-20190915.nc
>>> <http://good-usa-dlyprcp-20190915.nc>*
>>> ...
>>> obs_qty_table =
>>> "NA" ;
>>> ...
>>>
>>> Here's the Point-Stat config file you're using:
>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/PointStatConfig_gauges
>>> And in there you'll find that we're requesting obs with a quality
string
>> of
>>> 1, 2, or 3:
>>> * obs_quality = [ "1", "2", "3" ];*
>>>
>>> You can fix this behavior either by...
>>> (1) Including "NA" in the obs_quality list: obs_quality = [
"1",
>> "2",
>>> "3", "NA" ];
>>> (2) Or by changing the QC value in the ASCII obs from "NA" to a
value of
>> 1,
>>> 2, or 3.
>>>
>>> Hope that does it.
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> On Wed, Sep 18, 2019 at 3:43 PM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>
>>>> Hi George,
>>>>
>>>> Good stuff, the COPYABLE ENVIRONMENT FOR NEXT COMMAND. So
I did
>> as
>>>> you said, and the reason for not having any matching pairs is
that all
>>>> observations were rejected due to quality marker
>>>> (
>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/master_metplus.log.test
>> ).
>>>> Initially I thought it had something to do with all entries
having the
>>>> elev_now and height_now (in precip_ascii2nc.py, same ftp dir) set
to
>>>> '-9999.' (as in
>>>>
>>>>
>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/old/good-
usa-dlyprcp-20190915_formatted
>> )
>>>> so I went and changed both of these columns to '0.0', since
they're not
>>>> used in the verification anyway; and I also decided to have any
negative
>>>> precip values (-9999., or an unintentional negative precip amount
>>>> report) in the input to be encoded as '-9999.' in the
>>>> good-usa-dlyprcp-20190915.nc, and there are 69 such reports.
>>>>
>>>> It didn't make a difference though.
master_metplus.log.test shows
>>>> that
>>>> DEBUG 3: Observations processed = 7646
>>>> ....
>>>> DEBUG 3: Rejected: quality marker = 7646
>>>>
>>>> Can you tell by looking at good-usa-dlyprcp-20190915_formatted
what
>>>> makes every entry fail the quality marker? Maybe MET8.1 requires
a
>>>> different format for the point output. I originally got the
>>>> precip_ascii2nc.py from Mallory more than a year ago. Output
from it
>>>> (even with height and elev both set to '-9999.') works for MET7.0
>>>>
>>>> Thanks again -
>>>>
>>>> Ying
>>>>
>>>> On 9/18/19 12:05 PM, George McCabe via RT wrote:
>>>>> Hi Ying,
>>>>>
>>>>> It is hard to say why this run didn't work without comparing the
>>>> PointStat
>>>>> config file you used last time to what is set here. I would
recommend
>>>>> rerunning point_stat with verbosity increased to 3 so you can
see the
>>>>> reasons that the matched pairs were rejected. To do this, copy
all of
>> the
>>>>> export commands from the line after COPYABLE ENVIRONMENT FOR
NEXT
>>>> COMMAND:
>>>>> and paste that into a command prompt. Then copy the point_stat
command
>>>> but
>>>>> add '-v 3' to it. The output will now show how the matches pairs
were
>>>>> rejected, which is already shown in your old output. Compare the
logs
>> to
>>>>> see why they are being rejected. This will give you a clue as to
what
>> is
>>>>> different in your configuration between runs.
>>>>>
>>>>> On Wed, Sep 18, 2019 at 9:41 AM Ying Lin via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>>>
>>>>>> Thanks George, I see that my PointStatConfig was missing the
settings
>>>>>> for "LANDSF" and "WATERSF" in the message_type_group_map. Now
the
>> test
>>>>>> no longer has the err msgs.
>>>>>>
>>>>>> However, the point_stat step did not seem to have found any
matches -
>>>>>> the point_stat_*.txt and *.stat files all had only the header
line:
>> see
>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190918/
>>>>>>
>>>>>> A few weeks ago I ran the pcpRTMA vs. gauges in MET7.0
(couldn't get
>> it
>>>>>> to run in MET8.1 at the time, probably due to the undefined
default
>>>>>> values as you mentioned in the email below), and that ran fine
and
>>>>>> produced reasonable results. Output: point_stat_met7.0.out
>>>>>>
>>>>>> Thanks again for looking into this.
>>>>>>
>>>>>> Ying
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 9/17/19 11:42 AM, George McCabe via RT wrote:
>>>>>>> Hi Ying,
>>>>>>>
>>>>>>> There are new default values that need to be set in your
>>>>>>> PointStatConfig_gauges file that are required by the latest
version
>> of
>>>>>> MET.
>>>>>>> You should have an entry that looks like this:
>>>>>>>
>>>>>>> message_type_group_map = [
>>>>>>> { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";
>> },
>>>>>>> { key = "ANYAIR"; val = "AIRCAR,AIRCFT";
>> },
>>>>>>> { key = "ANYSFC"; val =
"ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET";
>> },
>>>>>>> { key = "ONLYSF"; val = "ADPSFC,SFCSHP";
>> },
>>>>>>> { key = "LANDSF"; val = "ADPSFC,MSONET";
>> },
>>>>>>> { key = "WATERSF"; val = "SFCSHP";
>> }
>>>>>>> ];
>>>>>>>
>>>>>>> You can refer to the default config files (located in
>> share/met/config
>>>>>>> under the location MET is installed) to see what all of the
defaults
>>>> are.
>>>>>>> You should add all of the default values to your config if
they are
>> not
>>>>>>> already set.
>>>>>>>
>>>>>>> On Tue, Sep 17, 2019 at 1:19 AM Ying Lin via RT
<met_help at ucar.edu>
>>>>>> wrote:
>>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>
>>>>>>>>
>>>>>>>> Hi George,
>>>>>>>>
>>>>>>>> OK, thanks, now we got pass that point. New err
msg:
>>>>>>>>
>>>>>>>> ERROR :
>>>>>>>> ERROR : PointStatVxOpt::set_vx_pd() ->
"message_type_group_map"
>> must
>>>>>>>> contain an entry for "LANDSF".
>>>>>>>> ERROR :
>>>>>>>>
>>>>>>>> Logs etc again in
>>>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
>>>>>>>>
>>>>>>>> Let's get back to this tomorrow.
>>>>>>>>
>>>>>>>> Ying
>>>>>>>>
>>>>>>>> On 9/16/19 8:53 PM, George McCabe via RT wrote:
>>>>>>>>> Hi Ying,
>>>>>>>>>
>>>>>>>>> In your PointStatConfig_gauges file, you need to remove the
quotes
>>>>>> around
>>>>>>>>> regrid = {
>>>>>>>>> to_grid = "${REGRID_TO_GRID}";
>>>>>>>>>
>>>>>>>>> it should be
>>>>>>>>>
>>>>>>>>> regrid = {
>>>>>>>>> to_grid = ${REGRID_TO_GRID};
>>>>>>>>>
>>>>>>>>> METplus checks if the value is NONE and adds quotes around
it if it
>>>> is
>>>>>>>> not
>>>>>>>>> set to NONE.
>>>>>>>>>
>>>>>>>>> On Mon, Sep 16, 2019 at 6:13 PM Ying Lin via RT
<met_help at ucar.edu
>>>>>>>> wrote:
>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027 >
>>>>>>>>>>
>>>>>>>>>> Hi George,
>>>>>>>>>>
>>>>>>>>>> Looks like
>>>>>>>>>>
>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>>>>>
>>>>>>>>>> fixed the issue with finding the gauge .nc file. Great!
>>>>>>>>>>
>>>>>>>>>> Setting the
>>>>>>>>>>
>>>>>>>>>> Now the issue remains is the PointStat err msg:
>>>>>>>>>>
>>>>>>>>>> DEBUG 1: Forecast File:
>>>>>>>>>>
>>>>>>>>>>
>>
/gpfs/dell2/emc/verification/noscrub/Ying.Lin/metplus.out/pcprtma/bucket/pcprtma.2019091512_A24h
>>>>>>>>>> DEBUG 1: Observation File:
>>>>>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc/
>>>>>>>>>> good-usa-dlyprcp-20190915.nc
>>>>>>>>>> DEBUG 2:
>>>>>>>>>> DEBUG 2:
>>>>>>>>>>
>>>>>>>>>>
>>
--------------------------------------------------------------------------------
>>>>>>>>>> DEBUG 2:
>>>>>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
>>>>>>>>>> ERROR :
>>>>>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
>>>>>>>>>> ERROR :
>>>>>>>>>>
>>>>>>>>>> I placed the latest configs, log etc in
>>>>>>>>>>
>> https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/
>>>>>>>>>> (older ones now under old/).
>>>>>>>>>>
>>>>>>>>>> Thanks again -
>>>>>>>>>>
>>>>>>>>>> Ying
>>>>>>>>>>
>>>>>>>>>> On 9/16/19 5:45 PM, George McCabe via RT wrote:
>>>>>>>>>>> I will take a look at the POLY issue. My guess is you need
to set
>>>> the
>>>>>>>>>> poly
>>>>>>>>>>> variable using {DMASK} instead of ${DMASK} since it is
>> referencing
>>>>>> the
>>>>>>>>>>> METplus config variable. I will let you know what I find.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 16, 2019 at 3:41 PM George McCabe
<mccabe at ucar.edu>
>>>>>> wrote:
>>>>>>>>>>>> Hi Ying,
>>>>>>>>>>>>
>>>>>>>>>>>> You should actually set:
>>>>>>>>>>>>
>>>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>>>>>>> instead of
>>>>>>>>>>>> OBS_FILE_WINDOW_BEGIN and END
>>>>>>>>>>>>
>>>>>>>>>>>> You SHOULD be able to set the latter, but there is a bug
that
>>>>>> prevents
>>>>>>>>>>>> that from working. I think you will need OBS_WINDOW_BEGIN
and
>> END
>>>> to
>>>>>>>> be
>>>>>>>>>>>> -2700 to 2700 so that PointStat can find the correct data
inside
>>>> the
>>>>>>>>>> files.
>>>>>>>>>>>> Does that make sense?
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 16, 2019 at 3:36 PM Ying Lin via RT <
>>>> met_help at ucar.edu>
>>>>>>>>>> wrote:
>>>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>>>>>>>>>>>>> Hi George,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I made a number of runs, here are three(to be
brief)
>> at
>>>>>>>>>>>>> 20190916195402 (run1), 20190916203610 (run2) and
20190916210024
>>>>>>>> (run3):
>>>>>>>>>>>>> Various configs/logs in
>>>>>>>>>>>>>
>>>>>>
https://ftp.emc.ncep.noaa.gov/mmb/precip/METplus.question/20190916/.
>>>>>>>>>>>>> I've also placed the pcprtma.2019091512_A24h Run3 log
>> complained
>>>>>>>> about:
>>>>>>>>>>>>> In run1, I changed
>>>>>>>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>>>>>>> to
>>>>>>>>>>>>> OBS_WINDOW_BEGIN = 0
>>>>>>>>>>>>> OBS_WINDOW_END = 0
>>>>>>>>>>>>>
>>>>>>>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]. The scripts
found
>>>> the
>>>>>>>>>>>>> current gauge .nc file, but while running PointStat and
reading
>>>> in
>>>>>>>>>>>>> /parm/PointStatConfig_gauges it complains of
>>>>>>>>>>>>> unexpected "$" in line 132 of config file
>>>>>>>>>>>>>
>>>>>>>>>>>>> Below is the offending line:
>>>>>>>>>>>>> //
>>>>>>>>>>>>> // Verification masking regions
>>>>>>>>>>>>> //
>>>>>>>>>>>>> mask = {
>>>>>>>>>>>>> grid = ${POINT_STAT_GRID};
>>>>>>>>>>>>> poly = ${POINT_STAT_POLY}; <=== line 132
>>>>>>>>>>>>> sid = [];
>>>>>>>>>>>>> llpnt = [];
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> Log file shows that the "poly" files for the mask was
found,
>> not
>>>>>> sure
>>>>>>>>>>>>> why line 132 was objectionable.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In run2, in pcprtma_24h.conf I have
>>>>>>>>>>>>>
>>>>>>>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>>>>>>>
>>>>>>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>>>>>>>>>>> OBS_FILE_WINDOW_END = 0
>>>>>>>>>>>>>
>>>>>>>>>>>>> and I defined POINT_STAT_POLY in PointStatConfig_gauges
instead
>>>> of
>>>>>> in
>>>>>>>>>>>>> pcprtma_24h.conf to get past the "error in line 132 in
above
>>>> run1.
>>>>>>>>>>>>> Now I'm getting the same old error as last week:
>>>>>>>>>>>>> 09/16 20:36:27.051 metplus.PointStat
>> (point_stat_wrapper.py:194)
>>>>>>>> ERROR:
>>>>>>>>>>>>> Could not find observation file in
>>>>>>>>>>>>> /gpfs/dell2/emc/verification/noscrub/Ying.Lin/gauge_nc
matching
>>>>>>>>>> template
>>>>>>>>>>>>> good-usa-dlyprcp-{valid?fmt=%Y%m%d}.nc
>>>>>>>>>>>>>
>>>>>>>>>>>>> Run 3: I went back to setting
>>>>>>>>>>>>> OBS_WINDOW_BEGIN = 0
>>>>>>>>>>>>> OBS_WINDOW_END = 0
>>>>>>>>>>>>> w/o setting the OBS_FILE_WINDOW_BEGIN[END]
>>>>>>>>>>>>> as in Run1, and left the mask setting as in Run2. This
time
>> the
>>>>>>>> script
>>>>>>>>>>>>> can find the gauge file, no complains about the mask
poly
>> files,
>>>>>> but
>>>>>>>>>>>>> does not seem to be able to find the gridded 24h precip:
>>>>>>>>>>>>> DEBUG 2:
>>>>>>>>>>>>> DEBUG 2:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>
--------------------------------------------------------------------------------
>>>>>>>>>>>>> DEBUG 2:
>>>>>>>>>>>>> DEBUG 2: Reading data for APCP_24(*,*).
>>>>>>>>>>>>> ERROR :
>>>>>>>>>>>>> ERROR : grd_file_type() -> file does not exist "NONE"
>>>>>>>>>>>>> ERROR :
>>>>>>>>>>>>>
>>>>>>>>>>>>> So in brief:
>>>>>>>>>>>>>
>>>>>>>>>>>>> * PointStat cannot read the 24h accumulation in
>>>>>>>>>> pcprtma.2019091512_A24h
>>>>>>>>>>>>> * Script can only find the gauges file when
>>>>>>>>>> OBS_WINDOW_BEGIN[_END]are
>>>>>>>>>>>>> set to zero. This is undesirable since it
narrows the
>>>> valid
>>>>>>>> time
>>>>>>>>>>>>> for each 24h gauge too much
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Ying
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 9/16/19 2:06 PM, George McCabe via RT wrote:
>>>>>>>>>>>>>> Hi Ying,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I determined the cause of your error. It is a
combination of
>>>>>>>>>>>>>> mis-configuration and a bug. You have set:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> OBS_WINDOW_BEGIN = -2700
>>>>>>>>>>>>>> OBS_WINDOW_END = 2700
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> to specify the range of data to be allowed to be read
in.
>>>> However,
>>>>>>>>>> thse
>>>>>>>>>>>>>> values also set the value of
OBS_FILE_WINDOW_[BEGIN/END] if
>>>> those
>>>>>>>> are
>>>>>>>>>>>>> not
>>>>>>>>>>>>>> set. The _FILE_WINDOW_ variables determine the window
of times
>>>> to
>>>>>>>> find
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> files based on their filename in case the filename
times don't
>>>>>>>> exactly
>>>>>>>>>>>>> line
>>>>>>>>>>>>>> up with the valid time. You are supposed to be able to
set
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> OBS_FILE_WINDOW_BEGIN = 0
>>>>>>>>>>>>>> OBS_FILE_WINDOW_END = 0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> to override the values. HOWEVER, there is a typo in the
code
>>>> that
>>>>>>>>>>>>> handles
>>>>>>>>>>>>>> these variables and PointStat actually uses
>>>> OBS_WINDOW_[BEGIN/END]
>>>>>>>>>>>>> instead.
>>>>>>>>>>>>>> I will fix this bug, but in the meantime, you should be
able
>> to
>>>>>> set
>>>>>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_BEGIN = 0
>>>>>>>>>>>>>> OBS_POINT_STAT_FILE_WINDOW_END = 0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and it should work as intended. Let me know if that
does not
>>>> work.
>>>>>>>>>>>>>> On Mon, Sep 16, 2019 at 10:12 AM John Halley Gotway via
RT <
>>>>>>>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Mon Sep 16 10:11:48 2019: Request 92027 was acted
upon.
>>>>>>>>>>>>>>> Transaction: Given to mccabe (George McCabe) by johnhg
>>>>>>>>>>>>>>> Queue: met_help
>>>>>>>>>>>>>>> Subject: python/v3.6.3 and METplus beta
version;
>> Re:
>>>>>>>> init_seq
>>>>>>>>>>>>> works -
>>>>>>>>>>>>>>> thanks!
>>>>>>>>>>>>>>> Owner: mccabe
>>>>>>>>>>>>>>> Requestors: mccabe at ucar.edu,
ying.lin at noaa.gov
>>>>>>>>>>>>>>> Status: new
>>>>>>>>>>>>>>> Ticket <URL:
>>>>>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92027
>>>>>>>>>>>>>>> This transaction appears to have no content
>>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Ying Lin
>>>>>>>>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
>>>>>> Branch
>>>>>>>>>>>>> NCWCP Cubicle No. 2015
>>>>>>>>>>>>> Ying.Lin at noaa.gov
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ying Lin
>>>>>>>>>> NCEP/EMC/Verification, Post-processing and Product
Generation
>> Branch
>>>>>>>>>> NCWCP Cubicle No. 2015
>>>>>>>>>> Ying.Lin at noaa.gov
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> --
>>>>>>>> Ying Lin
>>>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>>>> NCWCP Cubicle No. 2015
>>>>>>>> Ying.Lin at noaa.gov
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> --
>>>>>> Ying Lin
>>>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>>>> NCWCP Cubicle No. 2015
>>>>>> Ying.Lin at noaa.gov
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list