[Met_help] [rt.rap.ucar.edu #97794] History for processing anomaly resulting in zero matched pairs
John Halley Gotway via RT
met_help at ucar.edu
Thu Dec 10 09:57:02 MST 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
The last couple of days I've noticed that my METPlus routine has not been
processing ozone. Nothing has changed in the set-up, so I thought it might
be related to missing data, either obs. or from the model output. It turns
out that this is not the case.
Going through the log files I found something unexpected. Here is a sample
from demonstrating when it fails (i.e. no matched pairs)
DEBUG 3: Rejected: bad climo stdev = 0
DEBUG 3: Rejected: duplicates = 0
DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation type AIRNOW,
over region NEC, for interpolation method BILIN(4), using 0 matched pairs.
DEBUG 3: Number of matched pairs = 0
DEBUG 3: Observations processed = 42282
DEBUG 3: Rejected: station id = 0
DEBUG 3: Rejected: obs type = 0
DEBUG 3: Rejected: valid time = 42282
Here is an example when it is successful:
DEBUG 3: Rejected: bad climo mean = 0
DEBUG 3: Rejected: bad climo stdev = 0
DEBUG 3: Rejected: duplicates = 0
DEBUG 2: Processing OZCON/A8 versus COPO/A8, for observation type AIRNOW,
over region Urban, for interpolation method BILIN(4), using 145 matched
pairs.
DEBUG 3: Number of matched pairs = 145
DEBUG 3: Observations processed = 42091
DEBUG 3: Rejected: station id = 0
DEBUG 3: Rejected: obs type = 0
DEBUG 3: Rejected: valid time = 40328
If you look carefully you will see that the case where it failed shows
OZCONA8 while the successful case shows OZCON/A8.
I find nothing indicating the joining of these strings in the point_stat
config. If you look at my shared_config file you will see that I set the
obsvar and fcst var correctly:
FCST_VAR1_NAME = OZCON
FCST_VAR1_LEVELS = A1
OBS_VAR1_NAME= COPO
OBS_VAR1_LEVELS= A1
OBS_VAR1_OPTIONS = convert(x) = x * 10^9;
FCST_VAR2_NAME = OZCON
FCST_VAR2_LEVELS = A8
OBS_VAR2_NAME= COPO
OBS_VAR2_LEVELS= A8
OBS_VAR2_OPTIONS = convert(x) = x * 10^9;
I'm kind of at a loss here and was wondering what could have caused this.
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: processing anomaly resulting in zero matched pairs
From: John Halley Gotway
Time: Tue Dec 08 14:06:52 2020
Ed,
Nice find noticing that subtle difference in strings. The MET tools,
including point_stat, often include what we've called the "magic
string" in
the log message output. The magic string is just the "variable
name"/"level
string". It is not immediately obvious to me why you're seeing
"OZCONA8"
versus "OZCON/A8". So I'm not entirely sure. But I would expect the
mismatch in accumulation interval to be counted as a "level mismatch"
not
"valid time".
I'd recommend that we start by looking exactly where the log messages
are
pointing us. Let's make sure that when we run Point-Stat, the forecast
valid time matches up correctly with the valid time of point
observations.
Can you please send me the full path to the METplus log file on WCOSS
from
which you extracted these log messages? I'd like to look at the Point-
Stat
command that was run... and then look closely at the forecast and
observation data being compared. Hopefully then we'll be able to see
where
the problem lies... whether it be a problem in Point-Stat, METplus
wrappers, or your configuration.
Thanks,
John
On Mon, Dec 7, 2020 at 1:39 PM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Mon Dec 07 13:39:04 2020: Request 97794 was acted upon.
> Transaction: Ticket created by edward.strobach at noaa.gov
> Queue: met_help
> Subject: processing anomaly resulting in zero matched pairs
> Owner: Nobody
> Requestors: edward.strobach at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97794 >
>
>
> The last couple of days I've noticed that my METPlus routine has not
been
> processing ozone. Nothing has changed in the set-up, so I thought
it might
> be related to missing data, either obs. or from the model output.
It turns
> out that this is not the case.
>
> Going through the log files I found something unexpected. Here is a
sample
> from demonstrating when it fails (i.e. no matched pairs)
>
> DEBUG 3: Rejected: bad climo stdev = 0
> DEBUG 3: Rejected: duplicates = 0
> DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation type
AIRNOW,
> over region NEC, for interpolation method BILIN(4), using 0 matched
pairs.
> DEBUG 3: Number of matched pairs = 0
> DEBUG 3: Observations processed = 42282
> DEBUG 3: Rejected: station id = 0
> DEBUG 3: Rejected: obs type = 0
> DEBUG 3: Rejected: valid time = 42282
>
> Here is an example when it is successful:
>
> DEBUG 3: Rejected: bad climo mean = 0
> DEBUG 3: Rejected: bad climo stdev = 0
> DEBUG 3: Rejected: duplicates = 0
> DEBUG 2: Processing OZCON/A8 versus COPO/A8, for observation type
AIRNOW,
> over region Urban, for interpolation method BILIN(4), using 145
matched
> pairs.
> DEBUG 3: Number of matched pairs = 145
> DEBUG 3: Observations processed = 42091
> DEBUG 3: Rejected: station id = 0
> DEBUG 3: Rejected: obs type = 0
> DEBUG 3: Rejected: valid time = 40328
>
> If you look carefully you will see that the case where it failed
shows
> OZCONA8 while the successful case shows OZCON/A8.
>
> I find nothing indicating the joining of these strings in the
point_stat
> config. If you look at my shared_config file you will see that I
set the
> obsvar and fcst var correctly:
>
> FCST_VAR1_NAME = OZCON
> FCST_VAR1_LEVELS = A1
> OBS_VAR1_NAME= COPO
> OBS_VAR1_LEVELS= A1
> OBS_VAR1_OPTIONS = convert(x) = x * 10^9;
> FCST_VAR2_NAME = OZCON
> FCST_VAR2_LEVELS = A8
> OBS_VAR2_NAME= COPO
> OBS_VAR2_LEVELS= A8
> OBS_VAR2_OPTIONS = convert(x) = x * 10^9;
>
>
> I'm kind of at a loss here and was wondering what could have caused
this.
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: processing anomaly resulting in zero matched pairs
From: Edward Strobach - NOAA Affiliate
Time: Tue Dec 08 14:31:22 2020
Oddly enough this problem only lasted for two days. I saw this
morning
that the ozone stat files generated from Stat_Analysis were populated.
When this issue was happening, I saw that the point stat files were
also
not generated. PB2NC worked just fine. I didn't change anything but
the
problem somehow corrected itself. This problem occurred for all the
experiments I was running. I can still provide you additional
information
from the logs if you are still interested.
On Tue, Dec 8, 2020 at 4:07 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Ed,
>
> Nice find noticing that subtle difference in strings. The MET tools,
> including point_stat, often include what we've called the "magic
string" in
> the log message output. The magic string is just the "variable
name"/"level
> string". It is not immediately obvious to me why you're seeing
"OZCONA8"
> versus "OZCON/A8". So I'm not entirely sure. But I would expect the
> mismatch in accumulation interval to be counted as a "level
mismatch" not
> "valid time".
>
> I'd recommend that we start by looking exactly where the log
messages are
> pointing us. Let's make sure that when we run Point-Stat, the
forecast
> valid time matches up correctly with the valid time of point
observations.
>
> Can you please send me the full path to the METplus log file on
WCOSS from
> which you extracted these log messages? I'd like to look at the
Point-Stat
> command that was run... and then look closely at the forecast and
> observation data being compared. Hopefully then we'll be able to see
where
> the problem lies... whether it be a problem in Point-Stat, METplus
> wrappers, or your configuration.
>
> Thanks,
> John
>
> On Mon, Dec 7, 2020 at 1:39 PM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Dec 07 13:39:04 2020: Request 97794 was acted upon.
> > Transaction: Ticket created by edward.strobach at noaa.gov
> > Queue: met_help
> > Subject: processing anomaly resulting in zero matched pairs
> > Owner: Nobody
> > Requestors: edward.strobach at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97794 >
> >
> >
> > The last couple of days I've noticed that my METPlus routine has
not been
> > processing ozone. Nothing has changed in the set-up, so I thought
it
> might
> > be related to missing data, either obs. or from the model output.
It
> turns
> > out that this is not the case.
> >
> > Going through the log files I found something unexpected. Here is
a
> sample
> > from demonstrating when it fails (i.e. no matched pairs)
> >
> > DEBUG 3: Rejected: bad climo stdev = 0
> > DEBUG 3: Rejected: duplicates = 0
> > DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation type
AIRNOW,
> > over region NEC, for interpolation method BILIN(4), using 0
matched
> pairs.
> > DEBUG 3: Number of matched pairs = 0
> > DEBUG 3: Observations processed = 42282
> > DEBUG 3: Rejected: station id = 0
> > DEBUG 3: Rejected: obs type = 0
> > DEBUG 3: Rejected: valid time = 42282
> >
> > Here is an example when it is successful:
> >
> > DEBUG 3: Rejected: bad climo mean = 0
> > DEBUG 3: Rejected: bad climo stdev = 0
> > DEBUG 3: Rejected: duplicates = 0
> > DEBUG 2: Processing OZCON/A8 versus COPO/A8, for observation type
AIRNOW,
> > over region Urban, for interpolation method BILIN(4), using 145
matched
> > pairs.
> > DEBUG 3: Number of matched pairs = 145
> > DEBUG 3: Observations processed = 42091
> > DEBUG 3: Rejected: station id = 0
> > DEBUG 3: Rejected: obs type = 0
> > DEBUG 3: Rejected: valid time = 40328
> >
> > If you look carefully you will see that the case where it failed
shows
> > OZCONA8 while the successful case shows OZCON/A8.
> >
> > I find nothing indicating the joining of these strings in the
point_stat
> > config. If you look at my shared_config file you will see that I
set the
> > obsvar and fcst var correctly:
> >
> > FCST_VAR1_NAME = OZCON
> > FCST_VAR1_LEVELS = A1
> > OBS_VAR1_NAME= COPO
> > OBS_VAR1_LEVELS= A1
> > OBS_VAR1_OPTIONS = convert(x) = x * 10^9;
> > FCST_VAR2_NAME = OZCON
> > FCST_VAR2_LEVELS = A8
> > OBS_VAR2_NAME= COPO
> > OBS_VAR2_LEVELS= A8
> > OBS_VAR2_OPTIONS = convert(x) = x * 10^9;
> >
> >
> > I'm kind of at a loss here and was wondering what could have
caused this.
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: processing anomaly resulting in zero matched pairs
From: John Halley Gotway
Time: Tue Dec 08 14:58:30 2020
Up to you. Just let me know. If there's a need to go fill in the
missing
days, then we'd need to figure it out. If not, I suppose we can just
chalk
it up to 2020!
John
On Tue, Dec 8, 2020 at 2:31 PM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97794 >
>
> Oddly enough this problem only lasted for two days. I saw this
morning
> that the ozone stat files generated from Stat_Analysis were
populated.
> When this issue was happening, I saw that the point stat files were
also
> not generated. PB2NC worked just fine. I didn't change anything
but the
> problem somehow corrected itself. This problem occurred for all the
> experiments I was running. I can still provide you additional
information
> from the logs if you are still interested.
>
> On Tue, Dec 8, 2020 at 4:07 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Ed,
> >
> > Nice find noticing that subtle difference in strings. The MET
tools,
> > including point_stat, often include what we've called the "magic
string"
> in
> > the log message output. The magic string is just the "variable
> name"/"level
> > string". It is not immediately obvious to me why you're seeing
"OZCONA8"
> > versus "OZCON/A8". So I'm not entirely sure. But I would expect
the
> > mismatch in accumulation interval to be counted as a "level
mismatch" not
> > "valid time".
> >
> > I'd recommend that we start by looking exactly where the log
messages are
> > pointing us. Let's make sure that when we run Point-Stat, the
forecast
> > valid time matches up correctly with the valid time of point
> observations.
> >
> > Can you please send me the full path to the METplus log file on
WCOSS
> from
> > which you extracted these log messages? I'd like to look at the
> Point-Stat
> > command that was run... and then look closely at the forecast and
> > observation data being compared. Hopefully then we'll be able to
see
> where
> > the problem lies... whether it be a problem in Point-Stat, METplus
> > wrappers, or your configuration.
> >
> > Thanks,
> > John
> >
> > On Mon, Dec 7, 2020 at 1:39 PM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Dec 07 13:39:04 2020: Request 97794 was acted upon.
> > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > Queue: met_help
> > > Subject: processing anomaly resulting in zero matched pairs
> > > Owner: Nobody
> > > Requestors: edward.strobach at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97794
> >
> > >
> > >
> > > The last couple of days I've noticed that my METPlus routine has
not
> been
> > > processing ozone. Nothing has changed in the set-up, so I
thought it
> > might
> > > be related to missing data, either obs. or from the model
output. It
> > turns
> > > out that this is not the case.
> > >
> > > Going through the log files I found something unexpected. Here
is a
> > sample
> > > from demonstrating when it fails (i.e. no matched pairs)
> > >
> > > DEBUG 3: Rejected: bad climo stdev = 0
> > > DEBUG 3: Rejected: duplicates = 0
> > > DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation type
> AIRNOW,
> > > over region NEC, for interpolation method BILIN(4), using 0
matched
> > pairs.
> > > DEBUG 3: Number of matched pairs = 0
> > > DEBUG 3: Observations processed = 42282
> > > DEBUG 3: Rejected: station id = 0
> > > DEBUG 3: Rejected: obs type = 0
> > > DEBUG 3: Rejected: valid time = 42282
> > >
> > > Here is an example when it is successful:
> > >
> > > DEBUG 3: Rejected: bad climo mean = 0
> > > DEBUG 3: Rejected: bad climo stdev = 0
> > > DEBUG 3: Rejected: duplicates = 0
> > > DEBUG 2: Processing OZCON/A8 versus COPO/A8, for observation
type
> AIRNOW,
> > > over region Urban, for interpolation method BILIN(4), using 145
matched
> > > pairs.
> > > DEBUG 3: Number of matched pairs = 145
> > > DEBUG 3: Observations processed = 42091
> > > DEBUG 3: Rejected: station id = 0
> > > DEBUG 3: Rejected: obs type = 0
> > > DEBUG 3: Rejected: valid time = 40328
> > >
> > > If you look carefully you will see that the case where it failed
shows
> > > OZCONA8 while the successful case shows OZCON/A8.
> > >
> > > I find nothing indicating the joining of these strings in the
> point_stat
> > > config. If you look at my shared_config file you will see that
I set
> the
> > > obsvar and fcst var correctly:
> > >
> > > FCST_VAR1_NAME = OZCON
> > > FCST_VAR1_LEVELS = A1
> > > OBS_VAR1_NAME= COPO
> > > OBS_VAR1_LEVELS= A1
> > > OBS_VAR1_OPTIONS = convert(x) = x * 10^9;
> > > FCST_VAR2_NAME = OZCON
> > > FCST_VAR2_LEVELS = A8
> > > OBS_VAR2_NAME= COPO
> > > OBS_VAR2_LEVELS= A8
> > > OBS_VAR2_OPTIONS = convert(x) = x * 10^9;
> > >
> > >
> > > I'm kind of at a loss here and was wondering what could have
caused
> this.
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
------------------------------------------------
Subject: processing anomaly resulting in zero matched pairs
From: Edward Strobach - NOAA Affiliate
Time: Thu Dec 10 07:26:24 2020
Okay. If this happens again I'll probably want to know what went
wrong so
I can get a better understanding. Given mutual workloads and time
constraints I elect not to pursue this at this time.
On Tue, Dec 8, 2020 at 4:58 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Up to you. Just let me know. If there's a need to go fill in the
missing
> days, then we'd need to figure it out. If not, I suppose we can just
chalk
> it up to 2020!
>
> John
>
> On Tue, Dec 8, 2020 at 2:31 PM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97794 >
> >
> > Oddly enough this problem only lasted for two days. I saw this
morning
> > that the ozone stat files generated from Stat_Analysis were
populated.
> > When this issue was happening, I saw that the point stat files
were also
> > not generated. PB2NC worked just fine. I didn't change anything
but the
> > problem somehow corrected itself. This problem occurred for all
the
> > experiments I was running. I can still provide you additional
> information
> > from the logs if you are still interested.
> >
> > On Tue, Dec 8, 2020 at 4:07 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Ed,
> > >
> > > Nice find noticing that subtle difference in strings. The MET
tools,
> > > including point_stat, often include what we've called the "magic
> string"
> > in
> > > the log message output. The magic string is just the "variable
> > name"/"level
> > > string". It is not immediately obvious to me why you're seeing
> "OZCONA8"
> > > versus "OZCON/A8". So I'm not entirely sure. But I would expect
the
> > > mismatch in accumulation interval to be counted as a "level
mismatch"
> not
> > > "valid time".
> > >
> > > I'd recommend that we start by looking exactly where the log
messages
> are
> > > pointing us. Let's make sure that when we run Point-Stat, the
forecast
> > > valid time matches up correctly with the valid time of point
> > observations.
> > >
> > > Can you please send me the full path to the METplus log file on
WCOSS
> > from
> > > which you extracted these log messages? I'd like to look at the
> > Point-Stat
> > > command that was run... and then look closely at the forecast
and
> > > observation data being compared. Hopefully then we'll be able to
see
> > where
> > > the problem lies... whether it be a problem in Point-Stat,
METplus
> > > wrappers, or your configuration.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Dec 7, 2020 at 1:39 PM Edward Strobach - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Dec 07 13:39:04 2020: Request 97794 was acted upon.
> > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > > Queue: met_help
> > > > Subject: processing anomaly resulting in zero matched
pairs
> > > > Owner: Nobody
> > > > Requestors: edward.strobach at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97794
> > >
> > > >
> > > >
> > > > The last couple of days I've noticed that my METPlus routine
has not
> > been
> > > > processing ozone. Nothing has changed in the set-up, so I
thought it
> > > might
> > > > be related to missing data, either obs. or from the model
output. It
> > > turns
> > > > out that this is not the case.
> > > >
> > > > Going through the log files I found something unexpected.
Here is a
> > > sample
> > > > from demonstrating when it fails (i.e. no matched pairs)
> > > >
> > > > DEBUG 3: Rejected: bad climo stdev = 0
> > > > DEBUG 3: Rejected: duplicates = 0
> > > > DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation
type
> > AIRNOW,
> > > > over region NEC, for interpolation method BILIN(4), using 0
matched
> > > pairs.
> > > > DEBUG 3: Number of matched pairs = 0
> > > > DEBUG 3: Observations processed = 42282
> > > > DEBUG 3: Rejected: station id = 0
> > > > DEBUG 3: Rejected: obs type = 0
> > > > DEBUG 3: Rejected: valid time = 42282
> > > >
> > > > Here is an example when it is successful:
> > > >
> > > > DEBUG 3: Rejected: bad climo mean = 0
> > > > DEBUG 3: Rejected: bad climo stdev = 0
> > > > DEBUG 3: Rejected: duplicates = 0
> > > > DEBUG 2: Processing OZCON/A8 versus COPO/A8, for observation
type
> > AIRNOW,
> > > > over region Urban, for interpolation method BILIN(4), using
145
> matched
> > > > pairs.
> > > > DEBUG 3: Number of matched pairs = 145
> > > > DEBUG 3: Observations processed = 42091
> > > > DEBUG 3: Rejected: station id = 0
> > > > DEBUG 3: Rejected: obs type = 0
> > > > DEBUG 3: Rejected: valid time = 40328
> > > >
> > > > If you look carefully you will see that the case where it
failed
> shows
> > > > OZCONA8 while the successful case shows OZCON/A8.
> > > >
> > > > I find nothing indicating the joining of these strings in the
> > point_stat
> > > > config. If you look at my shared_config file you will see
that I set
> > the
> > > > obsvar and fcst var correctly:
> > > >
> > > > FCST_VAR1_NAME = OZCON
> > > > FCST_VAR1_LEVELS = A1
> > > > OBS_VAR1_NAME= COPO
> > > > OBS_VAR1_LEVELS= A1
> > > > OBS_VAR1_OPTIONS = convert(x) = x * 10^9;
> > > > FCST_VAR2_NAME = OZCON
> > > > FCST_VAR2_LEVELS = A8
> > > > OBS_VAR2_NAME= COPO
> > > > OBS_VAR2_LEVELS= A8
> > > > OBS_VAR2_OPTIONS = convert(x) = x * 10^9;
> > > >
> > > >
> > > > I'm kind of at a loss here and was wondering what could have
caused
> > this.
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
More information about the Met_help
mailing list