[Met_help] [rt.rap.ucar.edu #87334] History for Can MET remove the duplicated data before test?

Wed Oct 10 13:21:08 MDT 2018

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

To whom it may concern,
Some collected data inputting to MET are duplicated, is it possible to remove them while MET testing? I am not sure if the logical setting, duplicate_flag = NONE,can work for the issue and I don't know how to settle it down if the logical setting can work for it. Should I resolve the problem before the tested data entering the MET by myself? 
I am looking forward your news.
Thank you.
Sincerely
Yu

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Can MET remove the duplicated data before test?
From: John Halley Gotway
Time: Wed Oct 10 12:33:40 2018

Hello Yu,

I see you have a question about duplicate data being passed to MET.
I'm am
not sure that I completely understand your question, but by guess is
that
by "duplicate" data you mean duplicate point observations.

If that's the case, then you're on the right track.  When you run the
Point-Stat and Ensemble-Stat tools, the "duplicate_flag" option may be
set
to tell those tools how to handle duplicate point observations.

There is a related option called "obs_summary".  That is used when
multiple
observations occur at the same station within the time window.

Below I've cut-and-pasted a selection from the MET User's Guide about
these
settings.  Also, try running Point-Stat using verbosity level 3 (-v 3)
to
see a list of the duplicates that it found.

I would suggest using:
duplicate_flag = UNIQUE;
obs_summary = NEAREST;

That would (1) toss out any true duplicate observations and (2) only
use
the observation that closest in time to the forecast valid time.

Thanks,
John Halley Gotway

//
// The "duplicate_flag" entry specifies how to handle duplicate point
// observations in Point-Stat and Ensemble-Stat:
//
//    - "NONE" to use all point observations (legacy behavior)
//    - "UNIQUE" only use a single observation if two or more
observations
//      match. Matching observations are determined if they contain
identical
//      latitude, longitude, level, elevation, and time information.
//      They may contain different observation values or station IDs
//
// The reporting mechanism for this feature can be activated by
specifying
// a verbosity level of three or higher.  The report will show
information
// about where duplicates were detected and which observations were
used
// in those cases.
//
duplicate_flag = NONE;

//
// The "obs_summary" entry specifies how to compute statistics on
// observations that appear at a single location (lat,lon,level,elev)
// in Point-Stat and Ensemble-Stat.  Eight techniques are
// currently supported:
//
//    - "NONE" to use all point observations (legacy behavior)
//    - "NEAREST" use only the observation that has the valid
//      time closest to the forecast valid time
//    - "MIN" use only the observation that has the lowest value
//    - "MAX" use only the observation that has the highest value
//    - "UW_MEAN" compute an unweighted mean of the observations
//    - "DW_MEAN" compute a weighted mean of the observations based
//      on the time of the observation
//    - "MEDIAN" use the median observation
//    - "PERC" use the Nth percentile observation where N =
obs_perc_value
//
// The reporting mechanism for this feature can be activated by
specifying
// a verbosity level of three or higher.  The report will show
information
// about where duplicates were detected and which observations were
used
// in those cases.
//
obs_summary = NONE;

On Wed, Oct 10, 2018 at 11:58 AM Yu Xin via RT <met_help at ucar.edu>
wrote:

>
> Wed Oct 10 11:58:13 2018: Request 87334 was acted upon.
> Transaction: Ticket created by learnerxy at 163.com
>        Queue: met_help
>      Subject: Can MET remove the duplicated data before test?
>        Owner: Nobody
>   Requestors: learnerxy at 163.com
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87334 >
>
>
> To whom it may concern,
> Some collected data inputting to MET are duplicated, is it possible
to
> remove them while MET testing? I am not sure if the logical setting,
> duplicate_flag = NONE,can work for the issue and I don't know how to
settle
> it down if the logical setting can work for it. Should I resolve the
> problem before the tested data entering the MET by myself?
> I am looking forward your news.
> Thank you.
> Sincerely
> Yu
>

------------------------------------------------
Subject: Can MET remove the duplicated data before test?
From: Yu Xin
Time: Wed Oct 10 13:17:31 2018

Dear John,

You completely understand my issue and resolve the problem I
mentioned. Thank you so much. You can close the ticket now.

Sincerely
Yu

At 2018-10-11 02:33:40, "John Halley Gotway via RT"
<met_help at ucar.edu> wrote:
>Hello Yu,
>
>I see you have a question about duplicate data being passed to MET.
I'm am
>not sure that I completely understand your question, but by guess is
that
>by "duplicate" data you mean duplicate point observations.
>
>If that's the case, then you're on the right track.  When you run the
>Point-Stat and Ensemble-Stat tools, the "duplicate_flag" option may
be set
>to tell those tools how to handle duplicate point observations.
>
>There is a related option called "obs_summary".  That is used when
multiple
>observations occur at the same station within the time window.
>
>Below I've cut-and-pasted a selection from the MET User's Guide about
these
>settings.  Also, try running Point-Stat using verbosity level 3 (-v
3) to
>see a list of the duplicates that it found.
>
>I would suggest using:
>duplicate_flag = UNIQUE;
>obs_summary = NEAREST;
>
>That would (1) toss out any true duplicate observations and (2) only
use
>the observation that closest in time to the forecast valid time.
>
>Thanks,
>John Halley Gotway
>
>//
>// The "duplicate_flag" entry specifies how to handle duplicate point
>// observations in Point-Stat and Ensemble-Stat:
>//
>//    - "NONE" to use all point observations (legacy behavior)
>//    - "UNIQUE" only use a single observation if two or more
observations
>//      match. Matching observations are determined if they contain
>identical
>//      latitude, longitude, level, elevation, and time information.
>//      They may contain different observation values or station IDs
>//
>// The reporting mechanism for this feature can be activated by
specifying
>// a verbosity level of three or higher.  The report will show
information
>// about where duplicates were detected and which observations were
used
>// in those cases.
>//
>duplicate_flag = NONE;
>
>//
>// The "obs_summary" entry specifies how to compute statistics on
>// observations that appear at a single location (lat,lon,level,elev)
>// in Point-Stat and Ensemble-Stat.  Eight techniques are
>// currently supported:
>//
>//    - "NONE" to use all point observations (legacy behavior)
>//    - "NEAREST" use only the observation that has the valid
>//      time closest to the forecast valid time
>//    - "MIN" use only the observation that has the lowest value
>//    - "MAX" use only the observation that has the highest value
>//    - "UW_MEAN" compute an unweighted mean of the observations
>//    - "DW_MEAN" compute a weighted mean of the observations based
>//      on the time of the observation
>//    - "MEDIAN" use the median observation
>//    - "PERC" use the Nth percentile observation where N =
obs_perc_value
>//
>// The reporting mechanism for this feature can be activated by
specifying
>// a verbosity level of three or higher.  The report will show
information
>// about where duplicates were detected and which observations were
used
>// in those cases.
>//
>obs_summary = NONE;
>
>
>
>
>
>On Wed, Oct 10, 2018 at 11:58 AM Yu Xin via RT <met_help at ucar.edu>
wrote:
>
>>
>> Wed Oct 10 11:58:13 2018: Request 87334 was acted upon.
>> Transaction: Ticket created by learnerxy at 163.com
>>        Queue: met_help
>>      Subject: Can MET remove the duplicated data before test?
>>        Owner: Nobody
>>   Requestors: learnerxy at 163.com
>>       Status: new
>>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87334 >
>>
>>
>> To whom it may concern,
>> Some collected data inputting to MET are duplicated, is it possible
to
>> remove them while MET testing? I am not sure if the logical
setting,
>> duplicate_flag = NONE,can work for the issue and I don't know how
to settle
>> it down if the logical setting can work for it. Should I resolve
the
>> problem before the tested data entering the MET by myself?
>> I am looking forward your news.
>> Thank you.
>> Sincerely
>> Yu
>>

------------------------------------------------