[Met_help] [rt.rap.ucar.edu #60808] History for MADIS2NC and Duplicate observations
John Halley Gotway via RT
met_help at ucar.edu
Fri May 24 10:27:59 MDT 2013
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
MET Help,
I wanted to inform you of a few 'issues' I'm having with the METv4.0
package.
(1) I've been utilizing the software to process MADIS (metar) observations
using madis2nc (which seems to work fine). However, I'm not seeing the
precipitation, dew point, etc. observations being reformatted by madis2nc.
Hence, the ability to verify is limited to temperature, wind speed, hgt, u,
v, etc.
(2) There are 'duplicate' (i.e., same time) observations being processed by
point_stat (and also stat_analysis). I would like to have these
observations removed. Is there a way to do this with METv4.0?
Thanks in advance.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #60808] MADIS2NC and Duplicate observations
From: John Halley Gotway
Time: Thu Mar 28 15:13:21 2013
James,
Regarding the second point, the answer is yes. Please take a look in
the configuration file for Point-Stat at the "duplicate_flag" entry.
Listed below is an excerpt from the METv4.0/data/config/README file
explaining this option:
//
// The "duplicate_flag" entry specifies how to handle duplicate point
// observations in Point-Stat and Ensemble-Stat. Three techniques are
// currently supported:
//
// - "NONE" to use all point observations (legacy behavior)
// - "UNIQUE" if two or more observations match identically except
in the
// station ID field, only use a single observation
// - "SINGLE" if two or more observations appear at a single
location
// (lat, lon, level, elv), use only the observation that has the
valid
// time closest to the forecast valid time
//
// The reporting mechanism for this feature can be activated by
specifying
// a verbosity level of three or higher. The report will show
information
// about where duplicates were detected and which observations were
used
// in those cases.
//
duplicate_flag = NONE;
If you want to use only one observation for each station in your time
window, I'd suggest using the "SINGLE" setting.
Regarding the first point, would you be able to send me an example of
this behavior? I'd need:
- the input madis NetCDF file
- the command line you're using to call madis2nc
- a description of what's missing from the output NetCDF file
I'll try to replicate the behavior here and investigate the reason for
it. You could post this data to our anonymous ftp site following
these instructions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Thanks,
John
On 03/28/2013 02:05 PM, James Cipriani via RT wrote:
>
> Thu Mar 28 14:05:43 2013: Request 60808 was acted upon.
> Transaction: Ticket created by jcipriani25 at gmail.com
> Queue: met_help
> Subject: MADIS2NC and Duplicate observations
> Owner: Nobody
> Requestors: jcipriani25 at gmail.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60808 >
>
>
> MET Help,
> I wanted to inform you of a few 'issues' I'm having with the METv4.0
> package.
>
> (1) I've been utilizing the software to process MADIS (metar)
observations
> using madis2nc (which seems to work fine). However, I'm not seeing
the
> precipitation, dew point, etc. observations being reformatted by
madis2nc.
> Hence, the ability to verify is limited to temperature, wind speed,
hgt, u,
> v, etc.
> (2) There are 'duplicate' (i.e., same time) observations being
processed by
> point_stat (and also stat_analysis). I would like to have these
> observations removed. Is there a way to do this with METv4.0?
>
> Thanks in advance.
>
------------------------------------------------
Subject: MADIS2NC and Duplicate observations
From: James Cipriani
Time: Fri Mar 29 12:46:09 2013
John,
Thanks for the reply. I was able to successfully utilize the
'single/unique' option in the configuration file (I recall seeing a
post
related to this issue in v3.1 as well, hence the fix).
Regarding the madis2nc issue, here is some information:
(1) The netCDF file could be one from the online MADIS (metar)
archive,
which was referenced in a previous forum note. For example:
ftp://pftp.madis-
data.noaa.gov/archive/2012/07/05/point/metar/netcdf/20120705_1500.gz
(2) The madis2nc command (after renaming the file) could be something
like:
/bin/madis2nc madis_20120705_1500.nc metar_madis2nc_20120705_1500.nc
-type
metar -qc_dd K,V,B
(3) Some (most) of the sites in the newly-created netCDF file list the
following fields:
geopotential height
2-m temperature
10-m wind speed
wind-direction
10-m U
10-m V
MSLP
precipitation (after further analysis, I realized that this field was
present for some sites but not all)
However, I don't see derived dew point in the madis2nc files. Is
there a
way to extract this derived field from the raw MADIS netCDF files?
Thanks in advance.
On Thu, Mar 28, 2013 at 5:13 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> James,
>
> Regarding the second point, the answer is yes. Please take a look
in the
> configuration file for Point-Stat at the "duplicate_flag" entry.
>
> Listed below is an excerpt from the METv4.0/data/config/README file
> explaining this option:
>
> //
> // The "duplicate_flag" entry specifies how to handle duplicate
point
> // observations in Point-Stat and Ensemble-Stat. Three techniques
are
> // currently supported:
> //
> // - "NONE" to use all point observations (legacy behavior)
> // - "UNIQUE" if two or more observations match identically
except in
> the
> // station ID field, only use a single observation
> // - "SINGLE" if two or more observations appear at a single
location
> // (lat, lon, level, elv), use only the observation that has
the valid
> // time closest to the forecast valid time
> //
> // The reporting mechanism for this feature can be activated by
specifying
> // a verbosity level of three or higher. The report will show
information
> // about where duplicates were detected and which observations were
used
> // in those cases.
> //
> duplicate_flag = NONE;
>
> If you want to use only one observation for each station in your
time
> window, I'd suggest using the "SINGLE" setting.
>
> Regarding the first point, would you be able to send me an example
of this
> behavior? I'd need:
> - the input madis NetCDF file
> - the command line you're using to call madis2nc
> - a description of what's missing from the output NetCDF file
>
> I'll try to replicate the behavior here and investigate the reason
for it.
> You could post this data to our anonymous ftp site following these
> instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
>
> On 03/28/2013 02:05 PM, James Cipriani via RT wrote:
> >
> > Thu Mar 28 14:05:43 2013: Request 60808 was acted upon.
> > Transaction: Ticket created by jcipriani25 at gmail.com
> > Queue: met_help
> > Subject: MADIS2NC and Duplicate observations
> > Owner: Nobody
> > Requestors: jcipriani25 at gmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60808 >
> >
> >
> > MET Help,
> > I wanted to inform you of a few 'issues' I'm having with the
METv4.0
> > package.
> >
> > (1) I've been utilizing the software to process MADIS (metar)
> observations
> > using madis2nc (which seems to work fine). However, I'm not
seeing the
> > precipitation, dew point, etc. observations being reformatted by
> madis2nc.
> > Hence, the ability to verify is limited to temperature, wind
speed, hgt,
> u,
> > v, etc.
> > (2) There are 'duplicate' (i.e., same time) observations being
processed
> by
> > point_stat (and also stat_analysis). I would like to have these
> > observations removed. Is there a way to do this with METv4.0?
> >
> > Thanks in advance.
> >
>
>
------------------------------------------------
Subject: MADIS2NC and Duplicate observations
From: James Cipriani
Time: Fri Mar 29 13:11:40 2013
Found the issue for dew point.
In src/tools/other/madis2nc/madis2nc.cc, line 856 ... dew point GRIB
code
is listed as ' 7 ' ... but it should be ' 17 '. This is causing
point_stat
to not find dew point when attempting to verify.
Thanks for all your assistance.
---------- Forwarded message ----------
From: James Cipriani <jcipriani25 at gmail.com>
Date: Fri, Mar 29, 2013 at 2:46 PM
Subject: Re: [rt.rap.ucar.edu #60808] MADIS2NC and Duplicate
observations
To: met_help at ucar.edu
John,
Thanks for the reply. I was able to successfully utilize the
'single/unique' option in the configuration file (I recall seeing a
post
related to this issue in v3.1 as well, hence the fix).
Regarding the madis2nc issue, here is some information:
(1) The netCDF file could be one from the online MADIS (metar)
archive,
which was referenced in a previous forum note. For example:
ftp://pftp.madis-
data.noaa.gov/archive/2012/07/05/point/metar/netcdf/20120705_1500.gz
(2) The madis2nc command (after renaming the file) could be something
like:
/bin/madis2nc madis_20120705_1500.nc metar_madis2nc_20120705_1500.nc
-type
metar -qc_dd K,V,B
(3) Some (most) of the sites in the newly-created netCDF file list the
following fields:
geopotential height
2-m temperature
10-m wind speed
wind-direction
10-m U
10-m V
MSLP
precipitation (after further analysis, I realized that this field was
present for some sites but not all)
However, I don't see derived dew point in the madis2nc files. Is
there a
way to extract this derived field from the raw MADIS netCDF files?
Thanks in advance.
On Thu, Mar 28, 2013 at 5:13 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> James,
>
> Regarding the second point, the answer is yes. Please take a look
in the
> configuration file for Point-Stat at the "duplicate_flag" entry.
>
> Listed below is an excerpt from the METv4.0/data/config/README file
> explaining this option:
>
> //
> // The "duplicate_flag" entry specifies how to handle duplicate
point
> // observations in Point-Stat and Ensemble-Stat. Three techniques
are
> // currently supported:
> //
> // - "NONE" to use all point observations (legacy behavior)
> // - "UNIQUE" if two or more observations match identically
except in
> the
> // station ID field, only use a single observation
> // - "SINGLE" if two or more observations appear at a single
location
> // (lat, lon, level, elv), use only the observation that has
the valid
> // time closest to the forecast valid time
> //
> // The reporting mechanism for this feature can be activated by
specifying
> // a verbosity level of three or higher. The report will show
information
> // about where duplicates were detected and which observations were
used
> // in those cases.
> //
> duplicate_flag = NONE;
>
> If you want to use only one observation for each station in your
time
> window, I'd suggest using the "SINGLE" setting.
>
> Regarding the first point, would you be able to send me an example
of this
> behavior? I'd need:
> - the input madis NetCDF file
> - the command line you're using to call madis2nc
> - a description of what's missing from the output NetCDF file
>
> I'll try to replicate the behavior here and investigate the reason
for it.
> You could post this data to our anonymous ftp site following these
> instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
>
> On 03/28/2013 02:05 PM, James Cipriani via RT wrote:
> >
> > Thu Mar 28 14:05:43 2013: Request 60808 was acted upon.
> > Transaction: Ticket created by jcipriani25 at gmail.com
> > Queue: met_help
> > Subject: MADIS2NC and Duplicate observations
> > Owner: Nobody
> > Requestors: jcipriani25 at gmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60808 >
> >
> >
> > MET Help,
> > I wanted to inform you of a few 'issues' I'm having with the
METv4.0
> > package.
> >
> > (1) I've been utilizing the software to process MADIS (metar)
> observations
> > using madis2nc (which seems to work fine). However, I'm not
seeing the
> > precipitation, dew point, etc. observations being reformatted by
> madis2nc.
> > Hence, the ability to verify is limited to temperature, wind
speed, hgt,
> u,
> > v, etc.
> > (2) There are 'duplicate' (i.e., same time) observations being
processed
> by
> > point_stat (and also stat_analysis). I would like to have these
> > observations removed. Is there a way to do this with METv4.0?
> >
> > Thanks in advance.
> >
>
>
------------------------------------------------
Subject: Re: Fwd: [rt.rap.ucar.edu #60808] MADIS2NC and Duplicate observations
From: John Halley Gotway
Time: Fri Mar 29 13:36:10 2013
James,
That's great. Thanks for finding that and letting us know about it.
I've fixed it in the development version and will include it in the
bug fixes for METv4.0.
Is that the extent of the issues you were seeing? Or are you still
not seeing precip in the output when you expect to?
Thanks,
John
On 03/29/2013 01:11 PM, James Cipriani via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60808 >
>
> Found the issue for dew point.
> In src/tools/other/madis2nc/madis2nc.cc, line 856 ... dew point GRIB
code
> is listed as ' 7 ' ... but it should be ' 17 '. This is causing
point_stat
> to not find dew point when attempting to verify.
> Thanks for all your assistance.
>
> ---------- Forwarded message ----------
> From: James Cipriani <jcipriani25 at gmail.com>
> Date: Fri, Mar 29, 2013 at 2:46 PM
> Subject: Re: [rt.rap.ucar.edu #60808] MADIS2NC and Duplicate
observations
> To: met_help at ucar.edu
>
>
> John,
> Thanks for the reply. I was able to successfully utilize the
> 'single/unique' option in the configuration file (I recall seeing a
post
> related to this issue in v3.1 as well, hence the fix).
> Regarding the madis2nc issue, here is some information:
>
> (1) The netCDF file could be one from the online MADIS (metar)
archive,
> which was referenced in a previous forum note. For example:
>
> ftp://pftp.madis-
data.noaa.gov/archive/2012/07/05/point/metar/netcdf/20120705_1500.gz
>
> (2) The madis2nc command (after renaming the file) could be
something like:
>
> /bin/madis2nc madis_20120705_1500.nc metar_madis2nc_20120705_1500.nc
-type
> metar -qc_dd K,V,B
>
> (3) Some (most) of the sites in the newly-created netCDF file list
the
> following fields:
>
> geopotential height
> 2-m temperature
> 10-m wind speed
> wind-direction
> 10-m U
> 10-m V
> MSLP
> precipitation (after further analysis, I realized that this field
was
> present for some sites but not all)
>
> However, I don't see derived dew point in the madis2nc files. Is
there a
> way to extract this derived field from the raw MADIS netCDF files?
>
> Thanks in advance.
>
>
> On Thu, Mar 28, 2013 at 5:13 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> James,
>>
>> Regarding the second point, the answer is yes. Please take a look
in the
>> configuration file for Point-Stat at the "duplicate_flag" entry.
>>
>> Listed below is an excerpt from the METv4.0/data/config/README file
>> explaining this option:
>>
>> //
>> // The "duplicate_flag" entry specifies how to handle duplicate
point
>> // observations in Point-Stat and Ensemble-Stat. Three techniques
are
>> // currently supported:
>> //
>> // - "NONE" to use all point observations (legacy behavior)
>> // - "UNIQUE" if two or more observations match identically
except in
>> the
>> // station ID field, only use a single observation
>> // - "SINGLE" if two or more observations appear at a single
location
>> // (lat, lon, level, elv), use only the observation that has
the valid
>> // time closest to the forecast valid time
>> //
>> // The reporting mechanism for this feature can be activated by
specifying
>> // a verbosity level of three or higher. The report will show
information
>> // about where duplicates were detected and which observations were
used
>> // in those cases.
>> //
>> duplicate_flag = NONE;
>>
>> If you want to use only one observation for each station in your
time
>> window, I'd suggest using the "SINGLE" setting.
>>
>> Regarding the first point, would you be able to send me an example
of this
>> behavior? I'd need:
>> - the input madis NetCDF file
>> - the command line you're using to call madis2nc
>> - a description of what's missing from the output NetCDF file
>>
>> I'll try to replicate the behavior here and investigate the reason
for it.
>> You could post this data to our anonymous ftp site following
these
>> instructions:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Thanks,
>> John
>>
>>
>> On 03/28/2013 02:05 PM, James Cipriani via RT wrote:
>>>
>>> Thu Mar 28 14:05:43 2013: Request 60808 was acted upon.
>>> Transaction: Ticket created by jcipriani25 at gmail.com
>>> Queue: met_help
>>> Subject: MADIS2NC and Duplicate observations
>>> Owner: Nobody
>>> Requestors: jcipriani25 at gmail.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=60808 >
>>>
>>>
>>> MET Help,
>>> I wanted to inform you of a few 'issues' I'm having with the
METv4.0
>>> package.
>>>
>>> (1) I've been utilizing the software to process MADIS (metar)
>> observations
>>> using madis2nc (which seems to work fine). However, I'm not
seeing the
>>> precipitation, dew point, etc. observations being reformatted by
>> madis2nc.
>>> Hence, the ability to verify is limited to temperature, wind
speed, hgt,
>> u,
>>> v, etc.
>>> (2) There are 'duplicate' (i.e., same time) observations being
processed
>> by
>>> point_stat (and also stat_analysis). I would like to have these
>>> observations removed. Is there a way to do this with METv4.0?
>>>
>>> Thanks in advance.
>>>
>>
>>
------------------------------------------------
More information about the Met_help
mailing list