[Met_help] [rt.rap.ucar.edu #56945] History for ASCII2NC Question (and NetCDF)

[Paul Oldenburg via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi,

I am re-familiarizing myself with MET, after a long hiatus away from working with it.  I've got version 3.1 installed and am working through the User's Guide.

I have reached a stumbling block at PointStat.  I'm reading NCEP's observation data and can't seem to find the secret code for the config file, placed into the obs_field=[] line, to stop it from tossing the synoptic records I want, citing  a 'level mismatch'.  The fcst_field=["11/Z2"] works just fine for pulling surface temperature from the GriB field.  Yet the only things I retain when leaving obs_field blank (which defaults this to whatever I said in fcst_field) are a smattering of co-op reports-all the synoptic ones are tossed.

I'd really like to know when I state 'Z2', 'P500', 'L3', or what not in fcst_field and obs_field, how that specifically directs the pull of data from a GriB file or NetCDF file.  I'm guessing "11/Z2" acts on kpds5 and kpds7 on the GriB file, but it's unclear what that does to NetCDF.  And I've never understood the syntax of the NetCDF fields, such as "TMP(0,5,*,*)", or how to set it correctly.

Another question: is there any setting that allows me to limit every station encountered in the NetCDF file to retain only one record per station?  I see a few stations with records every 10 minutes, and those all get pulled in to the statistics calculations.  I realize that when I'm setting the temporal window width in seconds in the config file that I'm opening myself up to getting a lot more data than I want, but I have to use something a little wider than +- 60 seconds, centered at the top of the hour, to get all the surface synoptic reports at, say, 2353Z or 0000Z.  I see no optimal setting here to get one record from as many different stations as possible and avoiding one station having multiple records because the window I set was too wide to stop multiples from getting in.

Next thought: if NCEP's NetCDF obs data don't work for me, maybe I'd be better off converting our in-house observation data to NetCDF using ASCII2NC.  I looked at the examples provided and it's unclear what limitations there are on the input ASCII data... for example, are the call fields limited to 5 characters?  We have longer identifiers that we employ.  Also, given my above question about how to get the synoptic data out of NCEP's NetCDF observation files, should I choose to use ASCII2NC instead, is there a special setting for surface records (such as 0 meter height) that I need to set to make sure the corresponding setting for obs_field in the config file allows me to correctly pull the records I want?

Finally, how much does it cost to request a format statement within ASCII2NC to simplify the conversion? :)  One suggestion: if you use space limited inputs exclusively, how about a format line that allows me to name the fields in order, such as "CALLS LAT LON ELEVATION 11 7 33 34 52" etc., such that I could easily dictate to the converter which fields appear in which order.  Our standard is to put all the data in a single record, similar to a METAR report, whereas NetCDF would break the one METAR record into multiple records, one for each field it could pull from the METAR (e.g., temperature, wind, SLP, etc.).  Just a thought.

Thanks,
Matt

// SIGNED //
Matthew C. Sittel
Associate Scientist
University Corporation for Atmospheric Research
16WS/WXN, Offutt AFB, NE
(402) 294-3473  DSN: 271-3473



----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #56945] ASCII2NC Question (and NetCDF)
From: Paul Oldenburg
Time: Tue Jun 12 11:29:38 2012

Matt,

If your synoptic observations have a surface message type (ADPSFC,
SFCSHP or ONLYSF), then you need to set the
point_stat config file message_type to match that.  In this case,
point_stat basically ignores the level information
specified in the fcst_field/obs_field setting.  When using the
"[grib_code]/[level_type][level]" notation with MET point
NetCDF files:

* grib code is matched against the value in the second field of the
obs_arr variable
* level_type indicates whether or not to match against the third
(pressure/accumulation - P or A)
   or fourth (MSL - L or Z) field in obs_arr; this applies only when
the message_type is anything other
   than ADPSFC, SFCSHP or ONLYSF - see point_stat config file setting
message_type comments
* level is matched against the value in whatever field was specified
by level_type

If you are having trouble keeping the correct obs, try setting the
config obs_field/message_type settings to different
values until you find that it keeps the desired observations.

Regarding "Another question", there is no flag in METv3.1 to tell
point_stat to keep only the first obs from each
station.  The good news is that there is a feature in METv4.0 (due out
in a month) that will allow the user to do this.

Regarding "Next thought": there are no limits on the width of the
message_type or station_id fields, within reason.  Use
the message_type ADPSFC or SFCSHP if you want to designate it was a
"surface" observation, as explained above.

Regarding "Finally", in the future, the ascii2nc -format option that
you see in the usage statement may provide
functionality to support an arbitrary input field order.
Unfortunately, this option is not fully supported at this
time, so you will have to unleash your scripting ninja skills on the
input file to format it correctly.

Please let me know if you have any other questions.

Paul


On 06/12/2012 09:52 AM, matthew.sittel.ctr at offutt.af.mil via RT wrote:
>
> Tue Jun 12 09:52:42 2012: Request 56945 was acted upon.
> Transaction: Ticket created by matthew.sittel.ctr at offutt.af.mil
>         Queue: met_help
>       Subject: ASCII2NC Question (and NetCDF)
>         Owner: Nobody
>    Requestors: matthew.sittel.ctr at offutt.af.mil
>        Status: new
>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56945 >
>
>
> Hi,
>
> I am re-familiarizing myself with MET, after a long hiatus away from
working with it.  I've got version 3.1 installed and am working
through the User's Guide.
>
> I have reached a stumbling block at PointStat.  I'm reading NCEP's
observation data and can't seem to find the secret code for the config
file, placed into the obs_field=[] line, to stop it from tossing the
synoptic records I want, citing  a 'level mismatch'.  The
fcst_field=["11/Z2"] works just fine for pulling surface temperature
from the GriB field.  Yet the only things I retain when leaving
obs_field blank (which defaults this to whatever I said in fcst_field)
are a smattering of co-op reports-all the synoptic ones are tossed.
>
> I'd really like to know when I state 'Z2', 'P500', 'L3', or what not
in fcst_field and obs_field, how that specifically directs the pull of
data from a GriB file or NetCDF file.  I'm guessing "11/Z2" acts on
kpds5 and kpds7 on the GriB file, but it's unclear what that does to
NetCDF.  And I've never understood the syntax of the NetCDF fields,
such as "TMP(0,5,*,*)", or how to set it correctly.
>
> Another question: is there any setting that allows me to limit every
station encountered in the NetCDF file to retain only one record per
station?  I see a few stations with records every 10 minutes, and
those all get pulled in to the statistics calculations.  I realize
that when I'm setting the temporal window width in seconds in the
config file that I'm opening myself up to getting a lot more data than
I want, but I have to use something a little wider than +- 60 seconds,
centered at the top of the hour, to get all the surface synoptic
reports at, say, 2353Z or 0000Z.  I see no optimal setting here to get
one record from as many different stations as possible and avoiding
one station having multiple records because the window I set was too
wide to stop multiples from getting in.
>
> Next thought: if NCEP's NetCDF obs data don't work for me, maybe I'd
be better off converting our in-house observation data to NetCDF using
ASCII2NC.  I looked at the examples provided and it's unclear what
limitations there are on the input ASCII data... for example, are the
call fields limited to 5 characters?  We have longer identifiers that
we employ.  Also, given my above question about how to get the
synoptic data out of NCEP's NetCDF observation files, should I choose
to use ASCII2NC instead, is there a special setting for surface
records (such as 0 meter height) that I need to set to make sure the
corresponding setting for obs_field in the config file allows me to
correctly pull the records I want?
>
> Finally, how much does it cost to request a format statement within
ASCII2NC to simplify the conversion? :)  One suggestion: if you use
space limited inputs exclusively, how about a format line that allows
me to name the fields in order, such as "CALLS LAT LON ELEVATION 11 7
33 34 52" etc., such that I could easily dictate to the converter
which fields appear in which order.  Our standard is to put all the
data in a single record, similar to a METAR report, whereas NetCDF
would break the one METAR record into multiple records, one for each
field it could pull from the METAR (e.g., temperature, wind, SLP,
etc.).  Just a thought.
>
> Thanks,
> Matt
>
> // SIGNED //
> Matthew C. Sittel
> Associate Scientist
> University Corporation for Atmospheric Research
> 16WS/WXN, Offutt AFB, NE
> (402) 294-3473  DSN: 271-3473
>



------------------------------------------------