[Met_help] ascii2nc and point_stat: an update

[John Halley Gotway]

Jonathan,

Ok, the other way you could disable the computation of vector stats would be to NOT list UGRD followed by VGRD.  MET interprets U followed by V at the same level to mean that you want to compute
vector stats.  For example, if you were to list V followed by U, it would treat them separately as scalars.  I realize this seems a bit silly.

I just updated the code for the next release so that if you turn off VL1L2 and VAL1L2 in the output_flag, Point-Stat will not try to compute vector stats.  I think this method of disabling vector
stats is a lot more intuitive.

And thanks for sending those MADIS scripts.  I appreciate it.

Thanks,
John

Case, Jonathan (MSFC-VP61)[Other] wrote:
> John,
> 
> All is working well in point_stat now with the corrections to ascii2nc
> that you gave me.  THANK YOU!!
> 
> BTW, it didn't matter if I turned off the VL1L2 output in point_stat.
> The UGRD and VGRD still were unable to process through point_stat with
> the incorrectly formatted ascii2nc output.  
> 
> I'll send you the scripts and accompanying required MADIS files that
> have fields that are substituted by my script in a later email.  I'm
> adding some more comments prior to sending it your way.  It may not be
> the prettiest nor most efficient method, but it works anyway!  
> 
> Jonathan
> 
> 
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>> Sent: Wednesday, March 04, 2009 9:48 AM
>> To: Case, Jonathan (MSFC-VP61)[Other]; met_help at ucar.edu
>> Subject: Re: ascii2nc and point_stat: an update
>>
>> Jonathan,
>>
>> Thanks for the information about MADIS data.  That is definitely an
>> observation dataset that we've been hearing more about from users.  In
>> the long run, we'd like to make it easier to use MADIS
>> observations in MET - perhaps reading those observations directly.
> But
>> in the near term, it sounds like your scripts may work well.  Once you
>> have your scripts in good shape, if you'd be willing to
>> send us a copy, I'd really appreciate it.  We could post them
> somewhere
>> on the MET website as an example of how one can prepare MADIS
>> observations for use in MET.
>>
>> To answer your question, I believe that the only observations that
>> really need to be "grouped" together with the same header id are U and
>> V.  Point-Stat will just process through the observations in
>> the "obs_arr" variables one-by-one, and it'll read the header data to
>> which that observation points.  So U and V are the only ones with
>> restrictions.
>>
>> Also, I should have pointed out earlier, when you said that you
>> couldn't get Point-Stat to run at all with U and V... there's an easy
>> fix for that.  If you turn off the VL1L2 output in the
>> "output_flag" in the config file, Point-Stat should not attempt to
>> process the U and V as vectors.  Instead, it'll treat them as separate
>> scalars and generate an SL1L2 line for U and an SL1L2 line for
>> V.  So you shouldn't see that error message you were seeing.
>>
>> But in order to process them as a vectors, they do need to be in
> order,
>> U followed by V, with the same header id.
>>
>> Thanks,
>> John
>>
>>
>>
>>
>>
>> Case, Jonathan (MSFC-VP61)[Other] wrote:
>>> John,
>>>
>>> I'm still on Beta6 when you first pointed me to the v2 of MET.  So,
>>> hopefully that won't be an issue.
>>>
>>> As a follow-on question (before I test the fix out), are all the
>>> wind-related variables usually grouped together in the same hdr_id,
>>> while the remaining variables from a particular observation is
>> grouped
>>> into another hdr_id, or is the only necessary item having the u- and
>>> v-wind components in the same hdr_id?
>>>
>>> The following actions I took to sort the data might be helpful to
> MET
>>> users who use the archived MADIS observations, which I just
>> implemented
>>> into a shell script:
>>>
>>> (1) Since the MADIS data are processed by variable, I found it
> easier
>> to
>>> write each individual variable output into a separate file and then
>>> merged them into a single daily file, removing all the headers that
>>> accompanied each variable output.
>>>
>>> (2) Once I obtained the single daily file and in the column format
>>> required for ascii2nc, I sorted the output by column3 (valid_time)
>> using
>>> the following command:  "cat daily.txt | sort -k3,3 | uniq >
>>> new_output_filename"
>>>
>>> I need to use the unique command "uniq" because I found many
>> duplicate
>>> entries being output by the MADIS API.
>>> Removing these extra observations might also speed up ascii2nc.
>>>
>>> I'll let you know how the testing goes with your fixes.
>>> Jonathan
>>>
>>>> -----Original Message-----
>>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>>> Sent: Tuesday, March 03, 2009 6:30 PM
>>>> To: Case, Jonathan (MSFC-VP61)[Other]
>>>> Cc: met_help at ucar.edu
>>>> Subject: Re: ascii2nc and point_stat: an update
>>>>
>>>> Jonathan,
>>>>
>>>> Good catch.  I'm glad you found that.
>>>>
>>>> Please copy two attached files into the directory
>>>> METv2.0beta8/lib/vx_util, rebuild, and then try rerunning your data
>>>> through ASCII2NC and Point-Stat.
>>>>
>>>> Please let me know if that does the trick for you.
>>>>
>>>> In ASCII2NC we are supposed to be keeping track of the header data
>> and
>>>> only incrementing the header id when the header data actually
>> changes.
>>>> However, the lines that are doing the comparison to see if the
>> message
>>>> type, station id, and valid time have changed were not working
>>>> properly.
>>>> Hopefully this will fix it.
>>>>
>>>> The reason we require that the U and V component of winds be
> grouped
>>>> together is to make things a bit simpler.  Since we have to deal
>> with
>>>> them as pairs in a couple of places in the code:
>>>> - in Point-Stat in the computation of VL1L2 vector partial sums
>>>> - in PB2NC in deriving wind speed
>>>> It's easier to just require that one follows the other.  Otherwise,
>>>> we'd waste a lot of time hunting through thousands the observations
>>>> looking for U's and V's to match up.  I'm glad you were able to
>>> reorder
>>>> you data to accomodate that.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>>> John,
>>>>>
>>>>>
>>>>>
>>>>> Well, I tried sorting my daily output file by valid time.  As a
>>>>> result, I have each individual observation with all the variables
>>>>> grouped together on consecutive lines in the input file to
>> ascii2nc.
>>>>> However, the netcdf output from ascii2nc still did not group the
>>>>> common wind observations into the same hdr_id.  Could this be a
>>>> problem with the
>>>>> ascii2nc program in v2Beta?    It seems like the wind component
>> data
>>>>> need to be written to the same hdr_id in the obs_arr().
>>>>>
>>>>>
>>>>>
>>>>> For example, here's what I see in the PB2NC ncdump output:
>>>>>
>>>>>
>>>>>
>>>>> obs_arr =
>>>>>
>>>>>   0, 33, 1011.4, -9999, -5.1,
>>>>>
>>>>>   0, 34, 1011.4, -9999, 0,
>>>>>
>>>>>   0, 32, 1011.4, -9999, 5.1,
>>>>>
>>>>>   1, 11, 1011.4, 4.00206, 302.05,
>>>>>
>>>>>   1, 7, 1011.4, 4.00206, 4,
>>>>>
>>>>>   1, 17, 1011.4, 4.00206, 297.8651,
>>>>>
>>>>>   1, 2, 1011.4, 4.00206, 101186.8,
>>>>>
>>>>>   2, 33, 1012.1, -9999, -4,
>>>>>
>>>>>   2, 34, 1012.1, -9999, 0.7,
>>>>>
>>>>>   2, 32, 1012.1, -9999, 4.060788,
>>>>>
>>>>> .
>>>>>
>>>>> .
>>>>>
>>>>> .
>>>>>
>>>>>
>>>>>
>>>>> Notice that any wind information is grouped together under the
> same
>>>>> hdr_id (first column) whereas the other remaining scalar variables
>>>>> (T/Z/Td/PMSL) are grouped under their own common hdr_id.
>>>>>
>>>>>
>>>>>
>>>>> Meanwhile, the output from ascii2nc shows up as follows in an
>>> ncdump:
>>>>>
>>>>> obs_arr =
>>>>>
>>>>>   0, 11, -9999, 4, 301.15,
>>>>>
>>>>>   1, 17, -9999, 4, 296.15,
>>>>>
>>>>>   2, 2, -9999, 4, 101660,
>>>>>
>>>>>   3, 31, -9999, 4, 77.69,
>>>>>
>>>>>   4, 32, -9999, 4, 4.63,
>>>>>
>>>>>   5, 33, -9999, 4, -4.52,
>>>>>
>>>>>   6, 34, -9999, 4, -0.99,
>>>>>
>>>>>   7, 11, -9999, 319, 302.15,
>>>>>
>>>>>   8, 17, -9999, 319, 290.15,
>>>>>
>>>>>   9, 31, -9999, 319, 269.8,
>>>>>
>>>>>   10, 32, -9999, 319, 3.6,
>>>>>
>>>>>   11, 33, -9999, 319, 3.6,
>>>>>
>>>>>   12, 34, -9999, 319, 0.01,
>>>>>
>>>>> .
>>>>>
>>>>> .
>>>>>
>>>>> .
>>>>>
>>>>>
>>>>>
>>>>> Notice how each new line of data corresponding to a new variable
>> has
>>>> a
>>>>> new hdr_id, instead of grouping them together as in the PB2NC
>>> output.
>>>>> This seems to correlate well with the point_stat error message I
>>>>> receive when trying to process the UGRD/VGRD/WIND variables:
>>>>>
>>>>>
>>>>>
>>>>> ERROR: process_obs_file() -> when computing vector winds, each
> UGRD
>>>>> observation must be followed by a VGRD observation with the same
>>>>> header and at the same level
>>>>>
>>>>>
>>>>>
>>>>> I thought this additional feedback might be helpful towards your
>>>>> diagnosis.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Jonathan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ***********************************************************
>>>>> Jonathan Case, ENSCO, Inc.
>>>>> Aerospace Sciences & Engineering Division Short-term Prediction
>>>>> Research and Transition Center 320 Sparkman Drive, Room 3062
>>>>> Huntsville, AL 35805-1912
>>>>> Voice: (256) 961-7504   Fax: (256) 961-7788
>>>>> Emails: Jonathan.Case-1 at nasa.gov
>>>>>
>>>>>              case.jonathan at ensco.com
>>>>>
>>>>> ***********************************************************
>>>>>
>>>>>
>>>>>
>>>>>