[Met_help] ascii2nc and point_stat: an update

[John Halley Gotway]

Jonathan,

Thanks for the information about MADIS data.  That is definitely an observation dataset that we've been hearing more about from users.  In the long run, we'd like to make it easier to use MADIS
observations in MET - perhaps reading those observations directly.  But in the near term, it sounds like your scripts may work well.  Once you have your scripts in good shape, if you'd be willing to
send us a copy, I'd really appreciate it.  We could post them somewhere on the MET website as an example of how one can prepare MADIS observations for use in MET.

To answer your question, I believe that the only observations that really need to be "grouped" together with the same header id are U and V.  Point-Stat will just process through the observations in
the "obs_arr" variables one-by-one, and it'll read the header data to which that observation points.  So U and V are the only ones with restrictions.

Also, I should have pointed out earlier, when you said that you couldn't get Point-Stat to run at all with U and V... there's an easy fix for that.  If you turn off the VL1L2 output in the
"output_flag" in the config file, Point-Stat should not attempt to process the U and V as vectors.  Instead, it'll treat them as separate scalars and generate an SL1L2 line for U and an SL1L2 line for
V.  So you shouldn't see that error message you were seeing.

But in order to process them as a vectors, they do need to be in order, U followed by V, with the same header id.

Thanks,
John





Case, Jonathan (MSFC-VP61)[Other] wrote:
> John,
> 
> I'm still on Beta6 when you first pointed me to the v2 of MET.  So,
> hopefully that won't be an issue. 
> 
> As a follow-on question (before I test the fix out), are all the
> wind-related variables usually grouped together in the same hdr_id,
> while the remaining variables from a particular observation is grouped
> into another hdr_id, or is the only necessary item having the u- and
> v-wind components in the same hdr_id?  
> 
> The following actions I took to sort the data might be helpful to MET
> users who use the archived MADIS observations, which I just implemented
> into a shell script:
> 
> (1) Since the MADIS data are processed by variable, I found it easier to
> write each individual variable output into a separate file and then
> merged them into a single daily file, removing all the headers that
> accompanied each variable output.
> 
> (2) Once I obtained the single daily file and in the column format
> required for ascii2nc, I sorted the output by column3 (valid_time) using
> the following command:  "cat daily.txt | sort -k3,3 | uniq >
> new_output_filename"
> 
> I need to use the unique command "uniq" because I found many duplicate
> entries being output by the MADIS API. 
> Removing these extra observations might also speed up ascii2nc.  
> 
> I'll let you know how the testing goes with your fixes.
> Jonathan
> 
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>> Sent: Tuesday, March 03, 2009 6:30 PM
>> To: Case, Jonathan (MSFC-VP61)[Other]
>> Cc: met_help at ucar.edu
>> Subject: Re: ascii2nc and point_stat: an update
>>
>> Jonathan,
>>
>> Good catch.  I'm glad you found that.
>>
>> Please copy two attached files into the directory
>> METv2.0beta8/lib/vx_util, rebuild, and then try rerunning your data
>> through ASCII2NC and Point-Stat.
>>
>> Please let me know if that does the trick for you.
>>
>> In ASCII2NC we are supposed to be keeping track of the header data and
>> only incrementing the header id when the header data actually changes.
>> However, the lines that are doing the comparison to see if the message
>> type, station id, and valid time have changed were not working
>> properly.
>> Hopefully this will fix it.
>>
>> The reason we require that the U and V component of winds be grouped
>> together is to make things a bit simpler.  Since we have to deal with
>> them as pairs in a couple of places in the code:
>> - in Point-Stat in the computation of VL1L2 vector partial sums
>> - in PB2NC in deriving wind speed
>> It's easier to just require that one follows the other.  Otherwise,
>> we'd waste a lot of time hunting through thousands the observations
>> looking for U's and V's to match up.  I'm glad you were able to
> reorder
>> you data to accomodate that.
>>
>> Thanks,
>> John
>>
>>> John,
>>>
>>>
>>>
>>> Well, I tried sorting my daily output file by valid time.  As a
>>> result, I have each individual observation with all the variables
>>> grouped together on consecutive lines in the input file to ascii2nc.
>>> However, the netcdf output from ascii2nc still did not group the
>>> common wind observations into the same hdr_id.  Could this be a
>> problem with the
>>> ascii2nc program in v2Beta?    It seems like the wind component data
>>> need to be written to the same hdr_id in the obs_arr().
>>>
>>>
>>>
>>> For example, here's what I see in the PB2NC ncdump output:
>>>
>>>
>>>
>>> obs_arr =
>>>
>>>   0, 33, 1011.4, -9999, -5.1,
>>>
>>>   0, 34, 1011.4, -9999, 0,
>>>
>>>   0, 32, 1011.4, -9999, 5.1,
>>>
>>>   1, 11, 1011.4, 4.00206, 302.05,
>>>
>>>   1, 7, 1011.4, 4.00206, 4,
>>>
>>>   1, 17, 1011.4, 4.00206, 297.8651,
>>>
>>>   1, 2, 1011.4, 4.00206, 101186.8,
>>>
>>>   2, 33, 1012.1, -9999, -4,
>>>
>>>   2, 34, 1012.1, -9999, 0.7,
>>>
>>>   2, 32, 1012.1, -9999, 4.060788,
>>>
>>> .
>>>
>>> .
>>>
>>> .
>>>
>>>
>>>
>>> Notice that any wind information is grouped together under the same
>>> hdr_id (first column) whereas the other remaining scalar variables
>>> (T/Z/Td/PMSL) are grouped under their own common hdr_id.
>>>
>>>
>>>
>>> Meanwhile, the output from ascii2nc shows up as follows in an
> ncdump:
>>>
>>>
>>> obs_arr =
>>>
>>>   0, 11, -9999, 4, 301.15,
>>>
>>>   1, 17, -9999, 4, 296.15,
>>>
>>>   2, 2, -9999, 4, 101660,
>>>
>>>   3, 31, -9999, 4, 77.69,
>>>
>>>   4, 32, -9999, 4, 4.63,
>>>
>>>   5, 33, -9999, 4, -4.52,
>>>
>>>   6, 34, -9999, 4, -0.99,
>>>
>>>   7, 11, -9999, 319, 302.15,
>>>
>>>   8, 17, -9999, 319, 290.15,
>>>
>>>   9, 31, -9999, 319, 269.8,
>>>
>>>   10, 32, -9999, 319, 3.6,
>>>
>>>   11, 33, -9999, 319, 3.6,
>>>
>>>   12, 34, -9999, 319, 0.01,
>>>
>>> .
>>>
>>> .
>>>
>>> .
>>>
>>>
>>>
>>> Notice how each new line of data corresponding to a new variable has
>> a
>>> new hdr_id, instead of grouping them together as in the PB2NC
> output.
>>> This seems to correlate well with the point_stat error message I
>>> receive when trying to process the UGRD/VGRD/WIND variables:
>>>
>>>
>>>
>>> ERROR: process_obs_file() -> when computing vector winds, each UGRD
>>> observation must be followed by a VGRD observation with the same
>>> header and at the same level
>>>
>>>
>>>
>>> I thought this additional feedback might be helpful towards your
>>> diagnosis.
>>>
>>> Regards,
>>>
>>> Jonathan
>>>
>>>
>>>
>>>
>>>
>>> ***********************************************************
>>> Jonathan Case, ENSCO, Inc.
>>> Aerospace Sciences & Engineering Division Short-term Prediction
>>> Research and Transition Center 320 Sparkman Drive, Room 3062
>>> Huntsville, AL 35805-1912
>>> Voice: (256) 961-7504   Fax: (256) 961-7788
>>> Emails: Jonathan.Case-1 at nasa.gov
>>>
>>>              case.jonathan at ensco.com
>>>
>>> ***********************************************************
>>>
>>>
>>>
>>>