[Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)

[John Halley Gotway]

John,

Yes, it should print a warning message to the screen when an observation file is missing, and then process the observations in any additional observation files.  I definitely think logging the output
of the MET tools is a good idea.  Please let me know if you experience any more issues or need more clarification.

Thanks,
John

Raby, John (Civ, ARL/CISD) wrote:
> Classification: UNCLASSIFIED
> Caveats: NONE
> 
> John - 
> 
> Thanks for the call last Friday and for your quick analysis and solution. 
> 
> With the fix, you say it should print a warning then just continue
> processing instead of exiting. Does "continue processing" mean it will go
> ahead and process any additional observation files for that same hour passed
> via the optional argument?
> 
> Since we were calling Point-Stat 25 times in succession, we were unable to
> note the writing of errors in the info which was scrolling on the screen. We
> have re-written our script to log the output and error statements so we can
> go back and read it.
> 
> We will let you know about the results after recompiling with your fix.
> 
> R/
> John
> 
> 
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at ucar.edu] 
> Sent: Friday, April 23, 2010 3:22 PM
> To: Raby, John (Civ, ARL/CISD)
> Cc: Brown, Bob (Civ, ARL/CISD); met_help at ucar.edu
> Subject: Re: [Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)
> 
> John,
> 
> OK, I've made a minor change in the Point-Stat source code.  The updated
> version is attached.
> 
> Here's the original piece of code:
>       cerr << "\n\nERROR: process_obs_file() -> "
>            << "can't open observation netCDF file: "
>            << obs_file[i_nc] << "\n\n" << flush;
>       exit(1);
> 
> And here's the modified piece of code:
>       cout << "\n\nWARNING: process_obs_file() -> "
>            << "can't open observation netCDF file: "
>            << obs_file[i_nc] << "\n\n" << flush;
>       return;
> 
> When one of the observation files passed to it doesn't exist, instead of
> exiting with an error message, it'll just print a warning message and
> process the remaining observation files.  I suspect that if you look closely
> at the output for those hours where the PREPBUFR file does not exist, you'll
> find an error message alerting you to that fact.
> 
> Please try using this version of the code, recompile MET, and rerun.  And
> please let me know if this solves your problem.
> 
> Thanks,
> John
> 
> Raby, John (Civ, ARL/CISD) wrote:
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> John -
>>
>> We are unable to put together the data you requested now. 
>>
>> Ignoring the source of the observational (PrepBUFR or MADIS) the main 
>> issue is that if the required data argument contains no data for a 
>> given hour the software ignored the optional data for that same hour. 
>> Conversely, if there was required data for a given hour, then the 
>> optional data for that hour was ingested and used by Point-Stat. In 
>> short, the software refuses to add optional data to missing required data
> field.
>> Why should the lack of data in the required data for a given hour 
>> preclude the ingest of the optional data for that same hour?
>>
>> Due to the complexity of all of this, it would probably help to 
>> discuss it verbally. Please call me at this number (575) 678-2004.
>>
>> Thanks.
>>
>> John
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>> Sent: Friday, April 23, 2010 10:35 AM
>> To: Raby, John (Civ, ARL/CISD)
>> Cc: Brown, Bob (Civ, ARL/CISD); met_help at ucar.edu
>> Subject: Re: [Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)
>>
>> John,
>>
>> The order in which you pass point observation files to Point-Stat 
>> should have no impact on how the matched pairs are accumulated.  For 
>> each observation file passed to Point-Stat (the mandatory one or 
>> additional optional ones), we just call the "process_obs_file" routine 
>> once.  If the behavior of Point-Stat changes depending on the order of 
>> the point observation files, there may be a bug.
>>
>> I'm not entirely clear on the exact details of your situation.  Would 
>> you be able to provide me with an example of running Point-Stat two 
>> different ways, where changing the order of the point observation 
>> files has an impact on the results?  The simpler you're able to make 
>> the example, the easier it'll be to debug the issue.
>>
>> For this, I'd need:
>> - 1 forecast file
>> - 2 point observation files
>> - 1 Point-Stat configuration file
>> - 1 example of how you call Point-Stat on the command line
>>
>> And you could post them to our ftp following the directions I sent 
>> previously.
>>
>> Thanks,
>> John
>>
>> Raby, John (Civ, ARL/CISD) wrote:
>>> Classification: UNCLASSIFIED
>>> Caveats: NONE
>>>
>>> John -
>>>
>>> UPDATE: 
>>>
>>> We solved the problem by processing the PrepBUFR data to include the 
>>> metar data (ADPSFC) in addition to the ADPUPA" and "ANYAIR" data 
>>> mentioned in the email below. This ensured that there were all 25 
>>> files of data present data provided in the required argument.
>>>
>>> We changed the MADIS data collection to remove the metar data and 
>>> only receive the mesonet data and added this using the optional data 
>>> argument (-ncfile).
>>>
>>> Now the Point-Stat results are produced for all 25 hours (06Z-06Z).
>>>
>>> In summary, if the required PrepBUFR data argument contains no data 
>>> for a given hour the software ignored the optional MADIS data.
>>>
>>> Is this supposed to happen? Why should the lack of data in the 
>>> PrepBUFR data (required argument) preclude the ingest all 25 hours of 
>>> the
>> optional data?
>>> Please call me if we can clarify more for you or let me know if you 
>>> still want the data ftp'd to you.
>>>
>>> Thanks.
>>>
>>> R/
>>> John
>>>
>>> Mr John W. Raby, Meteorologist
>>> U.S. Army Research Laboratory
>>> White Sands Missile Range, NM 88011
>>> (575) 678-2004 DSN 258-2004
>>> FAX (575) 678-1230 DSN 258-1230
>>> Email: john.raby at us.army.mil
>>>
>>> Please begin using the email address within this signature block for 
>>> all future correspondence.  The current @arl.army.mil address will be 
>>> decommissioned in upcoming months.
>>>
>>> "When you can measure what you are speaking about and express it in 
>>> numbers, you know something about it, but when you cannot measure it, 
>>> when you cannot express it in number, your knowledge is of a meagre 
>>> and unsatisfactory kind". - Lord Kelvin
>>>
>>>  
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>> Sent: Thursday, April 22, 2010 8:58 AM
>>> To: Raby, John (Civ, ARL/CISD)
>>> Cc: met_help at ucar.edu
>>> Subject: Re: Request for Assistance
>>>
>>> John,
>>>
>>> Sure, we'd be happy to help you figure this out.  Would you be able 
>>> to send me some sample ASCII MADIS data, a sample forecast file, and 
>>> the Point-Stat configuration file you're using?
>>>
>>> I'll try running ASCII2NC and Point-Stat here and figure out what's 
>>> going on.
>>>
>>> You could post this sample data to our ftp site as follows:
>>> ftp ftp.rap.ucar.edu
>>> username=anonymous
>>> password="your email address"
>>> cd incoming/irap/met_help/raby_data
>>> put "your data files"
>>> bye
>>>
>>> Please let me know when the data is available, and I'll go take a look.
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> We are attempting to use both PrepBUFR and MADIS observational data 
>>> in MET Point_Stat to evaluate WRF performance over Utah for 2 
>>> domains, one nested inside the other.
>>>
>>> We process the PrepBUFR data for a 24 hour period (06Z to 06Z) We 
>>> explicitly retain only the message strings "ADPUPA" and "ANYAIR" 
>>> using the PB2NC configuration file settings in order to collect upper 
>>> air data (raobs, aircraft, acars, etc). After running PB2NC, we 
>>> typically find only 7 hourly files (12Z, 15Z, 17Z, 19Z, 20Z, 00Z and 
>>> 01Z) out of
>>> 25 possible which contain data (presumably because the upper air data 
>>> is not present for every hour).  These 7 include the 00Z and 12Z raob 
>>> data for 2 upper air stations inside the larger of our domains (KSLC 
>>> and
>> KLKN).
>>> We process the MADIS data to retain only surface mesonet and metar 
>>> data in
>>> 25 hourly text files (Point observation format), then run ASCII2NC to 
>>> convert the files to netcdf.
>>>
>>> We ran Point-Stat with the PrepBUFR upper air data files specified in 
>>> the required argument for the "obs_file" with the MADIS (surface 
>>> data) specified as additional NETCDF point observation files using the
> "-ncfile"
>> argument.
>>> The Point-Stat output contains results for only the hours 12Z, 15Z, 
>>> 17Z, 19Z, 20Z, 00Z and 01Z.
>>>
>>> We then tried running Point-Stat with the MADIS data specified in the 
>>> required argument and the PrepBUFR data specified as the optional 
>>> (additional observation file) argument, but this gave identical results.
>>>
>>> We expect results for all 25 hourly periods since we know that there 
>>> are MADIS surface observations for every hour.
>>>
>>> Can you assist us in resolving this problem?
>>>
>>> Thanks.
>>>
>>> R/
>>> John
>>>
>>> Mr John W. Raby, Meteorologist
>>> U.S. Army Research Laboratory
>>> White Sands Missile Range, NM 88011
>>> (575) 678-2004 DSN 258-2004
>>> FAX (575) 678-1230 DSN 258-1230
>>> Email: john.raby at us.army.mil
>>>
>>>
>>>
>>>
>>> Classification: UNCLASSIFIED
>>> Caveats: NONE
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> -
>>> --
>>>
>>> _______________________________________________
>>> Met_help mailing list
>>> Met_help at mailman.ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
> Classification: UNCLASSIFIED
> Caveats: NONE
> 
>