[Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)

John Halley Gotway johnhg at ucar.edu
Tue Apr 27 15:42:46 MDT 2010


John,

I believe the problem you're seeing should also be solved by the fix I sent.  Basically, you were calling Point-Stat by passing it two observation files, let's call them file1.nc and file2.nc.  The
fact that one was in the "mandatory" position and one was in the "optional" position (using -ncfile argument) should be irrelevant.  In the code, they're just put into a list of files.  When
processing each of those observation files in the list, originally Point-Stat would error out if one of the files didn't exist with a message like:
   ERROR: process_obs_file() -> can't open observation netCDF file: ../out/ascii2nc/sample_ascii2.nc

However, the change I sent you replaces that error with a simple warning message.  Now, rather than erroring out because of a non-existent file, it'll print a warning and continue processing the other
files.  The warning looks something like this:

   WARNING: process_obs_file() -> can't open observation netCDF file: ../out/ascii2nc/sample_ascii2.nc

Please try running with that fix and see if it addresses this issue.

Hope that helps.

John

Raby, John (Civ, ARL/CISD) wrote:
> Classification: UNCLASSIFIED
> Caveats: NONE
> 
> John - 
> 
> We are confused as our sysadmin, Bob Flanigan, was in the process of
> compiling and found that it appears that the patch was already compiled on
> 21 APR 2009. He is going ahead with the recompile to make sure it's
> implemented.
> 
> Yesterday, we found a mirror image of the same problem. This problem was a
> situation where the required data was fully populated for every hour, but
> the optional data was mostly empty (1 out of 25 hourly files).  Point-Stat
> only produced results for the 1 hour where there was optional and required
> data.
> 
> It appears that Point-Stat will only produce results where both datasets
> have data for a given hour. 
> 
> Please call us at (575) 678-2004.
> 
> Thanks.
> 
> R/
> John
> 
> 
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at ucar.edu] 
> Sent: Monday, April 26, 2010 10:40 AM
> To: Raby, John (Civ, ARL/CISD)
> Cc: Brown, Bob (Civ, ARL/CISD); met_help at ucar.edu
> Subject: Re: [Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)
> 
> John,
> 
> Yes, it should print a warning message to the screen when an observation
> file is missing, and then process the observations in any additional
> observation files.  I definitely think logging the output of the MET tools
> is a good idea.  Please let me know if you experience any more issues or
> need more clarification.
> 
> Thanks,
> John
> 
> Raby, John (Civ, ARL/CISD) wrote:
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> John -
>>
>> Thanks for the call last Friday and for your quick analysis and solution. 
>>
>> With the fix, you say it should print a warning then just continue 
>> processing instead of exiting. Does "continue processing" mean it will 
>> go ahead and process any additional observation files for that same 
>> hour passed via the optional argument?
>>
>> Since we were calling Point-Stat 25 times in succession, we were 
>> unable to note the writing of errors in the info which was scrolling 
>> on the screen. We have re-written our script to log the output and 
>> error statements so we can go back and read it.
>>
>> We will let you know about the results after recompiling with your fix.
>>
>> R/
>> John
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>> Sent: Friday, April 23, 2010 3:22 PM
>> To: Raby, John (Civ, ARL/CISD)
>> Cc: Brown, Bob (Civ, ARL/CISD); met_help at ucar.edu
>> Subject: Re: [Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)
>>
>> John,
>>
>> OK, I've made a minor change in the Point-Stat source code.  The 
>> updated version is attached.
>>
>> Here's the original piece of code:
>>       cerr << "\n\nERROR: process_obs_file() -> "
>>            << "can't open observation netCDF file: "
>>            << obs_file[i_nc] << "\n\n" << flush;
>>       exit(1);
>>
>> And here's the modified piece of code:
>>       cout << "\n\nWARNING: process_obs_file() -> "
>>            << "can't open observation netCDF file: "
>>            << obs_file[i_nc] << "\n\n" << flush;
>>       return;
>>
>> When one of the observation files passed to it doesn't exist, instead 
>> of exiting with an error message, it'll just print a warning message 
>> and process the remaining observation files.  I suspect that if you 
>> look closely at the output for those hours where the PREPBUFR file 
>> does not exist, you'll find an error message alerting you to that fact.
>>
>> Please try using this version of the code, recompile MET, and rerun.  
>> And please let me know if this solves your problem.
>>
>> Thanks,
>> John
>>
>> Raby, John (Civ, ARL/CISD) wrote:
>>> Classification: UNCLASSIFIED
>>> Caveats: NONE
>>>
>>> John -
>>>
>>> We are unable to put together the data you requested now. 
>>>
>>> Ignoring the source of the observational (PrepBUFR or MADIS) the main 
>>> issue is that if the required data argument contains no data for a 
>>> given hour the software ignored the optional data for that same hour.
>>> Conversely, if there was required data for a given hour, then the 
>>> optional data for that hour was ingested and used by Point-Stat. In 
>>> short, the software refuses to add optional data to missing required 
>>> data
>> field.
>>> Why should the lack of data in the required data for a given hour 
>>> preclude the ingest of the optional data for that same hour?
>>>
>>> Due to the complexity of all of this, it would probably help to 
>>> discuss it verbally. Please call me at this number (575) 678-2004.
>>>
>>> Thanks.
>>>
>>> John
>>>
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>> Sent: Friday, April 23, 2010 10:35 AM
>>> To: Raby, John (Civ, ARL/CISD)
>>> Cc: Brown, Bob (Civ, ARL/CISD); met_help at ucar.edu
>>> Subject: Re: [Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)
>>>
>>> John,
>>>
>>> The order in which you pass point observation files to Point-Stat 
>>> should have no impact on how the matched pairs are accumulated.  For 
>>> each observation file passed to Point-Stat (the mandatory one or 
>>> additional optional ones), we just call the "process_obs_file" 
>>> routine once.  If the behavior of Point-Stat changes depending on the 
>>> order of the point observation files, there may be a bug.
>>>
>>> I'm not entirely clear on the exact details of your situation.  Would 
>>> you be able to provide me with an example of running Point-Stat two 
>>> different ways, where changing the order of the point observation 
>>> files has an impact on the results?  The simpler you're able to make 
>>> the example, the easier it'll be to debug the issue.
>>>
>>> For this, I'd need:
>>> - 1 forecast file
>>> - 2 point observation files
>>> - 1 Point-Stat configuration file
>>> - 1 example of how you call Point-Stat on the command line
>>>
>>> And you could post them to our ftp following the directions I sent 
>>> previously.
>>>
>>> Thanks,
>>> John
>>>
>>> Raby, John (Civ, ARL/CISD) wrote:
>>>> Classification: UNCLASSIFIED
>>>> Caveats: NONE
>>>>
>>>> John -
>>>>
>>>> UPDATE: 
>>>>
>>>> We solved the problem by processing the PrepBUFR data to include the 
>>>> metar data (ADPSFC) in addition to the ADPUPA" and "ANYAIR" data 
>>>> mentioned in the email below. This ensured that there were all 25 
>>>> files of data present data provided in the required argument.
>>>>
>>>> We changed the MADIS data collection to remove the metar data and 
>>>> only receive the mesonet data and added this using the optional data 
>>>> argument (-ncfile).
>>>>
>>>> Now the Point-Stat results are produced for all 25 hours (06Z-06Z).
>>>>
>>>> In summary, if the required PrepBUFR data argument contains no data 
>>>> for a given hour the software ignored the optional MADIS data.
>>>>
>>>> Is this supposed to happen? Why should the lack of data in the 
>>>> PrepBUFR data (required argument) preclude the ingest all 25 hours 
>>>> of the
>>> optional data?
>>>> Please call me if we can clarify more for you or let me know if you 
>>>> still want the data ftp'd to you.
>>>>
>>>> Thanks.
>>>>
>>>> R/
>>>> John
>>>>
>>>> Mr John W. Raby, Meteorologist
>>>> U.S. Army Research Laboratory
>>>> White Sands Missile Range, NM 88011
>>>> (575) 678-2004 DSN 258-2004
>>>> FAX (575) 678-1230 DSN 258-1230
>>>> Email: john.raby at us.army.mil
>>>>
>>>> Please begin using the email address within this signature block for 
>>>> all future correspondence.  The current @arl.army.mil address will 
>>>> be decommissioned in upcoming months.
>>>>
>>>> "When you can measure what you are speaking about and express it in 
>>>> numbers, you know something about it, but when you cannot measure 
>>>> it, when you cannot express it in number, your knowledge is of a 
>>>> meagre and unsatisfactory kind". - Lord Kelvin
>>>>
>>>>  
>>>>
>>>> -----Original Message-----
>>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>>> Sent: Thursday, April 22, 2010 8:58 AM
>>>> To: Raby, John (Civ, ARL/CISD)
>>>> Cc: met_help at ucar.edu
>>>> Subject: Re: Request for Assistance
>>>>
>>>> John,
>>>>
>>>> Sure, we'd be happy to help you figure this out.  Would you be able 
>>>> to send me some sample ASCII MADIS data, a sample forecast file, and 
>>>> the Point-Stat configuration file you're using?
>>>>
>>>> I'll try running ASCII2NC and Point-Stat here and figure out what's 
>>>> going on.
>>>>
>>>> You could post this sample data to our ftp site as follows:
>>>> ftp ftp.rap.ucar.edu
>>>> username=anonymous
>>>> password="your email address"
>>>> cd incoming/irap/met_help/raby_data
>>>> put "your data files"
>>>> bye
>>>>
>>>> Please let me know when the data is available, and I'll go take a look.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>>
>>>> We are attempting to use both PrepBUFR and MADIS observational data 
>>>> in MET Point_Stat to evaluate WRF performance over Utah for 2 
>>>> domains, one nested inside the other.
>>>>
>>>> We process the PrepBUFR data for a 24 hour period (06Z to 06Z) We 
>>>> explicitly retain only the message strings "ADPUPA" and "ANYAIR"
>>>> using the PB2NC configuration file settings in order to collect 
>>>> upper air data (raobs, aircraft, acars, etc). After running PB2NC, 
>>>> we typically find only 7 hourly files (12Z, 15Z, 17Z, 19Z, 20Z, 00Z 
>>>> and
>>>> 01Z) out of
>>>> 25 possible which contain data (presumably because the upper air 
>>>> data is not present for every hour).  These 7 include the 00Z and 
>>>> 12Z raob data for 2 upper air stations inside the larger of our 
>>>> domains (KSLC and
>>> KLKN).
>>>> We process the MADIS data to retain only surface mesonet and metar 
>>>> data in
>>>> 25 hourly text files (Point observation format), then run ASCII2NC 
>>>> to convert the files to netcdf.
>>>>
>>>> We ran Point-Stat with the PrepBUFR upper air data files specified 
>>>> in the required argument for the "obs_file" with the MADIS (surface
>>>> data) specified as additional NETCDF point observation files using 
>>>> the
>> "-ncfile"
>>> argument.
>>>> The Point-Stat output contains results for only the hours 12Z, 15Z, 
>>>> 17Z, 19Z, 20Z, 00Z and 01Z.
>>>>
>>>> We then tried running Point-Stat with the MADIS data specified in 
>>>> the required argument and the PrepBUFR data specified as the 
>>>> optional (additional observation file) argument, but this gave identical
> results.
>>>> We expect results for all 25 hourly periods since we know that there 
>>>> are MADIS surface observations for every hour.
>>>>
>>>> Can you assist us in resolving this problem?
>>>>
>>>> Thanks.
>>>>
>>>> R/
>>>> John
>>>>
>>>> Mr John W. Raby, Meteorologist
>>>> U.S. Army Research Laboratory
>>>> White Sands Missile Range, NM 88011
>>>> (575) 678-2004 DSN 258-2004
>>>> FAX (575) 678-1230 DSN 258-1230
>>>> Email: john.raby at us.army.mil
>>>>
>>>>
>>>>
>>>>
>>>> Classification: UNCLASSIFIED
>>>> Caveats: NONE
>>>>
>>>>
>>>>
>>>>
>>>> --------------------------------------------------------------------
>>>> -
>>>> -
>>>> --
>>>>
>>>> _______________________________________________
>>>> Met_help mailing list
>>>> Met_help at mailman.ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>> Classification: UNCLASSIFIED
>>> Caveats: NONE
>>>
>>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
> Classification: UNCLASSIFIED
> Caveats: NONE
> 
> 


More information about the Met_help mailing list