[Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)
John Halley Gotway
johnhg at ucar.edu
Wed Apr 28 09:40:32 MDT 2010
John,
So it does sound like the patch I sent for some reason hasn't been applied to the binary you're running. I've re-attached that patched version of point_stat.cc. Please do the following:
- Use the attached version of point_stat.cc to overwrite the file METv2.0/src/point_stat/point_stat.cc.
- In the top-level METv2.0 directory, do a "make clean" to remove the existing executables in METv2.0/bin
- In the top-level METv2.0 directory, do a "make" to rebuild those executables.
- Try running Point-Stat on the command line by pointing to a real forecast file and bogus observation files. Something like:
METv2.0/bin/point_stat real_fcst.grb bogus_obs1.nc PointStatConfig -ncfile bogus_obs2.nc
You should see 2 WARNING messages like this:
WARNING: process_obs_file() -> can't open observation netCDF file: bogus_obs1.nc
WARNING: process_obs_file() -> can't open observation netCDF file: bogus_obs2.nc
But if you continue to see ERROR messages instead of WARNING messages, then something funny is going on.
Thanks,
John
Raby, John (Civ, ARL/CISD) wrote:
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> We recompiled MET with the fix yesterday and then tested it. We still are
> getting the error rather than the warning. Please call us at (575) 678-2004.
> Thanks.
> John
>
>
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at ucar.edu]
> Sent: Tuesday, April 27, 2010 3:43 PM
> To: Raby, John (Civ, ARL/CISD)
> Cc: Brown, Bob (Civ, ARL/CISD); met_help at ucar.edu
> Subject: Re: [Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)
>
> John,
>
> I believe the problem you're seeing should also be solved by the fix I sent.
> Basically, you were calling Point-Stat by passing it two observation files,
> let's call them file1.nc and file2.nc. The fact that one was in the
> "mandatory" position and one was in the "optional" position (using -ncfile
> argument) should be irrelevant. In the code, they're just put into a list
> of files. When processing each of those observation files in the list,
> originally Point-Stat would error out if one of the files didn't exist with
> a message like:
> ERROR: process_obs_file() -> can't open observation netCDF file:
> ../out/ascii2nc/sample_ascii2.nc
>
> However, the change I sent you replaces that error with a simple warning
> message. Now, rather than erroring out because of a non-existent file,
> it'll print a warning and continue processing the other files. The warning
> looks something like this:
>
> WARNING: process_obs_file() -> can't open observation netCDF file:
> ../out/ascii2nc/sample_ascii2.nc
>
> Please try running with that fix and see if it addresses this issue.
>
> Hope that helps.
>
> John
>
> Raby, John (Civ, ARL/CISD) wrote:
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> John -
>>
>> We are confused as our sysadmin, Bob Flanigan, was in the process of
>> compiling and found that it appears that the patch was already
>> compiled on
>> 21 APR 2009. He is going ahead with the recompile to make sure it's
>> implemented.
>>
>> Yesterday, we found a mirror image of the same problem. This problem
>> was a situation where the required data was fully populated for every
>> hour, but the optional data was mostly empty (1 out of 25 hourly
>> files). Point-Stat only produced results for the 1 hour where there
>> was optional and required data.
>>
>> It appears that Point-Stat will only produce results where both
>> datasets have data for a given hour.
>>
>> Please call us at (575) 678-2004.
>>
>> Thanks.
>>
>> R/
>> John
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>> Sent: Monday, April 26, 2010 10:40 AM
>> To: Raby, John (Civ, ARL/CISD)
>> Cc: Brown, Bob (Civ, ARL/CISD); met_help at ucar.edu
>> Subject: Re: [Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)
>>
>> John,
>>
>> Yes, it should print a warning message to the screen when an
>> observation file is missing, and then process the observations in any
>> additional observation files. I definitely think logging the output
>> of the MET tools is a good idea. Please let me know if you experience
>> any more issues or need more clarification.
>>
>> Thanks,
>> John
>>
>> Raby, John (Civ, ARL/CISD) wrote:
>>> Classification: UNCLASSIFIED
>>> Caveats: NONE
>>>
>>> John -
>>>
>>> Thanks for the call last Friday and for your quick analysis and solution.
>
>>> With the fix, you say it should print a warning then just continue
>>> processing instead of exiting. Does "continue processing" mean it
>>> will go ahead and process any additional observation files for that
>>> same hour passed via the optional argument?
>>>
>>> Since we were calling Point-Stat 25 times in succession, we were
>>> unable to note the writing of errors in the info which was scrolling
>>> on the screen. We have re-written our script to log the output and
>>> error statements so we can go back and read it.
>>>
>>> We will let you know about the results after recompiling with your fix.
>>>
>>> R/
>>> John
>>>
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>> Sent: Friday, April 23, 2010 3:22 PM
>>> To: Raby, John (Civ, ARL/CISD)
>>> Cc: Brown, Bob (Civ, ARL/CISD); met_help at ucar.edu
>>> Subject: Re: [Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)
>>>
>>> John,
>>>
>>> OK, I've made a minor change in the Point-Stat source code. The
>>> updated version is attached.
>>>
>>> Here's the original piece of code:
>>> cerr << "\n\nERROR: process_obs_file() -> "
>>> << "can't open observation netCDF file: "
>>> << obs_file[i_nc] << "\n\n" << flush;
>>> exit(1);
>>>
>>> And here's the modified piece of code:
>>> cout << "\n\nWARNING: process_obs_file() -> "
>>> << "can't open observation netCDF file: "
>>> << obs_file[i_nc] << "\n\n" << flush;
>>> return;
>>>
>>> When one of the observation files passed to it doesn't exist, instead
>>> of exiting with an error message, it'll just print a warning message
>>> and process the remaining observation files. I suspect that if you
>>> look closely at the output for those hours where the PREPBUFR file
>>> does not exist, you'll find an error message alerting you to that fact.
>>>
>>> Please try using this version of the code, recompile MET, and rerun.
>>> And please let me know if this solves your problem.
>>>
>>> Thanks,
>>> John
>>>
>>> Raby, John (Civ, ARL/CISD) wrote:
>>>> Classification: UNCLASSIFIED
>>>> Caveats: NONE
>>>>
>>>> John -
>>>>
>>>> We are unable to put together the data you requested now.
>>>>
>>>> Ignoring the source of the observational (PrepBUFR or MADIS) the
>>>> main issue is that if the required data argument contains no data
>>>> for a given hour the software ignored the optional data for that same
> hour.
>>>> Conversely, if there was required data for a given hour, then the
>>>> optional data for that hour was ingested and used by Point-Stat. In
>>>> short, the software refuses to add optional data to missing required
>>>> data
>>> field.
>>>> Why should the lack of data in the required data for a given hour
>>>> preclude the ingest of the optional data for that same hour?
>>>>
>>>> Due to the complexity of all of this, it would probably help to
>>>> discuss it verbally. Please call me at this number (575) 678-2004.
>>>>
>>>> Thanks.
>>>>
>>>> John
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>>> Sent: Friday, April 23, 2010 10:35 AM
>>>> To: Raby, John (Civ, ARL/CISD)
>>>> Cc: Brown, Bob (Civ, ARL/CISD); met_help at ucar.edu
>>>> Subject: Re: [Met_help] Request for Assistance-UPDATE (UNCLASSIFIED)
>>>>
>>>> John,
>>>>
>>>> The order in which you pass point observation files to Point-Stat
>>>> should have no impact on how the matched pairs are accumulated. For
>>>> each observation file passed to Point-Stat (the mandatory one or
>>>> additional optional ones), we just call the "process_obs_file"
>>>> routine once. If the behavior of Point-Stat changes depending on
>>>> the order of the point observation files, there may be a bug.
>>>>
>>>> I'm not entirely clear on the exact details of your situation.
>>>> Would you be able to provide me with an example of running
>>>> Point-Stat two different ways, where changing the order of the point
>>>> observation files has an impact on the results? The simpler you're
>>>> able to make the example, the easier it'll be to debug the issue.
>>>>
>>>> For this, I'd need:
>>>> - 1 forecast file
>>>> - 2 point observation files
>>>> - 1 Point-Stat configuration file
>>>> - 1 example of how you call Point-Stat on the command line
>>>>
>>>> And you could post them to our ftp following the directions I sent
>>>> previously.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> Raby, John (Civ, ARL/CISD) wrote:
>>>>> Classification: UNCLASSIFIED
>>>>> Caveats: NONE
>>>>>
>>>>> John -
>>>>>
>>>>> UPDATE:
>>>>>
>>>>> We solved the problem by processing the PrepBUFR data to include
>>>>> the metar data (ADPSFC) in addition to the ADPUPA" and "ANYAIR"
>>>>> data mentioned in the email below. This ensured that there were all
>>>>> 25 files of data present data provided in the required argument.
>>>>>
>>>>> We changed the MADIS data collection to remove the metar data and
>>>>> only receive the mesonet data and added this using the optional
>>>>> data argument (-ncfile).
>>>>>
>>>>> Now the Point-Stat results are produced for all 25 hours (06Z-06Z).
>>>>>
>>>>> In summary, if the required PrepBUFR data argument contains no data
>>>>> for a given hour the software ignored the optional MADIS data.
>>>>>
>>>>> Is this supposed to happen? Why should the lack of data in the
>>>>> PrepBUFR data (required argument) preclude the ingest all 25 hours
>>>>> of the
>>>> optional data?
>>>>> Please call me if we can clarify more for you or let me know if you
>>>>> still want the data ftp'd to you.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> R/
>>>>> John
>>>>>
>>>>> Mr John W. Raby, Meteorologist
>>>>> U.S. Army Research Laboratory
>>>>> White Sands Missile Range, NM 88011
>>>>> (575) 678-2004 DSN 258-2004
>>>>> FAX (575) 678-1230 DSN 258-1230
>>>>> Email: john.raby at us.army.mil
>>>>>
>>>>> Please begin using the email address within this signature block
>>>>> for all future correspondence. The current @arl.army.mil address
>>>>> will be decommissioned in upcoming months.
>>>>>
>>>>> "When you can measure what you are speaking about and express it in
>>>>> numbers, you know something about it, but when you cannot measure
>>>>> it, when you cannot express it in number, your knowledge is of a
>>>>> meagre and unsatisfactory kind". - Lord Kelvin
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>>>>> Sent: Thursday, April 22, 2010 8:58 AM
>>>>> To: Raby, John (Civ, ARL/CISD)
>>>>> Cc: met_help at ucar.edu
>>>>> Subject: Re: Request for Assistance
>>>>>
>>>>> John,
>>>>>
>>>>> Sure, we'd be happy to help you figure this out. Would you be able
>>>>> to send me some sample ASCII MADIS data, a sample forecast file,
>>>>> and the Point-Stat configuration file you're using?
>>>>>
>>>>> I'll try running ASCII2NC and Point-Stat here and figure out what's
>>>>> going on.
>>>>>
>>>>> You could post this sample data to our ftp site as follows:
>>>>> ftp ftp.rap.ucar.edu
>>>>> username=anonymous
>>>>> password="your email address"
>>>>> cd incoming/irap/met_help/raby_data put "your data files"
>>>>> bye
>>>>>
>>>>> Please let me know when the data is available, and I'll go take a look.
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>>
>>>>> We are attempting to use both PrepBUFR and MADIS observational data
>>>>> in MET Point_Stat to evaluate WRF performance over Utah for 2
>>>>> domains, one nested inside the other.
>>>>>
>>>>> We process the PrepBUFR data for a 24 hour period (06Z to 06Z) We
>>>>> explicitly retain only the message strings "ADPUPA" and "ANYAIR"
>>>>> using the PB2NC configuration file settings in order to collect
>>>>> upper air data (raobs, aircraft, acars, etc). After running PB2NC,
>>>>> we typically find only 7 hourly files (12Z, 15Z, 17Z, 19Z, 20Z, 00Z
>>>>> and
>>>>> 01Z) out of
>>>>> 25 possible which contain data (presumably because the upper air
>>>>> data is not present for every hour). These 7 include the 00Z and
>>>>> 12Z raob data for 2 upper air stations inside the larger of our
>>>>> domains (KSLC and
>>>> KLKN).
>>>>> We process the MADIS data to retain only surface mesonet and metar
>>>>> data in
>>>>> 25 hourly text files (Point observation format), then run ASCII2NC
>>>>> to convert the files to netcdf.
>>>>>
>>>>> We ran Point-Stat with the PrepBUFR upper air data files specified
>>>>> in the required argument for the "obs_file" with the MADIS (surface
>>>>> data) specified as additional NETCDF point observation files using
>>>>> the
>>> "-ncfile"
>>>> argument.
>>>>> The Point-Stat output contains results for only the hours 12Z, 15Z,
>>>>> 17Z, 19Z, 20Z, 00Z and 01Z.
>>>>>
>>>>> We then tried running Point-Stat with the MADIS data specified in
>>>>> the required argument and the PrepBUFR data specified as the
>>>>> optional (additional observation file) argument, but this gave
>>>>> identical
>> results.
>>>>> We expect results for all 25 hourly periods since we know that
>>>>> there are MADIS surface observations for every hour.
>>>>>
>>>>> Can you assist us in resolving this problem?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> R/
>>>>> John
>>>>>
>>>>> Mr John W. Raby, Meteorologist
>>>>> U.S. Army Research Laboratory
>>>>> White Sands Missile Range, NM 88011
>>>>> (575) 678-2004 DSN 258-2004
>>>>> FAX (575) 678-1230 DSN 258-1230
>>>>> Email: john.raby at us.army.mil
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Classification: UNCLASSIFIED
>>>>> Caveats: NONE
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------
>>>>> -
>>>>> -
>>>>> -
>>>>> --
>>>>>
>>>>> _______________________________________________
>>>>> Met_help mailing list
>>>>> Met_help at mailman.ucar.edu
>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>> Classification: UNCLASSIFIED
>>>> Caveats: NONE
>>>>
>>>>
>>> Classification: UNCLASSIFIED
>>> Caveats: NONE
>>>
>>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: point_stat.cc
Type: text/x-c++src
Size: 60564 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/met_help/attachments/20100428/5ac46ade/attachment-0001.bin
More information about the Met_help
mailing list