[Met_help] Point-Stat Tool --- Point Observation Component Issue
Waylon.Collins
Waylon.Collins at noaa.gov
Thu Aug 6 21:56:28 MDT 2009
John,
You don't need to create any additional tools; I decided to write a
shell script to run Point-Stat for multiple cases and then run
Stat_Analysis to aggregate the results, as you suggested.
However, I am having difficulty running Stat_Analysis. Using Point-Stat,
I generated STAT files for 30 days in a row. Then, I ran Stat_Analysis.
My objective is to use "aggregate_stat", yet I can't even get results
from "filter". Attached includes the output from running Stat_Analysis
using the configuration file with the "filter" option. Also attached is
the corresponding configuration file. Note that Stat_Analysis processes
my 30 STAT files, yet does not appear to read the lines. The file
following the -dump_row option had a size of 0. The filtering was set in
retrieve lines that exist in the STAT files, but to no avail. I also
attached the configuration file using in my Point-Stat run and included
one of the 30 STAT files.
Any idea what I am doing wrong?
Waylon
John Halley Gotway wrote:
> Waylon,
>
> The Point-Stat tool was designed to be run at a single moment in time. In the example you gave, for time 20090417_0600 (17 April 2009 at 0600 UTC), you'd run Point-Stat once passing to it the
> forecast file valid at that time and the set of observations you'd like to compare to those observations.
>
> If you'd like to compute quarterly or monthly verification statistics, you'll need to run Point-Stat once for each verification time, and then use the Stat-Analysis tool to aggregate the results.
>
> It depends on how much data we're talking about here, but one option would be to make use of the MPR (matched pair) line type. When you run Point-Stat, you have the option of dumping out the fcst and
> obs raw matched pair values. Then you could run a job in the Stat-Analysis tool to read in all those matched pairs for a month, for example, and compute whatever statistics you'd like from them.
>
> However, the MPR line type is pretty inefficient since all the header data is written out for each matched pair. So if space is an issue, you could instead make use of the SL1L2 partial sums and CTC
> contingency table counts line types from Point-Stat. Then you could run Stat-Analysis jobs to aggregate those lines over a month and compute statistics from them.
>
> As for passing multiple NetCDF files to the Point-Stat tool, please take a look at the usage statement for Point-Stat in the documentation. The "-ncfile" command line option can be used as many times
> as you like to pass additional NetCDF point observation files to Point-Stat to be used. So you could convert the hourly files and pass many of them to Point-Stat on the command line using "-ncfile".
> Or I could probably modify the AWIPS2NC tool to convert multiple input NetCDF files into a single output NetCDF file... but I'm pretty busy this week and next.
>
> Hope that helps.
>
> John
>
> Waylon.Collins wrote:
>
>> Hello John,
>>
>> Yes, I was successful this time. Thanks.
>>
>> The AWIPS netCDF observation files I have been sending you are for a
>> single moment in time (e.g. 20090417_0600 for 17 April 2009 at 0600
>> UTC). However, I will be generating either quarterly or monthly
>> verification statistics. How can I run Point-Stat in order to process
>> many netCDF files at once? Alternatively, I my plan is to install the
>> NCO (netCDF Operators) software (http://nco.sourceforge.net) and plan to
>> use the ncrcat (netCDF Record Concatenator), which is part of NCO, in
>> order to concatenates all of the hourly netCDF files into one netCDF
>> file before running your awips2nc program. Any other ideas?
>>
>> Waylon
>>
>> John Halley Gotway wrote:
>>
>>> Waylon,
>>>
>>> Thanks for sending me all that data. It made debugging the problem
>>> very easy. Sorry about this, but I had my counts off by 1. I was
>>> indexing the header id's from 1 to N, when I should have been
>>> doing it from 0 to N-1. This amounts to a 1 line change in the
>>> "awips2nc.cc" file. I've attached the updated version. Please
>>> replace METv2.0/src/awips2nc/awips2nc.cc with the attached version,
>>> recompile, and try running AWIPS2NC and Point-Stat again.
>>>
>>> Thanks,
>>> John
>>>
>>> Waylon.Collins wrote:
>>>
>>>
>>>> John,
>>>>
>>>> After compiling using the PGI Makefile, and after installing
>>>> METv2.0_patches_20090701.tar from the MET website, I ran awips2nc
>>>> without error. However, when I ran the Point Stat Tool as a test, I
>>>> encountered the attached error. I uploaded the associated files to the
>>>> /incoming/irap/met_help/collins_data subdirectory you prepared for me
>>>> earlier on ftp.rap.ucar.edu server. The file 20090417_0600 and
>>>> 20090417_0600.out are the input and output to the awips2nc program,
>>>> respectively. the *arw_wrfout* GRIB1 file is the forecast file. The
>>>> PointStatConfig_ARW file is the configuration file for Point Stat.
>>>>
>>>> Waylon
>>>>
>>>> John Halley Gotway wrote:
>>>>
>>>>
>>>>> Waylon,
>>>>>
>>>>> The updated PGI Makefile is attached. It includes additions in 4
>>>>> spots to include the building of the AWIPS2NC tool.
>>>>>
>>>>> Hope it all goes well.
>>>>>
>>>>> John
>>>>>
>>>>> Waylon.Collins wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Hello John,
>>>>>>
>>>>>> The Fortran Compiler that I use is the from the Portland Group (PGI),
>>>>>> although I do have gfortran installed on my computer (that runs the
>>>>>> Redhat Enterprise Linux v4 O/S.) I created both PGI (16 June 2009) and
>>>>>> gfortran (tonight) versions of both netCDF and libbufr.a and tried to
>>>>>> install MET again tonight using your new Makefile_gnu file and the
>>>>>> gfortran compiled versions of netCDF and BUFRLIB. I am having
>>>>>> difficulty, although it appears to be minor. Nevertheless, could you
>>>>>> prepare a Makefile_pgi file for the building of MET including
>>>>>> awips2nc?
>>>>>> Thanks again for your help.
>>>>>>
>>>>>> Waylon
>>>>>>
>>>>>> John Halley Gotway wrote:
>>>>>>
>>>>>>
>>>>>>> Waylon,
>>>>>>>
>>>>>>> I went ahead and just took a shot at it. After talking to a
>>>>>>> meteorologist here, I assumed that both report types, METAR and
>>>>>>> SPECI,
>>>>>>> indicate surface observations. I've attached some code for you to
>>>>>>> try out.
>>>>>>>
>>>>>>> Please copy the attached file "METv2.0_awips2nc.20090714.tar.gz",
>>>>>>> into
>>>>>>> the top-level METv2.0 directory. Then uncompress and untar it:
>>>>>>> gunzip METv2.0_awips2nc.20090714.tar.gz
>>>>>>> tar -xvf METv2.0_awips2nc.20090714.tar
>>>>>>>
>>>>>>> This will create a new directory "METv2.0/src/awips2nc". Also, it
>>>>>>> will overwrite the file "Makefile_gnu" with an updated version with
>>>>>>> commands for building the awips2nc tool. I wasn't sure which
>>>>>>> compilers you used to build MET... if it was something other than the
>>>>>>> GNU compilers, just look at the Makefile changes I made, and make
>>>>>>> similar ones for the version of the Makefile you're using.
>>>>>>>
>>>>>>> ... or let me know which compilers, and I can make the Makefile
>>>>>>> changes for you.
>>>>>>>
>>>>>>> Then just rebuild MET by typing "make" in the METv2.0 directory.
>>>>>>> That
>>>>>>> should create the "METv2.0/bin/awips2nc" executable which you can use
>>>>>>> to reformat your AWIPS point observation file. Type
>>>>>>> "METv2.0/bin/awips2nc" to see the usage statement. It's pretty
>>>>>>> straight-forward.
>>>>>>>
>>>>>>> Try it out and let me know how it goes. Right now it's assuming all
>>>>>>> of the point observations to be at the surface. Just let me know if
>>>>>>> we need to modify the logic to handle a wider variety of data
>>>>>>> types you may encounter.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> John Halley Gotway
>>>>>>> johnhg at ucar.edu
>>>>>>>
>>>>>>> John Halley Gotway wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Waylon,
>>>>>>>>
>>>>>>>> Thanks for checking in. I am working on it, but have been slowed by
>>>>>>>> needing to get ready for a MET tutorial we're giving next week.
>>>>>>>> I'll
>>>>>>>> get to work on it though and will get in touch with you when
>>>>>>>> I have something for you. I hope to make significant progress this
>>>>>>>> week.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>> Waylon.Collins at noaa.gov wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hello John,
>>>>>>>>>
>>>>>>>>> Just wondering if you made any progress developing software to
>>>>>>>>> convert AWIPS METAR netCDF files to Point Stat tool compatible
>>>>>>>>> netCDF files?
>>>>>>>>>
>>>>>>>>> Waylon
>>>>>>>>> Waylon Collins (Senior Forecaster)
>>>>>>>>> NOAA/National Weather Service
>>>>>>>>> 300 Pinson Drive
>>>>>>>>> Corpus Christi, TX 78406
>>>>>>>>> (United States)
>>>>>>>>> Telephone: (361) 289-0959 ----- Original Message -----
>>>>>>>>> From: John Halley Gotway <johnhg at rap.ucar.edu>
>>>>>>>>> Date: Thursday, June 25, 2009 1:43 pm
>>>>>>>>> Subject: Re: [Met_help] Point-Stat Tool --- Point Observation
>>>>>>>>> Component Issue
>>>>>>>>> To: "Waylon.Collins" <Waylon.Collins at noaa.gov>, met_help
>>>>>>>>> <met_help at ucar.edu>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Waylon,
>>>>>>>>>>
>>>>>>>>>> Sounds good. I'm busy this week with the WRF-User's Workshop.
>>>>>>>>>> But
>>>>>>>>>> I'll take a look at these files next week.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> Waylon.Collins wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hello John,
>>>>>>>>>>>
>>>>>>>>>>> Yes, lets pursue the strategy of developing a reformatting tool
>>>>>>>>>>> that
>>>>>>>>>>> would read an AWIPS netCDF METAR file and write a Point-Stat tool
>>>>>>>>>>> compatible netCDF file. I uploaded three (3) METAR files to the
>>>>>>>>>>> collins_data subdirectory on your FTP site. Go ahead and ignore
>>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>> QC
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> information and simply use the actual data values. The twelve
>>>>>>>>>>> (12)
>>>>>>>>>>> observation types you mentioned below are sufficient. Thanks for
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> your help.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Waylon
>>>>>>>>>>>
>>>>>>>>>>> John Halley Gotway wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Waylon,
>>>>>>>>>>>>
>>>>>>>>>>>> We do not currently have tools that will do either of the
>>>>>>>>>>>> things you
>>>>>>>>>>>> requested. However, I took at look at the NetCDF header you
>>>>>>>>>>>> sent,
>>>>>>>>>>>>
>>>>>>>>>> and
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> the format looks pretty straight-forward. I'd be happy to
>>>>>>>>>>>> work with you on a reformatting tool that would read a NetCDF
>>>>>>>>>>>> AWIPS
>>>>>>>>>>>> file and write a NetCDF file for input into the Point-Stat
>>>>>>>>>>>> tool. I
>>>>>>>>>>>> don't think it'd be too difficult. But I'd need you to send me
>>>>>>>>>>>> a few sample files for testing and answer some questions
>>>>>>>>>>>> about the
>>>>>>>>>>>> format.
>>>>>>>>>>>>
>>>>>>>>>>>> If you'd like to proceed with this, please send a few sample
>>>>>>>>>>>> files
>>>>>>>>>>>> either via email or by posting them on our anonymous FTP site:
>>>>>>>>>>>> ftp ftp.rap.ucar.edu
>>>>>>>>>>>> username = anonymous
>>>>>>>>>>>> password = "your email address"
>>>>>>>>>>>> cd incoming/irap/met_help/collins_data
>>>>>>>>>>>> put "your sample files"
>>>>>>>>>>>> bye
>>>>>>>>>>>>
>>>>>>>>>>>> Looking at the NetCDF header you sent, it looks like there's a
>>>>>>>>>>>> lot
>>>>>>>>>>>>
>>>>>>>>>> of
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> quality control information in the file. The goal of this
>>>>>>>>>>>> reformatting tool would be to extract the actual observation
>>>>>>>>>>>> values
>>>>>>>>>>>> you'd like to use in the verification. Would it be reasonable
>>>>>>>>>>>> initially to ignore the QC information and just use the actual
>>>>>>>>>>>> data
>>>>>>>>>>>> values?
>>>>>>>>>>>>
>>>>>>>>>>>> Here are the observation types I see that we would make sense to
>>>>>>>>>>>> extract to use in verification. Each is followed by the GRIB
>>>>>>>>>>>> code
>>>>>>>>>>>>
>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> which it would correspond:
>>>>>>>>>>>> (1) Sea Level Pressure (GRIB Code 2)
>>>>>>>>>>>> (2) Temperature (GRIB Code 11)
>>>>>>>>>>>> (3) Maximum Temperature (GRIB Code 15)
>>>>>>>>>>>> (4) Minimum Temperature (GRIB Code 16)
>>>>>>>>>>>> (5) Dewpoint (GRIB Code 17)
>>>>>>>>>>>> (6) Wind Direction (GRIB Code 31)
>>>>>>>>>>>> (7) Wind Speed (GRIB Code 32)
>>>>>>>>>>>> (8) Wind Gust (GRIB Code 180)
>>>>>>>>>>>> (9) 1-hourly precip (GRIB Code 61)
>>>>>>>>>>>> (10) 3-hourly precip (GRIB Code 61)
>>>>>>>>>>>> (11) 6-hourly precip (GRIB Code 61)
>>>>>>>>>>>> (12) 24-hourly precip (GRIB Code 61)
>>>>>>>>>>>>
>>>>>>>>>>>> Does that set of observation types look reasonable to you? Or
>>>>>>>>>>>> am I
>>>>>>>>>>>> missing something?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> John Halley Gotway
>>>>>>>>>>>> johnhg at ucar.edu
>>>>>>>>>>>>
>>>>>>>>>>>> Waylon.Collins wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear Sir or Madam,
>>>>>>>>>>>>>
>>>>>>>>>>>>> My goal is to generate verification statistics using the
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> Point-Stat Tool
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> by comparing point observations with WRF-ARW gridded forecast
>>>>>>>>>>>>> output.
>>>>>>>>>>>>> With respect to the Point-Stat Tool, I understand that the
>>>>>>>>>>>>> netCDF
>>>>>>>>>>>>> obs_file should be created from pb2nc or ascii2nc. I do not
>>>>>>>>>>>>> have
>>>>>>>>>>>>> access
>>>>>>>>>>>>> to data in prepbufr format, thus ascii2nc would be used.
>>>>>>>>>>>>> However,
>>>>>>>>>>>>>
>>>>>>>>>> I will
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> be preparing verification statistics based on 3 months of data,
>>>>>>>>>>>>> every
>>>>>>>>>>>>> 3-months, time resolution of 3 hours, for 5-10 observation
>>>>>>>>>>>>> points.
>>>>>>>>>>>>> Manually creating files to match the 10 column ASCII Point
>>>>>>>>>>>>> Observation
>>>>>>>>>>>>> Format as input into ascii2nc is not practical. I need an
>>>>>>>>>>>>> efficient
>>>>>>>>>>>>> method to create the obs_file as input into point_stat. I have
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> access to
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> the observations in netCDF format used by AWIPS. Attached is
>>>>>>>>>>>>> the
>>>>>>>>>>>>> output
>>>>>>>>>>>>> from the linux command "ncdump -h" applied to one of these
>>>>>>>>>>>>> files.
>>>>>>>>>>>>>
>>>>>>>>>> Do you
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> have software that will either (1) convert my AWIPS netCDF
>>>>>>>>>>>>> files
>>>>>>>>>>>>> into
>>>>>>>>>>>>> the netCDF flavor required by point_stat, or (2) input my AWIPS
>>>>>>>>>>>>> netCDF
>>>>>>>>>>>>> files and output an ASCII file containing the data in the 10
>>>>>>>>>>>>> column
>>>>>>>>>>>>> ASCII Point Observation Format required by ascii2nc?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Waylon
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Met_help mailing list
>>>>>>>>>>>>> Met_help at mailman.ucar.edu
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Met_help mailing list
>>>>>>>> Met_help at mailman.ucar.edu
>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>>>>>
>>>>>>>>
--
Waylon Collins (Senior Forecaster)
NOAA/National Weather Service
300 Pinson Drive
Corpus Christi, TX 78406
(United States)
Telephone: (361) 289-0959
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: STATAnalysisConfig_ARW
Url: http://mailman.ucar.edu/pipermail/met_help/attachments/20090806/75f39929/attachment-0003.pl
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: point_stat_ARW_240000L_20090531_120000V.stat
Url: http://mailman.ucar.edu/pipermail/met_help/attachments/20090806/75f39929/attachment-0004.pl
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: PointStatConfig_ARW
Url: http://mailman.ucar.edu/pipermail/met_help/attachments/20090806/75f39929/attachment-0005.pl
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: stat_analysis_output.txt
Url: http://mailman.ucar.edu/pipermail/met_help/attachments/20090806/75f39929/attachment-0001.txt
More information about the Met_help
mailing list