[Met_help] mask_id, CONUS.stations (or something like it - format)
John Halley Gotway
johnhg at ucar.edu
Tue May 18 12:47:46 MDT 2010
Mike,
It appears that the IBM I typically run on is not currently available. I believe they're doing maintenance. So I won't be able to test on there today.
Please let me know if you continue to experience that seg fault, and I'll test up there as soon as I can access it.
Thanks,
John
John Halley Gotway wrote:
> Mike,
>
> The latest set of patches can be found here:
> http://www.dtcenter.org/met/users/support/known_issues/METv2.0/index.php
>
> Just follow the instructions at the top of the page.
>
> The changes to the read TRI of 6 or 7 will be included in the next release, but not in those set of patches. I've attached the file with the changes. After untarring the patch files, copy the
> attached read_grib.cc into METv2.0/lib/vx_met_util/.
>
> I just went through those steps on LINUX: untar METv2.0, add on the patches, add on read_grib.cc, and compile. And your test data did run without any problems.
>
> When I get a chance, I'll try the same thing on an IBM.
>
> John
>
>
> Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
>> Maybe if I can the patch you're using. Where would I find that? Is it in the latest MET (as of when?)? Maybe I could see your AIX makefile to see any memory/bit flags you use?
>>
>> Here's the dbx coredump from the segfault:
>>
>> Segmentation fault in GCPairData::compute_interp(int,double,double,int,double,int,int) at 0x1007b7c8
>> 0x1007b7c8 (GCPairData::compute_interp(int,double,double,int,double,int,int)+0x12c) 7c84282e lwzx r4,r4,r5
>>
>> Thanks a lot,
>> Mike
>>
>> ______________________________________________
>>
>> Michael Shaw, Contractor
>> SAIC/NASA Support Scientist
>> michael.shaw.ctr at offutt.af.mil
>> 16WS Environmental Characterization
>> HQ AFWA-SAIC
>> 101 Nelson Drive
>> Offutt AFB, NE 68113-1023
>> 402-232-7690 Comm * 402-272-7690 DSN * 402-294-8230 Fax
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>> Sent: Tuesday, May 18, 2010 12:51 PM
>> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE
>> Subject: Re: mask_id, CONUS.stations (or something like it - format)
>>
>> Mike,
>>
>> Yep, I'm using all the data you sent. But I'm running on LINUX, not AIX. The METv2.0_patch is METv2.0 with the latest set of patches, plus a modified file to read the funny time range indicators
>> values of 6 and 7.
>>
>> I could try running this on an IBM, the bluefire supercomputer here at NCAR is the only one to which I have access.
>>
>> John
>>
>> Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
>>> Hm. You used all the files I'd sent? The LIS output file, the 2015*nc file, the PointStatConfig, all unmodified? Are you doing this on AIX? What does the "_patch" in the METv2.0_patch mean?
>>>
>>>
>>> Maybe I'll download a fresh copy of MET and see if there's something different in the version we have (or what I've further modified - I modified some of the date/time checking in the grib handler libraries, but haven't touched the pointstat source...maybe something in my grib handler mods...).
>>>
>>> Other thoughts are welcome as always. Thanks a lot for looking into this.
>>>
>>> ______________________________________________
>>>
>>> Michael Shaw, Contractor
>>> SAIC/NASA Support Scientist
>>> michael.shaw.ctr at offutt.af.mil
>>> 16WS Environmental Characterization
>>> HQ AFWA-SAIC
>>> 101 Nelson Drive
>>> Offutt AFB, NE 68113-1023
>>> 402-232-7690 Comm * 402-272-7690 DSN * 402-294-8230 Fax
>>>
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>> Sent: Tuesday, May 18, 2010 12:41 PM
>>> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE; met_help
>>> Subject: Re: mask_id, CONUS.stations (or something like it - format)
>>>
>>> Mike,
>>>
>>> Unfortunately (or fortunately) I'm not able to replicate the seg fault you're seeing. Here's what I see when I run your data:
>>>
>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>> METv2.0_patch/bin/point_stat \
>>> PS.AFWA_SC.U_DI.C_GP.LIS_GR.C0P25DEG_AR.GLOBAL_PA.LIS_DD.20100313_DT.0000_DF.GR1 \
>>> 2015_SCAN_SM.nc \
>>> PointStatConfig \
>>> -outdir out \
>>> -v 2
>>>
>>> Forecast File: PS.AFWA_SC.U_DI.C_GP.LIS_GR.C0P25DEG_AR.GLOBAL_PA.LIS_DD.20100313_DT.0000_DF.GR1
>>> Climatology File: none
>>> Configuration File: PointStatConfig
>>> Observation File: 2015_SCAN_SM.nc
>>>
>>> ----------------------------------------
>>>
>>> Reading records for ICWAT/R14.
>>> For ICWAT/R14 found 1 forecast levels and 0 climatology levels.
>>>
>>> ----------------------------------------
>>>
>>> Searching 64 observations from 64 PrepBufr messages.
>>>
>>> ----------------------------------------
>>>
>>> Processing ICWAT/R14 versus ICWAT/R14, for observation type ADPSFC, over region FULL, for interpolation method UW_MEAN(1), using 1 pairs.
>>> Processing ICWAT/R14 versus ICWAT/R14, for observation type ADPSFC, over region 2015, for interpolation method UW_MEAN(1), using 1 pairs.
>>>
>>> ----------------------------------------
>>>
>>> Output file: out/point_stat_000000L_20100313_000000V.stat
>>> Output file: out/point_stat_000000L_20100313_000000V_mpr.txt
>>>
>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>> Any thoughts on what I could do to help you debug this?
>>>
>>> Thanks,
>>> John
>>>
>>> Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
>>>> Hi John.
>>>>
>>>> I put the new obs file there. Thanks,
>>>> Mike
>>>>
>>>> ______________________________________________
>>>>
>>>> Michael Shaw, Contractor
>>>> SAIC/NASA Support Scientist
>>>> michael.shaw.ctr at offutt.af.mil
>>>> 16WS Environmental Characterization
>>>> HQ AFWA-SAIC
>>>> 101 Nelson Drive
>>>> Offutt AFB, NE 68113-1023
>>>> 402-232-7690 Comm * 402-272-7690 DSN * 402-294-8230 Fax
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>>> Sent: Friday, May 14, 2010 5:30 PM
>>>> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE
>>>> Subject: Re: mask_id, CONUS.stations (or something like it - format)
>>>>
>>>> Mike,
>>>>
>>>> Can you send me the observation file that's causing the core dump? I'll try to replicate it here an debug.
>>>>
>>>> John
>>>>
>>>> Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
>>>>> Thanks. "Searching 64 observations from 64 PrepBufr messages. Segmentation fault(coredump)" An error message might have helped here! But I know the grib is unorthodox, etc.; maybe it'll help for me to dig into the code again and make any needed mods while getting more familiar.
>>>>>
>>>>> I'll keep looking into it. Thanks for your help. And if you have other thoughts, feel free to offer them!
>>>>>
>>>>> ______________________________________________
>>>>>
>>>>> Michael Shaw, Contractor
>>>>> SAIC/NASA Support Scientist
>>>>> michael.shaw.ctr at offutt.af.mil
>>>>> 16WS Environmental Characterization
>>>>> HQ AFWA-SAIC
>>>>> 101 Nelson Drive
>>>>> Offutt AFB, NE 68113-1023
>>>>> 402-232-7690 Comm * 402-272-7690 DSN * 402-294-8230 Fax
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>>>> Sent: Friday, May 14, 2010 4:17 PM
>>>>> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE; met_help
>>>>> Subject: Re: mask_id, CONUS.stations (or something like it - format)
>>>>>
>>>>> Mike,
>>>>>
>>>>> The problem is the message type in the observation file you're using. I used ncdump to dump the observation file you sent me. In there you'll see that "hdr_typ" is set to 2015 for all the
>>>>> observations. That should probably be ADPSFC because that's the observation type you're looking for in the PointStatConfig file. Try changing hdr_typ to ADPSFC in your observations and run again.
>>>>>
>>>>> Also I see that this GRIB file uses time range indicator values that aren't support in MET yet. I'll add those definitions to the development version of the code for TRI = 6 and 7 as listed in the
>>>>> GRIB spec:
>>>>> http://www.nco.ncep.noaa.gov/pmb/docs/on388/table5.html
>>>>>
>>>>> John
>>>>>
>>>>> Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
>>>>>> Nope. That wasn't it. (degrees east)
>>>>>>
>>>>>> ______________________________________________
>>>>>>
>>>>>> Michael Shaw, Contractor
>>>>>> SAIC/NASA Support Scientist
>>>>>> michael.shaw.ctr at offutt.af.mil
>>>>>> 16WS Environmental Characterization
>>>>>> HQ AFWA-SAIC
>>>>>> 101 Nelson Drive
>>>>>> Offutt AFB, NE 68113-1023
>>>>>> 402-232-7690 Comm * 402-272-7690 DSN * 402-294-8230 Fax
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>>>>> Sent: Friday, May 14, 2010 3:10 PM
>>>>>> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE; met_help
>>>>>> Subject: Re: mask_id, CONUS.stations (or something like it - format)
>>>>>>
>>>>>> Mike,
>>>>>>
>>>>>> When you run Point-Stat, I'd suggest using the "-v 2" option which should print to the screen how many matched pairs it found. I'm guessing it found 0 matched pairs since you saw no output.
>>>>>>
>>>>>> There are many reasons why you may be getting 0 matched pairs, but it'd probably be fastest and easiest if you just send me a sample forecast file, a sample observation file, and the Point-Stat config
>>>>>> file you're using. I could probably figure out what's going on.
>>>>>>
>>>>>> If you'd like to do that, you could post it to our anonymous ftp site:
>>>>>>
>>>>>> ftp ftp.rap.ucar.edu/incoming/irap/met_help
>>>>>> username=anonymous
>>>>>> password="your email address"
>>>>>> mkdir shaw_data
>>>>>> cd shaw_data
>>>>>> put "your files"
>>>>>> bye
>>>>>>
>>>>>> If you do send files, just let me know when they're posted.
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>>
>>>>>> Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
>>>>>>> Hi John.
>>>>>>>
>>>>>>> Finally got around to trying this. An mpr file is generated with just a header in it (the column labels), but no data and no errors! Granted, these grib files are also the grib files with weird gribcodes and PR and TR values. But I would have thought giving point_Stat the grib abbreviation and the specific record number for the variable would have produced some data (correct or not) for this simple extract-and-put-in-0mpr-file operation...or at least reported an error message.
>>>>>>>
>>>>>>> Any thoughts?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>>>
>>>>>>> ______________________________________________
>>>>>>>
>>>>>>> Michael Shaw, Contractor
>>>>>>> SAIC/NASA Support Scientist
>>>>>>> michael.shaw.ctr at offutt.af.mil
>>>>>>> 16WS Environmental Characterization
>>>>>>> HQ AFWA-SAIC
>>>>>>> 101 Nelson Drive
>>>>>>> Offutt AFB, NE 68113-1023
>>>>>>> 402-232-7690 Comm * 402-272-7690 DSN * 402-294-8230 Fax
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>>>>>> Sent: Friday, May 14, 2010 2:14 PM
>>>>>>> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE
>>>>>>> Subject: Re: mask_id, CONUS.stations (or something like it - format)
>>>>>>>
>>>>>>> Mike,
>>>>>>>
>>>>>>> You know, I just realized the other day that that file doesn't exist in the tar file either. I'll ask around to see if we have a good list of the standard station names we could include in a future
>>>>>>> release. It really is just a list of station names in a file separated by whitespace. You could have them all on one line or put one on each line... however you'd like to format it. They just need
>>>>>>> to be separated by whitespace.
>>>>>>>
>>>>>>> John
>>>>>>>
>>>>>>> Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
>>>>>>>> Hi John.
>>>>>>>>
>>>>>>>> Sorry...but what's the CONUS.stations file (or a stations file for mask_sid) format have to look like? I'm not seeing this file (mentioned in the config file), for example, in the release/tarball, and not readily seeing much/anything about it the docs/tutorial. Is it really just a txt file with station numbers - SCAN, e.g. - on one line with spaces between them?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> ______________________________________________
>>>>>>>>
>>>>>>>> Michael Shaw, Contractor
>>>>>>>> SAIC/NASA Support Scientist
>>>>>>>> michael.shaw.ctr at offutt.af.mil
>>>>>>>> 16WS Environmental Characterization
>>>>>>>> HQ AFWA-SAIC
>>>>>>>> 101 Nelson Drive
>>>>>>>> Offutt AFB, NE 68113-1023
>>>>>>>> 402-232-7690 Comm * 402-272-7690 DSN * 402-294-8230 Fax
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>>>>>>> Sent: Friday, May 14, 2010 11:38 AM
>>>>>>>> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE; met_help
>>>>>>>> Subject: Re:
>>>>>>>>
>>>>>>>> Michael,
>>>>>>>>
>>>>>>>> Yes, that's basically correct. The main difference between Point-Stat and Grid-Stat is that Point-Stat is set up to handle point observations while Grid-Stat handles gridded observations. Beyond
>>>>>>>> that, they basically do the same thing: collect a bunch of fcst-obs matched pairs over a user-defined region and compute a variety of statistics from those sets of matched pairs. Both Point-Stat and
>>>>>>>> Grid-Stat are designed to be run once for each forecast valid time.
>>>>>>>>
>>>>>>>> I see that you're more interested in looking at performance through time at one or more individual points. And while MET wasn't optimized to solve that problem, there are ways to compute that. You
>>>>>>>> could run Point-Stat at each forecast valid time with the sole purpose of producing matched pairs at a set of observation locations. And then you could use the Stat-Analysis tool to read those
>>>>>>>> matched pairs through time for each observation location and compute any number of statistics. In order to do this, in the Point-Stat configuration file, you'd use the "mask_sid" (for station id) to
>>>>>>>> provide a list of the stations in which you're interested. And you'd set the "output_flag" parameter to only dump out the matched pair (MPR) line type. Then you'd run Stat-Analysis to read in those
>>>>>>>> MPR lines for each station, and recompute stats through time.
>>>>>>>>
>>>>>>>> I do know of at least one other users who's doing exactly that.
>>>>>>>>
>>>>>>>> Hope that helps clarify.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>> Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
>>>>>>>>> Hi John.
>>>>>>>>>
>>>>>>>>> Just getting back to some of this...and still wondering - point_stat isn't really for a point, it's for a set of points (i.e., a single point wouldn't be able to generate any statistics...with a sample size of 1 and all), right? So, the stat analysis part of MET is to then organize, say, rows (so a timeseries if the rows are for different times) of point_stat stats (which could only be done for a SET of points) and compute stats on those stats, right??
>>>>>>>>>
>>>>>>>>> I.e., there's no way of doing a single station/gridpoint pair timeseries statistical analysis with MET??
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> ______________________________________________
>>>>>>>>>
>>>>>>>>> Michael Shaw, Contractor
>>>>>>>>> SAIC/NASA Support Scientist
>>>>>>>>> michael.shaw.ctr at offutt.af.mil
>>>>>>>>> 16WS Environmental Characterization
>>>>>>>>> HQ AFWA-SAIC
>>>>>>>>> 101 Nelson Drive
>>>>>>>>> Offutt AFB, NE 68113-1023
>>>>>>>>> 402-232-7690 Comm * 402-272-7690 DSN * 402-294-8230 Fax
>>>>>>>>>
>>>>>>>>>
More information about the Met_help
mailing list