[Met_help] Zero output
John Halley Gotway
johnhg at ucar.edu
Thu Dec 17 13:00:49 MST 2009
Joe,
The weather's been pretty nice this week after a week of pretty bitter cold. 45 degrees feel pretty warm after -10.
I took a look at your data and found what's probably a typo in your job command. Try replacing "-interp_pnts 4" with "-interp_pnts 9". That'll give you some non-zero output.
Also, I'd suggest reading Chapter 8 of the MET User's Guide about the STAT-Analysis tool. I think you may not be doing exactly what you want to do. Please note that you can run it on the command
line for a single job. For example,
Here's a job that'll aggregate together all the SL1L2 lines for TMP/L2:
[1] ./stat_analysis -lookin /d1/johnhg/MET/MET_Help/eastman_data_20091217 -job aggregate -line_type SL1L2 -fcst_var TMP -fcst_lev L2
Here's a job that'll do the same thing, but write out continuous stats instead:
[2] ./stat_analysis -lookin /d1/johnhg/MET/MET_Help/eastman_data_20091217 -job aggregate_stat -line_type SL1L2 -out_line_type CNT -fcst_var TMP -fcst_lev L2
Please note that for UGRD and VGRD, you don't have any SL1L2 lines in your output. Instead, you have VL1L2 lines. You can read about that in Chapter 8. Also, it appears that you're dumping the raw
matched pair values to the stat files. You can really run a lot of jobs with the raw MPR lines!
For each job you run, I'd suggest specifying a single fcst_var and fcst_lev. Also, I'd suggest using the "-dump_row" option to dump the STAT lines used for the job to a file. It's a good idea to do
that to make sure that you're really running your jobs over the subset of data that you intend.
Let us know if more questions come up.
Thanks,
John
Joe Eastman wrote:
> Hi John,
>
> I put several stat files and my config file over there. Thanks for the help!
>
> What's the weather like in my old stomping grounds? Boy do I miss the Gondolier and Big City Burrito!
>
> Take care,
> Joe
>
>
> Dr. Joseph L. Eastman
>
> Senior Atmospheric Scientist
> WindLogics Inc.
>
> 201 4th St NW
>
> Grand Rapids, MN
>
> 55744
>
>
>
> c - 410.279.9702
>
> w - 651.556.4297
>
> jeastman at windlogics.com
>
>
>
>
>
>
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
> Sent: Wednesday, December 16, 2009 4:22 PM
> To: Joe Eastman
> Cc: met_help at ucar.edu
> Subject: RE: [Met_help] Zero output
>
> Joe,
>
> I have an idea of what might be going on, but I'd like to look more
> closely at your data to figure it out. The STAT-Analysis tool only reads
> the files that end in ".stat". Could you place a handful of your ".stat"
> files onto our anonymous ftp site? I'll also need to see the config file
> you're using, named "statanalysis".
>
> In case you forgot, here's how to access that:
> ftp ftp.rap.ucar.edu
> username = anonymous
> password = "your email address"
> cd incoming/irap/met_help/eastman_data
> put "your files"
> bye
>
> Thanks,
> John
>
>> Hi John,
>>
>> As promised I have moved to stage 2, Stat-Analysis. I used point_stat to
>> create the txt files for a 5 day forecast at 10 minute intervals. When I
>> try the stat_analysis tool I get the following:
>> stat_analysis -lookin stat -config statanalysis
>> WARNING: do_job_aggr_stat() -> no matching STAT lines found for job: -job
>> aggregate_stat -fcst_valid_beg 20020501_000000 -fcst_valid_end
>> 20020502_000000 -fcst_var UGRD -fcst_var VGRD -fcst_var TMP -fcst_lev L10
>> -fcst_lev L10 -fcst_lev L2 -interp_mthd DW_MEAN -interp_pnts 4 -line_type
>> SL1L2 -out_line_type CNT -out_alpha 0.050000 -rank_corr_flag 1
>>
>> Here is a sample listing of the directory stat (all 10 minute intervals
>> are there, just to long a list to include here:
>> oint_stat_wrftest_131009L_20020501_131009V_cnt.txt
>> point_stat_wrftest_990000L_20020505_030000V_cnt.txt
>> point_stat_wrftest_131009L_20020501_131009V_cts.txt
>> point_stat_wrftest_990000L_20020505_030000V_cts.txt
>> point_stat_wrftest_131009L_20020501_131009V_mpr.txt
>> point_stat_wrftest_990000L_20020505_030000V_mpr.txt
>> point_stat_wrftest_131009L_20020501_131009V_pct.txt
>> point_stat_wrftest_990000L_20020505_030000V_pct.txt
>> point_stat_wrftest_131009L_20020501_131009V_pjc.txt
>> point_stat_wrftest_990000L_20020505_030000V_pjc.txt
>> point_stat_wrftest_131009L_20020501_131009V_prc.txt
>> point_stat_wrftest_990000L_20020505_030000V_prc.txt
>> point_stat_wrftest_131009L_20020501_131009V_pstd.txt
>> point_stat_wrftest_990000L_20020505_030000V_pstd.txt
>> point_stat_wrftest_131009L_20020501_131009V_sal1l2.txt
>> point_stat_wrftest_990000L_20020505_030000V_sal1l2.txt
>> point_stat_wrftest_131009L_20020501_131009V_sl1l2.txt
>> point_stat_wrftest_990000L_20020505_030000V_sl1l2.txt
>> point_stat_wrftest_131009L_20020501_131009V.stat
>> point_stat_wrftest_990000L_20020505_030000V.stat
>> point_stat_wrftest_131009L_20020501_131009V_val1l2.txt
>> point_stat_wrftest_990000L_20020505_030000V_val1l2.txt
>> point_stat_wrftest_131009L_20020501_131009V_vl1l2.txt
>> point_stat_wrftest_990000L_20020505_030000V_vl1l2.txt
>>
>> Any clues?
>>
>> Take care,
>> Joe
>>
>>
>>
>> Dr. Joseph L. Eastman
>>
>> Senior Atmospheric Scientist
>> WindLogics Inc.
>>
>> 201 4th St NW
>>
>> Grand Rapids, MN
>>
>> 55744
>>
>>
>>
>> c - 410.279.9702
>>
>> w - 651.556.4297
>>
>> jeastman at windlogics.com
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>> Sent: Friday, December 04, 2009 12:46 PM
>> To: Joe Eastman
>> Cc: met_help
>> Subject: Re: [Met_help] Zero output
>>
>> Joe,
>>
>> OK, I see what's going on, and it's kind of subtle. In the Point-Stat
>> configuration file, I changed the "fcst_field" line:
>> FROM THIS: fcst_field[] = [ "UGRD/Z10", "VGRD/Z10", "TMP/Z2" ];
>> TO THIS: fcst_field[] = [ "UGRD/L10", "VGRD/L10", "TMP/L2" ];
>>
>> Once I did that, it ran fine and did find matched pairs. Here's why.
>>
>> The Point-Stat tool does matching for surface variables in a very
>> simplistic way. It's all based on message type. This is inherited from
>> the way NCEP does this type of matching. So by convention,
>> any observation with a message type of ADPSFC or SFCSHP is considered to
>> be at the surface. And by convention, 10-meter winds and 2-meter
>> temperatures are considered to be surface variables. So when
>> you say, "UGRD/Z10" that tells Point-Stat to get the U-component of wind
>> at 10-meters and treat it as a surface variable... i.e. only match it to
>> observations of type ADPSFC or SFCSHP.
>>
>> By changing it to "UGRD/L10" that tells Point-Stat to get the U-component
>> of wind with a level value of 10 and not to worry about only matching it
>> to surface observations. Instead, it matches it to
>> all observation types you request - MSONET in your case. I assume all of
>> your observations are actually at the surface which makes this work fine.
>>
>> So going forward, you're welcome to use "L10" and "L2". Or when building
>> your observation sets with ASCII2NC, you could use the "ADPSFC" message
>> type and keep using "Z10" and "Z2".
>>
>> Based on your data, here's a couple more suggestions:
>>
>> - I see that you have a whole month's worth of observations in your NetCDF
>> file. With data every 5 minutes, that's 4,773,360 observations for the
>> month! Point-Stat seems to handle it fine, but it
>> sure does make Point-Stat run slowly. Instead of storing a whole month's
>> worth of observations in each file, I'd suggest doing it daily or hourly
>> instead. That'd make each run of Point-Stat much
>> faster since it won't have to parse through millions of observations each
>> time it runs.
>>
>> - In the Point-Stat config file, you're using the "FULL" mask and the
>> "CONUS.poly" mask. I looked at your domain, and it's basically just all
>> of Oklahoma. Using "FULL" and "CONUS.poly" will have the
>> same effect, they both just give you the full domain. So get rid of the
>> "CONUS.poly" mask, and just use "FULL" by itself.
>>
>> Hope that helps.
>>
>> John
>>
>> Joe Eastman wrote:
>>> Hi John,
>>>
>>> The data should be up there shortly. I put a WRF grib file and the
>>> mesonet netCDF file in the specified directory.
>>>
>>> Thanks,
>>> Joe
>>>
>>> Dr. Joseph L. Eastman
>>>
>>> Senior Atmospheric Scientist
>>> WindLogics Inc.
>>>
>>> 201 4th St NW
>>>
>>> Grand Rapids, MN
>>>
>>> 55744
>>>
>>>
>>>
>>> c - 410.279.9702
>>>
>>> w - 651.556.4297
>>>
>>> jeastman at windlogics.com
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway [mailto:johnhg at ucar.edu]
>>> Sent: Friday, December 04, 2009 10:16 AM
>>> To: Joe Eastman
>>> Cc: met_help at ucar.edu
>>> Subject: Re: [Met_help] Zero output
>>>
>>> Joe,
>>>
>>> I see that you're running the Point-Stat tool but are getting zero
>>> matched pairs. There are many reasons why this could be happening, such
>>> as...
>>> - Perhaps the observations don't fall into the region you're verifying -
>>> maybe you have a sign wrong in the lat/lon location.
>>> - Perhaps the forecast and observation times don't match up.
>>> - Perhaps the observation message type doesn't match what you're trying
>>> to verify.
>>>
>>> It may be easiest for you to just send me a sample forecast and
>>> observation file, and I'll take a look and see what's going on. Could
>>> you post a sample forecast and observation file to our anonymous
>>> ftp site? Here's how you'd do that:
>>>
>>> ftp ftp.rap.ucar.edu
>>> username = anonymous
>>> password = "your email address"
>>> cd incoming/irap/met_help/eastman_data
>>> put "your forecast file"
>>> put "your observation file"
>>> bye
>>>
>>> To answer you're second question... the MET stats tools (Grid-Stat,
>>> Point-Stat, Wavelet-Stat, and MODE) were designed to run at a single
>>> point in time. So you should run Point-Stat once for each
>>> output time... in your case, every 10 minutes. To aggregate statistics
>>> through time, you'll need to run the STAT-Analysis tool. Now I see that
>>> you're trying to verify over the whole continental US
>>> as well as several individual stations. When verifying at stations,
>>> each individual station is treated as it's own "masking region". For
>>> each run of Point-Stat, you should only be getting 1 matched
>>> pair for each station. So you really will need to run your output
>>> through the STAT-Analysis tool to aggregate through time, which is what
>>> I assume you're really interested in. I realize that this
>>> design is cumbersome for what you're trying to do - but that's how it
>>> currently works.
>>>
>>> Based on what you appear to be doing, here are some suggestions:
>>> - If you haven't already done so, retrieve and install the latest set of
>>> patches for METv2.0 from:
>>> http://www.dtcenter.org/met/users/support/known_issues/METv2.0/index.php
>>> - Do NOT combine the GRIB files for multiple times into a single file.
>>> MET expects that GRIB output from separate times be in separate files.
>>> - I see that you've set "beg_ds" and "end_ds" to +/- 60 seconds in the
>>> config file which will match +/- 1 minute around the forecast valid
>>> time. That looks like a good choice to me for your data.
>>>
>>> Let's try to get a single case running for now, and once that's working
>>> well, I'll be happy to help you aggregate them through time using
>>> STAT-Analysis.
>>>
>>> Please let me know if you're able to post those forecast/observation
>>> files. And I'll take a look to see if I can provide better advice.
>>>
>>> Thanks,
>>> John Halley Gotway
>>> johnhg at ucar.edu
>>>
>>> Joe Eastman wrote:
>>>> Hello,
>>>>
>>>> I am new to using METv2.0, so this is probably something silly. I run
>>>> with the command:
>>>>
>>>> ~/METv2.0-update_bin/point_stat myj_d03_2002-05-05_06:00:00 okmet.nc
>>>> pointstat -outdir . -v 1000 >&m.out&
>>>>
>>>> What I end up with is zero output in the txt and stat files. It prints
>>>> the headers and nothing else. I have converted some Oklahoma mesonet
>>>> data successfully using the ascii2nc program. I included my station
>>>> file too. The other question that is not obvious to is how does one do
>>>> multiple time periods. I have data every 5 minutes. I write the grib
>>>> WRF output every 10 minutes and would like evaluation over all forecast
>>>> times in a single file. Is this possible? Do I need to write all WRF
>>>> output to a single grib file?
>>>>
>>>> Thanks for your time.
>>>>
>>>> Take care,
>>>> Joe
>>>>
>>>>
>>>> Dr. Joseph L. Eastman
>>>> Senior Atmospheric Scientist
>>>> WindLogics Inc.
>>>> 201 4th St NW
>>>> Grand Rapids, MN
>>>> 55744
>>>>
>>>> c - 410.279.9702
>>>> w - 651.556.4297
>>>> jeastman at windlogics.com<mailto:jeastman at windlogics.com>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> CONFIDENTIALITY NOTICE: This e-mail message is for the sole use of the
>>>> intended recipient(s) and may contain confidential and privileged
>>>> information. Any unauthorized review, use, disclosure or distribution
>>>> of any kind is strictly prohibited. If you are not the intended
>>>> recipient, please contact the sender via reply e-mail and destroy all
>>>> copies of the original message. Thank you.
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Met_help mailing list
>>>> Met_help at mailman.ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>
More information about the Met_help
mailing list