[Met_help] A problem
John Halley Gotway
johnhg at rap.ucar.edu
Tue Jul 14 06:45:54 MDT 2009
Hello,
As you can see in the output, we are computing a lot of statistics here! As we wrote the code, we tried to check all the possible error conditions (which are often cases of dividing by 0), but it's
certainly possible that we missed some. We should have caught all of those cases of "nan" that you're seeing and replaced them with "NA".
Would you be able to send me 2 things: the STAT-Analysis job that you're running (either through a config file or on the command line), and the actual STAT files over which you're running the job?
I'll need to run your data through the debugger to figure out what code changes are required to catch these error conditions.
Thanks for letting us know about this issue!
John Halley Gotway
johnhg at ucar.edu
zhxubinchaoshan wrote:
> Hello,I have run the stat_analysis Tools, and get the output files, such as "stat_analysis.out", but I found there is some problem in it :
> JOB_LIST: -job aggregate_stat -fcst_var RH -fcst_lev P1000-250 -line_type SL1L2 -out_line_type CNT -out_alpha 0.050000 -rank_corr_flag 1
> COL_NAME: TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL E90_BCU
> CNT: 2786 69.48891 68.71808 70.25974 NA NA 20.75872 20.22763 21.31865 NA NA 86.46482 nan nan NA NA nan nan nan NA NA 0.26125 0.22631 0.29552 NA NA NA NA 0 0 0 -16.97591 -17.72427 -16.22755 NA NA 20.15360 19.63800 20.69721 NA NA 0.80367 NA NA NA NA NA 694.20333 NA NA 406.02186 NA NA 26.34774 NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
> I found that some values are "nan". And if I deal with the UGRD and TMP, there is no problem, but if I deal with the SFPH and RH, there is the above problem. Is there any wrong in it?
> Thank you very much!
>
>
>
> 在2009-07-07，"John Halley Gotway" <johnhg at rap.ucar.edu> 写道：
>> Take a look in the MODE output file you sent me (mode_APCP_06_SFC_vs_APCP_06_SFC_060000L_20070707_060000V_060000A_obj.txt). And look in the column labeled "OBJECT_ID". You'll see that there are only
>> 3 entries: O001, O002, and O003. The 'O' in front means that these are observation objects, as opposed to 'F' for forecast objects.
>>
>> When you run your first job, you're passing the "-fcst" flag which says to only keep lines for forecast objects. And since the 3 lines in that file are all for observation objects, that's why there
>> are 0 lines kept. The file "job_summary_APCP_06_simple_fcst.txt" isn't created since there are no lines to dump.
>>
>> Make sense?
>>
>> Also, MODE-Analysis is most useful to run to summarize many MODE output files. For example, if you've run MODE on a month's worth of data and want to summarize the object attributes/differences,
>> MODE-Analysis can be very helpful. But if you're just looking at a single case, it may be easier to just open up that single output file and read the MODE output.
>>
>> Thanks,
>> John
>>
>> zhxubinchaoshan wrote:
>>> OK,I send you the two files, thanks for your help!
>>>
>>>
>>>
>>>
>>> 在2009-07-07，"John Halley Gotway" <johnhg at rap.ucar.edu> 写道：
>>>> Hello,
>>>>
>>>> The MODE-Analysis tool (and Stat-Analysis) may be run with or without a config file. Most of the time it's easiest to just run them from the command line without using a config file. I find that
>>>> it's only helpful to use the config file when I'm doing some complicated filtering on the data.
>>>>
>>>> So could you please send me two things:
>>>> (1) The command line (or script) you use to call MODE-Analysis.
>>>> (2) The MODE output file(s) that you're trying to analyze.
>>>>
>>>> I'll try running MODE-Analysis and tell you what's going on.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> zhxubinchaoshan wrote:
>>>>> Thanks for your reply!
>>>>> When I run the MODE-Analysis, I got the hint is that :
>>>>> *** Running MODE-Analysis to compute column summaries for simple forecast objects ***
>>>>>
>>>>>
>>>>>
>>>>> Total mode lines read = 3
>>>>> Total mode lines kept = 0
>>>>>
>>>>>
>>>>> *** Running MODE-Analysis to compute column summaries for simple observation objects ***
>>>>>
>>>>>
>>>>>
>>>>> Total mode lines read = 3
>>>>> Total mode lines kept = 3
>>>>>
>>>>> And then I failed to get the file job_summary_APCP_06_simple_fcst.txt, and all the variables in the file job_summary_APCP_06_simple_fcst.out are zero.
>>>>>
>>>>> I have sent you the file MODEAnalysisConfig,can you tell me what the problem is?
>>>>>
>>>>> And I want to know that if the MODE-Analysis is not necessary, because I found I can get the verification of WRF output after running the MODE tools.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Total mode lines read = 3
>>>>> Total mode lines kept = 3
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 在2009-07-07，"John Halley Gotway" <johnhg at rap.ucar.edu> 写道：
>>>>>> Great, I'm glad to hear that you got the lat/lon's figured out, and are able to run MODE!
>>>>>>
>>>>>> As for your questions on thresholding, let me explain a little...
>>>>>>
>>>>>> The TS or ETS is computed by the Grid-Stat and Point-Stat tools. The user selects a threshold value of interest, and that threshold is applied to forecast/observation matched pairs to determine hits,
>>>>>> misses, false alarms, and correct negatives. The ETS (and several other contingency table statistics) are derived from those values.
>>>>>>
>>>>>> In MODE, we use a two-step method to define objects. First, we apply a convolution filter to the raw fields. The user decides how much smoothing is to be done by setting the "fcst_conv_radius" and
>>>>>> "obs_conv_radius" parameters. Second, we threshold the convolved field using the "fcst_conv_thresh" and "obs_conv_thresh" parameters. And from that thresholded field, we define the objects.
>>>>>>
>>>>>> To answer your question directly, I believe the general answer is NO, those thresholds aren't doing exactly the same thing. Grid-Stat and Point-Stat are thresholding the RAW fcst/obs values. Whereas
>>>>>> MODE is thresholding the convolved, smoothed fields. So in general they won't result in exactly the same grid points being turned on.
>>>>>>
>>>>>> HOWEVER, if you set the convolution radii in MODE equal to 0, no convolution will be done, and you will be thresholding the raw field. But that generally isn't recommended because you'll likely end
>>>>>> up with a large number of very small objects that'll be difficult to work with.
>>>>>>
>>>>>> Hope that helps.
>>>>>>
>>>>>> John
>>>>>>
>>>>>> zhxubinchaoshan wrote:
>>>>>>> Thanks for your suggestions! And I have figure out the problem by your suggestion. The problem is that my lat's and lon's switched.
>>>>>>> In the calculating of TS or ETS, there is a parameter 'threshold'. And in the MODE tool, there is a parameter 'obs_conv_thresh'. Are they the same?
>>>>>>> As we know, in the forecast system, we generally make contrast between different threshold TS or ETS. So if I can make contrast between MODE results with different threshold, just by changing the parameter 'obs_conv_thresh'?
>>>>>>> Thank you very much!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 在2009-07-02，"John Halley Gotway" <johnhg at rap.ucar.edu> 写道：
>>>>>>>> I took a look at the data file files you sent. The problem is clearly in the generation of the observation file. The forecast file is fine... the only reason the image looks funny in the PostScript
>>>>>>>> file you sent is that you're masking the bad data values from the observation field onto the forecast. Try setting "mask_missing_flag = 0" in the MODE config file, rerun the data, and you'll see that
>>>>>>>> the forecast field is fine.
>>>>>>>>
>>>>>>>> So there's some problem in how you're generating the observation file. I see this type of problem all the time in when I'm converting data. You just have some indexing out of order when you're
>>>>>>>> creating the observation NetCDF file. I'd suggest playing around with the tool you're using to create the NetCDF file and look at how you're ordering the data. Perhaps you have your lat's and lon's
>>>>>>>> switched... or you have x where you should have nx - x.... or something like that.
>>>>>>>>
>>>>>>>> You can check how the observation NetCDF file looks by viewing it using "ncview". Once it looks good in "ncview" it should work fine for MODE.
>>>>>>>>
>>>>>>>> Good luck. I'm guessing it's a minor issue in how you're ordering the data, and once you figure it out, it'll all work fine.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>> zhxubinchaoshan wrote:
>>>>>>>>> Thanks for your reply!
>>>>>>>>> When I run MODE, I am comparing a forecast netCDF file to an observed netCDF file. The observed netCDF file is produced according to the pcp_combine output format.
>>>>>>>>> I send the MODE output to you, can you help me check how the problem come out?
>>>>>>>>> Thank you very much!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 在2009-07-01，"John Halley Gotway" <johnhg at rap.ucar.edu> 写道：
>>>>>>>>>> When you run MODE are you comparing a forecast GRIB file to an observed GRIB file? If that is the case, you can run the "copygb" utility to interpolate GRIB files from one grid to another.
>>>>>>>>>>
>>>>>>>>>> The "copygb" utility is included as part of the WRF-PostProcessor or may be downloaded/compiled separately.
>>>>>>>>>>
>>>>>>>>>> Here's some info about copygb:
>>>>>>>>>> http://www.cpc.noaa.gov/products/wesley/copygb.html
>>>>>>>>>> Slides 24-29 of the PDF file: http://www.mmm.ucar.edu/wrf/users/tutorial/200901/wpp.pdf
>>>>>>>>>>
>>>>>>>>>> And I've also attached the copygb documentation file that's included with the code.
>>>>>>>>>>
>>>>>>>>>> You can use it to regrid one dataset to another so that they're on the same grid when running MODE. Hope that helps.
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> zhxubinchaoshan wrote:
>>>>>>>>>>> Hello, I have interpolate my precipitation data from station observations to grid using the Cressman method. But the Cressman method I used is form GRADS. So the grid interpolated is not consistent with the one in WRF model. So when I run the MODE, I found the raw observation graphic is wrong.
>>>>>>>>>>> So my question is : Do I need to interpolate my precipitation data in consistent with the WRF model grid?
>>>>>>>>>>> And if I need to do that, is there any good method you can provide for me to do it?
>>>>>>>>>>> Thanks very much!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 在2009-06-30，"John Halley Gotway" <johnhg at rap.ucar.edu> 写道：
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> You are correct. The 2 most important parameters in the MODE configuration file are the convolution radius and the convolution threshold. They determine how the objects are defined from the raw field.
>>>>>>>>>>>>
>>>>>>>>>>>> And actually, there are 4 entries in the config file for these - 2 for the forecast field and 2 for the observation field:
>>>>>>>>>>>> fcst_conv_radius
>>>>>>>>>>>> obs_conv_radius
>>>>>>>>>>>> fcst_conv_thresh
>>>>>>>>>>>> obs_conv_thresh
>>>>>>>>>>>>
>>>>>>>>>>>> But if you're comparing the same field between the fcst and obs, just set the obs settings to the same values as the fcst.
>>>>>>>>>>>>
>>>>>>>>>>>> To address your question... unfortunately, there is no one answer based on the model resolution. Instead, it's up to the user to decide what scale of objects to analyze.
>>>>>>>>>>>>
>>>>>>>>>>>> But here are some guidelines:
>>>>>>>>>>>> - Setting the convolution radius lower will make the objects less smooth and more detailed.
>>>>>>>>>>>> - Setting the convolution radius higher will make the objects smoother.
>>>>>>>>>>>> - Setting the threshold lower will make the objects bigger.
>>>>>>>>>>>> - Setting the threshold higher will make the objects smaller.
>>>>>>>>>>>>
>>>>>>>>>>>> It's up to you to decide how you'd like to define the objects based on what type of features you'd like to extract from the field. For example, when running MODE on precip, you may define objects
>>>>>>>>>>>> that are pretty large and smooth to represent large systems or MCS's. Or you may define them to be pretty small and detailed to capture convection. You need to ask yourself the question, what
>>>>>>>>>>>> features am I trying to analyze in my output?
>>>>>>>>>>>>
>>>>>>>>>>>> You'll just need to play around with running MODE to get a sense of how the settings work. Choose a setting for the convolution radius (5, 10, 15, 20 grid squares?). Just pick one and set it. Then
>>>>>>>>>>>> set the convolution threshold to some value - for precip, just try "gt0.0". Then run MODE. And bring up the output PostScript file in a window. Next, adjust the convolution threshold, rerun MODE,
>>>>>>>>>>>> and observe how the objects change. Once you get a sense for how the convolution threshold works, try fixing that one and playing with the convolution radius, moving it up and down, and observing the
>>>>>>>>>>>> changes in the PostScript output. At first, I'd suggest keeping radius or threshold fixed, while you adjust the other one.
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry I can't provide more help. You'll need to play around with it to get a sense of what it's doing. If you try to run MODE to define a lot of objects (more than about 15 in each field), it'll
>>>>>>>>>>>> take a lot longer to run since there's more calculations to perform for each object.
>>>>>>>>>>>>
>>>>>>>>>>>> Once you do find some settings you like, feel free to send some sample data to us for advice on interpreting the output. If you do send some output, be sure to send the MODE configuration file you
>>>>>>>>>>>> used and the PostScript plots.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> John Halley Gotway
>>>>>>>>>>>> johnhg at ucar.edu
>>>>>>>>>>>>
>>>>>>>>>>>> zhxubinchaoshan wrote:
>>>>>>>>>>>>> Hello,I am using the MODE tool in MET for verification.
>>>>>>>>>>>>> And I have some problems about the parameters in it. In the WrfModeConfig, there are two important parameters.They are obs_conv_radius and obs_conv_thresh.I have read some referances, and I found that how to specify these two parameters is dependent on my model resolution. If my model resolution is 12km, how can I specify these two parameters?
>>>>>>>>>>>>> Thank you very much!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 在2009-05-13，"John Halley Gotway" <johnhg at rap.ucar.edu> 写道：
>>>>>>>>>>>>>> Great. Glad that did the trick for you.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> zhxubinchaoshan wrote:
>>>>>>>>>>>>>>> Thanks for your reply!It can work now!
>>>>>>>>>>>>>>> The problem is the "fcst_var". By your suggestion, I change "APCP_6" to "APCP_06", and then it can work successfully!
>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 在2009-05-11，"John Halley Gotway" <johnhg at rap.ucar.edu> 写道：
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What this warning message is telling you is that the Stat-Analysis tool is
>>>>>>>>>>>>>>>> looking in the directory "out/rain" for *.stat files, but it is not
>>>>>>>>>>>>>>>> finding any lines in those files that EXACTLY match the job you've
>>>>>>>>>>>>>>>> specified.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When I experience a problem like this, my usual approach is to take a step
>>>>>>>>>>>>>>>> back and under-specify the job just to make sure I'm matching some lines.
>>>>>>>>>>>>>>>> Then I take a look at the STAT lines that went into the job and adjust
>>>>>>>>>>>>>>> >from there. For example, you may try the following:
>>>>>>>>>>>>>>>> (1) Run this Stat-Analysis job on the command line:
>>>>>>>>>>>>>>>> ../bin/stat_analysis -lookin ./out/rain \
>>>>>>>>>>>>>>>> -job aggregate_stat -line_type FHO -out_line_type CTS \
>>>>>>>>>>>>>>>> -dump_row tmp.stat
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (2) Did this job run successfully? Did it find any input FHO lines to use?
>>>>>>>>>>>>>>>> If not, then maybe there aren't any FHO lines in your STAT files. If it
>>>>>>>>>>>>>>>> did find FHO lines, and the job ran fine, proceed to the next step.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (3) Take a look in the file "tmp.stat". How would you like to filter this
>>>>>>>>>>>>>>>> data down more? Do you see "APCP_6" in the column for "fcst_var"? Or
>>>>>>>>>>>>>>>> maybe it's actually "APCP_06"? Try adding that to the job:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ../bin/stat_analysis -lookin ./out/rain \
>>>>>>>>>>>>>>>> -job aggregate_stat -line_type FHO -fcst_var APCP_6 -out_line_type CTS \
>>>>>>>>>>>>>>>> -dump_row tmp.stat
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (4) Did this job run successfully, or was there a problem with how you
>>>>>>>>>>>>>>>> specified the "fcst_var". Now check the "tmp.stat" file again, and adjust
>>>>>>>>>>>>>>>> your job as necessary. One hint though, when running Stat-Analysis jobs
>>>>>>>>>>>>>>>> like this on the command line, you'll usually need to put a backslash
>>>>>>>>>>>>>>>> before special characters like this: -fcst_thresh "\>0.000"
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So I'd suggest approaching the problem like that. If you can't figure it
>>>>>>>>>>>>>>>> out, feel free to send along all of the STAT files in your "out/rain"
>>>>>>>>>>>>>>>> directory, the Stat-Analysis config file you're using, and the command
>>>>>>>>>>>>>>>> line you're using... and I'll try to figure out what's going on.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Good luck.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> John Halley Gotway
>>>>>>>>>>>>>>>> johnhg at ucar.edu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When I run the Stat_analysis Tool, I get the error below:
>>>>>>>>>>>>>>>>> WARNING: do_job_aggr_stat() -> no matching STAT lines found for job: -job
>>>>>>>>>>>>>>>>> aggregate_stat -fcst_var APCP_6 -fcst_thresh >0.000 -line_type FHO
>>>>>>>>>>>>>>>>> -dump_row ./out/stat_analysis/job_aggregate_stat_FHO_CTS.stat
>>>>>>>>>>>>>>>>> -out_line_type CTS -out_alpha 0.050000
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And my command is :
>>>>>>>>>>>>>>>>> ../bin/stat_analysis -config ./STATAnalysisConfig_07 -lookin ./out/rain
>>>>>>>>>>>>>>>>> -out ./out/stat_analysis/stat_analysis.out
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you tell me how to figure out the problem above?
>>>>>>>>>>>>>>>>> Thank you very much!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Met_help mailing list
>>>>>>>>>>>>>>>>> Met_help at mailman.ucar.edu
>>>>>>>>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>>>>>>>>>>>>>>
>
More information about the Met_help
mailing list