[Met_help] Re: Results of beta-test METv1.1

[John Halley Gotway]

Victor and Lorena,

Thanks for sending some of your data.  I was able to run Point-Stat with it and reproduce the spurious results you were seeing.

Based on that, I did find a bug in the routine that's computing the confidence intervals for the Contingency Table Statistics (i.e. CTS line).  I had placed two parentheses in the wrong position.  So 
unfortunately, that bug is producing these bad results.  After I fixed it, the confidence intervals look a lot more reasonable.  I also recomputed the confidence interval using an online calculator 
and got the exact same answer.  So I'm confident that they're being computed correctly now.

I can't thank you enough for letting me know about this bug.  I've posted a fix for METv1.0 to the known issues page of the MET website.  The same fix will work on the METv1.1beta version you have.

On the known issues page (http://www.dtcenter.org/met/users/support/known_issues/METv1.0/METv1.0_known_issues.php), please see the fix titled:
"Error in the confidence interval values computed for the CTS output line from Point-Stat and Grid-Stat"
Then you'll need to update your version of the file "compute_ci.cc" and rebuild.

For your information, we're using the Wilson Method for computing the confidence intervals in the CTS line.  Here's info on the Wilson Method:
http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

The equations listed under the "Wilson score interval" section of this page match what we're using.

Thanks again.  And please let me know if you run into any more problems or encounter questionable results.

John

Victor Homar wrote:
> Dear John,
> 
> We have tried out this version of MET and it has improved functionality 
> and versatility.
> As "responsible" beta testers, we have a few comments for you:
> 
> a) There is a bit of confusion about the format of the ASCII 
> observations file to feed into ASCII2NC.
> The help from the command ASCII2NC shows this order:
> Message_Type Station_ID Valid_Time  Lat(Deg North) Lon(Deg East) 
> Elevation(msl) Level Height(msl) Grib_Code Observation_Value
> 
> whereas in ascii2nc.cc source code lines 391-394, the ordering is 
> different.
>      obs_arr[1] = atof(dl[6]); // Grib Code
>      obs_arr[2] = atof(dl[7]); // Level: pressure level or accumulation 
> interval
>      obs_arr[3] = atof(dl[8]); // Meters above sea level
>      obs_arr[4] = atof(dl[9]); // Observation Value
> 
> Our tests confirm that this later order provides the expected results 
> and so the documentation of ASCII2NC should be corrected.
> 
> b) The point_stat output file *_cop.txt has an incorrectly labeled pair 
> of columns. Columns OY and ON actually show FY and FN. Please either 
> change point_stat.cc lines 2410 and 2411 to referr to something like 
> cts_info.cts.oy and cts_info.cts.on (instead of cts_info.cts.fy and 
> cts_info.cts.fn) or change the *_cod.txt header and comments in 
> point_stat.cc lines 2432 and 2433.
> 
> c) The calculation of the confidence intervals is another nice feature 
> of MET. However, we do not quite understand our preliminary results. For 
> instance, we get:
> VRS MODEL FCST_LEAD FCST_VALID      OBTYPE VX_MASK      
> CTS_T/ALPHA          VAR       LEVEL        = TOTAL           
> BASER           BASER_CL        BASER_CU  ...
> V01 WRF   300000    20011111_060000 ADPSFC PENINSULA    
> CTS>=2.000/0.050     APCP      A030         = 5               
> 0.80000         0.89003         1.47827        ...
> 
> It is a bit weird the BASER_CL is above BASER. It doesn't sound 
> reasonable that the confidence interval does not include the expected 
> value (BASER). The same applies for all confidence intervals we have 
> checked.
> 
> d)  The new PCP_COMBINE option to substract precip fields seems to work 
> fine. Actually, we could not use the new fields because these are in 
> netcdf format whereas point_stat uses GRIB gridded fields. Do you know 
> how can we use the PCP_COMBINE in POINT_STAT?
> 
> Again, thank you very much for your efforts,
> 
> Best,
> 
> Lorena and Victor.
> 
> 
> John Halley Gotway wrote:
>> Victor,
>>
>> I have a beta release of METv1.1 ready for you to start working with.
>>
>> The following changes are included in this beta version:
>> - PB2NC
>>    - Add the ability to process multiple PREPBUFR file at one time.
>> - ASCII2NC
>>    - Add new tool for converting ASCII observations to NetCDF for 
>> input into
>>      Point-Stat.
>> - PCP-Combine
>>    - Add "-subtract" and "-add" command line options to compute 
>> differences
>>      of accumulation intervals rather than just sums.
>>    - Add "-gc" command line option to allow the user to specify which 
>> grib
>>      code to use.
>> - Point-Stat
>>    - Add the ability to dump out the matched pair data directly to ASCII
>>      and VSDB files.
>> - Grid-Stat
>>    - Update to read NetCDF output of PCP-Combine
>> - MODE
>>    - Update to read NetCDF output of PCP-Combine
>>
>> So in your case, you'll use the "-subtract" option in PCP-Combine to 
>> do 30 hours of accumulation - 6 hours of accumulation.  And you'll use 
>> the ASCII2NC tool to convert your observations to NetCDF. I've 
>> modified that NetCDF point observation format to make it a little more 
>> simple.  I removed the entries that were not actually being used by 
>> Point-Stat.  Please use the version of the configuration files that 
>> are distributed with this code, rather than the ones from version 
>> 1.0.  The config file for PointStat in particular has changes slightly.
>>
>> You can find the code via FTP at:
>> ftp.rap.ucar.edu/incoming/irap/johnhg/METv1.1beta.20080206.tar.gz
>>
>> Please let me know if you run into any problems or have suggestions 
>> for changes.
>>
>> Thanks,
>> John Halley Gotway