[Met_help] Precip observations in MET
John Halley Gotway
johnhg at rap.ucar.edu
Thu Jan 31 11:27:27 MST 2008
Victor,
Great! I'm glad you were able to get it to work - albeit with a lot of "cheating".
We're working toward the next release of the MET (METv1.1) for early Spring 2008. I have a long list of things I'm trying to implement for that (see
http://www.dtcenter.org/met/users/support/future_releases/METv1.1/METv1.1_plans.php).
However, I would like to go ahead and send you a beta version of METv1.1 to try out on your data.
The 2 things in which you'll be most interested are:
- The addition of the ASCII2NC tool for converting ASCII obs to NetCDF.
- The addition of the "-subtract" option to the PCP-Combine tool so that you can subtract the 6 hours of accumulation from the 30 hours.
However I do need a couple of days still to clean things up and package them into a beta version. I hope to have it ready for you early next week.
The version I send you won't have everything implemented - but those 2 pieces you need will be there.
Does that sound reasonable?
Thanks,
John
Victor Homar wrote:
> Hi John,
>
> First things first: the patched met_stats.cc that you provided did
> compile and executed successfully right away. No problems on our side.
>
> Regarding the verification process...
> We got it! We made our first precipitation verification plot from MET
> (see attached pdf). A few obstacles hampering the process but nothing
> unsurmountable.
> We overcome the 30h accumulation problem using R to "cheat" on the WRF
> netcdf and substitute in the netcdf file the actual 30h acc. field for
> the 24h desired field. Then, wrfpostproc converts the netcdf to GRIB
> (thinking the 30h precip is correct) and we use the resulting GRIB file
> at 30h within MET. This GRIB file, together with the observations file
> your provided (which also cheats about the accumulated rain field, 30h
> instead of the actual 24h) computes the verification scores.
> Although there is quite a bit of cheating along the way, we believe the
> final scores correspond to our aimed results: verification of 24h acc
> rainfall observations with 24 acc. forecasts valid at 30h lead time.
>
> Now, we would need your program to create further netcdf observation
> files from ASCII reports in order to perform more detailed tests.
>
> Let us know if there is any other test we can do for you,
>
> Thanks a lot for your support,
>
> Lorena and Víctor.
> PS: By the way, we also get an "Floating Point Exception" when trying to
> run the wrfpostproc only on T+30h. Actually, we get the error whenever
> we do not include the T+00h (initial conditions) in the requested
> postprocess times. We don't think this concerns you but, please, let us
> know if you have a solution to fix it. Thanks.
>
>
> John Halley Gotway wrote:
>> Victor,
>>
>> Please use the attached version of met_stats.cc instead of the one I
>> just sent. I included another fix in there that's posted to the MET
>> website. So it contains your fix plus one other.
>>
>> Thanks,
>> John
>>
>> John Halley Gotway wrote:
>>> Victor,
>>>
>>> Thanks for posting that data. It helped me debug the problem.
>>>
>>> I see two issues here.
>>>
>>> The first issue is the segmentation fault you're getting when you run
>>> Point-Stat on your 30-hour precip field. Fortunately, I have an easy
>>> fix for you which should solve this. We just need to modify one line
>>> in one of the library files. We haven't tested Point-Stat on precip
>>> data before, so we didn't catch this bug. So thanks for finding it!
>>>
>>> Please replace the file: METv1.0/src/lib/vx_met_util/met_stats.cc
>>> with the version of met_stats.cc attached to this message.
>>> Then rebuild MET by doing a "make clean" followed by a "make".
>>> Point-Stat should now run on your data without the error.
>>>
>>> Please let me know if the fix works on your system. Once I verify
>>> that it works, I'll post the fix to the MET website.
>>>
>>> Also, I've attached a version of your observations that I modified to
>>> test the fix. I just changed your 24-hour accumulation observations
>>> to 30-hour accumulations. When I run Point-Stat using the 30-hour
>>> accumulation forecast and the 30-hour point observations, it finds
>>> 2498 data pairs.
>>>
>>> Second, you're wondering how you'd go about subtracting a 6-hour
>>> accumulation from a 30-hour accumulation to get a 24-hour
>>> accumulation. Right now, the answer is that you can't do that with
>>> MET. But, in a future release, you'll be able to. The Pcp-Combine
>>> tool within MET can be used to sum up accumulation intervals but not
>>> to do differences. However, we do realize the need to be able to do
>>> differences for the exact reason you've pointed out. We will
>>> implement that soon... I'll talk to the MET development team about
>>> getting it in the next release. So you'll ultimately be able to use
>>> Pcp-Combine to do differences.
>>>
>>> Please let me know if the fix worked for you.
>>>
>>> Thanks,
>>> John
>>>
>>> Victor Homar wrote:
>>>> Hi John,
>>>>
>>>> Thanks for your patience, we did not realized that detail regarding
>>>> GRIB/Netcdf formats and vertical levels.
>>>> Now, we converted the forecast file to GRIB but are still not
>>>> successfully running MET.
>>>> Let us explain our trials:
>>>>
>>>> We try the command:
>>>> ../bin/point_stat
>>>> ~/WRF-ARW/wrfpostprocV2/cas_John/postprd/wrfprs_d01.30
>>>> ~/verificacio/Observations_INM_rain_24h_mm.nc \
>>>> config/PointStatConfig -outdir ../out/prova_point_stat
>>>>
>>>> (with a vx_grib_code[] = [ "APCP/A030" ]; in config/PointStatConfig
>>>> file)
>>>>
>>>> but get a:
>>>> Forecast File:
>>>> /megadisk/people/lorena/WRF-ARW/wrfpostprocV2/cas_John/postprd/wrfprs_d01.30
>>>>
>>>> Observation File:
>>>> /megadisk/people/lorena/verificacio/Observations_INM_rain_24h_mm.nc
>>>> Configuration File: config/PointStatConfig
>>>> Climatology File: none
>>>> Segmentation fault (core dumped)
>>>>
>>>> If we change vx_grib_code[] = [ "APCP/A024" ]; the output is a bit
>>>> better but not much:
>>>> Forecast File:
>>>> /megadisk/people/lorena/WRF-ARW/wrfpostprocV2/cas_John/postprd/wrfprs_d01.30
>>>>
>>>> Observation File:
>>>> /megadisk/people/lorena/verificacio/Observations_INM_rain_24h_mm.nc
>>>> Configuration File: config/PointStatConfig
>>>> Climatology File: none
>>>> ***WARNING***: process_grib_codes() -> no records matching grib code
>>>> 61 with level indicator of A024 not found in grib file:
>>>> /megadisk/people/lorena/WRF-ARW/wrfpostprocV2/cas_John/postprd/wrfprs_d01.30
>>>>
>>>> Output VSDB file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V.vsdb
>>>> Output FHO file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_fho.txt
>>>> Output CTC file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_ctc.txt
>>>> Output CTP file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_ctp.txt
>>>> Output CFP file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_cfp.txt
>>>> Output COP file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_cop.txt
>>>> Output CTS file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_cts.txt
>>>> Output CNT file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_cnt.txt
>>>> Output SL1L2 file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_sl1l2.txt
>>>> Output SAL1L2 file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_sal1l2.txt
>>>> Output VL1L2 file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_vl1l2.txt
>>>> Output VAL1L2 file:
>>>> ../out/prova_point_stat/point_stat_300000L_20011111_060000V_val1l2.txt
>>>>
>>>>
>>>> Now the program point_stat runs well but it complains about not
>>>> finding 24h accumulated rain in the forecast file. This is actually
>>>> correct, but we don't know how to get a GRIB forecast file with 24h
>>>> accumulated rainfall (IC+06h to IC+30h) from our WRF output. Is
>>>> there any simple method to do that?
>>>>
>>>> We tried to get the desired field with the nco package directly over
>>>> WRF netCDF output but since WRF doesn't render total accumulated
>>>> rain (convective and gridscale rain), we got stuck there and don't
>>>> know how to proceed from here.
>>>>
>>>> Our forecast period is 30 h because we allow the first 6h of
>>>> forecast for the finer model scales to fill up and the simulation to
>>>> spin-up adequately. The verification period encompasses the last 24h
>>>> of that period.
>>>>
>>>> Do you have any suggestions on how to create a GRIB file with 24h
>>>> accumulated rain from IC+06h to IC+30h so we can feed it to point_stat?
>>>>
>>>> Many thanks,
>>>> Lorena and Víctor.
>>>> PS: We uploaded to our ftp server the two "wrfpostprocessed" GRIB
>>>> files from our WRF output file, valid at IC+06h (
>>>> ftp://eady.uib.es/pub/victor/MET/WRFPRS_d01.06 ) and IC+30h (
>>>> ftp://eady.uib.es/pub/victor/MET/WRFPRS_d01.30 ) just in case these
>>>> could help clarify the problem.
>>>>
>>>> John Halley Gotway wrote:
>>>>> Victor,
>>>>>
>>>>> I took the ASCII point observations you provided and converted them
>>>>> to NetCDF with the tool I'm working on. I've attached them as an
>>>>> example.
>>>>>
>>>>> However, the forecast file you provided is in NetCDF format. MET
>>>>> expects gridded data to be provided in Grib format. That's why we
>>>>> suggest the use the WRF-PostProcessor to destagger the WRF output
>>>>> files, interpolate to pressure levels, and write out Grid data.
>>>>>
>>>>> Do you have your gridded forecasts available in Grib1 format?
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> Victor Homar wrote:
>>>>>> Hi John,
>>>>>>
>>>>>> Thanks for your responses.
>>>>>>
>>>>>> Your plan to provide us with a tar ball with the whole thing to
>>>>>> test it sounds good.
>>>>>>
>>>>>> Answering your questions:
>>>>>> 1) The format makes good sense. The Message_Type field is a little
>>>>>> artificial for the particular set of stations we are dealing with
>>>>>> now. These are not synoptic stations but a particular network of
>>>>>> our Met Service (They call them "climatological network"). I guess
>>>>>> the ADPSFC code will work for all further steps along the
>>>>>> verification process.
>>>>>> Also, and thinking about general users, the format makes good
>>>>>> sense because it has been straight forward to adapt our data to it.
>>>>>>
>>>>>> 2) No suggestions, at first thought it looks like a reasonable and
>>>>>> versatile format to import generic observations.
>>>>>>
>>>>>> 3 and 4) I uploaded both an observations file and a WRF forecast
>>>>>> file to:
>>>>>>
>>>>>> ftp://eady.uib.es/pub/victor/MET/
>>>>>>
>>>>>> The WRF forecast file is a thinned file using the utility "ncks"
>>>>>> to include only the variables Times,RAINC,RAINNC,XLONG and XLAT.
>>>>>> Let me know if you need the big (3.5Gb) complete WRFout file.
>>>>>>
>>>>>> Thanks for all your help and have a nice weekend,
>>>>>>
>>>>>> Víctor.
>>>>>>
>>>>>> John Halley Gotway wrote:
>>>>>>> Victor,
>>>>>>>
>>>>>>> To answer your questions...
>>>>>>> This piece of code, like most of MET, would be written in C++.
>>>>>>> To test it, I'd like to try it out on some data internally and
>>>>>>> some of your data as well.
>>>>>>>
>>>>>>> I was working on this tool today, and I have some ideas. As for
>>>>>>> the format of the ASCII observation file, I'm thinking that each
>>>>>>> line could consist of the following 9 pieces of info:
>>>>>>> Message_Type Station_ID Valid_Time(YYYYMMDD_HHMMSS) Lat(Deg
>>>>>>> North) Lon(Deg East) Elevation(m) Accum(h)_or_Pressure(hPa)
>>>>>>> Grib_Code Observation_Value
>>>>>>>
>>>>>>> So for example, here are 4 lines of ASCII observation values (I
>>>>>>> made them up):
>>>>>>> ADPSFC SID1 20050810_120100 40.00 -105.00 -9999 12 61 2.01
>>>>>>> ADPSFC SID2 20050810_120500 41.00 -105.00 -9999 12 61 0.85
>>>>>>> ADPSFC SID3 20050810_121000 42.00 -105.00 -9999 12 61 0.90
>>>>>>> ADPSFC SID3 20050810_121000 42.00 -105.00 -9999 500 11 300
>>>>>>> They consist of 3 observation of 12-hourly accumulated precip
>>>>>>> (grib code = 61), and 1 observation of temperature (grib code =
>>>>>>> 11) at 500 hPa. I've attached the netCDF (sample_ascii2nc.nc)
>>>>>>> file that would be created from these 4 observations and could be
>>>>>>> used as input to Point-Stat.
>>>>>>>
>>>>>>> The elevation value is not currently used in Point-Stat, but we
>>>>>>> may want to use it in the future. So I've included it as one of
>>>>>>> the things to enter, but you can just put fill data in there
>>>>>>> right now.
>>>>>>>
>>>>>>> However, I've realized that Point-Stat needs to be tweaked
>>>>>>> slightly to perform point verification of accumulated
>>>>>>> precipitation. It wasn't matching the observations up
>>>>>>> correctly. So when I provide you with this tool to test on your
>>>>>>> end, I'll probably just send an updated tarball containing all of
>>>>>>> the necessary changes.
>>>>>>>
>>>>>>> My questions for you are:
>>>>>>> Does the ASCII observation format make sense, and would it be
>>>>>>> easy to use?
>>>>>>> Do you have any suggestions for changes or additions to the ASCII
>>>>>>> observation format?
>>>>>>> Would you be able to provide me with some of your rain-gauge
>>>>>>> observations formatted in this way?
>>>>>>> And would you be able to provide me with a sample forecast file
>>>>>>> to match up to those observations?
>>>>>>>
>>>>>>> One last thing to mention. I realize that the intermediate
>>>>>>> NetCDF file format contains a lot of unnecessary fields in it.
>>>>>>> We plan to clean up that file format in the near future and
>>>>>>> organize it a little better.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> John
>>>>>>>
>>>>>>> Victor Homar wrote:
>>>>>>>> John,
>>>>>>>>
>>>>>>>> We'll be glad to collaborate in creating such a tool which would
>>>>>>>> make our (non BUFR users) life much easier with MET.
>>>>>>>> So far, we tried to create a NetCDF file for Point-Stat but
>>>>>>>> could not figure out what are the minimum set of parameters per
>>>>>>>> observation that the NetCDF file should include.
>>>>>>>>
>>>>>>>> We are currently concerned with observations of 24h accumulated
>>>>>>>> precipitation and looking at page 3-10 of the MET User's Guide,
>>>>>>>> it is not intuitive to guess valid values for parameters such as
>>>>>>>> "Vertical level", "Pressure level"or "Reason code". Attached is
>>>>>>>> a simple file (we could provide more information regarding
>>>>>>>> raingauge station characteristics) showing an example of the
>>>>>>>> type of ASCII files we deal with. Obviously, variations from
>>>>>>>> this format could easily be done.
>>>>>>>>
>>>>>>>> What your plans to test the your code would be? Is the code
>>>>>>>> written in Fortran or C, or you use a higher level language?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Lorena and Víctor
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> John Halley Gotway wrote:
>>>>>>>>> Lorena and Victor,
>>>>>>>>>
>>>>>>>>> This question has come up a lot. We'd like to figure out an
>>>>>>>>> easy way for users to incorporate ASCII observations into MET.
>>>>>>>>> I'm wondering if you'd be willing to work with me on developing a
>>>>>>>>> conversion tool that takes ASCII observations in a format
>>>>>>>>> similar to the one you've described and reformat them into the
>>>>>>>>> NetCDF format that Point-Stat expects?
>>>>>>>>>
>>>>>>>>> So I'm wondering, if I was to provide you with some code for
>>>>>>>>> doing this, would you be willing to test it out on your system
>>>>>>>>> and let me know how things go?
>>>>>>>>>
>>>>>>>>> Hopefully we could figure out a solution and include it in the
>>>>>>>>> next release of MET.
>>>>>>>>>
>>>>>>>>> If you'd be willing to work with you on this, could you please
>>>>>>>>> send me a couple of your forecast files and corresponding sets
>>>>>>>>> of ASCII observations? I'll use them to test out the code I
>>>>>>>>> write before
>>>>>>>>> sending it to you. Otherwise, if you'd like to do it yourself,
>>>>>>>>> I could provide you with some more info on how to do it.
>>>>>>>>>
>>>>>>>>> Thanks and please let me know,
>>>>>>>>> John Halley Gotway
>>>> --
>>>> -----------------------------------------------------------------------
>>>> Víctor Homar Santaner
>>>> Grup de Meteorologia
>>>>
>>>> Edif. Mateu Orfila Tel: +34 971 17 1376
>>>> Universitat de les Illes Balears Fax: +34 971 17 3426
>>>> 07122 Palma de Mallorca (SPAIN) Email: Victor.Homar at uib.es
>>>>
>>>> Knowledge is contagious. Infect truth.
>>>> -----------------------------------------------------------------------
>>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Met_help mailing list
>>> Met_help at mailman.ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>
>
More information about the Met_help
mailing list