[Met_help] [rt.rap.ucar.edu #56967] History for Help i try to use MET to verify data precipitation model from weather radar
John Halley Gotway via RT
met_help at ucar.edu
Thu Aug 23 08:22:19 MDT 2012
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
To: Developmental Testbed Center - MET
Hello,
My name is Sebastian Paez, i am doing my master thesis at the University of Stuttgart. I am generating a precipitation forecast model based
on radar images (as 3darrays) from the DWD (German Weather Service). I want to use MET to verify my models with observations from DWD.
The information from DWD is provided as HFD5 file, which I transformed to NetCDF3 format using the pacakge NetCDF-python.
I installed MET and run the tutorial without big problems. But when i want to use my .nc files I got this error:
Met2dDataFileFactory::new_met_2d_data_file() -> unsupported gridded data file type "4"
Could you tell me if it's possible to use MET with my data. I am attaching to this e-mail my .nc file in case you wanna check it.
I hope you can help me to figure out if it's worth to continue with trying with MET as verification tool.
Thanks in advance
Sebastian Paez
WAREM - University of Stuttgart
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #56967] Help i try to use MET to verify data precipitation model from weather radar
From: Paul Oldenburg
Time: Wed Jun 13 12:51:06 2012
Sabastian,
MET can only read gridded model data that is in one of three formats:
GRIB, pinterp NetCDF and MET NetCDF. The last two
formats are basically special NetCDF formats with specific naming
conventions and certain variables and attributes
defined. In your case, I think converting your data to the last
format, MET NetCDF, would be the easiest route. We use
the statistical scripting language R (http://www.r-project.org/) with
the NetCDF library
(http://cran.r-project.org/web/packages/ncdf/index.html) to process
NetCDF data. There is a script available on our
website
(http://www.dtcenter.org/met/users/downloads/Rscripts/trmm2nc.R) that
uses these tools to convert precipitation
data in a text file into MET NetCDF. You could modify that script and
use it to convert your NetCDF data.
I think this approach would be the easiest way for you to verify your
model data using MET. Whether or not you choose
to do so probably depends on how well acquainted you are with R and
what your alternatives are. If you have another
means of reading and writing NetCDF files (for example
http://www.unidata.ucar.edu/software/netcdf-java/), that is also
a possible route. You can see examples of the MET NetCDF file format
by inspecting the files created by the MET tool
pcp_combine when you run the test scripts. They can be found in
$MET_BASE/out/pcp_combine, where $MET_BASE is the
directory where MET was built on your system. The goal of conversion
is to mimic the format of these files.
The model data file you sent seems to have bad data (all 9.96921e+36)
for the only variable present, r_adv. I'm sorry
for such a long and complicated answer. Let me know if you have any
questions. Good luck.
Paul
On 06/13/2012 06:53 AM, Sebastian Paez via RT wrote:
>
> Wed Jun 13 06:53:52 2012: Request 56967 was acted upon.
> Transaction: Ticket created by sebastian_paez at hotmail.com
> Queue: met_help
> Subject: Help i try to use MET to verify data precipitation
model from weather radar
> Owner: Nobody
> Requestors: sebastian_paez at hotmail.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>
>
>
> To: Developmental Testbed Center - MET
>
> Hello,
>
> My name is Sebastian Paez, i am doing my master thesis at the
University of Stuttgart. I am generating a precipitation forecast
model based
> on radar images (as 3darrays) from the DWD (German Weather Service).
I want to use MET to verify my models with observations from DWD.
> The information from DWD is provided as HFD5 file, which I
transformed to NetCDF3 format using the pacakge NetCDF-python.
>
> I installed MET and run the tutorial without big problems. But when
i want to use my .nc files I got this error:
>
>
>
> Met2dDataFileFactory::new_met_2d_data_file() -> unsupported
gridded data file type "4"
>
>
>
> Could you tell me if it's possible to use MET with my data. I am
attaching to this e-mail my .nc file in case you wanna check it.
>
> I hope you can help me to figure out if it's worth to continue with
trying with MET as verification tool.
>
> Thanks in advance
>
> Sebastian Paez
> WAREM - University of Stuttgart
>
>
------------------------------------------------
Subject: Help i try to use MET to verify data precipitation model from weather radar
From: Sebastian Paez
Time: Thu Jun 14 10:10:09 2012
Paul,
I created a .nc file with all attributes as you mentioned and it
worked.Now, my information is from Germany and therefore my
precipitation data use the "Polar Stereographic Projection" in
difference with the Lambert Conformal I used in the file I created.
Can you tell me the attributes related to the "
Polar Stereographic Projection" that should be included in the .nc
file or if you have an example file using
" Polar Stereographic Projection" can you send it to me?
I am attaching the file I created and that worked in MET with Lambert
Conformal Projection.
Thanks again for your help
Sebastian
> Subject: Re: [rt.rap.ucar.edu #56967] Help i try to use MET to
verify data precipitation model from weather radar
> From: met_help at ucar.edu
> To: sebastian_paez at hotmail.com
> Date: Wed, 13 Jun 2012 12:51:07 -0600
>
> Sabastian,
>
> MET can only read gridded model data that is in one of three
formats: GRIB, pinterp NetCDF and MET NetCDF. The last two
> formats are basically special NetCDF formats with specific naming
conventions and certain variables and attributes
> defined. In your case, I think converting your data to the last
format, MET NetCDF, would be the easiest route. We use
> the statistical scripting language R (http://www.r-project.org/)
with the NetCDF library
> (http://cran.r-project.org/web/packages/ncdf/index.html) to process
NetCDF data. There is a script available on our
> website
(http://www.dtcenter.org/met/users/downloads/Rscripts/trmm2nc.R) that
uses these tools to convert precipitation
> data in a text file into MET NetCDF. You could modify that script
and use it to convert your NetCDF data.
>
> I think this approach would be the easiest way for you to verify
your model data using MET. Whether or not you choose
> to do so probably depends on how well acquainted you are with R and
what your alternatives are. If you have another
> means of reading and writing NetCDF files (for example
http://www.unidata.ucar.edu/software/netcdf-java/), that is also
> a possible route. You can see examples of the MET NetCDF file
format by inspecting the files created by the MET tool
> pcp_combine when you run the test scripts. They can be found in
$MET_BASE/out/pcp_combine, where $MET_BASE is the
> directory where MET was built on your system. The goal of
conversion is to mimic the format of these files.
>
> The model data file you sent seems to have bad data (all
9.96921e+36) for the only variable present, r_adv. I'm sorry
> for such a long and complicated answer. Let me know if you have any
questions. Good luck.
>
> Paul
>
>
> On 06/13/2012 06:53 AM, Sebastian Paez via RT wrote:
> >
> > Wed Jun 13 06:53:52 2012: Request 56967 was acted upon.
> > Transaction: Ticket created by sebastian_paez at hotmail.com
> > Queue: met_help
> > Subject: Help i try to use MET to verify data precipitation
model from weather radar
> > Owner: Nobody
> > Requestors: sebastian_paez at hotmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >
> >
> >
> > To: Developmental Testbed Center - MET
> >
> > Hello,
> >
> > My name is Sebastian Paez, i am doing my master thesis at the
University of Stuttgart. I am generating a precipitation forecast
model based
> > on radar images (as 3darrays) from the DWD (German Weather
Service). I want to use MET to verify my models with observations from
DWD.
> > The information from DWD is provided as HFD5 file, which I
transformed to NetCDF3 format using the pacakge NetCDF-python.
> >
> > I installed MET and run the tutorial without big problems. But
when i want to use my .nc files I got this error:
> >
> >
> >
> > Met2dDataFileFactory::new_met_2d_data_file() -> unsupported
gridded data file type "4"
> >
> >
> >
> > Could you tell me if it's possible to use MET with my data. I am
attaching to this e-mail my .nc file in case you wanna check it.
> >
> > I hope you can help me to figure out if it's worth to continue
with trying with MET as verification tool.
> >
> > Thanks in advance
> >
> > Sebastian Paez
> > WAREM - University of Stuttgart
> >
> >
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #56967] Help i try to use MET to verify data precipitation model from weather radar
From: Paul Oldenburg
Time: Tue Jun 19 11:38:47 2012
Sebastian,
You can see an example of a polar stereographic MET NetCDF file in the
test script output file:
$MET_BASE/out/ensemble_stat/ensemble_stat_20100101_120000V_ens.nc
where MET_BASE is set to the base directory where MET was installed on
your system. Note that you must run the
$MET_BASE/scripts/test_all.sh script to generate this output file.
Not all of the global attributes (see below) in that
file are projection-related, but I recommend that you include all of
them, anyway. Please let me know if you have any
questions.
Paul
// global attributes:
:FileOrigins = "File
../out/ensemble_stat/ensemble_stat_20100101_120000V_ens.nc generated
20120613_154116 UTC on host orval by the MET ensemble_stat tool" ;
:MET_version = "V4.0" ;
:MET_tool = "ensemble_stat" ;
:Projection = "Polar Stereographic" ;
:hemisphere = "N" ;
:scale_lat = "60.000000 degrees_north" ;
:lat_pin = "31.357000" ;
:lon_pin = "-129.770000" ;
:x_pin = "0.000000" ;
:y_pin = "0.000000" ;
:lon_orient = "-120.500000" ;
:d_km = "10.395000 km" ;
:r_km = "6371.200000 km" ;
:nx = "169" ;
:ny = "154" ;
}
On 06/14/2012 10:10 AM, Sebastian Paez via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>
>
>
>
>
> Paul,
> I created a .nc file with all attributes as you mentioned and it
worked.Now, my information is from Germany and therefore my
precipitation data use the "Polar Stereographic Projection" in
difference with the Lambert Conformal I used in the file I created.
>
> Can you tell me the attributes related to the "
> Polar Stereographic Projection" that should be included in the .nc
file or if you have an example file using
> " Polar Stereographic Projection" can you send it to me?
> I am attaching the file I created and that worked in MET with
Lambert Conformal Projection.
> Thanks again for your help
>
> Sebastian
>
>> Subject: Re: [rt.rap.ucar.edu #56967] Help i try to use MET to
verify data precipitation model from weather radar
>> From: met_help at ucar.edu
>> To: sebastian_paez at hotmail.com
>> Date: Wed, 13 Jun 2012 12:51:07 -0600
>>
>> Sabastian,
>>
>> MET can only read gridded model data that is in one of three
formats: GRIB, pinterp NetCDF and MET NetCDF. The last two
>> formats are basically special NetCDF formats with specific naming
conventions and certain variables and attributes
>> defined. In your case, I think converting your data to the last
format, MET NetCDF, would be the easiest route. We use
>> the statistical scripting language R (http://www.r-project.org/)
with the NetCDF library
>> (http://cran.r-project.org/web/packages/ncdf/index.html) to process
NetCDF data. There is a script available on our
>> website
(http://www.dtcenter.org/met/users/downloads/Rscripts/trmm2nc.R) that
uses these tools to convert precipitation
>> data in a text file into MET NetCDF. You could modify that script
and use it to convert your NetCDF data.
>>
>> I think this approach would be the easiest way for you to verify
your model data using MET. Whether or not you choose
>> to do so probably depends on how well acquainted you are with R and
what your alternatives are. If you have another
>> means of reading and writing NetCDF files (for example
http://www.unidata.ucar.edu/software/netcdf-java/), that is also
>> a possible route. You can see examples of the MET NetCDF file
format by inspecting the files created by the MET tool
>> pcp_combine when you run the test scripts. They can be found in
$MET_BASE/out/pcp_combine, where $MET_BASE is the
>> directory where MET was built on your system. The goal of
conversion is to mimic the format of these files.
>>
>> The model data file you sent seems to have bad data (all
9.96921e+36) for the only variable present, r_adv. I'm sorry
>> for such a long and complicated answer. Let me know if you have
any questions. Good luck.
>>
>> Paul
>>
>>
>> On 06/13/2012 06:53 AM, Sebastian Paez via RT wrote:
>>>
>>> Wed Jun 13 06:53:52 2012: Request 56967 was acted upon.
>>> Transaction: Ticket created by sebastian_paez at hotmail.com
>>> Queue: met_help
>>> Subject: Help i try to use MET to verify data precipitation
model from weather radar
>>> Owner: Nobody
>>> Requestors: sebastian_paez at hotmail.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>
>>>
>>>
>>> To: Developmental Testbed Center - MET
>>>
>>> Hello,
>>>
>>> My name is Sebastian Paez, i am doing my master thesis at the
University of Stuttgart. I am generating a precipitation forecast
model based
>>> on radar images (as 3darrays) from the DWD (German Weather
Service). I want to use MET to verify my models with observations from
DWD.
>>> The information from DWD is provided as HFD5 file, which I
transformed to NetCDF3 format using the pacakge NetCDF-python.
>>>
>>> I installed MET and run the tutorial without big problems. But
when i want to use my .nc files I got this error:
>>>
>>>
>>>
>>> Met2dDataFileFactory::new_met_2d_data_file() -> unsupported
gridded data file type "4"
>>>
>>>
>>>
>>> Could you tell me if it's possible to use MET with my data. I am
attaching to this e-mail my .nc file in case you wanna check it.
>>>
>>> I hope you can help me to figure out if it's worth to continue
with trying with MET as verification tool.
>>>
>>> Thanks in advance
>>>
>>> Sebastian Paez
>>> WAREM - University of Stuttgart
>>>
>>>
>>
>>
>>
>
>
>
------------------------------------------------
Subject: Help i try to use MET to verify data precipitation model from weather radar
From: Sebastian Paez
Time: Fri Jul 13 11:31:43 2012
Hello Paul,
I have been working in the verification of my fields. I have one more
question regarding this issue:
- i still can not locate correctly the fields on the polar
stereographic projection. I would like to know the description of the
projection attributes (scale_lat, lat_pin, lon_pin, x_pin, y_pin,
lon_orient, d_km, r_km). Based on these descriptions i can improve the
location of my fields.
I am sending attached the file I am using so you can take a look. The
field is the 1h accumulation of precipitation using the weather
radar.
Best Regards
Sebastian Paez
> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET to
verify data precipitation model from weather radar
> From: met_help at ucar.edu
> To: sebastian_paez at hotmail.com
> Date: Tue, 3 Jul 2012 08:09:02 -0600
>
> According to our records, your request has been resolved. If you
have any
> further questions or concerns, please respond to this message.
------------------------------------------------
Subject: Help i try to use MET to verify data precipitation model from weather radar
From: John Halley Gotway
Time: Fri Jul 13 12:36:21 2012
Hello Sebastian,
Unfortunately, we don't have those parameters well documented anywhere
at this point, but I can give you a brief description of them. These
are the NetCDF global attributes used in MET to define a
polar stereographic grid. For reference, here's the values you have
set in the example NetCDF file you sent:
:Projection = "Polar Stereographic" ;
:hemisphere = "N" ;
:scale_lat = "60.000000 degrees_north" ;
:lat_pin = "47.340000" ;
:lon_pin = "7.980000" ;
:x_pin = "0.000000" ;
:y_pin = "0.000000" ;
:lon_orient = "8.000000" ;
:d_km = "10.395000" ;
:r_km = "6371.200000" ;
:nx = "290" ;
:ny = "290 grid_points" ;
- x_pin and y_pin: specify the (x,y) location of the reference lat/lon
provided in (lat_pin, lon_pin). Yours are set to (0,0), meaning that
(lat_pin, lon_pin) refers to the lower-left corner of the
grid. Setting (x_pin, y_pin) = (nx, ny) would mean that you're
specifying the lat/lon of the upper-right corner of the grid.
- lat_pin and lon_pin are the lat/lon of the (x,y) specified above.
Usually, it's the lower-left corner.
- scale_lat is the latitude at which the spacing of the grid (defined
in d_km) is defined.
- d_km is the spacing between grid points in kilometers at the
latitude defined in scale_lat.
- lon_orient specifies the longitude value that's vertical in your
grid.
- r_km is the assumed radius of a spherical earth.
- nx and ny specify the size of the grid.
I used the plot_data_plane tool to make a plot of the sample data you
sent, and it's attached to this message. Looks like MET thinks your
data is over eastern Europe. Is that correct?
Now I'm not sure where you're getting the grid definition for this
data. But if, for example, you have a GRIB file, you may find the
following page of the MET online tutorial to be of use:
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
Hopefully that helps. Do you have additional questions?
Thanks,
John Halley Gotway
On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>
>
> Hello Paul,
> I have been working in the verification of my fields. I have one
more question regarding this issue:
> - i still can not locate correctly the fields on the polar
stereographic projection. I would like to know the description of the
projection attributes (scale_lat, lat_pin, lon_pin, x_pin, y_pin,
lon_orient, d_km, r_km). Based on these descriptions i can improve the
location of my fields.
> I am sending attached the file I am using so you can take a look.
The field is the 1h accumulation of precipitation using the weather
radar.
> Best Regards
> Sebastian Paez
>
>
>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET
to verify data precipitation model from weather radar
>> From: met_help at ucar.edu
>> To: sebastian_paez at hotmail.com
>> Date: Tue, 3 Jul 2012 08:09:02 -0600
>>
>> According to our records, your request has been resolved. If you
have any
>> further questions or concerns, please respond to this message.
>
>
------------------------------------------------
Subject: Help i try to use MET to verify data precipitation model from weather radar
From: Sebastian Paez
Time: Sat Jul 14 07:37:09 2012
Hello John,
thanks for your answer, finally i could locate correctly the fields. I
am using radar data from the south of Germany.I am attaching the
corrected file so you can see it. One more question about this: Is it
possible to change the background image thatMODE and Wavelt produce?
Since my data is in a range of 290kmx290km i would like to have more
details on the back.
Another question, regarding Point Stat, with the same data i want to
verify it to gauge stations' data. That means I want to verifygrid
forecasts and point observations and i want the results for each
station (without interpolation). I was looking for an example but I
could not find this particular case. Maybe you can send me a typical
file *.stations for "mask_sid", so i can see what do i need.
Best RegardsSebastian Paez
> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> From: met_help at ucar.edu
> To: sebastian_paez at hotmail.com
> Date: Fri, 13 Jul 2012 12:36:21 -0600
>
> Hello Sebastian,
>
> Unfortunately, we don't have those parameters well documented
anywhere at this point, but I can give you a brief description of
them. These are the NetCDF global attributes used in MET to define a
> polar stereographic grid. For reference, here's the values you have
set in the example NetCDF file you sent:
> :Projection = "Polar Stereographic" ;
> :hemisphere = "N" ;
> :scale_lat = "60.000000 degrees_north" ;
> :lat_pin = "47.340000" ;
> :lon_pin = "7.980000" ;
> :x_pin = "0.000000" ;
> :y_pin = "0.000000" ;
> :lon_orient = "8.000000" ;
> :d_km = "10.395000" ;
> :r_km = "6371.200000" ;
> :nx = "290" ;
> :ny = "290 grid_points" ;
>
> - x_pin and y_pin: specify the (x,y) location of the reference
lat/lon provided in (lat_pin, lon_pin). Yours are set to (0,0),
meaning that (lat_pin, lon_pin) refers to the lower-left corner of the
> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that you're
specifying the lat/lon of the upper-right corner of the grid.
>
> - lat_pin and lon_pin are the lat/lon of the (x,y) specified above.
Usually, it's the lower-left corner.
>
> - scale_lat is the latitude at which the spacing of the grid
(defined in d_km) is defined.
>
> - d_km is the spacing between grid points in kilometers at the
latitude defined in scale_lat.
>
> - lon_orient specifies the longitude value that's vertical in your
grid.
>
> - r_km is the assumed radius of a spherical earth.
>
> - nx and ny specify the size of the grid.
>
> I used the plot_data_plane tool to make a plot of the sample data
you sent, and it's attached to this message. Looks like MET thinks
your data is over eastern Europe. Is that correct?
>
> Now I'm not sure where you're getting the grid definition for this
data. But if, for example, you have a GRIB file, you may find the
following page of the MET online tutorial to be of use:
>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
>
> Hopefully that helps. Do you have additional questions?
>
> Thanks,
> John Halley Gotway
>
> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >
> >
> > Hello Paul,
> > I have been working in the verification of my fields. I have one
more question regarding this issue:
> > - i still can not locate correctly the fields on the polar
stereographic projection. I would like to know the description of the
projection attributes (scale_lat, lat_pin, lon_pin, x_pin, y_pin,
lon_orient, d_km, r_km). Based on these descriptions i can improve the
location of my fields.
> > I am sending attached the file I am using so you can take a look.
The field is the 1h accumulation of precipitation using the weather
radar.
> > Best Regards
> > Sebastian Paez
> >
> >
> >> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET
to verify data precipitation model from weather radar
> >> From: met_help at ucar.edu
> >> To: sebastian_paez at hotmail.com
> >> Date: Tue, 3 Jul 2012 08:09:02 -0600
> >>
> >> According to our records, your request has been resolved. If you
have any
> >> further questions or concerns, please respond to this message.
> >
> >
>
>
------------------------------------------------
Subject: Help i try to use MET to verify data precipitation model from weather radar
From: Sebastian Paez
Time: Sat Jul 14 07:38:23 2012
Sorry, now the attachment
From: sebastian_paez at hotmail.com
To: met_help at ucar.edu
Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET
to verify data precipitation model from weather radar
Date: Sat, 14 Jul 2012 08:35:06 -0500
Hello John,
thanks for your answer, finally i could locate correctly the fields. I
am using radar data from the south of Germany.I am attaching the
corrected file so you can see it. One more question about this: Is it
possible to change the background image thatMODE and Wavelt produce?
Since my data is in a range of 290kmx290km i would like to have more
details on the back.
Another question, regarding Point Stat, with the same data i want to
verify it to gauge stations' data. That means I want to verifygrid
forecasts and point observations and i want the results for each
station (without interpolation). I was looking for an example but I
could not find this particular case. Maybe you can send me a typical
file *.stations for "mask_sid", so i can see what do i need.
Best RegardsSebastian Paez
> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> From: met_help at ucar.edu
> To: sebastian_paez at hotmail.com
> Date: Fri, 13 Jul 2012 12:36:21 -0600
>
> Hello Sebastian,
>
> Unfortunately, we don't have those parameters well documented
anywhere at this point, but I can give you a brief description of
them. These are the NetCDF global attributes used in MET to define a
> polar stereographic grid. For reference, here's the values you have
set in the example NetCDF file you sent:
> :Projection = "Polar Stereographic" ;
> :hemisphere = "N" ;
> :scale_lat = "60.000000 degrees_north" ;
> :lat_pin = "47.340000" ;
> :lon_pin = "7.980000" ;
> :x_pin = "0.000000" ;
> :y_pin = "0.000000" ;
> :lon_orient = "8.000000" ;
> :d_km = "10.395000" ;
> :r_km = "6371.200000" ;
> :nx = "290" ;
> :ny = "290 grid_points" ;
>
> - x_pin and y_pin: specify the (x,y) location of the reference
lat/lon provided in (lat_pin, lon_pin). Yours are set to (0,0),
meaning that (lat_pin, lon_pin) refers to the lower-left corner of the
> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that you're
specifying the lat/lon of the upper-right corner of the grid.
>
> - lat_pin and lon_pin are the lat/lon of the (x,y) specified above.
Usually, it's the lower-left corner.
>
> - scale_lat is the latitude at which the spacing of the grid
(defined in d_km) is defined.
>
> - d_km is the spacing between grid points in kilometers at the
latitude defined in scale_lat.
>
> - lon_orient specifies the longitude value that's vertical in your
grid.
>
> - r_km is the assumed radius of a spherical earth.
>
> - nx and ny specify the size of the grid.
>
> I used the plot_data_plane tool to make a plot of the sample data
you sent, and it's attached to this message. Looks like MET thinks
your data is over eastern Europe. Is that correct?
>
> Now I'm not sure where you're getting the grid definition for this
data. But if, for example, you have a GRIB file, you may find the
following page of the MET online tutorial to be of use:
>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
>
> Hopefully that helps. Do you have additional questions?
>
> Thanks,
> John Halley Gotway
>
> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >
> >
> > Hello Paul,
> > I have been working in the verification of my fields. I have one
more question regarding this issue:
> > - i still can not locate correctly the fields on the polar
stereographic projection. I would like to know the description of the
projection attributes (scale_lat, lat_pin, lon_pin, x_pin, y_pin,
lon_orient, d_km, r_km). Based on these descriptions i can improve the
location of my fields.
> > I am sending attached the file I am using so you can take a look.
The field is the 1h accumulation of precipitation using the weather
radar.
> > Best Regards
> > Sebastian Paez
> >
> >
> >> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET
to verify data precipitation model from weather radar
> >> From: met_help at ucar.edu
> >> To: sebastian_paez at hotmail.com
> >> Date: Tue, 3 Jul 2012 08:09:02 -0600
> >>
> >> According to our records, your request has been resolved. If you
have any
> >> further questions or concerns, please respond to this message.
> >
> >
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET to verify data precipitation model from weather radar
From: John Halley Gotway
Time: Mon Jul 16 06:45:50 2012
Paul,
Let me address your second question about verifying against point
observations and looking at the results for individual stations.
You'll use the Point-Stat tool to compare your gridded forecasts to
point observations - I assume you already have point observations
available to use. Since the addition of the MPR line type, I think
the mask_sid option has become much less useful. Instead, I'd
suggest just defining the masking region as 'grid = "FULL";'. That
will use all observations that fall in your forecast region. And be
sure to turn *ON* the MPR output line type to get information
about each station. Also, in the "mask" dictionary, set 'method =
UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match each
observation to the nearest forecast point with no interpolation
applied to the forecast values.
When you run Point-Stat, your output will include the matched forecast
and observation values for each station that went into the computation
of statistics. After you verify your forecast for many
output times, you have the option of running the STAT-Analysis tool to
compute statistics over each individual station, rather than doing an
aerial average, as is done in Point-Stat.
Next, you would like to use a more high resolution background map in
the output of MODE and Wavelet-Stat. The answer is, yes, it is
possible, but is likely pretty difficult. The map data in MET is
contained in the METv4.0/data/map directory. The default background
map consists of the country boundaries plus the USA state boundaries
and is drawn in the draw_map() function in
"METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc". Theoretically,
you could get a lat/lon representation of the addition map data you
want to plot and format it to look like one of those existing
map data files. You could use your new file to overwrite the existing
"usa_state_data" file and that should have the desired effect. Or you
could create a new file name and modify the C++ code to
plot it.
But that process might be difficult. Instead, you could just use the
NetCDF output of the MODE and Wavelet-Stat tools, read them using
whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
few), and then create whatever plot you'd like. We intentionally did
not put a lot of effort into making those plots very configurable.
Instead, we want to keep MET focused on the verification
aspect while providing users with the data they'd need to make
whatever plots they'd like.
Hope that helps.
Thanks,
John
On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>
>
> Sorry, now the attachment
>
> From: sebastian_paez at hotmail.com
> To: met_help at ucar.edu
> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> Date: Sat, 14 Jul 2012 08:35:06 -0500
>
>
>
>
>
> Hello John,
> thanks for your answer, finally i could locate correctly the fields.
I am using radar data from the south of Germany.I am attaching the
corrected file so you can see it. One more question about this: Is it
possible to change the background image thatMODE and Wavelt produce?
Since my data is in a range of 290kmx290km i would like to have more
details on the back.
> Another question, regarding Point Stat, with the same data i want to
verify it to gauge stations' data. That means I want to verifygrid
forecasts and point observations and i want the results for each
station (without interpolation). I was looking for an example but I
could not find this particular case. Maybe you can send me a typical
file *.stations for "mask_sid", so i can see what do i need.
> Best RegardsSebastian Paez
>
>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
>> From: met_help at ucar.edu
>> To: sebastian_paez at hotmail.com
>> Date: Fri, 13 Jul 2012 12:36:21 -0600
>>
>> Hello Sebastian,
>>
>> Unfortunately, we don't have those parameters well documented
anywhere at this point, but I can give you a brief description of
them. These are the NetCDF global attributes used in MET to define a
>> polar stereographic grid. For reference, here's the values you
have set in the example NetCDF file you sent:
>> :Projection = "Polar Stereographic" ;
>> :hemisphere = "N" ;
>> :scale_lat = "60.000000 degrees_north" ;
>> :lat_pin = "47.340000" ;
>> :lon_pin = "7.980000" ;
>> :x_pin = "0.000000" ;
>> :y_pin = "0.000000" ;
>> :lon_orient = "8.000000" ;
>> :d_km = "10.395000" ;
>> :r_km = "6371.200000" ;
>> :nx = "290" ;
>> :ny = "290 grid_points" ;
>>
>> - x_pin and y_pin: specify the (x,y) location of the reference
lat/lon provided in (lat_pin, lon_pin). Yours are set to (0,0),
meaning that (lat_pin, lon_pin) refers to the lower-left corner of the
>> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that you're
specifying the lat/lon of the upper-right corner of the grid.
>>
>> - lat_pin and lon_pin are the lat/lon of the (x,y) specified above.
Usually, it's the lower-left corner.
>>
>> - scale_lat is the latitude at which the spacing of the grid
(defined in d_km) is defined.
>>
>> - d_km is the spacing between grid points in kilometers at the
latitude defined in scale_lat.
>>
>> - lon_orient specifies the longitude value that's vertical in your
grid.
>>
>> - r_km is the assumed radius of a spherical earth.
>>
>> - nx and ny specify the size of the grid.
>>
>> I used the plot_data_plane tool to make a plot of the sample data
you sent, and it's attached to this message. Looks like MET thinks
your data is over eastern Europe. Is that correct?
>>
>> Now I'm not sure where you're getting the grid definition for this
data. But if, for example, you have a GRIB file, you may find the
following page of the MET online tutorial to be of use:
>>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
>>
>> Hopefully that helps. Do you have additional questions?
>>
>> Thanks,
>> John Halley Gotway
>>
>> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>
>>>
>>> Hello Paul,
>>> I have been working in the verification of my fields. I have one
more question regarding this issue:
>>> - i still can not locate correctly the fields on the polar
stereographic projection. I would like to know the description of the
projection attributes (scale_lat, lat_pin, lon_pin, x_pin, y_pin,
lon_orient, d_km, r_km). Based on these descriptions i can improve the
location of my fields.
>>> I am sending attached the file I am using so you can take a look.
The field is the 1h accumulation of precipitation using the weather
radar.
>>> Best Regards
>>> Sebastian Paez
>>>
>>>
>>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET
to verify data precipitation model from weather radar
>>>> From: met_help at ucar.edu
>>>> To: sebastian_paez at hotmail.com
>>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
>>>>
>>>> According to our records, your request has been resolved. If you
have any
>>>> further questions or concerns, please respond to this message.
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: Help i try to use MET to verify data precipitation model from weather radar
From: Sebastian Paez
Time: Mon Jul 16 08:59:49 2012
Hello John,
I configured the MET files and I have all netcdf files I created (obs
and fcsts). Then, I executed an script with a for loop using python to
verify all netcdf files I have (3 months hourly accumulated data). The
verification for each hourly accmulated files (obervations and
forecasts) is executed but the problem I found is that the output
files are created with the same name and therefore they are replacing
them selves every time. That means, at the end of the execution of the
script I only have the last output file and not a output file for each
hour.
I read in the chapter10 of the MET User's Guide some info, but I still
I could solve this situation.
I have two ideas;
- first, based on the example Script 10-3 on the User's Guide, it is
possible to set environment variables when the script runs. So my idea
was to access the "output_prefix" variable from the configuration file
and then change that variable with a loop. That would mean that I will
give a different name for each output file. In order to do this I
would need you to tell the name of the variable of output_prefix in
the MET configuration file, so I can use it as a environment variable
in the execution script.
- second, set the time in the netcdf files, forecast lead time,
forecast valid time, etc, and then based on this, run the script. Here
I have two problems, i can not find in thevariable attributes the
options to set this time attributes; and also the information from the
user's guide about how it runs it is not clear to me.
Could you help me with this problem and recommend me a solution. I am
attaching here the files I am going to verify(observations and
forecasts), the execution script i am running is below.
import os
korr = 'rawdata'
for imgnbr in range(11665,11666,1): command =
'/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/grid_stat
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_fcst_%d.nc
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_obs_%d.nc
\/home/ubuntu/Desktop/Master_Thesis/METv3.1/data/config/GridStatConfig_test
\-outdir
/home/ubuntu/Desktop/Master_Thesis/Results/1h_accumulations/%s/ \-v 2'
%(korr,korr,imgnbr,korr,korr,imgnbr,korr) os.system(command)
Bast Regards
Sebastian
> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> From: met_help at ucar.edu
> To: sebastian_paez at hotmail.com
> Date: Mon, 16 Jul 2012 06:45:51 -0600
>
> Paul,
>
> Let me address your second question about verifying against point
observations and looking at the results for individual stations.
You'll use the Point-Stat tool to compare your gridded forecasts to
> point observations - I assume you already have point observations
available to use. Since the addition of the MPR line type, I think
the mask_sid option has become much less useful. Instead, I'd
> suggest just defining the masking region as 'grid = "FULL";'. That
will use all observations that fall in your forecast region. And be
sure to turn *ON* the MPR output line type to get information
> about each station. Also, in the "mask" dictionary, set 'method =
UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match each
observation to the nearest forecast point with no interpolation
> applied to the forecast values.
>
> When you run Point-Stat, your output will include the matched
forecast and observation values for each station that went into the
computation of statistics. After you verify your forecast for many
> output times, you have the option of running the STAT-Analysis tool
to compute statistics over each individual station, rather than doing
an aerial average, as is done in Point-Stat.
>
> Next, you would like to use a more high resolution background map in
the output of MODE and Wavelet-Stat. The answer is, yes, it is
possible, but is likely pretty difficult. The map data in MET is
> contained in the METv4.0/data/map directory. The default background
map consists of the country boundaries plus the USA state boundaries
and is drawn in the draw_map() function in
> "METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc". Theoretically,
you could get a lat/lon representation of the addition map data you
want to plot and format it to look like one of those existing
> map data files. You could use your new file to overwrite the
existing "usa_state_data" file and that should have the desired
effect. Or you could create a new file name and modify the C++ code
to
> plot it.
>
> But that process might be difficult. Instead, you could just use
the NetCDF output of the MODE and Wavelet-Stat tools, read them using
whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
> few), and then create whatever plot you'd like. We intentionally
did not put a lot of effort into making those plots very configurable.
Instead, we want to keep MET focused on the verification
> aspect while providing users with the data they'd need to make
whatever plots they'd like.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >
> >
> > Sorry, now the attachment
> >
> > From: sebastian_paez at hotmail.com
> > To: met_help at ucar.edu
> > Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> > Date: Sat, 14 Jul 2012 08:35:06 -0500
> >
> >
> >
> >
> >
> > Hello John,
> > thanks for your answer, finally i could locate correctly the
fields. I am using radar data from the south of Germany.I am attaching
the corrected file so you can see it. One more question about this: Is
it possible to change the background image thatMODE and Wavelt
produce? Since my data is in a range of 290kmx290km i would like to
have more details on the back.
> > Another question, regarding Point Stat, with the same data i want
to verify it to gauge stations' data. That means I want to verifygrid
forecasts and point observations and i want the results for each
station (without interpolation). I was looking for an example but I
could not find this particular case. Maybe you can send me a typical
file *.stations for "mask_sid", so i can see what do i need.
> > Best RegardsSebastian Paez
> >
> >> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> >> From: met_help at ucar.edu
> >> To: sebastian_paez at hotmail.com
> >> Date: Fri, 13 Jul 2012 12:36:21 -0600
> >>
> >> Hello Sebastian,
> >>
> >> Unfortunately, we don't have those parameters well documented
anywhere at this point, but I can give you a brief description of
them. These are the NetCDF global attributes used in MET to define a
> >> polar stereographic grid. For reference, here's the values you
have set in the example NetCDF file you sent:
> >> :Projection = "Polar Stereographic" ;
> >> :hemisphere = "N" ;
> >> :scale_lat = "60.000000 degrees_north" ;
> >> :lat_pin = "47.340000" ;
> >> :lon_pin = "7.980000" ;
> >> :x_pin = "0.000000" ;
> >> :y_pin = "0.000000" ;
> >> :lon_orient = "8.000000" ;
> >> :d_km = "10.395000" ;
> >> :r_km = "6371.200000" ;
> >> :nx = "290" ;
> >> :ny = "290 grid_points" ;
> >>
> >> - x_pin and y_pin: specify the (x,y) location of the reference
lat/lon provided in (lat_pin, lon_pin). Yours are set to (0,0),
meaning that (lat_pin, lon_pin) refers to the lower-left corner of the
> >> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that you're
specifying the lat/lon of the upper-right corner of the grid.
> >>
> >> - lat_pin and lon_pin are the lat/lon of the (x,y) specified
above. Usually, it's the lower-left corner.
> >>
> >> - scale_lat is the latitude at which the spacing of the grid
(defined in d_km) is defined.
> >>
> >> - d_km is the spacing between grid points in kilometers at the
latitude defined in scale_lat.
> >>
> >> - lon_orient specifies the longitude value that's vertical in
your grid.
> >>
> >> - r_km is the assumed radius of a spherical earth.
> >>
> >> - nx and ny specify the size of the grid.
> >>
> >> I used the plot_data_plane tool to make a plot of the sample data
you sent, and it's attached to this message. Looks like MET thinks
your data is over eastern Europe. Is that correct?
> >>
> >> Now I'm not sure where you're getting the grid definition for
this data. But if, for example, you have a GRIB file, you may find
the following page of the MET online tutorial to be of use:
> >>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
> >>
> >> Hopefully that helps. Do you have additional questions?
> >>
> >> Thanks,
> >> John Halley Gotway
> >>
> >> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >>>
> >>>
> >>> Hello Paul,
> >>> I have been working in the verification of my fields. I have one
more question regarding this issue:
> >>> - i still can not locate correctly the fields on the polar
stereographic projection. I would like to know the description of the
projection attributes (scale_lat, lat_pin, lon_pin, x_pin, y_pin,
lon_orient, d_km, r_km). Based on these descriptions i can improve the
location of my fields.
> >>> I am sending attached the file I am using so you can take a
look. The field is the 1h accumulation of precipitation using the
weather radar.
> >>> Best Regards
> >>> Sebastian Paez
> >>>
> >>>
> >>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> >>>> From: met_help at ucar.edu
> >>>> To: sebastian_paez at hotmail.com
> >>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
> >>>>
> >>>> According to our records, your request has been resolved. If
you have any
> >>>> further questions or concerns, please respond to this message.
> >>>
> >>>
> >>
> >>
> >
> >
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET to verify data precipitation model from weather radar
From: John Halley Gotway
Time: Mon Jul 16 09:28:22 2012
Sebastian,
My apologies for calling you "Paul" in the last email. It was too
early this morning when I wrote it!
First, I'm glad you are able to get Grid-Stat to run on your data.
And second, I'm glad to hear that you've been able to script the calls
to Grid-Stat to verify over a 3 month time period.
Regarding the use of environment variables, yes, I think that's a
great idea. If you're having trouble with the output file names not
being unique, using environment variables to set the
"output_prefix" value in the config file is a great way to fix the
problem. In the Grid-Stat config file, set:
output_prefix = "${FCST_TIME}"; (or whatever environment variable
name you want to use)
And be sure to set the FCST_TIME environment variable in each
iteration of the loop. Hopefully that'll make the output file names
unique.
However, I'm concerned about why they are not already unique in the
first place. Doing an "ncdump -h
r_m1_1h_clutter_advcorr_fcst_11675.nc" on the forecast file you sent
me reveals the following
variable attributes for timing information:
float pr(lat, lon) ;
pr:init_time = "20050807_000000" ;
pr:init_time_ut = 1123459200 ;
pr:valid_time = "20050808_000000" ;
pr:accum_time = "120000" ;
pr:accum_time_sec = 43200 ;
You should make sure that the timing information in these variable
attributes are correct. Based on the command you sent me, I'm
guessing you're actually working with 2008 data, but the times listed
there say 2005. So in the process of making these NetCDF files, you
need to make sure the timing info is correct:
init_time = "YYYYMMDD_HHMMSS" ; for the model
initialization time
init_time_ut = unix_time ; unix time for the
model initialization time
valid_time = "YYYYMMDD_HHMMSS" ; for the forecast
valid time
valid_time_ut = unix_time ; unix time for the
forecast valid time
accum_time = "HHMMSS" ; for the accumulation
time. If no accumulation, just set to 000000
accum_time_sec = seconds ; for the accumulation
time in second.
Setting this timing info correctly is very important. MET reads this
info and writes it to the meta data in the statistics output.
I suspect that once you've corrected the timing information in the
variable attributes you will no longer have the problem of files
overwriting eachother.
Hope that helps.
John
On 07/16/2012 08:59 AM, Sebastian Paez via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>
>
> Hello John,
> I configured the MET files and I have all netcdf files I created
(obs and fcsts). Then, I executed an script with a for loop using
python to verify all netcdf files I have (3 months hourly accumulated
data). The verification for each hourly accmulated files (obervations
and forecasts) is executed but the problem I found is that the output
files are created with the same name and therefore they are replacing
them selves every time. That means, at the end of the execution of the
script I only have the last output file and not a output file for each
hour.
> I read in the chapter10 of the MET User's Guide some info, but I
still I could solve this situation.
> I have two ideas;
> - first, based on the example Script 10-3 on the User's Guide, it is
possible to set environment variables when the script runs. So my idea
was to access the "output_prefix" variable from the configuration file
and then change that variable with a loop. That would mean that I will
give a different name for each output file. In order to do this I
would need you to tell the name of the variable of output_prefix in
the MET configuration file, so I can use it as a environment variable
in the execution script.
> - second, set the time in the netcdf files, forecast lead time,
forecast valid time, etc, and then based on this, run the script. Here
I have two problems, i can not find in thevariable attributes the
options to set this time attributes; and also the information from the
user's guide about how it runs it is not clear to me.
> Could you help me with this problem and recommend me a solution. I
am attaching here the files I am going to verify(observations and
forecasts), the execution script i am running is below.
>
> import os
> korr = 'rawdata'
> for imgnbr in range(11665,11666,1): command =
'/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/grid_stat
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_fcst_%d.nc
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_obs_%d.nc
\/home/ubuntu/Desktop/Master_Thesis/METv3.1/data/config/GridStatConfig_test
\-outdir
/home/ubuntu/Desktop/Master_Thesis/Results/1h_accumulations/%s/ \-v 2'
%(korr,korr,imgnbr,korr,korr,imgnbr,korr) os.system(command)
>
> Bast Regards
> Sebastian
>
>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
>> From: met_help at ucar.edu
>> To: sebastian_paez at hotmail.com
>> Date: Mon, 16 Jul 2012 06:45:51 -0600
>>
>> Paul,
>>
>> Let me address your second question about verifying against point
observations and looking at the results for individual stations.
You'll use the Point-Stat tool to compare your gridded forecasts to
>> point observations - I assume you already have point observations
available to use. Since the addition of the MPR line type, I think
the mask_sid option has become much less useful. Instead, I'd
>> suggest just defining the masking region as 'grid = "FULL";'. That
will use all observations that fall in your forecast region. And be
sure to turn *ON* the MPR output line type to get information
>> about each station. Also, in the "mask" dictionary, set 'method =
UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match each
observation to the nearest forecast point with no interpolation
>> applied to the forecast values.
>>
>> When you run Point-Stat, your output will include the matched
forecast and observation values for each station that went into the
computation of statistics. After you verify your forecast for many
>> output times, you have the option of running the STAT-Analysis tool
to compute statistics over each individual station, rather than doing
an aerial average, as is done in Point-Stat.
>>
>> Next, you would like to use a more high resolution background map
in the output of MODE and Wavelet-Stat. The answer is, yes, it is
possible, but is likely pretty difficult. The map data in MET is
>> contained in the METv4.0/data/map directory. The default
background map consists of the country boundaries plus the USA state
boundaries and is drawn in the draw_map() function in
>> "METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc". Theoretically,
you could get a lat/lon representation of the addition map data you
want to plot and format it to look like one of those existing
>> map data files. You could use your new file to overwrite the
existing "usa_state_data" file and that should have the desired
effect. Or you could create a new file name and modify the C++ code
to
>> plot it.
>>
>> But that process might be difficult. Instead, you could just use
the NetCDF output of the MODE and Wavelet-Stat tools, read them using
whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
>> few), and then create whatever plot you'd like. We intentionally
did not put a lot of effort into making those plots very configurable.
Instead, we want to keep MET focused on the verification
>> aspect while providing users with the data they'd need to make
whatever plots they'd like.
>>
>> Hope that helps.
>>
>> Thanks,
>> John
>>
>>
>> On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>
>>>
>>> Sorry, now the attachment
>>>
>>> From: sebastian_paez at hotmail.com
>>> To: met_help at ucar.edu
>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
>>> Date: Sat, 14 Jul 2012 08:35:06 -0500
>>>
>>>
>>>
>>>
>>>
>>> Hello John,
>>> thanks for your answer, finally i could locate correctly the
fields. I am using radar data from the south of Germany.I am attaching
the corrected file so you can see it. One more question about this: Is
it possible to change the background image thatMODE and Wavelt
produce? Since my data is in a range of 290kmx290km i would like to
have more details on the back.
>>> Another question, regarding Point Stat, with the same data i want
to verify it to gauge stations' data. That means I want to verifygrid
forecasts and point observations and i want the results for each
station (without interpolation). I was looking for an example but I
could not find this particular case. Maybe you can send me a typical
file *.stations for "mask_sid", so i can see what do i need.
>>> Best RegardsSebastian Paez
>>>
>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
>>>> From: met_help at ucar.edu
>>>> To: sebastian_paez at hotmail.com
>>>> Date: Fri, 13 Jul 2012 12:36:21 -0600
>>>>
>>>> Hello Sebastian,
>>>>
>>>> Unfortunately, we don't have those parameters well documented
anywhere at this point, but I can give you a brief description of
them. These are the NetCDF global attributes used in MET to define a
>>>> polar stereographic grid. For reference, here's the values you
have set in the example NetCDF file you sent:
>>>> :Projection = "Polar Stereographic" ;
>>>> :hemisphere = "N" ;
>>>> :scale_lat = "60.000000 degrees_north" ;
>>>> :lat_pin = "47.340000" ;
>>>> :lon_pin = "7.980000" ;
>>>> :x_pin = "0.000000" ;
>>>> :y_pin = "0.000000" ;
>>>> :lon_orient = "8.000000" ;
>>>> :d_km = "10.395000" ;
>>>> :r_km = "6371.200000" ;
>>>> :nx = "290" ;
>>>> :ny = "290 grid_points" ;
>>>>
>>>> - x_pin and y_pin: specify the (x,y) location of the reference
lat/lon provided in (lat_pin, lon_pin). Yours are set to (0,0),
meaning that (lat_pin, lon_pin) refers to the lower-left corner of the
>>>> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that you're
specifying the lat/lon of the upper-right corner of the grid.
>>>>
>>>> - lat_pin and lon_pin are the lat/lon of the (x,y) specified
above. Usually, it's the lower-left corner.
>>>>
>>>> - scale_lat is the latitude at which the spacing of the grid
(defined in d_km) is defined.
>>>>
>>>> - d_km is the spacing between grid points in kilometers at the
latitude defined in scale_lat.
>>>>
>>>> - lon_orient specifies the longitude value that's vertical in
your grid.
>>>>
>>>> - r_km is the assumed radius of a spherical earth.
>>>>
>>>> - nx and ny specify the size of the grid.
>>>>
>>>> I used the plot_data_plane tool to make a plot of the sample data
you sent, and it's attached to this message. Looks like MET thinks
your data is over eastern Europe. Is that correct?
>>>>
>>>> Now I'm not sure where you're getting the grid definition for
this data. But if, for example, you have a GRIB file, you may find
the following page of the MET online tutorial to be of use:
>>>>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
>>>>
>>>> Hopefully that helps. Do you have additional questions?
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>>
>>>> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>>>
>>>>>
>>>>> Hello Paul,
>>>>> I have been working in the verification of my fields. I have one
more question regarding this issue:
>>>>> - i still can not locate correctly the fields on the polar
stereographic projection. I would like to know the description of the
projection attributes (scale_lat, lat_pin, lon_pin, x_pin, y_pin,
lon_orient, d_km, r_km). Based on these descriptions i can improve the
location of my fields.
>>>>> I am sending attached the file I am using so you can take a
look. The field is the 1h accumulation of precipitation using the
weather radar.
>>>>> Best Regards
>>>>> Sebastian Paez
>>>>>
>>>>>
>>>>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
>>>>>> From: met_help at ucar.edu
>>>>>> To: sebastian_paez at hotmail.com
>>>>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
>>>>>>
>>>>>> According to our records, your request has been resolved. If
you have any
>>>>>> further questions or concerns, please respond to this message.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: Help i try to use MET to verify data precipitation model from weather radar
From: Sebastian Paez
Time: Mon Jul 16 10:53:01 2012
John,
thanks a lot for your assistance, it is really helpful. The idea of
the environment variables worked but as I want to used the Stat
Analysis Tool, I think it is better to fix the time parameters.
Regarding the time parameters, I am still not sure what to use:I have
data from precipitation hourly accumulations starting 2008-05-
01T01:50:00 until 2008-07-31T23:50:00 in [mm/h]Every hour for example:
2008-05-01T01:50:00, then 2008-05-01T02:50:00, 2008-05-
01T03:50:00..... until the end of JulySo, in the case of the
observation file:
> pr:init_time = "20080501_015000" ; time when the MET model starts?
> pr:init_time_ut = ; what is unix time of the model?
> pr:valid_time = "20080501_015000" ; observation valid time, when
starts?> pr:valid_time_ut = ; what is unix time for the observation?
> pr:accum_time = "000000" ; what is refering time accumulation?
> pr:accum_time_sec = 43200 ;
what is refering time accumulation?
Also I was wondering what is meant with "lead time" for MET and if
that parameter needs to be defined somewhere?
Regards
Sebastian
> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> From: met_help at ucar.edu
> To: sebastian_paez at hotmail.com
> Date: Mon, 16 Jul 2012 09:28:22 -0600
>
> Sebastian,
>
> My apologies for calling you "Paul" in the last email. It was too
early this morning when I wrote it!
>
> First, I'm glad you are able to get Grid-Stat to run on your data.
And second, I'm glad to hear that you've been able to script the calls
to Grid-Stat to verify over a 3 month time period.
>
> Regarding the use of environment variables, yes, I think that's a
great idea. If you're having trouble with the output file names not
being unique, using environment variables to set the
> "output_prefix" value in the config file is a great way to fix the
problem. In the Grid-Stat config file, set:
>
> output_prefix = "${FCST_TIME}"; (or whatever environment
variable name you want to use)
>
> And be sure to set the FCST_TIME environment variable in each
iteration of the loop. Hopefully that'll make the output file names
unique.
>
> However, I'm concerned about why they are not already unique in the
first place. Doing an "ncdump -h
r_m1_1h_clutter_advcorr_fcst_11675.nc" on the forecast file you sent
me reveals the following
> variable attributes for timing information:
> float pr(lat, lon) ;
> pr:init_time = "20050807_000000" ;
> pr:init_time_ut = 1123459200 ;
> pr:valid_time = "20050808_000000" ;
> pr:accum_time = "120000" ;
> pr:accum_time_sec = 43200 ;
>
> You should make sure that the timing information in these variable
attributes are correct. Based on the command you sent me, I'm
guessing you're actually working with 2008 data, but the times listed
> there say 2005. So in the process of making these NetCDF files, you
need to make sure the timing info is correct:
>
> init_time = "YYYYMMDD_HHMMSS" ; for the model
initialization time
> init_time_ut = unix_time ; unix time for the
model initialization time
> valid_time = "YYYYMMDD_HHMMSS" ; for the forecast
valid time
> valid_time_ut = unix_time ; unix time for the
forecast valid time
> accum_time = "HHMMSS" ; for the
accumulation time. If no accumulation, just set to 000000
> accum_time_sec = seconds ; for the
accumulation time in second.
>
> Setting this timing info correctly is very important. MET reads
this info and writes it to the meta data in the statistics output.
>
> I suspect that once you've corrected the timing information in the
variable attributes you will no longer have the problem of files
overwriting eachother.
>
> Hope that helps.
>
> John
>
> On 07/16/2012 08:59 AM, Sebastian Paez via RT wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >
> >
> > Hello John,
> > I configured the MET files and I have all netcdf files I created
(obs and fcsts). Then, I executed an script with a for loop using
python to verify all netcdf files I have (3 months hourly accumulated
data). The verification for each hourly accmulated files (obervations
and forecasts) is executed but the problem I found is that the output
files are created with the same name and therefore they are replacing
them selves every time. That means, at the end of the execution of the
script I only have the last output file and not a output file for each
hour.
> > I read in the chapter10 of the MET User's Guide some info, but I
still I could solve this situation.
> > I have two ideas;
> > - first, based on the example Script 10-3 on the User's Guide, it
is possible to set environment variables when the script runs. So my
idea was to access the "output_prefix" variable from the configuration
file and then change that variable with a loop. That would mean that I
will give a different name for each output file. In order to do this I
would need you to tell the name of the variable of output_prefix in
the MET configuration file, so I can use it as a environment variable
in the execution script.
> > - second, set the time in the netcdf files, forecast lead time,
forecast valid time, etc, and then based on this, run the script. Here
I have two problems, i can not find in thevariable attributes the
options to set this time attributes; and also the information from the
user's guide about how it runs it is not clear to me.
> > Could you help me with this problem and recommend me a solution. I
am attaching here the files I am going to verify(observations and
forecasts), the execution script i am running is below.
> >
> > import os
> > korr = 'rawdata'
> > for imgnbr in range(11665,11666,1): command =
'/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/grid_stat
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_fcst_%d.nc
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_obs_%d.nc
\/home/ubuntu/Desktop/Master_Thesis/METv3.1/data/config/GridStatConfig_test
\-outdir
/home/ubuntu/Desktop/Master_Thesis/Results/1h_accumulations/%s/ \-v 2'
%(korr,korr,imgnbr,korr,korr,imgnbr,korr) os.system(command)
> >
> > Bast Regards
> > Sebastian
> >
> >> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> >> From: met_help at ucar.edu
> >> To: sebastian_paez at hotmail.com
> >> Date: Mon, 16 Jul 2012 06:45:51 -0600
> >>
> >> Paul,
> >>
> >> Let me address your second question about verifying against point
observations and looking at the results for individual stations.
You'll use the Point-Stat tool to compare your gridded forecasts to
> >> point observations - I assume you already have point observations
available to use. Since the addition of the MPR line type, I think
the mask_sid option has become much less useful. Instead, I'd
> >> suggest just defining the masking region as 'grid = "FULL";'.
That will use all observations that fall in your forecast region. And
be sure to turn *ON* the MPR output line type to get information
> >> about each station. Also, in the "mask" dictionary, set 'method
= UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match each
observation to the nearest forecast point with no interpolation
> >> applied to the forecast values.
> >>
> >> When you run Point-Stat, your output will include the matched
forecast and observation values for each station that went into the
computation of statistics. After you verify your forecast for many
> >> output times, you have the option of running the STAT-Analysis
tool to compute statistics over each individual station, rather than
doing an aerial average, as is done in Point-Stat.
> >>
> >> Next, you would like to use a more high resolution background map
in the output of MODE and Wavelet-Stat. The answer is, yes, it is
possible, but is likely pretty difficult. The map data in MET is
> >> contained in the METv4.0/data/map directory. The default
background map consists of the country boundaries plus the USA state
boundaries and is drawn in the draw_map() function in
> >> "METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc".
Theoretically, you could get a lat/lon representation of the addition
map data you want to plot and format it to look like one of those
existing
> >> map data files. You could use your new file to overwrite the
existing "usa_state_data" file and that should have the desired
effect. Or you could create a new file name and modify the C++ code
to
> >> plot it.
> >>
> >> But that process might be difficult. Instead, you could just use
the NetCDF output of the MODE and Wavelet-Stat tools, read them using
whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
> >> few), and then create whatever plot you'd like. We intentionally
did not put a lot of effort into making those plots very configurable.
Instead, we want to keep MET focused on the verification
> >> aspect while providing users with the data they'd need to make
whatever plots they'd like.
> >>
> >> Hope that helps.
> >>
> >> Thanks,
> >> John
> >>
> >>
> >> On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >>>
> >>>
> >>> Sorry, now the attachment
> >>>
> >>> From: sebastian_paez at hotmail.com
> >>> To: met_help at ucar.edu
> >>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
> >>> Date: Sat, 14 Jul 2012 08:35:06 -0500
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Hello John,
> >>> thanks for your answer, finally i could locate correctly the
fields. I am using radar data from the south of Germany.I am attaching
the corrected file so you can see it. One more question about this: Is
it possible to change the background image thatMODE and Wavelt
produce? Since my data is in a range of 290kmx290km i would like to
have more details on the back.
> >>> Another question, regarding Point Stat, with the same data i
want to verify it to gauge stations' data. That means I want to
verifygrid forecasts and point observations and i want the results for
each station (without interpolation). I was looking for an example but
I could not find this particular case. Maybe you can send me a typical
file *.stations for "mask_sid", so i can see what do i need.
> >>> Best RegardsSebastian Paez
> >>>
> >>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
> >>>> From: met_help at ucar.edu
> >>>> To: sebastian_paez at hotmail.com
> >>>> Date: Fri, 13 Jul 2012 12:36:21 -0600
> >>>>
> >>>> Hello Sebastian,
> >>>>
> >>>> Unfortunately, we don't have those parameters well documented
anywhere at this point, but I can give you a brief description of
them. These are the NetCDF global attributes used in MET to define a
> >>>> polar stereographic grid. For reference, here's the values you
have set in the example NetCDF file you sent:
> >>>> :Projection = "Polar Stereographic" ;
> >>>> :hemisphere = "N" ;
> >>>> :scale_lat = "60.000000 degrees_north" ;
> >>>> :lat_pin = "47.340000" ;
> >>>> :lon_pin = "7.980000" ;
> >>>> :x_pin = "0.000000" ;
> >>>> :y_pin = "0.000000" ;
> >>>> :lon_orient = "8.000000" ;
> >>>> :d_km = "10.395000" ;
> >>>> :r_km = "6371.200000" ;
> >>>> :nx = "290" ;
> >>>> :ny = "290 grid_points" ;
> >>>>
> >>>> - x_pin and y_pin: specify the (x,y) location of the reference
lat/lon provided in (lat_pin, lon_pin). Yours are set to (0,0),
meaning that (lat_pin, lon_pin) refers to the lower-left corner of the
> >>>> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that you're
specifying the lat/lon of the upper-right corner of the grid.
> >>>>
> >>>> - lat_pin and lon_pin are the lat/lon of the (x,y) specified
above. Usually, it's the lower-left corner.
> >>>>
> >>>> - scale_lat is the latitude at which the spacing of the grid
(defined in d_km) is defined.
> >>>>
> >>>> - d_km is the spacing between grid points in kilometers at the
latitude defined in scale_lat.
> >>>>
> >>>> - lon_orient specifies the longitude value that's vertical in
your grid.
> >>>>
> >>>> - r_km is the assumed radius of a spherical earth.
> >>>>
> >>>> - nx and ny specify the size of the grid.
> >>>>
> >>>> I used the plot_data_plane tool to make a plot of the sample
data you sent, and it's attached to this message. Looks like MET
thinks your data is over eastern Europe. Is that correct?
> >>>>
> >>>> Now I'm not sure where you're getting the grid definition for
this data. But if, for example, you have a GRIB file, you may find
the following page of the MET online tutorial to be of use:
> >>>>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
> >>>>
> >>>> Hopefully that helps. Do you have additional questions?
> >>>>
> >>>> Thanks,
> >>>> John Halley Gotway
> >>>>
> >>>> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
> >>>>>
> >>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967
>
> >>>>>
> >>>>>
> >>>>> Hello Paul,
> >>>>> I have been working in the verification of my fields. I have
one more question regarding this issue:
> >>>>> - i still can not locate correctly the fields on the polar
stereographic projection. I would like to know the description of the
projection attributes (scale_lat, lat_pin, lon_pin, x_pin, y_pin,
lon_orient, d_km, r_km). Based on these descriptions i can improve the
location of my fields.
> >>>>> I am sending attached the file I am using so you can take a
look. The field is the 1h accumulation of precipitation using the
weather radar.
> >>>>> Best Regards
> >>>>> Sebastian Paez
> >>>>>
> >>>>>
> >>>>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> >>>>>> From: met_help at ucar.edu
> >>>>>> To: sebastian_paez at hotmail.com
> >>>>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
> >>>>>>
> >>>>>> According to our records, your request has been resolved. If
you have any
> >>>>>> further questions or concerns, please respond to this
message.
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >
> >
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET to verify data precipitation model from weather radar
From: John Halley Gotway
Time: Mon Jul 16 15:09:57 2012
Sebastian,
Unix time is the number of seconds that have elapsed since January 1,
1970. It's commonly used in software applications for storing time
since
it's a convenient way to represent date/time with a single number.
Here's
a wikipedia entry about it:
http://en.wikipedia.org/wiki/Unix_time
On your unix machine, try running the following command:
date +%s
That gives you the current unix time.
We typically use the date command to convert between unix time and
year/month/day/hour/minute/second. The following two date commands
might
be helpful for you.
(1) Convert from the date/time listed to unix time:
date -d'2008-03-15 00:00:00 UTC' +%s
(2) Convert from unix time listed to the to date/time format
specified:
date -ud '1970-01-01 00:00:00 UTC + 1205539200 seconds'
+'%Y%m%d_%H%M%S'
The init time is the model initialization time. For example, users
often
run their models every 6 hours, 12 hours, or 24 hours, and look at
forecast lead times out to a certain number of hours.
It sounds like you're verifying hourly model output. But is the model
run
once every hour? Or is it perhaps run once at 00Z and it generates 24
1-hourly output files? The initialization time is the time when the
model
was run.
Next, the valid time is the time when the model is valid. The valid
time
is just the initialization time plus the lead time. MET reads the
init
and valid times and computes the lead time as valid - init.
The accumulation time is the amount of time over which the variable is
accumulated. If you're dealing with 1-hourly accumulated
precipitation:
accum_time = "010000"; (Hour, Minute, Second)
accum_time_sec = 3600; (seconds)
Some people have multiple accumulations in their model output. For
example, at 12Z you may have precip accumulated over the last 3 hours,
6
hours, 12 hours, and 24 hours all contained within the same output
file.
You'll need to look at the process that's generating your NetCDF files
and
fix it to encode the timing variable attributes correctly. Similar
date/time reformatting probably exists in python if you're more
comfortable with that.
Hopefully that gets you going in the right direction.
Thanks,
John
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>
>
> John,
> thanks a lot for your assistance, it is really helpful. The idea of
the
> environment variables worked but as I want to used the Stat Analysis
Tool,
> I think it is better to fix the time parameters.
> Regarding the time parameters, I am still not sure what to use:I
have data
> from precipitation hourly accumulations starting 2008-05-01T01:50:00
until
> 2008-07-31T23:50:00 in [mm/h]Every hour for example: 2008-05-
01T01:50:00,
> then 2008-05-01T02:50:00, 2008-05-01T03:50:00..... until the end of
> JulySo, in the case of the observation file:
>> pr:init_time = "20080501_015000" ; time when the MET model starts?
>> pr:init_time_ut = ; what is unix time of the model?
>> pr:valid_time = "20080501_015000" ; observation valid time, when
>> starts?> pr:valid_time_ut = ; what is unix time for the
observation?
>> pr:accum_time = "000000" ; what is refering time accumulation?
>> pr:accum_time_sec = 43200 ;
> what is refering time accumulation?
>
> Also I was wondering what is meant with "lead time" for MET and if
that
> parameter needs to be defined somewhere?
> Regards
> Sebastian
>
>
>
>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to
>> verify data precipitation model from weather radar
>> From: met_help at ucar.edu
>> To: sebastian_paez at hotmail.com
>> Date: Mon, 16 Jul 2012 09:28:22 -0600
>>
>> Sebastian,
>>
>> My apologies for calling you "Paul" in the last email. It was too
early
>> this morning when I wrote it!
>>
>> First, I'm glad you are able to get Grid-Stat to run on your data.
And
>> second, I'm glad to hear that you've been able to script the calls
to
>> Grid-Stat to verify over a 3 month time period.
>>
>> Regarding the use of environment variables, yes, I think that's a
great
>> idea. If you're having trouble with the output file names not
being
>> unique, using environment variables to set the
>> "output_prefix" value in the config file is a great way to fix the
>> problem. In the Grid-Stat config file, set:
>>
>> output_prefix = "${FCST_TIME}"; (or whatever environment
variable
>> name you want to use)
>>
>> And be sure to set the FCST_TIME environment variable in each
iteration
>> of the loop. Hopefully that'll make the output file names unique.
>>
>> However, I'm concerned about why they are not already unique in the
>> first place. Doing an "ncdump -h
r_m1_1h_clutter_advcorr_fcst_11675.nc"
>> on the forecast file you sent me reveals the following
>> variable attributes for timing information:
>> float pr(lat, lon) ;
>> pr:init_time = "20050807_000000" ;
>> pr:init_time_ut = 1123459200 ;
>> pr:valid_time = "20050808_000000" ;
>> pr:accum_time = "120000" ;
>> pr:accum_time_sec = 43200 ;
>>
>> You should make sure that the timing information in these variable
>> attributes are correct. Based on the command you sent me, I'm
guessing
>> you're actually working with 2008 data, but the times listed
>> there say 2005. So in the process of making these NetCDF files,
you
>> need to make sure the timing info is correct:
>>
>> init_time = "YYYYMMDD_HHMMSS" ; for the model
>> initialization time
>> init_time_ut = unix_time ; unix time for the
>> model initialization time
>> valid_time = "YYYYMMDD_HHMMSS" ; for the forecast
valid
>> time
>> valid_time_ut = unix_time ; unix time for the
>> forecast valid time
>> accum_time = "HHMMSS" ; for the
accumulation
>> time. If no accumulation, just set to 000000
>> accum_time_sec = seconds ; for the
accumulation
>> time in second.
>>
>> Setting this timing info correctly is very important. MET reads
this
>> info and writes it to the meta data in the statistics output.
>>
>> I suspect that once you've corrected the timing information in the
>> variable attributes you will no longer have the problem of files
>> overwriting eachother.
>>
>> Hope that helps.
>>
>> John
>>
>> On 07/16/2012 08:59 AM, Sebastian Paez via RT wrote:
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>> >
>> >
>> > Hello John,
>> > I configured the MET files and I have all netcdf files I created
(obs
>> and fcsts). Then, I executed an script with a for loop using python
to
>> verify all netcdf files I have (3 months hourly accumulated data).
The
>> verification for each hourly accmulated files (obervations and
>> forecasts) is executed but the problem I found is that the output
>> files are created with the same name and therefore they are
replacing
>> them selves every time. That means, at the end of the execution of
the
>> script I only have the last output file and not a output file for
each
>> hour.
>> > I read in the chapter10 of the MET User's Guide some info, but I
still
>> I could solve this situation.
>> > I have two ideas;
>> > - first, based on the example Script 10-3 on the User's Guide, it
is
>> possible to set environment variables when the script runs. So my
idea
>> was to access the "output_prefix" variable from the configuration
file
>> and then change that variable with a loop. That would mean that I
will
>> give a different name for each output file. In order to do this I
>> would need you to tell the name of the variable of output_prefix in
>> the MET configuration file, so I can use it as a environment
variable
>> in the execution script.
>> > - second, set the time in the netcdf files, forecast lead time,
>> forecast valid time, etc, and then based on this, run the script.
Here
>> I have two problems, i can not find in thevariable attributes the
>> options to set this time attributes; and also the information from
the
>> user's guide about how it runs it is not clear to me.
>> > Could you help me with this problem and recommend me a solution.
I am
>> attaching here the files I am going to verify(observations and
>> forecasts), the execution script i am running is below.
>> >
>> > import os
>> > korr = 'rawdata'
>> > for imgnbr in range(11665,11666,1): command =
>> '/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/grid_stat
>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_fcst_%d.nc
>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_obs_%d.nc
>>
\/home/ubuntu/Desktop/Master_Thesis/METv3.1/data/config/GridStatConfig_test
>> \-outdir
>> /home/ubuntu/Desktop/Master_Thesis/Results/1h_accumulations/%s/ \-v
2'
>> %(korr,korr,imgnbr,korr,korr,imgnbr,korr) os.system(command)
>> >
>> > Bast Regards
>> > Sebastian
>> >
>> >> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET
>> to verify data precipitation model from weather radar
>> >> From: met_help at ucar.edu
>> >> To: sebastian_paez at hotmail.com
>> >> Date: Mon, 16 Jul 2012 06:45:51 -0600
>> >>
>> >> Paul,
>> >>
>> >> Let me address your second question about verifying against
point
>> observations and looking at the results for individual stations.
>> You'll use the Point-Stat tool to compare your gridded forecasts to
>> >> point observations - I assume you already have point
observations
>> available to use. Since the addition of the MPR line type, I think
>> the mask_sid option has become much less useful. Instead, I'd
>> >> suggest just defining the masking region as 'grid = "FULL";'.
That
>> will use all observations that fall in your forecast region. And
be
>> sure to turn *ON* the MPR output line type to get information
>> >> about each station. Also, in the "mask" dictionary, set 'method
=
>> UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match each
>> observation to the nearest forecast point with no interpolation
>> >> applied to the forecast values.
>> >>
>> >> When you run Point-Stat, your output will include the matched
>> forecast and observation values for each station that went into the
>> computation of statistics. After you verify your forecast for many
>> >> output times, you have the option of running the STAT-Analysis
tool
>> to compute statistics over each individual station, rather than
doing
>> an aerial average, as is done in Point-Stat.
>> >>
>> >> Next, you would like to use a more high resolution background
map in
>> the output of MODE and Wavelet-Stat. The answer is, yes, it is
>> possible, but is likely pretty difficult. The map data in MET is
>> >> contained in the METv4.0/data/map directory. The default
background
>> map consists of the country boundaries plus the USA state
boundaries
>> and is drawn in the draw_map() function in
>> >> "METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc".
Theoretically,
>> you could get a lat/lon representation of the addition map data you
>> want to plot and format it to look like one of those existing
>> >> map data files. You could use your new file to overwrite the
>> existing "usa_state_data" file and that should have the desired
>> effect. Or you could create a new file name and modify the C++
code
>> to
>> >> plot it.
>> >>
>> >> But that process might be difficult. Instead, you could just
use the
>> NetCDF output of the MODE and Wavelet-Stat tools, read them using
>> whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
>> >> few), and then create whatever plot you'd like. We
intentionally did
>> not put a lot of effort into making those plots very configurable.
>> Instead, we want to keep MET focused on the verification
>> >> aspect while providing users with the data they'd need to make
>> whatever plots they'd like.
>> >>
>> >> Hope that helps.
>> >>
>> >> Thanks,
>> >> John
>> >>
>> >>
>> >> On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
>> >>>
>> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>> >>>
>> >>>
>> >>> Sorry, now the attachment
>> >>>
>> >>> From: sebastian_paez at hotmail.com
>> >>> To: met_help at ucar.edu
>> >>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use
>> MET to verify data precipitation model from weather radar
>> >>> Date: Sat, 14 Jul 2012 08:35:06 -0500
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Hello John,
>> >>> thanks for your answer, finally i could locate correctly the
fields.
>> I am using radar data from the south of Germany.I am attaching the
>> corrected file so you can see it. One more question about this: Is
>> it possible to change the background image thatMODE and Wavelt
>> produce? Since my data is in a range of 290kmx290km i would like to
>> have more details on the back.
>> >>> Another question, regarding Point Stat, with the same data i
want to
>> verify it to gauge stations' data. That means I want to verifygrid
>> forecasts and point observations and i want the results for each
>> station (without interpolation). I was looking for an example but I
>> could not find this particular case. Maybe you can send me a
typical
>> file *.stations for "mask_sid", so i can see what do i need.
>> >>> Best RegardsSebastian Paez
>> >>>
>> >>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use
>> MET to verify data precipitation model from weather radar
>> >>>> From: met_help at ucar.edu
>> >>>> To: sebastian_paez at hotmail.com
>> >>>> Date: Fri, 13 Jul 2012 12:36:21 -0600
>> >>>>
>> >>>> Hello Sebastian,
>> >>>>
>> >>>> Unfortunately, we don't have those parameters well documented
>> anywhere at this point, but I can give you a brief description of
>> them. These are the NetCDF global attributes used in MET to define
>> a
>> >>>> polar stereographic grid. For reference, here's the values
you
>> have set in the example NetCDF file you sent:
>> >>>> :Projection = "Polar Stereographic" ;
>> >>>> :hemisphere = "N" ;
>> >>>> :scale_lat = "60.000000 degrees_north" ;
>> >>>> :lat_pin = "47.340000" ;
>> >>>> :lon_pin = "7.980000" ;
>> >>>> :x_pin = "0.000000" ;
>> >>>> :y_pin = "0.000000" ;
>> >>>> :lon_orient = "8.000000" ;
>> >>>> :d_km = "10.395000" ;
>> >>>> :r_km = "6371.200000" ;
>> >>>> :nx = "290" ;
>> >>>> :ny = "290 grid_points" ;
>> >>>>
>> >>>> - x_pin and y_pin: specify the (x,y) location of the reference
>> lat/lon provided in (lat_pin, lon_pin). Yours are set to (0,0),
>> meaning that (lat_pin, lon_pin) refers to the lower-left corner of
>> the
>> >>>> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that
you're
>> specifying the lat/lon of the upper-right corner of the grid.
>> >>>>
>> >>>> - lat_pin and lon_pin are the lat/lon of the (x,y) specified
above.
>> Usually, it's the lower-left corner.
>> >>>>
>> >>>> - scale_lat is the latitude at which the spacing of the grid
>> (defined in d_km) is defined.
>> >>>>
>> >>>> - d_km is the spacing between grid points in kilometers at the
>> latitude defined in scale_lat.
>> >>>>
>> >>>> - lon_orient specifies the longitude value that's vertical in
your
>> grid.
>> >>>>
>> >>>> - r_km is the assumed radius of a spherical earth.
>> >>>>
>> >>>> - nx and ny specify the size of the grid.
>> >>>>
>> >>>> I used the plot_data_plane tool to make a plot of the sample
data
>> you sent, and it's attached to this message. Looks like MET thinks
>> your data is over eastern Europe. Is that correct?
>> >>>>
>> >>>> Now I'm not sure where you're getting the grid definition for
this
>> data. But if, for example, you have a GRIB file, you may find the
>> following page of the MET online tutorial to be of use:
>> >>>>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
>> >>>>
>> >>>> Hopefully that helps. Do you have additional questions?
>> >>>>
>> >>>> Thanks,
>> >>>> John Halley Gotway
>> >>>>
>> >>>> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
>> >>>>>
>> >>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967
>
>> >>>>>
>> >>>>>
>> >>>>> Hello Paul,
>> >>>>> I have been working in the verification of my fields. I have
one
>> more question regarding this issue:
>> >>>>> - i still can not locate correctly the fields on the polar
>> stereographic projection. I would like to know the description of
>> the projection attributes (scale_lat, lat_pin, lon_pin, x_pin,
>> y_pin, lon_orient, d_km, r_km). Based on these descriptions i can
>> improve the location of my fields.
>> >>>>> I am sending attached the file I am using so you can take a
look.
>> The field is the 1h accumulation of precipitation using the
>> weather radar.
>> >>>>> Best Regards
>> >>>>> Sebastian Paez
>> >>>>>
>> >>>>>
>> >>>>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET
>> to verify data precipitation model from weather radar
>> >>>>>> From: met_help at ucar.edu
>> >>>>>> To: sebastian_paez at hotmail.com
>> >>>>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
>> >>>>>>
>> >>>>>> According to our records, your request has been resolved. If
you
>> have any
>> >>>>>> further questions or concerns, please respond to this
message.
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >
>>
>>
>
>
------------------------------------------------
Subject: Help i try to use MET to verify data precipitation model from weather radar
From: Sebastian Paez
Time: Wed Jul 18 14:47:13 2012
John,
This is my third attempt. This time i am not including from Grid_Stat
results. I hope it works now.
Best Regards
Sebastian
From: sebastian_paez at hotmail.com
To: met_help at ucar.edu
Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET
to verify data precipitation model from weather radar
Date: Wed, 18 Jul 2012 11:16:08 -0500
John,
I got an error about this e-mail I sent you in the morning. I hope you
get it know.
Regards
Sebastian
From: sebastian_paez at hotmail.com
To: met_help at ucar.edu
Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET
to verify data precipitation model from weather radar
Date: Wed, 18 Jul 2012 05:14:50 -0500
John,
I changed the time attributes generated 3 .nc files. I used them in
Grid_Stat. Based on grid_stat i tried to use Stat_Analysis, but when
trying to get the "-job summary for CTS and CSI", I got an error: "no
valid data found in STAT lines". Please see image attached.I am also
attaching .nc files created and output files from Grid_Stat.
I would like to give you some info about my data. I created
precipitation forecasts for 1 hour based on weather radar images from
South Germany.The forecasts are accumulated every 15min, 0min-15min,
15min-30min, 30min-45min, 45min-60min. Nevertheless, in this 3 files
(13000,13001,13002), that I am attaching I am using accumulation of 1h
only as a test for the MET package.
It would be very helpful, If you can tell me where is the source of
the error in order to use Stat_Analysis.
Your assistance is highly appreciated.
Regards
Sebastian
> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> From: met_help at ucar.edu
> To: sebastian_paez at hotmail.com
> Date: Mon, 16 Jul 2012 15:09:57 -0600
>
> Sebastian,
>
> Unix time is the number of seconds that have elapsed since January
1,
> 1970. It's commonly used in software applications for storing time
since
> it's a convenient way to represent date/time with a single number.
Here's
> a wikipedia entry about it:
> http://en.wikipedia.org/wiki/Unix_time
>
> On your unix machine, try running the following command:
> date +%s
> That gives you the current unix time.
>
> We typically use the date command to convert between unix time and
> year/month/day/hour/minute/second. The following two date commands
might
> be helpful for you.
>
> (1) Convert from the date/time listed to unix time:
> date -d'2008-03-15 00:00:00 UTC' +%s
>
> (2) Convert from unix time listed to the to date/time format
specified:
> date -ud '1970-01-01 00:00:00 UTC + 1205539200 seconds'
+'%Y%m%d_%H%M%S'
>
> The init time is the model initialization time. For example, users
often
> run their models every 6 hours, 12 hours, or 24 hours, and look at
> forecast lead times out to a certain number of hours.
>
> It sounds like you're verifying hourly model output. But is the
model run
> once every hour? Or is it perhaps run once at 00Z and it generates
24
> 1-hourly output files? The initialization time is the time when the
model
> was run.
>
> Next, the valid time is the time when the model is valid. The valid
time
> is just the initialization time plus the lead time. MET reads the
init
> and valid times and computes the lead time as valid - init.
>
> The accumulation time is the amount of time over which the variable
is
> accumulated. If you're dealing with 1-hourly accumulated
precipitation:
> accum_time = "010000"; (Hour, Minute, Second)
> accum_time_sec = 3600; (seconds)
>
> Some people have multiple accumulations in their model output. For
> example, at 12Z you may have precip accumulated over the last 3
hours, 6
> hours, 12 hours, and 24 hours all contained within the same output
file.
>
> You'll need to look at the process that's generating your NetCDF
files and
> fix it to encode the timing variable attributes correctly. Similar
> date/time reformatting probably exists in python if you're more
> comfortable with that.
>
> Hopefully that gets you going in the right direction.
>
> Thanks,
> John
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >
> >
> > John,
> > thanks a lot for your assistance, it is really helpful. The idea
of the
> > environment variables worked but as I want to used the Stat
Analysis Tool,
> > I think it is better to fix the time parameters.
> > Regarding the time parameters, I am still not sure what to use:I
have data
> > from precipitation hourly accumulations starting 2008-05-
01T01:50:00 until
> > 2008-07-31T23:50:00 in [mm/h]Every hour for example: 2008-05-
01T01:50:00,
> > then 2008-05-01T02:50:00, 2008-05-01T03:50:00..... until the end
of
> > JulySo, in the case of the observation file:
> >> pr:init_time = "20080501_015000" ; time when the MET model
starts?
> >> pr:init_time_ut = ; what is unix time of the model?
> >> pr:valid_time = "20080501_015000" ; observation valid time, when
> >> starts?> pr:valid_time_ut = ; what is unix time for the
observation?
> >> pr:accum_time = "000000" ; what is refering time accumulation?
> >> pr:accum_time_sec = 43200 ;
> > what is refering time accumulation?
> >
> > Also I was wondering what is meant with "lead time" for MET and if
that
> > parameter needs to be defined somewhere?
> > Regards
> > Sebastian
> >
> >
> >
> >> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to
> >> verify data precipitation model from weather radar
> >> From: met_help at ucar.edu
> >> To: sebastian_paez at hotmail.com
> >> Date: Mon, 16 Jul 2012 09:28:22 -0600
> >>
> >> Sebastian,
> >>
> >> My apologies for calling you "Paul" in the last email. It was
too early
> >> this morning when I wrote it!
> >>
> >> First, I'm glad you are able to get Grid-Stat to run on your
data. And
> >> second, I'm glad to hear that you've been able to script the
calls to
> >> Grid-Stat to verify over a 3 month time period.
> >>
> >> Regarding the use of environment variables, yes, I think that's a
great
> >> idea. If you're having trouble with the output file names not
being
> >> unique, using environment variables to set the
> >> "output_prefix" value in the config file is a great way to fix
the
> >> problem. In the Grid-Stat config file, set:
> >>
> >> output_prefix = "${FCST_TIME}"; (or whatever environment
variable
> >> name you want to use)
> >>
> >> And be sure to set the FCST_TIME environment variable in each
iteration
> >> of the loop. Hopefully that'll make the output file names
unique.
> >>
> >> However, I'm concerned about why they are not already unique in
the
> >> first place. Doing an "ncdump -h
r_m1_1h_clutter_advcorr_fcst_11675.nc"
> >> on the forecast file you sent me reveals the following
> >> variable attributes for timing information:
> >> float pr(lat, lon) ;
> >> pr:init_time = "20050807_000000" ;
> >> pr:init_time_ut = 1123459200 ;
> >> pr:valid_time = "20050808_000000" ;
> >> pr:accum_time = "120000" ;
> >> pr:accum_time_sec = 43200 ;
> >>
> >> You should make sure that the timing information in these
variable
> >> attributes are correct. Based on the command you sent me, I'm
guessing
> >> you're actually working with 2008 data, but the times listed
> >> there say 2005. So in the process of making these NetCDF files,
you
> >> need to make sure the timing info is correct:
> >>
> >> init_time = "YYYYMMDD_HHMMSS" ; for the model
> >> initialization time
> >> init_time_ut = unix_time ; unix time for
the
> >> model initialization time
> >> valid_time = "YYYYMMDD_HHMMSS" ; for the
forecast valid
> >> time
> >> valid_time_ut = unix_time ; unix time for
the
> >> forecast valid time
> >> accum_time = "HHMMSS" ; for the
accumulation
> >> time. If no accumulation, just set to 000000
> >> accum_time_sec = seconds ; for the
accumulation
> >> time in second.
> >>
> >> Setting this timing info correctly is very important. MET reads
this
> >> info and writes it to the meta data in the statistics output.
> >>
> >> I suspect that once you've corrected the timing information in
the
> >> variable attributes you will no longer have the problem of files
> >> overwriting eachother.
> >>
> >> Hope that helps.
> >>
> >> John
> >>
> >> On 07/16/2012 08:59 AM, Sebastian Paez via RT wrote:
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >> >
> >> >
> >> > Hello John,
> >> > I configured the MET files and I have all netcdf files I
created (obs
> >> and fcsts). Then, I executed an script with a for loop using
python to
> >> verify all netcdf files I have (3 months hourly accumulated
data). The
> >> verification for each hourly accmulated files (obervations and
> >> forecasts) is executed but the problem I found is that the output
> >> files are created with the same name and therefore they are
replacing
> >> them selves every time. That means, at the end of the execution
of the
> >> script I only have the last output file and not a output file for
each
> >> hour.
> >> > I read in the chapter10 of the MET User's Guide some info, but
I still
> >> I could solve this situation.
> >> > I have two ideas;
> >> > - first, based on the example Script 10-3 on the User's Guide,
it is
> >> possible to set environment variables when the script runs. So my
idea
> >> was to access the "output_prefix" variable from the configuration
file
> >> and then change that variable with a loop. That would mean that I
will
> >> give a different name for each output file. In order to do this I
> >> would need you to tell the name of the variable of output_prefix
in
> >> the MET configuration file, so I can use it as a environment
variable
> >> in the execution script.
> >> > - second, set the time in the netcdf files, forecast lead time,
> >> forecast valid time, etc, and then based on this, run the script.
Here
> >> I have two problems, i can not find in thevariable attributes the
> >> options to set this time attributes; and also the information
from the
> >> user's guide about how it runs it is not clear to me.
> >> > Could you help me with this problem and recommend me a
solution. I am
> >> attaching here the files I am going to verify(observations and
> >> forecasts), the execution script i am running is below.
> >> >
> >> > import os
> >> > korr = 'rawdata'
> >> > for imgnbr in range(11665,11666,1): command =
> >> '/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/grid_stat
> >>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_fcst_%d.nc
> >>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_obs_%d.nc
> >>
\/home/ubuntu/Desktop/Master_Thesis/METv3.1/data/config/GridStatConfig_test
> >> \-outdir
> >> /home/ubuntu/Desktop/Master_Thesis/Results/1h_accumulations/%s/
\-v 2'
> >> %(korr,korr,imgnbr,korr,korr,imgnbr,korr) os.system(command)
> >> >
> >> > Bast Regards
> >> > Sebastian
> >> >
> >> >> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET
> >> to verify data precipitation model from weather radar
> >> >> From: met_help at ucar.edu
> >> >> To: sebastian_paez at hotmail.com
> >> >> Date: Mon, 16 Jul 2012 06:45:51 -0600
> >> >>
> >> >> Paul,
> >> >>
> >> >> Let me address your second question about verifying against
point
> >> observations and looking at the results for individual stations.
> >> You'll use the Point-Stat tool to compare your gridded forecasts
to
> >> >> point observations - I assume you already have point
observations
> >> available to use. Since the addition of the MPR line type, I
think
> >> the mask_sid option has become much less useful. Instead, I'd
> >> >> suggest just defining the masking region as 'grid = "FULL";'.
That
> >> will use all observations that fall in your forecast region. And
be
> >> sure to turn *ON* the MPR output line type to get information
> >> >> about each station. Also, in the "mask" dictionary, set
'method =
> >> UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match each
> >> observation to the nearest forecast point with no interpolation
> >> >> applied to the forecast values.
> >> >>
> >> >> When you run Point-Stat, your output will include the matched
> >> forecast and observation values for each station that went into
the
> >> computation of statistics. After you verify your forecast for
many
> >> >> output times, you have the option of running the STAT-Analysis
tool
> >> to compute statistics over each individual station, rather than
doing
> >> an aerial average, as is done in Point-Stat.
> >> >>
> >> >> Next, you would like to use a more high resolution background
map in
> >> the output of MODE and Wavelet-Stat. The answer is, yes, it is
> >> possible, but is likely pretty difficult. The map data in MET is
> >> >> contained in the METv4.0/data/map directory. The default
background
> >> map consists of the country boundaries plus the USA state
boundaries
> >> and is drawn in the draw_map() function in
> >> >> "METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc".
Theoretically,
> >> you could get a lat/lon representation of the addition map data
you
> >> want to plot and format it to look like one of those existing
> >> >> map data files. You could use your new file to overwrite the
> >> existing "usa_state_data" file and that should have the desired
> >> effect. Or you could create a new file name and modify the C++
code
> >> to
> >> >> plot it.
> >> >>
> >> >> But that process might be difficult. Instead, you could just
use the
> >> NetCDF output of the MODE and Wavelet-Stat tools, read them using
> >> whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
> >> >> few), and then create whatever plot you'd like. We
intentionally did
> >> not put a lot of effort into making those plots very
configurable.
> >> Instead, we want to keep MET focused on the verification
> >> >> aspect while providing users with the data they'd need to make
> >> whatever plots they'd like.
> >> >>
> >> >> Hope that helps.
> >> >>
> >> >> Thanks,
> >> >> John
> >> >>
> >> >>
> >> >> On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
> >> >>>
> >> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967
>
> >> >>>
> >> >>>
> >> >>> Sorry, now the attachment
> >> >>>
> >> >>> From: sebastian_paez at hotmail.com
> >> >>> To: met_help at ucar.edu
> >> >>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use
> >> MET to verify data precipitation model from weather radar
> >> >>> Date: Sat, 14 Jul 2012 08:35:06 -0500
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> Hello John,
> >> >>> thanks for your answer, finally i could locate correctly the
fields.
> >> I am using radar data from the south of Germany.I am attaching
the
> >> corrected file so you can see it. One more question about this:
Is
> >> it possible to change the background image thatMODE and Wavelt
> >> produce? Since my data is in a range of 290kmx290km i would like
to
> >> have more details on the back.
> >> >>> Another question, regarding Point Stat, with the same data i
want to
> >> verify it to gauge stations' data. That means I want to
verifygrid
> >> forecasts and point observations and i want the results for each
> >> station (without interpolation). I was looking for an example but
I
> >> could not find this particular case. Maybe you can send me a
typical
> >> file *.stations for "mask_sid", so i can see what do i need.
> >> >>> Best RegardsSebastian Paez
> >> >>>
> >> >>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use
> >> MET to verify data precipitation model from weather radar
> >> >>>> From: met_help at ucar.edu
> >> >>>> To: sebastian_paez at hotmail.com
> >> >>>> Date: Fri, 13 Jul 2012 12:36:21 -0600
> >> >>>>
> >> >>>> Hello Sebastian,
> >> >>>>
> >> >>>> Unfortunately, we don't have those parameters well
documented
> >> anywhere at this point, but I can give you a brief description of
> >> them. These are the NetCDF global attributes used in MET to
define
> >> a
> >> >>>> polar stereographic grid. For reference, here's the values
you
> >> have set in the example NetCDF file you sent:
> >> >>>> :Projection = "Polar Stereographic" ;
> >> >>>> :hemisphere = "N" ;
> >> >>>> :scale_lat = "60.000000 degrees_north" ;
> >> >>>> :lat_pin = "47.340000" ;
> >> >>>> :lon_pin = "7.980000" ;
> >> >>>> :x_pin = "0.000000" ;
> >> >>>> :y_pin = "0.000000" ;
> >> >>>> :lon_orient = "8.000000" ;
> >> >>>> :d_km = "10.395000" ;
> >> >>>> :r_km = "6371.200000" ;
> >> >>>> :nx = "290" ;
> >> >>>> :ny = "290 grid_points" ;
> >> >>>>
> >> >>>> - x_pin and y_pin: specify the (x,y) location of the
reference
> >> lat/lon provided in (lat_pin, lon_pin). Yours are set to (0,0),
> >> meaning that (lat_pin, lon_pin) refers to the lower-left corner
of
> >> the
> >> >>>> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that
you're
> >> specifying the lat/lon of the upper-right corner of the grid.
> >> >>>>
> >> >>>> - lat_pin and lon_pin are the lat/lon of the (x,y) specified
above.
> >> Usually, it's the lower-left corner.
> >> >>>>
> >> >>>> - scale_lat is the latitude at which the spacing of the grid
> >> (defined in d_km) is defined.
> >> >>>>
> >> >>>> - d_km is the spacing between grid points in kilometers at
the
> >> latitude defined in scale_lat.
> >> >>>>
> >> >>>> - lon_orient specifies the longitude value that's vertical
in your
> >> grid.
> >> >>>>
> >> >>>> - r_km is the assumed radius of a spherical earth.
> >> >>>>
> >> >>>> - nx and ny specify the size of the grid.
> >> >>>>
> >> >>>> I used the plot_data_plane tool to make a plot of the sample
data
> >> you sent, and it's attached to this message. Looks like MET
thinks
> >> your data is over eastern Europe. Is that correct?
> >> >>>>
> >> >>>> Now I'm not sure where you're getting the grid definition
for this
> >> data. But if, for example, you have a GRIB file, you may find
the
> >> following page of the MET online tutorial to be of use:
> >> >>>>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
> >> >>>>
> >> >>>> Hopefully that helps. Do you have additional questions?
> >> >>>>
> >> >>>> Thanks,
> >> >>>> John Halley Gotway
> >> >>>>
> >> >>>> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
> >> >>>>>
> >> >>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >> >>>>>
> >> >>>>>
> >> >>>>> Hello Paul,
> >> >>>>> I have been working in the verification of my fields. I
have one
> >> more question regarding this issue:
> >> >>>>> - i still can not locate correctly the fields on the polar
> >> stereographic projection. I would like to know the description of
> >> the projection attributes (scale_lat, lat_pin, lon_pin, x_pin,
> >> y_pin, lon_orient, d_km, r_km). Based on these descriptions i can
> >> improve the location of my fields.
> >> >>>>> I am sending attached the file I am using so you can take a
look.
> >> The field is the 1h accumulation of precipitation using the
> >> weather radar.
> >> >>>>> Best Regards
> >> >>>>> Sebastian Paez
> >> >>>>>
> >> >>>>>
> >> >>>>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET
> >> to verify data precipitation model from weather radar
> >> >>>>>> From: met_help at ucar.edu
> >> >>>>>> To: sebastian_paez at hotmail.com
> >> >>>>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
> >> >>>>>>
> >> >>>>>> According to our records, your request has been resolved.
If you
> >> have any
> >> >>>>>> further questions or concerns, please respond to this
message.
> >> >>>>>
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >
> >> >
> >>
> >>
> >
> >
>
>
>
------------------------------------------------
Subject: Re: FW: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET to verify data precipitation model from weather radar
From: John Halley Gotway
Time: Thu Jul 19 13:11:12 2012
Sebastian,
I received your message without the Grid-Stat results you were trying
to send. If you'd like to send me data, rather than attaching it to
an email message, please post it to our anonymous ftp site
using the following instructions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
I would be happy to take a look at your Grid-Stat output and the STAT-
Analysis job you're running to let you know why you're not getting any
results.
Please write me back once you've posted the output of Grid-Stat you're
using to our FTP site. Also, please send me the exactly STAT-Analysis
job you're trying to run. I'm happy to take a look.
Thanks,
John Halley Gotway
On 07/18/2012 02:47 PM, Sebastian Paez via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>
>
> John,
> This is my third attempt. This time i am not including from
Grid_Stat results. I hope it works now.
> Best Regards
> Sebastian
>
> From: sebastian_paez at hotmail.com
> To: met_help at ucar.edu
> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> Date: Wed, 18 Jul 2012 11:16:08 -0500
>
>
>
>
>
> John,
> I got an error about this e-mail I sent you in the morning. I hope
you get it know.
> Regards
> Sebastian
>
> From: sebastian_paez at hotmail.com
> To: met_help at ucar.edu
> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> Date: Wed, 18 Jul 2012 05:14:50 -0500
>
>
>
>
>
> John,
> I changed the time attributes generated 3 .nc files. I used them in
Grid_Stat. Based on grid_stat i tried to use Stat_Analysis, but when
trying to get the "-job summary for CTS and CSI", I got an error: "no
valid data found in STAT lines". Please see image attached.I am also
attaching .nc files created and output files from Grid_Stat.
> I would like to give you some info about my data. I created
precipitation forecasts for 1 hour based on weather radar images from
South Germany.The forecasts are accumulated every 15min, 0min-15min,
15min-30min, 30min-45min, 45min-60min. Nevertheless, in this 3 files
(13000,13001,13002), that I am attaching I am using accumulation of 1h
only as a test for the MET package.
>
> It would be very helpful, If you can tell me where is the source of
the error in order to use Stat_Analysis.
> Your assistance is highly appreciated.
> Regards
> Sebastian
>
>
>
>
>
>
>
>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
>> From: met_help at ucar.edu
>> To: sebastian_paez at hotmail.com
>> Date: Mon, 16 Jul 2012 15:09:57 -0600
>>
>> Sebastian,
>>
>> Unix time is the number of seconds that have elapsed since January
1,
>> 1970. It's commonly used in software applications for storing time
since
>> it's a convenient way to represent date/time with a single number.
Here's
>> a wikipedia entry about it:
>> http://en.wikipedia.org/wiki/Unix_time
>>
>> On your unix machine, try running the following command:
>> date +%s
>> That gives you the current unix time.
>>
>> We typically use the date command to convert between unix time and
>> year/month/day/hour/minute/second. The following two date commands
might
>> be helpful for you.
>>
>> (1) Convert from the date/time listed to unix time:
>> date -d'2008-03-15 00:00:00 UTC' +%s
>>
>> (2) Convert from unix time listed to the to date/time format
specified:
>> date -ud '1970-01-01 00:00:00 UTC + 1205539200 seconds'
+'%Y%m%d_%H%M%S'
>>
>> The init time is the model initialization time. For example, users
often
>> run their models every 6 hours, 12 hours, or 24 hours, and look at
>> forecast lead times out to a certain number of hours.
>>
>> It sounds like you're verifying hourly model output. But is the
model run
>> once every hour? Or is it perhaps run once at 00Z and it generates
24
>> 1-hourly output files? The initialization time is the time when
the model
>> was run.
>>
>> Next, the valid time is the time when the model is valid. The
valid time
>> is just the initialization time plus the lead time. MET reads the
init
>> and valid times and computes the lead time as valid - init.
>>
>> The accumulation time is the amount of time over which the variable
is
>> accumulated. If you're dealing with 1-hourly accumulated
precipitation:
>> accum_time = "010000"; (Hour, Minute, Second)
>> accum_time_sec = 3600; (seconds)
>>
>> Some people have multiple accumulations in their model output. For
>> example, at 12Z you may have precip accumulated over the last 3
hours, 6
>> hours, 12 hours, and 24 hours all contained within the same output
file.
>>
>> You'll need to look at the process that's generating your NetCDF
files and
>> fix it to encode the timing variable attributes correctly. Similar
>> date/time reformatting probably exists in python if you're more
>> comfortable with that.
>>
>> Hopefully that gets you going in the right direction.
>>
>> Thanks,
>> John
>>
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>
>>>
>>> John,
>>> thanks a lot for your assistance, it is really helpful. The idea
of the
>>> environment variables worked but as I want to used the Stat
Analysis Tool,
>>> I think it is better to fix the time parameters.
>>> Regarding the time parameters, I am still not sure what to use:I
have data
>>> from precipitation hourly accumulations starting 2008-05-
01T01:50:00 until
>>> 2008-07-31T23:50:00 in [mm/h]Every hour for example: 2008-05-
01T01:50:00,
>>> then 2008-05-01T02:50:00, 2008-05-01T03:50:00..... until the end
of
>>> JulySo, in the case of the observation file:
>>>> pr:init_time = "20080501_015000" ; time when the MET model
starts?
>>>> pr:init_time_ut = ; what is unix time of the model?
>>>> pr:valid_time = "20080501_015000" ; observation valid time, when
>>>> starts?> pr:valid_time_ut = ; what is unix time for the
observation?
>>>> pr:accum_time = "000000" ; what is refering time accumulation?
>>>> pr:accum_time_sec = 43200 ;
>>> what is refering time accumulation?
>>>
>>> Also I was wondering what is meant with "lead time" for MET and if
that
>>> parameter needs to be defined somewhere?
>>> Regards
>>> Sebastian
>>>
>>>
>>>
>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to
>>>> verify data precipitation model from weather radar
>>>> From: met_help at ucar.edu
>>>> To: sebastian_paez at hotmail.com
>>>> Date: Mon, 16 Jul 2012 09:28:22 -0600
>>>>
>>>> Sebastian,
>>>>
>>>> My apologies for calling you "Paul" in the last email. It was
too early
>>>> this morning when I wrote it!
>>>>
>>>> First, I'm glad you are able to get Grid-Stat to run on your
data. And
>>>> second, I'm glad to hear that you've been able to script the
calls to
>>>> Grid-Stat to verify over a 3 month time period.
>>>>
>>>> Regarding the use of environment variables, yes, I think that's a
great
>>>> idea. If you're having trouble with the output file names not
being
>>>> unique, using environment variables to set the
>>>> "output_prefix" value in the config file is a great way to fix
the
>>>> problem. In the Grid-Stat config file, set:
>>>>
>>>> output_prefix = "${FCST_TIME}"; (or whatever environment
variable
>>>> name you want to use)
>>>>
>>>> And be sure to set the FCST_TIME environment variable in each
iteration
>>>> of the loop. Hopefully that'll make the output file names
unique.
>>>>
>>>> However, I'm concerned about why they are not already unique in
the
>>>> first place. Doing an "ncdump -h
r_m1_1h_clutter_advcorr_fcst_11675.nc"
>>>> on the forecast file you sent me reveals the following
>>>> variable attributes for timing information:
>>>> float pr(lat, lon) ;
>>>> pr:init_time = "20050807_000000" ;
>>>> pr:init_time_ut = 1123459200 ;
>>>> pr:valid_time = "20050808_000000" ;
>>>> pr:accum_time = "120000" ;
>>>> pr:accum_time_sec = 43200 ;
>>>>
>>>> You should make sure that the timing information in these
variable
>>>> attributes are correct. Based on the command you sent me, I'm
guessing
>>>> you're actually working with 2008 data, but the times listed
>>>> there say 2005. So in the process of making these NetCDF files,
you
>>>> need to make sure the timing info is correct:
>>>>
>>>> init_time = "YYYYMMDD_HHMMSS" ; for the model
>>>> initialization time
>>>> init_time_ut = unix_time ; unix time for
the
>>>> model initialization time
>>>> valid_time = "YYYYMMDD_HHMMSS" ; for the
forecast valid
>>>> time
>>>> valid_time_ut = unix_time ; unix time for
the
>>>> forecast valid time
>>>> accum_time = "HHMMSS" ; for the
accumulation
>>>> time. If no accumulation, just set to 000000
>>>> accum_time_sec = seconds ; for the
accumulation
>>>> time in second.
>>>>
>>>> Setting this timing info correctly is very important. MET reads
this
>>>> info and writes it to the meta data in the statistics output.
>>>>
>>>> I suspect that once you've corrected the timing information in
the
>>>> variable attributes you will no longer have the problem of files
>>>> overwriting eachother.
>>>>
>>>> Hope that helps.
>>>>
>>>> John
>>>>
>>>> On 07/16/2012 08:59 AM, Sebastian Paez via RT wrote:
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>>>
>>>>>
>>>>> Hello John,
>>>>> I configured the MET files and I have all netcdf files I created
(obs
>>>> and fcsts). Then, I executed an script with a for loop using
python to
>>>> verify all netcdf files I have (3 months hourly accumulated
data). The
>>>> verification for each hourly accmulated files (obervations and
>>>> forecasts) is executed but the problem I found is that the output
>>>> files are created with the same name and therefore they are
replacing
>>>> them selves every time. That means, at the end of the execution
of the
>>>> script I only have the last output file and not a output file for
each
>>>> hour.
>>>>> I read in the chapter10 of the MET User's Guide some info, but I
still
>>>> I could solve this situation.
>>>>> I have two ideas;
>>>>> - first, based on the example Script 10-3 on the User's Guide,
it is
>>>> possible to set environment variables when the script runs. So my
idea
>>>> was to access the "output_prefix" variable from the configuration
file
>>>> and then change that variable with a loop. That would mean that I
will
>>>> give a different name for each output file. In order to do this I
>>>> would need you to tell the name of the variable of output_prefix
in
>>>> the MET configuration file, so I can use it as a environment
variable
>>>> in the execution script.
>>>>> - second, set the time in the netcdf files, forecast lead time,
>>>> forecast valid time, etc, and then based on this, run the script.
Here
>>>> I have two problems, i can not find in thevariable attributes the
>>>> options to set this time attributes; and also the information
from the
>>>> user's guide about how it runs it is not clear to me.
>>>>> Could you help me with this problem and recommend me a solution.
I am
>>>> attaching here the files I am going to verify(observations and
>>>> forecasts), the execution script i am running is below.
>>>>>
>>>>> import os
>>>>> korr = 'rawdata'
>>>>> for imgnbr in range(11665,11666,1): command =
>>>> '/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/grid_stat
>>>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_fcst_%d.nc
>>>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_obs_%d.nc
>>>>
\/home/ubuntu/Desktop/Master_Thesis/METv3.1/data/config/GridStatConfig_test
>>>> \-outdir
>>>> /home/ubuntu/Desktop/Master_Thesis/Results/1h_accumulations/%s/
\-v 2'
>>>> %(korr,korr,imgnbr,korr,korr,imgnbr,korr) os.system(command)
>>>>>
>>>>> Bast Regards
>>>>> Sebastian
>>>>>
>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET
>>>> to verify data precipitation model from weather radar
>>>>>> From: met_help at ucar.edu
>>>>>> To: sebastian_paez at hotmail.com
>>>>>> Date: Mon, 16 Jul 2012 06:45:51 -0600
>>>>>>
>>>>>> Paul,
>>>>>>
>>>>>> Let me address your second question about verifying against
point
>>>> observations and looking at the results for individual stations.
>>>> You'll use the Point-Stat tool to compare your gridded forecasts
to
>>>>>> point observations - I assume you already have point
observations
>>>> available to use. Since the addition of the MPR line type, I
think
>>>> the mask_sid option has become much less useful. Instead, I'd
>>>>>> suggest just defining the masking region as 'grid = "FULL";'.
That
>>>> will use all observations that fall in your forecast region. And
be
>>>> sure to turn *ON* the MPR output line type to get information
>>>>>> about each station. Also, in the "mask" dictionary, set
'method =
>>>> UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match each
>>>> observation to the nearest forecast point with no interpolation
>>>>>> applied to the forecast values.
>>>>>>
>>>>>> When you run Point-Stat, your output will include the matched
>>>> forecast and observation values for each station that went into
the
>>>> computation of statistics. After you verify your forecast for
many
>>>>>> output times, you have the option of running the STAT-Analysis
tool
>>>> to compute statistics over each individual station, rather than
doing
>>>> an aerial average, as is done in Point-Stat.
>>>>>>
>>>>>> Next, you would like to use a more high resolution background
map in
>>>> the output of MODE and Wavelet-Stat. The answer is, yes, it is
>>>> possible, but is likely pretty difficult. The map data in MET is
>>>>>> contained in the METv4.0/data/map directory. The default
background
>>>> map consists of the country boundaries plus the USA state
boundaries
>>>> and is drawn in the draw_map() function in
>>>>>> "METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc".
Theoretically,
>>>> you could get a lat/lon representation of the addition map data
you
>>>> want to plot and format it to look like one of those existing
>>>>>> map data files. You could use your new file to overwrite the
>>>> existing "usa_state_data" file and that should have the desired
>>>> effect. Or you could create a new file name and modify the C++
code
>>>> to
>>>>>> plot it.
>>>>>>
>>>>>> But that process might be difficult. Instead, you could just
use the
>>>> NetCDF output of the MODE and Wavelet-Stat tools, read them using
>>>> whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
>>>>>> few), and then create whatever plot you'd like. We
intentionally did
>>>> not put a lot of effort into making those plots very
configurable.
>>>> Instead, we want to keep MET focused on the verification
>>>>>> aspect while providing users with the data they'd need to make
>>>> whatever plots they'd like.
>>>>>>
>>>>>> Hope that helps.
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>>
>>>>>> On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
>>>>>>>
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967
>
>>>>>>>
>>>>>>>
>>>>>>> Sorry, now the attachment
>>>>>>>
>>>>>>> From: sebastian_paez at hotmail.com
>>>>>>> To: met_help at ucar.edu
>>>>>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use
>>>> MET to verify data precipitation model from weather radar
>>>>>>> Date: Sat, 14 Jul 2012 08:35:06 -0500
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hello John,
>>>>>>> thanks for your answer, finally i could locate correctly the
fields.
>>>> I am using radar data from the south of Germany.I am attaching
the
>>>> corrected file so you can see it. One more question about this:
Is
>>>> it possible to change the background image thatMODE and Wavelt
>>>> produce? Since my data is in a range of 290kmx290km i would like
to
>>>> have more details on the back.
>>>>>>> Another question, regarding Point Stat, with the same data i
want to
>>>> verify it to gauge stations' data. That means I want to
verifygrid
>>>> forecasts and point observations and i want the results for each
>>>> station (without interpolation). I was looking for an example but
I
>>>> could not find this particular case. Maybe you can send me a
typical
>>>> file *.stations for "mask_sid", so i can see what do i need.
>>>>>>> Best RegardsSebastian Paez
>>>>>>>
>>>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use
>>>> MET to verify data precipitation model from weather radar
>>>>>>>> From: met_help at ucar.edu
>>>>>>>> To: sebastian_paez at hotmail.com
>>>>>>>> Date: Fri, 13 Jul 2012 12:36:21 -0600
>>>>>>>>
>>>>>>>> Hello Sebastian,
>>>>>>>>
>>>>>>>> Unfortunately, we don't have those parameters well documented
>>>> anywhere at this point, but I can give you a brief description of
>>>> them. These are the NetCDF global attributes used in MET to
define
>>>> a
>>>>>>>> polar stereographic grid. For reference, here's the values
you
>>>> have set in the example NetCDF file you sent:
>>>>>>>> :Projection = "Polar Stereographic" ;
>>>>>>>> :hemisphere = "N" ;
>>>>>>>> :scale_lat = "60.000000 degrees_north" ;
>>>>>>>> :lat_pin = "47.340000" ;
>>>>>>>> :lon_pin = "7.980000" ;
>>>>>>>> :x_pin = "0.000000" ;
>>>>>>>> :y_pin = "0.000000" ;
>>>>>>>> :lon_orient = "8.000000" ;
>>>>>>>> :d_km = "10.395000" ;
>>>>>>>> :r_km = "6371.200000" ;
>>>>>>>> :nx = "290" ;
>>>>>>>> :ny = "290 grid_points" ;
>>>>>>>>
>>>>>>>> - x_pin and y_pin: specify the (x,y) location of the
reference
>>>> lat/lon provided in (lat_pin, lon_pin). Yours are set to (0,0),
>>>> meaning that (lat_pin, lon_pin) refers to the lower-left corner
of
>>>> the
>>>>>>>> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that
you're
>>>> specifying the lat/lon of the upper-right corner of the grid.
>>>>>>>>
>>>>>>>> - lat_pin and lon_pin are the lat/lon of the (x,y) specified
above.
>>>> Usually, it's the lower-left corner.
>>>>>>>>
>>>>>>>> - scale_lat is the latitude at which the spacing of the grid
>>>> (defined in d_km) is defined.
>>>>>>>>
>>>>>>>> - d_km is the spacing between grid points in kilometers at
the
>>>> latitude defined in scale_lat.
>>>>>>>>
>>>>>>>> - lon_orient specifies the longitude value that's vertical in
your
>>>> grid.
>>>>>>>>
>>>>>>>> - r_km is the assumed radius of a spherical earth.
>>>>>>>>
>>>>>>>> - nx and ny specify the size of the grid.
>>>>>>>>
>>>>>>>> I used the plot_data_plane tool to make a plot of the sample
data
>>>> you sent, and it's attached to this message. Looks like MET
thinks
>>>> your data is over eastern Europe. Is that correct?
>>>>>>>>
>>>>>>>> Now I'm not sure where you're getting the grid definition for
this
>>>> data. But if, for example, you have a GRIB file, you may find
the
>>>> following page of the MET online tutorial to be of use:
>>>>>>>>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
>>>>>>>>
>>>>>>>> Hopefully that helps. Do you have additional questions?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John Halley Gotway
>>>>>>>>
>>>>>>>> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
>>>>>>>>>
>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello Paul,
>>>>>>>>> I have been working in the verification of my fields. I have
one
>>>> more question regarding this issue:
>>>>>>>>> - i still can not locate correctly the fields on the polar
>>>> stereographic projection. I would like to know the description of
>>>> the projection attributes (scale_lat, lat_pin, lon_pin, x_pin,
>>>> y_pin, lon_orient, d_km, r_km). Based on these descriptions i can
>>>> improve the location of my fields.
>>>>>>>>> I am sending attached the file I am using so you can take a
look.
>>>> The field is the 1h accumulation of precipitation using the
>>>> weather radar.
>>>>>>>>> Best Regards
>>>>>>>>> Sebastian Paez
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET
>>>> to verify data precipitation model from weather radar
>>>>>>>>>> From: met_help at ucar.edu
>>>>>>>>>> To: sebastian_paez at hotmail.com
>>>>>>>>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
>>>>>>>>>>
>>>>>>>>>> According to our records, your request has been resolved.
If you
>>>> have any
>>>>>>>>>> further questions or concerns, please respond to this
message.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>
------------------------------------------------
Subject: Help i try to use MET to verify data precipitation model from weather radar
From: Sebastian Paez
Time: Thu Jul 19 18:04:19 2012
John,
I already upload to the ftp 3 sets of files (13000,13001,13002) with
the extensions (.stat,.txt,.cn) and the configuration file and the
execution script for the Stat_Analysis tool.
I hope you can help me in order to run properly Stat_Analysis.
Thanks again
Sebastian
> Subject: Re: FW: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
> From: met_help at ucar.edu
> To: sebastian_paez at hotmail.com
> Date: Thu, 19 Jul 2012 13:11:12 -0600
>
> Sebastian,
>
> I received your message without the Grid-Stat results you were
trying to send. If you'd like to send me data, rather than attaching
it to an email message, please post it to our anonymous ftp site
> using the following instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> I would be happy to take a look at your Grid-Stat output and the
STAT-Analysis job you're running to let you know why you're not
getting any results.
>
> Please write me back once you've posted the output of Grid-Stat
you're using to our FTP site. Also, please send me the exactly STAT-
Analysis job you're trying to run. I'm happy to take a look.
>
> Thanks,
> John Halley Gotway
>
> On 07/18/2012 02:47 PM, Sebastian Paez via RT wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >
> >
> > John,
> > This is my third attempt. This time i am not including from
Grid_Stat results. I hope it works now.
> > Best Regards
> > Sebastian
> >
> > From: sebastian_paez at hotmail.com
> > To: met_help at ucar.edu
> > Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> > Date: Wed, 18 Jul 2012 11:16:08 -0500
> >
> >
> >
> >
> >
> > John,
> > I got an error about this e-mail I sent you in the morning. I hope
you get it know.
> > Regards
> > Sebastian
> >
> > From: sebastian_paez at hotmail.com
> > To: met_help at ucar.edu
> > Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> > Date: Wed, 18 Jul 2012 05:14:50 -0500
> >
> >
> >
> >
> >
> > John,
> > I changed the time attributes generated 3 .nc files. I used them
in Grid_Stat. Based on grid_stat i tried to use Stat_Analysis, but
when trying to get the "-job summary for CTS and CSI", I got an error:
"no valid data found in STAT lines". Please see image attached.I am
also attaching .nc files created and output files from Grid_Stat.
> > I would like to give you some info about my data. I created
precipitation forecasts for 1 hour based on weather radar images from
South Germany.The forecasts are accumulated every 15min, 0min-15min,
15min-30min, 30min-45min, 45min-60min. Nevertheless, in this 3 files
(13000,13001,13002), that I am attaching I am using accumulation of 1h
only as a test for the MET package.
> >
> > It would be very helpful, If you can tell me where is the source
of the error in order to use Stat_Analysis.
> > Your assistance is highly appreciated.
> > Regards
> > Sebastian
> >
> >
> >
> >
> >
> >
> >
> >> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
> >> From: met_help at ucar.edu
> >> To: sebastian_paez at hotmail.com
> >> Date: Mon, 16 Jul 2012 15:09:57 -0600
> >>
> >> Sebastian,
> >>
> >> Unix time is the number of seconds that have elapsed since
January 1,
> >> 1970. It's commonly used in software applications for storing
time since
> >> it's a convenient way to represent date/time with a single
number. Here's
> >> a wikipedia entry about it:
> >> http://en.wikipedia.org/wiki/Unix_time
> >>
> >> On your unix machine, try running the following command:
> >> date +%s
> >> That gives you the current unix time.
> >>
> >> We typically use the date command to convert between unix time
and
> >> year/month/day/hour/minute/second. The following two date
commands might
> >> be helpful for you.
> >>
> >> (1) Convert from the date/time listed to unix time:
> >> date -d'2008-03-15 00:00:00 UTC' +%s
> >>
> >> (2) Convert from unix time listed to the to date/time format
specified:
> >> date -ud '1970-01-01 00:00:00 UTC + 1205539200 seconds'
+'%Y%m%d_%H%M%S'
> >>
> >> The init time is the model initialization time. For example,
users often
> >> run their models every 6 hours, 12 hours, or 24 hours, and look
at
> >> forecast lead times out to a certain number of hours.
> >>
> >> It sounds like you're verifying hourly model output. But is the
model run
> >> once every hour? Or is it perhaps run once at 00Z and it
generates 24
> >> 1-hourly output files? The initialization time is the time when
the model
> >> was run.
> >>
> >> Next, the valid time is the time when the model is valid. The
valid time
> >> is just the initialization time plus the lead time. MET reads
the init
> >> and valid times and computes the lead time as valid - init.
> >>
> >> The accumulation time is the amount of time over which the
variable is
> >> accumulated. If you're dealing with 1-hourly accumulated
precipitation:
> >> accum_time = "010000"; (Hour, Minute, Second)
> >> accum_time_sec = 3600; (seconds)
> >>
> >> Some people have multiple accumulations in their model output.
For
> >> example, at 12Z you may have precip accumulated over the last 3
hours, 6
> >> hours, 12 hours, and 24 hours all contained within the same
output file.
> >>
> >> You'll need to look at the process that's generating your NetCDF
files and
> >> fix it to encode the timing variable attributes correctly.
Similar
> >> date/time reformatting probably exists in python if you're more
> >> comfortable with that.
> >>
> >> Hopefully that gets you going in the right direction.
> >>
> >> Thanks,
> >> John
> >>
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >>>
> >>>
> >>> John,
> >>> thanks a lot for your assistance, it is really helpful. The idea
of the
> >>> environment variables worked but as I want to used the Stat
Analysis Tool,
> >>> I think it is better to fix the time parameters.
> >>> Regarding the time parameters, I am still not sure what to use:I
have data
> >>> from precipitation hourly accumulations starting 2008-05-
01T01:50:00 until
> >>> 2008-07-31T23:50:00 in [mm/h]Every hour for example: 2008-05-
01T01:50:00,
> >>> then 2008-05-01T02:50:00, 2008-05-01T03:50:00..... until the end
of
> >>> JulySo, in the case of the observation file:
> >>>> pr:init_time = "20080501_015000" ; time when the MET model
starts?
> >>>> pr:init_time_ut = ; what is unix time of the model?
> >>>> pr:valid_time = "20080501_015000" ; observation valid time,
when
> >>>> starts?> pr:valid_time_ut = ; what is unix time for the
observation?
> >>>> pr:accum_time = "000000" ; what is refering time accumulation?
> >>>> pr:accum_time_sec = 43200 ;
> >>> what is refering time accumulation?
> >>>
> >>> Also I was wondering what is meant with "lead time" for MET and
if that
> >>> parameter needs to be defined somewhere?
> >>> Regards
> >>> Sebastian
> >>>
> >>>
> >>>
> >>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to
> >>>> verify data precipitation model from weather radar
> >>>> From: met_help at ucar.edu
> >>>> To: sebastian_paez at hotmail.com
> >>>> Date: Mon, 16 Jul 2012 09:28:22 -0600
> >>>>
> >>>> Sebastian,
> >>>>
> >>>> My apologies for calling you "Paul" in the last email. It was
too early
> >>>> this morning when I wrote it!
> >>>>
> >>>> First, I'm glad you are able to get Grid-Stat to run on your
data. And
> >>>> second, I'm glad to hear that you've been able to script the
calls to
> >>>> Grid-Stat to verify over a 3 month time period.
> >>>>
> >>>> Regarding the use of environment variables, yes, I think that's
a great
> >>>> idea. If you're having trouble with the output file names not
being
> >>>> unique, using environment variables to set the
> >>>> "output_prefix" value in the config file is a great way to fix
the
> >>>> problem. In the Grid-Stat config file, set:
> >>>>
> >>>> output_prefix = "${FCST_TIME}"; (or whatever environment
variable
> >>>> name you want to use)
> >>>>
> >>>> And be sure to set the FCST_TIME environment variable in each
iteration
> >>>> of the loop. Hopefully that'll make the output file names
unique.
> >>>>
> >>>> However, I'm concerned about why they are not already unique in
the
> >>>> first place. Doing an "ncdump -h
r_m1_1h_clutter_advcorr_fcst_11675.nc"
> >>>> on the forecast file you sent me reveals the following
> >>>> variable attributes for timing information:
> >>>> float pr(lat, lon) ;
> >>>> pr:init_time = "20050807_000000" ;
> >>>> pr:init_time_ut = 1123459200 ;
> >>>> pr:valid_time = "20050808_000000" ;
> >>>> pr:accum_time = "120000" ;
> >>>> pr:accum_time_sec = 43200 ;
> >>>>
> >>>> You should make sure that the timing information in these
variable
> >>>> attributes are correct. Based on the command you sent me, I'm
guessing
> >>>> you're actually working with 2008 data, but the times listed
> >>>> there say 2005. So in the process of making these NetCDF
files, you
> >>>> need to make sure the timing info is correct:
> >>>>
> >>>> init_time = "YYYYMMDD_HHMMSS" ; for the
model
> >>>> initialization time
> >>>> init_time_ut = unix_time ; unix time
for the
> >>>> model initialization time
> >>>> valid_time = "YYYYMMDD_HHMMSS" ; for the
forecast valid
> >>>> time
> >>>> valid_time_ut = unix_time ; unix time
for the
> >>>> forecast valid time
> >>>> accum_time = "HHMMSS" ; for the
accumulation
> >>>> time. If no accumulation, just set to 000000
> >>>> accum_time_sec = seconds ; for the
accumulation
> >>>> time in second.
> >>>>
> >>>> Setting this timing info correctly is very important. MET
reads this
> >>>> info and writes it to the meta data in the statistics output.
> >>>>
> >>>> I suspect that once you've corrected the timing information in
the
> >>>> variable attributes you will no longer have the problem of
files
> >>>> overwriting eachother.
> >>>>
> >>>> Hope that helps.
> >>>>
> >>>> John
> >>>>
> >>>> On 07/16/2012 08:59 AM, Sebastian Paez via RT wrote:
> >>>>>
> >>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967
>
> >>>>>
> >>>>>
> >>>>> Hello John,
> >>>>> I configured the MET files and I have all netcdf files I
created (obs
> >>>> and fcsts). Then, I executed an script with a for loop using
python to
> >>>> verify all netcdf files I have (3 months hourly accumulated
data). The
> >>>> verification for each hourly accmulated files (obervations and
> >>>> forecasts) is executed but the problem I found is that the
output
> >>>> files are created with the same name and therefore they are
replacing
> >>>> them selves every time. That means, at the end of the execution
of the
> >>>> script I only have the last output file and not a output file
for each
> >>>> hour.
> >>>>> I read in the chapter10 of the MET User's Guide some info, but
I still
> >>>> I could solve this situation.
> >>>>> I have two ideas;
> >>>>> - first, based on the example Script 10-3 on the User's Guide,
it is
> >>>> possible to set environment variables when the script runs. So
my idea
> >>>> was to access the "output_prefix" variable from the
configuration file
> >>>> and then change that variable with a loop. That would mean that
I will
> >>>> give a different name for each output file. In order to do this
I
> >>>> would need you to tell the name of the variable of
output_prefix in
> >>>> the MET configuration file, so I can use it as a environment
variable
> >>>> in the execution script.
> >>>>> - second, set the time in the netcdf files, forecast lead
time,
> >>>> forecast valid time, etc, and then based on this, run the
script. Here
> >>>> I have two problems, i can not find in thevariable attributes
the
> >>>> options to set this time attributes; and also the information
from the
> >>>> user's guide about how it runs it is not clear to me.
> >>>>> Could you help me with this problem and recommend me a
solution. I am
> >>>> attaching here the files I am going to verify(observations and
> >>>> forecasts), the execution script i am running is below.
> >>>>>
> >>>>> import os
> >>>>> korr = 'rawdata'
> >>>>> for imgnbr in range(11665,11666,1): command =
> >>>> '/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/grid_stat
> >>>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_fcst_%d.nc
> >>>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_obs_%d.nc
> >>>>
\/home/ubuntu/Desktop/Master_Thesis/METv3.1/data/config/GridStatConfig_test
> >>>> \-outdir
> >>>> /home/ubuntu/Desktop/Master_Thesis/Results/1h_accumulations/%s/
\-v 2'
> >>>> %(korr,korr,imgnbr,korr,korr,imgnbr,korr) os.system(command)
> >>>>>
> >>>>> Bast Regards
> >>>>> Sebastian
> >>>>>
> >>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET
> >>>> to verify data precipitation model from weather radar
> >>>>>> From: met_help at ucar.edu
> >>>>>> To: sebastian_paez at hotmail.com
> >>>>>> Date: Mon, 16 Jul 2012 06:45:51 -0600
> >>>>>>
> >>>>>> Paul,
> >>>>>>
> >>>>>> Let me address your second question about verifying against
point
> >>>> observations and looking at the results for individual
stations.
> >>>> You'll use the Point-Stat tool to compare your gridded
forecasts to
> >>>>>> point observations - I assume you already have point
observations
> >>>> available to use. Since the addition of the MPR line type, I
think
> >>>> the mask_sid option has become much less useful. Instead, I'd
> >>>>>> suggest just defining the masking region as 'grid = "FULL";'.
That
> >>>> will use all observations that fall in your forecast region.
And be
> >>>> sure to turn *ON* the MPR output line type to get information
> >>>>>> about each station. Also, in the "mask" dictionary, set
'method =
> >>>> UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match
each
> >>>> observation to the nearest forecast point with no interpolation
> >>>>>> applied to the forecast values.
> >>>>>>
> >>>>>> When you run Point-Stat, your output will include the matched
> >>>> forecast and observation values for each station that went into
the
> >>>> computation of statistics. After you verify your forecast for
many
> >>>>>> output times, you have the option of running the STAT-
Analysis tool
> >>>> to compute statistics over each individual station, rather than
doing
> >>>> an aerial average, as is done in Point-Stat.
> >>>>>>
> >>>>>> Next, you would like to use a more high resolution background
map in
> >>>> the output of MODE and Wavelet-Stat. The answer is, yes, it is
> >>>> possible, but is likely pretty difficult. The map data in MET
is
> >>>>>> contained in the METv4.0/data/map directory. The default
background
> >>>> map consists of the country boundaries plus the USA state
boundaries
> >>>> and is drawn in the draw_map() function in
> >>>>>> "METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc".
Theoretically,
> >>>> you could get a lat/lon representation of the addition map data
you
> >>>> want to plot and format it to look like one of those existing
> >>>>>> map data files. You could use your new file to overwrite the
> >>>> existing "usa_state_data" file and that should have the desired
> >>>> effect. Or you could create a new file name and modify the C++
code
> >>>> to
> >>>>>> plot it.
> >>>>>>
> >>>>>> But that process might be difficult. Instead, you could just
use the
> >>>> NetCDF output of the MODE and Wavelet-Stat tools, read them
using
> >>>> whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
> >>>>>> few), and then create whatever plot you'd like. We
intentionally did
> >>>> not put a lot of effort into making those plots very
configurable.
> >>>> Instead, we want to keep MET focused on the verification
> >>>>>> aspect while providing users with the data they'd need to
make
> >>>> whatever plots they'd like.
> >>>>>>
> >>>>>> Hope that helps.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> John
> >>>>>>
> >>>>>>
> >>>>>> On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
> >>>>>>>
> >>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >>>>>>>
> >>>>>>>
> >>>>>>> Sorry, now the attachment
> >>>>>>>
> >>>>>>> From: sebastian_paez at hotmail.com
> >>>>>>> To: met_help at ucar.edu
> >>>>>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use
> >>>> MET to verify data precipitation model from weather radar
> >>>>>>> Date: Sat, 14 Jul 2012 08:35:06 -0500
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Hello John,
> >>>>>>> thanks for your answer, finally i could locate correctly the
fields.
> >>>> I am using radar data from the south of Germany.I am attaching
the
> >>>> corrected file so you can see it. One more question about this:
Is
> >>>> it possible to change the background image thatMODE and Wavelt
> >>>> produce? Since my data is in a range of 290kmx290km i would
like to
> >>>> have more details on the back.
> >>>>>>> Another question, regarding Point Stat, with the same data i
want to
> >>>> verify it to gauge stations' data. That means I want to
verifygrid
> >>>> forecasts and point observations and i want the results for
each
> >>>> station (without interpolation). I was looking for an example
but I
> >>>> could not find this particular case. Maybe you can send me a
typical
> >>>> file *.stations for "mask_sid", so i can see what do i need.
> >>>>>>> Best RegardsSebastian Paez
> >>>>>>>
> >>>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use
> >>>> MET to verify data precipitation model from weather radar
> >>>>>>>> From: met_help at ucar.edu
> >>>>>>>> To: sebastian_paez at hotmail.com
> >>>>>>>> Date: Fri, 13 Jul 2012 12:36:21 -0600
> >>>>>>>>
> >>>>>>>> Hello Sebastian,
> >>>>>>>>
> >>>>>>>> Unfortunately, we don't have those parameters well
documented
> >>>> anywhere at this point, but I can give you a brief description
of
> >>>> them. These are the NetCDF global attributes used in MET to
define
> >>>> a
> >>>>>>>> polar stereographic grid. For reference, here's the values
you
> >>>> have set in the example NetCDF file you sent:
> >>>>>>>> :Projection = "Polar Stereographic" ;
> >>>>>>>> :hemisphere = "N" ;
> >>>>>>>> :scale_lat = "60.000000 degrees_north"
;
> >>>>>>>> :lat_pin = "47.340000" ;
> >>>>>>>> :lon_pin = "7.980000" ;
> >>>>>>>> :x_pin = "0.000000" ;
> >>>>>>>> :y_pin = "0.000000" ;
> >>>>>>>> :lon_orient = "8.000000" ;
> >>>>>>>> :d_km = "10.395000" ;
> >>>>>>>> :r_km = "6371.200000" ;
> >>>>>>>> :nx = "290" ;
> >>>>>>>> :ny = "290 grid_points" ;
> >>>>>>>>
> >>>>>>>> - x_pin and y_pin: specify the (x,y) location of the
reference
> >>>> lat/lon provided in (lat_pin, lon_pin). Yours are set to
(0,0),
> >>>> meaning that (lat_pin, lon_pin) refers to the lower-left corner
of
> >>>> the
> >>>>>>>> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that
you're
> >>>> specifying the lat/lon of the upper-right corner of the grid.
> >>>>>>>>
> >>>>>>>> - lat_pin and lon_pin are the lat/lon of the (x,y)
specified above.
> >>>> Usually, it's the lower-left corner.
> >>>>>>>>
> >>>>>>>> - scale_lat is the latitude at which the spacing of the
grid
> >>>> (defined in d_km) is defined.
> >>>>>>>>
> >>>>>>>> - d_km is the spacing between grid points in kilometers at
the
> >>>> latitude defined in scale_lat.
> >>>>>>>>
> >>>>>>>> - lon_orient specifies the longitude value that's vertical
in your
> >>>> grid.
> >>>>>>>>
> >>>>>>>> - r_km is the assumed radius of a spherical earth.
> >>>>>>>>
> >>>>>>>> - nx and ny specify the size of the grid.
> >>>>>>>>
> >>>>>>>> I used the plot_data_plane tool to make a plot of the
sample data
> >>>> you sent, and it's attached to this message. Looks like MET
thinks
> >>>> your data is over eastern Europe. Is that correct?
> >>>>>>>>
> >>>>>>>> Now I'm not sure where you're getting the grid definition
for this
> >>>> data. But if, for example, you have a GRIB file, you may find
the
> >>>> following page of the MET online tutorial to be of use:
> >>>>>>>>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
> >>>>>>>>
> >>>>>>>> Hopefully that helps. Do you have additional questions?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> John Halley Gotway
> >>>>>>>>
> >>>>>>>> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
> >>>>>>>>>
> >>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hello Paul,
> >>>>>>>>> I have been working in the verification of my fields. I
have one
> >>>> more question regarding this issue:
> >>>>>>>>> - i still can not locate correctly the fields on the polar
> >>>> stereographic projection. I would like to know the description
of
> >>>> the projection attributes (scale_lat, lat_pin, lon_pin, x_pin,
> >>>> y_pin, lon_orient, d_km, r_km). Based on these descriptions i
can
> >>>> improve the location of my fields.
> >>>>>>>>> I am sending attached the file I am using so you can take
a look.
> >>>> The field is the 1h accumulation of precipitation using the
> >>>> weather radar.
> >>>>>>>>> Best Regards
> >>>>>>>>> Sebastian Paez
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET
> >>>> to verify data precipitation model from weather radar
> >>>>>>>>>> From: met_help at ucar.edu
> >>>>>>>>>> To: sebastian_paez at hotmail.com
> >>>>>>>>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
> >>>>>>>>>>
> >>>>>>>>>> According to our records, your request has been resolved.
If you
> >>>> have any
> >>>>>>>>>> further questions or concerns, please respond to this
message.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >
> >
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET to verify data precipitation model from weather radar
From: John Halley Gotway
Time: Fri Jul 20 08:11:36 2012
Sebastian,
Great. Thanks for sending those files.
The problem is in the STAT-Analysis config file here:
fcst_var[] = [ "pr(*,*)" ];
obs_var[] = [ "pr(*,*)" ];
If you change it to the following instead, you'll get results:
fcst_var[] = [ "pr" ];
obs_var[] = [ "pr" ];
More generally, let me tell you that the STAT-Analysis tool is mostly
just doing string matching on the *.stat ASCII output files from Grid-
Stat, Point-Stat, Ensemble-Stat, and Wavelet-Stat. If you
look in the .stat files you sent me in the "FCST_VAR" and "OBS_VAR"
columns, you'll see they just contain the string "pr".
Also, let me give you a few more tips on this...
- I generally prefer to run STAT-Analysis jobs on the command line
without using a config file. The config file is optional. When I
have a whole bunch of STAT-Analysis jobs I want to run, I might
group them together into a config file. The following command line
accomplishes the same thing you're doing with the config file:
/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/stat_analysis \
-lookin /media/Files2008/results/model1/1h_accumulation \
-out
/media/Files2008/analysis/model1/1h_accumulation/gridstat/rawdata_gridstat.out
\
-job summary -model model1 -fcst_var pr -obs_var pr -line_type CTS
-column CSI \
-dump_row
/media/Files2008/analysis/model1/1h_accumulation/gridstat/rawdata_gridstat.dump
\
-v 2
- You'll notice that I'm using the "-dump_row" option above. When
you're getting started with STAT-Analysis, it's always a good idea to
use -dump_row to write out the subset of STAT lines the tool
used in performing the job. In this case, you're looking at CSI from
the CTS line which depends on a threshold. Since you only used 1
categorical threshold in Grid-Stat (>0.010) for the pr variable,
there's isn't a problem. But if you'd used more than 1, this job
would print out summary information for CSI values computed using
different thresholds - which isn't very meaningful. In that case,
I'd recommend using "-fcst_thresh" and/or "-obs_thresh" to filter out
the lines you want. Using -dump_row enables you to inspect the lines
used for the job and make sure the job is doing exactly what
you intended.
Hope that helps clarify.
Thanks,
John
On 07/19/2012 06:04 PM, Sebastian Paez via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>
>
> John,
> I already upload to the ftp 3 sets of files (13000,13001,13002) with
the extensions (.stat,.txt,.cn) and the configuration file and the
execution script for the Stat_Analysis tool.
> I hope you can help me in order to run properly Stat_Analysis.
> Thanks again
> Sebastian
>
>> Subject: Re: FW: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
>> From: met_help at ucar.edu
>> To: sebastian_paez at hotmail.com
>> Date: Thu, 19 Jul 2012 13:11:12 -0600
>>
>> Sebastian,
>>
>> I received your message without the Grid-Stat results you were
trying to send. If you'd like to send me data, rather than attaching
it to an email message, please post it to our anonymous ftp site
>> using the following instructions:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> I would be happy to take a look at your Grid-Stat output and the
STAT-Analysis job you're running to let you know why you're not
getting any results.
>>
>> Please write me back once you've posted the output of Grid-Stat
you're using to our FTP site. Also, please send me the exactly STAT-
Analysis job you're trying to run. I'm happy to take a look.
>>
>> Thanks,
>> John Halley Gotway
>>
>> On 07/18/2012 02:47 PM, Sebastian Paez via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>
>>>
>>> John,
>>> This is my third attempt. This time i am not including from
Grid_Stat results. I hope it works now.
>>> Best Regards
>>> Sebastian
>>>
>>> From: sebastian_paez at hotmail.com
>>> To: met_help at ucar.edu
>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
>>> Date: Wed, 18 Jul 2012 11:16:08 -0500
>>>
>>>
>>>
>>>
>>>
>>> John,
>>> I got an error about this e-mail I sent you in the morning. I hope
you get it know.
>>> Regards
>>> Sebastian
>>>
>>> From: sebastian_paez at hotmail.com
>>> To: met_help at ucar.edu
>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
>>> Date: Wed, 18 Jul 2012 05:14:50 -0500
>>>
>>>
>>>
>>>
>>>
>>> John,
>>> I changed the time attributes generated 3 .nc files. I used them
in Grid_Stat. Based on grid_stat i tried to use Stat_Analysis, but
when trying to get the "-job summary for CTS and CSI", I got an error:
"no valid data found in STAT lines". Please see image attached.I am
also attaching .nc files created and output files from Grid_Stat.
>>> I would like to give you some info about my data. I created
precipitation forecasts for 1 hour based on weather radar images from
South Germany.The forecasts are accumulated every 15min, 0min-15min,
15min-30min, 30min-45min, 45min-60min. Nevertheless, in this 3 files
(13000,13001,13002), that I am attaching I am using accumulation of 1h
only as a test for the MET package.
>>>
>>> It would be very helpful, If you can tell me where is the source
of the error in order to use Stat_Analysis.
>>> Your assistance is highly appreciated.
>>> Regards
>>> Sebastian
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar
>>>> From: met_help at ucar.edu
>>>> To: sebastian_paez at hotmail.com
>>>> Date: Mon, 16 Jul 2012 15:09:57 -0600
>>>>
>>>> Sebastian,
>>>>
>>>> Unix time is the number of seconds that have elapsed since
January 1,
>>>> 1970. It's commonly used in software applications for storing
time since
>>>> it's a convenient way to represent date/time with a single
number. Here's
>>>> a wikipedia entry about it:
>>>> http://en.wikipedia.org/wiki/Unix_time
>>>>
>>>> On your unix machine, try running the following command:
>>>> date +%s
>>>> That gives you the current unix time.
>>>>
>>>> We typically use the date command to convert between unix time
and
>>>> year/month/day/hour/minute/second. The following two date
commands might
>>>> be helpful for you.
>>>>
>>>> (1) Convert from the date/time listed to unix time:
>>>> date -d'2008-03-15 00:00:00 UTC' +%s
>>>>
>>>> (2) Convert from unix time listed to the to date/time format
specified:
>>>> date -ud '1970-01-01 00:00:00 UTC + 1205539200 seconds'
+'%Y%m%d_%H%M%S'
>>>>
>>>> The init time is the model initialization time. For example,
users often
>>>> run their models every 6 hours, 12 hours, or 24 hours, and look
at
>>>> forecast lead times out to a certain number of hours.
>>>>
>>>> It sounds like you're verifying hourly model output. But is the
model run
>>>> once every hour? Or is it perhaps run once at 00Z and it
generates 24
>>>> 1-hourly output files? The initialization time is the time when
the model
>>>> was run.
>>>>
>>>> Next, the valid time is the time when the model is valid. The
valid time
>>>> is just the initialization time plus the lead time. MET reads
the init
>>>> and valid times and computes the lead time as valid - init.
>>>>
>>>> The accumulation time is the amount of time over which the
variable is
>>>> accumulated. If you're dealing with 1-hourly accumulated
precipitation:
>>>> accum_time = "010000"; (Hour, Minute, Second)
>>>> accum_time_sec = 3600; (seconds)
>>>>
>>>> Some people have multiple accumulations in their model output.
For
>>>> example, at 12Z you may have precip accumulated over the last 3
hours, 6
>>>> hours, 12 hours, and 24 hours all contained within the same
output file.
>>>>
>>>> You'll need to look at the process that's generating your NetCDF
files and
>>>> fix it to encode the timing variable attributes correctly.
Similar
>>>> date/time reformatting probably exists in python if you're more
>>>> comfortable with that.
>>>>
>>>> Hopefully that gets you going in the right direction.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>>>
>>>>>
>>>>> John,
>>>>> thanks a lot for your assistance, it is really helpful. The idea
of the
>>>>> environment variables worked but as I want to used the Stat
Analysis Tool,
>>>>> I think it is better to fix the time parameters.
>>>>> Regarding the time parameters, I am still not sure what to use:I
have data
>>>>> from precipitation hourly accumulations starting 2008-05-
01T01:50:00 until
>>>>> 2008-07-31T23:50:00 in [mm/h]Every hour for example: 2008-05-
01T01:50:00,
>>>>> then 2008-05-01T02:50:00, 2008-05-01T03:50:00..... until the end
of
>>>>> JulySo, in the case of the observation file:
>>>>>> pr:init_time = "20080501_015000" ; time when the MET model
starts?
>>>>>> pr:init_time_ut = ; what is unix time of the model?
>>>>>> pr:valid_time = "20080501_015000" ; observation valid time,
when
>>>>>> starts?> pr:valid_time_ut = ; what is unix time for the
observation?
>>>>>> pr:accum_time = "000000" ; what is refering time accumulation?
>>>>>> pr:accum_time_sec = 43200 ;
>>>>> what is refering time accumulation?
>>>>>
>>>>> Also I was wondering what is meant with "lead time" for MET and
if that
>>>>> parameter needs to be defined somewhere?
>>>>> Regards
>>>>> Sebastian
>>>>>
>>>>>
>>>>>
>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to
>>>>>> verify data precipitation model from weather radar
>>>>>> From: met_help at ucar.edu
>>>>>> To: sebastian_paez at hotmail.com
>>>>>> Date: Mon, 16 Jul 2012 09:28:22 -0600
>>>>>>
>>>>>> Sebastian,
>>>>>>
>>>>>> My apologies for calling you "Paul" in the last email. It was
too early
>>>>>> this morning when I wrote it!
>>>>>>
>>>>>> First, I'm glad you are able to get Grid-Stat to run on your
data. And
>>>>>> second, I'm glad to hear that you've been able to script the
calls to
>>>>>> Grid-Stat to verify over a 3 month time period.
>>>>>>
>>>>>> Regarding the use of environment variables, yes, I think that's
a great
>>>>>> idea. If you're having trouble with the output file names not
being
>>>>>> unique, using environment variables to set the
>>>>>> "output_prefix" value in the config file is a great way to fix
the
>>>>>> problem. In the Grid-Stat config file, set:
>>>>>>
>>>>>> output_prefix = "${FCST_TIME}"; (or whatever environment
variable
>>>>>> name you want to use)
>>>>>>
>>>>>> And be sure to set the FCST_TIME environment variable in each
iteration
>>>>>> of the loop. Hopefully that'll make the output file names
unique.
>>>>>>
>>>>>> However, I'm concerned about why they are not already unique in
the
>>>>>> first place. Doing an "ncdump -h
r_m1_1h_clutter_advcorr_fcst_11675.nc"
>>>>>> on the forecast file you sent me reveals the following
>>>>>> variable attributes for timing information:
>>>>>> float pr(lat, lon) ;
>>>>>> pr:init_time = "20050807_000000" ;
>>>>>> pr:init_time_ut = 1123459200 ;
>>>>>> pr:valid_time = "20050808_000000" ;
>>>>>> pr:accum_time = "120000" ;
>>>>>> pr:accum_time_sec = 43200 ;
>>>>>>
>>>>>> You should make sure that the timing information in these
variable
>>>>>> attributes are correct. Based on the command you sent me, I'm
guessing
>>>>>> you're actually working with 2008 data, but the times listed
>>>>>> there say 2005. So in the process of making these NetCDF
files, you
>>>>>> need to make sure the timing info is correct:
>>>>>>
>>>>>> init_time = "YYYYMMDD_HHMMSS" ; for the
model
>>>>>> initialization time
>>>>>> init_time_ut = unix_time ; unix time
for the
>>>>>> model initialization time
>>>>>> valid_time = "YYYYMMDD_HHMMSS" ; for the
forecast valid
>>>>>> time
>>>>>> valid_time_ut = unix_time ; unix time
for the
>>>>>> forecast valid time
>>>>>> accum_time = "HHMMSS" ; for the
accumulation
>>>>>> time. If no accumulation, just set to 000000
>>>>>> accum_time_sec = seconds ; for the
accumulation
>>>>>> time in second.
>>>>>>
>>>>>> Setting this timing info correctly is very important. MET
reads this
>>>>>> info and writes it to the meta data in the statistics output.
>>>>>>
>>>>>> I suspect that once you've corrected the timing information in
the
>>>>>> variable attributes you will no longer have the problem of
files
>>>>>> overwriting eachother.
>>>>>>
>>>>>> Hope that helps.
>>>>>>
>>>>>> John
>>>>>>
>>>>>> On 07/16/2012 08:59 AM, Sebastian Paez via RT wrote:
>>>>>>>
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967
>
>>>>>>>
>>>>>>>
>>>>>>> Hello John,
>>>>>>> I configured the MET files and I have all netcdf files I
created (obs
>>>>>> and fcsts). Then, I executed an script with a for loop using
python to
>>>>>> verify all netcdf files I have (3 months hourly accumulated
data). The
>>>>>> verification for each hourly accmulated files (obervations and
>>>>>> forecasts) is executed but the problem I found is that the
output
>>>>>> files are created with the same name and therefore they are
replacing
>>>>>> them selves every time. That means, at the end of the execution
of the
>>>>>> script I only have the last output file and not a output file
for each
>>>>>> hour.
>>>>>>> I read in the chapter10 of the MET User's Guide some info, but
I still
>>>>>> I could solve this situation.
>>>>>>> I have two ideas;
>>>>>>> - first, based on the example Script 10-3 on the User's Guide,
it is
>>>>>> possible to set environment variables when the script runs. So
my idea
>>>>>> was to access the "output_prefix" variable from the
configuration file
>>>>>> and then change that variable with a loop. That would mean that
I will
>>>>>> give a different name for each output file. In order to do this
I
>>>>>> would need you to tell the name of the variable of
output_prefix in
>>>>>> the MET configuration file, so I can use it as a environment
variable
>>>>>> in the execution script.
>>>>>>> - second, set the time in the netcdf files, forecast lead
time,
>>>>>> forecast valid time, etc, and then based on this, run the
script. Here
>>>>>> I have two problems, i can not find in thevariable attributes
the
>>>>>> options to set this time attributes; and also the information
from the
>>>>>> user's guide about how it runs it is not clear to me.
>>>>>>> Could you help me with this problem and recommend me a
solution. I am
>>>>>> attaching here the files I am going to verify(observations and
>>>>>> forecasts), the execution script i am running is below.
>>>>>>>
>>>>>>> import os
>>>>>>> korr = 'rawdata'
>>>>>>> for imgnbr in range(11665,11666,1): command =
>>>>>> '/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/grid_stat
>>>>>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_fcst_%d.nc
>>>>>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_obs_%d.nc
>>>>>>
\/home/ubuntu/Desktop/Master_Thesis/METv3.1/data/config/GridStatConfig_test
>>>>>> \-outdir
>>>>>> /home/ubuntu/Desktop/Master_Thesis/Results/1h_accumulations/%s/
\-v 2'
>>>>>> %(korr,korr,imgnbr,korr,korr,imgnbr,korr) os.system(command)
>>>>>>>
>>>>>>> Bast Regards
>>>>>>> Sebastian
>>>>>>>
>>>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET
>>>>>> to verify data precipitation model from weather radar
>>>>>>>> From: met_help at ucar.edu
>>>>>>>> To: sebastian_paez at hotmail.com
>>>>>>>> Date: Mon, 16 Jul 2012 06:45:51 -0600
>>>>>>>>
>>>>>>>> Paul,
>>>>>>>>
>>>>>>>> Let me address your second question about verifying against
point
>>>>>> observations and looking at the results for individual
stations.
>>>>>> You'll use the Point-Stat tool to compare your gridded
forecasts to
>>>>>>>> point observations - I assume you already have point
observations
>>>>>> available to use. Since the addition of the MPR line type, I
think
>>>>>> the mask_sid option has become much less useful. Instead, I'd
>>>>>>>> suggest just defining the masking region as 'grid = "FULL";'.
That
>>>>>> will use all observations that fall in your forecast region.
And be
>>>>>> sure to turn *ON* the MPR output line type to get information
>>>>>>>> about each station. Also, in the "mask" dictionary, set
'method =
>>>>>> UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match
each
>>>>>> observation to the nearest forecast point with no interpolation
>>>>>>>> applied to the forecast values.
>>>>>>>>
>>>>>>>> When you run Point-Stat, your output will include the matched
>>>>>> forecast and observation values for each station that went into
the
>>>>>> computation of statistics. After you verify your forecast for
many
>>>>>>>> output times, you have the option of running the STAT-
Analysis tool
>>>>>> to compute statistics over each individual station, rather than
doing
>>>>>> an aerial average, as is done in Point-Stat.
>>>>>>>>
>>>>>>>> Next, you would like to use a more high resolution background
map in
>>>>>> the output of MODE and Wavelet-Stat. The answer is, yes, it is
>>>>>> possible, but is likely pretty difficult. The map data in MET
is
>>>>>>>> contained in the METv4.0/data/map directory. The default
background
>>>>>> map consists of the country boundaries plus the USA state
boundaries
>>>>>> and is drawn in the draw_map() function in
>>>>>>>> "METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc".
Theoretically,
>>>>>> you could get a lat/lon representation of the addition map data
you
>>>>>> want to plot and format it to look like one of those existing
>>>>>>>> map data files. You could use your new file to overwrite the
>>>>>> existing "usa_state_data" file and that should have the desired
>>>>>> effect. Or you could create a new file name and modify the C++
code
>>>>>> to
>>>>>>>> plot it.
>>>>>>>>
>>>>>>>> But that process might be difficult. Instead, you could just
use the
>>>>>> NetCDF output of the MODE and Wavelet-Stat tools, read them
using
>>>>>> whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
>>>>>>>> few), and then create whatever plot you'd like. We
intentionally did
>>>>>> not put a lot of effort into making those plots very
configurable.
>>>>>> Instead, we want to keep MET focused on the verification
>>>>>>>> aspect while providing users with the data they'd need to
make
>>>>>> whatever plots they'd like.
>>>>>>>>
>>>>>>>> Hope that helps.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>>
>>>>>>>> On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
>>>>>>>>>
>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sorry, now the attachment
>>>>>>>>>
>>>>>>>>> From: sebastian_paez at hotmail.com
>>>>>>>>> To: met_help at ucar.edu
>>>>>>>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use
>>>>>> MET to verify data precipitation model from weather radar
>>>>>>>>> Date: Sat, 14 Jul 2012 08:35:06 -0500
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello John,
>>>>>>>>> thanks for your answer, finally i could locate correctly the
fields.
>>>>>> I am using radar data from the south of Germany.I am attaching
the
>>>>>> corrected file so you can see it. One more question about this:
Is
>>>>>> it possible to change the background image thatMODE and Wavelt
>>>>>> produce? Since my data is in a range of 290kmx290km i would
like to
>>>>>> have more details on the back.
>>>>>>>>> Another question, regarding Point Stat, with the same data i
want to
>>>>>> verify it to gauge stations' data. That means I want to
verifygrid
>>>>>> forecasts and point observations and i want the results for
each
>>>>>> station (without interpolation). I was looking for an example
but I
>>>>>> could not find this particular case. Maybe you can send me a
typical
>>>>>> file *.stations for "mask_sid", so i can see what do i need.
>>>>>>>>> Best RegardsSebastian Paez
>>>>>>>>>
>>>>>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use
>>>>>> MET to verify data precipitation model from weather radar
>>>>>>>>>> From: met_help at ucar.edu
>>>>>>>>>> To: sebastian_paez at hotmail.com
>>>>>>>>>> Date: Fri, 13 Jul 2012 12:36:21 -0600
>>>>>>>>>>
>>>>>>>>>> Hello Sebastian,
>>>>>>>>>>
>>>>>>>>>> Unfortunately, we don't have those parameters well
documented
>>>>>> anywhere at this point, but I can give you a brief description
of
>>>>>> them. These are the NetCDF global attributes used in MET to
define
>>>>>> a
>>>>>>>>>> polar stereographic grid. For reference, here's the values
you
>>>>>> have set in the example NetCDF file you sent:
>>>>>>>>>> :Projection = "Polar Stereographic" ;
>>>>>>>>>> :hemisphere = "N" ;
>>>>>>>>>> :scale_lat = "60.000000 degrees_north"
;
>>>>>>>>>> :lat_pin = "47.340000" ;
>>>>>>>>>> :lon_pin = "7.980000" ;
>>>>>>>>>> :x_pin = "0.000000" ;
>>>>>>>>>> :y_pin = "0.000000" ;
>>>>>>>>>> :lon_orient = "8.000000" ;
>>>>>>>>>> :d_km = "10.395000" ;
>>>>>>>>>> :r_km = "6371.200000" ;
>>>>>>>>>> :nx = "290" ;
>>>>>>>>>> :ny = "290 grid_points" ;
>>>>>>>>>>
>>>>>>>>>> - x_pin and y_pin: specify the (x,y) location of the
reference
>>>>>> lat/lon provided in (lat_pin, lon_pin). Yours are set to
(0,0),
>>>>>> meaning that (lat_pin, lon_pin) refers to the lower-left corner
of
>>>>>> the
>>>>>>>>>> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that
you're
>>>>>> specifying the lat/lon of the upper-right corner of the grid.
>>>>>>>>>>
>>>>>>>>>> - lat_pin and lon_pin are the lat/lon of the (x,y)
specified above.
>>>>>> Usually, it's the lower-left corner.
>>>>>>>>>>
>>>>>>>>>> - scale_lat is the latitude at which the spacing of the
grid
>>>>>> (defined in d_km) is defined.
>>>>>>>>>>
>>>>>>>>>> - d_km is the spacing between grid points in kilometers at
the
>>>>>> latitude defined in scale_lat.
>>>>>>>>>>
>>>>>>>>>> - lon_orient specifies the longitude value that's vertical
in your
>>>>>> grid.
>>>>>>>>>>
>>>>>>>>>> - r_km is the assumed radius of a spherical earth.
>>>>>>>>>>
>>>>>>>>>> - nx and ny specify the size of the grid.
>>>>>>>>>>
>>>>>>>>>> I used the plot_data_plane tool to make a plot of the
sample data
>>>>>> you sent, and it's attached to this message. Looks like MET
thinks
>>>>>> your data is over eastern Europe. Is that correct?
>>>>>>>>>>
>>>>>>>>>> Now I'm not sure where you're getting the grid definition
for this
>>>>>> data. But if, for example, you have a GRIB file, you may find
the
>>>>>> following page of the MET online tutorial to be of use:
>>>>>>>>>>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
>>>>>>>>>>
>>>>>>>>>> Hopefully that helps. Do you have additional questions?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> John Halley Gotway
>>>>>>>>>>
>>>>>>>>>> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
>>>>>>>>>>>
>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hello Paul,
>>>>>>>>>>> I have been working in the verification of my fields. I
have one
>>>>>> more question regarding this issue:
>>>>>>>>>>> - i still can not locate correctly the fields on the polar
>>>>>> stereographic projection. I would like to know the description
of
>>>>>> the projection attributes (scale_lat, lat_pin, lon_pin, x_pin,
>>>>>> y_pin, lon_orient, d_km, r_km). Based on these descriptions i
can
>>>>>> improve the location of my fields.
>>>>>>>>>>> I am sending attached the file I am using so you can take
a look.
>>>>>> The field is the 1h accumulation of precipitation using the
>>>>>> weather radar.
>>>>>>>>>>> Best Regards
>>>>>>>>>>> Sebastian Paez
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET
>>>>>> to verify data precipitation model from weather radar
>>>>>>>>>>>> From: met_help at ucar.edu
>>>>>>>>>>>> To: sebastian_paez at hotmail.com
>>>>>>>>>>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
>>>>>>>>>>>>
>>>>>>>>>>>> According to our records, your request has been resolved.
If you
>>>>>> have any
>>>>>>>>>>>> further questions or concerns, please respond to this
message.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
Subject: Help i try to use MET to verify data precipitation model from weather radar
From: Sebastian Paez
Time: Tue Aug 07 16:22:34 2012
John,
All calculations done si far with MET are ok,I only have one last
question:Is it possible to use Mode-Analysis Tool to get a summary of
CSI values calculated from MODE (cts files)? I tried to get it, but I
got a message of "bad column data csi".
RegardsSebastian
Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET
to verify data precipitation model from weather radar> From:
met_help at ucar.edu
> To: sebastian_paez at hotmail.com
> Date: Fri, 20 Jul 2012 08:11:36 -0600
>
> Sebastian,
>
> Great. Thanks for sending those files.
>
> The problem is in the STAT-Analysis config file here:
> fcst_var[] = [ "pr(*,*)" ];
> obs_var[] = [ "pr(*,*)" ];
>
> If you change it to the following instead, you'll get results:
> fcst_var[] = [ "pr" ];
> obs_var[] = [ "pr" ];
>
> More generally, let me tell you that the STAT-Analysis tool is
mostly just doing string matching on the *.stat ASCII output files
from Grid-Stat, Point-Stat, Ensemble-Stat, and Wavelet-Stat. If you
> look in the .stat files you sent me in the "FCST_VAR" and "OBS_VAR"
columns, you'll see they just contain the string "pr".
>
> Also, let me give you a few more tips on this...
>
> - I generally prefer to run STAT-Analysis jobs on the command line
without using a config file. The config file is optional. When I
have a whole bunch of STAT-Analysis jobs I want to run, I might
> group them together into a config file. The following command line
accomplishes the same thing you're doing with the config file:
>
> /home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/stat_analysis \
> -lookin /media/Files2008/results/model1/1h_accumulation \
> -out
/media/Files2008/analysis/model1/1h_accumulation/gridstat/rawdata_gridstat.out
\
> -job summary -model model1 -fcst_var pr -obs_var pr -line_type
CTS -column CSI \
> -dump_row
/media/Files2008/analysis/model1/1h_accumulation/gridstat/rawdata_gridstat.dump
\
> -v 2
>
> - You'll notice that I'm using the "-dump_row" option above. When
you're getting started with STAT-Analysis, it's always a good idea to
use -dump_row to write out the subset of STAT lines the tool
> used in performing the job. In this case, you're looking at CSI
from the CTS line which depends on a threshold. Since you only used 1
categorical threshold in Grid-Stat (>0.010) for the pr variable,
> there's isn't a problem. But if you'd used more than 1, this job
would print out summary information for CSI values computed using
different thresholds - which isn't very meaningful. In that case,
> I'd recommend using "-fcst_thresh" and/or "-obs_thresh" to filter
out the lines you want. Using -dump_row enables you to inspect the
lines used for the job and make sure the job is doing exactly what
> you intended.
>
> Hope that helps clarify.
>
> Thanks,
> John
>
>
> On 07/19/2012 06:04 PM, Sebastian Paez via RT wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >
> >
> > John,
> > I already upload to the ftp 3 sets of files (13000,13001,13002)
with the extensions (.stat,.txt,.cn) and the configuration file and
the execution script for the Stat_Analysis tool.
> > I hope you can help me in order to run properly Stat_Analysis.
> > Thanks again
> > Sebastian
> >
> >> Subject: Re: FW: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
> >> From: met_help at ucar.edu
> >> To: sebastian_paez at hotmail.com
> >> Date: Thu, 19 Jul 2012 13:11:12 -0600
> >>
> >> Sebastian,
> >>
> >> I received your message without the Grid-Stat results you were
trying to send. If you'd like to send me data, rather than attaching
it to an email message, please post it to our anonymous ftp site
> >> using the following instructions:
> >> http://www.dtcenter.org/met/users/support/met_help.php#ftp
> >>
> >> I would be happy to take a look at your Grid-Stat output and the
STAT-Analysis job you're running to let you know why you're not
getting any results.
> >>
> >> Please write me back once you've posted the output of Grid-Stat
you're using to our FTP site. Also, please send me the exactly STAT-
Analysis job you're trying to run. I'm happy to take a look.
> >>
> >> Thanks,
> >> John Halley Gotway
> >>
> >> On 07/18/2012 02:47 PM, Sebastian Paez via RT wrote:
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >>>
> >>>
> >>> John,
> >>> This is my third attempt. This time i am not including from
Grid_Stat results. I hope it works now.
> >>> Best Regards
> >>> Sebastian
> >>>
> >>> From: sebastian_paez at hotmail.com
> >>> To: met_help at ucar.edu
> >>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
> >>> Date: Wed, 18 Jul 2012 11:16:08 -0500
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> John,
> >>> I got an error about this e-mail I sent you in the morning. I
hope you get it know.
> >>> Regards
> >>> Sebastian
> >>>
> >>> From: sebastian_paez at hotmail.com
> >>> To: met_help at ucar.edu
> >>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
> >>> Date: Wed, 18 Jul 2012 05:14:50 -0500
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> John,
> >>> I changed the time attributes generated 3 .nc files. I used them
in Grid_Stat. Based on grid_stat i tried to use Stat_Analysis, but
when trying to get the "-job summary for CTS and CSI", I got an error:
"no valid data found in STAT lines". Please see image attached.I am
also attaching .nc files created and output files from Grid_Stat.
> >>> I would like to give you some info about my data. I created
precipitation forecasts for 1 hour based on weather radar images from
South Germany.The forecasts are accumulated every 15min, 0min-15min,
15min-30min, 30min-45min, 45min-60min. Nevertheless, in this 3 files
(13000,13001,13002), that I am attaching I am using accumulation of 1h
only as a test for the MET package.
> >>>
> >>> It would be very helpful, If you can tell me where is the source
of the error in order to use Stat_Analysis.
> >>> Your assistance is highly appreciated.
> >>> Regards
> >>> Sebastian
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
> >>>> From: met_help at ucar.edu
> >>>> To: sebastian_paez at hotmail.com
> >>>> Date: Mon, 16 Jul 2012 15:09:57 -0600
> >>>>
> >>>> Sebastian,
> >>>>
> >>>> Unix time is the number of seconds that have elapsed since
January 1,
> >>>> 1970. It's commonly used in software applications for storing
time since
> >>>> it's a convenient way to represent date/time with a single
number. Here's
> >>>> a wikipedia entry about it:
> >>>> http://en.wikipedia.org/wiki/Unix_time
> >>>>
> >>>> On your unix machine, try running the following command:
> >>>> date +%s
> >>>> That gives you the current unix time.
> >>>>
> >>>> We typically use the date command to convert between unix time
and
> >>>> year/month/day/hour/minute/second. The following two date
commands might
> >>>> be helpful for you.
> >>>>
> >>>> (1) Convert from the date/time listed to unix time:
> >>>> date -d'2008-03-15 00:00:00 UTC' +%s
> >>>>
> >>>> (2) Convert from unix time listed to the to date/time format
specified:
> >>>> date -ud '1970-01-01 00:00:00 UTC + 1205539200 seconds'
+'%Y%m%d_%H%M%S'
> >>>>
> >>>> The init time is the model initialization time. For example,
users often
> >>>> run their models every 6 hours, 12 hours, or 24 hours, and look
at
> >>>> forecast lead times out to a certain number of hours.
> >>>>
> >>>> It sounds like you're verifying hourly model output. But is
the model run
> >>>> once every hour? Or is it perhaps run once at 00Z and it
generates 24
> >>>> 1-hourly output files? The initialization time is the time
when the model
> >>>> was run.
> >>>>
> >>>> Next, the valid time is the time when the model is valid. The
valid time
> >>>> is just the initialization time plus the lead time. MET reads
the init
> >>>> and valid times and computes the lead time as valid - init.
> >>>>
> >>>> The accumulation time is the amount of time over which the
variable is
> >>>> accumulated. If you're dealing with 1-hourly accumulated
precipitation:
> >>>> accum_time = "010000"; (Hour, Minute, Second)
> >>>> accum_time_sec = 3600; (seconds)
> >>>>
> >>>> Some people have multiple accumulations in their model output.
For
> >>>> example, at 12Z you may have precip accumulated over the last 3
hours, 6
> >>>> hours, 12 hours, and 24 hours all contained within the same
output file.
> >>>>
> >>>> You'll need to look at the process that's generating your
NetCDF files and
> >>>> fix it to encode the timing variable attributes correctly.
Similar
> >>>> date/time reformatting probably exists in python if you're more
> >>>> comfortable with that.
> >>>>
> >>>> Hopefully that gets you going in the right direction.
> >>>>
> >>>> Thanks,
> >>>> John
> >>>>
> >>>>>
> >>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967
>
> >>>>>
> >>>>>
> >>>>> John,
> >>>>> thanks a lot for your assistance, it is really helpful. The
idea of the
> >>>>> environment variables worked but as I want to used the Stat
Analysis Tool,
> >>>>> I think it is better to fix the time parameters.
> >>>>> Regarding the time parameters, I am still not sure what to
use:I have data
> >>>>> from precipitation hourly accumulations starting 2008-05-
01T01:50:00 until
> >>>>> 2008-07-31T23:50:00 in [mm/h]Every hour for example: 2008-05-
01T01:50:00,
> >>>>> then 2008-05-01T02:50:00, 2008-05-01T03:50:00..... until the
end of
> >>>>> JulySo, in the case of the observation file:
> >>>>>> pr:init_time = "20080501_015000" ; time when the MET model
starts?
> >>>>>> pr:init_time_ut = ; what is unix time of the model?
> >>>>>> pr:valid_time = "20080501_015000" ; observation valid time,
when
> >>>>>> starts?> pr:valid_time_ut = ; what is unix time for the
observation?
> >>>>>> pr:accum_time = "000000" ; what is refering time
accumulation?
> >>>>>> pr:accum_time_sec = 43200 ;
> >>>>> what is refering time accumulation?
> >>>>>
> >>>>> Also I was wondering what is meant with "lead time" for MET
and if that
> >>>>> parameter needs to be defined somewhere?
> >>>>> Regards
> >>>>> Sebastian
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to
> >>>>>> verify data precipitation model from weather radar
> >>>>>> From: met_help at ucar.edu
> >>>>>> To: sebastian_paez at hotmail.com
> >>>>>> Date: Mon, 16 Jul 2012 09:28:22 -0600
> >>>>>>
> >>>>>> Sebastian,
> >>>>>>
> >>>>>> My apologies for calling you "Paul" in the last email. It
was too early
> >>>>>> this morning when I wrote it!
> >>>>>>
> >>>>>> First, I'm glad you are able to get Grid-Stat to run on your
data. And
> >>>>>> second, I'm glad to hear that you've been able to script the
calls to
> >>>>>> Grid-Stat to verify over a 3 month time period.
> >>>>>>
> >>>>>> Regarding the use of environment variables, yes, I think
that's a great
> >>>>>> idea. If you're having trouble with the output file names
not being
> >>>>>> unique, using environment variables to set the
> >>>>>> "output_prefix" value in the config file is a great way to
fix the
> >>>>>> problem. In the Grid-Stat config file, set:
> >>>>>>
> >>>>>> output_prefix = "${FCST_TIME}"; (or whatever
environment variable
> >>>>>> name you want to use)
> >>>>>>
> >>>>>> And be sure to set the FCST_TIME environment variable in each
iteration
> >>>>>> of the loop. Hopefully that'll make the output file names
unique.
> >>>>>>
> >>>>>> However, I'm concerned about why they are not already unique
in the
> >>>>>> first place. Doing an "ncdump -h
r_m1_1h_clutter_advcorr_fcst_11675.nc"
> >>>>>> on the forecast file you sent me reveals the following
> >>>>>> variable attributes for timing information:
> >>>>>> float pr(lat, lon) ;
> >>>>>> pr:init_time = "20050807_000000" ;
> >>>>>> pr:init_time_ut = 1123459200 ;
> >>>>>> pr:valid_time = "20050808_000000" ;
> >>>>>> pr:accum_time = "120000" ;
> >>>>>> pr:accum_time_sec = 43200 ;
> >>>>>>
> >>>>>> You should make sure that the timing information in these
variable
> >>>>>> attributes are correct. Based on the command you sent me,
I'm guessing
> >>>>>> you're actually working with 2008 data, but the times listed
> >>>>>> there say 2005. So in the process of making these NetCDF
files, you
> >>>>>> need to make sure the timing info is correct:
> >>>>>>
> >>>>>> init_time = "YYYYMMDD_HHMMSS" ; for the
model
> >>>>>> initialization time
> >>>>>> init_time_ut = unix_time ; unix time
for the
> >>>>>> model initialization time
> >>>>>> valid_time = "YYYYMMDD_HHMMSS" ; for the
forecast valid
> >>>>>> time
> >>>>>> valid_time_ut = unix_time ; unix time
for the
> >>>>>> forecast valid time
> >>>>>> accum_time = "HHMMSS" ; for the
accumulation
> >>>>>> time. If no accumulation, just set to 000000
> >>>>>> accum_time_sec = seconds ; for the
accumulation
> >>>>>> time in second.
> >>>>>>
> >>>>>> Setting this timing info correctly is very important. MET
reads this
> >>>>>> info and writes it to the meta data in the statistics output.
> >>>>>>
> >>>>>> I suspect that once you've corrected the timing information
in the
> >>>>>> variable attributes you will no longer have the problem of
files
> >>>>>> overwriting eachother.
> >>>>>>
> >>>>>> Hope that helps.
> >>>>>>
> >>>>>> John
> >>>>>>
> >>>>>> On 07/16/2012 08:59 AM, Sebastian Paez via RT wrote:
> >>>>>>>
> >>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >>>>>>>
> >>>>>>>
> >>>>>>> Hello John,
> >>>>>>> I configured the MET files and I have all netcdf files I
created (obs
> >>>>>> and fcsts). Then, I executed an script with a for loop using
python to
> >>>>>> verify all netcdf files I have (3 months hourly accumulated
data). The
> >>>>>> verification for each hourly accmulated files (obervations
and
> >>>>>> forecasts) is executed but the problem I found is that the
output
> >>>>>> files are created with the same name and therefore they are
replacing
> >>>>>> them selves every time. That means, at the end of the
execution of the
> >>>>>> script I only have the last output file and not a output file
for each
> >>>>>> hour.
> >>>>>>> I read in the chapter10 of the MET User's Guide some info,
but I still
> >>>>>> I could solve this situation.
> >>>>>>> I have two ideas;
> >>>>>>> - first, based on the example Script 10-3 on the User's
Guide, it is
> >>>>>> possible to set environment variables when the script runs.
So my idea
> >>>>>> was to access the "output_prefix" variable from the
configuration file
> >>>>>> and then change that variable with a loop. That would mean
that I will
> >>>>>> give a different name for each output file. In order to do
this I
> >>>>>> would need you to tell the name of the variable of
output_prefix in
> >>>>>> the MET configuration file, so I can use it as a environment
variable
> >>>>>> in the execution script.
> >>>>>>> - second, set the time in the netcdf files, forecast lead
time,
> >>>>>> forecast valid time, etc, and then based on this, run the
script. Here
> >>>>>> I have two problems, i can not find in thevariable attributes
the
> >>>>>> options to set this time attributes; and also the information
from the
> >>>>>> user's guide about how it runs it is not clear to me.
> >>>>>>> Could you help me with this problem and recommend me a
solution. I am
> >>>>>> attaching here the files I am going to verify(observations
and
> >>>>>> forecasts), the execution script i am running is below.
> >>>>>>>
> >>>>>>> import os
> >>>>>>> korr = 'rawdata'
> >>>>>>> for imgnbr in range(11665,11666,1): command =
> >>>>>> '/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/grid_stat
> >>>>>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_fcst_%d.nc
> >>>>>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_obs_%d.nc
> >>>>>>
\/home/ubuntu/Desktop/Master_Thesis/METv3.1/data/config/GridStatConfig_test
> >>>>>> \-outdir
> >>>>>>
/home/ubuntu/Desktop/Master_Thesis/Results/1h_accumulations/%s/ \-v 2'
> >>>>>> %(korr,korr,imgnbr,korr,korr,imgnbr,korr)
os.system(command)
> >>>>>>>
> >>>>>>> Bast Regards
> >>>>>>> Sebastian
> >>>>>>>
> >>>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use MET
> >>>>>> to verify data precipitation model from weather radar
> >>>>>>>> From: met_help at ucar.edu
> >>>>>>>> To: sebastian_paez at hotmail.com
> >>>>>>>> Date: Mon, 16 Jul 2012 06:45:51 -0600
> >>>>>>>>
> >>>>>>>> Paul,
> >>>>>>>>
> >>>>>>>> Let me address your second question about verifying against
point
> >>>>>> observations and looking at the results for individual
stations.
> >>>>>> You'll use the Point-Stat tool to compare your gridded
forecasts to
> >>>>>>>> point observations - I assume you already have point
observations
> >>>>>> available to use. Since the addition of the MPR line type, I
think
> >>>>>> the mask_sid option has become much less useful. Instead,
I'd
> >>>>>>>> suggest just defining the masking region as 'grid =
"FULL";'. That
> >>>>>> will use all observations that fall in your forecast region.
And be
> >>>>>> sure to turn *ON* the MPR output line type to get information
> >>>>>>>> about each station. Also, in the "mask" dictionary, set
'method =
> >>>>>> UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match
each
> >>>>>> observation to the nearest forecast point with no
interpolation
> >>>>>>>> applied to the forecast values.
> >>>>>>>>
> >>>>>>>> When you run Point-Stat, your output will include the
matched
> >>>>>> forecast and observation values for each station that went
into the
> >>>>>> computation of statistics. After you verify your forecast
for many
> >>>>>>>> output times, you have the option of running the STAT-
Analysis tool
> >>>>>> to compute statistics over each individual station, rather
than doing
> >>>>>> an aerial average, as is done in Point-Stat.
> >>>>>>>>
> >>>>>>>> Next, you would like to use a more high resolution
background map in
> >>>>>> the output of MODE and Wavelet-Stat. The answer is, yes, it
is
> >>>>>> possible, but is likely pretty difficult. The map data in
MET is
> >>>>>>>> contained in the METv4.0/data/map directory. The default
background
> >>>>>> map consists of the country boundaries plus the USA state
boundaries
> >>>>>> and is drawn in the draw_map() function in
> >>>>>>>> "METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc".
Theoretically,
> >>>>>> you could get a lat/lon representation of the addition map
data you
> >>>>>> want to plot and format it to look like one of those existing
> >>>>>>>> map data files. You could use your new file to overwrite
the
> >>>>>> existing "usa_state_data" file and that should have the
desired
> >>>>>> effect. Or you could create a new file name and modify the
C++ code
> >>>>>> to
> >>>>>>>> plot it.
> >>>>>>>>
> >>>>>>>> But that process might be difficult. Instead, you could
just use the
> >>>>>> NetCDF output of the MODE and Wavelet-Stat tools, read them
using
> >>>>>> whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
> >>>>>>>> few), and then create whatever plot you'd like. We
intentionally did
> >>>>>> not put a lot of effort into making those plots very
configurable.
> >>>>>> Instead, we want to keep MET focused on the verification
> >>>>>>>> aspect while providing users with the data they'd need to
make
> >>>>>> whatever plots they'd like.
> >>>>>>>>
> >>>>>>>> Hope that helps.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> John
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
> >>>>>>>>>
> >>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Sorry, now the attachment
> >>>>>>>>>
> >>>>>>>>> From: sebastian_paez at hotmail.com
> >>>>>>>>> To: met_help at ucar.edu
> >>>>>>>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use
> >>>>>> MET to verify data precipitation model from weather radar
> >>>>>>>>> Date: Sat, 14 Jul 2012 08:35:06 -0500
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hello John,
> >>>>>>>>> thanks for your answer, finally i could locate correctly
the fields.
> >>>>>> I am using radar data from the south of Germany.I am
attaching the
> >>>>>> corrected file so you can see it. One more question about
this: Is
> >>>>>> it possible to change the background image thatMODE and
Wavelt
> >>>>>> produce? Since my data is in a range of 290kmx290km i would
like to
> >>>>>> have more details on the back.
> >>>>>>>>> Another question, regarding Point Stat, with the same data
i want to
> >>>>>> verify it to gauge stations' data. That means I want to
verifygrid
> >>>>>> forecasts and point observations and i want the results for
each
> >>>>>> station (without interpolation). I was looking for an example
but I
> >>>>>> could not find this particular case. Maybe you can send me a
typical
> >>>>>> file *.stations for "mask_sid", so i can see what do i need.
> >>>>>>>>> Best RegardsSebastian Paez
> >>>>>>>>>
> >>>>>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i
try to use
> >>>>>> MET to verify data precipitation model from weather radar
> >>>>>>>>>> From: met_help at ucar.edu
> >>>>>>>>>> To: sebastian_paez at hotmail.com
> >>>>>>>>>> Date: Fri, 13 Jul 2012 12:36:21 -0600
> >>>>>>>>>>
> >>>>>>>>>> Hello Sebastian,
> >>>>>>>>>>
> >>>>>>>>>> Unfortunately, we don't have those parameters well
documented
> >>>>>> anywhere at this point, but I can give you a brief
description of
> >>>>>> them. These are the NetCDF global attributes used in MET to
define
> >>>>>> a
> >>>>>>>>>> polar stereographic grid. For reference, here's the
values you
> >>>>>> have set in the example NetCDF file you sent:
> >>>>>>>>>> :Projection = "Polar Stereographic"
;
> >>>>>>>>>> :hemisphere = "N" ;
> >>>>>>>>>> :scale_lat = "60.000000
degrees_north" ;
> >>>>>>>>>> :lat_pin = "47.340000" ;
> >>>>>>>>>> :lon_pin = "7.980000" ;
> >>>>>>>>>> :x_pin = "0.000000" ;
> >>>>>>>>>> :y_pin = "0.000000" ;
> >>>>>>>>>> :lon_orient = "8.000000" ;
> >>>>>>>>>> :d_km = "10.395000" ;
> >>>>>>>>>> :r_km = "6371.200000" ;
> >>>>>>>>>> :nx = "290" ;
> >>>>>>>>>> :ny = "290 grid_points" ;
> >>>>>>>>>>
> >>>>>>>>>> - x_pin and y_pin: specify the (x,y) location of the
reference
> >>>>>> lat/lon provided in (lat_pin, lon_pin). Yours are set to
(0,0),
> >>>>>> meaning that (lat_pin, lon_pin) refers to the lower-left
corner of
> >>>>>> the
> >>>>>>>>>> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that
you're
> >>>>>> specifying the lat/lon of the upper-right corner of the grid.
> >>>>>>>>>>
> >>>>>>>>>> - lat_pin and lon_pin are the lat/lon of the (x,y)
specified above.
> >>>>>> Usually, it's the lower-left corner.
> >>>>>>>>>>
> >>>>>>>>>> - scale_lat is the latitude at which the spacing of the
grid
> >>>>>> (defined in d_km) is defined.
> >>>>>>>>>>
> >>>>>>>>>> - d_km is the spacing between grid points in kilometers
at the
> >>>>>> latitude defined in scale_lat.
> >>>>>>>>>>
> >>>>>>>>>> - lon_orient specifies the longitude value that's
vertical in your
> >>>>>> grid.
> >>>>>>>>>>
> >>>>>>>>>> - r_km is the assumed radius of a spherical earth.
> >>>>>>>>>>
> >>>>>>>>>> - nx and ny specify the size of the grid.
> >>>>>>>>>>
> >>>>>>>>>> I used the plot_data_plane tool to make a plot of the
sample data
> >>>>>> you sent, and it's attached to this message. Looks like MET
thinks
> >>>>>> your data is over eastern Europe. Is that correct?
> >>>>>>>>>>
> >>>>>>>>>> Now I'm not sure where you're getting the grid definition
for this
> >>>>>> data. But if, for example, you have a GRIB file, you may
find the
> >>>>>> following page of the MET online tutorial to be of use:
> >>>>>>>>>>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
> >>>>>>>>>>
> >>>>>>>>>> Hopefully that helps. Do you have additional questions?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> John Halley Gotway
> >>>>>>>>>>
> >>>>>>>>>> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Hello Paul,
> >>>>>>>>>>> I have been working in the verification of my fields. I
have one
> >>>>>> more question regarding this issue:
> >>>>>>>>>>> - i still can not locate correctly the fields on the
polar
> >>>>>> stereographic projection. I would like to know the
description of
> >>>>>> the projection attributes (scale_lat, lat_pin, lon_pin,
x_pin,
> >>>>>> y_pin, lon_orient, d_km, r_km). Based on these descriptions i
can
> >>>>>> improve the location of my fields.
> >>>>>>>>>>> I am sending attached the file I am using so you can
take a look.
> >>>>>> The field is the 1h accumulation of precipitation using the
> >>>>>> weather radar.
> >>>>>>>>>>> Best Regards
> >>>>>>>>>>> Sebastian Paez
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use MET
> >>>>>> to verify data precipitation model from weather radar
> >>>>>>>>>>>> From: met_help at ucar.edu
> >>>>>>>>>>>> To: sebastian_paez at hotmail.com
> >>>>>>>>>>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
> >>>>>>>>>>>>
> >>>>>>>>>>>> According to our records, your request has been
resolved. If you
> >>>>>> have any
> >>>>>>>>>>>> further questions or concerns, please respond to this
message.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >
> >
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use MET to verify data precipitation model from weather radar
From: John Halley Gotway
Time: Wed Aug 08 07:24:18 2012
Sebastian,
Unfortunately, no. The MODE-Analysis tool is set up to read only the
"*_obj.txt" output files from MODE and not the "*_cts.txt" output.
However, you could easily use the same gridded data you passed
to the MODE tool as input to the Grid-Stat tool. And with the Grid-
Stat, you can generate any number of continuous and categorical
statistics that can easily be summarized using the STAT-Analysis tool.
Alternatively, you could just read the "*_cts.txt" files from MODE
into a statistical package (like Matlab, IDL, or R) and do additional
data processing.
Hope that helps.
John
On 08/07/2012 04:22 PM, Sebastian Paez via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>
>
> John,
> All calculations done si far with MET are ok,I only have one last
question:Is it possible to use Mode-Analysis Tool to get a summary of
CSI values calculated from MODE (cts files)? I tried to get it, but I
got a message of "bad column data csi".
> RegardsSebastian
>
> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to use
MET to verify data precipitation model from weather radar> From:
met_help at ucar.edu
>> To: sebastian_paez at hotmail.com
>> Date: Fri, 20 Jul 2012 08:11:36 -0600
>>
>> Sebastian,
>>
>> Great. Thanks for sending those files.
>>
>> The problem is in the STAT-Analysis config file here:
>> fcst_var[] = [ "pr(*,*)" ];
>> obs_var[] = [ "pr(*,*)" ];
>>
>> If you change it to the following instead, you'll get results:
>> fcst_var[] = [ "pr" ];
>> obs_var[] = [ "pr" ];
>>
>> More generally, let me tell you that the STAT-Analysis tool is
mostly just doing string matching on the *.stat ASCII output files
from Grid-Stat, Point-Stat, Ensemble-Stat, and Wavelet-Stat. If you
>> look in the .stat files you sent me in the "FCST_VAR" and "OBS_VAR"
columns, you'll see they just contain the string "pr".
>>
>> Also, let me give you a few more tips on this...
>>
>> - I generally prefer to run STAT-Analysis jobs on the command line
without using a config file. The config file is optional. When I
have a whole bunch of STAT-Analysis jobs I want to run, I might
>> group them together into a config file. The following command line
accomplishes the same thing you're doing with the config file:
>>
>> /home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/stat_analysis \
>> -lookin /media/Files2008/results/model1/1h_accumulation \
>> -out
/media/Files2008/analysis/model1/1h_accumulation/gridstat/rawdata_gridstat.out
\
>> -job summary -model model1 -fcst_var pr -obs_var pr -line_type
CTS -column CSI \
>> -dump_row
/media/Files2008/analysis/model1/1h_accumulation/gridstat/rawdata_gridstat.dump
\
>> -v 2
>>
>> - You'll notice that I'm using the "-dump_row" option above. When
you're getting started with STAT-Analysis, it's always a good idea to
use -dump_row to write out the subset of STAT lines the tool
>> used in performing the job. In this case, you're looking at CSI
from the CTS line which depends on a threshold. Since you only used 1
categorical threshold in Grid-Stat (>0.010) for the pr variable,
>> there's isn't a problem. But if you'd used more than 1, this job
would print out summary information for CSI values computed using
different thresholds - which isn't very meaningful. In that case,
>> I'd recommend using "-fcst_thresh" and/or "-obs_thresh" to filter
out the lines you want. Using -dump_row enables you to inspect the
lines used for the job and make sure the job is doing exactly what
>> you intended.
>>
>> Hope that helps clarify.
>>
>> Thanks,
>> John
>>
>>
>> On 07/19/2012 06:04 PM, Sebastian Paez via RT wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>
>>>
>>> John,
>>> I already upload to the ftp 3 sets of files (13000,13001,13002)
with the extensions (.stat,.txt,.cn) and the configuration file and
the execution script for the Stat_Analysis tool.
>>> I hope you can help me in order to run properly Stat_Analysis.
>>> Thanks again
>>> Sebastian
>>>
>>>> Subject: Re: FW: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
>>>> From: met_help at ucar.edu
>>>> To: sebastian_paez at hotmail.com
>>>> Date: Thu, 19 Jul 2012 13:11:12 -0600
>>>>
>>>> Sebastian,
>>>>
>>>> I received your message without the Grid-Stat results you were
trying to send. If you'd like to send me data, rather than attaching
it to an email message, please post it to our anonymous ftp site
>>>> using the following instructions:
>>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>
>>>> I would be happy to take a look at your Grid-Stat output and the
STAT-Analysis job you're running to let you know why you're not
getting any results.
>>>>
>>>> Please write me back once you've posted the output of Grid-Stat
you're using to our FTP site. Also, please send me the exactly STAT-
Analysis job you're trying to run. I'm happy to take a look.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>>
>>>> On 07/18/2012 02:47 PM, Sebastian Paez via RT wrote:
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>>>
>>>>>
>>>>> John,
>>>>> This is my third attempt. This time i am not including from
Grid_Stat results. I hope it works now.
>>>>> Best Regards
>>>>> Sebastian
>>>>>
>>>>> From: sebastian_paez at hotmail.com
>>>>> To: met_help at ucar.edu
>>>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
>>>>> Date: Wed, 18 Jul 2012 11:16:08 -0500
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> John,
>>>>> I got an error about this e-mail I sent you in the morning. I
hope you get it know.
>>>>> Regards
>>>>> Sebastian
>>>>>
>>>>> From: sebastian_paez at hotmail.com
>>>>> To: met_help at ucar.edu
>>>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
>>>>> Date: Wed, 18 Jul 2012 05:14:50 -0500
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> John,
>>>>> I changed the time attributes generated 3 .nc files. I used them
in Grid_Stat. Based on grid_stat i tried to use Stat_Analysis, but
when trying to get the "-job summary for CTS and CSI", I got an error:
"no valid data found in STAT lines". Please see image attached.I am
also attaching .nc files created and output files from Grid_Stat.
>>>>> I would like to give you some info about my data. I created
precipitation forecasts for 1 hour based on weather radar images from
South Germany.The forecasts are accumulated every 15min, 0min-15min,
15min-30min, 30min-45min, 45min-60min. Nevertheless, in this 3 files
(13000,13001,13002), that I am attaching I am using accumulation of 1h
only as a test for the MET package.
>>>>>
>>>>> It would be very helpful, If you can tell me where is the source
of the error in order to use Stat_Analysis.
>>>>> Your assistance is highly appreciated.
>>>>> Regards
>>>>> Sebastian
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to verify data precipitation model from weather radar
>>>>>> From: met_help at ucar.edu
>>>>>> To: sebastian_paez at hotmail.com
>>>>>> Date: Mon, 16 Jul 2012 15:09:57 -0600
>>>>>>
>>>>>> Sebastian,
>>>>>>
>>>>>> Unix time is the number of seconds that have elapsed since
January 1,
>>>>>> 1970. It's commonly used in software applications for storing
time since
>>>>>> it's a convenient way to represent date/time with a single
number. Here's
>>>>>> a wikipedia entry about it:
>>>>>> http://en.wikipedia.org/wiki/Unix_time
>>>>>>
>>>>>> On your unix machine, try running the following command:
>>>>>> date +%s
>>>>>> That gives you the current unix time.
>>>>>>
>>>>>> We typically use the date command to convert between unix time
and
>>>>>> year/month/day/hour/minute/second. The following two date
commands might
>>>>>> be helpful for you.
>>>>>>
>>>>>> (1) Convert from the date/time listed to unix time:
>>>>>> date -d'2008-03-15 00:00:00 UTC' +%s
>>>>>>
>>>>>> (2) Convert from unix time listed to the to date/time format
specified:
>>>>>> date -ud '1970-01-01 00:00:00 UTC + 1205539200 seconds'
+'%Y%m%d_%H%M%S'
>>>>>>
>>>>>> The init time is the model initialization time. For example,
users often
>>>>>> run their models every 6 hours, 12 hours, or 24 hours, and look
at
>>>>>> forecast lead times out to a certain number of hours.
>>>>>>
>>>>>> It sounds like you're verifying hourly model output. But is
the model run
>>>>>> once every hour? Or is it perhaps run once at 00Z and it
generates 24
>>>>>> 1-hourly output files? The initialization time is the time
when the model
>>>>>> was run.
>>>>>>
>>>>>> Next, the valid time is the time when the model is valid. The
valid time
>>>>>> is just the initialization time plus the lead time. MET reads
the init
>>>>>> and valid times and computes the lead time as valid - init.
>>>>>>
>>>>>> The accumulation time is the amount of time over which the
variable is
>>>>>> accumulated. If you're dealing with 1-hourly accumulated
precipitation:
>>>>>> accum_time = "010000"; (Hour, Minute, Second)
>>>>>> accum_time_sec = 3600; (seconds)
>>>>>>
>>>>>> Some people have multiple accumulations in their model output.
For
>>>>>> example, at 12Z you may have precip accumulated over the last 3
hours, 6
>>>>>> hours, 12 hours, and 24 hours all contained within the same
output file.
>>>>>>
>>>>>> You'll need to look at the process that's generating your
NetCDF files and
>>>>>> fix it to encode the timing variable attributes correctly.
Similar
>>>>>> date/time reformatting probably exists in python if you're more
>>>>>> comfortable with that.
>>>>>>
>>>>>> Hopefully that gets you going in the right direction.
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>>>
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967
>
>>>>>>>
>>>>>>>
>>>>>>> John,
>>>>>>> thanks a lot for your assistance, it is really helpful. The
idea of the
>>>>>>> environment variables worked but as I want to used the Stat
Analysis Tool,
>>>>>>> I think it is better to fix the time parameters.
>>>>>>> Regarding the time parameters, I am still not sure what to
use:I have data
>>>>>>> from precipitation hourly accumulations starting 2008-05-
01T01:50:00 until
>>>>>>> 2008-07-31T23:50:00 in [mm/h]Every hour for example: 2008-05-
01T01:50:00,
>>>>>>> then 2008-05-01T02:50:00, 2008-05-01T03:50:00..... until the
end of
>>>>>>> JulySo, in the case of the observation file:
>>>>>>>> pr:init_time = "20080501_015000" ; time when the MET model
starts?
>>>>>>>> pr:init_time_ut = ; what is unix time of the model?
>>>>>>>> pr:valid_time = "20080501_015000" ; observation valid time,
when
>>>>>>>> starts?> pr:valid_time_ut = ; what is unix time for the
observation?
>>>>>>>> pr:accum_time = "000000" ; what is refering time
accumulation?
>>>>>>>> pr:accum_time_sec = 43200 ;
>>>>>>> what is refering time accumulation?
>>>>>>>
>>>>>>> Also I was wondering what is meant with "lead time" for MET
and if that
>>>>>>> parameter needs to be defined somewhere?
>>>>>>> Regards
>>>>>>> Sebastian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try to
use MET to
>>>>>>>> verify data precipitation model from weather radar
>>>>>>>> From: met_help at ucar.edu
>>>>>>>> To: sebastian_paez at hotmail.com
>>>>>>>> Date: Mon, 16 Jul 2012 09:28:22 -0600
>>>>>>>>
>>>>>>>> Sebastian,
>>>>>>>>
>>>>>>>> My apologies for calling you "Paul" in the last email. It
was too early
>>>>>>>> this morning when I wrote it!
>>>>>>>>
>>>>>>>> First, I'm glad you are able to get Grid-Stat to run on your
data. And
>>>>>>>> second, I'm glad to hear that you've been able to script the
calls to
>>>>>>>> Grid-Stat to verify over a 3 month time period.
>>>>>>>>
>>>>>>>> Regarding the use of environment variables, yes, I think
that's a great
>>>>>>>> idea. If you're having trouble with the output file names
not being
>>>>>>>> unique, using environment variables to set the
>>>>>>>> "output_prefix" value in the config file is a great way to
fix the
>>>>>>>> problem. In the Grid-Stat config file, set:
>>>>>>>>
>>>>>>>> output_prefix = "${FCST_TIME}"; (or whatever
environment variable
>>>>>>>> name you want to use)
>>>>>>>>
>>>>>>>> And be sure to set the FCST_TIME environment variable in each
iteration
>>>>>>>> of the loop. Hopefully that'll make the output file names
unique.
>>>>>>>>
>>>>>>>> However, I'm concerned about why they are not already unique
in the
>>>>>>>> first place. Doing an "ncdump -h
r_m1_1h_clutter_advcorr_fcst_11675.nc"
>>>>>>>> on the forecast file you sent me reveals the following
>>>>>>>> variable attributes for timing information:
>>>>>>>> float pr(lat, lon) ;
>>>>>>>> pr:init_time = "20050807_000000" ;
>>>>>>>> pr:init_time_ut = 1123459200 ;
>>>>>>>> pr:valid_time = "20050808_000000" ;
>>>>>>>> pr:accum_time = "120000" ;
>>>>>>>> pr:accum_time_sec = 43200 ;
>>>>>>>>
>>>>>>>> You should make sure that the timing information in these
variable
>>>>>>>> attributes are correct. Based on the command you sent me,
I'm guessing
>>>>>>>> you're actually working with 2008 data, but the times listed
>>>>>>>> there say 2005. So in the process of making these NetCDF
files, you
>>>>>>>> need to make sure the timing info is correct:
>>>>>>>>
>>>>>>>> init_time = "YYYYMMDD_HHMMSS" ; for the
model
>>>>>>>> initialization time
>>>>>>>> init_time_ut = unix_time ; unix
time for the
>>>>>>>> model initialization time
>>>>>>>> valid_time = "YYYYMMDD_HHMMSS" ; for the
forecast valid
>>>>>>>> time
>>>>>>>> valid_time_ut = unix_time ; unix
time for the
>>>>>>>> forecast valid time
>>>>>>>> accum_time = "HHMMSS" ; for the
accumulation
>>>>>>>> time. If no accumulation, just set to 000000
>>>>>>>> accum_time_sec = seconds ; for the
accumulation
>>>>>>>> time in second.
>>>>>>>>
>>>>>>>> Setting this timing info correctly is very important. MET
reads this
>>>>>>>> info and writes it to the meta data in the statistics output.
>>>>>>>>
>>>>>>>> I suspect that once you've corrected the timing information
in the
>>>>>>>> variable attributes you will no longer have the problem of
files
>>>>>>>> overwriting eachother.
>>>>>>>>
>>>>>>>> Hope that helps.
>>>>>>>>
>>>>>>>> John
>>>>>>>>
>>>>>>>> On 07/16/2012 08:59 AM, Sebastian Paez via RT wrote:
>>>>>>>>>
>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello John,
>>>>>>>>> I configured the MET files and I have all netcdf files I
created (obs
>>>>>>>> and fcsts). Then, I executed an script with a for loop using
python to
>>>>>>>> verify all netcdf files I have (3 months hourly accumulated
data). The
>>>>>>>> verification for each hourly accmulated files (obervations
and
>>>>>>>> forecasts) is executed but the problem I found is that the
output
>>>>>>>> files are created with the same name and therefore they are
replacing
>>>>>>>> them selves every time. That means, at the end of the
execution of the
>>>>>>>> script I only have the last output file and not a output file
for each
>>>>>>>> hour.
>>>>>>>>> I read in the chapter10 of the MET User's Guide some info,
but I still
>>>>>>>> I could solve this situation.
>>>>>>>>> I have two ideas;
>>>>>>>>> - first, based on the example Script 10-3 on the User's
Guide, it is
>>>>>>>> possible to set environment variables when the script runs.
So my idea
>>>>>>>> was to access the "output_prefix" variable from the
configuration file
>>>>>>>> and then change that variable with a loop. That would mean
that I will
>>>>>>>> give a different name for each output file. In order to do
this I
>>>>>>>> would need you to tell the name of the variable of
output_prefix in
>>>>>>>> the MET configuration file, so I can use it as a environment
variable
>>>>>>>> in the execution script.
>>>>>>>>> - second, set the time in the netcdf files, forecast lead
time,
>>>>>>>> forecast valid time, etc, and then based on this, run the
script. Here
>>>>>>>> I have two problems, i can not find in thevariable attributes
the
>>>>>>>> options to set this time attributes; and also the information
from the
>>>>>>>> user's guide about how it runs it is not clear to me.
>>>>>>>>> Could you help me with this problem and recommend me a
solution. I am
>>>>>>>> attaching here the files I am going to verify(observations
and
>>>>>>>> forecasts), the execution script i am running is below.
>>>>>>>>>
>>>>>>>>> import os
>>>>>>>>> korr = 'rawdata'
>>>>>>>>> for imgnbr in range(11665,11666,1): command =
>>>>>>>> '/home/ubuntu/Desktop/Master_Thesis/METv3.1/bin/grid_stat
>>>>>>>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_fcst_%d.nc
>>>>>>>>
\/media/Files2008/1h_accumulations/m1/%s/r_m1_1h_%s_advcorr_obs_%d.nc
>>>>>>>>
\/home/ubuntu/Desktop/Master_Thesis/METv3.1/data/config/GridStatConfig_test
>>>>>>>> \-outdir
>>>>>>>>
/home/ubuntu/Desktop/Master_Thesis/Results/1h_accumulations/%s/ \-v 2'
>>>>>>>> %(korr,korr,imgnbr,korr,korr,imgnbr,korr)
os.system(command)
>>>>>>>>>
>>>>>>>>> Bast Regards
>>>>>>>>> Sebastian
>>>>>>>>>
>>>>>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use MET
>>>>>>>> to verify data precipitation model from weather radar
>>>>>>>>>> From: met_help at ucar.edu
>>>>>>>>>> To: sebastian_paez at hotmail.com
>>>>>>>>>> Date: Mon, 16 Jul 2012 06:45:51 -0600
>>>>>>>>>>
>>>>>>>>>> Paul,
>>>>>>>>>>
>>>>>>>>>> Let me address your second question about verifying against
point
>>>>>>>> observations and looking at the results for individual
stations.
>>>>>>>> You'll use the Point-Stat tool to compare your gridded
forecasts to
>>>>>>>>>> point observations - I assume you already have point
observations
>>>>>>>> available to use. Since the addition of the MPR line type, I
think
>>>>>>>> the mask_sid option has become much less useful. Instead,
I'd
>>>>>>>>>> suggest just defining the masking region as 'grid =
"FULL";'. That
>>>>>>>> will use all observations that fall in your forecast region.
And be
>>>>>>>> sure to turn *ON* the MPR output line type to get information
>>>>>>>>>> about each station. Also, in the "mask" dictionary, set
'method =
>>>>>>>> UW_MEAN;' and 'width = 1;' - that tells Point-Stat to match
each
>>>>>>>> observation to the nearest forecast point with no
interpolation
>>>>>>>>>> applied to the forecast values.
>>>>>>>>>>
>>>>>>>>>> When you run Point-Stat, your output will include the
matched
>>>>>>>> forecast and observation values for each station that went
into the
>>>>>>>> computation of statistics. After you verify your forecast
for many
>>>>>>>>>> output times, you have the option of running the STAT-
Analysis tool
>>>>>>>> to compute statistics over each individual station, rather
than doing
>>>>>>>> an aerial average, as is done in Point-Stat.
>>>>>>>>>>
>>>>>>>>>> Next, you would like to use a more high resolution
background map in
>>>>>>>> the output of MODE and Wavelet-Stat. The answer is, yes, it
is
>>>>>>>> possible, but is likely pretty difficult. The map data in
MET is
>>>>>>>>>> contained in the METv4.0/data/map directory. The default
background
>>>>>>>> map consists of the country boundaries plus the USA state
boundaries
>>>>>>>> and is drawn in the draw_map() function in
>>>>>>>>>> "METv4.0/src/libcode/vx_plot_util/vx_plot_util.cc".
Theoretically,
>>>>>>>> you could get a lat/lon representation of the addition map
data you
>>>>>>>> want to plot and format it to look like one of those existing
>>>>>>>>>> map data files. You could use your new file to overwrite
the
>>>>>>>> existing "usa_state_data" file and that should have the
desired
>>>>>>>> effect. Or you could create a new file name and modify the
C++ code
>>>>>>>> to
>>>>>>>>>> plot it.
>>>>>>>>>>
>>>>>>>>>> But that process might be difficult. Instead, you could
just use the
>>>>>>>> NetCDF output of the MODE and Wavelet-Stat tools, read them
using
>>>>>>>> whatever plotting tool you prefer (NCL, GRADS, IDL, to name a
>>>>>>>>>> few), and then create whatever plot you'd like. We
intentionally did
>>>>>>>> not put a lot of effort into making those plots very
configurable.
>>>>>>>> Instead, we want to keep MET focused on the verification
>>>>>>>>>> aspect while providing users with the data they'd need to
make
>>>>>>>> whatever plots they'd like.
>>>>>>>>>>
>>>>>>>>>> Hope that helps.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 07/14/2012 07:38 AM, Sebastian Paez via RT wrote:
>>>>>>>>>>>
>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sorry, now the attachment
>>>>>>>>>>>
>>>>>>>>>>> From: sebastian_paez at hotmail.com
>>>>>>>>>>> To: met_help at ucar.edu
>>>>>>>>>>> Subject: RE: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use
>>>>>>>> MET to verify data precipitation model from weather radar
>>>>>>>>>>> Date: Sat, 14 Jul 2012 08:35:06 -0500
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hello John,
>>>>>>>>>>> thanks for your answer, finally i could locate correctly
the fields.
>>>>>>>> I am using radar data from the south of Germany.I am
attaching the
>>>>>>>> corrected file so you can see it. One more question about
this: Is
>>>>>>>> it possible to change the background image thatMODE and
Wavelt
>>>>>>>> produce? Since my data is in a range of 290kmx290km i would
like to
>>>>>>>> have more details on the back.
>>>>>>>>>>> Another question, regarding Point Stat, with the same data
i want to
>>>>>>>> verify it to gauge stations' data. That means I want to
verifygrid
>>>>>>>> forecasts and point observations and i want the results for
each
>>>>>>>> station (without interpolation). I was looking for an example
but I
>>>>>>>> could not find this particular case. Maybe you can send me a
typical
>>>>>>>> file *.stations for "mask_sid", so i can see what do i need.
>>>>>>>>>>> Best RegardsSebastian Paez
>>>>>>>>>>>
>>>>>>>>>>>> Subject: Re: [rt.rap.ucar.edu #56967] Resolved: Help i
try to use
>>>>>>>> MET to verify data precipitation model from weather radar
>>>>>>>>>>>> From: met_help at ucar.edu
>>>>>>>>>>>> To: sebastian_paez at hotmail.com
>>>>>>>>>>>> Date: Fri, 13 Jul 2012 12:36:21 -0600
>>>>>>>>>>>>
>>>>>>>>>>>> Hello Sebastian,
>>>>>>>>>>>>
>>>>>>>>>>>> Unfortunately, we don't have those parameters well
documented
>>>>>>>> anywhere at this point, but I can give you a brief
description of
>>>>>>>> them. These are the NetCDF global attributes used in MET to
define
>>>>>>>> a
>>>>>>>>>>>> polar stereographic grid. For reference, here's the
values you
>>>>>>>> have set in the example NetCDF file you sent:
>>>>>>>>>>>> :Projection = "Polar Stereographic"
;
>>>>>>>>>>>> :hemisphere = "N" ;
>>>>>>>>>>>> :scale_lat = "60.000000
degrees_north" ;
>>>>>>>>>>>> :lat_pin = "47.340000" ;
>>>>>>>>>>>> :lon_pin = "7.980000" ;
>>>>>>>>>>>> :x_pin = "0.000000" ;
>>>>>>>>>>>> :y_pin = "0.000000" ;
>>>>>>>>>>>> :lon_orient = "8.000000" ;
>>>>>>>>>>>> :d_km = "10.395000" ;
>>>>>>>>>>>> :r_km = "6371.200000" ;
>>>>>>>>>>>> :nx = "290" ;
>>>>>>>>>>>> :ny = "290 grid_points" ;
>>>>>>>>>>>>
>>>>>>>>>>>> - x_pin and y_pin: specify the (x,y) location of the
reference
>>>>>>>> lat/lon provided in (lat_pin, lon_pin). Yours are set to
(0,0),
>>>>>>>> meaning that (lat_pin, lon_pin) refers to the lower-left
corner of
>>>>>>>> the
>>>>>>>>>>>> grid. Setting (x_pin, y_pin) = (nx, ny) would mean that
you're
>>>>>>>> specifying the lat/lon of the upper-right corner of the grid.
>>>>>>>>>>>>
>>>>>>>>>>>> - lat_pin and lon_pin are the lat/lon of the (x,y)
specified above.
>>>>>>>> Usually, it's the lower-left corner.
>>>>>>>>>>>>
>>>>>>>>>>>> - scale_lat is the latitude at which the spacing of the
grid
>>>>>>>> (defined in d_km) is defined.
>>>>>>>>>>>>
>>>>>>>>>>>> - d_km is the spacing between grid points in kilometers
at the
>>>>>>>> latitude defined in scale_lat.
>>>>>>>>>>>>
>>>>>>>>>>>> - lon_orient specifies the longitude value that's
vertical in your
>>>>>>>> grid.
>>>>>>>>>>>>
>>>>>>>>>>>> - r_km is the assumed radius of a spherical earth.
>>>>>>>>>>>>
>>>>>>>>>>>> - nx and ny specify the size of the grid.
>>>>>>>>>>>>
>>>>>>>>>>>> I used the plot_data_plane tool to make a plot of the
sample data
>>>>>>>> you sent, and it's attached to this message. Looks like MET
thinks
>>>>>>>> your data is over eastern Europe. Is that correct?
>>>>>>>>>>>>
>>>>>>>>>>>> Now I'm not sure where you're getting the grid definition
for this
>>>>>>>> data. But if, for example, you have a GRIB file, you may
find the
>>>>>>>> following page of the MET online tutorial to be of use:
>>>>>>>>>>>>
http://www.dtcenter.org/met/users/support/online_tutorial/METv4.0/copygb/run4.php
>>>>>>>>>>>>
>>>>>>>>>>>> Hopefully that helps. Do you have additional questions?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> John Halley Gotway
>>>>>>>>>>>>
>>>>>>>>>>>> On 07/13/2012 11:31 AM, Sebastian Paez via RT wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56967 >
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Paul,
>>>>>>>>>>>>> I have been working in the verification of my fields. I
have one
>>>>>>>> more question regarding this issue:
>>>>>>>>>>>>> - i still can not locate correctly the fields on the
polar
>>>>>>>> stereographic projection. I would like to know the
description of
>>>>>>>> the projection attributes (scale_lat, lat_pin, lon_pin,
x_pin,
>>>>>>>> y_pin, lon_orient, d_km, r_km). Based on these descriptions i
can
>>>>>>>> improve the location of my fields.
>>>>>>>>>>>>> I am sending attached the file I am using so you can
take a look.
>>>>>>>> The field is the 1h accumulation of precipitation using the
>>>>>>>> weather radar.
>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>> Sebastian Paez
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Subject: [rt.rap.ucar.edu #56967] Resolved: Help i try
to use MET
>>>>>>>> to verify data precipitation model from weather radar
>>>>>>>>>>>>>> From: met_help at ucar.edu
>>>>>>>>>>>>>> To: sebastian_paez at hotmail.com
>>>>>>>>>>>>>> Date: Tue, 3 Jul 2012 08:09:02 -0600
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> According to our records, your request has been
resolved. If you
>>>>>>>> have any
>>>>>>>>>>>>>> further questions or concerns, please respond to this
message.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
------------------------------------------------
More information about the Met_help
mailing list