[Met_help] [rt.rap.ucar.edu #49061] History for Need help for ASCII2NC and Point Stat tool
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Tue Sep 13 14:41:21 MDT 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear Paul,
I would like to thank you Paul for your necessary suggestions. I've
implemented them and now its work but I want use this file ( output from
ascii2nc) for Point Stat tool. I have tried to run point_stat, its running
and creating some stat files too but stats file having only headers no
statistics. I'm not getting why its so?
I've transferred data files as well as script file in
incoming/irap/met_help/singh_data. Here I want to run it for TMP and
Rainfall. The files for TMP are 20110601_060000_tt.dat
20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
20110601_060000_rf.dat
20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is WRF
outfile in GRIB1 format (output from WPPv3.2) named as
wrfprs_d01_2011-06-01-000_fcst01.grib1.
I want to ask you something on some other data. Suppose I have some ASCII
data having only 3 columns viz. Latitude, Longitude and Rainfall (say). Can
we convert it through ascii2nc for MET supported format? If yes, then what
we can take rest of the columns. I also transferred this kind of data named
as TRMM_16AUG2011.asc,
TRMM_16AUG2011.dat
I want to use point stat for this data also.
Thank you in advance.
On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <met_help at ucar.edu
> wrote:
> Harvir,
>
> I have identified several problems with the ascii2nc input file
> suflnd_20110601_06z_rrf.dat:
>
> 1. ascii2nc input columns must be space-delimited. Several of the columns
> in the input file are tab delimited. Do a
> find/replace on the file to change all tabs to spaces.
>
> 2. After you perform step 1, you will see the following error when you try
> to run ascii2nc:
>
> $ ascii2nc.sh
> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
> Processing 11414 observations.
>
>
> timestring_to_sec(const char *) -> can't parse time string "974.8"
>
>
> The reason for this error is that your file appears to contain accumulated
> precipitation observations (GRIB code 61 in
> column 7). In this case, ascii2nc is expecting an accumulation interval in
> column 8 in the format HH, for example 06 or
> 24. That is the reason for the error message above.
>
> 3. Column 9 should contain the height above ground of the observation
> station in meters. For accumulated
> precipitation, this information is ignored by ascii2nc because this is a
> "surface" observation. We typically use a fill
> value of -9999, indicating invalid data.
>
> 4. Column 10 should contain the observed values. I see valid observation
> values on some rows, but many are -99.0 which
> I interpret to mean invalid data. In this case, it does not make sense to
> include invalid observation data, and those
> lines should be deleted.
>
> I took the liberty of making a couple assumptions about your data and
> trying to adjust the input file that you sent me
> to something that ascii2nc can read. The assumptions I made are:
>
> 1. All of your observations are "surface" observations, so the message
> type (column 1) was set to ADPSFC
> 2. The accumulation interval is 6 hours (column 8 is set to 06)
> 3. The observation values are in mm, and left as-is
>
> If these assumptions, mainly #2 and #3, are incorrect, please make the
> appropriate adjustments. I attached the "fixed"
> file to this email for you to review and change as you wish. Please let me
> know if you have any questions.
>
> Paul
>
>
> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
> >
> >> Dear Paul,
> >>
> >> Accept My apology. Actually yesterday I forgot to attached th files.
> > However, as per your advice through ftp I have putted the same attached
> > files there also. and its path is
> > *incoming/irap/met_help/singh_data*
> >
> >> I'm sending you ASCII file and ascii2nc script. Please have close look
> on
> >> it if am wrong somewhere please let me know.
> >> And for TRMM rainfall I'll back to you later ...if I'll successful or
> not
> >> in pcp_combine ... However, I've run all the test scripts successfully.
> >> Thanx in advance...
> >>
> >> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
> >> met_help at ucar.edu> wrote:
> >>
> >>> Harvir,
> >>>
> >>> I'm afraid that MET cannot read TRMM data directly. The best way to
> read
> >>> TRMM data into MET would be to convert the
> >>> data from it's current format into a NetCDF file that has the same
> format
> >>> as the output of the MET pcp_combine tool. If
> >>> you ran the MET test scripts after you built MET, you should be able to
> >>> find pcp_combine output files in the sub-folder
> >>> out/pcp_combine under your MET build folder. If you are successful in
> >>> converting your data in this way, please send us
> >>> your code so that we can make it available for other MET users who
> might
> >>> want to use TRMM data. I'm sorry the answer to
> >>> this question is not more simple.
> >>>
> >>> Regarding your ascii2nc question, can you please send us the ascii2nc
> >>> input that is causing your error? Also, what is
> >>> the command line statement that you are using to invoke ascii2nc?
> >>>
> >>> Thanks,
> >>>
> >>> Paul
> >>>
> >>>
> >>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>
> >>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
> >>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>> Queue: met_help
> >>>> Subject: Getting Problem in converting TRMM rainfall to MET
> format
> >>>> Owner: Nobody
> >>>> Requestors: harvirduphy at gmail.com
> >>>> Status: new
> >>>> Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
> >>>>
> >>>>
> >>>> Dear Sir /Madam,
> >>>> I am using TRMM rainfall Netcdf or bin data for grid_stat MET tool?
> >>> which
> >>>> converter should did I use to convert TRMM Rainfall data (.nc or .bin)
> >>> to
> >>>> GRIB1 format? I've tried copygb through Unified Post Processing
> >>> (UPPv1.0),
> >>>> still I'm unable to convert it for METv3.0.1 usage.
> >>>> I want to use some ASCII DATA may be output of some models like GFS
> etc
> >>> ..
> >>>> I have extracted data in the desired 10 columns. I'm trying to use
> >>> *ascii2nc
> >>>> *this also giving some error like: open_met_ascii() -> line number 0
> >>> doesn't
> >>>> contain the expected number of colu
> >>>> N_items = 7
> >>>> N_chars = 512
> >>>> N_ints = 100
> >>>> LineNumber = 1
> >>>>
> >>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
> >>>> Item[ 1] = "48.68"
> >>>> Item[ 2] = "9.23 "
> >>>> Item[ 3] = "391.0 61 "
> >>>> Item[ 4] = "974.8"
> >>>>
> >>>> Item[ 5] = "1021.5"
> >>>> Item[ 6] = "-99.0"
> >>>>
> >>>> Offset[ 0] = 0
> >>>> Offset[ 1] = 32
> >>>> Offset[ 2] = 41
> >>>> Offset[ 3] = 47
> >>>> Offset[ 4] = 57
> >>>>
> >>>> Offset[ 5] = 64
> >>>> Offset[ 6] = 71
> >>>>
> >>>> Please Help me....
> >>>> Earliest reply will be appreciated.
> >>>>
> >>>> Thanks in advance for any help you can provide!
> >>>>
> >>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for Post
> Processing
> >>>> Tool "UPPv1.0 "*
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >
>
>
>
--
* Regards*
**************************************
**
* HARVIR SINGH <http://www.harvir007.webs.com>*
* **
***************************************
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and Point Stat tool
From: Paul Oldenburg
Time: Fri Aug 19 08:57:24 2011
Harvir,
Can you please send me the point_stat config file that you are trying
to use? I see in point_stat.sh that this file is
on your system at:
/gpfs1/home/harvir/SW/METv3.0.1/data/config/PointStatConfig_tmp
That will help me understand what you are trying to verify.
Thanks,
Paul
On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
>
> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
> Transaction: Ticket created by harvirduphy at gmail.com
> Queue: met_help
> Subject: Need help for ASCII2NC and Point Stat tool
> Owner: Nobody
> Requestors: harvirduphy at gmail.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>
>
> Dear Paul,
>
> I would like to thank you Paul for your necessary suggestions. I've
> implemented them and now its work but I want use this file ( output
from
> ascii2nc) for Point Stat tool. I have tried to run point_stat, its
running
> and creating some stat files too but stats file having only headers
no
> statistics. I'm not getting why its so?
> I've transferred data files as well as script file in
> incoming/irap/met_help/singh_data. Here I want to run it for TMP and
> Rainfall. The files for TMP are 20110601_060000_tt.dat
> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
> 20110601_060000_rf.dat
> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
> outfile in GRIB1 format (output from WPPv3.2) named as
> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>
>
>
> I want to ask you something on some other data. Suppose I have some
ASCII
> data having only 3 columns viz. Latitude, Longitude and Rainfall
(say). Can
> we convert it through ascii2nc for MET supported format? If yes,
then what
> we can take rest of the columns. I also transferred this kind of
data named
> as TRMM_16AUG2011.asc,
> TRMM_16AUG2011.dat
> I want to use point stat for this data also.
>
> Thank you in advance.
>
> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg}
<met_help at ucar.edu
>> wrote:
>
>> Harvir,
>>
>> I have identified several problems with the ascii2nc input file
>> suflnd_20110601_06z_rrf.dat:
>>
>> 1. ascii2nc input columns must be space-delimited. Several of the
columns
>> in the input file are tab delimited. Do a
>> find/replace on the file to change all tabs to spaces.
>>
>> 2. After you perform step 1, you will see the following error when
you try
>> to run ascii2nc:
>>
>> $ ascii2nc.sh
>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
>> Processing 11414 observations.
>>
>>
>> timestring_to_sec(const char *) -> can't parse time string "974.8"
>>
>>
>> The reason for this error is that your file appears to contain
accumulated
>> precipitation observations (GRIB code 61 in
>> column 7). In this case, ascii2nc is expecting an accumulation
interval in
>> column 8 in the format HH, for example 06 or
>> 24. That is the reason for the error message above.
>>
>> 3. Column 9 should contain the height above ground of the
observation
>> station in meters. For accumulated
>> precipitation, this information is ignored by ascii2nc because this
is a
>> "surface" observation. We typically use a fill
>> value of -9999, indicating invalid data.
>>
>> 4. Column 10 should contain the observed values. I see valid
observation
>> values on some rows, but many are -99.0 which
>> I interpret to mean invalid data. In this case, it does not make
sense to
>> include invalid observation data, and those
>> lines should be deleted.
>>
>> I took the liberty of making a couple assumptions about your data
and
>> trying to adjust the input file that you sent me
>> to something that ascii2nc can read. The assumptions I made are:
>>
>> 1. All of your observations are "surface" observations, so the
message
>> type (column 1) was set to ADPSFC
>> 2. The accumulation interval is 6 hours (column 8 is set to 06)
>> 3. The observation values are in mm, and left as-is
>>
>> If these assumptions, mainly #2 and #3, are incorrect, please make
the
>> appropriate adjustments. I attached the "fixed"
>> file to this email for you to review and change as you wish.
Please let me
>> know if you have any questions.
>>
>> Paul
>>
>>
>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
>>>
>>>> Dear Paul,
>>>>
>>>> Accept My apology. Actually yesterday I forgot to attached th
files.
>>> However, as per your advice through ftp I have putted the same
attached
>>> files there also. and its path is
>>> *incoming/irap/met_help/singh_data*
>>>
>>>> I'm sending you ASCII file and ascii2nc script. Please have close
look
>> on
>>>> it if am wrong somewhere please let me know.
>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
>> not
>>>> in pcp_combine ... However, I've run all the test scripts
successfully.
>>>> Thanx in advance...
>>>>
>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>> Harvir,
>>>>>
>>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
>> read
>>>>> TRMM data into MET would be to convert the
>>>>> data from it's current format into a NetCDF file that has the
same
>> format
>>>>> as the output of the MET pcp_combine tool. If
>>>>> you ran the MET test scripts after you built MET, you should be
able to
>>>>> find pcp_combine output files in the sub-folder
>>>>> out/pcp_combine under your MET build folder. If you are
successful in
>>>>> converting your data in this way, please send us
>>>>> your code so that we can make it available for other MET users
who
>> might
>>>>> want to use TRMM data. I'm sorry the answer to
>>>>> this question is not more simple.
>>>>>
>>>>> Regarding your ascii2nc question, can you please send us the
ascii2nc
>>>>> input that is causing your error? Also, what is
>>>>> the command line statement that you are using to invoke
ascii2nc?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>
>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>> Queue: met_help
>>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
>> format
>>>>>> Owner: Nobody
>>>>>> Requestors: harvirduphy at gmail.com
>>>>>> Status: new
>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>
>>>>>>
>>>>>> Dear Sir /Madam,
>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat MET
tool?
>>>>> which
>>>>>> converter should did I use to convert TRMM Rainfall data (.nc
or .bin)
>>>>> to
>>>>>> GRIB1 format? I've tried copygb through Unified Post Processing
>>>>> (UPPv1.0),
>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>> I want to use some ASCII DATA may be output of some models like
GFS
>> etc
>>>>> ..
>>>>>> I have extracted data in the desired 10 columns. I'm trying to
use
>>>>> *ascii2nc
>>>>>> *this also giving some error like: open_met_ascii() -> line
number 0
>>>>> doesn't
>>>>>> contain the expected number of colu
>>>>>> N_items = 7
>>>>>> N_chars = 512
>>>>>> N_ints = 100
>>>>>> LineNumber = 1
>>>>>>
>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
>>>>>> Item[ 1] = "48.68"
>>>>>> Item[ 2] = "9.23 "
>>>>>> Item[ 3] = "391.0 61 "
>>>>>> Item[ 4] = "974.8"
>>>>>>
>>>>>> Item[ 5] = "1021.5"
>>>>>> Item[ 6] = "-99.0"
>>>>>>
>>>>>> Offset[ 0] = 0
>>>>>> Offset[ 1] = 32
>>>>>> Offset[ 2] = 41
>>>>>> Offset[ 3] = 47
>>>>>> Offset[ 4] = 57
>>>>>>
>>>>>> Offset[ 5] = 64
>>>>>> Offset[ 6] = 71
>>>>>>
>>>>>> Please Help me....
>>>>>> Earliest reply will be appreciated.
>>>>>>
>>>>>> Thanks in advance for any help you can provide!
>>>>>>
>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for Post
>> Processing
>>>>>> Tool "UPPv1.0 "*
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>
>>
>>
>>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and Point Stat tool
From: Paul Oldenburg
Time: Fri Aug 19 09:11:55 2011
Harvir,
I see that your model data file has temperature data (TMP), but I do
not see accumulated rainfall data among the
forecast variables. Are you sure that this model data file should
contain rainfall information? Usually that has a
GRIB code of 61 (APCP). Below is a list of the forecast variables
that I see in your model data file.
Paul
$ wgrib wrfprs_d01_2011-06-01-000_fcst01.grib1 | perl
-e'while(<>){@g=split(/\:/); printf("%-12s%-8s\n",$g[4],$g[3]);}'
| sort -u
kpds5=11 TMP
kpds5=17 DPT
kpds5=1 PRES
kpds5=20 VIS
kpds5=211 REFD
kpds5=212 REFC
kpds5=2 PRMSL
kpds5=33 UGRD
kpds5=34 VGRD
kpds5=39 VVEL
kpds5=41 ABSV
kpds5=51 SPFH
kpds5=52 RH
kpds5=7 HGT
On 08/19/2011 08:57 AM, Paul Oldenburg wrote:
> Harvir,
>
> Can you please send me the point_stat config file that you are
trying to use? I see in point_stat.sh that this file is
> on your system at:
>
> /gpfs1/home/harvir/SW/METv3.0.1/data/config/PointStatConfig_tmp
>
> That will help me understand what you are trying to verify.
>
> Thanks,
>
> Paul
>
>
> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>
>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
>> Transaction: Ticket created by harvirduphy at gmail.com
>> Queue: met_help
>> Subject: Need help for ASCII2NC and Point Stat tool
>> Owner: Nobody
>> Requestors: harvirduphy at gmail.com
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>
>>
>> Dear Paul,
>>
>> I would like to thank you Paul for your necessary suggestions. I've
>> implemented them and now its work but I want use this file ( output
from
>> ascii2nc) for Point Stat tool. I have tried to run point_stat, its
running
>> and creating some stat files too but stats file having only headers
no
>> statistics. I'm not getting why its so?
>> I've transferred data files as well as script file in
>> incoming/irap/met_help/singh_data. Here I want to run it for TMP
and
>> Rainfall. The files for TMP are 20110601_060000_tt.dat
>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
>> 20110601_060000_rf.dat
>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
>> outfile in GRIB1 format (output from WPPv3.2) named as
>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>>
>>
>>
>> I want to ask you something on some other data. Suppose I have some
ASCII
>> data having only 3 columns viz. Latitude, Longitude and Rainfall
(say). Can
>> we convert it through ascii2nc for MET supported format? If yes,
then what
>> we can take rest of the columns. I also transferred this kind of
data named
>> as TRMM_16AUG2011.asc,
>> TRMM_16AUG2011.dat
>> I want to use point stat for this data also.
>>
>> Thank you in advance.
>>
>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg}
<met_help at ucar.edu
>>> wrote:
>>
>>> Harvir,
>>>
>>> I have identified several problems with the ascii2nc input file
>>> suflnd_20110601_06z_rrf.dat:
>>>
>>> 1. ascii2nc input columns must be space-delimited. Several of
the columns
>>> in the input file are tab delimited. Do a
>>> find/replace on the file to change all tabs to spaces.
>>>
>>> 2. After you perform step 1, you will see the following error
when you try
>>> to run ascii2nc:
>>>
>>> $ ascii2nc.sh
>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
>>> Processing 11414 observations.
>>>
>>>
>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
>>>
>>>
>>> The reason for this error is that your file appears to contain
accumulated
>>> precipitation observations (GRIB code 61 in
>>> column 7). In this case, ascii2nc is expecting an accumulation
interval in
>>> column 8 in the format HH, for example 06 or
>>> 24. That is the reason for the error message above.
>>>
>>> 3. Column 9 should contain the height above ground of the
observation
>>> station in meters. For accumulated
>>> precipitation, this information is ignored by ascii2nc because
this is a
>>> "surface" observation. We typically use a fill
>>> value of -9999, indicating invalid data.
>>>
>>> 4. Column 10 should contain the observed values. I see valid
observation
>>> values on some rows, but many are -99.0 which
>>> I interpret to mean invalid data. In this case, it does not make
sense to
>>> include invalid observation data, and those
>>> lines should be deleted.
>>>
>>> I took the liberty of making a couple assumptions about your data
and
>>> trying to adjust the input file that you sent me
>>> to something that ascii2nc can read. The assumptions I made are:
>>>
>>> 1. All of your observations are "surface" observations, so the
message
>>> type (column 1) was set to ADPSFC
>>> 2. The accumulation interval is 6 hours (column 8 is set to 06)
>>> 3. The observation values are in mm, and left as-is
>>>
>>> If these assumptions, mainly #2 and #3, are incorrect, please make
the
>>> appropriate adjustments. I attached the "fixed"
>>> file to this email for you to review and change as you wish.
Please let me
>>> know if you have any questions.
>>>
>>> Paul
>>>
>>>
>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
>>>>
>>>>> Dear Paul,
>>>>>
>>>>> Accept My apology. Actually yesterday I forgot to attached th
files.
>>>> However, as per your advice through ftp I have putted the same
attached
>>>> files there also. and its path is
>>>> *incoming/irap/met_help/singh_data*
>>>>
>>>>> I'm sending you ASCII file and ascii2nc script. Please have
close look
>>> on
>>>>> it if am wrong somewhere please let me know.
>>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
>>> not
>>>>> in pcp_combine ... However, I've run all the test scripts
successfully.
>>>>> Thanx in advance...
>>>>>
>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
>>>>> met_help at ucar.edu> wrote:
>>>>>
>>>>>> Harvir,
>>>>>>
>>>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
>>> read
>>>>>> TRMM data into MET would be to convert the
>>>>>> data from it's current format into a NetCDF file that has the
same
>>> format
>>>>>> as the output of the MET pcp_combine tool. If
>>>>>> you ran the MET test scripts after you built MET, you should be
able to
>>>>>> find pcp_combine output files in the sub-folder
>>>>>> out/pcp_combine under your MET build folder. If you are
successful in
>>>>>> converting your data in this way, please send us
>>>>>> your code so that we can make it available for other MET users
who
>>> might
>>>>>> want to use TRMM data. I'm sorry the answer to
>>>>>> this question is not more simple.
>>>>>>
>>>>>> Regarding your ascii2nc question, can you please send us the
ascii2nc
>>>>>> input that is causing your error? Also, what is
>>>>>> the command line statement that you are using to invoke
ascii2nc?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Paul
>>>>>>
>>>>>>
>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>>
>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>> Queue: met_help
>>>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
>>> format
>>>>>>> Owner: Nobody
>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>> Status: new
>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>>
>>>>>>>
>>>>>>> Dear Sir /Madam,
>>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat MET
tool?
>>>>>> which
>>>>>>> converter should did I use to convert TRMM Rainfall data (.nc
or .bin)
>>>>>> to
>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
>>>>>> (UPPv1.0),
>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>>> I want to use some ASCII DATA may be output of some models
like GFS
>>> etc
>>>>>> ..
>>>>>>> I have extracted data in the desired 10 columns. I'm trying to
use
>>>>>> *ascii2nc
>>>>>>> *this also giving some error like: open_met_ascii() -> line
number 0
>>>>>> doesn't
>>>>>>> contain the expected number of colu
>>>>>>> N_items = 7
>>>>>>> N_chars = 512
>>>>>>> N_ints = 100
>>>>>>> LineNumber = 1
>>>>>>>
>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
>>>>>>> Item[ 1] = "48.68"
>>>>>>> Item[ 2] = "9.23 "
>>>>>>> Item[ 3] = "391.0 61 "
>>>>>>> Item[ 4] = "974.8"
>>>>>>>
>>>>>>> Item[ 5] = "1021.5"
>>>>>>> Item[ 6] = "-99.0"
>>>>>>>
>>>>>>> Offset[ 0] = 0
>>>>>>> Offset[ 1] = 32
>>>>>>> Offset[ 2] = 41
>>>>>>> Offset[ 3] = 47
>>>>>>> Offset[ 4] = 57
>>>>>>>
>>>>>>> Offset[ 5] = 64
>>>>>>> Offset[ 6] = 71
>>>>>>>
>>>>>>> Please Help me....
>>>>>>> Earliest reply will be appreciated.
>>>>>>>
>>>>>>> Thanks in advance for any help you can provide!
>>>>>>>
>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for Post
>>> Processing
>>>>>>> Tool "UPPv1.0 "*
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>
>>>
>>>
>>>
>>
>>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and Point Stat tool
From: Paul Oldenburg
Time: Fri Aug 19 10:27:20 2011
Harvir,
I am concerned that you may not have a set of TMP observations that
will match the TMP data that
is in your model file. Here is the TMP data in your model data file
which does not appear to
contain surface data. Here is the list of levels in the data file:
$ wgrib wrfprs_d01_2011-06-01-000_fcst01.grib1 | grep TMP | perl
-e'while(<>){@g=split(/:/);
printf("%-12s%-12s%-12s%-18s\n", at g[(3,5,6,11)]);}'
TMP kpds6=109 kpds7=1 hybrid lev 1
TMP kpds6=109 kpds7=2 hybrid lev 2
TMP kpds6=100 kpds7=50 50 mb
TMP kpds6=100 kpds7=100 100 mb
TMP kpds6=100 kpds7=150 150 mb
...
TMP kpds6=100 kpds7=950 950 mb
TMP kpds6=100 kpds7=975 975 mb
TMP kpds6=100 kpds7=1000 1000 mb
In your observation text file, 20110601_060000_tt.dat, I see lines
that appear to be surface
observations, but that is based on speculation about what your fields
mean. I assume that the 9th
field (2nd to last), actually contains the pressure level of the
observation, which appears to be
a surface pressure. Is that the case? If so, you may need to
configure your model data
post-processing to dump surface temperature into the model data GRIB
file. I know that this is
confusing, so if you have any questions, please let me know.
Paul
> Harvir,
>
> I see that your model data file has temperature data (TMP), but I do
not see accumulated rainfall
> data among the
> forecast variables. Are you sure that this model data file should
contain rainfall information?
> Usually that has a
> GRIB code of 61 (APCP). Below is a list of the forecast variables
that I see in your model data
> file.
>
> Paul
>
>
> $ wgrib wrfprs_d01_2011-06-01-000_fcst01.grib1 | perl
-e'while(<>){@g=split(/\:/);
> printf("%-12s%-8s\n",$g[4],$g[3]);}'
> | sort -u
> kpds5=11 TMP
> kpds5=17 DPT
> kpds5=1 PRES
> kpds5=20 VIS
> kpds5=211 REFD
> kpds5=212 REFC
> kpds5=2 PRMSL
> kpds5=33 UGRD
> kpds5=34 VGRD
> kpds5=39 VVEL
> kpds5=41 ABSV
> kpds5=51 SPFH
> kpds5=52 RH
> kpds5=7 HGT
>
>
> On 08/19/2011 08:57 AM, Paul Oldenburg wrote:
>> Harvir,
>>
>> Can you please send me the point_stat config file that you are
trying to use? I see in
>> point_stat.sh that this file is
>> on your system at:
>>
>> /gpfs1/home/harvir/SW/METv3.0.1/data/config/PointStatConfig_tmp
>>
>> That will help me understand what you are trying to verify.
>>
>> Thanks,
>>
>> Paul
>>
>>
>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>
>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
>>> Transaction: Ticket created by harvirduphy at gmail.com
>>> Queue: met_help
>>> Subject: Need help for ASCII2NC and Point Stat tool
>>> Owner: Nobody
>>> Requestors: harvirduphy at gmail.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>
>>>
>>> Dear Paul,
>>>
>>> I would like to thank you Paul for your necessary suggestions.
I've
>>> implemented them and now its work but I want use this file (
output from
>>> ascii2nc) for Point Stat tool. I have tried to run point_stat, its
running
>>> and creating some stat files too but stats file having only
headers no
>>> statistics. I'm not getting why its so?
>>> I've transferred data files as well as script file in
>>> incoming/irap/met_help/singh_data. Here I want to run it for TMP
and
>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
>>> 20110601_060000_rf.dat
>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
>>> outfile in GRIB1 format (output from WPPv3.2) named as
>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>>>
>>>
>>>
>>> I want to ask you something on some other data. Suppose I have
some ASCII
>>> data having only 3 columns viz. Latitude, Longitude and Rainfall
(say). Can
>>> we convert it through ascii2nc for MET supported format? If yes,
then what
>>> we can take rest of the columns. I also transferred this kind of
data named
>>> as TRMM_16AUG2011.asc,
>>> TRMM_16AUG2011.dat
>>> I want to use point stat for this data also.
>>>
>>> Thank you in advance.
>>>
>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg}
<met_help at ucar.edu
>>>> wrote:
>>>
>>>> Harvir,
>>>>
>>>> I have identified several problems with the ascii2nc input file
>>>> suflnd_20110601_06z_rrf.dat:
>>>>
>>>> 1. ascii2nc input columns must be space-delimited. Several of
the columns
>>>> in the input file are tab delimited. Do a
>>>> find/replace on the file to change all tabs to spaces.
>>>>
>>>> 2. After you perform step 1, you will see the following error
when you try
>>>> to run ascii2nc:
>>>>
>>>> $ ascii2nc.sh
>>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
>>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
>>>> Processing 11414 observations.
>>>>
>>>>
>>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
>>>>
>>>>
>>>> The reason for this error is that your file appears to contain
accumulated
>>>> precipitation observations (GRIB code 61 in
>>>> column 7). In this case, ascii2nc is expecting an accumulation
interval in
>>>> column 8 in the format HH, for example 06 or
>>>> 24. That is the reason for the error message above.
>>>>
>>>> 3. Column 9 should contain the height above ground of the
observation
>>>> station in meters. For accumulated
>>>> precipitation, this information is ignored by ascii2nc because
this is a
>>>> "surface" observation. We typically use a fill
>>>> value of -9999, indicating invalid data.
>>>>
>>>> 4. Column 10 should contain the observed values. I see valid
observation
>>>> values on some rows, but many are -99.0 which
>>>> I interpret to mean invalid data. In this case, it does not make
sense to
>>>> include invalid observation data, and those
>>>> lines should be deleted.
>>>>
>>>> I took the liberty of making a couple assumptions about your data
and
>>>> trying to adjust the input file that you sent me
>>>> to something that ascii2nc can read. The assumptions I made are:
>>>>
>>>> 1. All of your observations are "surface" observations, so the
message
>>>> type (column 1) was set to ADPSFC
>>>> 2. The accumulation interval is 6 hours (column 8 is set to 06)
>>>> 3. The observation values are in mm, and left as-is
>>>>
>>>> If these assumptions, mainly #2 and #3, are incorrect, please
make the
>>>> appropriate adjustments. I attached the "fixed"
>>>> file to this email for you to review and change as you wish.
Please let me
>>>> know if you have any questions.
>>>>
>>>> Paul
>>>>
>>>>
>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
>>>>>
>>>>>> Dear Paul,
>>>>>>
>>>>>> Accept My apology. Actually yesterday I forgot to attached
th files.
>>>>> However, as per your advice through ftp I have putted the same
attached
>>>>> files there also. and its path is
>>>>> *incoming/irap/met_help/singh_data*
>>>>>
>>>>>> I'm sending you ASCII file and ascii2nc script. Please have
close look
>>>> on
>>>>>> it if am wrong somewhere please let me know.
>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
>>>> not
>>>>>> in pcp_combine ... However, I've run all the test scripts
successfully.
>>>>>> Thanx in advance...
>>>>>>
>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
>>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>>> Harvir,
>>>>>>>
>>>>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
>>>> read
>>>>>>> TRMM data into MET would be to convert the
>>>>>>> data from it's current format into a NetCDF file that has the
same
>>>> format
>>>>>>> as the output of the MET pcp_combine tool. If
>>>>>>> you ran the MET test scripts after you built MET, you should
be able to
>>>>>>> find pcp_combine output files in the sub-folder
>>>>>>> out/pcp_combine under your MET build folder. If you are
successful in
>>>>>>> converting your data in this way, please send us
>>>>>>> your code so that we can make it available for other MET users
who
>>>> might
>>>>>>> want to use TRMM data. I'm sorry the answer to
>>>>>>> this question is not more simple.
>>>>>>>
>>>>>>> Regarding your ascii2nc question, can you please send us the
ascii2nc
>>>>>>> input that is causing your error? Also, what is
>>>>>>> the command line statement that you are using to invoke
ascii2nc?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>>
>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>>>
>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>>> Queue: met_help
>>>>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
>>>> format
>>>>>>>> Owner: Nobody
>>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>>> Status: new
>>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dear Sir /Madam,
>>>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat MET
tool?
>>>>>>> which
>>>>>>>> converter should did I use to convert TRMM Rainfall data (.nc
or .bin)
>>>>>>> to
>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
>>>>>>> (UPPv1.0),
>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>>>> I want to use some ASCII DATA may be output of some models
like GFS
>>>> etc
>>>>>>> ..
>>>>>>>> I have extracted data in the desired 10 columns. I'm trying
to use
>>>>>>> *ascii2nc
>>>>>>>> *this also giving some error like: open_met_ascii() -> line
number 0
>>>>>>> doesn't
>>>>>>>> contain the expected number of colu
>>>>>>>> N_items = 7
>>>>>>>> N_chars = 512
>>>>>>>> N_ints = 100
>>>>>>>> LineNumber = 1
>>>>>>>>
>>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
>>>>>>>> Item[ 1] = "48.68"
>>>>>>>> Item[ 2] = "9.23 "
>>>>>>>> Item[ 3] = "391.0 61 "
>>>>>>>> Item[ 4] = "974.8"
>>>>>>>>
>>>>>>>> Item[ 5] = "1021.5"
>>>>>>>> Item[ 6] = "-99.0"
>>>>>>>>
>>>>>>>> Offset[ 0] = 0
>>>>>>>> Offset[ 1] = 32
>>>>>>>> Offset[ 2] = 41
>>>>>>>> Offset[ 3] = 47
>>>>>>>> Offset[ 4] = 57
>>>>>>>>
>>>>>>>> Offset[ 5] = 64
>>>>>>>> Offset[ 6] = 71
>>>>>>>>
>>>>>>>> Please Help me....
>>>>>>>> Earliest reply will be appreciated.
>>>>>>>>
>>>>>>>> Thanks in advance for any help you can provide!
>>>>>>>>
>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for Post
>>>> Processing
>>>>>>>> Tool "UPPv1.0 "*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: Paul Oldenburg
Time: Mon Aug 22 15:50:51 2011
Harvir,
Regarding your second request related to TRMM data, I wrote a small
script (attached) to convert TRMM_16AUG2011.asc into
a format that is readable by ascii2nc. Then, I plotted the
observation points using the MET utility plot_point_obs and
created the attached image showing the range of points. It is
important to note that I had to make several assumptions
about your observations to do this conversion. The assumptions are:
1. Your observations are surface observations (column 1 - message
type ADPSFC). This does make sense if your data is
accumulated precipitation, which I believe it is.
2. The valid time of your observations is 2011-08-11 00:00:00 UTC.
Obviously, the date is shown in your filename, but
I could not find the valid time.
3. The elevation in msl of all observing points is 0m, which is
definitely a bad assumption, but for "surface"
observations will not matter.
4. The accumulation interval of your observations is 6 hours (column
8). If this is not correct, please modify the
script I attached accordingly.
5. The height in msl of the observation points is 0m (see assumption
3).
Once you have model data that contains APCP with the same accumulation
interval as that of your observations, you should
be able to use this acii2nc-generate observations in point_stat.
Here are the steps to generate the ascii2nc NetCDF obs data file and
plot using the script I am sending. These commands
assume that the MET bin/ sub-folder is on your path.
$ chmod +x trmm_conv.pl
$ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
$ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
$ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15 45 45
120
Please let me know if you have any questions.
Thanks,
Paul
On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
>
> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
> Transaction: Ticket created by harvirduphy at gmail.com
> Queue: met_help
> Subject: Need help for ASCII2NC and Point Stat tool
> Owner: Nobody
> Requestors: harvirduphy at gmail.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>
>
> Dear Paul,
>
> I would like to thank you Paul for your necessary suggestions. I've
> implemented them and now its work but I want use this file ( output
from
> ascii2nc) for Point Stat tool. I have tried to run point_stat, its
running
> and creating some stat files too but stats file having only headers
no
> statistics. I'm not getting why its so?
> I've transferred data files as well as script file in
> incoming/irap/met_help/singh_data. Here I want to run it for TMP and
> Rainfall. The files for TMP are 20110601_060000_tt.dat
> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
> 20110601_060000_rf.dat
> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
> outfile in GRIB1 format (output from WPPv3.2) named as
> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>
>
>
> I want to ask you something on some other data. Suppose I have some
ASCII
> data having only 3 columns viz. Latitude, Longitude and Rainfall
(say). Can
> we convert it through ascii2nc for MET supported format? If yes,
then what
> we can take rest of the columns. I also transferred this kind of
data named
> as TRMM_16AUG2011.asc,
> TRMM_16AUG2011.dat
> I want to use point stat for this data also.
>
> Thank you in advance.
>
> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg}
<met_help at ucar.edu
>> wrote:
>
>> Harvir,
>>
>> I have identified several problems with the ascii2nc input file
>> suflnd_20110601_06z_rrf.dat:
>>
>> 1. ascii2nc input columns must be space-delimited. Several of the
columns
>> in the input file are tab delimited. Do a
>> find/replace on the file to change all tabs to spaces.
>>
>> 2. After you perform step 1, you will see the following error when
you try
>> to run ascii2nc:
>>
>> $ ascii2nc.sh
>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
>> Processing 11414 observations.
>>
>>
>> timestring_to_sec(const char *) -> can't parse time string "974.8"
>>
>>
>> The reason for this error is that your file appears to contain
accumulated
>> precipitation observations (GRIB code 61 in
>> column 7). In this case, ascii2nc is expecting an accumulation
interval in
>> column 8 in the format HH, for example 06 or
>> 24. That is the reason for the error message above.
>>
>> 3. Column 9 should contain the height above ground of the
observation
>> station in meters. For accumulated
>> precipitation, this information is ignored by ascii2nc because this
is a
>> "surface" observation. We typically use a fill
>> value of -9999, indicating invalid data.
>>
>> 4. Column 10 should contain the observed values. I see valid
observation
>> values on some rows, but many are -99.0 which
>> I interpret to mean invalid data. In this case, it does not make
sense to
>> include invalid observation data, and those
>> lines should be deleted.
>>
>> I took the liberty of making a couple assumptions about your data
and
>> trying to adjust the input file that you sent me
>> to something that ascii2nc can read. The assumptions I made are:
>>
>> 1. All of your observations are "surface" observations, so the
message
>> type (column 1) was set to ADPSFC
>> 2. The accumulation interval is 6 hours (column 8 is set to 06)
>> 3. The observation values are in mm, and left as-is
>>
>> If these assumptions, mainly #2 and #3, are incorrect, please make
the
>> appropriate adjustments. I attached the "fixed"
>> file to this email for you to review and change as you wish.
Please let me
>> know if you have any questions.
>>
>> Paul
>>
>>
>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
>>>
>>>> Dear Paul,
>>>>
>>>> Accept My apology. Actually yesterday I forgot to attached th
files.
>>> However, as per your advice through ftp I have putted the same
attached
>>> files there also. and its path is
>>> *incoming/irap/met_help/singh_data*
>>>
>>>> I'm sending you ASCII file and ascii2nc script. Please have close
look
>> on
>>>> it if am wrong somewhere please let me know.
>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
>> not
>>>> in pcp_combine ... However, I've run all the test scripts
successfully.
>>>> Thanx in advance...
>>>>
>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>> Harvir,
>>>>>
>>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
>> read
>>>>> TRMM data into MET would be to convert the
>>>>> data from it's current format into a NetCDF file that has the
same
>> format
>>>>> as the output of the MET pcp_combine tool. If
>>>>> you ran the MET test scripts after you built MET, you should be
able to
>>>>> find pcp_combine output files in the sub-folder
>>>>> out/pcp_combine under your MET build folder. If you are
successful in
>>>>> converting your data in this way, please send us
>>>>> your code so that we can make it available for other MET users
who
>> might
>>>>> want to use TRMM data. I'm sorry the answer to
>>>>> this question is not more simple.
>>>>>
>>>>> Regarding your ascii2nc question, can you please send us the
ascii2nc
>>>>> input that is causing your error? Also, what is
>>>>> the command line statement that you are using to invoke
ascii2nc?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>
>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>> Queue: met_help
>>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
>> format
>>>>>> Owner: Nobody
>>>>>> Requestors: harvirduphy at gmail.com
>>>>>> Status: new
>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>
>>>>>>
>>>>>> Dear Sir /Madam,
>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat MET
tool?
>>>>> which
>>>>>> converter should did I use to convert TRMM Rainfall data (.nc
or .bin)
>>>>> to
>>>>>> GRIB1 format? I've tried copygb through Unified Post Processing
>>>>> (UPPv1.0),
>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>> I want to use some ASCII DATA may be output of some models like
GFS
>> etc
>>>>> ..
>>>>>> I have extracted data in the desired 10 columns. I'm trying to
use
>>>>> *ascii2nc
>>>>>> *this also giving some error like: open_met_ascii() -> line
number 0
>>>>> doesn't
>>>>>> contain the expected number of colu
>>>>>> N_items = 7
>>>>>> N_chars = 512
>>>>>> N_ints = 100
>>>>>> LineNumber = 1
>>>>>>
>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
>>>>>> Item[ 1] = "48.68"
>>>>>> Item[ 2] = "9.23 "
>>>>>> Item[ 3] = "391.0 61 "
>>>>>> Item[ 4] = "974.8"
>>>>>>
>>>>>> Item[ 5] = "1021.5"
>>>>>> Item[ 6] = "-99.0"
>>>>>>
>>>>>> Offset[ 0] = 0
>>>>>> Offset[ 1] = 32
>>>>>> Offset[ 2] = 41
>>>>>> Offset[ 3] = 47
>>>>>> Offset[ 4] = 57
>>>>>>
>>>>>> Offset[ 5] = 64
>>>>>> Offset[ 6] = 71
>>>>>>
>>>>>> Please Help me....
>>>>>> Earliest reply will be appreciated.
>>>>>>
>>>>>> Thanks in advance for any help you can provide!
>>>>>>
>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for Post
>> Processing
>>>>>> Tool "UPPv1.0 "*
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>
>>
>>
>>
>
>
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: Harvir Singh
Time: Mon Aug 22 23:58:42 2011
Thank you Paul
I have transfer the required PointStatConfig_tmp
incoming/irap/met_help/singh_data. I'm trying to implement your
suggestions.
I'll let you know, if I would face any problem.
Thanx
HARVIR
On 19 August 2011 20:27, RAL HelpDesk {for Paul Oldenburg} <
met_help at ucar.edu> wrote:
> Harvir,
>
> Can you please send me the point_stat config file that you are
trying to
> use? I see in point_stat.sh that this file is
> on your system at:
>
> /gpfs1/home/harvir/SW/METv3.0.1/data/config/PointStatConfig_tmp
>
> That will help me understand what you are trying to verify.
>
> Thanks,
>
> Paul
>
>
> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >
> > Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
> > Transaction: Ticket created by harvirduphy at gmail.com
> > Queue: met_help
> > Subject: Need help for ASCII2NC and Point Stat tool
> > Owner: Nobody
> > Requestors: harvirduphy at gmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >
> >
> > Dear Paul,
> >
> > I would like to thank you Paul for your necessary suggestions.
I've
> > implemented them and now its work but I want use this file (
output from
> > ascii2nc) for Point Stat tool. I have tried to run point_stat, its
> running
> > and creating some stat files too but stats file having only
headers no
> > statistics. I'm not getting why its so?
> > I've transferred data files as well as script file in
> > incoming/irap/met_help/singh_data. Here I want to run it for TMP
and
> > Rainfall. The files for TMP are 20110601_060000_tt.dat
> > 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
> > 20110601_060000_rf.dat
> > 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
> > outfile in GRIB1 format (output from WPPv3.2) named as
> > wrfprs_d01_2011-06-01-000_fcst01.grib1.
> >
> >
> >
> > I want to ask you something on some other data. Suppose I have
some ASCII
> > data having only 3 columns viz. Latitude, Longitude and Rainfall
(say).
> Can
> > we convert it through ascii2nc for MET supported format? If yes,
then
> what
> > we can take rest of the columns. I also transferred this kind of
data
> named
> > as TRMM_16AUG2011.asc,
> > TRMM_16AUG2011.dat
> > I want to use point stat for this data also.
> >
> > Thank you in advance.
> >
> > On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
> met_help at ucar.edu
> >> wrote:
> >
> >> Harvir,
> >>
> >> I have identified several problems with the ascii2nc input file
> >> suflnd_20110601_06z_rrf.dat:
> >>
> >> 1. ascii2nc input columns must be space-delimited. Several of
the
> columns
> >> in the input file are tab delimited. Do a
> >> find/replace on the file to change all tabs to spaces.
> >>
> >> 2. After you perform step 1, you will see the following error
when you
> try
> >> to run ascii2nc:
> >>
> >> $ ascii2nc.sh
> >> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
> >> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
> >> Processing 11414 observations.
> >>
> >>
> >> timestring_to_sec(const char *) -> can't parse time string
"974.8"
> >>
> >>
> >> The reason for this error is that your file appears to contain
> accumulated
> >> precipitation observations (GRIB code 61 in
> >> column 7). In this case, ascii2nc is expecting an accumulation
interval
> in
> >> column 8 in the format HH, for example 06 or
> >> 24. That is the reason for the error message above.
> >>
> >> 3. Column 9 should contain the height above ground of the
observation
> >> station in meters. For accumulated
> >> precipitation, this information is ignored by ascii2nc because
this is a
> >> "surface" observation. We typically use a fill
> >> value of -9999, indicating invalid data.
> >>
> >> 4. Column 10 should contain the observed values. I see valid
> observation
> >> values on some rows, but many are -99.0 which
> >> I interpret to mean invalid data. In this case, it does not make
sense
> to
> >> include invalid observation data, and those
> >> lines should be deleted.
> >>
> >> I took the liberty of making a couple assumptions about your data
and
> >> trying to adjust the input file that you sent me
> >> to something that ascii2nc can read. The assumptions I made are:
> >>
> >> 1. All of your observations are "surface" observations, so the
message
> >> type (column 1) was set to ADPSFC
> >> 2. The accumulation interval is 6 hours (column 8 is set to 06)
> >> 3. The observation values are in mm, and left as-is
> >>
> >> If these assumptions, mainly #2 and #3, are incorrect, please
make the
> >> appropriate adjustments. I attached the "fixed"
> >> file to this email for you to review and change as you wish.
Please let
> me
> >> know if you have any questions.
> >>
> >> Paul
> >>
> >>
> >> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
> >>>
> >>>> Dear Paul,
> >>>>
> >>>> Accept My apology. Actually yesterday I forgot to attached
th
> files.
> >>> However, as per your advice through ftp I have putted the same
attached
> >>> files there also. and its path is
> >>> *incoming/irap/met_help/singh_data*
> >>>
> >>>> I'm sending you ASCII file and ascii2nc script. Please have
close look
> >> on
> >>>> it if am wrong somewhere please let me know.
> >>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
> >> not
> >>>> in pcp_combine ... However, I've run all the test scripts
> successfully.
> >>>> Thanx in advance...
> >>>>
> >>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
> >>>> met_help at ucar.edu> wrote:
> >>>>
> >>>>> Harvir,
> >>>>>
> >>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
> >> read
> >>>>> TRMM data into MET would be to convert the
> >>>>> data from it's current format into a NetCDF file that has the
same
> >> format
> >>>>> as the output of the MET pcp_combine tool. If
> >>>>> you ran the MET test scripts after you built MET, you should
be able
> to
> >>>>> find pcp_combine output files in the sub-folder
> >>>>> out/pcp_combine under your MET build folder. If you are
successful
> in
> >>>>> converting your data in this way, please send us
> >>>>> your code so that we can make it available for other MET users
who
> >> might
> >>>>> want to use TRMM data. I'm sorry the answer to
> >>>>> this question is not more simple.
> >>>>>
> >>>>> Regarding your ascii2nc question, can you please send us the
ascii2nc
> >>>>> input that is causing your error? Also, what is
> >>>>> the command line statement that you are using to invoke
ascii2nc?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Paul
> >>>>>
> >>>>>
> >>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>>>
> >>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
> >>>>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>>>> Queue: met_help
> >>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
> >> format
> >>>>>> Owner: Nobody
> >>>>>> Requestors: harvirduphy at gmail.com
> >>>>>> Status: new
> >>>>>> Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
> >>>>>>
> >>>>>>
> >>>>>> Dear Sir /Madam,
> >>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat MET
tool?
> >>>>> which
> >>>>>> converter should did I use to convert TRMM Rainfall data (.nc
or
> .bin)
> >>>>> to
> >>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
> >>>>> (UPPv1.0),
> >>>>>> still I'm unable to convert it for METv3.0.1 usage.
> >>>>>> I want to use some ASCII DATA may be output of some models
like GFS
> >> etc
> >>>>> ..
> >>>>>> I have extracted data in the desired 10 columns. I'm trying
to use
> >>>>> *ascii2nc
> >>>>>> *this also giving some error like: open_met_ascii() -> line
number 0
> >>>>> doesn't
> >>>>>> contain the expected number of colu
> >>>>>> N_items = 7
> >>>>>> N_chars = 512
> >>>>>> N_ints = 100
> >>>>>> LineNumber = 1
> >>>>>>
> >>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
> >>>>>> Item[ 1] = "48.68"
> >>>>>> Item[ 2] = "9.23 "
> >>>>>> Item[ 3] = "391.0 61 "
> >>>>>> Item[ 4] = "974.8"
> >>>>>>
> >>>>>> Item[ 5] = "1021.5"
> >>>>>> Item[ 6] = "-99.0"
> >>>>>>
> >>>>>> Offset[ 0] = 0
> >>>>>> Offset[ 1] = 32
> >>>>>> Offset[ 2] = 41
> >>>>>> Offset[ 3] = 47
> >>>>>> Offset[ 4] = 57
> >>>>>>
> >>>>>> Offset[ 5] = 64
> >>>>>> Offset[ 6] = 71
> >>>>>>
> >>>>>> Please Help me....
> >>>>>> Earliest reply will be appreciated.
> >>>>>>
> >>>>>> Thanks in advance for any help you can provide!
> >>>>>>
> >>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for Post
> >> Processing
> >>>>>> Tool "UPPv1.0 "*
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>
> >>
> >>
> >>
> >
> >
>
>
>
--
* Regards*
**************************************************************
**
* HARVIR SINGH <http://www.harvir007.webs.com>*
* To Know more <http://www.harvir007.webs.com> **
***************************************************************
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: Harvir Singh
Time: Tue Aug 23 00:10:08 2011
Dear Paul,
I've also transferred the forecast file named as
wrfprs_d01_2011-06-01-000_fcst01.grib1 in the same directory viz.
incoming/irap/met_help/singh_data.
On 19 August 2011 20:41, RAL HelpDesk {for Paul Oldenburg} <
met_help at ucar.edu> wrote:
> Harvir,
>
> I see that your model data file has temperature data (TMP), but I do
not
> see accumulated rainfall data among the
> forecast variables. Are you sure that this model data file should
contain
> rainfall information? Usually that has a
> GRIB code of 61 (APCP). Below is a list of the forecast variables
that I
> see in your model data file.
>
> Paul
>
>
> $ wgrib wrfprs_d01_2011-06-01-000_fcst01.grib1 | perl
> -e'while(<>){@g=split(/\:/); printf("%-12s%-8s\n",$g[4],$g[3]);}'
> | sort -u
> kpds5=11 TMP
> kpds5=17 DPT
> kpds5=1 PRES
> kpds5=20 VIS
> kpds5=211 REFD
> kpds5=212 REFC
> kpds5=2 PRMSL
> kpds5=33 UGRD
> kpds5=34 VGRD
> kpds5=39 VVEL
> kpds5=41 ABSV
> kpds5=51 SPFH
> kpds5=52 RH
> kpds5=7 HGT
>
>
> On 08/19/2011 08:57 AM, Paul Oldenburg wrote:
> > Harvir,
> >
> > Can you please send me the point_stat config file that you are
trying to
> use? I see in point_stat.sh that this file is
> > on your system at:
> >
> > /gpfs1/home/harvir/SW/METv3.0.1/data/config/PointStatConfig_tmp
> >
> > That will help me understand what you are trying to verify.
> >
> > Thanks,
> >
> > Paul
> >
> >
> > On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>
> >> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
> >> Transaction: Ticket created by harvirduphy at gmail.com
> >> Queue: met_help
> >> Subject: Need help for ASCII2NC and Point Stat tool
> >> Owner: Nobody
> >> Requestors: harvirduphy at gmail.com
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >>
> >>
> >> Dear Paul,
> >>
> >> I would like to thank you Paul for your necessary suggestions.
I've
> >> implemented them and now its work but I want use this file (
output from
> >> ascii2nc) for Point Stat tool. I have tried to run point_stat,
its
> running
> >> and creating some stat files too but stats file having only
headers no
> >> statistics. I'm not getting why its so?
> >> I've transferred data files as well as script file in
> >> incoming/irap/met_help/singh_data. Here I want to run it for TMP
and
> >> Rainfall. The files for TMP are 20110601_060000_tt.dat
> >> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
> >> 20110601_060000_rf.dat
> >> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
> >> outfile in GRIB1 format (output from WPPv3.2) named as
> >> wrfprs_d01_2011-06-01-000_fcst01.grib1.
> >>
> >>
> >>
> >> I want to ask you something on some other data. Suppose I have
some
> ASCII
> >> data having only 3 columns viz. Latitude, Longitude and Rainfall
(say).
> Can
> >> we convert it through ascii2nc for MET supported format? If yes,
then
> what
> >> we can take rest of the columns. I also transferred this kind of
data
> named
> >> as TRMM_16AUG2011.asc,
> >> TRMM_16AUG2011.dat
> >> I want to use point stat for this data also.
> >>
> >> Thank you in advance.
> >>
> >> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
> met_help at ucar.edu
> >>> wrote:
> >>
> >>> Harvir,
> >>>
> >>> I have identified several problems with the ascii2nc input file
> >>> suflnd_20110601_06z_rrf.dat:
> >>>
> >>> 1. ascii2nc input columns must be space-delimited. Several of
the
> columns
> >>> in the input file are tab delimited. Do a
> >>> find/replace on the file to change all tabs to spaces.
> >>>
> >>> 2. After you perform step 1, you will see the following error
when you
> try
> >>> to run ascii2nc:
> >>>
> >>> $ ascii2nc.sh
> >>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
> >>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
> >>> Processing 11414 observations.
> >>>
> >>>
> >>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
> >>>
> >>>
> >>> The reason for this error is that your file appears to contain
> accumulated
> >>> precipitation observations (GRIB code 61 in
> >>> column 7). In this case, ascii2nc is expecting an accumulation
> interval in
> >>> column 8 in the format HH, for example 06 or
> >>> 24. That is the reason for the error message above.
> >>>
> >>> 3. Column 9 should contain the height above ground of the
observation
> >>> station in meters. For accumulated
> >>> precipitation, this information is ignored by ascii2nc because
this is
> a
> >>> "surface" observation. We typically use a fill
> >>> value of -9999, indicating invalid data.
> >>>
> >>> 4. Column 10 should contain the observed values. I see valid
> observation
> >>> values on some rows, but many are -99.0 which
> >>> I interpret to mean invalid data. In this case, it does not
make sense
> to
> >>> include invalid observation data, and those
> >>> lines should be deleted.
> >>>
> >>> I took the liberty of making a couple assumptions about your
data and
> >>> trying to adjust the input file that you sent me
> >>> to something that ascii2nc can read. The assumptions I made
are:
> >>>
> >>> 1. All of your observations are "surface" observations, so the
message
> >>> type (column 1) was set to ADPSFC
> >>> 2. The accumulation interval is 6 hours (column 8 is set to 06)
> >>> 3. The observation values are in mm, and left as-is
> >>>
> >>> If these assumptions, mainly #2 and #3, are incorrect, please
make the
> >>> appropriate adjustments. I attached the "fixed"
> >>> file to this email for you to review and change as you wish.
Please
> let me
> >>> know if you have any questions.
> >>>
> >>> Paul
> >>>
> >>>
> >>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>
> >>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
> >>>>
> >>>>> Dear Paul,
> >>>>>
> >>>>> Accept My apology. Actually yesterday I forgot to attached
th
> files.
> >>>> However, as per your advice through ftp I have putted the same
> attached
> >>>> files there also. and its path is
> >>>> *incoming/irap/met_help/singh_data*
> >>>>
> >>>>> I'm sending you ASCII file and ascii2nc script. Please have
close
> look
> >>> on
> >>>>> it if am wrong somewhere please let me know.
> >>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
> >>> not
> >>>>> in pcp_combine ... However, I've run all the test scripts
> successfully.
> >>>>> Thanx in advance...
> >>>>>
> >>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
> >>>>> met_help at ucar.edu> wrote:
> >>>>>
> >>>>>> Harvir,
> >>>>>>
> >>>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
> >>> read
> >>>>>> TRMM data into MET would be to convert the
> >>>>>> data from it's current format into a NetCDF file that has the
same
> >>> format
> >>>>>> as the output of the MET pcp_combine tool. If
> >>>>>> you ran the MET test scripts after you built MET, you should
be able
> to
> >>>>>> find pcp_combine output files in the sub-folder
> >>>>>> out/pcp_combine under your MET build folder. If you are
successful
> in
> >>>>>> converting your data in this way, please send us
> >>>>>> your code so that we can make it available for other MET
users who
> >>> might
> >>>>>> want to use TRMM data. I'm sorry the answer to
> >>>>>> this question is not more simple.
> >>>>>>
> >>>>>> Regarding your ascii2nc question, can you please send us the
> ascii2nc
> >>>>>> input that is causing your error? Also, what is
> >>>>>> the command line statement that you are using to invoke
ascii2nc?
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Paul
> >>>>>>
> >>>>>>
> >>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh}
wrote:
> >>>>>>>
> >>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
> >>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>>>>> Queue: met_help
> >>>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
> >>> format
> >>>>>>> Owner: Nobody
> >>>>>>> Requestors: harvirduphy at gmail.com
> >>>>>>> Status: new
> >>>>>>> Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
> >>>>>>>
> >>>>>>>
> >>>>>>> Dear Sir /Madam,
> >>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat
MET tool?
> >>>>>> which
> >>>>>>> converter should did I use to convert TRMM Rainfall data
(.nc or
> .bin)
> >>>>>> to
> >>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
> >>>>>> (UPPv1.0),
> >>>>>>> still I'm unable to convert it for METv3.0.1 usage.
> >>>>>>> I want to use some ASCII DATA may be output of some models
like GFS
> >>> etc
> >>>>>> ..
> >>>>>>> I have extracted data in the desired 10 columns. I'm trying
to use
> >>>>>> *ascii2nc
> >>>>>>> *this also giving some error like: open_met_ascii() -> line
number
> 0
> >>>>>> doesn't
> >>>>>>> contain the expected number of colu
> >>>>>>> N_items = 7
> >>>>>>> N_chars = 512
> >>>>>>> N_ints = 100
> >>>>>>> LineNumber = 1
> >>>>>>>
> >>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
> >>>>>>> Item[ 1] = "48.68"
> >>>>>>> Item[ 2] = "9.23 "
> >>>>>>> Item[ 3] = "391.0 61 "
> >>>>>>> Item[ 4] = "974.8"
> >>>>>>>
> >>>>>>> Item[ 5] = "1021.5"
> >>>>>>> Item[ 6] = "-99.0"
> >>>>>>>
> >>>>>>> Offset[ 0] = 0
> >>>>>>> Offset[ 1] = 32
> >>>>>>> Offset[ 2] = 41
> >>>>>>> Offset[ 3] = 47
> >>>>>>> Offset[ 4] = 57
> >>>>>>>
> >>>>>>> Offset[ 5] = 64
> >>>>>>> Offset[ 6] = 71
> >>>>>>>
> >>>>>>> Please Help me....
> >>>>>>> Earliest reply will be appreciated.
> >>>>>>>
> >>>>>>> Thanks in advance for any help you can provide!
> >>>>>>>
> >>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for
Post
> >>> Processing
> >>>>>>> Tool "UPPv1.0 "*
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >
>
>
>
--
* Regards*
**************************************************************
**
* HARVIR SINGH <http://www.harvir007.webs.com>*
* To Know more <http://www.harvir007.webs.com> **
***************************************************************
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: Harvir Singh
Time: Tue Aug 23 04:55:19 2011
Dear Paul,
I have gone through your assumptions, might seems correct. But I am
having
some problems when I running Point_stat for rainfall, I am getting
some
error as:
$ sh point_stat.sh
ERROR: GCInfo::set_gcinfo() -> unsupported input file type "APCP/A24"
I have changed configure, "APCP/P24" "APCP/P1000", "APCP/P925" etc.
and I ma getting similar type of error. Suggest me how can I rectify
it.
I also tried for GRID_STAT and I got the error as follows
sh Grid_stat.sh
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073465844558
Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
Observation File: TRMM_16AUG2011.nc
Configuration File:
/gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum =
240000, Min = 0, Max = 148.8
ncattinq: ncid 4; varid -1; attname "Projection": Attribute not found
One more thing I want to ask you about plot_point_obs. Actually
plot_point_obs executable does not exits in my METv3.0.1/bin. Suppose
I need
it, then what I have to do?
Thanks
HARVIR
*PS: All the data files and script files have been transferred in
incoming/irap/met_help/singh_data
*
On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
met_help at ucar.edu> wrote:
> Harvir,
>
> Regarding your second request related to TRMM data, I wrote a small
script
> (attached) to convert TRMM_16AUG2011.asc into
> a format that is readable by ascii2nc. Then, I plotted the
observation
> points using the MET utility plot_point_obs and
> created the attached image showing the range of points. It is
important to
> note that I had to make several assumptions
> about your observations to do this conversion. The assumptions are:
>
> 1. Your observations are surface observations (column 1 - message
type
> ADPSFC). This does make sense if your data is
> accumulated precipitation, which I believe it is.
> 2. The valid time of your observations is 2011-08-11 00:00:00 UTC.
> Obviously, the date is shown in your filename, but
> I could not find the valid time.
> 3. The elevation in msl of all observing points is 0m, which is
definitely
> a bad assumption, but for "surface"
> observations will not matter.
> 4. The accumulation interval of your observations is 6 hours
(column 8).
> If this is not correct, please modify the
> script I attached accordingly.
> 5. The height in msl of the observation points is 0m (see
assumption 3).
>
> Once you have model data that contains APCP with the same
accumulation
> interval as that of your observations, you should
> be able to use this acii2nc-generate observations in point_stat.
>
> Here are the steps to generate the ascii2nc NetCDF obs data file and
plot
> using the script I am sending. These commands
> assume that the MET bin/ sub-folder is on your path.
>
> $ chmod +x trmm_conv.pl
> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15 45 45
120
>
> Please let me know if you have any questions.
>
> Thanks,
>
> Paul
>
>
> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >
> > Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
> > Transaction: Ticket created by harvirduphy at gmail.com
> > Queue: met_help
> > Subject: Need help for ASCII2NC and Point Stat tool
> > Owner: Nobody
> > Requestors: harvirduphy at gmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >
> >
> > Dear Paul,
> >
> > I would like to thank you Paul for your necessary suggestions.
I've
> > implemented them and now its work but I want use this file (
output from
> > ascii2nc) for Point Stat tool. I have tried to run point_stat, its
> running
> > and creating some stat files too but stats file having only
headers no
> > statistics. I'm not getting why its so?
> > I've transferred data files as well as script file in
> > incoming/irap/met_help/singh_data. Here I want to run it for TMP
and
> > Rainfall. The files for TMP are 20110601_060000_tt.dat
> > 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
> > 20110601_060000_rf.dat
> > 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
> > outfile in GRIB1 format (output from WPPv3.2) named as
> > wrfprs_d01_2011-06-01-000_fcst01.grib1.
> >
> >
> >
> > I want to ask you something on some other data. Suppose I have
some ASCII
> > data having only 3 columns viz. Latitude, Longitude and Rainfall
(say).
> Can
> > we convert it through ascii2nc for MET supported format? If yes,
then
> what
> > we can take rest of the columns. I also transferred this kind of
data
> named
> > as TRMM_16AUG2011.asc,
> > TRMM_16AUG2011.dat
> > I want to use point stat for this data also.
> >
> > Thank you in advance.
> >
> > On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
> met_help at ucar.edu
> >> wrote:
> >
> >> Harvir,
> >>
> >> I have identified several problems with the ascii2nc input file
> >> suflnd_20110601_06z_rrf.dat:
> >>
> >> 1. ascii2nc input columns must be space-delimited. Several of
the
> columns
> >> in the input file are tab delimited. Do a
> >> find/replace on the file to change all tabs to spaces.
> >>
> >> 2. After you perform step 1, you will see the following error
when you
> try
> >> to run ascii2nc:
> >>
> >> $ ascii2nc.sh
> >> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
> >> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
> >> Processing 11414 observations.
> >>
> >>
> >> timestring_to_sec(const char *) -> can't parse time string
"974.8"
> >>
> >>
> >> The reason for this error is that your file appears to contain
> accumulated
> >> precipitation observations (GRIB code 61 in
> >> column 7). In this case, ascii2nc is expecting an accumulation
interval
> in
> >> column 8 in the format HH, for example 06 or
> >> 24. That is the reason for the error message above.
> >>
> >> 3. Column 9 should contain the height above ground of the
observation
> >> station in meters. For accumulated
> >> precipitation, this information is ignored by ascii2nc because
this is a
> >> "surface" observation. We typically use a fill
> >> value of -9999, indicating invalid data.
> >>
> >> 4. Column 10 should contain the observed values. I see valid
> observation
> >> values on some rows, but many are -99.0 which
> >> I interpret to mean invalid data. In this case, it does not make
sense
> to
> >> include invalid observation data, and those
> >> lines should be deleted.
> >>
> >> I took the liberty of making a couple assumptions about your data
and
> >> trying to adjust the input file that you sent me
> >> to something that ascii2nc can read. The assumptions I made are:
> >>
> >> 1. All of your observations are "surface" observations, so the
message
> >> type (column 1) was set to ADPSFC
> >> 2. The accumulation interval is 6 hours (column 8 is set to 06)
> >> 3. The observation values are in mm, and left as-is
> >>
> >> If these assumptions, mainly #2 and #3, are incorrect, please
make the
> >> appropriate adjustments. I attached the "fixed"
> >> file to this email for you to review and change as you wish.
Please let
> me
> >> know if you have any questions.
> >>
> >> Paul
> >>
> >>
> >> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
> >>>
> >>>> Dear Paul,
> >>>>
> >>>> Accept My apology. Actually yesterday I forgot to attached
th
> files.
> >>> However, as per your advice through ftp I have putted the same
attached
> >>> files there also. and its path is
> >>> *incoming/irap/met_help/singh_data*
> >>>
> >>>> I'm sending you ASCII file and ascii2nc script. Please have
close look
> >> on
> >>>> it if am wrong somewhere please let me know.
> >>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
> >> not
> >>>> in pcp_combine ... However, I've run all the test scripts
> successfully.
> >>>> Thanx in advance...
> >>>>
> >>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
> >>>> met_help at ucar.edu> wrote:
> >>>>
> >>>>> Harvir,
> >>>>>
> >>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
> >> read
> >>>>> TRMM data into MET would be to convert the
> >>>>> data from it's current format into a NetCDF file that has the
same
> >> format
> >>>>> as the output of the MET pcp_combine tool. If
> >>>>> you ran the MET test scripts after you built MET, you should
be able
> to
> >>>>> find pcp_combine output files in the sub-folder
> >>>>> out/pcp_combine under your MET build folder. If you are
successful
> in
> >>>>> converting your data in this way, please send us
> >>>>> your code so that we can make it available for other MET users
who
> >> might
> >>>>> want to use TRMM data. I'm sorry the answer to
> >>>>> this question is not more simple.
> >>>>>
> >>>>> Regarding your ascii2nc question, can you please send us the
ascii2nc
> >>>>> input that is causing your error? Also, what is
> >>>>> the command line statement that you are using to invoke
ascii2nc?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Paul
> >>>>>
> >>>>>
> >>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>>>
> >>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
> >>>>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>>>> Queue: met_help
> >>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
> >> format
> >>>>>> Owner: Nobody
> >>>>>> Requestors: harvirduphy at gmail.com
> >>>>>> Status: new
> >>>>>> Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
> >>>>>>
> >>>>>>
> >>>>>> Dear Sir /Madam,
> >>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat MET
tool?
> >>>>> which
> >>>>>> converter should did I use to convert TRMM Rainfall data (.nc
or
> .bin)
> >>>>> to
> >>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
> >>>>> (UPPv1.0),
> >>>>>> still I'm unable to convert it for METv3.0.1 usage.
> >>>>>> I want to use some ASCII DATA may be output of some models
like GFS
> >> etc
> >>>>> ..
> >>>>>> I have extracted data in the desired 10 columns. I'm trying
to use
> >>>>> *ascii2nc
> >>>>>> *this also giving some error like: open_met_ascii() -> line
number 0
> >>>>> doesn't
> >>>>>> contain the expected number of colu
> >>>>>> N_items = 7
> >>>>>> N_chars = 512
> >>>>>> N_ints = 100
> >>>>>> LineNumber = 1
> >>>>>>
> >>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
> >>>>>> Item[ 1] = "48.68"
> >>>>>> Item[ 2] = "9.23 "
> >>>>>> Item[ 3] = "391.0 61 "
> >>>>>> Item[ 4] = "974.8"
> >>>>>>
> >>>>>> Item[ 5] = "1021.5"
> >>>>>> Item[ 6] = "-99.0"
> >>>>>>
> >>>>>> Offset[ 0] = 0
> >>>>>> Offset[ 1] = 32
> >>>>>> Offset[ 2] = 41
> >>>>>> Offset[ 3] = 47
> >>>>>> Offset[ 4] = 57
> >>>>>>
> >>>>>> Offset[ 5] = 64
> >>>>>> Offset[ 6] = 71
> >>>>>>
> >>>>>> Please Help me....
> >>>>>> Earliest reply will be appreciated.
> >>>>>>
> >>>>>> Thanks in advance for any help you can provide!
> >>>>>>
> >>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for Post
> >> Processing
> >>>>>> Tool "UPPv1.0 "*
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>
> >>
> >>
> >>
> >
> >
>
>
>
--
* Regards*
**************************************************************
**
* HARVIR SINGH <http://www.harvir007.webs.com>*
* To Know more <http://www.harvir007.webs.com> **
***************************************************************
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and Point Stat tool
From: Paul Oldenburg
Time: Tue Aug 23 08:20:36 2011
Harvir,
If you want to enable the compilation of plot_point_obs, edit your MET
Makefile to make the following change on about
line 92:
DISABLE_TOOLS = 0
Then, run make from the top-level in MET and it should build
plot_point_obs and put the executable in the bin/ sub-folder.
Paul
On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>
> Dear Paul,
> I have gone through your assumptions, might seems correct. But I am
having
> some problems when I running Point_stat for rainfall, I am getting
some
> error as:
>
> $ sh point_stat.sh
> ERROR: GCInfo::set_gcinfo() -> unsupported input file type
"APCP/A24"
> I have changed configure, "APCP/P24" "APCP/P1000", "APCP/P925" etc.
> and I ma getting similar type of error. Suggest me how can I rectify
it.
>
>
> I also tried for GRID_STAT and I got the error as follows
> sh Grid_stat.sh
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073465844558
> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
> Observation File: TRMM_16AUG2011.nc
> Configuration File:
> /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
> GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum =
> 240000, Min = 0, Max = 148.8
> ncattinq: ncid 4; varid -1; attname "Projection": Attribute not
found
>
>
> One more thing I want to ask you about plot_point_obs. Actually
> plot_point_obs executable does not exits in my METv3.0.1/bin.
Suppose I need
> it, then what I have to do?
>
> Thanks
> HARVIR
>
> *PS: All the data files and script files have been transferred in
> incoming/irap/met_help/singh_data
> *
> On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
> met_help at ucar.edu> wrote:
>
>> Harvir,
>>
>> Regarding your second request related to TRMM data, I wrote a small
script
>> (attached) to convert TRMM_16AUG2011.asc into
>> a format that is readable by ascii2nc. Then, I plotted the
observation
>> points using the MET utility plot_point_obs and
>> created the attached image showing the range of points. It is
important to
>> note that I had to make several assumptions
>> about your observations to do this conversion. The assumptions
are:
>>
>> 1. Your observations are surface observations (column 1 - message
type
>> ADPSFC). This does make sense if your data is
>> accumulated precipitation, which I believe it is.
>> 2. The valid time of your observations is 2011-08-11 00:00:00 UTC.
>> Obviously, the date is shown in your filename, but
>> I could not find the valid time.
>> 3. The elevation in msl of all observing points is 0m, which is
definitely
>> a bad assumption, but for "surface"
>> observations will not matter.
>> 4. The accumulation interval of your observations is 6 hours
(column 8).
>> If this is not correct, please modify the
>> script I attached accordingly.
>> 5. The height in msl of the observation points is 0m (see
assumption 3).
>>
>> Once you have model data that contains APCP with the same
accumulation
>> interval as that of your observations, you should
>> be able to use this acii2nc-generate observations in point_stat.
>>
>> Here are the steps to generate the ascii2nc NetCDF obs data file
and plot
>> using the script I am sending. These commands
>> assume that the MET bin/ sub-folder is on your path.
>>
>> $ chmod +x trmm_conv.pl
>> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
>> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
>> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15 45 45
120
>>
>> Please let me know if you have any questions.
>>
>> Thanks,
>>
>> Paul
>>
>>
>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>
>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
>>> Transaction: Ticket created by harvirduphy at gmail.com
>>> Queue: met_help
>>> Subject: Need help for ASCII2NC and Point Stat tool
>>> Owner: Nobody
>>> Requestors: harvirduphy at gmail.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>
>>>
>>> Dear Paul,
>>>
>>> I would like to thank you Paul for your necessary suggestions.
I've
>>> implemented them and now its work but I want use this file (
output from
>>> ascii2nc) for Point Stat tool. I have tried to run point_stat, its
>> running
>>> and creating some stat files too but stats file having only
headers no
>>> statistics. I'm not getting why its so?
>>> I've transferred data files as well as script file in
>>> incoming/irap/met_help/singh_data. Here I want to run it for TMP
and
>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
>>> 20110601_060000_rf.dat
>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
>>> outfile in GRIB1 format (output from WPPv3.2) named as
>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>>>
>>>
>>>
>>> I want to ask you something on some other data. Suppose I have
some ASCII
>>> data having only 3 columns viz. Latitude, Longitude and Rainfall
(say).
>> Can
>>> we convert it through ascii2nc for MET supported format? If yes,
then
>> what
>>> we can take rest of the columns. I also transferred this kind of
data
>> named
>>> as TRMM_16AUG2011.asc,
>>> TRMM_16AUG2011.dat
>>> I want to use point stat for this data also.
>>>
>>> Thank you in advance.
>>>
>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
>> met_help at ucar.edu
>>>> wrote:
>>>
>>>> Harvir,
>>>>
>>>> I have identified several problems with the ascii2nc input file
>>>> suflnd_20110601_06z_rrf.dat:
>>>>
>>>> 1. ascii2nc input columns must be space-delimited. Several of
the
>> columns
>>>> in the input file are tab delimited. Do a
>>>> find/replace on the file to change all tabs to spaces.
>>>>
>>>> 2. After you perform step 1, you will see the following error
when you
>> try
>>>> to run ascii2nc:
>>>>
>>>> $ ascii2nc.sh
>>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
>>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
>>>> Processing 11414 observations.
>>>>
>>>>
>>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
>>>>
>>>>
>>>> The reason for this error is that your file appears to contain
>> accumulated
>>>> precipitation observations (GRIB code 61 in
>>>> column 7). In this case, ascii2nc is expecting an accumulation
interval
>> in
>>>> column 8 in the format HH, for example 06 or
>>>> 24. That is the reason for the error message above.
>>>>
>>>> 3. Column 9 should contain the height above ground of the
observation
>>>> station in meters. For accumulated
>>>> precipitation, this information is ignored by ascii2nc because
this is a
>>>> "surface" observation. We typically use a fill
>>>> value of -9999, indicating invalid data.
>>>>
>>>> 4. Column 10 should contain the observed values. I see valid
>> observation
>>>> values on some rows, but many are -99.0 which
>>>> I interpret to mean invalid data. In this case, it does not make
sense
>> to
>>>> include invalid observation data, and those
>>>> lines should be deleted.
>>>>
>>>> I took the liberty of making a couple assumptions about your data
and
>>>> trying to adjust the input file that you sent me
>>>> to something that ascii2nc can read. The assumptions I made are:
>>>>
>>>> 1. All of your observations are "surface" observations, so the
message
>>>> type (column 1) was set to ADPSFC
>>>> 2. The accumulation interval is 6 hours (column 8 is set to 06)
>>>> 3. The observation values are in mm, and left as-is
>>>>
>>>> If these assumptions, mainly #2 and #3, are incorrect, please
make the
>>>> appropriate adjustments. I attached the "fixed"
>>>> file to this email for you to review and change as you wish.
Please let
>> me
>>>> know if you have any questions.
>>>>
>>>> Paul
>>>>
>>>>
>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
>>>>>
>>>>>> Dear Paul,
>>>>>>
>>>>>> Accept My apology. Actually yesterday I forgot to attached
th
>> files.
>>>>> However, as per your advice through ftp I have putted the same
attached
>>>>> files there also. and its path is
>>>>> *incoming/irap/met_help/singh_data*
>>>>>
>>>>>> I'm sending you ASCII file and ascii2nc script. Please have
close look
>>>> on
>>>>>> it if am wrong somewhere please let me know.
>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
>>>> not
>>>>>> in pcp_combine ... However, I've run all the test scripts
>> successfully.
>>>>>> Thanx in advance...
>>>>>>
>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
>>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>>> Harvir,
>>>>>>>
>>>>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
>>>> read
>>>>>>> TRMM data into MET would be to convert the
>>>>>>> data from it's current format into a NetCDF file that has the
same
>>>> format
>>>>>>> as the output of the MET pcp_combine tool. If
>>>>>>> you ran the MET test scripts after you built MET, you should
be able
>> to
>>>>>>> find pcp_combine output files in the sub-folder
>>>>>>> out/pcp_combine under your MET build folder. If you are
successful
>> in
>>>>>>> converting your data in this way, please send us
>>>>>>> your code so that we can make it available for other MET users
who
>>>> might
>>>>>>> want to use TRMM data. I'm sorry the answer to
>>>>>>> this question is not more simple.
>>>>>>>
>>>>>>> Regarding your ascii2nc question, can you please send us the
ascii2nc
>>>>>>> input that is causing your error? Also, what is
>>>>>>> the command line statement that you are using to invoke
ascii2nc?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>>
>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>>>
>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>>> Queue: met_help
>>>>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
>>>> format
>>>>>>>> Owner: Nobody
>>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>>> Status: new
>>>>>>>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dear Sir /Madam,
>>>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat MET
tool?
>>>>>>> which
>>>>>>>> converter should did I use to convert TRMM Rainfall data (.nc
or
>> .bin)
>>>>>>> to
>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
>>>>>>> (UPPv1.0),
>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>>>> I want to use some ASCII DATA may be output of some models
like GFS
>>>> etc
>>>>>>> ..
>>>>>>>> I have extracted data in the desired 10 columns. I'm trying
to use
>>>>>>> *ascii2nc
>>>>>>>> *this also giving some error like: open_met_ascii() -> line
number 0
>>>>>>> doesn't
>>>>>>>> contain the expected number of colu
>>>>>>>> N_items = 7
>>>>>>>> N_chars = 512
>>>>>>>> N_ints = 100
>>>>>>>> LineNumber = 1
>>>>>>>>
>>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
>>>>>>>> Item[ 1] = "48.68"
>>>>>>>> Item[ 2] = "9.23 "
>>>>>>>> Item[ 3] = "391.0 61 "
>>>>>>>> Item[ 4] = "974.8"
>>>>>>>>
>>>>>>>> Item[ 5] = "1021.5"
>>>>>>>> Item[ 6] = "-99.0"
>>>>>>>>
>>>>>>>> Offset[ 0] = 0
>>>>>>>> Offset[ 1] = 32
>>>>>>>> Offset[ 2] = 41
>>>>>>>> Offset[ 3] = 47
>>>>>>>> Offset[ 4] = 57
>>>>>>>>
>>>>>>>> Offset[ 5] = 64
>>>>>>>> Offset[ 6] = 71
>>>>>>>>
>>>>>>>> Please Help me....
>>>>>>>> Earliest reply will be appreciated.
>>>>>>>>
>>>>>>>> Thanks in advance for any help you can provide!
>>>>>>>>
>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for Post
>>>> Processing
>>>>>>>> Tool "UPPv1.0 "*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and Point Stat tool
From: Paul Oldenburg
Time: Tue Aug 23 10:25:53 2011
Harvir,
There are a couple points to discuss before we continue. First, I
recommend that you use point_stat at this time to
perform verification on your model data. The reason is that
converting your observation data to a format that grid_stat
can read is more difficult because it must contain projection
information that matches your model data projection. Why
do you want to run grid_stat? Are you interested in calculating
neighborhood statistics or just basic verification
statistics like RMSE and the Equitable Threat Score? If you only want
basic statistics, point_stat will work just fine.
Second, for rainfall data, you said that the assumptions that I have
made about your observations are correct. If that
is the case, and the accumulation interval of your observation data is
6 hours, then there is a mis-match with the
accumulation interval in your model data, which is 24 hours (see 0-
24hr acc):
$ wgrib wrfprs_d01_2011-06-01_000_fcst01.grib1 | grep APCP
252:323426:d=11053100:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr acc:NAve=0
If you are sure that your rainfall observations have an accumulation
interval of 6 hours, then you must combine four of
your 6 hour observation files together to form rainfall observations
with an accumulation interval of 24 hours. A good
way to do this is to use the MET tool pcp_combine after you run
ascii2nc on your rainfall observation files.
Finally, I tried to apply the model data that you gave me with surface
temperature to the observation data you gave me
last week, 20110601_060000_tt.dat. I put the files involved in this
experiment are on the RAL FTP site in the archive
ftp.rap.ucar.edu:/incoming/irap/met_help/singh_data/PointStatVx.tar.gz.
If you download this and unpack it, you should
be able to reproduce the output file that I included,
point_stat_240000L_20110601_000000V.stat. This file contains
basic verification statistics, like continuous and categorical
statistics. I used the threshold 273.15 K, which is 0 C.
Note that for this experiment to be valid, I had to open the time
window to +/- 6 hours because your model data is valid
at 20110601_000000 and your observations are valid at 20110601_060000.
This may or may not be an acceptable compromise
to your for your work. Also, I assumed that the temperature
observations were all on the surface (message type ADPUPA).
It also looked like your observations were in degrees Celsius, so I
converted them to Kelvin. Please let me know if
you have any questions or trouble following my example. Please look
carefully at the files included in the archive,
especially PointStatConfig_tmp and point_stat.sh.
Paul
$ tar xzvf PointStatVx.tar.gz
$ cat obs_20110601_060000_tt.dat | ./obs_conv.pl >
obs_20110601_060000_tt.txt
$ ascii2nc obs_20110601_060000_tt.txt obs_20110601_060000_tt.nc -v 3
Creating NetCDF Observation file: obs_20110601_060000_tt.nc
Reading ASCII Observation file: obs_20110601_060000_tt.txt
Processing 1099 observations.
Finished processing 1099 observations.
$ ./point_stat.sh
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=231822158
Forecast File: wrfprs_d01_2011-06-01_000_fcst01.grib1
Climatology File: none
Configuration File: PointStatConfig_tmp
Observation File: obs_20110601_060000_tt.nc
--------------------------------------------------------------------------------
Reading records for TMP/Z2.
GRIB Record 244: Init = 20110531_000000, Valid = 20110601_000000,
Accum = 000000, Min = 292.579, Max = 303.165
For TMP/Z2 found 1 forecast levels and 0 climatology levels.
--------------------------------------------------------------------------------
Searching 1099 observations from 1099 PrepBufr messages.
--------------------------------------------------------------------------------
Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC, over
region FULL, for interpolation method UW_MEAN(1),
using 58 pairs.
Number of matched pairs = 58
Observations processed = 1099
Rejected: GRIB code = 0
Rejected: valid time = 0
Rejected: bad obs value = 0
Rejected: off the grid = 1041
Rejected: level mismatch = 0
Rejected: message type = 0
Rejected: masking region = 0
Rejected: bad fcst value = 0
Computing Categorical Statistics.
Computing Continuous Statistics.
Computing Scalar Partial Sums.
--------------------------------------------------------------------------------
Output file: ./point_stat_240000L_20110601_000000V.stat
On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>
> Dear Paul,
> I have gone through your assumptions, might seems correct. But I am
having
> some problems when I running Point_stat for rainfall, I am getting
some
> error as:
>
> $ sh point_stat.sh
> ERROR: GCInfo::set_gcinfo() -> unsupported input file type
"APCP/A24"
> I have changed configure, "APCP/P24" "APCP/P1000", "APCP/P925" etc.
> and I ma getting similar type of error. Suggest me how can I rectify
it.
>
>
> I also tried for GRID_STAT and I got the error as follows
> sh Grid_stat.sh
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073465844558
> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
> Observation File: TRMM_16AUG2011.nc
> Configuration File:
> /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
> GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum =
> 240000, Min = 0, Max = 148.8
> ncattinq: ncid 4; varid -1; attname "Projection": Attribute not
found
>
>
> One more thing I want to ask you about plot_point_obs. Actually
> plot_point_obs executable does not exits in my METv3.0.1/bin.
Suppose I need
> it, then what I have to do?
>
> Thanks
> HARVIR
>
> *PS: All the data files and script files have been transferred in
> incoming/irap/met_help/singh_data
> *
> On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
> met_help at ucar.edu> wrote:
>
>> Harvir,
>>
>> Regarding your second request related to TRMM data, I wrote a small
script
>> (attached) to convert TRMM_16AUG2011.asc into
>> a format that is readable by ascii2nc. Then, I plotted the
observation
>> points using the MET utility plot_point_obs and
>> created the attached image showing the range of points. It is
important to
>> note that I had to make several assumptions
>> about your observations to do this conversion. The assumptions
are:
>>
>> 1. Your observations are surface observations (column 1 - message
type
>> ADPSFC). This does make sense if your data is
>> accumulated precipitation, which I believe it is.
>> 2. The valid time of your observations is 2011-08-11 00:00:00 UTC.
>> Obviously, the date is shown in your filename, but
>> I could not find the valid time.
>> 3. The elevation in msl of all observing points is 0m, which is
definitely
>> a bad assumption, but for "surface"
>> observations will not matter.
>> 4. The accumulation interval of your observations is 6 hours
(column 8).
>> If this is not correct, please modify the
>> script I attached accordingly.
>> 5. The height in msl of the observation points is 0m (see
assumption 3).
>>
>> Once you have model data that contains APCP with the same
accumulation
>> interval as that of your observations, you should
>> be able to use this acii2nc-generate observations in point_stat.
>>
>> Here are the steps to generate the ascii2nc NetCDF obs data file
and plot
>> using the script I am sending. These commands
>> assume that the MET bin/ sub-folder is on your path.
>>
>> $ chmod +x trmm_conv.pl
>> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
>> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
>> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15 45 45
120
>>
>> Please let me know if you have any questions.
>>
>> Thanks,
>>
>> Paul
>>
>>
>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>
>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
>>> Transaction: Ticket created by harvirduphy at gmail.com
>>> Queue: met_help
>>> Subject: Need help for ASCII2NC and Point Stat tool
>>> Owner: Nobody
>>> Requestors: harvirduphy at gmail.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>
>>>
>>> Dear Paul,
>>>
>>> I would like to thank you Paul for your necessary suggestions.
I've
>>> implemented them and now its work but I want use this file (
output from
>>> ascii2nc) for Point Stat tool. I have tried to run point_stat, its
>> running
>>> and creating some stat files too but stats file having only
headers no
>>> statistics. I'm not getting why its so?
>>> I've transferred data files as well as script file in
>>> incoming/irap/met_help/singh_data. Here I want to run it for TMP
and
>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
>>> 20110601_060000_rf.dat
>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
>>> outfile in GRIB1 format (output from WPPv3.2) named as
>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>>>
>>>
>>>
>>> I want to ask you something on some other data. Suppose I have
some ASCII
>>> data having only 3 columns viz. Latitude, Longitude and Rainfall
(say).
>> Can
>>> we convert it through ascii2nc for MET supported format? If yes,
then
>> what
>>> we can take rest of the columns. I also transferred this kind of
data
>> named
>>> as TRMM_16AUG2011.asc,
>>> TRMM_16AUG2011.dat
>>> I want to use point stat for this data also.
>>>
>>> Thank you in advance.
>>>
>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
>> met_help at ucar.edu
>>>> wrote:
>>>
>>>> Harvir,
>>>>
>>>> I have identified several problems with the ascii2nc input file
>>>> suflnd_20110601_06z_rrf.dat:
>>>>
>>>> 1. ascii2nc input columns must be space-delimited. Several of
the
>> columns
>>>> in the input file are tab delimited. Do a
>>>> find/replace on the file to change all tabs to spaces.
>>>>
>>>> 2. After you perform step 1, you will see the following error
when you
>> try
>>>> to run ascii2nc:
>>>>
>>>> $ ascii2nc.sh
>>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
>>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
>>>> Processing 11414 observations.
>>>>
>>>>
>>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
>>>>
>>>>
>>>> The reason for this error is that your file appears to contain
>> accumulated
>>>> precipitation observations (GRIB code 61 in
>>>> column 7). In this case, ascii2nc is expecting an accumulation
interval
>> in
>>>> column 8 in the format HH, for example 06 or
>>>> 24. That is the reason for the error message above.
>>>>
>>>> 3. Column 9 should contain the height above ground of the
observation
>>>> station in meters. For accumulated
>>>> precipitation, this information is ignored by ascii2nc because
this is a
>>>> "surface" observation. We typically use a fill
>>>> value of -9999, indicating invalid data.
>>>>
>>>> 4. Column 10 should contain the observed values. I see valid
>> observation
>>>> values on some rows, but many are -99.0 which
>>>> I interpret to mean invalid data. In this case, it does not make
sense
>> to
>>>> include invalid observation data, and those
>>>> lines should be deleted.
>>>>
>>>> I took the liberty of making a couple assumptions about your data
and
>>>> trying to adjust the input file that you sent me
>>>> to something that ascii2nc can read. The assumptions I made are:
>>>>
>>>> 1. All of your observations are "surface" observations, so the
message
>>>> type (column 1) was set to ADPSFC
>>>> 2. The accumulation interval is 6 hours (column 8 is set to 06)
>>>> 3. The observation values are in mm, and left as-is
>>>>
>>>> If these assumptions, mainly #2 and #3, are incorrect, please
make the
>>>> appropriate adjustments. I attached the "fixed"
>>>> file to this email for you to review and change as you wish.
Please let
>> me
>>>> know if you have any questions.
>>>>
>>>> Paul
>>>>
>>>>
>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
>>>>>
>>>>>> Dear Paul,
>>>>>>
>>>>>> Accept My apology. Actually yesterday I forgot to attached
th
>> files.
>>>>> However, as per your advice through ftp I have putted the same
attached
>>>>> files there also. and its path is
>>>>> *incoming/irap/met_help/singh_data*
>>>>>
>>>>>> I'm sending you ASCII file and ascii2nc script. Please have
close look
>>>> on
>>>>>> it if am wrong somewhere please let me know.
>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
>>>> not
>>>>>> in pcp_combine ... However, I've run all the test scripts
>> successfully.
>>>>>> Thanx in advance...
>>>>>>
>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
>>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>>> Harvir,
>>>>>>>
>>>>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
>>>> read
>>>>>>> TRMM data into MET would be to convert the
>>>>>>> data from it's current format into a NetCDF file that has the
same
>>>> format
>>>>>>> as the output of the MET pcp_combine tool. If
>>>>>>> you ran the MET test scripts after you built MET, you should
be able
>> to
>>>>>>> find pcp_combine output files in the sub-folder
>>>>>>> out/pcp_combine under your MET build folder. If you are
successful
>> in
>>>>>>> converting your data in this way, please send us
>>>>>>> your code so that we can make it available for other MET users
who
>>>> might
>>>>>>> want to use TRMM data. I'm sorry the answer to
>>>>>>> this question is not more simple.
>>>>>>>
>>>>>>> Regarding your ascii2nc question, can you please send us the
ascii2nc
>>>>>>> input that is causing your error? Also, what is
>>>>>>> the command line statement that you are using to invoke
ascii2nc?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>>
>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>>>
>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>>> Queue: met_help
>>>>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
>>>> format
>>>>>>>> Owner: Nobody
>>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>>> Status: new
>>>>>>>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dear Sir /Madam,
>>>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat MET
tool?
>>>>>>> which
>>>>>>>> converter should did I use to convert TRMM Rainfall data (.nc
or
>> .bin)
>>>>>>> to
>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
>>>>>>> (UPPv1.0),
>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>>>> I want to use some ASCII DATA may be output of some models
like GFS
>>>> etc
>>>>>>> ..
>>>>>>>> I have extracted data in the desired 10 columns. I'm trying
to use
>>>>>>> *ascii2nc
>>>>>>>> *this also giving some error like: open_met_ascii() -> line
number 0
>>>>>>> doesn't
>>>>>>>> contain the expected number of colu
>>>>>>>> N_items = 7
>>>>>>>> N_chars = 512
>>>>>>>> N_ints = 100
>>>>>>>> LineNumber = 1
>>>>>>>>
>>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
>>>>>>>> Item[ 1] = "48.68"
>>>>>>>> Item[ 2] = "9.23 "
>>>>>>>> Item[ 3] = "391.0 61 "
>>>>>>>> Item[ 4] = "974.8"
>>>>>>>>
>>>>>>>> Item[ 5] = "1021.5"
>>>>>>>> Item[ 6] = "-99.0"
>>>>>>>>
>>>>>>>> Offset[ 0] = 0
>>>>>>>> Offset[ 1] = 32
>>>>>>>> Offset[ 2] = 41
>>>>>>>> Offset[ 3] = 47
>>>>>>>> Offset[ 4] = 57
>>>>>>>>
>>>>>>>> Offset[ 5] = 64
>>>>>>>> Offset[ 6] = 71
>>>>>>>>
>>>>>>>> Please Help me....
>>>>>>>> Earliest reply will be appreciated.
>>>>>>>>
>>>>>>>> Thanks in advance for any help you can provide!
>>>>>>>>
>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for Post
>>>> Processing
>>>>>>>> Tool "UPPv1.0 "*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: Harvir Singh
Time: Fri Aug 26 07:02:04 2011
Dear Paul,
Thank You for your prompt reply. Ofcourse yes I've to compute Threat
score
too.. For this point of time I just want to go with point stat. and
I'm
trying it for WRF ARW core (output data) it. I've tried point_stat,
its
running but giving headers only in stat txt files. I'm unable to
rectify it.
Please help me to come out of this problem. I've transfrred data in
incoming/irap/met_help/singh_data
thanx in advance!
---------- Forwarded message ----------
From: RAL HelpDesk {for Paul Oldenburg} <met_help at ucar.edu>
Date: 23 August 2011 21:55
Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and Point
Stat
tool
To: harvirduphy at gmail.com
Harvir,
There are a couple points to discuss before we continue. First, I
recommend
that you use point_stat at this time to
perform verification on your model data. The reason is that
converting your
observation data to a format that grid_stat
can read is more difficult because it must contain projection
information
that matches your model data projection. Why
do you want to run grid_stat? Are you interested in calculating
neighborhood statistics or just basic verification
statistics like RMSE and the Equitable Threat Score? If you only want
basic
statistics, point_stat will work just fine.
Second, for rainfall data, you said that the assumptions that I have
made
about your observations are correct. If that
is the case, and the accumulation interval of your observation data is
6
hours, then there is a mis-match with the
accumulation interval in your model data, which is 24 hours (see 0-
24hr
acc):
$ wgrib wrfprs_d01_2011-06-01_000_fcst01.grib1 | grep APCP
252:323426:d=11053100:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
acc:NAve=0
If you are sure that your rainfall observations have an accumulation
interval of 6 hours, then you must combine four of
your 6 hour observation files together to form rainfall observations
with an
accumulation interval of 24 hours. A good
way to do this is to use the MET tool pcp_combine after you run
ascii2nc on
your rainfall observation files.
Finally, I tried to apply the model data that you gave me with surface
temperature to the observation data you gave me
last week, 20110601_060000_tt.dat. I put the files involved in this
experiment are on the RAL FTP site in the archive
ftp.rap.ucar.edu:/incoming/irap/met_help/singh_data/PointStatVx.tar.gz.
If
you download this and unpack it, you should
be able to reproduce the output file that I included,
point_stat_240000L_20110601_000000V.stat. This file contains
basic verification statistics, like continuous and categorical
statistics.
I used the threshold 273.15 K, which is 0 C.
Note that for this experiment to be valid, I had to open the time
window to
+/- 6 hours because your model data is valid
at 20110601_000000 and your observations are valid at 20110601_060000.
This
may or may not be an acceptable compromise
to your for your work. Also, I assumed that the temperature
observations
were all on the surface (message type ADPUPA).
It also looked like your observations were in degrees Celsius, so I
converted them to Kelvin. Please let me know if
you have any questions or trouble following my example. Please look
carefully at the files included in the archive,
especially PointStatConfig_tmp and point_stat.sh.
Paul
$ tar xzvf PointStatVx.tar.gz
$ cat obs_20110601_060000_tt.dat | ./obs_conv.pl >
obs_20110601_060000_tt.txt
$ ascii2nc obs_20110601_060000_tt.txt obs_20110601_060000_tt.nc -v 3
Creating NetCDF Observation file: obs_20110601_060000_tt.nc
Reading ASCII Observation file: obs_20110601_060000_tt.txt
Processing 1099 observations.
Finished processing 1099 observations.
$ ./point_stat.sh
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=231822158
Forecast File: wrfprs_d01_2011-06-01_000_fcst01.grib1
Climatology File: none
Configuration File: PointStatConfig_tmp
Observation File: obs_20110601_060000_tt.nc
--------------------------------------------------------------------------------
Reading records for TMP/Z2.
GRIB Record 244: Init = 20110531_000000, Valid = 20110601_000000,
Accum =
000000, Min = 292.579, Max = 303.165
For TMP/Z2 found 1 forecast levels and 0 climatology levels.
--------------------------------------------------------------------------------
Searching 1099 observations from 1099 PrepBufr messages.
--------------------------------------------------------------------------------
Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC, over
region
FULL, for interpolation method UW_MEAN(1),
using 58 pairs.
Number of matched pairs = 58
Observations processed = 1099
Rejected: GRIB code = 0
Rejected: valid time = 0
Rejected: bad obs value = 0
Rejected: off the grid = 1041
Rejected: level mismatch = 0
Rejected: message type = 0
Rejected: masking region = 0
Rejected: bad fcst value = 0
Computing Categorical Statistics.
Computing Continuous Statistics.
Computing Scalar Partial Sums.
--------------------------------------------------------------------------------
Output file: ./point_stat_240000L_20110601_000000V.stat
On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>
> Dear Paul,
> I have gone through your assumptions, might seems correct. But I am
having
> some problems when I running Point_stat for rainfall, I am getting
some
> error as:
>
> $ sh point_stat.sh
> ERROR: GCInfo::set_gcinfo() -> unsupported input file type
"APCP/A24"
> I have changed configure, "APCP/P24" "APCP/P1000", "APCP/P925" etc.
> and I ma getting similar type of error. Suggest me how can I rectify
it.
>
>
> I also tried for GRID_STAT and I got the error as follows
> sh Grid_stat.sh
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073465844558
> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
> Observation File: TRMM_16AUG2011.nc
> Configuration File:
> /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
> GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum =
> 240000, Min = 0, Max = 148.8
> ncattinq: ncid 4; varid -1; attname "Projection": Attribute not
found
>
>
> One more thing I want to ask you about plot_point_obs. Actually
> plot_point_obs executable does not exits in my METv3.0.1/bin.
Suppose I
need
> it, then what I have to do?
>
> Thanks
> HARVIR
>
> *PS: All the data files and script files have been transferred in
> incoming/irap/met_help/singh_data
> *
> On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
> met_help at ucar.edu> wrote:
>
>> Harvir,
>>
>> Regarding your second request related to TRMM data, I wrote a small
script
>> (attached) to convert TRMM_16AUG2011.asc into
>> a format that is readable by ascii2nc. Then, I plotted the
observation
>> points using the MET utility plot_point_obs and
>> created the attached image showing the range of points. It is
important
to
>> note that I had to make several assumptions
>> about your observations to do this conversion. The assumptions
are:
>>
>> 1. Your observations are surface observations (column 1 - message
type
>> ADPSFC). This does make sense if your data is
>> accumulated precipitation, which I believe it is.
>> 2. The valid time of your observations is 2011-08-11 00:00:00 UTC.
>> Obviously, the date is shown in your filename, but
>> I could not find the valid time.
>> 3. The elevation in msl of all observing points is 0m, which is
definitely
>> a bad assumption, but for "surface"
>> observations will not matter.
>> 4. The accumulation interval of your observations is 6 hours
(column 8).
>> If this is not correct, please modify the
>> script I attached accordingly.
>> 5. The height in msl of the observation points is 0m (see
assumption 3).
>>
>> Once you have model data that contains APCP with the same
accumulation
>> interval as that of your observations, you should
>> be able to use this acii2nc-generate observations in point_stat.
>>
>> Here are the steps to generate the ascii2nc NetCDF obs data file
and plot
>> using the script I am sending. These commands
>> assume that the MET bin/ sub-folder is on your path.
>>
>> $ chmod +x trmm_conv.pl
>> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
>> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
>> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15 45 45
120
>>
>> Please let me know if you have any questions.
>>
>> Thanks,
>>
>> Paul
>>
>>
>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>
>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
>>> Transaction: Ticket created by harvirduphy at gmail.com
>>> Queue: met_help
>>> Subject: Need help for ASCII2NC and Point Stat tool
>>> Owner: Nobody
>>> Requestors: harvirduphy at gmail.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>
>>>
>>> Dear Paul,
>>>
>>> I would like to thank you Paul for your necessary suggestions.
I've
>>> implemented them and now its work but I want use this file (
output from
>>> ascii2nc) for Point Stat tool. I have tried to run point_stat, its
>> running
>>> and creating some stat files too but stats file having only
headers no
>>> statistics. I'm not getting why its so?
>>> I've transferred data files as well as script file in
>>> incoming/irap/met_help/singh_data. Here I want to run it for TMP
and
>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
>>> 20110601_060000_rf.dat
>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
>>> outfile in GRIB1 format (output from WPPv3.2) named as
>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>>>
>>>
>>>
>>> I want to ask you something on some other data. Suppose I have
some
ASCII
>>> data having only 3 columns viz. Latitude, Longitude and Rainfall
(say).
>> Can
>>> we convert it through ascii2nc for MET supported format? If yes,
then
>> what
>>> we can take rest of the columns. I also transferred this kind of
data
>> named
>>> as TRMM_16AUG2011.asc,
>>> TRMM_16AUG2011.dat
>>> I want to use point stat for this data also.
>>>
>>> Thank you in advance.
>>>
>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
>> met_help at ucar.edu
>>>> wrote:
>>>
>>>> Harvir,
>>>>
>>>> I have identified several problems with the ascii2nc input file
>>>> suflnd_20110601_06z_rrf.dat:
>>>>
>>>> 1. ascii2nc input columns must be space-delimited. Several of
the
>> columns
>>>> in the input file are tab delimited. Do a
>>>> find/replace on the file to change all tabs to spaces.
>>>>
>>>> 2. After you perform step 1, you will see the following error
when you
>> try
>>>> to run ascii2nc:
>>>>
>>>> $ ascii2nc.sh
>>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
>>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
>>>> Processing 11414 observations.
>>>>
>>>>
>>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
>>>>
>>>>
>>>> The reason for this error is that your file appears to contain
>> accumulated
>>>> precipitation observations (GRIB code 61 in
>>>> column 7). In this case, ascii2nc is expecting an accumulation
interval
>> in
>>>> column 8 in the format HH, for example 06 or
>>>> 24. That is the reason for the error message above.
>>>>
>>>> 3. Column 9 should contain the height above ground of the
observation
>>>> station in meters. For accumulated
>>>> precipitation, this information is ignored by ascii2nc because
this is
a
>>>> "surface" observation. We typically use a fill
>>>> value of -9999, indicating invalid data.
>>>>
>>>> 4. Column 10 should contain the observed values. I see valid
>> observation
>>>> values on some rows, but many are -99.0 which
>>>> I interpret to mean invalid data. In this case, it does not make
sense
>> to
>>>> include invalid observation data, and those
>>>> lines should be deleted.
>>>>
>>>> I took the liberty of making a couple assumptions about your data
and
>>>> trying to adjust the input file that you sent me
>>>> to something that ascii2nc can read. The assumptions I made are:
>>>>
>>>> 1. All of your observations are "surface" observations, so the
message
>>>> type (column 1) was set to ADPSFC
>>>> 2. The accumulation interval is 6 hours (column 8 is set to 06)
>>>> 3. The observation values are in mm, and left as-is
>>>>
>>>> If these assumptions, mainly #2 and #3, are incorrect, please
make the
>>>> appropriate adjustments. I attached the "fixed"
>>>> file to this email for you to review and change as you wish.
Please
let
>> me
>>>> know if you have any questions.
>>>>
>>>> Paul
>>>>
>>>>
>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
>>>>>
>>>>>> Dear Paul,
>>>>>>
>>>>>> Accept My apology. Actually yesterday I forgot to attached
th
>> files.
>>>>> However, as per your advice through ftp I have putted the same
attached
>>>>> files there also. and its path is
>>>>> *incoming/irap/met_help/singh_data*
>>>>>
>>>>>> I'm sending you ASCII file and ascii2nc script. Please have
close
look
>>>> on
>>>>>> it if am wrong somewhere please let me know.
>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
>>>> not
>>>>>> in pcp_combine ... However, I've run all the test scripts
>> successfully.
>>>>>> Thanx in advance...
>>>>>>
>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
>>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>>> Harvir,
>>>>>>>
>>>>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
>>>> read
>>>>>>> TRMM data into MET would be to convert the
>>>>>>> data from it's current format into a NetCDF file that has the
same
>>>> format
>>>>>>> as the output of the MET pcp_combine tool. If
>>>>>>> you ran the MET test scripts after you built MET, you should
be able
>> to
>>>>>>> find pcp_combine output files in the sub-folder
>>>>>>> out/pcp_combine under your MET build folder. If you are
successful
>> in
>>>>>>> converting your data in this way, please send us
>>>>>>> your code so that we can make it available for other MET users
who
>>>> might
>>>>>>> want to use TRMM data. I'm sorry the answer to
>>>>>>> this question is not more simple.
>>>>>>>
>>>>>>> Regarding your ascii2nc question, can you please send us the
ascii2nc
>>>>>>> input that is causing your error? Also, what is
>>>>>>> the command line statement that you are using to invoke
ascii2nc?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>>
>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>>>
>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>>> Queue: met_help
>>>>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
>>>> format
>>>>>>>> Owner: Nobody
>>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>>> Status: new
>>>>>>>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dear Sir /Madam,
>>>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat MET
tool?
>>>>>>> which
>>>>>>>> converter should did I use to convert TRMM Rainfall data (.nc
or
>> .bin)
>>>>>>> to
>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
>>>>>>> (UPPv1.0),
>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>>>> I want to use some ASCII DATA may be output of some models
like GFS
>>>> etc
>>>>>>> ..
>>>>>>>> I have extracted data in the desired 10 columns. I'm trying
to use
>>>>>>> *ascii2nc
>>>>>>>> *this also giving some error like: open_met_ascii() -> line
number
0
>>>>>>> doesn't
>>>>>>>> contain the expected number of colu
>>>>>>>> N_items = 7
>>>>>>>> N_chars = 512
>>>>>>>> N_ints = 100
>>>>>>>> LineNumber = 1
>>>>>>>>
>>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
>>>>>>>> Item[ 1] = "48.68"
>>>>>>>> Item[ 2] = "9.23 "
>>>>>>>> Item[ 3] = "391.0 61 "
>>>>>>>> Item[ 4] = "974.8"
>>>>>>>>
>>>>>>>> Item[ 5] = "1021.5"
>>>>>>>> Item[ 6] = "-99.0"
>>>>>>>>
>>>>>>>> Offset[ 0] = 0
>>>>>>>> Offset[ 1] = 32
>>>>>>>> Offset[ 2] = 41
>>>>>>>> Offset[ 3] = 47
>>>>>>>> Offset[ 4] = 57
>>>>>>>>
>>>>>>>> Offset[ 5] = 64
>>>>>>>> Offset[ 6] = 71
>>>>>>>>
>>>>>>>> Please Help me....
>>>>>>>> Earliest reply will be appreciated.
>>>>>>>>
>>>>>>>> Thanks in advance for any help you can provide!
>>>>>>>>
>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for Post
>>>> Processing
>>>>>>>> Tool "UPPv1.0 "*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>
--
* Regards*
**************************************************************
**
* HARVIR SINGH <http://www.harvir007.webs.com>*
* To Know more <http://www.harvir007.webs.com> **
***************************************************************
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: Harvir Singh
Time: Fri Aug 26 07:04:09 2011
sorry I've not mentioned the variable, its rainfall
On 26 August 2011 18:31, Harvir Singh <harvirduphy at gmail.com> wrote:
> Dear Paul,
>
> Thank You for your prompt reply. Ofcourse yes I've to compute Threat
score
> too.. For this point of time I just want to go with point stat. and
I'm
> trying it for WRF ARW core (output data) it. I've tried point_stat,
its
> running but giving headers only in stat txt files. I'm unable to
rectify it.
> Please help me to come out of this problem. I've transfrred data in
> incoming/irap/met_help/singh_data
> thanx in advance!
>
>
> ---------- Forwarded message ----------
> From: RAL HelpDesk {for Paul Oldenburg} <met_help at ucar.edu>
> Date: 23 August 2011 21:55
> Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and
Point
> Stat tool
> To: harvirduphy at gmail.com
>
>
> Harvir,
>
> There are a couple points to discuss before we continue. First, I
> recommend that you use point_stat at this time to
> perform verification on your model data. The reason is that
converting
> your observation data to a format that grid_stat
> can read is more difficult because it must contain projection
information
> that matches your model data projection. Why
> do you want to run grid_stat? Are you interested in calculating
> neighborhood statistics or just basic verification
> statistics like RMSE and the Equitable Threat Score? If you only
want
> basic statistics, point_stat will work just fine.
>
> Second, for rainfall data, you said that the assumptions that I have
made
> about your observations are correct. If that
> is the case, and the accumulation interval of your observation data
is 6
> hours, then there is a mis-match with the
> accumulation interval in your model data, which is 24 hours (see 0-
24hr
> acc):
>
> $ wgrib wrfprs_d01_2011-06-01_000_fcst01.grib1 | grep APCP
>
252:323426:d=11053100:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
> acc:NAve=0
>
> If you are sure that your rainfall observations have an accumulation
> interval of 6 hours, then you must combine four of
> your 6 hour observation files together to form rainfall observations
with
> an accumulation interval of 24 hours. A good
> way to do this is to use the MET tool pcp_combine after you run
ascii2nc on
> your rainfall observation files.
>
> Finally, I tried to apply the model data that you gave me with
surface
> temperature to the observation data you gave me
> last week, 20110601_060000_tt.dat. I put the files involved in this
> experiment are on the RAL FTP site in the archive
>
ftp.rap.ucar.edu:/incoming/irap/met_help/singh_data/PointStatVx.tar.gz.
> If you download this and unpack it, you should
> be able to reproduce the output file that I included,
> point_stat_240000L_20110601_000000V.stat. This file contains
> basic verification statistics, like continuous and categorical
statistics.
> I used the threshold 273.15 K, which is 0 C.
>
> Note that for this experiment to be valid, I had to open the time
window to
> +/- 6 hours because your model data is valid
> at 20110601_000000 and your observations are valid at
20110601_060000.
> This may or may not be an acceptable compromise
> to your for your work. Also, I assumed that the temperature
observations
> were all on the surface (message type ADPUPA).
> It also looked like your observations were in degrees Celsius, so I
> converted them to Kelvin. Please let me know if
> you have any questions or trouble following my example. Please look
> carefully at the files included in the archive,
> especially PointStatConfig_tmp and point_stat.sh.
>
> Paul
>
>
> $ tar xzvf PointStatVx.tar.gz
> $ cat obs_20110601_060000_tt.dat | ./obs_conv.pl >
> obs_20110601_060000_tt.txt
> $ ascii2nc obs_20110601_060000_tt.txt obs_20110601_060000_tt.nc -v 3
> Creating NetCDF Observation file: obs_20110601_060000_tt.nc
> Reading ASCII Observation file: obs_20110601_060000_tt.txt
> Processing 1099 observations.
> Finished processing 1099 observations.
> $ ./point_stat.sh
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=231822158
> Forecast File: wrfprs_d01_2011-06-01_000_fcst01.grib1
> Climatology File: none
> Configuration File: PointStatConfig_tmp
> Observation File: obs_20110601_060000_tt.nc
>
>
>
--------------------------------------------------------------------------------
>
> Reading records for TMP/Z2.
> GRIB Record 244: Init = 20110531_000000, Valid = 20110601_000000,
Accum =
> 000000, Min = 292.579, Max = 303.165
> For TMP/Z2 found 1 forecast levels and 0 climatology levels.
>
>
>
--------------------------------------------------------------------------------
>
> Searching 1099 observations from 1099 PrepBufr messages.
>
>
>
--------------------------------------------------------------------------------
>
> Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC, over
region
> FULL, for interpolation method UW_MEAN(1),
> using 58 pairs.
> Number of matched pairs = 58
> Observations processed = 1099
> Rejected: GRIB code = 0
> Rejected: valid time = 0
> Rejected: bad obs value = 0
> Rejected: off the grid = 1041
> Rejected: level mismatch = 0
> Rejected: message type = 0
> Rejected: masking region = 0
> Rejected: bad fcst value = 0
> Computing Categorical Statistics.
> Computing Continuous Statistics.
> Computing Scalar Partial Sums.
>
>
>
--------------------------------------------------------------------------------
>
> Output file: ./point_stat_240000L_20110601_000000V.stat
>
>
> On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >
> > Dear Paul,
> > I have gone through your assumptions, might seems correct. But I
am
> having
> > some problems when I running Point_stat for rainfall, I am getting
some
> > error as:
> >
> > $ sh point_stat.sh
> > ERROR: GCInfo::set_gcinfo() -> unsupported input file type
"APCP/A24"
> > I have changed configure, "APCP/P24" "APCP/P1000", "APCP/P925"
etc.
> > and I ma getting similar type of error. Suggest me how can I
rectify it.
> >
> >
> > I also tried for GRID_STAT and I got the error as follows
> > sh Grid_stat.sh
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=18446744073465844558
> > Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
> > Observation File: TRMM_16AUG2011.nc
> > Configuration File:
> > /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
> > GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum =
> > 240000, Min = 0, Max = 148.8
> > ncattinq: ncid 4; varid -1; attname "Projection": Attribute not
found
> >
> >
> > One more thing I want to ask you about plot_point_obs. Actually
> > plot_point_obs executable does not exits in my METv3.0.1/bin.
Suppose I
> need
> > it, then what I have to do?
> >
> > Thanks
> > HARVIR
> >
> > *PS: All the data files and script files have been transferred in
> > incoming/irap/met_help/singh_data
> > *
> > On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
> > met_help at ucar.edu> wrote:
> >
> >> Harvir,
> >>
> >> Regarding your second request related to TRMM data, I wrote a
small
> script
> >> (attached) to convert TRMM_16AUG2011.asc into
> >> a format that is readable by ascii2nc. Then, I plotted the
observation
> >> points using the MET utility plot_point_obs and
> >> created the attached image showing the range of points. It is
important
> to
> >> note that I had to make several assumptions
> >> about your observations to do this conversion. The assumptions
are:
> >>
> >> 1. Your observations are surface observations (column 1 -
message type
> >> ADPSFC). This does make sense if your data is
> >> accumulated precipitation, which I believe it is.
> >> 2. The valid time of your observations is 2011-08-11 00:00:00
UTC.
> >> Obviously, the date is shown in your filename, but
> >> I could not find the valid time.
> >> 3. The elevation in msl of all observing points is 0m, which is
> definitely
> >> a bad assumption, but for "surface"
> >> observations will not matter.
> >> 4. The accumulation interval of your observations is 6 hours
(column
> 8).
> >> If this is not correct, please modify the
> >> script I attached accordingly.
> >> 5. The height in msl of the observation points is 0m (see
assumption
> 3).
> >>
> >> Once you have model data that contains APCP with the same
accumulation
> >> interval as that of your observations, you should
> >> be able to use this acii2nc-generate observations in point_stat.
> >>
> >> Here are the steps to generate the ascii2nc NetCDF obs data file
and
> plot
> >> using the script I am sending. These commands
> >> assume that the MET bin/ sub-folder is on your path.
> >>
> >> $ chmod +x trmm_conv.pl
> >> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
> >> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
> >> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15 45
45 120
> >>
> >> Please let me know if you have any questions.
> >>
> >> Thanks,
> >>
> >> Paul
> >>
> >>
> >> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>
> >>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
> >>> Transaction: Ticket created by harvirduphy at gmail.com
> >>> Queue: met_help
> >>> Subject: Need help for ASCII2NC and Point Stat tool
> >>> Owner: Nobody
> >>> Requestors: harvirduphy at gmail.com
> >>> Status: new
> >>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061>
> >>>
> >>>
> >>> Dear Paul,
> >>>
> >>> I would like to thank you Paul for your necessary suggestions.
I've
> >>> implemented them and now its work but I want use this file (
output
> from
> >>> ascii2nc) for Point Stat tool. I have tried to run point_stat,
its
> >> running
> >>> and creating some stat files too but stats file having only
headers no
> >>> statistics. I'm not getting why its so?
> >>> I've transferred data files as well as script file in
> >>> incoming/irap/met_help/singh_data. Here I want to run it for TMP
and
> >>> Rainfall. The files for TMP are 20110601_060000_tt.dat
> >>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
> >>> 20110601_060000_rf.dat
> >>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file
is WRF
> >>> outfile in GRIB1 format (output from WPPv3.2) named as
> >>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
> >>>
> >>>
> >>>
> >>> I want to ask you something on some other data. Suppose I have
some
> ASCII
> >>> data having only 3 columns viz. Latitude, Longitude and Rainfall
(say).
> >> Can
> >>> we convert it through ascii2nc for MET supported format? If yes,
then
> >> what
> >>> we can take rest of the columns. I also transferred this kind of
data
> >> named
> >>> as TRMM_16AUG2011.asc,
> >>> TRMM_16AUG2011.dat
> >>> I want to use point stat for this data also.
> >>>
> >>> Thank you in advance.
> >>>
> >>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
> >> met_help at ucar.edu
> >>>> wrote:
> >>>
> >>>> Harvir,
> >>>>
> >>>> I have identified several problems with the ascii2nc input file
> >>>> suflnd_20110601_06z_rrf.dat:
> >>>>
> >>>> 1. ascii2nc input columns must be space-delimited. Several of
the
> >> columns
> >>>> in the input file are tab delimited. Do a
> >>>> find/replace on the file to change all tabs to spaces.
> >>>>
> >>>> 2. After you perform step 1, you will see the following error
when
> you
> >> try
> >>>> to run ascii2nc:
> >>>>
> >>>> $ ascii2nc.sh
> >>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
> >>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
> >>>> Processing 11414 observations.
> >>>>
> >>>>
> >>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
> >>>>
> >>>>
> >>>> The reason for this error is that your file appears to contain
> >> accumulated
> >>>> precipitation observations (GRIB code 61 in
> >>>> column 7). In this case, ascii2nc is expecting an accumulation
> interval
> >> in
> >>>> column 8 in the format HH, for example 06 or
> >>>> 24. That is the reason for the error message above.
> >>>>
> >>>> 3. Column 9 should contain the height above ground of the
observation
> >>>> station in meters. For accumulated
> >>>> precipitation, this information is ignored by ascii2nc because
this is
> a
> >>>> "surface" observation. We typically use a fill
> >>>> value of -9999, indicating invalid data.
> >>>>
> >>>> 4. Column 10 should contain the observed values. I see valid
> >> observation
> >>>> values on some rows, but many are -99.0 which
> >>>> I interpret to mean invalid data. In this case, it does not
make
> sense
> >> to
> >>>> include invalid observation data, and those
> >>>> lines should be deleted.
> >>>>
> >>>> I took the liberty of making a couple assumptions about your
data and
> >>>> trying to adjust the input file that you sent me
> >>>> to something that ascii2nc can read. The assumptions I made
are:
> >>>>
> >>>> 1. All of your observations are "surface" observations, so the
> message
> >>>> type (column 1) was set to ADPSFC
> >>>> 2. The accumulation interval is 6 hours (column 8 is set to
06)
> >>>> 3. The observation values are in mm, and left as-is
> >>>>
> >>>> If these assumptions, mainly #2 and #3, are incorrect, please
make the
> >>>> appropriate adjustments. I attached the "fixed"
> >>>> file to this email for you to review and change as you wish.
Please
> let
> >> me
> >>>> know if you have any questions.
> >>>>
> >>>> Paul
> >>>>
> >>>>
> >>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>>
> >>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689
>
> >>>>>
> >>>>>> Dear Paul,
> >>>>>>
> >>>>>> Accept My apology. Actually yesterday I forgot to attached
th
> >> files.
> >>>>> However, as per your advice through ftp I have putted the same
> attached
> >>>>> files there also. and its path is
> >>>>> *incoming/irap/met_help/singh_data*
> >>>>>
> >>>>>> I'm sending you ASCII file and ascii2nc script. Please have
close
> look
> >>>> on
> >>>>>> it if am wrong somewhere please let me know.
> >>>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful
> or
> >>>> not
> >>>>>> in pcp_combine ... However, I've run all the test scripts
> >> successfully.
> >>>>>> Thanx in advance...
> >>>>>>
> >>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
> >>>>>> met_help at ucar.edu> wrote:
> >>>>>>
> >>>>>>> Harvir,
> >>>>>>>
> >>>>>>> I'm afraid that MET cannot read TRMM data directly. The
best way
> to
> >>>> read
> >>>>>>> TRMM data into MET would be to convert the
> >>>>>>> data from it's current format into a NetCDF file that has
the same
> >>>> format
> >>>>>>> as the output of the MET pcp_combine tool. If
> >>>>>>> you ran the MET test scripts after you built MET, you should
be
> able
> >> to
> >>>>>>> find pcp_combine output files in the sub-folder
> >>>>>>> out/pcp_combine under your MET build folder. If you are
successful
> >> in
> >>>>>>> converting your data in this way, please send us
> >>>>>>> your code so that we can make it available for other MET
users who
> >>>> might
> >>>>>>> want to use TRMM data. I'm sorry the answer to
> >>>>>>> this question is not more simple.
> >>>>>>>
> >>>>>>> Regarding your ascii2nc question, can you please send us the
> ascii2nc
> >>>>>>> input that is causing your error? Also, what is
> >>>>>>> the command line statement that you are using to invoke
ascii2nc?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Paul
> >>>>>>>
> >>>>>>>
> >>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh}
wrote:
> >>>>>>>>
> >>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
> >>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>>>>>> Queue: met_help
> >>>>>>>> Subject: Getting Problem in converting TRMM rainfall
to MET
> >>>> format
> >>>>>>>> Owner: Nobody
> >>>>>>>> Requestors: harvirduphy at gmail.com
> >>>>>>>> Status: new
> >>>>>>>> Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Dear Sir /Madam,
> >>>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat
MET
> tool?
> >>>>>>> which
> >>>>>>>> converter should did I use to convert TRMM Rainfall data
(.nc or
> >> .bin)
> >>>>>>> to
> >>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
> >>>>>>> (UPPv1.0),
> >>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
> >>>>>>>> I want to use some ASCII DATA may be output of some models
like
> GFS
> >>>> etc
> >>>>>>> ..
> >>>>>>>> I have extracted data in the desired 10 columns. I'm trying
to use
> >>>>>>> *ascii2nc
> >>>>>>>> *this also giving some error like: open_met_ascii() -> line
number
> 0
> >>>>>>> doesn't
> >>>>>>>> contain the expected number of colu
> >>>>>>>> N_items = 7
> >>>>>>>> N_chars = 512
> >>>>>>>> N_ints = 100
> >>>>>>>> LineNumber = 1
> >>>>>>>>
> >>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
> >>>>>>>> Item[ 1] = "48.68"
> >>>>>>>> Item[ 2] = "9.23 "
> >>>>>>>> Item[ 3] = "391.0 61 "
> >>>>>>>> Item[ 4] = "974.8"
> >>>>>>>>
> >>>>>>>> Item[ 5] = "1021.5"
> >>>>>>>> Item[ 6] = "-99.0"
> >>>>>>>>
> >>>>>>>> Offset[ 0] = 0
> >>>>>>>> Offset[ 1] = 32
> >>>>>>>> Offset[ 2] = 41
> >>>>>>>> Offset[ 3] = 47
> >>>>>>>> Offset[ 4] = 57
> >>>>>>>>
> >>>>>>>> Offset[ 5] = 64
> >>>>>>>> Offset[ 6] = 71
> >>>>>>>>
> >>>>>>>> Please Help me....
> >>>>>>>> Earliest reply will be appreciated.
> >>>>>>>>
> >>>>>>>> Thanks in advance for any help you can provide!
> >>>>>>>>
> >>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for
Post
> >>>> Processing
> >>>>>>>> Tool "UPPv1.0 "*
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >
> >
>
>
>
>
>
> --
> * Regards*
> **************************************************************
> **
> * HARVIR SINGH <http://www.harvir007.webs.com>*
> * To Know more <http://www.harvir007.webs.com> **
>
> ***************************************************************
>
>
--
* Regards*
**************************************************************
**
* HARVIR SINGH <http://www.harvir007.webs.com>*
* To Know more <http://www.harvir007.webs.com> **
***************************************************************
------------------------------------------------
Subject: Re: Fwd: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and Point Stat tool
From: Paul Oldenburg
Time: Fri Aug 26 08:35:45 2011
Harvir,
In the last email I sent you, I said that your rainfall observation
data in your model data file,
wrfprs_d01_2011-06-01_000_fcst01.grib1, had an accumulation interval
of 24 hours. That means that the amount of
rainfall in the model data file is from a 24 hour period.
Do you know what the accumulation interval of the rainfall data you
sent me in either obs_20110601_060000_rf.dat or
TRMM_16AUG2011.dat is? I see the following information at the top of
the file TRMM_16AUG2011.dat:
Selected parameter: TRMM TMPA-RT Daily Accumulated Rainfall
Selected area: lat=[16.161S,45.311N], lon=[44.182E,121.960E]
Selected time period: (16Aug2011-16Aug2011)
Does that mean that it is 24 hour accumulated rainfall from 2011-08-15
00:00:00 to 2011-08-16 00:00:00? If so, then it
has a 24 hour accumulation interval, but it does not match the date in
your model data file. Do you have a TRMM file
that is valid at the same time as your model data file?
I need to know the answers to these questions before I can make
further progress helping you verify rainfall. It is
very important that both the accumulation interval and the valid time
of your model and observation data match.
Thanks,
Paul
On 08/26/2011 07:02 AM, RAL HelpDesk {for Harvir Singh} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>
> Dear Paul,
>
> Thank You for your prompt reply. Ofcourse yes I've to compute Threat
score
> too.. For this point of time I just want to go with point stat. and
I'm
> trying it for WRF ARW core (output data) it. I've tried point_stat,
its
> running but giving headers only in stat txt files. I'm unable to
rectify it.
> Please help me to come out of this problem. I've transfrred data in
> incoming/irap/met_help/singh_data
> thanx in advance!
>
> ---------- Forwarded message ----------
> From: RAL HelpDesk {for Paul Oldenburg} <met_help at ucar.edu>
> Date: 23 August 2011 21:55
> Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and
Point Stat
> tool
> To: harvirduphy at gmail.com
>
>
> Harvir,
>
> There are a couple points to discuss before we continue. First, I
recommend
> that you use point_stat at this time to
> perform verification on your model data. The reason is that
converting your
> observation data to a format that grid_stat
> can read is more difficult because it must contain projection
information
> that matches your model data projection. Why
> do you want to run grid_stat? Are you interested in calculating
> neighborhood statistics or just basic verification
> statistics like RMSE and the Equitable Threat Score? If you only
want basic
> statistics, point_stat will work just fine.
>
> Second, for rainfall data, you said that the assumptions that I have
made
> about your observations are correct. If that
> is the case, and the accumulation interval of your observation data
is 6
> hours, then there is a mis-match with the
> accumulation interval in your model data, which is 24 hours (see 0-
24hr
> acc):
>
> $ wgrib wrfprs_d01_2011-06-01_000_fcst01.grib1 | grep APCP
>
252:323426:d=11053100:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
> acc:NAve=0
>
> If you are sure that your rainfall observations have an accumulation
> interval of 6 hours, then you must combine four of
> your 6 hour observation files together to form rainfall observations
with an
> accumulation interval of 24 hours. A good
> way to do this is to use the MET tool pcp_combine after you run
ascii2nc on
> your rainfall observation files.
>
> Finally, I tried to apply the model data that you gave me with
surface
> temperature to the observation data you gave me
> last week, 20110601_060000_tt.dat. I put the files involved in this
> experiment are on the RAL FTP site in the archive
>
ftp.rap.ucar.edu:/incoming/irap/met_help/singh_data/PointStatVx.tar.gz.
If
> you download this and unpack it, you should
> be able to reproduce the output file that I included,
> point_stat_240000L_20110601_000000V.stat. This file contains
> basic verification statistics, like continuous and categorical
statistics.
> I used the threshold 273.15 K, which is 0 C.
>
> Note that for this experiment to be valid, I had to open the time
window to
> +/- 6 hours because your model data is valid
> at 20110601_000000 and your observations are valid at
20110601_060000. This
> may or may not be an acceptable compromise
> to your for your work. Also, I assumed that the temperature
observations
> were all on the surface (message type ADPUPA).
> It also looked like your observations were in degrees Celsius, so I
> converted them to Kelvin. Please let me know if
> you have any questions or trouble following my example. Please look
> carefully at the files included in the archive,
> especially PointStatConfig_tmp and point_stat.sh.
>
> Paul
>
>
> $ tar xzvf PointStatVx.tar.gz
> $ cat obs_20110601_060000_tt.dat | ./obs_conv.pl >
> obs_20110601_060000_tt.txt
> $ ascii2nc obs_20110601_060000_tt.txt obs_20110601_060000_tt.nc -v 3
> Creating NetCDF Observation file: obs_20110601_060000_tt.nc
> Reading ASCII Observation file: obs_20110601_060000_tt.txt
> Processing 1099 observations.
> Finished processing 1099 observations.
> $ ./point_stat.sh
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=231822158
> Forecast File: wrfprs_d01_2011-06-01_000_fcst01.grib1
> Climatology File: none
> Configuration File: PointStatConfig_tmp
> Observation File: obs_20110601_060000_tt.nc
>
>
--------------------------------------------------------------------------------
>
> Reading records for TMP/Z2.
> GRIB Record 244: Init = 20110531_000000, Valid = 20110601_000000,
Accum =
> 000000, Min = 292.579, Max = 303.165
> For TMP/Z2 found 1 forecast levels and 0 climatology levels.
>
>
--------------------------------------------------------------------------------
>
> Searching 1099 observations from 1099 PrepBufr messages.
>
>
--------------------------------------------------------------------------------
>
> Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC, over
region
> FULL, for interpolation method UW_MEAN(1),
> using 58 pairs.
> Number of matched pairs = 58
> Observations processed = 1099
> Rejected: GRIB code = 0
> Rejected: valid time = 0
> Rejected: bad obs value = 0
> Rejected: off the grid = 1041
> Rejected: level mismatch = 0
> Rejected: message type = 0
> Rejected: masking region = 0
> Rejected: bad fcst value = 0
> Computing Categorical Statistics.
> Computing Continuous Statistics.
> Computing Scalar Partial Sums.
>
>
--------------------------------------------------------------------------------
>
> Output file: ./point_stat_240000L_20110601_000000V.stat
>
>
> On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>
>> Dear Paul,
>> I have gone through your assumptions, might seems correct. But I am
having
>> some problems when I running Point_stat for rainfall, I am getting
some
>> error as:
>>
>> $ sh point_stat.sh
>> ERROR: GCInfo::set_gcinfo() -> unsupported input file type
"APCP/A24"
>> I have changed configure, "APCP/P24" "APCP/P1000", "APCP/P925" etc.
>> and I ma getting similar type of error. Suggest me how can I
rectify it.
>>
>>
>> I also tried for GRID_STAT and I got the error as follows
>> sh Grid_stat.sh
>> GSL_RNG_TYPE=mt19937
>> GSL_RNG_SEED=18446744073465844558
>> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
>> Observation File: TRMM_16AUG2011.nc
>> Configuration File:
>> /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
>> GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum =
>> 240000, Min = 0, Max = 148.8
>> ncattinq: ncid 4; varid -1; attname "Projection": Attribute not
found
>>
>>
>> One more thing I want to ask you about plot_point_obs. Actually
>> plot_point_obs executable does not exits in my METv3.0.1/bin.
Suppose I
> need
>> it, then what I have to do?
>>
>> Thanks
>> HARVIR
>>
>> *PS: All the data files and script files have been transferred in
>> incoming/irap/met_help/singh_data
>> *
>> On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
>> met_help at ucar.edu> wrote:
>>
>>> Harvir,
>>>
>>> Regarding your second request related to TRMM data, I wrote a
small
> script
>>> (attached) to convert TRMM_16AUG2011.asc into
>>> a format that is readable by ascii2nc. Then, I plotted the
observation
>>> points using the MET utility plot_point_obs and
>>> created the attached image showing the range of points. It is
important
> to
>>> note that I had to make several assumptions
>>> about your observations to do this conversion. The assumptions
are:
>>>
>>> 1. Your observations are surface observations (column 1 - message
type
>>> ADPSFC). This does make sense if your data is
>>> accumulated precipitation, which I believe it is.
>>> 2. The valid time of your observations is 2011-08-11 00:00:00
UTC.
>>> Obviously, the date is shown in your filename, but
>>> I could not find the valid time.
>>> 3. The elevation in msl of all observing points is 0m, which is
> definitely
>>> a bad assumption, but for "surface"
>>> observations will not matter.
>>> 4. The accumulation interval of your observations is 6 hours
(column 8).
>>> If this is not correct, please modify the
>>> script I attached accordingly.
>>> 5. The height in msl of the observation points is 0m (see
assumption 3).
>>>
>>> Once you have model data that contains APCP with the same
accumulation
>>> interval as that of your observations, you should
>>> be able to use this acii2nc-generate observations in point_stat.
>>>
>>> Here are the steps to generate the ascii2nc NetCDF obs data file
and plot
>>> using the script I am sending. These commands
>>> assume that the MET bin/ sub-folder is on your path.
>>>
>>> $ chmod +x trmm_conv.pl
>>> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
>>> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
>>> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15 45
45 120
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>>
>>> Paul
>>>
>>>
>>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>
>>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>> Queue: met_help
>>>> Subject: Need help for ASCII2NC and Point Stat tool
>>>> Owner: Nobody
>>>> Requestors: harvirduphy at gmail.com
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>>
>>>>
>>>> Dear Paul,
>>>>
>>>> I would like to thank you Paul for your necessary suggestions.
I've
>>>> implemented them and now its work but I want use this file (
output from
>>>> ascii2nc) for Point Stat tool. I have tried to run point_stat,
its
>>> running
>>>> and creating some stat files too but stats file having only
headers no
>>>> statistics. I'm not getting why its so?
>>>> I've transferred data files as well as script file in
>>>> incoming/irap/met_help/singh_data. Here I want to run it for TMP
and
>>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
>>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
>>>> 20110601_060000_rf.dat
>>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file is
WRF
>>>> outfile in GRIB1 format (output from WPPv3.2) named as
>>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>>>>
>>>>
>>>>
>>>> I want to ask you something on some other data. Suppose I have
some
> ASCII
>>>> data having only 3 columns viz. Latitude, Longitude and Rainfall
(say).
>>> Can
>>>> we convert it through ascii2nc for MET supported format? If yes,
then
>>> what
>>>> we can take rest of the columns. I also transferred this kind of
data
>>> named
>>>> as TRMM_16AUG2011.asc,
>>>> TRMM_16AUG2011.dat
>>>> I want to use point stat for this data also.
>>>>
>>>> Thank you in advance.
>>>>
>>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
>>> met_help at ucar.edu
>>>>> wrote:
>>>>
>>>>> Harvir,
>>>>>
>>>>> I have identified several problems with the ascii2nc input file
>>>>> suflnd_20110601_06z_rrf.dat:
>>>>>
>>>>> 1. ascii2nc input columns must be space-delimited. Several of
the
>>> columns
>>>>> in the input file are tab delimited. Do a
>>>>> find/replace on the file to change all tabs to spaces.
>>>>>
>>>>> 2. After you perform step 1, you will see the following error
when you
>>> try
>>>>> to run ascii2nc:
>>>>>
>>>>> $ ascii2nc.sh
>>>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
>>>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
>>>>> Processing 11414 observations.
>>>>>
>>>>>
>>>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
>>>>>
>>>>>
>>>>> The reason for this error is that your file appears to contain
>>> accumulated
>>>>> precipitation observations (GRIB code 61 in
>>>>> column 7). In this case, ascii2nc is expecting an accumulation
> interval
>>> in
>>>>> column 8 in the format HH, for example 06 or
>>>>> 24. That is the reason for the error message above.
>>>>>
>>>>> 3. Column 9 should contain the height above ground of the
observation
>>>>> station in meters. For accumulated
>>>>> precipitation, this information is ignored by ascii2nc because
this is
> a
>>>>> "surface" observation. We typically use a fill
>>>>> value of -9999, indicating invalid data.
>>>>>
>>>>> 4. Column 10 should contain the observed values. I see valid
>>> observation
>>>>> values on some rows, but many are -99.0 which
>>>>> I interpret to mean invalid data. In this case, it does not
make sense
>>> to
>>>>> include invalid observation data, and those
>>>>> lines should be deleted.
>>>>>
>>>>> I took the liberty of making a couple assumptions about your
data and
>>>>> trying to adjust the input file that you sent me
>>>>> to something that ascii2nc can read. The assumptions I made
are:
>>>>>
>>>>> 1. All of your observations are "surface" observations, so the
message
>>>>> type (column 1) was set to ADPSFC
>>>>> 2. The accumulation interval is 6 hours (column 8 is set to 06)
>>>>> 3. The observation values are in mm, and left as-is
>>>>>
>>>>> If these assumptions, mainly #2 and #3, are incorrect, please
make the
>>>>> appropriate adjustments. I attached the "fixed"
>>>>> file to this email for you to review and change as you wish.
Please
> let
>>> me
>>>>> know if you have any questions.
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
>>>>>>
>>>>>>> Dear Paul,
>>>>>>>
>>>>>>> Accept My apology. Actually yesterday I forgot to attached
th
>>> files.
>>>>>> However, as per your advice through ftp I have putted the same
> attached
>>>>>> files there also. and its path is
>>>>>> *incoming/irap/met_help/singh_data*
>>>>>>
>>>>>>> I'm sending you ASCII file and ascii2nc script. Please have
close
> look
>>>>> on
>>>>>>> it if am wrong somewhere please let me know.
>>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful or
>>>>> not
>>>>>>> in pcp_combine ... However, I've run all the test scripts
>>> successfully.
>>>>>>> Thanx in advance...
>>>>>>>
>>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>
>>>>>>>> Harvir,
>>>>>>>>
>>>>>>>> I'm afraid that MET cannot read TRMM data directly. The best
way to
>>>>> read
>>>>>>>> TRMM data into MET would be to convert the
>>>>>>>> data from it's current format into a NetCDF file that has the
same
>>>>> format
>>>>>>>> as the output of the MET pcp_combine tool. If
>>>>>>>> you ran the MET test scripts after you built MET, you should
be able
>>> to
>>>>>>>> find pcp_combine output files in the sub-folder
>>>>>>>> out/pcp_combine under your MET build folder. If you are
successful
>>> in
>>>>>>>> converting your data in this way, please send us
>>>>>>>> your code so that we can make it available for other MET
users who
>>>>> might
>>>>>>>> want to use TRMM data. I'm sorry the answer to
>>>>>>>> this question is not more simple.
>>>>>>>>
>>>>>>>> Regarding your ascii2nc question, can you please send us the
> ascii2nc
>>>>>>>> input that is causing your error? Also, what is
>>>>>>>> the command line statement that you are using to invoke
ascii2nc?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Paul
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh}
wrote:
>>>>>>>>>
>>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
>>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>>>> Queue: met_help
>>>>>>>>> Subject: Getting Problem in converting TRMM rainfall to
MET
>>>>> format
>>>>>>>>> Owner: Nobody
>>>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>>>> Status: new
>>>>>>>>> Ticket <URL:
>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dear Sir /Madam,
>>>>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat
MET tool?
>>>>>>>> which
>>>>>>>>> converter should did I use to convert TRMM Rainfall data
(.nc or
>>> .bin)
>>>>>>>> to
>>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
>>>>>>>> (UPPv1.0),
>>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>>>>> I want to use some ASCII DATA may be output of some models
like GFS
>>>>> etc
>>>>>>>> ..
>>>>>>>>> I have extracted data in the desired 10 columns. I'm trying
to use
>>>>>>>> *ascii2nc
>>>>>>>>> *this also giving some error like: open_met_ascii() -> line
number
> 0
>>>>>>>> doesn't
>>>>>>>>> contain the expected number of colu
>>>>>>>>> N_items = 7
>>>>>>>>> N_chars = 512
>>>>>>>>> N_ints = 100
>>>>>>>>> LineNumber = 1
>>>>>>>>>
>>>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
>>>>>>>>> Item[ 1] = "48.68"
>>>>>>>>> Item[ 2] = "9.23 "
>>>>>>>>> Item[ 3] = "391.0 61 "
>>>>>>>>> Item[ 4] = "974.8"
>>>>>>>>>
>>>>>>>>> Item[ 5] = "1021.5"
>>>>>>>>> Item[ 6] = "-99.0"
>>>>>>>>>
>>>>>>>>> Offset[ 0] = 0
>>>>>>>>> Offset[ 1] = 32
>>>>>>>>> Offset[ 2] = 41
>>>>>>>>> Offset[ 3] = 47
>>>>>>>>> Offset[ 4] = 57
>>>>>>>>>
>>>>>>>>> Offset[ 5] = 64
>>>>>>>>> Offset[ 6] = 71
>>>>>>>>>
>>>>>>>>> Please Help me....
>>>>>>>>> Earliest reply will be appreciated.
>>>>>>>>>
>>>>>>>>> Thanks in advance for any help you can provide!
>>>>>>>>>
>>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for
Post
>>>>> Processing
>>>>>>>>> Tool "UPPv1.0 "*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
>
>
>
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: Harvir Singh
Time: Mon Aug 29 06:29:18 2011
Dear Paul,
With the reference to your earlier mails, wrfprs_d01_2011-06-01_000_
>
> fcst01.grib1 was for forecast file, obs_20110601_060000_rf.dat
observation
> file from GTS data and
> TRMM_16AUG2011.dat is from TRMM rainfall data.
>
Now I've transferred new set of data files viz.
Model forecast file: wrfprs_d01_2011-08-16_000_fcst01.grib1
Observation file : TRMM_RF_16AUG2011_0.25.txt
and TRMM Rainfall data is from 00Z 16 AUG 2011 to 00Z 17 AUG 2011 and
model
data is for 24 hr acumulated..
thanx
On 26 August 2011 20:05, RAL HelpDesk {for Paul Oldenburg} <
met_help at ucar.edu> wrote:
> Harvir,
>
> In the last email I sent you, I said that your rainfall observation
data in
> your model data file,
> wrfprs_d01_2011-06-01_000_fcst01.grib1, had an accumulation interval
of 24
> hours. That means that the amount of
> rainfall in the model data file is from a 24 hour period.
>
> Do you know what the accumulation interval of the rainfall data you
sent me
> in either obs_20110601_060000_rf.dat or
> TRMM_16AUG2011.dat is? I see the following information at the top
of the
> file TRMM_16AUG2011.dat:
>
> Selected parameter: TRMM TMPA-RT Daily Accumulated Rainfall
> Selected area: lat=[16.161S,45.311N], lon=[44.182E,121.960E]
> Selected time period: (16Aug2011-16Aug2011)
>
> Does that mean that it is 24 hour accumulated rainfall from 2011-08-
15
> 00:00:00 to 2011-08-16 00:00:00? If so, then it
> has a 24 hour accumulation interval, but it does not match the date
in your
> model data file. Do you have a TRMM file
> that is valid at the same time as your model data file?
>
> I need to know the answers to these questions before I can make
further
> progress helping you verify rainfall. It is
> very important that both the accumulation interval and the valid
time of
> your model and observation data match.
>
> Thanks,
>
> Paul
>
>
> On 08/26/2011 07:02 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >
> > Dear Paul,
> >
> > Thank You for your prompt reply. Ofcourse yes I've to compute
Threat
> score
> > too.. For this point of time I just want to go with point stat.
and I'm
> > trying it for WRF ARW core (output data) it. I've tried
point_stat, its
> > running but giving headers only in stat txt files. I'm unable to
rectify
> it.
> > Please help me to come out of this problem. I've transfrred data
in
> > incoming/irap/met_help/singh_data
> > thanx in advance!
> >
> > ---------- Forwarded message ----------
> > From: RAL HelpDesk {for Paul Oldenburg} <met_help at ucar.edu>
> > Date: 23 August 2011 21:55
> > Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and
Point
> Stat
> > tool
> > To: harvirduphy at gmail.com
> >
> >
> > Harvir,
> >
> > There are a couple points to discuss before we continue. First, I
> recommend
> > that you use point_stat at this time to
> > perform verification on your model data. The reason is that
converting
> your
> > observation data to a format that grid_stat
> > can read is more difficult because it must contain projection
information
> > that matches your model data projection. Why
> > do you want to run grid_stat? Are you interested in calculating
> > neighborhood statistics or just basic verification
> > statistics like RMSE and the Equitable Threat Score? If you only
want
> basic
> > statistics, point_stat will work just fine.
> >
> > Second, for rainfall data, you said that the assumptions that I
have made
> > about your observations are correct. If that
> > is the case, and the accumulation interval of your observation
data is 6
> > hours, then there is a mis-match with the
> > accumulation interval in your model data, which is 24 hours (see
0-24hr
> > acc):
> >
> > $ wgrib wrfprs_d01_2011-06-01_000_fcst01.grib1 | grep APCP
> >
>
252:323426:d=11053100:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
> > acc:NAve=0
> >
> > If you are sure that your rainfall observations have an
accumulation
> > interval of 6 hours, then you must combine four of
> > your 6 hour observation files together to form rainfall
observations with
> an
> > accumulation interval of 24 hours. A good
> > way to do this is to use the MET tool pcp_combine after you run
ascii2nc
> on
> > your rainfall observation files.
> >
> > Finally, I tried to apply the model data that you gave me with
surface
> > temperature to the observation data you gave me
> > last week, 20110601_060000_tt.dat. I put the files involved in
this
> > experiment are on the RAL FTP site in the archive
> >
ftp.rap.ucar.edu:/incoming/irap/met_help/singh_data/PointStatVx.tar.gz.
> If
> > you download this and unpack it, you should
> > be able to reproduce the output file that I included,
> > point_stat_240000L_20110601_000000V.stat. This file contains
> > basic verification statistics, like continuous and categorical
> statistics.
> > I used the threshold 273.15 K, which is 0 C.
> >
> > Note that for this experiment to be valid, I had to open the time
window
> to
> > +/- 6 hours because your model data is valid
> > at 20110601_000000 and your observations are valid at
20110601_060000.
> This
> > may or may not be an acceptable compromise
> > to your for your work. Also, I assumed that the temperature
observations
> > were all on the surface (message type ADPUPA).
> > It also looked like your observations were in degrees Celsius, so
I
> > converted them to Kelvin. Please let me know if
> > you have any questions or trouble following my example. Please
look
> > carefully at the files included in the archive,
> > especially PointStatConfig_tmp and point_stat.sh.
> >
> > Paul
> >
> >
> > $ tar xzvf PointStatVx.tar.gz
> > $ cat obs_20110601_060000_tt.dat | ./obs_conv.pl >
> > obs_20110601_060000_tt.txt
> > $ ascii2nc obs_20110601_060000_tt.txt obs_20110601_060000_tt.nc -v
3
> > Creating NetCDF Observation file: obs_20110601_060000_tt.nc
> > Reading ASCII Observation file: obs_20110601_060000_tt.txt
> > Processing 1099 observations.
> > Finished processing 1099 observations.
> > $ ./point_stat.sh
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=231822158
> > Forecast File: wrfprs_d01_2011-06-01_000_fcst01.grib1
> > Climatology File: none
> > Configuration File: PointStatConfig_tmp
> > Observation File: obs_20110601_060000_tt.nc
> >
> >
>
--------------------------------------------------------------------------------
> >
> > Reading records for TMP/Z2.
> > GRIB Record 244: Init = 20110531_000000, Valid = 20110601_000000,
Accum =
> > 000000, Min = 292.579, Max = 303.165
> > For TMP/Z2 found 1 forecast levels and 0 climatology levels.
> >
> >
>
--------------------------------------------------------------------------------
> >
> > Searching 1099 observations from 1099 PrepBufr messages.
> >
> >
>
--------------------------------------------------------------------------------
> >
> > Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC, over
region
> > FULL, for interpolation method UW_MEAN(1),
> > using 58 pairs.
> > Number of matched pairs = 58
> > Observations processed = 1099
> > Rejected: GRIB code = 0
> > Rejected: valid time = 0
> > Rejected: bad obs value = 0
> > Rejected: off the grid = 1041
> > Rejected: level mismatch = 0
> > Rejected: message type = 0
> > Rejected: masking region = 0
> > Rejected: bad fcst value = 0
> > Computing Categorical Statistics.
> > Computing Continuous Statistics.
> > Computing Scalar Partial Sums.
> >
> >
>
--------------------------------------------------------------------------------
> >
> > Output file: ./point_stat_240000L_20110601_000000V.stat
> >
> >
> > On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >>
> >> Dear Paul,
> >> I have gone through your assumptions, might seems correct. But I
am
> having
> >> some problems when I running Point_stat for rainfall, I am
getting some
> >> error as:
> >>
> >> $ sh point_stat.sh
> >> ERROR: GCInfo::set_gcinfo() -> unsupported input file type
"APCP/A24"
> >> I have changed configure, "APCP/P24" "APCP/P1000", "APCP/P925"
etc.
> >> and I ma getting similar type of error. Suggest me how can I
rectify it.
> >>
> >>
> >> I also tried for GRID_STAT and I got the error as follows
> >> sh Grid_stat.sh
> >> GSL_RNG_TYPE=mt19937
> >> GSL_RNG_SEED=18446744073465844558
> >> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
> >> Observation File: TRMM_16AUG2011.nc
> >> Configuration File:
> >> /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
> >> GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum
> =
> >> 240000, Min = 0, Max = 148.8
> >> ncattinq: ncid 4; varid -1; attname "Projection": Attribute not
found
> >>
> >>
> >> One more thing I want to ask you about plot_point_obs. Actually
> >> plot_point_obs executable does not exits in my METv3.0.1/bin.
Suppose I
> > need
> >> it, then what I have to do?
> >>
> >> Thanks
> >> HARVIR
> >>
> >> *PS: All the data files and script files have been transferred in
> >> incoming/irap/met_help/singh_data
> >> *
> >> On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
> >> met_help at ucar.edu> wrote:
> >>
> >>> Harvir,
> >>>
> >>> Regarding your second request related to TRMM data, I wrote a
small
> > script
> >>> (attached) to convert TRMM_16AUG2011.asc into
> >>> a format that is readable by ascii2nc. Then, I plotted the
observation
> >>> points using the MET utility plot_point_obs and
> >>> created the attached image showing the range of points. It is
> important
> > to
> >>> note that I had to make several assumptions
> >>> about your observations to do this conversion. The assumptions
are:
> >>>
> >>> 1. Your observations are surface observations (column 1 -
message type
> >>> ADPSFC). This does make sense if your data is
> >>> accumulated precipitation, which I believe it is.
> >>> 2. The valid time of your observations is 2011-08-11 00:00:00
UTC.
> >>> Obviously, the date is shown in your filename, but
> >>> I could not find the valid time.
> >>> 3. The elevation in msl of all observing points is 0m, which is
> > definitely
> >>> a bad assumption, but for "surface"
> >>> observations will not matter.
> >>> 4. The accumulation interval of your observations is 6 hours
(column
> 8).
> >>> If this is not correct, please modify the
> >>> script I attached accordingly.
> >>> 5. The height in msl of the observation points is 0m (see
assumption
> 3).
> >>>
> >>> Once you have model data that contains APCP with the same
accumulation
> >>> interval as that of your observations, you should
> >>> be able to use this acii2nc-generate observations in point_stat.
> >>>
> >>> Here are the steps to generate the ascii2nc NetCDF obs data file
and
> plot
> >>> using the script I am sending. These commands
> >>> assume that the MET bin/ sub-folder is on your path.
> >>>
> >>> $ chmod +x trmm_conv.pl
> >>> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
> >>> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
> >>> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15 45
45 120
> >>>
> >>> Please let me know if you have any questions.
> >>>
> >>> Thanks,
> >>>
> >>> Paul
> >>>
> >>>
> >>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>
> >>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
> >>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>> Queue: met_help
> >>>> Subject: Need help for ASCII2NC and Point Stat tool
> >>>> Owner: Nobody
> >>>> Requestors: harvirduphy at gmail.com
> >>>> Status: new
> >>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061>
> >>>>
> >>>>
> >>>> Dear Paul,
> >>>>
> >>>> I would like to thank you Paul for your necessary suggestions.
I've
> >>>> implemented them and now its work but I want use this file (
output
> from
> >>>> ascii2nc) for Point Stat tool. I have tried to run point_stat,
its
> >>> running
> >>>> and creating some stat files too but stats file having only
headers no
> >>>> statistics. I'm not getting why its so?
> >>>> I've transferred data files as well as script file in
> >>>> incoming/irap/met_help/singh_data. Here I want to run it for
TMP and
> >>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
> >>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
> >>>> 20110601_060000_rf.dat
> >>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file
is WRF
> >>>> outfile in GRIB1 format (output from WPPv3.2) named as
> >>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
> >>>>
> >>>>
> >>>>
> >>>> I want to ask you something on some other data. Suppose I have
some
> > ASCII
> >>>> data having only 3 columns viz. Latitude, Longitude and
Rainfall
> (say).
> >>> Can
> >>>> we convert it through ascii2nc for MET supported format? If
yes, then
> >>> what
> >>>> we can take rest of the columns. I also transferred this kind
of data
> >>> named
> >>>> as TRMM_16AUG2011.asc,
> >>>> TRMM_16AUG2011.dat
> >>>> I want to use point stat for this data also.
> >>>>
> >>>> Thank you in advance.
> >>>>
> >>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
> >>> met_help at ucar.edu
> >>>>> wrote:
> >>>>
> >>>>> Harvir,
> >>>>>
> >>>>> I have identified several problems with the ascii2nc input
file
> >>>>> suflnd_20110601_06z_rrf.dat:
> >>>>>
> >>>>> 1. ascii2nc input columns must be space-delimited. Several
of the
> >>> columns
> >>>>> in the input file are tab delimited. Do a
> >>>>> find/replace on the file to change all tabs to spaces.
> >>>>>
> >>>>> 2. After you perform step 1, you will see the following error
when
> you
> >>> try
> >>>>> to run ascii2nc:
> >>>>>
> >>>>> $ ascii2nc.sh
> >>>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
> >>>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
> >>>>> Processing 11414 observations.
> >>>>>
> >>>>>
> >>>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
> >>>>>
> >>>>>
> >>>>> The reason for this error is that your file appears to contain
> >>> accumulated
> >>>>> precipitation observations (GRIB code 61 in
> >>>>> column 7). In this case, ascii2nc is expecting an
accumulation
> > interval
> >>> in
> >>>>> column 8 in the format HH, for example 06 or
> >>>>> 24. That is the reason for the error message above.
> >>>>>
> >>>>> 3. Column 9 should contain the height above ground of the
> observation
> >>>>> station in meters. For accumulated
> >>>>> precipitation, this information is ignored by ascii2nc because
this
> is
> > a
> >>>>> "surface" observation. We typically use a fill
> >>>>> value of -9999, indicating invalid data.
> >>>>>
> >>>>> 4. Column 10 should contain the observed values. I see valid
> >>> observation
> >>>>> values on some rows, but many are -99.0 which
> >>>>> I interpret to mean invalid data. In this case, it does not
make
> sense
> >>> to
> >>>>> include invalid observation data, and those
> >>>>> lines should be deleted.
> >>>>>
> >>>>> I took the liberty of making a couple assumptions about your
data and
> >>>>> trying to adjust the input file that you sent me
> >>>>> to something that ascii2nc can read. The assumptions I made
are:
> >>>>>
> >>>>> 1. All of your observations are "surface" observations, so
the
> message
> >>>>> type (column 1) was set to ADPSFC
> >>>>> 2. The accumulation interval is 6 hours (column 8 is set to
06)
> >>>>> 3. The observation values are in mm, and left as-is
> >>>>>
> >>>>> If these assumptions, mainly #2 and #3, are incorrect, please
make
> the
> >>>>> appropriate adjustments. I attached the "fixed"
> >>>>> file to this email for you to review and change as you wish.
Please
> > let
> >>> me
> >>>>> know if you have any questions.
> >>>>>
> >>>>> Paul
> >>>>>
> >>>>>
> >>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>>>
> >>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689
>
> >>>>>>
> >>>>>>> Dear Paul,
> >>>>>>>
> >>>>>>> Accept My apology. Actually yesterday I forgot to
attached th
> >>> files.
> >>>>>> However, as per your advice through ftp I have putted the
same
> > attached
> >>>>>> files there also. and its path is
> >>>>>> *incoming/irap/met_help/singh_data*
> >>>>>>
> >>>>>>> I'm sending you ASCII file and ascii2nc script. Please have
close
> > look
> >>>>> on
> >>>>>>> it if am wrong somewhere please let me know.
> >>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful
> or
> >>>>> not
> >>>>>>> in pcp_combine ... However, I've run all the test scripts
> >>> successfully.
> >>>>>>> Thanx in advance...
> >>>>>>>
> >>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
> >>>>>>> met_help at ucar.edu> wrote:
> >>>>>>>
> >>>>>>>> Harvir,
> >>>>>>>>
> >>>>>>>> I'm afraid that MET cannot read TRMM data directly. The
best way
> to
> >>>>> read
> >>>>>>>> TRMM data into MET would be to convert the
> >>>>>>>> data from it's current format into a NetCDF file that has
the same
> >>>>> format
> >>>>>>>> as the output of the MET pcp_combine tool. If
> >>>>>>>> you ran the MET test scripts after you built MET, you
should be
> able
> >>> to
> >>>>>>>> find pcp_combine output files in the sub-folder
> >>>>>>>> out/pcp_combine under your MET build folder. If you are
> successful
> >>> in
> >>>>>>>> converting your data in this way, please send us
> >>>>>>>> your code so that we can make it available for other MET
users who
> >>>>> might
> >>>>>>>> want to use TRMM data. I'm sorry the answer to
> >>>>>>>> this question is not more simple.
> >>>>>>>>
> >>>>>>>> Regarding your ascii2nc question, can you please send us
the
> > ascii2nc
> >>>>>>>> input that is causing your error? Also, what is
> >>>>>>>> the command line statement that you are using to invoke
ascii2nc?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Paul
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh}
wrote:
> >>>>>>>>>
> >>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
> >>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>>>>>>> Queue: met_help
> >>>>>>>>> Subject: Getting Problem in converting TRMM rainfall
to MET
> >>>>> format
> >>>>>>>>> Owner: Nobody
> >>>>>>>>> Requestors: harvirduphy at gmail.com
> >>>>>>>>> Status: new
> >>>>>>>>> Ticket <URL:
> >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Dear Sir /Madam,
> >>>>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat
MET
> tool?
> >>>>>>>> which
> >>>>>>>>> converter should did I use to convert TRMM Rainfall data
(.nc or
> >>> .bin)
> >>>>>>>> to
> >>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
> >>>>>>>> (UPPv1.0),
> >>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
> >>>>>>>>> I want to use some ASCII DATA may be output of some models
like
> GFS
> >>>>> etc
> >>>>>>>> ..
> >>>>>>>>> I have extracted data in the desired 10 columns. I'm
trying to
> use
> >>>>>>>> *ascii2nc
> >>>>>>>>> *this also giving some error like: open_met_ascii() ->
line
> number
> > 0
> >>>>>>>> doesn't
> >>>>>>>>> contain the expected number of colu
> >>>>>>>>> N_items = 7
> >>>>>>>>> N_chars = 512
> >>>>>>>>> N_ints = 100
> >>>>>>>>> LineNumber = 1
> >>>>>>>>>
> >>>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
> >>>>>>>>> Item[ 1] = "48.68"
> >>>>>>>>> Item[ 2] = "9.23 "
> >>>>>>>>> Item[ 3] = "391.0 61 "
> >>>>>>>>> Item[ 4] = "974.8"
> >>>>>>>>>
> >>>>>>>>> Item[ 5] = "1021.5"
> >>>>>>>>> Item[ 6] = "-99.0"
> >>>>>>>>>
> >>>>>>>>> Offset[ 0] = 0
> >>>>>>>>> Offset[ 1] = 32
> >>>>>>>>> Offset[ 2] = 41
> >>>>>>>>> Offset[ 3] = 47
> >>>>>>>>> Offset[ 4] = 57
> >>>>>>>>>
> >>>>>>>>> Offset[ 5] = 64
> >>>>>>>>> Offset[ 6] = 71
> >>>>>>>>>
> >>>>>>>>> Please Help me....
> >>>>>>>>> Earliest reply will be appreciated.
> >>>>>>>>>
> >>>>>>>>> Thanks in advance for any help you can provide!
> >>>>>>>>>
> >>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for
Post
> >>>>> Processing
> >>>>>>>>> Tool "UPPv1.0 "*
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >
> >
> >
> >
> >
>
>
>
--
* Regards*
**************************************************************
**
* HARVIR SINGH <http://www.harvir007.webs.com>*
* To Know more <http://www.harvir007.webs.com> **
***************************************************************
------------------------------------------------
Subject: Re: Fwd: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and Point Stat tool
From: Paul Oldenburg
Time: Tue Aug 30 13:55:32 2011
Harvir,
If I understand you correctly, you are saying that your observation
data is accumulated rainfall from 20110816_000000 to
20110817_000000. Your model data appears to contain accumulated
rainfall from 20110815_000000 to 20110816_000000.
Obviously, these time windows do not overlap. Do you have the TRMM
data file for the day before, so it matches the
model data?
Paul
On 08/29/2011 06:29 AM, RAL HelpDesk {for Harvir Singh} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>
> Dear Paul,
> With the reference to your earlier mails, wrfprs_d01_2011-06-01_000_
>>
>> fcst01.grib1 was for forecast file, obs_20110601_060000_rf.dat
observation
>> file from GTS data and
>> TRMM_16AUG2011.dat is from TRMM rainfall data.
>>
>
>
> Now I've transferred new set of data files viz.
> Model forecast file: wrfprs_d01_2011-08-16_000_fcst01.grib1
> Observation file : TRMM_RF_16AUG2011_0.25.txt
>
> and TRMM Rainfall data is from 00Z 16 AUG 2011 to 00Z 17 AUG 2011
and model
> data is for 24 hr acumulated..
> thanx
>
>
>
> On 26 August 2011 20:05, RAL HelpDesk {for Paul Oldenburg} <
> met_help at ucar.edu> wrote:
>
>> Harvir,
>>
>> In the last email I sent you, I said that your rainfall observation
data in
>> your model data file,
>> wrfprs_d01_2011-06-01_000_fcst01.grib1, had an accumulation
interval of 24
>> hours. That means that the amount of
>> rainfall in the model data file is from a 24 hour period.
>>
>> Do you know what the accumulation interval of the rainfall data you
sent me
>> in either obs_20110601_060000_rf.dat or
>> TRMM_16AUG2011.dat is? I see the following information at the top
of the
>> file TRMM_16AUG2011.dat:
>>
>> Selected parameter: TRMM TMPA-RT Daily Accumulated Rainfall
>> Selected area: lat=[16.161S,45.311N], lon=[44.182E,121.960E]
>> Selected time period: (16Aug2011-16Aug2011)
>>
>> Does that mean that it is 24 hour accumulated rainfall from 2011-
08-15
>> 00:00:00 to 2011-08-16 00:00:00? If so, then it
>> has a 24 hour accumulation interval, but it does not match the date
in your
>> model data file. Do you have a TRMM file
>> that is valid at the same time as your model data file?
>>
>> I need to know the answers to these questions before I can make
further
>> progress helping you verify rainfall. It is
>> very important that both the accumulation interval and the valid
time of
>> your model and observation data match.
>>
>> Thanks,
>>
>> Paul
>>
>>
>> On 08/26/2011 07:02 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>
>>> Dear Paul,
>>>
>>> Thank You for your prompt reply. Ofcourse yes I've to compute
Threat
>> score
>>> too.. For this point of time I just want to go with point stat.
and I'm
>>> trying it for WRF ARW core (output data) it. I've tried
point_stat, its
>>> running but giving headers only in stat txt files. I'm unable to
rectify
>> it.
>>> Please help me to come out of this problem. I've transfrred data
in
>>> incoming/irap/met_help/singh_data
>>> thanx in advance!
>>>
>>> ---------- Forwarded message ----------
>>> From: RAL HelpDesk {for Paul Oldenburg} <met_help at ucar.edu>
>>> Date: 23 August 2011 21:55
>>> Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and
Point
>> Stat
>>> tool
>>> To: harvirduphy at gmail.com
>>>
>>>
>>> Harvir,
>>>
>>> There are a couple points to discuss before we continue. First, I
>> recommend
>>> that you use point_stat at this time to
>>> perform verification on your model data. The reason is that
converting
>> your
>>> observation data to a format that grid_stat
>>> can read is more difficult because it must contain projection
information
>>> that matches your model data projection. Why
>>> do you want to run grid_stat? Are you interested in calculating
>>> neighborhood statistics or just basic verification
>>> statistics like RMSE and the Equitable Threat Score? If you only
want
>> basic
>>> statistics, point_stat will work just fine.
>>>
>>> Second, for rainfall data, you said that the assumptions that I
have made
>>> about your observations are correct. If that
>>> is the case, and the accumulation interval of your observation
data is 6
>>> hours, then there is a mis-match with the
>>> accumulation interval in your model data, which is 24 hours (see
0-24hr
>>> acc):
>>>
>>> $ wgrib wrfprs_d01_2011-06-01_000_fcst01.grib1 | grep APCP
>>>
>>
252:323426:d=11053100:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
>>> acc:NAve=0
>>>
>>> If you are sure that your rainfall observations have an
accumulation
>>> interval of 6 hours, then you must combine four of
>>> your 6 hour observation files together to form rainfall
observations with
>> an
>>> accumulation interval of 24 hours. A good
>>> way to do this is to use the MET tool pcp_combine after you run
ascii2nc
>> on
>>> your rainfall observation files.
>>>
>>> Finally, I tried to apply the model data that you gave me with
surface
>>> temperature to the observation data you gave me
>>> last week, 20110601_060000_tt.dat. I put the files involved in
this
>>> experiment are on the RAL FTP site in the archive
>>>
ftp.rap.ucar.edu:/incoming/irap/met_help/singh_data/PointStatVx.tar.gz.
>> If
>>> you download this and unpack it, you should
>>> be able to reproduce the output file that I included,
>>> point_stat_240000L_20110601_000000V.stat. This file contains
>>> basic verification statistics, like continuous and categorical
>> statistics.
>>> I used the threshold 273.15 K, which is 0 C.
>>>
>>> Note that for this experiment to be valid, I had to open the time
window
>> to
>>> +/- 6 hours because your model data is valid
>>> at 20110601_000000 and your observations are valid at
20110601_060000.
>> This
>>> may or may not be an acceptable compromise
>>> to your for your work. Also, I assumed that the temperature
observations
>>> were all on the surface (message type ADPUPA).
>>> It also looked like your observations were in degrees Celsius, so
I
>>> converted them to Kelvin. Please let me know if
>>> you have any questions or trouble following my example. Please
look
>>> carefully at the files included in the archive,
>>> especially PointStatConfig_tmp and point_stat.sh.
>>>
>>> Paul
>>>
>>>
>>> $ tar xzvf PointStatVx.tar.gz
>>> $ cat obs_20110601_060000_tt.dat | ./obs_conv.pl >
>>> obs_20110601_060000_tt.txt
>>> $ ascii2nc obs_20110601_060000_tt.txt obs_20110601_060000_tt.nc -v
3
>>> Creating NetCDF Observation file: obs_20110601_060000_tt.nc
>>> Reading ASCII Observation file: obs_20110601_060000_tt.txt
>>> Processing 1099 observations.
>>> Finished processing 1099 observations.
>>> $ ./point_stat.sh
>>> GSL_RNG_TYPE=mt19937
>>> GSL_RNG_SEED=231822158
>>> Forecast File: wrfprs_d01_2011-06-01_000_fcst01.grib1
>>> Climatology File: none
>>> Configuration File: PointStatConfig_tmp
>>> Observation File: obs_20110601_060000_tt.nc
>>>
>>>
>>
--------------------------------------------------------------------------------
>>>
>>> Reading records for TMP/Z2.
>>> GRIB Record 244: Init = 20110531_000000, Valid = 20110601_000000,
Accum =
>>> 000000, Min = 292.579, Max = 303.165
>>> For TMP/Z2 found 1 forecast levels and 0 climatology levels.
>>>
>>>
>>
--------------------------------------------------------------------------------
>>>
>>> Searching 1099 observations from 1099 PrepBufr messages.
>>>
>>>
>>
--------------------------------------------------------------------------------
>>>
>>> Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC, over
region
>>> FULL, for interpolation method UW_MEAN(1),
>>> using 58 pairs.
>>> Number of matched pairs = 58
>>> Observations processed = 1099
>>> Rejected: GRIB code = 0
>>> Rejected: valid time = 0
>>> Rejected: bad obs value = 0
>>> Rejected: off the grid = 1041
>>> Rejected: level mismatch = 0
>>> Rejected: message type = 0
>>> Rejected: masking region = 0
>>> Rejected: bad fcst value = 0
>>> Computing Categorical Statistics.
>>> Computing Continuous Statistics.
>>> Computing Scalar Partial Sums.
>>>
>>>
>>
--------------------------------------------------------------------------------
>>>
>>> Output file: ./point_stat_240000L_20110601_000000V.stat
>>>
>>>
>>> On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>>
>>>> Dear Paul,
>>>> I have gone through your assumptions, might seems correct. But I
am
>> having
>>>> some problems when I running Point_stat for rainfall, I am
getting some
>>>> error as:
>>>>
>>>> $ sh point_stat.sh
>>>> ERROR: GCInfo::set_gcinfo() -> unsupported input file type
"APCP/A24"
>>>> I have changed configure, "APCP/P24" "APCP/P1000", "APCP/P925"
etc.
>>>> and I ma getting similar type of error. Suggest me how can I
rectify it.
>>>>
>>>>
>>>> I also tried for GRID_STAT and I got the error as follows
>>>> sh Grid_stat.sh
>>>> GSL_RNG_TYPE=mt19937
>>>> GSL_RNG_SEED=18446744073465844558
>>>> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
>>>> Observation File: TRMM_16AUG2011.nc
>>>> Configuration File:
>>>> /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
>>>> GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum
>> =
>>>> 240000, Min = 0, Max = 148.8
>>>> ncattinq: ncid 4; varid -1; attname "Projection": Attribute not
found
>>>>
>>>>
>>>> One more thing I want to ask you about plot_point_obs. Actually
>>>> plot_point_obs executable does not exits in my METv3.0.1/bin.
Suppose I
>>> need
>>>> it, then what I have to do?
>>>>
>>>> Thanks
>>>> HARVIR
>>>>
>>>> *PS: All the data files and script files have been transferred in
>>>> incoming/irap/met_help/singh_data
>>>> *
>>>> On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>> Harvir,
>>>>>
>>>>> Regarding your second request related to TRMM data, I wrote a
small
>>> script
>>>>> (attached) to convert TRMM_16AUG2011.asc into
>>>>> a format that is readable by ascii2nc. Then, I plotted the
observation
>>>>> points using the MET utility plot_point_obs and
>>>>> created the attached image showing the range of points. It is
>> important
>>> to
>>>>> note that I had to make several assumptions
>>>>> about your observations to do this conversion. The assumptions
are:
>>>>>
>>>>> 1. Your observations are surface observations (column 1 -
message type
>>>>> ADPSFC). This does make sense if your data is
>>>>> accumulated precipitation, which I believe it is.
>>>>> 2. The valid time of your observations is 2011-08-11 00:00:00
UTC.
>>>>> Obviously, the date is shown in your filename, but
>>>>> I could not find the valid time.
>>>>> 3. The elevation in msl of all observing points is 0m, which is
>>> definitely
>>>>> a bad assumption, but for "surface"
>>>>> observations will not matter.
>>>>> 4. The accumulation interval of your observations is 6 hours
(column
>> 8).
>>>>> If this is not correct, please modify the
>>>>> script I attached accordingly.
>>>>> 5. The height in msl of the observation points is 0m (see
assumption
>> 3).
>>>>>
>>>>> Once you have model data that contains APCP with the same
accumulation
>>>>> interval as that of your observations, you should
>>>>> be able to use this acii2nc-generate observations in point_stat.
>>>>>
>>>>> Here are the steps to generate the ascii2nc NetCDF obs data file
and
>> plot
>>>>> using the script I am sending. These commands
>>>>> assume that the MET bin/ sub-folder is on your path.
>>>>>
>>>>> $ chmod +x trmm_conv.pl
>>>>> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
>>>>> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
>>>>> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15 45
45 120
>>>>>
>>>>> Please let me know if you have any questions.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>
>>>>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>> Queue: met_help
>>>>>> Subject: Need help for ASCII2NC and Point Stat tool
>>>>>> Owner: Nobody
>>>>>> Requestors: harvirduphy at gmail.com
>>>>>> Status: new
>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061>
>>>>>>
>>>>>>
>>>>>> Dear Paul,
>>>>>>
>>>>>> I would like to thank you Paul for your necessary suggestions.
I've
>>>>>> implemented them and now its work but I want use this file (
output
>> from
>>>>>> ascii2nc) for Point Stat tool. I have tried to run point_stat,
its
>>>>> running
>>>>>> and creating some stat files too but stats file having only
headers no
>>>>>> statistics. I'm not getting why its so?
>>>>>> I've transferred data files as well as script file in
>>>>>> incoming/irap/met_help/singh_data. Here I want to run it for
TMP and
>>>>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
>>>>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
>>>>>> 20110601_060000_rf.dat
>>>>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast file
is WRF
>>>>>> outfile in GRIB1 format (output from WPPv3.2) named as
>>>>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I want to ask you something on some other data. Suppose I have
some
>>> ASCII
>>>>>> data having only 3 columns viz. Latitude, Longitude and
Rainfall
>> (say).
>>>>> Can
>>>>>> we convert it through ascii2nc for MET supported format? If
yes, then
>>>>> what
>>>>>> we can take rest of the columns. I also transferred this kind
of data
>>>>> named
>>>>>> as TRMM_16AUG2011.asc,
>>>>>> TRMM_16AUG2011.dat
>>>>>> I want to use point stat for this data also.
>>>>>>
>>>>>> Thank you in advance.
>>>>>>
>>>>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
>>>>> met_help at ucar.edu
>>>>>>> wrote:
>>>>>>
>>>>>>> Harvir,
>>>>>>>
>>>>>>> I have identified several problems with the ascii2nc input
file
>>>>>>> suflnd_20110601_06z_rrf.dat:
>>>>>>>
>>>>>>> 1. ascii2nc input columns must be space-delimited. Several
of the
>>>>> columns
>>>>>>> in the input file are tab delimited. Do a
>>>>>>> find/replace on the file to change all tabs to spaces.
>>>>>>>
>>>>>>> 2. After you perform step 1, you will see the following error
when
>> you
>>>>> try
>>>>>>> to run ascii2nc:
>>>>>>>
>>>>>>> $ ascii2nc.sh
>>>>>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
>>>>>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
>>>>>>> Processing 11414 observations.
>>>>>>>
>>>>>>>
>>>>>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
>>>>>>>
>>>>>>>
>>>>>>> The reason for this error is that your file appears to contain
>>>>> accumulated
>>>>>>> precipitation observations (GRIB code 61 in
>>>>>>> column 7). In this case, ascii2nc is expecting an
accumulation
>>> interval
>>>>> in
>>>>>>> column 8 in the format HH, for example 06 or
>>>>>>> 24. That is the reason for the error message above.
>>>>>>>
>>>>>>> 3. Column 9 should contain the height above ground of the
>> observation
>>>>>>> station in meters. For accumulated
>>>>>>> precipitation, this information is ignored by ascii2nc because
this
>> is
>>> a
>>>>>>> "surface" observation. We typically use a fill
>>>>>>> value of -9999, indicating invalid data.
>>>>>>>
>>>>>>> 4. Column 10 should contain the observed values. I see valid
>>>>> observation
>>>>>>> values on some rows, but many are -99.0 which
>>>>>>> I interpret to mean invalid data. In this case, it does not
make
>> sense
>>>>> to
>>>>>>> include invalid observation data, and those
>>>>>>> lines should be deleted.
>>>>>>>
>>>>>>> I took the liberty of making a couple assumptions about your
data and
>>>>>>> trying to adjust the input file that you sent me
>>>>>>> to something that ascii2nc can read. The assumptions I made
are:
>>>>>>>
>>>>>>> 1. All of your observations are "surface" observations, so
the
>> message
>>>>>>> type (column 1) was set to ADPSFC
>>>>>>> 2. The accumulation interval is 6 hours (column 8 is set to
06)
>>>>>>> 3. The observation values are in mm, and left as-is
>>>>>>>
>>>>>>> If these assumptions, mainly #2 and #3, are incorrect, please
make
>> the
>>>>>>> appropriate adjustments. I attached the "fixed"
>>>>>>> file to this email for you to review and change as you wish.
Please
>>> let
>>>>> me
>>>>>>> know if you have any questions.
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>>
>>>>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>>>
>>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689
>
>>>>>>>>
>>>>>>>>> Dear Paul,
>>>>>>>>>
>>>>>>>>> Accept My apology. Actually yesterday I forgot to
attached th
>>>>> files.
>>>>>>>> However, as per your advice through ftp I have putted the
same
>>> attached
>>>>>>>> files there also. and its path is
>>>>>>>> *incoming/irap/met_help/singh_data*
>>>>>>>>
>>>>>>>>> I'm sending you ASCII file and ascii2nc script. Please have
close
>>> look
>>>>>>> on
>>>>>>>>> it if am wrong somewhere please let me know.
>>>>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
successful
>> or
>>>>>>> not
>>>>>>>>> in pcp_combine ... However, I've run all the test scripts
>>>>> successfully.
>>>>>>>>> Thanx in advance...
>>>>>>>>>
>>>>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg} <
>>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>>
>>>>>>>>>> Harvir,
>>>>>>>>>>
>>>>>>>>>> I'm afraid that MET cannot read TRMM data directly. The
best way
>> to
>>>>>>> read
>>>>>>>>>> TRMM data into MET would be to convert the
>>>>>>>>>> data from it's current format into a NetCDF file that has
the same
>>>>>>> format
>>>>>>>>>> as the output of the MET pcp_combine tool. If
>>>>>>>>>> you ran the MET test scripts after you built MET, you
should be
>> able
>>>>> to
>>>>>>>>>> find pcp_combine output files in the sub-folder
>>>>>>>>>> out/pcp_combine under your MET build folder. If you are
>> successful
>>>>> in
>>>>>>>>>> converting your data in this way, please send us
>>>>>>>>>> your code so that we can make it available for other MET
users who
>>>>>>> might
>>>>>>>>>> want to use TRMM data. I'm sorry the answer to
>>>>>>>>>> this question is not more simple.
>>>>>>>>>>
>>>>>>>>>> Regarding your ascii2nc question, can you please send us
the
>>> ascii2nc
>>>>>>>>>> input that is causing your error? Also, what is
>>>>>>>>>> the command line statement that you are using to invoke
ascii2nc?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Paul
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh}
wrote:
>>>>>>>>>>>
>>>>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
>>>>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>>>>>> Queue: met_help
>>>>>>>>>>> Subject: Getting Problem in converting TRMM rainfall
to MET
>>>>>>> format
>>>>>>>>>>> Owner: Nobody
>>>>>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>>>>>> Status: new
>>>>>>>>>>> Ticket <URL:
>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Dear Sir /Madam,
>>>>>>>>>>> I am using TRMM rainfall Netcdf or bin data for grid_stat
MET
>> tool?
>>>>>>>>>> which
>>>>>>>>>>> converter should did I use to convert TRMM Rainfall data
(.nc or
>>>>> .bin)
>>>>>>>>>> to
>>>>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
>>>>>>>>>> (UPPv1.0),
>>>>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>>>>>>> I want to use some ASCII DATA may be output of some models
like
>> GFS
>>>>>>> etc
>>>>>>>>>> ..
>>>>>>>>>>> I have extracted data in the desired 10 columns. I'm
trying to
>> use
>>>>>>>>>> *ascii2nc
>>>>>>>>>>> *this also giving some error like: open_met_ascii() ->
line
>> number
>>> 0
>>>>>>>>>> doesn't
>>>>>>>>>>> contain the expected number of colu
>>>>>>>>>>> N_items = 7
>>>>>>>>>>> N_chars = 512
>>>>>>>>>>> N_ints = 100
>>>>>>>>>>> LineNumber = 1
>>>>>>>>>>>
>>>>>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000 "
>>>>>>>>>>> Item[ 1] = "48.68"
>>>>>>>>>>> Item[ 2] = "9.23 "
>>>>>>>>>>> Item[ 3] = "391.0 61 "
>>>>>>>>>>> Item[ 4] = "974.8"
>>>>>>>>>>>
>>>>>>>>>>> Item[ 5] = "1021.5"
>>>>>>>>>>> Item[ 6] = "-99.0"
>>>>>>>>>>>
>>>>>>>>>>> Offset[ 0] = 0
>>>>>>>>>>> Offset[ 1] = 32
>>>>>>>>>>> Offset[ 2] = 41
>>>>>>>>>>> Offset[ 3] = 47
>>>>>>>>>>> Offset[ 4] = 57
>>>>>>>>>>>
>>>>>>>>>>> Offset[ 5] = 64
>>>>>>>>>>> Offset[ 6] = 71
>>>>>>>>>>>
>>>>>>>>>>> Please Help me....
>>>>>>>>>>> Earliest reply will be appreciated.
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance for any help you can provide!
>>>>>>>>>>>
>>>>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for
Post
>>>>>>> Processing
>>>>>>>>>>> Tool "UPPv1.0 "*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: Harvir Singh
Time: Thu Sep 01 06:26:56 2011
Dear Paul,
Yes I have the TRMM data file for the day before and I've transferred
to
met_help. The TRMM rainfall I'm using, is available on web also i.e. :
http://disc2.nascom.nasa.gov/Giovanni/tovas/realtime.3B42RT.shtml
(ASCII)
and for 15th Aug 2011 data @
http://disc2.nascom.nasa.gov/daac-
bin/Giovanni/tovas/Giovanni_cgi.pl?west=45.0&north=45.0&east=121.0&south=-
16.0¶ms=0%7C3B42RT&plot_type=Area+Plot&byr=2011&bmo=08&bdy=14&bhr=00&eyr=2011&emo=08&edy=15&ehr=00&begin_date=2008%2F10%2F01%2F00&end_date=2011%2F08%2F31%2F12&cbar=cdyn&cmin=&cmax=&yaxis=ydyn&ymin=&ymax=&yint=&ascres=0.25x0.25&global_cfg=tovas.global.cfg.pl&instance_id=realtime&prod_id=3B42RT&action=ASCII+Output.
Actually I've tried a lot to work on Point_stat but every time I
struck
somewhere.. so I've some questions:
1. This METv3.0.1 software is only meant for WRF-NMM or WRF-ARW as
well.
For this point of time I'm using WRF-ARW output to WRF Post
Processing
ie WPPv3.2 and converting to GRIB1 format. And for observation data
I'm
downloading ASCII data as above said link. When I'm converting ASCII
data to
NetCDF I've also taken several assumptions for surface data like:
Massage type: ADPSFC
Station ID: TRMM
Date : 20110816_000000
Lat
Lon
Elevation: -9999 ( as data is not available)
g.c. : 61 for Total precipitation
Level : 24 hour ( for 24 accumulation rainfall)
height : -9999 ( as data is not available)
for both forecast and observation I've used 245x305 grid points at the
interval of 0.25 degree within the Indian region ie 16S to 45N and
45E to
121E.
And fore converting WRF-ARF outfiles by WPPv3.2 have used copygb with
the
given information as:
grid='255 0 305 245 -16000 45000 128 45000 121000 250 250 64'
${POSTEXEC}/copygb.exe -g"${grid}" -x
WRFPRS_${domain}_${YY}-${MM}-${DD}.${fhr}_fcst01 WRFPRS
_fcst01/wrfprs_${domain}_${YY}-${MM}-${DD}_${fhr}_fcst01.grib1
2. Here I'm using Gridded data either from TRMM or from GTS or AWS etc
...
then why I can't use Grid_stat?
I'm not getting it. I may be using some wrong information during the
data
format conversion i.e. creating 10 columns for ASCII2NC. I've tried it
for
several combinations for point stat as NetCDF as forecast file and
GRIB1 as
observation file and vice-verso, also tried several combinations in
PointStatConfug_rf as:
// fcst_field[] = ["APCP_A24(*,*)"];
fcst_field[] = ["APCP/A24"];
//obs_field[] = ["APCP/A24"];
//obs_field[] = [];
obs_field[] = ["APCP_A24(0,5,*,*)"];
and many more....
Thanx
HARVIR
On 31 August 2011 01:25, RAL HelpDesk {for Paul Oldenburg} <
met_help at ucar.edu> wrote:
> Harvir,
>
> If I understand you correctly, you are saying that your observation
data is
> accumulated rainfall from 20110816_000000 to
> 20110817_000000. Your model data appears to contain accumulated
rainfall
> from 20110815_000000 to 20110816_000000.
> Obviously, these time windows do not overlap. Do you have the TRMM
data
> file for the day before, so it matches the
> model data?
>
> Paul
>
>
> On 08/29/2011 06:29 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >
> > Dear Paul,
> > With the reference to your earlier mails, wrfprs_d01_2011-06-
01_000_
> >>
> >> fcst01.grib1 was for forecast file, obs_20110601_060000_rf.dat
> observation
> >> file from GTS data and
> >> TRMM_16AUG2011.dat is from TRMM rainfall data.
> >>
> >
> >
> > Now I've transferred new set of data files viz.
> > Model forecast file: wrfprs_d01_2011-08-16_000_fcst01.grib1
> > Observation file : TRMM_RF_16AUG2011_0.25.txt
> >
> > and TRMM Rainfall data is from 00Z 16 AUG 2011 to 00Z 17 AUG 2011
and
> model
> > data is for 24 hr acumulated..
> > thanx
> >
> >
> >
> > On 26 August 2011 20:05, RAL HelpDesk {for Paul Oldenburg} <
> > met_help at ucar.edu> wrote:
> >
> >> Harvir,
> >>
> >> In the last email I sent you, I said that your rainfall
observation data
> in
> >> your model data file,
> >> wrfprs_d01_2011-06-01_000_fcst01.grib1, had an accumulation
interval of
> 24
> >> hours. That means that the amount of
> >> rainfall in the model data file is from a 24 hour period.
> >>
> >> Do you know what the accumulation interval of the rainfall data
you sent
> me
> >> in either obs_20110601_060000_rf.dat or
> >> TRMM_16AUG2011.dat is? I see the following information at the
top of
> the
> >> file TRMM_16AUG2011.dat:
> >>
> >> Selected parameter: TRMM TMPA-RT Daily Accumulated Rainfall
> >> Selected area: lat=[16.161S,45.311N], lon=[44.182E,121.960E]
> >> Selected time period: (16Aug2011-16Aug2011)
> >>
> >> Does that mean that it is 24 hour accumulated rainfall from 2011-
08-15
> >> 00:00:00 to 2011-08-16 00:00:00? If so, then it
> >> has a 24 hour accumulation interval, but it does not match the
date in
> your
> >> model data file. Do you have a TRMM file
> >> that is valid at the same time as your model data file?
> >>
> >> I need to know the answers to these questions before I can make
further
> >> progress helping you verify rainfall. It is
> >> very important that both the accumulation interval and the valid
time of
> >> your model and observation data match.
> >>
> >> Thanks,
> >>
> >> Paul
> >>
> >>
> >> On 08/26/2011 07:02 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >>>
> >>> Dear Paul,
> >>>
> >>> Thank You for your prompt reply. Ofcourse yes I've to compute
Threat
> >> score
> >>> too.. For this point of time I just want to go with point stat.
and I'm
> >>> trying it for WRF ARW core (output data) it. I've tried
point_stat, its
> >>> running but giving headers only in stat txt files. I'm unable to
> rectify
> >> it.
> >>> Please help me to come out of this problem. I've transfrred data
in
> >>> incoming/irap/met_help/singh_data
> >>> thanx in advance!
> >>>
> >>> ---------- Forwarded message ----------
> >>> From: RAL HelpDesk {for Paul Oldenburg} <met_help at ucar.edu>
> >>> Date: 23 August 2011 21:55
> >>> Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and
Point
> >> Stat
> >>> tool
> >>> To: harvirduphy at gmail.com
> >>>
> >>>
> >>> Harvir,
> >>>
> >>> There are a couple points to discuss before we continue. First,
I
> >> recommend
> >>> that you use point_stat at this time to
> >>> perform verification on your model data. The reason is that
converting
> >> your
> >>> observation data to a format that grid_stat
> >>> can read is more difficult because it must contain projection
> information
> >>> that matches your model data projection. Why
> >>> do you want to run grid_stat? Are you interested in calculating
> >>> neighborhood statistics or just basic verification
> >>> statistics like RMSE and the Equitable Threat Score? If you
only want
> >> basic
> >>> statistics, point_stat will work just fine.
> >>>
> >>> Second, for rainfall data, you said that the assumptions that I
have
> made
> >>> about your observations are correct. If that
> >>> is the case, and the accumulation interval of your observation
data is
> 6
> >>> hours, then there is a mis-match with the
> >>> accumulation interval in your model data, which is 24 hours (see
0-24hr
> >>> acc):
> >>>
> >>> $ wgrib wrfprs_d01_2011-06-01_000_fcst01.grib1 | grep APCP
> >>>
> >>
>
252:323426:d=11053100:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
> >>> acc:NAve=0
> >>>
> >>> If you are sure that your rainfall observations have an
accumulation
> >>> interval of 6 hours, then you must combine four of
> >>> your 6 hour observation files together to form rainfall
observations
> with
> >> an
> >>> accumulation interval of 24 hours. A good
> >>> way to do this is to use the MET tool pcp_combine after you run
> ascii2nc
> >> on
> >>> your rainfall observation files.
> >>>
> >>> Finally, I tried to apply the model data that you gave me with
surface
> >>> temperature to the observation data you gave me
> >>> last week, 20110601_060000_tt.dat. I put the files involved in
this
> >>> experiment are on the RAL FTP site in the archive
> >>> ftp.rap.ucar.edu:
> /incoming/irap/met_help/singh_data/PointStatVx.tar.gz.
> >> If
> >>> you download this and unpack it, you should
> >>> be able to reproduce the output file that I included,
> >>> point_stat_240000L_20110601_000000V.stat. This file contains
> >>> basic verification statistics, like continuous and categorical
> >> statistics.
> >>> I used the threshold 273.15 K, which is 0 C.
> >>>
> >>> Note that for this experiment to be valid, I had to open the
time
> window
> >> to
> >>> +/- 6 hours because your model data is valid
> >>> at 20110601_000000 and your observations are valid at
20110601_060000.
> >> This
> >>> may or may not be an acceptable compromise
> >>> to your for your work. Also, I assumed that the temperature
> observations
> >>> were all on the surface (message type ADPUPA).
> >>> It also looked like your observations were in degrees Celsius,
so I
> >>> converted them to Kelvin. Please let me know if
> >>> you have any questions or trouble following my example. Please
look
> >>> carefully at the files included in the archive,
> >>> especially PointStatConfig_tmp and point_stat.sh.
> >>>
> >>> Paul
> >>>
> >>>
> >>> $ tar xzvf PointStatVx.tar.gz
> >>> $ cat obs_20110601_060000_tt.dat | ./obs_conv.pl >
> >>> obs_20110601_060000_tt.txt
> >>> $ ascii2nc obs_20110601_060000_tt.txt obs_20110601_060000_tt.nc
-v 3
> >>> Creating NetCDF Observation file: obs_20110601_060000_tt.nc
> >>> Reading ASCII Observation file: obs_20110601_060000_tt.txt
> >>> Processing 1099 observations.
> >>> Finished processing 1099 observations.
> >>> $ ./point_stat.sh
> >>> GSL_RNG_TYPE=mt19937
> >>> GSL_RNG_SEED=231822158
> >>> Forecast File: wrfprs_d01_2011-06-01_000_fcst01.grib1
> >>> Climatology File: none
> >>> Configuration File: PointStatConfig_tmp
> >>> Observation File: obs_20110601_060000_tt.nc
> >>>
> >>>
> >>
>
--------------------------------------------------------------------------------
> >>>
> >>> Reading records for TMP/Z2.
> >>> GRIB Record 244: Init = 20110531_000000, Valid =
20110601_000000, Accum
> =
> >>> 000000, Min = 292.579, Max = 303.165
> >>> For TMP/Z2 found 1 forecast levels and 0 climatology levels.
> >>>
> >>>
> >>
>
--------------------------------------------------------------------------------
> >>>
> >>> Searching 1099 observations from 1099 PrepBufr messages.
> >>>
> >>>
> >>
>
--------------------------------------------------------------------------------
> >>>
> >>> Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC,
over
> region
> >>> FULL, for interpolation method UW_MEAN(1),
> >>> using 58 pairs.
> >>> Number of matched pairs = 58
> >>> Observations processed = 1099
> >>> Rejected: GRIB code = 0
> >>> Rejected: valid time = 0
> >>> Rejected: bad obs value = 0
> >>> Rejected: off the grid = 1041
> >>> Rejected: level mismatch = 0
> >>> Rejected: message type = 0
> >>> Rejected: masking region = 0
> >>> Rejected: bad fcst value = 0
> >>> Computing Categorical Statistics.
> >>> Computing Continuous Statistics.
> >>> Computing Scalar Partial Sums.
> >>>
> >>>
> >>
>
--------------------------------------------------------------------------------
> >>>
> >>> Output file: ./point_stat_240000L_20110601_000000V.stat
> >>>
> >>>
> >>> On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>
> >>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >>>>
> >>>> Dear Paul,
> >>>> I have gone through your assumptions, might seems correct. But
I am
> >> having
> >>>> some problems when I running Point_stat for rainfall, I am
getting
> some
> >>>> error as:
> >>>>
> >>>> $ sh point_stat.sh
> >>>> ERROR: GCInfo::set_gcinfo() -> unsupported input file type
"APCP/A24"
> >>>> I have changed configure, "APCP/P24" "APCP/P1000", "APCP/P925"
etc.
> >>>> and I ma getting similar type of error. Suggest me how can I
rectify
> it.
> >>>>
> >>>>
> >>>> I also tried for GRID_STAT and I got the error as follows
> >>>> sh Grid_stat.sh
> >>>> GSL_RNG_TYPE=mt19937
> >>>> GSL_RNG_SEED=18446744073465844558
> >>>> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
> >>>> Observation File: TRMM_16AUG2011.nc
> >>>> Configuration File:
> >>>> /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
> >>>> GRIB Record 252: Init = 20110815_000000, Valid =
20110816_000000,
> Accum
> >> =
> >>>> 240000, Min = 0, Max = 148.8
> >>>> ncattinq: ncid 4; varid -1; attname "Projection": Attribute not
found
> >>>>
> >>>>
> >>>> One more thing I want to ask you about plot_point_obs. Actually
> >>>> plot_point_obs executable does not exits in my METv3.0.1/bin.
Suppose
> I
> >>> need
> >>>> it, then what I have to do?
> >>>>
> >>>> Thanks
> >>>> HARVIR
> >>>>
> >>>> *PS: All the data files and script files have been transferred
in
> >>>> incoming/irap/met_help/singh_data
> >>>> *
> >>>> On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
> >>>> met_help at ucar.edu> wrote:
> >>>>
> >>>>> Harvir,
> >>>>>
> >>>>> Regarding your second request related to TRMM data, I wrote a
small
> >>> script
> >>>>> (attached) to convert TRMM_16AUG2011.asc into
> >>>>> a format that is readable by ascii2nc. Then, I plotted the
> observation
> >>>>> points using the MET utility plot_point_obs and
> >>>>> created the attached image showing the range of points. It is
> >> important
> >>> to
> >>>>> note that I had to make several assumptions
> >>>>> about your observations to do this conversion. The
assumptions are:
> >>>>>
> >>>>> 1. Your observations are surface observations (column 1 -
message
> type
> >>>>> ADPSFC). This does make sense if your data is
> >>>>> accumulated precipitation, which I believe it is.
> >>>>> 2. The valid time of your observations is 2011-08-11 00:00:00
UTC.
> >>>>> Obviously, the date is shown in your filename, but
> >>>>> I could not find the valid time.
> >>>>> 3. The elevation in msl of all observing points is 0m, which
is
> >>> definitely
> >>>>> a bad assumption, but for "surface"
> >>>>> observations will not matter.
> >>>>> 4. The accumulation interval of your observations is 6 hours
(column
> >> 8).
> >>>>> If this is not correct, please modify the
> >>>>> script I attached accordingly.
> >>>>> 5. The height in msl of the observation points is 0m (see
assumption
> >> 3).
> >>>>>
> >>>>> Once you have model data that contains APCP with the same
> accumulation
> >>>>> interval as that of your observations, you should
> >>>>> be able to use this acii2nc-generate observations in
point_stat.
> >>>>>
> >>>>> Here are the steps to generate the ascii2nc NetCDF obs data
file and
> >> plot
> >>>>> using the script I am sending. These commands
> >>>>> assume that the MET bin/ sub-folder is on your path.
> >>>>>
> >>>>> $ chmod +x trmm_conv.pl
> >>>>> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
> >>>>> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
> >>>>> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15
45 45
> 120
> >>>>>
> >>>>> Please let me know if you have any questions.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Paul
> >>>>>
> >>>>>
> >>>>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>>>
> >>>>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
> >>>>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>>>> Queue: met_help
> >>>>>> Subject: Need help for ASCII2NC and Point Stat tool
> >>>>>> Owner: Nobody
> >>>>>> Requestors: harvirduphy at gmail.com
> >>>>>> Status: new
> >>>>>> Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061>
> >>>>>>
> >>>>>>
> >>>>>> Dear Paul,
> >>>>>>
> >>>>>> I would like to thank you Paul for your necessary
suggestions. I've
> >>>>>> implemented them and now its work but I want use this file (
output
> >> from
> >>>>>> ascii2nc) for Point Stat tool. I have tried to run
point_stat, its
> >>>>> running
> >>>>>> and creating some stat files too but stats file having only
headers
> no
> >>>>>> statistics. I'm not getting why its so?
> >>>>>> I've transferred data files as well as script file in
> >>>>>> incoming/irap/met_help/singh_data. Here I want to run it for
TMP and
> >>>>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
> >>>>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
> >>>>>> 20110601_060000_rf.dat
> >>>>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast
file is
> WRF
> >>>>>> outfile in GRIB1 format (output from WPPv3.2) named as
> >>>>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I want to ask you something on some other data. Suppose I
have some
> >>> ASCII
> >>>>>> data having only 3 columns viz. Latitude, Longitude and
Rainfall
> >> (say).
> >>>>> Can
> >>>>>> we convert it through ascii2nc for MET supported format? If
yes,
> then
> >>>>> what
> >>>>>> we can take rest of the columns. I also transferred this kind
of
> data
> >>>>> named
> >>>>>> as TRMM_16AUG2011.asc,
> >>>>>> TRMM_16AUG2011.dat
> >>>>>> I want to use point stat for this data also.
> >>>>>>
> >>>>>> Thank you in advance.
> >>>>>>
> >>>>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
> >>>>> met_help at ucar.edu
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> Harvir,
> >>>>>>>
> >>>>>>> I have identified several problems with the ascii2nc input
file
> >>>>>>> suflnd_20110601_06z_rrf.dat:
> >>>>>>>
> >>>>>>> 1. ascii2nc input columns must be space-delimited. Several
of the
> >>>>> columns
> >>>>>>> in the input file are tab delimited. Do a
> >>>>>>> find/replace on the file to change all tabs to spaces.
> >>>>>>>
> >>>>>>> 2. After you perform step 1, you will see the following
error when
> >> you
> >>>>> try
> >>>>>>> to run ascii2nc:
> >>>>>>>
> >>>>>>> $ ascii2nc.sh
> >>>>>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
> >>>>>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
> >>>>>>> Processing 11414 observations.
> >>>>>>>
> >>>>>>>
> >>>>>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
> >>>>>>>
> >>>>>>>
> >>>>>>> The reason for this error is that your file appears to
contain
> >>>>> accumulated
> >>>>>>> precipitation observations (GRIB code 61 in
> >>>>>>> column 7). In this case, ascii2nc is expecting an
accumulation
> >>> interval
> >>>>> in
> >>>>>>> column 8 in the format HH, for example 06 or
> >>>>>>> 24. That is the reason for the error message above.
> >>>>>>>
> >>>>>>> 3. Column 9 should contain the height above ground of the
> >> observation
> >>>>>>> station in meters. For accumulated
> >>>>>>> precipitation, this information is ignored by ascii2nc
because this
> >> is
> >>> a
> >>>>>>> "surface" observation. We typically use a fill
> >>>>>>> value of -9999, indicating invalid data.
> >>>>>>>
> >>>>>>> 4. Column 10 should contain the observed values. I see
valid
> >>>>> observation
> >>>>>>> values on some rows, but many are -99.0 which
> >>>>>>> I interpret to mean invalid data. In this case, it does not
make
> >> sense
> >>>>> to
> >>>>>>> include invalid observation data, and those
> >>>>>>> lines should be deleted.
> >>>>>>>
> >>>>>>> I took the liberty of making a couple assumptions about your
data
> and
> >>>>>>> trying to adjust the input file that you sent me
> >>>>>>> to something that ascii2nc can read. The assumptions I made
are:
> >>>>>>>
> >>>>>>> 1. All of your observations are "surface" observations, so
the
> >> message
> >>>>>>> type (column 1) was set to ADPSFC
> >>>>>>> 2. The accumulation interval is 6 hours (column 8 is set to
06)
> >>>>>>> 3. The observation values are in mm, and left as-is
> >>>>>>>
> >>>>>>> If these assumptions, mainly #2 and #3, are incorrect,
please make
> >> the
> >>>>>>> appropriate adjustments. I attached the "fixed"
> >>>>>>> file to this email for you to review and change as you wish.
> Please
> >>> let
> >>>>> me
> >>>>>>> know if you have any questions.
> >>>>>>>
> >>>>>>> Paul
> >>>>>>>
> >>>>>>>
> >>>>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh}
wrote:
> >>>>>>>>
> >>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
> >>>>>>>>
> >>>>>>>>> Dear Paul,
> >>>>>>>>>
> >>>>>>>>> Accept My apology. Actually yesterday I forgot to
attached th
> >>>>> files.
> >>>>>>>> However, as per your advice through ftp I have putted the
same
> >>> attached
> >>>>>>>> files there also. and its path is
> >>>>>>>> *incoming/irap/met_help/singh_data*
> >>>>>>>>
> >>>>>>>>> I'm sending you ASCII file and ascii2nc script. Please
have close
> >>> look
> >>>>>>> on
> >>>>>>>>> it if am wrong somewhere please let me know.
> >>>>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
> successful
> >> or
> >>>>>>> not
> >>>>>>>>> in pcp_combine ... However, I've run all the test scripts
> >>>>> successfully.
> >>>>>>>>> Thanx in advance...
> >>>>>>>>>
> >>>>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg}
<
> >>>>>>>>> met_help at ucar.edu> wrote:
> >>>>>>>>>
> >>>>>>>>>> Harvir,
> >>>>>>>>>>
> >>>>>>>>>> I'm afraid that MET cannot read TRMM data directly. The
best
> way
> >> to
> >>>>>>> read
> >>>>>>>>>> TRMM data into MET would be to convert the
> >>>>>>>>>> data from it's current format into a NetCDF file that has
the
> same
> >>>>>>> format
> >>>>>>>>>> as the output of the MET pcp_combine tool. If
> >>>>>>>>>> you ran the MET test scripts after you built MET, you
should be
> >> able
> >>>>> to
> >>>>>>>>>> find pcp_combine output files in the sub-folder
> >>>>>>>>>> out/pcp_combine under your MET build folder. If you are
> >> successful
> >>>>> in
> >>>>>>>>>> converting your data in this way, please send us
> >>>>>>>>>> your code so that we can make it available for other MET
users
> who
> >>>>>>> might
> >>>>>>>>>> want to use TRMM data. I'm sorry the answer to
> >>>>>>>>>> this question is not more simple.
> >>>>>>>>>>
> >>>>>>>>>> Regarding your ascii2nc question, can you please send us
the
> >>> ascii2nc
> >>>>>>>>>> input that is causing your error? Also, what is
> >>>>>>>>>> the command line statement that you are using to invoke
> ascii2nc?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>>
> >>>>>>>>>> Paul
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh}
wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
> >>>>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>>>>>>>>> Queue: met_help
> >>>>>>>>>>> Subject: Getting Problem in converting TRMM
rainfall to
> MET
> >>>>>>> format
> >>>>>>>>>>> Owner: Nobody
> >>>>>>>>>>> Requestors: harvirduphy at gmail.com
> >>>>>>>>>>> Status: new
> >>>>>>>>>>> Ticket <URL:
> >>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Dear Sir /Madam,
> >>>>>>>>>>> I am using TRMM rainfall Netcdf or bin data for
grid_stat MET
> >> tool?
> >>>>>>>>>> which
> >>>>>>>>>>> converter should did I use to convert TRMM Rainfall data
(.nc
> or
> >>>>> .bin)
> >>>>>>>>>> to
> >>>>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
> >>>>>>>>>> (UPPv1.0),
> >>>>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
> >>>>>>>>>>> I want to use some ASCII DATA may be output of some
models like
> >> GFS
> >>>>>>> etc
> >>>>>>>>>> ..
> >>>>>>>>>>> I have extracted data in the desired 10 columns. I'm
trying to
> >> use
> >>>>>>>>>> *ascii2nc
> >>>>>>>>>>> *this also giving some error like: open_met_ascii() ->
line
> >> number
> >>> 0
> >>>>>>>>>> doesn't
> >>>>>>>>>>> contain the expected number of colu
> >>>>>>>>>>> N_items = 7
> >>>>>>>>>>> N_chars = 512
> >>>>>>>>>>> N_ints = 100
> >>>>>>>>>>> LineNumber = 1
> >>>>>>>>>>>
> >>>>>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000
"
> >>>>>>>>>>> Item[ 1] = "48.68"
> >>>>>>>>>>> Item[ 2] = "9.23 "
> >>>>>>>>>>> Item[ 3] = "391.0 61 "
> >>>>>>>>>>> Item[ 4] = "974.8"
> >>>>>>>>>>>
> >>>>>>>>>>> Item[ 5] = "1021.5"
> >>>>>>>>>>> Item[ 6] = "-99.0"
> >>>>>>>>>>>
> >>>>>>>>>>> Offset[ 0] = 0
> >>>>>>>>>>> Offset[ 1] = 32
> >>>>>>>>>>> Offset[ 2] = 41
> >>>>>>>>>>> Offset[ 3] = 47
> >>>>>>>>>>> Offset[ 4] = 57
> >>>>>>>>>>>
> >>>>>>>>>>> Offset[ 5] = 64
> >>>>>>>>>>> Offset[ 6] = 71
> >>>>>>>>>>>
> >>>>>>>>>>> Please Help me....
> >>>>>>>>>>> Earliest reply will be appreciated.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks in advance for any help you can provide!
> >>>>>>>>>>>
> >>>>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for
Post
> >>>>>>> Processing
> >>>>>>>>>>> Tool "UPPv1.0 "*
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >
> >
>
>
>
------------------------------------------------
Subject: Re: Fwd: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and Point Stat tool
From: Paul Oldenburg
Time: Thu Sep 01 09:42:55 2011
Harvir,
There are a couple issues with the data that you sent me:
1. The TRMM data files you sent, TRMM_RF_15AUG2011_0.25x.dat and
TRMM_RF_16AUG2011_0.25x.dat, contain rainfall data for
3 hour accumulation intervals. You can see this in the header of
those files:
$ head TRMM_RF_16AUG2011_0.25x.dat
Selected parameter: 3-hourly TMPA-RT Accumulated Rainfall
...
So, I went to the website you sent and downloaded 24 hour (daily)
rainfall instead:
http://disc2.nascom.nasa.gov/Giovanni/tovas/realtime.3B42RT_daily.shtml
Then, I performed the steps shown below (with output), including
running point_stat with a modified version your
point_stat config file, PointStatConfig_rf. All of the input and
output files that I used and created are included in
the tarball PointStat_run.tar.gz on the FTP. Please download this and
try to follow my process.
$ cat TRMM_RF_20110816.dat | ./trmm_conv.pl > TRMM_RF_20110816.txt
$ ascii2nc TRMM_RF_20110816.txt TRMM_RF_20110816.nc -v 3
Creating NetCDF Observation file: TRMM_RF_20110816.nc
Reading ASCII Observation file: TRMM_RF_20110816.txt
Processing 74725 observations.
Finished processing 74725 observations.
$ plot_point_obs TRMM_RF_20110816.nc TRMM_RF_20110816.ps -dim -20 35
55 130
Opening netCDF file: TRMM_RF_20110816.nc
Processing 74725 observations at 74725 locations.
Observation GRIB codes: ALL
Observation message types: ALL
Creating postscript file: TRMM_RF_20110816.ps
Finished plotting 74725 locations.
$ $MET_BASE/bin/point_stat \
wrfprs_d01_2011-08-16_000_fcst01.grib1 \
TRMM_RF_20110816.nc \
PointStatConfig_rf \
-outdir . -v 3
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073636306766
Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
Climatology File: none
Configuration File: PointStatConfig_rf
Observation File: TRMM_RF_20110816.nc
--------------------------------------------------------------------------------
Reading records for APCP_24/A24.
GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum = 240000, Min = 0, Max = 235.7
For APCP_24/A24 found 1 forecast levels and 0 climatology levels.
--------------------------------------------------------------------------------
Searching 74725 observations from 74725 PrepBufr messages.
--------------------------------------------------------------------------------
Processing APCP_24/A24 versus APCP_24/A24, for observation type
ADPSFC, over region FULL, for interpolation method
UW_MEAN(1), using 66238 pairs.
Number of matched pairs = 66238
Observations processed = 74725
Rejected: GRIB code = 0
Rejected: valid time = 0
Rejected: bad obs value = 0
Rejected: off the grid = 0
Rejected: level mismatch = 0
Rejected: message type = 0
Rejected: masking region = 0
Rejected: bad fcst value = 8487
Computing Categorical Statistics.
Computing Continuous Statistics.
Computing Scalar Partial Sums.
--------------------------------------------------------------------------------
Output file: ./point_stat_240000L_20110816_000000V.stat
I want to call your attention to the fact that I could only verify one
of your model files,
wrfprs_d01_2011-08-16_000_fcst01.grib1. It appears that you do not
have consistent accumulation intervals:
$ wgrib wrfprs_d01_2011-08-15_000_fcst01.grib1 | grep APCP
252:28117402:d=11081500:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=0:TimeU=1:sfc:0-
0hr acc:NAve=0
$ wgrib wrfprs_d01_2011-08-16_000_fcst01.grib1 | grep APCP
252:28581058:d=11081500:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr acc:NAve=0
$ wgrib wrfprs_d01_2011-08-17_000_fcst01.grib1 | grep APCP
252:28514820:d=11081500:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=48:TimeU=1:sfc:0-
48hr acc:NAve=0
You can see near the end of the lines that the accumulation intervals
fluctuate: 0-0hr acc, 0-24hr acc, 0-48hr acc. I
am not an experienced WPP user, so I'm not sure how to change the
post-processing configuration so that the accumulation
interval is always 24 hours.
As for your question about grid_stat, it is possible for you to use
grid_stat, but it will not be easy. First of all,
the MET tool ascii2nc only outputs NetCDF data that can be used by
point_stat, not grid_stat. To use grid_stat with the
TRMM ascii data, you must put it into a NetCDF file that has a format
identical to the output of the MET tool
pcp_combine. MET does not contain any tools that perform this task,
so this is the difficult part. If I were going to
try to do this, I would probably use a data analysis package like R.
The point_stat tool is capable of calculating a lot of verification
statistics, so maybe it will suit your needs.
Please let me know if you have any other questions.
Paul
On 09/01/2011 06:26 AM, RAL HelpDesk {for Harvir Singh} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>
> Dear Paul,
>
> Yes I have the TRMM data file for the day before and I've
transferred to
> met_help. The TRMM rainfall I'm using, is available on web also i.e.
:
> http://disc2.nascom.nasa.gov/Giovanni/tovas/realtime.3B42RT.shtml
(ASCII)
> and for 15th Aug 2011 data @
> http://disc2.nascom.nasa.gov/daac-
bin/Giovanni/tovas/Giovanni_cgi.pl?west=45.0&north=45.0&east=121.0&south=-
16.0¶ms=0%7C3B42RT&plot_type=Area+Plot&byr=2011&bmo=08&bdy=14&bhr=00&eyr=2011&emo=08&edy=15&ehr=00&begin_date=2008%2F10%2F01%2F00&end_date=2011%2F08%2F31%2F12&cbar=cdyn&cmin=&cmax=&yaxis=ydyn&ymin=&ymax=&yint=&ascres=0.25x0.25&global_cfg=tovas.global.cfg.pl&instance_id=realtime&prod_id=3B42RT&action=ASCII+Output.
>
> Actually I've tried a lot to work on Point_stat but every time I
struck
> somewhere.. so I've some questions:
> 1. This METv3.0.1 software is only meant for WRF-NMM or WRF-ARW as
well.
>
> For this point of time I'm using WRF-ARW output to WRF Post
Processing
> ie WPPv3.2 and converting to GRIB1 format. And for observation data
I'm
> downloading ASCII data as above said link. When I'm converting ASCII
data to
> NetCDF I've also taken several assumptions for surface data like:
> Massage type: ADPSFC
> Station ID: TRMM
> Date : 20110816_000000
> Lat
> Lon
> Elevation: -9999 ( as data is not available)
> g.c. : 61 for Total precipitation
> Level : 24 hour ( for 24 accumulation rainfall)
> height : -9999 ( as data is not available)
>
>
> for both forecast and observation I've used 245x305 grid points at
the
> interval of 0.25 degree within the Indian region ie 16S to 45N and
45E to
> 121E.
> And fore converting WRF-ARF outfiles by WPPv3.2 have used copygb
with the
> given information as:
> grid='255 0 305 245 -16000 45000 128 45000 121000 250 250 64'
> ${POSTEXEC}/copygb.exe -g"${grid}" -x
> WRFPRS_${domain}_${YY}-${MM}-${DD}.${fhr}_fcst01 WRFPRS
> _fcst01/wrfprs_${domain}_${YY}-${MM}-${DD}_${fhr}_fcst01.grib1
>
>
> 2. Here I'm using Gridded data either from TRMM or from GTS or AWS
etc ...
> then why I can't use Grid_stat?
>
>
> I'm not getting it. I may be using some wrong information during the
data
> format conversion i.e. creating 10 columns for ASCII2NC. I've tried
it for
> several combinations for point stat as NetCDF as forecast file and
GRIB1 as
> observation file and vice-verso, also tried several combinations in
> PointStatConfug_rf as:
> // fcst_field[] = ["APCP_A24(*,*)"];
> fcst_field[] = ["APCP/A24"];
> //obs_field[] = ["APCP/A24"];
> //obs_field[] = [];
> obs_field[] = ["APCP_A24(0,5,*,*)"];
> and many more....
>
> Thanx
> HARVIR
>
> On 31 August 2011 01:25, RAL HelpDesk {for Paul Oldenburg} <
> met_help at ucar.edu> wrote:
>
>> Harvir,
>>
>> If I understand you correctly, you are saying that your observation
data is
>> accumulated rainfall from 20110816_000000 to
>> 20110817_000000. Your model data appears to contain accumulated
rainfall
>> from 20110815_000000 to 20110816_000000.
>> Obviously, these time windows do not overlap. Do you have the TRMM
data
>> file for the day before, so it matches the
>> model data?
>>
>> Paul
>>
>>
>> On 08/29/2011 06:29 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>
>>> Dear Paul,
>>> With the reference to your earlier mails, wrfprs_d01_2011-06-
01_000_
>>>>
>>>> fcst01.grib1 was for forecast file, obs_20110601_060000_rf.dat
>> observation
>>>> file from GTS data and
>>>> TRMM_16AUG2011.dat is from TRMM rainfall data.
>>>>
>>>
>>>
>>> Now I've transferred new set of data files viz.
>>> Model forecast file: wrfprs_d01_2011-08-16_000_fcst01.grib1
>>> Observation file : TRMM_RF_16AUG2011_0.25.txt
>>>
>>> and TRMM Rainfall data is from 00Z 16 AUG 2011 to 00Z 17 AUG 2011
and
>> model
>>> data is for 24 hr acumulated..
>>> thanx
>>>
>>>
>>>
>>> On 26 August 2011 20:05, RAL HelpDesk {for Paul Oldenburg} <
>>> met_help at ucar.edu> wrote:
>>>
>>>> Harvir,
>>>>
>>>> In the last email I sent you, I said that your rainfall
observation data
>> in
>>>> your model data file,
>>>> wrfprs_d01_2011-06-01_000_fcst01.grib1, had an accumulation
interval of
>> 24
>>>> hours. That means that the amount of
>>>> rainfall in the model data file is from a 24 hour period.
>>>>
>>>> Do you know what the accumulation interval of the rainfall data
you sent
>> me
>>>> in either obs_20110601_060000_rf.dat or
>>>> TRMM_16AUG2011.dat is? I see the following information at the
top of
>> the
>>>> file TRMM_16AUG2011.dat:
>>>>
>>>> Selected parameter: TRMM TMPA-RT Daily Accumulated Rainfall
>>>> Selected area: lat=[16.161S,45.311N], lon=[44.182E,121.960E]
>>>> Selected time period: (16Aug2011-16Aug2011)
>>>>
>>>> Does that mean that it is 24 hour accumulated rainfall from 2011-
08-15
>>>> 00:00:00 to 2011-08-16 00:00:00? If so, then it
>>>> has a 24 hour accumulation interval, but it does not match the
date in
>> your
>>>> model data file. Do you have a TRMM file
>>>> that is valid at the same time as your model data file?
>>>>
>>>> I need to know the answers to these questions before I can make
further
>>>> progress helping you verify rainfall. It is
>>>> very important that both the accumulation interval and the valid
time of
>>>> your model and observation data match.
>>>>
>>>> Thanks,
>>>>
>>>> Paul
>>>>
>>>>
>>>> On 08/26/2011 07:02 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>>>
>>>>> Dear Paul,
>>>>>
>>>>> Thank You for your prompt reply. Ofcourse yes I've to compute
Threat
>>>> score
>>>>> too.. For this point of time I just want to go with point stat.
and I'm
>>>>> trying it for WRF ARW core (output data) it. I've tried
point_stat, its
>>>>> running but giving headers only in stat txt files. I'm unable to
>> rectify
>>>> it.
>>>>> Please help me to come out of this problem. I've transfrred data
in
>>>>> incoming/irap/met_help/singh_data
>>>>> thanx in advance!
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: RAL HelpDesk {for Paul Oldenburg} <met_help at ucar.edu>
>>>>> Date: 23 August 2011 21:55
>>>>> Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC and
Point
>>>> Stat
>>>>> tool
>>>>> To: harvirduphy at gmail.com
>>>>>
>>>>>
>>>>> Harvir,
>>>>>
>>>>> There are a couple points to discuss before we continue. First,
I
>>>> recommend
>>>>> that you use point_stat at this time to
>>>>> perform verification on your model data. The reason is that
converting
>>>> your
>>>>> observation data to a format that grid_stat
>>>>> can read is more difficult because it must contain projection
>> information
>>>>> that matches your model data projection. Why
>>>>> do you want to run grid_stat? Are you interested in calculating
>>>>> neighborhood statistics or just basic verification
>>>>> statistics like RMSE and the Equitable Threat Score? If you
only want
>>>> basic
>>>>> statistics, point_stat will work just fine.
>>>>>
>>>>> Second, for rainfall data, you said that the assumptions that I
have
>> made
>>>>> about your observations are correct. If that
>>>>> is the case, and the accumulation interval of your observation
data is
>> 6
>>>>> hours, then there is a mis-match with the
>>>>> accumulation interval in your model data, which is 24 hours (see
0-24hr
>>>>> acc):
>>>>>
>>>>> $ wgrib wrfprs_d01_2011-06-01_000_fcst01.grib1 | grep APCP
>>>>>
>>>>
>>
252:323426:d=11053100:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
>>>>> acc:NAve=0
>>>>>
>>>>> If you are sure that your rainfall observations have an
accumulation
>>>>> interval of 6 hours, then you must combine four of
>>>>> your 6 hour observation files together to form rainfall
observations
>> with
>>>> an
>>>>> accumulation interval of 24 hours. A good
>>>>> way to do this is to use the MET tool pcp_combine after you run
>> ascii2nc
>>>> on
>>>>> your rainfall observation files.
>>>>>
>>>>> Finally, I tried to apply the model data that you gave me with
surface
>>>>> temperature to the observation data you gave me
>>>>> last week, 20110601_060000_tt.dat. I put the files involved in
this
>>>>> experiment are on the RAL FTP site in the archive
>>>>> ftp.rap.ucar.edu:
>> /incoming/irap/met_help/singh_data/PointStatVx.tar.gz.
>>>> If
>>>>> you download this and unpack it, you should
>>>>> be able to reproduce the output file that I included,
>>>>> point_stat_240000L_20110601_000000V.stat. This file contains
>>>>> basic verification statistics, like continuous and categorical
>>>> statistics.
>>>>> I used the threshold 273.15 K, which is 0 C.
>>>>>
>>>>> Note that for this experiment to be valid, I had to open the
time
>> window
>>>> to
>>>>> +/- 6 hours because your model data is valid
>>>>> at 20110601_000000 and your observations are valid at
20110601_060000.
>>>> This
>>>>> may or may not be an acceptable compromise
>>>>> to your for your work. Also, I assumed that the temperature
>> observations
>>>>> were all on the surface (message type ADPUPA).
>>>>> It also looked like your observations were in degrees Celsius,
so I
>>>>> converted them to Kelvin. Please let me know if
>>>>> you have any questions or trouble following my example. Please
look
>>>>> carefully at the files included in the archive,
>>>>> especially PointStatConfig_tmp and point_stat.sh.
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>> $ tar xzvf PointStatVx.tar.gz
>>>>> $ cat obs_20110601_060000_tt.dat | ./obs_conv.pl >
>>>>> obs_20110601_060000_tt.txt
>>>>> $ ascii2nc obs_20110601_060000_tt.txt obs_20110601_060000_tt.nc
-v 3
>>>>> Creating NetCDF Observation file: obs_20110601_060000_tt.nc
>>>>> Reading ASCII Observation file: obs_20110601_060000_tt.txt
>>>>> Processing 1099 observations.
>>>>> Finished processing 1099 observations.
>>>>> $ ./point_stat.sh
>>>>> GSL_RNG_TYPE=mt19937
>>>>> GSL_RNG_SEED=231822158
>>>>> Forecast File: wrfprs_d01_2011-06-01_000_fcst01.grib1
>>>>> Climatology File: none
>>>>> Configuration File: PointStatConfig_tmp
>>>>> Observation File: obs_20110601_060000_tt.nc
>>>>>
>>>>>
>>>>
>>
--------------------------------------------------------------------------------
>>>>>
>>>>> Reading records for TMP/Z2.
>>>>> GRIB Record 244: Init = 20110531_000000, Valid =
20110601_000000, Accum
>> =
>>>>> 000000, Min = 292.579, Max = 303.165
>>>>> For TMP/Z2 found 1 forecast levels and 0 climatology levels.
>>>>>
>>>>>
>>>>
>>
--------------------------------------------------------------------------------
>>>>>
>>>>> Searching 1099 observations from 1099 PrepBufr messages.
>>>>>
>>>>>
>>>>
>>
--------------------------------------------------------------------------------
>>>>>
>>>>> Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC,
over
>> region
>>>>> FULL, for interpolation method UW_MEAN(1),
>>>>> using 58 pairs.
>>>>> Number of matched pairs = 58
>>>>> Observations processed = 1099
>>>>> Rejected: GRIB code = 0
>>>>> Rejected: valid time = 0
>>>>> Rejected: bad obs value = 0
>>>>> Rejected: off the grid = 1041
>>>>> Rejected: level mismatch = 0
>>>>> Rejected: message type = 0
>>>>> Rejected: masking region = 0
>>>>> Rejected: bad fcst value = 0
>>>>> Computing Categorical Statistics.
>>>>> Computing Continuous Statistics.
>>>>> Computing Scalar Partial Sums.
>>>>>
>>>>>
>>>>
>>
--------------------------------------------------------------------------------
>>>>>
>>>>> Output file: ./point_stat_240000L_20110601_000000V.stat
>>>>>
>>>>>
>>>>> On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>>>>
>>>>>> Dear Paul,
>>>>>> I have gone through your assumptions, might seems correct. But
I am
>>>> having
>>>>>> some problems when I running Point_stat for rainfall, I am
getting
>> some
>>>>>> error as:
>>>>>>
>>>>>> $ sh point_stat.sh
>>>>>> ERROR: GCInfo::set_gcinfo() -> unsupported input file type
"APCP/A24"
>>>>>> I have changed configure, "APCP/P24" "APCP/P1000", "APCP/P925"
etc.
>>>>>> and I ma getting similar type of error. Suggest me how can I
rectify
>> it.
>>>>>>
>>>>>>
>>>>>> I also tried for GRID_STAT and I got the error as follows
>>>>>> sh Grid_stat.sh
>>>>>> GSL_RNG_TYPE=mt19937
>>>>>> GSL_RNG_SEED=18446744073465844558
>>>>>> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
>>>>>> Observation File: TRMM_16AUG2011.nc
>>>>>> Configuration File:
>>>>>> /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
>>>>>> GRIB Record 252: Init = 20110815_000000, Valid =
20110816_000000,
>> Accum
>>>> =
>>>>>> 240000, Min = 0, Max = 148.8
>>>>>> ncattinq: ncid 4; varid -1; attname "Projection": Attribute not
found
>>>>>>
>>>>>>
>>>>>> One more thing I want to ask you about plot_point_obs. Actually
>>>>>> plot_point_obs executable does not exits in my METv3.0.1/bin.
Suppose
>> I
>>>>> need
>>>>>> it, then what I have to do?
>>>>>>
>>>>>> Thanks
>>>>>> HARVIR
>>>>>>
>>>>>> *PS: All the data files and script files have been transferred
in
>>>>>> incoming/irap/met_help/singh_data
>>>>>> *
>>>>>> On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
>>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>>> Harvir,
>>>>>>>
>>>>>>> Regarding your second request related to TRMM data, I wrote a
small
>>>>> script
>>>>>>> (attached) to convert TRMM_16AUG2011.asc into
>>>>>>> a format that is readable by ascii2nc. Then, I plotted the
>> observation
>>>>>>> points using the MET utility plot_point_obs and
>>>>>>> created the attached image showing the range of points. It is
>>>> important
>>>>> to
>>>>>>> note that I had to make several assumptions
>>>>>>> about your observations to do this conversion. The
assumptions are:
>>>>>>>
>>>>>>> 1. Your observations are surface observations (column 1 -
message
>> type
>>>>>>> ADPSFC). This does make sense if your data is
>>>>>>> accumulated precipitation, which I believe it is.
>>>>>>> 2. The valid time of your observations is 2011-08-11 00:00:00
UTC.
>>>>>>> Obviously, the date is shown in your filename, but
>>>>>>> I could not find the valid time.
>>>>>>> 3. The elevation in msl of all observing points is 0m, which
is
>>>>> definitely
>>>>>>> a bad assumption, but for "surface"
>>>>>>> observations will not matter.
>>>>>>> 4. The accumulation interval of your observations is 6 hours
(column
>>>> 8).
>>>>>>> If this is not correct, please modify the
>>>>>>> script I attached accordingly.
>>>>>>> 5. The height in msl of the observation points is 0m (see
assumption
>>>> 3).
>>>>>>>
>>>>>>> Once you have model data that contains APCP with the same
>> accumulation
>>>>>>> interval as that of your observations, you should
>>>>>>> be able to use this acii2nc-generate observations in
point_stat.
>>>>>>>
>>>>>>> Here are the steps to generate the ascii2nc NetCDF obs data
file and
>>>> plot
>>>>>>> using the script I am sending. These commands
>>>>>>> assume that the MET bin/ sub-folder is on your path.
>>>>>>>
>>>>>>> $ chmod +x trmm_conv.pl
>>>>>>> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl > TRMM_16AUG2011.txt
>>>>>>> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
>>>>>>> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim -15
45 45
>> 120
>>>>>>>
>>>>>>> Please let me know if you have any questions.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>>
>>>>>>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>>>
>>>>>>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>>> Queue: met_help
>>>>>>>> Subject: Need help for ASCII2NC and Point Stat tool
>>>>>>>> Owner: Nobody
>>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>>> Status: new
>>>>>>>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dear Paul,
>>>>>>>>
>>>>>>>> I would like to thank you Paul for your necessary
suggestions. I've
>>>>>>>> implemented them and now its work but I want use this file (
output
>>>> from
>>>>>>>> ascii2nc) for Point Stat tool. I have tried to run
point_stat, its
>>>>>>> running
>>>>>>>> and creating some stat files too but stats file having only
headers
>> no
>>>>>>>> statistics. I'm not getting why its so?
>>>>>>>> I've transferred data files as well as script file in
>>>>>>>> incoming/irap/met_help/singh_data. Here I want to run it for
TMP and
>>>>>>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
>>>>>>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
>>>>>>>> 20110601_060000_rf.dat
>>>>>>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast
file is
>> WRF
>>>>>>>> outfile in GRIB1 format (output from WPPv3.2) named as
>>>>>>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I want to ask you something on some other data. Suppose I
have some
>>>>> ASCII
>>>>>>>> data having only 3 columns viz. Latitude, Longitude and
Rainfall
>>>> (say).
>>>>>>> Can
>>>>>>>> we convert it through ascii2nc for MET supported format? If
yes,
>> then
>>>>>>> what
>>>>>>>> we can take rest of the columns. I also transferred this kind
of
>> data
>>>>>>> named
>>>>>>>> as TRMM_16AUG2011.asc,
>>>>>>>> TRMM_16AUG2011.dat
>>>>>>>> I want to use point stat for this data also.
>>>>>>>>
>>>>>>>> Thank you in advance.
>>>>>>>>
>>>>>>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
>>>>>>> met_help at ucar.edu
>>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Harvir,
>>>>>>>>>
>>>>>>>>> I have identified several problems with the ascii2nc input
file
>>>>>>>>> suflnd_20110601_06z_rrf.dat:
>>>>>>>>>
>>>>>>>>> 1. ascii2nc input columns must be space-delimited. Several
of the
>>>>>>> columns
>>>>>>>>> in the input file are tab delimited. Do a
>>>>>>>>> find/replace on the file to change all tabs to spaces.
>>>>>>>>>
>>>>>>>>> 2. After you perform step 1, you will see the following
error when
>>>> you
>>>>>>> try
>>>>>>>>> to run ascii2nc:
>>>>>>>>>
>>>>>>>>> $ ascii2nc.sh
>>>>>>>>> Creating NetCDF Observation file: suflnd_20110601_06z_RF.nc
>>>>>>>>> Reading ASCII Observation file: suflnd_20110601_06z_rrf.dat
>>>>>>>>> Processing 11414 observations.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> timestring_to_sec(const char *) -> can't parse time string
"974.8"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The reason for this error is that your file appears to
contain
>>>>>>> accumulated
>>>>>>>>> precipitation observations (GRIB code 61 in
>>>>>>>>> column 7). In this case, ascii2nc is expecting an
accumulation
>>>>> interval
>>>>>>> in
>>>>>>>>> column 8 in the format HH, for example 06 or
>>>>>>>>> 24. That is the reason for the error message above.
>>>>>>>>>
>>>>>>>>> 3. Column 9 should contain the height above ground of the
>>>> observation
>>>>>>>>> station in meters. For accumulated
>>>>>>>>> precipitation, this information is ignored by ascii2nc
because this
>>>> is
>>>>> a
>>>>>>>>> "surface" observation. We typically use a fill
>>>>>>>>> value of -9999, indicating invalid data.
>>>>>>>>>
>>>>>>>>> 4. Column 10 should contain the observed values. I see
valid
>>>>>>> observation
>>>>>>>>> values on some rows, but many are -99.0 which
>>>>>>>>> I interpret to mean invalid data. In this case, it does not
make
>>>> sense
>>>>>>> to
>>>>>>>>> include invalid observation data, and those
>>>>>>>>> lines should be deleted.
>>>>>>>>>
>>>>>>>>> I took the liberty of making a couple assumptions about your
data
>> and
>>>>>>>>> trying to adjust the input file that you sent me
>>>>>>>>> to something that ascii2nc can read. The assumptions I made
are:
>>>>>>>>>
>>>>>>>>> 1. All of your observations are "surface" observations, so
the
>>>> message
>>>>>>>>> type (column 1) was set to ADPSFC
>>>>>>>>> 2. The accumulation interval is 6 hours (column 8 is set to
06)
>>>>>>>>> 3. The observation values are in mm, and left as-is
>>>>>>>>>
>>>>>>>>> If these assumptions, mainly #2 and #3, are incorrect,
please make
>>>> the
>>>>>>>>> appropriate adjustments. I attached the "fixed"
>>>>>>>>> file to this email for you to review and change as you wish.
>> Please
>>>>> let
>>>>>>> me
>>>>>>>>> know if you have any questions.
>>>>>>>>>
>>>>>>>>> Paul
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh}
wrote:
>>>>>>>>>>
>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
>>>>>>>>>>
>>>>>>>>>>> Dear Paul,
>>>>>>>>>>>
>>>>>>>>>>> Accept My apology. Actually yesterday I forgot to
attached th
>>>>>>> files.
>>>>>>>>>> However, as per your advice through ftp I have putted the
same
>>>>> attached
>>>>>>>>>> files there also. and its path is
>>>>>>>>>> *incoming/irap/met_help/singh_data*
>>>>>>>>>>
>>>>>>>>>>> I'm sending you ASCII file and ascii2nc script. Please
have close
>>>>> look
>>>>>>>>> on
>>>>>>>>>>> it if am wrong somewhere please let me know.
>>>>>>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
>> successful
>>>> or
>>>>>>>>> not
>>>>>>>>>>> in pcp_combine ... However, I've run all the test scripts
>>>>>>> successfully.
>>>>>>>>>>> Thanx in advance...
>>>>>>>>>>>
>>>>>>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul Oldenburg}
<
>>>>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Harvir,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm afraid that MET cannot read TRMM data directly. The
best
>> way
>>>> to
>>>>>>>>> read
>>>>>>>>>>>> TRMM data into MET would be to convert the
>>>>>>>>>>>> data from it's current format into a NetCDF file that has
the
>> same
>>>>>>>>> format
>>>>>>>>>>>> as the output of the MET pcp_combine tool. If
>>>>>>>>>>>> you ran the MET test scripts after you built MET, you
should be
>>>> able
>>>>>>> to
>>>>>>>>>>>> find pcp_combine output files in the sub-folder
>>>>>>>>>>>> out/pcp_combine under your MET build folder. If you are
>>>> successful
>>>>>>> in
>>>>>>>>>>>> converting your data in this way, please send us
>>>>>>>>>>>> your code so that we can make it available for other MET
users
>> who
>>>>>>>>> might
>>>>>>>>>>>> want to use TRMM data. I'm sorry the answer to
>>>>>>>>>>>> this question is not more simple.
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding your ascii2nc question, can you please send us
the
>>>>> ascii2nc
>>>>>>>>>>>> input that is causing your error? Also, what is
>>>>>>>>>>>> the command line statement that you are using to invoke
>> ascii2nc?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Paul
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh}
wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted upon.
>>>>>>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>>>>>>>> Queue: met_help
>>>>>>>>>>>>> Subject: Getting Problem in converting TRMM
rainfall to
>> MET
>>>>>>>>> format
>>>>>>>>>>>>> Owner: Nobody
>>>>>>>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>>>>>>>> Status: new
>>>>>>>>>>>>> Ticket <URL:
>>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dear Sir /Madam,
>>>>>>>>>>>>> I am using TRMM rainfall Netcdf or bin data for
grid_stat MET
>>>> tool?
>>>>>>>>>>>> which
>>>>>>>>>>>>> converter should did I use to convert TRMM Rainfall data
(.nc
>> or
>>>>>>> .bin)
>>>>>>>>>>>> to
>>>>>>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
Processing
>>>>>>>>>>>> (UPPv1.0),
>>>>>>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>>>>>>>>> I want to use some ASCII DATA may be output of some
models like
>>>> GFS
>>>>>>>>> etc
>>>>>>>>>>>> ..
>>>>>>>>>>>>> I have extracted data in the desired 10 columns. I'm
trying to
>>>> use
>>>>>>>>>>>> *ascii2nc
>>>>>>>>>>>>> *this also giving some error like: open_met_ascii() ->
line
>>>> number
>>>>> 0
>>>>>>>>>>>> doesn't
>>>>>>>>>>>>> contain the expected number of colu
>>>>>>>>>>>>> N_items = 7
>>>>>>>>>>>>> N_chars = 512
>>>>>>>>>>>>> N_ints = 100
>>>>>>>>>>>>> LineNumber = 1
>>>>>>>>>>>>>
>>>>>>>>>>>>> Item[ 0] = "SIDL210 10738 20110601_030000
"
>>>>>>>>>>>>> Item[ 1] = "48.68"
>>>>>>>>>>>>> Item[ 2] = "9.23 "
>>>>>>>>>>>>> Item[ 3] = "391.0 61 "
>>>>>>>>>>>>> Item[ 4] = "974.8"
>>>>>>>>>>>>>
>>>>>>>>>>>>> Item[ 5] = "1021.5"
>>>>>>>>>>>>> Item[ 6] = "-99.0"
>>>>>>>>>>>>>
>>>>>>>>>>>>> Offset[ 0] = 0
>>>>>>>>>>>>> Offset[ 1] = 32
>>>>>>>>>>>>> Offset[ 2] = 41
>>>>>>>>>>>>> Offset[ 3] = 47
>>>>>>>>>>>>> Offset[ 4] = 57
>>>>>>>>>>>>>
>>>>>>>>>>>>> Offset[ 5] = 64
>>>>>>>>>>>>> Offset[ 6] = 71
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please Help me....
>>>>>>>>>>>>> Earliest reply will be appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks in advance for any help you can provide!
>>>>>>>>>>>>>
>>>>>>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1, for
Post
>>>>>>>>> Processing
>>>>>>>>>>>>> Tool "UPPv1.0 "*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: Harvir Singh
Time: Fri Sep 02 06:49:22 2011
Dear Paul,
I would like to thank you for your invaluable guidance. The
Point_stats
seems working perfectly but now I have been told to work on
Grid_stats.
Could you suggest me " How can I implement for Grid_stats for same
TRMM
data?"
or How can I run grid_stat for TRMM data?
Thanx
HARVIR
On 1 September 2011 21:12, RAL HelpDesk {for Paul Oldenburg} <
met_help at ucar.edu> wrote:
> Harvir,
>
> There are a couple issues with the data that you sent me:
>
> 1. The TRMM data files you sent, TRMM_RF_15AUG2011_0.25x.dat and
> TRMM_RF_16AUG2011_0.25x.dat, contain rainfall data for
> 3 hour accumulation intervals. You can see this in the header of
those
> files:
>
> $ head TRMM_RF_16AUG2011_0.25x.dat
> Selected parameter: 3-hourly TMPA-RT Accumulated Rainfall
> ...
>
> So, I went to the website you sent and downloaded 24 hour (daily)
rainfall
> instead:
>
>
http://disc2.nascom.nasa.gov/Giovanni/tovas/realtime.3B42RT_daily.shtml
>
> Then, I performed the steps shown below (with output), including
running
> point_stat with a modified version your
> point_stat config file, PointStatConfig_rf. All of the input and
output
> files that I used and created are included in
> the tarball PointStat_run.tar.gz on the FTP. Please download this
and try
> to follow my process.
>
> $ cat TRMM_RF_20110816.dat | ./trmm_conv.pl > TRMM_RF_20110816.txt
> $ ascii2nc TRMM_RF_20110816.txt TRMM_RF_20110816.nc -v 3
> Creating NetCDF Observation file: TRMM_RF_20110816.nc
> Reading ASCII Observation file: TRMM_RF_20110816.txt
> Processing 74725 observations.
> Finished processing 74725 observations.
> $ plot_point_obs TRMM_RF_20110816.nc TRMM_RF_20110816.ps -dim -20 35
55 130
> Opening netCDF file: TRMM_RF_20110816.nc
> Processing 74725 observations at 74725 locations.
> Observation GRIB codes: ALL
> Observation message types: ALL
> Creating postscript file: TRMM_RF_20110816.ps
> Finished plotting 74725 locations.
> $ $MET_BASE/bin/point_stat \
> wrfprs_d01_2011-08-16_000_fcst01.grib1 \
> TRMM_RF_20110816.nc \
> PointStatConfig_rf \
> -outdir . -v 3
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073636306766
> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
> Climatology File: none
> Configuration File: PointStatConfig_rf
> Observation File: TRMM_RF_20110816.nc
>
>
>
--------------------------------------------------------------------------------
>
> Reading records for APCP_24/A24.
> GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum =
> 240000, Min = 0, Max = 235.7
> For APCP_24/A24 found 1 forecast levels and 0 climatology levels.
>
>
>
--------------------------------------------------------------------------------
>
> Searching 74725 observations from 74725 PrepBufr messages.
>
>
>
--------------------------------------------------------------------------------
>
> Processing APCP_24/A24 versus APCP_24/A24, for observation type
ADPSFC,
> over region FULL, for interpolation method
> UW_MEAN(1), using 66238 pairs.
> Number of matched pairs = 66238
> Observations processed = 74725
> Rejected: GRIB code = 0
> Rejected: valid time = 0
> Rejected: bad obs value = 0
> Rejected: off the grid = 0
> Rejected: level mismatch = 0
> Rejected: message type = 0
> Rejected: masking region = 0
> Rejected: bad fcst value = 8487
> Computing Categorical Statistics.
> Computing Continuous Statistics.
> Computing Scalar Partial Sums.
>
>
>
--------------------------------------------------------------------------------
>
> Output file: ./point_stat_240000L_20110816_000000V.stat
>
>
> I want to call your attention to the fact that I could only verify
one of
> your model files,
> wrfprs_d01_2011-08-16_000_fcst01.grib1. It appears that you do not
have
> consistent accumulation intervals:
>
> $ wgrib wrfprs_d01_2011-08-15_000_fcst01.grib1 | grep APCP
>
252:28117402:d=11081500:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=0:TimeU=1:sfc:0-
0hr
> acc:NAve=0
> $ wgrib wrfprs_d01_2011-08-16_000_fcst01.grib1 | grep APCP
>
252:28581058:d=11081500:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
> acc:NAve=0
> $ wgrib wrfprs_d01_2011-08-17_000_fcst01.grib1 | grep APCP
>
252:28514820:d=11081500:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=48:TimeU=1:sfc:0-
48hr
> acc:NAve=0
>
> You can see near the end of the lines that the accumulation
intervals
> fluctuate: 0-0hr acc, 0-24hr acc, 0-48hr acc. I
> am not an experienced WPP user, so I'm not sure how to change the
> post-processing configuration so that the accumulation
> interval is always 24 hours.
>
> As for your question about grid_stat, it is possible for you to use
> grid_stat, but it will not be easy. First of all,
> the MET tool ascii2nc only outputs NetCDF data that can be used by
> point_stat, not grid_stat. To use grid_stat with the
> TRMM ascii data, you must put it into a NetCDF file that has a
format
> identical to the output of the MET tool
> pcp_combine. MET does not contain any tools that perform this task,
so
> this is the difficult part. If I were going to
> try to do this, I would probably use a data analysis package like R.
>
> The point_stat tool is capable of calculating a lot of verification
> statistics, so maybe it will suit your needs.
>
> Please let me know if you have any other questions.
>
> Paul
>
>
> On 09/01/2011 06:26 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >
> > Dear Paul,
> >
> > Yes I have the TRMM data file for the day before and I've
transferred to
> > met_help. The TRMM rainfall I'm using, is available on web also
i.e. :
> > http://disc2.nascom.nasa.gov/Giovanni/tovas/realtime.3B42RT.shtml
(ASCII)
> > and for 15th Aug 2011 data @
> >
> http://disc2.nascom.nasa.gov/daac-
bin/Giovanni/tovas/Giovanni_cgi.pl?west=45.0&north=45.0&east=121.0&south=-
16.0¶ms=0%7C3B42RT&plot_type=Area+Plot&byr=2011&bmo=08&bdy=14&bhr=00&eyr=2011&emo=08&edy=15&ehr=00&begin_date=2008%2F10%2F01%2F00&end_date=2011%2F08%2F31%2F12&cbar=cdyn&cmin=&cmax=&yaxis=ydyn&ymin=&ymax=&yint=&ascres=0.25x0.25&global_cfg=tovas.global.cfg.pl&instance_id=realtime&prod_id=3B42RT&action=ASCII+Output
> .
> >
> > Actually I've tried a lot to work on Point_stat but every time I
struck
> > somewhere.. so I've some questions:
> > 1. This METv3.0.1 software is only meant for WRF-NMM or WRF-ARW as
well.
> >
> > For this point of time I'm using WRF-ARW output to WRF Post
> Processing
> > ie WPPv3.2 and converting to GRIB1 format. And for observation
data I'm
> > downloading ASCII data as above said link. When I'm converting
ASCII data
> to
> > NetCDF I've also taken several assumptions for surface data like:
> > Massage type: ADPSFC
> > Station ID: TRMM
> > Date : 20110816_000000
> > Lat
> > Lon
> > Elevation: -9999 ( as data is not available)
> > g.c. : 61 for Total precipitation
> > Level : 24 hour ( for 24 accumulation rainfall)
> > height : -9999 ( as data is not available)
> >
> >
> > for both forecast and observation I've used 245x305 grid points at
the
> > interval of 0.25 degree within the Indian region ie 16S to 45N
and 45E
> to
> > 121E.
> > And fore converting WRF-ARF outfiles by WPPv3.2 have used copygb
with the
> > given information as:
> > grid='255 0 305 245 -16000 45000 128 45000 121000 250 250 64'
> > ${POSTEXEC}/copygb.exe -g"${grid}" -x
> > WRFPRS_${domain}_${YY}-${MM}-${DD}.${fhr}_fcst01 WRFPRS
> > _fcst01/wrfprs_${domain}_${YY}-${MM}-${DD}_${fhr}_fcst01.grib1
> >
> >
> > 2. Here I'm using Gridded data either from TRMM or from GTS or AWS
etc
> ...
> > then why I can't use Grid_stat?
> >
> >
> > I'm not getting it. I may be using some wrong information during
the data
> > format conversion i.e. creating 10 columns for ASCII2NC. I've
tried it
> for
> > several combinations for point stat as NetCDF as forecast file and
GRIB1
> as
> > observation file and vice-verso, also tried several combinations
in
> > PointStatConfug_rf as:
> > // fcst_field[] = ["APCP_A24(*,*)"];
> > fcst_field[] = ["APCP/A24"];
> > //obs_field[] = ["APCP/A24"];
> > //obs_field[] = [];
> > obs_field[] = ["APCP_A24(0,5,*,*)"];
> > and many more....
> >
> > Thanx
> > HARVIR
> >
> > On 31 August 2011 01:25, RAL HelpDesk {for Paul Oldenburg} <
> > met_help at ucar.edu> wrote:
> >
> >> Harvir,
> >>
> >> If I understand you correctly, you are saying that your
observation data
> is
> >> accumulated rainfall from 20110816_000000 to
> >> 20110817_000000. Your model data appears to contain accumulated
> rainfall
> >> from 20110815_000000 to 20110816_000000.
> >> Obviously, these time windows do not overlap. Do you have the
TRMM data
> >> file for the day before, so it matches the
> >> model data?
> >>
> >> Paul
> >>
> >>
> >> On 08/29/2011 06:29 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
> >>>
> >>> Dear Paul,
> >>> With the reference to your earlier mails, wrfprs_d01_2011-06-
01_000_
> >>>>
> >>>> fcst01.grib1 was for forecast file, obs_20110601_060000_rf.dat
> >> observation
> >>>> file from GTS data and
> >>>> TRMM_16AUG2011.dat is from TRMM rainfall data.
> >>>>
> >>>
> >>>
> >>> Now I've transferred new set of data files viz.
> >>> Model forecast file: wrfprs_d01_2011-08-16_000_fcst01.grib1
> >>> Observation file : TRMM_RF_16AUG2011_0.25.txt
> >>>
> >>> and TRMM Rainfall data is from 00Z 16 AUG 2011 to 00Z 17 AUG
2011 and
> >> model
> >>> data is for 24 hr acumulated..
> >>> thanx
> >>>
> >>>
> >>>
> >>> On 26 August 2011 20:05, RAL HelpDesk {for Paul Oldenburg} <
> >>> met_help at ucar.edu> wrote:
> >>>
> >>>> Harvir,
> >>>>
> >>>> In the last email I sent you, I said that your rainfall
observation
> data
> >> in
> >>>> your model data file,
> >>>> wrfprs_d01_2011-06-01_000_fcst01.grib1, had an accumulation
interval
> of
> >> 24
> >>>> hours. That means that the amount of
> >>>> rainfall in the model data file is from a 24 hour period.
> >>>>
> >>>> Do you know what the accumulation interval of the rainfall data
you
> sent
> >> me
> >>>> in either obs_20110601_060000_rf.dat or
> >>>> TRMM_16AUG2011.dat is? I see the following information at the
top of
> >> the
> >>>> file TRMM_16AUG2011.dat:
> >>>>
> >>>> Selected parameter: TRMM TMPA-RT Daily Accumulated Rainfall
> >>>> Selected area: lat=[16.161S,45.311N], lon=[44.182E,121.960E]
> >>>> Selected time period: (16Aug2011-16Aug2011)
> >>>>
> >>>> Does that mean that it is 24 hour accumulated rainfall from
2011-08-15
> >>>> 00:00:00 to 2011-08-16 00:00:00? If so, then it
> >>>> has a 24 hour accumulation interval, but it does not match the
date in
> >> your
> >>>> model data file. Do you have a TRMM file
> >>>> that is valid at the same time as your model data file?
> >>>>
> >>>> I need to know the answers to these questions before I can make
> further
> >>>> progress helping you verify rainfall. It is
> >>>> very important that both the accumulation interval and the
valid time
> of
> >>>> your model and observation data match.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Paul
> >>>>
> >>>>
> >>>> On 08/26/2011 07:02 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>>
> >>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061
>
> >>>>>
> >>>>> Dear Paul,
> >>>>>
> >>>>> Thank You for your prompt reply. Ofcourse yes I've to compute
Threat
> >>>> score
> >>>>> too.. For this point of time I just want to go with point
stat. and
> I'm
> >>>>> trying it for WRF ARW core (output data) it. I've tried
point_stat,
> its
> >>>>> running but giving headers only in stat txt files. I'm unable
to
> >> rectify
> >>>> it.
> >>>>> Please help me to come out of this problem. I've transfrred
data in
> >>>>> incoming/irap/met_help/singh_data
> >>>>> thanx in advance!
> >>>>>
> >>>>> ---------- Forwarded message ----------
> >>>>> From: RAL HelpDesk {for Paul Oldenburg} <met_help at ucar.edu>
> >>>>> Date: 23 August 2011 21:55
> >>>>> Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC
and
> Point
> >>>> Stat
> >>>>> tool
> >>>>> To: harvirduphy at gmail.com
> >>>>>
> >>>>>
> >>>>> Harvir,
> >>>>>
> >>>>> There are a couple points to discuss before we continue.
First, I
> >>>> recommend
> >>>>> that you use point_stat at this time to
> >>>>> perform verification on your model data. The reason is that
> converting
> >>>> your
> >>>>> observation data to a format that grid_stat
> >>>>> can read is more difficult because it must contain projection
> >> information
> >>>>> that matches your model data projection. Why
> >>>>> do you want to run grid_stat? Are you interested in
calculating
> >>>>> neighborhood statistics or just basic verification
> >>>>> statistics like RMSE and the Equitable Threat Score? If you
only
> want
> >>>> basic
> >>>>> statistics, point_stat will work just fine.
> >>>>>
> >>>>> Second, for rainfall data, you said that the assumptions that
I have
> >> made
> >>>>> about your observations are correct. If that
> >>>>> is the case, and the accumulation interval of your observation
data
> is
> >> 6
> >>>>> hours, then there is a mis-match with the
> >>>>> accumulation interval in your model data, which is 24 hours
(see
> 0-24hr
> >>>>> acc):
> >>>>>
> >>>>> $ wgrib wrfprs_d01_2011-06-01_000_fcst01.grib1 | grep APCP
> >>>>>
> >>>>
> >>
>
252:323426:d=11053100:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
> >>>>> acc:NAve=0
> >>>>>
> >>>>> If you are sure that your rainfall observations have an
accumulation
> >>>>> interval of 6 hours, then you must combine four of
> >>>>> your 6 hour observation files together to form rainfall
observations
> >> with
> >>>> an
> >>>>> accumulation interval of 24 hours. A good
> >>>>> way to do this is to use the MET tool pcp_combine after you
run
> >> ascii2nc
> >>>> on
> >>>>> your rainfall observation files.
> >>>>>
> >>>>> Finally, I tried to apply the model data that you gave me with
> surface
> >>>>> temperature to the observation data you gave me
> >>>>> last week, 20110601_060000_tt.dat. I put the files involved
in this
> >>>>> experiment are on the RAL FTP site in the archive
> >>>>> ftp.rap.ucar.edu:
> >> /incoming/irap/met_help/singh_data/PointStatVx.tar.gz.
> >>>> If
> >>>>> you download this and unpack it, you should
> >>>>> be able to reproduce the output file that I included,
> >>>>> point_stat_240000L_20110601_000000V.stat. This file contains
> >>>>> basic verification statistics, like continuous and categorical
> >>>> statistics.
> >>>>> I used the threshold 273.15 K, which is 0 C.
> >>>>>
> >>>>> Note that for this experiment to be valid, I had to open the
time
> >> window
> >>>> to
> >>>>> +/- 6 hours because your model data is valid
> >>>>> at 20110601_000000 and your observations are valid at
> 20110601_060000.
> >>>> This
> >>>>> may or may not be an acceptable compromise
> >>>>> to your for your work. Also, I assumed that the temperature
> >> observations
> >>>>> were all on the surface (message type ADPUPA).
> >>>>> It also looked like your observations were in degrees
Celsius, so I
> >>>>> converted them to Kelvin. Please let me know if
> >>>>> you have any questions or trouble following my example.
Please look
> >>>>> carefully at the files included in the archive,
> >>>>> especially PointStatConfig_tmp and point_stat.sh.
> >>>>>
> >>>>> Paul
> >>>>>
> >>>>>
> >>>>> $ tar xzvf PointStatVx.tar.gz
> >>>>> $ cat obs_20110601_060000_tt.dat | ./obs_conv.pl >
> >>>>> obs_20110601_060000_tt.txt
> >>>>> $ ascii2nc obs_20110601_060000_tt.txt
obs_20110601_060000_tt.nc -v 3
> >>>>> Creating NetCDF Observation file: obs_20110601_060000_tt.nc
> >>>>> Reading ASCII Observation file: obs_20110601_060000_tt.txt
> >>>>> Processing 1099 observations.
> >>>>> Finished processing 1099 observations.
> >>>>> $ ./point_stat.sh
> >>>>> GSL_RNG_TYPE=mt19937
> >>>>> GSL_RNG_SEED=231822158
> >>>>> Forecast File: wrfprs_d01_2011-06-01_000_fcst01.grib1
> >>>>> Climatology File: none
> >>>>> Configuration File: PointStatConfig_tmp
> >>>>> Observation File: obs_20110601_060000_tt.nc
> >>>>>
> >>>>>
> >>>>
> >>
>
--------------------------------------------------------------------------------
> >>>>>
> >>>>> Reading records for TMP/Z2.
> >>>>> GRIB Record 244: Init = 20110531_000000, Valid =
20110601_000000,
> Accum
> >> =
> >>>>> 000000, Min = 292.579, Max = 303.165
> >>>>> For TMP/Z2 found 1 forecast levels and 0 climatology levels.
> >>>>>
> >>>>>
> >>>>
> >>
>
--------------------------------------------------------------------------------
> >>>>>
> >>>>> Searching 1099 observations from 1099 PrepBufr messages.
> >>>>>
> >>>>>
> >>>>
> >>
>
--------------------------------------------------------------------------------
> >>>>>
> >>>>> Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC,
over
> >> region
> >>>>> FULL, for interpolation method UW_MEAN(1),
> >>>>> using 58 pairs.
> >>>>> Number of matched pairs = 58
> >>>>> Observations processed = 1099
> >>>>> Rejected: GRIB code = 0
> >>>>> Rejected: valid time = 0
> >>>>> Rejected: bad obs value = 0
> >>>>> Rejected: off the grid = 1041
> >>>>> Rejected: level mismatch = 0
> >>>>> Rejected: message type = 0
> >>>>> Rejected: masking region = 0
> >>>>> Rejected: bad fcst value = 0
> >>>>> Computing Categorical Statistics.
> >>>>> Computing Continuous Statistics.
> >>>>> Computing Scalar Partial Sums.
> >>>>>
> >>>>>
> >>>>
> >>
>
--------------------------------------------------------------------------------
> >>>>>
> >>>>> Output file: ./point_stat_240000L_20110601_000000V.stat
> >>>>>
> >>>>>
> >>>>> On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
> >>>>>>
> >>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061
>
> >>>>>>
> >>>>>> Dear Paul,
> >>>>>> I have gone through your assumptions, might seems correct.
But I am
> >>>> having
> >>>>>> some problems when I running Point_stat for rainfall, I am
getting
> >> some
> >>>>>> error as:
> >>>>>>
> >>>>>> $ sh point_stat.sh
> >>>>>> ERROR: GCInfo::set_gcinfo() -> unsupported input file type
> "APCP/A24"
> >>>>>> I have changed configure, "APCP/P24" "APCP/P1000",
"APCP/P925" etc.
> >>>>>> and I ma getting similar type of error. Suggest me how can I
rectify
> >> it.
> >>>>>>
> >>>>>>
> >>>>>> I also tried for GRID_STAT and I got the error as follows
> >>>>>> sh Grid_stat.sh
> >>>>>> GSL_RNG_TYPE=mt19937
> >>>>>> GSL_RNG_SEED=18446744073465844558
> >>>>>> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
> >>>>>> Observation File: TRMM_16AUG2011.nc
> >>>>>> Configuration File:
> >>>>>> /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
> >>>>>> GRIB Record 252: Init = 20110815_000000, Valid =
20110816_000000,
> >> Accum
> >>>> =
> >>>>>> 240000, Min = 0, Max = 148.8
> >>>>>> ncattinq: ncid 4; varid -1; attname "Projection": Attribute
not
> found
> >>>>>>
> >>>>>>
> >>>>>> One more thing I want to ask you about plot_point_obs.
Actually
> >>>>>> plot_point_obs executable does not exits in my METv3.0.1/bin.
> Suppose
> >> I
> >>>>> need
> >>>>>> it, then what I have to do?
> >>>>>>
> >>>>>> Thanks
> >>>>>> HARVIR
> >>>>>>
> >>>>>> *PS: All the data files and script files have been
transferred in
> >>>>>> incoming/irap/met_help/singh_data
> >>>>>> *
> >>>>>> On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
> >>>>>> met_help at ucar.edu> wrote:
> >>>>>>
> >>>>>>> Harvir,
> >>>>>>>
> >>>>>>> Regarding your second request related to TRMM data, I wrote
a small
> >>>>> script
> >>>>>>> (attached) to convert TRMM_16AUG2011.asc into
> >>>>>>> a format that is readable by ascii2nc. Then, I plotted the
> >> observation
> >>>>>>> points using the MET utility plot_point_obs and
> >>>>>>> created the attached image showing the range of points. It
is
> >>>> important
> >>>>> to
> >>>>>>> note that I had to make several assumptions
> >>>>>>> about your observations to do this conversion. The
assumptions
> are:
> >>>>>>>
> >>>>>>> 1. Your observations are surface observations (column 1 -
message
> >> type
> >>>>>>> ADPSFC). This does make sense if your data is
> >>>>>>> accumulated precipitation, which I believe it is.
> >>>>>>> 2. The valid time of your observations is 2011-08-11
00:00:00 UTC.
> >>>>>>> Obviously, the date is shown in your filename, but
> >>>>>>> I could not find the valid time.
> >>>>>>> 3. The elevation in msl of all observing points is 0m,
which is
> >>>>> definitely
> >>>>>>> a bad assumption, but for "surface"
> >>>>>>> observations will not matter.
> >>>>>>> 4. The accumulation interval of your observations is 6
hours
> (column
> >>>> 8).
> >>>>>>> If this is not correct, please modify the
> >>>>>>> script I attached accordingly.
> >>>>>>> 5. The height in msl of the observation points is 0m (see
> assumption
> >>>> 3).
> >>>>>>>
> >>>>>>> Once you have model data that contains APCP with the same
> >> accumulation
> >>>>>>> interval as that of your observations, you should
> >>>>>>> be able to use this acii2nc-generate observations in
point_stat.
> >>>>>>>
> >>>>>>> Here are the steps to generate the ascii2nc NetCDF obs data
file
> and
> >>>> plot
> >>>>>>> using the script I am sending. These commands
> >>>>>>> assume that the MET bin/ sub-folder is on your path.
> >>>>>>>
> >>>>>>> $ chmod +x trmm_conv.pl
> >>>>>>> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl >
TRMM_16AUG2011.txt
> >>>>>>> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
> >>>>>>> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim
-15 45 45
> >> 120
> >>>>>>>
> >>>>>>> Please let me know if you have any questions.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Paul
> >>>>>>>
> >>>>>>>
> >>>>>>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh}
wrote:
> >>>>>>>>
> >>>>>>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
> >>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>>>>>> Queue: met_help
> >>>>>>>> Subject: Need help for ASCII2NC and Point Stat tool
> >>>>>>>> Owner: Nobody
> >>>>>>>> Requestors: harvirduphy at gmail.com
> >>>>>>>> Status: new
> >>>>>>>> Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Dear Paul,
> >>>>>>>>
> >>>>>>>> I would like to thank you Paul for your necessary
suggestions.
> I've
> >>>>>>>> implemented them and now its work but I want use this file
(
> output
> >>>> from
> >>>>>>>> ascii2nc) for Point Stat tool. I have tried to run
point_stat, its
> >>>>>>> running
> >>>>>>>> and creating some stat files too but stats file having only
> headers
> >> no
> >>>>>>>> statistics. I'm not getting why its so?
> >>>>>>>> I've transferred data files as well as script file in
> >>>>>>>> incoming/irap/met_help/singh_data. Here I want to run it
for TMP
> and
> >>>>>>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
> >>>>>>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
> >>>>>>>> 20110601_060000_rf.dat
> >>>>>>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast
file is
> >> WRF
> >>>>>>>> outfile in GRIB1 format (output from WPPv3.2) named as
> >>>>>>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I want to ask you something on some other data. Suppose I
have
> some
> >>>>> ASCII
> >>>>>>>> data having only 3 columns viz. Latitude, Longitude and
Rainfall
> >>>> (say).
> >>>>>>> Can
> >>>>>>>> we convert it through ascii2nc for MET supported format? If
yes,
> >> then
> >>>>>>> what
> >>>>>>>> we can take rest of the columns. I also transferred this
kind of
> >> data
> >>>>>>> named
> >>>>>>>> as TRMM_16AUG2011.asc,
> >>>>>>>> TRMM_16AUG2011.dat
> >>>>>>>> I want to use point stat for this data also.
> >>>>>>>>
> >>>>>>>> Thank you in advance.
> >>>>>>>>
> >>>>>>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
> >>>>>>> met_help at ucar.edu
> >>>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Harvir,
> >>>>>>>>>
> >>>>>>>>> I have identified several problems with the ascii2nc input
file
> >>>>>>>>> suflnd_20110601_06z_rrf.dat:
> >>>>>>>>>
> >>>>>>>>> 1. ascii2nc input columns must be space-delimited.
Several of
> the
> >>>>>>> columns
> >>>>>>>>> in the input file are tab delimited. Do a
> >>>>>>>>> find/replace on the file to change all tabs to spaces.
> >>>>>>>>>
> >>>>>>>>> 2. After you perform step 1, you will see the following
error
> when
> >>>> you
> >>>>>>> try
> >>>>>>>>> to run ascii2nc:
> >>>>>>>>>
> >>>>>>>>> $ ascii2nc.sh
> >>>>>>>>> Creating NetCDF Observation file:
suflnd_20110601_06z_RF.nc
> >>>>>>>>> Reading ASCII Observation file:
suflnd_20110601_06z_rrf.dat
> >>>>>>>>> Processing 11414 observations.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> timestring_to_sec(const char *) -> can't parse time
string
> "974.8"
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> The reason for this error is that your file appears to
contain
> >>>>>>> accumulated
> >>>>>>>>> precipitation observations (GRIB code 61 in
> >>>>>>>>> column 7). In this case, ascii2nc is expecting an
accumulation
> >>>>> interval
> >>>>>>> in
> >>>>>>>>> column 8 in the format HH, for example 06 or
> >>>>>>>>> 24. That is the reason for the error message above.
> >>>>>>>>>
> >>>>>>>>> 3. Column 9 should contain the height above ground of the
> >>>> observation
> >>>>>>>>> station in meters. For accumulated
> >>>>>>>>> precipitation, this information is ignored by ascii2nc
because
> this
> >>>> is
> >>>>> a
> >>>>>>>>> "surface" observation. We typically use a fill
> >>>>>>>>> value of -9999, indicating invalid data.
> >>>>>>>>>
> >>>>>>>>> 4. Column 10 should contain the observed values. I see
valid
> >>>>>>> observation
> >>>>>>>>> values on some rows, but many are -99.0 which
> >>>>>>>>> I interpret to mean invalid data. In this case, it does
not make
> >>>> sense
> >>>>>>> to
> >>>>>>>>> include invalid observation data, and those
> >>>>>>>>> lines should be deleted.
> >>>>>>>>>
> >>>>>>>>> I took the liberty of making a couple assumptions about
your data
> >> and
> >>>>>>>>> trying to adjust the input file that you sent me
> >>>>>>>>> to something that ascii2nc can read. The assumptions I
made are:
> >>>>>>>>>
> >>>>>>>>> 1. All of your observations are "surface" observations,
so the
> >>>> message
> >>>>>>>>> type (column 1) was set to ADPSFC
> >>>>>>>>> 2. The accumulation interval is 6 hours (column 8 is set
to 06)
> >>>>>>>>> 3. The observation values are in mm, and left as-is
> >>>>>>>>>
> >>>>>>>>> If these assumptions, mainly #2 and #3, are incorrect,
please
> make
> >>>> the
> >>>>>>>>> appropriate adjustments. I attached the "fixed"
> >>>>>>>>> file to this email for you to review and change as you
wish.
> >> Please
> >>>>> let
> >>>>>>> me
> >>>>>>>>> know if you have any questions.
> >>>>>>>>>
> >>>>>>>>> Paul
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh}
wrote:
> >>>>>>>>>>
> >>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
> >>>>>>>>>>
> >>>>>>>>>>> Dear Paul,
> >>>>>>>>>>>
> >>>>>>>>>>> Accept My apology. Actually yesterday I forgot to
attached
> th
> >>>>>>> files.
> >>>>>>>>>> However, as per your advice through ftp I have putted the
same
> >>>>> attached
> >>>>>>>>>> files there also. and its path is
> >>>>>>>>>> *incoming/irap/met_help/singh_data*
> >>>>>>>>>>
> >>>>>>>>>>> I'm sending you ASCII file and ascii2nc script. Please
have
> close
> >>>>> look
> >>>>>>>>> on
> >>>>>>>>>>> it if am wrong somewhere please let me know.
> >>>>>>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
> >> successful
> >>>> or
> >>>>>>>>> not
> >>>>>>>>>>> in pcp_combine ... However, I've run all the test
scripts
> >>>>>>> successfully.
> >>>>>>>>>>> Thanx in advance...
> >>>>>>>>>>>
> >>>>>>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul
Oldenburg} <
> >>>>>>>>>>> met_help at ucar.edu> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Harvir,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm afraid that MET cannot read TRMM data directly.
The best
> >> way
> >>>> to
> >>>>>>>>> read
> >>>>>>>>>>>> TRMM data into MET would be to convert the
> >>>>>>>>>>>> data from it's current format into a NetCDF file that
has the
> >> same
> >>>>>>>>> format
> >>>>>>>>>>>> as the output of the MET pcp_combine tool. If
> >>>>>>>>>>>> you ran the MET test scripts after you built MET, you
should
> be
> >>>> able
> >>>>>>> to
> >>>>>>>>>>>> find pcp_combine output files in the sub-folder
> >>>>>>>>>>>> out/pcp_combine under your MET build folder. If you
are
> >>>> successful
> >>>>>>> in
> >>>>>>>>>>>> converting your data in this way, please send us
> >>>>>>>>>>>> your code so that we can make it available for other
MET users
> >> who
> >>>>>>>>> might
> >>>>>>>>>>>> want to use TRMM data. I'm sorry the answer to
> >>>>>>>>>>>> this question is not more simple.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Regarding your ascii2nc question, can you please send
us the
> >>>>> ascii2nc
> >>>>>>>>>>>> input that is causing your error? Also, what is
> >>>>>>>>>>>> the command line statement that you are using to invoke
> >> ascii2nc?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Paul
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh}
wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted
upon.
> >>>>>>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
> >>>>>>>>>>>>> Queue: met_help
> >>>>>>>>>>>>> Subject: Getting Problem in converting TRMM
rainfall to
> >> MET
> >>>>>>>>> format
> >>>>>>>>>>>>> Owner: Nobody
> >>>>>>>>>>>>> Requestors: harvirduphy at gmail.com
> >>>>>>>>>>>>> Status: new
> >>>>>>>>>>>>> Ticket <URL:
> >>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Dear Sir /Madam,
> >>>>>>>>>>>>> I am using TRMM rainfall Netcdf or bin data for
grid_stat MET
> >>>> tool?
> >>>>>>>>>>>> which
> >>>>>>>>>>>>> converter should did I use to convert TRMM Rainfall
data (.nc
> >> or
> >>>>>>> .bin)
> >>>>>>>>>>>> to
> >>>>>>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
> Processing
> >>>>>>>>>>>> (UPPv1.0),
> >>>>>>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
> >>>>>>>>>>>>> I want to use some ASCII DATA may be output of some
models
> like
> >>>> GFS
> >>>>>>>>> etc
> >>>>>>>>>>>> ..
> >>>>>>>>>>>>> I have extracted data in the desired 10 columns. I'm
trying
> to
> >>>> use
> >>>>>>>>>>>> *ascii2nc
> >>>>>>>>>>>>> *this also giving some error like: open_met_ascii() ->
line
> >>>> number
> >>>>> 0
> >>>>>>>>>>>> doesn't
> >>>>>>>>>>>>> contain the expected number of colu
> >>>>>>>>>>>>> N_items = 7
> >>>>>>>>>>>>> N_chars = 512
> >>>>>>>>>>>>> N_ints = 100
> >>>>>>>>>>>>> LineNumber = 1
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Item[ 0] = "SIDL210 10738
20110601_030000 "
> >>>>>>>>>>>>> Item[ 1] = "48.68"
> >>>>>>>>>>>>> Item[ 2] = "9.23 "
> >>>>>>>>>>>>> Item[ 3] = "391.0 61 "
> >>>>>>>>>>>>> Item[ 4] = "974.8"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Item[ 5] = "1021.5"
> >>>>>>>>>>>>> Item[ 6] = "-99.0"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Offset[ 0] = 0
> >>>>>>>>>>>>> Offset[ 1] = 32
> >>>>>>>>>>>>> Offset[ 2] = 41
> >>>>>>>>>>>>> Offset[ 3] = 47
> >>>>>>>>>>>>> Offset[ 4] = 57
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Offset[ 5] = 64
> >>>>>>>>>>>>> Offset[ 6] = 71
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Please Help me....
> >>>>>>>>>>>>> Earliest reply will be appreciated.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks in advance for any help you can provide!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1,
for Post
> >>>>>>>>> Processing
> >>>>>>>>>>>>> Tool "UPPv1.0 "*
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
>
>
>
--
* Regards*
**************************************************************
**
* HARVIR SINGH <http://www.harvir007.webs.com>*
* To Know more <http://www.harvir007.webs.com> **
***************************************************************
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: John Halley Gotway
Time: Fri Sep 02 15:42:58 2011
Harvir,
Good news. We were able to put together a script in R to reformat
that data from the ASCII point format from Giovanni site into a NetCDF
format that MET can read. The R script to perform this task
is attached. I've also posted a tarball for you on our ftp site to
give you a more detailed example of using it on your data:
ftp.rap.ucar.edu/incoming/irap/met_help/for_singh.tar.gz
In there, you'll find a script named "run_met.sh". In this script, I
show an example of performing the following steps:
(1) Using 'wgrib' to extract out the single 24-hour accumulation
record from your forecast GRIB file.
(2) Using 'copygb' to regrid that data onto the 0.25x0.25 degree grid
on which the TRMM observations reside.
(3) Call 'trmm2nc.R' to reformat ASCII TRMM point data into the NetCDF
format accepted by MET.
(4) Calling 'grid_stat' to verify your model versus the TRMM data.
(4) Calling 'mode' to do an object-based verification.
Please take a look at what I've sent and let me know if any questions
arise. Please note that to get the Rscript to run, you'll need to
have the R 'ncdf' package installed on your machine. That
package enables R to interface with NetCDF files.
Hopefully this will get you going on running your data through MET's
Grid-Stat and MODE tools.
Thanks,
John Halley Gotway
met_help at ucar.edu
On 09/02/2011 06:49 AM, RAL HelpDesk {for Harvir Singh} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>
> Dear Paul,
>
> I would like to thank you for your invaluable guidance. The
Point_stats
> seems working perfectly but now I have been told to work on
Grid_stats.
> Could you suggest me " How can I implement for Grid_stats for same
TRMM
> data?"
> or How can I run grid_stat for TRMM data?
> Thanx
> HARVIR
>
>
> On 1 September 2011 21:12, RAL HelpDesk {for Paul Oldenburg} <
> met_help at ucar.edu> wrote:
>
>> Harvir,
>>
>> There are a couple issues with the data that you sent me:
>>
>> 1. The TRMM data files you sent, TRMM_RF_15AUG2011_0.25x.dat and
>> TRMM_RF_16AUG2011_0.25x.dat, contain rainfall data for
>> 3 hour accumulation intervals. You can see this in the header of
those
>> files:
>>
>> $ head TRMM_RF_16AUG2011_0.25x.dat
>> Selected parameter: 3-hourly TMPA-RT Accumulated Rainfall
>> ...
>>
>> So, I went to the website you sent and downloaded 24 hour (daily)
rainfall
>> instead:
>>
>>
http://disc2.nascom.nasa.gov/Giovanni/tovas/realtime.3B42RT_daily.shtml
>>
>> Then, I performed the steps shown below (with output), including
running
>> point_stat with a modified version your
>> point_stat config file, PointStatConfig_rf. All of the input and
output
>> files that I used and created are included in
>> the tarball PointStat_run.tar.gz on the FTP. Please download this
and try
>> to follow my process.
>>
>> $ cat TRMM_RF_20110816.dat | ./trmm_conv.pl > TRMM_RF_20110816.txt
>> $ ascii2nc TRMM_RF_20110816.txt TRMM_RF_20110816.nc -v 3
>> Creating NetCDF Observation file: TRMM_RF_20110816.nc
>> Reading ASCII Observation file: TRMM_RF_20110816.txt
>> Processing 74725 observations.
>> Finished processing 74725 observations.
>> $ plot_point_obs TRMM_RF_20110816.nc TRMM_RF_20110816.ps -dim -20
35 55 130
>> Opening netCDF file: TRMM_RF_20110816.nc
>> Processing 74725 observations at 74725 locations.
>> Observation GRIB codes: ALL
>> Observation message types: ALL
>> Creating postscript file: TRMM_RF_20110816.ps
>> Finished plotting 74725 locations.
>> $ $MET_BASE/bin/point_stat \
>> wrfprs_d01_2011-08-16_000_fcst01.grib1 \
>> TRMM_RF_20110816.nc \
>> PointStatConfig_rf \
>> -outdir . -v 3
>> GSL_RNG_TYPE=mt19937
>> GSL_RNG_SEED=18446744073636306766
>> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
>> Climatology File: none
>> Configuration File: PointStatConfig_rf
>> Observation File: TRMM_RF_20110816.nc
>>
>>
>>
--------------------------------------------------------------------------------
>>
>> Reading records for APCP_24/A24.
>> GRIB Record 252: Init = 20110815_000000, Valid = 20110816_000000,
Accum =
>> 240000, Min = 0, Max = 235.7
>> For APCP_24/A24 found 1 forecast levels and 0 climatology levels.
>>
>>
>>
--------------------------------------------------------------------------------
>>
>> Searching 74725 observations from 74725 PrepBufr messages.
>>
>>
>>
--------------------------------------------------------------------------------
>>
>> Processing APCP_24/A24 versus APCP_24/A24, for observation type
ADPSFC,
>> over region FULL, for interpolation method
>> UW_MEAN(1), using 66238 pairs.
>> Number of matched pairs = 66238
>> Observations processed = 74725
>> Rejected: GRIB code = 0
>> Rejected: valid time = 0
>> Rejected: bad obs value = 0
>> Rejected: off the grid = 0
>> Rejected: level mismatch = 0
>> Rejected: message type = 0
>> Rejected: masking region = 0
>> Rejected: bad fcst value = 8487
>> Computing Categorical Statistics.
>> Computing Continuous Statistics.
>> Computing Scalar Partial Sums.
>>
>>
>>
--------------------------------------------------------------------------------
>>
>> Output file: ./point_stat_240000L_20110816_000000V.stat
>>
>>
>> I want to call your attention to the fact that I could only verify
one of
>> your model files,
>> wrfprs_d01_2011-08-16_000_fcst01.grib1. It appears that you do not
have
>> consistent accumulation intervals:
>>
>> $ wgrib wrfprs_d01_2011-08-15_000_fcst01.grib1 | grep APCP
>>
252:28117402:d=11081500:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=0:TimeU=1:sfc:0-
0hr
>> acc:NAve=0
>> $ wgrib wrfprs_d01_2011-08-16_000_fcst01.grib1 | grep APCP
>>
252:28581058:d=11081500:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
>> acc:NAve=0
>> $ wgrib wrfprs_d01_2011-08-17_000_fcst01.grib1 | grep APCP
>>
252:28514820:d=11081500:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=48:TimeU=1:sfc:0-
48hr
>> acc:NAve=0
>>
>> You can see near the end of the lines that the accumulation
intervals
>> fluctuate: 0-0hr acc, 0-24hr acc, 0-48hr acc. I
>> am not an experienced WPP user, so I'm not sure how to change the
>> post-processing configuration so that the accumulation
>> interval is always 24 hours.
>>
>> As for your question about grid_stat, it is possible for you to use
>> grid_stat, but it will not be easy. First of all,
>> the MET tool ascii2nc only outputs NetCDF data that can be used by
>> point_stat, not grid_stat. To use grid_stat with the
>> TRMM ascii data, you must put it into a NetCDF file that has a
format
>> identical to the output of the MET tool
>> pcp_combine. MET does not contain any tools that perform this
task, so
>> this is the difficult part. If I were going to
>> try to do this, I would probably use a data analysis package like
R.
>>
>> The point_stat tool is capable of calculating a lot of verification
>> statistics, so maybe it will suit your needs.
>>
>> Please let me know if you have any other questions.
>>
>> Paul
>>
>>
>> On 09/01/2011 06:26 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>
>>> Dear Paul,
>>>
>>> Yes I have the TRMM data file for the day before and I've
transferred to
>>> met_help. The TRMM rainfall I'm using, is available on web also
i.e. :
>>> http://disc2.nascom.nasa.gov/Giovanni/tovas/realtime.3B42RT.shtml
(ASCII)
>>> and for 15th Aug 2011 data @
>>>
>> http://disc2.nascom.nasa.gov/daac-
bin/Giovanni/tovas/Giovanni_cgi.pl?west=45.0&north=45.0&east=121.0&south=-
16.0¶ms=0%7C3B42RT&plot_type=Area+Plot&byr=2011&bmo=08&bdy=14&bhr=00&eyr=2011&emo=08&edy=15&ehr=00&begin_date=2008%2F10%2F01%2F00&end_date=2011%2F08%2F31%2F12&cbar=cdyn&cmin=&cmax=&yaxis=ydyn&ymin=&ymax=&yint=&ascres=0.25x0.25&global_cfg=tovas.global.cfg.pl&instance_id=realtime&prod_id=3B42RT&action=ASCII+Output
>> .
>>>
>>> Actually I've tried a lot to work on Point_stat but every time I
struck
>>> somewhere.. so I've some questions:
>>> 1. This METv3.0.1 software is only meant for WRF-NMM or WRF-ARW as
well.
>>>
>>> For this point of time I'm using WRF-ARW output to WRF Post
>> Processing
>>> ie WPPv3.2 and converting to GRIB1 format. And for observation
data I'm
>>> downloading ASCII data as above said link. When I'm converting
ASCII data
>> to
>>> NetCDF I've also taken several assumptions for surface data like:
>>> Massage type: ADPSFC
>>> Station ID: TRMM
>>> Date : 20110816_000000
>>> Lat
>>> Lon
>>> Elevation: -9999 ( as data is not available)
>>> g.c. : 61 for Total precipitation
>>> Level : 24 hour ( for 24 accumulation rainfall)
>>> height : -9999 ( as data is not available)
>>>
>>>
>>> for both forecast and observation I've used 245x305 grid points at
the
>>> interval of 0.25 degree within the Indian region ie 16S to 45N
and 45E
>> to
>>> 121E.
>>> And fore converting WRF-ARF outfiles by WPPv3.2 have used copygb
with the
>>> given information as:
>>> grid='255 0 305 245 -16000 45000 128 45000 121000 250 250 64'
>>> ${POSTEXEC}/copygb.exe -g"${grid}" -x
>>> WRFPRS_${domain}_${YY}-${MM}-${DD}.${fhr}_fcst01 WRFPRS
>>> _fcst01/wrfprs_${domain}_${YY}-${MM}-${DD}_${fhr}_fcst01.grib1
>>>
>>>
>>> 2. Here I'm using Gridded data either from TRMM or from GTS or AWS
etc
>> ...
>>> then why I can't use Grid_stat?
>>>
>>>
>>> I'm not getting it. I may be using some wrong information during
the data
>>> format conversion i.e. creating 10 columns for ASCII2NC. I've
tried it
>> for
>>> several combinations for point stat as NetCDF as forecast file and
GRIB1
>> as
>>> observation file and vice-verso, also tried several combinations
in
>>> PointStatConfug_rf as:
>>> // fcst_field[] = ["APCP_A24(*,*)"];
>>> fcst_field[] = ["APCP/A24"];
>>> //obs_field[] = ["APCP/A24"];
>>> //obs_field[] = [];
>>> obs_field[] = ["APCP_A24(0,5,*,*)"];
>>> and many more....
>>>
>>> Thanx
>>> HARVIR
>>>
>>> On 31 August 2011 01:25, RAL HelpDesk {for Paul Oldenburg} <
>>> met_help at ucar.edu> wrote:
>>>
>>>> Harvir,
>>>>
>>>> If I understand you correctly, you are saying that your
observation data
>> is
>>>> accumulated rainfall from 20110816_000000 to
>>>> 20110817_000000. Your model data appears to contain accumulated
>> rainfall
>>>> from 20110815_000000 to 20110816_000000.
>>>> Obviously, these time windows do not overlap. Do you have the
TRMM data
>>>> file for the day before, so it matches the
>>>> model data?
>>>>
>>>> Paul
>>>>
>>>>
>>>> On 08/29/2011 06:29 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061 >
>>>>>
>>>>> Dear Paul,
>>>>> With the reference to your earlier mails, wrfprs_d01_2011-06-
01_000_
>>>>>>
>>>>>> fcst01.grib1 was for forecast file, obs_20110601_060000_rf.dat
>>>> observation
>>>>>> file from GTS data and
>>>>>> TRMM_16AUG2011.dat is from TRMM rainfall data.
>>>>>>
>>>>>
>>>>>
>>>>> Now I've transferred new set of data files viz.
>>>>> Model forecast file: wrfprs_d01_2011-08-16_000_fcst01.grib1
>>>>> Observation file : TRMM_RF_16AUG2011_0.25.txt
>>>>>
>>>>> and TRMM Rainfall data is from 00Z 16 AUG 2011 to 00Z 17 AUG
2011 and
>>>> model
>>>>> data is for 24 hr acumulated..
>>>>> thanx
>>>>>
>>>>>
>>>>>
>>>>> On 26 August 2011 20:05, RAL HelpDesk {for Paul Oldenburg} <
>>>>> met_help at ucar.edu> wrote:
>>>>>
>>>>>> Harvir,
>>>>>>
>>>>>> In the last email I sent you, I said that your rainfall
observation
>> data
>>>> in
>>>>>> your model data file,
>>>>>> wrfprs_d01_2011-06-01_000_fcst01.grib1, had an accumulation
interval
>> of
>>>> 24
>>>>>> hours. That means that the amount of
>>>>>> rainfall in the model data file is from a 24 hour period.
>>>>>>
>>>>>> Do you know what the accumulation interval of the rainfall data
you
>> sent
>>>> me
>>>>>> in either obs_20110601_060000_rf.dat or
>>>>>> TRMM_16AUG2011.dat is? I see the following information at the
top of
>>>> the
>>>>>> file TRMM_16AUG2011.dat:
>>>>>>
>>>>>> Selected parameter: TRMM TMPA-RT Daily Accumulated Rainfall
>>>>>> Selected area: lat=[16.161S,45.311N], lon=[44.182E,121.960E]
>>>>>> Selected time period: (16Aug2011-16Aug2011)
>>>>>>
>>>>>> Does that mean that it is 24 hour accumulated rainfall from
2011-08-15
>>>>>> 00:00:00 to 2011-08-16 00:00:00? If so, then it
>>>>>> has a 24 hour accumulation interval, but it does not match the
date in
>>>> your
>>>>>> model data file. Do you have a TRMM file
>>>>>> that is valid at the same time as your model data file?
>>>>>>
>>>>>> I need to know the answers to these questions before I can make
>> further
>>>>>> progress helping you verify rainfall. It is
>>>>>> very important that both the accumulation interval and the
valid time
>> of
>>>>>> your model and observation data match.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Paul
>>>>>>
>>>>>>
>>>>>> On 08/26/2011 07:02 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>>
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061
>
>>>>>>>
>>>>>>> Dear Paul,
>>>>>>>
>>>>>>> Thank You for your prompt reply. Ofcourse yes I've to compute
Threat
>>>>>> score
>>>>>>> too.. For this point of time I just want to go with point
stat. and
>> I'm
>>>>>>> trying it for WRF ARW core (output data) it. I've tried
point_stat,
>> its
>>>>>>> running but giving headers only in stat txt files. I'm unable
to
>>>> rectify
>>>>>> it.
>>>>>>> Please help me to come out of this problem. I've transfrred
data in
>>>>>>> incoming/irap/met_help/singh_data
>>>>>>> thanx in advance!
>>>>>>>
>>>>>>> ---------- Forwarded message ----------
>>>>>>> From: RAL HelpDesk {for Paul Oldenburg} <met_help at ucar.edu>
>>>>>>> Date: 23 August 2011 21:55
>>>>>>> Subject: Re: [rt.rap.ucar.edu #49061] Need help for ASCII2NC
and
>> Point
>>>>>> Stat
>>>>>>> tool
>>>>>>> To: harvirduphy at gmail.com
>>>>>>>
>>>>>>>
>>>>>>> Harvir,
>>>>>>>
>>>>>>> There are a couple points to discuss before we continue.
First, I
>>>>>> recommend
>>>>>>> that you use point_stat at this time to
>>>>>>> perform verification on your model data. The reason is that
>> converting
>>>>>> your
>>>>>>> observation data to a format that grid_stat
>>>>>>> can read is more difficult because it must contain projection
>>>> information
>>>>>>> that matches your model data projection. Why
>>>>>>> do you want to run grid_stat? Are you interested in
calculating
>>>>>>> neighborhood statistics or just basic verification
>>>>>>> statistics like RMSE and the Equitable Threat Score? If you
only
>> want
>>>>>> basic
>>>>>>> statistics, point_stat will work just fine.
>>>>>>>
>>>>>>> Second, for rainfall data, you said that the assumptions that
I have
>>>> made
>>>>>>> about your observations are correct. If that
>>>>>>> is the case, and the accumulation interval of your observation
data
>> is
>>>> 6
>>>>>>> hours, then there is a mis-match with the
>>>>>>> accumulation interval in your model data, which is 24 hours
(see
>> 0-24hr
>>>>>>> acc):
>>>>>>>
>>>>>>> $ wgrib wrfprs_d01_2011-06-01_000_fcst01.grib1 | grep APCP
>>>>>>>
>>>>>>
>>>>
>>
252:323426:d=11053100:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
>>>>>>> acc:NAve=0
>>>>>>>
>>>>>>> If you are sure that your rainfall observations have an
accumulation
>>>>>>> interval of 6 hours, then you must combine four of
>>>>>>> your 6 hour observation files together to form rainfall
observations
>>>> with
>>>>>> an
>>>>>>> accumulation interval of 24 hours. A good
>>>>>>> way to do this is to use the MET tool pcp_combine after you
run
>>>> ascii2nc
>>>>>> on
>>>>>>> your rainfall observation files.
>>>>>>>
>>>>>>> Finally, I tried to apply the model data that you gave me with
>> surface
>>>>>>> temperature to the observation data you gave me
>>>>>>> last week, 20110601_060000_tt.dat. I put the files involved
in this
>>>>>>> experiment are on the RAL FTP site in the archive
>>>>>>> ftp.rap.ucar.edu:
>>>> /incoming/irap/met_help/singh_data/PointStatVx.tar.gz.
>>>>>> If
>>>>>>> you download this and unpack it, you should
>>>>>>> be able to reproduce the output file that I included,
>>>>>>> point_stat_240000L_20110601_000000V.stat. This file contains
>>>>>>> basic verification statistics, like continuous and categorical
>>>>>> statistics.
>>>>>>> I used the threshold 273.15 K, which is 0 C.
>>>>>>>
>>>>>>> Note that for this experiment to be valid, I had to open the
time
>>>> window
>>>>>> to
>>>>>>> +/- 6 hours because your model data is valid
>>>>>>> at 20110601_000000 and your observations are valid at
>> 20110601_060000.
>>>>>> This
>>>>>>> may or may not be an acceptable compromise
>>>>>>> to your for your work. Also, I assumed that the temperature
>>>> observations
>>>>>>> were all on the surface (message type ADPUPA).
>>>>>>> It also looked like your observations were in degrees
Celsius, so I
>>>>>>> converted them to Kelvin. Please let me know if
>>>>>>> you have any questions or trouble following my example.
Please look
>>>>>>> carefully at the files included in the archive,
>>>>>>> especially PointStatConfig_tmp and point_stat.sh.
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>>
>>>>>>> $ tar xzvf PointStatVx.tar.gz
>>>>>>> $ cat obs_20110601_060000_tt.dat | ./obs_conv.pl >
>>>>>>> obs_20110601_060000_tt.txt
>>>>>>> $ ascii2nc obs_20110601_060000_tt.txt
obs_20110601_060000_tt.nc -v 3
>>>>>>> Creating NetCDF Observation file: obs_20110601_060000_tt.nc
>>>>>>> Reading ASCII Observation file: obs_20110601_060000_tt.txt
>>>>>>> Processing 1099 observations.
>>>>>>> Finished processing 1099 observations.
>>>>>>> $ ./point_stat.sh
>>>>>>> GSL_RNG_TYPE=mt19937
>>>>>>> GSL_RNG_SEED=231822158
>>>>>>> Forecast File: wrfprs_d01_2011-06-01_000_fcst01.grib1
>>>>>>> Climatology File: none
>>>>>>> Configuration File: PointStatConfig_tmp
>>>>>>> Observation File: obs_20110601_060000_tt.nc
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
--------------------------------------------------------------------------------
>>>>>>>
>>>>>>> Reading records for TMP/Z2.
>>>>>>> GRIB Record 244: Init = 20110531_000000, Valid =
20110601_000000,
>> Accum
>>>> =
>>>>>>> 000000, Min = 292.579, Max = 303.165
>>>>>>> For TMP/Z2 found 1 forecast levels and 0 climatology levels.
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
--------------------------------------------------------------------------------
>>>>>>>
>>>>>>> Searching 1099 observations from 1099 PrepBufr messages.
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
--------------------------------------------------------------------------------
>>>>>>>
>>>>>>> Processing TMP/Z2 versus TMP/Z2, for observation type ADPSFC,
over
>>>> region
>>>>>>> FULL, for interpolation method UW_MEAN(1),
>>>>>>> using 58 pairs.
>>>>>>> Number of matched pairs = 58
>>>>>>> Observations processed = 1099
>>>>>>> Rejected: GRIB code = 0
>>>>>>> Rejected: valid time = 0
>>>>>>> Rejected: bad obs value = 0
>>>>>>> Rejected: off the grid = 1041
>>>>>>> Rejected: level mismatch = 0
>>>>>>> Rejected: message type = 0
>>>>>>> Rejected: masking region = 0
>>>>>>> Rejected: bad fcst value = 0
>>>>>>> Computing Categorical Statistics.
>>>>>>> Computing Continuous Statistics.
>>>>>>> Computing Scalar Partial Sums.
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
--------------------------------------------------------------------------------
>>>>>>>
>>>>>>> Output file: ./point_stat_240000L_20110601_000000V.stat
>>>>>>>
>>>>>>>
>>>>>>> On 08/23/2011 04:55 AM, RAL HelpDesk {for Harvir Singh} wrote:
>>>>>>>>
>>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061
>
>>>>>>>>
>>>>>>>> Dear Paul,
>>>>>>>> I have gone through your assumptions, might seems correct.
But I am
>>>>>> having
>>>>>>>> some problems when I running Point_stat for rainfall, I am
getting
>>>> some
>>>>>>>> error as:
>>>>>>>>
>>>>>>>> $ sh point_stat.sh
>>>>>>>> ERROR: GCInfo::set_gcinfo() -> unsupported input file type
>> "APCP/A24"
>>>>>>>> I have changed configure, "APCP/P24" "APCP/P1000",
"APCP/P925" etc.
>>>>>>>> and I ma getting similar type of error. Suggest me how can I
rectify
>>>> it.
>>>>>>>>
>>>>>>>>
>>>>>>>> I also tried for GRID_STAT and I got the error as follows
>>>>>>>> sh Grid_stat.sh
>>>>>>>> GSL_RNG_TYPE=mt19937
>>>>>>>> GSL_RNG_SEED=18446744073465844558
>>>>>>>> Forecast File: wrfprs_d01_2011-08-16_000_fcst01.grib1
>>>>>>>> Observation File: TRMM_16AUG2011.nc
>>>>>>>> Configuration File:
>>>>>>>> /gpfs1/home/harvir/SW/METv3.0.1/data/config/GridStatConfig_rf
>>>>>>>> GRIB Record 252: Init = 20110815_000000, Valid =
20110816_000000,
>>>> Accum
>>>>>> =
>>>>>>>> 240000, Min = 0, Max = 148.8
>>>>>>>> ncattinq: ncid 4; varid -1; attname "Projection": Attribute
not
>> found
>>>>>>>>
>>>>>>>>
>>>>>>>> One more thing I want to ask you about plot_point_obs.
Actually
>>>>>>>> plot_point_obs executable does not exits in my METv3.0.1/bin.
>> Suppose
>>>> I
>>>>>>> need
>>>>>>>> it, then what I have to do?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> HARVIR
>>>>>>>>
>>>>>>>> *PS: All the data files and script files have been
transferred in
>>>>>>>> incoming/irap/met_help/singh_data
>>>>>>>> *
>>>>>>>> On 23 August 2011 03:20, RAL HelpDesk {for Paul Oldenburg} <
>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>
>>>>>>>>> Harvir,
>>>>>>>>>
>>>>>>>>> Regarding your second request related to TRMM data, I wrote
a small
>>>>>>> script
>>>>>>>>> (attached) to convert TRMM_16AUG2011.asc into
>>>>>>>>> a format that is readable by ascii2nc. Then, I plotted the
>>>> observation
>>>>>>>>> points using the MET utility plot_point_obs and
>>>>>>>>> created the attached image showing the range of points. It
is
>>>>>> important
>>>>>>> to
>>>>>>>>> note that I had to make several assumptions
>>>>>>>>> about your observations to do this conversion. The
assumptions
>> are:
>>>>>>>>>
>>>>>>>>> 1. Your observations are surface observations (column 1 -
message
>>>> type
>>>>>>>>> ADPSFC). This does make sense if your data is
>>>>>>>>> accumulated precipitation, which I believe it is.
>>>>>>>>> 2. The valid time of your observations is 2011-08-11
00:00:00 UTC.
>>>>>>>>> Obviously, the date is shown in your filename, but
>>>>>>>>> I could not find the valid time.
>>>>>>>>> 3. The elevation in msl of all observing points is 0m,
which is
>>>>>>> definitely
>>>>>>>>> a bad assumption, but for "surface"
>>>>>>>>> observations will not matter.
>>>>>>>>> 4. The accumulation interval of your observations is 6
hours
>> (column
>>>>>> 8).
>>>>>>>>> If this is not correct, please modify the
>>>>>>>>> script I attached accordingly.
>>>>>>>>> 5. The height in msl of the observation points is 0m (see
>> assumption
>>>>>> 3).
>>>>>>>>>
>>>>>>>>> Once you have model data that contains APCP with the same
>>>> accumulation
>>>>>>>>> interval as that of your observations, you should
>>>>>>>>> be able to use this acii2nc-generate observations in
point_stat.
>>>>>>>>>
>>>>>>>>> Here are the steps to generate the ascii2nc NetCDF obs data
file
>> and
>>>>>> plot
>>>>>>>>> using the script I am sending. These commands
>>>>>>>>> assume that the MET bin/ sub-folder is on your path.
>>>>>>>>>
>>>>>>>>> $ chmod +x trmm_conv.pl
>>>>>>>>> $ cat TRMM_16AUG2011.asc | ./trmm_conv.pl >
TRMM_16AUG2011.txt
>>>>>>>>> $ ascii2nc TRMM_16AUG2011.txt TRMM_16AUG2011.nc -v 3
>>>>>>>>> $ plot_point_obs TRMM_16AUG2011.nc TRMM_16AUG2011.ps -dim
-15 45 45
>>>> 120
>>>>>>>>>
>>>>>>>>> Please let me know if you have any questions.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Paul
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 08/19/2011 05:10 AM, RAL HelpDesk {for Harvir Singh}
wrote:
>>>>>>>>>>
>>>>>>>>>> Fri Aug 19 05:10:33 2011: Request 49061 was acted upon.
>>>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>>>>> Queue: met_help
>>>>>>>>>> Subject: Need help for ASCII2NC and Point Stat tool
>>>>>>>>>> Owner: Nobody
>>>>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>>>>> Status: new
>>>>>>>>>> Ticket <URL:
>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49061>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Dear Paul,
>>>>>>>>>>
>>>>>>>>>> I would like to thank you Paul for your necessary
suggestions.
>> I've
>>>>>>>>>> implemented them and now its work but I want use this file
(
>> output
>>>>>> from
>>>>>>>>>> ascii2nc) for Point Stat tool. I have tried to run
point_stat, its
>>>>>>>>> running
>>>>>>>>>> and creating some stat files too but stats file having only
>> headers
>>>> no
>>>>>>>>>> statistics. I'm not getting why its so?
>>>>>>>>>> I've transferred data files as well as script file in
>>>>>>>>>> incoming/irap/met_help/singh_data. Here I want to run it
for TMP
>> and
>>>>>>>>>> Rainfall. The files for TMP are 20110601_060000_tt.dat
>>>>>>>>>> 20110601_060000_tt.nc( Created by ASCII2NC) for Rainfall
>>>>>>>>>> 20110601_060000_rf.dat
>>>>>>>>>> 20110601_060000_rf.nc ( Created by ASCII2NC) and forecast
file is
>>>> WRF
>>>>>>>>>> outfile in GRIB1 format (output from WPPv3.2) named as
>>>>>>>>>> wrfprs_d01_2011-06-01-000_fcst01.grib1.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I want to ask you something on some other data. Suppose I
have
>> some
>>>>>>> ASCII
>>>>>>>>>> data having only 3 columns viz. Latitude, Longitude and
Rainfall
>>>>>> (say).
>>>>>>>>> Can
>>>>>>>>>> we convert it through ascii2nc for MET supported format? If
yes,
>>>> then
>>>>>>>>> what
>>>>>>>>>> we can take rest of the columns. I also transferred this
kind of
>>>> data
>>>>>>>>> named
>>>>>>>>>> as TRMM_16AUG2011.asc,
>>>>>>>>>> TRMM_16AUG2011.dat
>>>>>>>>>> I want to use point stat for this data also.
>>>>>>>>>>
>>>>>>>>>> Thank you in advance.
>>>>>>>>>>
>>>>>>>>>> On 3 August 2011 23:16, RAL HelpDesk {for Paul Oldenburg} <
>>>>>>>>> met_help at ucar.edu
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Harvir,
>>>>>>>>>>>
>>>>>>>>>>> I have identified several problems with the ascii2nc input
file
>>>>>>>>>>> suflnd_20110601_06z_rrf.dat:
>>>>>>>>>>>
>>>>>>>>>>> 1. ascii2nc input columns must be space-delimited.
Several of
>> the
>>>>>>>>> columns
>>>>>>>>>>> in the input file are tab delimited. Do a
>>>>>>>>>>> find/replace on the file to change all tabs to spaces.
>>>>>>>>>>>
>>>>>>>>>>> 2. After you perform step 1, you will see the following
error
>> when
>>>>>> you
>>>>>>>>> try
>>>>>>>>>>> to run ascii2nc:
>>>>>>>>>>>
>>>>>>>>>>> $ ascii2nc.sh
>>>>>>>>>>> Creating NetCDF Observation file:
suflnd_20110601_06z_RF.nc
>>>>>>>>>>> Reading ASCII Observation file:
suflnd_20110601_06z_rrf.dat
>>>>>>>>>>> Processing 11414 observations.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> timestring_to_sec(const char *) -> can't parse time
string
>> "974.8"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The reason for this error is that your file appears to
contain
>>>>>>>>> accumulated
>>>>>>>>>>> precipitation observations (GRIB code 61 in
>>>>>>>>>>> column 7). In this case, ascii2nc is expecting an
accumulation
>>>>>>> interval
>>>>>>>>> in
>>>>>>>>>>> column 8 in the format HH, for example 06 or
>>>>>>>>>>> 24. That is the reason for the error message above.
>>>>>>>>>>>
>>>>>>>>>>> 3. Column 9 should contain the height above ground of the
>>>>>> observation
>>>>>>>>>>> station in meters. For accumulated
>>>>>>>>>>> precipitation, this information is ignored by ascii2nc
because
>> this
>>>>>> is
>>>>>>> a
>>>>>>>>>>> "surface" observation. We typically use a fill
>>>>>>>>>>> value of -9999, indicating invalid data.
>>>>>>>>>>>
>>>>>>>>>>> 4. Column 10 should contain the observed values. I see
valid
>>>>>>>>> observation
>>>>>>>>>>> values on some rows, but many are -99.0 which
>>>>>>>>>>> I interpret to mean invalid data. In this case, it does
not make
>>>>>> sense
>>>>>>>>> to
>>>>>>>>>>> include invalid observation data, and those
>>>>>>>>>>> lines should be deleted.
>>>>>>>>>>>
>>>>>>>>>>> I took the liberty of making a couple assumptions about
your data
>>>> and
>>>>>>>>>>> trying to adjust the input file that you sent me
>>>>>>>>>>> to something that ascii2nc can read. The assumptions I
made are:
>>>>>>>>>>>
>>>>>>>>>>> 1. All of your observations are "surface" observations,
so the
>>>>>> message
>>>>>>>>>>> type (column 1) was set to ADPSFC
>>>>>>>>>>> 2. The accumulation interval is 6 hours (column 8 is set
to 06)
>>>>>>>>>>> 3. The observation values are in mm, and left as-is
>>>>>>>>>>>
>>>>>>>>>>> If these assumptions, mainly #2 and #3, are incorrect,
please
>> make
>>>>>> the
>>>>>>>>>>> appropriate adjustments. I attached the "fixed"
>>>>>>>>>>> file to this email for you to review and change as you
wish.
>>>> Please
>>>>>>> let
>>>>>>>>> me
>>>>>>>>>>> know if you have any questions.
>>>>>>>>>>>
>>>>>>>>>>> Paul
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 08/03/2011 12:44 AM, RAL HelpDesk {for Harvir Singh}
wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689 >
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear Paul,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Accept My apology. Actually yesterday I forgot to
attached
>> th
>>>>>>>>> files.
>>>>>>>>>>>> However, as per your advice through ftp I have putted the
same
>>>>>>> attached
>>>>>>>>>>>> files there also. and its path is
>>>>>>>>>>>> *incoming/irap/met_help/singh_data*
>>>>>>>>>>>>
>>>>>>>>>>>>> I'm sending you ASCII file and ascii2nc script. Please
have
>> close
>>>>>>> look
>>>>>>>>>>> on
>>>>>>>>>>>>> it if am wrong somewhere please let me know.
>>>>>>>>>>>>> And for TRMM rainfall I'll back to you later ...if I'll
>>>> successful
>>>>>> or
>>>>>>>>>>> not
>>>>>>>>>>>>> in pcp_combine ... However, I've run all the test
scripts
>>>>>>>>> successfully.
>>>>>>>>>>>>> Thanx in advance...
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 1 August 2011 20:54, RAL HelpDesk {for Paul
Oldenburg} <
>>>>>>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Harvir,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm afraid that MET cannot read TRMM data directly.
The best
>>>> way
>>>>>> to
>>>>>>>>>>> read
>>>>>>>>>>>>>> TRMM data into MET would be to convert the
>>>>>>>>>>>>>> data from it's current format into a NetCDF file that
has the
>>>> same
>>>>>>>>>>> format
>>>>>>>>>>>>>> as the output of the MET pcp_combine tool. If
>>>>>>>>>>>>>> you ran the MET test scripts after you built MET, you
should
>> be
>>>>>> able
>>>>>>>>> to
>>>>>>>>>>>>>> find pcp_combine output files in the sub-folder
>>>>>>>>>>>>>> out/pcp_combine under your MET build folder. If you
are
>>>>>> successful
>>>>>>>>> in
>>>>>>>>>>>>>> converting your data in this way, please send us
>>>>>>>>>>>>>> your code so that we can make it available for other
MET users
>>>> who
>>>>>>>>>>> might
>>>>>>>>>>>>>> want to use TRMM data. I'm sorry the answer to
>>>>>>>>>>>>>> this question is not more simple.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding your ascii2nc question, can you please send
us the
>>>>>>> ascii2nc
>>>>>>>>>>>>>> input that is causing your error? Also, what is
>>>>>>>>>>>>>> the command line statement that you are using to invoke
>>>> ascii2nc?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Paul
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 08/01/2011 03:36 AM, RAL HelpDesk {for Harvir Singh}
wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Mon Aug 01 03:36:57 2011: Request 48689 was acted
upon.
>>>>>>>>>>>>>>> Transaction: Ticket created by harvirduphy at gmail.com
>>>>>>>>>>>>>>> Queue: met_help
>>>>>>>>>>>>>>> Subject: Getting Problem in converting TRMM
rainfall to
>>>> MET
>>>>>>>>>>> format
>>>>>>>>>>>>>>> Owner: Nobody
>>>>>>>>>>>>>>> Requestors: harvirduphy at gmail.com
>>>>>>>>>>>>>>> Status: new
>>>>>>>>>>>>>>> Ticket <URL:
>>>>>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=48689>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Dear Sir /Madam,
>>>>>>>>>>>>>>> I am using TRMM rainfall Netcdf or bin data for
grid_stat MET
>>>>>> tool?
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>> converter should did I use to convert TRMM Rainfall
data (.nc
>>>> or
>>>>>>>>> .bin)
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> GRIB1 format? I've tried copygb through Unified Post
>> Processing
>>>>>>>>>>>>>> (UPPv1.0),
>>>>>>>>>>>>>>> still I'm unable to convert it for METv3.0.1 usage.
>>>>>>>>>>>>>>> I want to use some ASCII DATA may be output of some
models
>> like
>>>>>> GFS
>>>>>>>>>>> etc
>>>>>>>>>>>>>> ..
>>>>>>>>>>>>>>> I have extracted data in the desired 10 columns. I'm
trying
>> to
>>>>>> use
>>>>>>>>>>>>>> *ascii2nc
>>>>>>>>>>>>>>> *this also giving some error like: open_met_ascii() ->
line
>>>>>> number
>>>>>>> 0
>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>> contain the expected number of colu
>>>>>>>>>>>>>>> N_items = 7
>>>>>>>>>>>>>>> N_chars = 512
>>>>>>>>>>>>>>> N_ints = 100
>>>>>>>>>>>>>>> LineNumber = 1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Item[ 0] = "SIDL210 10738
20110601_030000 "
>>>>>>>>>>>>>>> Item[ 1] = "48.68"
>>>>>>>>>>>>>>> Item[ 2] = "9.23 "
>>>>>>>>>>>>>>> Item[ 3] = "391.0 61 "
>>>>>>>>>>>>>>> Item[ 4] = "974.8"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Item[ 5] = "1021.5"
>>>>>>>>>>>>>>> Item[ 6] = "-99.0"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Offset[ 0] = 0
>>>>>>>>>>>>>>> Offset[ 1] = 32
>>>>>>>>>>>>>>> Offset[ 2] = 41
>>>>>>>>>>>>>>> Offset[ 3] = 47
>>>>>>>>>>>>>>> Offset[ 4] = 57
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Offset[ 5] = 64
>>>>>>>>>>>>>>> Offset[ 6] = 71
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please Help me....
>>>>>>>>>>>>>>> Earliest reply will be appreciated.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks in advance for any help you can provide!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *PS:* *I'm working on AIX-5.3 Machine and METv3.0.1,
for Post
>>>>>>>>>>> Processing
>>>>>>>>>>>>>>> Tool "UPPv1.0 "*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>
>>
>>
>
>
------------------------------------------------
Subject: Need help for ASCII2NC and Point Stat tool
From: John Halley Gotway
Time: Fri Sep 02 15:42:58 2011
########################################################################
##
## Name: trmm2nc.R
##
## Description:
## This script should be called via an Rscript command to
reformat
## ASCII TRMM precipitation pcp into a NetCDF format that MET
## can read. The input pcp for this script can be attained via:
## 24-hour accumulations:
##
http://disc2.nascom.nasa.gov/Giovanni/tovas/realtime.3B42RT_daily.shtml
## 03-hour accumulations:
##
http://disc2.nascom.nasa.gov/Giovanni/tovas/realtime.3B42RT.shtml
##
## Usage:
## Rscript trmm2nc.R
## trmm_file
## nc_file
## [-save]
##
## Arguments:
## "trmm_file" is an ASCII file containing TRMM accumulated
precipitation pcp.
## "nc_file" is the out NetCDF file to be written.
## "-save" to call save.image().
##
## Details:
##
## Examples:
## Rscript trmm2nc.R \
## trmm_sample.txt \
## trmm_sample.nc
##
## Author:
## John Halley Gotway (johnhg at ucar.edu), NCAR-RAL-DTC
## 09/02/2011
##
########################################################################
########################################################################
#
# Required libraries.
#
########################################################################
library(ncdf)
########################################################################
#
# Constants and command line arguments
#
########################################################################
hdr_len = 5 # Each pcp file begins with 5 header lines.
hdr_file = "trmm.hdr" # Temporary header file.
missing = -9999 # Missing pcp value to be used in MET.
save = FALSE # If set to TRUE, call save.image()
########################################################################
#
# Handle the arguments.
#
########################################################################
# Retreive the arguments
args = commandArgs(TRUE)
# Check the number of arguments
if(length(args) < 2) {
cat("Usage: trmm2nc.R\n")
cat(" trmm_file\n")
cat(" nc_file\n")
cat(" [-save]\n")
cat(" where \"trmm_file\" is an ASCII file containing TRMM
accumulated precipitation pcp.\n")
cat(" \"nc_file\" is the out NetCDF file to be
written.\n")
cat(" \"-save\" to call save.image().\n\n")
quit()
}
# Store the input file names
trmm_file = args[1]
nc_file = args[2]
# Parse optional arguments
for(i in 1:length(args)) {
if(args[i] == "-save") {
save = TRUE
}
}
########################################################################
#
# Process the header pcp.
#
########################################################################
# Read the header pcp
sys_cmd = paste("head -", hdr_len, " ", trmm_file, " > ", hdr_file,
sep="")
system(sys_cmd)
hdr = read.table(hdr_file, fill=TRUE)
sys_cmd = paste("rm -f", hdr_file)
system(sys_cmd)
# Determine the accumlation interval
if(as.character(hdr[1,3]) == "TRMM") {
acc_hr = 24
} else if(as.character(hdr[1,3]) == "3-hourly") {
acc_hr = 3
} else {
cat("\n\nERROR: Can\'t figure out the accumulation interval!\n\n")
exit(1)
}
acc_sec = acc_hr * 3600
acc_str = sprintf("%.2i", acc_hr)
# Parse the valid time and compute the initialization time.
if(acc_hr == 24) {
valid = as.POSIXct(strptime(unlist(strsplit(as.character(hdr[3,4]),
'-'))[2],
format="%d%b%Y)", tz="GMT"))
} else {
valid = as.POSIXct(strptime(unlist(strsplit(as.character(hdr[3,4]),
'-'))[2],
format="%HZ%d%b%Y)", tz="GMT"))
}
init = valid - acc_sec
########################################################################
#
# Process the accumulated precipitation pcp.
#
########################################################################
# Read the TRMM file
cat(paste("Reading:", trmm_file, "\n"))
trmm = read.table(trmm_file, header=TRUE, skip=hdr_len)
# Determine lat/lon dimensions
lat = unique(trmm$Latitude)
lon = unique(trmm$Longitude)
dlat = lat[2]-lat[1]
dlon = lon[2]-lon[1]
nlat = length(lat)
nlon = length(lon)
# Slice the 1D array into a 2D array
pcp = array(trmm$AccRain, dim=c(nlon, nlat), dimnames=c("lon", "lat"))
pcp[pcp < missing] = missing
########################################################################
#
# Create the NetCDF output file.
#
########################################################################
# Define dimensions
dim_lat = dim.def.ncdf("lat", "degrees_north", lat,
create_dimvar=TRUE, longname="latitude")
dim_lon = dim.def.ncdf("lon", "degrees_east", lon,
create_dimvar=TRUE, longname="longitude")
# Define variables
var_pcp = var.def.ncdf(paste("APCP_", acc_str, sep=''), "kg/m^2",
list(dim_lon, dim_lat), missing,
longname="Total precipitation", prec="single")
# Define file
nc = create.ncdf(nc_file, var_pcp)
# Add variable attributes
att.put.ncdf(nc, var_pcp, "name", "APCP")
att.put.ncdf(nc, var_pcp, "level", paste("A", acc_hr, sep=''))
att.put.ncdf(nc, var_pcp, "grib_code", 61)
att.put.ncdf(nc, var_pcp, "_FillValue", missing, prec="single")
att.put.ncdf(nc, var_pcp, "init_time", format(init, "%Y%m%d_%H%M%S",
tz="GMT"), prec="text")
att.put.ncdf(nc, var_pcp, "init_time_ut", as.numeric(init),
prec="double")
att.put.ncdf(nc, var_pcp, "valid_time", format(valid, "%Y%m%d_%H%M%S",
tz="GMT"), prec="text")
att.put.ncdf(nc, var_pcp, "valid_time_ut", as.numeric(valid),
prec="double")
att.put.ncdf(nc, var_pcp, "accum_time", paste(acc_str, "0000",
sep=''))
att.put.ncdf(nc, var_pcp, "accum_time_sec", acc_sec)
# Add global attributes
cur_time = Sys.time()
att.put.ncdf(nc, 0, "FileOrigins", paste("File", nc_file, "generated",
format(Sys.time(), "%Y%m%d_%H%M%S"),
"on host", Sys.info()[4], "by
the Rscript trmm2nc.R"))
att.put.ncdf(nc, 0, "MET_version", "V3.0.1")
att.put.ncdf(nc, 0, "Projection", "LatLon", prec="text")
att.put.ncdf(nc, 0, "lat_ll", paste(min(lat), "degrees_north"),
prec="text")
att.put.ncdf(nc, 0, "lon_ll", paste(min(lon), "degrees_east"),
prec="text")
att.put.ncdf(nc, 0, "delta_lat", paste(dlat, "degrees"), prec="text")
att.put.ncdf(nc, 0, "delta_lon", paste(dlon, "degrees"), prec="text")
att.put.ncdf(nc, 0, "Nlat", paste(nlat, "grid_points"), prec="text")
att.put.ncdf(nc, 0, "Nlon", paste(nlon, "grid_points"), prec="text")
# Add pcp to the file
put.var.ncdf(nc, var_pcp, pcp)
# Close the file
close.ncdf(nc)
cat(paste("Writing:", nc_file, "\n"))
########################################################################
#
# Finished.
#
########################################################################
# Optionally, save all of the pcp to an .RData file
if(save == TRUE) save.image()
------------------------------------------------
More information about the Met_help
mailing list