[Met_help] [rt.rap.ucar.edu #53771] History for Problem with Point_stat and APCP time intervals in GRIB1

Paul Oldenburg via RT met_help at ucar.edu
Mon Feb 20 12:33:29 MST 2012 

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Dear MetHelp,

I am using MET tools for an ensemble forecast verification. Each 
ensemble member is an execution instance of WRF 3.0 with 96 h leap time 
and 12 km horizontal resolution. My present task is about accumulated 
precipitation on 6, 12 and 24 hours.

In order to verify each ensemble member I dispose of accumulated 
observation data on 1h, 6h, 12h and 24h. Observation files are ASCII 
formatted as expected. These files are inputs for ascii2nc MET tool, so 
NetCDF observation files are available as output. Forecast files are 
GRIB1 and contain accumulated variable APCP on times: 0-1hr, 0-2hr, 
0-3hr, ... , 0-96hr, i.e.:
534:17771418:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=1:TimeU=1:sfc:0-1hr 
acc:NAve=0
811:27601724:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=2:TimeU=1:sfc:0-2hr 
acc:NAve=0
...

I know it is possible to transform these accumulated times using GRIB2 
and wgrib2 tool (ncep_norm parameter) but it is not possible with GRIB1 
and wgrib. In addition MET tools don't accept GRIB2 files as input. So 
in order to transform from '0-n hr' to '(n-1)-n hr' accumulated times, I 
extract each period from GRIB1 file and subtract using pcp_combine.

For APCP24 (as an instance) I extract from original GRIB1 file:

6905:248457188:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-24hr 
acc:NAve=0
13553:488793736:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=48:TimeU=1:sfc:0-48hr 
acc:NAve=0
20201:731549400:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=72:TimeU=1:sfc:0-72hr 
acc:NAve=0
26849:971441062:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=96:TimeU=1:sfc:0-96hr 
acc:NAve=0

to four individual GRIB1 files and subtract each one:
$ pcp_combine -subtract wrfprs_d02_m00_2011102600_APCP_0-48hr_acc.grib 
48 wrfprs_d02_m00_2011102600_APCP_0-24hr_acc.grib 24 
wrfprs_d02_m00_2011102600_APCP24_24-48hr_acc.nc

Reading input file: wrfprs_d02_m00_2011102600_APCP_0-48hr_acc.grib
Reading input file: wrfprs_d02_m00_2011102600_APCP_0-24hr_acc.grib
Performing subtraction command.
Writing output file: wrfprs_d02_m00_2011102600_APCP24_24-48hr_acc.nc

$ pcp_combine -subtract wrfprs_d02_m00_2011102600_APCP_0-72hr_acc.grib 
72 wrfprs_d02_m00_2011102600_APCP_0-48hr_acc.grib 48 
wrfprs_d02_m00_2011102600_APCP24_48-72hr_acc.nc
...
and so on.

This approach produces four NetCDF files which will be processed by 
point_stat, together with observation file. So I do execute point_stat 
four times and generate four different configuration files. Adding 
APCP12 and APCP06 variables (23 new files) for each ensemble member 
imply too many files and to much expent time on I/O.

MET tools are really quick computing statistics but this approach expend 
too much time reading and writing files, so my question: Is there any 
tool to join again all files into one, in order to minimize read/write time?

I tried to include a temporal dimension on NetCDF, using CDO tools and 
declaring 'fcst_field[] = [ "APCP_24(24,*,*) APCP_24(48,*,*) 
APCP_24(72,*,*) APCP_24(96,*,*)" ];' in point_stat configuration file, 
but it does not work.

Another idea I have had is creating a NetCDF forecast file from scratch, 
similar to pb2nc or ascii2nc output files and including (or not) an 
'initialization time' variable. I have not tried yet.

Any clue? Thanks in advance.


Víctor M. Álvarez
Numerical Department - MeteoGalicia
Consellería de Medio Ambiente, Territorio e Infraestruturas - Xunta de 
Galicia
Santiago de Compostela - Spain



----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #53771] Problem with Point_stat and APCP time intervals in GRIB1
From: Paul Oldenburg
Time: Thu Feb 16 09:33:59 2012

Victor,

There is no MET tool to join pcp_combine NetCDF output files together.
If you want to perform this process, you will
have to create such a tool yourself.  However, I am skeptical that
combining the inputs to point_stat will speed up or
simplify your process.  Instead, you may want to consider using
environment variables inside of your point_stat config.
  Even though you will have to call point_stat once for each input
NetCDF file, you can use a single config file with a
parameterized value in the fcst_field and obs_field, if necessary:

fcst_field[] = [ "${PS_APCP_FIELD}" ];

Before you call point_stat, remember to set the environment variable
PS_APCP_FIELD.

If you are interested in manually combining the pcp_combine NetCDF
output files, try putting multiple 2d vars in a
single NetCDF file and then give them different variable names.  For
example:

APCP_24_f24(lat,lon)
APCP_24_f48(lat,lon)
APCP_24_f72(lat,lon)
APCP_24_f96(lat,lon)
APCP_12_f12(lat,lon)
...
APCP_12_f96(lat,lon)
APCP_06_f06(lat,lon)
...
APCP_06_f96(lat,lon)

Make sure that the timing information stored as variable attributes is
correct so point_stat will know how to handle the
records.  Again, we are not sure if doing this will actually speed up
your process or shrink the size of the data on
your disk. Verifying the multitude of files will require separate
calls to point_stat, but one config file can be used
for many calls.  I hope my explanation makes sense.  Please let me
know if you have any questions.

Paul



On 02/16/2012 04:44 AM, Victor M. Alvarez via RT wrote:
>
> Thu Feb 16 04:44:45 2012: Request 53771 was acted upon.
> Transaction: Ticket created by victor.alvarez at meteogalicia.es
>         Queue: met_help
>       Subject: Problem with Point_stat and APCP time intervals in
GRIB1
>         Owner: Nobody
>    Requestors: victor.alvarez at meteogalicia.es
>        Status: new
>   Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=53771>
>
>
> Dear MetHelp,
>
> I am using MET tools for an ensemble forecast verification. Each
> ensemble member is an execution instance of WRF 3.0 with 96 h leap
time
> and 12 km horizontal resolution. My present task is about
accumulated
> precipitation on 6, 12 and 24 hours.
>
> In order to verify each ensemble member I dispose of accumulated
> observation data on 1h, 6h, 12h and 24h. Observation files are ASCII
> formatted as expected. These files are inputs for ascii2nc MET tool,
so
> NetCDF observation files are available as output. Forecast files are
> GRIB1 and contain accumulated variable APCP on times: 0-1hr, 0-2hr,
> 0-3hr, ... , 0-96hr, i.e.:
>
534:17771418:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=1:TimeU=1:sfc:0-
1hr
> acc:NAve=0
>
811:27601724:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=2:TimeU=1:sfc:0-
2hr
> acc:NAve=0
> ...
>
> I know it is possible to transform these accumulated times using
GRIB2
> and wgrib2 tool (ncep_norm parameter) but it is not possible with
GRIB1
> and wgrib. In addition MET tools don't accept GRIB2 files as input.
So
> in order to transform from '0-n hr' to '(n-1)-n hr' accumulated
times, I
> extract each period from GRIB1 file and subtract using pcp_combine.
>
> For APCP24 (as an instance) I extract from original GRIB1 file:
>
>
6905:248457188:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=24:TimeU=1:sfc:0-
24hr
> acc:NAve=0
>
13553:488793736:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=48:TimeU=1:sfc:0-
48hr
> acc:NAve=0
>
20201:731549400:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=72:TimeU=1:sfc:0-
72hr
> acc:NAve=0
>
26849:971441062:d=11102600:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=96:TimeU=1:sfc:0-
96hr
> acc:NAve=0
>
> to four individual GRIB1 files and subtract each one:
> $ pcp_combine -subtract wrfprs_d02_m00_2011102600_APCP_0-
48hr_acc.grib
> 48 wrfprs_d02_m00_2011102600_APCP_0-24hr_acc.grib 24
> wrfprs_d02_m00_2011102600_APCP24_24-48hr_acc.nc
>
> Reading input file: wrfprs_d02_m00_2011102600_APCP_0-48hr_acc.grib
> Reading input file: wrfprs_d02_m00_2011102600_APCP_0-24hr_acc.grib
> Performing subtraction command.
> Writing output file: wrfprs_d02_m00_2011102600_APCP24_24-48hr_acc.nc
>
> $ pcp_combine -subtract wrfprs_d02_m00_2011102600_APCP_0-
72hr_acc.grib
> 72 wrfprs_d02_m00_2011102600_APCP_0-48hr_acc.grib 48
> wrfprs_d02_m00_2011102600_APCP24_48-72hr_acc.nc
> ...
> and so on.
>
> This approach produces four NetCDF files which will be processed by
> point_stat, together with observation file. So I do execute
point_stat
> four times and generate four different configuration files. Adding
> APCP12 and APCP06 variables (23 new files) for each ensemble member
> imply too many files and to much expent time on I/O.
>
> MET tools are really quick computing statistics but this approach
expend
> too much time reading and writing files, so my question: Is there
any
> tool to join again all files into one, in order to minimize
read/write time?
>
> I tried to include a temporal dimension on NetCDF, using CDO tools
and
> declaring 'fcst_field[] = [ "APCP_24(24,*,*) APCP_24(48,*,*)
> APCP_24(72,*,*) APCP_24(96,*,*)" ];' in point_stat configuration
file,
> but it does not work.
>
> Another idea I have had is creating a NetCDF forecast file from
scratch,
> similar to pb2nc or ascii2nc output files and including (or not) an
> 'initialization time' variable. I have not tried yet.
>
> Any clue? Thanks in advance.
>
>
> Víctor M. Álvarez
> Numerical Department - MeteoGalicia
> Consellería de Medio Ambiente, Territorio e Infraestruturas - Xunta
de
> Galicia
> Santiago de Compostela - Spain
>


------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #53771] Problem with Point_stat and APCP time intervals in GRIB1
From: Victor M. Alvarez
Time: Fri Feb 17 04:38:07 2012

Paul,

thanks for your help. Off course your explanation make sense.

Environmental variable is a good idea though it doesn't avoid running
point_stat multiple times. I'll try putting together multiple
variables
into a single NetCDF. Such approach forces to parse output stat files
in
order to change again variable names to APCP_24. So I'm going to edit
my
scripts and test times.

Thanks again.


Víctor M. Álvarez
Numerical Department - MeteoGalicia
Consellería de Medio Ambiente, Territorio e Infraestruturas - Xunta de
Galicia
Santiago de Compostela - Spain



Paul Oldenburg via RT escribió:
> Victor,
>
> There is no MET tool to join pcp_combine NetCDF output files
together.  If you want to perform this process, you will
> have to create such a tool yourself.  However, I am skeptical that
combining the inputs to point_stat will speed up or
> simplify your process.  Instead, you may want to consider using
environment variables inside of your point_stat config.
>   Even though you will have to call point_stat once for each input
NetCDF file, you can use a single config file with a
> parameterized value in the fcst_field and obs_field, if necessary:
>
> fcst_field[] = [ "${PS_APCP_FIELD}" ];
>
> Before you call point_stat, remember to set the environment variable
PS_APCP_FIELD.
>
> If you are interested in manually combining the pcp_combine NetCDF
output files, try putting multiple 2d vars in a
> single NetCDF file and then give them different variable names.  For
example:
>
> APCP_24_f24(lat,lon)
> APCP_24_f48(lat,lon)
> APCP_24_f72(lat,lon)
> APCP_24_f96(lat,lon)
> APCP_12_f12(lat,lon)
> ...
> APCP_12_f96(lat,lon)
> APCP_06_f06(lat,lon)
> ...
> APCP_06_f96(lat,lon)
>
> Make sure that the timing information stored as variable attributes
is correct so point_stat will know how to handle the
> records.  Again, we are not sure if doing this will actually speed
up your process or shrink the size of the data on
> your disk. Verifying the multitude of files will require separate
calls to point_stat, but one config file can be used
> for many calls.  I hope my explanation makes sense.  Please let me
know if you have any questions.
>
> Paul
>
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #53771] Problem with Point_stat and APCP time  intervals in GRIB1
From: John Halley Gotway
Time: Fri Feb 17 09:36:56 2012

Victor,

Just want to jump in here quickly with a suggestion.

You mentioned the need to modify the STAT file output from Point-Stat
to
change the name for example from "APCP_24_f24" back to "APCP_24".
We've
added some logic in METv3.1 that will make this step unnecessary.  We
added the logic mostly to simplify handling the output of the
Ensemble-Stat tool, but it applies in this situation as well.

As long as your NetCDF variables include a "name" attribute, that
value
will be used by Point-Stat and written out in the "fcst_var" output
column.  For example, for the "APCP_24_f24" variable, just make sure
the
"name" variable attribute it set to "APCP_24" and that should be what
appears in the output from Point-Stat.

Hope that helps.

Thanks,
John

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=53771 >
>
> Paul,
>
> thanks for your help. Off course your explanation make sense.
>
> Environmental variable is a good idea though it doesn't avoid
running
> point_stat multiple times. I'll try putting together multiple
variables
> into a single NetCDF. Such approach forces to parse output stat
files in
> order to change again variable names to APCP_24. So I'm going to
edit my
> scripts and test times.
>
> Thanks again.
>
>
> Víctor M. Álvarez
> Numerical Department - MeteoGalicia
> Consellería de Medio Ambiente, Territorio e Infraestruturas - Xunta
de
> Galicia
> Santiago de Compostela - Spain
>
>
>
> Paul Oldenburg via RT escribió:
>> Victor,
>>
>> There is no MET tool to join pcp_combine NetCDF output files
together.
>> If you want to perform this process, you will
>> have to create such a tool yourself.  However, I am skeptical that
>> combining the inputs to point_stat will speed up or
>> simplify your process.  Instead, you may want to consider using
>> environment variables inside of your point_stat config.
>>   Even though you will have to call point_stat once for each input
>> NetCDF file, you can use a single config file with a
>> parameterized value in the fcst_field and obs_field, if necessary:
>>
>> fcst_field[] = [ "${PS_APCP_FIELD}" ];
>>
>> Before you call point_stat, remember to set the environment
variable
>> PS_APCP_FIELD.
>>
>> If you are interested in manually combining the pcp_combine NetCDF
>> output files, try putting multiple 2d vars in a
>> single NetCDF file and then give them different variable names.
For
>> example:
>>
>> APCP_24_f24(lat,lon)
>> APCP_24_f48(lat,lon)
>> APCP_24_f72(lat,lon)
>> APCP_24_f96(lat,lon)
>> APCP_12_f12(lat,lon)
>> ...
>> APCP_12_f96(lat,lon)
>> APCP_06_f06(lat,lon)
>> ...
>> APCP_06_f96(lat,lon)
>>
>> Make sure that the timing information stored as variable attributes
is
>> correct so point_stat will know how to handle the
>> records.  Again, we are not sure if doing this will actually speed
up
>> your process or shrink the size of the data on
>> your disk. Verifying the multitude of files will require separate
calls
>> to point_stat, but one config file can be used
>> for many calls.  I hope my explanation makes sense.  Please let me
know
>> if you have any questions.
>>
>> Paul
>>
>>
>



------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #53771] Problem with Point_stat and APCP time intervals in GRIB1
From: Victor M. Alvarez
Time: Mon Feb 20 01:52:37 2012

Thanks John for your suggestion.

This added logic is a good reason to install MET v3.1. It simplifies
my
scripts and also unifies treatment of inputs files to point_stat.
Particularly APCP_ENS_MEAN variable from ensemble_stat.

Thanks again for your valuable suggestions.


Víctor M. Álvarez
Numerical Department - MeteoGalicia
Consellería de Medio Ambiente, Territorio e Infraestruturas - Xunta de
Galicia
Santiago de Compostela - Spain


John Halley Gotway via RT escribió:
> Victor,
>
> Just want to jump in here quickly with a suggestion.
>
> You mentioned the need to modify the STAT file output from Point-
Stat to
> change the name for example from "APCP_24_f24" back to "APCP_24".
We've
> added some logic in METv3.1 that will make this step unnecessary.
We
> added the logic mostly to simplify handling the output of the
> Ensemble-Stat tool, but it applies in this situation as well.
>
> As long as your NetCDF variables include a "name" attribute, that
value
> will be used by Point-Stat and written out in the "fcst_var" output
> column.  For example, for the "APCP_24_f24" variable, just make sure
the
> "name" variable attribute it set to "APCP_24" and that should be
what
> appears in the output from Point-Stat.
>
> Hope that helps.
>
> Thanks,
> John

------------------------------------------------

More information about the Met_help mailing list