[Met_help] [rt.rap.ucar.edu #78971] History for using pcp combine on GFS grib files
John Halley Gotway via RT
met_help at ucar.edu
Wed Jan 18 09:54:56 MST 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear MET help,
I am trying to use pcp_combine on GFS output files, with file naming
convention of pgbqFFF.gfs.YYYYMMDDHH, where FFF is the forecast hour. The
GFS output is 6-hr accumulation, but I would like to generate a netcdf file
with 24 hour accumulations for each gfs run, where YYYYMMDDHH specifies the
initialization date of the GFS.
As for observations, I have 24 hour accumulated precipitation in this file
format, already processed from NCAR.
ST4.YYYYMMDDHH.24h
I've looked up the syntax of pcp_combine but its hard to depict how to do
it. Would anyone be able to help me? My end goal is to then run this
through MET MODE to look at equitable threat scores for various precip
thresholds per 24 hours to see the forecast impact of two GFS experiments.
Thank you so much,
--
Andrew Kren, PhD
Research Scientist I, Global Observing Systems Analysis (GOSA) Group
NOAA ESRL Global Systems Division (Rm 3C515)
325 Broadway, Boulder, CO 80305
(303) 497-5418
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: using pcp combine on GFS grib files
From: John Halley Gotway
Time: Thu Dec 22 13:53:33 2016
Hello Andrew,
I apologize for the delay in answering your question. Many of our
staff
are out of the office over the holidays.
I see that you have questions about running GFS data through the
pcp_combine tool. You'd like to use pcp_combine to put GFS into 24-
hour
intervals for comparison against 24-hourly StageIV precipitation.
However, be aware that the 24-hour StageIV data is defined as an
accumulation from 12Z on one day to 12Z on the next day:
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpanl/stage4/
Therefore, you can only use the 24-hour StageIV data to evaluate 12Z
to 12Z
accumulations from your model. Alternatively, you could use the 6-
hour
StageIV accumulations to evaluate any 24 hour accumulation from your
model. For the latter, you'd just need to run the 6-hour StageIV
files
through pcp_combine to generate your desired 24-hour accumulation.
Let me go through an example. Run pcp_combine to compute 24-hour
accumulations for GFS. Let's suppose I'm processing the 20150220 00Z
initialization of GFS.
met-5.2/bin/pcp_combine \
-sum 20150220_00 06 20150221_00 24 \
gfs_APCP_24_20150220_00_F00_F24.nc \
-pcprx "gfs_4_20150220_00.*grb2" \
-pcpdir /d1/SBU/GFS/model_data/20150220
Here I'm telling pcp_combine to look in the
/d1/SBU/GFS/model_data/20150220
directory at files which match this regular expression
"gfs_4_20150220_00.*grb2". That directory contains data for 00, 06,
12,
and 18 hour initializations, but the "-pcprx" option narrows the
search
down to the 00 hour initialization which makes it run faster. It
inspects
all the matching files, looking for 6-hour APCP data to sum up to a
24-hour
accumulation valid at 20150221_00. This results in a 24-hour
accumulation
between forecast hours 0 and 24.
The following command will compute the 24-hour accumulation between
forecast hours 12 and 36:
met-5.2/bin/pcp_combine \
-sum 20150220_00 06 20150221_12 24 \
gfs_APCP_24_20150220_00_F12_F36.nc \
-pcprx "gfs_4_20150220_00.*grb2" \
-pcpdir /d1/SBU/GFS/model_data/20150220
The "-sum" command is meant to make things easier by searching your
directory for you. But instead of using "-sum", you could use the "-
add"
command and just explicitly list the 4 files from which you'd like to
extract 6-hour APCP and add them up to 24. In my directory structure,
the
previous "-sum" job could be rewritten with "-add" like this:
cd /d1/SBU/GFS/model_data/20150220
met-5.2/bin/pcp_combine -add \
/d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_018.grb2 06 \
/d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_024.grb2 06 \
/d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_030.grb2 06 \
/d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_036.grb2 06 \
gfs_APCP_24_20150220_00_F12_F36_add_option.nc
Here I'm explicitly telling pcp_combine which files to read and what
accumulation interval (6 hours) to extract from them. The resulting
output
should be identical to the output of the "-sum" command.
Hope that helps get you going.
Just let me know what other issues or questions arise.
Thanks,
John Halley Gotway
On Thu, Dec 15, 2016 at 11:35 AM, Andrew Kren - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Thu Dec 15 11:35:19 2016: Request 78971 was acted upon.
> Transaction: Ticket created by andrew.kren at noaa.gov
> Queue: met_help
> Subject: using pcp combine on GFS grib files
> Owner: Nobody
> Requestors: andrew.kren at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78971 >
>
>
> Dear MET help,
>
> I am trying to use pcp_combine on GFS output files, with file naming
> convention of pgbqFFF.gfs.YYYYMMDDHH, where FFF is the forecast
hour. The
> GFS output is 6-hr accumulation, but I would like to generate a
netcdf file
> with 24 hour accumulations for each gfs run, where YYYYMMDDHH
specifies the
> initialization date of the GFS.
>
> As for observations, I have 24 hour accumulated precipitation in
this file
> format, already processed from NCAR.
>
> ST4.YYYYMMDDHH.24h
>
> I've looked up the syntax of pcp_combine but its hard to depict how
to do
> it. Would anyone be able to help me? My end goal is to then run this
> through MET MODE to look at equitable threat scores for various
precip
> thresholds per 24 hours to see the forecast impact of two GFS
experiments.
>
> Thank you so much,
>
> --
> Andrew Kren, PhD
> Research Scientist I, Global Observing Systems Analysis (GOSA) Group
> NOAA ESRL Global Systems Division (Rm 3C515)
> 325 Broadway, Boulder, CO 80305
> (303) 497-5418
>
>
------------------------------------------------
Subject: using pcp combine on GFS grib files
From: Andrew Kren - NOAA Affiliate
Time: Thu Dec 22 15:15:48 2016
Could you show me an example for accumulating the 6hr stage 4 data? Or
would it be the same? Thanks so much!
Andrew Kren
From: John Halley Gotway via RT
Sent: Thursday, December 22, 2016 1:53 PM
To: andrew.kren at noaa.gov
Subject: Re: [rt.rap.ucar.edu #78971] using pcp combine on GFS grib
files
Hello Andrew,
I apologize for the delay in answering your question. Many of our
staff
are out of the office over the holidays.
I see that you have questions about running GFS data through the
pcp_combine tool. You'd like to use pcp_combine to put GFS into 24-
hour
intervals for comparison against 24-hourly StageIV precipitation.
However, be aware that the 24-hour StageIV data is defined as an
accumulation from 12Z on one day to 12Z on the next day:
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpanl/stage4/
Therefore, you can only use the 24-hour StageIV data to evaluate 12Z
to 12Z
accumulations from your model. Alternatively, you could use the 6-
hour
StageIV accumulations to evaluate any 24 hour accumulation from your
model. For the latter, you'd just need to run the 6-hour StageIV
files
through pcp_combine to generate your desired 24-hour accumulation.
Let me go through an example. Run pcp_combine to compute 24-hour
accumulations for GFS. Let's suppose I'm processing the 20150220 00Z
initialization of GFS.
met-5.2/bin/pcp_combine \
-sum 20150220_00 06 20150221_00 24 \
gfs_APCP_24_20150220_00_F00_F24.nc \
-pcprx "gfs_4_20150220_00.*grb2" \
-pcpdir /d1/SBU/GFS/model_data/20150220
Here I'm telling pcp_combine to look in the
/d1/SBU/GFS/model_data/20150220
directory at files which match this regular expression
"gfs_4_20150220_00.*grb2". That directory contains data for 00, 06,
12,
and 18 hour initializations, but the "-pcprx" option narrows the
search
down to the 00 hour initialization which makes it run faster. It
inspects
all the matching files, looking for 6-hour APCP data to sum up to a
24-hour
accumulation valid at 20150221_00. This results in a 24-hour
accumulation
between forecast hours 0 and 24.
The following command will compute the 24-hour accumulation between
forecast hours 12 and 36:
met-5.2/bin/pcp_combine \
-sum 20150220_00 06 20150221_12 24 \
gfs_APCP_24_20150220_00_F12_F36.nc \
-pcprx "gfs_4_20150220_00.*grb2" \
-pcpdir /d1/SBU/GFS/model_data/20150220
The "-sum" command is meant to make things easier by searching your
directory for you. But instead of using "-sum", you could use the "-
add"
command and just explicitly list the 4 files from which you'd like to
extract 6-hour APCP and add them up to 24. In my directory structure,
the
previous "-sum" job could be rewritten with "-add" like this:
cd /d1/SBU/GFS/model_data/20150220
met-5.2/bin/pcp_combine -add \
/d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_018.grb2 06 \
/d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_024.grb2 06 \
/d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_030.grb2 06 \
/d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_036.grb2 06 \
gfs_APCP_24_20150220_00_F12_F36_add_option.nc
Here I'm explicitly telling pcp_combine which files to read and what
accumulation interval (6 hours) to extract from them. The resulting
output
should be identical to the output of the "-sum" command.
Hope that helps get you going.
Just let me know what other issues or questions arise.
Thanks,
John Halley Gotway
On Thu, Dec 15, 2016 at 11:35 AM, Andrew Kren - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Thu Dec 15 11:35:19 2016: Request 78971 was acted upon.
> Transaction: Ticket created by andrew.kren at noaa.gov
> Queue: met_help
> Subject: using pcp combine on GFS grib files
> Owner: Nobody
> Requestors: andrew.kren at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78971 >
>
>
> Dear MET help,
>
> I am trying to use pcp_combine on GFS output files, with file naming
> convention of pgbqFFF.gfs.YYYYMMDDHH, where FFF is the forecast
hour. The
> GFS output is 6-hr accumulation, but I would like to generate a
netcdf file
> with 24 hour accumulations for each gfs run, where YYYYMMDDHH
specifies the
> initialization date of the GFS.
>
> As for observations, I have 24 hour accumulated precipitation in
this file
> format, already processed from NCAR.
>
> ST4.YYYYMMDDHH.24h
>
> I've looked up the syntax of pcp_combine but its hard to depict how
to do
> it. Would anyone be able to help me? My end goal is to then run this
> through MET MODE to look at equitable threat scores for various
precip
> thresholds per 24 hours to see the forecast impact of two GFS
experiments.
>
> Thank you so much,
>
> --
> Andrew Kren, PhD
> Research Scientist I, Global Observing Systems Analysis (GOSA) Group
> NOAA ESRL Global Systems Division (Rm 3C515)
> 325 Broadway, Boulder, CO 80305
> (303) 497-5418
>
>
------------------------------------------------
Subject: using pcp combine on GFS grib files
From: John Halley Gotway
Time: Thu Dec 22 16:11:53 2016
Andrew,
It's basically the same. The only difference is that you specify the
initialization time using all 0's. Usually pcp_combine requires that
the
initialization time stays constant since you generally wouldn't want
to mix
data from different model initializations.
Here's some sample commands to grab some StageIV data, unzip it, and
run
pcp_combine on it:
mkdir test_st4
cd test_st4
wget
ftp://ftp.emc.ncep.noaa.gov/mmb/precip/st2n4.arch/201610/ST4.20161015ST4.20161015
tar -xvf ST4.20161015
gunzip ST4*.06h.gz
# Add 2 files together
met-5.2/bin/pcp_combine -add ST4.2016101512.06h 06 ST4.2016101518.06h
06
ST4.2016101518.APCP_12.nc
# Use the sum command instead
met-5.2/bin/pcp_combine -sum 00000000_000000 06 20161015_18 12
ST4.2016101518.APCP_12_SUM.nc -pcprx "ST4.*.06h"
Thanks,
John
On Thu, Dec 22, 2016 at 3:15 PM, Andrew Kren - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78971 >
>
> Could you show me an example for accumulating the 6hr stage 4 data?
Or
> would it be the same? Thanks so much!
>
> Andrew Kren
>
> From: John Halley Gotway via RT
> Sent: Thursday, December 22, 2016 1:53 PM
> To: andrew.kren at noaa.gov
> Subject: Re: [rt.rap.ucar.edu #78971] using pcp combine on GFS grib
files
>
> Hello Andrew,
>
> I apologize for the delay in answering your question. Many of our
staff
> are out of the office over the holidays.
>
> I see that you have questions about running GFS data through the
> pcp_combine tool. You'd like to use pcp_combine to put GFS into 24-
hour
> intervals for comparison against 24-hourly StageIV precipitation.
>
> However, be aware that the 24-hour StageIV data is defined as an
> accumulation from 12Z on one day to 12Z on the next day:
> http://www.emc.ncep.noaa.gov/mmb/ylin/pcpanl/stage4/
>
> Therefore, you can only use the 24-hour StageIV data to evaluate 12Z
to 12Z
> accumulations from your model. Alternatively, you could use the 6-
hour
> StageIV accumulations to evaluate any 24 hour accumulation from your
> model. For the latter, you'd just need to run the 6-hour StageIV
files
> through pcp_combine to generate your desired 24-hour accumulation.
>
> Let me go through an example. Run pcp_combine to compute 24-hour
> accumulations for GFS. Let's suppose I'm processing the 20150220
00Z
> initialization of GFS.
>
> met-5.2/bin/pcp_combine \
> -sum 20150220_00 06 20150221_00 24 \
> gfs_APCP_24_20150220_00_F00_F24.nc \
> -pcprx "gfs_4_20150220_00.*grb2" \
> -pcpdir /d1/SBU/GFS/model_data/20150220
>
> Here I'm telling pcp_combine to look in the /d1/SBU/GFS/model_data/
> 20150220
> directory at files which match this regular expression
> "gfs_4_20150220_00.*grb2". That directory contains data for 00, 06,
12,
> and 18 hour initializations, but the "-pcprx" option narrows the
search
> down to the 00 hour initialization which makes it run faster. It
inspects
> all the matching files, looking for 6-hour APCP data to sum up to a
24-hour
> accumulation valid at 20150221_00. This results in a 24-hour
accumulation
> between forecast hours 0 and 24.
>
> The following command will compute the 24-hour accumulation between
> forecast hours 12 and 36:
>
> met-5.2/bin/pcp_combine \
> -sum 20150220_00 06 20150221_12 24 \
> gfs_APCP_24_20150220_00_F12_F36.nc \
> -pcprx "gfs_4_20150220_00.*grb2" \
> -pcpdir /d1/SBU/GFS/model_data/20150220
>
> The "-sum" command is meant to make things easier by searching your
> directory for you. But instead of using "-sum", you could use the
"-add"
> command and just explicitly list the 4 files from which you'd like
to
> extract 6-hour APCP and add them up to 24. In my directory
structure, the
> previous "-sum" job could be rewritten with "-add" like this:
>
> cd /d1/SBU/GFS/model_data/20150220
> met-5.2/bin/pcp_combine -add \
> /d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_018.grb2 06
\
> /d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_024.grb2 06
\
> /d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_030.grb2 06
\
> /d1/SBU/GFS/model_data/20150220/gfs_4_20150220_0000_036.grb2 06
\
> gfs_APCP_24_20150220_00_F12_F36_add_option.nc
>
> Here I'm explicitly telling pcp_combine which files to read and what
> accumulation interval (6 hours) to extract from them. The resulting
output
> should be identical to the output of the "-sum" command.
>
> Hope that helps get you going.
>
> Just let me know what other issues or questions arise.
>
> Thanks,
> John Halley Gotway
>
>
> On Thu, Dec 15, 2016 at 11:35 AM, Andrew Kren - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Dec 15 11:35:19 2016: Request 78971 was acted upon.
> > Transaction: Ticket created by andrew.kren at noaa.gov
> > Queue: met_help
> > Subject: using pcp combine on GFS grib files
> > Owner: Nobody
> > Requestors: andrew.kren at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78971 >
> >
> >
> > Dear MET help,
> >
> > I am trying to use pcp_combine on GFS output files, with file
naming
> > convention of pgbqFFF.gfs.YYYYMMDDHH, where FFF is the forecast
hour. The
> > GFS output is 6-hr accumulation, but I would like to generate a
netcdf
> file
> > with 24 hour accumulations for each gfs run, where YYYYMMDDHH
specifies
> the
> > initialization date of the GFS.
> >
> > As for observations, I have 24 hour accumulated precipitation in
this
> file
> > format, already processed from NCAR.
> >
> > ST4.YYYYMMDDHH.24h
> >
> > I've looked up the syntax of pcp_combine but its hard to depict
how to do
> > it. Would anyone be able to help me? My end goal is to then run
this
> > through MET MODE to look at equitable threat scores for various
precip
> > thresholds per 24 hours to see the forecast impact of two GFS
> experiments.
> >
> > Thank you so much,
> >
> > --
> > Andrew Kren, PhD
> > Research Scientist I, Global Observing Systems Analysis (GOSA)
Group
> > NOAA ESRL Global Systems Division (Rm 3C515)
> > 325 Broadway, Boulder, CO 80305
> > (303) 497-5418
> >
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list