[Met_help] [rt.rap.ucar.edu #85603] History for PCP Slowness
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:03:31 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John, I am running PCP combine in MET 6.1 to sum WRF 01hr precip forecasts. My command line is below:
/h/WXQC/met-6.1/bin/pcp_combine -sum 20180618_000000 01 20180618_120000 12 /h/data/global/WXQC/data/met/tmp/precip_sum -pcpdir /h/data/global/WXQC/data/wrf/t47 -pcprx qc_180618* -v 2
It seems to me that the -pcprx term is not working. If I run the summation using files in the t47 directory that contains several days of WRF data, the summation is very slow even though I am telling MET (I believe) to only look at the 20180618 files. If I place just the 20180618 files in a subdirectory leaving everything else as is, the summation goes much faster. I was hoping I wouldn't have to move files in to a temporary directory to speed up the processing. The files are listed below:
-rw-rw-r-- 1 qcteam qcteam 29759004 Jun 19 14:31 qc_180618_us057g1011t47b000000000
-rw-rw-r-- 1 qcteam qcteam 32304984 Jun 19 14:31 qc_180618_us057g1011t47b000000100
-rw-rw-r-- 1 qcteam qcteam 32802972 Jun 19 14:31 qc_180618_us057g1011t47b000000200
-rw-rw-r-- 1 qcteam qcteam 32996634 Jun 19 14:31 qc_180618_us057g1011t47b000000300
-rw-rw-r-- 1 qcteam qcteam 33273294 Jun 19 14:31 qc_180618_us057g1011t47b000000400
-rw-rw-r-- 1 qcteam qcteam 33715950 Jun 19 14:31 qc_180618_us057g1011t47b000000500
-rw-rw-r-- 1 qcteam qcteam 34213938 Jun 19 14:31 qc_180618_us057g1011t47b000000600
-rw-rw-r-- 1 qcteam qcteam 34213938 Jun 19 14:31 qc_180618_us057g1011t47b000000700
-rw-rw-r-- 1 qcteam qcteam 34269270 Jun 19 14:31 qc_180618_us057g1011t47b000000800
-rw-rw-r-- 1 qcteam qcteam 34213938 Jun 19 14:31 qc_180618_us057g1011t47b000000900
-rw-rw-r-- 1 qcteam qcteam 34324602 Jun 19 14:31 qc_180618_us057g1011t47b000001000
-rw-rw-r-- 1 qcteam qcteam 34628928 Jun 19 14:31 qc_180618_us057g1011t47b000001100
-rw-rw-r-- 1 qcteam qcteam 34462932 Jun 19 14:31 qc_180618_us057g1011t47b000001200
-rw-rw-r-- 1 qcteam qcteam 34352268 Jun 19 14:31 qc_180618_us057g1011t47b000001300
-rw-rw-r-- 1 qcteam qcteam 34269270 Jun 19 14:31 qc_180618_us057g1011t47b000001400
-rw-rw-r-- 1 qcteam qcteam 34545930 Jun 19 14:31 qc_180618_us057g1011t47b000001500
-rw-rw-r-- 1 qcteam qcteam 34711926 Jun 19 14:31 qc_180618_us057g1011t47b000001600
-rw-rw-r-- 1 qcteam qcteam 34711926 Jun 19 14:31 qc_180618_us057g1011t47b000001700
-rw-rw-r-- 1 qcteam qcteam 34739592 Jun 19 14:31 qc_180618_us057g1011t47b000001800
-rw-rw-r-- 1 qcteam qcteam 34518264 Jun 19 14:31 qc_180618_us057g1011t47b000001900
-rw-rw-r-- 1 qcteam qcteam 34407600 Jun 19 14:31 qc_180618_us057g1011t47b000002000
-rw-rw-r-- 1 qcteam qcteam 34573596 Jun 19 14:31 qc_180618_us057g1011t47b000002100
-rw-rw-r-- 1 qcteam qcteam 34435266 Jun 19 14:31 qc_180618_us057g1011t47b000002200
-rw-rw-r-- 1 qcteam qcteam 34407600 Jun 19 14:31 qc_180618_us057g1011t47b000002300
-rw-rw-r-- 1 qcteam qcteam 34324602 Jun 19 14:31 qc_180618_us057g1011t47b000002400
-rw-rw-r-- 1 qcteam qcteam 34269270 Jun 19 14:31 qc_180618_us057g1011t47b000002500
-rw-rw-r-- 1 qcteam qcteam 34462932 Jun 19 14:31 qc_180618_us057g1011t47b000002600
Am I coding the -pcprx field correctly?
Thanks
Bob
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: PCP Slowness
From: John Halley Gotway
Time: Wed Jun 20 10:35:24 2018
Bob,
I think you should remove the '*' from the -pcprx option and that
should
fix it. Regular expressions can get very tricky. I actually think
that if
you cut-and-paste the exact command you sent on the command line,
you'd get
an error from pcp_combine. You'd need to enclose it in single quotes
to
prevent command line expansion of the star. So please try running the
following commands on the command line:
time /h/WXQC/met-6.1/bin/pcp_combine -sum 20180618_000000 01
20180618_120000 12 /h/data/global/WXQC/data/met/tmp/precip_sum -pcpdir
/h/data/global/WXQC/data/wrf/t47 -pcprx 'qc_180618*' -v 2
time /h/WXQC/met-6.1/bin/pcp_combine -sum 20180618_000000 01
20180618_120000 12 /h/data/global/WXQC/data/met/tmp/precip_sum -pcpdir
/h/data/global/WXQC/data/wrf/t47 -pcprx 'qc_180618' -v 2
The first uses the '*' and encloses the regular expression in single
quotes... the second gets rid of the star. If all goes well, the
second
command should run much faster than the first because we're
successfully
skipping the files which don't match. Please let me know how it goes.
Thanks,
John
On Tue, Jun 19, 2018 at 1:12 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Tue Jun 19 13:11:30 2018: Request 85603 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: PCP Slowness
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=85603 >
>
>
> John, I am running PCP combine in MET 6.1 to sum WRF 01hr precip
> forecasts. My command line is below:
>
> /h/WXQC/met-6.1/bin/pcp_combine -sum 20180618_000000 01
20180618_120000
> 12 /h/data/global/WXQC/data/met/tmp/precip_sum -pcpdir
> /h/data/global/WXQC/data/wrf/t47 -pcprx qc_180618* -v 2
>
> It seems to me that the -pcprx term is not working. If I run the
> summation using files in the t47 directory that contains several
days of
> WRF data, the summation is very slow even though I am telling MET (I
> believe) to only look at the 20180618 files. If I place just the
20180618
> files in a subdirectory leaving everything else as is, the summation
goes
> much faster. I was hoping I wouldn't have to move files in to a
> temporary directory to speed up the processing. The files are
listed below:
>
> -rw-rw-r-- 1 qcteam qcteam 29759004 Jun 19 14:31
> qc_180618_us057g1011t47b000000000
> -rw-rw-r-- 1 qcteam qcteam 32304984 Jun 19 14:31
> qc_180618_us057g1011t47b000000100
> -rw-rw-r-- 1 qcteam qcteam 32802972 Jun 19 14:31
> qc_180618_us057g1011t47b000000200
> -rw-rw-r-- 1 qcteam qcteam 32996634 Jun 19 14:31
> qc_180618_us057g1011t47b000000300
> -rw-rw-r-- 1 qcteam qcteam 33273294 Jun 19 14:31
> qc_180618_us057g1011t47b000000400
> -rw-rw-r-- 1 qcteam qcteam 33715950 Jun 19 14:31
> qc_180618_us057g1011t47b000000500
> -rw-rw-r-- 1 qcteam qcteam 34213938 Jun 19 14:31
> qc_180618_us057g1011t47b000000600
> -rw-rw-r-- 1 qcteam qcteam 34213938 Jun 19 14:31
> qc_180618_us057g1011t47b000000700
> -rw-rw-r-- 1 qcteam qcteam 34269270 Jun 19 14:31
> qc_180618_us057g1011t47b000000800
> -rw-rw-r-- 1 qcteam qcteam 34213938 Jun 19 14:31
> qc_180618_us057g1011t47b000000900
> -rw-rw-r-- 1 qcteam qcteam 34324602 Jun 19 14:31
> qc_180618_us057g1011t47b000001000
> -rw-rw-r-- 1 qcteam qcteam 34628928 Jun 19 14:31
> qc_180618_us057g1011t47b000001100
> -rw-rw-r-- 1 qcteam qcteam 34462932 Jun 19 14:31
> qc_180618_us057g1011t47b000001200
> -rw-rw-r-- 1 qcteam qcteam 34352268 Jun 19 14:31
> qc_180618_us057g1011t47b000001300
> -rw-rw-r-- 1 qcteam qcteam 34269270 Jun 19 14:31
> qc_180618_us057g1011t47b000001400
> -rw-rw-r-- 1 qcteam qcteam 34545930 Jun 19 14:31
> qc_180618_us057g1011t47b000001500
> -rw-rw-r-- 1 qcteam qcteam 34711926 Jun 19 14:31
> qc_180618_us057g1011t47b000001600
> -rw-rw-r-- 1 qcteam qcteam 34711926 Jun 19 14:31
> qc_180618_us057g1011t47b000001700
> -rw-rw-r-- 1 qcteam qcteam 34739592 Jun 19 14:31
> qc_180618_us057g1011t47b000001800
> -rw-rw-r-- 1 qcteam qcteam 34518264 Jun 19 14:31
> qc_180618_us057g1011t47b000001900
> -rw-rw-r-- 1 qcteam qcteam 34407600 Jun 19 14:31
> qc_180618_us057g1011t47b000002000
> -rw-rw-r-- 1 qcteam qcteam 34573596 Jun 19 14:31
> qc_180618_us057g1011t47b000002100
> -rw-rw-r-- 1 qcteam qcteam 34435266 Jun 19 14:31
> qc_180618_us057g1011t47b000002200
> -rw-rw-r-- 1 qcteam qcteam 34407600 Jun 19 14:31
> qc_180618_us057g1011t47b000002300
> -rw-rw-r-- 1 qcteam qcteam 34324602 Jun 19 14:31
> qc_180618_us057g1011t47b000002400
> -rw-rw-r-- 1 qcteam qcteam 34269270 Jun 19 14:31
> qc_180618_us057g1011t47b000002500
> -rw-rw-r-- 1 qcteam qcteam 34462932 Jun 19 14:31
> qc_180618_us057g1011t47b000002600
>
> Am I coding the -pcprx field correctly?
>
> Thanks
> Bob
>
>
>
------------------------------------------------
Subject: PCP Slowness
From: John Halley Gotway
Time: Wed Jun 20 10:58:38 2018
Bob,
Just to follow up as to why that '*' character matters so much. On
the
command line, we're used to using '*' as a wildcard, meaning match
anything. In regular expressions, the '*' has a very specific
meaning. It
means match "0 or more of the preceding character". So using the star
at
the end of -pcprx 'qc_180618*' means that there can be zero or more
8's.
So it'll match all the files that show up by running this 'ls'
command:
ls /h/data/global/WXQC/data/wrf/t47/qc_18061*
There are some useful online utilities for checking regular
expressions
that are pretty useful:
https://regex101.com/
But there still is confusion because there are 'regular' regular
expressions... and then 'extended' regular expressions with slightly
different notation and rules! While they're very powerful and
convenient,
they can be very tricky.
Thanks,
John
On Wed, Jun 20, 2018 at 10:35 AM The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Wed Jun 20 10:35:24 2018: Request 85603 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: PCP Slowness
> Owner: johnhg
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=85603 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
More information about the Met_help
mailing list