[Met_help] [rt.rap.ucar.edu #99429] History for How to set time for hourly 12-24 hours data ?
George McCabe via RT
met_help at ucar.edu
Mon Jul 19 09:37:31 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
Could you let me know how to set time on processing hourly Obs. data for
lasting 12 -24hours using "PB2NC", and do hourly verification for lasting
12-24 hours using Point-stat?
such as:
VALID_TIME_FMT = %Y%m%d%H
VALID_BEG = 2015071500
VALID_END = 2015071500
VALID_INCREMENT = 1M
LEAD_SEQ = 0
Thanks!
Zhuming
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Tue Apr 06 16:23:00 2021
Hi Zhuming,
To run the tools hourly you would set the VALID_INCREMENT value for
1H.
Here is an example that runs once per hour for a day from 2015-07-15
0Z to
2017-07-16 0Z:
VALID_TIME_FMT = %Y%m%d%H
VALID_BEG = 2015071500
VALID_END = 2015071600
VALID_INCREMENT = 1H
LEAD_SEQ = 0
Each run time will use the PB2NC_INPUT_TEMPLATE to find files whose
filenames match the time you need.
To read prepBUFR observations for 12 hours, you will want to set
PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range around the
valid
time to process. These values depend on what is contained inside your
prepBUFR files. If you want to use data from the valid time of the
filename
backwards 12 hours, you would set:
PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
For a runtime of 2015071512, this would be substituted as:
valid_beg: 20150715_00
valid_end: 20150715_12
If you want to create an output file for each hour in PB2NC you would
need
to make sure that the hour is specified in the PB2NC_OUTPUT_TEMPLATE.
PointStat has similar config variables to set the valid beg and end
range
to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
If your prepBUFR files are separated out into multiple files that need
to
be read into PB2NC, you can set a file window around the valid time to
allow files to be passed into the app. More information on that
functionality is here:
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
If you are still having trouble getting this configured, you could
send me
the paths of the files you are trying to read and what data is
contained in
them so I can give you suggestions on what to set for the
configuration.
Thanks,
George
On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> Transaction: Given to mccabe (George McCabe) by jpresto
> Queue: met_help
> Subject: How to set time for hourly 12-24 hours data ?
> Owner: mccabe
> Requestors: zhuming at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
>
> This transaction appears to have no content
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Tue Apr 06 17:16:50 2021
Hi George,
Thanks for your reply!
I'm still confused, I need to output hourly obs data to do hourly
verification.
The original GDAS Prepbufr data has 4 files one day, should with 6
hours
obs data in one file.
After "PB2NC", I can see pbs.2015071500.nc (obs time is 00? or 6 times
of
00,01,02,03,04,05?), pbs.2015071506.nc...
my config file : cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
Thanks,
Zhuming
On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> To run the tools hourly you would set the VALID_INCREMENT value for
1H.
> Here is an example that runs once per hour for a day from 2015-07-15
0Z to
> 2017-07-16 0Z:
>
> VALID_TIME_FMT = %Y%m%d%H
>
> VALID_BEG = 2015071500
>
> VALID_END = 2015071600
>
> VALID_INCREMENT = 1H
>
> LEAD_SEQ = 0
>
> Each run time will use the PB2NC_INPUT_TEMPLATE to find files whose
> filenames match the time you need.
>
> To read prepBUFR observations for 12 hours, you will want to set
> PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range around
the valid
> time to process. These values depend on what is contained inside
your
> prepBUFR files. If you want to use data from the valid time of the
filename
> backwards 12 hours, you would set:
>
> PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
>
> For a runtime of 2015071512, this would be substituted as:
>
> valid_beg: 20150715_00
> valid_end: 20150715_12
>
> If you want to create an output file for each hour in PB2NC you
would need
> to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
>
> PointStat has similar config variables to set the valid beg and end
range
> to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
>
> If your prepBUFR files are separated out into multiple files that
need to
> be read into PB2NC, you can set a file window around the valid time
to
> allow files to be passed into the app. More information on that
> functionality is here:
>
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
>
> If you are still having trouble getting this configured, you could
send me
> the paths of the files you are trying to read and what data is
contained in
> them so I can give you suggestions on what to set for the
configuration.
>
> Thanks,
> George
>
>
>
> On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > Transaction: Given to mccabe (George McCabe) by jpresto
> > Queue: met_help
> > Subject: How to set time for hourly 12-24 hours data ?
> > Owner: mccabe
> > Requestors: zhuming at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> >
> > This transaction appears to have no content
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Tue Apr 06 18:34:09 2021
Hi Zhuming,
Thanks for the information. You will need to use the truncate keyword
to
find the files, then set the valid beg/end range to 1 hour. This is
what I
changed from your PB2NC1.conf config file:
*PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
{valid?fmt=%Y%m%d_%H?shift=1H}*
*PB2NC_INPUT_TEMPLATE =
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
My config file is on cheyenne here:
/glade/u/home/mccabe/PB2NC1.conf
I just started running it and the output is being written here:
/glade/u/home/mccabe/data/out/pb2nc1
For PointStat, you will want to loop over the same times. I noticed
you
specify message type in the VAR<n>_OPTIONS, but you do not need to do
this
since there is a POINT_STAT_MESSAGE_TYPE variable that defines this.
This
is what I changed in your PointStat1.conf file:
*VALID_END = 2015071600*
*VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
=POINT_STAT_OBS_VALID_BEG =*
*POINT_STAT_MESSAGE_TYPE = ADPSFC*
*#OBS_VAR1_OPTIONS = message_type = ["ADPSFC"];#OBS_VAR2_OPTIONS =
message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
/glade/scratch/zhuming/metplus4/pb2nc1*
*OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
Or you can view my file in the same directory. I usually try to test
with a
single run time to make sure it is working properly, so I set
VALID_END
back to 2015071500. I ran it and it produced an output file, but there
were
no values besides the header. I checked the log file and saw this:
*DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for observation type
ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
processed = 393373DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected: obs type = 390528DEBUG 3: Rejected: valid time =
0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the grid
=
2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
mismatch
= 259DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected: message
type
= 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3: Rejected: bad
climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
--------------------------------------------------------------------------------DEBUG
2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for observation
type
ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
processed = 393373*
*DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected: obs type
=
382965DEBUG 3: Rejected: valid time = 0DEBUG 3: Rejected: bad obs
value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3: Rejected:
topography = 0DEBUG 3: Rejected: level mismatch = 1168DEBUG 3:
Rejected: quality marker = 0DEBUG 3: Rejected: message type =
0DEBUG 3:
Rejected: masking region = 0DEBUG 3: Rejected: bad fcst value =
0DEBUG 3:
Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo stdev =
0DEBUG 3:
Rejected: duplicates = 0DEBUG 2:DEBUG 2:
--------------------------------------------------------------------------------DEBUG
2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for observation
type
ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
processed = 393373DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected: obs type = 382965DEBUG 3: Rejected: valid time =
0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the grid
=
9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
mismatch
= 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected:
bad
fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected: bad
climo stdev = 0*
*DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
--------------------------------------------------------------------------------DEBUG
2:DEBUG 1: Output file:
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
This describes the reason why each observation was rejected. Some
observations didn't match the message type you requested, some were
not
within the forecast grid, and some did not match the vertical level.
You
will have to make adjustments to your settings so that the data
matches up
properly. Maybe the grid you are using in the PB2NC config file does
not
match the forecast grid and you are missing values. If that is the
case,
you will have to reconfigure and rerun PB2NC to get output on the
correct
gr5id. Maybe the observation points don't line up with the vertical
level
of the model fields. You can use a range of level values for the
observation data to allow data within that range, i.e. Z0-10.
Let me know if you are still having issues getting this to work and I
can
help out.
Thanks,
George
On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> Thanks for your reply!
>
> I'm still confused, I need to output hourly obs data to do hourly
> verification.
> The original GDAS Prepbufr data has 4 files one day, should with 6
hours
> obs data in one file.
>
> After "PB2NC", I can see pbs.2015071500.nc (obs time is 00? or 6
times of
> 00,01,02,03,04,05?), pbs.2015071506.nc...
>
> my config file :
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
>
> Thanks,
> Zhuming
>
> On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > To run the tools hourly you would set the VALID_INCREMENT value
for 1H.
> > Here is an example that runs once per hour for a day from 2015-07-
15 0Z
> to
> > 2017-07-16 0Z:
> >
> > VALID_TIME_FMT = %Y%m%d%H
> >
> > VALID_BEG = 2015071500
> >
> > VALID_END = 2015071600
> >
> > VALID_INCREMENT = 1H
> >
> > LEAD_SEQ = 0
> >
> > Each run time will use the PB2NC_INPUT_TEMPLATE to find files
whose
> > filenames match the time you need.
> >
> > To read prepBUFR observations for 12 hours, you will want to set
> > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range around
the
> valid
> > time to process. These values depend on what is contained inside
your
> > prepBUFR files. If you want to use data from the valid time of the
> filename
> > backwards 12 hours, you would set:
> >
> > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> >
> > For a runtime of 2015071512, this would be substituted as:
> >
> > valid_beg: 20150715_00
> > valid_end: 20150715_12
> >
> > If you want to create an output file for each hour in PB2NC you
would
> need
> > to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
> >
> > PointStat has similar config variables to set the valid beg and
end range
> > to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
> >
> > If your prepBUFR files are separated out into multiple files that
need to
> > be read into PB2NC, you can set a file window around the valid
time to
> > allow files to be passed into the app. More information on that
> > functionality is here:
> >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> >
> > If you are still having trouble getting this configured, you could
send
> me
> > the paths of the files you are trying to read and what data is
contained
> in
> > them so I can give you suggestions on what to set for the
configuration.
> >
> > Thanks,
> > George
> >
> >
> >
> > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > Queue: met_help
> > > Subject: How to set time for hourly 12-24 hours data ?
> > > Owner: mccabe
> > > Requestors: zhuming at ucar.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Wed Apr 07 17:31:37 2021
Hi George,
Now makes me confused is why processed obs of several times are
missing.
There has files of time at 00,01,02, but no 03,04,05; has 06,07,08,
but no
09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with 21,no
22,23.
>From yours and mines:
ls -l /glade/u/home/mccabe/data/out/pb2nc1/
-rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52 pbs.2015071500.nc
-rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56 pbs.2015071501.nc
-rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00 pbs.2015071502.nc
-rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14 pbs.2015071506.nc
-rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18 pbs.2015071507.nc
-rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22 pbs.2015071508.nc
-rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36 pbs.2015071512.nc
-rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40 pbs.2015071513.nc
-rw-r--r-- 1 mccabe ral 96 Apr 6 18:40 pbs.2015071514.nc
-rw-r--r-- 1 mccabe ral 96 Apr 6 18:42 pbs.2015071515.nc
cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
total 96112
-rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11 pbs.2015071500.nc
-rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14 pbs.2015071501.nc
-rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18 pbs.2015071502.nc
-rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33 pbs.2015071506.nc
-rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36 pbs.2015071507.nc
-rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40 pbs.2015071508.nc
-rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55 pbs.2015071512.nc
-rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59 pbs.2015071513.nc
-rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03 pbs.2015071514.nc
-rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18 pbs.2015071518.nc
-rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22 pbs.2015071519.nc
-rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27 pbs.2015071520.nc
-rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31 pbs.2015071521.nc
-rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43 pbs.2015071600.nc
Thanks,
Zhuming
On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> Thanks for the information. You will need to use the truncate
keyword to
> find the files, then set the valid beg/end range to 1 hour. This is
what I
> changed from your PB2NC1.conf config file:
>
>
> *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> {valid?fmt=%Y%m%d_%H?shift=1H}*
>
> *PB2NC_INPUT_TEMPLATE =
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
>
> My config file is on cheyenne here:
> /glade/u/home/mccabe/PB2NC1.conf
>
> I just started running it and the output is being written here:
> /glade/u/home/mccabe/data/out/pb2nc1
>
>
> For PointStat, you will want to loop over the same times. I noticed
you
> specify message type in the VAR<n>_OPTIONS, but you do not need to
do this
> since there is a POINT_STAT_MESSAGE_TYPE variable that defines this.
This
> is what I changed in your PointStat1.conf file:
>
> *VALID_END = 2015071600*
>
>
>
> *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
=POINT_STAT_OBS_VALID_BEG =*
>
>
> *POINT_STAT_MESSAGE_TYPE = ADPSFC*
>
>
>
>
>
> *#OBS_VAR1_OPTIONS = message_type = ["ADPSFC"];#OBS_VAR2_OPTIONS =
> message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
> ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> /glade/scratch/zhuming/metplus4/pb2nc1*
>
> *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
>
> Or you can view my file in the same directory. I usually try to test
with a
> single run time to make sure it is working properly, so I set
VALID_END
> back to 2015071500. I ran it and it produced an output file, but
there were
> no values besides the header. I checked the log file and saw this:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for observation
type
> ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
> matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> processed = 393373DEBUG 3: Rejected: station id = 0DEBUG 3:
> Rejected: obs type = 390528DEBUG 3: Rejected: valid time
=
> 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the
grid =
> 2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
mismatch
> = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message type
> = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
> value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3: Rejected:
bad
> climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG
2:
>
>
--------------------------------------------------------------------------------DEBUG
> 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
observation type
> ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
> matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> processed = 393373*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected: obs type
=
> 382965DEBUG 3: Rejected: valid time = 0DEBUG 3: Rejected: bad
obs
> value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3:
Rejected:
> topography = 0DEBUG 3: Rejected: level mismatch = 1168DEBUG 3:
> Rejected: quality marker = 0DEBUG 3: Rejected: message type =
0DEBUG 3:
> Rejected: masking region = 0DEBUG 3: Rejected: bad fcst value =
0DEBUG 3:
> Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo stdev =
0DEBUG 3:
> Rejected: duplicates = 0DEBUG 2:DEBUG 2:
>
>
--------------------------------------------------------------------------------DEBUG
> 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
observation type
> ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
> matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> processed = 393373DEBUG 3: Rejected: station id = 0DEBUG 3:
> Rejected: obs type = 382965DEBUG 3: Rejected: valid time
=
> 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the
grid =
> 9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
mismatch
> = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
> type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected:
bad
> fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected: bad
> climo stdev = 0*
>
>
>
>
> *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
>
>
--------------------------------------------------------------------------------DEBUG
> 2:DEBUG 1: Output file:
>
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
>
>
>
> This describes the reason why each observation was rejected. Some
> observations didn't match the message type you requested, some were
not
> within the forecast grid, and some did not match the vertical level.
You
> will have to make adjustments to your settings so that the data
matches up
> properly. Maybe the grid you are using in the PB2NC config file does
not
> match the forecast grid and you are missing values. If that is the
case,
> you will have to reconfigure and rerun PB2NC to get output on the
correct
> gr5id. Maybe the observation points don't line up with the vertical
level
> of the model fields. You can use a range of level values for the
> observation data to allow data within that range, i.e. Z0-10.
>
> Let me know if you are still having issues getting this to work and
I can
> help out.
>
> Thanks,
> George
>
>
> On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > Thanks for your reply!
> >
> > I'm still confused, I need to output hourly obs data to do hourly
> > verification.
> > The original GDAS Prepbufr data has 4 files one day, should with 6
hours
> > obs data in one file.
> >
> > After "PB2NC", I can see pbs.2015071500.nc (obs time is 00? or 6
times
> of
> > 00,01,02,03,04,05?), pbs.2015071506.nc...
> >
> > my config file :
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> >
> > Thanks,
> > Zhuming
> >
> > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > To run the tools hourly you would set the VALID_INCREMENT value
for 1H.
> > > Here is an example that runs once per hour for a day from 2015-
07-15 0Z
> > to
> > > 2017-07-16 0Z:
> > >
> > > VALID_TIME_FMT = %Y%m%d%H
> > >
> > > VALID_BEG = 2015071500
> > >
> > > VALID_END = 2015071600
> > >
> > > VALID_INCREMENT = 1H
> > >
> > > LEAD_SEQ = 0
> > >
> > > Each run time will use the PB2NC_INPUT_TEMPLATE to find files
whose
> > > filenames match the time you need.
> > >
> > > To read prepBUFR observations for 12 hours, you will want to set
> > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around the
> > valid
> > > time to process. These values depend on what is contained inside
your
> > > prepBUFR files. If you want to use data from the valid time of
the
> > filename
> > > backwards 12 hours, you would set:
> > >
> > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > >
> > > For a runtime of 2015071512, this would be substituted as:
> > >
> > > valid_beg: 20150715_00
> > > valid_end: 20150715_12
> > >
> > > If you want to create an output file for each hour in PB2NC you
would
> > need
> > > to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
> > >
> > > PointStat has similar config variables to set the valid beg and
end
> range
> > > to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
> > >
> > > If your prepBUFR files are separated out into multiple files
that need
> to
> > > be read into PB2NC, you can set a file window around the valid
time to
> > > allow files to be passed into the app. More information on that
> > > functionality is here:
> > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > >
> > > If you are still having trouble getting this configured, you
could send
> > me
> > > the paths of the files you are trying to read and what data is
> contained
> > in
> > > them so I can give you suggestions on what to set for the
> configuration.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > >
> > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > Queue: met_help
> > > > Subject: How to set time for hourly 12-24 hours data ?
> > > > Owner: mccabe
> > > > Requestors: zhuming at ucar.edu
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >
> > > >
> > > >
> > > > This transaction appears to have no content
> > > >
> > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Apr 07 17:42:33 2021
Hi Zhuming,
Looking at the log output, it looks like the data is stored in the
prepBUFR
files differently that you may have thought:
DEBUG 2: PrepBufr Time Center: 20150715_000000
DEBUG 2: Searching Time Window: 20150715_030000 to
20150715_040000
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90%
95%
100%
DEBUG 2: Total Messages processed = 590836
DEBUG 2: Rejected based on message type = 0
DEBUG 2: Rejected based on station id = 0
DEBUG 2: Rejected based on valid time = 590815
DEBUG 2: Rejected based on masking grid = 20
DEBUG 2: Rejected based on masking polygon = 0
DEBUG 2: Rejected based on elevation = 0
DEBUG 2: Rejected based on pb report type = 1
DEBUG 2: Rejected based on input report type = 0
DEBUG 2: Rejected based on instrument type = 0
DEBUG 2: Rejected based on zero observations = 0
DEBUG 2: Total Messages retained = 0
DEBUG 2: Total observations retained or derived = 0
DEBUG 1: Obs time between *20150714_210000 and 20150715_030000*
WARNING:
WARNING: process_pbfile() -> No PrepBufr messages retained from file:
/glade/collections/rda/data/ds337.0/prepnr/2015/prepbufr.gdas.2015071500.nr
WARNING:
ERROR :
ERROR : write_netcdf_hdr_data() -> No PrepBufr messages retained.
Nothing
to write.
ERROR :
The prepbufr.gdas.2015071500.nr file actually contains data from
20150714_210000 to 20150715_030000, not 20150715_000000 to
20150715_060000.
To get the correct file for each run, you will have to shift the valid
time
before truncating it so that it gets the correct file for each run.
Instead
of setting:
PB2NC_INPUT_TEMPLATE =
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
You will have to also shift forward 3 hours, i.e.:
PB2NC_INPUT_TEMPLATE =
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
This says to add 3 hours to the current valid time before truncating
to the
nearest 6 hours.
Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
File: prepbufr.gdas.2015071500.nr
Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
File: prepbufr.gdas.2015071500.nr
Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
File: prepbufr.gdas.2015071500.nr
Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
File: prepbufr.gdas.2015071506.nr
Let me know if that doesn't work.
Thanks,
George
On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> Now makes me confused is why processed obs of several times are
missing.
> There has files of time at 00,01,02, but no 03,04,05; has 06,07,08,
but no
> 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with 21,no
22,23.
> From yours and mines:
>
> ls -l /glade/u/home/mccabe/data/out/pb2nc1/
>
>
> -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52 pbs.2015071500.nc
>
> -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56 pbs.2015071501.nc
>
> -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00 pbs.2015071502.nc
>
> -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14 pbs.2015071506.nc
>
> -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18 pbs.2015071507.nc
>
> -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22 pbs.2015071508.nc
>
> -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36 pbs.2015071512.nc
>
> -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40 pbs.2015071513.nc
>
> -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40 pbs.2015071514.nc
>
> -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42 pbs.2015071515.nc
>
>
> cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
>
> total 96112
>
> -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11 pbs.2015071500.nc
>
> -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14 pbs.2015071501.nc
>
> -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18 pbs.2015071502.nc
>
> -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33 pbs.2015071506.nc
>
> -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36 pbs.2015071507.nc
>
> -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40 pbs.2015071508.nc
>
> -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55 pbs.2015071512.nc
>
> -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59 pbs.2015071513.nc
>
> -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03 pbs.2015071514.nc
>
> -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18 pbs.2015071518.nc
>
> -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22 pbs.2015071519.nc
>
> -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27 pbs.2015071520.nc
>
> -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31 pbs.2015071521.nc
>
> -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43 pbs.2015071600.nc
>
>
>
> Thanks,
>
> Zhuming
>
> On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > Thanks for the information. You will need to use the truncate
keyword to
> > find the files, then set the valid beg/end range to 1 hour. This
is what
> I
> > changed from your PB2NC1.conf config file:
> >
> >
> > *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > {valid?fmt=%Y%m%d_%H?shift=1H}*
> >
> > *PB2NC_INPUT_TEMPLATE =
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> >
> > My config file is on cheyenne here:
> > /glade/u/home/mccabe/PB2NC1.conf
> >
> > I just started running it and the output is being written here:
> > /glade/u/home/mccabe/data/out/pb2nc1
> >
> >
> > For PointStat, you will want to loop over the same times. I
noticed you
> > specify message type in the VAR<n>_OPTIONS, but you do not need to
do
> this
> > since there is a POINT_STAT_MESSAGE_TYPE variable that defines
this. This
> > is what I changed in your PointStat1.conf file:
> >
> > *VALID_END = 2015071600*
> >
> >
> >
> > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
=POINT_STAT_OBS_VALID_BEG
> =*
> >
> >
> > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> >
> >
> >
> >
> >
> > *#OBS_VAR1_OPTIONS = message_type = ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
> > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > /glade/scratch/zhuming/metplus4/pb2nc1*
> >
> > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> >
> > Or you can view my file in the same directory. I usually try to
test
> with a
> > single run time to make sure it is working properly, so I set
VALID_END
> > back to 2015071500. I ran it and it produced an output file, but
there
> were
> > no values besides the header. I checked the log file and saw this:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for observation
type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
0
> > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> > processed = 393373DEBUG 3: Rejected: station id = 0DEBUG
3:
> > Rejected: obs type = 390528DEBUG 3: Rejected: valid time
=
> > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the
grid
> =
> > 2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
> mismatch
> > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
> type
> > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
> > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3: Rejected:
bad
> > climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG
2:
> >
> >
>
--------------------------------------------------------------------------------DEBUG
> > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
observation
> type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
0
> > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> > processed = 393373*
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected: obs type
> =
> > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3: Rejected: bad
obs
> > value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3:
Rejected:
> > topography = 0DEBUG 3: Rejected: level mismatch = 1168DEBUG
3:
> > Rejected: quality marker = 0DEBUG 3: Rejected: message type =
0DEBUG
> 3:
> > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst value =
0DEBUG
> 3:
> > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo stdev =
0DEBUG
> 3:
> > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> >
> >
>
--------------------------------------------------------------------------------DEBUG
> > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
observation
> type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
0
> > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> > processed = 393373DEBUG 3: Rejected: station id = 0DEBUG
3:
> > Rejected: obs type = 382965DEBUG 3: Rejected: valid time
=
> > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the
grid
> =
> > 9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
> mismatch
> > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
> > type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected: bad
> > fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected:
> bad
> > climo stdev = 0*
> >
> >
> >
> >
> > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> >
> >
>
--------------------------------------------------------------------------------DEBUG
> > 2:DEBUG 1: Output file:
> >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> >
> >
> >
> > This describes the reason why each observation was rejected. Some
> > observations didn't match the message type you requested, some
were not
> > within the forecast grid, and some did not match the vertical
level. You
> > will have to make adjustments to your settings so that the data
matches
> up
> > properly. Maybe the grid you are using in the PB2NC config file
does not
> > match the forecast grid and you are missing values. If that is the
case,
> > you will have to reconfigure and rerun PB2NC to get output on the
correct
> > gr5id. Maybe the observation points don't line up with the
vertical level
> > of the model fields. You can use a range of level values for the
> > observation data to allow data within that range, i.e. Z0-10.
> >
> > Let me know if you are still having issues getting this to work
and I can
> > help out.
> >
> > Thanks,
> > George
> >
> >
> > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > Thanks for your reply!
> > >
> > > I'm still confused, I need to output hourly obs data to do
hourly
> > > verification.
> > > The original GDAS Prepbufr data has 4 files one day, should with
6
> hours
> > > obs data in one file.
> > >
> > > After "PB2NC", I can see pbs.2015071500.nc (obs time is 00? or 6
times
> > of
> > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > >
> > > my config file :
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > >
> > > Thanks,
> > > Zhuming
> > >
> > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > To run the tools hourly you would set the VALID_INCREMENT
value for
> 1H.
> > > > Here is an example that runs once per hour for a day from
2015-07-15
> 0Z
> > > to
> > > > 2017-07-16 0Z:
> > > >
> > > > VALID_TIME_FMT = %Y%m%d%H
> > > >
> > > > VALID_BEG = 2015071500
> > > >
> > > > VALID_END = 2015071600
> > > >
> > > > VALID_INCREMENT = 1H
> > > >
> > > > LEAD_SEQ = 0
> > > >
> > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find files
whose
> > > > filenames match the time you need.
> > > >
> > > > To read prepBUFR observations for 12 hours, you will want to
set
> > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around the
> > > valid
> > > > time to process. These values depend on what is contained
inside your
> > > > prepBUFR files. If you want to use data from the valid time of
the
> > > filename
> > > > backwards 12 hours, you would set:
> > > >
> > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > >
> > > > For a runtime of 2015071512, this would be substituted as:
> > > >
> > > > valid_beg: 20150715_00
> > > > valid_end: 20150715_12
> > > >
> > > > If you want to create an output file for each hour in PB2NC
you would
> > > need
> > > > to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
> > > >
> > > > PointStat has similar config variables to set the valid beg
and end
> > range
> > > > to use (POINT_STAT_OBS_VALID_BEG and
POINT_STAT_OBS_VALID_END).
> > > >
> > > > If your prepBUFR files are separated out into multiple files
that
> need
> > to
> > > > be read into PB2NC, you can set a file window around the valid
time
> to
> > > > allow files to be passed into the app. More information on
that
> > > > functionality is here:
> > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > >
> > > > If you are still having trouble getting this configured, you
could
> send
> > > me
> > > > the paths of the files you are trying to read and what data is
> > contained
> > > in
> > > > them so I can give you suggestions on what to set for the
> > configuration.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > >
> > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > > Queue: met_help
> > > > > Subject: How to set time for hourly 12-24 hours data ?
> > > > > Owner: mccabe
> > > > > Requestors: zhuming at ucar.edu
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >
> > > > >
> > > > >
> > > > > This transaction appears to have no content
> > > > >
> > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Thu Apr 08 09:45:31 2021
Hi George,
Processing Obs data using "PB2NC" are right now.
Now I'm using "PointStat" , I set VALID_BEG = 2015071500, VALID_END =
2015071501. It looks for the forecast data 2015071500/
wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
wrfout_d02_2015-07-15_01:00:00_cf.nc.
But my forecast data are all put in dir of 2015071500/.
So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get the right
fcst
data?
FCST_POINT_STAT_INPUT_TEMPLATE =
{valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:00_cf.nc
config file: /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
log file:
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
Thanks,
Zhuming
On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> Looking at the log output, it looks like the data is stored in the
prepBUFR
> files differently that you may have thought:
>
> DEBUG 2: PrepBufr Time Center: 20150715_000000
> DEBUG 2: Searching Time Window: 20150715_030000 to
20150715_040000
> 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85%
90% 95%
> 100%
>
> DEBUG 2: Total Messages processed = 590836
> DEBUG 2: Rejected based on message type = 0
> DEBUG 2: Rejected based on station id = 0
> DEBUG 2: Rejected based on valid time = 590815
> DEBUG 2: Rejected based on masking grid = 20
> DEBUG 2: Rejected based on masking polygon = 0
> DEBUG 2: Rejected based on elevation = 0
> DEBUG 2: Rejected based on pb report type = 1
> DEBUG 2: Rejected based on input report type = 0
> DEBUG 2: Rejected based on instrument type = 0
> DEBUG 2: Rejected based on zero observations = 0
> DEBUG 2: Total Messages retained = 0
> DEBUG 2: Total observations retained or derived = 0
> DEBUG 1: Obs time between *20150714_210000 and 20150715_030000*
> WARNING:
> WARNING: process_pbfile() -> No PrepBufr messages retained from
file:
> /glade/collections/rda/data/ds337.0/prepnr/2015/
> prepbufr.gdas.2015071500.nr
> WARNING:
> ERROR :
> ERROR : write_netcdf_hdr_data() -> No PrepBufr messages retained.
Nothing
> to write.
> ERROR :
>
> The prepbufr.gdas.2015071500.nr file actually contains data from
> 20150714_210000 to 20150715_030000, not 20150715_000000 to
20150715_060000.
>
> To get the correct file for each run, you will have to shift the
valid time
> before truncating it so that it gets the correct file for each run.
Instead
> of setting:
>
> PB2NC_INPUT_TEMPLATE =
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
>
> You will have to also shift forward 3 hours, i.e.:
>
> PB2NC_INPUT_TEMPLATE =
>
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
>
> This says to add 3 hours to the current valid time before truncating
to the
> nearest 6 hours.
>
> Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> File: prepbufr.gdas.2015071500.nr
>
> Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> File: prepbufr.gdas.2015071500.nr
>
> Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> File: prepbufr.gdas.2015071500.nr
>
> Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> File: prepbufr.gdas.2015071506.nr
>
> Let me know if that doesn't work.
>
> Thanks,
> George
>
>
> On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > Now makes me confused is why processed obs of several times are
missing.
> > There has files of time at 00,01,02, but no 03,04,05; has
06,07,08, but
> no
> > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with 21,no
22,23.
> > From yours and mines:
> >
> > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> >
> >
> > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52 pbs.2015071500.nc
> >
> > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56 pbs.2015071501.nc
> >
> > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00 pbs.2015071502.nc
> >
> > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14 pbs.2015071506.nc
> >
> > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18 pbs.2015071507.nc
> >
> > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22 pbs.2015071508.nc
> >
> > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36 pbs.2015071512.nc
> >
> > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40 pbs.2015071513.nc
> >
> > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40 pbs.2015071514.nc
> >
> > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42 pbs.2015071515.nc
> >
> >
> > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> >
> > total 96112
> >
> > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11 pbs.2015071500.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14 pbs.2015071501.nc
> >
> > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18 pbs.2015071502.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33 pbs.2015071506.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36 pbs.2015071507.nc
> >
> > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40 pbs.2015071508.nc
> >
> > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55 pbs.2015071512.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59 pbs.2015071513.nc
> >
> > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03 pbs.2015071514.nc
> >
> > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18 pbs.2015071518.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22 pbs.2015071519.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27 pbs.2015071520.nc
> >
> > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31 pbs.2015071521.nc
> >
> > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43 pbs.2015071600.nc
> >
> >
> >
> > Thanks,
> >
> > Zhuming
> >
> > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > Thanks for the information. You will need to use the truncate
keyword
> to
> > > find the files, then set the valid beg/end range to 1 hour. This
is
> what
> > I
> > > changed from your PB2NC1.conf config file:
> > >
> > >
> > > *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > >
> > > *PB2NC_INPUT_TEMPLATE =
> > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > >
> > > My config file is on cheyenne here:
> > > /glade/u/home/mccabe/PB2NC1.conf
> > >
> > > I just started running it and the output is being written here:
> > > /glade/u/home/mccabe/data/out/pb2nc1
> > >
> > >
> > > For PointStat, you will want to loop over the same times. I
noticed you
> > > specify message type in the VAR<n>_OPTIONS, but you do not need
to do
> > this
> > > since there is a POINT_STAT_MESSAGE_TYPE variable that defines
this.
> This
> > > is what I changed in your PointStat1.conf file:
> > >
> > > *VALID_END = 2015071600*
> > >
> > >
> > >
> > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
=POINT_STAT_OBS_VALID_BEG
> > =*
> > >
> > >
> > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > >
> > >
> > >
> > >
> > >
> > > *#OBS_VAR1_OPTIONS = message_type = ["ADPSFC"];#OBS_VAR2_OPTIONS
=
> > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
> > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > >
> > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > >
> > > Or you can view my file in the same directory. I usually try to
test
> > with a
> > > single run time to make sure it is working properly, so I set
VALID_END
> > > back to 2015071500. I ran it and it produced an output file, but
there
> > were
> > > no values besides the header. I checked the log file and saw
this:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for observation
type
> > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> Observations
> > > processed = 393373DEBUG 3: Rejected: station id = 0DEBUG
3:
> > > Rejected: obs type = 390528DEBUG 3: Rejected: valid time
=
> > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off
the grid
> > =
> > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
level
> > mismatch
> > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
> > type
> > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected:
bad fcst
> > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected: bad
> > > climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
observation
> > type
> > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> Observations
> > > processed = 393373*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected: obs
type
> > =
> > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3: Rejected:
bad obs
> > > value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3:
Rejected:
> > > topography = 0DEBUG 3: Rejected: level mismatch =
1168DEBUG 3:
> > > Rejected: quality marker = 0DEBUG 3: Rejected: message type
=
> 0DEBUG
> > 3:
> > > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst value
=
> 0DEBUG
> > 3:
> > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo stdev
=
> 0DEBUG
> > 3:
> > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
observation
> > type
> > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> Observations
> > > processed = 393373DEBUG 3: Rejected: station id = 0DEBUG
3:
> > > Rejected: obs type = 382965DEBUG 3: Rejected: valid time
=
> > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off
the grid
> > =
> > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
level
> > mismatch
> > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
> > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected: bad
> > > fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected:
> > bad
> > > climo stdev = 0*
> > >
> > >
> > >
> > >
> > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > 2:DEBUG 1: Output file:
> > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > >
> > >
> > >
> > > This describes the reason why each observation was rejected.
Some
> > > observations didn't match the message type you requested, some
were not
> > > within the forecast grid, and some did not match the vertical
level.
> You
> > > will have to make adjustments to your settings so that the data
matches
> > up
> > > properly. Maybe the grid you are using in the PB2NC config file
does
> not
> > > match the forecast grid and you are missing values. If that is
the
> case,
> > > you will have to reconfigure and rerun PB2NC to get output on
the
> correct
> > > gr5id. Maybe the observation points don't line up with the
vertical
> level
> > > of the model fields. You can use a range of level values for the
> > > observation data to allow data within that range, i.e. Z0-10.
> > >
> > > Let me know if you are still having issues getting this to work
and I
> can
> > > help out.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >
> > > > Hi George,
> > > >
> > > > Thanks for your reply!
> > > >
> > > > I'm still confused, I need to output hourly obs data to do
hourly
> > > > verification.
> > > > The original GDAS Prepbufr data has 4 files one day, should
with 6
> > hours
> > > > obs data in one file.
> > > >
> > > > After "PB2NC", I can see pbs.2015071500.nc (obs time is 00? or
6
> times
> > > of
> > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > >
> > > > my config file :
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > >
> > > > Thanks,
> > > > Zhuming
> > > >
> > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > Hi Zhuming,
> > > > >
> > > > > To run the tools hourly you would set the VALID_INCREMENT
value for
> > 1H.
> > > > > Here is an example that runs once per hour for a day from
> 2015-07-15
> > 0Z
> > > > to
> > > > > 2017-07-16 0Z:
> > > > >
> > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > >
> > > > > VALID_BEG = 2015071500
> > > > >
> > > > > VALID_END = 2015071600
> > > > >
> > > > > VALID_INCREMENT = 1H
> > > > >
> > > > > LEAD_SEQ = 0
> > > > >
> > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find
files whose
> > > > > filenames match the time you need.
> > > > >
> > > > > To read prepBUFR observations for 12 hours, you will want to
set
> > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around
> the
> > > > valid
> > > > > time to process. These values depend on what is contained
inside
> your
> > > > > prepBUFR files. If you want to use data from the valid time
of the
> > > > filename
> > > > > backwards 12 hours, you would set:
> > > > >
> > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > >
> > > > > For a runtime of 2015071512, this would be substituted as:
> > > > >
> > > > > valid_beg: 20150715_00
> > > > > valid_end: 20150715_12
> > > > >
> > > > > If you want to create an output file for each hour in PB2NC
you
> would
> > > > need
> > > > > to make sure that the hour is specified in the
> PB2NC_OUTPUT_TEMPLATE.
> > > > >
> > > > > PointStat has similar config variables to set the valid beg
and end
> > > range
> > > > > to use (POINT_STAT_OBS_VALID_BEG and
POINT_STAT_OBS_VALID_END).
> > > > >
> > > > > If your prepBUFR files are separated out into multiple files
that
> > need
> > > to
> > > > > be read into PB2NC, you can set a file window around the
valid time
> > to
> > > > > allow files to be passed into the app. More information on
that
> > > > > functionality is here:
> > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > > >
> > > > > If you are still having trouble getting this configured, you
could
> > send
> > > > me
> > > > > the paths of the files you are trying to read and what data
is
> > > contained
> > > > in
> > > > > them so I can give you suggestions on what to set for the
> > > configuration.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > > > Queue: met_help
> > > > > > Subject: How to set time for hourly 12-24 hours data
?
> > > > > > Owner: mccabe
> > > > > > Requestors: zhuming at ucar.edu
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > >
> > > > > >
> > > > > >
> > > > > > This transaction appears to have no content
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Thu Apr 08 10:04:44 2021
Hi Zhuming,
Typically directories that contain data for an entire day are named
with
YYYYMMDD, so your filename template tag would be {valid?fmt=%Y%m%d},
however that may not always be the case, as you are seeing here. There
are
a couple ways to handle this.
If you know that the directory will always be YYYYMMDD with 00 at the
end,
you can simply put 00 in the template instead of adding %H in the tag,
i.e.
FCST_POINT_STAT_INPUT_TEMPLATE =
{valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:00_cf.nc
If the HH value can vary, for instance if there are 2 directories
created
per day (i.e. 2015071400 and 2015071412), then you can use the
truncate
keyword to handle it.
If directories are created every 24 hours:
FCST_POINT_STAT_INPUT_TEMPLATE =
{valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
00_cf.nc
If directories are created every 12 hours:
FCST_POINT_STAT_INPUT_TEMPLATE =
{valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
00_cf.nc
Let me know if that doesn't work for you.
Thanks,
George
On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> Processing Obs data using "PB2NC" are right now.
>
> Now I'm using "PointStat" , I set VALID_BEG = 2015071500, VALID_END
=
> 2015071501. It looks for the forecast data 2015071500/
> wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> wrfout_d02_2015-07-15_01:00:00_cf.nc.
> But my forecast data are all put in dir of 2015071500/.
> So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get the right
fcst
> data?
>
> FCST_POINT_STAT_INPUT_TEMPLATE =
> {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:00_cf.nc
>
>
> config file: /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
>
> log file:
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
>
>
> Thanks,
>
> Zhuming
>
> On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > Looking at the log output, it looks like the data is stored in the
> prepBUFR
> > files differently that you may have thought:
> >
> > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > DEBUG 2: Searching Time Window: 20150715_030000 to
> 20150715_040000
> > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85%
90%
> 95%
> > 100%
> >
> > DEBUG 2: Total Messages processed = 590836
> > DEBUG 2: Rejected based on message type = 0
> > DEBUG 2: Rejected based on station id = 0
> > DEBUG 2: Rejected based on valid time = 590815
> > DEBUG 2: Rejected based on masking grid = 20
> > DEBUG 2: Rejected based on masking polygon = 0
> > DEBUG 2: Rejected based on elevation = 0
> > DEBUG 2: Rejected based on pb report type = 1
> > DEBUG 2: Rejected based on input report type = 0
> > DEBUG 2: Rejected based on instrument type = 0
> > DEBUG 2: Rejected based on zero observations = 0
> > DEBUG 2: Total Messages retained = 0
> > DEBUG 2: Total observations retained or derived = 0
> > DEBUG 1: Obs time between *20150714_210000 and 20150715_030000*
> > WARNING:
> > WARNING: process_pbfile() -> No PrepBufr messages retained from
file:
> > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > prepbufr.gdas.2015071500.nr
> > WARNING:
> > ERROR :
> > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages retained.
> Nothing
> > to write.
> > ERROR :
> >
> > The prepbufr.gdas.2015071500.nr file actually contains data from
> > 20150714_210000 to 20150715_030000, not 20150715_000000 to
> 20150715_060000.
> >
> > To get the correct file for each run, you will have to shift the
valid
> time
> > before truncating it so that it gets the correct file for each
run.
> Instead
> > of setting:
> >
> > PB2NC_INPUT_TEMPLATE =
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> >
> > You will have to also shift forward 3 hours, i.e.:
> >
> > PB2NC_INPUT_TEMPLATE =
> >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> >
> > This says to add 3 hours to the current valid time before
truncating to
> the
> > nearest 6 hours.
> >
> > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > File: prepbufr.gdas.2015071500.nr
> >
> > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > File: prepbufr.gdas.2015071500.nr
> >
> > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > File: prepbufr.gdas.2015071500.nr
> >
> > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > File: prepbufr.gdas.2015071506.nr
> >
> > Let me know if that doesn't work.
> >
> > Thanks,
> > George
> >
> >
> > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > Now makes me confused is why processed obs of several times are
> missing.
> > > There has files of time at 00,01,02, but no 03,04,05; has
06,07,08, but
> > no
> > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with
21,no
> 22,23.
> > > From yours and mines:
> > >
> > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > >
> > >
> > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52 pbs.2015071500.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56 pbs.2015071501.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00 pbs.2015071502.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14 pbs.2015071506.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18 pbs.2015071507.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22 pbs.2015071508.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36 pbs.2015071512.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40 pbs.2015071513.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40 pbs.2015071514.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42 pbs.2015071515.nc
> > >
> > >
> > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > >
> > > total 96112
> > >
> > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
pbs.2015071500.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
pbs.2015071501.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
pbs.2015071502.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
pbs.2015071506.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
pbs.2015071507.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
pbs.2015071508.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
pbs.2015071512.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
pbs.2015071513.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
pbs.2015071514.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
pbs.2015071518.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
pbs.2015071519.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
pbs.2015071520.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
pbs.2015071521.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
pbs.2015071600.nc
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Zhuming
> > >
> > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > Thanks for the information. You will need to use the truncate
keyword
> > to
> > > > find the files, then set the valid beg/end range to 1 hour.
This is
> > what
> > > I
> > > > changed from your PB2NC1.conf config file:
> > > >
> > > >
> > > > *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > >
> > > > *PB2NC_INPUT_TEMPLATE =
> > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > >
> > > > My config file is on cheyenne here:
> > > > /glade/u/home/mccabe/PB2NC1.conf
> > > >
> > > > I just started running it and the output is being written
here:
> > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > >
> > > >
> > > > For PointStat, you will want to loop over the same times. I
noticed
> you
> > > > specify message type in the VAR<n>_OPTIONS, but you do not
need to do
> > > this
> > > > since there is a POINT_STAT_MESSAGE_TYPE variable that defines
this.
> > This
> > > > is what I changed in your PointStat1.conf file:
> > > >
> > > > *VALID_END = 2015071600*
> > > >
> > > >
> > > >
> > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> =POINT_STAT_OBS_VALID_BEG
> > > =*
> > > >
> > > >
> > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *#OBS_VAR1_OPTIONS = message_type =
["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
> > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > >
> > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > >
> > > > Or you can view my file in the same directory. I usually try
to test
> > > with a
> > > > single run time to make sure it is working properly, so I set
> VALID_END
> > > > back to 2015071500. I ran it and it produced an output file,
but
> there
> > > were
> > > > no values besides the header. I checked the log file and saw
this:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
observation type
> > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > Observations
> > > > processed = 393373DEBUG 3: Rejected: station id =
0DEBUG 3:
> > > > Rejected: obs type = 390528DEBUG 3: Rejected: valid
time
> =
> > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off
the
> grid
> > > =
> > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
level
> > > mismatch
> > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
> message
> > > type
> > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected:
bad
> fcst
> > > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected:
> bad
> > > > climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG 2:
> > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
> observation
> > > type
> > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > Observations
> > > > processed = 393373*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected: obs
type
> > > =
> > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3: Rejected:
bad
> obs
> > > > value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3:
> Rejected:
> > > > topography = 0DEBUG 3: Rejected: level mismatch =
1168DEBUG 3:
> > > > Rejected: quality marker = 0DEBUG 3: Rejected: message type
=
> > 0DEBUG
> > > 3:
> > > > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst value
=
> > 0DEBUG
> > > 3:
> > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo
stdev =
> > 0DEBUG
> > > 3:
> > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
> observation
> > > type
> > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > Observations
> > > > processed = 393373DEBUG 3: Rejected: station id =
0DEBUG 3:
> > > > Rejected: obs type = 382965DEBUG 3: Rejected: valid
time
> =
> > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off
the
> grid
> > > =
> > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
level
> > > mismatch
> > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> message
> > > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected:
> bad
> > > > fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
> Rejected:
> > > bad
> > > > climo stdev = 0*
> > > >
> > > >
> > > >
> > > >
> > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > 2:DEBUG 1: Output file:
> > > >
> > > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > >
> > > >
> > > >
> > > > This describes the reason why each observation was rejected.
Some
> > > > observations didn't match the message type you requested, some
were
> not
> > > > within the forecast grid, and some did not match the vertical
level.
> > You
> > > > will have to make adjustments to your settings so that the
data
> matches
> > > up
> > > > properly. Maybe the grid you are using in the PB2NC config
file does
> > not
> > > > match the forecast grid and you are missing values. If that is
the
> > case,
> > > > you will have to reconfigure and rerun PB2NC to get output on
the
> > correct
> > > > gr5id. Maybe the observation points don't line up with the
vertical
> > level
> > > > of the model fields. You can use a range of level values for
the
> > > > observation data to allow data within that range, i.e. Z0-10.
> > > >
> > > > Let me know if you are still having issues getting this to
work and I
> > can
> > > > help out.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >
> > > > > Hi George,
> > > > >
> > > > > Thanks for your reply!
> > > > >
> > > > > I'm still confused, I need to output hourly obs data to do
hourly
> > > > > verification.
> > > > > The original GDAS Prepbufr data has 4 files one day, should
with 6
> > > hours
> > > > > obs data in one file.
> > > > >
> > > > > After "PB2NC", I can see pbs.2015071500.nc (obs time is 00?
or 6
> > times
> > > > of
> > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > >
> > > > > my config file :
> cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > >
> > > > > Thanks,
> > > > > Zhuming
> > > > >
> > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Zhuming,
> > > > > >
> > > > > > To run the tools hourly you would set the VALID_INCREMENT
value
> for
> > > 1H.
> > > > > > Here is an example that runs once per hour for a day from
> > 2015-07-15
> > > 0Z
> > > > > to
> > > > > > 2017-07-16 0Z:
> > > > > >
> > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > >
> > > > > > VALID_BEG = 2015071500
> > > > > >
> > > > > > VALID_END = 2015071600
> > > > > >
> > > > > > VALID_INCREMENT = 1H
> > > > > >
> > > > > > LEAD_SEQ = 0
> > > > > >
> > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find
files
> whose
> > > > > > filenames match the time you need.
> > > > > >
> > > > > > To read prepBUFR observations for 12 hours, you will want
to set
> > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around
> > the
> > > > > valid
> > > > > > time to process. These values depend on what is contained
inside
> > your
> > > > > > prepBUFR files. If you want to use data from the valid
time of
> the
> > > > > filename
> > > > > > backwards 12 hours, you would set:
> > > > > >
> > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > >
> > > > > > For a runtime of 2015071512, this would be substituted as:
> > > > > >
> > > > > > valid_beg: 20150715_00
> > > > > > valid_end: 20150715_12
> > > > > >
> > > > > > If you want to create an output file for each hour in
PB2NC you
> > would
> > > > > need
> > > > > > to make sure that the hour is specified in the
> > PB2NC_OUTPUT_TEMPLATE.
> > > > > >
> > > > > > PointStat has similar config variables to set the valid
beg and
> end
> > > > range
> > > > > > to use (POINT_STAT_OBS_VALID_BEG and
POINT_STAT_OBS_VALID_END).
> > > > > >
> > > > > > If your prepBUFR files are separated out into multiple
files that
> > > need
> > > > to
> > > > > > be read into PB2NC, you can set a file window around the
valid
> time
> > > to
> > > > > > allow files to be passed into the app. More information on
that
> > > > > > functionality is here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > > > >
> > > > > > If you are still having trouble getting this configured,
you
> could
> > > send
> > > > > me
> > > > > > the paths of the files you are trying to read and what
data is
> > > > contained
> > > > > in
> > > > > > them so I can give you suggestions on what to set for the
> > > > configuration.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > > > > Queue: met_help
> > > > > > > Subject: How to set time for hourly 12-24 hours
data ?
> > > > > > > Owner: mccabe
> > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This transaction appears to have no content
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Apr 14 13:56:25 2021
Hi Zhuming,
Just following up on this ticket. Were you able to get this resolved?
Let me know if you need more assistance.
Thanks,
George
On Thu Apr 08 10:04:44 2021, mccabe wrote:
> Hi Zhuming,
>
> Typically directories that contain data for an entire day are named
> with
> YYYYMMDD, so your filename template tag would be
{valid?fmt=%Y%m%d},
> however that may not always be the case, as you are seeing here.
There
> are
> a couple ways to handle this.
>
> If you know that the directory will always be YYYYMMDD with 00 at
the
> end,
> you can simply put 00 in the template instead of adding %H in the
tag,
> i.e.
>
> FCST_POINT_STAT_INPUT_TEMPLATE =
> {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:00_cf.nc
>
> If the HH value can vary, for instance if there are 2 directories
> created
> per day (i.e. 2015071400 and 2015071412), then you can use the
> truncate
> keyword to handle it.
>
> If directories are created every 24 hours:
>
> FCST_POINT_STAT_INPUT_TEMPLATE =
> {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> %d_%H}:00:
> 00_cf.nc
>
> If directories are created every 12 hours:
>
> FCST_POINT_STAT_INPUT_TEMPLATE =
> {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> %d_%H}:00:
> 00_cf.nc
>
> Let me know if that doesn't work for you.
>
> Thanks,
> George
>
> On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > Processing Obs data using "PB2NC" are right now.
> >
> > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
VALID_END =
> > 2015071501. It looks for the forecast data 2015071500/
> > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > But my forecast data are all put in dir of 2015071500/.
> > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get the
right
> > fcst
> > data?
> >
> > FCST_POINT_STAT_INPUT_TEMPLATE =
> > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> >
> >
> > config file: /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> >
> > log file:
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> >
> >
> > Thanks,
> >
> > Zhuming
> >
> > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > Looking at the log output, it looks like the data is stored in
the
> > prepBUFR
> > > files differently that you may have thought:
> > >
> > > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > > DEBUG 2: Searching Time Window: 20150715_030000 to
> > 20150715_040000
> > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80%
85%
> > > 90%
> > 95%
> > > 100%
> > >
> > > DEBUG 2: Total Messages processed = 590836
> > > DEBUG 2: Rejected based on message type = 0
> > > DEBUG 2: Rejected based on station id = 0
> > > DEBUG 2: Rejected based on valid time = 590815
> > > DEBUG 2: Rejected based on masking grid = 20
> > > DEBUG 2: Rejected based on masking polygon = 0
> > > DEBUG 2: Rejected based on elevation = 0
> > > DEBUG 2: Rejected based on pb report type = 1
> > > DEBUG 2: Rejected based on input report type = 0
> > > DEBUG 2: Rejected based on instrument type = 0
> > > DEBUG 2: Rejected based on zero observations = 0
> > > DEBUG 2: Total Messages retained = 0
> > > DEBUG 2: Total observations retained or derived = 0
> > > DEBUG 1: Obs time between *20150714_210000 and 20150715_030000*
> > > WARNING:
> > > WARNING: process_pbfile() -> No PrepBufr messages retained from
> > > file:
> > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > prepbufr.gdas.2015071500.nr
> > > WARNING:
> > > ERROR :
> > > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages
retained.
> > Nothing
> > > to write.
> > > ERROR :
> > >
> > > The prepbufr.gdas.2015071500.nr file actually contains data from
> > > 20150714_210000 to 20150715_030000, not 20150715_000000 to
> > 20150715_060000.
> > >
> > > To get the correct file for each run, you will have to shift the
> > > valid
> > time
> > > before truncating it so that it gets the correct file for each
run.
> > Instead
> > > of setting:
> > >
> > > PB2NC_INPUT_TEMPLATE =
> > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > >
> > > You will have to also shift forward 3 hours, i.e.:
> > >
> > > PB2NC_INPUT_TEMPLATE =
> > >
> > >
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > >
> > > This says to add 3 hours to the current valid time before
> > > truncating to
> > the
> > > nearest 6 hours.
> > >
> > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > File: prepbufr.gdas.2015071500.nr
> > >
> > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > File: prepbufr.gdas.2015071500.nr
> > >
> > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > File: prepbufr.gdas.2015071500.nr
> > >
> > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > File: prepbufr.gdas.2015071506.nr
> > >
> > > Let me know if that doesn't work.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >
> > > > Hi George,
> > > >
> > > > Now makes me confused is why processed obs of several times
are
> > missing.
> > > > There has files of time at 00,01,02, but no 03,04,05; has
> > > > 06,07,08, but
> > > no
> > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with
> > > > 21,no
> > 22,23.
> > > > From yours and mines:
> > > >
> > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > >
> > > >
> > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
pbs.2015071500.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
pbs.2015071501.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
pbs.2015071502.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
pbs.2015071506.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
pbs.2015071507.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
pbs.2015071508.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
pbs.2015071512.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
pbs.2015071513.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
pbs.2015071514.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
pbs.2015071515.nc
> > > >
> > > >
> > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > >
> > > > total 96112
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
pbs.2015071500.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
pbs.2015071501.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
pbs.2015071502.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
pbs.2015071506.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
pbs.2015071507.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
pbs.2015071508.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
pbs.2015071512.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
pbs.2015071513.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
pbs.2015071514.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
pbs.2015071518.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
pbs.2015071519.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
pbs.2015071520.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
pbs.2015071521.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
pbs.2015071600.nc
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Zhuming
> > > >
> > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> > > > <met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > Hi Zhuming,
> > > > >
> > > > > Thanks for the information. You will need to use the
truncate
> > > > > keyword
> > > to
> > > > > find the files, then set the valid beg/end range to 1 hour.
> > > > > This is
> > > what
> > > > I
> > > > > changed from your PB2NC1.conf config file:
> > > > >
> > > > >
> > > > > *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > >
> > > > > *PB2NC_INPUT_TEMPLATE =
> > > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > >
> > > > > My config file is on cheyenne here:
> > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > >
> > > > > I just started running it and the output is being written
here:
> > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > >
> > > > >
> > > > > For PointStat, you will want to loop over the same times. I
> > > > > noticed
> > you
> > > > > specify message type in the VAR<n>_OPTIONS, but you do not
need
> > > > > to do
> > > > this
> > > > > since there is a POINT_STAT_MESSAGE_TYPE variable that
defines
> > > > > this.
> > > This
> > > > > is what I changed in your PointStat1.conf file:
> > > > >
> > > > > *VALID_END = 2015071600*
> > > > >
> > > > >
> > > > >
> > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > =POINT_STAT_OBS_VALID_BEG
> > > > =*
> > > > >
> > > > >
> > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
> > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > >
> > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > > >
> > > > > Or you can view my file in the same directory. I usually try
to
> > > > > test
> > > > with a
> > > > > single run time to make sure it is working properly, so I
set
> > VALID_END
> > > > > back to 2015071500. I ran it and it produced an output file,
> > > > > but
> > there
> > > > were
> > > > > no values besides the header. I checked the log file and saw
> > > > > this:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
> > > > > observation type
> > > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
> > > > > using 0
> > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > > Observations
> > > > > processed = 393373DEBUG 3: Rejected: station id =
> > > > > 0DEBUG 3:
> > > > > Rejected: obs type = 390528DEBUG 3: Rejected: valid
time
> > =
> > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected:
off
> > > > > the
> > grid
> > > > =
> > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
> > > > > level
> > > > mismatch
> > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > message
> > > > type
> > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected:
bad
> > fcst
> > > > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
> > > > > Rejected:
> > bad
> > > > > climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG
> > > > > 2:DEBUG 2:
> > > > >
> > > > >
> > > >
> > >
> >
--------------------------------------------------------------------------------DEBUG
> > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
> > observation
> > > > type
> > > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
> > > > > using 0
> > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > > Observations
> > > > > processed = 393373*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected:
obs
> > > > > type
> > > > =
> > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3:
Rejected:
> > > > > bad
> > obs
> > > > > value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3:
> > Rejected:
> > > > > topography = 0DEBUG 3: Rejected: level mismatch =
> > > > > 1168DEBUG 3:
> > > > > Rejected: quality marker = 0DEBUG 3: Rejected: message type
> > > > > =
> > > 0DEBUG
> > > > 3:
> > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst
value
> > > > > =
> > > 0DEBUG
> > > > 3:
> > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo
stdev
> > > > > =
> > > 0DEBUG
> > > > 3:
> > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > >
> > > > >
> > > >
> > >
> >
--------------------------------------------------------------------------------DEBUG
> > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
> > observation
> > > > type
> > > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
> > > > > using 0
> > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > > Observations
> > > > > processed = 393373DEBUG 3: Rejected: station id =
> > > > > 0DEBUG 3:
> > > > > Rejected: obs type = 382965DEBUG 3: Rejected: valid
time
> > =
> > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected:
off
> > > > > the
> > grid
> > > > =
> > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
> > > > > level
> > > > mismatch
> > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > message
> > > > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
> > > > > Rejected:
> > bad
> > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG
3:
> > Rejected:
> > > > bad
> > > > > climo stdev = 0*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > >
> > > > >
> > > >
> > >
> >
--------------------------------------------------------------------------------DEBUG
> > > > > 2:DEBUG 1: Output file:
> > > > >
> > > > >
> > > >
> > >
> >
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > >
> > > > >
> > > > >
> > > > > This describes the reason why each observation was rejected.
> > > > > Some
> > > > > observations didn't match the message type you requested,
some
> > > > > were
> > not
> > > > > within the forecast grid, and some did not match the
vertical
> > > > > level.
> > > You
> > > > > will have to make adjustments to your settings so that the
data
> > matches
> > > > up
> > > > > properly. Maybe the grid you are using in the PB2NC config
file
> > > > > does
> > > not
> > > > > match the forecast grid and you are missing values. If that
is
> > > > > the
> > > case,
> > > > > you will have to reconfigure and rerun PB2NC to get output
on
> > > > > the
> > > correct
> > > > > gr5id. Maybe the observation points don't line up with the
> > > > > vertical
> > > level
> > > > > of the model fields. You can use a range of level values for
> > > > > the
> > > > > observation data to allow data within that range, i.e. Z0-
10.
> > > > >
> > > > > Let me know if you are still having issues getting this to
work
> > > > > and I
> > > can
> > > > > help out.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > >
> > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > >
> > > > > >
> > > > > > Hi George,
> > > > > >
> > > > > > Thanks for your reply!
> > > > > >
> > > > > > I'm still confused, I need to output hourly obs data to do
> > > > > > hourly
> > > > > > verification.
> > > > > > The original GDAS Prepbufr data has 4 files one day,
should
> > > > > > with 6
> > > > hours
> > > > > > obs data in one file.
> > > > > >
> > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs time is
00?
> > > > > > or 6
> > > times
> > > > > of
> > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > > >
> > > > > > my config file :
> > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > >
> > > > > > Thanks,
> > > > > > Zhuming
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Zhuming,
> > > > > > >
> > > > > > > To run the tools hourly you would set the
VALID_INCREMENT
> > > > > > > value
> > for
> > > > 1H.
> > > > > > > Here is an example that runs once per hour for a day
from
> > > 2015-07-15
> > > > 0Z
> > > > > > to
> > > > > > > 2017-07-16 0Z:
> > > > > > >
> > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > >
> > > > > > > VALID_BEG = 2015071500
> > > > > > >
> > > > > > > VALID_END = 2015071600
> > > > > > >
> > > > > > > VALID_INCREMENT = 1H
> > > > > > >
> > > > > > > LEAD_SEQ = 0
> > > > > > >
> > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find
> > > > > > > files
> > whose
> > > > > > > filenames match the time you need.
> > > > > > >
> > > > > > > To read prepBUFR observations for 12 hours, you will
want
> > > > > > > to set
> > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the
range
> > > > > > > around
> > > the
> > > > > > valid
> > > > > > > time to process. These values depend on what is
contained
> > > > > > > inside
> > > your
> > > > > > > prepBUFR files. If you want to use data from the valid
time
> > > > > > > of
> > the
> > > > > > filename
> > > > > > > backwards 12 hours, you would set:
> > > > > > >
> > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > >
> > > > > > > For a runtime of 2015071512, this would be substituted
as:
> > > > > > >
> > > > > > > valid_beg: 20150715_00
> > > > > > > valid_end: 20150715_12
> > > > > > >
> > > > > > > If you want to create an output file for each hour in
PB2NC
> > > > > > > you
> > > would
> > > > > > need
> > > > > > > to make sure that the hour is specified in the
> > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > >
> > > > > > > PointStat has similar config variables to set the valid
beg
> > > > > > > and
> > end
> > > > > range
> > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > >
> > > > > > > If your prepBUFR files are separated out into multiple
> > > > > > > files that
> > > > need
> > > > > to
> > > > > > > be read into PB2NC, you can set a file window around the
> > > > > > > valid
> > time
> > > > to
> > > > > > > allow files to be passed into the app. More information
on
> > > > > > > that
> > > > > > > functionality is here:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > windows-to-find-valid-files
> > > > > > >
> > > > > > > If you are still having trouble getting this configured,
> > > > > > > you
> > could
> > > > send
> > > > > > me
> > > > > > > the paths of the files you are trying to read and what
data
> > > > > > > is
> > > > > contained
> > > > > > in
> > > > > > > them so I can give you suggestions on what to set for
the
> > > > > configuration.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted
upon.
> > > > > > > > Transaction: Given to mccabe (George McCabe) by
jpresto
> > > > > > > > Queue: met_help
> > > > > > > > Subject: How to set time for hourly 12-24 hours
data
> > > > > > > > ?
> > > > > > > > Owner: mccabe
> > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > This transaction appears to have no content
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not
> > > > > > > feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
> > > obliged to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Wed Apr 14 15:17:49 2021
Hi George,
Thanks for your email, I just would like to send you for help.
I already resolved the last problem with the WRF forecast, thank you
for
your help.
I also need to verify the MPAS model forecast. These 12 hour
MAPS forecasts are stored in a one file, such as
2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
2015071512/mpas_wrf_cf_2015071512.nc
It seems MET is doing fine on the first 2 hours, but no matched pairs
for
later hours.
I'm not sure what's wrong with it? the time is not matched?
see the log file
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
Thanks,
Zhuming
On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> Just following up on this ticket. Were you able to get this
resolved? Let
> me know if you need more assistance.
>
> Thanks,
> George
>
> On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > Hi Zhuming,
> >
> > Typically directories that contain data for an entire day are
named
> > with
> > YYYYMMDD, so your filename template tag would be
{valid?fmt=%Y%m%d},
> > however that may not always be the case, as you are seeing here.
There
> > are
> > a couple ways to handle this.
> >
> > If you know that the directory will always be YYYYMMDD with 00 at
the
> > end,
> > you can simply put 00 in the template instead of adding %H in the
tag,
> > i.e.
> >
> > FCST_POINT_STAT_INPUT_TEMPLATE =
> > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> >
> > If the HH value can vary, for instance if there are 2 directories
> > created
> > per day (i.e. 2015071400 and 2015071412), then you can use the
> > truncate
> > keyword to handle it.
> >
> > If directories are created every 24 hours:
> >
> > FCST_POINT_STAT_INPUT_TEMPLATE =
> > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > %d_%H}:00:
> > 00_cf.nc
> >
> > If directories are created every 12 hours:
> >
> > FCST_POINT_STAT_INPUT_TEMPLATE =
> > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > %d_%H}:00:
> > 00_cf.nc
> >
> > Let me know if that doesn't work for you.
> >
> > Thanks,
> > George
> >
> > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > Processing Obs data using "PB2NC" are right now.
> > >
> > > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
VALID_END =
> > > 2015071501. It looks for the forecast data 2015071500/
> > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > But my forecast data are all put in dir of 2015071500/.
> > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get the
right
> > > fcst
> > > data?
> > >
> > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> > >
> > >
> > > config file: /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > >
> > > log file:
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > >
> > >
> > > Thanks,
> > >
> > > Zhuming
> > >
> > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > Looking at the log output, it looks like the data is stored in
the
> > > prepBUFR
> > > > files differently that you may have thought:
> > > >
> > > > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > > > DEBUG 2: Searching Time Window: 20150715_030000 to
> > > 20150715_040000
> > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80%
85%
> > > > 90%
> > > 95%
> > > > 100%
> > > >
> > > > DEBUG 2: Total Messages processed = 590836
> > > > DEBUG 2: Rejected based on message type = 0
> > > > DEBUG 2: Rejected based on station id = 0
> > > > DEBUG 2: Rejected based on valid time = 590815
> > > > DEBUG 2: Rejected based on masking grid = 20
> > > > DEBUG 2: Rejected based on masking polygon = 0
> > > > DEBUG 2: Rejected based on elevation = 0
> > > > DEBUG 2: Rejected based on pb report type = 1
> > > > DEBUG 2: Rejected based on input report type = 0
> > > > DEBUG 2: Rejected based on instrument type = 0
> > > > DEBUG 2: Rejected based on zero observations = 0
> > > > DEBUG 2: Total Messages retained = 0
> > > > DEBUG 2: Total observations retained or derived = 0
> > > > DEBUG 1: Obs time between *20150714_210000 and
20150715_030000*
> > > > WARNING:
> > > > WARNING: process_pbfile() -> No PrepBufr messages retained
from
> > > > file:
> > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > prepbufr.gdas.2015071500.nr
> > > > WARNING:
> > > > ERROR :
> > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages
retained.
> > > Nothing
> > > > to write.
> > > > ERROR :
> > > >
> > > > The prepbufr.gdas.2015071500.nr file actually contains data
from
> > > > 20150714_210000 to 20150715_030000, not 20150715_000000 to
> > > 20150715_060000.
> > > >
> > > > To get the correct file for each run, you will have to shift
the
> > > > valid
> > > time
> > > > before truncating it so that it gets the correct file for each
run.
> > > Instead
> > > > of setting:
> > > >
> > > > PB2NC_INPUT_TEMPLATE =
> > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > >
> > > > You will have to also shift forward 3 hours, i.e.:
> > > >
> > > > PB2NC_INPUT_TEMPLATE =
> > > >
> > > >
> > >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > >
> > > > This says to add 3 hours to the current valid time before
> > > > truncating to
> > > the
> > > > nearest 6 hours.
> > > >
> > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > > File: prepbufr.gdas.2015071500.nr
> > > >
> > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > > File: prepbufr.gdas.2015071500.nr
> > > >
> > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > > File: prepbufr.gdas.2015071500.nr
> > > >
> > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > > File: prepbufr.gdas.2015071506.nr
> > > >
> > > > Let me know if that doesn't work.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >
> > > > > Hi George,
> > > > >
> > > > > Now makes me confused is why processed obs of several times
are
> > > missing.
> > > > > There has files of time at 00,01,02, but no 03,04,05; has
> > > > > 06,07,08, but
> > > > no
> > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with
> > > > > 21,no
> > > 22,23.
> > > > > From yours and mines:
> > > > >
> > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > > >
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
pbs.2015071500.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
pbs.2015071501.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
pbs.2015071502.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
pbs.2015071506.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
pbs.2015071507.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
pbs.2015071508.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
pbs.2015071512.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
pbs.2015071513.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
pbs.2015071514.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
pbs.2015071515.nc
> > > > >
> > > > >
> > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > >
> > > > > total 96112
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
pbs.2015071500.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
pbs.2015071501.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
pbs.2015071502.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
pbs.2015071506.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
pbs.2015071507.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
pbs.2015071508.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
pbs.2015071512.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
pbs.2015071513.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
pbs.2015071514.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
pbs.2015071518.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
pbs.2015071519.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
pbs.2015071520.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
pbs.2015071521.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
pbs.2015071600.nc
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Zhuming
> > > > >
> > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> > > > > <met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Zhuming,
> > > > > >
> > > > > > Thanks for the information. You will need to use the
truncate
> > > > > > keyword
> > > > to
> > > > > > find the files, then set the valid beg/end range to 1
hour.
> > > > > > This is
> > > > what
> > > > > I
> > > > > > changed from your PB2NC1.conf config file:
> > > > > >
> > > > > >
> > > > > > *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
=
> > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > >
> > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > >
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > >
> > > > > > My config file is on cheyenne here:
> > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > >
> > > > > > I just started running it and the output is being written
here:
> > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > >
> > > > > >
> > > > > > For PointStat, you will want to loop over the same times.
I
> > > > > > noticed
> > > you
> > > > > > specify message type in the VAR<n>_OPTIONS, but you do not
need
> > > > > > to do
> > > > > this
> > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable that
defines
> > > > > > this.
> > > > This
> > > > > > is what I changed in your PointStat1.conf file:
> > > > > >
> > > > > > *VALID_END = 2015071600*
> > > > > >
> > > > > >
> > > > > >
> > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > > =POINT_STAT_OBS_VALID_BEG
> > > > > =*
> > > > > >
> > > > > >
> > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type
=
> > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > >
> > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > > > >
> > > > > > Or you can view my file in the same directory. I usually
try to
> > > > > > test
> > > > > with a
> > > > > > single run time to make sure it is working properly, so I
set
> > > VALID_END
> > > > > > back to 2015071500. I ran it and it produced an output
file,
> > > > > > but
> > > there
> > > > > were
> > > > > > no values besides the header. I checked the log file and
saw
> > > > > > this:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
> > > > > > observation type
> > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > using 0
> > > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG
3:
> > > > Observations
> > > > > > processed = 393373DEBUG 3: Rejected: station id =
> > > > > > 0DEBUG 3:
> > > > > > Rejected: obs type = 390528DEBUG 3: Rejected: valid
time
> > > =
> > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected:
off
> > > > > > the
> > > grid
> > > > > =
> > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > level
> > > > > mismatch
> > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > message
> > > > > type
> > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected: bad
> > > fcst
> > > > > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
> > > > > > Rejected:
> > > bad
> > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG
> > > > > > 2:DEBUG 2:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
> > > observation
> > > > > type
> > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > using 0
> > > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG
3:
> > > > Observations
> > > > > > processed = 393373*
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected:
obs
> > > > > > type
> > > > > =
> > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3:
Rejected:
> > > > > > bad
> > > obs
> > > > > > value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG
3:
> > > Rejected:
> > > > > > topography = 0DEBUG 3: Rejected: level mismatch =
> > > > > > 1168DEBUG 3:
> > > > > > Rejected: quality marker = 0DEBUG 3: Rejected: message
type
> > > > > > =
> > > > 0DEBUG
> > > > > 3:
> > > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst
value
> > > > > > =
> > > > 0DEBUG
> > > > > 3:
> > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo
stdev
> > > > > > =
> > > > 0DEBUG
> > > > > 3:
> > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
> > > observation
> > > > > type
> > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > using 0
> > > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG
3:
> > > > Observations
> > > > > > processed = 393373DEBUG 3: Rejected: station id =
> > > > > > 0DEBUG 3:
> > > > > > Rejected: obs type = 382965DEBUG 3: Rejected: valid
time
> > > =
> > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected:
off
> > > > > > the
> > > grid
> > > > > =
> > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > level
> > > > > mismatch
> > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > message
> > > > > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
> > > > > > Rejected:
> > > bad
> > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG
3:
> > > Rejected:
> > > > > bad
> > > > > > climo stdev = 0*
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > 2:DEBUG 1: Output file:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > >
> > > > > >
> > > > > >
> > > > > > This describes the reason why each observation was
rejected.
> > > > > > Some
> > > > > > observations didn't match the message type you requested,
some
> > > > > > were
> > > not
> > > > > > within the forecast grid, and some did not match the
vertical
> > > > > > level.
> > > > You
> > > > > > will have to make adjustments to your settings so that the
data
> > > matches
> > > > > up
> > > > > > properly. Maybe the grid you are using in the PB2NC config
file
> > > > > > does
> > > > not
> > > > > > match the forecast grid and you are missing values. If
that is
> > > > > > the
> > > > case,
> > > > > > you will have to reconfigure and rerun PB2NC to get output
on
> > > > > > the
> > > > correct
> > > > > > gr5id. Maybe the observation points don't line up with the
> > > > > > vertical
> > > > level
> > > > > > of the model fields. You can use a range of level values
for
> > > > > > the
> > > > > > observation data to allow data within that range, i.e. Z0-
10.
> > > > > >
> > > > > > Let me know if you are still having issues getting this to
work
> > > > > > and I
> > > > can
> > > > > > help out.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > >
> > > > > > >
> > > > > > > Hi George,
> > > > > > >
> > > > > > > Thanks for your reply!
> > > > > > >
> > > > > > > I'm still confused, I need to output hourly obs data to
do
> > > > > > > hourly
> > > > > > > verification.
> > > > > > > The original GDAS Prepbufr data has 4 files one day,
should
> > > > > > > with 6
> > > > > hours
> > > > > > > obs data in one file.
> > > > > > >
> > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs time is
00?
> > > > > > > or 6
> > > > times
> > > > > > of
> > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > > > >
> > > > > > > my config file :
> > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Zhuming
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Zhuming,
> > > > > > > >
> > > > > > > > To run the tools hourly you would set the
VALID_INCREMENT
> > > > > > > > value
> > > for
> > > > > 1H.
> > > > > > > > Here is an example that runs once per hour for a day
from
> > > > 2015-07-15
> > > > > 0Z
> > > > > > > to
> > > > > > > > 2017-07-16 0Z:
> > > > > > > >
> > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > > >
> > > > > > > > VALID_BEG = 2015071500
> > > > > > > >
> > > > > > > > VALID_END = 2015071600
> > > > > > > >
> > > > > > > > VALID_INCREMENT = 1H
> > > > > > > >
> > > > > > > > LEAD_SEQ = 0
> > > > > > > >
> > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to
find
> > > > > > > > files
> > > whose
> > > > > > > > filenames match the time you need.
> > > > > > > >
> > > > > > > > To read prepBUFR observations for 12 hours, you will
want
> > > > > > > > to set
> > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the
range
> > > > > > > > around
> > > > the
> > > > > > > valid
> > > > > > > > time to process. These values depend on what is
contained
> > > > > > > > inside
> > > > your
> > > > > > > > prepBUFR files. If you want to use data from the valid
time
> > > > > > > > of
> > > the
> > > > > > > filename
> > > > > > > > backwards 12 hours, you would set:
> > > > > > > >
> > > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > > >
> > > > > > > > For a runtime of 2015071512, this would be substituted
as:
> > > > > > > >
> > > > > > > > valid_beg: 20150715_00
> > > > > > > > valid_end: 20150715_12
> > > > > > > >
> > > > > > > > If you want to create an output file for each hour in
PB2NC
> > > > > > > > you
> > > > would
> > > > > > > need
> > > > > > > > to make sure that the hour is specified in the
> > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > > >
> > > > > > > > PointStat has similar config variables to set the
valid beg
> > > > > > > > and
> > > end
> > > > > > range
> > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > > >
> > > > > > > > If your prepBUFR files are separated out into multiple
> > > > > > > > files that
> > > > > need
> > > > > > to
> > > > > > > > be read into PB2NC, you can set a file window around
the
> > > > > > > > valid
> > > time
> > > > > to
> > > > > > > > allow files to be passed into the app. More
information on
> > > > > > > > that
> > > > > > > > functionality is here:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > windows-to-find-valid-files
> > > > > > > >
> > > > > > > > If you are still having trouble getting this
configured,
> > > > > > > > you
> > > could
> > > > > send
> > > > > > > me
> > > > > > > > the paths of the files you are trying to read and what
data
> > > > > > > > is
> > > > > > contained
> > > > > > > in
> > > > > > > > them so I can give you suggestions on what to set for
the
> > > > > > configuration.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted
upon.
> > > > > > > > > Transaction: Given to mccabe (George McCabe) by
jpresto
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: How to set time for hourly 12-24 hours
data
> > > > > > > > > ?
> > > > > > > > > Owner: mccabe
> > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This transaction appears to have no content
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> > > > > > > > feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> > > > obliged to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Apr 14 15:41:29 2021
Hi Zhuming,
It looks like you are requesting the first time in the 12 hour file
for
each run:
FCST_VAR1_LEVELS = (0,*,*)
...
FCST_VAR2_LEVELS = (0,*,*)
The first dimension in these files is the time offset. You will need
the
correct index that corresponds to each valid time. Fortunately, MET
makes
this easy by allowing you to specify the valid time for a time
dimension
explicitly and it will find the correct data. You can set:
FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
...
FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
And it will substitute the valid time for each run. Let me know if you
have
any questions.
Thanks,
George
On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> Thanks for your email, I just would like to send you for help.
>
> I already resolved the last problem with the WRF forecast, thank you
for
> your help.
>
> I also need to verify the MPAS model forecast. These 12 hour
> MAPS forecasts are stored in a one file, such as
> 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> 2015071512/mpas_wrf_cf_2015071512.nc
>
> It seems MET is doing fine on the first 2 hours, but no matched
pairs for
> later hours.
> I'm not sure what's wrong with it? the time is not matched?
>
> see the log file
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
>
> Thanks,
> Zhuming
>
> On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > Just following up on this ticket. Were you able to get this
resolved? Let
> > me know if you need more assistance.
> >
> > Thanks,
> > George
> >
> > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > Hi Zhuming,
> > >
> > > Typically directories that contain data for an entire day are
named
> > > with
> > > YYYYMMDD, so your filename template tag would be
{valid?fmt=%Y%m%d},
> > > however that may not always be the case, as you are seeing here.
There
> > > are
> > > a couple ways to handle this.
> > >
> > > If you know that the directory will always be YYYYMMDD with 00
at the
> > > end,
> > > you can simply put 00 in the template instead of adding %H in
the tag,
> > > i.e.
> > >
> > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> > >
> > > If the HH value can vary, for instance if there are 2
directories
> > > created
> > > per day (i.e. 2015071400 and 2015071412), then you can use the
> > > truncate
> > > keyword to handle it.
> > >
> > > If directories are created every 24 hours:
> > >
> > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > %d_%H}:00:
> > > 00_cf.nc
> > >
> > > If directories are created every 12 hours:
> > >
> > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > %d_%H}:00:
> > > 00_cf.nc
> > >
> > > Let me know if that doesn't work for you.
> > >
> > > Thanks,
> > > George
> > >
> > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >
> > > > Hi George,
> > > >
> > > > Processing Obs data using "PB2NC" are right now.
> > > >
> > > > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
VALID_END =
> > > > 2015071501. It looks for the forecast data 2015071500/
> > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > > But my forecast data are all put in dir of 2015071500/.
> > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get the
right
> > > > fcst
> > > > data?
> > > >
> > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> > > >
> > > >
> > > > config file: /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > >
> > > > log file:
> > > >
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Zhuming
> > > >
> > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Zhuming,
> > > > >
> > > > > Looking at the log output, it looks like the data is stored
in the
> > > > prepBUFR
> > > > > files differently that you may have thought:
> > > > >
> > > > > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > > > > DEBUG 2: Searching Time Window: 20150715_030000 to
> > > > 20150715_040000
> > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75%
80% 85%
> > > > > 90%
> > > > 95%
> > > > > 100%
> > > > >
> > > > > DEBUG 2: Total Messages processed = 590836
> > > > > DEBUG 2: Rejected based on message type = 0
> > > > > DEBUG 2: Rejected based on station id = 0
> > > > > DEBUG 2: Rejected based on valid time = 590815
> > > > > DEBUG 2: Rejected based on masking grid = 20
> > > > > DEBUG 2: Rejected based on masking polygon = 0
> > > > > DEBUG 2: Rejected based on elevation = 0
> > > > > DEBUG 2: Rejected based on pb report type = 1
> > > > > DEBUG 2: Rejected based on input report type = 0
> > > > > DEBUG 2: Rejected based on instrument type = 0
> > > > > DEBUG 2: Rejected based on zero observations = 0
> > > > > DEBUG 2: Total Messages retained = 0
> > > > > DEBUG 2: Total observations retained or derived = 0
> > > > > DEBUG 1: Obs time between *20150714_210000 and
20150715_030000*
> > > > > WARNING:
> > > > > WARNING: process_pbfile() -> No PrepBufr messages retained
from
> > > > > file:
> > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > > prepbufr.gdas.2015071500.nr
> > > > > WARNING:
> > > > > ERROR :
> > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages
retained.
> > > > Nothing
> > > > > to write.
> > > > > ERROR :
> > > > >
> > > > > The prepbufr.gdas.2015071500.nr file actually contains data
from
> > > > > 20150714_210000 to 20150715_030000, not 20150715_000000 to
> > > > 20150715_060000.
> > > > >
> > > > > To get the correct file for each run, you will have to shift
the
> > > > > valid
> > > > time
> > > > > before truncating it so that it gets the correct file for
each run.
> > > > Instead
> > > > > of setting:
> > > > >
> > > > > PB2NC_INPUT_TEMPLATE =
> > > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > > >
> > > > > You will have to also shift forward 3 hours, i.e.:
> > > > >
> > > > > PB2NC_INPUT_TEMPLATE =
> > > > >
> > > > >
> > > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > > >
> > > > > This says to add 3 hours to the current valid time before
> > > > > truncating to
> > > > the
> > > > > nearest 6 hours.
> > > > >
> > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > > > File: prepbufr.gdas.2015071500.nr
> > > > >
> > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > > > File: prepbufr.gdas.2015071500.nr
> > > > >
> > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > > > File: prepbufr.gdas.2015071500.nr
> > > > >
> > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > > > File: prepbufr.gdas.2015071506.nr
> > > > >
> > > > > Let me know if that doesn't work.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > >
> > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > > >
> > > > > > Hi George,
> > > > > >
> > > > > > Now makes me confused is why processed obs of several
times are
> > > > missing.
> > > > > > There has files of time at 00,01,02, but no 03,04,05; has
> > > > > > 06,07,08, but
> > > > > no
> > > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but
with
> > > > > > 21,no
> > > > 22,23.
> > > > > > From yours and mines:
> > > > > >
> > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > > > >
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
pbs.2015071500.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
pbs.2015071501.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
pbs.2015071502.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
pbs.2015071506.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
pbs.2015071507.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
pbs.2015071508.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
pbs.2015071512.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
pbs.2015071513.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
pbs.2015071514.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
pbs.2015071515.nc
> > > > > >
> > > > > >
> > > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > > >
> > > > > > total 96112
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> pbs.2015071500.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> pbs.2015071501.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> pbs.2015071502.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> pbs.2015071506.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> pbs.2015071507.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> pbs.2015071508.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> pbs.2015071512.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> pbs.2015071513.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> pbs.2015071514.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> pbs.2015071518.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> pbs.2015071519.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> pbs.2015071520.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> pbs.2015071521.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> pbs.2015071600.nc
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Zhuming
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> > > > > > <met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Zhuming,
> > > > > > >
> > > > > > > Thanks for the information. You will need to use the
truncate
> > > > > > > keyword
> > > > > to
> > > > > > > find the files, then set the valid beg/end range to 1
hour.
> > > > > > > This is
> > > > > what
> > > > > > I
> > > > > > > changed from your PB2NC1.conf config file:
> > > > > > >
> > > > > > >
> > > > > > > *PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > > >
> > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > > >
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > > >
> > > > > > > My config file is on cheyenne here:
> > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > > >
> > > > > > > I just started running it and the output is being
written here:
> > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > > >
> > > > > > >
> > > > > > > For PointStat, you will want to loop over the same
times. I
> > > > > > > noticed
> > > > you
> > > > > > > specify message type in the VAR<n>_OPTIONS, but you do
not need
> > > > > > > to do
> > > > > > this
> > > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable that
defines
> > > > > > > this.
> > > > > This
> > > > > > > is what I changed in your PointStat1.conf file:
> > > > > > >
> > > > > > > *VALID_END = 2015071600*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > > > =POINT_STAT_OBS_VALID_BEG
> > > > > > =*
> > > > > > >
> > > > > > >
> > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
message_type =
> > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > > >
> > > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > > > > >
> > > > > > > Or you can view my file in the same directory. I usually
try to
> > > > > > > test
> > > > > > with a
> > > > > > > single run time to make sure it is working properly, so
I set
> > > > VALID_END
> > > > > > > back to 2015071500. I ran it and it produced an output
file,
> > > > > > > but
> > > > there
> > > > > > were
> > > > > > > no values besides the header. I checked the log file and
saw
> > > > > > > this:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
> > > > > > > observation type
> > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > using 0
> > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > Observations
> > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > 0DEBUG 3:
> > > > > > > Rejected: obs type = 390528DEBUG 3: Rejected:
valid time
> > > > =
> > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected: off
> > > > > > > the
> > > > grid
> > > > > > =
> > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > > level
> > > > > > mismatch
> > > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > > message
> > > > > > type
> > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected: bad
> > > > fcst
> > > > > > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
> > > > > > > Rejected:
> > > > bad
> > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates =
0DEBUG
> > > > > > > 2:DEBUG 2:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10,
for
> > > > observation
> > > > > > type
> > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > using 0
> > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > Observations
> > > > > > > processed = 393373*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected: obs
> > > > > > > type
> > > > > > =
> > > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3:
Rejected:
> > > > > > > bad
> > > > obs
> > > > > > > value = 0DEBUG 3: Rejected: off the grid =
9240DEBUG 3:
> > > > Rejected:
> > > > > > > topography = 0DEBUG 3: Rejected: level mismatch =
> > > > > > > 1168DEBUG 3:
> > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected: message
type
> > > > > > > =
> > > > > 0DEBUG
> > > > > > 3:
> > > > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst
value
> > > > > > > =
> > > > > 0DEBUG
> > > > > > 3:
> > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad
climo stdev
> > > > > > > =
> > > > > 0DEBUG
> > > > > > 3:
> > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10,
for
> > > > observation
> > > > > > type
> > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > using 0
> > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > Observations
> > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > 0DEBUG 3:
> > > > > > > Rejected: obs type = 382965DEBUG 3: Rejected:
valid time
> > > > =
> > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected: off
> > > > > > > the
> > > > grid
> > > > > > =
> > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > > level
> > > > > > mismatch
> > > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > > message
> > > > > > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG
3:
> > > > > > > Rejected:
> > > > bad
> > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG 3:
> > > > Rejected:
> > > > > > bad
> > > > > > > climo stdev = 0*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > 2:DEBUG 1: Output file:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This describes the reason why each observation was
rejected.
> > > > > > > Some
> > > > > > > observations didn't match the message type you
requested, some
> > > > > > > were
> > > > not
> > > > > > > within the forecast grid, and some did not match the
vertical
> > > > > > > level.
> > > > > You
> > > > > > > will have to make adjustments to your settings so that
the data
> > > > matches
> > > > > > up
> > > > > > > properly. Maybe the grid you are using in the PB2NC
config file
> > > > > > > does
> > > > > not
> > > > > > > match the forecast grid and you are missing values. If
that is
> > > > > > > the
> > > > > case,
> > > > > > > you will have to reconfigure and rerun PB2NC to get
output on
> > > > > > > the
> > > > > correct
> > > > > > > gr5id. Maybe the observation points don't line up with
the
> > > > > > > vertical
> > > > > level
> > > > > > > of the model fields. You can use a range of level values
for
> > > > > > > the
> > > > > > > observation data to allow data within that range, i.e.
Z0-10.
> > > > > > >
> > > > > > > Let me know if you are still having issues getting this
to work
> > > > > > > and I
> > > > > can
> > > > > > > help out.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi George,
> > > > > > > >
> > > > > > > > Thanks for your reply!
> > > > > > > >
> > > > > > > > I'm still confused, I need to output hourly obs data
to do
> > > > > > > > hourly
> > > > > > > > verification.
> > > > > > > > The original GDAS Prepbufr data has 4 files one day,
should
> > > > > > > > with 6
> > > > > > hours
> > > > > > > > obs data in one file.
> > > > > > > >
> > > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs time
is 00?
> > > > > > > > or 6
> > > > > times
> > > > > > > of
> > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > > > > >
> > > > > > > > my config file :
> > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Zhuming
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Zhuming,
> > > > > > > > >
> > > > > > > > > To run the tools hourly you would set the
VALID_INCREMENT
> > > > > > > > > value
> > > > for
> > > > > > 1H.
> > > > > > > > > Here is an example that runs once per hour for a day
from
> > > > > 2015-07-15
> > > > > > 0Z
> > > > > > > > to
> > > > > > > > > 2017-07-16 0Z:
> > > > > > > > >
> > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > > > >
> > > > > > > > > VALID_BEG = 2015071500
> > > > > > > > >
> > > > > > > > > VALID_END = 2015071600
> > > > > > > > >
> > > > > > > > > VALID_INCREMENT = 1H
> > > > > > > > >
> > > > > > > > > LEAD_SEQ = 0
> > > > > > > > >
> > > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to
find
> > > > > > > > > files
> > > > whose
> > > > > > > > > filenames match the time you need.
> > > > > > > > >
> > > > > > > > > To read prepBUFR observations for 12 hours, you will
want
> > > > > > > > > to set
> > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the
range
> > > > > > > > > around
> > > > > the
> > > > > > > > valid
> > > > > > > > > time to process. These values depend on what is
contained
> > > > > > > > > inside
> > > > > your
> > > > > > > > > prepBUFR files. If you want to use data from the
valid time
> > > > > > > > > of
> > > > the
> > > > > > > > filename
> > > > > > > > > backwards 12 hours, you would set:
> > > > > > > > >
> > > > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H
}
> > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > > > >
> > > > > > > > > For a runtime of 2015071512, this would be
substituted as:
> > > > > > > > >
> > > > > > > > > valid_beg: 20150715_00
> > > > > > > > > valid_end: 20150715_12
> > > > > > > > >
> > > > > > > > > If you want to create an output file for each hour
in PB2NC
> > > > > > > > > you
> > > > > would
> > > > > > > > need
> > > > > > > > > to make sure that the hour is specified in the
> > > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > > > >
> > > > > > > > > PointStat has similar config variables to set the
valid beg
> > > > > > > > > and
> > > > end
> > > > > > > range
> > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > > > >
> > > > > > > > > If your prepBUFR files are separated out into
multiple
> > > > > > > > > files that
> > > > > > need
> > > > > > > to
> > > > > > > > > be read into PB2NC, you can set a file window around
the
> > > > > > > > > valid
> > > > time
> > > > > > to
> > > > > > > > > allow files to be passed into the app. More
information on
> > > > > > > > > that
> > > > > > > > > functionality is here:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > > windows-to-find-valid-files
> > > > > > > > >
> > > > > > > > > If you are still having trouble getting this
configured,
> > > > > > > > > you
> > > > could
> > > > > > send
> > > > > > > > me
> > > > > > > > > the paths of the files you are trying to read and
what data
> > > > > > > > > is
> > > > > > > contained
> > > > > > > > in
> > > > > > > > > them so I can give you suggestions on what to set
for the
> > > > > > > configuration.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted
upon.
> > > > > > > > > > Transaction: Given to mccabe (George McCabe) by
jpresto
> > > > > > > > > > Queue: met_help
> > > > > > > > > > Subject: How to set time for hourly 12-24
hours data
> > > > > > > > > > ?
> > > > > > > > > > Owner: mccabe
> > > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > > > Status: new
> > > > > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This transaction appears to have no content
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > > > > > > > > feel
> > > > > > obliged
> > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > > > > obliged to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> >
> >
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Wed Apr 14 16:13:51 2021
Hi George,
It still can't find the correct time. I tried
FCST_VAR1_LEVELS = (*,*,*)
FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
Thanks!
Zhuming
On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> It looks like you are requesting the first time in the 12 hour file
for
> each run:
>
> FCST_VAR1_LEVELS = (0,*,*)
> ...
> FCST_VAR2_LEVELS = (0,*,*)
>
> The first dimension in these files is the time offset. You will need
the
> correct index that corresponds to each valid time. Fortunately, MET
makes
> this easy by allowing you to specify the valid time for a time
dimension
> explicitly and it will find the correct data. You can set:
>
> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> ...
> FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
>
> And it will substitute the valid time for each run. Let me know if
you have
> any questions.
>
> Thanks,
> George
>
> On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > Thanks for your email, I just would like to send you for help.
> >
> > I already resolved the last problem with the WRF forecast, thank
you for
> > your help.
> >
> > I also need to verify the MPAS model forecast. These 12 hour
> > MAPS forecasts are stored in a one file, such as
> > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> > 2015071512/mpas_wrf_cf_2015071512.nc
> >
> > It seems MET is doing fine on the first 2 hours, but no matched
pairs for
> > later hours.
> > I'm not sure what's wrong with it? the time is not matched?
> >
> > see the log file
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> >
> > Thanks,
> > Zhuming
> >
> > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > Just following up on this ticket. Were you able to get this
resolved?
> Let
> > > me know if you need more assistance.
> > >
> > > Thanks,
> > > George
> > >
> > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > > Hi Zhuming,
> > > >
> > > > Typically directories that contain data for an entire day are
named
> > > > with
> > > > YYYYMMDD, so your filename template tag would be
{valid?fmt=%Y%m%d},
> > > > however that may not always be the case, as you are seeing
here.
> There
> > > > are
> > > > a couple ways to handle this.
> > > >
> > > > If you know that the directory will always be YYYYMMDD with 00
at the
> > > > end,
> > > > you can simply put 00 in the template instead of adding %H in
the
> tag,
> > > > i.e.
> > > >
> > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> > > >
> > > > If the HH value can vary, for instance if there are 2
directories
> > > > created
> > > > per day (i.e. 2015071400 and 2015071412), then you can use
the
> > > > truncate
> > > > keyword to handle it.
> > > >
> > > > If directories are created every 24 hours:
> > > >
> > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > > %d_%H}:00:
> > > > 00_cf.nc
> > > >
> > > > If directories are created every 12 hours:
> > > >
> > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > > %d_%H}:00:
> > > > 00_cf.nc
> > > >
> > > > Let me know if that doesn't work for you.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >
> > > > > Hi George,
> > > > >
> > > > > Processing Obs data using "PB2NC" are right now.
> > > > >
> > > > > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
> VALID_END =
> > > > > 2015071501. It looks for the forecast data 2015071500/
> > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > > > But my forecast data are all put in dir of 2015071500/.
> > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get
the right
> > > > > fcst
> > > > > data?
> > > > >
> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> 00_cf.nc
> > > > >
> > > > >
> > > > > config file:
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > >
> > > > > log file:
> > > > >
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Zhuming
> > > > >
> > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Zhuming,
> > > > > >
> > > > > > Looking at the log output, it looks like the data is
stored in
> the
> > > > > prepBUFR
> > > > > > files differently that you may have thought:
> > > > > >
> > > > > > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > > > > > DEBUG 2: Searching Time Window: 20150715_030000 to
> > > > > 20150715_040000
> > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75%
80%
> 85%
> > > > > > 90%
> > > > > 95%
> > > > > > 100%
> > > > > >
> > > > > > DEBUG 2: Total Messages processed = 590836
> > > > > > DEBUG 2: Rejected based on message type = 0
> > > > > > DEBUG 2: Rejected based on station id = 0
> > > > > > DEBUG 2: Rejected based on valid time = 590815
> > > > > > DEBUG 2: Rejected based on masking grid = 20
> > > > > > DEBUG 2: Rejected based on masking polygon = 0
> > > > > > DEBUG 2: Rejected based on elevation = 0
> > > > > > DEBUG 2: Rejected based on pb report type = 1
> > > > > > DEBUG 2: Rejected based on input report type = 0
> > > > > > DEBUG 2: Rejected based on instrument type = 0
> > > > > > DEBUG 2: Rejected based on zero observations = 0
> > > > > > DEBUG 2: Total Messages retained = 0
> > > > > > DEBUG 2: Total observations retained or derived = 0
> > > > > > DEBUG 1: Obs time between *20150714_210000 and
20150715_030000*
> > > > > > WARNING:
> > > > > > WARNING: process_pbfile() -> No PrepBufr messages retained
from
> > > > > > file:
> > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > > > prepbufr.gdas.2015071500.nr
> > > > > > WARNING:
> > > > > > ERROR :
> > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages
> retained.
> > > > > Nothing
> > > > > > to write.
> > > > > > ERROR :
> > > > > >
> > > > > > The prepbufr.gdas.2015071500.nr file actually contains
data from
> > > > > > 20150714_210000 to 20150715_030000, not 20150715_000000 to
> > > > > 20150715_060000.
> > > > > >
> > > > > > To get the correct file for each run, you will have to
shift the
> > > > > > valid
> > > > > time
> > > > > > before truncating it so that it gets the correct file for
each
> run.
> > > > > Instead
> > > > > > of setting:
> > > > > >
> > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > >
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > > > >
> > > > > > You will have to also shift forward 3 hours, i.e.:
> > > > > >
> > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > >
> > > > > >
> > > > >
> > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > > > >
> > > > > > This says to add 3 hours to the current valid time before
> > > > > > truncating to
> > > > > the
> > > > > > nearest 6 hours.
> > > > > >
> > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > >
> > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > >
> > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > >
> > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > > > > File: prepbufr.gdas.2015071506.nr
> > > > > >
> > > > > > Let me know if that doesn't work.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > >
> > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >
> > > > > > >
> > > > > > > Hi George,
> > > > > > >
> > > > > > > Now makes me confused is why processed obs of several
times are
> > > > > missing.
> > > > > > > There has files of time at 00,01,02, but no 03,04,05;
has
> > > > > > > 06,07,08, but
> > > > > > no
> > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but
with
> > > > > > > 21,no
> > > > > 22,23.
> > > > > > > From yours and mines:
> > > > > > >
> > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > > > > >
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> pbs.2015071500.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> pbs.2015071501.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> pbs.2015071502.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> pbs.2015071506.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> pbs.2015071507.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> pbs.2015071508.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> pbs.2015071512.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> pbs.2015071513.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> pbs.2015071514.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> pbs.2015071515.nc
> > > > > > >
> > > > > > >
> > > > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > > > >
> > > > > > > total 96112
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> > pbs.2015071500.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> > pbs.2015071501.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> > pbs.2015071502.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> > pbs.2015071506.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> > pbs.2015071507.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> > pbs.2015071508.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> > pbs.2015071512.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> > pbs.2015071513.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> > pbs.2015071514.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> > pbs.2015071518.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> > pbs.2015071519.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> > pbs.2015071520.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> > pbs.2015071521.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> > pbs.2015071600.nc
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Zhuming
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> > > > > > > <met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Zhuming,
> > > > > > > >
> > > > > > > > Thanks for the information. You will need to use the
truncate
> > > > > > > > keyword
> > > > > > to
> > > > > > > > find the files, then set the valid beg/end range to 1
hour.
> > > > > > > > This is
> > > > > > what
> > > > > > > I
> > > > > > > > changed from your PB2NC1.conf config file:
> > > > > > > >
> > > > > > > >
> > > > > > > > *PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > > > >
> > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > > > >
> > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > > > >
> > > > > > > > My config file is on cheyenne here:
> > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > > > >
> > > > > > > > I just started running it and the output is being
written
> here:
> > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > > > >
> > > > > > > >
> > > > > > > > For PointStat, you will want to loop over the same
times. I
> > > > > > > > noticed
> > > > > you
> > > > > > > > specify message type in the VAR<n>_OPTIONS, but you do
not
> need
> > > > > > > > to do
> > > > > > > this
> > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable that
> defines
> > > > > > > > this.
> > > > > > This
> > > > > > > > is what I changed in your PointStat1.conf file:
> > > > > > > >
> > > > > > > > *VALID_END = 2015071600*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > > > > =POINT_STAT_OBS_VALID_BEG
> > > > > > > =*
> > > > > > > >
> > > > > > > >
> > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
message_type =
> > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > > > >
> > > > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > > > > > >
> > > > > > > > Or you can view my file in the same directory. I
usually try
> to
> > > > > > > > test
> > > > > > > with a
> > > > > > > > single run time to make sure it is working properly,
so I set
> > > > > VALID_END
> > > > > > > > back to 2015071500. I ran it and it produced an output
file,
> > > > > > > > but
> > > > > there
> > > > > > > were
> > > > > > > > no values besides the header. I checked the log file
and saw
> > > > > > > > this:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
> > > > > > > > observation type
> > > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > > using 0
> > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > > Observations
> > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > > 0DEBUG 3:
> > > > > > > > Rejected: obs type = 390528DEBUG 3: Rejected:
valid
> time
> > > > > =
> > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected: off
> > > > > > > > the
> > > > > grid
> > > > > > > =
> > > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > > > level
> > > > > > > mismatch
> > > > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > > > message
> > > > > > > type
> > > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected:
> bad
> > > > > fcst
> > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG
3:
> > > > > > > > Rejected:
> > > > > bad
> > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates =
0DEBUG
> > > > > > > > 2:DEBUG 2:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10,
for
> > > > > observation
> > > > > > > type
> > > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > > using 0
> > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > > Observations
> > > > > > > > processed = 393373*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected: obs
> > > > > > > > type
> > > > > > > =
> > > > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3:
> Rejected:
> > > > > > > > bad
> > > > > obs
> > > > > > > > value = 0DEBUG 3: Rejected: off the grid =
9240DEBUG 3:
> > > > > Rejected:
> > > > > > > > topography = 0DEBUG 3: Rejected: level mismatch
=
> > > > > > > > 1168DEBUG 3:
> > > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected:
message type
> > > > > > > > =
> > > > > > 0DEBUG
> > > > > > > 3:
> > > > > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
> value
> > > > > > > > =
> > > > > > 0DEBUG
> > > > > > > 3:
> > > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad
climo
> stdev
> > > > > > > > =
> > > > > > 0DEBUG
> > > > > > > 3:
> > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10,
for
> > > > > observation
> > > > > > > type
> > > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > > using 0
> > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > > Observations
> > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > > 0DEBUG 3:
> > > > > > > > Rejected: obs type = 382965DEBUG 3: Rejected:
valid
> time
> > > > > =
> > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected: off
> > > > > > > > the
> > > > > grid
> > > > > > > =
> > > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > > > level
> > > > > > > mismatch
> > > > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
> Rejected:
> > > > > message
> > > > > > > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG
3:
> > > > > > > > Rejected:
> > > > > bad
> > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG 3:
> > > > > Rejected:
> > > > > > > bad
> > > > > > > > climo stdev = 0*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG
2:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > 2:DEBUG 1: Output file:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > This describes the reason why each observation was
rejected.
> > > > > > > > Some
> > > > > > > > observations didn't match the message type you
requested,
> some
> > > > > > > > were
> > > > > not
> > > > > > > > within the forecast grid, and some did not match the
vertical
> > > > > > > > level.
> > > > > > You
> > > > > > > > will have to make adjustments to your settings so that
the
> data
> > > > > matches
> > > > > > > up
> > > > > > > > properly. Maybe the grid you are using in the PB2NC
config
> file
> > > > > > > > does
> > > > > > not
> > > > > > > > match the forecast grid and you are missing values. If
that
> is
> > > > > > > > the
> > > > > > case,
> > > > > > > > you will have to reconfigure and rerun PB2NC to get
output on
> > > > > > > > the
> > > > > > correct
> > > > > > > > gr5id. Maybe the observation points don't line up with
the
> > > > > > > > vertical
> > > > > > level
> > > > > > > > of the model fields. You can use a range of level
values for
> > > > > > > > the
> > > > > > > > observation data to allow data within that range, i.e.
Z0-10.
> > > > > > > >
> > > > > > > > Let me know if you are still having issues getting
this to
> work
> > > > > > > > and I
> > > > > > can
> > > > > > > > help out.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi George,
> > > > > > > > >
> > > > > > > > > Thanks for your reply!
> > > > > > > > >
> > > > > > > > > I'm still confused, I need to output hourly obs data
to do
> > > > > > > > > hourly
> > > > > > > > > verification.
> > > > > > > > > The original GDAS Prepbufr data has 4 files one day,
should
> > > > > > > > > with 6
> > > > > > > hours
> > > > > > > > > obs data in one file.
> > > > > > > > >
> > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs time
is
> 00?
> > > > > > > > > or 6
> > > > > > times
> > > > > > > > of
> > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > > > > > >
> > > > > > > > > my config file :
> > > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Zhuming
> > > > > > > > >
> > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Zhuming,
> > > > > > > > > >
> > > > > > > > > > To run the tools hourly you would set the
VALID_INCREMENT
> > > > > > > > > > value
> > > > > for
> > > > > > > 1H.
> > > > > > > > > > Here is an example that runs once per hour for a
day from
> > > > > > 2015-07-15
> > > > > > > 0Z
> > > > > > > > > to
> > > > > > > > > > 2017-07-16 0Z:
> > > > > > > > > >
> > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > > > > >
> > > > > > > > > > VALID_BEG = 2015071500
> > > > > > > > > >
> > > > > > > > > > VALID_END = 2015071600
> > > > > > > > > >
> > > > > > > > > > VALID_INCREMENT = 1H
> > > > > > > > > >
> > > > > > > > > > LEAD_SEQ = 0
> > > > > > > > > >
> > > > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to
find
> > > > > > > > > > files
> > > > > whose
> > > > > > > > > > filenames match the time you need.
> > > > > > > > > >
> > > > > > > > > > To read prepBUFR observations for 12 hours, you
will want
> > > > > > > > > > to set
> > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify
the
> range
> > > > > > > > > > around
> > > > > > the
> > > > > > > > > valid
> > > > > > > > > > time to process. These values depend on what is
contained
> > > > > > > > > > inside
> > > > > > your
> > > > > > > > > > prepBUFR files. If you want to use data from the
valid
> time
> > > > > > > > > > of
> > > > > the
> > > > > > > > > filename
> > > > > > > > > > backwards 12 hours, you would set:
> > > > > > > > > >
> > > > > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-
12H }
> > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > > > > >
> > > > > > > > > > For a runtime of 2015071512, this would be
substituted
> as:
> > > > > > > > > >
> > > > > > > > > > valid_beg: 20150715_00
> > > > > > > > > > valid_end: 20150715_12
> > > > > > > > > >
> > > > > > > > > > If you want to create an output file for each hour
in
> PB2NC
> > > > > > > > > > you
> > > > > > would
> > > > > > > > > need
> > > > > > > > > > to make sure that the hour is specified in the
> > > > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > > > > >
> > > > > > > > > > PointStat has similar config variables to set the
valid
> beg
> > > > > > > > > > and
> > > > > end
> > > > > > > > range
> > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > > > > >
> > > > > > > > > > If your prepBUFR files are separated out into
multiple
> > > > > > > > > > files that
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > be read into PB2NC, you can set a file window
around the
> > > > > > > > > > valid
> > > > > time
> > > > > > > to
> > > > > > > > > > allow files to be passed into the app. More
information
> on
> > > > > > > > > > that
> > > > > > > > > > functionality is here:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > > > windows-to-find-valid-files
> > > > > > > > > >
> > > > > > > > > > If you are still having trouble getting this
configured,
> > > > > > > > > > you
> > > > > could
> > > > > > > send
> > > > > > > > > me
> > > > > > > > > > the paths of the files you are trying to read and
what
> data
> > > > > > > > > > is
> > > > > > > > contained
> > > > > > > > > in
> > > > > > > > > > them so I can give you suggestions on what to set
for the
> > > > > > > > configuration.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > George
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik
via RT <
> > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was
acted upon.
> > > > > > > > > > > Transaction: Given to mccabe (George McCabe) by
jpresto
> > > > > > > > > > > Queue: met_help
> > > > > > > > > > > Subject: How to set time for hourly 12-24
hours
> data
> > > > > > > > > > > ?
> > > > > > > > > > > Owner: mccabe
> > > > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > > > > Status: new
> > > > > > > > > > > Ticket <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > This transaction appears to have no content
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > 303-497-2768
> > > > > > > > > > ---
> > > > > > > > > > My working day may not be your working day. Please
do not
> > > > > > > > > > feel
> > > > > > > obliged
> > > > > > > > to
> > > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > > > > obliged to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Apr 14 17:29:02 2021
Hi Zhuming,
I tried to get this to work on cheyenne with your data, but was not
successful. I am able to read the data with plot_data_plane and
generate an
image, so I think there may be a bug in the code. I have forwarded
info to
one of our C++ developers to look into it. I will report back when I
know
more.
Thanks,
George
On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> It still can't find the correct time. I tried
>
> FCST_VAR1_LEVELS = (*,*,*)
> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
>
> Thanks!
>
> Zhuming
>
> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > It looks like you are requesting the first time in the 12 hour
file for
> > each run:
> >
> > FCST_VAR1_LEVELS = (0,*,*)
> > ...
> > FCST_VAR2_LEVELS = (0,*,*)
> >
> > The first dimension in these files is the time offset. You will
need the
> > correct index that corresponds to each valid time. Fortunately,
MET makes
> > this easy by allowing you to specify the valid time for a time
dimension
> > explicitly and it will find the correct data. You can set:
> >
> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > ...
> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> >
> > And it will substitute the valid time for each run. Let me know if
you
> have
> > any questions.
> >
> > Thanks,
> > George
> >
> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > Thanks for your email, I just would like to send you for help.
> > >
> > > I already resolved the last problem with the WRF forecast, thank
you
> for
> > > your help.
> > >
> > > I also need to verify the MPAS model forecast. These 12 hour
> > > MAPS forecasts are stored in a one file, such as
> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> > > 2015071512/mpas_wrf_cf_2015071512.nc
> > >
> > > It seems MET is doing fine on the first 2 hours, but no matched
pairs
> for
> > > later hours.
> > > I'm not sure what's wrong with it? the time is not matched?
> > >
> > > see the log file
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > > config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> > >
> > > Thanks,
> > > Zhuming
> > >
> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > Just following up on this ticket. Were you able to get this
resolved?
> > Let
> > > > me know if you need more assistance.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > > > Hi Zhuming,
> > > > >
> > > > > Typically directories that contain data for an entire day
are named
> > > > > with
> > > > > YYYYMMDD, so your filename template tag would be
> {valid?fmt=%Y%m%d},
> > > > > however that may not always be the case, as you are seeing
here.
> > There
> > > > > are
> > > > > a couple ways to handle this.
> > > > >
> > > > > If you know that the directory will always be YYYYMMDD with
00 at
> the
> > > > > end,
> > > > > you can simply put 00 in the template instead of adding %H
in the
> > tag,
> > > > > i.e.
> > > > >
> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> 00_cf.nc
> > > > >
> > > > > If the HH value can vary, for instance if there are 2
directories
> > > > > created
> > > > > per day (i.e. 2015071400 and 2015071412), then you can use
the
> > > > > truncate
> > > > > keyword to handle it.
> > > > >
> > > > > If directories are created every 24 hours:
> > > > >
> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-
%m-
> > > > > %d_%H}:00:
> > > > > 00_cf.nc
> > > > >
> > > > > If directories are created every 12 hours:
> > > > >
> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-
%m-
> > > > > %d_%H}:00:
> > > > > 00_cf.nc
> > > > >
> > > > > Let me know if that doesn't work for you.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > > >
> > > > > > Hi George,
> > > > > >
> > > > > > Processing Obs data using "PB2NC" are right now.
> > > > > >
> > > > > > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
> > VALID_END =
> > > > > > 2015071501. It looks for the forecast data 2015071500/
> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > > > > But my forecast data are all put in dir of 2015071500/.
> > > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get
the
> right
> > > > > > fcst
> > > > > > data?
> > > > > >
> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> > 00_cf.nc
> > > > > >
> > > > > >
> > > > > > config file:
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > >
> > > > > > log file:
> > > > > >
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Zhuming
> > > > > >
> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > > > > > <met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Zhuming,
> > > > > > >
> > > > > > > Looking at the log output, it looks like the data is
stored in
> > the
> > > > > > prepBUFR
> > > > > > > files differently that you may have thought:
> > > > > > >
> > > > > > > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > > > > > > DEBUG 2: Searching Time Window: 20150715_030000
to
> > > > > > 20150715_040000
> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70%
75% 80%
> > 85%
> > > > > > > 90%
> > > > > > 95%
> > > > > > > 100%
> > > > > > >
> > > > > > > DEBUG 2: Total Messages processed = 590836
> > > > > > > DEBUG 2: Rejected based on message type = 0
> > > > > > > DEBUG 2: Rejected based on station id = 0
> > > > > > > DEBUG 2: Rejected based on valid time = 590815
> > > > > > > DEBUG 2: Rejected based on masking grid = 20
> > > > > > > DEBUG 2: Rejected based on masking polygon = 0
> > > > > > > DEBUG 2: Rejected based on elevation = 0
> > > > > > > DEBUG 2: Rejected based on pb report type = 1
> > > > > > > DEBUG 2: Rejected based on input report type = 0
> > > > > > > DEBUG 2: Rejected based on instrument type = 0
> > > > > > > DEBUG 2: Rejected based on zero observations = 0
> > > > > > > DEBUG 2: Total Messages retained = 0
> > > > > > > DEBUG 2: Total observations retained or derived = 0
> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
20150715_030000*
> > > > > > > WARNING:
> > > > > > > WARNING: process_pbfile() -> No PrepBufr messages
retained from
> > > > > > > file:
> > > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > > > > prepbufr.gdas.2015071500.nr
> > > > > > > WARNING:
> > > > > > > ERROR :
> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages
> > retained.
> > > > > > Nothing
> > > > > > > to write.
> > > > > > > ERROR :
> > > > > > >
> > > > > > > The prepbufr.gdas.2015071500.nr file actually contains
data
> from
> > > > > > > 20150714_210000 to 20150715_030000, not 20150715_000000
to
> > > > > > 20150715_060000.
> > > > > > >
> > > > > > > To get the correct file for each run, you will have to
shift
> the
> > > > > > > valid
> > > > > > time
> > > > > > > before truncating it so that it gets the correct file
for each
> > run.
> > > > > > Instead
> > > > > > > of setting:
> > > > > > >
> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > > >
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > > > > >
> > > > > > > You will have to also shift forward 3 hours, i.e.:
> > > > > > >
> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > > > > >
> > > > > > > This says to add 3 hours to the current valid time
before
> > > > > > > truncating to
> > > > > > the
> > > > > > > nearest 6 hours.
> > > > > > >
> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > >
> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > >
> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > >
> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > > > > > File: prepbufr.gdas.2015071506.nr
> > > > > > >
> > > > > > > Let me know if that doesn't work.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >
> > > > > > > >
> > > > > > > > Hi George,
> > > > > > > >
> > > > > > > > Now makes me confused is why processed obs of several
times
> are
> > > > > > missing.
> > > > > > > > There has files of time at 00,01,02, but no 03,04,05;
has
> > > > > > > > 06,07,08, but
> > > > > > > no
> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20,
but with
> > > > > > > > 21,no
> > > > > > 22,23.
> > > > > > > > From yours and mines:
> > > > > > > >
> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > > > > > >
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> > pbs.2015071500.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> > pbs.2015071501.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> > pbs.2015071502.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> > pbs.2015071506.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> > pbs.2015071507.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> > pbs.2015071508.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> > pbs.2015071512.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> > pbs.2015071513.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> > pbs.2015071514.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> > pbs.2015071515.nc
> > > > > > > >
> > > > > > > >
> > > > > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > > > > >
> > > > > > > > total 96112
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> > > pbs.2015071500.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> > > pbs.2015071501.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> > > pbs.2015071502.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> > > pbs.2015071506.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> > > pbs.2015071507.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> > > pbs.2015071508.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> > > pbs.2015071512.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> > > pbs.2015071513.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> > > pbs.2015071514.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> > > pbs.2015071518.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> > > pbs.2015071519.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> > > pbs.2015071520.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> > > pbs.2015071521.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> > > pbs.2015071600.nc
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Zhuming
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> > > > > > > > <met_help at ucar.edu
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Zhuming,
> > > > > > > > >
> > > > > > > > > Thanks for the information. You will need to use the
> truncate
> > > > > > > > > keyword
> > > > > > > to
> > > > > > > > > find the files, then set the valid beg/end range to
1 hour.
> > > > > > > > > This is
> > > > > > > what
> > > > > > > > I
> > > > > > > > > changed from your PB2NC1.conf config file:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > > > > >
> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > > > > >
> > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > > > > >
> > > > > > > > > My config file is on cheyenne here:
> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > > > > >
> > > > > > > > > I just started running it and the output is being
written
> > here:
> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > For PointStat, you will want to loop over the same
times. I
> > > > > > > > > noticed
> > > > > > you
> > > > > > > > > specify message type in the VAR<n>_OPTIONS, but you
do not
> > need
> > > > > > > > > to do
> > > > > > > > this
> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable
that
> > defines
> > > > > > > > > this.
> > > > > > > This
> > > > > > > > > is what I changed in your PointStat1.conf file:
> > > > > > > > >
> > > > > > > > > *VALID_END = 2015071600*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > > > > > =POINT_STAT_OBS_VALID_BEG
> > > > > > > > =*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
message_type
> =
> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > > > > >
> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > > > > > > >
> > > > > > > > > Or you can view my file in the same directory. I
usually
> try
> > to
> > > > > > > > > test
> > > > > > > > with a
> > > > > > > > > single run time to make sure it is working properly,
so I
> set
> > > > > > VALID_END
> > > > > > > > > back to 2015071500. I ran it and it produced an
output
> file,
> > > > > > > > > but
> > > > > > there
> > > > > > > > were
> > > > > > > > > no values besides the header. I checked the log file
and
> saw
> > > > > > > > > this:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
> > > > > > > > > observation type
> > > > > > > > > ADPSFC, over region FULL, for interpolation method
> BILIN(4),
> > > > > > > > > using 0
> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
> 3:
> > > > > > > Observations
> > > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > > > 0DEBUG 3:
> > > > > > > > > Rejected: obs type = 390528DEBUG 3: Rejected:
valid
> > time
> > > > > > =
> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected:
> off
> > > > > > > > > the
> > > > > > grid
> > > > > > > > =
> > > > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3:
> Rejected:
> > > > > > > > > level
> > > > > > > > mismatch
> > > > > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
> Rejected:
> > > > > > message
> > > > > > > > type
> > > > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected:
> > bad
> > > > > > fcst
> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG 3:
> > > > > > > > > Rejected:
> > > > > > bad
> > > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates =
0DEBUG
> > > > > > > > > 2:DEBUG 2:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus
UGRD/Z10, for
> > > > > > observation
> > > > > > > > type
> > > > > > > > > ADPSFC, over region FULL, for interpolation method
> BILIN(4),
> > > > > > > > > using 0
> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
> 3:
> > > > > > > Observations
> > > > > > > > > processed = 393373*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected:
> obs
> > > > > > > > > type
> > > > > > > > =
> > > > > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3:
> > Rejected:
> > > > > > > > > bad
> > > > > > obs
> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid =
9240DEBUG
> 3:
> > > > > > Rejected:
> > > > > > > > > topography = 0DEBUG 3: Rejected: level mismatch
=
> > > > > > > > > 1168DEBUG 3:
> > > > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected:
message
> type
> > > > > > > > > =
> > > > > > > 0DEBUG
> > > > > > > > 3:
> > > > > > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
> > value
> > > > > > > > > =
> > > > > > > 0DEBUG
> > > > > > > > 3:
> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad
climo
> > stdev
> > > > > > > > > =
> > > > > > > 0DEBUG
> > > > > > > > 3:
> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus
VGRD/Z10, for
> > > > > > observation
> > > > > > > > type
> > > > > > > > > ADPSFC, over region FULL, for interpolation method
> BILIN(4),
> > > > > > > > > using 0
> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
> 3:
> > > > > > > Observations
> > > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > > > 0DEBUG 3:
> > > > > > > > > Rejected: obs type = 382965DEBUG 3: Rejected:
valid
> > time
> > > > > > =
> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected:
> off
> > > > > > > > > the
> > > > > > grid
> > > > > > > > =
> > > > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3:
> Rejected:
> > > > > > > > > level
> > > > > > > > mismatch
> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
> > Rejected:
> > > > > > message
> > > > > > > > > type = 0DEBUG 3: Rejected: masking region =
0DEBUG 3:
> > > > > > > > > Rejected:
> > > > > > bad
> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG
> 3:
> > > > > > Rejected:
> > > > > > > > bad
> > > > > > > > > climo stdev = 0*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG
2:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > > 2:DEBUG 1: Output file:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This describes the reason why each observation was
> rejected.
> > > > > > > > > Some
> > > > > > > > > observations didn't match the message type you
requested,
> > some
> > > > > > > > > were
> > > > > > not
> > > > > > > > > within the forecast grid, and some did not match the
> vertical
> > > > > > > > > level.
> > > > > > > You
> > > > > > > > > will have to make adjustments to your settings so
that the
> > data
> > > > > > matches
> > > > > > > > up
> > > > > > > > > properly. Maybe the grid you are using in the PB2NC
config
> > file
> > > > > > > > > does
> > > > > > > not
> > > > > > > > > match the forecast grid and you are missing values.
If that
> > is
> > > > > > > > > the
> > > > > > > case,
> > > > > > > > > you will have to reconfigure and rerun PB2NC to get
output
> on
> > > > > > > > > the
> > > > > > > correct
> > > > > > > > > gr5id. Maybe the observation points don't line up
with the
> > > > > > > > > vertical
> > > > > > > level
> > > > > > > > > of the model fields. You can use a range of level
values
> for
> > > > > > > > > the
> > > > > > > > > observation data to allow data within that range,
i.e.
> Z0-10.
> > > > > > > > >
> > > > > > > > > Let me know if you are still having issues getting
this to
> > work
> > > > > > > > > and I
> > > > > > > can
> > > > > > > > > help out.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi George,
> > > > > > > > > >
> > > > > > > > > > Thanks for your reply!
> > > > > > > > > >
> > > > > > > > > > I'm still confused, I need to output hourly obs
data to
> do
> > > > > > > > > > hourly
> > > > > > > > > > verification.
> > > > > > > > > > The original GDAS Prepbufr data has 4 files one
day,
> should
> > > > > > > > > > with 6
> > > > > > > > hours
> > > > > > > > > > obs data in one file.
> > > > > > > > > >
> > > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs
time is
> > 00?
> > > > > > > > > > or 6
> > > > > > > times
> > > > > > > > > of
> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > > > > > > >
> > > > > > > > > > my config file :
> > > > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > > > > > and
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Zhuming
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Zhuming,
> > > > > > > > > > >
> > > > > > > > > > > To run the tools hourly you would set the
> VALID_INCREMENT
> > > > > > > > > > > value
> > > > > > for
> > > > > > > > 1H.
> > > > > > > > > > > Here is an example that runs once per hour for a
day
> from
> > > > > > > 2015-07-15
> > > > > > > > 0Z
> > > > > > > > > > to
> > > > > > > > > > > 2017-07-16 0Z:
> > > > > > > > > > >
> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > > > > > >
> > > > > > > > > > > VALID_BEG = 2015071500
> > > > > > > > > > >
> > > > > > > > > > > VALID_END = 2015071600
> > > > > > > > > > >
> > > > > > > > > > > VALID_INCREMENT = 1H
> > > > > > > > > > >
> > > > > > > > > > > LEAD_SEQ = 0
> > > > > > > > > > >
> > > > > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE
to find
> > > > > > > > > > > files
> > > > > > whose
> > > > > > > > > > > filenames match the time you need.
> > > > > > > > > > >
> > > > > > > > > > > To read prepBUFR observations for 12 hours, you
will
> want
> > > > > > > > > > > to set
> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify
the
> > range
> > > > > > > > > > > around
> > > > > > > the
> > > > > > > > > > valid
> > > > > > > > > > > time to process. These values depend on what is
> contained
> > > > > > > > > > > inside
> > > > > > > your
> > > > > > > > > > > prepBUFR files. If you want to use data from the
valid
> > time
> > > > > > > > > > > of
> > > > > > the
> > > > > > > > > > filename
> > > > > > > > > > > backwards 12 hours, you would set:
> > > > > > > > > > >
> > > > > > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-
12H }
> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > > > > > >
> > > > > > > > > > > For a runtime of 2015071512, this would be
substituted
> > as:
> > > > > > > > > > >
> > > > > > > > > > > valid_beg: 20150715_00
> > > > > > > > > > > valid_end: 20150715_12
> > > > > > > > > > >
> > > > > > > > > > > If you want to create an output file for each
hour in
> > PB2NC
> > > > > > > > > > > you
> > > > > > > would
> > > > > > > > > > need
> > > > > > > > > > > to make sure that the hour is specified in the
> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > > > > > >
> > > > > > > > > > > PointStat has similar config variables to set
the valid
> > beg
> > > > > > > > > > > and
> > > > > > end
> > > > > > > > > range
> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > > > > > >
> > > > > > > > > > > If your prepBUFR files are separated out into
multiple
> > > > > > > > > > > files that
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > be read into PB2NC, you can set a file window
around
> the
> > > > > > > > > > > valid
> > > > > > time
> > > > > > > > to
> > > > > > > > > > > allow files to be passed into the app. More
information
> > on
> > > > > > > > > > > that
> > > > > > > > > > > functionality is here:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > > > > windows-to-find-valid-files
> > > > > > > > > > >
> > > > > > > > > > > If you are still having trouble getting this
> configured,
> > > > > > > > > > > you
> > > > > > could
> > > > > > > > send
> > > > > > > > > > me
> > > > > > > > > > > the paths of the files you are trying to read
and what
> > data
> > > > > > > > > > > is
> > > > > > > > > contained
> > > > > > > > > > in
> > > > > > > > > > > them so I can give you suggestions on what to
set for
> the
> > > > > > > > > configuration.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > George
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik
via RT
> <
> > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was
acted
> upon.
> > > > > > > > > > > > Transaction: Given to mccabe (George McCabe)
by
> jpresto
> > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > Subject: How to set time for hourly 12-24
hours
> > data
> > > > > > > > > > > > ?
> > > > > > > > > > > > Owner: mccabe
> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > > > > > Status: new
> > > > > > > > > > > > Ticket <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > This transaction appears to have no content
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > 303-497-2768
> > > > > > > > > > > ---
> > > > > > > > > > > My working day may not be your working day.
Please do
> not
> > > > > > > > > > > feel
> > > > > > > > obliged
> > > > > > > > > to
> > > > > > > > > > > reply to this email outside of your normal
working
> hours.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > > > > obliged to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Thu Apr 15 13:52:17 2021
Hi Zhuming,
There was indeed a bug in the code. Howard has fixed the issue with
this
pull request: https://github.com/dtcenter/MET/pull/1756
This fix will be included in the MET 10.0.0-beta5 release that is
tentatively scheduled for next week. This beta version can be
installed on
cheyenne so you can test that the update fixes your issue. I will
follow up
and let you know when the beta release is available on cheyenne.
Thanks,
George
On Wed, Apr 14, 2021 at 5:28 PM George McCabe <mccabe at ucar.edu> wrote:
> Hi Zhuming,
>
> I tried to get this to work on cheyenne with your data, but was not
> successful. I am able to read the data with plot_data_plane and
generate an
> image, so I think there may be a bug in the code. I have forwarded
info to
> one of our C++ developers to look into it. I will report back when I
know
> more.
>
> Thanks,
> George
>
> On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>>
>> Hi George,
>>
>> It still can't find the correct time. I tried
>>
>> FCST_VAR1_LEVELS = (*,*,*)
>> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
>> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
>> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
>>
>> Thanks!
>>
>> Zhuming
>>
>> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>> > Hi Zhuming,
>> >
>> > It looks like you are requesting the first time in the 12 hour
file for
>> > each run:
>> >
>> > FCST_VAR1_LEVELS = (0,*,*)
>> > ...
>> > FCST_VAR2_LEVELS = (0,*,*)
>> >
>> > The first dimension in these files is the time offset. You will
need the
>> > correct index that corresponds to each valid time. Fortunately,
MET
>> makes
>> > this easy by allowing you to specify the valid time for a time
dimension
>> > explicitly and it will find the correct data. You can set:
>> >
>> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
>> > ...
>> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
>> >
>> > And it will substitute the valid time for each run. Let me know
if you
>> have
>> > any questions.
>> >
>> > Thanks,
>> > George
>> >
>> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
>> met_help at ucar.edu
>> > >
>> > wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>> > >
>> > > Hi George,
>> > >
>> > > Thanks for your email, I just would like to send you for help.
>> > >
>> > > I already resolved the last problem with the WRF forecast,
thank you
>> for
>> > > your help.
>> > >
>> > > I also need to verify the MPAS model forecast. These 12 hour
>> > > MAPS forecasts are stored in a one file, such as
>> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
>> > > 2015071512/mpas_wrf_cf_2015071512.nc
>> > >
>> > > It seems MET is doing fine on the first 2 hours, but no matched
pairs
>> for
>> > > later hours.
>> > > I'm not sure what's wrong with it? the time is not matched?
>> > >
>> > > see the log file
>> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
>> > > config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
>> > >
>> > > Thanks,
>> > > Zhuming
>> > >
>> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
>> met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Hi Zhuming,
>> > > >
>> > > > Just following up on this ticket. Were you able to get this
>> resolved?
>> > Let
>> > > > me know if you need more assistance.
>> > > >
>> > > > Thanks,
>> > > > George
>> > > >
>> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
>> > > > > Hi Zhuming,
>> > > > >
>> > > > > Typically directories that contain data for an entire day
are
>> named
>> > > > > with
>> > > > > YYYYMMDD, so your filename template tag would be
>> {valid?fmt=%Y%m%d},
>> > > > > however that may not always be the case, as you are seeing
here.
>> > There
>> > > > > are
>> > > > > a couple ways to handle this.
>> > > > >
>> > > > > If you know that the directory will always be YYYYMMDD with
00 at
>> the
>> > > > > end,
>> > > > > you can simply put 00 in the template instead of adding %H
in the
>> > tag,
>> > > > > i.e.
>> > > > >
>> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
>> > > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
>> 00_cf.nc
>> > > > >
>> > > > > If the HH value can vary, for instance if there are 2
directories
>> > > > > created
>> > > > > per day (i.e. 2015071400 and 2015071412), then you can use
the
>> > > > > truncate
>> > > > > keyword to handle it.
>> > > > >
>> > > > > If directories are created every 24 hours:
>> > > > >
>> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
>> > > > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-
%m-
>> > > > > %d_%H}:00:
>> > > > > 00_cf.nc
>> > > > >
>> > > > > If directories are created every 12 hours:
>> > > > >
>> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
>> > > > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-
%m-
>> > > > > %d_%H}:00:
>> > > > > 00_cf.nc
>> > > > >
>> > > > > Let me know if that doesn't work for you.
>> > > > >
>> > > > > Thanks,
>> > > > > George
>> > > > >
>> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
>> > > > > <met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>> > > > > >
>> > > > > > Hi George,
>> > > > > >
>> > > > > > Processing Obs data using "PB2NC" are right now.
>> > > > > >
>> > > > > > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
>> > VALID_END =
>> > > > > > 2015071501. It looks for the forecast data 2015071500/
>> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
>> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
>> > > > > > But my forecast data are all put in dir of 2015071500/.
>> > > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get
the
>> right
>> > > > > > fcst
>> > > > > > data?
>> > > > > >
>> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
>> > > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
>> > 00_cf.nc
>> > > > > >
>> > > > > >
>> > > > > > config file:
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
>> > > > > >
>> > > > > > log file:
>> > > > > >
>> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
>> > > > > >
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Zhuming
>> > > > > >
>> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
>> > > > > > <met_help at ucar.edu>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Zhuming,
>> > > > > > >
>> > > > > > > Looking at the log output, it looks like the data is
stored in
>> > the
>> > > > > > prepBUFR
>> > > > > > > files differently that you may have thought:
>> > > > > > >
>> > > > > > > DEBUG 2: PrepBufr Time Center: 20150715_000000
>> > > > > > > DEBUG 2: Searching Time Window: 20150715_030000
to
>> > > > > > 20150715_040000
>> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70%
75% 80%
>> > 85%
>> > > > > > > 90%
>> > > > > > 95%
>> > > > > > > 100%
>> > > > > > >
>> > > > > > > DEBUG 2: Total Messages processed =
590836
>> > > > > > > DEBUG 2: Rejected based on message type = 0
>> > > > > > > DEBUG 2: Rejected based on station id = 0
>> > > > > > > DEBUG 2: Rejected based on valid time =
590815
>> > > > > > > DEBUG 2: Rejected based on masking grid = 20
>> > > > > > > DEBUG 2: Rejected based on masking polygon = 0
>> > > > > > > DEBUG 2: Rejected based on elevation = 0
>> > > > > > > DEBUG 2: Rejected based on pb report type = 1
>> > > > > > > DEBUG 2: Rejected based on input report type = 0
>> > > > > > > DEBUG 2: Rejected based on instrument type = 0
>> > > > > > > DEBUG 2: Rejected based on zero observations = 0
>> > > > > > > DEBUG 2: Total Messages retained = 0
>> > > > > > > DEBUG 2: Total observations retained or derived = 0
>> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
>> 20150715_030000*
>> > > > > > > WARNING:
>> > > > > > > WARNING: process_pbfile() -> No PrepBufr messages
retained
>> from
>> > > > > > > file:
>> > > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
>> > > > > > > prepbufr.gdas.2015071500.nr
>> > > > > > > WARNING:
>> > > > > > > ERROR :
>> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr
messages
>> > retained.
>> > > > > > Nothing
>> > > > > > > to write.
>> > > > > > > ERROR :
>> > > > > > >
>> > > > > > > The prepbufr.gdas.2015071500.nr file actually contains
data
>> from
>> > > > > > > 20150714_210000 to 20150715_030000, not 20150715_000000
to
>> > > > > > 20150715_060000.
>> > > > > > >
>> > > > > > > To get the correct file for each run, you will have to
shift
>> the
>> > > > > > > valid
>> > > > > > time
>> > > > > > > before truncating it so that it gets the correct file
for each
>> > run.
>> > > > > > Instead
>> > > > > > > of setting:
>> > > > > > >
>> > > > > > > PB2NC_INPUT_TEMPLATE =
>> > > > > > >
>> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
>> > > > > > >
>> > > > > > > You will have to also shift forward 3 hours, i.e.:
>> > > > > > >
>> > > > > > > PB2NC_INPUT_TEMPLATE =
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
>> > > > > > >
>> > > > > > > This says to add 3 hours to the current valid time
before
>> > > > > > > truncating to
>> > > > > > the
>> > > > > > > nearest 6 hours.
>> > > > > > >
>> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
>> > > > > > > File: prepbufr.gdas.2015071500.nr
>> > > > > > >
>> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
>> > > > > > > File: prepbufr.gdas.2015071500.nr
>> > > > > > >
>> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
>> > > > > > > File: prepbufr.gdas.2015071500.nr
>> > > > > > >
>> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
>> > > > > > > File: prepbufr.gdas.2015071506.nr
>> > > > > > >
>> > > > > > > Let me know if that doesn't work.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > George
>> > > > > > >
>> > > > > > >
>> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT
<
>> > > > > > met_help at ucar.edu>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > >
>> > > > > > > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>> > >
>> > > > > > > >
>> > > > > > > > Hi George,
>> > > > > > > >
>> > > > > > > > Now makes me confused is why processed obs of several
times
>> are
>> > > > > > missing.
>> > > > > > > > There has files of time at 00,01,02, but no 03,04,05;
has
>> > > > > > > > 06,07,08, but
>> > > > > > > no
>> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20,
but with
>> > > > > > > > 21,no
>> > > > > > 22,23.
>> > > > > > > > From yours and mines:
>> > > > > > > >
>> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
>> > pbs.2015071500.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
>> > pbs.2015071501.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
>> > pbs.2015071502.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
>> > pbs.2015071506.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
>> > pbs.2015071507.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
>> > pbs.2015071508.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
>> > pbs.2015071512.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
>> > pbs.2015071513.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
>> > pbs.2015071514.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
>> > pbs.2015071515.nc
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
>> > > > > > > >
>> > > > > > > > total 96112
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
>> > > pbs.2015071500.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
>> > > pbs.2015071501.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
>> > > pbs.2015071502.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
>> > > pbs.2015071506.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
>> > > pbs.2015071507.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
>> > > pbs.2015071508.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
>> > > pbs.2015071512.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
>> > > pbs.2015071513.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
>> > > pbs.2015071514.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
>> > > pbs.2015071518.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
>> > > pbs.2015071519.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
>> > > pbs.2015071520.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
>> > > pbs.2015071521.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
>> > > pbs.2015071600.nc
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > >
>> > > > > > > > Zhuming
>> > > > > > > >
>> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
>> > > > > > > > <met_help at ucar.edu
>> > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Zhuming,
>> > > > > > > > >
>> > > > > > > > > Thanks for the information. You will need to use
the
>> truncate
>> > > > > > > > > keyword
>> > > > > > > to
>> > > > > > > > > find the files, then set the valid beg/end range to
1
>> hour.
>> > > > > > > > > This is
>> > > > > > > what
>> > > > > > > > I
>> > > > > > > > > changed from your PB2NC1.conf config file:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
>> =
>> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
>> > > > > > > > >
>> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
>> > > > > > > > >
>> > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
>> > > > > > > > >
>> > > > > > > > > My config file is on cheyenne here:
>> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
>> > > > > > > > >
>> > > > > > > > > I just started running it and the output is being
written
>> > here:
>> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > For PointStat, you will want to loop over the same
times.
>> I
>> > > > > > > > > noticed
>> > > > > > you
>> > > > > > > > > specify message type in the VAR<n>_OPTIONS, but you
do not
>> > need
>> > > > > > > > > to do
>> > > > > > > > this
>> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable
that
>> > defines
>> > > > > > > > > this.
>> > > > > > > This
>> > > > > > > > > is what I changed in your PointStat1.conf file:
>> > > > > > > > >
>> > > > > > > > > *VALID_END = 2015071600*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
>> > > > > > =POINT_STAT_OBS_VALID_BEG
>> > > > > > > > =*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
>> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
>> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
>> message_type =
>> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
>> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
>> > > > > > > > >
>> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
>> > > > > > > > >
>> > > > > > > > > Or you can view my file in the same directory. I
usually
>> try
>> > to
>> > > > > > > > > test
>> > > > > > > > with a
>> > > > > > > > > single run time to make sure it is working
properly, so I
>> set
>> > > > > > VALID_END
>> > > > > > > > > back to 2015071500. I ran it and it produced an
output
>> file,
>> > > > > > > > > but
>> > > > > > there
>> > > > > > > > were
>> > > > > > > > > no values besides the header. I checked the log
file and
>> saw
>> > > > > > > > > this:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2,
for
>> > > > > > > > > observation type
>> > > > > > > > > ADPSFC, over region FULL, for interpolation method
>> BILIN(4),
>> > > > > > > > > using 0
>> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
>> 3:
>> > > > > > > Observations
>> > > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
>> > > > > > > > > 0DEBUG 3:
>> > > > > > > > > Rejected: obs type = 390528DEBUG 3:
Rejected: valid
>> > time
>> > > > > > =
>> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected:
>> off
>> > > > > > > > > the
>> > > > > > grid
>> > > > > > > > =
>> > > > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3:
>> Rejected:
>> > > > > > > > > level
>> > > > > > > > mismatch
>> > > > > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
>> Rejected:
>> > > > > > message
>> > > > > > > > type
>> > > > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
>> Rejected:
>> > bad
>> > > > > > fcst
>> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG 3:
>> > > > > > > > > Rejected:
>> > > > > > bad
>> > > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates =
0DEBUG
>> > > > > > > > > 2:DEBUG 2:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------DEBUG
>> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus
UGRD/Z10, for
>> > > > > > observation
>> > > > > > > > type
>> > > > > > > > > ADPSFC, over region FULL, for interpolation method
>> BILIN(4),
>> > > > > > > > > using 0
>> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
>> 3:
>> > > > > > > Observations
>> > > > > > > > > processed = 393373*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected:
>> obs
>> > > > > > > > > type
>> > > > > > > > =
>> > > > > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG
3:
>> > Rejected:
>> > > > > > > > > bad
>> > > > > > obs
>> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid =
9240DEBUG
>> 3:
>> > > > > > Rejected:
>> > > > > > > > > topography = 0DEBUG 3: Rejected: level
mismatch =
>> > > > > > > > > 1168DEBUG 3:
>> > > > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected:
message
>> type
>> > > > > > > > > =
>> > > > > > > 0DEBUG
>> > > > > > > > 3:
>> > > > > > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
>> > value
>> > > > > > > > > =
>> > > > > > > 0DEBUG
>> > > > > > > > 3:
>> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad
climo
>> > stdev
>> > > > > > > > > =
>> > > > > > > 0DEBUG
>> > > > > > > > 3:
>> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------DEBUG
>> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus
VGRD/Z10, for
>> > > > > > observation
>> > > > > > > > type
>> > > > > > > > > ADPSFC, over region FULL, for interpolation method
>> BILIN(4),
>> > > > > > > > > using 0
>> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
>> 3:
>> > > > > > > Observations
>> > > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
>> > > > > > > > > 0DEBUG 3:
>> > > > > > > > > Rejected: obs type = 382965DEBUG 3:
Rejected: valid
>> > time
>> > > > > > =
>> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected:
>> off
>> > > > > > > > > the
>> > > > > > grid
>> > > > > > > > =
>> > > > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3:
>> Rejected:
>> > > > > > > > > level
>> > > > > > > > mismatch
>> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG
3:
>> > Rejected:
>> > > > > > message
>> > > > > > > > > type = 0DEBUG 3: Rejected: masking region =
0DEBUG 3:
>> > > > > > > > > Rejected:
>> > > > > > bad
>> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean =
>> 0DEBUG 3:
>> > > > > > Rejected:
>> > > > > > > > bad
>> > > > > > > > > climo stdev = 0*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG 2:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------DEBUG
>> > > > > > > > > 2:DEBUG 1: Output file:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > This describes the reason why each observation was
>> rejected.
>> > > > > > > > > Some
>> > > > > > > > > observations didn't match the message type you
requested,
>> > some
>> > > > > > > > > were
>> > > > > > not
>> > > > > > > > > within the forecast grid, and some did not match
the
>> vertical
>> > > > > > > > > level.
>> > > > > > > You
>> > > > > > > > > will have to make adjustments to your settings so
that the
>> > data
>> > > > > > matches
>> > > > > > > > up
>> > > > > > > > > properly. Maybe the grid you are using in the PB2NC
config
>> > file
>> > > > > > > > > does
>> > > > > > > not
>> > > > > > > > > match the forecast grid and you are missing values.
If
>> that
>> > is
>> > > > > > > > > the
>> > > > > > > case,
>> > > > > > > > > you will have to reconfigure and rerun PB2NC to get
>> output on
>> > > > > > > > > the
>> > > > > > > correct
>> > > > > > > > > gr5id. Maybe the observation points don't line up
with the
>> > > > > > > > > vertical
>> > > > > > > level
>> > > > > > > > > of the model fields. You can use a range of level
values
>> for
>> > > > > > > > > the
>> > > > > > > > > observation data to allow data within that range,
i.e.
>> Z0-10.
>> > > > > > > > >
>> > > > > > > > > Let me know if you are still having issues getting
this to
>> > work
>> > > > > > > > > and I
>> > > > > > > can
>> > > > > > > > > help out.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > > George
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via
RT <
>> > > > > > > > met_help at ucar.edu>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Hi George,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for your reply!
>> > > > > > > > > >
>> > > > > > > > > > I'm still confused, I need to output hourly obs
data to
>> do
>> > > > > > > > > > hourly
>> > > > > > > > > > verification.
>> > > > > > > > > > The original GDAS Prepbufr data has 4 files one
day,
>> should
>> > > > > > > > > > with 6
>> > > > > > > > hours
>> > > > > > > > > > obs data in one file.
>> > > > > > > > > >
>> > > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs
time is
>> > 00?
>> > > > > > > > > > or 6
>> > > > > > > times
>> > > > > > > > > of
>> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
>> > > > > > > > > >
>> > > > > > > > > > my config file :
>> > > > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
>> > > > > > > > > > and
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Zhuming
>> > > > > > > > > >
>> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via
RT <
>> > > > > > > met_help at ucar.edu
>> > > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Zhuming,
>> > > > > > > > > > >
>> > > > > > > > > > > To run the tools hourly you would set the
>> VALID_INCREMENT
>> > > > > > > > > > > value
>> > > > > > for
>> > > > > > > > 1H.
>> > > > > > > > > > > Here is an example that runs once per hour for
a day
>> from
>> > > > > > > 2015-07-15
>> > > > > > > > 0Z
>> > > > > > > > > > to
>> > > > > > > > > > > 2017-07-16 0Z:
>> > > > > > > > > > >
>> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
>> > > > > > > > > > >
>> > > > > > > > > > > VALID_BEG = 2015071500
>> > > > > > > > > > >
>> > > > > > > > > > > VALID_END = 2015071600
>> > > > > > > > > > >
>> > > > > > > > > > > VALID_INCREMENT = 1H
>> > > > > > > > > > >
>> > > > > > > > > > > LEAD_SEQ = 0
>> > > > > > > > > > >
>> > > > > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE
to
>> find
>> > > > > > > > > > > files
>> > > > > > whose
>> > > > > > > > > > > filenames match the time you need.
>> > > > > > > > > > >
>> > > > > > > > > > > To read prepBUFR observations for 12 hours, you
will
>> want
>> > > > > > > > > > > to set
>> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to
specify the
>> > range
>> > > > > > > > > > > around
>> > > > > > > the
>> > > > > > > > > > valid
>> > > > > > > > > > > time to process. These values depend on what is
>> contained
>> > > > > > > > > > > inside
>> > > > > > > your
>> > > > > > > > > > > prepBUFR files. If you want to use data from
the valid
>> > time
>> > > > > > > > > > > of
>> > > > > > the
>> > > > > > > > > > filename
>> > > > > > > > > > > backwards 12 hours, you would set:
>> > > > > > > > > > >
>> > > > > > > > > > > PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H?shift=-12H }
>> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
>> > > > > > > > > > >
>> > > > > > > > > > > For a runtime of 2015071512, this would be
substituted
>> > as:
>> > > > > > > > > > >
>> > > > > > > > > > > valid_beg: 20150715_00
>> > > > > > > > > > > valid_end: 20150715_12
>> > > > > > > > > > >
>> > > > > > > > > > > If you want to create an output file for each
hour in
>> > PB2NC
>> > > > > > > > > > > you
>> > > > > > > would
>> > > > > > > > > > need
>> > > > > > > > > > > to make sure that the hour is specified in the
>> > > > > > > PB2NC_OUTPUT_TEMPLATE.
>> > > > > > > > > > >
>> > > > > > > > > > > PointStat has similar config variables to set
the
>> valid
>> > beg
>> > > > > > > > > > > and
>> > > > > > end
>> > > > > > > > > range
>> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
>> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
>> > > > > > > > > > >
>> > > > > > > > > > > If your prepBUFR files are separated out into
multiple
>> > > > > > > > > > > files that
>> > > > > > > > need
>> > > > > > > > > to
>> > > > > > > > > > > be read into PB2NC, you can set a file window
around
>> the
>> > > > > > > > > > > valid
>> > > > > > time
>> > > > > > > > to
>> > > > > > > > > > > allow files to be passed into the app. More
>> information
>> > on
>> > > > > > > > > > > that
>> > > > > > > > > > > functionality is here:
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
>> > > > > > windows-to-find-valid-files
>> > > > > > > > > > >
>> > > > > > > > > > > If you are still having trouble getting this
>> configured,
>> > > > > > > > > > > you
>> > > > > > could
>> > > > > > > > send
>> > > > > > > > > > me
>> > > > > > > > > > > the paths of the files you are trying to read
and what
>> > data
>> > > > > > > > > > > is
>> > > > > > > > > contained
>> > > > > > > > > > in
>> > > > > > > > > > > them so I can give you suggestions on what to
set for
>> the
>> > > > > > > > > configuration.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > > George
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik
via
>> RT <
>> > > > > > > > > > met_help at ucar.edu>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was
acted
>> upon.
>> > > > > > > > > > > > Transaction: Given to mccabe (George McCabe)
by
>> jpresto
>> > > > > > > > > > > > Queue: met_help
>> > > > > > > > > > > > Subject: How to set time for hourly 12-
24 hours
>> > data
>> > > > > > > > > > > > ?
>> > > > > > > > > > > > Owner: mccabe
>> > > > > > > > > > > > Requestors: zhuming at ucar.edu
>> > > > > > > > > > > > Status: new
>> > > > > > > > > > > > Ticket <URL:
>> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>> > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > This transaction appears to have no content
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > > George McCabe - Software Engineer III
>> > > > > > > > > > > National Center for Atmospheric Research
>> > > > > > > > > > > Research Applications Laboratory
>> > > > > > > > > > > 303-497-2768
>> > > > > > > > > > > ---
>> > > > > > > > > > > My working day may not be your working day.
Please do
>> not
>> > > > > > > > > > > feel
>> > > > > > > > obliged
>> > > > > > > > > to
>> > > > > > > > > > > reply to this email outside of your normal
working
>> hours.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > George McCabe - Software Engineer III
>> > > > > > > > > National Center for Atmospheric Research
>> > > > > > > > > Research Applications Laboratory
>> > > > > > > > > 303-497-2768
>> > > > > > > > > ---
>> > > > > > > > > My working day may not be your working day. Please
do not
>> > feel
>> > > > > > obliged
>> > > > > > > to
>> > > > > > > > > reply to this email outside of your normal working
hours.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > George McCabe - Software Engineer III
>> > > > > > > National Center for Atmospheric Research
>> > > > > > > Research Applications Laboratory
>> > > > > > > 303-497-2768
>> > > > > > > ---
>> > > > > > > My working day may not be your working day. Please do
not feel
>> > > > > > > obliged to
>> > > > > > > reply to this email outside of your normal working
hours.
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> > --
>> > George McCabe - Software Engineer III
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > 303-497-2768
>> > ---
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>> >
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Thu Apr 22 09:52:34 2021
Hi George,
My verification result shows that there is a big error on a non-
standard
level, so my question is:
Is the forecast grid match to Obs, or obs station match to Forecast? I
need
to do is the forecast grid interpolated to the obs location and
vertical
levels. So how to set the levels?
Thanks,
Zhuming
FCST_VAR3_NAME = v_10m_tr
FCST_VAR3_LEVELS = (0,*,*)
FCST_VAR3_THRESH = >=5, <5
OBS_VAR3_NAME = VGRD
OBS_VAR3_LEVELS = Z10
OBS_VAR3_THRESH = >=5, <5
OBS_VAR3_OPTIONS = message_type = ["ADPSFC"];
FCST_VAR3_NAME = v_tr_p
FCST_VAR3_LEVELS =
(0,0,*,*),(0,1,*,*),(0,2,*,*),(0,3,*,*),(0,4,*,*),(0,5,*,*),(0,6,*,*),(0,7,*,*),(0,8,*,*),(0,9,*,*)
FCST_VAR3_THRESH = >=5, <5
OBS_VAR3_NAME = VGRD
OBS_VAR3_LEVELS =
P900-950,P825-875,P675-725,P575-625,P475-525,P375-425,P275-325,P225-
275,P175-225,P75-125
OBS_VAR3_THRESH = >=5, <5
OBS_VAR3_OPTIONS = message_type = ["ADPUPA"];
On Thu, Apr 15, 2021 at 1:52 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> There was indeed a bug in the code. Howard has fixed the issue with
this
> pull request: https://github.com/dtcenter/MET/pull/1756
> This fix will be included in the MET 10.0.0-beta5 release that is
> tentatively scheduled for next week. This beta version can be
installed on
> cheyenne so you can test that the update fixes your issue. I will
follow up
> and let you know when the beta release is available on cheyenne.
>
> Thanks,
> George
>
> On Wed, Apr 14, 2021 at 5:28 PM George McCabe <mccabe at ucar.edu>
wrote:
>
> > Hi Zhuming,
> >
> > I tried to get this to work on cheyenne with your data, but was
not
> > successful. I am able to read the data with plot_data_plane and
generate
> an
> > image, so I think there may be a bug in the code. I have forwarded
info
> to
> > one of our C++ developers to look into it. I will report back when
I know
> > more.
> >
> > Thanks,
> > George
> >
> > On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu>
> > wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >>
> >> Hi George,
> >>
> >> It still can't find the correct time. I tried
> >>
> >> FCST_VAR1_LEVELS = (*,*,*)
> >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
> >>
> >> Thanks!
> >>
> >> Zhuming
> >>
> >> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT
<met_help at ucar.edu
> >
> >> wrote:
> >>
> >> > Hi Zhuming,
> >> >
> >> > It looks like you are requesting the first time in the 12 hour
file
> for
> >> > each run:
> >> >
> >> > FCST_VAR1_LEVELS = (0,*,*)
> >> > ...
> >> > FCST_VAR2_LEVELS = (0,*,*)
> >> >
> >> > The first dimension in these files is the time offset. You will
need
> the
> >> > correct index that corresponds to each valid time. Fortunately,
MET
> >> makes
> >> > this easy by allowing you to specify the valid time for a time
> dimension
> >> > explicitly and it will find the correct data. You can set:
> >> >
> >> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> >> > ...
> >> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> >> >
> >> > And it will substitute the valid time for each run. Let me know
if you
> >> have
> >> > any questions.
> >> >
> >> > Thanks,
> >> > George
> >> >
> >> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
> >> met_help at ucar.edu
> >> > >
> >> > wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> >> > >
> >> > > Hi George,
> >> > >
> >> > > Thanks for your email, I just would like to send you for
help.
> >> > >
> >> > > I already resolved the last problem with the WRF forecast,
thank you
> >> for
> >> > > your help.
> >> > >
> >> > > I also need to verify the MPAS model forecast. These 12 hour
> >> > > MAPS forecasts are stored in a one file, such as
> >> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> >> > > 2015071512/mpas_wrf_cf_2015071512.nc
> >> > >
> >> > > It seems MET is doing fine on the first 2 hours, but no
matched
> pairs
> >> for
> >> > > later hours.
> >> > > I'm not sure what's wrong with it? the time is not matched?
> >> > >
> >> > > see the log file
> >> > >
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> >> > > config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> >> > >
> >> > > Thanks,
> >> > > Zhuming
> >> > >
> >> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
> >> met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > > Hi Zhuming,
> >> > > >
> >> > > > Just following up on this ticket. Were you able to get this
> >> resolved?
> >> > Let
> >> > > > me know if you need more assistance.
> >> > > >
> >> > > > Thanks,
> >> > > > George
> >> > > >
> >> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> >> > > > > Hi Zhuming,
> >> > > > >
> >> > > > > Typically directories that contain data for an entire day
are
> >> named
> >> > > > > with
> >> > > > > YYYYMMDD, so your filename template tag would be
> >> {valid?fmt=%Y%m%d},
> >> > > > > however that may not always be the case, as you are
seeing here.
> >> > There
> >> > > > > are
> >> > > > > a couple ways to handle this.
> >> > > > >
> >> > > > > If you know that the directory will always be YYYYMMDD
with 00
> at
> >> the
> >> > > > > end,
> >> > > > > you can simply put 00 in the template instead of adding
%H in
> the
> >> > tag,
> >> > > > > i.e.
> >> > > > >
> >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> >> > > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> >> 00_cf.nc
> >> > > > >
> >> > > > > If the HH value can vary, for instance if there are 2
> directories
> >> > > > > created
> >> > > > > per day (i.e. 2015071400 and 2015071412), then you can
use the
> >> > > > > truncate
> >> > > > > keyword to handle it.
> >> > > > >
> >> > > > > If directories are created every 24 hours:
> >> > > > >
> >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> >> > > > >
{valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> >> > > > > %d_%H}:00:
> >> > > > > 00_cf.nc
> >> > > > >
> >> > > > > If directories are created every 12 hours:
> >> > > > >
> >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> >> > > > >
{valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> >> > > > > %d_%H}:00:
> >> > > > > 00_cf.nc
> >> > > > >
> >> > > > > Let me know if that doesn't work for you.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > George
> >> > > > >
> >> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> >> > > > > <met_help at ucar.edu>
> >> > > > > wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >
> >> > > > > >
> >> > > > > > Hi George,
> >> > > > > >
> >> > > > > > Processing Obs data using "PB2NC" are right now.
> >> > > > > >
> >> > > > > > Now I'm using "PointStat" , I set VALID_BEG =
2015071500,
> >> > VALID_END =
> >> > > > > > 2015071501. It looks for the forecast data 2015071500/
> >> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> >> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> >> > > > > > But my forecast data are all put in dir of 2015071500/.
> >> > > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to
get the
> >> right
> >> > > > > > fcst
> >> > > > > > data?
> >> > > > > >
> >> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> >> > > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> >> > 00_cf.nc
> >> > > > > >
> >> > > > > >
> >> > > > > > config file:
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> >> > > > > >
> >> > > > > > log file:
> >> > > > > >
> >> > >
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> >> > > > > >
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > >
> >> > > > > > Zhuming
> >> > > > > >
> >> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> >> > > > > > <met_help at ucar.edu>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Zhuming,
> >> > > > > > >
> >> > > > > > > Looking at the log output, it looks like the data is
stored
> in
> >> > the
> >> > > > > > prepBUFR
> >> > > > > > > files differently that you may have thought:
> >> > > > > > >
> >> > > > > > > DEBUG 2: PrepBufr Time Center:
20150715_000000
> >> > > > > > > DEBUG 2: Searching Time Window:
20150715_030000 to
> >> > > > > > 20150715_040000
> >> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65%
70% 75%
> 80%
> >> > 85%
> >> > > > > > > 90%
> >> > > > > > 95%
> >> > > > > > > 100%
> >> > > > > > >
> >> > > > > > > DEBUG 2: Total Messages processed =
590836
> >> > > > > > > DEBUG 2: Rejected based on message type = 0
> >> > > > > > > DEBUG 2: Rejected based on station id = 0
> >> > > > > > > DEBUG 2: Rejected based on valid time =
590815
> >> > > > > > > DEBUG 2: Rejected based on masking grid = 20
> >> > > > > > > DEBUG 2: Rejected based on masking polygon = 0
> >> > > > > > > DEBUG 2: Rejected based on elevation = 0
> >> > > > > > > DEBUG 2: Rejected based on pb report type = 1
> >> > > > > > > DEBUG 2: Rejected based on input report type = 0
> >> > > > > > > DEBUG 2: Rejected based on instrument type = 0
> >> > > > > > > DEBUG 2: Rejected based on zero observations = 0
> >> > > > > > > DEBUG 2: Total Messages retained = 0
> >> > > > > > > DEBUG 2: Total observations retained or derived = 0
> >> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
> >> 20150715_030000*
> >> > > > > > > WARNING:
> >> > > > > > > WARNING: process_pbfile() -> No PrepBufr messages
retained
> >> from
> >> > > > > > > file:
> >> > > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> >> > > > > > > prepbufr.gdas.2015071500.nr
> >> > > > > > > WARNING:
> >> > > > > > > ERROR :
> >> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr
messages
> >> > retained.
> >> > > > > > Nothing
> >> > > > > > > to write.
> >> > > > > > > ERROR :
> >> > > > > > >
> >> > > > > > > The prepbufr.gdas.2015071500.nr file actually
contains data
> >> from
> >> > > > > > > 20150714_210000 to 20150715_030000, not
20150715_000000 to
> >> > > > > > 20150715_060000.
> >> > > > > > >
> >> > > > > > > To get the correct file for each run, you will have
to shift
> >> the
> >> > > > > > > valid
> >> > > > > > time
> >> > > > > > > before truncating it so that it gets the correct file
for
> each
> >> > run.
> >> > > > > > Instead
> >> > > > > > > of setting:
> >> > > > > > >
> >> > > > > > > PB2NC_INPUT_TEMPLATE =
> >> > > > > > >
> >> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> >> > > > > > >
> >> > > > > > > You will have to also shift forward 3 hours, i.e.:
> >> > > > > > >
> >> > > > > > > PB2NC_INPUT_TEMPLATE =
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> >> > > > > > >
> >> > > > > > > This says to add 3 hours to the current valid time
before
> >> > > > > > > truncating to
> >> > > > > > the
> >> > > > > > > nearest 6 hours.
> >> > > > > > >
> >> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> >> > > > > > > File: prepbufr.gdas.2015071500.nr
> >> > > > > > >
> >> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> >> > > > > > > File: prepbufr.gdas.2015071500.nr
> >> > > > > > >
> >> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> >> > > > > > > File: prepbufr.gdas.2015071500.nr
> >> > > > > > >
> >> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> >> > > > > > > File: prepbufr.gdas.2015071506.nr
> >> > > > > > >
> >> > > > > > > Let me know if that doesn't work.
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > George
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via
RT <
> >> > > > > > met_help at ucar.edu>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > >
> >> > > > > > > > <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >> > >
> >> > > > > > > >
> >> > > > > > > > Hi George,
> >> > > > > > > >
> >> > > > > > > > Now makes me confused is why processed obs of
several
> times
> >> are
> >> > > > > > missing.
> >> > > > > > > > There has files of time at 00,01,02, but no
03,04,05; has
> >> > > > > > > > 06,07,08, but
> >> > > > > > > no
> >> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20,
but
> with
> >> > > > > > > > 21,no
> >> > > > > > 22,23.
> >> > > > > > > > From yours and mines:
> >> > > > > > > >
> >> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> >> > pbs.2015071500.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> >> > pbs.2015071501.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> >> > pbs.2015071502.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> >> > pbs.2015071506.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> >> > pbs.2015071507.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> >> > pbs.2015071508.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> >> > pbs.2015071512.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> >> > pbs.2015071513.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> >> > pbs.2015071514.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> >> > pbs.2015071515.nc
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> >> > > > > > > >
> >> > > > > > > > total 96112
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> >> > > pbs.2015071500.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> >> > > pbs.2015071501.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> >> > > pbs.2015071502.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> >> > > pbs.2015071506.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> >> > > pbs.2015071507.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> >> > > pbs.2015071508.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> >> > > pbs.2015071512.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> >> > > pbs.2015071513.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> >> > > pbs.2015071514.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> >> > > pbs.2015071518.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> >> > > pbs.2015071519.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> >> > > pbs.2015071520.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> >> > > pbs.2015071521.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> >> > > pbs.2015071600.nc
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > >
> >> > > > > > > > Zhuming
> >> > > > > > > >
> >> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> >> > > > > > > > <met_help at ucar.edu
> >> > > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Zhuming,
> >> > > > > > > > >
> >> > > > > > > > > Thanks for the information. You will need to use
the
> >> truncate
> >> > > > > > > > > keyword
> >> > > > > > > to
> >> > > > > > > > > find the files, then set the valid beg/end range
to 1
> >> hour.
> >> > > > > > > > > This is
> >> > > > > > > what
> >> > > > > > > > I
> >> > > > > > > > > changed from your PB2NC1.conf config file:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *PB2NC_VALID_BEGIN =
> {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
> >> =
> >> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> >> > > > > > > > >
> >> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> >> > > > > > > > >
> >> > > >
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> >> > > > > > > > >
> >> > > > > > > > > My config file is on cheyenne here:
> >> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> >> > > > > > > > >
> >> > > > > > > > > I just started running it and the output is being
> written
> >> > here:
> >> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > For PointStat, you will want to loop over the
same
> times.
> >> I
> >> > > > > > > > > noticed
> >> > > > > > you
> >> > > > > > > > > specify message type in the VAR<n>_OPTIONS, but
you do
> not
> >> > need
> >> > > > > > > > > to do
> >> > > > > > > > this
> >> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable
that
> >> > defines
> >> > > > > > > > > this.
> >> > > > > > > This
> >> > > > > > > > > is what I changed in your PointStat1.conf file:
> >> > > > > > > > >
> >> > > > > > > > > *VALID_END = 2015071600*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> >> > > > > > =POINT_STAT_OBS_VALID_BEG
> >> > > > > > > > =*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> >> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> >> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
> >> message_type =
> >> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> >> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> >> > > > > > > > >
> >> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> >> > > > > > > > >
> >> > > > > > > > > Or you can view my file in the same directory. I
usually
> >> try
> >> > to
> >> > > > > > > > > test
> >> > > > > > > > with a
> >> > > > > > > > > single run time to make sure it is working
properly, so
> I
> >> set
> >> > > > > > VALID_END
> >> > > > > > > > > back to 2015071500. I ran it and it produced an
output
> >> file,
> >> > > > > > > > > but
> >> > > > > > there
> >> > > > > > > > were
> >> > > > > > > > > no values besides the header. I checked the log
file and
> >> saw
> >> > > > > > > > > this:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2,
for
> >> > > > > > > > > observation type
> >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> >> BILIN(4),
> >> > > > > > > > > using 0
> >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> 0DEBUG
> >> 3:
> >> > > > > > > Observations
> >> > > > > > > > > processed = 393373DEBUG 3: Rejected: station
id
> =
> >> > > > > > > > > 0DEBUG 3:
> >> > > > > > > > > Rejected: obs type = 390528DEBUG 3:
Rejected:
> valid
> >> > time
> >> > > > > > =
> >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
> Rejected:
> >> off
> >> > > > > > > > > the
> >> > > > > > grid
> >> > > > > > > > =
> >> > > > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG
3:
> >> Rejected:
> >> > > > > > > > > level
> >> > > > > > > > mismatch
> >> > > > > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG
3:
> >> Rejected:
> >> > > > > > message
> >> > > > > > > > type
> >> > > > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
> >> Rejected:
> >> > bad
> >> > > > > > fcst
> >> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG 3:
> >> > > > > > > > > Rejected:
> >> > > > > > bad
> >> > > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates
=
> 0DEBUG
> >> > > > > > > > > 2:DEBUG 2:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
--------------------------------------------------------------------------------DEBUG
> >> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus
UGRD/Z10,
> for
> >> > > > > > observation
> >> > > > > > > > type
> >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> >> BILIN(4),
> >> > > > > > > > > using 0
> >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> 0DEBUG
> >> 3:
> >> > > > > > > Observations
> >> > > > > > > > > processed = 393373*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
> Rejected:
> >> obs
> >> > > > > > > > > type
> >> > > > > > > > =
> >> > > > > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG
3:
> >> > Rejected:
> >> > > > > > > > > bad
> >> > > > > > obs
> >> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid =
> 9240DEBUG
> >> 3:
> >> > > > > > Rejected:
> >> > > > > > > > > topography = 0DEBUG 3: Rejected: level
mismatch =
> >> > > > > > > > > 1168DEBUG 3:
> >> > > > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected:
message
> >> type
> >> > > > > > > > > =
> >> > > > > > > 0DEBUG
> >> > > > > > > > 3:
> >> > > > > > > > > Rejected: masking region = 0DEBUG 3: Rejected:
bad fcst
> >> > value
> >> > > > > > > > > =
> >> > > > > > > 0DEBUG
> >> > > > > > > > 3:
> >> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected:
bad
> climo
> >> > stdev
> >> > > > > > > > > =
> >> > > > > > > 0DEBUG
> >> > > > > > > > 3:
> >> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
--------------------------------------------------------------------------------DEBUG
> >> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus
VGRD/Z10,
> for
> >> > > > > > observation
> >> > > > > > > > type
> >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> >> BILIN(4),
> >> > > > > > > > > using 0
> >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> 0DEBUG
> >> 3:
> >> > > > > > > Observations
> >> > > > > > > > > processed = 393373DEBUG 3: Rejected: station
id
> =
> >> > > > > > > > > 0DEBUG 3:
> >> > > > > > > > > Rejected: obs type = 382965DEBUG 3:
Rejected:
> valid
> >> > time
> >> > > > > > =
> >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
> Rejected:
> >> off
> >> > > > > > > > > the
> >> > > > > > grid
> >> > > > > > > > =
> >> > > > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG
3:
> >> Rejected:
> >> > > > > > > > > level
> >> > > > > > > > mismatch
> >> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG
3:
> >> > Rejected:
> >> > > > > > message
> >> > > > > > > > > type = 0DEBUG 3: Rejected: masking region =
0DEBUG
> 3:
> >> > > > > > > > > Rejected:
> >> > > > > > bad
> >> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean
=
> >> 0DEBUG 3:
> >> > > > > > Rejected:
> >> > > > > > > > bad
> >> > > > > > > > > climo stdev = 0*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG 2:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
--------------------------------------------------------------------------------DEBUG
> >> > > > > > > > > 2:DEBUG 1: Output file:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > This describes the reason why each observation
was
> >> rejected.
> >> > > > > > > > > Some
> >> > > > > > > > > observations didn't match the message type you
> requested,
> >> > some
> >> > > > > > > > > were
> >> > > > > > not
> >> > > > > > > > > within the forecast grid, and some did not match
the
> >> vertical
> >> > > > > > > > > level.
> >> > > > > > > You
> >> > > > > > > > > will have to make adjustments to your settings so
that
> the
> >> > data
> >> > > > > > matches
> >> > > > > > > > up
> >> > > > > > > > > properly. Maybe the grid you are using in the
PB2NC
> config
> >> > file
> >> > > > > > > > > does
> >> > > > > > > not
> >> > > > > > > > > match the forecast grid and you are missing
values. If
> >> that
> >> > is
> >> > > > > > > > > the
> >> > > > > > > case,
> >> > > > > > > > > you will have to reconfigure and rerun PB2NC to
get
> >> output on
> >> > > > > > > > > the
> >> > > > > > > correct
> >> > > > > > > > > gr5id. Maybe the observation points don't line up
with
> the
> >> > > > > > > > > vertical
> >> > > > > > > level
> >> > > > > > > > > of the model fields. You can use a range of level
values
> >> for
> >> > > > > > > > > the
> >> > > > > > > > > observation data to allow data within that range,
i.e.
> >> Z0-10.
> >> > > > > > > > >
> >> > > > > > > > > Let me know if you are still having issues
getting this
> to
> >> > work
> >> > > > > > > > > and I
> >> > > > > > > can
> >> > > > > > > > > help out.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > > George
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu
via RT
> <
> >> > > > > > > > met_help at ucar.edu>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Hi George,
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks for your reply!
> >> > > > > > > > > >
> >> > > > > > > > > > I'm still confused, I need to output hourly obs
data
> to
> >> do
> >> > > > > > > > > > hourly
> >> > > > > > > > > > verification.
> >> > > > > > > > > > The original GDAS Prepbufr data has 4 files one
day,
> >> should
> >> > > > > > > > > > with 6
> >> > > > > > > > hours
> >> > > > > > > > > > obs data in one file.
> >> > > > > > > > > >
> >> > > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs
time
> is
> >> > 00?
> >> > > > > > > > > > or 6
> >> > > > > > > times
> >> > > > > > > > > of
> >> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> >> > > > > > > > > >
> >> > > > > > > > > > my config file :
> >> > > > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> >> > > > > > > > > > and
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Zhuming
> >> > > > > > > > > >
> >> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe
via RT <
> >> > > > > > > met_help at ucar.edu
> >> > > > > > > > >
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Zhuming,
> >> > > > > > > > > > >
> >> > > > > > > > > > > To run the tools hourly you would set the
> >> VALID_INCREMENT
> >> > > > > > > > > > > value
> >> > > > > > for
> >> > > > > > > > 1H.
> >> > > > > > > > > > > Here is an example that runs once per hour
for a day
> >> from
> >> > > > > > > 2015-07-15
> >> > > > > > > > 0Z
> >> > > > > > > > > > to
> >> > > > > > > > > > > 2017-07-16 0Z:
> >> > > > > > > > > > >
> >> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> >> > > > > > > > > > >
> >> > > > > > > > > > > VALID_BEG = 2015071500
> >> > > > > > > > > > >
> >> > > > > > > > > > > VALID_END = 2015071600
> >> > > > > > > > > > >
> >> > > > > > > > > > > VALID_INCREMENT = 1H
> >> > > > > > > > > > >
> >> > > > > > > > > > > LEAD_SEQ = 0
> >> > > > > > > > > > >
> >> > > > > > > > > > > Each run time will use the
PB2NC_INPUT_TEMPLATE to
> >> find
> >> > > > > > > > > > > files
> >> > > > > > whose
> >> > > > > > > > > > > filenames match the time you need.
> >> > > > > > > > > > >
> >> > > > > > > > > > > To read prepBUFR observations for 12 hours,
you will
> >> want
> >> > > > > > > > > > > to set
> >> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to
specify the
> >> > range
> >> > > > > > > > > > > around
> >> > > > > > > the
> >> > > > > > > > > > valid
> >> > > > > > > > > > > time to process. These values depend on what
is
> >> contained
> >> > > > > > > > > > > inside
> >> > > > > > > your
> >> > > > > > > > > > > prepBUFR files. If you want to use data from
the
> valid
> >> > time
> >> > > > > > > > > > > of
> >> > > > > > the
> >> > > > > > > > > > filename
> >> > > > > > > > > > > backwards 12 hours, you would set:
> >> > > > > > > > > > >
> >> > > > > > > > > > > PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H?shift=-12H
> }
> >> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> >> > > > > > > > > > >
> >> > > > > > > > > > > For a runtime of 2015071512, this would be
> substituted
> >> > as:
> >> > > > > > > > > > >
> >> > > > > > > > > > > valid_beg: 20150715_00
> >> > > > > > > > > > > valid_end: 20150715_12
> >> > > > > > > > > > >
> >> > > > > > > > > > > If you want to create an output file for each
hour
> in
> >> > PB2NC
> >> > > > > > > > > > > you
> >> > > > > > > would
> >> > > > > > > > > > need
> >> > > > > > > > > > > to make sure that the hour is specified in
the
> >> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> >> > > > > > > > > > >
> >> > > > > > > > > > > PointStat has similar config variables to set
the
> >> valid
> >> > beg
> >> > > > > > > > > > > and
> >> > > > > > end
> >> > > > > > > > > range
> >> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> >> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> >> > > > > > > > > > >
> >> > > > > > > > > > > If your prepBUFR files are separated out into
> multiple
> >> > > > > > > > > > > files that
> >> > > > > > > > need
> >> > > > > > > > > to
> >> > > > > > > > > > > be read into PB2NC, you can set a file window
around
> >> the
> >> > > > > > > > > > > valid
> >> > > > > > time
> >> > > > > > > > to
> >> > > > > > > > > > > allow files to be passed into the app. More
> >> information
> >> > on
> >> > > > > > > > > > > that
> >> > > > > > > > > > > functionality is here:
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> >> > > > > > windows-to-find-valid-files
> >> > > > > > > > > > >
> >> > > > > > > > > > > If you are still having trouble getting this
> >> configured,
> >> > > > > > > > > > > you
> >> > > > > > could
> >> > > > > > > > send
> >> > > > > > > > > > me
> >> > > > > > > > > > > the paths of the files you are trying to read
and
> what
> >> > data
> >> > > > > > > > > > > is
> >> > > > > > > > > contained
> >> > > > > > > > > > in
> >> > > > > > > > > > > them so I can give you suggestions on what to
set
> for
> >> the
> >> > > > > > > > > configuration.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks,
> >> > > > > > > > > > > George
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie
Prestopnik via
> >> RT <
> >> > > > > > > > > > met_help at ucar.edu>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was
acted
> >> upon.
> >> > > > > > > > > > > > Transaction: Given to mccabe (George
McCabe) by
> >> jpresto
> >> > > > > > > > > > > > Queue: met_help
> >> > > > > > > > > > > > Subject: How to set time for hourly
12-24
> hours
> >> > data
> >> > > > > > > > > > > > ?
> >> > > > > > > > > > > > Owner: mccabe
> >> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> >> > > > > > > > > > > > Status: new
> >> > > > > > > > > > > > Ticket <URL:
> >> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >> > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > This transaction appears to have no content
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > > George McCabe - Software Engineer III
> >> > > > > > > > > > > National Center for Atmospheric Research
> >> > > > > > > > > > > Research Applications Laboratory
> >> > > > > > > > > > > 303-497-2768
> >> > > > > > > > > > > ---
> >> > > > > > > > > > > My working day may not be your working day.
Please
> do
> >> not
> >> > > > > > > > > > > feel
> >> > > > > > > > obliged
> >> > > > > > > > > to
> >> > > > > > > > > > > reply to this email outside of your normal
working
> >> hours.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > > George McCabe - Software Engineer III
> >> > > > > > > > > National Center for Atmospheric Research
> >> > > > > > > > > Research Applications Laboratory
> >> > > > > > > > > 303-497-2768
> >> > > > > > > > > ---
> >> > > > > > > > > My working day may not be your working day.
Please do
> not
> >> > feel
> >> > > > > > obliged
> >> > > > > > > to
> >> > > > > > > > > reply to this email outside of your normal
working
> hours.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > George McCabe - Software Engineer III
> >> > > > > > > National Center for Atmospheric Research
> >> > > > > > > Research Applications Laboratory
> >> > > > > > > 303-497-2768
> >> > > > > > > ---
> >> > > > > > > My working day may not be your working day. Please do
not
> feel
> >> > > > > > > obliged to
> >> > > > > > > reply to this email outside of your normal working
hours.
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > George McCabe - Software Engineer III
> >> > National Center for Atmospheric Research
> >> > Research Applications Laboratory
> >> > 303-497-2768
> >> > ---
> >> > My working day may not be your working day. Please do not feel
obliged
> >> to
> >> > reply to this email outside of your normal working hours.
> >> >
> >> >
> >>
> >>
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: John Halley Gotway
Time: Thu Apr 22 11:40:23 2021
Zhuming,
This is John HG. George asked me to take a look at this question.
Since
you're asking about "non-standard" levels, I assume you're asking
about the
verification of the pressure levels against the ADPUPA message type.
In general, the logic is this...
For each point observation, loop through all the verification tasks
and see
if it should be used.
For example, let's talk about the 4 level entry for VAR3:
FCST_VAR3_NAME = v_tr_p
FCST_VAR3_LEVELS = ...(0,3,*,*)...
OBS_VAR3_NAME = VGRD
OBS_VAR3_LEVELS = ...P575-625...
The point observation will be used here if it's a "VGRD" observation
and
has a level value between 575 and 625. This verification task has been
configured to only verify a single forecast level of data:
v_tr_p(0,3,*,*).
So no vertical interpolation is done. MET interpolates horizontally
from
the forecast grid points to the observation lat/lon location (using
user-configurable interpolation options).
To put it simply, all obs between 575 and 625 mb are compared directly
to
the forecast values from v_tr_p(0,3,*,*). If the model output is
actually
on pressure levels, then that's a reasonable thing to do. If the
model's
vertical coordinate is something other than pressure levels, then
that's a
potential source for large errors.
Note that MET does do vertical interpolation when processing the GRIB
output from the Unified Post Processor tool. But all the details of
that
depend on how the configuration file is setup. I'll explain below.
To date we have avoided including direct knowledge about the vertical
coordinate of each modelling system in the verification code. The goal
is
to create verification tools applicable to many models, rather than
customizing them for each. We do realize that desire for the latter
and are
exploring ways to pull data from the IODA data assimilation system
instead,
which necessarily knows a lot about each model.
So let's say you were using GRIB data, and configured it like this:
VAR3_NAME = VGRD
VAR3_LEVELS = P500-1000
Since we haven't prefixed these options with FCST_ and OBS_, the same
settings are applied to both.
MET would read ALL forecast GRIB records from the model file with
pressures
between 500 and 1000. Perhaps, that's every 100mb or perhaps every
50mb? It
reads whatever data falls in that range. For each point observation
with a
pressure level between 500 and 1000, it'll first find the actual model
pressure level above/below it. It'll interpolate the forecast
data horizontally to the obs lat/lon location for both. And then it'll
interpolate those values above/below vertically to the actual
observation
level, linear in the log of pressure.
This issue is that the processing of pressure LAYERS is not currently
supported for NetCDF data I believe. And the code could be enhanced to
support that. But again, this only makes sense if the model's vertical
coordinate is actually pressure levels and not something else.
Hope that helps clarify.
Thanks,
John
On Thu, Apr 22, 2021 at 9:52 AM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> My verification result shows that there is a big error on a non-
standard
> level, so my question is:
> Is the forecast grid match to Obs, or obs station match to Forecast?
I need
> to do is the forecast grid interpolated to the obs location and
vertical
> levels. So how to set the levels?
>
> Thanks,
> Zhuming
>
> FCST_VAR3_NAME = v_10m_tr
>
> FCST_VAR3_LEVELS = (0,*,*)
>
> FCST_VAR3_THRESH = >=5, <5
>
> OBS_VAR3_NAME = VGRD
>
> OBS_VAR3_LEVELS = Z10
>
> OBS_VAR3_THRESH = >=5, <5
>
> OBS_VAR3_OPTIONS = message_type = ["ADPSFC"];
>
>
> FCST_VAR3_NAME = v_tr_p
>
> FCST_VAR3_LEVELS =
>
>
(0,0,*,*),(0,1,*,*),(0,2,*,*),(0,3,*,*),(0,4,*,*),(0,5,*,*),(0,6,*,*),(0,7,*,*),(0,8,*,*),(0,9,*,*)
>
> FCST_VAR3_THRESH = >=5, <5
>
> OBS_VAR3_NAME = VGRD
>
> OBS_VAR3_LEVELS =
>
> P900-950,P825-875,P675-725,P575-625,P475-525,P375-425,P275-325,P225-
275,P175-225,P75-125
>
> OBS_VAR3_THRESH = >=5, <5
>
> OBS_VAR3_OPTIONS = message_type = ["ADPUPA"];
>
>
>
> On Thu, Apr 15, 2021 at 1:52 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > There was indeed a bug in the code. Howard has fixed the issue
with this
> > pull request: https://github.com/dtcenter/MET/pull/1756
> > This fix will be included in the MET 10.0.0-beta5 release that is
> > tentatively scheduled for next week. This beta version can be
installed
> on
> > cheyenne so you can test that the update fixes your issue. I will
follow
> up
> > and let you know when the beta release is available on cheyenne.
> >
> > Thanks,
> > George
> >
> > On Wed, Apr 14, 2021 at 5:28 PM George McCabe <mccabe at ucar.edu>
wrote:
> >
> > > Hi Zhuming,
> > >
> > > I tried to get this to work on cheyenne with your data, but was
not
> > > successful. I am able to read the data with plot_data_plane and
> generate
> > an
> > > image, so I think there may be a bug in the code. I have
forwarded info
> > to
> > > one of our C++ developers to look into it. I will report back
when I
> know
> > > more.
> > >
> > > Thanks,
> > > George
> > >
> > > On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >>
> > >> Hi George,
> > >>
> > >> It still can't find the correct time. I tried
> > >>
> > >> FCST_VAR1_LEVELS = (*,*,*)
> > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
> > >>
> > >> Thanks!
> > >>
> > >> Zhuming
> > >>
> > >> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > >> wrote:
> > >>
> > >> > Hi Zhuming,
> > >> >
> > >> > It looks like you are requesting the first time in the 12
hour file
> > for
> > >> > each run:
> > >> >
> > >> > FCST_VAR1_LEVELS = (0,*,*)
> > >> > ...
> > >> > FCST_VAR2_LEVELS = (0,*,*)
> > >> >
> > >> > The first dimension in these files is the time offset. You
will need
> > the
> > >> > correct index that corresponds to each valid time.
Fortunately, MET
> > >> makes
> > >> > this easy by allowing you to specify the valid time for a
time
> > dimension
> > >> > explicitly and it will find the correct data. You can set:
> > >> >
> > >> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > >> > ...
> > >> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > >> >
> > >> > And it will substitute the valid time for each run. Let me
know if
> you
> > >> have
> > >> > any questions.
> > >> >
> > >> > Thanks,
> > >> > George
> > >> >
> > >> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
> > >> met_help at ucar.edu
> > >> > >
> > >> > wrote:
> > >> >
> > >> > >
> > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >> > >
> > >> > > Hi George,
> > >> > >
> > >> > > Thanks for your email, I just would like to send you for
help.
> > >> > >
> > >> > > I already resolved the last problem with the WRF forecast,
thank
> you
> > >> for
> > >> > > your help.
> > >> > >
> > >> > > I also need to verify the MPAS model forecast. These 12
hour
> > >> > > MAPS forecasts are stored in a one file, such as
> > >> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> > >> > > 2015071512/mpas_wrf_cf_2015071512.nc
> > >> > >
> > >> > > It seems MET is doing fine on the first 2 hours, but no
matched
> > pairs
> > >> for
> > >> > > later hours.
> > >> > > I'm not sure what's wrong with it? the time is not matched?
> > >> > >
> > >> > > see the log file
> > >> > >
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > >> > > config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> > >> > >
> > >> > > Thanks,
> > >> > > Zhuming
> > >> > >
> > >> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
> > >> met_help at ucar.edu>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Zhuming,
> > >> > > >
> > >> > > > Just following up on this ticket. Were you able to get
this
> > >> resolved?
> > >> > Let
> > >> > > > me know if you need more assistance.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > George
> > >> > > >
> > >> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > >> > > > > Hi Zhuming,
> > >> > > > >
> > >> > > > > Typically directories that contain data for an entire
day are
> > >> named
> > >> > > > > with
> > >> > > > > YYYYMMDD, so your filename template tag would be
> > >> {valid?fmt=%Y%m%d},
> > >> > > > > however that may not always be the case, as you are
seeing
> here.
> > >> > There
> > >> > > > > are
> > >> > > > > a couple ways to handle this.
> > >> > > > >
> > >> > > > > If you know that the directory will always be YYYYMMDD
with 00
> > at
> > >> the
> > >> > > > > end,
> > >> > > > > you can simply put 00 in the template instead of adding
%H in
> > the
> > >> > tag,
> > >> > > > > i.e.
> > >> > > > >
> > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > >> > > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> > >> 00_cf.nc
> > >> > > > >
> > >> > > > > If the HH value can vary, for instance if there are 2
> > directories
> > >> > > > > created
> > >> > > > > per day (i.e. 2015071400 and 2015071412), then you can
use
> the
> > >> > > > > truncate
> > >> > > > > keyword to handle it.
> > >> > > > >
> > >> > > > > If directories are created every 24 hours:
> > >> > > > >
> > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > >> > > > >
{valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > >> > > > > %d_%H}:00:
> > >> > > > > 00_cf.nc
> > >> > > > >
> > >> > > > > If directories are created every 12 hours:
> > >> > > > >
> > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > >> > > > >
{valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > >> > > > > %d_%H}:00:
> > >> > > > > 00_cf.nc
> > >> > > > >
> > >> > > > > Let me know if that doesn't work for you.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > George
> > >> > > > >
> > >> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> > >> > > > > <met_help at ucar.edu>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > >
> > >> > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >
> > >> > > > > >
> > >> > > > > > Hi George,
> > >> > > > > >
> > >> > > > > > Processing Obs data using "PB2NC" are right now.
> > >> > > > > >
> > >> > > > > > Now I'm using "PointStat" , I set VALID_BEG =
2015071500,
> > >> > VALID_END =
> > >> > > > > > 2015071501. It looks for the forecast data
2015071500/
> > >> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > >> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > >> > > > > > But my forecast data are all put in dir of
2015071500/.
> > >> > > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to
get
> the
> > >> right
> > >> > > > > > fcst
> > >> > > > > > data?
> > >> > > > > >
> > >> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > >> > > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> > >> > 00_cf.nc
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > config file:
> /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > >> > > > > >
> > >> > > > > > log file:
> > >> > > > > >
> > >> > >
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > >
> > >> > > > > > Zhuming
> > >> > > > > >
> > >> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > >> > > > > > <met_help at ucar.edu>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Zhuming,
> > >> > > > > > >
> > >> > > > > > > Looking at the log output, it looks like the data
is
> stored
> > in
> > >> > the
> > >> > > > > > prepBUFR
> > >> > > > > > > files differently that you may have thought:
> > >> > > > > > >
> > >> > > > > > > DEBUG 2: PrepBufr Time Center:
20150715_000000
> > >> > > > > > > DEBUG 2: Searching Time Window:
20150715_030000 to
> > >> > > > > > 20150715_040000
> > >> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65%
70% 75%
> > 80%
> > >> > 85%
> > >> > > > > > > 90%
> > >> > > > > > 95%
> > >> > > > > > > 100%
> > >> > > > > > >
> > >> > > > > > > DEBUG 2: Total Messages processed =
590836
> > >> > > > > > > DEBUG 2: Rejected based on message type = 0
> > >> > > > > > > DEBUG 2: Rejected based on station id = 0
> > >> > > > > > > DEBUG 2: Rejected based on valid time =
590815
> > >> > > > > > > DEBUG 2: Rejected based on masking grid =
20
> > >> > > > > > > DEBUG 2: Rejected based on masking polygon = 0
> > >> > > > > > > DEBUG 2: Rejected based on elevation = 0
> > >> > > > > > > DEBUG 2: Rejected based on pb report type = 1
> > >> > > > > > > DEBUG 2: Rejected based on input report type = 0
> > >> > > > > > > DEBUG 2: Rejected based on instrument type = 0
> > >> > > > > > > DEBUG 2: Rejected based on zero observations = 0
> > >> > > > > > > DEBUG 2: Total Messages retained = 0
> > >> > > > > > > DEBUG 2: Total observations retained or derived = 0
> > >> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
> > >> 20150715_030000*
> > >> > > > > > > WARNING:
> > >> > > > > > > WARNING: process_pbfile() -> No PrepBufr messages
retained
> > >> from
> > >> > > > > > > file:
> > >> > > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > >> > > > > > > prepbufr.gdas.2015071500.nr
> > >> > > > > > > WARNING:
> > >> > > > > > > ERROR :
> > >> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr
messages
> > >> > retained.
> > >> > > > > > Nothing
> > >> > > > > > > to write.
> > >> > > > > > > ERROR :
> > >> > > > > > >
> > >> > > > > > > The prepbufr.gdas.2015071500.nr file actually
contains
> data
> > >> from
> > >> > > > > > > 20150714_210000 to 20150715_030000, not
20150715_000000 to
> > >> > > > > > 20150715_060000.
> > >> > > > > > >
> > >> > > > > > > To get the correct file for each run, you will have
to
> shift
> > >> the
> > >> > > > > > > valid
> > >> > > > > > time
> > >> > > > > > > before truncating it so that it gets the correct
file for
> > each
> > >> > run.
> > >> > > > > > Instead
> > >> > > > > > > of setting:
> > >> > > > > > >
> > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > >> > > > > > >
> > >> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > >> > > > > > >
> > >> > > > > > > You will have to also shift forward 3 hours, i.e.:
> > >> > > > > > >
> > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > >> > > > > > >
> > >> > > > > > > This says to add 3 hours to the current valid time
before
> > >> > > > > > > truncating to
> > >> > > > > > the
> > >> > > > > > > nearest 6 hours.
> > >> > > > > > >
> > >> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > >> > > > > > >
> > >> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > >> > > > > > >
> > >> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > >> > > > > > >
> > >> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > >> > > > > > > File: prepbufr.gdas.2015071506.nr
> > >> > > > > > >
> > >> > > > > > > Let me know if that doesn't work.
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > George
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via
RT <
> > >> > > > > > met_help at ucar.edu>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > <URL:
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >> > >
> > >> > > > > > > >
> > >> > > > > > > > Hi George,
> > >> > > > > > > >
> > >> > > > > > > > Now makes me confused is why processed obs of
several
> > times
> > >> are
> > >> > > > > > missing.
> > >> > > > > > > > There has files of time at 00,01,02, but no
03,04,05;
> has
> > >> > > > > > > > 06,07,08, but
> > >> > > > > > > no
> > >> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has
18,19,20, but
> > with
> > >> > > > > > > > 21,no
> > >> > > > > > 22,23.
> > >> > > > > > > > From yours and mines:
> > >> > > > > > > >
> > >> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> > >> > pbs.2015071500.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> > >> > pbs.2015071501.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> > >> > pbs.2015071502.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> > >> > pbs.2015071506.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> > >> > pbs.2015071507.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> > >> > pbs.2015071508.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> > >> > pbs.2015071512.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> > >> > pbs.2015071513.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> > >> > pbs.2015071514.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> > >> > pbs.2015071515.nc
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > >> > > > > > > >
> > >> > > > > > > > total 96112
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> > >> > > pbs.2015071500.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> > >> > > pbs.2015071501.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> > >> > > pbs.2015071502.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> > >> > > pbs.2015071506.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> > >> > > pbs.2015071507.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> > >> > > pbs.2015071508.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> > >> > > pbs.2015071512.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> > >> > > pbs.2015071513.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> > >> > > pbs.2015071514.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> > >> > > pbs.2015071518.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> > >> > > pbs.2015071519.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> > >> > > pbs.2015071520.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> > >> > > pbs.2015071521.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> > >> > > pbs.2015071600.nc
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > >
> > >> > > > > > > > Zhuming
> > >> > > > > > > >
> > >> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via
RT
> > >> > > > > > > > <met_help at ucar.edu
> > >> > > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi Zhuming,
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks for the information. You will need to
use the
> > >> truncate
> > >> > > > > > > > > keyword
> > >> > > > > > > to
> > >> > > > > > > > > find the files, then set the valid beg/end
range to 1
> > >> hour.
> > >> > > > > > > > > This is
> > >> > > > > > > what
> > >> > > > > > > > I
> > >> > > > > > > > > changed from your PB2NC1.conf config file:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *PB2NC_VALID_BEGIN =
> > {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
> > >> =
> > >> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > >> > > > > > > > >
> > >> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > >> > > > > > > > >
> > >> > > >
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > >> > > > > > > > >
> > >> > > > > > > > > My config file is on cheyenne here:
> > >> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > >> > > > > > > > >
> > >> > > > > > > > > I just started running it and the output is
being
> > written
> > >> > here:
> > >> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > For PointStat, you will want to loop over the
same
> > times.
> > >> I
> > >> > > > > > > > > noticed
> > >> > > > > > you
> > >> > > > > > > > > specify message type in the VAR<n>_OPTIONS, but
you do
> > not
> > >> > need
> > >> > > > > > > > > to do
> > >> > > > > > > > this
> > >> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE
variable that
> > >> > defines
> > >> > > > > > > > > this.
> > >> > > > > > > This
> > >> > > > > > > > > is what I changed in your PointStat1.conf file:
> > >> > > > > > > > >
> > >> > > > > > > > > *VALID_END = 2015071600*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > >> > > > > > =POINT_STAT_OBS_VALID_BEG
> > >> > > > > > > > =*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > >> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > >> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
> > >> message_type =
> > >> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > >> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > >> > > > > > > > >
> > >> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR =
{OUTPUT_BASE}/pb2nc1*
> > >> > > > > > > > >
> > >> > > > > > > > > Or you can view my file in the same directory.
I
> usually
> > >> try
> > >> > to
> > >> > > > > > > > > test
> > >> > > > > > > > with a
> > >> > > > > > > > > single run time to make sure it is working
properly,
> so
> > I
> > >> set
> > >> > > > > > VALID_END
> > >> > > > > > > > > back to 2015071500. I ran it and it produced an
output
> > >> file,
> > >> > > > > > > > > but
> > >> > > > > > there
> > >> > > > > > > > were
> > >> > > > > > > > > no values besides the header. I checked the log
file
> and
> > >> saw
> > >> > > > > > > > > this:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus
TMP/Z2, for
> > >> > > > > > > > > observation type
> > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > >> BILIN(4),
> > >> > > > > > > > > using 0
> > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> > 0DEBUG
> > >> 3:
> > >> > > > > > > Observations
> > >> > > > > > > > > processed = 393373DEBUG 3: Rejected: station
id
> > =
> > >> > > > > > > > > 0DEBUG 3:
> > >> > > > > > > > > Rejected: obs type = 390528DEBUG 3:
Rejected:
> > valid
> > >> > time
> > >> > > > > > =
> > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
> > Rejected:
> > >> off
> > >> > > > > > > > > the
> > >> > > > > > grid
> > >> > > > > > > > =
> > >> > > > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG
3:
> > >> Rejected:
> > >> > > > > > > > > level
> > >> > > > > > > > mismatch
> > >> > > > > > > > > = 259DEBUG 3: Rejected: quality marker =
0DEBUG 3:
> > >> Rejected:
> > >> > > > > > message
> > >> > > > > > > > type
> > >> > > > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG
3:
> > >> Rejected:
> > >> > bad
> > >> > > > > > fcst
> > >> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG
> 3:
> > >> > > > > > > > > Rejected:
> > >> > > > > > bad
> > >> > > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates
=
> > 0DEBUG
> > >> > > > > > > > > 2:DEBUG 2:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
--------------------------------------------------------------------------------DEBUG
> > >> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus
UGRD/Z10,
> > for
> > >> > > > > > observation
> > >> > > > > > > > type
> > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > >> BILIN(4),
> > >> > > > > > > > > using 0
> > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> > 0DEBUG
> > >> 3:
> > >> > > > > > > Observations
> > >> > > > > > > > > processed = 393373*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
> > Rejected:
> > >> obs
> > >> > > > > > > > > type
> > >> > > > > > > > =
> > >> > > > > > > > > 382965DEBUG 3: Rejected: valid time =
0DEBUG 3:
> > >> > Rejected:
> > >> > > > > > > > > bad
> > >> > > > > > obs
> > >> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid =
> > 9240DEBUG
> > >> 3:
> > >> > > > > > Rejected:
> > >> > > > > > > > > topography = 0DEBUG 3: Rejected: level
mismatch
> =
> > >> > > > > > > > > 1168DEBUG 3:
> > >> > > > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected:
> message
> > >> type
> > >> > > > > > > > > =
> > >> > > > > > > 0DEBUG
> > >> > > > > > > > 3:
> > >> > > > > > > > > Rejected: masking region = 0DEBUG 3: Rejected:
bad
> fcst
> > >> > value
> > >> > > > > > > > > =
> > >> > > > > > > 0DEBUG
> > >> > > > > > > > 3:
> > >> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected:
bad
> > climo
> > >> > stdev
> > >> > > > > > > > > =
> > >> > > > > > > 0DEBUG
> > >> > > > > > > > 3:
> > >> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
--------------------------------------------------------------------------------DEBUG
> > >> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus
VGRD/Z10,
> > for
> > >> > > > > > observation
> > >> > > > > > > > type
> > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > >> BILIN(4),
> > >> > > > > > > > > using 0
> > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> > 0DEBUG
> > >> 3:
> > >> > > > > > > Observations
> > >> > > > > > > > > processed = 393373DEBUG 3: Rejected: station
id
> > =
> > >> > > > > > > > > 0DEBUG 3:
> > >> > > > > > > > > Rejected: obs type = 382965DEBUG 3:
Rejected:
> > valid
> > >> > time
> > >> > > > > > =
> > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
> > Rejected:
> > >> off
> > >> > > > > > > > > the
> > >> > > > > > grid
> > >> > > > > > > > =
> > >> > > > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG
3:
> > >> Rejected:
> > >> > > > > > > > > level
> > >> > > > > > > > mismatch
> > >> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker =
0DEBUG 3:
> > >> > Rejected:
> > >> > > > > > message
> > >> > > > > > > > > type = 0DEBUG 3: Rejected: masking region =
0DEBUG
> > 3:
> > >> > > > > > > > > Rejected:
> > >> > > > > > bad
> > >> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo
mean =
> > >> 0DEBUG 3:
> > >> > > > > > Rejected:
> > >> > > > > > > > bad
> > >> > > > > > > > > climo stdev = 0*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG
> 2:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
--------------------------------------------------------------------------------DEBUG
> > >> > > > > > > > > 2:DEBUG 1: Output file:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > This describes the reason why each observation
was
> > >> rejected.
> > >> > > > > > > > > Some
> > >> > > > > > > > > observations didn't match the message type you
> > requested,
> > >> > some
> > >> > > > > > > > > were
> > >> > > > > > not
> > >> > > > > > > > > within the forecast grid, and some did not
match the
> > >> vertical
> > >> > > > > > > > > level.
> > >> > > > > > > You
> > >> > > > > > > > > will have to make adjustments to your settings
so that
> > the
> > >> > data
> > >> > > > > > matches
> > >> > > > > > > > up
> > >> > > > > > > > > properly. Maybe the grid you are using in the
PB2NC
> > config
> > >> > file
> > >> > > > > > > > > does
> > >> > > > > > > not
> > >> > > > > > > > > match the forecast grid and you are missing
values. If
> > >> that
> > >> > is
> > >> > > > > > > > > the
> > >> > > > > > > case,
> > >> > > > > > > > > you will have to reconfigure and rerun PB2NC to
get
> > >> output on
> > >> > > > > > > > > the
> > >> > > > > > > correct
> > >> > > > > > > > > gr5id. Maybe the observation points don't line
up with
> > the
> > >> > > > > > > > > vertical
> > >> > > > > > > level
> > >> > > > > > > > > of the model fields. You can use a range of
level
> values
> > >> for
> > >> > > > > > > > > the
> > >> > > > > > > > > observation data to allow data within that
range, i.e.
> > >> Z0-10.
> > >> > > > > > > > >
> > >> > > > > > > > > Let me know if you are still having issues
getting
> this
> > to
> > >> > work
> > >> > > > > > > > > and I
> > >> > > > > > > can
> > >> > > > > > > > > help out.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > > George
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu
via
> RT
> > <
> > >> > > > > > > > met_help at ucar.edu>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > <URL:
> > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > Hi George,
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks for your reply!
> > >> > > > > > > > > >
> > >> > > > > > > > > > I'm still confused, I need to output hourly
obs data
> > to
> > >> do
> > >> > > > > > > > > > hourly
> > >> > > > > > > > > > verification.
> > >> > > > > > > > > > The original GDAS Prepbufr data has 4 files
one day,
> > >> should
> > >> > > > > > > > > > with 6
> > >> > > > > > > > hours
> > >> > > > > > > > > > obs data in one file.
> > >> > > > > > > > > >
> > >> > > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc
(obs
> time
> > is
> > >> > 00?
> > >> > > > > > > > > > or 6
> > >> > > > > > > times
> > >> > > > > > > > > of
> > >> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > >> > > > > > > > > >
> > >> > > > > > > > > > my config file :
> > >> > > > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > >> > > > > > > > > > and
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Zhuming
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe
via RT
> <
> > >> > > > > > > met_help at ucar.edu
> > >> > > > > > > > >
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi Zhuming,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > To run the tools hourly you would set the
> > >> VALID_INCREMENT
> > >> > > > > > > > > > > value
> > >> > > > > > for
> > >> > > > > > > > 1H.
> > >> > > > > > > > > > > Here is an example that runs once per hour
for a
> day
> > >> from
> > >> > > > > > > 2015-07-15
> > >> > > > > > > > 0Z
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > 2017-07-16 0Z:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > VALID_BEG = 2015071500
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > VALID_END = 2015071600
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > VALID_INCREMENT = 1H
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > LEAD_SEQ = 0
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Each run time will use the
PB2NC_INPUT_TEMPLATE to
> > >> find
> > >> > > > > > > > > > > files
> > >> > > > > > whose
> > >> > > > > > > > > > > filenames match the time you need.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > To read prepBUFR observations for 12 hours,
you
> will
> > >> want
> > >> > > > > > > > > > > to set
> > >> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to
specify
> the
> > >> > range
> > >> > > > > > > > > > > around
> > >> > > > > > > the
> > >> > > > > > > > > > valid
> > >> > > > > > > > > > > time to process. These values depend on
what is
> > >> contained
> > >> > > > > > > > > > > inside
> > >> > > > > > > your
> > >> > > > > > > > > > > prepBUFR files. If you want to use data
from the
> > valid
> > >> > time
> > >> > > > > > > > > > > of
> > >> > > > > > the
> > >> > > > > > > > > > filename
> > >> > > > > > > > > > > backwards 12 hours, you would set:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > PB2NC_VALID_BEGIN =
> {valid?fmt=%Y%m%d_%H?shift=-12H
> > }
> > >> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > For a runtime of 2015071512, this would be
> > substituted
> > >> > as:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > valid_beg: 20150715_00
> > >> > > > > > > > > > > valid_end: 20150715_12
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > If you want to create an output file for
each hour
> > in
> > >> > PB2NC
> > >> > > > > > > > > > > you
> > >> > > > > > > would
> > >> > > > > > > > > > need
> > >> > > > > > > > > > > to make sure that the hour is specified in
the
> > >> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > PointStat has similar config variables to
set the
> > >> valid
> > >> > beg
> > >> > > > > > > > > > > and
> > >> > > > > > end
> > >> > > > > > > > > range
> > >> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > >> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > If your prepBUFR files are separated out
into
> > multiple
> > >> > > > > > > > > > > files that
> > >> > > > > > > > need
> > >> > > > > > > > > to
> > >> > > > > > > > > > > be read into PB2NC, you can set a file
window
> around
> > >> the
> > >> > > > > > > > > > > valid
> > >> > > > > > time
> > >> > > > > > > > to
> > >> > > > > > > > > > > allow files to be passed into the app. More
> > >> information
> > >> > on
> > >> > > > > > > > > > > that
> > >> > > > > > > > > > > functionality is here:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > >> > > > > > windows-to-find-valid-files
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > If you are still having trouble getting
this
> > >> configured,
> > >> > > > > > > > > > > you
> > >> > > > > > could
> > >> > > > > > > > send
> > >> > > > > > > > > > me
> > >> > > > > > > > > > > the paths of the files you are trying to
read and
> > what
> > >> > data
> > >> > > > > > > > > > > is
> > >> > > > > > > > > contained
> > >> > > > > > > > > > in
> > >> > > > > > > > > > > them so I can give you suggestions on what
to set
> > for
> > >> the
> > >> > > > > > > > > configuration.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > George
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie
Prestopnik
> via
> > >> RT <
> > >> > > > > > > > > > met_help at ucar.edu>
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429
was
> acted
> > >> upon.
> > >> > > > > > > > > > > > Transaction: Given to mccabe (George
McCabe) by
> > >> jpresto
> > >> > > > > > > > > > > > Queue: met_help
> > >> > > > > > > > > > > > Subject: How to set time for hourly
12-24
> > hours
> > >> > data
> > >> > > > > > > > > > > > ?
> > >> > > > > > > > > > > > Owner: mccabe
> > >> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> > >> > > > > > > > > > > > Status: new
> > >> > > > > > > > > > > > Ticket <URL:
> > >> > > > > > > > >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > This transaction appears to have no
content
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > --
> > >> > > > > > > > > > > George McCabe - Software Engineer III
> > >> > > > > > > > > > > National Center for Atmospheric Research
> > >> > > > > > > > > > > Research Applications Laboratory
> > >> > > > > > > > > > > 303-497-2768
> > >> > > > > > > > > > > ---
> > >> > > > > > > > > > > My working day may not be your working day.
Please
> > do
> > >> not
> > >> > > > > > > > > > > feel
> > >> > > > > > > > obliged
> > >> > > > > > > > > to
> > >> > > > > > > > > > > reply to this email outside of your normal
working
> > >> hours.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > --
> > >> > > > > > > > > George McCabe - Software Engineer III
> > >> > > > > > > > > National Center for Atmospheric Research
> > >> > > > > > > > > Research Applications Laboratory
> > >> > > > > > > > > 303-497-2768
> > >> > > > > > > > > ---
> > >> > > > > > > > > My working day may not be your working day.
Please do
> > not
> > >> > feel
> > >> > > > > > obliged
> > >> > > > > > > to
> > >> > > > > > > > > reply to this email outside of your normal
working
> > hours.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > George McCabe - Software Engineer III
> > >> > > > > > > National Center for Atmospheric Research
> > >> > > > > > > Research Applications Laboratory
> > >> > > > > > > 303-497-2768
> > >> > > > > > > ---
> > >> > > > > > > My working day may not be your working day. Please
do not
> > feel
> > >> > > > > > > obliged to
> > >> > > > > > > reply to this email outside of your normal working
hours.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > George McCabe - Software Engineer III
> > >> > National Center for Atmospheric Research
> > >> > Research Applications Laboratory
> > >> > 303-497-2768
> > >> > ---
> > >> > My working day may not be your working day. Please do not
feel
> obliged
> > >> to
> > >> > reply to this email outside of your normal working hours.
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Mon May 03 15:47:14 2021
Hi Zhuming,
The METplus 4.0.0-beta5 release is now available on Cheyenne. If you
are
able to test that this version works and fixes some of the issues you
mentioned in this ticket before the 4.0.0 release next week, that
would be
great! Information on how to use the new version can be found here:
https://dtcenter.org/community-code/metplus/metplus-4-0-existing-
builds
(Under NCAR > CHEYENNE > METplus-4.0.0-beta5)
Here are the commands to set it up:
module use /glade/p/ral/jntp/MET/METplus/modulefiles
module load metplus/4.0.0-beta5
ncar_pylib
Please let me know if you have any issues or questions.
Thanks,
George
On Thu, Apr 22, 2021 at 11:40 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Zhuming,
>
> This is John HG. George asked me to take a look at this question.
Since
> you're asking about "non-standard" levels, I assume you're asking
about the
> verification of the pressure levels against the ADPUPA message type.
>
> In general, the logic is this...
> For each point observation, loop through all the verification tasks
and see
> if it should be used.
> For example, let's talk about the 4 level entry for VAR3:
>
> FCST_VAR3_NAME = v_tr_p
> FCST_VAR3_LEVELS = ...(0,3,*,*)...
> OBS_VAR3_NAME = VGRD
> OBS_VAR3_LEVELS = ...P575-625...
>
> The point observation will be used here if it's a "VGRD" observation
and
> has a level value between 575 and 625. This verification task has
been
> configured to only verify a single forecast level of data:
v_tr_p(0,3,*,*).
> So no vertical interpolation is done. MET interpolates horizontally
from
> the forecast grid points to the observation lat/lon location (using
> user-configurable interpolation options).
>
> To put it simply, all obs between 575 and 625 mb are compared
directly to
> the forecast values from v_tr_p(0,3,*,*). If the model output is
actually
> on pressure levels, then that's a reasonable thing to do. If the
model's
> vertical coordinate is something other than pressure levels, then
that's a
> potential source for large errors.
>
> Note that MET does do vertical interpolation when processing the
GRIB
> output from the Unified Post Processor tool. But all the details of
that
> depend on how the configuration file is setup. I'll explain below.
>
> To date we have avoided including direct knowledge about the
vertical
> coordinate of each modelling system in the verification code. The
goal is
> to create verification tools applicable to many models, rather than
> customizing them for each. We do realize that desire for the latter
and are
> exploring ways to pull data from the IODA data assimilation system
instead,
> which necessarily knows a lot about each model.
>
> So let's say you were using GRIB data, and configured it like this:
>
> VAR3_NAME = VGRD
> VAR3_LEVELS = P500-1000
>
> Since we haven't prefixed these options with FCST_ and OBS_, the
same
> settings are applied to both.
>
> MET would read ALL forecast GRIB records from the model file with
pressures
> between 500 and 1000. Perhaps, that's every 100mb or perhaps every
50mb? It
> reads whatever data falls in that range. For each point observation
with a
> pressure level between 500 and 1000, it'll first find the actual
model
> pressure level above/below it. It'll interpolate the forecast
> data horizontally to the obs lat/lon location for both. And then
it'll
> interpolate those values above/below vertically to the actual
observation
> level, linear in the log of pressure.
>
> This issue is that the processing of pressure LAYERS is not
currently
> supported for NetCDF data I believe. And the code could be enhanced
to
> support that. But again, this only makes sense if the model's
vertical
> coordinate is actually pressure levels and not something else.
>
> Hope that helps clarify.
>
> Thanks,
> John
>
> On Thu, Apr 22, 2021 at 9:52 AM zhuming at ucar.edu via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > My verification result shows that there is a big error on a non-
standard
> > level, so my question is:
> > Is the forecast grid match to Obs, or obs station match to
Forecast? I
> need
> > to do is the forecast grid interpolated to the obs location and
vertical
> > levels. So how to set the levels?
> >
> > Thanks,
> > Zhuming
> >
> > FCST_VAR3_NAME = v_10m_tr
> >
> > FCST_VAR3_LEVELS = (0,*,*)
> >
> > FCST_VAR3_THRESH = >=5, <5
> >
> > OBS_VAR3_NAME = VGRD
> >
> > OBS_VAR3_LEVELS = Z10
> >
> > OBS_VAR3_THRESH = >=5, <5
> >
> > OBS_VAR3_OPTIONS = message_type = ["ADPSFC"];
> >
> >
> > FCST_VAR3_NAME = v_tr_p
> >
> > FCST_VAR3_LEVELS =
> >
> >
>
(0,0,*,*),(0,1,*,*),(0,2,*,*),(0,3,*,*),(0,4,*,*),(0,5,*,*),(0,6,*,*),(0,7,*,*),(0,8,*,*),(0,9,*,*)
> >
> > FCST_VAR3_THRESH = >=5, <5
> >
> > OBS_VAR3_NAME = VGRD
> >
> > OBS_VAR3_LEVELS =
> >
> >
> P900-950,P825-875,P675-725,P575-625,P475-525,P375-425,P275-325,P225-
275,P175-225,P75-125
> >
> > OBS_VAR3_THRESH = >=5, <5
> >
> > OBS_VAR3_OPTIONS = message_type = ["ADPUPA"];
> >
> >
> >
> > On Thu, Apr 15, 2021 at 1:52 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > There was indeed a bug in the code. Howard has fixed the issue
with
> this
> > > pull request: https://github.com/dtcenter/MET/pull/1756
> > > This fix will be included in the MET 10.0.0-beta5 release that
is
> > > tentatively scheduled for next week. This beta version can be
installed
> > on
> > > cheyenne so you can test that the update fixes your issue. I
will
> follow
> > up
> > > and let you know when the beta release is available on cheyenne.
> > >
> > > Thanks,
> > > George
> > >
> > > On Wed, Apr 14, 2021 at 5:28 PM George McCabe <mccabe at ucar.edu>
wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > I tried to get this to work on cheyenne with your data, but
was not
> > > > successful. I am able to read the data with plot_data_plane
and
> > generate
> > > an
> > > > image, so I think there may be a bug in the code. I have
forwarded
> info
> > > to
> > > > one of our C++ developers to look into it. I will report back
when I
> > know
> > > > more.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >>
> > > >> Hi George,
> > > >>
> > > >> It still can't find the correct time. I tried
> > > >>
> > > >> FCST_VAR1_LEVELS = (*,*,*)
> > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Zhuming
> > > >>
> > > >> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > >> wrote:
> > > >>
> > > >> > Hi Zhuming,
> > > >> >
> > > >> > It looks like you are requesting the first time in the 12
hour
> file
> > > for
> > > >> > each run:
> > > >> >
> > > >> > FCST_VAR1_LEVELS = (0,*,*)
> > > >> > ...
> > > >> > FCST_VAR2_LEVELS = (0,*,*)
> > > >> >
> > > >> > The first dimension in these files is the time offset. You
will
> need
> > > the
> > > >> > correct index that corresponds to each valid time.
Fortunately,
> MET
> > > >> makes
> > > >> > this easy by allowing you to specify the valid time for a
time
> > > dimension
> > > >> > explicitly and it will find the correct data. You can set:
> > > >> >
> > > >> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > >> > ...
> > > >> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > >> >
> > > >> > And it will substitute the valid time for each run. Let me
know if
> > you
> > > >> have
> > > >> > any questions.
> > > >> >
> > > >> > Thanks,
> > > >> > George
> > > >> >
> > > >> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
> > > >> met_help at ucar.edu
> > > >> > >
> > > >> > wrote:
> > > >> >
> > > >> > >
> > > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > >> > >
> > > >> > > Hi George,
> > > >> > >
> > > >> > > Thanks for your email, I just would like to send you for
help.
> > > >> > >
> > > >> > > I already resolved the last problem with the WRF
forecast, thank
> > you
> > > >> for
> > > >> > > your help.
> > > >> > >
> > > >> > > I also need to verify the MPAS model forecast. These 12
hour
> > > >> > > MAPS forecasts are stored in a one file, such as
> > > >> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> > > >> > > 2015071512/mpas_wrf_cf_2015071512.nc
> > > >> > >
> > > >> > > It seems MET is doing fine on the first 2 hours, but no
matched
> > > pairs
> > > >> for
> > > >> > > later hours.
> > > >> > > I'm not sure what's wrong with it? the time is not
matched?
> > > >> > >
> > > >> > > see the log file
> > > >> > >
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > > >> > > config file
/gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Zhuming
> > > >> > >
> > > >> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
> > > >> met_help at ucar.edu>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi Zhuming,
> > > >> > > >
> > > >> > > > Just following up on this ticket. Were you able to get
this
> > > >> resolved?
> > > >> > Let
> > > >> > > > me know if you need more assistance.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > George
> > > >> > > >
> > > >> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > >> > > > > Hi Zhuming,
> > > >> > > > >
> > > >> > > > > Typically directories that contain data for an entire
day
> are
> > > >> named
> > > >> > > > > with
> > > >> > > > > YYYYMMDD, so your filename template tag would be
> > > >> {valid?fmt=%Y%m%d},
> > > >> > > > > however that may not always be the case, as you are
seeing
> > here.
> > > >> > There
> > > >> > > > > are
> > > >> > > > > a couple ways to handle this.
> > > >> > > > >
> > > >> > > > > If you know that the directory will always be
YYYYMMDD with
> 00
> > > at
> > > >> the
> > > >> > > > > end,
> > > >> > > > > you can simply put 00 in the template instead of
adding %H
> in
> > > the
> > > >> > tag,
> > > >> > > > > i.e.
> > > >> > > > >
> > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > >> > > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> > > >> 00_cf.nc
> > > >> > > > >
> > > >> > > > > If the HH value can vary, for instance if there are 2
> > > directories
> > > >> > > > > created
> > > >> > > > > per day (i.e. 2015071400 and 2015071412), then you
can use
> > the
> > > >> > > > > truncate
> > > >> > > > > keyword to handle it.
> > > >> > > > >
> > > >> > > > > If directories are created every 24 hours:
> > > >> > > > >
> > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > >> > > > >
> {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > >> > > > > %d_%H}:00:
> > > >> > > > > 00_cf.nc
> > > >> > > > >
> > > >> > > > > If directories are created every 12 hours:
> > > >> > > > >
> > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > >> > > > >
> {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > >> > > > > %d_%H}:00:
> > > >> > > > > 00_cf.nc
> > > >> > > > >
> > > >> > > > > Let me know if that doesn't work for you.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > George
> > > >> > > > >
> > > >> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via
RT
> > > >> > > > > <met_help at ucar.edu>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > >
> > > >> > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >
> > > >> > > > > >
> > > >> > > > > > Hi George,
> > > >> > > > > >
> > > >> > > > > > Processing Obs data using "PB2NC" are right now.
> > > >> > > > > >
> > > >> > > > > > Now I'm using "PointStat" , I set VALID_BEG =
2015071500,
> > > >> > VALID_END =
> > > >> > > > > > 2015071501. It looks for the forecast data
2015071500/
> > > >> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and
2015071501/
> > > >> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > >> > > > > > But my forecast data are all put in dir of
2015071500/.
> > > >> > > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE
to get
> > the
> > > >> right
> > > >> > > > > > fcst
> > > >> > > > > > data?
> > > >> > > > > >
> > > >> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > >> > > > > >
> {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> > > >> > 00_cf.nc
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > config file:
> > /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > >> > > > > >
> > > >> > > > > > log file:
> > > >> > > > > >
> > > >> > >
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > >
> > > >> > > > > > Zhuming
> > > >> > > > > >
> > > >> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > > >> > > > > > <met_help at ucar.edu>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi Zhuming,
> > > >> > > > > > >
> > > >> > > > > > > Looking at the log output, it looks like the data
is
> > stored
> > > in
> > > >> > the
> > > >> > > > > > prepBUFR
> > > >> > > > > > > files differently that you may have thought:
> > > >> > > > > > >
> > > >> > > > > > > DEBUG 2: PrepBufr Time Center:
20150715_000000
> > > >> > > > > > > DEBUG 2: Searching Time Window:
20150715_030000
> to
> > > >> > > > > > 20150715_040000
> > > >> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60%
65% 70%
> 75%
> > > 80%
> > > >> > 85%
> > > >> > > > > > > 90%
> > > >> > > > > > 95%
> > > >> > > > > > > 100%
> > > >> > > > > > >
> > > >> > > > > > > DEBUG 2: Total Messages processed =
590836
> > > >> > > > > > > DEBUG 2: Rejected based on message type =
0
> > > >> > > > > > > DEBUG 2: Rejected based on station id =
0
> > > >> > > > > > > DEBUG 2: Rejected based on valid time =
590815
> > > >> > > > > > > DEBUG 2: Rejected based on masking grid =
20
> > > >> > > > > > > DEBUG 2: Rejected based on masking polygon =
0
> > > >> > > > > > > DEBUG 2: Rejected based on elevation =
0
> > > >> > > > > > > DEBUG 2: Rejected based on pb report type =
1
> > > >> > > > > > > DEBUG 2: Rejected based on input report type =
0
> > > >> > > > > > > DEBUG 2: Rejected based on instrument type =
0
> > > >> > > > > > > DEBUG 2: Rejected based on zero observations =
0
> > > >> > > > > > > DEBUG 2: Total Messages retained =
0
> > > >> > > > > > > DEBUG 2: Total observations retained or derived =
0
> > > >> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
> > > >> 20150715_030000*
> > > >> > > > > > > WARNING:
> > > >> > > > > > > WARNING: process_pbfile() -> No PrepBufr messages
> retained
> > > >> from
> > > >> > > > > > > file:
> > > >> > > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > >> > > > > > > prepbufr.gdas.2015071500.nr
> > > >> > > > > > > WARNING:
> > > >> > > > > > > ERROR :
> > > >> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr
messages
> > > >> > retained.
> > > >> > > > > > Nothing
> > > >> > > > > > > to write.
> > > >> > > > > > > ERROR :
> > > >> > > > > > >
> > > >> > > > > > > The prepbufr.gdas.2015071500.nr file actually
contains
> > data
> > > >> from
> > > >> > > > > > > 20150714_210000 to 20150715_030000, not
20150715_000000
> to
> > > >> > > > > > 20150715_060000.
> > > >> > > > > > >
> > > >> > > > > > > To get the correct file for each run, you will
have to
> > shift
> > > >> the
> > > >> > > > > > > valid
> > > >> > > > > > time
> > > >> > > > > > > before truncating it so that it gets the correct
file
> for
> > > each
> > > >> > run.
> > > >> > > > > > Instead
> > > >> > > > > > > of setting:
> > > >> > > > > > >
> > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > >> > > > > > >
> > > >> >
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > >> > > > > > >
> > > >> > > > > > > You will have to also shift forward 3 hours,
i.e.:
> > > >> > > > > > >
> > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > >> > > > > > >
> > > >> > > > > > > This says to add 3 hours to the current valid
time
> before
> > > >> > > > > > > truncating to
> > > >> > > > > > the
> > > >> > > > > > > nearest 6 hours.
> > > >> > > > > > >
> > > >> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > >> > > > > > >
> > > >> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > >> > > > > > >
> > > >> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > >> > > > > > >
> > > >> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > >> > > > > > > File: prepbufr.gdas.2015071506.nr
> > > >> > > > > > >
> > > >> > > > > > > Let me know if that doesn't work.
> > > >> > > > > > >
> > > >> > > > > > > Thanks,
> > > >> > > > > > > George
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu
via RT
> <
> > > >> > > > > > met_help at ucar.edu>
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > <URL:
> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >> > >
> > > >> > > > > > > >
> > > >> > > > > > > > Hi George,
> > > >> > > > > > > >
> > > >> > > > > > > > Now makes me confused is why processed obs of
several
> > > times
> > > >> are
> > > >> > > > > > missing.
> > > >> > > > > > > > There has files of time at 00,01,02, but no
03,04,05;
> > has
> > > >> > > > > > > > 06,07,08, but
> > > >> > > > > > > no
> > > >> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has
18,19,20,
> but
> > > with
> > > >> > > > > > > > 21,no
> > > >> > > > > > 22,23.
> > > >> > > > > > > > From yours and mines:
> > > >> > > > > > > >
> > > >> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> > > >> > pbs.2015071500.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> > > >> > pbs.2015071501.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> > > >> > pbs.2015071502.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> > > >> > pbs.2015071506.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> > > >> > pbs.2015071507.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> > > >> > pbs.2015071508.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> > > >> > pbs.2015071512.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> > > >> > pbs.2015071513.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> > > >> > pbs.2015071514.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> > > >> > pbs.2015071515.nc
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > >> > > > > > > >
> > > >> > > > > > > > total 96112
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> > > >> > > pbs.2015071500.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> > > >> > > pbs.2015071501.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> > > >> > > pbs.2015071502.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> > > >> > > pbs.2015071506.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> > > >> > > pbs.2015071507.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> > > >> > > pbs.2015071508.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> > > >> > > pbs.2015071512.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> > > >> > > pbs.2015071513.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> > > >> > > pbs.2015071514.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> > > >> > > pbs.2015071518.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> > > >> > > pbs.2015071519.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> > > >> > > pbs.2015071520.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> > > >> > > pbs.2015071521.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> > > >> > > pbs.2015071600.nc
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks,
> > > >> > > > > > > >
> > > >> > > > > > > > Zhuming
> > > >> > > > > > > >
> > > >> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe
via RT
> > > >> > > > > > > > <met_help at ucar.edu
> > > >> > > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi Zhuming,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks for the information. You will need to
use the
> > > >> truncate
> > > >> > > > > > > > > keyword
> > > >> > > > > > > to
> > > >> > > > > > > > > find the files, then set the valid beg/end
range to
> 1
> > > >> hour.
> > > >> > > > > > > > > This is
> > > >> > > > > > > what
> > > >> > > > > > > > I
> > > >> > > > > > > > > changed from your PB2NC1.conf config file:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *PB2NC_VALID_BEGIN =
> > > {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
> > > >> =
> > > >> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > >> > > > > > > > >
> > > >> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > >> > > > > > > > >
> > > >> > > >
> > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > >> > > > > > > > >
> > > >> > > > > > > > > My config file is on cheyenne here:
> > > >> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > >> > > > > > > > >
> > > >> > > > > > > > > I just started running it and the output is
being
> > > written
> > > >> > here:
> > > >> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > For PointStat, you will want to loop over the
same
> > > times.
> > > >> I
> > > >> > > > > > > > > noticed
> > > >> > > > > > you
> > > >> > > > > > > > > specify message type in the VAR<n>_OPTIONS,
but you
> do
> > > not
> > > >> > need
> > > >> > > > > > > > > to do
> > > >> > > > > > > > this
> > > >> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE
variable
> that
> > > >> > defines
> > > >> > > > > > > > > this.
> > > >> > > > > > > This
> > > >> > > > > > > > > is what I changed in your PointStat1.conf
file:
> > > >> > > > > > > > >
> > > >> > > > > > > > > *VALID_END = 2015071600*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > > >> > > > > > =POINT_STAT_OBS_VALID_BEG
> > > >> > > > > > > > =*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > >> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > >> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
> > > >> message_type =
> > > >> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > >> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > >> > > > > > > > >
> > > >> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR =
{OUTPUT_BASE}/pb2nc1*
> > > >> > > > > > > > >
> > > >> > > > > > > > > Or you can view my file in the same
directory. I
> > usually
> > > >> try
> > > >> > to
> > > >> > > > > > > > > test
> > > >> > > > > > > > with a
> > > >> > > > > > > > > single run time to make sure it is working
properly,
> > so
> > > I
> > > >> set
> > > >> > > > > > VALID_END
> > > >> > > > > > > > > back to 2015071500. I ran it and it produced
an
> output
> > > >> file,
> > > >> > > > > > > > > but
> > > >> > > > > > there
> > > >> > > > > > > > were
> > > >> > > > > > > > > no values besides the header. I checked the
log file
> > and
> > > >> saw
> > > >> > > > > > > > > this:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus
TMP/Z2, for
> > > >> > > > > > > > > observation type
> > > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > > >> BILIN(4),
> > > >> > > > > > > > > using 0
> > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs =
> > > 0DEBUG
> > > >> 3:
> > > >> > > > > > > Observations
> > > >> > > > > > > > > processed = 393373DEBUG 3: Rejected:
station id
> > > =
> > > >> > > > > > > > > 0DEBUG 3:
> > > >> > > > > > > > > Rejected: obs type = 390528DEBUG 3:
Rejected:
> > > valid
> > > >> > time
> > > >> > > > > > =
> > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG
3:
> > > Rejected:
> > > >> off
> > > >> > > > > > > > > the
> > > >> > > > > > grid
> > > >> > > > > > > > =
> > > >> > > > > > > > > 2586DEBUG 3: Rejected: topography =
0DEBUG 3:
> > > >> Rejected:
> > > >> > > > > > > > > level
> > > >> > > > > > > > mismatch
> > > >> > > > > > > > > = 259DEBUG 3: Rejected: quality marker =
0DEBUG 3:
> > > >> Rejected:
> > > >> > > > > > message
> > > >> > > > > > > > type
> > > >> > > > > > > > > = 0DEBUG 3: Rejected: masking region =
0DEBUG 3:
> > > >> Rejected:
> > > >> > bad
> > > >> > > > > > fcst
> > > >> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean
=
> 0DEBUG
> > 3:
> > > >> > > > > > > > > Rejected:
> > > >> > > > > > bad
> > > >> > > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates
=
> > > 0DEBUG
> > > >> > > > > > > > > 2:DEBUG 2:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > >> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus
> UGRD/Z10,
> > > for
> > > >> > > > > > observation
> > > >> > > > > > > > type
> > > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > > >> BILIN(4),
> > > >> > > > > > > > > using 0
> > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs =
> > > 0DEBUG
> > > >> 3:
> > > >> > > > > > > Observations
> > > >> > > > > > > > > processed = 393373*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG
3:
> > > Rejected:
> > > >> obs
> > > >> > > > > > > > > type
> > > >> > > > > > > > =
> > > >> > > > > > > > > 382965DEBUG 3: Rejected: valid time =
0DEBUG 3:
> > > >> > Rejected:
> > > >> > > > > > > > > bad
> > > >> > > > > > obs
> > > >> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid
=
> > > 9240DEBUG
> > > >> 3:
> > > >> > > > > > Rejected:
> > > >> > > > > > > > > topography = 0DEBUG 3: Rejected: level
mismatch
> > =
> > > >> > > > > > > > > 1168DEBUG 3:
> > > >> > > > > > > > > Rejected: quality marker = 0DEBUG 3:
Rejected:
> > message
> > > >> type
> > > >> > > > > > > > > =
> > > >> > > > > > > 0DEBUG
> > > >> > > > > > > > 3:
> > > >> > > > > > > > > Rejected: masking region = 0DEBUG 3:
Rejected: bad
> > fcst
> > > >> > value
> > > >> > > > > > > > > =
> > > >> > > > > > > 0DEBUG
> > > >> > > > > > > > 3:
> > > >> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3:
Rejected: bad
> > > climo
> > > >> > stdev
> > > >> > > > > > > > > =
> > > >> > > > > > > 0DEBUG
> > > >> > > > > > > > 3:
> > > >> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > >> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus
> VGRD/Z10,
> > > for
> > > >> > > > > > observation
> > > >> > > > > > > > type
> > > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > > >> BILIN(4),
> > > >> > > > > > > > > using 0
> > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs =
> > > 0DEBUG
> > > >> 3:
> > > >> > > > > > > Observations
> > > >> > > > > > > > > processed = 393373DEBUG 3: Rejected:
station id
> > > =
> > > >> > > > > > > > > 0DEBUG 3:
> > > >> > > > > > > > > Rejected: obs type = 382965DEBUG 3:
Rejected:
> > > valid
> > > >> > time
> > > >> > > > > > =
> > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG
3:
> > > Rejected:
> > > >> off
> > > >> > > > > > > > > the
> > > >> > > > > > grid
> > > >> > > > > > > > =
> > > >> > > > > > > > > 9240DEBUG 3: Rejected: topography =
0DEBUG 3:
> > > >> Rejected:
> > > >> > > > > > > > > level
> > > >> > > > > > > > mismatch
> > > >> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker =
0DEBUG 3:
> > > >> > Rejected:
> > > >> > > > > > message
> > > >> > > > > > > > > type = 0DEBUG 3: Rejected: masking region
=
> 0DEBUG
> > > 3:
> > > >> > > > > > > > > Rejected:
> > > >> > > > > > bad
> > > >> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo
mean =
> > > >> 0DEBUG 3:
> > > >> > > > > > Rejected:
> > > >> > > > > > > > bad
> > > >> > > > > > > > > climo stdev = 0*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG
> > 2:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > >> > > > > > > > > 2:DEBUG 1: Output file:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > This describes the reason why each
observation was
> > > >> rejected.
> > > >> > > > > > > > > Some
> > > >> > > > > > > > > observations didn't match the message type
you
> > > requested,
> > > >> > some
> > > >> > > > > > > > > were
> > > >> > > > > > not
> > > >> > > > > > > > > within the forecast grid, and some did not
match the
> > > >> vertical
> > > >> > > > > > > > > level.
> > > >> > > > > > > You
> > > >> > > > > > > > > will have to make adjustments to your
settings so
> that
> > > the
> > > >> > data
> > > >> > > > > > matches
> > > >> > > > > > > > up
> > > >> > > > > > > > > properly. Maybe the grid you are using in the
PB2NC
> > > config
> > > >> > file
> > > >> > > > > > > > > does
> > > >> > > > > > > not
> > > >> > > > > > > > > match the forecast grid and you are missing
values.
> If
> > > >> that
> > > >> > is
> > > >> > > > > > > > > the
> > > >> > > > > > > case,
> > > >> > > > > > > > > you will have to reconfigure and rerun PB2NC
to get
> > > >> output on
> > > >> > > > > > > > > the
> > > >> > > > > > > correct
> > > >> > > > > > > > > gr5id. Maybe the observation points don't
line up
> with
> > > the
> > > >> > > > > > > > > vertical
> > > >> > > > > > > level
> > > >> > > > > > > > > of the model fields. You can use a range of
level
> > values
> > > >> for
> > > >> > > > > > > > > the
> > > >> > > > > > > > > observation data to allow data within that
range,
> i.e.
> > > >> Z0-10.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Let me know if you are still having issues
getting
> > this
> > > to
> > > >> > work
> > > >> > > > > > > > > and I
> > > >> > > > > > > can
> > > >> > > > > > > > > help out.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks,
> > > >> > > > > > > > > George
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM
zhuming at ucar.edu via
> > RT
> > > <
> > > >> > > > > > > > met_help at ucar.edu>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > <URL:
> > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Hi George,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks for your reply!
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > I'm still confused, I need to output hourly
obs
> data
> > > to
> > > >> do
> > > >> > > > > > > > > > hourly
> > > >> > > > > > > > > > verification.
> > > >> > > > > > > > > > The original GDAS Prepbufr data has 4 files
one
> day,
> > > >> should
> > > >> > > > > > > > > > with 6
> > > >> > > > > > > > hours
> > > >> > > > > > > > > > obs data in one file.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc
(obs
> > time
> > > is
> > > >> > 00?
> > > >> > > > > > > > > > or 6
> > > >> > > > > > > times
> > > >> > > > > > > > > of
> > > >> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > my config file :
> > > >> > > > > >
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > >> > > > > > > > > > and
> /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks,
> > > >> > > > > > > > > > Zhuming
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George
McCabe via
> RT
> > <
> > > >> > > > > > > met_help at ucar.edu
> > > >> > > > > > > > >
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi Zhuming,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > To run the tools hourly you would set the
> > > >> VALID_INCREMENT
> > > >> > > > > > > > > > > value
> > > >> > > > > > for
> > > >> > > > > > > > 1H.
> > > >> > > > > > > > > > > Here is an example that runs once per
hour for a
> > day
> > > >> from
> > > >> > > > > > > 2015-07-15
> > > >> > > > > > > > 0Z
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > 2017-07-16 0Z:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > VALID_BEG = 2015071500
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > VALID_END = 2015071600
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > VALID_INCREMENT = 1H
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > LEAD_SEQ = 0
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Each run time will use the
PB2NC_INPUT_TEMPLATE
> to
> > > >> find
> > > >> > > > > > > > > > > files
> > > >> > > > > > whose
> > > >> > > > > > > > > > > filenames match the time you need.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > To read prepBUFR observations for 12
hours, you
> > will
> > > >> want
> > > >> > > > > > > > > > > to set
> > > >> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to
specify
> > the
> > > >> > range
> > > >> > > > > > > > > > > around
> > > >> > > > > > > the
> > > >> > > > > > > > > > valid
> > > >> > > > > > > > > > > time to process. These values depend on
what is
> > > >> contained
> > > >> > > > > > > > > > > inside
> > > >> > > > > > > your
> > > >> > > > > > > > > > > prepBUFR files. If you want to use data
from the
> > > valid
> > > >> > time
> > > >> > > > > > > > > > > of
> > > >> > > > > > the
> > > >> > > > > > > > > > filename
> > > >> > > > > > > > > > > backwards 12 hours, you would set:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > PB2NC_VALID_BEGIN =
> > {valid?fmt=%Y%m%d_%H?shift=-12H
> > > }
> > > >> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > For a runtime of 2015071512, this would
be
> > > substituted
> > > >> > as:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > valid_beg: 20150715_00
> > > >> > > > > > > > > > > valid_end: 20150715_12
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > If you want to create an output file for
each
> hour
> > > in
> > > >> > PB2NC
> > > >> > > > > > > > > > > you
> > > >> > > > > > > would
> > > >> > > > > > > > > > need
> > > >> > > > > > > > > > > to make sure that the hour is specified
in the
> > > >> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > PointStat has similar config variables to
set
> the
> > > >> valid
> > > >> > beg
> > > >> > > > > > > > > > > and
> > > >> > > > > > end
> > > >> > > > > > > > > range
> > > >> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > >> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > If your prepBUFR files are separated out
into
> > > multiple
> > > >> > > > > > > > > > > files that
> > > >> > > > > > > > need
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > be read into PB2NC, you can set a file
window
> > around
> > > >> the
> > > >> > > > > > > > > > > valid
> > > >> > > > > > time
> > > >> > > > > > > > to
> > > >> > > > > > > > > > > allow files to be passed into the app.
More
> > > >> information
> > > >> > on
> > > >> > > > > > > > > > > that
> > > >> > > > > > > > > > > functionality is here:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > >> > > > > > windows-to-find-valid-files
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > If you are still having trouble getting
this
> > > >> configured,
> > > >> > > > > > > > > > > you
> > > >> > > > > > could
> > > >> > > > > > > > send
> > > >> > > > > > > > > > me
> > > >> > > > > > > > > > > the paths of the files you are trying to
read
> and
> > > what
> > > >> > data
> > > >> > > > > > > > > > > is
> > > >> > > > > > > > > contained
> > > >> > > > > > > > > > in
> > > >> > > > > > > > > > > them so I can give you suggestions on
what to
> set
> > > for
> > > >> the
> > > >> > > > > > > > > configuration.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > George
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie
Prestopnik
> > via
> > > >> RT <
> > > >> > > > > > > > > > met_help at ucar.edu>
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429
was
> > acted
> > > >> upon.
> > > >> > > > > > > > > > > > Transaction: Given to mccabe (George
McCabe)
> by
> > > >> jpresto
> > > >> > > > > > > > > > > > Queue: met_help
> > > >> > > > > > > > > > > > Subject: How to set time for
hourly 12-24
> > > hours
> > > >> > data
> > > >> > > > > > > > > > > > ?
> > > >> > > > > > > > > > > > Owner: mccabe
> > > >> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> > > >> > > > > > > > > > > > Status: new
> > > >> > > > > > > > > > > > Ticket <URL:
> > > >> > > > > > > > >
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > This transaction appears to have no
content
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > --
> > > >> > > > > > > > > > > George McCabe - Software Engineer III
> > > >> > > > > > > > > > > National Center for Atmospheric Research
> > > >> > > > > > > > > > > Research Applications Laboratory
> > > >> > > > > > > > > > > 303-497-2768
> > > >> > > > > > > > > > > ---
> > > >> > > > > > > > > > > My working day may not be your working
day.
> Please
> > > do
> > > >> not
> > > >> > > > > > > > > > > feel
> > > >> > > > > > > > obliged
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > reply to this email outside of your
normal
> working
> > > >> hours.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > --
> > > >> > > > > > > > > George McCabe - Software Engineer III
> > > >> > > > > > > > > National Center for Atmospheric Research
> > > >> > > > > > > > > Research Applications Laboratory
> > > >> > > > > > > > > 303-497-2768
> > > >> > > > > > > > > ---
> > > >> > > > > > > > > My working day may not be your working day.
Please
> do
> > > not
> > > >> > feel
> > > >> > > > > > obliged
> > > >> > > > > > > to
> > > >> > > > > > > > > reply to this email outside of your normal
working
> > > hours.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > --
> > > >> > > > > > > George McCabe - Software Engineer III
> > > >> > > > > > > National Center for Atmospheric Research
> > > >> > > > > > > Research Applications Laboratory
> > > >> > > > > > > 303-497-2768
> > > >> > > > > > > ---
> > > >> > > > > > > My working day may not be your working day.
Please do
> not
> > > feel
> > > >> > > > > > > obliged to
> > > >> > > > > > > reply to this email outside of your normal
working
> hours.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > George McCabe - Software Engineer III
> > > >> > National Center for Atmospheric Research
> > > >> > Research Applications Laboratory
> > > >> > 303-497-2768
> > > >> > ---
> > > >> > My working day may not be your working day. Please do not
feel
> > obliged
> > > >> to
> > > >> > reply to this email outside of your normal working hours.
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Wed May 19 14:43:44 2021
Hi George,
I tried running MET new version beta5. It seems something is wrong.
see log file
/glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
config file in /glade/work/zhuming/METplus4
Thanks!
Zhuming
(NPL) cheyenne1:/glade/work/zhuming/METplus4>run_metplus.py -c
./PointStat.conf -c ./defaults.conf
Running METplus 4.0.0-beta5
Starting METplus v4.0.0-beta5
05/19 14:34:36.341 metplus (config_metplus.py:117) INFO: Starting
METplus
configuration setup.
05/19 14:34:36.343 metplus (config_metplus.py:247) INFO:
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/defaults.conf:
Parsed
this file
05/19 14:34:36.344 metplus (config_metplus.py:247) INFO:
./PointStat.conf:
Parsed this file
05/19 14:34:36.345 metplus (config_metplus.py:247) INFO:
./defaults.conf:
Parsed this file
05/19 14:34:36.346 metplus (config_metplus.py:436) INFO: Adding:
config.LOG_TIMESTAMP='20210519143436'
05/19 14:34:36.346 metplus (config_metplus.py:442) INFO: Replace
[config]
LOG_METPLUS with
'/glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436'
05/19 14:34:36.348 metplus (config_metplus.py:314) INFO: Setting [dir]
METPLUS_BASE to /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta5
05/19 14:34:36.348 metplus (config_metplus.py:314) INFO: Setting [dir]
PARM_BASE to /gpfs/fs1/work/zhuming/METplus4/parm
05/19 14:34:36.348 metplus (config_metplus.py:321) INFO: METPLUS_CONF:
/glade/scratch/zhuming/metplus4/metplus_final.conf written here.
05/19 14:34:36.349 metplus (config_metplus.py:137) INFO: Completed
METplus
configuration setup.
05/19 14:34:36.349 metplus (met_util.py:68) INFO: Running METplus
v4.0.0-beta5 called with command:
/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta5/ush/run_metplus.py
-c
./PointStat.conf -c ./defaults.conf
05/19 14:34:36.349 metplus (met_util.py:70) INFO: Log file:
/glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
05/19 14:34:36.358 metplus.Usage (met_util.py:186) ERROR: Fatal error
occurred
Traceback (most recent call last):
File
"/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/util/met_util.py",
line 128, in run_metplus
instance=instance)
File
"/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/wrappers/usage_wrapper.py",
line 19, in __init__
self.available_processes = list(set(val for val in
met_util.LOWER_TO_WRAPPER_NAME.values()))
AttributeError: module 'metplus.util.met_util' has no attribute
'LOWER_TO_WRAPPER_NAME'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/util/met_util.py",
line 136, in run_metplus
raise NameError("There was a problem loading "
NameError: There was a problem loading Usage wrapper.
05/19 14:34:36Z run-METplus-metplus.Usage: ERROR: Fatal error
occurred
05/19 14:34:36.359 metplus.Usage (met_util.py:187) INFO: Check the log
file
for more information:
/glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
05/19 14:34:36.360 metplus (met_util.py:834) INFO:
/glade/scratch/zhuming/metplus4/metplus_final.conf: write metplus.conf
here
05/19 14:34:36.365 metplus (met_util.py:218) ERROR: METplus has
finished
running but had 1 error.
05/19 14:34:36Z run-METplus-metplus: ERROR: METplus has finished
running
but had 1 error.
05/19 14:34:36.365 metplus (met_util.py:219) INFO: Check the log file
for
more information:
/glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
(NPL) cheyenne1:/glade/work/zhuming/METplus4>
On Mon, May 3, 2021 at 3:47 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> The METplus 4.0.0-beta5 release is now available on Cheyenne. If you
are
> able to test that this version works and fixes some of the issues
you
> mentioned in this ticket before the 4.0.0 release next week, that
would be
> great! Information on how to use the new version can be found here:
>
> https://dtcenter.org/community-code/metplus/metplus-4-0-existing-
builds
> (Under NCAR > CHEYENNE > METplus-4.0.0-beta5)
>
> Here are the commands to set it up:
>
> module use /glade/p/ral/jntp/MET/METplus/modulefiles
> module load metplus/4.0.0-beta5
> ncar_pylib
>
> Please let me know if you have any issues or questions.
>
> Thanks,
> George
>
> On Thu, Apr 22, 2021 at 11:40 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Zhuming,
> >
> > This is John HG. George asked me to take a look at this question.
Since
> > you're asking about "non-standard" levels, I assume you're asking
about
> the
> > verification of the pressure levels against the ADPUPA message
type.
> >
> > In general, the logic is this...
> > For each point observation, loop through all the verification
tasks and
> see
> > if it should be used.
> > For example, let's talk about the 4 level entry for VAR3:
> >
> > FCST_VAR3_NAME = v_tr_p
> > FCST_VAR3_LEVELS = ...(0,3,*,*)...
> > OBS_VAR3_NAME = VGRD
> > OBS_VAR3_LEVELS = ...P575-625...
> >
> > The point observation will be used here if it's a "VGRD"
observation and
> > has a level value between 575 and 625. This verification task has
been
> > configured to only verify a single forecast level of data:
> v_tr_p(0,3,*,*).
> > So no vertical interpolation is done. MET interpolates
horizontally from
> > the forecast grid points to the observation lat/lon location
(using
> > user-configurable interpolation options).
> >
> > To put it simply, all obs between 575 and 625 mb are compared
directly to
> > the forecast values from v_tr_p(0,3,*,*). If the model output is
actually
> > on pressure levels, then that's a reasonable thing to do. If the
model's
> > vertical coordinate is something other than pressure levels, then
that's
> a
> > potential source for large errors.
> >
> > Note that MET does do vertical interpolation when processing the
GRIB
> > output from the Unified Post Processor tool. But all the details
of that
> > depend on how the configuration file is setup. I'll explain below.
> >
> > To date we have avoided including direct knowledge about the
vertical
> > coordinate of each modelling system in the verification code. The
goal is
> > to create verification tools applicable to many models, rather
than
> > customizing them for each. We do realize that desire for the
latter and
> are
> > exploring ways to pull data from the IODA data assimilation system
> instead,
> > which necessarily knows a lot about each model.
> >
> > So let's say you were using GRIB data, and configured it like
this:
> >
> > VAR3_NAME = VGRD
> > VAR3_LEVELS = P500-1000
> >
> > Since we haven't prefixed these options with FCST_ and OBS_, the
same
> > settings are applied to both.
> >
> > MET would read ALL forecast GRIB records from the model file with
> pressures
> > between 500 and 1000. Perhaps, that's every 100mb or perhaps every
50mb?
> It
> > reads whatever data falls in that range. For each point
observation with
> a
> > pressure level between 500 and 1000, it'll first find the actual
model
> > pressure level above/below it. It'll interpolate the forecast
> > data horizontally to the obs lat/lon location for both. And then
it'll
> > interpolate those values above/below vertically to the actual
observation
> > level, linear in the log of pressure.
> >
> > This issue is that the processing of pressure LAYERS is not
currently
> > supported for NetCDF data I believe. And the code could be
enhanced to
> > support that. But again, this only makes sense if the model's
vertical
> > coordinate is actually pressure levels and not something else.
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John
> >
> > On Thu, Apr 22, 2021 at 9:52 AM zhuming at ucar.edu via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > My verification result shows that there is a big error on a
> non-standard
> > > level, so my question is:
> > > Is the forecast grid match to Obs, or obs station match to
Forecast? I
> > need
> > > to do is the forecast grid interpolated to the obs location and
> vertical
> > > levels. So how to set the levels?
> > >
> > > Thanks,
> > > Zhuming
> > >
> > > FCST_VAR3_NAME = v_10m_tr
> > >
> > > FCST_VAR3_LEVELS = (0,*,*)
> > >
> > > FCST_VAR3_THRESH = >=5, <5
> > >
> > > OBS_VAR3_NAME = VGRD
> > >
> > > OBS_VAR3_LEVELS = Z10
> > >
> > > OBS_VAR3_THRESH = >=5, <5
> > >
> > > OBS_VAR3_OPTIONS = message_type = ["ADPSFC"];
> > >
> > >
> > > FCST_VAR3_NAME = v_tr_p
> > >
> > > FCST_VAR3_LEVELS =
> > >
> > >
> >
>
(0,0,*,*),(0,1,*,*),(0,2,*,*),(0,3,*,*),(0,4,*,*),(0,5,*,*),(0,6,*,*),(0,7,*,*),(0,8,*,*),(0,9,*,*)
> > >
> > > FCST_VAR3_THRESH = >=5, <5
> > >
> > > OBS_VAR3_NAME = VGRD
> > >
> > > OBS_VAR3_LEVELS =
> > >
> > >
> >
> P900-950,P825-875,P675-725,P575-625,P475-525,P375-425,P275-325,P225-
275,P175-225,P75-125
> > >
> > > OBS_VAR3_THRESH = >=5, <5
> > >
> > > OBS_VAR3_OPTIONS = message_type = ["ADPUPA"];
> > >
> > >
> > >
> > > On Thu, Apr 15, 2021 at 1:52 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > There was indeed a bug in the code. Howard has fixed the issue
with
> > this
> > > > pull request: https://github.com/dtcenter/MET/pull/1756
> > > > This fix will be included in the MET 10.0.0-beta5 release that
is
> > > > tentatively scheduled for next week. This beta version can be
> installed
> > > on
> > > > cheyenne so you can test that the update fixes your issue. I
will
> > follow
> > > up
> > > > and let you know when the beta release is available on
cheyenne.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Wed, Apr 14, 2021 at 5:28 PM George McCabe
<mccabe at ucar.edu>
> wrote:
> > > >
> > > > > Hi Zhuming,
> > > > >
> > > > > I tried to get this to work on cheyenne with your data, but
was not
> > > > > successful. I am able to read the data with plot_data_plane
and
> > > generate
> > > > an
> > > > > image, so I think there may be a bug in the code. I have
forwarded
> > info
> > > > to
> > > > > one of our C++ developers to look into it. I will report
back when
> I
> > > know
> > > > > more.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >>
> > > > >> Hi George,
> > > > >>
> > > > >> It still can't find the correct time. I tried
> > > > >>
> > > > >> FCST_VAR1_LEVELS = (*,*,*)
> > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >> Zhuming
> > > > >>
> > > > >> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Zhuming,
> > > > >> >
> > > > >> > It looks like you are requesting the first time in the 12
hour
> > file
> > > > for
> > > > >> > each run:
> > > > >> >
> > > > >> > FCST_VAR1_LEVELS = (0,*,*)
> > > > >> > ...
> > > > >> > FCST_VAR2_LEVELS = (0,*,*)
> > > > >> >
> > > > >> > The first dimension in these files is the time offset.
You will
> > need
> > > > the
> > > > >> > correct index that corresponds to each valid time.
Fortunately,
> > MET
> > > > >> makes
> > > > >> > this easy by allowing you to specify the valid time for a
time
> > > > dimension
> > > > >> > explicitly and it will find the correct data. You can
set:
> > > > >> >
> > > > >> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > >> > ...
> > > > >> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > >> >
> > > > >> > And it will substitute the valid time for each run. Let
me know
> if
> > > you
> > > > >> have
> > > > >> > any questions.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > George
> > > > >> >
> > > > >> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
> > > > >> met_help at ucar.edu
> > > > >> > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > >
> > > > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >
> > > > >> > >
> > > > >> > > Hi George,
> > > > >> > >
> > > > >> > > Thanks for your email, I just would like to send you
for help.
> > > > >> > >
> > > > >> > > I already resolved the last problem with the WRF
forecast,
> thank
> > > you
> > > > >> for
> > > > >> > > your help.
> > > > >> > >
> > > > >> > > I also need to verify the MPAS model forecast. These 12
hour
> > > > >> > > MAPS forecasts are stored in a one file, such as
> > > > >> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> > > > >> > > 2015071512/mpas_wrf_cf_2015071512.nc
> > > > >> > >
> > > > >> > > It seems MET is doing fine on the first 2 hours, but no
> matched
> > > > pairs
> > > > >> for
> > > > >> > > later hours.
> > > > >> > > I'm not sure what's wrong with it? the time is not
matched?
> > > > >> > >
> > > > >> > > see the log file
> > > > >> > >
> > > >
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > > > >> > > config file
/gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Zhuming
> > > > >> > >
> > > > >> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
> > > > >> met_help at ucar.edu>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Hi Zhuming,
> > > > >> > > >
> > > > >> > > > Just following up on this ticket. Were you able to
get this
> > > > >> resolved?
> > > > >> > Let
> > > > >> > > > me know if you need more assistance.
> > > > >> > > >
> > > > >> > > > Thanks,
> > > > >> > > > George
> > > > >> > > >
> > > > >> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > > >> > > > > Hi Zhuming,
> > > > >> > > > >
> > > > >> > > > > Typically directories that contain data for an
entire day
> > are
> > > > >> named
> > > > >> > > > > with
> > > > >> > > > > YYYYMMDD, so your filename template tag would be
> > > > >> {valid?fmt=%Y%m%d},
> > > > >> > > > > however that may not always be the case, as you are
seeing
> > > here.
> > > > >> > There
> > > > >> > > > > are
> > > > >> > > > > a couple ways to handle this.
> > > > >> > > > >
> > > > >> > > > > If you know that the directory will always be
YYYYMMDD
> with
> > 00
> > > > at
> > > > >> the
> > > > >> > > > > end,
> > > > >> > > > > you can simply put 00 in the template instead of
adding %H
> > in
> > > > the
> > > > >> > tag,
> > > > >> > > > > i.e.
> > > > >> > > > >
> > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > >> > > > >
> {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> > > > >> 00_cf.nc
> > > > >> > > > >
> > > > >> > > > > If the HH value can vary, for instance if there are
2
> > > > directories
> > > > >> > > > > created
> > > > >> > > > > per day (i.e. 2015071400 and 2015071412), then you
can
> use
> > > the
> > > > >> > > > > truncate
> > > > >> > > > > keyword to handle it.
> > > > >> > > > >
> > > > >> > > > > If directories are created every 24 hours:
> > > > >> > > > >
> > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > >> > > > >
> > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > > >> > > > > %d_%H}:00:
> > > > >> > > > > 00_cf.nc
> > > > >> > > > >
> > > > >> > > > > If directories are created every 12 hours:
> > > > >> > > > >
> > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > >> > > > >
> > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > > >> > > > > %d_%H}:00:
> > > > >> > > > > 00_cf.nc
> > > > >> > > > >
> > > > >> > > > > Let me know if that doesn't work for you.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > George
> > > > >> > > > >
> > > > >> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via
RT
> > > > >> > > > > <met_help at ucar.edu>
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > >
> > > > >> > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > >
> > > > >> > > > > >
> > > > >> > > > > > Hi George,
> > > > >> > > > > >
> > > > >> > > > > > Processing Obs data using "PB2NC" are right now.
> > > > >> > > > > >
> > > > >> > > > > > Now I'm using "PointStat" , I set VALID_BEG =
> 2015071500,
> > > > >> > VALID_END =
> > > > >> > > > > > 2015071501. It looks for the forecast data
2015071500/
> > > > >> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and
2015071501/
> > > > >> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > > >> > > > > > But my forecast data are all put in dir of
2015071500/.
> > > > >> > > > > > So how could I set
FCST_POINT_STAT_INPUT_TEMPLATE to
> get
> > > the
> > > > >> right
> > > > >> > > > > > fcst
> > > > >> > > > > > data?
> > > > >> > > > > >
> > > > >> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > >> > > > > >
> > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> > > > >> > 00_cf.nc
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > config file:
> > > /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > >> > > > > >
> > > > >> > > > > > log file:
> > > > >> > > > > >
> > > > >> > >
> > > >
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Thanks,
> > > > >> > > > > >
> > > > >> > > > > > Zhuming
> > > > >> > > > > >
> > > > >> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via
RT
> > > > >> > > > > > <met_help at ucar.edu>
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Hi Zhuming,
> > > > >> > > > > > >
> > > > >> > > > > > > Looking at the log output, it looks like the
data is
> > > stored
> > > > in
> > > > >> > the
> > > > >> > > > > > prepBUFR
> > > > >> > > > > > > files differently that you may have thought:
> > > > >> > > > > > >
> > > > >> > > > > > > DEBUG 2: PrepBufr Time Center:
> 20150715_000000
> > > > >> > > > > > > DEBUG 2: Searching Time Window:
> 20150715_030000
> > to
> > > > >> > > > > > 20150715_040000
> > > > >> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60%
65% 70%
> > 75%
> > > > 80%
> > > > >> > 85%
> > > > >> > > > > > > 90%
> > > > >> > > > > > 95%
> > > > >> > > > > > > 100%
> > > > >> > > > > > >
> > > > >> > > > > > > DEBUG 2: Total Messages processed
=
> 590836
> > > > >> > > > > > > DEBUG 2: Rejected based on message type
= 0
> > > > >> > > > > > > DEBUG 2: Rejected based on station id
= 0
> > > > >> > > > > > > DEBUG 2: Rejected based on valid time
=
> 590815
> > > > >> > > > > > > DEBUG 2: Rejected based on masking grid
= 20
> > > > >> > > > > > > DEBUG 2: Rejected based on masking polygon
= 0
> > > > >> > > > > > > DEBUG 2: Rejected based on elevation
= 0
> > > > >> > > > > > > DEBUG 2: Rejected based on pb report type
= 1
> > > > >> > > > > > > DEBUG 2: Rejected based on input report type
= 0
> > > > >> > > > > > > DEBUG 2: Rejected based on instrument type
= 0
> > > > >> > > > > > > DEBUG 2: Rejected based on zero observations
= 0
> > > > >> > > > > > > DEBUG 2: Total Messages retained
= 0
> > > > >> > > > > > > DEBUG 2: Total observations retained or derived
= 0
> > > > >> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
> > > > >> 20150715_030000*
> > > > >> > > > > > > WARNING:
> > > > >> > > > > > > WARNING: process_pbfile() -> No PrepBufr
messages
> > retained
> > > > >> from
> > > > >> > > > > > > file:
> > > > >> > > > > > >
/glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > >> > > > > > > prepbufr.gdas.2015071500.nr
> > > > >> > > > > > > WARNING:
> > > > >> > > > > > > ERROR :
> > > > >> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr
> messages
> > > > >> > retained.
> > > > >> > > > > > Nothing
> > > > >> > > > > > > to write.
> > > > >> > > > > > > ERROR :
> > > > >> > > > > > >
> > > > >> > > > > > > The prepbufr.gdas.2015071500.nr file actually
> contains
> > > data
> > > > >> from
> > > > >> > > > > > > 20150714_210000 to 20150715_030000, not
> 20150715_000000
> > to
> > > > >> > > > > > 20150715_060000.
> > > > >> > > > > > >
> > > > >> > > > > > > To get the correct file for each run, you will
have to
> > > shift
> > > > >> the
> > > > >> > > > > > > valid
> > > > >> > > > > > time
> > > > >> > > > > > > before truncating it so that it gets the
correct file
> > for
> > > > each
> > > > >> > run.
> > > > >> > > > > > Instead
> > > > >> > > > > > > of setting:
> > > > >> > > > > > >
> > > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > >> > > > > > >
> > > > >> >
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > > >> > > > > > >
> > > > >> > > > > > > You will have to also shift forward 3 hours,
i.e.:
> > > > >> > > > > > >
> > > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > > >> > > > > > >
> > > > >> > > > > > > This says to add 3 hours to the current valid
time
> > before
> > > > >> > > > > > > truncating to
> > > > >> > > > > > the
> > > > >> > > > > > > nearest 6 hours.
> > > > >> > > > > > >
> > > > >> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > >> > > > > > >
> > > > >> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > >> > > > > > >
> > > > >> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > >> > > > > > >
> > > > >> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > > >> > > > > > > File: prepbufr.gdas.2015071506.nr
> > > > >> > > > > > >
> > > > >> > > > > > > Let me know if that doesn't work.
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks,
> > > > >> > > > > > > George
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu
via
> RT
> > <
> > > > >> > > > > > met_help at ucar.edu>
> > > > >> > > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > <URL:
> > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > >> > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > Hi George,
> > > > >> > > > > > > >
> > > > >> > > > > > > > Now makes me confused is why processed obs of
> several
> > > > times
> > > > >> are
> > > > >> > > > > > missing.
> > > > >> > > > > > > > There has files of time at 00,01,02, but no
> 03,04,05;
> > > has
> > > > >> > > > > > > > 06,07,08, but
> > > > >> > > > > > > no
> > > > >> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has
18,19,20,
> > but
> > > > with
> > > > >> > > > > > > > 21,no
> > > > >> > > > > > 22,23.
> > > > >> > > > > > > > From yours and mines:
> > > > >> > > > > > > >
> > > > >> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> > > > >> > pbs.2015071500.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> > > > >> > pbs.2015071501.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> > > > >> > pbs.2015071502.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> > > > >> > pbs.2015071506.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> > > > >> > pbs.2015071507.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> > > > >> > pbs.2015071508.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> > > > >> > pbs.2015071512.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> > > > >> > pbs.2015071513.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> > > > >> > pbs.2015071514.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> > > > >> > pbs.2015071515.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > >
cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > >> > > > > > > >
> > > > >> > > > > > > > total 96112
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7
15:11
> > > > >> > > pbs.2015071500.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7
15:14
> > > > >> > > pbs.2015071501.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7
15:18
> > > > >> > > pbs.2015071502.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7
15:33
> > > > >> > > pbs.2015071506.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7
15:36
> > > > >> > > pbs.2015071507.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7
15:40
> > > > >> > > pbs.2015071508.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7
15:55
> > > > >> > > pbs.2015071512.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7
15:59
> > > > >> > > pbs.2015071513.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7
16:03
> > > > >> > > pbs.2015071514.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7
16:18
> > > > >> > > pbs.2015071518.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7
16:22
> > > > >> > > pbs.2015071519.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7
16:27
> > > > >> > > pbs.2015071520.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7
16:31
> > > > >> > > pbs.2015071521.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7
16:43
> > > > >> > > pbs.2015071600.nc
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thanks,
> > > > >> > > > > > > >
> > > > >> > > > > > > > Zhuming
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe
via RT
> > > > >> > > > > > > > <met_help at ucar.edu
> > > > >> > > > > > >
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Hi Zhuming,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks for the information. You will need
to use
> the
> > > > >> truncate
> > > > >> > > > > > > > > keyword
> > > > >> > > > > > > to
> > > > >> > > > > > > > > find the files, then set the valid beg/end
range
> to
> > 1
> > > > >> hour.
> > > > >> > > > > > > > > This is
> > > > >> > > > > > > what
> > > > >> > > > > > > > I
> > > > >> > > > > > > > > changed from your PB2NC1.conf config file:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *PB2NC_VALID_BEGIN =
> > > > {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
> > > > >> =
> > > > >> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > >> > > > > > > > >
> > > > >> > > >
> > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > My config file is on cheyenne here:
> > > > >> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I just started running it and the output is
being
> > > > written
> > > > >> > here:
> > > > >> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > For PointStat, you will want to loop over
the same
> > > > times.
> > > > >> I
> > > > >> > > > > > > > > noticed
> > > > >> > > > > > you
> > > > >> > > > > > > > > specify message type in the VAR<n>_OPTIONS,
but
> you
> > do
> > > > not
> > > > >> > need
> > > > >> > > > > > > > > to do
> > > > >> > > > > > > > this
> > > > >> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE
variable
> > that
> > > > >> > defines
> > > > >> > > > > > > > > this.
> > > > >> > > > > > > This
> > > > >> > > > > > > > > is what I changed in your PointStat1.conf
file:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *VALID_END = 2015071600*
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *VALID_INCREMENT =
1HPOINT_STAT_OBS_VALID_BEG
> > > > >> > > > > > =POINT_STAT_OBS_VALID_BEG
> > > > >> > > > > > > > =*
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > >> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > >> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS
=
> > > > >> message_type =
> > > > >> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > >> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR =
{OUTPUT_BASE}/pb2nc1*
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Or you can view my file in the same
directory. I
> > > usually
> > > > >> try
> > > > >> > to
> > > > >> > > > > > > > > test
> > > > >> > > > > > > > with a
> > > > >> > > > > > > > > single run time to make sure it is working
> properly,
> > > so
> > > > I
> > > > >> set
> > > > >> > > > > > VALID_END
> > > > >> > > > > > > > > back to 2015071500. I ran it and it
produced an
> > output
> > > > >> file,
> > > > >> > > > > > > > > but
> > > > >> > > > > > there
> > > > >> > > > > > > > were
> > > > >> > > > > > > > > no values besides the header. I checked the
log
> file
> > > and
> > > > >> saw
> > > > >> > > > > > > > > this:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus
TMP/Z2,
> for
> > > > >> > > > > > > > > observation type
> > > > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > > > >> BILIN(4),
> > > > >> > > > > > > > > using 0
> > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs =
> > > > 0DEBUG
> > > > >> 3:
> > > > >> > > > > > > Observations
> > > > >> > > > > > > > > processed = 393373DEBUG 3: Rejected:
station id
> > > > =
> > > > >> > > > > > > > > 0DEBUG 3:
> > > > >> > > > > > > > > Rejected: obs type = 390528DEBUG 3:
> Rejected:
> > > > valid
> > > > >> > time
> > > > >> > > > > > =
> > > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value =
0DEBUG 3:
> > > > Rejected:
> > > > >> off
> > > > >> > > > > > > > > the
> > > > >> > > > > > grid
> > > > >> > > > > > > > =
> > > > >> > > > > > > > > 2586DEBUG 3: Rejected: topography =
0DEBUG 3:
> > > > >> Rejected:
> > > > >> > > > > > > > > level
> > > > >> > > > > > > > mismatch
> > > > >> > > > > > > > > = 259DEBUG 3: Rejected: quality marker =
0DEBUG
> 3:
> > > > >> Rejected:
> > > > >> > > > > > message
> > > > >> > > > > > > > type
> > > > >> > > > > > > > > = 0DEBUG 3: Rejected: masking region =
0DEBUG 3:
> > > > >> Rejected:
> > > > >> > bad
> > > > >> > > > > > fcst
> > > > >> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean
=
> > 0DEBUG
> > > 3:
> > > > >> > > > > > > > > Rejected:
> > > > >> > > > > > bad
> > > > >> > > > > > > > > climo stdev = 0DEBUG 3: Rejected:
duplicates
> =
> > > > 0DEBUG
> > > > >> > > > > > > > > 2:DEBUG 2:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > >> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*)
versus
> > UGRD/Z10,
> > > > for
> > > > >> > > > > > observation
> > > > >> > > > > > > > type
> > > > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > > > >> BILIN(4),
> > > > >> > > > > > > > > using 0
> > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs =
> > > > 0DEBUG
> > > > >> 3:
> > > > >> > > > > > > Observations
> > > > >> > > > > > > > > processed = 393373*
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *DEBUG 3: Rejected: station id =
0DEBUG 3:
> > > > Rejected:
> > > > >> obs
> > > > >> > > > > > > > > type
> > > > >> > > > > > > > =
> > > > >> > > > > > > > > 382965DEBUG 3: Rejected: valid time =
0DEBUG
> 3:
> > > > >> > Rejected:
> > > > >> > > > > > > > > bad
> > > > >> > > > > > obs
> > > > >> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid
=
> > > > 9240DEBUG
> > > > >> 3:
> > > > >> > > > > > Rejected:
> > > > >> > > > > > > > > topography = 0DEBUG 3: Rejected: level
> mismatch
> > > =
> > > > >> > > > > > > > > 1168DEBUG 3:
> > > > >> > > > > > > > > Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > message
> > > > >> type
> > > > >> > > > > > > > > =
> > > > >> > > > > > > 0DEBUG
> > > > >> > > > > > > > 3:
> > > > >> > > > > > > > > Rejected: masking region = 0DEBUG 3:
Rejected:
> bad
> > > fcst
> > > > >> > value
> > > > >> > > > > > > > > =
> > > > >> > > > > > > 0DEBUG
> > > > >> > > > > > > > 3:
> > > > >> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3:
Rejected:
> bad
> > > > climo
> > > > >> > stdev
> > > > >> > > > > > > > > =
> > > > >> > > > > > > 0DEBUG
> > > > >> > > > > > > > 3:
> > > > >> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG
2:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > >> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*)
versus
> > VGRD/Z10,
> > > > for
> > > > >> > > > > > observation
> > > > >> > > > > > > > type
> > > > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > > > >> BILIN(4),
> > > > >> > > > > > > > > using 0
> > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs =
> > > > 0DEBUG
> > > > >> 3:
> > > > >> > > > > > > Observations
> > > > >> > > > > > > > > processed = 393373DEBUG 3: Rejected:
station id
> > > > =
> > > > >> > > > > > > > > 0DEBUG 3:
> > > > >> > > > > > > > > Rejected: obs type = 382965DEBUG 3:
> Rejected:
> > > > valid
> > > > >> > time
> > > > >> > > > > > =
> > > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value =
0DEBUG 3:
> > > > Rejected:
> > > > >> off
> > > > >> > > > > > > > > the
> > > > >> > > > > > grid
> > > > >> > > > > > > > =
> > > > >> > > > > > > > > 9240DEBUG 3: Rejected: topography =
0DEBUG 3:
> > > > >> Rejected:
> > > > >> > > > > > > > > level
> > > > >> > > > > > > > mismatch
> > > > >> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker =
0DEBUG
> 3:
> > > > >> > Rejected:
> > > > >> > > > > > message
> > > > >> > > > > > > > > type = 0DEBUG 3: Rejected: masking
region =
> > 0DEBUG
> > > > 3:
> > > > >> > > > > > > > > Rejected:
> > > > >> > > > > > bad
> > > > >> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo
mean
> =
> > > > >> 0DEBUG 3:
> > > > >> > > > > > Rejected:
> > > > >> > > > > > > > bad
> > > > >> > > > > > > > > climo stdev = 0*
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > *DEBUG 3: Rejected: duplicates =
0DEBUG
> 2:DEBUG
> > > 2:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > >> > > > > > > > > 2:DEBUG 1: Output file:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > This describes the reason why each
observation was
> > > > >> rejected.
> > > > >> > > > > > > > > Some
> > > > >> > > > > > > > > observations didn't match the message type
you
> > > > requested,
> > > > >> > some
> > > > >> > > > > > > > > were
> > > > >> > > > > > not
> > > > >> > > > > > > > > within the forecast grid, and some did not
match
> the
> > > > >> vertical
> > > > >> > > > > > > > > level.
> > > > >> > > > > > > You
> > > > >> > > > > > > > > will have to make adjustments to your
settings so
> > that
> > > > the
> > > > >> > data
> > > > >> > > > > > matches
> > > > >> > > > > > > > up
> > > > >> > > > > > > > > properly. Maybe the grid you are using in
the
> PB2NC
> > > > config
> > > > >> > file
> > > > >> > > > > > > > > does
> > > > >> > > > > > > not
> > > > >> > > > > > > > > match the forecast grid and you are missing
> values.
> > If
> > > > >> that
> > > > >> > is
> > > > >> > > > > > > > > the
> > > > >> > > > > > > case,
> > > > >> > > > > > > > > you will have to reconfigure and rerun
PB2NC to
> get
> > > > >> output on
> > > > >> > > > > > > > > the
> > > > >> > > > > > > correct
> > > > >> > > > > > > > > gr5id. Maybe the observation points don't
line up
> > with
> > > > the
> > > > >> > > > > > > > > vertical
> > > > >> > > > > > > level
> > > > >> > > > > > > > > of the model fields. You can use a range of
level
> > > values
> > > > >> for
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > observation data to allow data within that
range,
> > i.e.
> > > > >> Z0-10.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Let me know if you are still having issues
getting
> > > this
> > > > to
> > > > >> > work
> > > > >> > > > > > > > > and I
> > > > >> > > > > > > can
> > > > >> > > > > > > > > help out.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks,
> > > > >> > > > > > > > > George
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM
zhuming at ucar.edu
> via
> > > RT
> > > > <
> > > > >> > > > > > > > met_help at ucar.edu>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > <URL:
> > > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Hi George,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks for your reply!
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > I'm still confused, I need to output
hourly obs
> > data
> > > > to
> > > > >> do
> > > > >> > > > > > > > > > hourly
> > > > >> > > > > > > > > > verification.
> > > > >> > > > > > > > > > The original GDAS Prepbufr data has 4
files one
> > day,
> > > > >> should
> > > > >> > > > > > > > > > with 6
> > > > >> > > > > > > > hours
> > > > >> > > > > > > > > > obs data in one file.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > After "PB2NC", I can see
pbs.2015071500.nc (obs
> > > time
> > > > is
> > > > >> > 00?
> > > > >> > > > > > > > > > or 6
> > > > >> > > > > > > times
> > > > >> > > > > > > > > of
> > > > >> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > my config file :
> > > > >> > > > > >
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > >> > > > > > > > > > and
> > /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > Zhuming
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George
McCabe via
> > RT
> > > <
> > > > >> > > > > > > met_help at ucar.edu
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > Hi Zhuming,
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > To run the tools hourly you would set
the
> > > > >> VALID_INCREMENT
> > > > >> > > > > > > > > > > value
> > > > >> > > > > > for
> > > > >> > > > > > > > 1H.
> > > > >> > > > > > > > > > > Here is an example that runs once per
hour
> for a
> > > day
> > > > >> from
> > > > >> > > > > > > 2015-07-15
> > > > >> > > > > > > > 0Z
> > > > >> > > > > > > > > > to
> > > > >> > > > > > > > > > > 2017-07-16 0Z:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > VALID_BEG = 2015071500
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > VALID_END = 2015071600
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > VALID_INCREMENT = 1H
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > LEAD_SEQ = 0
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Each run time will use the
> PB2NC_INPUT_TEMPLATE
> > to
> > > > >> find
> > > > >> > > > > > > > > > > files
> > > > >> > > > > > whose
> > > > >> > > > > > > > > > > filenames match the time you need.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > To read prepBUFR observations for 12
hours,
> you
> > > will
> > > > >> want
> > > > >> > > > > > > > > > > to set
> > > > >> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END
to
> specify
> > > the
> > > > >> > range
> > > > >> > > > > > > > > > > around
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > valid
> > > > >> > > > > > > > > > > time to process. These values depend on
what
> is
> > > > >> contained
> > > > >> > > > > > > > > > > inside
> > > > >> > > > > > > your
> > > > >> > > > > > > > > > > prepBUFR files. If you want to use data
from
> the
> > > > valid
> > > > >> > time
> > > > >> > > > > > > > > > > of
> > > > >> > > > > > the
> > > > >> > > > > > > > > > filename
> > > > >> > > > > > > > > > > backwards 12 hours, you would set:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > PB2NC_VALID_BEGIN =
> > > {valid?fmt=%Y%m%d_%H?shift=-12H
> > > > }
> > > > >> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > For a runtime of 2015071512, this would
be
> > > > substituted
> > > > >> > as:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > valid_beg: 20150715_00
> > > > >> > > > > > > > > > > valid_end: 20150715_12
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > If you want to create an output file
for each
> > hour
> > > > in
> > > > >> > PB2NC
> > > > >> > > > > > > > > > > you
> > > > >> > > > > > > would
> > > > >> > > > > > > > > > need
> > > > >> > > > > > > > > > > to make sure that the hour is specified
in the
> > > > >> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > PointStat has similar config variables
to set
> > the
> > > > >> valid
> > > > >> > beg
> > > > >> > > > > > > > > > > and
> > > > >> > > > > > end
> > > > >> > > > > > > > > range
> > > > >> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > >> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > If your prepBUFR files are separated
out into
> > > > multiple
> > > > >> > > > > > > > > > > files that
> > > > >> > > > > > > > need
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > be read into PB2NC, you can set a file
window
> > > around
> > > > >> the
> > > > >> > > > > > > > > > > valid
> > > > >> > > > > > time
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > > allow files to be passed into the app.
More
> > > > >> information
> > > > >> > on
> > > > >> > > > > > > > > > > that
> > > > >> > > > > > > > > > > functionality is here:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > > >> > > > > > windows-to-find-valid-files
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > If you are still having trouble getting
this
> > > > >> configured,
> > > > >> > > > > > > > > > > you
> > > > >> > > > > > could
> > > > >> > > > > > > > send
> > > > >> > > > > > > > > > me
> > > > >> > > > > > > > > > > the paths of the files you are trying
to read
> > and
> > > > what
> > > > >> > data
> > > > >> > > > > > > > > > > is
> > > > >> > > > > > > > > contained
> > > > >> > > > > > > > > > in
> > > > >> > > > > > > > > > > them so I can give you suggestions on
what to
> > set
> > > > for
> > > > >> the
> > > > >> > > > > > > > > configuration.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > > George
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie
> Prestopnik
> > > via
> > > > >> RT <
> > > > >> > > > > > > > > > met_help at ucar.edu>
> > > > >> > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request
99429 was
> > > acted
> > > > >> upon.
> > > > >> > > > > > > > > > > > Transaction: Given to mccabe (George
McCabe)
> > by
> > > > >> jpresto
> > > > >> > > > > > > > > > > > Queue: met_help
> > > > >> > > > > > > > > > > > Subject: How to set time for
hourly
> 12-24
> > > > hours
> > > > >> > data
> > > > >> > > > > > > > > > > > ?
> > > > >> > > > > > > > > > > > Owner: mccabe
> > > > >> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> > > > >> > > > > > > > > > > > Status: new
> > > > >> > > > > > > > > > > > Ticket <URL:
> > > > >> > > > > > > > >
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > This transaction appears to have no
content
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > --
> > > > >> > > > > > > > > > > George McCabe - Software Engineer III
> > > > >> > > > > > > > > > > National Center for Atmospheric
Research
> > > > >> > > > > > > > > > > Research Applications Laboratory
> > > > >> > > > > > > > > > > 303-497-2768
> > > > >> > > > > > > > > > > ---
> > > > >> > > > > > > > > > > My working day may not be your working
day.
> > Please
> > > > do
> > > > >> not
> > > > >> > > > > > > > > > > feel
> > > > >> > > > > > > > obliged
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > reply to this email outside of your
normal
> > working
> > > > >> hours.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > --
> > > > >> > > > > > > > > George McCabe - Software Engineer III
> > > > >> > > > > > > > > National Center for Atmospheric Research
> > > > >> > > > > > > > > Research Applications Laboratory
> > > > >> > > > > > > > > 303-497-2768
> > > > >> > > > > > > > > ---
> > > > >> > > > > > > > > My working day may not be your working day.
Please
> > do
> > > > not
> > > > >> > feel
> > > > >> > > > > > obliged
> > > > >> > > > > > > to
> > > > >> > > > > > > > > reply to this email outside of your normal
working
> > > > hours.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > --
> > > > >> > > > > > > George McCabe - Software Engineer III
> > > > >> > > > > > > National Center for Atmospheric Research
> > > > >> > > > > > > Research Applications Laboratory
> > > > >> > > > > > > 303-497-2768
> > > > >> > > > > > > ---
> > > > >> > > > > > > My working day may not be your working day.
Please do
> > not
> > > > feel
> > > > >> > > > > > > obliged to
> > > > >> > > > > > > reply to this email outside of your normal
working
> > hours.
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > George McCabe - Software Engineer III
> > > > >> > National Center for Atmospheric Research
> > > > >> > Research Applications Laboratory
> > > > >> > 303-497-2768
> > > > >> > ---
> > > > >> > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > >> to
> > > > >> > reply to this email outside of your normal working hours.
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed May 19 15:03:32 2021
Hi Zhuming,
There are a few things going on here:
1) There is a bug in the Usage wrapper that just prints the list of
possible wrappers. Thank you for finding this. I will fix this for the
next
development release.
2) The reason you are getting the Usage wrapper instead of running
PointStat is you are passing in the defaults.conf file last.
PointStat.conf has:
PROCESS_LIST = PointStat
and defaults.conf has:
PROCESS_LIST = Usage
Since defaults.conf was read after PointStat.conf, the value in
defaults.conf is used. The defaults.conf file in the METplus directory
(under parm/metplus_config) is actually read first automatically
before
reading any of the config files you pass into the command, so you do
not
ever need to include it in the command.
3) You no longer need to put '-c' before your config files on the
command
line. It doesn't hurt to have them, but they are ignored.
run_metplus.py ./PointStat.conf
4) The official 4.0.0 release is now available on cheyenne, so you can
use
that instead. You can set it up by running the following:
module use /glade/p/ral/jntp/MET/METplus/modulefiles
module load metplus/4.0.0
ncar_pylib
Let me know if you have any other issues.
Thanks,
George
On Wed, May 19, 2021 at 2:44 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> I tried running MET new version beta5. It seems something is wrong.
> see log file
> /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
> config file in /glade/work/zhuming/METplus4
>
> Thanks!
> Zhuming
>
>
> (NPL) cheyenne1:/glade/work/zhuming/METplus4>run_metplus.py -c
> ./PointStat.conf -c ./defaults.conf
>
> Running METplus 4.0.0-beta5
>
> Starting METplus v4.0.0-beta5
>
> 05/19 14:34:36.341 metplus (config_metplus.py:117) INFO: Starting
METplus
> configuration setup.
>
> 05/19 14:34:36.343 metplus (config_metplus.py:247) INFO:
> /gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/defaults.conf:
Parsed
> this file
>
> 05/19 14:34:36.344 metplus (config_metplus.py:247) INFO:
./PointStat.conf:
> Parsed this file
>
> 05/19 14:34:36.345 metplus (config_metplus.py:247) INFO:
./defaults.conf:
> Parsed this file
>
> 05/19 14:34:36.346 metplus (config_metplus.py:436) INFO: Adding:
> config.LOG_TIMESTAMP='20210519143436'
>
> 05/19 14:34:36.346 metplus (config_metplus.py:442) INFO: Replace
[config]
> LOG_METPLUS with
> '/glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436'
>
> 05/19 14:34:36.348 metplus (config_metplus.py:314) INFO: Setting
[dir]
> METPLUS_BASE to /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta5
>
> 05/19 14:34:36.348 metplus (config_metplus.py:314) INFO: Setting
[dir]
> PARM_BASE to /gpfs/fs1/work/zhuming/METplus4/parm
>
> 05/19 14:34:36.348 metplus (config_metplus.py:321) INFO:
METPLUS_CONF:
> /glade/scratch/zhuming/metplus4/metplus_final.conf written here.
>
> 05/19 14:34:36.349 metplus (config_metplus.py:137) INFO: Completed
METplus
> configuration setup.
>
> 05/19 14:34:36.349 metplus (met_util.py:68) INFO: Running METplus
> v4.0.0-beta5 called with command:
> /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta5/ush/run_metplus.py
-c
> ./PointStat.conf -c ./defaults.conf
>
> 05/19 14:34:36.349 metplus (met_util.py:70) INFO: Log file:
> /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
>
> 05/19 14:34:36.358 metplus.Usage (met_util.py:186) ERROR: Fatal
error
> occurred
>
> Traceback (most recent call last):
>
> File
>
> "/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/util/met_util.py",
> line 128, in run_metplus
>
> instance=instance)
>
> File
>
> "/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/wrappers/usage_wrapper.py",
> line 19, in __init__
>
> self.available_processes = list(set(val for val in
> met_util.LOWER_TO_WRAPPER_NAME.values()))
>
> AttributeError: module 'metplus.util.met_util' has no attribute
> 'LOWER_TO_WRAPPER_NAME'
>
>
> During handling of the above exception, another exception occurred:
>
>
> Traceback (most recent call last):
>
> File
>
> "/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/util/met_util.py",
> line 136, in run_metplus
>
> raise NameError("There was a problem loading "
>
> NameError: There was a problem loading Usage wrapper.
>
> 05/19 14:34:36Z run-METplus-metplus.Usage: ERROR: Fatal error
occurred
>
> 05/19 14:34:36.359 metplus.Usage (met_util.py:187) INFO: Check the
log file
> for more information:
> /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
>
> 05/19 14:34:36.360 metplus (met_util.py:834) INFO:
> /glade/scratch/zhuming/metplus4/metplus_final.conf: write
metplus.conf here
>
> 05/19 14:34:36.365 metplus (met_util.py:218) ERROR: METplus has
finished
> running but had 1 error.
>
> 05/19 14:34:36Z run-METplus-metplus: ERROR: METplus has finished
running
> but had 1 error.
>
> 05/19 14:34:36.365 metplus (met_util.py:219) INFO: Check the log
file for
> more information:
> /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
>
> (NPL) cheyenne1:/glade/work/zhuming/METplus4>
>
> On Mon, May 3, 2021 at 3:47 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > The METplus 4.0.0-beta5 release is now available on Cheyenne. If
you are
> > able to test that this version works and fixes some of the issues
you
> > mentioned in this ticket before the 4.0.0 release next week, that
would
> be
> > great! Information on how to use the new version can be found
here:
> >
> > https://dtcenter.org/community-code/metplus/metplus-4-0-existing-
builds
> > (Under NCAR > CHEYENNE > METplus-4.0.0-beta5)
> >
> > Here are the commands to set it up:
> >
> > module use /glade/p/ral/jntp/MET/METplus/modulefiles
> > module load metplus/4.0.0-beta5
> > ncar_pylib
> >
> > Please let me know if you have any issues or questions.
> >
> > Thanks,
> > George
> >
> > On Thu, Apr 22, 2021 at 11:40 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Zhuming,
> > >
> > > This is John HG. George asked me to take a look at this
question. Since
> > > you're asking about "non-standard" levels, I assume you're
asking about
> > the
> > > verification of the pressure levels against the ADPUPA message
type.
> > >
> > > In general, the logic is this...
> > > For each point observation, loop through all the verification
tasks and
> > see
> > > if it should be used.
> > > For example, let's talk about the 4 level entry for VAR3:
> > >
> > > FCST_VAR3_NAME = v_tr_p
> > > FCST_VAR3_LEVELS = ...(0,3,*,*)...
> > > OBS_VAR3_NAME = VGRD
> > > OBS_VAR3_LEVELS = ...P575-625...
> > >
> > > The point observation will be used here if it's a "VGRD"
observation
> and
> > > has a level value between 575 and 625. This verification task
has been
> > > configured to only verify a single forecast level of data:
> > v_tr_p(0,3,*,*).
> > > So no vertical interpolation is done. MET interpolates
horizontally
> from
> > > the forecast grid points to the observation lat/lon location
(using
> > > user-configurable interpolation options).
> > >
> > > To put it simply, all obs between 575 and 625 mb are compared
directly
> to
> > > the forecast values from v_tr_p(0,3,*,*). If the model output is
> actually
> > > on pressure levels, then that's a reasonable thing to do. If the
> model's
> > > vertical coordinate is something other than pressure levels,
then
> that's
> > a
> > > potential source for large errors.
> > >
> > > Note that MET does do vertical interpolation when processing the
GRIB
> > > output from the Unified Post Processor tool. But all the details
of
> that
> > > depend on how the configuration file is setup. I'll explain
below.
> > >
> > > To date we have avoided including direct knowledge about the
vertical
> > > coordinate of each modelling system in the verification code.
The goal
> is
> > > to create verification tools applicable to many models, rather
than
> > > customizing them for each. We do realize that desire for the
latter and
> > are
> > > exploring ways to pull data from the IODA data assimilation
system
> > instead,
> > > which necessarily knows a lot about each model.
> > >
> > > So let's say you were using GRIB data, and configured it like
this:
> > >
> > > VAR3_NAME = VGRD
> > > VAR3_LEVELS = P500-1000
> > >
> > > Since we haven't prefixed these options with FCST_ and OBS_, the
same
> > > settings are applied to both.
> > >
> > > MET would read ALL forecast GRIB records from the model file
with
> > pressures
> > > between 500 and 1000. Perhaps, that's every 100mb or perhaps
every
> 50mb?
> > It
> > > reads whatever data falls in that range. For each point
observation
> with
> > a
> > > pressure level between 500 and 1000, it'll first find the actual
model
> > > pressure level above/below it. It'll interpolate the forecast
> > > data horizontally to the obs lat/lon location for both. And then
it'll
> > > interpolate those values above/below vertically to the actual
> observation
> > > level, linear in the log of pressure.
> > >
> > > This issue is that the processing of pressure LAYERS is not
currently
> > > supported for NetCDF data I believe. And the code could be
enhanced to
> > > support that. But again, this only makes sense if the model's
vertical
> > > coordinate is actually pressure levels and not something else.
> > >
> > > Hope that helps clarify.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Apr 22, 2021 at 9:52 AM zhuming at ucar.edu via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >
> > > > Hi George,
> > > >
> > > > My verification result shows that there is a big error on a
> > non-standard
> > > > level, so my question is:
> > > > Is the forecast grid match to Obs, or obs station match to
Forecast?
> I
> > > need
> > > > to do is the forecast grid interpolated to the obs location
and
> > vertical
> > > > levels. So how to set the levels?
> > > >
> > > > Thanks,
> > > > Zhuming
> > > >
> > > > FCST_VAR3_NAME = v_10m_tr
> > > >
> > > > FCST_VAR3_LEVELS = (0,*,*)
> > > >
> > > > FCST_VAR3_THRESH = >=5, <5
> > > >
> > > > OBS_VAR3_NAME = VGRD
> > > >
> > > > OBS_VAR3_LEVELS = Z10
> > > >
> > > > OBS_VAR3_THRESH = >=5, <5
> > > >
> > > > OBS_VAR3_OPTIONS = message_type = ["ADPSFC"];
> > > >
> > > >
> > > > FCST_VAR3_NAME = v_tr_p
> > > >
> > > > FCST_VAR3_LEVELS =
> > > >
> > > >
> > >
> >
>
(0,0,*,*),(0,1,*,*),(0,2,*,*),(0,3,*,*),(0,4,*,*),(0,5,*,*),(0,6,*,*),(0,7,*,*),(0,8,*,*),(0,9,*,*)
> > > >
> > > > FCST_VAR3_THRESH = >=5, <5
> > > >
> > > > OBS_VAR3_NAME = VGRD
> > > >
> > > > OBS_VAR3_LEVELS =
> > > >
> > > >
> > >
> >
> P900-950,P825-875,P675-725,P575-625,P475-525,P375-425,P275-325,P225-
275,P175-225,P75-125
> > > >
> > > > OBS_VAR3_THRESH = >=5, <5
> > > >
> > > > OBS_VAR3_OPTIONS = message_type = ["ADPUPA"];
> > > >
> > > >
> > > >
> > > > On Thu, Apr 15, 2021 at 1:52 PM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Zhuming,
> > > > >
> > > > > There was indeed a bug in the code. Howard has fixed the
issue with
> > > this
> > > > > pull request: https://github.com/dtcenter/MET/pull/1756
> > > > > This fix will be included in the MET 10.0.0-beta5 release
that is
> > > > > tentatively scheduled for next week. This beta version can
be
> > installed
> > > > on
> > > > > cheyenne so you can test that the update fixes your issue. I
will
> > > follow
> > > > up
> > > > > and let you know when the beta release is available on
cheyenne.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Wed, Apr 14, 2021 at 5:28 PM George McCabe
<mccabe at ucar.edu>
> > wrote:
> > > > >
> > > > > > Hi Zhuming,
> > > > > >
> > > > > > I tried to get this to work on cheyenne with your data,
but was
> not
> > > > > > successful. I am able to read the data with
plot_data_plane and
> > > > generate
> > > > > an
> > > > > > image, so I think there may be a bug in the code. I have
> forwarded
> > > info
> > > > > to
> > > > > > one of our C++ developers to look into it. I will report
back
> when
> > I
> > > > know
> > > > > > more.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > >>
> > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > > >>
> > > > > >> Hi George,
> > > > > >>
> > > > > >> It still can't find the correct time. I tried
> > > > > >>
> > > > > >> FCST_VAR1_LEVELS = (*,*,*)
> > > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> > > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
> > > > > >>
> > > > > >> Thanks!
> > > > > >>
> > > > > >> Zhuming
> > > > > >>
> > > > > >> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Zhuming,
> > > > > >> >
> > > > > >> > It looks like you are requesting the first time in the
12 hour
> > > file
> > > > > for
> > > > > >> > each run:
> > > > > >> >
> > > > > >> > FCST_VAR1_LEVELS = (0,*,*)
> > > > > >> > ...
> > > > > >> > FCST_VAR2_LEVELS = (0,*,*)
> > > > > >> >
> > > > > >> > The first dimension in these files is the time offset.
You
> will
> > > need
> > > > > the
> > > > > >> > correct index that corresponds to each valid time.
> Fortunately,
> > > MET
> > > > > >> makes
> > > > > >> > this easy by allowing you to specify the valid time for
a time
> > > > > dimension
> > > > > >> > explicitly and it will find the correct data. You can
set:
> > > > > >> >
> > > > > >> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > > >> > ...
> > > > > >> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > > >> >
> > > > > >> > And it will substitute the valid time for each run. Let
me
> know
> > if
> > > > you
> > > > > >> have
> > > > > >> > any questions.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > George
> > > > > >> >
> > > > > >> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT
<
> > > > > >> met_help at ucar.edu
> > > > > >> > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > >
> > > > > >> > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >
> > > > > >> > >
> > > > > >> > > Hi George,
> > > > > >> > >
> > > > > >> > > Thanks for your email, I just would like to send you
for
> help.
> > > > > >> > >
> > > > > >> > > I already resolved the last problem with the WRF
forecast,
> > thank
> > > > you
> > > > > >> for
> > > > > >> > > your help.
> > > > > >> > >
> > > > > >> > > I also need to verify the MPAS model forecast. These
12 hour
> > > > > >> > > MAPS forecasts are stored in a one file, such as
> > > > > >> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h
forecasts
> > > > > >> > > 2015071512/mpas_wrf_cf_2015071512.nc
> > > > > >> > >
> > > > > >> > > It seems MET is doing fine on the first 2 hours, but
no
> > matched
> > > > > pairs
> > > > > >> for
> > > > > >> > > later hours.
> > > > > >> > > I'm not sure what's wrong with it? the time is not
matched?
> > > > > >> > >
> > > > > >> > > see the log file
> > > > > >> > >
> > > > >
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > > > > >> > > config file
/gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Zhuming
> > > > > >> > >
> > > > > >> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT
<
> > > > > >> met_help at ucar.edu>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi Zhuming,
> > > > > >> > > >
> > > > > >> > > > Just following up on this ticket. Were you able to
get
> this
> > > > > >> resolved?
> > > > > >> > Let
> > > > > >> > > > me know if you need more assistance.
> > > > > >> > > >
> > > > > >> > > > Thanks,
> > > > > >> > > > George
> > > > > >> > > >
> > > > > >> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > > > >> > > > > Hi Zhuming,
> > > > > >> > > > >
> > > > > >> > > > > Typically directories that contain data for an
entire
> day
> > > are
> > > > > >> named
> > > > > >> > > > > with
> > > > > >> > > > > YYYYMMDD, so your filename template tag would be
> > > > > >> {valid?fmt=%Y%m%d},
> > > > > >> > > > > however that may not always be the case, as you
are
> seeing
> > > > here.
> > > > > >> > There
> > > > > >> > > > > are
> > > > > >> > > > > a couple ways to handle this.
> > > > > >> > > > >
> > > > > >> > > > > If you know that the directory will always be
YYYYMMDD
> > with
> > > 00
> > > > > at
> > > > > >> the
> > > > > >> > > > > end,
> > > > > >> > > > > you can simply put 00 in the template instead of
adding
> %H
> > > in
> > > > > the
> > > > > >> > tag,
> > > > > >> > > > > i.e.
> > > > > >> > > > >
> > > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > >> > > > >
> > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> > > > > >> 00_cf.nc
> > > > > >> > > > >
> > > > > >> > > > > If the HH value can vary, for instance if there
are 2
> > > > > directories
> > > > > >> > > > > created
> > > > > >> > > > > per day (i.e. 2015071400 and 2015071412), then
you can
> > use
> > > > the
> > > > > >> > > > > truncate
> > > > > >> > > > > keyword to handle it.
> > > > > >> > > > >
> > > > > >> > > > > If directories are created every 24 hours:
> > > > > >> > > > >
> > > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > >> > > > >
> > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > > > >> > > > > %d_%H}:00:
> > > > > >> > > > > 00_cf.nc
> > > > > >> > > > >
> > > > > >> > > > > If directories are created every 12 hours:
> > > > > >> > > > >
> > > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > >> > > > >
> > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > > > >> > > > > %d_%H}:00:
> > > > > >> > > > > 00_cf.nc
> > > > > >> > > > >
> > > > > >> > > > > Let me know if that doesn't work for you.
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > George
> > > > > >> > > > >
> > > > > >> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu
via RT
> > > > > >> > > > > <met_help at ucar.edu>
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Hi George,
> > > > > >> > > > > >
> > > > > >> > > > > > Processing Obs data using "PB2NC" are right
now.
> > > > > >> > > > > >
> > > > > >> > > > > > Now I'm using "PointStat" , I set VALID_BEG =
> > 2015071500,
> > > > > >> > VALID_END =
> > > > > >> > > > > > 2015071501. It looks for the forecast data
2015071500/
> > > > > >> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and
2015071501/
> > > > > >> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > > > >> > > > > > But my forecast data are all put in dir of
> 2015071500/.
> > > > > >> > > > > > So how could I set
FCST_POINT_STAT_INPUT_TEMPLATE to
> > get
> > > > the
> > > > > >> right
> > > > > >> > > > > > fcst
> > > > > >> > > > > > data?
> > > > > >> > > > > >
> > > > > >> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > >> > > > > >
> > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> > > > > >> > 00_cf.nc
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > config file:
> > > > /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > >> > > > > >
> > > > > >> > > > > > log file:
> > > > > >> > > > > >
> > > > > >> > >
> > > > >
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Thanks,
> > > > > >> > > > > >
> > > > > >> > > > > > Zhuming
> > > > > >> > > > > >
> > > > > >> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe
via RT
> > > > > >> > > > > > <met_help at ucar.edu>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi Zhuming,
> > > > > >> > > > > > >
> > > > > >> > > > > > > Looking at the log output, it looks like the
data is
> > > > stored
> > > > > in
> > > > > >> > the
> > > > > >> > > > > > prepBUFR
> > > > > >> > > > > > > files differently that you may have thought:
> > > > > >> > > > > > >
> > > > > >> > > > > > > DEBUG 2: PrepBufr Time Center:
> > 20150715_000000
> > > > > >> > > > > > > DEBUG 2: Searching Time Window:
> > 20150715_030000
> > > to
> > > > > >> > > > > > 20150715_040000
> > > > > >> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55%
60% 65%
> 70%
> > > 75%
> > > > > 80%
> > > > > >> > 85%
> > > > > >> > > > > > > 90%
> > > > > >> > > > > > 95%
> > > > > >> > > > > > > 100%
> > > > > >> > > > > > >
> > > > > >> > > > > > > DEBUG 2: Total Messages processed
=
> > 590836
> > > > > >> > > > > > > DEBUG 2: Rejected based on message type
= 0
> > > > > >> > > > > > > DEBUG 2: Rejected based on station id
= 0
> > > > > >> > > > > > > DEBUG 2: Rejected based on valid time
=
> > 590815
> > > > > >> > > > > > > DEBUG 2: Rejected based on masking grid
= 20
> > > > > >> > > > > > > DEBUG 2: Rejected based on masking polygon
= 0
> > > > > >> > > > > > > DEBUG 2: Rejected based on elevation
= 0
> > > > > >> > > > > > > DEBUG 2: Rejected based on pb report type
= 1
> > > > > >> > > > > > > DEBUG 2: Rejected based on input report type
= 0
> > > > > >> > > > > > > DEBUG 2: Rejected based on instrument type
= 0
> > > > > >> > > > > > > DEBUG 2: Rejected based on zero observations
= 0
> > > > > >> > > > > > > DEBUG 2: Total Messages retained
= 0
> > > > > >> > > > > > > DEBUG 2: Total observations retained or
derived = 0
> > > > > >> > > > > > > DEBUG 1: Obs time between *20150714_210000
and
> > > > > >> 20150715_030000*
> > > > > >> > > > > > > WARNING:
> > > > > >> > > > > > > WARNING: process_pbfile() -> No PrepBufr
messages
> > > retained
> > > > > >> from
> > > > > >> > > > > > > file:
> > > > > >> > > > > > >
/glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > > >> > > > > > > prepbufr.gdas.2015071500.nr
> > > > > >> > > > > > > WARNING:
> > > > > >> > > > > > > ERROR :
> > > > > >> > > > > > > ERROR : write_netcdf_hdr_data() -> No
PrepBufr
> > messages
> > > > > >> > retained.
> > > > > >> > > > > > Nothing
> > > > > >> > > > > > > to write.
> > > > > >> > > > > > > ERROR :
> > > > > >> > > > > > >
> > > > > >> > > > > > > The prepbufr.gdas.2015071500.nr file actually
> > contains
> > > > data
> > > > > >> from
> > > > > >> > > > > > > 20150714_210000 to 20150715_030000, not
> > 20150715_000000
> > > to
> > > > > >> > > > > > 20150715_060000.
> > > > > >> > > > > > >
> > > > > >> > > > > > > To get the correct file for each run, you
will have
> to
> > > > shift
> > > > > >> the
> > > > > >> > > > > > > valid
> > > > > >> > > > > > time
> > > > > >> > > > > > > before truncating it so that it gets the
correct
> file
> > > for
> > > > > each
> > > > > >> > run.
> > > > > >> > > > > > Instead
> > > > > >> > > > > > > of setting:
> > > > > >> > > > > > >
> > > > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > >> > > > > > >
> > > > > >> >
> > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > > > >> > > > > > >
> > > > > >> > > > > > > You will have to also shift forward 3 hours,
i.e.:
> > > > > >> > > > > > >
> > > > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > > > >> > > > > > >
> > > > > >> > > > > > > This says to add 3 hours to the current valid
time
> > > before
> > > > > >> > > > > > > truncating to
> > > > > >> > > > > > the
> > > > > >> > > > > > > nearest 6 hours.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H =
0Z
> > > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > >> > > > > > >
> > > > > >> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H =
0Z
> > > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > >> > > > > > >
> > > > > >> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H =
0Z
> > > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > >> > > > > > >
> > > > > >> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H =
6Z
> > > > > >> > > > > > > File: prepbufr.gdas.2015071506.nr
> > > > > >> > > > > > >
> > > > > >> > > > > > > Let me know if that doesn't work.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Thanks,
> > > > > >> > > > > > > George
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM
zhuming at ucar.edu via
> > RT
> > > <
> > > > > >> > > > > > met_help at ucar.edu>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > <URL:
> > > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > >> > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Hi George,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Now makes me confused is why processed obs
of
> > several
> > > > > times
> > > > > >> are
> > > > > >> > > > > > missing.
> > > > > >> > > > > > > > There has files of time at 00,01,02, but no
> > 03,04,05;
> > > > has
> > > > > >> > > > > > > > 06,07,08, but
> > > > > >> > > > > > > no
> > > > > >> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has
> 18,19,20,
> > > but
> > > > > with
> > > > > >> > > > > > > > 21,no
> > > > > >> > > > > > 22,23.
> > > > > >> > > > > > > > From yours and mines:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6
17:52
> > > > > >> > pbs.2015071500.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6
17:56
> > > > > >> > pbs.2015071501.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6
18:00
> > > > > >> > pbs.2015071502.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6
18:14
> > > > > >> > pbs.2015071506.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6
18:18
> > > > > >> > pbs.2015071507.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6
18:22
> > > > > >> > pbs.2015071508.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6
18:36
> > > > > >> > pbs.2015071512.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6
18:40
> > > > > >> > pbs.2015071513.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6
18:40
> > > > > >> > pbs.2015071514.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6
18:42
> > > > > >> > pbs.2015071515.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > total 96112
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7
15:11
> > > > > >> > > pbs.2015071500.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7
15:14
> > > > > >> > > pbs.2015071501.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7
15:18
> > > > > >> > > pbs.2015071502.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7
15:33
> > > > > >> > > pbs.2015071506.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7
15:36
> > > > > >> > > pbs.2015071507.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7
15:40
> > > > > >> > > pbs.2015071508.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7
15:55
> > > > > >> > > pbs.2015071512.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7
15:59
> > > > > >> > > pbs.2015071513.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7
16:03
> > > > > >> > > pbs.2015071514.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7
16:18
> > > > > >> > > pbs.2015071518.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7
16:22
> > > > > >> > > pbs.2015071519.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7
16:27
> > > > > >> > > pbs.2015071520.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7
16:31
> > > > > >> > > pbs.2015071521.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7
16:43
> > > > > >> > > pbs.2015071600.nc
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Thanks,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Zhuming
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George
McCabe via
> RT
> > > > > >> > > > > > > > <met_help at ucar.edu
> > > > > >> > > > > > >
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Hi Zhuming,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks for the information. You will need
to use
> > the
> > > > > >> truncate
> > > > > >> > > > > > > > > keyword
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > find the files, then set the valid
beg/end range
> > to
> > > 1
> > > > > >> hour.
> > > > > >> > > > > > > > > This is
> > > > > >> > > > > > > what
> > > > > >> > > > > > > > I
> > > > > >> > > > > > > > > changed from your PB2NC1.conf config
file:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *PB2NC_VALID_BEGIN =
> > > > > {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
> > > > > >> =
> > > > > >> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > >> > > > > > > > >
> > > > > >> > > >
> > > > >
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > My config file is on cheyenne here:
> > > > > >> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I just started running it and the output
is
> being
> > > > > written
> > > > > >> > here:
> > > > > >> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > For PointStat, you will want to loop over
the
> same
> > > > > times.
> > > > > >> I
> > > > > >> > > > > > > > > noticed
> > > > > >> > > > > > you
> > > > > >> > > > > > > > > specify message type in the
VAR<n>_OPTIONS, but
> > you
> > > do
> > > > > not
> > > > > >> > need
> > > > > >> > > > > > > > > to do
> > > > > >> > > > > > > > this
> > > > > >> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE
> variable
> > > that
> > > > > >> > defines
> > > > > >> > > > > > > > > this.
> > > > > >> > > > > > > This
> > > > > >> > > > > > > > > is what I changed in your PointStat1.conf
file:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *VALID_END = 2015071600*
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *VALID_INCREMENT =
1HPOINT_STAT_OBS_VALID_BEG
> > > > > >> > > > > > =POINT_STAT_OBS_VALID_BEG
> > > > > >> > > > > > > > =*
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > >> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > >> > > > > > > > > message_type =
["ADPSFC"];#OBS_VAR3_OPTIONS =
> > > > > >> message_type =
> > > > > >> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > >> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR =
> {OUTPUT_BASE}/pb2nc1*
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Or you can view my file in the same
directory. I
> > > > usually
> > > > > >> try
> > > > > >> > to
> > > > > >> > > > > > > > > test
> > > > > >> > > > > > > > with a
> > > > > >> > > > > > > > > single run time to make sure it is
working
> > properly,
> > > > so
> > > > > I
> > > > > >> set
> > > > > >> > > > > > VALID_END
> > > > > >> > > > > > > > > back to 2015071500. I ran it and it
produced an
> > > output
> > > > > >> file,
> > > > > >> > > > > > > > > but
> > > > > >> > > > > > there
> > > > > >> > > > > > > > were
> > > > > >> > > > > > > > > no values besides the header. I checked
the log
> > file
> > > > and
> > > > > >> saw
> > > > > >> > > > > > > > > this:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus
TMP/Z2,
> > for
> > > > > >> > > > > > > > > observation type
> > > > > >> > > > > > > > > ADPSFC, over region FULL, for
interpolation
> method
> > > > > >> BILIN(4),
> > > > > >> > > > > > > > > using 0
> > > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs
> =
> > > > > 0DEBUG
> > > > > >> 3:
> > > > > >> > > > > > > Observations
> > > > > >> > > > > > > > > processed = 393373DEBUG 3: Rejected:
station
> id
> > > > > =
> > > > > >> > > > > > > > > 0DEBUG 3:
> > > > > >> > > > > > > > > Rejected: obs type = 390528DEBUG
3:
> > Rejected:
> > > > > valid
> > > > > >> > time
> > > > > >> > > > > > =
> > > > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value =
0DEBUG 3:
> > > > > Rejected:
> > > > > >> off
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > grid
> > > > > >> > > > > > > > =
> > > > > >> > > > > > > > > 2586DEBUG 3: Rejected: topography =
0DEBUG
> 3:
> > > > > >> Rejected:
> > > > > >> > > > > > > > > level
> > > > > >> > > > > > > > mismatch
> > > > > >> > > > > > > > > = 259DEBUG 3: Rejected: quality marker =
0DEBUG
> > 3:
> > > > > >> Rejected:
> > > > > >> > > > > > message
> > > > > >> > > > > > > > type
> > > > > >> > > > > > > > > = 0DEBUG 3: Rejected: masking region =
0DEBUG
> 3:
> > > > > >> Rejected:
> > > > > >> > bad
> > > > > >> > > > > > fcst
> > > > > >> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo
mean =
> > > 0DEBUG
> > > > 3:
> > > > > >> > > > > > > > > Rejected:
> > > > > >> > > > > > bad
> > > > > >> > > > > > > > > climo stdev = 0DEBUG 3: Rejected:
duplicates
> > =
> > > > > 0DEBUG
> > > > > >> > > > > > > > > 2:DEBUG 2:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > >> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*)
versus
> > > UGRD/Z10,
> > > > > for
> > > > > >> > > > > > observation
> > > > > >> > > > > > > > type
> > > > > >> > > > > > > > > ADPSFC, over region FULL, for
interpolation
> method
> > > > > >> BILIN(4),
> > > > > >> > > > > > > > > using 0
> > > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs
> =
> > > > > 0DEBUG
> > > > > >> 3:
> > > > > >> > > > > > > Observations
> > > > > >> > > > > > > > > processed = 393373*
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *DEBUG 3: Rejected: station id =
0DEBUG 3:
> > > > > Rejected:
> > > > > >> obs
> > > > > >> > > > > > > > > type
> > > > > >> > > > > > > > =
> > > > > >> > > > > > > > > 382965DEBUG 3: Rejected: valid time
=
> 0DEBUG
> > 3:
> > > > > >> > Rejected:
> > > > > >> > > > > > > > > bad
> > > > > >> > > > > > obs
> > > > > >> > > > > > > > > value = 0DEBUG 3: Rejected: off the
grid =
> > > > > 9240DEBUG
> > > > > >> 3:
> > > > > >> > > > > > Rejected:
> > > > > >> > > > > > > > > topography = 0DEBUG 3: Rejected:
level
> > mismatch
> > > > =
> > > > > >> > > > > > > > > 1168DEBUG 3:
> > > > > >> > > > > > > > > Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > > message
> > > > > >> type
> > > > > >> > > > > > > > > =
> > > > > >> > > > > > > 0DEBUG
> > > > > >> > > > > > > > 3:
> > > > > >> > > > > > > > > Rejected: masking region = 0DEBUG 3:
Rejected:
> > bad
> > > > fcst
> > > > > >> > value
> > > > > >> > > > > > > > > =
> > > > > >> > > > > > > 0DEBUG
> > > > > >> > > > > > > > 3:
> > > > > >> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3:
Rejected:
> > bad
> > > > > climo
> > > > > >> > stdev
> > > > > >> > > > > > > > > =
> > > > > >> > > > > > > 0DEBUG
> > > > > >> > > > > > > > 3:
> > > > > >> > > > > > > > > Rejected: duplicates = 0DEBUG
2:DEBUG 2:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > >> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*)
versus
> > > VGRD/Z10,
> > > > > for
> > > > > >> > > > > > observation
> > > > > >> > > > > > > > type
> > > > > >> > > > > > > > > ADPSFC, over region FULL, for
interpolation
> method
> > > > > >> BILIN(4),
> > > > > >> > > > > > > > > using 0
> > > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs
> =
> > > > > 0DEBUG
> > > > > >> 3:
> > > > > >> > > > > > > Observations
> > > > > >> > > > > > > > > processed = 393373DEBUG 3: Rejected:
station
> id
> > > > > =
> > > > > >> > > > > > > > > 0DEBUG 3:
> > > > > >> > > > > > > > > Rejected: obs type = 382965DEBUG
3:
> > Rejected:
> > > > > valid
> > > > > >> > time
> > > > > >> > > > > > =
> > > > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value =
0DEBUG 3:
> > > > > Rejected:
> > > > > >> off
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > grid
> > > > > >> > > > > > > > =
> > > > > >> > > > > > > > > 9240DEBUG 3: Rejected: topography =
0DEBUG
> 3:
> > > > > >> Rejected:
> > > > > >> > > > > > > > > level
> > > > > >> > > > > > > > mismatch
> > > > > >> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker
=
> 0DEBUG
> > 3:
> > > > > >> > Rejected:
> > > > > >> > > > > > message
> > > > > >> > > > > > > > > type = 0DEBUG 3: Rejected: masking
region =
> > > 0DEBUG
> > > > > 3:
> > > > > >> > > > > > > > > Rejected:
> > > > > >> > > > > > bad
> > > > > >> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad
climo mean
> > =
> > > > > >> 0DEBUG 3:
> > > > > >> > > > > > Rejected:
> > > > > >> > > > > > > > bad
> > > > > >> > > > > > > > > climo stdev = 0*
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > *DEBUG 3: Rejected: duplicates =
0DEBUG
> > 2:DEBUG
> > > > 2:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > >> > > > > > > > > 2:DEBUG 1: Output file:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > This describes the reason why each
observation
> was
> > > > > >> rejected.
> > > > > >> > > > > > > > > Some
> > > > > >> > > > > > > > > observations didn't match the message
type you
> > > > > requested,
> > > > > >> > some
> > > > > >> > > > > > > > > were
> > > > > >> > > > > > not
> > > > > >> > > > > > > > > within the forecast grid, and some did
not match
> > the
> > > > > >> vertical
> > > > > >> > > > > > > > > level.
> > > > > >> > > > > > > You
> > > > > >> > > > > > > > > will have to make adjustments to your
settings
> so
> > > that
> > > > > the
> > > > > >> > data
> > > > > >> > > > > > matches
> > > > > >> > > > > > > > up
> > > > > >> > > > > > > > > properly. Maybe the grid you are using in
the
> > PB2NC
> > > > > config
> > > > > >> > file
> > > > > >> > > > > > > > > does
> > > > > >> > > > > > > not
> > > > > >> > > > > > > > > match the forecast grid and you are
missing
> > values.
> > > If
> > > > > >> that
> > > > > >> > is
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > case,
> > > > > >> > > > > > > > > you will have to reconfigure and rerun
PB2NC to
> > get
> > > > > >> output on
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > correct
> > > > > >> > > > > > > > > gr5id. Maybe the observation points don't
line
> up
> > > with
> > > > > the
> > > > > >> > > > > > > > > vertical
> > > > > >> > > > > > > level
> > > > > >> > > > > > > > > of the model fields. You can use a range
of
> level
> > > > values
> > > > > >> for
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > observation data to allow data within
that
> range,
> > > i.e.
> > > > > >> Z0-10.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Let me know if you are still having
issues
> getting
> > > > this
> > > > > to
> > > > > >> > work
> > > > > >> > > > > > > > > and I
> > > > > >> > > > > > > can
> > > > > >> > > > > > > > > help out.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks,
> > > > > >> > > > > > > > > George
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM
zhuming at ucar.edu
> > via
> > > > RT
> > > > > <
> > > > > >> > > > > > > > met_help at ucar.edu>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > <URL:
> > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Hi George,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks for your reply!
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > I'm still confused, I need to output
hourly
> obs
> > > data
> > > > > to
> > > > > >> do
> > > > > >> > > > > > > > > > hourly
> > > > > >> > > > > > > > > > verification.
> > > > > >> > > > > > > > > > The original GDAS Prepbufr data has 4
files
> one
> > > day,
> > > > > >> should
> > > > > >> > > > > > > > > > with 6
> > > > > >> > > > > > > > hours
> > > > > >> > > > > > > > > > obs data in one file.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > After "PB2NC", I can see
pbs.2015071500.nc
> (obs
> > > > time
> > > > > is
> > > > > >> > 00?
> > > > > >> > > > > > > > > > or 6
> > > > > >> > > > > > > times
> > > > > >> > > > > > > > > of
> > > > > >> > > > > > > > > > 00,01,02,03,04,05?),
pbs.2015071506.nc...
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > my config file :
> > > > > >> > > > > >
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > >> > > > > > > > > > and
> > > /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > Zhuming
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George
McCabe
> via
> > > RT
> > > > <
> > > > > >> > > > > > > met_help at ucar.edu
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > Hi Zhuming,
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > To run the tools hourly you would set
the
> > > > > >> VALID_INCREMENT
> > > > > >> > > > > > > > > > > value
> > > > > >> > > > > > for
> > > > > >> > > > > > > > 1H.
> > > > > >> > > > > > > > > > > Here is an example that runs once per
hour
> > for a
> > > > day
> > > > > >> from
> > > > > >> > > > > > > 2015-07-15
> > > > > >> > > > > > > > 0Z
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > 2017-07-16 0Z:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > VALID_BEG = 2015071500
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > VALID_END = 2015071600
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > VALID_INCREMENT = 1H
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > LEAD_SEQ = 0
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Each run time will use the
> > PB2NC_INPUT_TEMPLATE
> > > to
> > > > > >> find
> > > > > >> > > > > > > > > > > files
> > > > > >> > > > > > whose
> > > > > >> > > > > > > > > > > filenames match the time you need.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > To read prepBUFR observations for 12
hours,
> > you
> > > > will
> > > > > >> want
> > > > > >> > > > > > > > > > > to set
> > > > > >> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END
to
> > specify
> > > > the
> > > > > >> > range
> > > > > >> > > > > > > > > > > around
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > valid
> > > > > >> > > > > > > > > > > time to process. These values depend
on what
> > is
> > > > > >> contained
> > > > > >> > > > > > > > > > > inside
> > > > > >> > > > > > > your
> > > > > >> > > > > > > > > > > prepBUFR files. If you want to use
data from
> > the
> > > > > valid
> > > > > >> > time
> > > > > >> > > > > > > > > > > of
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > > filename
> > > > > >> > > > > > > > > > > backwards 12 hours, you would set:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > PB2NC_VALID_BEGIN =
> > > > {valid?fmt=%Y%m%d_%H?shift=-12H
> > > > > }
> > > > > >> > > > > > > > > > > PB2NC_VALID_END =
{valid?fmt=%Y%m%d_%H}
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > For a runtime of 2015071512, this
would be
> > > > > substituted
> > > > > >> > as:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > valid_beg: 20150715_00
> > > > > >> > > > > > > > > > > valid_end: 20150715_12
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > If you want to create an output file
for
> each
> > > hour
> > > > > in
> > > > > >> > PB2NC
> > > > > >> > > > > > > > > > > you
> > > > > >> > > > > > > would
> > > > > >> > > > > > > > > > need
> > > > > >> > > > > > > > > > > to make sure that the hour is
specified in
> the
> > > > > >> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > PointStat has similar config
variables to
> set
> > > the
> > > > > >> valid
> > > > > >> > beg
> > > > > >> > > > > > > > > > > and
> > > > > >> > > > > > end
> > > > > >> > > > > > > > > range
> > > > > >> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > >> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > If your prepBUFR files are separated
out
> into
> > > > > multiple
> > > > > >> > > > > > > > > > > files that
> > > > > >> > > > > > > > need
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > be read into PB2NC, you can set a
file
> window
> > > > around
> > > > > >> the
> > > > > >> > > > > > > > > > > valid
> > > > > >> > > > > > time
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > > allow files to be passed into the
app. More
> > > > > >> information
> > > > > >> > on
> > > > > >> > > > > > > > > > > that
> > > > > >> > > > > > > > > > > functionality is here:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > > > >> > > > > > windows-to-find-valid-files
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > If you are still having trouble
getting this
> > > > > >> configured,
> > > > > >> > > > > > > > > > > you
> > > > > >> > > > > > could
> > > > > >> > > > > > > > send
> > > > > >> > > > > > > > > > me
> > > > > >> > > > > > > > > > > the paths of the files you are trying
to
> read
> > > and
> > > > > what
> > > > > >> > data
> > > > > >> > > > > > > > > > > is
> > > > > >> > > > > > > > > contained
> > > > > >> > > > > > > > > > in
> > > > > >> > > > > > > > > > > them so I can give you suggestions on
what
> to
> > > set
> > > > > for
> > > > > >> the
> > > > > >> > > > > > > > > configuration.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > > George
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie
> > Prestopnik
> > > > via
> > > > > >> RT <
> > > > > >> > > > > > > > > > met_help at ucar.edu>
> > > > > >> > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request
99429
> was
> > > > acted
> > > > > >> upon.
> > > > > >> > > > > > > > > > > > Transaction: Given to mccabe
(George
> McCabe)
> > > by
> > > > > >> jpresto
> > > > > >> > > > > > > > > > > > Queue: met_help
> > > > > >> > > > > > > > > > > > Subject: How to set time for
hourly
> > 12-24
> > > > > hours
> > > > > >> > data
> > > > > >> > > > > > > > > > > > ?
> > > > > >> > > > > > > > > > > > Owner: mccabe
> > > > > >> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > >> > > > > > > > > > > > Status: new
> > > > > >> > > > > > > > > > > > Ticket <URL:
> > > > > >> > > > > > > > >
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > This transaction appears to have no
> content
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > --
> > > > > >> > > > > > > > > > > George McCabe - Software Engineer III
> > > > > >> > > > > > > > > > > National Center for Atmospheric
Research
> > > > > >> > > > > > > > > > > Research Applications Laboratory
> > > > > >> > > > > > > > > > > 303-497-2768
> > > > > >> > > > > > > > > > > ---
> > > > > >> > > > > > > > > > > My working day may not be your
working day.
> > > Please
> > > > > do
> > > > > >> not
> > > > > >> > > > > > > > > > > feel
> > > > > >> > > > > > > > obliged
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > reply to this email outside of your
normal
> > > working
> > > > > >> hours.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > --
> > > > > >> > > > > > > > > George McCabe - Software Engineer III
> > > > > >> > > > > > > > > National Center for Atmospheric Research
> > > > > >> > > > > > > > > Research Applications Laboratory
> > > > > >> > > > > > > > > 303-497-2768
> > > > > >> > > > > > > > > ---
> > > > > >> > > > > > > > > My working day may not be your working
day.
> Please
> > > do
> > > > > not
> > > > > >> > feel
> > > > > >> > > > > > obliged
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > reply to this email outside of your
normal
> working
> > > > > hours.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > --
> > > > > >> > > > > > > George McCabe - Software Engineer III
> > > > > >> > > > > > > National Center for Atmospheric Research
> > > > > >> > > > > > > Research Applications Laboratory
> > > > > >> > > > > > > 303-497-2768
> > > > > >> > > > > > > ---
> > > > > >> > > > > > > My working day may not be your working day.
Please
> do
> > > not
> > > > > feel
> > > > > >> > > > > > > obliged to
> > > > > >> > > > > > > reply to this email outside of your normal
working
> > > hours.
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> > --
> > > > > >> > George McCabe - Software Engineer III
> > > > > >> > National Center for Atmospheric Research
> > > > > >> > Research Applications Laboratory
> > > > > >> > 303-497-2768
> > > > > >> > ---
> > > > > >> > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > >> to
> > > > > >> > reply to this email outside of your normal working
hours.
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Wed Jun 16 10:11:38 2021
Hi George,
I still have questions about setting time on PointStat.
I did run WRF and MPAS for one and half months, twice a day (00utc
and
12utc) , for a total of about 90 runs.
Every run lasts 48hrs (forecast lead 48hrs) and output every 3 hrs.
See
data in /glade/scratch/zhuming/pecan/ctrl/fcst/cf.
So my question is how to set time for so many runs?
see config file /glade/work/zhuming/METplus4/PointStat1.conf
Thanks,
Zhuming
On Wed, May 19, 2021 at 3:03 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> There are a few things going on here:
>
> 1) There is a bug in the Usage wrapper that just prints the list of
> possible wrappers. Thank you for finding this. I will fix this for
the next
> development release.
>
> 2) The reason you are getting the Usage wrapper instead of running
> PointStat is you are passing in the defaults.conf file last.
>
> PointStat.conf has:
> PROCESS_LIST = PointStat
>
> and defaults.conf has:
> PROCESS_LIST = Usage
>
> Since defaults.conf was read after PointStat.conf, the value in
> defaults.conf is used. The defaults.conf file in the METplus
directory
> (under parm/metplus_config) is actually read first automatically
before
> reading any of the config files you pass into the command, so you do
not
> ever need to include it in the command.
>
> 3) You no longer need to put '-c' before your config files on the
command
> line. It doesn't hurt to have them, but they are ignored.
>
> run_metplus.py ./PointStat.conf
>
> 4) The official 4.0.0 release is now available on cheyenne, so you
can use
> that instead. You can set it up by running the following:
>
> module use /glade/p/ral/jntp/MET/METplus/modulefiles
> module load metplus/4.0.0
> ncar_pylib
>
> Let me know if you have any other issues.
>
> Thanks,
> George
>
> On Wed, May 19, 2021 at 2:44 PM zhuming at ucar.edu via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > I tried running MET new version beta5. It seems something is
wrong.
> > see log file
> > /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
> > config file in /glade/work/zhuming/METplus4
> >
> > Thanks!
> > Zhuming
> >
> >
> > (NPL) cheyenne1:/glade/work/zhuming/METplus4>run_metplus.py -c
> > ./PointStat.conf -c ./defaults.conf
> >
> > Running METplus 4.0.0-beta5
> >
> > Starting METplus v4.0.0-beta5
> >
> > 05/19 14:34:36.341 metplus (config_metplus.py:117) INFO: Starting
METplus
> > configuration setup.
> >
> > 05/19 14:34:36.343 metplus (config_metplus.py:247) INFO:
> > /gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/defaults.conf:
Parsed
> > this file
> >
> > 05/19 14:34:36.344 metplus (config_metplus.py:247) INFO:
> ./PointStat.conf:
> > Parsed this file
> >
> > 05/19 14:34:36.345 metplus (config_metplus.py:247) INFO:
./defaults.conf:
> > Parsed this file
> >
> > 05/19 14:34:36.346 metplus (config_metplus.py:436) INFO: Adding:
> > config.LOG_TIMESTAMP='20210519143436'
> >
> > 05/19 14:34:36.346 metplus (config_metplus.py:442) INFO: Replace
[config]
> > LOG_METPLUS with
> > '/glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436'
> >
> > 05/19 14:34:36.348 metplus (config_metplus.py:314) INFO: Setting
[dir]
> > METPLUS_BASE to /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta5
> >
> > 05/19 14:34:36.348 metplus (config_metplus.py:314) INFO: Setting
[dir]
> > PARM_BASE to /gpfs/fs1/work/zhuming/METplus4/parm
> >
> > 05/19 14:34:36.348 metplus (config_metplus.py:321) INFO:
METPLUS_CONF:
> > /glade/scratch/zhuming/metplus4/metplus_final.conf written here.
> >
> > 05/19 14:34:36.349 metplus (config_metplus.py:137) INFO: Completed
> METplus
> > configuration setup.
> >
> > 05/19 14:34:36.349 metplus (met_util.py:68) INFO: Running METplus
> > v4.0.0-beta5 called with command:
> > /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/ush/run_metplus.py -c
> > ./PointStat.conf -c ./defaults.conf
> >
> > 05/19 14:34:36.349 metplus (met_util.py:70) INFO: Log file:
> > /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
> >
> > 05/19 14:34:36.358 metplus.Usage (met_util.py:186) ERROR: Fatal
error
> > occurred
> >
> > Traceback (most recent call last):
> >
> > File
> >
> >
> "/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/util/met_util.py",
> > line 128, in run_metplus
> >
> > instance=instance)
> >
> > File
> >
> >
> "/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/wrappers/usage_wrapper.py",
> > line 19, in __init__
> >
> > self.available_processes = list(set(val for val in
> > met_util.LOWER_TO_WRAPPER_NAME.values()))
> >
> > AttributeError: module 'metplus.util.met_util' has no attribute
> > 'LOWER_TO_WRAPPER_NAME'
> >
> >
> > During handling of the above exception, another exception
occurred:
> >
> >
> > Traceback (most recent call last):
> >
> > File
> >
> >
> "/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/util/met_util.py",
> > line 136, in run_metplus
> >
> > raise NameError("There was a problem loading "
> >
> > NameError: There was a problem loading Usage wrapper.
> >
> > 05/19 14:34:36Z run-METplus-metplus.Usage: ERROR: Fatal error
occurred
> >
> > 05/19 14:34:36.359 metplus.Usage (met_util.py:187) INFO: Check the
log
> file
> > for more information:
> > /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
> >
> > 05/19 14:34:36.360 metplus (met_util.py:834) INFO:
> > /glade/scratch/zhuming/metplus4/metplus_final.conf: write
metplus.conf
> here
> >
> > 05/19 14:34:36.365 metplus (met_util.py:218) ERROR: METplus has
finished
> > running but had 1 error.
> >
> > 05/19 14:34:36Z run-METplus-metplus: ERROR: METplus has finished
running
> > but had 1 error.
> >
> > 05/19 14:34:36.365 metplus (met_util.py:219) INFO: Check the log
file for
> > more information:
> > /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
> >
> > (NPL) cheyenne1:/glade/work/zhuming/METplus4>
> >
> > On Mon, May 3, 2021 at 3:47 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > The METplus 4.0.0-beta5 release is now available on Cheyenne. If
you
> are
> > > able to test that this version works and fixes some of the
issues you
> > > mentioned in this ticket before the 4.0.0 release next week,
that would
> > be
> > > great! Information on how to use the new version can be found
here:
> > >
> > >
> https://dtcenter.org/community-code/metplus/metplus-4-0-existing-
builds
> > > (Under NCAR > CHEYENNE > METplus-4.0.0-beta5)
> > >
> > > Here are the commands to set it up:
> > >
> > > module use /glade/p/ral/jntp/MET/METplus/modulefiles
> > > module load metplus/4.0.0-beta5
> > > ncar_pylib
> > >
> > > Please let me know if you have any issues or questions.
> > >
> > > Thanks,
> > > George
> > >
> > > On Thu, Apr 22, 2021 at 11:40 AM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >
> > > > Zhuming,
> > > >
> > > > This is John HG. George asked me to take a look at this
question.
> Since
> > > > you're asking about "non-standard" levels, I assume you're
asking
> about
> > > the
> > > > verification of the pressure levels against the ADPUPA message
type.
> > > >
> > > > In general, the logic is this...
> > > > For each point observation, loop through all the verification
tasks
> and
> > > see
> > > > if it should be used.
> > > > For example, let's talk about the 4 level entry for VAR3:
> > > >
> > > > FCST_VAR3_NAME = v_tr_p
> > > > FCST_VAR3_LEVELS = ...(0,3,*,*)...
> > > > OBS_VAR3_NAME = VGRD
> > > > OBS_VAR3_LEVELS = ...P575-625...
> > > >
> > > > The point observation will be used here if it's a "VGRD"
observation
> > and
> > > > has a level value between 575 and 625. This verification task
has
> been
> > > > configured to only verify a single forecast level of data:
> > > v_tr_p(0,3,*,*).
> > > > So no vertical interpolation is done. MET interpolates
horizontally
> > from
> > > > the forecast grid points to the observation lat/lon location
(using
> > > > user-configurable interpolation options).
> > > >
> > > > To put it simply, all obs between 575 and 625 mb are compared
> directly
> > to
> > > > the forecast values from v_tr_p(0,3,*,*). If the model output
is
> > actually
> > > > on pressure levels, then that's a reasonable thing to do. If
the
> > model's
> > > > vertical coordinate is something other than pressure levels,
then
> > that's
> > > a
> > > > potential source for large errors.
> > > >
> > > > Note that MET does do vertical interpolation when processing
the GRIB
> > > > output from the Unified Post Processor tool. But all the
details of
> > that
> > > > depend on how the configuration file is setup. I'll explain
below.
> > > >
> > > > To date we have avoided including direct knowledge about the
vertical
> > > > coordinate of each modelling system in the verification code.
The
> goal
> > is
> > > > to create verification tools applicable to many models, rather
than
> > > > customizing them for each. We do realize that desire for the
latter
> and
> > > are
> > > > exploring ways to pull data from the IODA data assimilation
system
> > > instead,
> > > > which necessarily knows a lot about each model.
> > > >
> > > > So let's say you were using GRIB data, and configured it like
this:
> > > >
> > > > VAR3_NAME = VGRD
> > > > VAR3_LEVELS = P500-1000
> > > >
> > > > Since we haven't prefixed these options with FCST_ and OBS_,
the same
> > > > settings are applied to both.
> > > >
> > > > MET would read ALL forecast GRIB records from the model file
with
> > > pressures
> > > > between 500 and 1000. Perhaps, that's every 100mb or perhaps
every
> > 50mb?
> > > It
> > > > reads whatever data falls in that range. For each point
observation
> > with
> > > a
> > > > pressure level between 500 and 1000, it'll first find the
actual
> model
> > > > pressure level above/below it. It'll interpolate the forecast
> > > > data horizontally to the obs lat/lon location for both. And
then
> it'll
> > > > interpolate those values above/below vertically to the actual
> > observation
> > > > level, linear in the log of pressure.
> > > >
> > > > This issue is that the processing of pressure LAYERS is not
currently
> > > > supported for NetCDF data I believe. And the code could be
enhanced
> to
> > > > support that. But again, this only makes sense if the model's
> vertical
> > > > coordinate is actually pressure levels and not something else.
> > > >
> > > > Hope that helps clarify.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Apr 22, 2021 at 9:52 AM zhuming at ucar.edu via RT <
> > > met_help at ucar.edu
> > > > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >
> > > > > Hi George,
> > > > >
> > > > > My verification result shows that there is a big error on a
> > > non-standard
> > > > > level, so my question is:
> > > > > Is the forecast grid match to Obs, or obs station match to
> Forecast?
> > I
> > > > need
> > > > > to do is the forecast grid interpolated to the obs location
and
> > > vertical
> > > > > levels. So how to set the levels?
> > > > >
> > > > > Thanks,
> > > > > Zhuming
> > > > >
> > > > > FCST_VAR3_NAME = v_10m_tr
> > > > >
> > > > > FCST_VAR3_LEVELS = (0,*,*)
> > > > >
> > > > > FCST_VAR3_THRESH = >=5, <5
> > > > >
> > > > > OBS_VAR3_NAME = VGRD
> > > > >
> > > > > OBS_VAR3_LEVELS = Z10
> > > > >
> > > > > OBS_VAR3_THRESH = >=5, <5
> > > > >
> > > > > OBS_VAR3_OPTIONS = message_type = ["ADPSFC"];
> > > > >
> > > > >
> > > > > FCST_VAR3_NAME = v_tr_p
> > > > >
> > > > > FCST_VAR3_LEVELS =
> > > > >
> > > > >
> > > >
> > >
> >
>
(0,0,*,*),(0,1,*,*),(0,2,*,*),(0,3,*,*),(0,4,*,*),(0,5,*,*),(0,6,*,*),(0,7,*,*),(0,8,*,*),(0,9,*,*)
> > > > >
> > > > > FCST_VAR3_THRESH = >=5, <5
> > > > >
> > > > > OBS_VAR3_NAME = VGRD
> > > > >
> > > > > OBS_VAR3_LEVELS =
> > > > >
> > > > >
> > > >
> > >
> >
> P900-950,P825-875,P675-725,P575-625,P475-525,P375-425,P275-325,P225-
275,P175-225,P75-125
> > > > >
> > > > > OBS_VAR3_THRESH = >=5, <5
> > > > >
> > > > > OBS_VAR3_OPTIONS = message_type = ["ADPUPA"];
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Apr 15, 2021 at 1:52 PM George McCabe via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Zhuming,
> > > > > >
> > > > > > There was indeed a bug in the code. Howard has fixed the
issue
> with
> > > > this
> > > > > > pull request: https://github.com/dtcenter/MET/pull/1756
> > > > > > This fix will be included in the MET 10.0.0-beta5 release
that is
> > > > > > tentatively scheduled for next week. This beta version can
be
> > > installed
> > > > > on
> > > > > > cheyenne so you can test that the update fixes your issue.
I will
> > > > follow
> > > > > up
> > > > > > and let you know when the beta release is available on
cheyenne.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Wed, Apr 14, 2021 at 5:28 PM George McCabe
<mccabe at ucar.edu>
> > > wrote:
> > > > > >
> > > > > > > Hi Zhuming,
> > > > > > >
> > > > > > > I tried to get this to work on cheyenne with your data,
but was
> > not
> > > > > > > successful. I am able to read the data with
plot_data_plane and
> > > > > generate
> > > > > > an
> > > > > > > image, so I think there may be a bug in the code. I have
> > forwarded
> > > > info
> > > > > > to
> > > > > > > one of our C++ developers to look into it. I will report
back
> > when
> > > I
> > > > > know
> > > > > > > more.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > > On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT
<
> > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > >>
> > > > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >
> > > > > > >>
> > > > > > >> Hi George,
> > > > > > >>
> > > > > > >> It still can't find the correct time. I tried
> > > > > > >>
> > > > > > >> FCST_VAR1_LEVELS = (*,*,*)
> > > > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> > > > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >>
> > > > > > >> Zhuming
> > > > > > >>
> > > > > > >> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Zhuming,
> > > > > > >> >
> > > > > > >> > It looks like you are requesting the first time in
the 12
> hour
> > > > file
> > > > > > for
> > > > > > >> > each run:
> > > > > > >> >
> > > > > > >> > FCST_VAR1_LEVELS = (0,*,*)
> > > > > > >> > ...
> > > > > > >> > FCST_VAR2_LEVELS = (0,*,*)
> > > > > > >> >
> > > > > > >> > The first dimension in these files is the time
offset. You
> > will
> > > > need
> > > > > > the
> > > > > > >> > correct index that corresponds to each valid time.
> > Fortunately,
> > > > MET
> > > > > > >> makes
> > > > > > >> > this easy by allowing you to specify the valid time
for a
> time
> > > > > > dimension
> > > > > > >> > explicitly and it will find the correct data. You can
set:
> > > > > > >> >
> > > > > > >> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > > > >> > ...
> > > > > > >> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > > > >> >
> > > > > > >> > And it will substitute the valid time for each run.
Let me
> > know
> > > if
> > > > > you
> > > > > > >> have
> > > > > > >> > any questions.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > George
> > > > > > >> >
> > > > > > >> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via
RT <
> > > > > > >> met_help at ucar.edu
> > > > > > >> > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >
> > > > > > >> > >
> > > > > > >> > > Hi George,
> > > > > > >> > >
> > > > > > >> > > Thanks for your email, I just would like to send
you for
> > help.
> > > > > > >> > >
> > > > > > >> > > I already resolved the last problem with the WRF
forecast,
> > > thank
> > > > > you
> > > > > > >> for
> > > > > > >> > > your help.
> > > > > > >> > >
> > > > > > >> > > I also need to verify the MPAS model forecast.
These 12
> hour
> > > > > > >> > > MAPS forecasts are stored in a one file, such as
> > > > > > >> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h
forecasts
> > > > > > >> > > 2015071512/mpas_wrf_cf_2015071512.nc
> > > > > > >> > >
> > > > > > >> > > It seems MET is doing fine on the first 2 hours,
but no
> > > matched
> > > > > > pairs
> > > > > > >> for
> > > > > > >> > > later hours.
> > > > > > >> > > I'm not sure what's wrong with it? the time is not
> matched?
> > > > > > >> > >
> > > > > > >> > > see the log file
> > > > > > >> > >
> > > > > >
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > > > > > >> > > config file
> /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Zhuming
> > > > > > >> > >
> > > > > > >> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via
RT <
> > > > > > >> met_help at ucar.edu>
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hi Zhuming,
> > > > > > >> > > >
> > > > > > >> > > > Just following up on this ticket. Were you able
to get
> > this
> > > > > > >> resolved?
> > > > > > >> > Let
> > > > > > >> > > > me know if you need more assistance.
> > > > > > >> > > >
> > > > > > >> > > > Thanks,
> > > > > > >> > > > George
> > > > > > >> > > >
> > > > > > >> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > > > > >> > > > > Hi Zhuming,
> > > > > > >> > > > >
> > > > > > >> > > > > Typically directories that contain data for an
entire
> > day
> > > > are
> > > > > > >> named
> > > > > > >> > > > > with
> > > > > > >> > > > > YYYYMMDD, so your filename template tag would
be
> > > > > > >> {valid?fmt=%Y%m%d},
> > > > > > >> > > > > however that may not always be the case, as you
are
> > seeing
> > > > > here.
> > > > > > >> > There
> > > > > > >> > > > > are
> > > > > > >> > > > > a couple ways to handle this.
> > > > > > >> > > > >
> > > > > > >> > > > > If you know that the directory will always be
YYYYMMDD
> > > with
> > > > 00
> > > > > > at
> > > > > > >> the
> > > > > > >> > > > > end,
> > > > > > >> > > > > you can simply put 00 in the template instead
of
> adding
> > %H
> > > > in
> > > > > > the
> > > > > > >> > tag,
> > > > > > >> > > > > i.e.
> > > > > > >> > > > >
> > > > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > > >> > > > >
> > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> > > > > > >> 00_cf.nc
> > > > > > >> > > > >
> > > > > > >> > > > > If the HH value can vary, for instance if there
are 2
> > > > > > directories
> > > > > > >> > > > > created
> > > > > > >> > > > > per day (i.e. 2015071400 and 2015071412), then
you
> can
> > > use
> > > > > the
> > > > > > >> > > > > truncate
> > > > > > >> > > > > keyword to handle it.
> > > > > > >> > > > >
> > > > > > >> > > > > If directories are created every 24 hours:
> > > > > > >> > > > >
> > > > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > > >> > > > >
> > > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > > > > >> > > > > %d_%H}:00:
> > > > > > >> > > > > 00_cf.nc
> > > > > > >> > > > >
> > > > > > >> > > > > If directories are created every 12 hours:
> > > > > > >> > > > >
> > > > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > > >> > > > >
> > > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > > > > >> > > > > %d_%H}:00:
> > > > > > >> > > > > 00_cf.nc
> > > > > > >> > > > >
> > > > > > >> > > > > Let me know if that doesn't work for you.
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > George
> > > > > > >> > > > >
> > > > > > >> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu
via
> RT
> > > > > > >> > > > > <met_help at ucar.edu>
> > > > > > >> > > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > Hi George,
> > > > > > >> > > > > >
> > > > > > >> > > > > > Processing Obs data using "PB2NC" are right
now.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Now I'm using "PointStat" , I set VALID_BEG =
> > > 2015071500,
> > > > > > >> > VALID_END =
> > > > > > >> > > > > > 2015071501. It looks for the forecast data
> 2015071500/
> > > > > > >> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and
> 2015071501/
> > > > > > >> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > > > > >> > > > > > But my forecast data are all put in dir of
> > 2015071500/.
> > > > > > >> > > > > > So how could I set
FCST_POINT_STAT_INPUT_TEMPLATE
> to
> > > get
> > > > > the
> > > > > > >> right
> > > > > > >> > > > > > fcst
> > > > > > >> > > > > > data?
> > > > > > >> > > > > >
> > > > > > >> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > > >> > > > > >
> > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> > > > > > >> > 00_cf.nc
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > config file:
> > > > > /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > >> > > > > >
> > > > > > >> > > > > > log file:
> > > > > > >> > > > > >
> > > > > > >> > >
> > > > > >
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thanks,
> > > > > > >> > > > > >
> > > > > > >> > > > > > Zhuming
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe
via RT
> > > > > > >> > > > > > <met_help at ucar.edu>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Hi Zhuming,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Looking at the log output, it looks like
the data
> is
> > > > > stored
> > > > > > in
> > > > > > >> > the
> > > > > > >> > > > > > prepBUFR
> > > > > > >> > > > > > > files differently that you may have
thought:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > DEBUG 2: PrepBufr Time Center:
> > > 20150715_000000
> > > > > > >> > > > > > > DEBUG 2: Searching Time Window:
> > > 20150715_030000
> > > > to
> > > > > > >> > > > > > 20150715_040000
> > > > > > >> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55%
60% 65%
> > 70%
> > > > 75%
> > > > > > 80%
> > > > > > >> > 85%
> > > > > > >> > > > > > > 90%
> > > > > > >> > > > > > 95%
> > > > > > >> > > > > > > 100%
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > DEBUG 2: Total Messages processed
=
> > > 590836
> > > > > > >> > > > > > > DEBUG 2: Rejected based on message type
=
> 0
> > > > > > >> > > > > > > DEBUG 2: Rejected based on station id
=
> 0
> > > > > > >> > > > > > > DEBUG 2: Rejected based on valid time
=
> > > 590815
> > > > > > >> > > > > > > DEBUG 2: Rejected based on masking grid
=
> 20
> > > > > > >> > > > > > > DEBUG 2: Rejected based on masking polygon
=
> 0
> > > > > > >> > > > > > > DEBUG 2: Rejected based on elevation
=
> 0
> > > > > > >> > > > > > > DEBUG 2: Rejected based on pb report type
=
> 1
> > > > > > >> > > > > > > DEBUG 2: Rejected based on input report
type =
> 0
> > > > > > >> > > > > > > DEBUG 2: Rejected based on instrument type
=
> 0
> > > > > > >> > > > > > > DEBUG 2: Rejected based on zero
observations =
> 0
> > > > > > >> > > > > > > DEBUG 2: Total Messages retained
=
> 0
> > > > > > >> > > > > > > DEBUG 2: Total observations retained or
derived =
> 0
> > > > > > >> > > > > > > DEBUG 1: Obs time between *20150714_210000
and
> > > > > > >> 20150715_030000*
> > > > > > >> > > > > > > WARNING:
> > > > > > >> > > > > > > WARNING: process_pbfile() -> No PrepBufr
messages
> > > > retained
> > > > > > >> from
> > > > > > >> > > > > > > file:
> > > > > > >> > > > > > >
/glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > > > >> > > > > > > prepbufr.gdas.2015071500.nr
> > > > > > >> > > > > > > WARNING:
> > > > > > >> > > > > > > ERROR :
> > > > > > >> > > > > > > ERROR : write_netcdf_hdr_data() -> No
PrepBufr
> > > messages
> > > > > > >> > retained.
> > > > > > >> > > > > > Nothing
> > > > > > >> > > > > > > to write.
> > > > > > >> > > > > > > ERROR :
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > The prepbufr.gdas.2015071500.nr file
actually
> > > contains
> > > > > data
> > > > > > >> from
> > > > > > >> > > > > > > 20150714_210000 to 20150715_030000, not
> > > 20150715_000000
> > > > to
> > > > > > >> > > > > > 20150715_060000.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > To get the correct file for each run, you
will
> have
> > to
> > > > > shift
> > > > > > >> the
> > > > > > >> > > > > > > valid
> > > > > > >> > > > > > time
> > > > > > >> > > > > > > before truncating it so that it gets the
correct
> > file
> > > > for
> > > > > > each
> > > > > > >> > run.
> > > > > > >> > > > > > Instead
> > > > > > >> > > > > > > of setting:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > > >> > > > > > >
> > > > > > >> >
> > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > You will have to also shift forward 3
hours, i.e.:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > This says to add 3 hours to the current
valid time
> > > > before
> > > > > > >> > > > > > > truncating to
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > nearest 6 hours.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H =
0Z
> > > > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H
= 0Z
> > > > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H
= 0Z
> > > > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H
= 6Z
> > > > > > >> > > > > > > File: prepbufr.gdas.2015071506.nr
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Let me know if that doesn't work.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks,
> > > > > > >> > > > > > > George
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM
zhuming at ucar.edu
> via
> > > RT
> > > > <
> > > > > > >> > > > > > met_help at ucar.edu>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > <URL:
> > > > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > >> > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Hi George,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Now makes me confused is why processed
obs of
> > > several
> > > > > > times
> > > > > > >> are
> > > > > > >> > > > > > missing.
> > > > > > >> > > > > > > > There has files of time at 00,01,02, but
no
> > > 03,04,05;
> > > > > has
> > > > > > >> > > > > > > > 06,07,08, but
> > > > > > >> > > > > > > no
> > > > > > >> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has
> > 18,19,20,
> > > > but
> > > > > > with
> > > > > > >> > > > > > > > 21,no
> > > > > > >> > > > > > 22,23.
> > > > > > >> > > > > > > > From yours and mines:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > ls -l
/glade/u/home/mccabe/data/out/pb2nc1/
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6
17:52
> > > > > > >> > pbs.2015071500.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6
17:56
> > > > > > >> > pbs.2015071501.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6
18:00
> > > > > > >> > pbs.2015071502.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6
18:14
> > > > > > >> > pbs.2015071506.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6
18:18
> > > > > > >> > pbs.2015071507.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6
18:22
> > > > > > >> > pbs.2015071508.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6
18:36
> > > > > > >> > pbs.2015071512.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6
18:40
> > > > > > >> > pbs.2015071513.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6
18:40
> > > > > > >> > pbs.2015071514.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6
18:42
> > > > > > >> > pbs.2015071515.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > total 96112
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7
15:11
> > > > > > >> > > pbs.2015071500.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7
15:14
> > > > > > >> > > pbs.2015071501.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7
15:18
> > > > > > >> > > pbs.2015071502.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7
15:33
> > > > > > >> > > pbs.2015071506.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7
15:36
> > > > > > >> > > pbs.2015071507.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7
15:40
> > > > > > >> > > pbs.2015071508.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7
15:55
> > > > > > >> > > pbs.2015071512.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7
15:59
> > > > > > >> > > pbs.2015071513.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7
16:03
> > > > > > >> > > pbs.2015071514.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7
16:18
> > > > > > >> > > pbs.2015071518.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7
16:22
> > > > > > >> > > pbs.2015071519.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7
16:27
> > > > > > >> > > pbs.2015071520.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7
16:31
> > > > > > >> > > pbs.2015071521.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7
16:43
> > > > > > >> > > pbs.2015071600.nc
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Thanks,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Zhuming
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George
McCabe via
> > RT
> > > > > > >> > > > > > > > <met_help at ucar.edu
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Hi Zhuming,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks for the information. You will
need to
> use
> > > the
> > > > > > >> truncate
> > > > > > >> > > > > > > > > keyword
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > find the files, then set the valid
beg/end
> range
> > > to
> > > > 1
> > > > > > >> hour.
> > > > > > >> > > > > > > > > This is
> > > > > > >> > > > > > > what
> > > > > > >> > > > > > > > I
> > > > > > >> > > > > > > > > changed from your PB2NC1.conf config
file:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > *PB2NC_VALID_BEGIN =
> > > > > > {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
> > > > > > >> =
> > > > > > >> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > > >> > > > > > > > >
> > > > > > >> > > >
> > > > > >
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > My config file is on cheyenne here:
> > > > > > >> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I just started running it and the
output is
> > being
> > > > > > written
> > > > > > >> > here:
> > > > > > >> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > For PointStat, you will want to loop
over the
> > same
> > > > > > times.
> > > > > > >> I
> > > > > > >> > > > > > > > > noticed
> > > > > > >> > > > > > you
> > > > > > >> > > > > > > > > specify message type in the
VAR<n>_OPTIONS,
> but
> > > you
> > > > do
> > > > > > not
> > > > > > >> > need
> > > > > > >> > > > > > > > > to do
> > > > > > >> > > > > > > > this
> > > > > > >> > > > > > > > > since there is a
POINT_STAT_MESSAGE_TYPE
> > variable
> > > > that
> > > > > > >> > defines
> > > > > > >> > > > > > > > > this.
> > > > > > >> > > > > > > This
> > > > > > >> > > > > > > > > is what I changed in your
PointStat1.conf
> file:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > *VALID_END = 2015071600*
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > *VALID_INCREMENT =
1HPOINT_STAT_OBS_VALID_BEG
> > > > > > >> > > > > > =POINT_STAT_OBS_VALID_BEG
> > > > > > >> > > > > > > > =*
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > > >> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > > >> > > > > > > > > message_type =
["ADPSFC"];#OBS_VAR3_OPTIONS =
> > > > > > >> message_type =
> > > > > > >> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > > >> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR =
> > {OUTPUT_BASE}/pb2nc1*
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Or you can view my file in the same
> directory. I
> > > > > usually
> > > > > > >> try
> > > > > > >> > to
> > > > > > >> > > > > > > > > test
> > > > > > >> > > > > > > > with a
> > > > > > >> > > > > > > > > single run time to make sure it is
working
> > > properly,
> > > > > so
> > > > > > I
> > > > > > >> set
> > > > > > >> > > > > > VALID_END
> > > > > > >> > > > > > > > > back to 2015071500. I ran it and it
produced
> an
> > > > output
> > > > > > >> file,
> > > > > > >> > > > > > > > > but
> > > > > > >> > > > > > there
> > > > > > >> > > > > > > > were
> > > > > > >> > > > > > > > > no values besides the header. I checked
the
> log
> > > file
> > > > > and
> > > > > > >> saw
> > > > > > >> > > > > > > > > this:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*)
versus
> TMP/Z2,
> > > for
> > > > > > >> > > > > > > > > observation type
> > > > > > >> > > > > > > > > ADPSFC, over region FULL, for
interpolation
> > method
> > > > > > >> BILIN(4),
> > > > > > >> > > > > > > > > using 0
> > > > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of
matched pairs
> > =
> > > > > > 0DEBUG
> > > > > > >> 3:
> > > > > > >> > > > > > > Observations
> > > > > > >> > > > > > > > > processed = 393373DEBUG 3: Rejected:
> station
> > id
> > > > > > =
> > > > > > >> > > > > > > > > 0DEBUG 3:
> > > > > > >> > > > > > > > > Rejected: obs type = 390528DEBUG
3:
> > > Rejected:
> > > > > > valid
> > > > > > >> > time
> > > > > > >> > > > > > =
> > > > > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value =
0DEBUG
> 3:
> > > > > > Rejected:
> > > > > > >> off
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > grid
> > > > > > >> > > > > > > > =
> > > > > > >> > > > > > > > > 2586DEBUG 3: Rejected: topography
=
> 0DEBUG
> > 3:
> > > > > > >> Rejected:
> > > > > > >> > > > > > > > > level
> > > > > > >> > > > > > > > mismatch
> > > > > > >> > > > > > > > > = 259DEBUG 3: Rejected: quality marker
=
> 0DEBUG
> > > 3:
> > > > > > >> Rejected:
> > > > > > >> > > > > > message
> > > > > > >> > > > > > > > type
> > > > > > >> > > > > > > > > = 0DEBUG 3: Rejected: masking region =
0DEBUG
> > 3:
> > > > > > >> Rejected:
> > > > > > >> > bad
> > > > > > >> > > > > > fcst
> > > > > > >> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo
mean =
> > > > 0DEBUG
> > > > > 3:
> > > > > > >> > > > > > > > > Rejected:
> > > > > > >> > > > > > bad
> > > > > > >> > > > > > > > > climo stdev = 0DEBUG 3: Rejected:
duplicates
> > > =
> > > > > > 0DEBUG
> > > > > > >> > > > > > > > > 2:DEBUG 2:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > >> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*)
versus
> > > > UGRD/Z10,
> > > > > > for
> > > > > > >> > > > > > observation
> > > > > > >> > > > > > > > type
> > > > > > >> > > > > > > > > ADPSFC, over region FULL, for
interpolation
> > method
> > > > > > >> BILIN(4),
> > > > > > >> > > > > > > > > using 0
> > > > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of
matched pairs
> > =
> > > > > > 0DEBUG
> > > > > > >> 3:
> > > > > > >> > > > > > > Observations
> > > > > > >> > > > > > > > > processed = 393373*
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > *DEBUG 3: Rejected: station id =
0DEBUG
> 3:
> > > > > > Rejected:
> > > > > > >> obs
> > > > > > >> > > > > > > > > type
> > > > > > >> > > > > > > > =
> > > > > > >> > > > > > > > > 382965DEBUG 3: Rejected: valid time
=
> > 0DEBUG
> > > 3:
> > > > > > >> > Rejected:
> > > > > > >> > > > > > > > > bad
> > > > > > >> > > > > > obs
> > > > > > >> > > > > > > > > value = 0DEBUG 3: Rejected: off the
grid
> =
> > > > > > 9240DEBUG
> > > > > > >> 3:
> > > > > > >> > > > > > Rejected:
> > > > > > >> > > > > > > > > topography = 0DEBUG 3: Rejected:
level
> > > mismatch
> > > > > =
> > > > > > >> > > > > > > > > 1168DEBUG 3:
> > > > > > >> > > > > > > > > Rejected: quality marker = 0DEBUG 3:
> Rejected:
> > > > > message
> > > > > > >> type
> > > > > > >> > > > > > > > > =
> > > > > > >> > > > > > > 0DEBUG
> > > > > > >> > > > > > > > 3:
> > > > > > >> > > > > > > > > Rejected: masking region = 0DEBUG 3:
> Rejected:
> > > bad
> > > > > fcst
> > > > > > >> > value
> > > > > > >> > > > > > > > > =
> > > > > > >> > > > > > > 0DEBUG
> > > > > > >> > > > > > > > 3:
> > > > > > >> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3:
> Rejected:
> > > bad
> > > > > > climo
> > > > > > >> > stdev
> > > > > > >> > > > > > > > > =
> > > > > > >> > > > > > > 0DEBUG
> > > > > > >> > > > > > > > 3:
> > > > > > >> > > > > > > > > Rejected: duplicates = 0DEBUG
2:DEBUG 2:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > >> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*)
versus
> > > > VGRD/Z10,
> > > > > > for
> > > > > > >> > > > > > observation
> > > > > > >> > > > > > > > type
> > > > > > >> > > > > > > > > ADPSFC, over region FULL, for
interpolation
> > method
> > > > > > >> BILIN(4),
> > > > > > >> > > > > > > > > using 0
> > > > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of
matched pairs
> > =
> > > > > > 0DEBUG
> > > > > > >> 3:
> > > > > > >> > > > > > > Observations
> > > > > > >> > > > > > > > > processed = 393373DEBUG 3: Rejected:
> station
> > id
> > > > > > =
> > > > > > >> > > > > > > > > 0DEBUG 3:
> > > > > > >> > > > > > > > > Rejected: obs type = 382965DEBUG
3:
> > > Rejected:
> > > > > > valid
> > > > > > >> > time
> > > > > > >> > > > > > =
> > > > > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value =
0DEBUG
> 3:
> > > > > > Rejected:
> > > > > > >> off
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > grid
> > > > > > >> > > > > > > > =
> > > > > > >> > > > > > > > > 9240DEBUG 3: Rejected: topography
=
> 0DEBUG
> > 3:
> > > > > > >> Rejected:
> > > > > > >> > > > > > > > > level
> > > > > > >> > > > > > > > mismatch
> > > > > > >> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker
=
> > 0DEBUG
> > > 3:
> > > > > > >> > Rejected:
> > > > > > >> > > > > > message
> > > > > > >> > > > > > > > > type = 0DEBUG 3: Rejected: masking
region
> =
> > > > 0DEBUG
> > > > > > 3:
> > > > > > >> > > > > > > > > Rejected:
> > > > > > >> > > > > > bad
> > > > > > >> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad
climo
> mean
> > > =
> > > > > > >> 0DEBUG 3:
> > > > > > >> > > > > > Rejected:
> > > > > > >> > > > > > > > bad
> > > > > > >> > > > > > > > > climo stdev = 0*
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > *DEBUG 3: Rejected: duplicates =
0DEBUG
> > > 2:DEBUG
> > > > > 2:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > >> > > > > > > > > 2:DEBUG 1: Output file:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > This describes the reason why each
observation
> > was
> > > > > > >> rejected.
> > > > > > >> > > > > > > > > Some
> > > > > > >> > > > > > > > > observations didn't match the message
type you
> > > > > > requested,
> > > > > > >> > some
> > > > > > >> > > > > > > > > were
> > > > > > >> > > > > > not
> > > > > > >> > > > > > > > > within the forecast grid, and some did
not
> match
> > > the
> > > > > > >> vertical
> > > > > > >> > > > > > > > > level.
> > > > > > >> > > > > > > You
> > > > > > >> > > > > > > > > will have to make adjustments to your
settings
> > so
> > > > that
> > > > > > the
> > > > > > >> > data
> > > > > > >> > > > > > matches
> > > > > > >> > > > > > > > up
> > > > > > >> > > > > > > > > properly. Maybe the grid you are using
in the
> > > PB2NC
> > > > > > config
> > > > > > >> > file
> > > > > > >> > > > > > > > > does
> > > > > > >> > > > > > > not
> > > > > > >> > > > > > > > > match the forecast grid and you are
missing
> > > values.
> > > > If
> > > > > > >> that
> > > > > > >> > is
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > case,
> > > > > > >> > > > > > > > > you will have to reconfigure and rerun
PB2NC
> to
> > > get
> > > > > > >> output on
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > correct
> > > > > > >> > > > > > > > > gr5id. Maybe the observation points
don't line
> > up
> > > > with
> > > > > > the
> > > > > > >> > > > > > > > > vertical
> > > > > > >> > > > > > > level
> > > > > > >> > > > > > > > > of the model fields. You can use a
range of
> > level
> > > > > values
> > > > > > >> for
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > observation data to allow data within
that
> > range,
> > > > i.e.
> > > > > > >> Z0-10.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Let me know if you are still having
issues
> > getting
> > > > > this
> > > > > > to
> > > > > > >> > work
> > > > > > >> > > > > > > > > and I
> > > > > > >> > > > > > > can
> > > > > > >> > > > > > > > > help out.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > George
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM
> zhuming at ucar.edu
> > > via
> > > > > RT
> > > > > > <
> > > > > > >> > > > > > > > met_help at ucar.edu>
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > <URL:
> > > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Hi George,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks for your reply!
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > I'm still confused, I need to output
hourly
> > obs
> > > > data
> > > > > > to
> > > > > > >> do
> > > > > > >> > > > > > > > > > hourly
> > > > > > >> > > > > > > > > > verification.
> > > > > > >> > > > > > > > > > The original GDAS Prepbufr data has 4
files
> > one
> > > > day,
> > > > > > >> should
> > > > > > >> > > > > > > > > > with 6
> > > > > > >> > > > > > > > hours
> > > > > > >> > > > > > > > > > obs data in one file.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > After "PB2NC", I can see
pbs.2015071500.nc
> > (obs
> > > > > time
> > > > > > is
> > > > > > >> > 00?
> > > > > > >> > > > > > > > > > or 6
> > > > > > >> > > > > > > times
> > > > > > >> > > > > > > > > of
> > > > > > >> > > > > > > > > > 00,01,02,03,04,05?),
pbs.2015071506.nc...
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > my config file :
> > > > > > >> > > > > >
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > >> > > > > > > > > > and
> > > > /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > > Zhuming
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George
McCabe
> > via
> > > > RT
> > > > > <
> > > > > > >> > > > > > > met_help at ucar.edu
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > Hi Zhuming,
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > To run the tools hourly you would
set the
> > > > > > >> VALID_INCREMENT
> > > > > > >> > > > > > > > > > > value
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > > 1H.
> > > > > > >> > > > > > > > > > > Here is an example that runs once
per hour
> > > for a
> > > > > day
> > > > > > >> from
> > > > > > >> > > > > > > 2015-07-15
> > > > > > >> > > > > > > > 0Z
> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > > > > > > 2017-07-16 0Z:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > VALID_BEG = 2015071500
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > VALID_END = 2015071600
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > VALID_INCREMENT = 1H
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > LEAD_SEQ = 0
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Each run time will use the
> > > PB2NC_INPUT_TEMPLATE
> > > > to
> > > > > > >> find
> > > > > > >> > > > > > > > > > > files
> > > > > > >> > > > > > whose
> > > > > > >> > > > > > > > > > > filenames match the time you need.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > To read prepBUFR observations for
12
> hours,
> > > you
> > > > > will
> > > > > > >> want
> > > > > > >> > > > > > > > > > > to set
> > > > > > >> > > > > > > > > > > PB2NC_VALID_BEGIN and
PB2NC_VALID_END to
> > > specify
> > > > > the
> > > > > > >> > range
> > > > > > >> > > > > > > > > > > around
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > valid
> > > > > > >> > > > > > > > > > > time to process. These values
depend on
> what
> > > is
> > > > > > >> contained
> > > > > > >> > > > > > > > > > > inside
> > > > > > >> > > > > > > your
> > > > > > >> > > > > > > > > > > prepBUFR files. If you want to use
data
> from
> > > the
> > > > > > valid
> > > > > > >> > time
> > > > > > >> > > > > > > > > > > of
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > > > filename
> > > > > > >> > > > > > > > > > > backwards 12 hours, you would set:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > PB2NC_VALID_BEGIN =
> > > > > {valid?fmt=%Y%m%d_%H?shift=-12H
> > > > > > }
> > > > > > >> > > > > > > > > > > PB2NC_VALID_END =
{valid?fmt=%Y%m%d_%H}
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > For a runtime of 2015071512, this
would be
> > > > > > substituted
> > > > > > >> > as:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > valid_beg: 20150715_00
> > > > > > >> > > > > > > > > > > valid_end: 20150715_12
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > If you want to create an output
file for
> > each
> > > > hour
> > > > > > in
> > > > > > >> > PB2NC
> > > > > > >> > > > > > > > > > > you
> > > > > > >> > > > > > > would
> > > > > > >> > > > > > > > > > need
> > > > > > >> > > > > > > > > > > to make sure that the hour is
specified in
> > the
> > > > > > >> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > PointStat has similar config
variables to
> > set
> > > > the
> > > > > > >> valid
> > > > > > >> > beg
> > > > > > >> > > > > > > > > > > and
> > > > > > >> > > > > > end
> > > > > > >> > > > > > > > > range
> > > > > > >> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG
and
> > > > > > >> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > If your prepBUFR files are
separated out
> > into
> > > > > > multiple
> > > > > > >> > > > > > > > > > > files that
> > > > > > >> > > > > > > > need
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > be read into PB2NC, you can set a
file
> > window
> > > > > around
> > > > > > >> the
> > > > > > >> > > > > > > > > > > valid
> > > > > > >> > > > > > time
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > > > allow files to be passed into the
app.
> More
> > > > > > >> information
> > > > > > >> > on
> > > > > > >> > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > functionality is here:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > > > > >> > > > > > windows-to-find-valid-files
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > If you are still having trouble
getting
> this
> > > > > > >> configured,
> > > > > > >> > > > > > > > > > > you
> > > > > > >> > > > > > could
> > > > > > >> > > > > > > > send
> > > > > > >> > > > > > > > > > me
> > > > > > >> > > > > > > > > > > the paths of the files you are
trying to
> > read
> > > > and
> > > > > > what
> > > > > > >> > data
> > > > > > >> > > > > > > > > > > is
> > > > > > >> > > > > > > > > contained
> > > > > > >> > > > > > > > > > in
> > > > > > >> > > > > > > > > > > them so I can give you suggestions
on what
> > to
> > > > set
> > > > > > for
> > > > > > >> the
> > > > > > >> > > > > > > > > configuration.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > > > George
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM
Julie
> > > Prestopnik
> > > > > via
> > > > > > >> RT <
> > > > > > >> > > > > > > > > > met_help at ucar.edu>
> > > > > > >> > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request
99429
> > was
> > > > > acted
> > > > > > >> upon.
> > > > > > >> > > > > > > > > > > > Transaction: Given to mccabe
(George
> > McCabe)
> > > > by
> > > > > > >> jpresto
> > > > > > >> > > > > > > > > > > > Queue: met_help
> > > > > > >> > > > > > > > > > > > Subject: How to set time for
hourly
> > > 12-24
> > > > > > hours
> > > > > > >> > data
> > > > > > >> > > > > > > > > > > > ?
> > > > > > >> > > > > > > > > > > > Owner: mccabe
> > > > > > >> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > >> > > > > > > > > > > > Status: new
> > > > > > >> > > > > > > > > > > > Ticket <URL:
> > > > > > >> > > > > > > > >
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > This transaction appears to have
no
> > content
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > --
> > > > > > >> > > > > > > > > > > George McCabe - Software Engineer
III
> > > > > > >> > > > > > > > > > > National Center for Atmospheric
Research
> > > > > > >> > > > > > > > > > > Research Applications Laboratory
> > > > > > >> > > > > > > > > > > 303-497-2768
> > > > > > >> > > > > > > > > > > ---
> > > > > > >> > > > > > > > > > > My working day may not be your
working
> day.
> > > > Please
> > > > > > do
> > > > > > >> not
> > > > > > >> > > > > > > > > > > feel
> > > > > > >> > > > > > > > obliged
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > reply to this email outside of your
normal
> > > > working
> > > > > > >> hours.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > --
> > > > > > >> > > > > > > > > George McCabe - Software Engineer III
> > > > > > >> > > > > > > > > National Center for Atmospheric
Research
> > > > > > >> > > > > > > > > Research Applications Laboratory
> > > > > > >> > > > > > > > > 303-497-2768
> > > > > > >> > > > > > > > > ---
> > > > > > >> > > > > > > > > My working day may not be your working
day.
> > Please
> > > > do
> > > > > > not
> > > > > > >> > feel
> > > > > > >> > > > > > obliged
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > reply to this email outside of your
normal
> > working
> > > > > > hours.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > --
> > > > > > >> > > > > > > George McCabe - Software Engineer III
> > > > > > >> > > > > > > National Center for Atmospheric Research
> > > > > > >> > > > > > > Research Applications Laboratory
> > > > > > >> > > > > > > 303-497-2768
> > > > > > >> > > > > > > ---
> > > > > > >> > > > > > > My working day may not be your working day.
Please
> > do
> > > > not
> > > > > > feel
> > > > > > >> > > > > > > obliged to
> > > > > > >> > > > > > > reply to this email outside of your normal
working
> > > > hours.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > George McCabe - Software Engineer III
> > > > > > >> > National Center for Atmospheric Research
> > > > > > >> > Research Applications Laboratory
> > > > > > >> > 303-497-2768
> > > > > > >> > ---
> > > > > > >> > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > >> to
> > > > > > >> > reply to this email outside of your normal working
hours.
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Jun 16 14:40:25 2021
Hi Zhuming,
If I understand what you are trying to run correctly, then then the
LEAD_SEQ variable and the the INIT_ variables are what you will want
to
utilize. Here is some information about those settings:
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#looping-
by-initialization-time
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#lead-
seq
You can configure PointStat to loop over a set of initialization
times,
then loop over a set of forecast leads for each initialization time.
Instead of setting LOOP_BY = VALID, you will want to set
*LOOP_BY = INIT*
when this is set, you will need to set the corresponding INIT_
variables
instead of the VALID_ variables.
*INIT_TIME_FMT = %Y%m%d%H*
*INIT_BEG = 2021061500*
*INIT_END = 2021061600*
*INIT_INCREMENT = 12H*
This will process the following initialization times:
Init: 2021-06-15 @0Z
Init: 2021-06-15 @12Z
Init: 2021-06-16 @0Z
You can define the list of forecast leads to process. If you want to
process every 3rd forecast lead from 0-48, you can set:
*LEAD_SEQ = begin_end_incr(0,48,3)*
This is the equivalent of setting:
* LEAD_SEQ = 0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48*
For each initialization time, each forecast lead in the list will be
processed. In this example, it will process:
Init: 2021-06-15 @0Z, Lead: 0, Valid: 2021-06-15 @0Z
Init: 2021-06-15 @0Z, Lead: 3, Valid: 2021-06-15 @3Z
Init: 2021-06-15 @0Z, Lead: 6, Valid: 2021-06-15 @6Z
...
Init: 2021-06-15 @0Z, Lead: 48, Valid: 2021-06-17 @0Z
Init: 2021-06-15 @12Z, Lead: 0, Valid: 2021-06-15 @12Z
Init: 2021-06-15 @12Z, Lead: 3, Valid: 2021-06-15 @15Z
Init: 2021-06-15 @12Z, Lead: 6, Valid: 2021-06-15 @18Z
...
Init: 2021-06-15 @12Z, Lead: 48, Valid: 2021-06-17 @12Z
...
For each run, the initialization and lead times are used to compute
the
valid time, so you can reference the valid time to find observation
data
that corresponds to each run time.
I hope that helps! Let me know if I am misunderstanding what you are
trying
to configure.
Thanks,
George
On Wed, Jun 16, 2021 at 10:12 AM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> I still have questions about setting time on PointStat.
>
> I did run WRF and MPAS for one and half months, twice a day (00utc
and
> 12utc) , for a total of about 90 runs.
> Every run lasts 48hrs (forecast lead 48hrs) and output every 3 hrs.
See
> data in /glade/scratch/zhuming/pecan/ctrl/fcst/cf.
> So my question is how to set time for so many runs?
>
> see config file /glade/work/zhuming/METplus4/PointStat1.conf
>
> Thanks,
> Zhuming
>
>
> On Wed, May 19, 2021 at 3:03 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > There are a few things going on here:
> >
> > 1) There is a bug in the Usage wrapper that just prints the list
of
> > possible wrappers. Thank you for finding this. I will fix this for
the
> next
> > development release.
> >
> > 2) The reason you are getting the Usage wrapper instead of running
> > PointStat is you are passing in the defaults.conf file last.
> >
> > PointStat.conf has:
> > PROCESS_LIST = PointStat
> >
> > and defaults.conf has:
> > PROCESS_LIST = Usage
> >
> > Since defaults.conf was read after PointStat.conf, the value in
> > defaults.conf is used. The defaults.conf file in the METplus
directory
> > (under parm/metplus_config) is actually read first automatically
before
> > reading any of the config files you pass into the command, so you
do not
> > ever need to include it in the command.
> >
> > 3) You no longer need to put '-c' before your config files on the
command
> > line. It doesn't hurt to have them, but they are ignored.
> >
> > run_metplus.py ./PointStat.conf
> >
> > 4) The official 4.0.0 release is now available on cheyenne, so you
can
> use
> > that instead. You can set it up by running the following:
> >
> > module use /glade/p/ral/jntp/MET/METplus/modulefiles
> > module load metplus/4.0.0
> > ncar_pylib
> >
> > Let me know if you have any other issues.
> >
> > Thanks,
> > George
> >
> > On Wed, May 19, 2021 at 2:44 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > I tried running MET new version beta5. It seems something is
wrong.
> > > see log file
> > > /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
> > > config file in /glade/work/zhuming/METplus4
> > >
> > > Thanks!
> > > Zhuming
> > >
> > >
> > > (NPL) cheyenne1:/glade/work/zhuming/METplus4>run_metplus.py -c
> > > ./PointStat.conf -c ./defaults.conf
> > >
> > > Running METplus 4.0.0-beta5
> > >
> > > Starting METplus v4.0.0-beta5
> > >
> > > 05/19 14:34:36.341 metplus (config_metplus.py:117) INFO:
Starting
> METplus
> > > configuration setup.
> > >
> > > 05/19 14:34:36.343 metplus (config_metplus.py:247) INFO:
> > >
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/defaults.conf:
> Parsed
> > > this file
> > >
> > > 05/19 14:34:36.344 metplus (config_metplus.py:247) INFO:
> > ./PointStat.conf:
> > > Parsed this file
> > >
> > > 05/19 14:34:36.345 metplus (config_metplus.py:247) INFO:
> ./defaults.conf:
> > > Parsed this file
> > >
> > > 05/19 14:34:36.346 metplus (config_metplus.py:436) INFO: Adding:
> > > config.LOG_TIMESTAMP='20210519143436'
> > >
> > > 05/19 14:34:36.346 metplus (config_metplus.py:442) INFO: Replace
> [config]
> > > LOG_METPLUS with
> > >
'/glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436'
> > >
> > > 05/19 14:34:36.348 metplus (config_metplus.py:314) INFO: Setting
[dir]
> > > METPLUS_BASE to /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5
> > >
> > > 05/19 14:34:36.348 metplus (config_metplus.py:314) INFO: Setting
[dir]
> > > PARM_BASE to /gpfs/fs1/work/zhuming/METplus4/parm
> > >
> > > 05/19 14:34:36.348 metplus (config_metplus.py:321) INFO:
METPLUS_CONF:
> > > /glade/scratch/zhuming/metplus4/metplus_final.conf written here.
> > >
> > > 05/19 14:34:36.349 metplus (config_metplus.py:137) INFO:
Completed
> > METplus
> > > configuration setup.
> > >
> > > 05/19 14:34:36.349 metplus (met_util.py:68) INFO: Running
METplus
> > > v4.0.0-beta5 called with command:
> > > /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/ush/run_metplus.py -c
> > > ./PointStat.conf -c ./defaults.conf
> > >
> > > 05/19 14:34:36.349 metplus (met_util.py:70) INFO: Log file:
> > > /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
> > >
> > > 05/19 14:34:36.358 metplus.Usage (met_util.py:186) ERROR: Fatal
error
> > > occurred
> > >
> > > Traceback (most recent call last):
> > >
> > > File
> > >
> > >
> >
> "/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/util/met_util.py",
> > > line 128, in run_metplus
> > >
> > > instance=instance)
> > >
> > > File
> > >
> > >
> >
> "/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/wrappers/usage_wrapper.py",
> > > line 19, in __init__
> > >
> > > self.available_processes = list(set(val for val in
> > > met_util.LOWER_TO_WRAPPER_NAME.values()))
> > >
> > > AttributeError: module 'metplus.util.met_util' has no attribute
> > > 'LOWER_TO_WRAPPER_NAME'
> > >
> > >
> > > During handling of the above exception, another exception
occurred:
> > >
> > >
> > > Traceback (most recent call last):
> > >
> > > File
> > >
> > >
> >
> "/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/metplus/util/met_util.py",
> > > line 136, in run_metplus
> > >
> > > raise NameError("There was a problem loading "
> > >
> > > NameError: There was a problem loading Usage wrapper.
> > >
> > > 05/19 14:34:36Z run-METplus-metplus.Usage: ERROR: Fatal error
occurred
> > >
> > > 05/19 14:34:36.359 metplus.Usage (met_util.py:187) INFO: Check
the log
> > file
> > > for more information:
> > > /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
> > >
> > > 05/19 14:34:36.360 metplus (met_util.py:834) INFO:
> > > /glade/scratch/zhuming/metplus4/metplus_final.conf: write
metplus.conf
> > here
> > >
> > > 05/19 14:34:36.365 metplus (met_util.py:218) ERROR: METplus has
> finished
> > > running but had 1 error.
> > >
> > > 05/19 14:34:36Z run-METplus-metplus: ERROR: METplus has
finished
> running
> > > but had 1 error.
> > >
> > > 05/19 14:34:36.365 metplus (met_util.py:219) INFO: Check the log
file
> for
> > > more information:
> > > /glade/scratch/zhuming/metplus4/logs/metplus.log.20210519143436
> > >
> > > (NPL) cheyenne1:/glade/work/zhuming/METplus4>
> > >
> > > On Mon, May 3, 2021 at 3:47 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > The METplus 4.0.0-beta5 release is now available on Cheyenne.
If you
> > are
> > > > able to test that this version works and fixes some of the
issues you
> > > > mentioned in this ticket before the 4.0.0 release next week,
that
> would
> > > be
> > > > great! Information on how to use the new version can be found
here:
> > > >
> > > >
> > https://dtcenter.org/community-code/metplus/metplus-4-0-existing-
builds
> > > > (Under NCAR > CHEYENNE > METplus-4.0.0-beta5)
> > > >
> > > > Here are the commands to set it up:
> > > >
> > > > module use /glade/p/ral/jntp/MET/METplus/modulefiles
> > > > module load metplus/4.0.0-beta5
> > > > ncar_pylib
> > > >
> > > > Please let me know if you have any issues or questions.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Thu, Apr 22, 2021 at 11:40 AM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >
> > > > > Zhuming,
> > > > >
> > > > > This is John HG. George asked me to take a look at this
question.
> > Since
> > > > > you're asking about "non-standard" levels, I assume you're
asking
> > about
> > > > the
> > > > > verification of the pressure levels against the ADPUPA
message
> type.
> > > > >
> > > > > In general, the logic is this...
> > > > > For each point observation, loop through all the
verification tasks
> > and
> > > > see
> > > > > if it should be used.
> > > > > For example, let's talk about the 4 level entry for VAR3:
> > > > >
> > > > > FCST_VAR3_NAME = v_tr_p
> > > > > FCST_VAR3_LEVELS = ...(0,3,*,*)...
> > > > > OBS_VAR3_NAME = VGRD
> > > > > OBS_VAR3_LEVELS = ...P575-625...
> > > > >
> > > > > The point observation will be used here if it's a "VGRD"
> observation
> > > and
> > > > > has a level value between 575 and 625. This verification
task has
> > been
> > > > > configured to only verify a single forecast level of data:
> > > > v_tr_p(0,3,*,*).
> > > > > So no vertical interpolation is done. MET interpolates
horizontally
> > > from
> > > > > the forecast grid points to the observation lat/lon location
(using
> > > > > user-configurable interpolation options).
> > > > >
> > > > > To put it simply, all obs between 575 and 625 mb are
compared
> > directly
> > > to
> > > > > the forecast values from v_tr_p(0,3,*,*). If the model
output is
> > > actually
> > > > > on pressure levels, then that's a reasonable thing to do. If
the
> > > model's
> > > > > vertical coordinate is something other than pressure levels,
then
> > > that's
> > > > a
> > > > > potential source for large errors.
> > > > >
> > > > > Note that MET does do vertical interpolation when processing
the
> GRIB
> > > > > output from the Unified Post Processor tool. But all the
details of
> > > that
> > > > > depend on how the configuration file is setup. I'll explain
below.
> > > > >
> > > > > To date we have avoided including direct knowledge about the
> vertical
> > > > > coordinate of each modelling system in the verification
code. The
> > goal
> > > is
> > > > > to create verification tools applicable to many models,
rather than
> > > > > customizing them for each. We do realize that desire for the
latter
> > and
> > > > are
> > > > > exploring ways to pull data from the IODA data assimilation
system
> > > > instead,
> > > > > which necessarily knows a lot about each model.
> > > > >
> > > > > So let's say you were using GRIB data, and configured it
like this:
> > > > >
> > > > > VAR3_NAME = VGRD
> > > > > VAR3_LEVELS = P500-1000
> > > > >
> > > > > Since we haven't prefixed these options with FCST_ and OBS_,
the
> same
> > > > > settings are applied to both.
> > > > >
> > > > > MET would read ALL forecast GRIB records from the model file
with
> > > > pressures
> > > > > between 500 and 1000. Perhaps, that's every 100mb or perhaps
every
> > > 50mb?
> > > > It
> > > > > reads whatever data falls in that range. For each point
observation
> > > with
> > > > a
> > > > > pressure level between 500 and 1000, it'll first find the
actual
> > model
> > > > > pressure level above/below it. It'll interpolate the
forecast
> > > > > data horizontally to the obs lat/lon location for both. And
then
> > it'll
> > > > > interpolate those values above/below vertically to the
actual
> > > observation
> > > > > level, linear in the log of pressure.
> > > > >
> > > > > This issue is that the processing of pressure LAYERS is not
> currently
> > > > > supported for NetCDF data I believe. And the code could be
enhanced
> > to
> > > > > support that. But again, this only makes sense if the
model's
> > vertical
> > > > > coordinate is actually pressure levels and not something
else.
> > > > >
> > > > > Hope that helps clarify.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Apr 22, 2021 at 9:52 AM zhuming at ucar.edu via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > > >
> > > > > > Hi George,
> > > > > >
> > > > > > My verification result shows that there is a big error on
a
> > > > non-standard
> > > > > > level, so my question is:
> > > > > > Is the forecast grid match to Obs, or obs station match to
> > Forecast?
> > > I
> > > > > need
> > > > > > to do is the forecast grid interpolated to the obs
location and
> > > > vertical
> > > > > > levels. So how to set the levels?
> > > > > >
> > > > > > Thanks,
> > > > > > Zhuming
> > > > > >
> > > > > > FCST_VAR3_NAME = v_10m_tr
> > > > > >
> > > > > > FCST_VAR3_LEVELS = (0,*,*)
> > > > > >
> > > > > > FCST_VAR3_THRESH = >=5, <5
> > > > > >
> > > > > > OBS_VAR3_NAME = VGRD
> > > > > >
> > > > > > OBS_VAR3_LEVELS = Z10
> > > > > >
> > > > > > OBS_VAR3_THRESH = >=5, <5
> > > > > >
> > > > > > OBS_VAR3_OPTIONS = message_type = ["ADPSFC"];
> > > > > >
> > > > > >
> > > > > > FCST_VAR3_NAME = v_tr_p
> > > > > >
> > > > > > FCST_VAR3_LEVELS =
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
(0,0,*,*),(0,1,*,*),(0,2,*,*),(0,3,*,*),(0,4,*,*),(0,5,*,*),(0,6,*,*),(0,7,*,*),(0,8,*,*),(0,9,*,*)
> > > > > >
> > > > > > FCST_VAR3_THRESH = >=5, <5
> > > > > >
> > > > > > OBS_VAR3_NAME = VGRD
> > > > > >
> > > > > > OBS_VAR3_LEVELS =
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> P900-950,P825-875,P675-725,P575-625,P475-525,P375-425,P275-325,P225-
275,P175-225,P75-125
> > > > > >
> > > > > > OBS_VAR3_THRESH = >=5, <5
> > > > > >
> > > > > > OBS_VAR3_OPTIONS = message_type = ["ADPUPA"];
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Apr 15, 2021 at 1:52 PM George McCabe via RT <
> > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Zhuming,
> > > > > > >
> > > > > > > There was indeed a bug in the code. Howard has fixed the
issue
> > with
> > > > > this
> > > > > > > pull request: https://github.com/dtcenter/MET/pull/1756
> > > > > > > This fix will be included in the MET 10.0.0-beta5
release that
> is
> > > > > > > tentatively scheduled for next week. This beta version
can be
> > > > installed
> > > > > > on
> > > > > > > cheyenne so you can test that the update fixes your
issue. I
> will
> > > > > follow
> > > > > > up
> > > > > > > and let you know when the beta release is available on
> cheyenne.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > > On Wed, Apr 14, 2021 at 5:28 PM George McCabe
<mccabe at ucar.edu
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Zhuming,
> > > > > > > >
> > > > > > > > I tried to get this to work on cheyenne with your
data, but
> was
> > > not
> > > > > > > > successful. I am able to read the data with
plot_data_plane
> and
> > > > > > generate
> > > > > > > an
> > > > > > > > image, so I think there may be a bug in the code. I
have
> > > forwarded
> > > > > info
> > > > > > > to
> > > > > > > > one of our C++ developers to look into it. I will
report back
> > > when
> > > > I
> > > > > > know
> > > > > > > > more.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > > On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via
RT <
> > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >>
> > > > > > > >> <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >
> > > > > > > >>
> > > > > > > >> Hi George,
> > > > > > > >>
> > > > > > > >> It still can't find the correct time. I tried
> > > > > > > >>
> > > > > > > >> FCST_VAR1_LEVELS = (*,*,*)
> > > > > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> > > > > > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
> > > > > > > >>
> > > > > > > >> Thanks!
> > > > > > > >>
> > > > > > > >> Zhuming
> > > > > > > >>
> > > > > > > >> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Hi Zhuming,
> > > > > > > >> >
> > > > > > > >> > It looks like you are requesting the first time in
the 12
> > hour
> > > > > file
> > > > > > > for
> > > > > > > >> > each run:
> > > > > > > >> >
> > > > > > > >> > FCST_VAR1_LEVELS = (0,*,*)
> > > > > > > >> > ...
> > > > > > > >> > FCST_VAR2_LEVELS = (0,*,*)
> > > > > > > >> >
> > > > > > > >> > The first dimension in these files is the time
offset. You
> > > will
> > > > > need
> > > > > > > the
> > > > > > > >> > correct index that corresponds to each valid time.
> > > Fortunately,
> > > > > MET
> > > > > > > >> makes
> > > > > > > >> > this easy by allowing you to specify the valid time
for a
> > time
> > > > > > > dimension
> > > > > > > >> > explicitly and it will find the correct data. You
can set:
> > > > > > > >> >
> > > > > > > >> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > > > > >> > ...
> > > > > > > >> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > > > > > >> >
> > > > > > > >> > And it will substitute the valid time for each run.
Let me
> > > know
> > > > if
> > > > > > you
> > > > > > > >> have
> > > > > > > >> > any questions.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > George
> > > > > > > >> >
> > > > > > > >> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu
via RT <
> > > > > > > >> met_help at ucar.edu
> > > > > > > >> > >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > >
> > > > > > > >> > >
> > > > > > > >> > > Hi George,
> > > > > > > >> > >
> > > > > > > >> > > Thanks for your email, I just would like to send
you for
> > > help.
> > > > > > > >> > >
> > > > > > > >> > > I already resolved the last problem with the WRF
> forecast,
> > > > thank
> > > > > > you
> > > > > > > >> for
> > > > > > > >> > > your help.
> > > > > > > >> > >
> > > > > > > >> > > I also need to verify the MPAS model forecast.
These 12
> > hour
> > > > > > > >> > > MAPS forecasts are stored in a one file, such as
> > > > > > > >> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h
forecasts
> > > > > > > >> > > 2015071512/mpas_wrf_cf_2015071512.nc
> > > > > > > >> > >
> > > > > > > >> > > It seems MET is doing fine on the first 2 hours,
but no
> > > > matched
> > > > > > > pairs
> > > > > > > >> for
> > > > > > > >> > > later hours.
> > > > > > > >> > > I'm not sure what's wrong with it? the time is
not
> > matched?
> > > > > > > >> > >
> > > > > > > >> > > see the log file
> > > > > > > >> > >
> > > > > > >
> > > >
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > > > > > > >> > > config file
> > /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Zhuming
> > > > > > > >> > >
> > > > > > > >> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via
RT <
> > > > > > > >> met_help at ucar.edu>
> > > > > > > >> > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Hi Zhuming,
> > > > > > > >> > > >
> > > > > > > >> > > > Just following up on this ticket. Were you able
to get
> > > this
> > > > > > > >> resolved?
> > > > > > > >> > Let
> > > > > > > >> > > > me know if you need more assistance.
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks,
> > > > > > > >> > > > George
> > > > > > > >> > > >
> > > > > > > >> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > > > > > >> > > > > Hi Zhuming,
> > > > > > > >> > > > >
> > > > > > > >> > > > > Typically directories that contain data for
an
> entire
> > > day
> > > > > are
> > > > > > > >> named
> > > > > > > >> > > > > with
> > > > > > > >> > > > > YYYYMMDD, so your filename template tag would
be
> > > > > > > >> {valid?fmt=%Y%m%d},
> > > > > > > >> > > > > however that may not always be the case, as
you are
> > > seeing
> > > > > > here.
> > > > > > > >> > There
> > > > > > > >> > > > > are
> > > > > > > >> > > > > a couple ways to handle this.
> > > > > > > >> > > > >
> > > > > > > >> > > > > If you know that the directory will always be
> YYYYMMDD
> > > > with
> > > > > 00
> > > > > > > at
> > > > > > > >> the
> > > > > > > >> > > > > end,
> > > > > > > >> > > > > you can simply put 00 in the template instead
of
> > adding
> > > %H
> > > > > in
> > > > > > > the
> > > > > > > >> > tag,
> > > > > > > >> > > > > i.e.
> > > > > > > >> > > > >
> > > > > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > > > >> > > > >
> > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> > > > > > > >> 00_cf.nc
> > > > > > > >> > > > >
> > > > > > > >> > > > > If the HH value can vary, for instance if
there are
> 2
> > > > > > > directories
> > > > > > > >> > > > > created
> > > > > > > >> > > > > per day (i.e. 2015071400 and 2015071412),
then you
> > can
> > > > use
> > > > > > the
> > > > > > > >> > > > > truncate
> > > > > > > >> > > > > keyword to handle it.
> > > > > > > >> > > > >
> > > > > > > >> > > > > If directories are created every 24 hours:
> > > > > > > >> > > > >
> > > > > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > > > >> > > > >
> > > > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-
%m-
> > > > > > > >> > > > > %d_%H}:00:
> > > > > > > >> > > > > 00_cf.nc
> > > > > > > >> > > > >
> > > > > > > >> > > > > If directories are created every 12 hours:
> > > > > > > >> > > > >
> > > > > > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > > > >> > > > >
> > > > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-
%m-
> > > > > > > >> > > > > %d_%H}:00:
> > > > > > > >> > > > > 00_cf.nc
> > > > > > > >> > > > >
> > > > > > > >> > > > > Let me know if that doesn't work for you.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > George
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Thu, Apr 8, 2021 at 9:46 AM
zhuming at ucar.edu via
> > RT
> > > > > > > >> > > > > <met_help at ucar.edu>
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Hi George,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Processing Obs data using "PB2NC" are right
now.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Now I'm using "PointStat" , I set VALID_BEG
=
> > > > 2015071500,
> > > > > > > >> > VALID_END =
> > > > > > > >> > > > > > 2015071501. It looks for the forecast data
> > 2015071500/
> > > > > > > >> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and
> > 2015071501/
> > > > > > > >> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > > > > > >> > > > > > But my forecast data are all put in dir of
> > > 2015071500/.
> > > > > > > >> > > > > > So how could I set
FCST_POINT_STAT_INPUT_TEMPLATE
> > to
> > > > get
> > > > > > the
> > > > > > > >> right
> > > > > > > >> > > > > > fcst
> > > > > > > >> > > > > > data?
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > > > >> > > > > >
> > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> > > > > > > >> > 00_cf.nc
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > config file:
> > > > > > /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > log file:
> > > > > > > >> > > > > >
> > > > > > > >> > >
> > > > > > >
> > > >
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Zhuming
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George
McCabe via
> RT
> > > > > > > >> > > > > > <met_help at ucar.edu>
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Hi Zhuming,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Looking at the log output, it looks like
the
> data
> > is
> > > > > > stored
> > > > > > > in
> > > > > > > >> > the
> > > > > > > >> > > > > > prepBUFR
> > > > > > > >> > > > > > > files differently that you may have
thought:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > DEBUG 2: PrepBufr Time Center:
> > > > 20150715_000000
> > > > > > > >> > > > > > > DEBUG 2: Searching Time Window:
> > > > 20150715_030000
> > > > > to
> > > > > > > >> > > > > > 20150715_040000
> > > > > > > >> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
55% 60%
> 65%
> > > 70%
> > > > > 75%
> > > > > > > 80%
> > > > > > > >> > 85%
> > > > > > > >> > > > > > > 90%
> > > > > > > >> > > > > > 95%
> > > > > > > >> > > > > > > 100%
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > DEBUG 2: Total Messages processed
> =
> > > > 590836
> > > > > > > >> > > > > > > DEBUG 2: Rejected based on message type
> =
> > 0
> > > > > > > >> > > > > > > DEBUG 2: Rejected based on station id
> =
> > 0
> > > > > > > >> > > > > > > DEBUG 2: Rejected based on valid time
> =
> > > > 590815
> > > > > > > >> > > > > > > DEBUG 2: Rejected based on masking grid
> =
> > 20
> > > > > > > >> > > > > > > DEBUG 2: Rejected based on masking
polygon
> =
> > 0
> > > > > > > >> > > > > > > DEBUG 2: Rejected based on elevation
> =
> > 0
> > > > > > > >> > > > > > > DEBUG 2: Rejected based on pb report type
> =
> > 1
> > > > > > > >> > > > > > > DEBUG 2: Rejected based on input report
type
> =
> > 0
> > > > > > > >> > > > > > > DEBUG 2: Rejected based on instrument
type
> =
> > 0
> > > > > > > >> > > > > > > DEBUG 2: Rejected based on zero
observations
> =
> > 0
> > > > > > > >> > > > > > > DEBUG 2: Total Messages retained
> =
> > 0
> > > > > > > >> > > > > > > DEBUG 2: Total observations retained or
derived
> =
> > 0
> > > > > > > >> > > > > > > DEBUG 1: Obs time between
*20150714_210000 and
> > > > > > > >> 20150715_030000*
> > > > > > > >> > > > > > > WARNING:
> > > > > > > >> > > > > > > WARNING: process_pbfile() -> No PrepBufr
> messages
> > > > > retained
> > > > > > > >> from
> > > > > > > >> > > > > > > file:
> > > > > > > >> > > > > > >
/glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > > > > >> > > > > > > prepbufr.gdas.2015071500.nr
> > > > > > > >> > > > > > > WARNING:
> > > > > > > >> > > > > > > ERROR :
> > > > > > > >> > > > > > > ERROR : write_netcdf_hdr_data() -> No
PrepBufr
> > > > messages
> > > > > > > >> > retained.
> > > > > > > >> > > > > > Nothing
> > > > > > > >> > > > > > > to write.
> > > > > > > >> > > > > > > ERROR :
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > The prepbufr.gdas.2015071500.nr file
actually
> > > > contains
> > > > > > data
> > > > > > > >> from
> > > > > > > >> > > > > > > 20150714_210000 to 20150715_030000, not
> > > > 20150715_000000
> > > > > to
> > > > > > > >> > > > > > 20150715_060000.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > To get the correct file for each run, you
will
> > have
> > > to
> > > > > > shift
> > > > > > > >> the
> > > > > > > >> > > > > > > valid
> > > > > > > >> > > > > > time
> > > > > > > >> > > > > > > before truncating it so that it gets the
correct
> > > file
> > > > > for
> > > > > > > each
> > > > > > > >> > run.
> > > > > > > >> > > > > > Instead
> > > > > > > >> > > > > > > of setting:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > > > >> > > > > > >
> > > > > > > >> >
> > > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > You will have to also shift forward 3
hours,
> i.e.:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > This says to add 3 hours to the current
valid
> time
> > > > > before
> > > > > > > >> > > > > > > truncating to
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > nearest 6 hours.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H
= 0Z
> > > > > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate
6H = 0Z
> > > > > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate
6H = 0Z
> > > > > > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate
6H = 6Z
> > > > > > > >> > > > > > > File: prepbufr.gdas.2015071506.nr
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Let me know if that doesn't work.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Thanks,
> > > > > > > >> > > > > > > George
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM
zhuming at ucar.edu
> > via
> > > > RT
> > > > > <
> > > > > > > >> > > > > > met_help at ucar.edu>
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > <URL:
> > > > > > > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > >> > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Hi George,
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Now makes me confused is why processed
obs of
> > > > several
> > > > > > > times
> > > > > > > >> are
> > > > > > > >> > > > > > missing.
> > > > > > > >> > > > > > > > There has files of time at 00,01,02,
but no
> > > > 03,04,05;
> > > > > > has
> > > > > > > >> > > > > > > > 06,07,08, but
> > > > > > > >> > > > > > > no
> > > > > > > >> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17;
has
> > > 18,19,20,
> > > > > but
> > > > > > > with
> > > > > > > >> > > > > > > > 21,no
> > > > > > > >> > > > > > 22,23.
> > > > > > > >> > > > > > > > From yours and mines:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > ls -l
/glade/u/home/mccabe/data/out/pb2nc1/
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6
17:52
> > > > > > > >> > pbs.2015071500.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6
17:56
> > > > > > > >> > pbs.2015071501.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6
18:00
> > > > > > > >> > pbs.2015071502.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6
18:14
> > > > > > > >> > pbs.2015071506.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6
18:18
> > > > > > > >> > pbs.2015071507.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6
18:22
> > > > > > > >> > pbs.2015071508.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6
18:36
> > > > > > > >> > pbs.2015071512.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6
18:40
> > > > > > > >> > pbs.2015071513.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6
18:40
> > > > > > > >> > pbs.2015071514.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6
18:42
> > > > > > > >> > pbs.2015071515.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > total 96112
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr
7
> 15:11
> > > > > > > >> > > pbs.2015071500.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr
7
> 15:14
> > > > > > > >> > > pbs.2015071501.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr
7
> 15:18
> > > > > > > >> > > pbs.2015071502.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr
7
> 15:33
> > > > > > > >> > > pbs.2015071506.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr
7
> 15:36
> > > > > > > >> > > pbs.2015071507.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr
7
> 15:40
> > > > > > > >> > > pbs.2015071508.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr
7
> 15:55
> > > > > > > >> > > pbs.2015071512.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr
7
> 15:59
> > > > > > > >> > > pbs.2015071513.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr
7
> 16:03
> > > > > > > >> > > pbs.2015071514.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr
7
> 16:18
> > > > > > > >> > > pbs.2015071518.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr
7
> 16:22
> > > > > > > >> > > pbs.2015071519.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr
7
> 16:27
> > > > > > > >> > > pbs.2015071520.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr
7
> 16:31
> > > > > > > >> > > pbs.2015071521.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr
7
> 16:43
> > > > > > > >> > > pbs.2015071600.nc
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Thanks,
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Zhuming
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George
McCabe
> via
> > > RT
> > > > > > > >> > > > > > > > <met_help at ucar.edu
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > Hi Zhuming,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Thanks for the information. You will
need to
> > use
> > > > the
> > > > > > > >> truncate
> > > > > > > >> > > > > > > > > keyword
> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > > > > > find the files, then set the valid
beg/end
> > range
> > > > to
> > > > > 1
> > > > > > > >> hour.
> > > > > > > >> > > > > > > > > This is
> > > > > > > >> > > > > > > what
> > > > > > > >> > > > > > > > I
> > > > > > > >> > > > > > > > > changed from your PB2NC1.conf config
file:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > *PB2NC_VALID_BEGIN =
> > > > > > > {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
> > > > > > > >> =
> > > > > > > >> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > >
> > > > > > >
> > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > My config file is on cheyenne here:
> > > > > > > >> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > I just started running it and the
output is
> > > being
> > > > > > > written
> > > > > > > >> > here:
> > > > > > > >> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > For PointStat, you will want to loop
over
> the
> > > same
> > > > > > > times.
> > > > > > > >> I
> > > > > > > >> > > > > > > > > noticed
> > > > > > > >> > > > > > you
> > > > > > > >> > > > > > > > > specify message type in the
VAR<n>_OPTIONS,
> > but
> > > > you
> > > > > do
> > > > > > > not
> > > > > > > >> > need
> > > > > > > >> > > > > > > > > to do
> > > > > > > >> > > > > > > > this
> > > > > > > >> > > > > > > > > since there is a
POINT_STAT_MESSAGE_TYPE
> > > variable
> > > > > that
> > > > > > > >> > defines
> > > > > > > >> > > > > > > > > this.
> > > > > > > >> > > > > > > This
> > > > > > > >> > > > > > > > > is what I changed in your
PointStat1.conf
> > file:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > *VALID_END = 2015071600*
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > *VALID_INCREMENT =
> 1HPOINT_STAT_OBS_VALID_BEG
> > > > > > > >> > > > > > =POINT_STAT_OBS_VALID_BEG
> > > > > > > >> > > > > > > > =*
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > > > >> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > > > >> > > > > > > > > message_type =
["ADPSFC"];#OBS_VAR3_OPTIONS
> =
> > > > > > > >> message_type =
> > > > > > > >> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR
=
> > > > > > > >> > > > > > > > >
/glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR =
> > > {OUTPUT_BASE}/pb2nc1*
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Or you can view my file in the same
> > directory. I
> > > > > > usually
> > > > > > > >> try
> > > > > > > >> > to
> > > > > > > >> > > > > > > > > test
> > > > > > > >> > > > > > > > with a
> > > > > > > >> > > > > > > > > single run time to make sure it is
working
> > > > properly,
> > > > > > so
> > > > > > > I
> > > > > > > >> set
> > > > > > > >> > > > > > VALID_END
> > > > > > > >> > > > > > > > > back to 2015071500. I ran it and it
produced
> > an
> > > > > output
> > > > > > > >> file,
> > > > > > > >> > > > > > > > > but
> > > > > > > >> > > > > > there
> > > > > > > >> > > > > > > > were
> > > > > > > >> > > > > > > > > no values besides the header. I
checked the
> > log
> > > > file
> > > > > > and
> > > > > > > >> saw
> > > > > > > >> > > > > > > > > this:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*)
versus
> > TMP/Z2,
> > > > for
> > > > > > > >> > > > > > > > > observation type
> > > > > > > >> > > > > > > > > ADPSFC, over region FULL, for
interpolation
> > > method
> > > > > > > >> BILIN(4),
> > > > > > > >> > > > > > > > > using 0
> > > > > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of
matched
> pairs
> > > =
> > > > > > > 0DEBUG
> > > > > > > >> 3:
> > > > > > > >> > > > > > > Observations
> > > > > > > >> > > > > > > > > processed = 393373DEBUG 3:
Rejected:
> > station
> > > id
> > > > > > > =
> > > > > > > >> > > > > > > > > 0DEBUG 3:
> > > > > > > >> > > > > > > > > Rejected: obs type =
390528DEBUG 3:
> > > > Rejected:
> > > > > > > valid
> > > > > > > >> > time
> > > > > > > >> > > > > > =
> > > > > > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value =
0DEBUG
> > 3:
> > > > > > > Rejected:
> > > > > > > >> off
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > grid
> > > > > > > >> > > > > > > > =
> > > > > > > >> > > > > > > > > 2586DEBUG 3: Rejected: topography
=
> > 0DEBUG
> > > 3:
> > > > > > > >> Rejected:
> > > > > > > >> > > > > > > > > level
> > > > > > > >> > > > > > > > mismatch
> > > > > > > >> > > > > > > > > = 259DEBUG 3: Rejected: quality
marker =
> > 0DEBUG
> > > > 3:
> > > > > > > >> Rejected:
> > > > > > > >> > > > > > message
> > > > > > > >> > > > > > > > type
> > > > > > > >> > > > > > > > > = 0DEBUG 3: Rejected: masking region
=
> 0DEBUG
> > > 3:
> > > > > > > >> Rejected:
> > > > > > > >> > bad
> > > > > > > >> > > > > > fcst
> > > > > > > >> > > > > > > > > value = 0DEBUG 3: Rejected: bad
climo
> mean =
> > > > > 0DEBUG
> > > > > > 3:
> > > > > > > >> > > > > > > > > Rejected:
> > > > > > > >> > > > > > bad
> > > > > > > >> > > > > > > > > climo stdev = 0DEBUG 3: Rejected:
duplicates
> > > > =
> > > > > > > 0DEBUG
> > > > > > > >> > > > > > > > > 2:DEBUG 2:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > >> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*)
versus
> > > > > UGRD/Z10,
> > > > > > > for
> > > > > > > >> > > > > > observation
> > > > > > > >> > > > > > > > type
> > > > > > > >> > > > > > > > > ADPSFC, over region FULL, for
interpolation
> > > method
> > > > > > > >> BILIN(4),
> > > > > > > >> > > > > > > > > using 0
> > > > > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of
matched
> pairs
> > > =
> > > > > > > 0DEBUG
> > > > > > > >> 3:
> > > > > > > >> > > > > > > Observations
> > > > > > > >> > > > > > > > > processed = 393373*
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > *DEBUG 3: Rejected: station id =
0DEBUG
> > 3:
> > > > > > > Rejected:
> > > > > > > >> obs
> > > > > > > >> > > > > > > > > type
> > > > > > > >> > > > > > > > =
> > > > > > > >> > > > > > > > > 382965DEBUG 3: Rejected: valid time
=
> > > 0DEBUG
> > > > 3:
> > > > > > > >> > Rejected:
> > > > > > > >> > > > > > > > > bad
> > > > > > > >> > > > > > obs
> > > > > > > >> > > > > > > > > value = 0DEBUG 3: Rejected: off the
grid
> > =
> > > > > > > 9240DEBUG
> > > > > > > >> 3:
> > > > > > > >> > > > > > Rejected:
> > > > > > > >> > > > > > > > > topography = 0DEBUG 3: Rejected:
level
> > > > mismatch
> > > > > > =
> > > > > > > >> > > > > > > > > 1168DEBUG 3:
> > > > > > > >> > > > > > > > > Rejected: quality marker = 0DEBUG 3:
> > Rejected:
> > > > > > message
> > > > > > > >> type
> > > > > > > >> > > > > > > > > =
> > > > > > > >> > > > > > > 0DEBUG
> > > > > > > >> > > > > > > > 3:
> > > > > > > >> > > > > > > > > Rejected: masking region = 0DEBUG 3:
> > Rejected:
> > > > bad
> > > > > > fcst
> > > > > > > >> > value
> > > > > > > >> > > > > > > > > =
> > > > > > > >> > > > > > > 0DEBUG
> > > > > > > >> > > > > > > > 3:
> > > > > > > >> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3:
> > Rejected:
> > > > bad
> > > > > > > climo
> > > > > > > >> > stdev
> > > > > > > >> > > > > > > > > =
> > > > > > > >> > > > > > > 0DEBUG
> > > > > > > >> > > > > > > > 3:
> > > > > > > >> > > > > > > > > Rejected: duplicates = 0DEBUG
2:DEBUG
> 2:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > >> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*)
versus
> > > > > VGRD/Z10,
> > > > > > > for
> > > > > > > >> > > > > > observation
> > > > > > > >> > > > > > > > type
> > > > > > > >> > > > > > > > > ADPSFC, over region FULL, for
interpolation
> > > method
> > > > > > > >> BILIN(4),
> > > > > > > >> > > > > > > > > using 0
> > > > > > > >> > > > > > > > > matched pairs.DEBUG 3: Number of
matched
> pairs
> > > =
> > > > > > > 0DEBUG
> > > > > > > >> 3:
> > > > > > > >> > > > > > > Observations
> > > > > > > >> > > > > > > > > processed = 393373DEBUG 3:
Rejected:
> > station
> > > id
> > > > > > > =
> > > > > > > >> > > > > > > > > 0DEBUG 3:
> > > > > > > >> > > > > > > > > Rejected: obs type =
382965DEBUG 3:
> > > > Rejected:
> > > > > > > valid
> > > > > > > >> > time
> > > > > > > >> > > > > > =
> > > > > > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value =
0DEBUG
> > 3:
> > > > > > > Rejected:
> > > > > > > >> off
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > grid
> > > > > > > >> > > > > > > > =
> > > > > > > >> > > > > > > > > 9240DEBUG 3: Rejected: topography
=
> > 0DEBUG
> > > 3:
> > > > > > > >> Rejected:
> > > > > > > >> > > > > > > > > level
> > > > > > > >> > > > > > > > mismatch
> > > > > > > >> > > > > > > > > = 1168DEBUG 3: Rejected: quality
marker =
> > > 0DEBUG
> > > > 3:
> > > > > > > >> > Rejected:
> > > > > > > >> > > > > > message
> > > > > > > >> > > > > > > > > type = 0DEBUG 3: Rejected: masking
region
> > =
> > > > > 0DEBUG
> > > > > > > 3:
> > > > > > > >> > > > > > > > > Rejected:
> > > > > > > >> > > > > > bad
> > > > > > > >> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad
climo
> > mean
> > > > =
> > > > > > > >> 0DEBUG 3:
> > > > > > > >> > > > > > Rejected:
> > > > > > > >> > > > > > > > bad
> > > > > > > >> > > > > > > > > climo stdev = 0*
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > *DEBUG 3: Rejected: duplicates =
0DEBUG
> > > > 2:DEBUG
> > > > > > 2:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > >> > > > > > > > > 2:DEBUG 1: Output file:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > This describes the reason why each
> observation
> > > was
> > > > > > > >> rejected.
> > > > > > > >> > > > > > > > > Some
> > > > > > > >> > > > > > > > > observations didn't match the message
type
> you
> > > > > > > requested,
> > > > > > > >> > some
> > > > > > > >> > > > > > > > > were
> > > > > > > >> > > > > > not
> > > > > > > >> > > > > > > > > within the forecast grid, and some
did not
> > match
> > > > the
> > > > > > > >> vertical
> > > > > > > >> > > > > > > > > level.
> > > > > > > >> > > > > > > You
> > > > > > > >> > > > > > > > > will have to make adjustments to your
> settings
> > > so
> > > > > that
> > > > > > > the
> > > > > > > >> > data
> > > > > > > >> > > > > > matches
> > > > > > > >> > > > > > > > up
> > > > > > > >> > > > > > > > > properly. Maybe the grid you are
using in
> the
> > > > PB2NC
> > > > > > > config
> > > > > > > >> > file
> > > > > > > >> > > > > > > > > does
> > > > > > > >> > > > > > > not
> > > > > > > >> > > > > > > > > match the forecast grid and you are
missing
> > > > values.
> > > > > If
> > > > > > > >> that
> > > > > > > >> > is
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > case,
> > > > > > > >> > > > > > > > > you will have to reconfigure and
rerun PB2NC
> > to
> > > > get
> > > > > > > >> output on
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > correct
> > > > > > > >> > > > > > > > > gr5id. Maybe the observation points
don't
> line
> > > up
> > > > > with
> > > > > > > the
> > > > > > > >> > > > > > > > > vertical
> > > > > > > >> > > > > > > level
> > > > > > > >> > > > > > > > > of the model fields. You can use a
range of
> > > level
> > > > > > values
> > > > > > > >> for
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > observation data to allow data within
that
> > > range,
> > > > > i.e.
> > > > > > > >> Z0-10.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Let me know if you are still having
issues
> > > getting
> > > > > > this
> > > > > > > to
> > > > > > > >> > work
> > > > > > > >> > > > > > > > > and I
> > > > > > > >> > > > > > > can
> > > > > > > >> > > > > > > > > help out.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > > George
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM
> > zhuming at ucar.edu
> > > > via
> > > > > > RT
> > > > > > > <
> > > > > > > >> > > > > > > > met_help at ucar.edu>
> > > > > > > >> > > > > > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > <URL:
> > > > > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Hi George,
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Thanks for your reply!
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > I'm still confused, I need to
output
> hourly
> > > obs
> > > > > data
> > > > > > > to
> > > > > > > >> do
> > > > > > > >> > > > > > > > > > hourly
> > > > > > > >> > > > > > > > > > verification.
> > > > > > > >> > > > > > > > > > The original GDAS Prepbufr data has
4
> files
> > > one
> > > > > day,
> > > > > > > >> should
> > > > > > > >> > > > > > > > > > with 6
> > > > > > > >> > > > > > > > hours
> > > > > > > >> > > > > > > > > > obs data in one file.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > After "PB2NC", I can see
> pbs.2015071500.nc
> > > (obs
> > > > > > time
> > > > > > > is
> > > > > > > >> > 00?
> > > > > > > >> > > > > > > > > > or 6
> > > > > > > >> > > > > > > times
> > > > > > > >> > > > > > > > > of
> > > > > > > >> > > > > > > > > > 00,01,02,03,04,05?),
pbs.2015071506.nc...
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > my config file :
> > > > > > > >> > > > > >
> cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > > >> > > > > > > > > > and
> > > > > /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > > > Zhuming
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM
George
> McCabe
> > > via
> > > > > RT
> > > > > > <
> > > > > > > >> > > > > > > met_help at ucar.edu
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Hi Zhuming,
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > To run the tools hourly you would
set
> the
> > > > > > > >> VALID_INCREMENT
> > > > > > > >> > > > > > > > > > > value
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > > > 1H.
> > > > > > > >> > > > > > > > > > > Here is an example that runs once
per
> hour
> > > > for a
> > > > > > day
> > > > > > > >> from
> > > > > > > >> > > > > > > 2015-07-15
> > > > > > > >> > > > > > > > 0Z
> > > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > 2017-07-16 0Z:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > VALID_BEG = 2015071500
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > VALID_END = 2015071600
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > VALID_INCREMENT = 1H
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > LEAD_SEQ = 0
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Each run time will use the
> > > > PB2NC_INPUT_TEMPLATE
> > > > > to
> > > > > > > >> find
> > > > > > > >> > > > > > > > > > > files
> > > > > > > >> > > > > > whose
> > > > > > > >> > > > > > > > > > > filenames match the time you
need.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > To read prepBUFR observations for
12
> > hours,
> > > > you
> > > > > > will
> > > > > > > >> want
> > > > > > > >> > > > > > > > > > > to set
> > > > > > > >> > > > > > > > > > > PB2NC_VALID_BEGIN and
PB2NC_VALID_END to
> > > > specify
> > > > > > the
> > > > > > > >> > range
> > > > > > > >> > > > > > > > > > > around
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > valid
> > > > > > > >> > > > > > > > > > > time to process. These values
depend on
> > what
> > > > is
> > > > > > > >> contained
> > > > > > > >> > > > > > > > > > > inside
> > > > > > > >> > > > > > > your
> > > > > > > >> > > > > > > > > > > prepBUFR files. If you want to
use data
> > from
> > > > the
> > > > > > > valid
> > > > > > > >> > time
> > > > > > > >> > > > > > > > > > > of
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > > > > filename
> > > > > > > >> > > > > > > > > > > backwards 12 hours, you would
set:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > PB2NC_VALID_BEGIN =
> > > > > > {valid?fmt=%Y%m%d_%H?shift=-12H
> > > > > > > }
> > > > > > > >> > > > > > > > > > > PB2NC_VALID_END =
{valid?fmt=%Y%m%d_%H}
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > For a runtime of 2015071512, this
would
> be
> > > > > > > substituted
> > > > > > > >> > as:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > valid_beg: 20150715_00
> > > > > > > >> > > > > > > > > > > valid_end: 20150715_12
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > If you want to create an output
file for
> > > each
> > > > > hour
> > > > > > > in
> > > > > > > >> > PB2NC
> > > > > > > >> > > > > > > > > > > you
> > > > > > > >> > > > > > > would
> > > > > > > >> > > > > > > > > > need
> > > > > > > >> > > > > > > > > > > to make sure that the hour is
specified
> in
> > > the
> > > > > > > >> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > PointStat has similar config
variables
> to
> > > set
> > > > > the
> > > > > > > >> valid
> > > > > > > >> > beg
> > > > > > > >> > > > > > > > > > > and
> > > > > > > >> > > > > > end
> > > > > > > >> > > > > > > > > range
> > > > > > > >> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG
and
> > > > > > > >> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > If your prepBUFR files are
separated out
> > > into
> > > > > > > multiple
> > > > > > > >> > > > > > > > > > > files that
> > > > > > > >> > > > > > > > need
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > > be read into PB2NC, you can set a
file
> > > window
> > > > > > around
> > > > > > > >> the
> > > > > > > >> > > > > > > > > > > valid
> > > > > > > >> > > > > > time
> > > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > > > > > > allow files to be passed into the
app.
> > More
> > > > > > > >> information
> > > > > > > >> > on
> > > > > > > >> > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > functionality is here:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > > > > > >> > > > > > windows-to-find-valid-files
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > If you are still having trouble
getting
> > this
> > > > > > > >> configured,
> > > > > > > >> > > > > > > > > > > you
> > > > > > > >> > > > > > could
> > > > > > > >> > > > > > > > send
> > > > > > > >> > > > > > > > > > me
> > > > > > > >> > > > > > > > > > > the paths of the files you are
trying to
> > > read
> > > > > and
> > > > > > > what
> > > > > > > >> > data
> > > > > > > >> > > > > > > > > > > is
> > > > > > > >> > > > > > > > > contained
> > > > > > > >> > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > them so I can give you
suggestions on
> what
> > > to
> > > > > set
> > > > > > > for
> > > > > > > >> the
> > > > > > > >> > > > > > > > > configuration.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > > > > George
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM
Julie
> > > > Prestopnik
> > > > > > via
> > > > > > > >> RT <
> > > > > > > >> > > > > > > > > > met_help at ucar.edu>
> > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Tue Apr 06 14:16:42 2021:
Request
> 99429
> > > was
> > > > > > acted
> > > > > > > >> upon.
> > > > > > > >> > > > > > > > > > > > Transaction: Given to mccabe
(George
> > > McCabe)
> > > > > by
> > > > > > > >> jpresto
> > > > > > > >> > > > > > > > > > > > Queue: met_help
> > > > > > > >> > > > > > > > > > > > Subject: How to set time
for
> hourly
> > > > 12-24
> > > > > > > hours
> > > > > > > >> > data
> > > > > > > >> > > > > > > > > > > > ?
> > > > > > > >> > > > > > > > > > > > Owner: mccabe
> > > > > > > >> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > >> > > > > > > > > > > > Status: new
> > > > > > > >> > > > > > > > > > > > Ticket <URL:
> > > > > > > >> > > > > > > > >
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > This transaction appears to
have no
> > > content
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > --
> > > > > > > >> > > > > > > > > > > George McCabe - Software Engineer
III
> > > > > > > >> > > > > > > > > > > National Center for Atmospheric
Research
> > > > > > > >> > > > > > > > > > > Research Applications Laboratory
> > > > > > > >> > > > > > > > > > > 303-497-2768
> > > > > > > >> > > > > > > > > > > ---
> > > > > > > >> > > > > > > > > > > My working day may not be your
working
> > day.
> > > > > Please
> > > > > > > do
> > > > > > > >> not
> > > > > > > >> > > > > > > > > > > feel
> > > > > > > >> > > > > > > > obliged
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > > reply to this email outside of
your
> normal
> > > > > working
> > > > > > > >> hours.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > --
> > > > > > > >> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > >> > > > > > > > > National Center for Atmospheric
Research
> > > > > > > >> > > > > > > > > Research Applications Laboratory
> > > > > > > >> > > > > > > > > 303-497-2768
> > > > > > > >> > > > > > > > > ---
> > > > > > > >> > > > > > > > > My working day may not be your
working day.
> > > Please
> > > > > do
> > > > > > > not
> > > > > > > >> > feel
> > > > > > > >> > > > > > obliged
> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > > > > > reply to this email outside of your
normal
> > > working
> > > > > > > hours.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > --
> > > > > > > >> > > > > > > George McCabe - Software Engineer III
> > > > > > > >> > > > > > > National Center for Atmospheric Research
> > > > > > > >> > > > > > > Research Applications Laboratory
> > > > > > > >> > > > > > > 303-497-2768
> > > > > > > >> > > > > > > ---
> > > > > > > >> > > > > > > My working day may not be your working
day.
> Please
> > > do
> > > > > not
> > > > > > > feel
> > > > > > > >> > > > > > > obliged to
> > > > > > > >> > > > > > > reply to this email outside of your
normal
> working
> > > > > hours.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > George McCabe - Software Engineer III
> > > > > > > >> > National Center for Atmospheric Research
> > > > > > > >> > Research Applications Laboratory
> > > > > > > >> > 303-497-2768
> > > > > > > >> > ---
> > > > > > > >> > My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > >> to
> > > > > > > >> > reply to this email outside of your normal working
hours.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Fri Jun 18 17:32:20 2021
Hello George,
Thank you for your help. but I still need your help.
(1) Could I change the names of the stat output file? I prefer the
output
filename including the initial time (such as 2015060100), and the
forecast
time, but they only have forecast time as follows.
cheyenne2:/glade/scratch/zhuming/metplus4/point_stat_wrf_sfc/2015060100>l
*.stat
-rw-r--r-- 1 zhuming ncar 929063 Jun 16 18:54
point_stat_000000L_20150601_000000V.stat
-rw-r--r-- 1 zhuming ncar 920032 Jun 16 18:54
point_stat_000000L_20150601_030000V.stat
-rw-r--r-- 1 zhuming ncar 926411 Jun 16 18:54
point_stat_000000L_20150601_060000V.stat
-rw-r--r-- 1 zhuming ncar 928897 Jun 16 18:55
point_stat_000000L_20150601_090000V.stat
-rw-r--r-- 1 zhuming ncar 929075 Jun 16 18:55
point_stat_000000L_20150601_120000V.stat
-rw-r--r-- 1 zhuming ncar 912980 Jun 16 18:55
point_stat_000000L_20150601_150000V.stat
-rw-r--r-- 1 zhuming ncar 902466 Jun 16 18:56
point_stat_000000L_20150601_180000V.stat
-rw-r--r-- 1 zhuming ncar 899886 Jun 16 18:56
point_stat_000000L_20150601_210000V.stat
-rw-r--r-- 1 zhuming ncar 951104 Jun 16 18:56
point_stat_000000L_20150602_000000V.stat
-rw-r--r-- 1 zhuming ncar 909688 Jun 16 18:56
point_stat_000000L_20150602_030000V.stat
-rw-r--r-- 1 zhuming ncar 931520 Jun 16 18:57
point_stat_000000L_20150602_060000V.stat
-rw-r--r-- 1 zhuming ncar 912593 Jun 16 18:57
point_stat_000000L_20150602_090000V.stat
-rw-r--r-- 1 zhuming ncar 905290 Jun 16 18:57
point_stat_000000L_20150602_120000V.stat
-rw-r--r-- 1 zhuming ncar 908887 Jun 16 18:57
point_stat_000000L_20150602_150000V.stat
-rw-r--r-- 1 zhuming ncar 890838 Jun 16 18:58
point_stat_000000L_20150602_180000V.stat
-rw-r--r-- 1 zhuming ncar 904740 Jun 16 18:58
point_stat_000000L_20150602_210000V.stat
-rw-r--r-- 1 zhuming ncar 929844 Jun 16 18:58
point_stat_000000L_20150603_000000V.stat
(2) I need to do precipitation verification. I have precipitation obs
Stage4 data (Conus domain, with hourly, 6hr and 24hr). I also have 48
hr
precipitation forecast data (small domain with an output interval of
3hrs). I can find several config files of prescription verification
with
Stage4.
So my question is which one is better for me to use, and do I need to
convert/subdomain Stage4 obs data to my small forecast domain with 3
hrs
interval? What is pcp_combine?
l
/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/parm/use_cases/model_applications/precipitation/
*StgIV*
-rw-rw-r-- 1 jpresto p48503002 4606 May 3 14:51
GridStat_fcstHREFmean_obsStgIV_Gempak.conf
-rw-rw-r-- 1 jpresto p48503002 5380 May 3 14:51
GridStat_fcstHREFmean_obsStgIV_NetCDF.conf
-rw-rw-r-- 1 jpresto p48503002 4443 May 3 14:51
GridStat_fcstHRRR-TLE_obsStgIV_GRIB.conf
Thanks a lot!
Zhuming
On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> To run the tools hourly you would set the VALID_INCREMENT value for
1H.
> Here is an example that runs once per hour for a day from 2015-07-15
0Z to
> 2017-07-16 0Z:
>
> VALID_TIME_FMT = %Y%m%d%H
>
> VALID_BEG = 2015071500
>
> VALID_END = 2015071600
>
> VALID_INCREMENT = 1H
>
> LEAD_SEQ = 0
>
> Each run time will use the PB2NC_INPUT_TEMPLATE to find files whose
> filenames match the time you need.
>
> To read prepBUFR observations for 12 hours, you will want to set
> PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range around
the valid
> time to process. These values depend on what is contained inside
your
> prepBUFR files. If you want to use data from the valid time of the
filename
> backwards 12 hours, you would set:
>
> PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
>
> For a runtime of 2015071512, this would be substituted as:
>
> valid_beg: 20150715_00
> valid_end: 20150715_12
>
> If you want to create an output file for each hour in PB2NC you
would need
> to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
>
> PointStat has similar config variables to set the valid beg and end
range
> to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
>
> If your prepBUFR files are separated out into multiple files that
need to
> be read into PB2NC, you can set a file window around the valid time
to
> allow files to be passed into the app. More information on that
> functionality is here:
>
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
>
> If you are still having trouble getting this configured, you could
send me
> the paths of the files you are trying to read and what data is
contained in
> them so I can give you suggestions on what to set for the
configuration.
>
> Thanks,
> George
>
>
>
> On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > Transaction: Given to mccabe (George McCabe) by jpresto
> > Queue: met_help
> > Subject: How to set time for hourly 12-24 hours data ?
> > Owner: mccabe
> > Requestors: zhuming at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> >
> > This transaction appears to have no content
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Mon Jun 21 11:21:51 2021
Hi Zhuming,
*(1) Could I change the names of the stat output file? I prefer the
output
filename including the initial time (such as 2015060100), and the
forecast
time, but they only have forecast time as follows. *
The output stat files are named after the valid time because they
contain
output that is generated from comparing forecast and observation data
at a
common valid time. However, you can set the output_prefix to include
the
forecast lead and init time so that this information will also be
included
in the output file names. You can set something like this:
*POINT_STAT_OUTPUT_PREFIX = init{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H} *
This extra information will be added after "point_stat_" in the output
filenames.
* (2) I need to do precipitation verification. I have precipitation
obs
Stage4 data (Conus domain, with hourly, 6hr and 24hr). I also have 48
hr
precipitation forecast data (small domain with an output interval of
3hrs). I can find several config files of prescription verification
with
Stage4.*
* So my question is which one is better for me to use, and do I need
to
convert/subdomain Stage4 obs data to my small forecast domain with 3
hrs
interval? What is pcp_combine? *
I think the best example config file to use depends on what you are
trying
to accomplish in your verification. These examples are useful to
determine
how to configure the wrappers to read the same input data sets, but
the
actual analysis and set of run times depends on your needs.
There are configurations for multiple tools including PointStat and
GridStat to regrid data on the fly. See this page (search for
"regrid"):
https://met.readthedocs.io/en/latest/Users_Guide/config_options.html?highlight=output_prefix#settings-
common-to-multiple-tools
It sounds like in your case, you would want to regrid the data to the
forecast domain since it is smaller than the observation domain. So
the MET
config value would be "to_grid = FCST;" To configure this with
METplus, see
this section of the METplus User's Guide (search for
METPLUS_REGRID_DICT):
https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#grid-
stat-met-conf
GRID_STAT_REGRID_TO_GRID = FCST
Regridding on the fly does take additional processing time. If you
find
yourself needing to regrid the same file multiple times, it is better
to
use another tool called RegridDataPlane that can perform the
regridding and
output a file that can be read by the other MET tools more quickly.
Again,
this depends on your use case.
PCPCombine is a tool that (among other things) can be used to compute
different precipitation accumulations so that the forecast and
observation
accumulations match. For example, if you have hourly observation
precip
accumulations and 3 hourly forecast accumulations, you can run
PCPCombine
on the observation data to create 3 hourly observation accumulations
then
compare that result to the forecast. More information on the tool can
be
found here:
https://met.readthedocs.io/en/latest/Users_Guide/reformat_grid.html#pcp-
combine-tool
I hope this information helps. If you have a clear idea of the
verification
you are trying to configure, I can help you set up your config file to
accomplish that.
Thanks,
George
On Fri, Jun 18, 2021 at 5:32 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hello George,
>
> Thank you for your help. but I still need your help.
>
> (1) Could I change the names of the stat output file? I prefer the
output
> filename including the initial time (such as 2015060100), and the
forecast
> time, but they only have forecast time as follows.
>
>
cheyenne2:/glade/scratch/zhuming/metplus4/point_stat_wrf_sfc/2015060100>l
> *.stat
>
> -rw-r--r-- 1 zhuming ncar 929063 Jun 16 18:54
> point_stat_000000L_20150601_000000V.stat
>
> -rw-r--r-- 1 zhuming ncar 920032 Jun 16 18:54
> point_stat_000000L_20150601_030000V.stat
>
> -rw-r--r-- 1 zhuming ncar 926411 Jun 16 18:54
> point_stat_000000L_20150601_060000V.stat
>
> -rw-r--r-- 1 zhuming ncar 928897 Jun 16 18:55
> point_stat_000000L_20150601_090000V.stat
>
> -rw-r--r-- 1 zhuming ncar 929075 Jun 16 18:55
> point_stat_000000L_20150601_120000V.stat
>
> -rw-r--r-- 1 zhuming ncar 912980 Jun 16 18:55
> point_stat_000000L_20150601_150000V.stat
>
> -rw-r--r-- 1 zhuming ncar 902466 Jun 16 18:56
> point_stat_000000L_20150601_180000V.stat
>
> -rw-r--r-- 1 zhuming ncar 899886 Jun 16 18:56
> point_stat_000000L_20150601_210000V.stat
>
> -rw-r--r-- 1 zhuming ncar 951104 Jun 16 18:56
> point_stat_000000L_20150602_000000V.stat
>
> -rw-r--r-- 1 zhuming ncar 909688 Jun 16 18:56
> point_stat_000000L_20150602_030000V.stat
>
> -rw-r--r-- 1 zhuming ncar 931520 Jun 16 18:57
> point_stat_000000L_20150602_060000V.stat
>
> -rw-r--r-- 1 zhuming ncar 912593 Jun 16 18:57
> point_stat_000000L_20150602_090000V.stat
>
> -rw-r--r-- 1 zhuming ncar 905290 Jun 16 18:57
> point_stat_000000L_20150602_120000V.stat
>
> -rw-r--r-- 1 zhuming ncar 908887 Jun 16 18:57
> point_stat_000000L_20150602_150000V.stat
>
> -rw-r--r-- 1 zhuming ncar 890838 Jun 16 18:58
> point_stat_000000L_20150602_180000V.stat
>
> -rw-r--r-- 1 zhuming ncar 904740 Jun 16 18:58
> point_stat_000000L_20150602_210000V.stat
>
> -rw-r--r-- 1 zhuming ncar 929844 Jun 16 18:58
> point_stat_000000L_20150603_000000V.stat
>
>
> (2) I need to do precipitation verification. I have precipitation
obs
> Stage4 data (Conus domain, with hourly, 6hr and 24hr). I also have
48 hr
> precipitation forecast data (small domain with an output interval of
> 3hrs). I can find several config files of prescription verification
with
> Stage4.
>
> So my question is which one is better for me to use, and do I need
to
> convert/subdomain Stage4 obs data to my small forecast domain with 3
hrs
> interval? What is pcp_combine?
>
>
> l
>
> /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/parm/use_cases/model_applications/precipitation/
> *StgIV*
>
> -rw-rw-r-- 1 jpresto p48503002 4606 May 3 14:51
> GridStat_fcstHREFmean_obsStgIV_Gempak.conf
>
> -rw-rw-r-- 1 jpresto p48503002 5380 May 3 14:51
> GridStat_fcstHREFmean_obsStgIV_NetCDF.conf
>
> -rw-rw-r-- 1 jpresto p48503002 4443 May 3 14:51
> GridStat_fcstHRRR-TLE_obsStgIV_GRIB.conf
>
>
>
> Thanks a lot!
>
> Zhuming
>
> On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > To run the tools hourly you would set the VALID_INCREMENT value
for 1H.
> > Here is an example that runs once per hour for a day from 2015-07-
15 0Z
> to
> > 2017-07-16 0Z:
> >
> > VALID_TIME_FMT = %Y%m%d%H
> >
> > VALID_BEG = 2015071500
> >
> > VALID_END = 2015071600
> >
> > VALID_INCREMENT = 1H
> >
> > LEAD_SEQ = 0
> >
> > Each run time will use the PB2NC_INPUT_TEMPLATE to find files
whose
> > filenames match the time you need.
> >
> > To read prepBUFR observations for 12 hours, you will want to set
> > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range around
the
> valid
> > time to process. These values depend on what is contained inside
your
> > prepBUFR files. If you want to use data from the valid time of the
> filename
> > backwards 12 hours, you would set:
> >
> > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> >
> > For a runtime of 2015071512, this would be substituted as:
> >
> > valid_beg: 20150715_00
> > valid_end: 20150715_12
> >
> > If you want to create an output file for each hour in PB2NC you
would
> need
> > to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
> >
> > PointStat has similar config variables to set the valid beg and
end range
> > to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
> >
> > If your prepBUFR files are separated out into multiple files that
need to
> > be read into PB2NC, you can set a file window around the valid
time to
> > allow files to be passed into the app. More information on that
> > functionality is here:
> >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> >
> > If you are still having trouble getting this configured, you could
send
> me
> > the paths of the files you are trying to read and what data is
contained
> in
> > them so I can give you suggestions on what to set for the
configuration.
> >
> > Thanks,
> > George
> >
> >
> >
> > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > Queue: met_help
> > > Subject: How to set time for hourly 12-24 hours data ?
> > > Owner: mccabe
> > > Requestors: zhuming at ucar.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Wed Jun 23 10:38:57 2021
Hello George,
Thanks for your clear description of precipitation verification.
I would like to do are
(1) create 3 hourly obs precipitation from hourly observation precip
using
"PCPCombine"
(2) regrid these 3 hourly precip obs. to my small fcst domain/grid
using
"RegridDataPlane"
(3) do precip verification using "GridStat". (Need to output 3 hourly
precip fcst? )
Looking at the PCPCombine config files in met_tool_wrapper/PCPCombine,
I
chose "PCPCombine_add.conf" as my config file to revise, so how to
change
the time to fit mine?
Thanks!
Zhuming
ls parm/use_cases/met_tool_wrapper/PCPCombine/
PCPCombine_add.conf PCPCombine_derive.conf
PCPCombine_python_embedding.conf PCPCombine_sum.conf
sum_IMERG_V06_HDF5.py
PCPCombine_bucket.conf PCPCombine_loop_custom.conf
PCPCombine_subtract.conf PCPCombine_user_defined.conf
stage4 precip obs: /glade/scratch/zhuming/pecan/st4
wrf fcst: /glade/scratch/zhuming/pecan/ctrl/fcst/cf/
On Mon, Jun 21, 2021 at 11:21 AM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
>
>
>
> *(1) Could I change the names of the stat output file? I prefer the
output
> filename including the initial time (such as 2015060100), and the
forecast
> time, but they only have forecast time as follows. *
>
> The output stat files are named after the valid time because they
contain
> output that is generated from comparing forecast and observation
data at a
> common valid time. However, you can set the output_prefix to include
the
> forecast lead and init time so that this information will also be
included
> in the output file names. You can set something like this:
>
>
> *POINT_STAT_OUTPUT_PREFIX = init{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H}
*
>
> This extra information will be added after "point_stat_" in the
output
> filenames.
>
>
>
>
>
>
>
> * (2) I need to do precipitation verification. I have precipitation
obs
> Stage4 data (Conus domain, with hourly, 6hr and 24hr). I also have
48 hr
> precipitation forecast data (small domain with an output interval of
> 3hrs). I can find several config files of prescription verification
with
> Stage4.*
>
>
>
> * So my question is which one is better for me to use, and do I need
to
> convert/subdomain Stage4 obs data to my small forecast domain with 3
hrs
> interval? What is pcp_combine? *
>
> I think the best example config file to use depends on what you are
trying
> to accomplish in your verification. These examples are useful to
determine
> how to configure the wrappers to read the same input data sets, but
the
> actual analysis and set of run times depends on your needs.
>
> There are configurations for multiple tools including PointStat and
> GridStat to regrid data on the fly. See this page (search for
"regrid"):
>
>
https://met.readthedocs.io/en/latest/Users_Guide/config_options.html?highlight=output_prefix#settings-
common-to-multiple-tools
>
> It sounds like in your case, you would want to regrid the data to
the
> forecast domain since it is smaller than the observation domain. So
the MET
> config value would be "to_grid = FCST;" To configure this with
METplus, see
> this section of the METplus User's Guide (search for
METPLUS_REGRID_DICT):
>
>
https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#grid-
stat-met-conf
>
> GRID_STAT_REGRID_TO_GRID = FCST
>
> Regridding on the fly does take additional processing time. If you
find
> yourself needing to regrid the same file multiple times, it is
better to
> use another tool called RegridDataPlane that can perform the
regridding and
> output a file that can be read by the other MET tools more quickly.
Again,
> this depends on your use case.
>
> PCPCombine is a tool that (among other things) can be used to
compute
> different precipitation accumulations so that the forecast and
observation
> accumulations match. For example, if you have hourly observation
precip
> accumulations and 3 hourly forecast accumulations, you can run
PCPCombine
> on the observation data to create 3 hourly observation accumulations
then
> compare that result to the forecast. More information on the tool
can be
> found here:
>
>
https://met.readthedocs.io/en/latest/Users_Guide/reformat_grid.html#pcp-
combine-tool
>
> I hope this information helps. If you have a clear idea of the
verification
> you are trying to configure, I can help you set up your config file
to
> accomplish that.
>
> Thanks,
> George
>
> On Fri, Jun 18, 2021 at 5:32 PM zhuming at ucar.edu via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hello George,
> >
> > Thank you for your help. but I still need your help.
> >
> > (1) Could I change the names of the stat output file? I prefer the
output
> > filename including the initial time (such as 2015060100), and the
> forecast
> > time, but they only have forecast time as follows.
> >
> >
cheyenne2:/glade/scratch/zhuming/metplus4/point_stat_wrf_sfc/2015060100>l
> > *.stat
> >
> > -rw-r--r-- 1 zhuming ncar 929063 Jun 16 18:54
> > point_stat_000000L_20150601_000000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 920032 Jun 16 18:54
> > point_stat_000000L_20150601_030000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 926411 Jun 16 18:54
> > point_stat_000000L_20150601_060000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 928897 Jun 16 18:55
> > point_stat_000000L_20150601_090000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 929075 Jun 16 18:55
> > point_stat_000000L_20150601_120000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 912980 Jun 16 18:55
> > point_stat_000000L_20150601_150000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 902466 Jun 16 18:56
> > point_stat_000000L_20150601_180000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 899886 Jun 16 18:56
> > point_stat_000000L_20150601_210000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 951104 Jun 16 18:56
> > point_stat_000000L_20150602_000000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 909688 Jun 16 18:56
> > point_stat_000000L_20150602_030000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 931520 Jun 16 18:57
> > point_stat_000000L_20150602_060000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 912593 Jun 16 18:57
> > point_stat_000000L_20150602_090000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 905290 Jun 16 18:57
> > point_stat_000000L_20150602_120000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 908887 Jun 16 18:57
> > point_stat_000000L_20150602_150000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 890838 Jun 16 18:58
> > point_stat_000000L_20150602_180000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 904740 Jun 16 18:58
> > point_stat_000000L_20150602_210000V.stat
> >
> > -rw-r--r-- 1 zhuming ncar 929844 Jun 16 18:58
> > point_stat_000000L_20150603_000000V.stat
> >
> >
> > (2) I need to do precipitation verification. I have precipitation
obs
> > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I also have
48 hr
> > precipitation forecast data (small domain with an output interval
of
> > 3hrs). I can find several config files of prescription
verification with
> > Stage4.
> >
> > So my question is which one is better for me to use, and do I need
to
> > convert/subdomain Stage4 obs data to my small forecast domain with
3 hrs
> > interval? What is pcp_combine?
> >
> >
> > l
> >
> >
> /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/parm/use_cases/model_applications/precipitation/
> > *StgIV*
> >
> > -rw-rw-r-- 1 jpresto p48503002 4606 May 3 14:51
> > GridStat_fcstHREFmean_obsStgIV_Gempak.conf
> >
> > -rw-rw-r-- 1 jpresto p48503002 5380 May 3 14:51
> > GridStat_fcstHREFmean_obsStgIV_NetCDF.conf
> >
> > -rw-rw-r-- 1 jpresto p48503002 4443 May 3 14:51
> > GridStat_fcstHRRR-TLE_obsStgIV_GRIB.conf
> >
> >
> >
> > Thanks a lot!
> >
> > Zhuming
> >
> > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > To run the tools hourly you would set the VALID_INCREMENT value
for 1H.
> > > Here is an example that runs once per hour for a day from 2015-
07-15 0Z
> > to
> > > 2017-07-16 0Z:
> > >
> > > VALID_TIME_FMT = %Y%m%d%H
> > >
> > > VALID_BEG = 2015071500
> > >
> > > VALID_END = 2015071600
> > >
> > > VALID_INCREMENT = 1H
> > >
> > > LEAD_SEQ = 0
> > >
> > > Each run time will use the PB2NC_INPUT_TEMPLATE to find files
whose
> > > filenames match the time you need.
> > >
> > > To read prepBUFR observations for 12 hours, you will want to set
> > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around the
> > valid
> > > time to process. These values depend on what is contained inside
your
> > > prepBUFR files. If you want to use data from the valid time of
the
> > filename
> > > backwards 12 hours, you would set:
> > >
> > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > >
> > > For a runtime of 2015071512, this would be substituted as:
> > >
> > > valid_beg: 20150715_00
> > > valid_end: 20150715_12
> > >
> > > If you want to create an output file for each hour in PB2NC you
would
> > need
> > > to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
> > >
> > > PointStat has similar config variables to set the valid beg and
end
> range
> > > to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
> > >
> > > If your prepBUFR files are separated out into multiple files
that need
> to
> > > be read into PB2NC, you can set a file window around the valid
time to
> > > allow files to be passed into the app. More information on that
> > > functionality is here:
> > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > >
> > > If you are still having trouble getting this configured, you
could send
> > me
> > > the paths of the files you are trying to read and what data is
> contained
> > in
> > > them so I can give you suggestions on what to set for the
> configuration.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > >
> > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > Queue: met_help
> > > > Subject: How to set time for hourly 12-24 hours data ?
> > > > Owner: mccabe
> > > > Requestors: zhuming at ucar.edu
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >
> > > >
> > > >
> > > > This transaction appears to have no content
> > > >
> > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Jun 23 10:45:40 2021
Hi Zhuming,
Yes, I'd recommend using the PCPCombine_add example and modify it to
match
your data. Let me know if you have any issues getting it configured
and I
can help.
Thanks,
George
On Wed, Jun 23, 2021 at 10:39 AM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hello George,
>
> Thanks for your clear description of precipitation verification.
>
> I would like to do are
>
> (1) create 3 hourly obs precipitation from hourly observation precip
using
> "PCPCombine"
> (2) regrid these 3 hourly precip obs. to my small fcst domain/grid
using
> "RegridDataPlane"
> (3) do precip verification using "GridStat". (Need to output 3
hourly
> precip fcst? )
>
> Looking at the PCPCombine config files in
met_tool_wrapper/PCPCombine, I
> chose "PCPCombine_add.conf" as my config file to revise, so how to
change
> the time to fit mine?
>
> Thanks!
> Zhuming
>
> ls parm/use_cases/met_tool_wrapper/PCPCombine/
>
> PCPCombine_add.conf PCPCombine_derive.conf
> PCPCombine_python_embedding.conf PCPCombine_sum.conf
> sum_IMERG_V06_HDF5.py
>
> PCPCombine_bucket.conf PCPCombine_loop_custom.conf
> PCPCombine_subtract.conf PCPCombine_user_defined.conf
>
> stage4 precip obs: /glade/scratch/zhuming/pecan/st4
>
> wrf fcst: /glade/scratch/zhuming/pecan/ctrl/fcst/cf/
>
> On Mon, Jun 21, 2021 at 11:21 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> >
> >
> >
> > *(1) Could I change the names of the stat output file? I prefer
the
> output
> > filename including the initial time (such as 2015060100), and the
> forecast
> > time, but they only have forecast time as follows. *
> >
> > The output stat files are named after the valid time because they
contain
> > output that is generated from comparing forecast and observation
data at
> a
> > common valid time. However, you can set the output_prefix to
include the
> > forecast lead and init time so that this information will also be
> included
> > in the output file names. You can set something like this:
> >
> >
> > *POINT_STAT_OUTPUT_PREFIX =
init{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H} *
> >
> > This extra information will be added after "point_stat_" in the
output
> > filenames.
> >
> >
> >
> >
> >
> >
> >
> > * (2) I need to do precipitation verification. I have
precipitation obs
> > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I also have
48 hr
> > precipitation forecast data (small domain with an output interval
of
> > 3hrs). I can find several config files of prescription
verification with
> > Stage4.*
> >
> >
> >
> > * So my question is which one is better for me to use, and do I
need to
> > convert/subdomain Stage4 obs data to my small forecast domain with
3 hrs
> > interval? What is pcp_combine? *
> >
> > I think the best example config file to use depends on what you
are
> trying
> > to accomplish in your verification. These examples are useful to
> determine
> > how to configure the wrappers to read the same input data sets,
but the
> > actual analysis and set of run times depends on your needs.
> >
> > There are configurations for multiple tools including PointStat
and
> > GridStat to regrid data on the fly. See this page (search for
"regrid"):
> >
> >
>
https://met.readthedocs.io/en/latest/Users_Guide/config_options.html?highlight=output_prefix#settings-
common-to-multiple-tools
> >
> > It sounds like in your case, you would want to regrid the data to
the
> > forecast domain since it is smaller than the observation domain.
So the
> MET
> > config value would be "to_grid = FCST;" To configure this with
METplus,
> see
> > this section of the METplus User's Guide (search for
> METPLUS_REGRID_DICT):
> >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#grid-
stat-met-conf
> >
> > GRID_STAT_REGRID_TO_GRID = FCST
> >
> > Regridding on the fly does take additional processing time. If you
find
> > yourself needing to regrid the same file multiple times, it is
better to
> > use another tool called RegridDataPlane that can perform the
regridding
> and
> > output a file that can be read by the other MET tools more
quickly.
> Again,
> > this depends on your use case.
> >
> > PCPCombine is a tool that (among other things) can be used to
compute
> > different precipitation accumulations so that the forecast and
> observation
> > accumulations match. For example, if you have hourly observation
precip
> > accumulations and 3 hourly forecast accumulations, you can run
PCPCombine
> > on the observation data to create 3 hourly observation
accumulations then
> > compare that result to the forecast. More information on the tool
can be
> > found here:
> >
> >
>
https://met.readthedocs.io/en/latest/Users_Guide/reformat_grid.html#pcp-
combine-tool
> >
> > I hope this information helps. If you have a clear idea of the
> verification
> > you are trying to configure, I can help you set up your config
file to
> > accomplish that.
> >
> > Thanks,
> > George
> >
> > On Fri, Jun 18, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hello George,
> > >
> > > Thank you for your help. but I still need your help.
> > >
> > > (1) Could I change the names of the stat output file? I prefer
the
> output
> > > filename including the initial time (such as 2015060100), and
the
> > forecast
> > > time, but they only have forecast time as follows.
> > >
> > >
>
cheyenne2:/glade/scratch/zhuming/metplus4/point_stat_wrf_sfc/2015060100>l
> > > *.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 929063 Jun 16 18:54
> > > point_stat_000000L_20150601_000000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 920032 Jun 16 18:54
> > > point_stat_000000L_20150601_030000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 926411 Jun 16 18:54
> > > point_stat_000000L_20150601_060000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 928897 Jun 16 18:55
> > > point_stat_000000L_20150601_090000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 929075 Jun 16 18:55
> > > point_stat_000000L_20150601_120000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 912980 Jun 16 18:55
> > > point_stat_000000L_20150601_150000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 902466 Jun 16 18:56
> > > point_stat_000000L_20150601_180000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 899886 Jun 16 18:56
> > > point_stat_000000L_20150601_210000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 951104 Jun 16 18:56
> > > point_stat_000000L_20150602_000000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 909688 Jun 16 18:56
> > > point_stat_000000L_20150602_030000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 931520 Jun 16 18:57
> > > point_stat_000000L_20150602_060000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 912593 Jun 16 18:57
> > > point_stat_000000L_20150602_090000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 905290 Jun 16 18:57
> > > point_stat_000000L_20150602_120000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 908887 Jun 16 18:57
> > > point_stat_000000L_20150602_150000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 890838 Jun 16 18:58
> > > point_stat_000000L_20150602_180000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 904740 Jun 16 18:58
> > > point_stat_000000L_20150602_210000V.stat
> > >
> > > -rw-r--r-- 1 zhuming ncar 929844 Jun 16 18:58
> > > point_stat_000000L_20150603_000000V.stat
> > >
> > >
> > > (2) I need to do precipitation verification. I have
precipitation obs
> > > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I also
have 48
> hr
> > > precipitation forecast data (small domain with an output
interval of
> > > 3hrs). I can find several config files of prescription
verification
> with
> > > Stage4.
> > >
> > > So my question is which one is better for me to use, and do I
need to
> > > convert/subdomain Stage4 obs data to my small forecast domain
with 3
> hrs
> > > interval? What is pcp_combine?
> > >
> > >
> > > l
> > >
> > >
> >
> /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/parm/use_cases/model_applications/precipitation/
> > > *StgIV*
> > >
> > > -rw-rw-r-- 1 jpresto p48503002 4606 May 3 14:51
> > > GridStat_fcstHREFmean_obsStgIV_Gempak.conf
> > >
> > > -rw-rw-r-- 1 jpresto p48503002 5380 May 3 14:51
> > > GridStat_fcstHREFmean_obsStgIV_NetCDF.conf
> > >
> > > -rw-rw-r-- 1 jpresto p48503002 4443 May 3 14:51
> > > GridStat_fcstHRRR-TLE_obsStgIV_GRIB.conf
> > >
> > >
> > >
> > > Thanks a lot!
> > >
> > > Zhuming
> > >
> > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > To run the tools hourly you would set the VALID_INCREMENT
value for
> 1H.
> > > > Here is an example that runs once per hour for a day from
2015-07-15
> 0Z
> > > to
> > > > 2017-07-16 0Z:
> > > >
> > > > VALID_TIME_FMT = %Y%m%d%H
> > > >
> > > > VALID_BEG = 2015071500
> > > >
> > > > VALID_END = 2015071600
> > > >
> > > > VALID_INCREMENT = 1H
> > > >
> > > > LEAD_SEQ = 0
> > > >
> > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find files
whose
> > > > filenames match the time you need.
> > > >
> > > > To read prepBUFR observations for 12 hours, you will want to
set
> > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around the
> > > valid
> > > > time to process. These values depend on what is contained
inside your
> > > > prepBUFR files. If you want to use data from the valid time of
the
> > > filename
> > > > backwards 12 hours, you would set:
> > > >
> > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > >
> > > > For a runtime of 2015071512, this would be substituted as:
> > > >
> > > > valid_beg: 20150715_00
> > > > valid_end: 20150715_12
> > > >
> > > > If you want to create an output file for each hour in PB2NC
you would
> > > need
> > > > to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
> > > >
> > > > PointStat has similar config variables to set the valid beg
and end
> > range
> > > > to use (POINT_STAT_OBS_VALID_BEG and
POINT_STAT_OBS_VALID_END).
> > > >
> > > > If your prepBUFR files are separated out into multiple files
that
> need
> > to
> > > > be read into PB2NC, you can set a file window around the valid
time
> to
> > > > allow files to be passed into the app. More information on
that
> > > > functionality is here:
> > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > >
> > > > If you are still having trouble getting this configured, you
could
> send
> > > me
> > > > the paths of the files you are trying to read and what data is
> > contained
> > > in
> > > > them so I can give you suggestions on what to set for the
> > configuration.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > >
> > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > > Queue: met_help
> > > > > Subject: How to set time for hourly 12-24 hours data ?
> > > > > Owner: mccabe
> > > > > Requestors: zhuming at ucar.edu
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >
> > > > >
> > > > >
> > > > > This transaction appears to have no content
> > > > >
> > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Wed Jun 23 15:17:22 2021
Hi George,
Could you help on using "PCPCombine"? not sure why it has so many
ERRORs:
PCPCombine had 25 errors.
config file : /glade/work/zhuming/METplus4/PCPCombine_add.conf
log file :
/glade/scratch/zhuming/metplus4/logs/metplus.log.20210623151104
Thanks,
Zhuming
On Wed, Jun 23, 2021 at 10:45 AM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> Yes, I'd recommend using the PCPCombine_add example and modify it to
match
> your data. Let me know if you have any issues getting it configured
and I
> can help.
>
> Thanks,
> George
>
> On Wed, Jun 23, 2021 at 10:39 AM zhuming at ucar.edu via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hello George,
> >
> > Thanks for your clear description of precipitation verification.
> >
> > I would like to do are
> >
> > (1) create 3 hourly obs precipitation from hourly observation
precip
> using
> > "PCPCombine"
> > (2) regrid these 3 hourly precip obs. to my small fcst domain/grid
using
> > "RegridDataPlane"
> > (3) do precip verification using "GridStat". (Need to output 3
hourly
> > precip fcst? )
> >
> > Looking at the PCPCombine config files in
met_tool_wrapper/PCPCombine, I
> > chose "PCPCombine_add.conf" as my config file to revise, so how to
change
> > the time to fit mine?
> >
> > Thanks!
> > Zhuming
> >
> > ls parm/use_cases/met_tool_wrapper/PCPCombine/
> >
> > PCPCombine_add.conf PCPCombine_derive.conf
> > PCPCombine_python_embedding.conf PCPCombine_sum.conf
> > sum_IMERG_V06_HDF5.py
> >
> > PCPCombine_bucket.conf PCPCombine_loop_custom.conf
> > PCPCombine_subtract.conf PCPCombine_user_defined.conf
> >
> > stage4 precip obs: /glade/scratch/zhuming/pecan/st4
> >
> > wrf fcst: /glade/scratch/zhuming/pecan/ctrl/fcst/cf/
> >
> > On Mon, Jun 21, 2021 at 11:21 AM George McCabe via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > >
> > >
> > >
> > > *(1) Could I change the names of the stat output file? I prefer
the
> > output
> > > filename including the initial time (such as 2015060100), and
the
> > forecast
> > > time, but they only have forecast time as follows. *
> > >
> > > The output stat files are named after the valid time because
they
> contain
> > > output that is generated from comparing forecast and observation
data
> at
> > a
> > > common valid time. However, you can set the output_prefix to
include
> the
> > > forecast lead and init time so that this information will also
be
> > included
> > > in the output file names. You can set something like this:
> > >
> > >
> > > *POINT_STAT_OUTPUT_PREFIX =
init{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H} *
> > >
> > > This extra information will be added after "point_stat_" in the
output
> > > filenames.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > * (2) I need to do precipitation verification. I have
precipitation obs
> > > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I also
have 48
> hr
> > > precipitation forecast data (small domain with an output
interval of
> > > 3hrs). I can find several config files of prescription
verification
> with
> > > Stage4.*
> > >
> > >
> > >
> > > * So my question is which one is better for me to use, and do I
need to
> > > convert/subdomain Stage4 obs data to my small forecast domain
with 3
> hrs
> > > interval? What is pcp_combine? *
> > >
> > > I think the best example config file to use depends on what you
are
> > trying
> > > to accomplish in your verification. These examples are useful to
> > determine
> > > how to configure the wrappers to read the same input data sets,
but the
> > > actual analysis and set of run times depends on your needs.
> > >
> > > There are configurations for multiple tools including PointStat
and
> > > GridStat to regrid data on the fly. See this page (search for
> "regrid"):
> > >
> > >
> >
>
https://met.readthedocs.io/en/latest/Users_Guide/config_options.html?highlight=output_prefix#settings-
common-to-multiple-tools
> > >
> > > It sounds like in your case, you would want to regrid the data
to the
> > > forecast domain since it is smaller than the observation domain.
So the
> > MET
> > > config value would be "to_grid = FCST;" To configure this with
METplus,
> > see
> > > this section of the METplus User's Guide (search for
> > METPLUS_REGRID_DICT):
> > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#grid-
stat-met-conf
> > >
> > > GRID_STAT_REGRID_TO_GRID = FCST
> > >
> > > Regridding on the fly does take additional processing time. If
you find
> > > yourself needing to regrid the same file multiple times, it is
better
> to
> > > use another tool called RegridDataPlane that can perform the
regridding
> > and
> > > output a file that can be read by the other MET tools more
quickly.
> > Again,
> > > this depends on your use case.
> > >
> > > PCPCombine is a tool that (among other things) can be used to
compute
> > > different precipitation accumulations so that the forecast and
> > observation
> > > accumulations match. For example, if you have hourly observation
precip
> > > accumulations and 3 hourly forecast accumulations, you can run
> PCPCombine
> > > on the observation data to create 3 hourly observation
accumulations
> then
> > > compare that result to the forecast. More information on the
tool can
> be
> > > found here:
> > >
> > >
> >
>
https://met.readthedocs.io/en/latest/Users_Guide/reformat_grid.html#pcp-
combine-tool
> > >
> > > I hope this information helps. If you have a clear idea of the
> > verification
> > > you are trying to configure, I can help you set up your config
file to
> > > accomplish that.
> > >
> > > Thanks,
> > > George
> > >
> > > On Fri, Jun 18, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >
> > > > Hello George,
> > > >
> > > > Thank you for your help. but I still need your help.
> > > >
> > > > (1) Could I change the names of the stat output file? I prefer
the
> > output
> > > > filename including the initial time (such as 2015060100), and
the
> > > forecast
> > > > time, but they only have forecast time as follows.
> > > >
> > > >
> >
cheyenne2:/glade/scratch/zhuming/metplus4/point_stat_wrf_sfc/2015060100>l
> > > > *.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 929063 Jun 16 18:54
> > > > point_stat_000000L_20150601_000000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 920032 Jun 16 18:54
> > > > point_stat_000000L_20150601_030000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 926411 Jun 16 18:54
> > > > point_stat_000000L_20150601_060000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 928897 Jun 16 18:55
> > > > point_stat_000000L_20150601_090000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 929075 Jun 16 18:55
> > > > point_stat_000000L_20150601_120000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 912980 Jun 16 18:55
> > > > point_stat_000000L_20150601_150000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 902466 Jun 16 18:56
> > > > point_stat_000000L_20150601_180000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 899886 Jun 16 18:56
> > > > point_stat_000000L_20150601_210000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 951104 Jun 16 18:56
> > > > point_stat_000000L_20150602_000000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 909688 Jun 16 18:56
> > > > point_stat_000000L_20150602_030000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 931520 Jun 16 18:57
> > > > point_stat_000000L_20150602_060000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 912593 Jun 16 18:57
> > > > point_stat_000000L_20150602_090000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 905290 Jun 16 18:57
> > > > point_stat_000000L_20150602_120000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 908887 Jun 16 18:57
> > > > point_stat_000000L_20150602_150000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 890838 Jun 16 18:58
> > > > point_stat_000000L_20150602_180000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 904740 Jun 16 18:58
> > > > point_stat_000000L_20150602_210000V.stat
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 929844 Jun 16 18:58
> > > > point_stat_000000L_20150603_000000V.stat
> > > >
> > > >
> > > > (2) I need to do precipitation verification. I have
precipitation obs
> > > > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I also
have 48
> > hr
> > > > precipitation forecast data (small domain with an output
interval of
> > > > 3hrs). I can find several config files of prescription
verification
> > with
> > > > Stage4.
> > > >
> > > > So my question is which one is better for me to use, and do I
need to
> > > > convert/subdomain Stage4 obs data to my small forecast domain
with 3
> > hrs
> > > > interval? What is pcp_combine?
> > > >
> > > >
> > > > l
> > > >
> > > >
> > >
> >
> /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/parm/use_cases/model_applications/precipitation/
> > > > *StgIV*
> > > >
> > > > -rw-rw-r-- 1 jpresto p48503002 4606 May 3 14:51
> > > > GridStat_fcstHREFmean_obsStgIV_Gempak.conf
> > > >
> > > > -rw-rw-r-- 1 jpresto p48503002 5380 May 3 14:51
> > > > GridStat_fcstHREFmean_obsStgIV_NetCDF.conf
> > > >
> > > > -rw-rw-r-- 1 jpresto p48503002 4443 May 3 14:51
> > > > GridStat_fcstHRRR-TLE_obsStgIV_GRIB.conf
> > > >
> > > >
> > > >
> > > > Thanks a lot!
> > > >
> > > > Zhuming
> > > >
> > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > Hi Zhuming,
> > > > >
> > > > > To run the tools hourly you would set the VALID_INCREMENT
value for
> > 1H.
> > > > > Here is an example that runs once per hour for a day from
> 2015-07-15
> > 0Z
> > > > to
> > > > > 2017-07-16 0Z:
> > > > >
> > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > >
> > > > > VALID_BEG = 2015071500
> > > > >
> > > > > VALID_END = 2015071600
> > > > >
> > > > > VALID_INCREMENT = 1H
> > > > >
> > > > > LEAD_SEQ = 0
> > > > >
> > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find
files whose
> > > > > filenames match the time you need.
> > > > >
> > > > > To read prepBUFR observations for 12 hours, you will want to
set
> > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around
> the
> > > > valid
> > > > > time to process. These values depend on what is contained
inside
> your
> > > > > prepBUFR files. If you want to use data from the valid time
of the
> > > > filename
> > > > > backwards 12 hours, you would set:
> > > > >
> > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > >
> > > > > For a runtime of 2015071512, this would be substituted as:
> > > > >
> > > > > valid_beg: 20150715_00
> > > > > valid_end: 20150715_12
> > > > >
> > > > > If you want to create an output file for each hour in PB2NC
you
> would
> > > > need
> > > > > to make sure that the hour is specified in the
> PB2NC_OUTPUT_TEMPLATE.
> > > > >
> > > > > PointStat has similar config variables to set the valid beg
and end
> > > range
> > > > > to use (POINT_STAT_OBS_VALID_BEG and
POINT_STAT_OBS_VALID_END).
> > > > >
> > > > > If your prepBUFR files are separated out into multiple files
that
> > need
> > > to
> > > > > be read into PB2NC, you can set a file window around the
valid time
> > to
> > > > > allow files to be passed into the app. More information on
that
> > > > > functionality is here:
> > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > > >
> > > > > If you are still having trouble getting this configured, you
could
> > send
> > > > me
> > > > > the paths of the files you are trying to read and what data
is
> > > contained
> > > > in
> > > > > them so I can give you suggestions on what to set for the
> > > configuration.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > > > Queue: met_help
> > > > > > Subject: How to set time for hourly 12-24 hours data
?
> > > > > > Owner: mccabe
> > > > > > Requestors: zhuming at ucar.edu
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > >
> > > > > >
> > > > > >
> > > > > > This transaction appears to have no content
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Mon Jun 28 13:54:28 2021
Hi Zhuming,
The errors in your log file suggest that it cannot find the desired
field
from the input file:
DEBUG 1: Reading data (name="RAINRATE"; level="Surface";) from input
file:
/glade/scratch/zhuming/pecan/st4/ST4.2015060200.01h
ERROR :
ERROR : VarInfo::set_level_info_grib() - failed to parse level string
'Surface'
ERROR :
I ran wgrib on the input file and saw that there is a single field in
this
file named APCP:
(NPL) mccabe at cheyenne6:~> wgrib
/glade/scratch/zhuming/pecan/st4/ST4.2015060200.01h
1:0:d=15060123:*APCP*:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=1:TimeU=1:sfc:0-
1hr
acc:NAve=0
You should change your configuration file to match the fields that are
found in the file:
OBS_PCP_COMBINE_INPUT_NAMES = APCP
OBS_PCP_COMBINE_INPUT_LEVELS = L0
On Wed, Jun 23, 2021 at 3:17 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> Could you help on using "PCPCombine"? not sure why it has so many
ERRORs:
> PCPCombine had 25 errors.
>
> config file : /glade/work/zhuming/METplus4/PCPCombine_add.conf
> log file :
/glade/scratch/zhuming/metplus4/logs/metplus.log.20210623151104
>
> Thanks,
> Zhuming
>
> On Wed, Jun 23, 2021 at 10:45 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > Yes, I'd recommend using the PCPCombine_add example and modify it
to
> match
> > your data. Let me know if you have any issues getting it
configured and I
> > can help.
> >
> > Thanks,
> > George
> >
> > On Wed, Jun 23, 2021 at 10:39 AM zhuming at ucar.edu via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hello George,
> > >
> > > Thanks for your clear description of precipitation
verification.
> > >
> > > I would like to do are
> > >
> > > (1) create 3 hourly obs precipitation from hourly observation
precip
> > using
> > > "PCPCombine"
> > > (2) regrid these 3 hourly precip obs. to my small fcst
domain/grid
> using
> > > "RegridDataPlane"
> > > (3) do precip verification using "GridStat". (Need to output 3
hourly
> > > precip fcst? )
> > >
> > > Looking at the PCPCombine config files in
met_tool_wrapper/PCPCombine,
> I
> > > chose "PCPCombine_add.conf" as my config file to revise, so how
to
> change
> > > the time to fit mine?
> > >
> > > Thanks!
> > > Zhuming
> > >
> > > ls parm/use_cases/met_tool_wrapper/PCPCombine/
> > >
> > > PCPCombine_add.conf PCPCombine_derive.conf
> > > PCPCombine_python_embedding.conf PCPCombine_sum.conf
> > > sum_IMERG_V06_HDF5.py
> > >
> > > PCPCombine_bucket.conf PCPCombine_loop_custom.conf
> > > PCPCombine_subtract.conf PCPCombine_user_defined.conf
> > >
> > > stage4 precip obs: /glade/scratch/zhuming/pecan/st4
> > >
> > > wrf fcst: /glade/scratch/zhuming/pecan/ctrl/fcst/cf/
> > >
> > > On Mon, Jun 21, 2021 at 11:21 AM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > >
> > > >
> > > >
> > > > *(1) Could I change the names of the stat output file? I
prefer the
> > > output
> > > > filename including the initial time (such as 2015060100), and
the
> > > forecast
> > > > time, but they only have forecast time as follows. *
> > > >
> > > > The output stat files are named after the valid time because
they
> > contain
> > > > output that is generated from comparing forecast and
observation data
> > at
> > > a
> > > > common valid time. However, you can set the output_prefix to
include
> > the
> > > > forecast lead and init time so that this information will also
be
> > > included
> > > > in the output file names. You can set something like this:
> > > >
> > > >
> > > > *POINT_STAT_OUTPUT_PREFIX =
init{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H} *
> > > >
> > > > This extra information will be added after "point_stat_" in
the
> output
> > > > filenames.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > * (2) I need to do precipitation verification. I have
precipitation
> obs
> > > > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I also
have 48
> > hr
> > > > precipitation forecast data (small domain with an output
interval of
> > > > 3hrs). I can find several config files of prescription
verification
> > with
> > > > Stage4.*
> > > >
> > > >
> > > >
> > > > * So my question is which one is better for me to use, and do
I need
> to
> > > > convert/subdomain Stage4 obs data to my small forecast domain
with 3
> > hrs
> > > > interval? What is pcp_combine? *
> > > >
> > > > I think the best example config file to use depends on what
you are
> > > trying
> > > > to accomplish in your verification. These examples are useful
to
> > > determine
> > > > how to configure the wrappers to read the same input data
sets, but
> the
> > > > actual analysis and set of run times depends on your needs.
> > > >
> > > > There are configurations for multiple tools including
PointStat and
> > > > GridStat to regrid data on the fly. See this page (search for
> > "regrid"):
> > > >
> > > >
> > >
> >
>
https://met.readthedocs.io/en/latest/Users_Guide/config_options.html?highlight=output_prefix#settings-
common-to-multiple-tools
> > > >
> > > > It sounds like in your case, you would want to regrid the data
to the
> > > > forecast domain since it is smaller than the observation
domain. So
> the
> > > MET
> > > > config value would be "to_grid = FCST;" To configure this with
> METplus,
> > > see
> > > > this section of the METplus User's Guide (search for
> > > METPLUS_REGRID_DICT):
> > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#grid-
stat-met-conf
> > > >
> > > > GRID_STAT_REGRID_TO_GRID = FCST
> > > >
> > > > Regridding on the fly does take additional processing time. If
you
> find
> > > > yourself needing to regrid the same file multiple times, it is
better
> > to
> > > > use another tool called RegridDataPlane that can perform the
> regridding
> > > and
> > > > output a file that can be read by the other MET tools more
quickly.
> > > Again,
> > > > this depends on your use case.
> > > >
> > > > PCPCombine is a tool that (among other things) can be used to
compute
> > > > different precipitation accumulations so that the forecast and
> > > observation
> > > > accumulations match. For example, if you have hourly
observation
> precip
> > > > accumulations and 3 hourly forecast accumulations, you can run
> > PCPCombine
> > > > on the observation data to create 3 hourly observation
accumulations
> > then
> > > > compare that result to the forecast. More information on the
tool can
> > be
> > > > found here:
> > > >
> > > >
> > >
> >
>
https://met.readthedocs.io/en/latest/Users_Guide/reformat_grid.html#pcp-
combine-tool
> > > >
> > > > I hope this information helps. If you have a clear idea of the
> > > verification
> > > > you are trying to configure, I can help you set up your config
file
> to
> > > > accomplish that.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Fri, Jun 18, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > met_help at ucar.edu
> > > > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >
> > > > > Hello George,
> > > > >
> > > > > Thank you for your help. but I still need your help.
> > > > >
> > > > > (1) Could I change the names of the stat output file? I
prefer the
> > > output
> > > > > filename including the initial time (such as 2015060100),
and the
> > > > forecast
> > > > > time, but they only have forecast time as follows.
> > > > >
> > > > >
> > >
>
cheyenne2:/glade/scratch/zhuming/metplus4/point_stat_wrf_sfc/2015060100>l
> > > > > *.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 929063 Jun 16 18:54
> > > > > point_stat_000000L_20150601_000000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 920032 Jun 16 18:54
> > > > > point_stat_000000L_20150601_030000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 926411 Jun 16 18:54
> > > > > point_stat_000000L_20150601_060000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 928897 Jun 16 18:55
> > > > > point_stat_000000L_20150601_090000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 929075 Jun 16 18:55
> > > > > point_stat_000000L_20150601_120000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 912980 Jun 16 18:55
> > > > > point_stat_000000L_20150601_150000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 902466 Jun 16 18:56
> > > > > point_stat_000000L_20150601_180000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 899886 Jun 16 18:56
> > > > > point_stat_000000L_20150601_210000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 951104 Jun 16 18:56
> > > > > point_stat_000000L_20150602_000000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 909688 Jun 16 18:56
> > > > > point_stat_000000L_20150602_030000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 931520 Jun 16 18:57
> > > > > point_stat_000000L_20150602_060000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 912593 Jun 16 18:57
> > > > > point_stat_000000L_20150602_090000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 905290 Jun 16 18:57
> > > > > point_stat_000000L_20150602_120000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 908887 Jun 16 18:57
> > > > > point_stat_000000L_20150602_150000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 890838 Jun 16 18:58
> > > > > point_stat_000000L_20150602_180000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 904740 Jun 16 18:58
> > > > > point_stat_000000L_20150602_210000V.stat
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 929844 Jun 16 18:58
> > > > > point_stat_000000L_20150603_000000V.stat
> > > > >
> > > > >
> > > > > (2) I need to do precipitation verification. I have
precipitation
> obs
> > > > > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I
also have
> 48
> > > hr
> > > > > precipitation forecast data (small domain with an output
interval
> of
> > > > > 3hrs). I can find several config files of prescription
> verification
> > > with
> > > > > Stage4.
> > > > >
> > > > > So my question is which one is better for me to use, and do
I need
> to
> > > > > convert/subdomain Stage4 obs data to my small forecast
domain with
> 3
> > > hrs
> > > > > interval? What is pcp_combine?
> > > > >
> > > > >
> > > > > l
> > > > >
> > > > >
> > > >
> > >
> >
> /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/parm/use_cases/model_applications/precipitation/
> > > > > *StgIV*
> > > > >
> > > > > -rw-rw-r-- 1 jpresto p48503002 4606 May 3 14:51
> > > > > GridStat_fcstHREFmean_obsStgIV_Gempak.conf
> > > > >
> > > > > -rw-rw-r-- 1 jpresto p48503002 5380 May 3 14:51
> > > > > GridStat_fcstHREFmean_obsStgIV_NetCDF.conf
> > > > >
> > > > > -rw-rw-r-- 1 jpresto p48503002 4443 May 3 14:51
> > > > > GridStat_fcstHRRR-TLE_obsStgIV_GRIB.conf
> > > > >
> > > > >
> > > > >
> > > > > Thanks a lot!
> > > > >
> > > > > Zhuming
> > > > >
> > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Zhuming,
> > > > > >
> > > > > > To run the tools hourly you would set the VALID_INCREMENT
value
> for
> > > 1H.
> > > > > > Here is an example that runs once per hour for a day from
> > 2015-07-15
> > > 0Z
> > > > > to
> > > > > > 2017-07-16 0Z:
> > > > > >
> > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > >
> > > > > > VALID_BEG = 2015071500
> > > > > >
> > > > > > VALID_END = 2015071600
> > > > > >
> > > > > > VALID_INCREMENT = 1H
> > > > > >
> > > > > > LEAD_SEQ = 0
> > > > > >
> > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find
files
> whose
> > > > > > filenames match the time you need.
> > > > > >
> > > > > > To read prepBUFR observations for 12 hours, you will want
to set
> > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around
> > the
> > > > > valid
> > > > > > time to process. These values depend on what is contained
inside
> > your
> > > > > > prepBUFR files. If you want to use data from the valid
time of
> the
> > > > > filename
> > > > > > backwards 12 hours, you would set:
> > > > > >
> > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > >
> > > > > > For a runtime of 2015071512, this would be substituted as:
> > > > > >
> > > > > > valid_beg: 20150715_00
> > > > > > valid_end: 20150715_12
> > > > > >
> > > > > > If you want to create an output file for each hour in
PB2NC you
> > would
> > > > > need
> > > > > > to make sure that the hour is specified in the
> > PB2NC_OUTPUT_TEMPLATE.
> > > > > >
> > > > > > PointStat has similar config variables to set the valid
beg and
> end
> > > > range
> > > > > > to use (POINT_STAT_OBS_VALID_BEG and
POINT_STAT_OBS_VALID_END).
> > > > > >
> > > > > > If your prepBUFR files are separated out into multiple
files that
> > > need
> > > > to
> > > > > > be read into PB2NC, you can set a file window around the
valid
> time
> > > to
> > > > > > allow files to be passed into the app. More information on
that
> > > > > > functionality is here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > > > >
> > > > > > If you are still having trouble getting this configured,
you
> could
> > > send
> > > > > me
> > > > > > the paths of the files you are trying to read and what
data is
> > > > contained
> > > > > in
> > > > > > them so I can give you suggestions on what to set for the
> > > > configuration.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > > > > Queue: met_help
> > > > > > > Subject: How to set time for hourly 12-24 hours
data ?
> > > > > > > Owner: mccabe
> > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This transaction appears to have no content
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Tue Jul 13 11:55:42 2021
Hello George,
I finished "PCPCombine" , "RegridDataPlane", and "GridStat". But I'm
not
sure if it's right or not, could you check for me?
Thanks!
Zhuming
Also I have question about Gridstat:
(1) What's the threshold for? only check for the precipitation>12.7mm?
BOTH_VAR1_THRESH = gt12.7, gt25.4, gt50.8, gt76.2, gt152.4
(2) what's the mask data for?
REGRID_DATA_PLANE_VERIF_GRID =
{INPUT_BASE}/model_applications/precipitation/mask/CONUS_HRRRTLE.nc
see log file:
/glade/scratch/zhuming/metplus4/logs/metplus.log.20210713113230
and config files:
/glade/work/zhuming/METplus4/GridStat_wrf_obsStgIV_GRIB.conf
/glade/work/zhuming/METplus4/PCPCombine_add.conf
/glade/work/zhuming/METplus4/RegridDataPlane.conf
On Mon, Jun 28, 2021 at 1:54 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> The errors in your log file suggest that it cannot find the desired
field
> from the input file:
>
> DEBUG 1: Reading data (name="RAINRATE"; level="Surface";) from input
file:
> /glade/scratch/zhuming/pecan/st4/ST4.2015060200.01h
> ERROR :
> ERROR : VarInfo::set_level_info_grib() - failed to parse level
string
> 'Surface'
> ERROR :
>
> I ran wgrib on the input file and saw that there is a single field
in this
> file named APCP:
>
> (NPL) mccabe at cheyenne6:~> wgrib
> /glade/scratch/zhuming/pecan/st4/ST4.2015060200.01h
>
>
1:0:d=15060123:*APCP*:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=1:TimeU=1:sfc:0-
1hr
> acc:NAve=0
>
> You should change your configuration file to match the fields that
are
> found in the file:
>
> OBS_PCP_COMBINE_INPUT_NAMES = APCP
>
> OBS_PCP_COMBINE_INPUT_LEVELS = L0
>
> On Wed, Jun 23, 2021 at 3:17 PM zhuming at ucar.edu via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > Could you help on using "PCPCombine"? not sure why it has so many
ERRORs:
> > PCPCombine had 25 errors.
> >
> > config file : /glade/work/zhuming/METplus4/PCPCombine_add.conf
> > log file :
> /glade/scratch/zhuming/metplus4/logs/metplus.log.20210623151104
> >
> > Thanks,
> > Zhuming
> >
> > On Wed, Jun 23, 2021 at 10:45 AM George McCabe via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > Yes, I'd recommend using the PCPCombine_add example and modify
it to
> > match
> > > your data. Let me know if you have any issues getting it
configured
> and I
> > > can help.
> > >
> > > Thanks,
> > > George
> > >
> > > On Wed, Jun 23, 2021 at 10:39 AM zhuming at ucar.edu via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >
> > > > Hello George,
> > > >
> > > > Thanks for your clear description of precipitation
verification.
> > > >
> > > > I would like to do are
> > > >
> > > > (1) create 3 hourly obs precipitation from hourly observation
precip
> > > using
> > > > "PCPCombine"
> > > > (2) regrid these 3 hourly precip obs. to my small fcst
domain/grid
> > using
> > > > "RegridDataPlane"
> > > > (3) do precip verification using "GridStat". (Need to output 3
hourly
> > > > precip fcst? )
> > > >
> > > > Looking at the PCPCombine config files in
> met_tool_wrapper/PCPCombine,
> > I
> > > > chose "PCPCombine_add.conf" as my config file to revise, so
how to
> > change
> > > > the time to fit mine?
> > > >
> > > > Thanks!
> > > > Zhuming
> > > >
> > > > ls parm/use_cases/met_tool_wrapper/PCPCombine/
> > > >
> > > > PCPCombine_add.conf PCPCombine_derive.conf
> > > > PCPCombine_python_embedding.conf PCPCombine_sum.conf
> > > > sum_IMERG_V06_HDF5.py
> > > >
> > > > PCPCombine_bucket.conf PCPCombine_loop_custom.conf
> > > > PCPCombine_subtract.conf PCPCombine_user_defined.conf
> > > >
> > > > stage4 precip obs: /glade/scratch/zhuming/pecan/st4
> > > >
> > > > wrf fcst: /glade/scratch/zhuming/pecan/ctrl/fcst/cf/
> > > >
> > > > On Mon, Jun 21, 2021 at 11:21 AM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > > > Hi Zhuming,
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *(1) Could I change the names of the stat output file? I
prefer the
> > > > output
> > > > > filename including the initial time (such as 2015060100),
and the
> > > > forecast
> > > > > time, but they only have forecast time as follows. *
> > > > >
> > > > > The output stat files are named after the valid time because
they
> > > contain
> > > > > output that is generated from comparing forecast and
observation
> data
> > > at
> > > > a
> > > > > common valid time. However, you can set the output_prefix to
> include
> > > the
> > > > > forecast lead and init time so that this information will
also be
> > > > included
> > > > > in the output file names. You can set something like this:
> > > > >
> > > > >
> > > > > *POINT_STAT_OUTPUT_PREFIX =
> init{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H} *
> > > > >
> > > > > This extra information will be added after "point_stat_" in
the
> > output
> > > > > filenames.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > * (2) I need to do precipitation verification. I have
precipitation
> > obs
> > > > > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I
also have
> 48
> > > hr
> > > > > precipitation forecast data (small domain with an output
interval
> of
> > > > > 3hrs). I can find several config files of prescription
> verification
> > > with
> > > > > Stage4.*
> > > > >
> > > > >
> > > > >
> > > > > * So my question is which one is better for me to use, and
do I
> need
> > to
> > > > > convert/subdomain Stage4 obs data to my small forecast
domain with
> 3
> > > hrs
> > > > > interval? What is pcp_combine? *
> > > > >
> > > > > I think the best example config file to use depends on what
you are
> > > > trying
> > > > > to accomplish in your verification. These examples are
useful to
> > > > determine
> > > > > how to configure the wrappers to read the same input data
sets, but
> > the
> > > > > actual analysis and set of run times depends on your needs.
> > > > >
> > > > > There are configurations for multiple tools including
PointStat and
> > > > > GridStat to regrid data on the fly. See this page (search
for
> > > "regrid"):
> > > > >
> > > > >
> > > >
> > >
> >
>
https://met.readthedocs.io/en/latest/Users_Guide/config_options.html?highlight=output_prefix#settings-
common-to-multiple-tools
> > > > >
> > > > > It sounds like in your case, you would want to regrid the
data to
> the
> > > > > forecast domain since it is smaller than the observation
domain. So
> > the
> > > > MET
> > > > > config value would be "to_grid = FCST;" To configure this
with
> > METplus,
> > > > see
> > > > > this section of the METplus User's Guide (search for
> > > > METPLUS_REGRID_DICT):
> > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#grid-
stat-met-conf
> > > > >
> > > > > GRID_STAT_REGRID_TO_GRID = FCST
> > > > >
> > > > > Regridding on the fly does take additional processing time.
If you
> > find
> > > > > yourself needing to regrid the same file multiple times, it
is
> better
> > > to
> > > > > use another tool called RegridDataPlane that can perform the
> > regridding
> > > > and
> > > > > output a file that can be read by the other MET tools more
quickly.
> > > > Again,
> > > > > this depends on your use case.
> > > > >
> > > > > PCPCombine is a tool that (among other things) can be used
to
> compute
> > > > > different precipitation accumulations so that the forecast
and
> > > > observation
> > > > > accumulations match. For example, if you have hourly
observation
> > precip
> > > > > accumulations and 3 hourly forecast accumulations, you can
run
> > > PCPCombine
> > > > > on the observation data to create 3 hourly observation
> accumulations
> > > then
> > > > > compare that result to the forecast. More information on the
tool
> can
> > > be
> > > > > found here:
> > > > >
> > > > >
> > > >
> > >
> >
>
https://met.readthedocs.io/en/latest/Users_Guide/reformat_grid.html#pcp-
combine-tool
> > > > >
> > > > > I hope this information helps. If you have a clear idea of
the
> > > > verification
> > > > > you are trying to configure, I can help you set up your
config file
> > to
> > > > > accomplish that.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Fri, Jun 18, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > > >
> > > > > > Hello George,
> > > > > >
> > > > > > Thank you for your help. but I still need your help.
> > > > > >
> > > > > > (1) Could I change the names of the stat output file? I
prefer
> the
> > > > output
> > > > > > filename including the initial time (such as 2015060100),
and the
> > > > > forecast
> > > > > > time, but they only have forecast time as follows.
> > > > > >
> > > > > >
> > > >
> >
cheyenne2:/glade/scratch/zhuming/metplus4/point_stat_wrf_sfc/2015060100>l
> > > > > > *.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 929063 Jun 16 18:54
> > > > > > point_stat_000000L_20150601_000000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 920032 Jun 16 18:54
> > > > > > point_stat_000000L_20150601_030000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 926411 Jun 16 18:54
> > > > > > point_stat_000000L_20150601_060000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 928897 Jun 16 18:55
> > > > > > point_stat_000000L_20150601_090000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 929075 Jun 16 18:55
> > > > > > point_stat_000000L_20150601_120000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 912980 Jun 16 18:55
> > > > > > point_stat_000000L_20150601_150000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 902466 Jun 16 18:56
> > > > > > point_stat_000000L_20150601_180000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 899886 Jun 16 18:56
> > > > > > point_stat_000000L_20150601_210000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 951104 Jun 16 18:56
> > > > > > point_stat_000000L_20150602_000000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 909688 Jun 16 18:56
> > > > > > point_stat_000000L_20150602_030000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 931520 Jun 16 18:57
> > > > > > point_stat_000000L_20150602_060000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 912593 Jun 16 18:57
> > > > > > point_stat_000000L_20150602_090000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 905290 Jun 16 18:57
> > > > > > point_stat_000000L_20150602_120000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 908887 Jun 16 18:57
> > > > > > point_stat_000000L_20150602_150000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 890838 Jun 16 18:58
> > > > > > point_stat_000000L_20150602_180000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 904740 Jun 16 18:58
> > > > > > point_stat_000000L_20150602_210000V.stat
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 929844 Jun 16 18:58
> > > > > > point_stat_000000L_20150603_000000V.stat
> > > > > >
> > > > > >
> > > > > > (2) I need to do precipitation verification. I have
precipitation
> > obs
> > > > > > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I
also
> have
> > 48
> > > > hr
> > > > > > precipitation forecast data (small domain with an output
interval
> > of
> > > > > > 3hrs). I can find several config files of prescription
> > verification
> > > > with
> > > > > > Stage4.
> > > > > >
> > > > > > So my question is which one is better for me to use, and
do I
> need
> > to
> > > > > > convert/subdomain Stage4 obs data to my small forecast
domain
> with
> > 3
> > > > hrs
> > > > > > interval? What is pcp_combine?
> > > > > >
> > > > > >
> > > > > > l
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/parm/use_cases/model_applications/precipitation/
> > > > > > *StgIV*
> > > > > >
> > > > > > -rw-rw-r-- 1 jpresto p48503002 4606 May 3 14:51
> > > > > > GridStat_fcstHREFmean_obsStgIV_Gempak.conf
> > > > > >
> > > > > > -rw-rw-r-- 1 jpresto p48503002 5380 May 3 14:51
> > > > > > GridStat_fcstHREFmean_obsStgIV_NetCDF.conf
> > > > > >
> > > > > > -rw-rw-r-- 1 jpresto p48503002 4443 May 3 14:51
> > > > > > GridStat_fcstHRRR-TLE_obsStgIV_GRIB.conf
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks a lot!
> > > > > >
> > > > > > Zhuming
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Zhuming,
> > > > > > >
> > > > > > > To run the tools hourly you would set the
VALID_INCREMENT value
> > for
> > > > 1H.
> > > > > > > Here is an example that runs once per hour for a day
from
> > > 2015-07-15
> > > > 0Z
> > > > > > to
> > > > > > > 2017-07-16 0Z:
> > > > > > >
> > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > >
> > > > > > > VALID_BEG = 2015071500
> > > > > > >
> > > > > > > VALID_END = 2015071600
> > > > > > >
> > > > > > > VALID_INCREMENT = 1H
> > > > > > >
> > > > > > > LEAD_SEQ = 0
> > > > > > >
> > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find
files
> > whose
> > > > > > > filenames match the time you need.
> > > > > > >
> > > > > > > To read prepBUFR observations for 12 hours, you will
want to
> set
> > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the
range
> around
> > > the
> > > > > > valid
> > > > > > > time to process. These values depend on what is
contained
> inside
> > > your
> > > > > > > prepBUFR files. If you want to use data from the valid
time of
> > the
> > > > > > filename
> > > > > > > backwards 12 hours, you would set:
> > > > > > >
> > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > >
> > > > > > > For a runtime of 2015071512, this would be substituted
as:
> > > > > > >
> > > > > > > valid_beg: 20150715_00
> > > > > > > valid_end: 20150715_12
> > > > > > >
> > > > > > > If you want to create an output file for each hour in
PB2NC you
> > > would
> > > > > > need
> > > > > > > to make sure that the hour is specified in the
> > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > >
> > > > > > > PointStat has similar config variables to set the valid
beg and
> > end
> > > > > range
> > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
POINT_STAT_OBS_VALID_END).
> > > > > > >
> > > > > > > If your prepBUFR files are separated out into multiple
files
> that
> > > > need
> > > > > to
> > > > > > > be read into PB2NC, you can set a file window around the
valid
> > time
> > > > to
> > > > > > > allow files to be passed into the app. More information
on that
> > > > > > > functionality is here:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > > > > >
> > > > > > > If you are still having trouble getting this configured,
you
> > could
> > > > send
> > > > > > me
> > > > > > > the paths of the files you are trying to read and what
data is
> > > > > contained
> > > > > > in
> > > > > > > them so I can give you suggestions on what to set for
the
> > > > > configuration.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted
upon.
> > > > > > > > Transaction: Given to mccabe (George McCabe) by
jpresto
> > > > > > > > Queue: met_help
> > > > > > > > Subject: How to set time for hourly 12-24 hours
data ?
> > > > > > > > Owner: mccabe
> > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > This transaction appears to have no content
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Jul 14 09:07:45 2021
Hi Zhuming,
It looks like the log file you provided only runs GridStat portion. It
is
using the data generated from the other steps (PCPCombine and
RegridDataPlane) and it is producing output results, so I don't see
anything wrong with what you are running. Whether or not the output
are
"right" or not depends on what you are trying to accomplish with your
use
case. I would recommend reviewing the MET User's Guide to understand
what
output can be produces with the tools to help determine what you want
to
configure. https://met.readthedocs.io/en/latest/Users_Guide/grid-
stat.html
The VAR<n>_THRESH variables are used to set the categorical threshold
(cat_thresh) values. It is used to bin the data into groups that can
be
analyzed. More information on this topic can be found in MET User's
Guide.
https://met.readthedocs.io/en/latest/Users_Guide/appendixC.html#met-
verification-measures-for-categorical-dichotomous-variables
In RegridDataPlane, the verification grid defines the "to_grid" output
grid
to use when regridding the data (See
https://met.readthedocs.io/en/latest/Users_Guide/reformat_grid.html#required-
arguments-for-regrid-data-plane).
This can be the path to a file that MET can read to use the grid of
that
data or an explicit grid definition. In this example, the tool is
regridding the input data to the grid of the mask data. See the
section
titled "regrid" on this page for more information:
https://met.readthedocs.io/en/latest/Users_Guide/config_options.html#settings-
common-to-multiple-tools
Also FYI:
We switched from providing support through this email address to
providing
support through GitHub Discussions. In the future, you will need to
create
a free GitHub account if you don’t have one already and post your
questions
to the METplus Components Discussion Forum:
https://github.com/dtcenter/METplus/discussions
To post a new question, click on the green button in the upper right
corner
of the page labeled "New discussion" and select "Incoming" as the
category.
<https://github.com/dtcenter/METplus/discussions>
Thanks,
George
On Tue, Jul 13, 2021 at 11:55 AM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hello George,
>
> I finished "PCPCombine" , "RegridDataPlane", and "GridStat". But I'm
not
> sure if it's right or not, could you check for me?
>
> Thanks!
> Zhuming
>
> Also I have question about Gridstat:
>
> (1) What's the threshold for? only check for the
precipitation>12.7mm?
>
> BOTH_VAR1_THRESH = gt12.7, gt25.4, gt50.8, gt76.2, gt152.4
>
>
> (2) what's the mask data for?
>
> REGRID_DATA_PLANE_VERIF_GRID =
> {INPUT_BASE}/model_applications/precipitation/mask/CONUS_HRRRTLE.nc
>
> see log file:
> /glade/scratch/zhuming/metplus4/logs/metplus.log.20210713113230
>
>
> and config files:
>
> /glade/work/zhuming/METplus4/GridStat_wrf_obsStgIV_GRIB.conf
>
> /glade/work/zhuming/METplus4/PCPCombine_add.conf
>
> /glade/work/zhuming/METplus4/RegridDataPlane.conf
>
> On Mon, Jun 28, 2021 at 1:54 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > The errors in your log file suggest that it cannot find the
desired field
> > from the input file:
> >
> > DEBUG 1: Reading data (name="RAINRATE"; level="Surface";) from
input
> file:
> > /glade/scratch/zhuming/pecan/st4/ST4.2015060200.01h
> > ERROR :
> > ERROR : VarInfo::set_level_info_grib() - failed to parse level
string
> > 'Surface'
> > ERROR :
> >
> > I ran wgrib on the input file and saw that there is a single field
in
> this
> > file named APCP:
> >
> > (NPL) mccabe at cheyenne6:~> wgrib
> > /glade/scratch/zhuming/pecan/st4/ST4.2015060200.01h
> >
> >
>
1:0:d=15060123:*APCP*:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=1:TimeU=1:sfc:0-
1hr
> > acc:NAve=0
> >
> > You should change your configuration file to match the fields that
are
> > found in the file:
> >
> > OBS_PCP_COMBINE_INPUT_NAMES = APCP
> >
> > OBS_PCP_COMBINE_INPUT_LEVELS = L0
> >
> > On Wed, Jun 23, 2021 at 3:17 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > Could you help on using "PCPCombine"? not sure why it has so
many
> ERRORs:
> > > PCPCombine had 25 errors.
> > >
> > > config file : /glade/work/zhuming/METplus4/PCPCombine_add.conf
> > > log file :
> > /glade/scratch/zhuming/metplus4/logs/metplus.log.20210623151104
> > >
> > > Thanks,
> > > Zhuming
> > >
> > > On Wed, Jun 23, 2021 at 10:45 AM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > Yes, I'd recommend using the PCPCombine_add example and modify
it to
> > > match
> > > > your data. Let me know if you have any issues getting it
configured
> > and I
> > > > can help.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Wed, Jun 23, 2021 at 10:39 AM zhuming at ucar.edu via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >
> > > > > Hello George,
> > > > >
> > > > > Thanks for your clear description of precipitation
verification.
> > > > >
> > > > > I would like to do are
> > > > >
> > > > > (1) create 3 hourly obs precipitation from hourly
observation
> precip
> > > > using
> > > > > "PCPCombine"
> > > > > (2) regrid these 3 hourly precip obs. to my small fcst
domain/grid
> > > using
> > > > > "RegridDataPlane"
> > > > > (3) do precip verification using "GridStat". (Need to output
3
> hourly
> > > > > precip fcst? )
> > > > >
> > > > > Looking at the PCPCombine config files in
> > met_tool_wrapper/PCPCombine,
> > > I
> > > > > chose "PCPCombine_add.conf" as my config file to revise, so
how to
> > > change
> > > > > the time to fit mine?
> > > > >
> > > > > Thanks!
> > > > > Zhuming
> > > > >
> > > > > ls parm/use_cases/met_tool_wrapper/PCPCombine/
> > > > >
> > > > > PCPCombine_add.conf PCPCombine_derive.conf
> > > > > PCPCombine_python_embedding.conf PCPCombine_sum.conf
> > > > > sum_IMERG_V06_HDF5.py
> > > > >
> > > > > PCPCombine_bucket.conf PCPCombine_loop_custom.conf
> > > > > PCPCombine_subtract.conf
PCPCombine_user_defined.conf
> > > > >
> > > > > stage4 precip obs: /glade/scratch/zhuming/pecan/st4
> > > > >
> > > > > wrf fcst: /glade/scratch/zhuming/pecan/ctrl/fcst/cf/
> > > > >
> > > > > On Mon, Jun 21, 2021 at 11:21 AM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Zhuming,
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *(1) Could I change the names of the stat output file? I
prefer
> the
> > > > > output
> > > > > > filename including the initial time (such as 2015060100),
and the
> > > > > forecast
> > > > > > time, but they only have forecast time as follows. *
> > > > > >
> > > > > > The output stat files are named after the valid time
because they
> > > > contain
> > > > > > output that is generated from comparing forecast and
observation
> > data
> > > > at
> > > > > a
> > > > > > common valid time. However, you can set the output_prefix
to
> > include
> > > > the
> > > > > > forecast lead and init time so that this information will
also be
> > > > > included
> > > > > > in the output file names. You can set something like this:
> > > > > >
> > > > > >
> > > > > > *POINT_STAT_OUTPUT_PREFIX =
> > init{init?fmt=%Y%m%d%H}_F{lead?fmt=%3H} *
> > > > > >
> > > > > > This extra information will be added after "point_stat_"
in the
> > > output
> > > > > > filenames.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > * (2) I need to do precipitation verification. I have
> precipitation
> > > obs
> > > > > > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I
also
> have
> > 48
> > > > hr
> > > > > > precipitation forecast data (small domain with an output
interval
> > of
> > > > > > 3hrs). I can find several config files of prescription
> > verification
> > > > with
> > > > > > Stage4.*
> > > > > >
> > > > > >
> > > > > >
> > > > > > * So my question is which one is better for me to use, and
do I
> > need
> > > to
> > > > > > convert/subdomain Stage4 obs data to my small forecast
domain
> with
> > 3
> > > > hrs
> > > > > > interval? What is pcp_combine? *
> > > > > >
> > > > > > I think the best example config file to use depends on
what you
> are
> > > > > trying
> > > > > > to accomplish in your verification. These examples are
useful to
> > > > > determine
> > > > > > how to configure the wrappers to read the same input data
sets,
> but
> > > the
> > > > > > actual analysis and set of run times depends on your
needs.
> > > > > >
> > > > > > There are configurations for multiple tools including
PointStat
> and
> > > > > > GridStat to regrid data on the fly. See this page (search
for
> > > > "regrid"):
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://met.readthedocs.io/en/latest/Users_Guide/config_options.html?highlight=output_prefix#settings-
common-to-multiple-tools
> > > > > >
> > > > > > It sounds like in your case, you would want to regrid the
data to
> > the
> > > > > > forecast domain since it is smaller than the observation
domain.
> So
> > > the
> > > > > MET
> > > > > > config value would be "to_grid = FCST;" To configure this
with
> > > METplus,
> > > > > see
> > > > > > this section of the METplus User's Guide (search for
> > > > > METPLUS_REGRID_DICT):
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#grid-
stat-met-conf
> > > > > >
> > > > > > GRID_STAT_REGRID_TO_GRID = FCST
> > > > > >
> > > > > > Regridding on the fly does take additional processing
time. If
> you
> > > find
> > > > > > yourself needing to regrid the same file multiple times,
it is
> > better
> > > > to
> > > > > > use another tool called RegridDataPlane that can perform
the
> > > regridding
> > > > > and
> > > > > > output a file that can be read by the other MET tools more
> quickly.
> > > > > Again,
> > > > > > this depends on your use case.
> > > > > >
> > > > > > PCPCombine is a tool that (among other things) can be used
to
> > compute
> > > > > > different precipitation accumulations so that the forecast
and
> > > > > observation
> > > > > > accumulations match. For example, if you have hourly
observation
> > > precip
> > > > > > accumulations and 3 hourly forecast accumulations, you can
run
> > > > PCPCombine
> > > > > > on the observation data to create 3 hourly observation
> > accumulations
> > > > then
> > > > > > compare that result to the forecast. More information on
the tool
> > can
> > > > be
> > > > > > found here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://met.readthedocs.io/en/latest/Users_Guide/reformat_grid.html#pcp-
combine-tool
> > > > > >
> > > > > > I hope this information helps. If you have a clear idea of
the
> > > > > verification
> > > > > > you are trying to configure, I can help you set up your
config
> file
> > > to
> > > > > > accomplish that.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Fri, Jun 18, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >
> > > > > > >
> > > > > > > Hello George,
> > > > > > >
> > > > > > > Thank you for your help. but I still need your help.
> > > > > > >
> > > > > > > (1) Could I change the names of the stat output file? I
prefer
> > the
> > > > > output
> > > > > > > filename including the initial time (such as
2015060100), and
> the
> > > > > > forecast
> > > > > > > time, but they only have forecast time as follows.
> > > > > > >
> > > > > > >
> > > > >
> > >
>
cheyenne2:/glade/scratch/zhuming/metplus4/point_stat_wrf_sfc/2015060100>l
> > > > > > > *.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 929063 Jun 16 18:54
> > > > > > > point_stat_000000L_20150601_000000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 920032 Jun 16 18:54
> > > > > > > point_stat_000000L_20150601_030000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 926411 Jun 16 18:54
> > > > > > > point_stat_000000L_20150601_060000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 928897 Jun 16 18:55
> > > > > > > point_stat_000000L_20150601_090000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 929075 Jun 16 18:55
> > > > > > > point_stat_000000L_20150601_120000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 912980 Jun 16 18:55
> > > > > > > point_stat_000000L_20150601_150000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 902466 Jun 16 18:56
> > > > > > > point_stat_000000L_20150601_180000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 899886 Jun 16 18:56
> > > > > > > point_stat_000000L_20150601_210000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 951104 Jun 16 18:56
> > > > > > > point_stat_000000L_20150602_000000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 909688 Jun 16 18:56
> > > > > > > point_stat_000000L_20150602_030000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 931520 Jun 16 18:57
> > > > > > > point_stat_000000L_20150602_060000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 912593 Jun 16 18:57
> > > > > > > point_stat_000000L_20150602_090000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 905290 Jun 16 18:57
> > > > > > > point_stat_000000L_20150602_120000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 908887 Jun 16 18:57
> > > > > > > point_stat_000000L_20150602_150000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 890838 Jun 16 18:58
> > > > > > > point_stat_000000L_20150602_180000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 904740 Jun 16 18:58
> > > > > > > point_stat_000000L_20150602_210000V.stat
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 929844 Jun 16 18:58
> > > > > > > point_stat_000000L_20150603_000000V.stat
> > > > > > >
> > > > > > >
> > > > > > > (2) I need to do precipitation verification. I have
> precipitation
> > > obs
> > > > > > > Stage4 data (Conus domain, with hourly, 6hr and 24hr). I
also
> > have
> > > 48
> > > > > hr
> > > > > > > precipitation forecast data (small domain with an output
> interval
> > > of
> > > > > > > 3hrs). I can find several config files of prescription
> > > verification
> > > > > with
> > > > > > > Stage4.
> > > > > > >
> > > > > > > So my question is which one is better for me to use, and
do I
> > need
> > > to
> > > > > > > convert/subdomain Stage4 obs data to my small forecast
domain
> > with
> > > 3
> > > > > hrs
> > > > > > > interval? What is pcp_combine?
> > > > > > >
> > > > > > >
> > > > > > > l
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta5/parm/use_cases/model_applications/precipitation/
> > > > > > > *StgIV*
> > > > > > >
> > > > > > > -rw-rw-r-- 1 jpresto p48503002 4606 May 3 14:51
> > > > > > > GridStat_fcstHREFmean_obsStgIV_Gempak.conf
> > > > > > >
> > > > > > > -rw-rw-r-- 1 jpresto p48503002 5380 May 3 14:51
> > > > > > > GridStat_fcstHREFmean_obsStgIV_NetCDF.conf
> > > > > > >
> > > > > > > -rw-rw-r-- 1 jpresto p48503002 4443 May 3 14:51
> > > > > > > GridStat_fcstHRRR-TLE_obsStgIV_GRIB.conf
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks a lot!
> > > > > > >
> > > > > > > Zhuming
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Zhuming,
> > > > > > > >
> > > > > > > > To run the tools hourly you would set the
VALID_INCREMENT
> value
> > > for
> > > > > 1H.
> > > > > > > > Here is an example that runs once per hour for a day
from
> > > > 2015-07-15
> > > > > 0Z
> > > > > > > to
> > > > > > > > 2017-07-16 0Z:
> > > > > > > >
> > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > > >
> > > > > > > > VALID_BEG = 2015071500
> > > > > > > >
> > > > > > > > VALID_END = 2015071600
> > > > > > > >
> > > > > > > > VALID_INCREMENT = 1H
> > > > > > > >
> > > > > > > > LEAD_SEQ = 0
> > > > > > > >
> > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to
find files
> > > whose
> > > > > > > > filenames match the time you need.
> > > > > > > >
> > > > > > > > To read prepBUFR observations for 12 hours, you will
want to
> > set
> > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the
range
> > around
> > > > the
> > > > > > > valid
> > > > > > > > time to process. These values depend on what is
contained
> > inside
> > > > your
> > > > > > > > prepBUFR files. If you want to use data from the valid
time
> of
> > > the
> > > > > > > filename
> > > > > > > > backwards 12 hours, you would set:
> > > > > > > >
> > > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > > >
> > > > > > > > For a runtime of 2015071512, this would be substituted
as:
> > > > > > > >
> > > > > > > > valid_beg: 20150715_00
> > > > > > > > valid_end: 20150715_12
> > > > > > > >
> > > > > > > > If you want to create an output file for each hour in
PB2NC
> you
> > > > would
> > > > > > > need
> > > > > > > > to make sure that the hour is specified in the
> > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > > >
> > > > > > > > PointStat has similar config variables to set the
valid beg
> and
> > > end
> > > > > > range
> > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> POINT_STAT_OBS_VALID_END).
> > > > > > > >
> > > > > > > > If your prepBUFR files are separated out into multiple
files
> > that
> > > > > need
> > > > > > to
> > > > > > > > be read into PB2NC, you can set a file window around
the
> valid
> > > time
> > > > > to
> > > > > > > > allow files to be passed into the app. More
information on
> that
> > > > > > > > functionality is here:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > > > > > >
> > > > > > > > If you are still having trouble getting this
configured, you
> > > could
> > > > > send
> > > > > > > me
> > > > > > > > the paths of the files you are trying to read and what
data
> is
> > > > > > contained
> > > > > > > in
> > > > > > > > them so I can give you suggestions on what to set for
the
> > > > > > configuration.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted
upon.
> > > > > > > > > Transaction: Given to mccabe (George McCabe) by
jpresto
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: How to set time for hourly 12-24 hours
data ?
> > > > > > > > > Owner: mccabe
> > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This transaction appears to have no content
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list