[Met_help] [rt.rap.ucar.edu #99429] History for How to set time for hourly 12-24 hours data ?
George McCabe via RT
met_help at ucar.edu
Wed May 12 07:28:39 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
Could you let me know how to set time on processing hourly Obs. data for
lasting 12 -24hours using "PB2NC", and do hourly verification for lasting
12-24 hours using Point-stat?
such as:
VALID_TIME_FMT = %Y%m%d%H
VALID_BEG = 2015071500
VALID_END = 2015071500
VALID_INCREMENT = 1M
LEAD_SEQ = 0
Thanks!
Zhuming
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Tue Apr 06 16:23:00 2021
Hi Zhuming,
To run the tools hourly you would set the VALID_INCREMENT value for
1H.
Here is an example that runs once per hour for a day from 2015-07-15
0Z to
2017-07-16 0Z:
VALID_TIME_FMT = %Y%m%d%H
VALID_BEG = 2015071500
VALID_END = 2015071600
VALID_INCREMENT = 1H
LEAD_SEQ = 0
Each run time will use the PB2NC_INPUT_TEMPLATE to find files whose
filenames match the time you need.
To read prepBUFR observations for 12 hours, you will want to set
PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range around the
valid
time to process. These values depend on what is contained inside your
prepBUFR files. If you want to use data from the valid time of the
filename
backwards 12 hours, you would set:
PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
For a runtime of 2015071512, this would be substituted as:
valid_beg: 20150715_00
valid_end: 20150715_12
If you want to create an output file for each hour in PB2NC you would
need
to make sure that the hour is specified in the PB2NC_OUTPUT_TEMPLATE.
PointStat has similar config variables to set the valid beg and end
range
to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
If your prepBUFR files are separated out into multiple files that need
to
be read into PB2NC, you can set a file window around the valid time to
allow files to be passed into the app. More information on that
functionality is here:
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
If you are still having trouble getting this configured, you could
send me
the paths of the files you are trying to read and what data is
contained in
them so I can give you suggestions on what to set for the
configuration.
Thanks,
George
On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> Transaction: Given to mccabe (George McCabe) by jpresto
> Queue: met_help
> Subject: How to set time for hourly 12-24 hours data ?
> Owner: mccabe
> Requestors: zhuming at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
>
> This transaction appears to have no content
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Tue Apr 06 17:16:50 2021
Hi George,
Thanks for your reply!
I'm still confused, I need to output hourly obs data to do hourly
verification.
The original GDAS Prepbufr data has 4 files one day, should with 6
hours
obs data in one file.
After "PB2NC", I can see pbs.2015071500.nc (obs time is 00? or 6 times
of
00,01,02,03,04,05?), pbs.2015071506.nc...
my config file : cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
Thanks,
Zhuming
On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> To run the tools hourly you would set the VALID_INCREMENT value for
1H.
> Here is an example that runs once per hour for a day from 2015-07-15
0Z to
> 2017-07-16 0Z:
>
> VALID_TIME_FMT = %Y%m%d%H
>
> VALID_BEG = 2015071500
>
> VALID_END = 2015071600
>
> VALID_INCREMENT = 1H
>
> LEAD_SEQ = 0
>
> Each run time will use the PB2NC_INPUT_TEMPLATE to find files whose
> filenames match the time you need.
>
> To read prepBUFR observations for 12 hours, you will want to set
> PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range around
the valid
> time to process. These values depend on what is contained inside
your
> prepBUFR files. If you want to use data from the valid time of the
filename
> backwards 12 hours, you would set:
>
> PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
>
> For a runtime of 2015071512, this would be substituted as:
>
> valid_beg: 20150715_00
> valid_end: 20150715_12
>
> If you want to create an output file for each hour in PB2NC you
would need
> to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
>
> PointStat has similar config variables to set the valid beg and end
range
> to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
>
> If your prepBUFR files are separated out into multiple files that
need to
> be read into PB2NC, you can set a file window around the valid time
to
> allow files to be passed into the app. More information on that
> functionality is here:
>
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
>
> If you are still having trouble getting this configured, you could
send me
> the paths of the files you are trying to read and what data is
contained in
> them so I can give you suggestions on what to set for the
configuration.
>
> Thanks,
> George
>
>
>
> On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > Transaction: Given to mccabe (George McCabe) by jpresto
> > Queue: met_help
> > Subject: How to set time for hourly 12-24 hours data ?
> > Owner: mccabe
> > Requestors: zhuming at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> >
> > This transaction appears to have no content
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Tue Apr 06 18:34:09 2021
Hi Zhuming,
Thanks for the information. You will need to use the truncate keyword
to
find the files, then set the valid beg/end range to 1 hour. This is
what I
changed from your PB2NC1.conf config file:
*PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
{valid?fmt=%Y%m%d_%H?shift=1H}*
*PB2NC_INPUT_TEMPLATE =
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
My config file is on cheyenne here:
/glade/u/home/mccabe/PB2NC1.conf
I just started running it and the output is being written here:
/glade/u/home/mccabe/data/out/pb2nc1
For PointStat, you will want to loop over the same times. I noticed
you
specify message type in the VAR<n>_OPTIONS, but you do not need to do
this
since there is a POINT_STAT_MESSAGE_TYPE variable that defines this.
This
is what I changed in your PointStat1.conf file:
*VALID_END = 2015071600*
*VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
=POINT_STAT_OBS_VALID_BEG =*
*POINT_STAT_MESSAGE_TYPE = ADPSFC*
*#OBS_VAR1_OPTIONS = message_type = ["ADPSFC"];#OBS_VAR2_OPTIONS =
message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
/glade/scratch/zhuming/metplus4/pb2nc1*
*OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
Or you can view my file in the same directory. I usually try to test
with a
single run time to make sure it is working properly, so I set
VALID_END
back to 2015071500. I ran it and it produced an output file, but there
were
no values besides the header. I checked the log file and saw this:
*DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for observation type
ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
processed = 393373DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected: obs type = 390528DEBUG 3: Rejected: valid time =
0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the grid
=
2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
mismatch
= 259DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected: message
type
= 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3: Rejected: bad
climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
--------------------------------------------------------------------------------DEBUG
2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for observation
type
ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
processed = 393373*
*DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected: obs type
=
382965DEBUG 3: Rejected: valid time = 0DEBUG 3: Rejected: bad obs
value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3: Rejected:
topography = 0DEBUG 3: Rejected: level mismatch = 1168DEBUG 3:
Rejected: quality marker = 0DEBUG 3: Rejected: message type =
0DEBUG 3:
Rejected: masking region = 0DEBUG 3: Rejected: bad fcst value =
0DEBUG 3:
Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo stdev =
0DEBUG 3:
Rejected: duplicates = 0DEBUG 2:DEBUG 2:
--------------------------------------------------------------------------------DEBUG
2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for observation
type
ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
processed = 393373DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected: obs type = 382965DEBUG 3: Rejected: valid time =
0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the grid
=
9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
mismatch
= 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected:
bad
fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected: bad
climo stdev = 0*
*DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
--------------------------------------------------------------------------------DEBUG
2:DEBUG 1: Output file:
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
This describes the reason why each observation was rejected. Some
observations didn't match the message type you requested, some were
not
within the forecast grid, and some did not match the vertical level.
You
will have to make adjustments to your settings so that the data
matches up
properly. Maybe the grid you are using in the PB2NC config file does
not
match the forecast grid and you are missing values. If that is the
case,
you will have to reconfigure and rerun PB2NC to get output on the
correct
gr5id. Maybe the observation points don't line up with the vertical
level
of the model fields. You can use a range of level values for the
observation data to allow data within that range, i.e. Z0-10.
Let me know if you are still having issues getting this to work and I
can
help out.
Thanks,
George
On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> Thanks for your reply!
>
> I'm still confused, I need to output hourly obs data to do hourly
> verification.
> The original GDAS Prepbufr data has 4 files one day, should with 6
hours
> obs data in one file.
>
> After "PB2NC", I can see pbs.2015071500.nc (obs time is 00? or 6
times of
> 00,01,02,03,04,05?), pbs.2015071506.nc...
>
> my config file :
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
>
> Thanks,
> Zhuming
>
> On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > To run the tools hourly you would set the VALID_INCREMENT value
for 1H.
> > Here is an example that runs once per hour for a day from 2015-07-
15 0Z
> to
> > 2017-07-16 0Z:
> >
> > VALID_TIME_FMT = %Y%m%d%H
> >
> > VALID_BEG = 2015071500
> >
> > VALID_END = 2015071600
> >
> > VALID_INCREMENT = 1H
> >
> > LEAD_SEQ = 0
> >
> > Each run time will use the PB2NC_INPUT_TEMPLATE to find files
whose
> > filenames match the time you need.
> >
> > To read prepBUFR observations for 12 hours, you will want to set
> > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range around
the
> valid
> > time to process. These values depend on what is contained inside
your
> > prepBUFR files. If you want to use data from the valid time of the
> filename
> > backwards 12 hours, you would set:
> >
> > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> >
> > For a runtime of 2015071512, this would be substituted as:
> >
> > valid_beg: 20150715_00
> > valid_end: 20150715_12
> >
> > If you want to create an output file for each hour in PB2NC you
would
> need
> > to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
> >
> > PointStat has similar config variables to set the valid beg and
end range
> > to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
> >
> > If your prepBUFR files are separated out into multiple files that
need to
> > be read into PB2NC, you can set a file window around the valid
time to
> > allow files to be passed into the app. More information on that
> > functionality is here:
> >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> >
> > If you are still having trouble getting this configured, you could
send
> me
> > the paths of the files you are trying to read and what data is
contained
> in
> > them so I can give you suggestions on what to set for the
configuration.
> >
> > Thanks,
> > George
> >
> >
> >
> > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > Queue: met_help
> > > Subject: How to set time for hourly 12-24 hours data ?
> > > Owner: mccabe
> > > Requestors: zhuming at ucar.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Wed Apr 07 17:31:37 2021
Hi George,
Now makes me confused is why processed obs of several times are
missing.
There has files of time at 00,01,02, but no 03,04,05; has 06,07,08,
but no
09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with 21,no
22,23.
>From yours and mines:
ls -l /glade/u/home/mccabe/data/out/pb2nc1/
-rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52 pbs.2015071500.nc
-rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56 pbs.2015071501.nc
-rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00 pbs.2015071502.nc
-rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14 pbs.2015071506.nc
-rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18 pbs.2015071507.nc
-rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22 pbs.2015071508.nc
-rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36 pbs.2015071512.nc
-rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40 pbs.2015071513.nc
-rw-r--r-- 1 mccabe ral 96 Apr 6 18:40 pbs.2015071514.nc
-rw-r--r-- 1 mccabe ral 96 Apr 6 18:42 pbs.2015071515.nc
cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
total 96112
-rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11 pbs.2015071500.nc
-rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14 pbs.2015071501.nc
-rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18 pbs.2015071502.nc
-rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33 pbs.2015071506.nc
-rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36 pbs.2015071507.nc
-rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40 pbs.2015071508.nc
-rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55 pbs.2015071512.nc
-rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59 pbs.2015071513.nc
-rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03 pbs.2015071514.nc
-rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18 pbs.2015071518.nc
-rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22 pbs.2015071519.nc
-rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27 pbs.2015071520.nc
-rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31 pbs.2015071521.nc
-rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43 pbs.2015071600.nc
Thanks,
Zhuming
On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> Thanks for the information. You will need to use the truncate
keyword to
> find the files, then set the valid beg/end range to 1 hour. This is
what I
> changed from your PB2NC1.conf config file:
>
>
> *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> {valid?fmt=%Y%m%d_%H?shift=1H}*
>
> *PB2NC_INPUT_TEMPLATE =
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
>
> My config file is on cheyenne here:
> /glade/u/home/mccabe/PB2NC1.conf
>
> I just started running it and the output is being written here:
> /glade/u/home/mccabe/data/out/pb2nc1
>
>
> For PointStat, you will want to loop over the same times. I noticed
you
> specify message type in the VAR<n>_OPTIONS, but you do not need to
do this
> since there is a POINT_STAT_MESSAGE_TYPE variable that defines this.
This
> is what I changed in your PointStat1.conf file:
>
> *VALID_END = 2015071600*
>
>
>
> *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
=POINT_STAT_OBS_VALID_BEG =*
>
>
> *POINT_STAT_MESSAGE_TYPE = ADPSFC*
>
>
>
>
>
> *#OBS_VAR1_OPTIONS = message_type = ["ADPSFC"];#OBS_VAR2_OPTIONS =
> message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
> ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> /glade/scratch/zhuming/metplus4/pb2nc1*
>
> *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
>
> Or you can view my file in the same directory. I usually try to test
with a
> single run time to make sure it is working properly, so I set
VALID_END
> back to 2015071500. I ran it and it produced an output file, but
there were
> no values besides the header. I checked the log file and saw this:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for observation
type
> ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
> matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> processed = 393373DEBUG 3: Rejected: station id = 0DEBUG 3:
> Rejected: obs type = 390528DEBUG 3: Rejected: valid time
=
> 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the
grid =
> 2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
mismatch
> = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message type
> = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
> value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3: Rejected:
bad
> climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG
2:
>
>
--------------------------------------------------------------------------------DEBUG
> 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
observation type
> ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
> matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> processed = 393373*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected: obs type
=
> 382965DEBUG 3: Rejected: valid time = 0DEBUG 3: Rejected: bad
obs
> value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3:
Rejected:
> topography = 0DEBUG 3: Rejected: level mismatch = 1168DEBUG 3:
> Rejected: quality marker = 0DEBUG 3: Rejected: message type =
0DEBUG 3:
> Rejected: masking region = 0DEBUG 3: Rejected: bad fcst value =
0DEBUG 3:
> Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo stdev =
0DEBUG 3:
> Rejected: duplicates = 0DEBUG 2:DEBUG 2:
>
>
--------------------------------------------------------------------------------DEBUG
> 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
observation type
> ADPSFC, over region FULL, for interpolation method BILIN(4), using 0
> matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> processed = 393373DEBUG 3: Rejected: station id = 0DEBUG 3:
> Rejected: obs type = 382965DEBUG 3: Rejected: valid time
=
> 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the
grid =
> 9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
mismatch
> = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
> type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected:
bad
> fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected: bad
> climo stdev = 0*
>
>
>
>
> *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
>
>
--------------------------------------------------------------------------------DEBUG
> 2:DEBUG 1: Output file:
>
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
>
>
>
> This describes the reason why each observation was rejected. Some
> observations didn't match the message type you requested, some were
not
> within the forecast grid, and some did not match the vertical level.
You
> will have to make adjustments to your settings so that the data
matches up
> properly. Maybe the grid you are using in the PB2NC config file does
not
> match the forecast grid and you are missing values. If that is the
case,
> you will have to reconfigure and rerun PB2NC to get output on the
correct
> gr5id. Maybe the observation points don't line up with the vertical
level
> of the model fields. You can use a range of level values for the
> observation data to allow data within that range, i.e. Z0-10.
>
> Let me know if you are still having issues getting this to work and
I can
> help out.
>
> Thanks,
> George
>
>
> On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > Thanks for your reply!
> >
> > I'm still confused, I need to output hourly obs data to do hourly
> > verification.
> > The original GDAS Prepbufr data has 4 files one day, should with 6
hours
> > obs data in one file.
> >
> > After "PB2NC", I can see pbs.2015071500.nc (obs time is 00? or 6
times
> of
> > 00,01,02,03,04,05?), pbs.2015071506.nc...
> >
> > my config file :
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> >
> > Thanks,
> > Zhuming
> >
> > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > To run the tools hourly you would set the VALID_INCREMENT value
for 1H.
> > > Here is an example that runs once per hour for a day from 2015-
07-15 0Z
> > to
> > > 2017-07-16 0Z:
> > >
> > > VALID_TIME_FMT = %Y%m%d%H
> > >
> > > VALID_BEG = 2015071500
> > >
> > > VALID_END = 2015071600
> > >
> > > VALID_INCREMENT = 1H
> > >
> > > LEAD_SEQ = 0
> > >
> > > Each run time will use the PB2NC_INPUT_TEMPLATE to find files
whose
> > > filenames match the time you need.
> > >
> > > To read prepBUFR observations for 12 hours, you will want to set
> > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around the
> > valid
> > > time to process. These values depend on what is contained inside
your
> > > prepBUFR files. If you want to use data from the valid time of
the
> > filename
> > > backwards 12 hours, you would set:
> > >
> > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > >
> > > For a runtime of 2015071512, this would be substituted as:
> > >
> > > valid_beg: 20150715_00
> > > valid_end: 20150715_12
> > >
> > > If you want to create an output file for each hour in PB2NC you
would
> > need
> > > to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
> > >
> > > PointStat has similar config variables to set the valid beg and
end
> range
> > > to use (POINT_STAT_OBS_VALID_BEG and POINT_STAT_OBS_VALID_END).
> > >
> > > If your prepBUFR files are separated out into multiple files
that need
> to
> > > be read into PB2NC, you can set a file window around the valid
time to
> > > allow files to be passed into the app. More information on that
> > > functionality is here:
> > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > >
> > > If you are still having trouble getting this configured, you
could send
> > me
> > > the paths of the files you are trying to read and what data is
> contained
> > in
> > > them so I can give you suggestions on what to set for the
> configuration.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > >
> > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > Queue: met_help
> > > > Subject: How to set time for hourly 12-24 hours data ?
> > > > Owner: mccabe
> > > > Requestors: zhuming at ucar.edu
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >
> > > >
> > > >
> > > > This transaction appears to have no content
> > > >
> > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Apr 07 17:42:33 2021
Hi Zhuming,
Looking at the log output, it looks like the data is stored in the
prepBUFR
files differently that you may have thought:
DEBUG 2: PrepBufr Time Center: 20150715_000000
DEBUG 2: Searching Time Window: 20150715_030000 to
20150715_040000
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90%
95%
100%
DEBUG 2: Total Messages processed = 590836
DEBUG 2: Rejected based on message type = 0
DEBUG 2: Rejected based on station id = 0
DEBUG 2: Rejected based on valid time = 590815
DEBUG 2: Rejected based on masking grid = 20
DEBUG 2: Rejected based on masking polygon = 0
DEBUG 2: Rejected based on elevation = 0
DEBUG 2: Rejected based on pb report type = 1
DEBUG 2: Rejected based on input report type = 0
DEBUG 2: Rejected based on instrument type = 0
DEBUG 2: Rejected based on zero observations = 0
DEBUG 2: Total Messages retained = 0
DEBUG 2: Total observations retained or derived = 0
DEBUG 1: Obs time between *20150714_210000 and 20150715_030000*
WARNING:
WARNING: process_pbfile() -> No PrepBufr messages retained from file:
/glade/collections/rda/data/ds337.0/prepnr/2015/prepbufr.gdas.2015071500.nr
WARNING:
ERROR :
ERROR : write_netcdf_hdr_data() -> No PrepBufr messages retained.
Nothing
to write.
ERROR :
The prepbufr.gdas.2015071500.nr file actually contains data from
20150714_210000 to 20150715_030000, not 20150715_000000 to
20150715_060000.
To get the correct file for each run, you will have to shift the valid
time
before truncating it so that it gets the correct file for each run.
Instead
of setting:
PB2NC_INPUT_TEMPLATE =
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
You will have to also shift forward 3 hours, i.e.:
PB2NC_INPUT_TEMPLATE =
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
This says to add 3 hours to the current valid time before truncating
to the
nearest 6 hours.
Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
File: prepbufr.gdas.2015071500.nr
Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
File: prepbufr.gdas.2015071500.nr
Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
File: prepbufr.gdas.2015071500.nr
Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
File: prepbufr.gdas.2015071506.nr
Let me know if that doesn't work.
Thanks,
George
On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> Now makes me confused is why processed obs of several times are
missing.
> There has files of time at 00,01,02, but no 03,04,05; has 06,07,08,
but no
> 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with 21,no
22,23.
> From yours and mines:
>
> ls -l /glade/u/home/mccabe/data/out/pb2nc1/
>
>
> -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52 pbs.2015071500.nc
>
> -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56 pbs.2015071501.nc
>
> -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00 pbs.2015071502.nc
>
> -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14 pbs.2015071506.nc
>
> -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18 pbs.2015071507.nc
>
> -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22 pbs.2015071508.nc
>
> -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36 pbs.2015071512.nc
>
> -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40 pbs.2015071513.nc
>
> -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40 pbs.2015071514.nc
>
> -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42 pbs.2015071515.nc
>
>
> cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
>
> total 96112
>
> -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11 pbs.2015071500.nc
>
> -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14 pbs.2015071501.nc
>
> -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18 pbs.2015071502.nc
>
> -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33 pbs.2015071506.nc
>
> -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36 pbs.2015071507.nc
>
> -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40 pbs.2015071508.nc
>
> -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55 pbs.2015071512.nc
>
> -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59 pbs.2015071513.nc
>
> -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03 pbs.2015071514.nc
>
> -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18 pbs.2015071518.nc
>
> -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22 pbs.2015071519.nc
>
> -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27 pbs.2015071520.nc
>
> -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31 pbs.2015071521.nc
>
> -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43 pbs.2015071600.nc
>
>
>
> Thanks,
>
> Zhuming
>
> On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > Thanks for the information. You will need to use the truncate
keyword to
> > find the files, then set the valid beg/end range to 1 hour. This
is what
> I
> > changed from your PB2NC1.conf config file:
> >
> >
> > *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > {valid?fmt=%Y%m%d_%H?shift=1H}*
> >
> > *PB2NC_INPUT_TEMPLATE =
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> >
> > My config file is on cheyenne here:
> > /glade/u/home/mccabe/PB2NC1.conf
> >
> > I just started running it and the output is being written here:
> > /glade/u/home/mccabe/data/out/pb2nc1
> >
> >
> > For PointStat, you will want to loop over the same times. I
noticed you
> > specify message type in the VAR<n>_OPTIONS, but you do not need to
do
> this
> > since there is a POINT_STAT_MESSAGE_TYPE variable that defines
this. This
> > is what I changed in your PointStat1.conf file:
> >
> > *VALID_END = 2015071600*
> >
> >
> >
> > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
=POINT_STAT_OBS_VALID_BEG
> =*
> >
> >
> > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> >
> >
> >
> >
> >
> > *#OBS_VAR1_OPTIONS = message_type = ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
> > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > /glade/scratch/zhuming/metplus4/pb2nc1*
> >
> > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> >
> > Or you can view my file in the same directory. I usually try to
test
> with a
> > single run time to make sure it is working properly, so I set
VALID_END
> > back to 2015071500. I ran it and it produced an output file, but
there
> were
> > no values besides the header. I checked the log file and saw this:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for observation
type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
0
> > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> > processed = 393373DEBUG 3: Rejected: station id = 0DEBUG
3:
> > Rejected: obs type = 390528DEBUG 3: Rejected: valid time
=
> > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the
grid
> =
> > 2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
> mismatch
> > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
> type
> > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
> > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3: Rejected:
bad
> > climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG
2:
> >
> >
>
--------------------------------------------------------------------------------DEBUG
> > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
observation
> type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
0
> > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> > processed = 393373*
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected: obs type
> =
> > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3: Rejected: bad
obs
> > value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3:
Rejected:
> > topography = 0DEBUG 3: Rejected: level mismatch = 1168DEBUG
3:
> > Rejected: quality marker = 0DEBUG 3: Rejected: message type =
0DEBUG
> 3:
> > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst value =
0DEBUG
> 3:
> > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo stdev =
0DEBUG
> 3:
> > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> >
> >
>
--------------------------------------------------------------------------------DEBUG
> > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
observation
> type
> > ADPSFC, over region FULL, for interpolation method BILIN(4), using
0
> > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
Observations
> > processed = 393373DEBUG 3: Rejected: station id = 0DEBUG
3:
> > Rejected: obs type = 382965DEBUG 3: Rejected: valid time
=
> > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off the
grid
> =
> > 9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected: level
> mismatch
> > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
> > type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected: bad
> > fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected:
> bad
> > climo stdev = 0*
> >
> >
> >
> >
> > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> >
> >
>
--------------------------------------------------------------------------------DEBUG
> > 2:DEBUG 1: Output file:
> >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> >
> >
> >
> > This describes the reason why each observation was rejected. Some
> > observations didn't match the message type you requested, some
were not
> > within the forecast grid, and some did not match the vertical
level. You
> > will have to make adjustments to your settings so that the data
matches
> up
> > properly. Maybe the grid you are using in the PB2NC config file
does not
> > match the forecast grid and you are missing values. If that is the
case,
> > you will have to reconfigure and rerun PB2NC to get output on the
correct
> > gr5id. Maybe the observation points don't line up with the
vertical level
> > of the model fields. You can use a range of level values for the
> > observation data to allow data within that range, i.e. Z0-10.
> >
> > Let me know if you are still having issues getting this to work
and I can
> > help out.
> >
> > Thanks,
> > George
> >
> >
> > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > Thanks for your reply!
> > >
> > > I'm still confused, I need to output hourly obs data to do
hourly
> > > verification.
> > > The original GDAS Prepbufr data has 4 files one day, should with
6
> hours
> > > obs data in one file.
> > >
> > > After "PB2NC", I can see pbs.2015071500.nc (obs time is 00? or 6
times
> > of
> > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > >
> > > my config file :
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > >
> > > Thanks,
> > > Zhuming
> > >
> > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > To run the tools hourly you would set the VALID_INCREMENT
value for
> 1H.
> > > > Here is an example that runs once per hour for a day from
2015-07-15
> 0Z
> > > to
> > > > 2017-07-16 0Z:
> > > >
> > > > VALID_TIME_FMT = %Y%m%d%H
> > > >
> > > > VALID_BEG = 2015071500
> > > >
> > > > VALID_END = 2015071600
> > > >
> > > > VALID_INCREMENT = 1H
> > > >
> > > > LEAD_SEQ = 0
> > > >
> > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find files
whose
> > > > filenames match the time you need.
> > > >
> > > > To read prepBUFR observations for 12 hours, you will want to
set
> > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around the
> > > valid
> > > > time to process. These values depend on what is contained
inside your
> > > > prepBUFR files. If you want to use data from the valid time of
the
> > > filename
> > > > backwards 12 hours, you would set:
> > > >
> > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > >
> > > > For a runtime of 2015071512, this would be substituted as:
> > > >
> > > > valid_beg: 20150715_00
> > > > valid_end: 20150715_12
> > > >
> > > > If you want to create an output file for each hour in PB2NC
you would
> > > need
> > > > to make sure that the hour is specified in the
PB2NC_OUTPUT_TEMPLATE.
> > > >
> > > > PointStat has similar config variables to set the valid beg
and end
> > range
> > > > to use (POINT_STAT_OBS_VALID_BEG and
POINT_STAT_OBS_VALID_END).
> > > >
> > > > If your prepBUFR files are separated out into multiple files
that
> need
> > to
> > > > be read into PB2NC, you can set a file window around the valid
time
> to
> > > > allow files to be passed into the app. More information on
that
> > > > functionality is here:
> > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > >
> > > > If you are still having trouble getting this configured, you
could
> send
> > > me
> > > > the paths of the files you are trying to read and what data is
> > contained
> > > in
> > > > them so I can give you suggestions on what to set for the
> > configuration.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > >
> > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > > Queue: met_help
> > > > > Subject: How to set time for hourly 12-24 hours data ?
> > > > > Owner: mccabe
> > > > > Requestors: zhuming at ucar.edu
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >
> > > > >
> > > > >
> > > > > This transaction appears to have no content
> > > > >
> > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Thu Apr 08 09:45:31 2021
Hi George,
Processing Obs data using "PB2NC" are right now.
Now I'm using "PointStat" , I set VALID_BEG = 2015071500, VALID_END =
2015071501. It looks for the forecast data 2015071500/
wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
wrfout_d02_2015-07-15_01:00:00_cf.nc.
But my forecast data are all put in dir of 2015071500/.
So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get the right
fcst
data?
FCST_POINT_STAT_INPUT_TEMPLATE =
{valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:00_cf.nc
config file: /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
log file:
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
Thanks,
Zhuming
On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> Looking at the log output, it looks like the data is stored in the
prepBUFR
> files differently that you may have thought:
>
> DEBUG 2: PrepBufr Time Center: 20150715_000000
> DEBUG 2: Searching Time Window: 20150715_030000 to
20150715_040000
> 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85%
90% 95%
> 100%
>
> DEBUG 2: Total Messages processed = 590836
> DEBUG 2: Rejected based on message type = 0
> DEBUG 2: Rejected based on station id = 0
> DEBUG 2: Rejected based on valid time = 590815
> DEBUG 2: Rejected based on masking grid = 20
> DEBUG 2: Rejected based on masking polygon = 0
> DEBUG 2: Rejected based on elevation = 0
> DEBUG 2: Rejected based on pb report type = 1
> DEBUG 2: Rejected based on input report type = 0
> DEBUG 2: Rejected based on instrument type = 0
> DEBUG 2: Rejected based on zero observations = 0
> DEBUG 2: Total Messages retained = 0
> DEBUG 2: Total observations retained or derived = 0
> DEBUG 1: Obs time between *20150714_210000 and 20150715_030000*
> WARNING:
> WARNING: process_pbfile() -> No PrepBufr messages retained from
file:
> /glade/collections/rda/data/ds337.0/prepnr/2015/
> prepbufr.gdas.2015071500.nr
> WARNING:
> ERROR :
> ERROR : write_netcdf_hdr_data() -> No PrepBufr messages retained.
Nothing
> to write.
> ERROR :
>
> The prepbufr.gdas.2015071500.nr file actually contains data from
> 20150714_210000 to 20150715_030000, not 20150715_000000 to
20150715_060000.
>
> To get the correct file for each run, you will have to shift the
valid time
> before truncating it so that it gets the correct file for each run.
Instead
> of setting:
>
> PB2NC_INPUT_TEMPLATE =
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
>
> You will have to also shift forward 3 hours, i.e.:
>
> PB2NC_INPUT_TEMPLATE =
>
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
>
> This says to add 3 hours to the current valid time before truncating
to the
> nearest 6 hours.
>
> Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> File: prepbufr.gdas.2015071500.nr
>
> Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> File: prepbufr.gdas.2015071500.nr
>
> Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> File: prepbufr.gdas.2015071500.nr
>
> Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> File: prepbufr.gdas.2015071506.nr
>
> Let me know if that doesn't work.
>
> Thanks,
> George
>
>
> On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > Now makes me confused is why processed obs of several times are
missing.
> > There has files of time at 00,01,02, but no 03,04,05; has
06,07,08, but
> no
> > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with 21,no
22,23.
> > From yours and mines:
> >
> > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> >
> >
> > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52 pbs.2015071500.nc
> >
> > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56 pbs.2015071501.nc
> >
> > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00 pbs.2015071502.nc
> >
> > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14 pbs.2015071506.nc
> >
> > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18 pbs.2015071507.nc
> >
> > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22 pbs.2015071508.nc
> >
> > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36 pbs.2015071512.nc
> >
> > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40 pbs.2015071513.nc
> >
> > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40 pbs.2015071514.nc
> >
> > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42 pbs.2015071515.nc
> >
> >
> > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> >
> > total 96112
> >
> > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11 pbs.2015071500.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14 pbs.2015071501.nc
> >
> > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18 pbs.2015071502.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33 pbs.2015071506.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36 pbs.2015071507.nc
> >
> > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40 pbs.2015071508.nc
> >
> > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55 pbs.2015071512.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59 pbs.2015071513.nc
> >
> > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03 pbs.2015071514.nc
> >
> > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18 pbs.2015071518.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22 pbs.2015071519.nc
> >
> > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27 pbs.2015071520.nc
> >
> > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31 pbs.2015071521.nc
> >
> > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43 pbs.2015071600.nc
> >
> >
> >
> > Thanks,
> >
> > Zhuming
> >
> > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > Thanks for the information. You will need to use the truncate
keyword
> to
> > > find the files, then set the valid beg/end range to 1 hour. This
is
> what
> > I
> > > changed from your PB2NC1.conf config file:
> > >
> > >
> > > *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > >
> > > *PB2NC_INPUT_TEMPLATE =
> > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > >
> > > My config file is on cheyenne here:
> > > /glade/u/home/mccabe/PB2NC1.conf
> > >
> > > I just started running it and the output is being written here:
> > > /glade/u/home/mccabe/data/out/pb2nc1
> > >
> > >
> > > For PointStat, you will want to loop over the same times. I
noticed you
> > > specify message type in the VAR<n>_OPTIONS, but you do not need
to do
> > this
> > > since there is a POINT_STAT_MESSAGE_TYPE variable that defines
this.
> This
> > > is what I changed in your PointStat1.conf file:
> > >
> > > *VALID_END = 2015071600*
> > >
> > >
> > >
> > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
=POINT_STAT_OBS_VALID_BEG
> > =*
> > >
> > >
> > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > >
> > >
> > >
> > >
> > >
> > > *#OBS_VAR1_OPTIONS = message_type = ["ADPSFC"];#OBS_VAR2_OPTIONS
=
> > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
> > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > >
> > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > >
> > > Or you can view my file in the same directory. I usually try to
test
> > with a
> > > single run time to make sure it is working properly, so I set
VALID_END
> > > back to 2015071500. I ran it and it produced an output file, but
there
> > were
> > > no values besides the header. I checked the log file and saw
this:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for observation
type
> > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> Observations
> > > processed = 393373DEBUG 3: Rejected: station id = 0DEBUG
3:
> > > Rejected: obs type = 390528DEBUG 3: Rejected: valid time
=
> > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off
the grid
> > =
> > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
level
> > mismatch
> > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
> > type
> > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected:
bad fcst
> > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected: bad
> > > climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
observation
> > type
> > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> Observations
> > > processed = 393373*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected: obs
type
> > =
> > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3: Rejected:
bad obs
> > > value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3:
Rejected:
> > > topography = 0DEBUG 3: Rejected: level mismatch =
1168DEBUG 3:
> > > Rejected: quality marker = 0DEBUG 3: Rejected: message type
=
> 0DEBUG
> > 3:
> > > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst value
=
> 0DEBUG
> > 3:
> > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo stdev
=
> 0DEBUG
> > 3:
> > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
observation
> > type
> > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> Observations
> > > processed = 393373DEBUG 3: Rejected: station id = 0DEBUG
3:
> > > Rejected: obs type = 382965DEBUG 3: Rejected: valid time
=
> > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off
the grid
> > =
> > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
level
> > mismatch
> > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
message
> > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected: bad
> > > fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected:
> > bad
> > > climo stdev = 0*
> > >
> > >
> > >
> > >
> > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > 2:DEBUG 1: Output file:
> > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > >
> > >
> > >
> > > This describes the reason why each observation was rejected.
Some
> > > observations didn't match the message type you requested, some
were not
> > > within the forecast grid, and some did not match the vertical
level.
> You
> > > will have to make adjustments to your settings so that the data
matches
> > up
> > > properly. Maybe the grid you are using in the PB2NC config file
does
> not
> > > match the forecast grid and you are missing values. If that is
the
> case,
> > > you will have to reconfigure and rerun PB2NC to get output on
the
> correct
> > > gr5id. Maybe the observation points don't line up with the
vertical
> level
> > > of the model fields. You can use a range of level values for the
> > > observation data to allow data within that range, i.e. Z0-10.
> > >
> > > Let me know if you are still having issues getting this to work
and I
> can
> > > help out.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >
> > > > Hi George,
> > > >
> > > > Thanks for your reply!
> > > >
> > > > I'm still confused, I need to output hourly obs data to do
hourly
> > > > verification.
> > > > The original GDAS Prepbufr data has 4 files one day, should
with 6
> > hours
> > > > obs data in one file.
> > > >
> > > > After "PB2NC", I can see pbs.2015071500.nc (obs time is 00? or
6
> times
> > > of
> > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > >
> > > > my config file :
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > >
> > > > Thanks,
> > > > Zhuming
> > > >
> > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > Hi Zhuming,
> > > > >
> > > > > To run the tools hourly you would set the VALID_INCREMENT
value for
> > 1H.
> > > > > Here is an example that runs once per hour for a day from
> 2015-07-15
> > 0Z
> > > > to
> > > > > 2017-07-16 0Z:
> > > > >
> > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > >
> > > > > VALID_BEG = 2015071500
> > > > >
> > > > > VALID_END = 2015071600
> > > > >
> > > > > VALID_INCREMENT = 1H
> > > > >
> > > > > LEAD_SEQ = 0
> > > > >
> > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find
files whose
> > > > > filenames match the time you need.
> > > > >
> > > > > To read prepBUFR observations for 12 hours, you will want to
set
> > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around
> the
> > > > valid
> > > > > time to process. These values depend on what is contained
inside
> your
> > > > > prepBUFR files. If you want to use data from the valid time
of the
> > > > filename
> > > > > backwards 12 hours, you would set:
> > > > >
> > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > >
> > > > > For a runtime of 2015071512, this would be substituted as:
> > > > >
> > > > > valid_beg: 20150715_00
> > > > > valid_end: 20150715_12
> > > > >
> > > > > If you want to create an output file for each hour in PB2NC
you
> would
> > > > need
> > > > > to make sure that the hour is specified in the
> PB2NC_OUTPUT_TEMPLATE.
> > > > >
> > > > > PointStat has similar config variables to set the valid beg
and end
> > > range
> > > > > to use (POINT_STAT_OBS_VALID_BEG and
POINT_STAT_OBS_VALID_END).
> > > > >
> > > > > If your prepBUFR files are separated out into multiple files
that
> > need
> > > to
> > > > > be read into PB2NC, you can set a file window around the
valid time
> > to
> > > > > allow files to be passed into the app. More information on
that
> > > > > functionality is here:
> > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > > >
> > > > > If you are still having trouble getting this configured, you
could
> > send
> > > > me
> > > > > the paths of the files you are trying to read and what data
is
> > > contained
> > > > in
> > > > > them so I can give you suggestions on what to set for the
> > > configuration.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > > > Queue: met_help
> > > > > > Subject: How to set time for hourly 12-24 hours data
?
> > > > > > Owner: mccabe
> > > > > > Requestors: zhuming at ucar.edu
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > >
> > > > > >
> > > > > >
> > > > > > This transaction appears to have no content
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Thu Apr 08 10:04:44 2021
Hi Zhuming,
Typically directories that contain data for an entire day are named
with
YYYYMMDD, so your filename template tag would be {valid?fmt=%Y%m%d},
however that may not always be the case, as you are seeing here. There
are
a couple ways to handle this.
If you know that the directory will always be YYYYMMDD with 00 at the
end,
you can simply put 00 in the template instead of adding %H in the tag,
i.e.
FCST_POINT_STAT_INPUT_TEMPLATE =
{valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:00_cf.nc
If the HH value can vary, for instance if there are 2 directories
created
per day (i.e. 2015071400 and 2015071412), then you can use the
truncate
keyword to handle it.
If directories are created every 24 hours:
FCST_POINT_STAT_INPUT_TEMPLATE =
{valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
00_cf.nc
If directories are created every 12 hours:
FCST_POINT_STAT_INPUT_TEMPLATE =
{valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
00_cf.nc
Let me know if that doesn't work for you.
Thanks,
George
On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> Processing Obs data using "PB2NC" are right now.
>
> Now I'm using "PointStat" , I set VALID_BEG = 2015071500, VALID_END
=
> 2015071501. It looks for the forecast data 2015071500/
> wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> wrfout_d02_2015-07-15_01:00:00_cf.nc.
> But my forecast data are all put in dir of 2015071500/.
> So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get the right
fcst
> data?
>
> FCST_POINT_STAT_INPUT_TEMPLATE =
> {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:00_cf.nc
>
>
> config file: /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
>
> log file:
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
>
>
> Thanks,
>
> Zhuming
>
> On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > Looking at the log output, it looks like the data is stored in the
> prepBUFR
> > files differently that you may have thought:
> >
> > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > DEBUG 2: Searching Time Window: 20150715_030000 to
> 20150715_040000
> > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85%
90%
> 95%
> > 100%
> >
> > DEBUG 2: Total Messages processed = 590836
> > DEBUG 2: Rejected based on message type = 0
> > DEBUG 2: Rejected based on station id = 0
> > DEBUG 2: Rejected based on valid time = 590815
> > DEBUG 2: Rejected based on masking grid = 20
> > DEBUG 2: Rejected based on masking polygon = 0
> > DEBUG 2: Rejected based on elevation = 0
> > DEBUG 2: Rejected based on pb report type = 1
> > DEBUG 2: Rejected based on input report type = 0
> > DEBUG 2: Rejected based on instrument type = 0
> > DEBUG 2: Rejected based on zero observations = 0
> > DEBUG 2: Total Messages retained = 0
> > DEBUG 2: Total observations retained or derived = 0
> > DEBUG 1: Obs time between *20150714_210000 and 20150715_030000*
> > WARNING:
> > WARNING: process_pbfile() -> No PrepBufr messages retained from
file:
> > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > prepbufr.gdas.2015071500.nr
> > WARNING:
> > ERROR :
> > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages retained.
> Nothing
> > to write.
> > ERROR :
> >
> > The prepbufr.gdas.2015071500.nr file actually contains data from
> > 20150714_210000 to 20150715_030000, not 20150715_000000 to
> 20150715_060000.
> >
> > To get the correct file for each run, you will have to shift the
valid
> time
> > before truncating it so that it gets the correct file for each
run.
> Instead
> > of setting:
> >
> > PB2NC_INPUT_TEMPLATE =
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> >
> > You will have to also shift forward 3 hours, i.e.:
> >
> > PB2NC_INPUT_TEMPLATE =
> >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> >
> > This says to add 3 hours to the current valid time before
truncating to
> the
> > nearest 6 hours.
> >
> > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > File: prepbufr.gdas.2015071500.nr
> >
> > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > File: prepbufr.gdas.2015071500.nr
> >
> > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > File: prepbufr.gdas.2015071500.nr
> >
> > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > File: prepbufr.gdas.2015071506.nr
> >
> > Let me know if that doesn't work.
> >
> > Thanks,
> > George
> >
> >
> > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > Now makes me confused is why processed obs of several times are
> missing.
> > > There has files of time at 00,01,02, but no 03,04,05; has
06,07,08, but
> > no
> > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with
21,no
> 22,23.
> > > From yours and mines:
> > >
> > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > >
> > >
> > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52 pbs.2015071500.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56 pbs.2015071501.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00 pbs.2015071502.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14 pbs.2015071506.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18 pbs.2015071507.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22 pbs.2015071508.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36 pbs.2015071512.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40 pbs.2015071513.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40 pbs.2015071514.nc
> > >
> > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42 pbs.2015071515.nc
> > >
> > >
> > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > >
> > > total 96112
> > >
> > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
pbs.2015071500.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
pbs.2015071501.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
pbs.2015071502.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
pbs.2015071506.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
pbs.2015071507.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
pbs.2015071508.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
pbs.2015071512.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
pbs.2015071513.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
pbs.2015071514.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
pbs.2015071518.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
pbs.2015071519.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
pbs.2015071520.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
pbs.2015071521.nc
> > >
> > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
pbs.2015071600.nc
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Zhuming
> > >
> > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > Thanks for the information. You will need to use the truncate
keyword
> > to
> > > > find the files, then set the valid beg/end range to 1 hour.
This is
> > what
> > > I
> > > > changed from your PB2NC1.conf config file:
> > > >
> > > >
> > > > *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > >
> > > > *PB2NC_INPUT_TEMPLATE =
> > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > >
> > > > My config file is on cheyenne here:
> > > > /glade/u/home/mccabe/PB2NC1.conf
> > > >
> > > > I just started running it and the output is being written
here:
> > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > >
> > > >
> > > > For PointStat, you will want to loop over the same times. I
noticed
> you
> > > > specify message type in the VAR<n>_OPTIONS, but you do not
need to do
> > > this
> > > > since there is a POINT_STAT_MESSAGE_TYPE variable that defines
this.
> > This
> > > > is what I changed in your PointStat1.conf file:
> > > >
> > > > *VALID_END = 2015071600*
> > > >
> > > >
> > > >
> > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> =POINT_STAT_OBS_VALID_BEG
> > > =*
> > > >
> > > >
> > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *#OBS_VAR1_OPTIONS = message_type =
["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
> > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > >
> > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > >
> > > > Or you can view my file in the same directory. I usually try
to test
> > > with a
> > > > single run time to make sure it is working properly, so I set
> VALID_END
> > > > back to 2015071500. I ran it and it produced an output file,
but
> there
> > > were
> > > > no values besides the header. I checked the log file and saw
this:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
observation type
> > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > Observations
> > > > processed = 393373DEBUG 3: Rejected: station id =
0DEBUG 3:
> > > > Rejected: obs type = 390528DEBUG 3: Rejected: valid
time
> =
> > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off
the
> grid
> > > =
> > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
level
> > > mismatch
> > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3: Rejected:
> message
> > > type
> > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected:
bad
> fcst
> > > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
Rejected:
> bad
> > > > climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG 2:
> > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
> observation
> > > type
> > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > Observations
> > > > processed = 393373*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected: obs
type
> > > =
> > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3: Rejected:
bad
> obs
> > > > value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3:
> Rejected:
> > > > topography = 0DEBUG 3: Rejected: level mismatch =
1168DEBUG 3:
> > > > Rejected: quality marker = 0DEBUG 3: Rejected: message type
=
> > 0DEBUG
> > > 3:
> > > > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst value
=
> > 0DEBUG
> > > 3:
> > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo
stdev =
> > 0DEBUG
> > > 3:
> > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
> observation
> > > type
> > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
using 0
> > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > Observations
> > > > processed = 393373DEBUG 3: Rejected: station id =
0DEBUG 3:
> > > > Rejected: obs type = 382965DEBUG 3: Rejected: valid
time
> =
> > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected: off
the
> grid
> > > =
> > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
level
> > > mismatch
> > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> message
> > > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected:
> bad
> > > > fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
> Rejected:
> > > bad
> > > > climo stdev = 0*
> > > >
> > > >
> > > >
> > > >
> > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > 2:DEBUG 1: Output file:
> > > >
> > > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > >
> > > >
> > > >
> > > > This describes the reason why each observation was rejected.
Some
> > > > observations didn't match the message type you requested, some
were
> not
> > > > within the forecast grid, and some did not match the vertical
level.
> > You
> > > > will have to make adjustments to your settings so that the
data
> matches
> > > up
> > > > properly. Maybe the grid you are using in the PB2NC config
file does
> > not
> > > > match the forecast grid and you are missing values. If that is
the
> > case,
> > > > you will have to reconfigure and rerun PB2NC to get output on
the
> > correct
> > > > gr5id. Maybe the observation points don't line up with the
vertical
> > level
> > > > of the model fields. You can use a range of level values for
the
> > > > observation data to allow data within that range, i.e. Z0-10.
> > > >
> > > > Let me know if you are still having issues getting this to
work and I
> > can
> > > > help out.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >
> > > > > Hi George,
> > > > >
> > > > > Thanks for your reply!
> > > > >
> > > > > I'm still confused, I need to output hourly obs data to do
hourly
> > > > > verification.
> > > > > The original GDAS Prepbufr data has 4 files one day, should
with 6
> > > hours
> > > > > obs data in one file.
> > > > >
> > > > > After "PB2NC", I can see pbs.2015071500.nc (obs time is 00?
or 6
> > times
> > > > of
> > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > >
> > > > > my config file :
> cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > >
> > > > > Thanks,
> > > > > Zhuming
> > > > >
> > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Zhuming,
> > > > > >
> > > > > > To run the tools hourly you would set the VALID_INCREMENT
value
> for
> > > 1H.
> > > > > > Here is an example that runs once per hour for a day from
> > 2015-07-15
> > > 0Z
> > > > > to
> > > > > > 2017-07-16 0Z:
> > > > > >
> > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > >
> > > > > > VALID_BEG = 2015071500
> > > > > >
> > > > > > VALID_END = 2015071600
> > > > > >
> > > > > > VALID_INCREMENT = 1H
> > > > > >
> > > > > > LEAD_SEQ = 0
> > > > > >
> > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find
files
> whose
> > > > > > filenames match the time you need.
> > > > > >
> > > > > > To read prepBUFR observations for 12 hours, you will want
to set
> > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the range
around
> > the
> > > > > valid
> > > > > > time to process. These values depend on what is contained
inside
> > your
> > > > > > prepBUFR files. If you want to use data from the valid
time of
> the
> > > > > filename
> > > > > > backwards 12 hours, you would set:
> > > > > >
> > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > >
> > > > > > For a runtime of 2015071512, this would be substituted as:
> > > > > >
> > > > > > valid_beg: 20150715_00
> > > > > > valid_end: 20150715_12
> > > > > >
> > > > > > If you want to create an output file for each hour in
PB2NC you
> > would
> > > > > need
> > > > > > to make sure that the hour is specified in the
> > PB2NC_OUTPUT_TEMPLATE.
> > > > > >
> > > > > > PointStat has similar config variables to set the valid
beg and
> end
> > > > range
> > > > > > to use (POINT_STAT_OBS_VALID_BEG and
POINT_STAT_OBS_VALID_END).
> > > > > >
> > > > > > If your prepBUFR files are separated out into multiple
files that
> > > need
> > > > to
> > > > > > be read into PB2NC, you can set a file window around the
valid
> time
> > > to
> > > > > > allow files to be passed into the app. More information on
that
> > > > > > functionality is here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
windows-to-find-valid-files
> > > > > >
> > > > > > If you are still having trouble getting this configured,
you
> could
> > > send
> > > > > me
> > > > > > the paths of the files you are trying to read and what
data is
> > > > contained
> > > > > in
> > > > > > them so I can give you suggestions on what to set for the
> > > > configuration.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted upon.
> > > > > > > Transaction: Given to mccabe (George McCabe) by jpresto
> > > > > > > Queue: met_help
> > > > > > > Subject: How to set time for hourly 12-24 hours
data ?
> > > > > > > Owner: mccabe
> > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This transaction appears to have no content
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Apr 14 13:56:25 2021
Hi Zhuming,
Just following up on this ticket. Were you able to get this resolved?
Let me know if you need more assistance.
Thanks,
George
On Thu Apr 08 10:04:44 2021, mccabe wrote:
> Hi Zhuming,
>
> Typically directories that contain data for an entire day are named
> with
> YYYYMMDD, so your filename template tag would be
{valid?fmt=%Y%m%d},
> however that may not always be the case, as you are seeing here.
There
> are
> a couple ways to handle this.
>
> If you know that the directory will always be YYYYMMDD with 00 at
the
> end,
> you can simply put 00 in the template instead of adding %H in the
tag,
> i.e.
>
> FCST_POINT_STAT_INPUT_TEMPLATE =
> {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:00_cf.nc
>
> If the HH value can vary, for instance if there are 2 directories
> created
> per day (i.e. 2015071400 and 2015071412), then you can use the
> truncate
> keyword to handle it.
>
> If directories are created every 24 hours:
>
> FCST_POINT_STAT_INPUT_TEMPLATE =
> {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> %d_%H}:00:
> 00_cf.nc
>
> If directories are created every 12 hours:
>
> FCST_POINT_STAT_INPUT_TEMPLATE =
> {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> %d_%H}:00:
> 00_cf.nc
>
> Let me know if that doesn't work for you.
>
> Thanks,
> George
>
> On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > Processing Obs data using "PB2NC" are right now.
> >
> > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
VALID_END =
> > 2015071501. It looks for the forecast data 2015071500/
> > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > But my forecast data are all put in dir of 2015071500/.
> > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get the
right
> > fcst
> > data?
> >
> > FCST_POINT_STAT_INPUT_TEMPLATE =
> > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> >
> >
> > config file: /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> >
> > log file:
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> >
> >
> > Thanks,
> >
> > Zhuming
> >
> > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > Looking at the log output, it looks like the data is stored in
the
> > prepBUFR
> > > files differently that you may have thought:
> > >
> > > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > > DEBUG 2: Searching Time Window: 20150715_030000 to
> > 20150715_040000
> > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80%
85%
> > > 90%
> > 95%
> > > 100%
> > >
> > > DEBUG 2: Total Messages processed = 590836
> > > DEBUG 2: Rejected based on message type = 0
> > > DEBUG 2: Rejected based on station id = 0
> > > DEBUG 2: Rejected based on valid time = 590815
> > > DEBUG 2: Rejected based on masking grid = 20
> > > DEBUG 2: Rejected based on masking polygon = 0
> > > DEBUG 2: Rejected based on elevation = 0
> > > DEBUG 2: Rejected based on pb report type = 1
> > > DEBUG 2: Rejected based on input report type = 0
> > > DEBUG 2: Rejected based on instrument type = 0
> > > DEBUG 2: Rejected based on zero observations = 0
> > > DEBUG 2: Total Messages retained = 0
> > > DEBUG 2: Total observations retained or derived = 0
> > > DEBUG 1: Obs time between *20150714_210000 and 20150715_030000*
> > > WARNING:
> > > WARNING: process_pbfile() -> No PrepBufr messages retained from
> > > file:
> > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > prepbufr.gdas.2015071500.nr
> > > WARNING:
> > > ERROR :
> > > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages
retained.
> > Nothing
> > > to write.
> > > ERROR :
> > >
> > > The prepbufr.gdas.2015071500.nr file actually contains data from
> > > 20150714_210000 to 20150715_030000, not 20150715_000000 to
> > 20150715_060000.
> > >
> > > To get the correct file for each run, you will have to shift the
> > > valid
> > time
> > > before truncating it so that it gets the correct file for each
run.
> > Instead
> > > of setting:
> > >
> > > PB2NC_INPUT_TEMPLATE =
> > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > >
> > > You will have to also shift forward 3 hours, i.e.:
> > >
> > > PB2NC_INPUT_TEMPLATE =
> > >
> > >
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > >
> > > This says to add 3 hours to the current valid time before
> > > truncating to
> > the
> > > nearest 6 hours.
> > >
> > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > File: prepbufr.gdas.2015071500.nr
> > >
> > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > File: prepbufr.gdas.2015071500.nr
> > >
> > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > File: prepbufr.gdas.2015071500.nr
> > >
> > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > File: prepbufr.gdas.2015071506.nr
> > >
> > > Let me know if that doesn't work.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >
> > > > Hi George,
> > > >
> > > > Now makes me confused is why processed obs of several times
are
> > missing.
> > > > There has files of time at 00,01,02, but no 03,04,05; has
> > > > 06,07,08, but
> > > no
> > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with
> > > > 21,no
> > 22,23.
> > > > From yours and mines:
> > > >
> > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > >
> > > >
> > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
pbs.2015071500.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
pbs.2015071501.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
pbs.2015071502.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
pbs.2015071506.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
pbs.2015071507.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
pbs.2015071508.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
pbs.2015071512.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
pbs.2015071513.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
pbs.2015071514.nc
> > > >
> > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
pbs.2015071515.nc
> > > >
> > > >
> > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > >
> > > > total 96112
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
pbs.2015071500.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
pbs.2015071501.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
pbs.2015071502.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
pbs.2015071506.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
pbs.2015071507.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
pbs.2015071508.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
pbs.2015071512.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
pbs.2015071513.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
pbs.2015071514.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
pbs.2015071518.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
pbs.2015071519.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
pbs.2015071520.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
pbs.2015071521.nc
> > > >
> > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
pbs.2015071600.nc
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Zhuming
> > > >
> > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> > > > <met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > Hi Zhuming,
> > > > >
> > > > > Thanks for the information. You will need to use the
truncate
> > > > > keyword
> > > to
> > > > > find the files, then set the valid beg/end range to 1 hour.
> > > > > This is
> > > what
> > > > I
> > > > > changed from your PB2NC1.conf config file:
> > > > >
> > > > >
> > > > > *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > >
> > > > > *PB2NC_INPUT_TEMPLATE =
> > > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > >
> > > > > My config file is on cheyenne here:
> > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > >
> > > > > I just started running it and the output is being written
here:
> > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > >
> > > > >
> > > > > For PointStat, you will want to loop over the same times. I
> > > > > noticed
> > you
> > > > > specify message type in the VAR<n>_OPTIONS, but you do not
need
> > > > > to do
> > > > this
> > > > > since there is a POINT_STAT_MESSAGE_TYPE variable that
defines
> > > > > this.
> > > This
> > > > > is what I changed in your PointStat1.conf file:
> > > > >
> > > > > *VALID_END = 2015071600*
> > > > >
> > > > >
> > > > >
> > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > =POINT_STAT_OBS_VALID_BEG
> > > > =*
> > > > >
> > > > >
> > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type =
> > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > >
> > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > > >
> > > > > Or you can view my file in the same directory. I usually try
to
> > > > > test
> > > > with a
> > > > > single run time to make sure it is working properly, so I
set
> > VALID_END
> > > > > back to 2015071500. I ran it and it produced an output file,
> > > > > but
> > there
> > > > were
> > > > > no values besides the header. I checked the log file and saw
> > > > > this:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
> > > > > observation type
> > > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
> > > > > using 0
> > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > > Observations
> > > > > processed = 393373DEBUG 3: Rejected: station id =
> > > > > 0DEBUG 3:
> > > > > Rejected: obs type = 390528DEBUG 3: Rejected: valid
time
> > =
> > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected:
off
> > > > > the
> > grid
> > > > =
> > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
> > > > > level
> > > > mismatch
> > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > message
> > > > type
> > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3: Rejected:
bad
> > fcst
> > > > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
> > > > > Rejected:
> > bad
> > > > > climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG
> > > > > 2:DEBUG 2:
> > > > >
> > > > >
> > > >
> > >
> >
--------------------------------------------------------------------------------DEBUG
> > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
> > observation
> > > > type
> > > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
> > > > > using 0
> > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > > Observations
> > > > > processed = 393373*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected:
obs
> > > > > type
> > > > =
> > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3:
Rejected:
> > > > > bad
> > obs
> > > > > value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG 3:
> > Rejected:
> > > > > topography = 0DEBUG 3: Rejected: level mismatch =
> > > > > 1168DEBUG 3:
> > > > > Rejected: quality marker = 0DEBUG 3: Rejected: message type
> > > > > =
> > > 0DEBUG
> > > > 3:
> > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst
value
> > > > > =
> > > 0DEBUG
> > > > 3:
> > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo
stdev
> > > > > =
> > > 0DEBUG
> > > > 3:
> > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > >
> > > > >
> > > >
> > >
> >
--------------------------------------------------------------------------------DEBUG
> > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
> > observation
> > > > type
> > > > > ADPSFC, over region FULL, for interpolation method BILIN(4),
> > > > > using 0
> > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG 3:
> > > Observations
> > > > > processed = 393373DEBUG 3: Rejected: station id =
> > > > > 0DEBUG 3:
> > > > > Rejected: obs type = 382965DEBUG 3: Rejected: valid
time
> > =
> > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected:
off
> > > > > the
> > grid
> > > > =
> > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3: Rejected:
> > > > > level
> > > > mismatch
> > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > message
> > > > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
> > > > > Rejected:
> > bad
> > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG
3:
> > Rejected:
> > > > bad
> > > > > climo stdev = 0*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > >
> > > > >
> > > >
> > >
> >
--------------------------------------------------------------------------------DEBUG
> > > > > 2:DEBUG 1: Output file:
> > > > >
> > > > >
> > > >
> > >
> >
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > >
> > > > >
> > > > >
> > > > > This describes the reason why each observation was rejected.
> > > > > Some
> > > > > observations didn't match the message type you requested,
some
> > > > > were
> > not
> > > > > within the forecast grid, and some did not match the
vertical
> > > > > level.
> > > You
> > > > > will have to make adjustments to your settings so that the
data
> > matches
> > > > up
> > > > > properly. Maybe the grid you are using in the PB2NC config
file
> > > > > does
> > > not
> > > > > match the forecast grid and you are missing values. If that
is
> > > > > the
> > > case,
> > > > > you will have to reconfigure and rerun PB2NC to get output
on
> > > > > the
> > > correct
> > > > > gr5id. Maybe the observation points don't line up with the
> > > > > vertical
> > > level
> > > > > of the model fields. You can use a range of level values for
> > > > > the
> > > > > observation data to allow data within that range, i.e. Z0-
10.
> > > > >
> > > > > Let me know if you are still having issues getting this to
work
> > > > > and I
> > > can
> > > > > help out.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > >
> > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > >
> > > > > >
> > > > > > Hi George,
> > > > > >
> > > > > > Thanks for your reply!
> > > > > >
> > > > > > I'm still confused, I need to output hourly obs data to do
> > > > > > hourly
> > > > > > verification.
> > > > > > The original GDAS Prepbufr data has 4 files one day,
should
> > > > > > with 6
> > > > hours
> > > > > > obs data in one file.
> > > > > >
> > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs time is
00?
> > > > > > or 6
> > > times
> > > > > of
> > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > > >
> > > > > > my config file :
> > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > >
> > > > > > Thanks,
> > > > > > Zhuming
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Zhuming,
> > > > > > >
> > > > > > > To run the tools hourly you would set the
VALID_INCREMENT
> > > > > > > value
> > for
> > > > 1H.
> > > > > > > Here is an example that runs once per hour for a day
from
> > > 2015-07-15
> > > > 0Z
> > > > > > to
> > > > > > > 2017-07-16 0Z:
> > > > > > >
> > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > >
> > > > > > > VALID_BEG = 2015071500
> > > > > > >
> > > > > > > VALID_END = 2015071600
> > > > > > >
> > > > > > > VALID_INCREMENT = 1H
> > > > > > >
> > > > > > > LEAD_SEQ = 0
> > > > > > >
> > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to find
> > > > > > > files
> > whose
> > > > > > > filenames match the time you need.
> > > > > > >
> > > > > > > To read prepBUFR observations for 12 hours, you will
want
> > > > > > > to set
> > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the
range
> > > > > > > around
> > > the
> > > > > > valid
> > > > > > > time to process. These values depend on what is
contained
> > > > > > > inside
> > > your
> > > > > > > prepBUFR files. If you want to use data from the valid
time
> > > > > > > of
> > the
> > > > > > filename
> > > > > > > backwards 12 hours, you would set:
> > > > > > >
> > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > >
> > > > > > > For a runtime of 2015071512, this would be substituted
as:
> > > > > > >
> > > > > > > valid_beg: 20150715_00
> > > > > > > valid_end: 20150715_12
> > > > > > >
> > > > > > > If you want to create an output file for each hour in
PB2NC
> > > > > > > you
> > > would
> > > > > > need
> > > > > > > to make sure that the hour is specified in the
> > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > >
> > > > > > > PointStat has similar config variables to set the valid
beg
> > > > > > > and
> > end
> > > > > range
> > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > >
> > > > > > > If your prepBUFR files are separated out into multiple
> > > > > > > files that
> > > > need
> > > > > to
> > > > > > > be read into PB2NC, you can set a file window around the
> > > > > > > valid
> > time
> > > > to
> > > > > > > allow files to be passed into the app. More information
on
> > > > > > > that
> > > > > > > functionality is here:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > windows-to-find-valid-files
> > > > > > >
> > > > > > > If you are still having trouble getting this configured,
> > > > > > > you
> > could
> > > > send
> > > > > > me
> > > > > > > the paths of the files you are trying to read and what
data
> > > > > > > is
> > > > > contained
> > > > > > in
> > > > > > > them so I can give you suggestions on what to set for
the
> > > > > configuration.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted
upon.
> > > > > > > > Transaction: Given to mccabe (George McCabe) by
jpresto
> > > > > > > > Queue: met_help
> > > > > > > > Subject: How to set time for hourly 12-24 hours
data
> > > > > > > > ?
> > > > > > > > Owner: mccabe
> > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > This transaction appears to have no content
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not
> > > > > > > feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
> > > obliged to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Wed Apr 14 15:17:49 2021
Hi George,
Thanks for your email, I just would like to send you for help.
I already resolved the last problem with the WRF forecast, thank you
for
your help.
I also need to verify the MPAS model forecast. These 12 hour
MAPS forecasts are stored in a one file, such as
2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
2015071512/mpas_wrf_cf_2015071512.nc
It seems MET is doing fine on the first 2 hours, but no matched pairs
for
later hours.
I'm not sure what's wrong with it? the time is not matched?
see the log file
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
Thanks,
Zhuming
On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> Just following up on this ticket. Were you able to get this
resolved? Let
> me know if you need more assistance.
>
> Thanks,
> George
>
> On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > Hi Zhuming,
> >
> > Typically directories that contain data for an entire day are
named
> > with
> > YYYYMMDD, so your filename template tag would be
{valid?fmt=%Y%m%d},
> > however that may not always be the case, as you are seeing here.
There
> > are
> > a couple ways to handle this.
> >
> > If you know that the directory will always be YYYYMMDD with 00 at
the
> > end,
> > you can simply put 00 in the template instead of adding %H in the
tag,
> > i.e.
> >
> > FCST_POINT_STAT_INPUT_TEMPLATE =
> > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> >
> > If the HH value can vary, for instance if there are 2 directories
> > created
> > per day (i.e. 2015071400 and 2015071412), then you can use the
> > truncate
> > keyword to handle it.
> >
> > If directories are created every 24 hours:
> >
> > FCST_POINT_STAT_INPUT_TEMPLATE =
> > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > %d_%H}:00:
> > 00_cf.nc
> >
> > If directories are created every 12 hours:
> >
> > FCST_POINT_STAT_INPUT_TEMPLATE =
> > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > %d_%H}:00:
> > 00_cf.nc
> >
> > Let me know if that doesn't work for you.
> >
> > Thanks,
> > George
> >
> > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > Processing Obs data using "PB2NC" are right now.
> > >
> > > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
VALID_END =
> > > 2015071501. It looks for the forecast data 2015071500/
> > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > But my forecast data are all put in dir of 2015071500/.
> > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get the
right
> > > fcst
> > > data?
> > >
> > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> > >
> > >
> > > config file: /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > >
> > > log file:
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > >
> > >
> > > Thanks,
> > >
> > > Zhuming
> > >
> > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > Looking at the log output, it looks like the data is stored in
the
> > > prepBUFR
> > > > files differently that you may have thought:
> > > >
> > > > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > > > DEBUG 2: Searching Time Window: 20150715_030000 to
> > > 20150715_040000
> > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80%
85%
> > > > 90%
> > > 95%
> > > > 100%
> > > >
> > > > DEBUG 2: Total Messages processed = 590836
> > > > DEBUG 2: Rejected based on message type = 0
> > > > DEBUG 2: Rejected based on station id = 0
> > > > DEBUG 2: Rejected based on valid time = 590815
> > > > DEBUG 2: Rejected based on masking grid = 20
> > > > DEBUG 2: Rejected based on masking polygon = 0
> > > > DEBUG 2: Rejected based on elevation = 0
> > > > DEBUG 2: Rejected based on pb report type = 1
> > > > DEBUG 2: Rejected based on input report type = 0
> > > > DEBUG 2: Rejected based on instrument type = 0
> > > > DEBUG 2: Rejected based on zero observations = 0
> > > > DEBUG 2: Total Messages retained = 0
> > > > DEBUG 2: Total observations retained or derived = 0
> > > > DEBUG 1: Obs time between *20150714_210000 and
20150715_030000*
> > > > WARNING:
> > > > WARNING: process_pbfile() -> No PrepBufr messages retained
from
> > > > file:
> > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > prepbufr.gdas.2015071500.nr
> > > > WARNING:
> > > > ERROR :
> > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages
retained.
> > > Nothing
> > > > to write.
> > > > ERROR :
> > > >
> > > > The prepbufr.gdas.2015071500.nr file actually contains data
from
> > > > 20150714_210000 to 20150715_030000, not 20150715_000000 to
> > > 20150715_060000.
> > > >
> > > > To get the correct file for each run, you will have to shift
the
> > > > valid
> > > time
> > > > before truncating it so that it gets the correct file for each
run.
> > > Instead
> > > > of setting:
> > > >
> > > > PB2NC_INPUT_TEMPLATE =
> > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > >
> > > > You will have to also shift forward 3 hours, i.e.:
> > > >
> > > > PB2NC_INPUT_TEMPLATE =
> > > >
> > > >
> > >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > >
> > > > This says to add 3 hours to the current valid time before
> > > > truncating to
> > > the
> > > > nearest 6 hours.
> > > >
> > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > > File: prepbufr.gdas.2015071500.nr
> > > >
> > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > > File: prepbufr.gdas.2015071500.nr
> > > >
> > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > > File: prepbufr.gdas.2015071500.nr
> > > >
> > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > > File: prepbufr.gdas.2015071506.nr
> > > >
> > > > Let me know if that doesn't work.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >
> > > > > Hi George,
> > > > >
> > > > > Now makes me confused is why processed obs of several times
are
> > > missing.
> > > > > There has files of time at 00,01,02, but no 03,04,05; has
> > > > > 06,07,08, but
> > > > no
> > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but with
> > > > > 21,no
> > > 22,23.
> > > > > From yours and mines:
> > > > >
> > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > > >
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
pbs.2015071500.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
pbs.2015071501.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
pbs.2015071502.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
pbs.2015071506.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
pbs.2015071507.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
pbs.2015071508.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
pbs.2015071512.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
pbs.2015071513.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
pbs.2015071514.nc
> > > > >
> > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
pbs.2015071515.nc
> > > > >
> > > > >
> > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > >
> > > > > total 96112
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
pbs.2015071500.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
pbs.2015071501.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
pbs.2015071502.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
pbs.2015071506.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
pbs.2015071507.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
pbs.2015071508.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
pbs.2015071512.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
pbs.2015071513.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
pbs.2015071514.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
pbs.2015071518.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
pbs.2015071519.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
pbs.2015071520.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
pbs.2015071521.nc
> > > > >
> > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
pbs.2015071600.nc
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Zhuming
> > > > >
> > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> > > > > <met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Zhuming,
> > > > > >
> > > > > > Thanks for the information. You will need to use the
truncate
> > > > > > keyword
> > > > to
> > > > > > find the files, then set the valid beg/end range to 1
hour.
> > > > > > This is
> > > > what
> > > > > I
> > > > > > changed from your PB2NC1.conf config file:
> > > > > >
> > > > > >
> > > > > > *PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
=
> > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > >
> > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > >
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > >
> > > > > > My config file is on cheyenne here:
> > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > >
> > > > > > I just started running it and the output is being written
here:
> > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > >
> > > > > >
> > > > > > For PointStat, you will want to loop over the same times.
I
> > > > > > noticed
> > > you
> > > > > > specify message type in the VAR<n>_OPTIONS, but you do not
need
> > > > > > to do
> > > > > this
> > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable that
defines
> > > > > > this.
> > > > This
> > > > > > is what I changed in your PointStat1.conf file:
> > > > > >
> > > > > > *VALID_END = 2015071600*
> > > > > >
> > > > > >
> > > > > >
> > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > > =POINT_STAT_OBS_VALID_BEG
> > > > > =*
> > > > > >
> > > > > >
> > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS = message_type
=
> > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > >
> > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > > > >
> > > > > > Or you can view my file in the same directory. I usually
try to
> > > > > > test
> > > > > with a
> > > > > > single run time to make sure it is working properly, so I
set
> > > VALID_END
> > > > > > back to 2015071500. I ran it and it produced an output
file,
> > > > > > but
> > > there
> > > > > were
> > > > > > no values besides the header. I checked the log file and
saw
> > > > > > this:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
> > > > > > observation type
> > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > using 0
> > > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG
3:
> > > > Observations
> > > > > > processed = 393373DEBUG 3: Rejected: station id =
> > > > > > 0DEBUG 3:
> > > > > > Rejected: obs type = 390528DEBUG 3: Rejected: valid
time
> > > =
> > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected:
off
> > > > > > the
> > > grid
> > > > > =
> > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > level
> > > > > mismatch
> > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > message
> > > > > type
> > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected: bad
> > > fcst
> > > > > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
> > > > > > Rejected:
> > > bad
> > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates = 0DEBUG
> > > > > > 2:DEBUG 2:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10, for
> > > observation
> > > > > type
> > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > using 0
> > > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG
3:
> > > > Observations
> > > > > > processed = 393373*
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3: Rejected:
obs
> > > > > > type
> > > > > =
> > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3:
Rejected:
> > > > > > bad
> > > obs
> > > > > > value = 0DEBUG 3: Rejected: off the grid = 9240DEBUG
3:
> > > Rejected:
> > > > > > topography = 0DEBUG 3: Rejected: level mismatch =
> > > > > > 1168DEBUG 3:
> > > > > > Rejected: quality marker = 0DEBUG 3: Rejected: message
type
> > > > > > =
> > > > 0DEBUG
> > > > > 3:
> > > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst
value
> > > > > > =
> > > > 0DEBUG
> > > > > 3:
> > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad climo
stdev
> > > > > > =
> > > > 0DEBUG
> > > > > 3:
> > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10, for
> > > observation
> > > > > type
> > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > using 0
> > > > > > matched pairs.DEBUG 3: Number of matched pairs = 0DEBUG
3:
> > > > Observations
> > > > > > processed = 393373DEBUG 3: Rejected: station id =
> > > > > > 0DEBUG 3:
> > > > > > Rejected: obs type = 382965DEBUG 3: Rejected: valid
time
> > > =
> > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3: Rejected:
off
> > > > > > the
> > > grid
> > > > > =
> > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > level
> > > > > mismatch
> > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > message
> > > > > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
> > > > > > Rejected:
> > > bad
> > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG
3:
> > > Rejected:
> > > > > bad
> > > > > > climo stdev = 0*
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > 2:DEBUG 1: Output file:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > >
> > > > > >
> > > > > >
> > > > > > This describes the reason why each observation was
rejected.
> > > > > > Some
> > > > > > observations didn't match the message type you requested,
some
> > > > > > were
> > > not
> > > > > > within the forecast grid, and some did not match the
vertical
> > > > > > level.
> > > > You
> > > > > > will have to make adjustments to your settings so that the
data
> > > matches
> > > > > up
> > > > > > properly. Maybe the grid you are using in the PB2NC config
file
> > > > > > does
> > > > not
> > > > > > match the forecast grid and you are missing values. If
that is
> > > > > > the
> > > > case,
> > > > > > you will have to reconfigure and rerun PB2NC to get output
on
> > > > > > the
> > > > correct
> > > > > > gr5id. Maybe the observation points don't line up with the
> > > > > > vertical
> > > > level
> > > > > > of the model fields. You can use a range of level values
for
> > > > > > the
> > > > > > observation data to allow data within that range, i.e. Z0-
10.
> > > > > >
> > > > > > Let me know if you are still having issues getting this to
work
> > > > > > and I
> > > > can
> > > > > > help out.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > >
> > > > > > >
> > > > > > > Hi George,
> > > > > > >
> > > > > > > Thanks for your reply!
> > > > > > >
> > > > > > > I'm still confused, I need to output hourly obs data to
do
> > > > > > > hourly
> > > > > > > verification.
> > > > > > > The original GDAS Prepbufr data has 4 files one day,
should
> > > > > > > with 6
> > > > > hours
> > > > > > > obs data in one file.
> > > > > > >
> > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs time is
00?
> > > > > > > or 6
> > > > times
> > > > > > of
> > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > > > >
> > > > > > > my config file :
> > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Zhuming
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Zhuming,
> > > > > > > >
> > > > > > > > To run the tools hourly you would set the
VALID_INCREMENT
> > > > > > > > value
> > > for
> > > > > 1H.
> > > > > > > > Here is an example that runs once per hour for a day
from
> > > > 2015-07-15
> > > > > 0Z
> > > > > > > to
> > > > > > > > 2017-07-16 0Z:
> > > > > > > >
> > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > > >
> > > > > > > > VALID_BEG = 2015071500
> > > > > > > >
> > > > > > > > VALID_END = 2015071600
> > > > > > > >
> > > > > > > > VALID_INCREMENT = 1H
> > > > > > > >
> > > > > > > > LEAD_SEQ = 0
> > > > > > > >
> > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to
find
> > > > > > > > files
> > > whose
> > > > > > > > filenames match the time you need.
> > > > > > > >
> > > > > > > > To read prepBUFR observations for 12 hours, you will
want
> > > > > > > > to set
> > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the
range
> > > > > > > > around
> > > > the
> > > > > > > valid
> > > > > > > > time to process. These values depend on what is
contained
> > > > > > > > inside
> > > > your
> > > > > > > > prepBUFR files. If you want to use data from the valid
time
> > > > > > > > of
> > > the
> > > > > > > filename
> > > > > > > > backwards 12 hours, you would set:
> > > > > > > >
> > > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H }
> > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > > >
> > > > > > > > For a runtime of 2015071512, this would be substituted
as:
> > > > > > > >
> > > > > > > > valid_beg: 20150715_00
> > > > > > > > valid_end: 20150715_12
> > > > > > > >
> > > > > > > > If you want to create an output file for each hour in
PB2NC
> > > > > > > > you
> > > > would
> > > > > > > need
> > > > > > > > to make sure that the hour is specified in the
> > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > > >
> > > > > > > > PointStat has similar config variables to set the
valid beg
> > > > > > > > and
> > > end
> > > > > > range
> > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > > >
> > > > > > > > If your prepBUFR files are separated out into multiple
> > > > > > > > files that
> > > > > need
> > > > > > to
> > > > > > > > be read into PB2NC, you can set a file window around
the
> > > > > > > > valid
> > > time
> > > > > to
> > > > > > > > allow files to be passed into the app. More
information on
> > > > > > > > that
> > > > > > > > functionality is here:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > windows-to-find-valid-files
> > > > > > > >
> > > > > > > > If you are still having trouble getting this
configured,
> > > > > > > > you
> > > could
> > > > > send
> > > > > > > me
> > > > > > > > the paths of the files you are trying to read and what
data
> > > > > > > > is
> > > > > > contained
> > > > > > > in
> > > > > > > > them so I can give you suggestions on what to set for
the
> > > > > > configuration.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted
upon.
> > > > > > > > > Transaction: Given to mccabe (George McCabe) by
jpresto
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: How to set time for hourly 12-24 hours
data
> > > > > > > > > ?
> > > > > > > > > Owner: mccabe
> > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This transaction appears to have no content
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> > > > > > > > feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> > > > obliged to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Apr 14 15:41:29 2021
Hi Zhuming,
It looks like you are requesting the first time in the 12 hour file
for
each run:
FCST_VAR1_LEVELS = (0,*,*)
...
FCST_VAR2_LEVELS = (0,*,*)
The first dimension in these files is the time offset. You will need
the
correct index that corresponds to each valid time. Fortunately, MET
makes
this easy by allowing you to specify the valid time for a time
dimension
explicitly and it will find the correct data. You can set:
FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
...
FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
And it will substitute the valid time for each run. Let me know if you
have
any questions.
Thanks,
George
On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> Thanks for your email, I just would like to send you for help.
>
> I already resolved the last problem with the WRF forecast, thank you
for
> your help.
>
> I also need to verify the MPAS model forecast. These 12 hour
> MAPS forecasts are stored in a one file, such as
> 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> 2015071512/mpas_wrf_cf_2015071512.nc
>
> It seems MET is doing fine on the first 2 hours, but no matched
pairs for
> later hours.
> I'm not sure what's wrong with it? the time is not matched?
>
> see the log file
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
>
> Thanks,
> Zhuming
>
> On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > Just following up on this ticket. Were you able to get this
resolved? Let
> > me know if you need more assistance.
> >
> > Thanks,
> > George
> >
> > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > Hi Zhuming,
> > >
> > > Typically directories that contain data for an entire day are
named
> > > with
> > > YYYYMMDD, so your filename template tag would be
{valid?fmt=%Y%m%d},
> > > however that may not always be the case, as you are seeing here.
There
> > > are
> > > a couple ways to handle this.
> > >
> > > If you know that the directory will always be YYYYMMDD with 00
at the
> > > end,
> > > you can simply put 00 in the template instead of adding %H in
the tag,
> > > i.e.
> > >
> > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> > >
> > > If the HH value can vary, for instance if there are 2
directories
> > > created
> > > per day (i.e. 2015071400 and 2015071412), then you can use the
> > > truncate
> > > keyword to handle it.
> > >
> > > If directories are created every 24 hours:
> > >
> > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > %d_%H}:00:
> > > 00_cf.nc
> > >
> > > If directories are created every 12 hours:
> > >
> > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > %d_%H}:00:
> > > 00_cf.nc
> > >
> > > Let me know if that doesn't work for you.
> > >
> > > Thanks,
> > > George
> > >
> > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >
> > > > Hi George,
> > > >
> > > > Processing Obs data using "PB2NC" are right now.
> > > >
> > > > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
VALID_END =
> > > > 2015071501. It looks for the forecast data 2015071500/
> > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > > But my forecast data are all put in dir of 2015071500/.
> > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get the
right
> > > > fcst
> > > > data?
> > > >
> > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> > > >
> > > >
> > > > config file: /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > >
> > > > log file:
> > > >
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Zhuming
> > > >
> > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Zhuming,
> > > > >
> > > > > Looking at the log output, it looks like the data is stored
in the
> > > > prepBUFR
> > > > > files differently that you may have thought:
> > > > >
> > > > > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > > > > DEBUG 2: Searching Time Window: 20150715_030000 to
> > > > 20150715_040000
> > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75%
80% 85%
> > > > > 90%
> > > > 95%
> > > > > 100%
> > > > >
> > > > > DEBUG 2: Total Messages processed = 590836
> > > > > DEBUG 2: Rejected based on message type = 0
> > > > > DEBUG 2: Rejected based on station id = 0
> > > > > DEBUG 2: Rejected based on valid time = 590815
> > > > > DEBUG 2: Rejected based on masking grid = 20
> > > > > DEBUG 2: Rejected based on masking polygon = 0
> > > > > DEBUG 2: Rejected based on elevation = 0
> > > > > DEBUG 2: Rejected based on pb report type = 1
> > > > > DEBUG 2: Rejected based on input report type = 0
> > > > > DEBUG 2: Rejected based on instrument type = 0
> > > > > DEBUG 2: Rejected based on zero observations = 0
> > > > > DEBUG 2: Total Messages retained = 0
> > > > > DEBUG 2: Total observations retained or derived = 0
> > > > > DEBUG 1: Obs time between *20150714_210000 and
20150715_030000*
> > > > > WARNING:
> > > > > WARNING: process_pbfile() -> No PrepBufr messages retained
from
> > > > > file:
> > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > > prepbufr.gdas.2015071500.nr
> > > > > WARNING:
> > > > > ERROR :
> > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages
retained.
> > > > Nothing
> > > > > to write.
> > > > > ERROR :
> > > > >
> > > > > The prepbufr.gdas.2015071500.nr file actually contains data
from
> > > > > 20150714_210000 to 20150715_030000, not 20150715_000000 to
> > > > 20150715_060000.
> > > > >
> > > > > To get the correct file for each run, you will have to shift
the
> > > > > valid
> > > > time
> > > > > before truncating it so that it gets the correct file for
each run.
> > > > Instead
> > > > > of setting:
> > > > >
> > > > > PB2NC_INPUT_TEMPLATE =
> > > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > > >
> > > > > You will have to also shift forward 3 hours, i.e.:
> > > > >
> > > > > PB2NC_INPUT_TEMPLATE =
> > > > >
> > > > >
> > > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > > >
> > > > > This says to add 3 hours to the current valid time before
> > > > > truncating to
> > > > the
> > > > > nearest 6 hours.
> > > > >
> > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > > > File: prepbufr.gdas.2015071500.nr
> > > > >
> > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > > > File: prepbufr.gdas.2015071500.nr
> > > > >
> > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > > > File: prepbufr.gdas.2015071500.nr
> > > > >
> > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > > > File: prepbufr.gdas.2015071506.nr
> > > > >
> > > > > Let me know if that doesn't work.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > >
> > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > > >
> > > > > > Hi George,
> > > > > >
> > > > > > Now makes me confused is why processed obs of several
times are
> > > > missing.
> > > > > > There has files of time at 00,01,02, but no 03,04,05; has
> > > > > > 06,07,08, but
> > > > > no
> > > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but
with
> > > > > > 21,no
> > > > 22,23.
> > > > > > From yours and mines:
> > > > > >
> > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > > > >
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
pbs.2015071500.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
pbs.2015071501.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
pbs.2015071502.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
pbs.2015071506.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
pbs.2015071507.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
pbs.2015071508.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
pbs.2015071512.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
pbs.2015071513.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
pbs.2015071514.nc
> > > > > >
> > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
pbs.2015071515.nc
> > > > > >
> > > > > >
> > > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > > >
> > > > > > total 96112
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> pbs.2015071500.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> pbs.2015071501.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> pbs.2015071502.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> pbs.2015071506.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> pbs.2015071507.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> pbs.2015071508.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> pbs.2015071512.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> pbs.2015071513.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> pbs.2015071514.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> pbs.2015071518.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> pbs.2015071519.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> pbs.2015071520.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> pbs.2015071521.nc
> > > > > >
> > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> pbs.2015071600.nc
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Zhuming
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> > > > > > <met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Zhuming,
> > > > > > >
> > > > > > > Thanks for the information. You will need to use the
truncate
> > > > > > > keyword
> > > > > to
> > > > > > > find the files, then set the valid beg/end range to 1
hour.
> > > > > > > This is
> > > > > what
> > > > > > I
> > > > > > > changed from your PB2NC1.conf config file:
> > > > > > >
> > > > > > >
> > > > > > > *PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > > >
> > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > > >
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > > >
> > > > > > > My config file is on cheyenne here:
> > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > > >
> > > > > > > I just started running it and the output is being
written here:
> > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > > >
> > > > > > >
> > > > > > > For PointStat, you will want to loop over the same
times. I
> > > > > > > noticed
> > > > you
> > > > > > > specify message type in the VAR<n>_OPTIONS, but you do
not need
> > > > > > > to do
> > > > > > this
> > > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable that
defines
> > > > > > > this.
> > > > > This
> > > > > > > is what I changed in your PointStat1.conf file:
> > > > > > >
> > > > > > > *VALID_END = 2015071600*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > > > =POINT_STAT_OBS_VALID_BEG
> > > > > > =*
> > > > > > >
> > > > > > >
> > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
message_type =
> > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > > >
> > > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > > > > >
> > > > > > > Or you can view my file in the same directory. I usually
try to
> > > > > > > test
> > > > > > with a
> > > > > > > single run time to make sure it is working properly, so
I set
> > > > VALID_END
> > > > > > > back to 2015071500. I ran it and it produced an output
file,
> > > > > > > but
> > > > there
> > > > > > were
> > > > > > > no values besides the header. I checked the log file and
saw
> > > > > > > this:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
> > > > > > > observation type
> > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > using 0
> > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > Observations
> > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > 0DEBUG 3:
> > > > > > > Rejected: obs type = 390528DEBUG 3: Rejected:
valid time
> > > > =
> > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected: off
> > > > > > > the
> > > > grid
> > > > > > =
> > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > > level
> > > > > > mismatch
> > > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > > message
> > > > > > type
> > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected: bad
> > > > fcst
> > > > > > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG 3:
> > > > > > > Rejected:
> > > > bad
> > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates =
0DEBUG
> > > > > > > 2:DEBUG 2:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10,
for
> > > > observation
> > > > > > type
> > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > using 0
> > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > Observations
> > > > > > > processed = 393373*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected: obs
> > > > > > > type
> > > > > > =
> > > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3:
Rejected:
> > > > > > > bad
> > > > obs
> > > > > > > value = 0DEBUG 3: Rejected: off the grid =
9240DEBUG 3:
> > > > Rejected:
> > > > > > > topography = 0DEBUG 3: Rejected: level mismatch =
> > > > > > > 1168DEBUG 3:
> > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected: message
type
> > > > > > > =
> > > > > 0DEBUG
> > > > > > 3:
> > > > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad fcst
value
> > > > > > > =
> > > > > 0DEBUG
> > > > > > 3:
> > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad
climo stdev
> > > > > > > =
> > > > > 0DEBUG
> > > > > > 3:
> > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10,
for
> > > > observation
> > > > > > type
> > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > using 0
> > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > Observations
> > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > 0DEBUG 3:
> > > > > > > Rejected: obs type = 382965DEBUG 3: Rejected:
valid time
> > > > =
> > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected: off
> > > > > > > the
> > > > grid
> > > > > > =
> > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > > level
> > > > > > mismatch
> > > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > > message
> > > > > > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG
3:
> > > > > > > Rejected:
> > > > bad
> > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG 3:
> > > > Rejected:
> > > > > > bad
> > > > > > > climo stdev = 0*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > 2:DEBUG 1: Output file:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This describes the reason why each observation was
rejected.
> > > > > > > Some
> > > > > > > observations didn't match the message type you
requested, some
> > > > > > > were
> > > > not
> > > > > > > within the forecast grid, and some did not match the
vertical
> > > > > > > level.
> > > > > You
> > > > > > > will have to make adjustments to your settings so that
the data
> > > > matches
> > > > > > up
> > > > > > > properly. Maybe the grid you are using in the PB2NC
config file
> > > > > > > does
> > > > > not
> > > > > > > match the forecast grid and you are missing values. If
that is
> > > > > > > the
> > > > > case,
> > > > > > > you will have to reconfigure and rerun PB2NC to get
output on
> > > > > > > the
> > > > > correct
> > > > > > > gr5id. Maybe the observation points don't line up with
the
> > > > > > > vertical
> > > > > level
> > > > > > > of the model fields. You can use a range of level values
for
> > > > > > > the
> > > > > > > observation data to allow data within that range, i.e.
Z0-10.
> > > > > > >
> > > > > > > Let me know if you are still having issues getting this
to work
> > > > > > > and I
> > > > > can
> > > > > > > help out.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi George,
> > > > > > > >
> > > > > > > > Thanks for your reply!
> > > > > > > >
> > > > > > > > I'm still confused, I need to output hourly obs data
to do
> > > > > > > > hourly
> > > > > > > > verification.
> > > > > > > > The original GDAS Prepbufr data has 4 files one day,
should
> > > > > > > > with 6
> > > > > > hours
> > > > > > > > obs data in one file.
> > > > > > > >
> > > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs time
is 00?
> > > > > > > > or 6
> > > > > times
> > > > > > > of
> > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > > > > >
> > > > > > > > my config file :
> > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Zhuming
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Zhuming,
> > > > > > > > >
> > > > > > > > > To run the tools hourly you would set the
VALID_INCREMENT
> > > > > > > > > value
> > > > for
> > > > > > 1H.
> > > > > > > > > Here is an example that runs once per hour for a day
from
> > > > > 2015-07-15
> > > > > > 0Z
> > > > > > > > to
> > > > > > > > > 2017-07-16 0Z:
> > > > > > > > >
> > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > > > >
> > > > > > > > > VALID_BEG = 2015071500
> > > > > > > > >
> > > > > > > > > VALID_END = 2015071600
> > > > > > > > >
> > > > > > > > > VALID_INCREMENT = 1H
> > > > > > > > >
> > > > > > > > > LEAD_SEQ = 0
> > > > > > > > >
> > > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to
find
> > > > > > > > > files
> > > > whose
> > > > > > > > > filenames match the time you need.
> > > > > > > > >
> > > > > > > > > To read prepBUFR observations for 12 hours, you will
want
> > > > > > > > > to set
> > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify the
range
> > > > > > > > > around
> > > > > the
> > > > > > > > valid
> > > > > > > > > time to process. These values depend on what is
contained
> > > > > > > > > inside
> > > > > your
> > > > > > > > > prepBUFR files. If you want to use data from the
valid time
> > > > > > > > > of
> > > > the
> > > > > > > > filename
> > > > > > > > > backwards 12 hours, you would set:
> > > > > > > > >
> > > > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-12H
}
> > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > > > >
> > > > > > > > > For a runtime of 2015071512, this would be
substituted as:
> > > > > > > > >
> > > > > > > > > valid_beg: 20150715_00
> > > > > > > > > valid_end: 20150715_12
> > > > > > > > >
> > > > > > > > > If you want to create an output file for each hour
in PB2NC
> > > > > > > > > you
> > > > > would
> > > > > > > > need
> > > > > > > > > to make sure that the hour is specified in the
> > > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > > > >
> > > > > > > > > PointStat has similar config variables to set the
valid beg
> > > > > > > > > and
> > > > end
> > > > > > > range
> > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > > > >
> > > > > > > > > If your prepBUFR files are separated out into
multiple
> > > > > > > > > files that
> > > > > > need
> > > > > > > to
> > > > > > > > > be read into PB2NC, you can set a file window around
the
> > > > > > > > > valid
> > > > time
> > > > > > to
> > > > > > > > > allow files to be passed into the app. More
information on
> > > > > > > > > that
> > > > > > > > > functionality is here:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > > windows-to-find-valid-files
> > > > > > > > >
> > > > > > > > > If you are still having trouble getting this
configured,
> > > > > > > > > you
> > > > could
> > > > > > send
> > > > > > > > me
> > > > > > > > > the paths of the files you are trying to read and
what data
> > > > > > > > > is
> > > > > > > contained
> > > > > > > > in
> > > > > > > > > them so I can give you suggestions on what to set
for the
> > > > > > > configuration.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was acted
upon.
> > > > > > > > > > Transaction: Given to mccabe (George McCabe) by
jpresto
> > > > > > > > > > Queue: met_help
> > > > > > > > > > Subject: How to set time for hourly 12-24
hours data
> > > > > > > > > > ?
> > > > > > > > > > Owner: mccabe
> > > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > > > Status: new
> > > > > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This transaction appears to have no content
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > > > > > > > > feel
> > > > > > obliged
> > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > > > > obliged to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> >
> >
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Wed Apr 14 16:13:51 2021
Hi George,
It still can't find the correct time. I tried
FCST_VAR1_LEVELS = (*,*,*)
FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
Thanks!
Zhuming
On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> It looks like you are requesting the first time in the 12 hour file
for
> each run:
>
> FCST_VAR1_LEVELS = (0,*,*)
> ...
> FCST_VAR2_LEVELS = (0,*,*)
>
> The first dimension in these files is the time offset. You will need
the
> correct index that corresponds to each valid time. Fortunately, MET
makes
> this easy by allowing you to specify the valid time for a time
dimension
> explicitly and it will find the correct data. You can set:
>
> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> ...
> FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
>
> And it will substitute the valid time for each run. Let me know if
you have
> any questions.
>
> Thanks,
> George
>
> On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > Thanks for your email, I just would like to send you for help.
> >
> > I already resolved the last problem with the WRF forecast, thank
you for
> > your help.
> >
> > I also need to verify the MPAS model forecast. These 12 hour
> > MAPS forecasts are stored in a one file, such as
> > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> > 2015071512/mpas_wrf_cf_2015071512.nc
> >
> > It seems MET is doing fine on the first 2 hours, but no matched
pairs for
> > later hours.
> > I'm not sure what's wrong with it? the time is not matched?
> >
> > see the log file
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> >
> > Thanks,
> > Zhuming
> >
> > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > Just following up on this ticket. Were you able to get this
resolved?
> Let
> > > me know if you need more assistance.
> > >
> > > Thanks,
> > > George
> > >
> > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > > Hi Zhuming,
> > > >
> > > > Typically directories that contain data for an entire day are
named
> > > > with
> > > > YYYYMMDD, so your filename template tag would be
{valid?fmt=%Y%m%d},
> > > > however that may not always be the case, as you are seeing
here.
> There
> > > > are
> > > > a couple ways to handle this.
> > > >
> > > > If you know that the directory will always be YYYYMMDD with 00
at the
> > > > end,
> > > > you can simply put 00 in the template instead of adding %H in
the
> tag,
> > > > i.e.
> > > >
> > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:00_cf.nc
> > > >
> > > > If the HH value can vary, for instance if there are 2
directories
> > > > created
> > > > per day (i.e. 2015071400 and 2015071412), then you can use
the
> > > > truncate
> > > > keyword to handle it.
> > > >
> > > > If directories are created every 24 hours:
> > > >
> > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > > %d_%H}:00:
> > > > 00_cf.nc
> > > >
> > > > If directories are created every 12 hours:
> > > >
> > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > > %d_%H}:00:
> > > > 00_cf.nc
> > > >
> > > > Let me know if that doesn't work for you.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > >
> > > > > Hi George,
> > > > >
> > > > > Processing Obs data using "PB2NC" are right now.
> > > > >
> > > > > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
> VALID_END =
> > > > > 2015071501. It looks for the forecast data 2015071500/
> > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > > > But my forecast data are all put in dir of 2015071500/.
> > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get
the right
> > > > > fcst
> > > > > data?
> > > > >
> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> 00_cf.nc
> > > > >
> > > > >
> > > > > config file:
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > >
> > > > > log file:
> > > > >
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Zhuming
> > > > >
> > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Zhuming,
> > > > > >
> > > > > > Looking at the log output, it looks like the data is
stored in
> the
> > > > > prepBUFR
> > > > > > files differently that you may have thought:
> > > > > >
> > > > > > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > > > > > DEBUG 2: Searching Time Window: 20150715_030000 to
> > > > > 20150715_040000
> > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75%
80%
> 85%
> > > > > > 90%
> > > > > 95%
> > > > > > 100%
> > > > > >
> > > > > > DEBUG 2: Total Messages processed = 590836
> > > > > > DEBUG 2: Rejected based on message type = 0
> > > > > > DEBUG 2: Rejected based on station id = 0
> > > > > > DEBUG 2: Rejected based on valid time = 590815
> > > > > > DEBUG 2: Rejected based on masking grid = 20
> > > > > > DEBUG 2: Rejected based on masking polygon = 0
> > > > > > DEBUG 2: Rejected based on elevation = 0
> > > > > > DEBUG 2: Rejected based on pb report type = 1
> > > > > > DEBUG 2: Rejected based on input report type = 0
> > > > > > DEBUG 2: Rejected based on instrument type = 0
> > > > > > DEBUG 2: Rejected based on zero observations = 0
> > > > > > DEBUG 2: Total Messages retained = 0
> > > > > > DEBUG 2: Total observations retained or derived = 0
> > > > > > DEBUG 1: Obs time between *20150714_210000 and
20150715_030000*
> > > > > > WARNING:
> > > > > > WARNING: process_pbfile() -> No PrepBufr messages retained
from
> > > > > > file:
> > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > > > prepbufr.gdas.2015071500.nr
> > > > > > WARNING:
> > > > > > ERROR :
> > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages
> retained.
> > > > > Nothing
> > > > > > to write.
> > > > > > ERROR :
> > > > > >
> > > > > > The prepbufr.gdas.2015071500.nr file actually contains
data from
> > > > > > 20150714_210000 to 20150715_030000, not 20150715_000000 to
> > > > > 20150715_060000.
> > > > > >
> > > > > > To get the correct file for each run, you will have to
shift the
> > > > > > valid
> > > > > time
> > > > > > before truncating it so that it gets the correct file for
each
> run.
> > > > > Instead
> > > > > > of setting:
> > > > > >
> > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > >
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > > > >
> > > > > > You will have to also shift forward 3 hours, i.e.:
> > > > > >
> > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > >
> > > > > >
> > > > >
> > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > > > >
> > > > > > This says to add 3 hours to the current valid time before
> > > > > > truncating to
> > > > > the
> > > > > > nearest 6 hours.
> > > > > >
> > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > >
> > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > >
> > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > >
> > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > > > > File: prepbufr.gdas.2015071506.nr
> > > > > >
> > > > > > Let me know if that doesn't work.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > >
> > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >
> > > > > > >
> > > > > > > Hi George,
> > > > > > >
> > > > > > > Now makes me confused is why processed obs of several
times are
> > > > > missing.
> > > > > > > There has files of time at 00,01,02, but no 03,04,05;
has
> > > > > > > 06,07,08, but
> > > > > > no
> > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20, but
with
> > > > > > > 21,no
> > > > > 22,23.
> > > > > > > From yours and mines:
> > > > > > >
> > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > > > > >
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> pbs.2015071500.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> pbs.2015071501.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> pbs.2015071502.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> pbs.2015071506.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> pbs.2015071507.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> pbs.2015071508.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> pbs.2015071512.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> pbs.2015071513.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> pbs.2015071514.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> pbs.2015071515.nc
> > > > > > >
> > > > > > >
> > > > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > > > >
> > > > > > > total 96112
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> > pbs.2015071500.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> > pbs.2015071501.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> > pbs.2015071502.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> > pbs.2015071506.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> > pbs.2015071507.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> > pbs.2015071508.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> > pbs.2015071512.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> > pbs.2015071513.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> > pbs.2015071514.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> > pbs.2015071518.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> > pbs.2015071519.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> > pbs.2015071520.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> > pbs.2015071521.nc
> > > > > > >
> > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> > pbs.2015071600.nc
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Zhuming
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> > > > > > > <met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Zhuming,
> > > > > > > >
> > > > > > > > Thanks for the information. You will need to use the
truncate
> > > > > > > > keyword
> > > > > > to
> > > > > > > > find the files, then set the valid beg/end range to 1
hour.
> > > > > > > > This is
> > > > > > what
> > > > > > > I
> > > > > > > > changed from your PB2NC1.conf config file:
> > > > > > > >
> > > > > > > >
> > > > > > > > *PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > > > >
> > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > > > >
> > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > > > >
> > > > > > > > My config file is on cheyenne here:
> > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > > > >
> > > > > > > > I just started running it and the output is being
written
> here:
> > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > > > >
> > > > > > > >
> > > > > > > > For PointStat, you will want to loop over the same
times. I
> > > > > > > > noticed
> > > > > you
> > > > > > > > specify message type in the VAR<n>_OPTIONS, but you do
not
> need
> > > > > > > > to do
> > > > > > > this
> > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable that
> defines
> > > > > > > > this.
> > > > > > This
> > > > > > > > is what I changed in your PointStat1.conf file:
> > > > > > > >
> > > > > > > > *VALID_END = 2015071600*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > > > > =POINT_STAT_OBS_VALID_BEG
> > > > > > > =*
> > > > > > > >
> > > > > > > >
> > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
message_type =
> > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > > > >
> > > > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > > > > > >
> > > > > > > > Or you can view my file in the same directory. I
usually try
> to
> > > > > > > > test
> > > > > > > with a
> > > > > > > > single run time to make sure it is working properly,
so I set
> > > > > VALID_END
> > > > > > > > back to 2015071500. I ran it and it produced an output
file,
> > > > > > > > but
> > > > > there
> > > > > > > were
> > > > > > > > no values besides the header. I checked the log file
and saw
> > > > > > > > this:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
> > > > > > > > observation type
> > > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > > using 0
> > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > > Observations
> > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > > 0DEBUG 3:
> > > > > > > > Rejected: obs type = 390528DEBUG 3: Rejected:
valid
> time
> > > > > =
> > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected: off
> > > > > > > > the
> > > > > grid
> > > > > > > =
> > > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > > > level
> > > > > > > mismatch
> > > > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
Rejected:
> > > > > message
> > > > > > > type
> > > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected:
> bad
> > > > > fcst
> > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean = 0DEBUG
3:
> > > > > > > > Rejected:
> > > > > bad
> > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates =
0DEBUG
> > > > > > > > 2:DEBUG 2:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus UGRD/Z10,
for
> > > > > observation
> > > > > > > type
> > > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > > using 0
> > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > > Observations
> > > > > > > > processed = 393373*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected: obs
> > > > > > > > type
> > > > > > > =
> > > > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3:
> Rejected:
> > > > > > > > bad
> > > > > obs
> > > > > > > > value = 0DEBUG 3: Rejected: off the grid =
9240DEBUG 3:
> > > > > Rejected:
> > > > > > > > topography = 0DEBUG 3: Rejected: level mismatch
=
> > > > > > > > 1168DEBUG 3:
> > > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected:
message type
> > > > > > > > =
> > > > > > 0DEBUG
> > > > > > > 3:
> > > > > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
> value
> > > > > > > > =
> > > > > > 0DEBUG
> > > > > > > 3:
> > > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad
climo
> stdev
> > > > > > > > =
> > > > > > 0DEBUG
> > > > > > > 3:
> > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus VGRD/Z10,
for
> > > > > observation
> > > > > > > type
> > > > > > > > ADPSFC, over region FULL, for interpolation method
BILIN(4),
> > > > > > > > using 0
> > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG 3:
> > > > > > Observations
> > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > > 0DEBUG 3:
> > > > > > > > Rejected: obs type = 382965DEBUG 3: Rejected:
valid
> time
> > > > > =
> > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected: off
> > > > > > > > the
> > > > > grid
> > > > > > > =
> > > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3:
Rejected:
> > > > > > > > level
> > > > > > > mismatch
> > > > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
> Rejected:
> > > > > message
> > > > > > > > type = 0DEBUG 3: Rejected: masking region = 0DEBUG
3:
> > > > > > > > Rejected:
> > > > > bad
> > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG 3:
> > > > > Rejected:
> > > > > > > bad
> > > > > > > > climo stdev = 0*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG
2:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > 2:DEBUG 1: Output file:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > This describes the reason why each observation was
rejected.
> > > > > > > > Some
> > > > > > > > observations didn't match the message type you
requested,
> some
> > > > > > > > were
> > > > > not
> > > > > > > > within the forecast grid, and some did not match the
vertical
> > > > > > > > level.
> > > > > > You
> > > > > > > > will have to make adjustments to your settings so that
the
> data
> > > > > matches
> > > > > > > up
> > > > > > > > properly. Maybe the grid you are using in the PB2NC
config
> file
> > > > > > > > does
> > > > > > not
> > > > > > > > match the forecast grid and you are missing values. If
that
> is
> > > > > > > > the
> > > > > > case,
> > > > > > > > you will have to reconfigure and rerun PB2NC to get
output on
> > > > > > > > the
> > > > > > correct
> > > > > > > > gr5id. Maybe the observation points don't line up with
the
> > > > > > > > vertical
> > > > > > level
> > > > > > > > of the model fields. You can use a range of level
values for
> > > > > > > > the
> > > > > > > > observation data to allow data within that range, i.e.
Z0-10.
> > > > > > > >
> > > > > > > > Let me know if you are still having issues getting
this to
> work
> > > > > > > > and I
> > > > > > can
> > > > > > > > help out.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi George,
> > > > > > > > >
> > > > > > > > > Thanks for your reply!
> > > > > > > > >
> > > > > > > > > I'm still confused, I need to output hourly obs data
to do
> > > > > > > > > hourly
> > > > > > > > > verification.
> > > > > > > > > The original GDAS Prepbufr data has 4 files one day,
should
> > > > > > > > > with 6
> > > > > > > hours
> > > > > > > > > obs data in one file.
> > > > > > > > >
> > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs time
is
> 00?
> > > > > > > > > or 6
> > > > > > times
> > > > > > > > of
> > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > > > > > >
> > > > > > > > > my config file :
> > > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > > > > and /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Zhuming
> > > > > > > > >
> > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Zhuming,
> > > > > > > > > >
> > > > > > > > > > To run the tools hourly you would set the
VALID_INCREMENT
> > > > > > > > > > value
> > > > > for
> > > > > > > 1H.
> > > > > > > > > > Here is an example that runs once per hour for a
day from
> > > > > > 2015-07-15
> > > > > > > 0Z
> > > > > > > > > to
> > > > > > > > > > 2017-07-16 0Z:
> > > > > > > > > >
> > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > > > > >
> > > > > > > > > > VALID_BEG = 2015071500
> > > > > > > > > >
> > > > > > > > > > VALID_END = 2015071600
> > > > > > > > > >
> > > > > > > > > > VALID_INCREMENT = 1H
> > > > > > > > > >
> > > > > > > > > > LEAD_SEQ = 0
> > > > > > > > > >
> > > > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE to
find
> > > > > > > > > > files
> > > > > whose
> > > > > > > > > > filenames match the time you need.
> > > > > > > > > >
> > > > > > > > > > To read prepBUFR observations for 12 hours, you
will want
> > > > > > > > > > to set
> > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify
the
> range
> > > > > > > > > > around
> > > > > > the
> > > > > > > > > valid
> > > > > > > > > > time to process. These values depend on what is
contained
> > > > > > > > > > inside
> > > > > > your
> > > > > > > > > > prepBUFR files. If you want to use data from the
valid
> time
> > > > > > > > > > of
> > > > > the
> > > > > > > > > filename
> > > > > > > > > > backwards 12 hours, you would set:
> > > > > > > > > >
> > > > > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-
12H }
> > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > > > > >
> > > > > > > > > > For a runtime of 2015071512, this would be
substituted
> as:
> > > > > > > > > >
> > > > > > > > > > valid_beg: 20150715_00
> > > > > > > > > > valid_end: 20150715_12
> > > > > > > > > >
> > > > > > > > > > If you want to create an output file for each hour
in
> PB2NC
> > > > > > > > > > you
> > > > > > would
> > > > > > > > > need
> > > > > > > > > > to make sure that the hour is specified in the
> > > > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > > > > >
> > > > > > > > > > PointStat has similar config variables to set the
valid
> beg
> > > > > > > > > > and
> > > > > end
> > > > > > > > range
> > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > > > > >
> > > > > > > > > > If your prepBUFR files are separated out into
multiple
> > > > > > > > > > files that
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > be read into PB2NC, you can set a file window
around the
> > > > > > > > > > valid
> > > > > time
> > > > > > > to
> > > > > > > > > > allow files to be passed into the app. More
information
> on
> > > > > > > > > > that
> > > > > > > > > > functionality is here:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > > > windows-to-find-valid-files
> > > > > > > > > >
> > > > > > > > > > If you are still having trouble getting this
configured,
> > > > > > > > > > you
> > > > > could
> > > > > > > send
> > > > > > > > > me
> > > > > > > > > > the paths of the files you are trying to read and
what
> data
> > > > > > > > > > is
> > > > > > > > contained
> > > > > > > > > in
> > > > > > > > > > them so I can give you suggestions on what to set
for the
> > > > > > > > configuration.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > George
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik
via RT <
> > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was
acted upon.
> > > > > > > > > > > Transaction: Given to mccabe (George McCabe) by
jpresto
> > > > > > > > > > > Queue: met_help
> > > > > > > > > > > Subject: How to set time for hourly 12-24
hours
> data
> > > > > > > > > > > ?
> > > > > > > > > > > Owner: mccabe
> > > > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > > > > Status: new
> > > > > > > > > > > Ticket <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > This transaction appears to have no content
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > 303-497-2768
> > > > > > > > > > ---
> > > > > > > > > > My working day may not be your working day. Please
do not
> > > > > > > > > > feel
> > > > > > > obliged
> > > > > > > > to
> > > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > > > > obliged to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Wed Apr 14 17:29:02 2021
Hi Zhuming,
I tried to get this to work on cheyenne with your data, but was not
successful. I am able to read the data with plot_data_plane and
generate an
image, so I think there may be a bug in the code. I have forwarded
info to
one of our C++ developers to look into it. I will report back when I
know
more.
Thanks,
George
On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> It still can't find the correct time. I tried
>
> FCST_VAR1_LEVELS = (*,*,*)
> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
>
> Thanks!
>
> Zhuming
>
> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > It looks like you are requesting the first time in the 12 hour
file for
> > each run:
> >
> > FCST_VAR1_LEVELS = (0,*,*)
> > ...
> > FCST_VAR2_LEVELS = (0,*,*)
> >
> > The first dimension in these files is the time offset. You will
need the
> > correct index that corresponds to each valid time. Fortunately,
MET makes
> > this easy by allowing you to specify the valid time for a time
dimension
> > explicitly and it will find the correct data. You can set:
> >
> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > ...
> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> >
> > And it will substitute the valid time for each run. Let me know if
you
> have
> > any questions.
> >
> > Thanks,
> > George
> >
> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >
> > > Hi George,
> > >
> > > Thanks for your email, I just would like to send you for help.
> > >
> > > I already resolved the last problem with the WRF forecast, thank
you
> for
> > > your help.
> > >
> > > I also need to verify the MPAS model forecast. These 12 hour
> > > MAPS forecasts are stored in a one file, such as
> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> > > 2015071512/mpas_wrf_cf_2015071512.nc
> > >
> > > It seems MET is doing fine on the first 2 hours, but no matched
pairs
> for
> > > later hours.
> > > I'm not sure what's wrong with it? the time is not matched?
> > >
> > > see the log file
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > > config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> > >
> > > Thanks,
> > > Zhuming
> > >
> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > Just following up on this ticket. Were you able to get this
resolved?
> > Let
> > > > me know if you need more assistance.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > > > Hi Zhuming,
> > > > >
> > > > > Typically directories that contain data for an entire day
are named
> > > > > with
> > > > > YYYYMMDD, so your filename template tag would be
> {valid?fmt=%Y%m%d},
> > > > > however that may not always be the case, as you are seeing
here.
> > There
> > > > > are
> > > > > a couple ways to handle this.
> > > > >
> > > > > If you know that the directory will always be YYYYMMDD with
00 at
> the
> > > > > end,
> > > > > you can simply put 00 in the template instead of adding %H
in the
> > tag,
> > > > > i.e.
> > > > >
> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> 00_cf.nc
> > > > >
> > > > > If the HH value can vary, for instance if there are 2
directories
> > > > > created
> > > > > per day (i.e. 2015071400 and 2015071412), then you can use
the
> > > > > truncate
> > > > > keyword to handle it.
> > > > >
> > > > > If directories are created every 24 hours:
> > > > >
> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-
%m-
> > > > > %d_%H}:00:
> > > > > 00_cf.nc
> > > > >
> > > > > If directories are created every 12 hours:
> > > > >
> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-
%m-
> > > > > %d_%H}:00:
> > > > > 00_cf.nc
> > > > >
> > > > > Let me know if that doesn't work for you.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > > > >
> > > > > > Hi George,
> > > > > >
> > > > > > Processing Obs data using "PB2NC" are right now.
> > > > > >
> > > > > > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
> > VALID_END =
> > > > > > 2015071501. It looks for the forecast data 2015071500/
> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > > > > But my forecast data are all put in dir of 2015071500/.
> > > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get
the
> right
> > > > > > fcst
> > > > > > data?
> > > > > >
> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> > 00_cf.nc
> > > > > >
> > > > > >
> > > > > > config file:
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > >
> > > > > > log file:
> > > > > >
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Zhuming
> > > > > >
> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > > > > > <met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Zhuming,
> > > > > > >
> > > > > > > Looking at the log output, it looks like the data is
stored in
> > the
> > > > > > prepBUFR
> > > > > > > files differently that you may have thought:
> > > > > > >
> > > > > > > DEBUG 2: PrepBufr Time Center: 20150715_000000
> > > > > > > DEBUG 2: Searching Time Window: 20150715_030000
to
> > > > > > 20150715_040000
> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70%
75% 80%
> > 85%
> > > > > > > 90%
> > > > > > 95%
> > > > > > > 100%
> > > > > > >
> > > > > > > DEBUG 2: Total Messages processed = 590836
> > > > > > > DEBUG 2: Rejected based on message type = 0
> > > > > > > DEBUG 2: Rejected based on station id = 0
> > > > > > > DEBUG 2: Rejected based on valid time = 590815
> > > > > > > DEBUG 2: Rejected based on masking grid = 20
> > > > > > > DEBUG 2: Rejected based on masking polygon = 0
> > > > > > > DEBUG 2: Rejected based on elevation = 0
> > > > > > > DEBUG 2: Rejected based on pb report type = 1
> > > > > > > DEBUG 2: Rejected based on input report type = 0
> > > > > > > DEBUG 2: Rejected based on instrument type = 0
> > > > > > > DEBUG 2: Rejected based on zero observations = 0
> > > > > > > DEBUG 2: Total Messages retained = 0
> > > > > > > DEBUG 2: Total observations retained or derived = 0
> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
20150715_030000*
> > > > > > > WARNING:
> > > > > > > WARNING: process_pbfile() -> No PrepBufr messages
retained from
> > > > > > > file:
> > > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > > > > > prepbufr.gdas.2015071500.nr
> > > > > > > WARNING:
> > > > > > > ERROR :
> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr messages
> > retained.
> > > > > > Nothing
> > > > > > > to write.
> > > > > > > ERROR :
> > > > > > >
> > > > > > > The prepbufr.gdas.2015071500.nr file actually contains
data
> from
> > > > > > > 20150714_210000 to 20150715_030000, not 20150715_000000
to
> > > > > > 20150715_060000.
> > > > > > >
> > > > > > > To get the correct file for each run, you will have to
shift
> the
> > > > > > > valid
> > > > > > time
> > > > > > > before truncating it so that it gets the correct file
for each
> > run.
> > > > > > Instead
> > > > > > > of setting:
> > > > > > >
> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > > >
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > > > > >
> > > > > > > You will have to also shift forward 3 hours, i.e.:
> > > > > > >
> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > > > > >
> > > > > > > This says to add 3 hours to the current valid time
before
> > > > > > > truncating to
> > > > > > the
> > > > > > > nearest 6 hours.
> > > > > > >
> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > >
> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > >
> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > > > > >
> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > > > > > File: prepbufr.gdas.2015071506.nr
> > > > > > >
> > > > > > > Let me know if that doesn't work.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >
> > > > > > > >
> > > > > > > > Hi George,
> > > > > > > >
> > > > > > > > Now makes me confused is why processed obs of several
times
> are
> > > > > > missing.
> > > > > > > > There has files of time at 00,01,02, but no 03,04,05;
has
> > > > > > > > 06,07,08, but
> > > > > > > no
> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20,
but with
> > > > > > > > 21,no
> > > > > > 22,23.
> > > > > > > > From yours and mines:
> > > > > > > >
> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > > > > > >
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> > pbs.2015071500.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> > pbs.2015071501.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> > pbs.2015071502.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> > pbs.2015071506.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> > pbs.2015071507.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> > pbs.2015071508.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> > pbs.2015071512.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> > pbs.2015071513.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> > pbs.2015071514.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> > pbs.2015071515.nc
> > > > > > > >
> > > > > > > >
> > > > > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > > > > > >
> > > > > > > > total 96112
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> > > pbs.2015071500.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> > > pbs.2015071501.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> > > pbs.2015071502.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> > > pbs.2015071506.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> > > pbs.2015071507.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> > > pbs.2015071508.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> > > pbs.2015071512.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> > > pbs.2015071513.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> > > pbs.2015071514.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> > > pbs.2015071518.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> > > pbs.2015071519.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> > > pbs.2015071520.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> > > pbs.2015071521.nc
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> > > pbs.2015071600.nc
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Zhuming
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> > > > > > > > <met_help at ucar.edu
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Zhuming,
> > > > > > > > >
> > > > > > > > > Thanks for the information. You will need to use the
> truncate
> > > > > > > > > keyword
> > > > > > > to
> > > > > > > > > find the files, then set the valid beg/end range to
1 hour.
> > > > > > > > > This is
> > > > > > > what
> > > > > > > > I
> > > > > > > > > changed from your PB2NC1.conf config file:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H}PB2NC_VALID_END =
> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > > > > > > >
> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > > > > > > >
> > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > > > > > > >
> > > > > > > > > My config file is on cheyenne here:
> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > > > > > > >
> > > > > > > > > I just started running it and the output is being
written
> > here:
> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > For PointStat, you will want to loop over the same
times. I
> > > > > > > > > noticed
> > > > > > you
> > > > > > > > > specify message type in the VAR<n>_OPTIONS, but you
do not
> > need
> > > > > > > > > to do
> > > > > > > > this
> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable
that
> > defines
> > > > > > > > > this.
> > > > > > > This
> > > > > > > > > is what I changed in your PointStat1.conf file:
> > > > > > > > >
> > > > > > > > > *VALID_END = 2015071600*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > > > > > =POINT_STAT_OBS_VALID_BEG
> > > > > > > > =*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
message_type
> =
> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > > > > > > >
> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> > > > > > > > >
> > > > > > > > > Or you can view my file in the same directory. I
usually
> try
> > to
> > > > > > > > > test
> > > > > > > > with a
> > > > > > > > > single run time to make sure it is working properly,
so I
> set
> > > > > > VALID_END
> > > > > > > > > back to 2015071500. I ran it and it produced an
output
> file,
> > > > > > > > > but
> > > > > > there
> > > > > > > > were
> > > > > > > > > no values besides the header. I checked the log file
and
> saw
> > > > > > > > > this:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2, for
> > > > > > > > > observation type
> > > > > > > > > ADPSFC, over region FULL, for interpolation method
> BILIN(4),
> > > > > > > > > using 0
> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
> 3:
> > > > > > > Observations
> > > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > > > 0DEBUG 3:
> > > > > > > > > Rejected: obs type = 390528DEBUG 3: Rejected:
valid
> > time
> > > > > > =
> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected:
> off
> > > > > > > > > the
> > > > > > grid
> > > > > > > > =
> > > > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3:
> Rejected:
> > > > > > > > > level
> > > > > > > > mismatch
> > > > > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
> Rejected:
> > > > > > message
> > > > > > > > type
> > > > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
Rejected:
> > bad
> > > > > > fcst
> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG 3:
> > > > > > > > > Rejected:
> > > > > > bad
> > > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates =
0DEBUG
> > > > > > > > > 2:DEBUG 2:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus
UGRD/Z10, for
> > > > > > observation
> > > > > > > > type
> > > > > > > > > ADPSFC, over region FULL, for interpolation method
> BILIN(4),
> > > > > > > > > using 0
> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
> 3:
> > > > > > > Observations
> > > > > > > > > processed = 393373*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected:
> obs
> > > > > > > > > type
> > > > > > > > =
> > > > > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG 3:
> > Rejected:
> > > > > > > > > bad
> > > > > > obs
> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid =
9240DEBUG
> 3:
> > > > > > Rejected:
> > > > > > > > > topography = 0DEBUG 3: Rejected: level mismatch
=
> > > > > > > > > 1168DEBUG 3:
> > > > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected:
message
> type
> > > > > > > > > =
> > > > > > > 0DEBUG
> > > > > > > > 3:
> > > > > > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
> > value
> > > > > > > > > =
> > > > > > > 0DEBUG
> > > > > > > > 3:
> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad
climo
> > stdev
> > > > > > > > > =
> > > > > > > 0DEBUG
> > > > > > > > 3:
> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus
VGRD/Z10, for
> > > > > > observation
> > > > > > > > type
> > > > > > > > > ADPSFC, over region FULL, for interpolation method
> BILIN(4),
> > > > > > > > > using 0
> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
> 3:
> > > > > > > Observations
> > > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
> > > > > > > > > 0DEBUG 3:
> > > > > > > > > Rejected: obs type = 382965DEBUG 3: Rejected:
valid
> > time
> > > > > > =
> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected:
> off
> > > > > > > > > the
> > > > > > grid
> > > > > > > > =
> > > > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3:
> Rejected:
> > > > > > > > > level
> > > > > > > > mismatch
> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG 3:
> > Rejected:
> > > > > > message
> > > > > > > > > type = 0DEBUG 3: Rejected: masking region =
0DEBUG 3:
> > > > > > > > > Rejected:
> > > > > > bad
> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG
> 3:
> > > > > > Rejected:
> > > > > > > > bad
> > > > > > > > > climo stdev = 0*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG 2:DEBUG
2:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > > > > > 2:DEBUG 1: Output file:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This describes the reason why each observation was
> rejected.
> > > > > > > > > Some
> > > > > > > > > observations didn't match the message type you
requested,
> > some
> > > > > > > > > were
> > > > > > not
> > > > > > > > > within the forecast grid, and some did not match the
> vertical
> > > > > > > > > level.
> > > > > > > You
> > > > > > > > > will have to make adjustments to your settings so
that the
> > data
> > > > > > matches
> > > > > > > > up
> > > > > > > > > properly. Maybe the grid you are using in the PB2NC
config
> > file
> > > > > > > > > does
> > > > > > > not
> > > > > > > > > match the forecast grid and you are missing values.
If that
> > is
> > > > > > > > > the
> > > > > > > case,
> > > > > > > > > you will have to reconfigure and rerun PB2NC to get
output
> on
> > > > > > > > > the
> > > > > > > correct
> > > > > > > > > gr5id. Maybe the observation points don't line up
with the
> > > > > > > > > vertical
> > > > > > > level
> > > > > > > > > of the model fields. You can use a range of level
values
> for
> > > > > > > > > the
> > > > > > > > > observation data to allow data within that range,
i.e.
> Z0-10.
> > > > > > > > >
> > > > > > > > > Let me know if you are still having issues getting
this to
> > work
> > > > > > > > > and I
> > > > > > > can
> > > > > > > > > help out.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi George,
> > > > > > > > > >
> > > > > > > > > > Thanks for your reply!
> > > > > > > > > >
> > > > > > > > > > I'm still confused, I need to output hourly obs
data to
> do
> > > > > > > > > > hourly
> > > > > > > > > > verification.
> > > > > > > > > > The original GDAS Prepbufr data has 4 files one
day,
> should
> > > > > > > > > > with 6
> > > > > > > > hours
> > > > > > > > > > obs data in one file.
> > > > > > > > > >
> > > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs
time is
> > 00?
> > > > > > > > > > or 6
> > > > > > > times
> > > > > > > > > of
> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > > > > > > > >
> > > > > > > > > > my config file :
> > > > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > > > > > > > > and
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Zhuming
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Zhuming,
> > > > > > > > > > >
> > > > > > > > > > > To run the tools hourly you would set the
> VALID_INCREMENT
> > > > > > > > > > > value
> > > > > > for
> > > > > > > > 1H.
> > > > > > > > > > > Here is an example that runs once per hour for a
day
> from
> > > > > > > 2015-07-15
> > > > > > > > 0Z
> > > > > > > > > > to
> > > > > > > > > > > 2017-07-16 0Z:
> > > > > > > > > > >
> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > > > > > > > > >
> > > > > > > > > > > VALID_BEG = 2015071500
> > > > > > > > > > >
> > > > > > > > > > > VALID_END = 2015071600
> > > > > > > > > > >
> > > > > > > > > > > VALID_INCREMENT = 1H
> > > > > > > > > > >
> > > > > > > > > > > LEAD_SEQ = 0
> > > > > > > > > > >
> > > > > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE
to find
> > > > > > > > > > > files
> > > > > > whose
> > > > > > > > > > > filenames match the time you need.
> > > > > > > > > > >
> > > > > > > > > > > To read prepBUFR observations for 12 hours, you
will
> want
> > > > > > > > > > > to set
> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to specify
the
> > range
> > > > > > > > > > > around
> > > > > > > the
> > > > > > > > > > valid
> > > > > > > > > > > time to process. These values depend on what is
> contained
> > > > > > > > > > > inside
> > > > > > > your
> > > > > > > > > > > prepBUFR files. If you want to use data from the
valid
> > time
> > > > > > > > > > > of
> > > > > > the
> > > > > > > > > > filename
> > > > > > > > > > > backwards 12 hours, you would set:
> > > > > > > > > > >
> > > > > > > > > > > PB2NC_VALID_BEGIN = {valid?fmt=%Y%m%d_%H?shift=-
12H }
> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > > > > > > > > >
> > > > > > > > > > > For a runtime of 2015071512, this would be
substituted
> > as:
> > > > > > > > > > >
> > > > > > > > > > > valid_beg: 20150715_00
> > > > > > > > > > > valid_end: 20150715_12
> > > > > > > > > > >
> > > > > > > > > > > If you want to create an output file for each
hour in
> > PB2NC
> > > > > > > > > > > you
> > > > > > > would
> > > > > > > > > > need
> > > > > > > > > > > to make sure that the hour is specified in the
> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> > > > > > > > > > >
> > > > > > > > > > > PointStat has similar config variables to set
the valid
> > beg
> > > > > > > > > > > and
> > > > > > end
> > > > > > > > > range
> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > > > > > > > > >
> > > > > > > > > > > If your prepBUFR files are separated out into
multiple
> > > > > > > > > > > files that
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > be read into PB2NC, you can set a file window
around
> the
> > > > > > > > > > > valid
> > > > > > time
> > > > > > > > to
> > > > > > > > > > > allow files to be passed into the app. More
information
> > on
> > > > > > > > > > > that
> > > > > > > > > > > functionality is here:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > > > > windows-to-find-valid-files
> > > > > > > > > > >
> > > > > > > > > > > If you are still having trouble getting this
> configured,
> > > > > > > > > > > you
> > > > > > could
> > > > > > > > send
> > > > > > > > > > me
> > > > > > > > > > > the paths of the files you are trying to read
and what
> > data
> > > > > > > > > > > is
> > > > > > > > > contained
> > > > > > > > > > in
> > > > > > > > > > > them so I can give you suggestions on what to
set for
> the
> > > > > > > > > configuration.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > George
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik
via RT
> <
> > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was
acted
> upon.
> > > > > > > > > > > > Transaction: Given to mccabe (George McCabe)
by
> jpresto
> > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > Subject: How to set time for hourly 12-24
hours
> > data
> > > > > > > > > > > > ?
> > > > > > > > > > > > Owner: mccabe
> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> > > > > > > > > > > > Status: new
> > > > > > > > > > > > Ticket <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > This transaction appears to have no content
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > 303-497-2768
> > > > > > > > > > > ---
> > > > > > > > > > > My working day may not be your working day.
Please do
> not
> > > > > > > > > > > feel
> > > > > > > > obliged
> > > > > > > > > to
> > > > > > > > > > > reply to this email outside of your normal
working
> hours.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > > > > obliged to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Thu Apr 15 13:52:17 2021
Hi Zhuming,
There was indeed a bug in the code. Howard has fixed the issue with
this
pull request: https://github.com/dtcenter/MET/pull/1756
This fix will be included in the MET 10.0.0-beta5 release that is
tentatively scheduled for next week. This beta version can be
installed on
cheyenne so you can test that the update fixes your issue. I will
follow up
and let you know when the beta release is available on cheyenne.
Thanks,
George
On Wed, Apr 14, 2021 at 5:28 PM George McCabe <mccabe at ucar.edu> wrote:
> Hi Zhuming,
>
> I tried to get this to work on cheyenne with your data, but was not
> successful. I am able to read the data with plot_data_plane and
generate an
> image, so I think there may be a bug in the code. I have forwarded
info to
> one of our C++ developers to look into it. I will report back when I
know
> more.
>
> Thanks,
> George
>
> On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>>
>> Hi George,
>>
>> It still can't find the correct time. I tried
>>
>> FCST_VAR1_LEVELS = (*,*,*)
>> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
>> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
>> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
>>
>> Thanks!
>>
>> Zhuming
>>
>> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>> > Hi Zhuming,
>> >
>> > It looks like you are requesting the first time in the 12 hour
file for
>> > each run:
>> >
>> > FCST_VAR1_LEVELS = (0,*,*)
>> > ...
>> > FCST_VAR2_LEVELS = (0,*,*)
>> >
>> > The first dimension in these files is the time offset. You will
need the
>> > correct index that corresponds to each valid time. Fortunately,
MET
>> makes
>> > this easy by allowing you to specify the valid time for a time
dimension
>> > explicitly and it will find the correct data. You can set:
>> >
>> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
>> > ...
>> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
>> >
>> > And it will substitute the valid time for each run. Let me know
if you
>> have
>> > any questions.
>> >
>> > Thanks,
>> > George
>> >
>> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
>> met_help at ucar.edu
>> > >
>> > wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>> > >
>> > > Hi George,
>> > >
>> > > Thanks for your email, I just would like to send you for help.
>> > >
>> > > I already resolved the last problem with the WRF forecast,
thank you
>> for
>> > > your help.
>> > >
>> > > I also need to verify the MPAS model forecast. These 12 hour
>> > > MAPS forecasts are stored in a one file, such as
>> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
>> > > 2015071512/mpas_wrf_cf_2015071512.nc
>> > >
>> > > It seems MET is doing fine on the first 2 hours, but no matched
pairs
>> for
>> > > later hours.
>> > > I'm not sure what's wrong with it? the time is not matched?
>> > >
>> > > see the log file
>> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
>> > > config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
>> > >
>> > > Thanks,
>> > > Zhuming
>> > >
>> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
>> met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Hi Zhuming,
>> > > >
>> > > > Just following up on this ticket. Were you able to get this
>> resolved?
>> > Let
>> > > > me know if you need more assistance.
>> > > >
>> > > > Thanks,
>> > > > George
>> > > >
>> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
>> > > > > Hi Zhuming,
>> > > > >
>> > > > > Typically directories that contain data for an entire day
are
>> named
>> > > > > with
>> > > > > YYYYMMDD, so your filename template tag would be
>> {valid?fmt=%Y%m%d},
>> > > > > however that may not always be the case, as you are seeing
here.
>> > There
>> > > > > are
>> > > > > a couple ways to handle this.
>> > > > >
>> > > > > If you know that the directory will always be YYYYMMDD with
00 at
>> the
>> > > > > end,
>> > > > > you can simply put 00 in the template instead of adding %H
in the
>> > tag,
>> > > > > i.e.
>> > > > >
>> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
>> > > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
>> 00_cf.nc
>> > > > >
>> > > > > If the HH value can vary, for instance if there are 2
directories
>> > > > > created
>> > > > > per day (i.e. 2015071400 and 2015071412), then you can use
the
>> > > > > truncate
>> > > > > keyword to handle it.
>> > > > >
>> > > > > If directories are created every 24 hours:
>> > > > >
>> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
>> > > > > {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-
%m-
>> > > > > %d_%H}:00:
>> > > > > 00_cf.nc
>> > > > >
>> > > > > If directories are created every 12 hours:
>> > > > >
>> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
>> > > > > {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-
%m-
>> > > > > %d_%H}:00:
>> > > > > 00_cf.nc
>> > > > >
>> > > > > Let me know if that doesn't work for you.
>> > > > >
>> > > > > Thanks,
>> > > > > George
>> > > > >
>> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
>> > > > > <met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>> > > > > >
>> > > > > > Hi George,
>> > > > > >
>> > > > > > Processing Obs data using "PB2NC" are right now.
>> > > > > >
>> > > > > > Now I'm using "PointStat" , I set VALID_BEG = 2015071500,
>> > VALID_END =
>> > > > > > 2015071501. It looks for the forecast data 2015071500/
>> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
>> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
>> > > > > > But my forecast data are all put in dir of 2015071500/.
>> > > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to get
the
>> right
>> > > > > > fcst
>> > > > > > data?
>> > > > > >
>> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
>> > > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
>> > 00_cf.nc
>> > > > > >
>> > > > > >
>> > > > > > config file:
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
>> > > > > >
>> > > > > > log file:
>> > > > > >
>> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
>> > > > > >
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Zhuming
>> > > > > >
>> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
>> > > > > > <met_help at ucar.edu>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Zhuming,
>> > > > > > >
>> > > > > > > Looking at the log output, it looks like the data is
stored in
>> > the
>> > > > > > prepBUFR
>> > > > > > > files differently that you may have thought:
>> > > > > > >
>> > > > > > > DEBUG 2: PrepBufr Time Center: 20150715_000000
>> > > > > > > DEBUG 2: Searching Time Window: 20150715_030000
to
>> > > > > > 20150715_040000
>> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70%
75% 80%
>> > 85%
>> > > > > > > 90%
>> > > > > > 95%
>> > > > > > > 100%
>> > > > > > >
>> > > > > > > DEBUG 2: Total Messages processed =
590836
>> > > > > > > DEBUG 2: Rejected based on message type = 0
>> > > > > > > DEBUG 2: Rejected based on station id = 0
>> > > > > > > DEBUG 2: Rejected based on valid time =
590815
>> > > > > > > DEBUG 2: Rejected based on masking grid = 20
>> > > > > > > DEBUG 2: Rejected based on masking polygon = 0
>> > > > > > > DEBUG 2: Rejected based on elevation = 0
>> > > > > > > DEBUG 2: Rejected based on pb report type = 1
>> > > > > > > DEBUG 2: Rejected based on input report type = 0
>> > > > > > > DEBUG 2: Rejected based on instrument type = 0
>> > > > > > > DEBUG 2: Rejected based on zero observations = 0
>> > > > > > > DEBUG 2: Total Messages retained = 0
>> > > > > > > DEBUG 2: Total observations retained or derived = 0
>> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
>> 20150715_030000*
>> > > > > > > WARNING:
>> > > > > > > WARNING: process_pbfile() -> No PrepBufr messages
retained
>> from
>> > > > > > > file:
>> > > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
>> > > > > > > prepbufr.gdas.2015071500.nr
>> > > > > > > WARNING:
>> > > > > > > ERROR :
>> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr
messages
>> > retained.
>> > > > > > Nothing
>> > > > > > > to write.
>> > > > > > > ERROR :
>> > > > > > >
>> > > > > > > The prepbufr.gdas.2015071500.nr file actually contains
data
>> from
>> > > > > > > 20150714_210000 to 20150715_030000, not 20150715_000000
to
>> > > > > > 20150715_060000.
>> > > > > > >
>> > > > > > > To get the correct file for each run, you will have to
shift
>> the
>> > > > > > > valid
>> > > > > > time
>> > > > > > > before truncating it so that it gets the correct file
for each
>> > run.
>> > > > > > Instead
>> > > > > > > of setting:
>> > > > > > >
>> > > > > > > PB2NC_INPUT_TEMPLATE =
>> > > > > > >
>> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
>> > > > > > >
>> > > > > > > You will have to also shift forward 3 hours, i.e.:
>> > > > > > >
>> > > > > > > PB2NC_INPUT_TEMPLATE =
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
>> > > > > > >
>> > > > > > > This says to add 3 hours to the current valid time
before
>> > > > > > > truncating to
>> > > > > > the
>> > > > > > > nearest 6 hours.
>> > > > > > >
>> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
>> > > > > > > File: prepbufr.gdas.2015071500.nr
>> > > > > > >
>> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
>> > > > > > > File: prepbufr.gdas.2015071500.nr
>> > > > > > >
>> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
>> > > > > > > File: prepbufr.gdas.2015071500.nr
>> > > > > > >
>> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
>> > > > > > > File: prepbufr.gdas.2015071506.nr
>> > > > > > >
>> > > > > > > Let me know if that doesn't work.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > George
>> > > > > > >
>> > > > > > >
>> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via RT
<
>> > > > > > met_help at ucar.edu>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > >
>> > > > > > > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>> > >
>> > > > > > > >
>> > > > > > > > Hi George,
>> > > > > > > >
>> > > > > > > > Now makes me confused is why processed obs of several
times
>> are
>> > > > > > missing.
>> > > > > > > > There has files of time at 00,01,02, but no 03,04,05;
has
>> > > > > > > > 06,07,08, but
>> > > > > > > no
>> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20,
but with
>> > > > > > > > 21,no
>> > > > > > 22,23.
>> > > > > > > > From yours and mines:
>> > > > > > > >
>> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
>> > pbs.2015071500.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
>> > pbs.2015071501.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
>> > pbs.2015071502.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
>> > pbs.2015071506.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
>> > pbs.2015071507.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
>> > pbs.2015071508.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
>> > pbs.2015071512.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
>> > pbs.2015071513.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
>> > pbs.2015071514.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
>> > pbs.2015071515.nc
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
>> > > > > > > >
>> > > > > > > > total 96112
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
>> > > pbs.2015071500.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
>> > > pbs.2015071501.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
>> > > pbs.2015071502.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
>> > > pbs.2015071506.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
>> > > pbs.2015071507.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
>> > > pbs.2015071508.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
>> > > pbs.2015071512.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
>> > > pbs.2015071513.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
>> > > pbs.2015071514.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
>> > > pbs.2015071518.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
>> > > pbs.2015071519.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
>> > > pbs.2015071520.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
>> > > pbs.2015071521.nc
>> > > > > > > >
>> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
>> > > pbs.2015071600.nc
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > >
>> > > > > > > > Zhuming
>> > > > > > > >
>> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
>> > > > > > > > <met_help at ucar.edu
>> > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Zhuming,
>> > > > > > > > >
>> > > > > > > > > Thanks for the information. You will need to use
the
>> truncate
>> > > > > > > > > keyword
>> > > > > > > to
>> > > > > > > > > find the files, then set the valid beg/end range to
1
>> hour.
>> > > > > > > > > This is
>> > > > > > > what
>> > > > > > > > I
>> > > > > > > > > changed from your PB2NC1.conf config file:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
>> =
>> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
>> > > > > > > > >
>> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
>> > > > > > > > >
>> > > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
>> > > > > > > > >
>> > > > > > > > > My config file is on cheyenne here:
>> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
>> > > > > > > > >
>> > > > > > > > > I just started running it and the output is being
written
>> > here:
>> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > For PointStat, you will want to loop over the same
times.
>> I
>> > > > > > > > > noticed
>> > > > > > you
>> > > > > > > > > specify message type in the VAR<n>_OPTIONS, but you
do not
>> > need
>> > > > > > > > > to do
>> > > > > > > > this
>> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable
that
>> > defines
>> > > > > > > > > this.
>> > > > > > > This
>> > > > > > > > > is what I changed in your PointStat1.conf file:
>> > > > > > > > >
>> > > > > > > > > *VALID_END = 2015071600*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
>> > > > > > =POINT_STAT_OBS_VALID_BEG
>> > > > > > > > =*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
>> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
>> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
>> message_type =
>> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
>> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
>> > > > > > > > >
>> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
>> > > > > > > > >
>> > > > > > > > > Or you can view my file in the same directory. I
usually
>> try
>> > to
>> > > > > > > > > test
>> > > > > > > > with a
>> > > > > > > > > single run time to make sure it is working
properly, so I
>> set
>> > > > > > VALID_END
>> > > > > > > > > back to 2015071500. I ran it and it produced an
output
>> file,
>> > > > > > > > > but
>> > > > > > there
>> > > > > > > > were
>> > > > > > > > > no values besides the header. I checked the log
file and
>> saw
>> > > > > > > > > this:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2,
for
>> > > > > > > > > observation type
>> > > > > > > > > ADPSFC, over region FULL, for interpolation method
>> BILIN(4),
>> > > > > > > > > using 0
>> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
>> 3:
>> > > > > > > Observations
>> > > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
>> > > > > > > > > 0DEBUG 3:
>> > > > > > > > > Rejected: obs type = 390528DEBUG 3:
Rejected: valid
>> > time
>> > > > > > =
>> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected:
>> off
>> > > > > > > > > the
>> > > > > > grid
>> > > > > > > > =
>> > > > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG 3:
>> Rejected:
>> > > > > > > > > level
>> > > > > > > > mismatch
>> > > > > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG 3:
>> Rejected:
>> > > > > > message
>> > > > > > > > type
>> > > > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
>> Rejected:
>> > bad
>> > > > > > fcst
>> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG 3:
>> > > > > > > > > Rejected:
>> > > > > > bad
>> > > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates =
0DEBUG
>> > > > > > > > > 2:DEBUG 2:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------DEBUG
>> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus
UGRD/Z10, for
>> > > > > > observation
>> > > > > > > > type
>> > > > > > > > > ADPSFC, over region FULL, for interpolation method
>> BILIN(4),
>> > > > > > > > > using 0
>> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
>> 3:
>> > > > > > > Observations
>> > > > > > > > > processed = 393373*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
Rejected:
>> obs
>> > > > > > > > > type
>> > > > > > > > =
>> > > > > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG
3:
>> > Rejected:
>> > > > > > > > > bad
>> > > > > > obs
>> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid =
9240DEBUG
>> 3:
>> > > > > > Rejected:
>> > > > > > > > > topography = 0DEBUG 3: Rejected: level
mismatch =
>> > > > > > > > > 1168DEBUG 3:
>> > > > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected:
message
>> type
>> > > > > > > > > =
>> > > > > > > 0DEBUG
>> > > > > > > > 3:
>> > > > > > > > > Rejected: masking region = 0DEBUG 3: Rejected: bad
fcst
>> > value
>> > > > > > > > > =
>> > > > > > > 0DEBUG
>> > > > > > > > 3:
>> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected: bad
climo
>> > stdev
>> > > > > > > > > =
>> > > > > > > 0DEBUG
>> > > > > > > > 3:
>> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------DEBUG
>> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus
VGRD/Z10, for
>> > > > > > observation
>> > > > > > > > type
>> > > > > > > > > ADPSFC, over region FULL, for interpolation method
>> BILIN(4),
>> > > > > > > > > using 0
>> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs =
0DEBUG
>> 3:
>> > > > > > > Observations
>> > > > > > > > > processed = 393373DEBUG 3: Rejected: station id
=
>> > > > > > > > > 0DEBUG 3:
>> > > > > > > > > Rejected: obs type = 382965DEBUG 3:
Rejected: valid
>> > time
>> > > > > > =
>> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
Rejected:
>> off
>> > > > > > > > > the
>> > > > > > grid
>> > > > > > > > =
>> > > > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG 3:
>> Rejected:
>> > > > > > > > > level
>> > > > > > > > mismatch
>> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG
3:
>> > Rejected:
>> > > > > > message
>> > > > > > > > > type = 0DEBUG 3: Rejected: masking region =
0DEBUG 3:
>> > > > > > > > > Rejected:
>> > > > > > bad
>> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean =
>> 0DEBUG 3:
>> > > > > > Rejected:
>> > > > > > > > bad
>> > > > > > > > > climo stdev = 0*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG 2:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------DEBUG
>> > > > > > > > > 2:DEBUG 1: Output file:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > This describes the reason why each observation was
>> rejected.
>> > > > > > > > > Some
>> > > > > > > > > observations didn't match the message type you
requested,
>> > some
>> > > > > > > > > were
>> > > > > > not
>> > > > > > > > > within the forecast grid, and some did not match
the
>> vertical
>> > > > > > > > > level.
>> > > > > > > You
>> > > > > > > > > will have to make adjustments to your settings so
that the
>> > data
>> > > > > > matches
>> > > > > > > > up
>> > > > > > > > > properly. Maybe the grid you are using in the PB2NC
config
>> > file
>> > > > > > > > > does
>> > > > > > > not
>> > > > > > > > > match the forecast grid and you are missing values.
If
>> that
>> > is
>> > > > > > > > > the
>> > > > > > > case,
>> > > > > > > > > you will have to reconfigure and rerun PB2NC to get
>> output on
>> > > > > > > > > the
>> > > > > > > correct
>> > > > > > > > > gr5id. Maybe the observation points don't line up
with the
>> > > > > > > > > vertical
>> > > > > > > level
>> > > > > > > > > of the model fields. You can use a range of level
values
>> for
>> > > > > > > > > the
>> > > > > > > > > observation data to allow data within that range,
i.e.
>> Z0-10.
>> > > > > > > > >
>> > > > > > > > > Let me know if you are still having issues getting
this to
>> > work
>> > > > > > > > > and I
>> > > > > > > can
>> > > > > > > > > help out.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > > George
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu via
RT <
>> > > > > > > > met_help at ucar.edu>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Hi George,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for your reply!
>> > > > > > > > > >
>> > > > > > > > > > I'm still confused, I need to output hourly obs
data to
>> do
>> > > > > > > > > > hourly
>> > > > > > > > > > verification.
>> > > > > > > > > > The original GDAS Prepbufr data has 4 files one
day,
>> should
>> > > > > > > > > > with 6
>> > > > > > > > hours
>> > > > > > > > > > obs data in one file.
>> > > > > > > > > >
>> > > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs
time is
>> > 00?
>> > > > > > > > > > or 6
>> > > > > > > times
>> > > > > > > > > of
>> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
>> > > > > > > > > >
>> > > > > > > > > > my config file :
>> > > > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
>> > > > > > > > > > and
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Zhuming
>> > > > > > > > > >
>> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe via
RT <
>> > > > > > > met_help at ucar.edu
>> > > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Zhuming,
>> > > > > > > > > > >
>> > > > > > > > > > > To run the tools hourly you would set the
>> VALID_INCREMENT
>> > > > > > > > > > > value
>> > > > > > for
>> > > > > > > > 1H.
>> > > > > > > > > > > Here is an example that runs once per hour for
a day
>> from
>> > > > > > > 2015-07-15
>> > > > > > > > 0Z
>> > > > > > > > > > to
>> > > > > > > > > > > 2017-07-16 0Z:
>> > > > > > > > > > >
>> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
>> > > > > > > > > > >
>> > > > > > > > > > > VALID_BEG = 2015071500
>> > > > > > > > > > >
>> > > > > > > > > > > VALID_END = 2015071600
>> > > > > > > > > > >
>> > > > > > > > > > > VALID_INCREMENT = 1H
>> > > > > > > > > > >
>> > > > > > > > > > > LEAD_SEQ = 0
>> > > > > > > > > > >
>> > > > > > > > > > > Each run time will use the PB2NC_INPUT_TEMPLATE
to
>> find
>> > > > > > > > > > > files
>> > > > > > whose
>> > > > > > > > > > > filenames match the time you need.
>> > > > > > > > > > >
>> > > > > > > > > > > To read prepBUFR observations for 12 hours, you
will
>> want
>> > > > > > > > > > > to set
>> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to
specify the
>> > range
>> > > > > > > > > > > around
>> > > > > > > the
>> > > > > > > > > > valid
>> > > > > > > > > > > time to process. These values depend on what is
>> contained
>> > > > > > > > > > > inside
>> > > > > > > your
>> > > > > > > > > > > prepBUFR files. If you want to use data from
the valid
>> > time
>> > > > > > > > > > > of
>> > > > > > the
>> > > > > > > > > > filename
>> > > > > > > > > > > backwards 12 hours, you would set:
>> > > > > > > > > > >
>> > > > > > > > > > > PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H?shift=-12H }
>> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
>> > > > > > > > > > >
>> > > > > > > > > > > For a runtime of 2015071512, this would be
substituted
>> > as:
>> > > > > > > > > > >
>> > > > > > > > > > > valid_beg: 20150715_00
>> > > > > > > > > > > valid_end: 20150715_12
>> > > > > > > > > > >
>> > > > > > > > > > > If you want to create an output file for each
hour in
>> > PB2NC
>> > > > > > > > > > > you
>> > > > > > > would
>> > > > > > > > > > need
>> > > > > > > > > > > to make sure that the hour is specified in the
>> > > > > > > PB2NC_OUTPUT_TEMPLATE.
>> > > > > > > > > > >
>> > > > > > > > > > > PointStat has similar config variables to set
the
>> valid
>> > beg
>> > > > > > > > > > > and
>> > > > > > end
>> > > > > > > > > range
>> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
>> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
>> > > > > > > > > > >
>> > > > > > > > > > > If your prepBUFR files are separated out into
multiple
>> > > > > > > > > > > files that
>> > > > > > > > need
>> > > > > > > > > to
>> > > > > > > > > > > be read into PB2NC, you can set a file window
around
>> the
>> > > > > > > > > > > valid
>> > > > > > time
>> > > > > > > > to
>> > > > > > > > > > > allow files to be passed into the app. More
>> information
>> > on
>> > > > > > > > > > > that
>> > > > > > > > > > > functionality is here:
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
>> > > > > > windows-to-find-valid-files
>> > > > > > > > > > >
>> > > > > > > > > > > If you are still having trouble getting this
>> configured,
>> > > > > > > > > > > you
>> > > > > > could
>> > > > > > > > send
>> > > > > > > > > > me
>> > > > > > > > > > > the paths of the files you are trying to read
and what
>> > data
>> > > > > > > > > > > is
>> > > > > > > > > contained
>> > > > > > > > > > in
>> > > > > > > > > > > them so I can give you suggestions on what to
set for
>> the
>> > > > > > > > > configuration.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > > George
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie Prestopnik
via
>> RT <
>> > > > > > > > > > met_help at ucar.edu>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was
acted
>> upon.
>> > > > > > > > > > > > Transaction: Given to mccabe (George McCabe)
by
>> jpresto
>> > > > > > > > > > > > Queue: met_help
>> > > > > > > > > > > > Subject: How to set time for hourly 12-
24 hours
>> > data
>> > > > > > > > > > > > ?
>> > > > > > > > > > > > Owner: mccabe
>> > > > > > > > > > > > Requestors: zhuming at ucar.edu
>> > > > > > > > > > > > Status: new
>> > > > > > > > > > > > Ticket <URL:
>> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>> > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > This transaction appears to have no content
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > > George McCabe - Software Engineer III
>> > > > > > > > > > > National Center for Atmospheric Research
>> > > > > > > > > > > Research Applications Laboratory
>> > > > > > > > > > > 303-497-2768
>> > > > > > > > > > > ---
>> > > > > > > > > > > My working day may not be your working day.
Please do
>> not
>> > > > > > > > > > > feel
>> > > > > > > > obliged
>> > > > > > > > > to
>> > > > > > > > > > > reply to this email outside of your normal
working
>> hours.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > George McCabe - Software Engineer III
>> > > > > > > > > National Center for Atmospheric Research
>> > > > > > > > > Research Applications Laboratory
>> > > > > > > > > 303-497-2768
>> > > > > > > > > ---
>> > > > > > > > > My working day may not be your working day. Please
do not
>> > feel
>> > > > > > obliged
>> > > > > > > to
>> > > > > > > > > reply to this email outside of your normal working
hours.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > George McCabe - Software Engineer III
>> > > > > > > National Center for Atmospheric Research
>> > > > > > > Research Applications Laboratory
>> > > > > > > 303-497-2768
>> > > > > > > ---
>> > > > > > > My working day may not be your working day. Please do
not feel
>> > > > > > > obliged to
>> > > > > > > reply to this email outside of your normal working
hours.
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> > --
>> > George McCabe - Software Engineer III
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > 303-497-2768
>> > ---
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>> >
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: zhuming at ucar.edu
Time: Thu Apr 22 09:52:34 2021
Hi George,
My verification result shows that there is a big error on a non-
standard
level, so my question is:
Is the forecast grid match to Obs, or obs station match to Forecast? I
need
to do is the forecast grid interpolated to the obs location and
vertical
levels. So how to set the levels?
Thanks,
Zhuming
FCST_VAR3_NAME = v_10m_tr
FCST_VAR3_LEVELS = (0,*,*)
FCST_VAR3_THRESH = >=5, <5
OBS_VAR3_NAME = VGRD
OBS_VAR3_LEVELS = Z10
OBS_VAR3_THRESH = >=5, <5
OBS_VAR3_OPTIONS = message_type = ["ADPSFC"];
FCST_VAR3_NAME = v_tr_p
FCST_VAR3_LEVELS =
(0,0,*,*),(0,1,*,*),(0,2,*,*),(0,3,*,*),(0,4,*,*),(0,5,*,*),(0,6,*,*),(0,7,*,*),(0,8,*,*),(0,9,*,*)
FCST_VAR3_THRESH = >=5, <5
OBS_VAR3_NAME = VGRD
OBS_VAR3_LEVELS =
P900-950,P825-875,P675-725,P575-625,P475-525,P375-425,P275-325,P225-
275,P175-225,P75-125
OBS_VAR3_THRESH = >=5, <5
OBS_VAR3_OPTIONS = message_type = ["ADPUPA"];
On Thu, Apr 15, 2021 at 1:52 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Zhuming,
>
> There was indeed a bug in the code. Howard has fixed the issue with
this
> pull request: https://github.com/dtcenter/MET/pull/1756
> This fix will be included in the MET 10.0.0-beta5 release that is
> tentatively scheduled for next week. This beta version can be
installed on
> cheyenne so you can test that the update fixes your issue. I will
follow up
> and let you know when the beta release is available on cheyenne.
>
> Thanks,
> George
>
> On Wed, Apr 14, 2021 at 5:28 PM George McCabe <mccabe at ucar.edu>
wrote:
>
> > Hi Zhuming,
> >
> > I tried to get this to work on cheyenne with your data, but was
not
> > successful. I am able to read the data with plot_data_plane and
generate
> an
> > image, so I think there may be a bug in the code. I have forwarded
info
> to
> > one of our C++ developers to look into it. I will report back when
I know
> > more.
> >
> > Thanks,
> > George
> >
> > On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT <
> met_help at ucar.edu>
> > wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >>
> >> Hi George,
> >>
> >> It still can't find the correct time. I tried
> >>
> >> FCST_VAR1_LEVELS = (*,*,*)
> >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
> >>
> >> Thanks!
> >>
> >> Zhuming
> >>
> >> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT
<met_help at ucar.edu
> >
> >> wrote:
> >>
> >> > Hi Zhuming,
> >> >
> >> > It looks like you are requesting the first time in the 12 hour
file
> for
> >> > each run:
> >> >
> >> > FCST_VAR1_LEVELS = (0,*,*)
> >> > ...
> >> > FCST_VAR2_LEVELS = (0,*,*)
> >> >
> >> > The first dimension in these files is the time offset. You will
need
> the
> >> > correct index that corresponds to each valid time. Fortunately,
MET
> >> makes
> >> > this easy by allowing you to specify the valid time for a time
> dimension
> >> > explicitly and it will find the correct data. You can set:
> >> >
> >> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> >> > ...
> >> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> >> >
> >> > And it will substitute the valid time for each run. Let me know
if you
> >> have
> >> > any questions.
> >> >
> >> > Thanks,
> >> > George
> >> >
> >> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
> >> met_help at ucar.edu
> >> > >
> >> > wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> >> > >
> >> > > Hi George,
> >> > >
> >> > > Thanks for your email, I just would like to send you for
help.
> >> > >
> >> > > I already resolved the last problem with the WRF forecast,
thank you
> >> for
> >> > > your help.
> >> > >
> >> > > I also need to verify the MPAS model forecast. These 12 hour
> >> > > MAPS forecasts are stored in a one file, such as
> >> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> >> > > 2015071512/mpas_wrf_cf_2015071512.nc
> >> > >
> >> > > It seems MET is doing fine on the first 2 hours, but no
matched
> pairs
> >> for
> >> > > later hours.
> >> > > I'm not sure what's wrong with it? the time is not matched?
> >> > >
> >> > > see the log file
> >> > >
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> >> > > config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> >> > >
> >> > > Thanks,
> >> > > Zhuming
> >> > >
> >> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
> >> met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > > Hi Zhuming,
> >> > > >
> >> > > > Just following up on this ticket. Were you able to get this
> >> resolved?
> >> > Let
> >> > > > me know if you need more assistance.
> >> > > >
> >> > > > Thanks,
> >> > > > George
> >> > > >
> >> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> >> > > > > Hi Zhuming,
> >> > > > >
> >> > > > > Typically directories that contain data for an entire day
are
> >> named
> >> > > > > with
> >> > > > > YYYYMMDD, so your filename template tag would be
> >> {valid?fmt=%Y%m%d},
> >> > > > > however that may not always be the case, as you are
seeing here.
> >> > There
> >> > > > > are
> >> > > > > a couple ways to handle this.
> >> > > > >
> >> > > > > If you know that the directory will always be YYYYMMDD
with 00
> at
> >> the
> >> > > > > end,
> >> > > > > you can simply put 00 in the template instead of adding
%H in
> the
> >> > tag,
> >> > > > > i.e.
> >> > > > >
> >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> >> > > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> >> 00_cf.nc
> >> > > > >
> >> > > > > If the HH value can vary, for instance if there are 2
> directories
> >> > > > > created
> >> > > > > per day (i.e. 2015071400 and 2015071412), then you can
use the
> >> > > > > truncate
> >> > > > > keyword to handle it.
> >> > > > >
> >> > > > > If directories are created every 24 hours:
> >> > > > >
> >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> >> > > > >
{valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> >> > > > > %d_%H}:00:
> >> > > > > 00_cf.nc
> >> > > > >
> >> > > > > If directories are created every 12 hours:
> >> > > > >
> >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> >> > > > >
{valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> >> > > > > %d_%H}:00:
> >> > > > > 00_cf.nc
> >> > > > >
> >> > > > > Let me know if that doesn't work for you.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > George
> >> > > > >
> >> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> >> > > > > <met_help at ucar.edu>
> >> > > > > wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >
> >> > > > > >
> >> > > > > > Hi George,
> >> > > > > >
> >> > > > > > Processing Obs data using "PB2NC" are right now.
> >> > > > > >
> >> > > > > > Now I'm using "PointStat" , I set VALID_BEG =
2015071500,
> >> > VALID_END =
> >> > > > > > 2015071501. It looks for the forecast data 2015071500/
> >> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> >> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> >> > > > > > But my forecast data are all put in dir of 2015071500/.
> >> > > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to
get the
> >> right
> >> > > > > > fcst
> >> > > > > > data?
> >> > > > > >
> >> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> >> > > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> >> > 00_cf.nc
> >> > > > > >
> >> > > > > >
> >> > > > > > config file:
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> >> > > > > >
> >> > > > > > log file:
> >> > > > > >
> >> > >
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> >> > > > > >
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > >
> >> > > > > > Zhuming
> >> > > > > >
> >> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> >> > > > > > <met_help at ucar.edu>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Zhuming,
> >> > > > > > >
> >> > > > > > > Looking at the log output, it looks like the data is
stored
> in
> >> > the
> >> > > > > > prepBUFR
> >> > > > > > > files differently that you may have thought:
> >> > > > > > >
> >> > > > > > > DEBUG 2: PrepBufr Time Center:
20150715_000000
> >> > > > > > > DEBUG 2: Searching Time Window:
20150715_030000 to
> >> > > > > > 20150715_040000
> >> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65%
70% 75%
> 80%
> >> > 85%
> >> > > > > > > 90%
> >> > > > > > 95%
> >> > > > > > > 100%
> >> > > > > > >
> >> > > > > > > DEBUG 2: Total Messages processed =
590836
> >> > > > > > > DEBUG 2: Rejected based on message type = 0
> >> > > > > > > DEBUG 2: Rejected based on station id = 0
> >> > > > > > > DEBUG 2: Rejected based on valid time =
590815
> >> > > > > > > DEBUG 2: Rejected based on masking grid = 20
> >> > > > > > > DEBUG 2: Rejected based on masking polygon = 0
> >> > > > > > > DEBUG 2: Rejected based on elevation = 0
> >> > > > > > > DEBUG 2: Rejected based on pb report type = 1
> >> > > > > > > DEBUG 2: Rejected based on input report type = 0
> >> > > > > > > DEBUG 2: Rejected based on instrument type = 0
> >> > > > > > > DEBUG 2: Rejected based on zero observations = 0
> >> > > > > > > DEBUG 2: Total Messages retained = 0
> >> > > > > > > DEBUG 2: Total observations retained or derived = 0
> >> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
> >> 20150715_030000*
> >> > > > > > > WARNING:
> >> > > > > > > WARNING: process_pbfile() -> No PrepBufr messages
retained
> >> from
> >> > > > > > > file:
> >> > > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> >> > > > > > > prepbufr.gdas.2015071500.nr
> >> > > > > > > WARNING:
> >> > > > > > > ERROR :
> >> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr
messages
> >> > retained.
> >> > > > > > Nothing
> >> > > > > > > to write.
> >> > > > > > > ERROR :
> >> > > > > > >
> >> > > > > > > The prepbufr.gdas.2015071500.nr file actually
contains data
> >> from
> >> > > > > > > 20150714_210000 to 20150715_030000, not
20150715_000000 to
> >> > > > > > 20150715_060000.
> >> > > > > > >
> >> > > > > > > To get the correct file for each run, you will have
to shift
> >> the
> >> > > > > > > valid
> >> > > > > > time
> >> > > > > > > before truncating it so that it gets the correct file
for
> each
> >> > run.
> >> > > > > > Instead
> >> > > > > > > of setting:
> >> > > > > > >
> >> > > > > > > PB2NC_INPUT_TEMPLATE =
> >> > > > > > >
> >> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> >> > > > > > >
> >> > > > > > > You will have to also shift forward 3 hours, i.e.:
> >> > > > > > >
> >> > > > > > > PB2NC_INPUT_TEMPLATE =
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> >> > > > > > >
> >> > > > > > > This says to add 3 hours to the current valid time
before
> >> > > > > > > truncating to
> >> > > > > > the
> >> > > > > > > nearest 6 hours.
> >> > > > > > >
> >> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> >> > > > > > > File: prepbufr.gdas.2015071500.nr
> >> > > > > > >
> >> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> >> > > > > > > File: prepbufr.gdas.2015071500.nr
> >> > > > > > >
> >> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> >> > > > > > > File: prepbufr.gdas.2015071500.nr
> >> > > > > > >
> >> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> >> > > > > > > File: prepbufr.gdas.2015071506.nr
> >> > > > > > >
> >> > > > > > > Let me know if that doesn't work.
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > George
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via
RT <
> >> > > > > > met_help at ucar.edu>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > >
> >> > > > > > > > <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >> > >
> >> > > > > > > >
> >> > > > > > > > Hi George,
> >> > > > > > > >
> >> > > > > > > > Now makes me confused is why processed obs of
several
> times
> >> are
> >> > > > > > missing.
> >> > > > > > > > There has files of time at 00,01,02, but no
03,04,05; has
> >> > > > > > > > 06,07,08, but
> >> > > > > > > no
> >> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has 18,19,20,
but
> with
> >> > > > > > > > 21,no
> >> > > > > > 22,23.
> >> > > > > > > > From yours and mines:
> >> > > > > > > >
> >> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> >> > pbs.2015071500.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> >> > pbs.2015071501.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> >> > pbs.2015071502.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> >> > pbs.2015071506.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> >> > pbs.2015071507.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> >> > pbs.2015071508.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> >> > pbs.2015071512.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> >> > pbs.2015071513.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> >> > pbs.2015071514.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> >> > pbs.2015071515.nc
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> >> > > > > > > >
> >> > > > > > > > total 96112
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> >> > > pbs.2015071500.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> >> > > pbs.2015071501.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> >> > > pbs.2015071502.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> >> > > pbs.2015071506.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> >> > > pbs.2015071507.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> >> > > pbs.2015071508.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> >> > > pbs.2015071512.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> >> > > pbs.2015071513.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> >> > > pbs.2015071514.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> >> > > pbs.2015071518.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> >> > > pbs.2015071519.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> >> > > pbs.2015071520.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> >> > > pbs.2015071521.nc
> >> > > > > > > >
> >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> >> > > pbs.2015071600.nc
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > >
> >> > > > > > > > Zhuming
> >> > > > > > > >
> >> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via RT
> >> > > > > > > > <met_help at ucar.edu
> >> > > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Zhuming,
> >> > > > > > > > >
> >> > > > > > > > > Thanks for the information. You will need to use
the
> >> truncate
> >> > > > > > > > > keyword
> >> > > > > > > to
> >> > > > > > > > > find the files, then set the valid beg/end range
to 1
> >> hour.
> >> > > > > > > > > This is
> >> > > > > > > what
> >> > > > > > > > I
> >> > > > > > > > > changed from your PB2NC1.conf config file:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *PB2NC_VALID_BEGIN =
> {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
> >> =
> >> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> >> > > > > > > > >
> >> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> >> > > > > > > > >
> >> > > >
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> >> > > > > > > > >
> >> > > > > > > > > My config file is on cheyenne here:
> >> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> >> > > > > > > > >
> >> > > > > > > > > I just started running it and the output is being
> written
> >> > here:
> >> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > For PointStat, you will want to loop over the
same
> times.
> >> I
> >> > > > > > > > > noticed
> >> > > > > > you
> >> > > > > > > > > specify message type in the VAR<n>_OPTIONS, but
you do
> not
> >> > need
> >> > > > > > > > > to do
> >> > > > > > > > this
> >> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE variable
that
> >> > defines
> >> > > > > > > > > this.
> >> > > > > > > This
> >> > > > > > > > > is what I changed in your PointStat1.conf file:
> >> > > > > > > > >
> >> > > > > > > > > *VALID_END = 2015071600*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> >> > > > > > =POINT_STAT_OBS_VALID_BEG
> >> > > > > > > > =*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> >> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> >> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
> >> message_type =
> >> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> >> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> >> > > > > > > > >
> >> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR = {OUTPUT_BASE}/pb2nc1*
> >> > > > > > > > >
> >> > > > > > > > > Or you can view my file in the same directory. I
usually
> >> try
> >> > to
> >> > > > > > > > > test
> >> > > > > > > > with a
> >> > > > > > > > > single run time to make sure it is working
properly, so
> I
> >> set
> >> > > > > > VALID_END
> >> > > > > > > > > back to 2015071500. I ran it and it produced an
output
> >> file,
> >> > > > > > > > > but
> >> > > > > > there
> >> > > > > > > > were
> >> > > > > > > > > no values besides the header. I checked the log
file and
> >> saw
> >> > > > > > > > > this:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus TMP/Z2,
for
> >> > > > > > > > > observation type
> >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> >> BILIN(4),
> >> > > > > > > > > using 0
> >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> 0DEBUG
> >> 3:
> >> > > > > > > Observations
> >> > > > > > > > > processed = 393373DEBUG 3: Rejected: station
id
> =
> >> > > > > > > > > 0DEBUG 3:
> >> > > > > > > > > Rejected: obs type = 390528DEBUG 3:
Rejected:
> valid
> >> > time
> >> > > > > > =
> >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
> Rejected:
> >> off
> >> > > > > > > > > the
> >> > > > > > grid
> >> > > > > > > > =
> >> > > > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG
3:
> >> Rejected:
> >> > > > > > > > > level
> >> > > > > > > > mismatch
> >> > > > > > > > > = 259DEBUG 3: Rejected: quality marker = 0DEBUG
3:
> >> Rejected:
> >> > > > > > message
> >> > > > > > > > type
> >> > > > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG 3:
> >> Rejected:
> >> > bad
> >> > > > > > fcst
> >> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG 3:
> >> > > > > > > > > Rejected:
> >> > > > > > bad
> >> > > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates
=
> 0DEBUG
> >> > > > > > > > > 2:DEBUG 2:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
--------------------------------------------------------------------------------DEBUG
> >> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus
UGRD/Z10,
> for
> >> > > > > > observation
> >> > > > > > > > type
> >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> >> BILIN(4),
> >> > > > > > > > > using 0
> >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> 0DEBUG
> >> 3:
> >> > > > > > > Observations
> >> > > > > > > > > processed = 393373*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
> Rejected:
> >> obs
> >> > > > > > > > > type
> >> > > > > > > > =
> >> > > > > > > > > 382965DEBUG 3: Rejected: valid time = 0DEBUG
3:
> >> > Rejected:
> >> > > > > > > > > bad
> >> > > > > > obs
> >> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid =
> 9240DEBUG
> >> 3:
> >> > > > > > Rejected:
> >> > > > > > > > > topography = 0DEBUG 3: Rejected: level
mismatch =
> >> > > > > > > > > 1168DEBUG 3:
> >> > > > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected:
message
> >> type
> >> > > > > > > > > =
> >> > > > > > > 0DEBUG
> >> > > > > > > > 3:
> >> > > > > > > > > Rejected: masking region = 0DEBUG 3: Rejected:
bad fcst
> >> > value
> >> > > > > > > > > =
> >> > > > > > > 0DEBUG
> >> > > > > > > > 3:
> >> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected:
bad
> climo
> >> > stdev
> >> > > > > > > > > =
> >> > > > > > > 0DEBUG
> >> > > > > > > > 3:
> >> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
--------------------------------------------------------------------------------DEBUG
> >> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus
VGRD/Z10,
> for
> >> > > > > > observation
> >> > > > > > > > type
> >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> >> BILIN(4),
> >> > > > > > > > > using 0
> >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> 0DEBUG
> >> 3:
> >> > > > > > > Observations
> >> > > > > > > > > processed = 393373DEBUG 3: Rejected: station
id
> =
> >> > > > > > > > > 0DEBUG 3:
> >> > > > > > > > > Rejected: obs type = 382965DEBUG 3:
Rejected:
> valid
> >> > time
> >> > > > > > =
> >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
> Rejected:
> >> off
> >> > > > > > > > > the
> >> > > > > > grid
> >> > > > > > > > =
> >> > > > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG
3:
> >> Rejected:
> >> > > > > > > > > level
> >> > > > > > > > mismatch
> >> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker = 0DEBUG
3:
> >> > Rejected:
> >> > > > > > message
> >> > > > > > > > > type = 0DEBUG 3: Rejected: masking region =
0DEBUG
> 3:
> >> > > > > > > > > Rejected:
> >> > > > > > bad
> >> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo mean
=
> >> 0DEBUG 3:
> >> > > > > > Rejected:
> >> > > > > > > > bad
> >> > > > > > > > > climo stdev = 0*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG 2:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
--------------------------------------------------------------------------------DEBUG
> >> > > > > > > > > 2:DEBUG 1: Output file:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > This describes the reason why each observation
was
> >> rejected.
> >> > > > > > > > > Some
> >> > > > > > > > > observations didn't match the message type you
> requested,
> >> > some
> >> > > > > > > > > were
> >> > > > > > not
> >> > > > > > > > > within the forecast grid, and some did not match
the
> >> vertical
> >> > > > > > > > > level.
> >> > > > > > > You
> >> > > > > > > > > will have to make adjustments to your settings so
that
> the
> >> > data
> >> > > > > > matches
> >> > > > > > > > up
> >> > > > > > > > > properly. Maybe the grid you are using in the
PB2NC
> config
> >> > file
> >> > > > > > > > > does
> >> > > > > > > not
> >> > > > > > > > > match the forecast grid and you are missing
values. If
> >> that
> >> > is
> >> > > > > > > > > the
> >> > > > > > > case,
> >> > > > > > > > > you will have to reconfigure and rerun PB2NC to
get
> >> output on
> >> > > > > > > > > the
> >> > > > > > > correct
> >> > > > > > > > > gr5id. Maybe the observation points don't line up
with
> the
> >> > > > > > > > > vertical
> >> > > > > > > level
> >> > > > > > > > > of the model fields. You can use a range of level
values
> >> for
> >> > > > > > > > > the
> >> > > > > > > > > observation data to allow data within that range,
i.e.
> >> Z0-10.
> >> > > > > > > > >
> >> > > > > > > > > Let me know if you are still having issues
getting this
> to
> >> > work
> >> > > > > > > > > and I
> >> > > > > > > can
> >> > > > > > > > > help out.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > > George
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu
via RT
> <
> >> > > > > > > > met_help at ucar.edu>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Hi George,
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks for your reply!
> >> > > > > > > > > >
> >> > > > > > > > > > I'm still confused, I need to output hourly obs
data
> to
> >> do
> >> > > > > > > > > > hourly
> >> > > > > > > > > > verification.
> >> > > > > > > > > > The original GDAS Prepbufr data has 4 files one
day,
> >> should
> >> > > > > > > > > > with 6
> >> > > > > > > > hours
> >> > > > > > > > > > obs data in one file.
> >> > > > > > > > > >
> >> > > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc (obs
time
> is
> >> > 00?
> >> > > > > > > > > > or 6
> >> > > > > > > times
> >> > > > > > > > > of
> >> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> >> > > > > > > > > >
> >> > > > > > > > > > my config file :
> >> > > > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> >> > > > > > > > > > and
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Zhuming
> >> > > > > > > > > >
> >> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe
via RT <
> >> > > > > > > met_help at ucar.edu
> >> > > > > > > > >
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Zhuming,
> >> > > > > > > > > > >
> >> > > > > > > > > > > To run the tools hourly you would set the
> >> VALID_INCREMENT
> >> > > > > > > > > > > value
> >> > > > > > for
> >> > > > > > > > 1H.
> >> > > > > > > > > > > Here is an example that runs once per hour
for a day
> >> from
> >> > > > > > > 2015-07-15
> >> > > > > > > > 0Z
> >> > > > > > > > > > to
> >> > > > > > > > > > > 2017-07-16 0Z:
> >> > > > > > > > > > >
> >> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> >> > > > > > > > > > >
> >> > > > > > > > > > > VALID_BEG = 2015071500
> >> > > > > > > > > > >
> >> > > > > > > > > > > VALID_END = 2015071600
> >> > > > > > > > > > >
> >> > > > > > > > > > > VALID_INCREMENT = 1H
> >> > > > > > > > > > >
> >> > > > > > > > > > > LEAD_SEQ = 0
> >> > > > > > > > > > >
> >> > > > > > > > > > > Each run time will use the
PB2NC_INPUT_TEMPLATE to
> >> find
> >> > > > > > > > > > > files
> >> > > > > > whose
> >> > > > > > > > > > > filenames match the time you need.
> >> > > > > > > > > > >
> >> > > > > > > > > > > To read prepBUFR observations for 12 hours,
you will
> >> want
> >> > > > > > > > > > > to set
> >> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to
specify the
> >> > range
> >> > > > > > > > > > > around
> >> > > > > > > the
> >> > > > > > > > > > valid
> >> > > > > > > > > > > time to process. These values depend on what
is
> >> contained
> >> > > > > > > > > > > inside
> >> > > > > > > your
> >> > > > > > > > > > > prepBUFR files. If you want to use data from
the
> valid
> >> > time
> >> > > > > > > > > > > of
> >> > > > > > the
> >> > > > > > > > > > filename
> >> > > > > > > > > > > backwards 12 hours, you would set:
> >> > > > > > > > > > >
> >> > > > > > > > > > > PB2NC_VALID_BEGIN =
{valid?fmt=%Y%m%d_%H?shift=-12H
> }
> >> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> >> > > > > > > > > > >
> >> > > > > > > > > > > For a runtime of 2015071512, this would be
> substituted
> >> > as:
> >> > > > > > > > > > >
> >> > > > > > > > > > > valid_beg: 20150715_00
> >> > > > > > > > > > > valid_end: 20150715_12
> >> > > > > > > > > > >
> >> > > > > > > > > > > If you want to create an output file for each
hour
> in
> >> > PB2NC
> >> > > > > > > > > > > you
> >> > > > > > > would
> >> > > > > > > > > > need
> >> > > > > > > > > > > to make sure that the hour is specified in
the
> >> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> >> > > > > > > > > > >
> >> > > > > > > > > > > PointStat has similar config variables to set
the
> >> valid
> >> > beg
> >> > > > > > > > > > > and
> >> > > > > > end
> >> > > > > > > > > range
> >> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> >> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> >> > > > > > > > > > >
> >> > > > > > > > > > > If your prepBUFR files are separated out into
> multiple
> >> > > > > > > > > > > files that
> >> > > > > > > > need
> >> > > > > > > > > to
> >> > > > > > > > > > > be read into PB2NC, you can set a file window
around
> >> the
> >> > > > > > > > > > > valid
> >> > > > > > time
> >> > > > > > > > to
> >> > > > > > > > > > > allow files to be passed into the app. More
> >> information
> >> > on
> >> > > > > > > > > > > that
> >> > > > > > > > > > > functionality is here:
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> >> > > > > > windows-to-find-valid-files
> >> > > > > > > > > > >
> >> > > > > > > > > > > If you are still having trouble getting this
> >> configured,
> >> > > > > > > > > > > you
> >> > > > > > could
> >> > > > > > > > send
> >> > > > > > > > > > me
> >> > > > > > > > > > > the paths of the files you are trying to read
and
> what
> >> > data
> >> > > > > > > > > > > is
> >> > > > > > > > > contained
> >> > > > > > > > > > in
> >> > > > > > > > > > > them so I can give you suggestions on what to
set
> for
> >> the
> >> > > > > > > > > configuration.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks,
> >> > > > > > > > > > > George
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie
Prestopnik via
> >> RT <
> >> > > > > > > > > > met_help at ucar.edu>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429 was
acted
> >> upon.
> >> > > > > > > > > > > > Transaction: Given to mccabe (George
McCabe) by
> >> jpresto
> >> > > > > > > > > > > > Queue: met_help
> >> > > > > > > > > > > > Subject: How to set time for hourly
12-24
> hours
> >> > data
> >> > > > > > > > > > > > ?
> >> > > > > > > > > > > > Owner: mccabe
> >> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> >> > > > > > > > > > > > Status: new
> >> > > > > > > > > > > > Ticket <URL:
> >> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> >> > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > This transaction appears to have no content
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > > George McCabe - Software Engineer III
> >> > > > > > > > > > > National Center for Atmospheric Research
> >> > > > > > > > > > > Research Applications Laboratory
> >> > > > > > > > > > > 303-497-2768
> >> > > > > > > > > > > ---
> >> > > > > > > > > > > My working day may not be your working day.
Please
> do
> >> not
> >> > > > > > > > > > > feel
> >> > > > > > > > obliged
> >> > > > > > > > > to
> >> > > > > > > > > > > reply to this email outside of your normal
working
> >> hours.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > > George McCabe - Software Engineer III
> >> > > > > > > > > National Center for Atmospheric Research
> >> > > > > > > > > Research Applications Laboratory
> >> > > > > > > > > 303-497-2768
> >> > > > > > > > > ---
> >> > > > > > > > > My working day may not be your working day.
Please do
> not
> >> > feel
> >> > > > > > obliged
> >> > > > > > > to
> >> > > > > > > > > reply to this email outside of your normal
working
> hours.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > George McCabe - Software Engineer III
> >> > > > > > > National Center for Atmospheric Research
> >> > > > > > > Research Applications Laboratory
> >> > > > > > > 303-497-2768
> >> > > > > > > ---
> >> > > > > > > My working day may not be your working day. Please do
not
> feel
> >> > > > > > > obliged to
> >> > > > > > > reply to this email outside of your normal working
hours.
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > George McCabe - Software Engineer III
> >> > National Center for Atmospheric Research
> >> > Research Applications Laboratory
> >> > 303-497-2768
> >> > ---
> >> > My working day may not be your working day. Please do not feel
obliged
> >> to
> >> > reply to this email outside of your normal working hours.
> >> >
> >> >
> >>
> >>
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: John Halley Gotway
Time: Thu Apr 22 11:40:23 2021
Zhuming,
This is John HG. George asked me to take a look at this question.
Since
you're asking about "non-standard" levels, I assume you're asking
about the
verification of the pressure levels against the ADPUPA message type.
In general, the logic is this...
For each point observation, loop through all the verification tasks
and see
if it should be used.
For example, let's talk about the 4 level entry for VAR3:
FCST_VAR3_NAME = v_tr_p
FCST_VAR3_LEVELS = ...(0,3,*,*)...
OBS_VAR3_NAME = VGRD
OBS_VAR3_LEVELS = ...P575-625...
The point observation will be used here if it's a "VGRD" observation
and
has a level value between 575 and 625. This verification task has been
configured to only verify a single forecast level of data:
v_tr_p(0,3,*,*).
So no vertical interpolation is done. MET interpolates horizontally
from
the forecast grid points to the observation lat/lon location (using
user-configurable interpolation options).
To put it simply, all obs between 575 and 625 mb are compared directly
to
the forecast values from v_tr_p(0,3,*,*). If the model output is
actually
on pressure levels, then that's a reasonable thing to do. If the
model's
vertical coordinate is something other than pressure levels, then
that's a
potential source for large errors.
Note that MET does do vertical interpolation when processing the GRIB
output from the Unified Post Processor tool. But all the details of
that
depend on how the configuration file is setup. I'll explain below.
To date we have avoided including direct knowledge about the vertical
coordinate of each modelling system in the verification code. The goal
is
to create verification tools applicable to many models, rather than
customizing them for each. We do realize that desire for the latter
and are
exploring ways to pull data from the IODA data assimilation system
instead,
which necessarily knows a lot about each model.
So let's say you were using GRIB data, and configured it like this:
VAR3_NAME = VGRD
VAR3_LEVELS = P500-1000
Since we haven't prefixed these options with FCST_ and OBS_, the same
settings are applied to both.
MET would read ALL forecast GRIB records from the model file with
pressures
between 500 and 1000. Perhaps, that's every 100mb or perhaps every
50mb? It
reads whatever data falls in that range. For each point observation
with a
pressure level between 500 and 1000, it'll first find the actual model
pressure level above/below it. It'll interpolate the forecast
data horizontally to the obs lat/lon location for both. And then it'll
interpolate those values above/below vertically to the actual
observation
level, linear in the log of pressure.
This issue is that the processing of pressure LAYERS is not currently
supported for NetCDF data I believe. And the code could be enhanced to
support that. But again, this only makes sense if the model's vertical
coordinate is actually pressure levels and not something else.
Hope that helps clarify.
Thanks,
John
On Thu, Apr 22, 2021 at 9:52 AM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Hi George,
>
> My verification result shows that there is a big error on a non-
standard
> level, so my question is:
> Is the forecast grid match to Obs, or obs station match to Forecast?
I need
> to do is the forecast grid interpolated to the obs location and
vertical
> levels. So how to set the levels?
>
> Thanks,
> Zhuming
>
> FCST_VAR3_NAME = v_10m_tr
>
> FCST_VAR3_LEVELS = (0,*,*)
>
> FCST_VAR3_THRESH = >=5, <5
>
> OBS_VAR3_NAME = VGRD
>
> OBS_VAR3_LEVELS = Z10
>
> OBS_VAR3_THRESH = >=5, <5
>
> OBS_VAR3_OPTIONS = message_type = ["ADPSFC"];
>
>
> FCST_VAR3_NAME = v_tr_p
>
> FCST_VAR3_LEVELS =
>
>
(0,0,*,*),(0,1,*,*),(0,2,*,*),(0,3,*,*),(0,4,*,*),(0,5,*,*),(0,6,*,*),(0,7,*,*),(0,8,*,*),(0,9,*,*)
>
> FCST_VAR3_THRESH = >=5, <5
>
> OBS_VAR3_NAME = VGRD
>
> OBS_VAR3_LEVELS =
>
> P900-950,P825-875,P675-725,P575-625,P475-525,P375-425,P275-325,P225-
275,P175-225,P75-125
>
> OBS_VAR3_THRESH = >=5, <5
>
> OBS_VAR3_OPTIONS = message_type = ["ADPUPA"];
>
>
>
> On Thu, Apr 15, 2021 at 1:52 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Zhuming,
> >
> > There was indeed a bug in the code. Howard has fixed the issue
with this
> > pull request: https://github.com/dtcenter/MET/pull/1756
> > This fix will be included in the MET 10.0.0-beta5 release that is
> > tentatively scheduled for next week. This beta version can be
installed
> on
> > cheyenne so you can test that the update fixes your issue. I will
follow
> up
> > and let you know when the beta release is available on cheyenne.
> >
> > Thanks,
> > George
> >
> > On Wed, Apr 14, 2021 at 5:28 PM George McCabe <mccabe at ucar.edu>
wrote:
> >
> > > Hi Zhuming,
> > >
> > > I tried to get this to work on cheyenne with your data, but was
not
> > > successful. I am able to read the data with plot_data_plane and
> generate
> > an
> > > image, so I think there may be a bug in the code. I have
forwarded info
> > to
> > > one of our C++ developers to look into it. I will report back
when I
> know
> > > more.
> > >
> > > Thanks,
> > > George
> > >
> > > On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >>
> > >> Hi George,
> > >>
> > >> It still can't find the correct time. I tried
> > >>
> > >> FCST_VAR1_LEVELS = (*,*,*)
> > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
> > >>
> > >> Thanks!
> > >>
> > >> Zhuming
> > >>
> > >> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > >> wrote:
> > >>
> > >> > Hi Zhuming,
> > >> >
> > >> > It looks like you are requesting the first time in the 12
hour file
> > for
> > >> > each run:
> > >> >
> > >> > FCST_VAR1_LEVELS = (0,*,*)
> > >> > ...
> > >> > FCST_VAR2_LEVELS = (0,*,*)
> > >> >
> > >> > The first dimension in these files is the time offset. You
will need
> > the
> > >> > correct index that corresponds to each valid time.
Fortunately, MET
> > >> makes
> > >> > this easy by allowing you to specify the valid time for a
time
> > dimension
> > >> > explicitly and it will find the correct data. You can set:
> > >> >
> > >> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > >> > ...
> > >> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > >> >
> > >> > And it will substitute the valid time for each run. Let me
know if
> you
> > >> have
> > >> > any questions.
> > >> >
> > >> > Thanks,
> > >> > George
> > >> >
> > >> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
> > >> met_help at ucar.edu
> > >> > >
> > >> > wrote:
> > >> >
> > >> > >
> > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > >> > >
> > >> > > Hi George,
> > >> > >
> > >> > > Thanks for your email, I just would like to send you for
help.
> > >> > >
> > >> > > I already resolved the last problem with the WRF forecast,
thank
> you
> > >> for
> > >> > > your help.
> > >> > >
> > >> > > I also need to verify the MPAS model forecast. These 12
hour
> > >> > > MAPS forecasts are stored in a one file, such as
> > >> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> > >> > > 2015071512/mpas_wrf_cf_2015071512.nc
> > >> > >
> > >> > > It seems MET is doing fine on the first 2 hours, but no
matched
> > pairs
> > >> for
> > >> > > later hours.
> > >> > > I'm not sure what's wrong with it? the time is not matched?
> > >> > >
> > >> > > see the log file
> > >> > >
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > >> > > config file /gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> > >> > >
> > >> > > Thanks,
> > >> > > Zhuming
> > >> > >
> > >> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
> > >> met_help at ucar.edu>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Zhuming,
> > >> > > >
> > >> > > > Just following up on this ticket. Were you able to get
this
> > >> resolved?
> > >> > Let
> > >> > > > me know if you need more assistance.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > George
> > >> > > >
> > >> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > >> > > > > Hi Zhuming,
> > >> > > > >
> > >> > > > > Typically directories that contain data for an entire
day are
> > >> named
> > >> > > > > with
> > >> > > > > YYYYMMDD, so your filename template tag would be
> > >> {valid?fmt=%Y%m%d},
> > >> > > > > however that may not always be the case, as you are
seeing
> here.
> > >> > There
> > >> > > > > are
> > >> > > > > a couple ways to handle this.
> > >> > > > >
> > >> > > > > If you know that the directory will always be YYYYMMDD
with 00
> > at
> > >> the
> > >> > > > > end,
> > >> > > > > you can simply put 00 in the template instead of adding
%H in
> > the
> > >> > tag,
> > >> > > > > i.e.
> > >> > > > >
> > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > >> > > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> > >> 00_cf.nc
> > >> > > > >
> > >> > > > > If the HH value can vary, for instance if there are 2
> > directories
> > >> > > > > created
> > >> > > > > per day (i.e. 2015071400 and 2015071412), then you can
use
> the
> > >> > > > > truncate
> > >> > > > > keyword to handle it.
> > >> > > > >
> > >> > > > > If directories are created every 24 hours:
> > >> > > > >
> > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > >> > > > >
{valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > >> > > > > %d_%H}:00:
> > >> > > > > 00_cf.nc
> > >> > > > >
> > >> > > > > If directories are created every 12 hours:
> > >> > > > >
> > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > >> > > > >
{valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > >> > > > > %d_%H}:00:
> > >> > > > > 00_cf.nc
> > >> > > > >
> > >> > > > > Let me know if that doesn't work for you.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > George
> > >> > > > >
> > >> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via RT
> > >> > > > > <met_help at ucar.edu>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > >
> > >> > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >
> > >> > > > > >
> > >> > > > > > Hi George,
> > >> > > > > >
> > >> > > > > > Processing Obs data using "PB2NC" are right now.
> > >> > > > > >
> > >> > > > > > Now I'm using "PointStat" , I set VALID_BEG =
2015071500,
> > >> > VALID_END =
> > >> > > > > > 2015071501. It looks for the forecast data
2015071500/
> > >> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and 2015071501/
> > >> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > >> > > > > > But my forecast data are all put in dir of
2015071500/.
> > >> > > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE to
get
> the
> > >> right
> > >> > > > > > fcst
> > >> > > > > > data?
> > >> > > > > >
> > >> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > >> > > > > > {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> > >> > 00_cf.nc
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > config file:
> /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > >> > > > > >
> > >> > > > > > log file:
> > >> > > > > >
> > >> > >
> >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > >
> > >> > > > > > Zhuming
> > >> > > > > >
> > >> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > >> > > > > > <met_help at ucar.edu>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Zhuming,
> > >> > > > > > >
> > >> > > > > > > Looking at the log output, it looks like the data
is
> stored
> > in
> > >> > the
> > >> > > > > > prepBUFR
> > >> > > > > > > files differently that you may have thought:
> > >> > > > > > >
> > >> > > > > > > DEBUG 2: PrepBufr Time Center:
20150715_000000
> > >> > > > > > > DEBUG 2: Searching Time Window:
20150715_030000 to
> > >> > > > > > 20150715_040000
> > >> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65%
70% 75%
> > 80%
> > >> > 85%
> > >> > > > > > > 90%
> > >> > > > > > 95%
> > >> > > > > > > 100%
> > >> > > > > > >
> > >> > > > > > > DEBUG 2: Total Messages processed =
590836
> > >> > > > > > > DEBUG 2: Rejected based on message type = 0
> > >> > > > > > > DEBUG 2: Rejected based on station id = 0
> > >> > > > > > > DEBUG 2: Rejected based on valid time =
590815
> > >> > > > > > > DEBUG 2: Rejected based on masking grid =
20
> > >> > > > > > > DEBUG 2: Rejected based on masking polygon = 0
> > >> > > > > > > DEBUG 2: Rejected based on elevation = 0
> > >> > > > > > > DEBUG 2: Rejected based on pb report type = 1
> > >> > > > > > > DEBUG 2: Rejected based on input report type = 0
> > >> > > > > > > DEBUG 2: Rejected based on instrument type = 0
> > >> > > > > > > DEBUG 2: Rejected based on zero observations = 0
> > >> > > > > > > DEBUG 2: Total Messages retained = 0
> > >> > > > > > > DEBUG 2: Total observations retained or derived = 0
> > >> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
> > >> 20150715_030000*
> > >> > > > > > > WARNING:
> > >> > > > > > > WARNING: process_pbfile() -> No PrepBufr messages
retained
> > >> from
> > >> > > > > > > file:
> > >> > > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > >> > > > > > > prepbufr.gdas.2015071500.nr
> > >> > > > > > > WARNING:
> > >> > > > > > > ERROR :
> > >> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr
messages
> > >> > retained.
> > >> > > > > > Nothing
> > >> > > > > > > to write.
> > >> > > > > > > ERROR :
> > >> > > > > > >
> > >> > > > > > > The prepbufr.gdas.2015071500.nr file actually
contains
> data
> > >> from
> > >> > > > > > > 20150714_210000 to 20150715_030000, not
20150715_000000 to
> > >> > > > > > 20150715_060000.
> > >> > > > > > >
> > >> > > > > > > To get the correct file for each run, you will have
to
> shift
> > >> the
> > >> > > > > > > valid
> > >> > > > > > time
> > >> > > > > > > before truncating it so that it gets the correct
file for
> > each
> > >> > run.
> > >> > > > > > Instead
> > >> > > > > > > of setting:
> > >> > > > > > >
> > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > >> > > > > > >
> > >> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > >> > > > > > >
> > >> > > > > > > You will have to also shift forward 3 hours, i.e.:
> > >> > > > > > >
> > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > >> > > > > > >
> > >> > > > > > > This says to add 3 hours to the current valid time
before
> > >> > > > > > > truncating to
> > >> > > > > > the
> > >> > > > > > > nearest 6 hours.
> > >> > > > > > >
> > >> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > >> > > > > > >
> > >> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > >> > > > > > >
> > >> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > >> > > > > > >
> > >> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > >> > > > > > > File: prepbufr.gdas.2015071506.nr
> > >> > > > > > >
> > >> > > > > > > Let me know if that doesn't work.
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > George
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu via
RT <
> > >> > > > > > met_help at ucar.edu>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > <URL:
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >> > >
> > >> > > > > > > >
> > >> > > > > > > > Hi George,
> > >> > > > > > > >
> > >> > > > > > > > Now makes me confused is why processed obs of
several
> > times
> > >> are
> > >> > > > > > missing.
> > >> > > > > > > > There has files of time at 00,01,02, but no
03,04,05;
> has
> > >> > > > > > > > 06,07,08, but
> > >> > > > > > > no
> > >> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has
18,19,20, but
> > with
> > >> > > > > > > > 21,no
> > >> > > > > > 22,23.
> > >> > > > > > > > From yours and mines:
> > >> > > > > > > >
> > >> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> > >> > pbs.2015071500.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> > >> > pbs.2015071501.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> > >> > pbs.2015071502.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> > >> > pbs.2015071506.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> > >> > pbs.2015071507.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> > >> > pbs.2015071508.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> > >> > pbs.2015071512.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> > >> > pbs.2015071513.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> > >> > pbs.2015071514.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> > >> > pbs.2015071515.nc
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > >> > > > > > > >
> > >> > > > > > > > total 96112
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> > >> > > pbs.2015071500.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> > >> > > pbs.2015071501.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> > >> > > pbs.2015071502.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> > >> > > pbs.2015071506.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> > >> > > pbs.2015071507.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> > >> > > pbs.2015071508.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> > >> > > pbs.2015071512.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> > >> > > pbs.2015071513.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> > >> > > pbs.2015071514.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> > >> > > pbs.2015071518.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> > >> > > pbs.2015071519.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> > >> > > pbs.2015071520.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> > >> > > pbs.2015071521.nc
> > >> > > > > > > >
> > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> > >> > > pbs.2015071600.nc
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > >
> > >> > > > > > > > Zhuming
> > >> > > > > > > >
> > >> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe via
RT
> > >> > > > > > > > <met_help at ucar.edu
> > >> > > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi Zhuming,
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks for the information. You will need to
use the
> > >> truncate
> > >> > > > > > > > > keyword
> > >> > > > > > > to
> > >> > > > > > > > > find the files, then set the valid beg/end
range to 1
> > >> hour.
> > >> > > > > > > > > This is
> > >> > > > > > > what
> > >> > > > > > > > I
> > >> > > > > > > > > changed from your PB2NC1.conf config file:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *PB2NC_VALID_BEGIN =
> > {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
> > >> =
> > >> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > >> > > > > > > > >
> > >> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > >> > > > > > > > >
> > >> > > >
> >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > >> > > > > > > > >
> > >> > > > > > > > > My config file is on cheyenne here:
> > >> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > >> > > > > > > > >
> > >> > > > > > > > > I just started running it and the output is
being
> > written
> > >> > here:
> > >> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > For PointStat, you will want to loop over the
same
> > times.
> > >> I
> > >> > > > > > > > > noticed
> > >> > > > > > you
> > >> > > > > > > > > specify message type in the VAR<n>_OPTIONS, but
you do
> > not
> > >> > need
> > >> > > > > > > > > to do
> > >> > > > > > > > this
> > >> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE
variable that
> > >> > defines
> > >> > > > > > > > > this.
> > >> > > > > > > This
> > >> > > > > > > > > is what I changed in your PointStat1.conf file:
> > >> > > > > > > > >
> > >> > > > > > > > > *VALID_END = 2015071600*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > >> > > > > > =POINT_STAT_OBS_VALID_BEG
> > >> > > > > > > > =*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > >> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > >> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
> > >> message_type =
> > >> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > >> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > >> > > > > > > > >
> > >> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR =
{OUTPUT_BASE}/pb2nc1*
> > >> > > > > > > > >
> > >> > > > > > > > > Or you can view my file in the same directory.
I
> usually
> > >> try
> > >> > to
> > >> > > > > > > > > test
> > >> > > > > > > > with a
> > >> > > > > > > > > single run time to make sure it is working
properly,
> so
> > I
> > >> set
> > >> > > > > > VALID_END
> > >> > > > > > > > > back to 2015071500. I ran it and it produced an
output
> > >> file,
> > >> > > > > > > > > but
> > >> > > > > > there
> > >> > > > > > > > were
> > >> > > > > > > > > no values besides the header. I checked the log
file
> and
> > >> saw
> > >> > > > > > > > > this:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus
TMP/Z2, for
> > >> > > > > > > > > observation type
> > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > >> BILIN(4),
> > >> > > > > > > > > using 0
> > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> > 0DEBUG
> > >> 3:
> > >> > > > > > > Observations
> > >> > > > > > > > > processed = 393373DEBUG 3: Rejected: station
id
> > =
> > >> > > > > > > > > 0DEBUG 3:
> > >> > > > > > > > > Rejected: obs type = 390528DEBUG 3:
Rejected:
> > valid
> > >> > time
> > >> > > > > > =
> > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
> > Rejected:
> > >> off
> > >> > > > > > > > > the
> > >> > > > > > grid
> > >> > > > > > > > =
> > >> > > > > > > > > 2586DEBUG 3: Rejected: topography = 0DEBUG
3:
> > >> Rejected:
> > >> > > > > > > > > level
> > >> > > > > > > > mismatch
> > >> > > > > > > > > = 259DEBUG 3: Rejected: quality marker =
0DEBUG 3:
> > >> Rejected:
> > >> > > > > > message
> > >> > > > > > > > type
> > >> > > > > > > > > = 0DEBUG 3: Rejected: masking region = 0DEBUG
3:
> > >> Rejected:
> > >> > bad
> > >> > > > > > fcst
> > >> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean =
0DEBUG
> 3:
> > >> > > > > > > > > Rejected:
> > >> > > > > > bad
> > >> > > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates
=
> > 0DEBUG
> > >> > > > > > > > > 2:DEBUG 2:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
--------------------------------------------------------------------------------DEBUG
> > >> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus
UGRD/Z10,
> > for
> > >> > > > > > observation
> > >> > > > > > > > type
> > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > >> BILIN(4),
> > >> > > > > > > > > using 0
> > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> > 0DEBUG
> > >> 3:
> > >> > > > > > > Observations
> > >> > > > > > > > > processed = 393373*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG 3:
> > Rejected:
> > >> obs
> > >> > > > > > > > > type
> > >> > > > > > > > =
> > >> > > > > > > > > 382965DEBUG 3: Rejected: valid time =
0DEBUG 3:
> > >> > Rejected:
> > >> > > > > > > > > bad
> > >> > > > > > obs
> > >> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid =
> > 9240DEBUG
> > >> 3:
> > >> > > > > > Rejected:
> > >> > > > > > > > > topography = 0DEBUG 3: Rejected: level
mismatch
> =
> > >> > > > > > > > > 1168DEBUG 3:
> > >> > > > > > > > > Rejected: quality marker = 0DEBUG 3: Rejected:
> message
> > >> type
> > >> > > > > > > > > =
> > >> > > > > > > 0DEBUG
> > >> > > > > > > > 3:
> > >> > > > > > > > > Rejected: masking region = 0DEBUG 3: Rejected:
bad
> fcst
> > >> > value
> > >> > > > > > > > > =
> > >> > > > > > > 0DEBUG
> > >> > > > > > > > 3:
> > >> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3: Rejected:
bad
> > climo
> > >> > stdev
> > >> > > > > > > > > =
> > >> > > > > > > 0DEBUG
> > >> > > > > > > > 3:
> > >> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
--------------------------------------------------------------------------------DEBUG
> > >> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus
VGRD/Z10,
> > for
> > >> > > > > > observation
> > >> > > > > > > > type
> > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > >> BILIN(4),
> > >> > > > > > > > > using 0
> > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched pairs
=
> > 0DEBUG
> > >> 3:
> > >> > > > > > > Observations
> > >> > > > > > > > > processed = 393373DEBUG 3: Rejected: station
id
> > =
> > >> > > > > > > > > 0DEBUG 3:
> > >> > > > > > > > > Rejected: obs type = 382965DEBUG 3:
Rejected:
> > valid
> > >> > time
> > >> > > > > > =
> > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG 3:
> > Rejected:
> > >> off
> > >> > > > > > > > > the
> > >> > > > > > grid
> > >> > > > > > > > =
> > >> > > > > > > > > 9240DEBUG 3: Rejected: topography = 0DEBUG
3:
> > >> Rejected:
> > >> > > > > > > > > level
> > >> > > > > > > > mismatch
> > >> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker =
0DEBUG 3:
> > >> > Rejected:
> > >> > > > > > message
> > >> > > > > > > > > type = 0DEBUG 3: Rejected: masking region =
0DEBUG
> > 3:
> > >> > > > > > > > > Rejected:
> > >> > > > > > bad
> > >> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo
mean =
> > >> 0DEBUG 3:
> > >> > > > > > Rejected:
> > >> > > > > > > > bad
> > >> > > > > > > > > climo stdev = 0*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG
> 2:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
--------------------------------------------------------------------------------DEBUG
> > >> > > > > > > > > 2:DEBUG 1: Output file:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > This describes the reason why each observation
was
> > >> rejected.
> > >> > > > > > > > > Some
> > >> > > > > > > > > observations didn't match the message type you
> > requested,
> > >> > some
> > >> > > > > > > > > were
> > >> > > > > > not
> > >> > > > > > > > > within the forecast grid, and some did not
match the
> > >> vertical
> > >> > > > > > > > > level.
> > >> > > > > > > You
> > >> > > > > > > > > will have to make adjustments to your settings
so that
> > the
> > >> > data
> > >> > > > > > matches
> > >> > > > > > > > up
> > >> > > > > > > > > properly. Maybe the grid you are using in the
PB2NC
> > config
> > >> > file
> > >> > > > > > > > > does
> > >> > > > > > > not
> > >> > > > > > > > > match the forecast grid and you are missing
values. If
> > >> that
> > >> > is
> > >> > > > > > > > > the
> > >> > > > > > > case,
> > >> > > > > > > > > you will have to reconfigure and rerun PB2NC to
get
> > >> output on
> > >> > > > > > > > > the
> > >> > > > > > > correct
> > >> > > > > > > > > gr5id. Maybe the observation points don't line
up with
> > the
> > >> > > > > > > > > vertical
> > >> > > > > > > level
> > >> > > > > > > > > of the model fields. You can use a range of
level
> values
> > >> for
> > >> > > > > > > > > the
> > >> > > > > > > > > observation data to allow data within that
range, i.e.
> > >> Z0-10.
> > >> > > > > > > > >
> > >> > > > > > > > > Let me know if you are still having issues
getting
> this
> > to
> > >> > work
> > >> > > > > > > > > and I
> > >> > > > > > > can
> > >> > > > > > > > > help out.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > > George
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM zhuming at ucar.edu
via
> RT
> > <
> > >> > > > > > > > met_help at ucar.edu>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > <URL:
> > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > Hi George,
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks for your reply!
> > >> > > > > > > > > >
> > >> > > > > > > > > > I'm still confused, I need to output hourly
obs data
> > to
> > >> do
> > >> > > > > > > > > > hourly
> > >> > > > > > > > > > verification.
> > >> > > > > > > > > > The original GDAS Prepbufr data has 4 files
one day,
> > >> should
> > >> > > > > > > > > > with 6
> > >> > > > > > > > hours
> > >> > > > > > > > > > obs data in one file.
> > >> > > > > > > > > >
> > >> > > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc
(obs
> time
> > is
> > >> > 00?
> > >> > > > > > > > > > or 6
> > >> > > > > > > times
> > >> > > > > > > > > of
> > >> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > >> > > > > > > > > >
> > >> > > > > > > > > > my config file :
> > >> > > > > > cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > >> > > > > > > > > > and
/gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Zhuming
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George McCabe
via RT
> <
> > >> > > > > > > met_help at ucar.edu
> > >> > > > > > > > >
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi Zhuming,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > To run the tools hourly you would set the
> > >> VALID_INCREMENT
> > >> > > > > > > > > > > value
> > >> > > > > > for
> > >> > > > > > > > 1H.
> > >> > > > > > > > > > > Here is an example that runs once per hour
for a
> day
> > >> from
> > >> > > > > > > 2015-07-15
> > >> > > > > > > > 0Z
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > 2017-07-16 0Z:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > VALID_BEG = 2015071500
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > VALID_END = 2015071600
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > VALID_INCREMENT = 1H
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > LEAD_SEQ = 0
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Each run time will use the
PB2NC_INPUT_TEMPLATE to
> > >> find
> > >> > > > > > > > > > > files
> > >> > > > > > whose
> > >> > > > > > > > > > > filenames match the time you need.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > To read prepBUFR observations for 12 hours,
you
> will
> > >> want
> > >> > > > > > > > > > > to set
> > >> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to
specify
> the
> > >> > range
> > >> > > > > > > > > > > around
> > >> > > > > > > the
> > >> > > > > > > > > > valid
> > >> > > > > > > > > > > time to process. These values depend on
what is
> > >> contained
> > >> > > > > > > > > > > inside
> > >> > > > > > > your
> > >> > > > > > > > > > > prepBUFR files. If you want to use data
from the
> > valid
> > >> > time
> > >> > > > > > > > > > > of
> > >> > > > > > the
> > >> > > > > > > > > > filename
> > >> > > > > > > > > > > backwards 12 hours, you would set:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > PB2NC_VALID_BEGIN =
> {valid?fmt=%Y%m%d_%H?shift=-12H
> > }
> > >> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > For a runtime of 2015071512, this would be
> > substituted
> > >> > as:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > valid_beg: 20150715_00
> > >> > > > > > > > > > > valid_end: 20150715_12
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > If you want to create an output file for
each hour
> > in
> > >> > PB2NC
> > >> > > > > > > > > > > you
> > >> > > > > > > would
> > >> > > > > > > > > > need
> > >> > > > > > > > > > > to make sure that the hour is specified in
the
> > >> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > PointStat has similar config variables to
set the
> > >> valid
> > >> > beg
> > >> > > > > > > > > > > and
> > >> > > > > > end
> > >> > > > > > > > > range
> > >> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > >> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > If your prepBUFR files are separated out
into
> > multiple
> > >> > > > > > > > > > > files that
> > >> > > > > > > > need
> > >> > > > > > > > > to
> > >> > > > > > > > > > > be read into PB2NC, you can set a file
window
> around
> > >> the
> > >> > > > > > > > > > > valid
> > >> > > > > > time
> > >> > > > > > > > to
> > >> > > > > > > > > > > allow files to be passed into the app. More
> > >> information
> > >> > on
> > >> > > > > > > > > > > that
> > >> > > > > > > > > > > functionality is here:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > >> > > > > > windows-to-find-valid-files
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > If you are still having trouble getting
this
> > >> configured,
> > >> > > > > > > > > > > you
> > >> > > > > > could
> > >> > > > > > > > send
> > >> > > > > > > > > > me
> > >> > > > > > > > > > > the paths of the files you are trying to
read and
> > what
> > >> > data
> > >> > > > > > > > > > > is
> > >> > > > > > > > > contained
> > >> > > > > > > > > > in
> > >> > > > > > > > > > > them so I can give you suggestions on what
to set
> > for
> > >> the
> > >> > > > > > > > > configuration.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > George
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie
Prestopnik
> via
> > >> RT <
> > >> > > > > > > > > > met_help at ucar.edu>
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429
was
> acted
> > >> upon.
> > >> > > > > > > > > > > > Transaction: Given to mccabe (George
McCabe) by
> > >> jpresto
> > >> > > > > > > > > > > > Queue: met_help
> > >> > > > > > > > > > > > Subject: How to set time for hourly
12-24
> > hours
> > >> > data
> > >> > > > > > > > > > > > ?
> > >> > > > > > > > > > > > Owner: mccabe
> > >> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> > >> > > > > > > > > > > > Status: new
> > >> > > > > > > > > > > > Ticket <URL:
> > >> > > > > > > > >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > This transaction appears to have no
content
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > --
> > >> > > > > > > > > > > George McCabe - Software Engineer III
> > >> > > > > > > > > > > National Center for Atmospheric Research
> > >> > > > > > > > > > > Research Applications Laboratory
> > >> > > > > > > > > > > 303-497-2768
> > >> > > > > > > > > > > ---
> > >> > > > > > > > > > > My working day may not be your working day.
Please
> > do
> > >> not
> > >> > > > > > > > > > > feel
> > >> > > > > > > > obliged
> > >> > > > > > > > > to
> > >> > > > > > > > > > > reply to this email outside of your normal
working
> > >> hours.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > --
> > >> > > > > > > > > George McCabe - Software Engineer III
> > >> > > > > > > > > National Center for Atmospheric Research
> > >> > > > > > > > > Research Applications Laboratory
> > >> > > > > > > > > 303-497-2768
> > >> > > > > > > > > ---
> > >> > > > > > > > > My working day may not be your working day.
Please do
> > not
> > >> > feel
> > >> > > > > > obliged
> > >> > > > > > > to
> > >> > > > > > > > > reply to this email outside of your normal
working
> > hours.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > George McCabe - Software Engineer III
> > >> > > > > > > National Center for Atmospheric Research
> > >> > > > > > > Research Applications Laboratory
> > >> > > > > > > 303-497-2768
> > >> > > > > > > ---
> > >> > > > > > > My working day may not be your working day. Please
do not
> > feel
> > >> > > > > > > obliged to
> > >> > > > > > > reply to this email outside of your normal working
hours.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > George McCabe - Software Engineer III
> > >> > National Center for Atmospheric Research
> > >> > Research Applications Laboratory
> > >> > 303-497-2768
> > >> > ---
> > >> > My working day may not be your working day. Please do not
feel
> obliged
> > >> to
> > >> > reply to this email outside of your normal working hours.
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
------------------------------------------------
Subject: How to set time for hourly 12-24 hours data ?
From: George McCabe
Time: Mon May 03 15:47:14 2021
Hi Zhuming,
The METplus 4.0.0-beta5 release is now available on Cheyenne. If you
are
able to test that this version works and fixes some of the issues you
mentioned in this ticket before the 4.0.0 release next week, that
would be
great! Information on how to use the new version can be found here:
https://dtcenter.org/community-code/metplus/metplus-4-0-existing-
builds
(Under NCAR > CHEYENNE > METplus-4.0.0-beta5)
Here are the commands to set it up:
module use /glade/p/ral/jntp/MET/METplus/modulefiles
module load metplus/4.0.0-beta5
ncar_pylib
Please let me know if you have any issues or questions.
Thanks,
George
On Thu, Apr 22, 2021 at 11:40 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
>
> Zhuming,
>
> This is John HG. George asked me to take a look at this question.
Since
> you're asking about "non-standard" levels, I assume you're asking
about the
> verification of the pressure levels against the ADPUPA message type.
>
> In general, the logic is this...
> For each point observation, loop through all the verification tasks
and see
> if it should be used.
> For example, let's talk about the 4 level entry for VAR3:
>
> FCST_VAR3_NAME = v_tr_p
> FCST_VAR3_LEVELS = ...(0,3,*,*)...
> OBS_VAR3_NAME = VGRD
> OBS_VAR3_LEVELS = ...P575-625...
>
> The point observation will be used here if it's a "VGRD" observation
and
> has a level value between 575 and 625. This verification task has
been
> configured to only verify a single forecast level of data:
v_tr_p(0,3,*,*).
> So no vertical interpolation is done. MET interpolates horizontally
from
> the forecast grid points to the observation lat/lon location (using
> user-configurable interpolation options).
>
> To put it simply, all obs between 575 and 625 mb are compared
directly to
> the forecast values from v_tr_p(0,3,*,*). If the model output is
actually
> on pressure levels, then that's a reasonable thing to do. If the
model's
> vertical coordinate is something other than pressure levels, then
that's a
> potential source for large errors.
>
> Note that MET does do vertical interpolation when processing the
GRIB
> output from the Unified Post Processor tool. But all the details of
that
> depend on how the configuration file is setup. I'll explain below.
>
> To date we have avoided including direct knowledge about the
vertical
> coordinate of each modelling system in the verification code. The
goal is
> to create verification tools applicable to many models, rather than
> customizing them for each. We do realize that desire for the latter
and are
> exploring ways to pull data from the IODA data assimilation system
instead,
> which necessarily knows a lot about each model.
>
> So let's say you were using GRIB data, and configured it like this:
>
> VAR3_NAME = VGRD
> VAR3_LEVELS = P500-1000
>
> Since we haven't prefixed these options with FCST_ and OBS_, the
same
> settings are applied to both.
>
> MET would read ALL forecast GRIB records from the model file with
pressures
> between 500 and 1000. Perhaps, that's every 100mb or perhaps every
50mb? It
> reads whatever data falls in that range. For each point observation
with a
> pressure level between 500 and 1000, it'll first find the actual
model
> pressure level above/below it. It'll interpolate the forecast
> data horizontally to the obs lat/lon location for both. And then
it'll
> interpolate those values above/below vertically to the actual
observation
> level, linear in the log of pressure.
>
> This issue is that the processing of pressure LAYERS is not
currently
> supported for NetCDF data I believe. And the code could be enhanced
to
> support that. But again, this only makes sense if the model's
vertical
> coordinate is actually pressure levels and not something else.
>
> Hope that helps clarify.
>
> Thanks,
> John
>
> On Thu, Apr 22, 2021 at 9:52 AM zhuming at ucar.edu via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> >
> > Hi George,
> >
> > My verification result shows that there is a big error on a non-
standard
> > level, so my question is:
> > Is the forecast grid match to Obs, or obs station match to
Forecast? I
> need
> > to do is the forecast grid interpolated to the obs location and
vertical
> > levels. So how to set the levels?
> >
> > Thanks,
> > Zhuming
> >
> > FCST_VAR3_NAME = v_10m_tr
> >
> > FCST_VAR3_LEVELS = (0,*,*)
> >
> > FCST_VAR3_THRESH = >=5, <5
> >
> > OBS_VAR3_NAME = VGRD
> >
> > OBS_VAR3_LEVELS = Z10
> >
> > OBS_VAR3_THRESH = >=5, <5
> >
> > OBS_VAR3_OPTIONS = message_type = ["ADPSFC"];
> >
> >
> > FCST_VAR3_NAME = v_tr_p
> >
> > FCST_VAR3_LEVELS =
> >
> >
>
(0,0,*,*),(0,1,*,*),(0,2,*,*),(0,3,*,*),(0,4,*,*),(0,5,*,*),(0,6,*,*),(0,7,*,*),(0,8,*,*),(0,9,*,*)
> >
> > FCST_VAR3_THRESH = >=5, <5
> >
> > OBS_VAR3_NAME = VGRD
> >
> > OBS_VAR3_LEVELS =
> >
> >
> P900-950,P825-875,P675-725,P575-625,P475-525,P375-425,P275-325,P225-
275,P175-225,P75-125
> >
> > OBS_VAR3_THRESH = >=5, <5
> >
> > OBS_VAR3_OPTIONS = message_type = ["ADPUPA"];
> >
> >
> >
> > On Thu, Apr 15, 2021 at 1:52 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Zhuming,
> > >
> > > There was indeed a bug in the code. Howard has fixed the issue
with
> this
> > > pull request: https://github.com/dtcenter/MET/pull/1756
> > > This fix will be included in the MET 10.0.0-beta5 release that
is
> > > tentatively scheduled for next week. This beta version can be
installed
> > on
> > > cheyenne so you can test that the update fixes your issue. I
will
> follow
> > up
> > > and let you know when the beta release is available on cheyenne.
> > >
> > > Thanks,
> > > George
> > >
> > > On Wed, Apr 14, 2021 at 5:28 PM George McCabe <mccabe at ucar.edu>
wrote:
> > >
> > > > Hi Zhuming,
> > > >
> > > > I tried to get this to work on cheyenne with your data, but
was not
> > > > successful. I am able to read the data with plot_data_plane
and
> > generate
> > > an
> > > > image, so I think there may be a bug in the code. I have
forwarded
> info
> > > to
> > > > one of our C++ developers to look into it. I will report back
when I
> > know
> > > > more.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Wed, Apr 14, 2021 at 4:13 PM zhuming at ucar.edu via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
>
> > > >>
> > > >> Hi George,
> > > >>
> > > >> It still can't find the correct time. I tried
> > > >>
> > > >> FCST_VAR1_LEVELS = (*,*,*)
> > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M},*,*)
> > > >> FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d%H%M},*,*)
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Zhuming
> > > >>
> > > >> On Wed, Apr 14, 2021 at 3:41 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > >> wrote:
> > > >>
> > > >> > Hi Zhuming,
> > > >> >
> > > >> > It looks like you are requesting the first time in the 12
hour
> file
> > > for
> > > >> > each run:
> > > >> >
> > > >> > FCST_VAR1_LEVELS = (0,*,*)
> > > >> > ...
> > > >> > FCST_VAR2_LEVELS = (0,*,*)
> > > >> >
> > > >> > The first dimension in these files is the time offset. You
will
> need
> > > the
> > > >> > correct index that corresponds to each valid time.
Fortunately,
> MET
> > > >> makes
> > > >> > this easy by allowing you to specify the valid time for a
time
> > > dimension
> > > >> > explicitly and it will find the correct data. You can set:
> > > >> >
> > > >> > FCST_VAR1_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > >> > ...
> > > >> > FCST_VAR2_LEVELS = ({valid?fmt=%Y%m%d_%H%M%S},*,*)
> > > >> >
> > > >> > And it will substitute the valid time for each run. Let me
know if
> > you
> > > >> have
> > > >> > any questions.
> > > >> >
> > > >> > Thanks,
> > > >> > George
> > > >> >
> > > >> > On Wed, Apr 14, 2021 at 3:18 PM zhuming at ucar.edu via RT <
> > > >> met_help at ucar.edu
> > > >> > >
> > > >> > wrote:
> > > >> >
> > > >> > >
> > > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429 >
> > > >> > >
> > > >> > > Hi George,
> > > >> > >
> > > >> > > Thanks for your email, I just would like to send you for
help.
> > > >> > >
> > > >> > > I already resolved the last problem with the WRF
forecast, thank
> > you
> > > >> for
> > > >> > > your help.
> > > >> > >
> > > >> > > I also need to verify the MPAS model forecast. These 12
hour
> > > >> > > MAPS forecasts are stored in a one file, such as
> > > >> > > 2015071500/mpas_wrf_cf_2015071500.nc, has 12h forecasts
> > > >> > > 2015071512/mpas_wrf_cf_2015071512.nc
> > > >> > >
> > > >> > > It seems MET is doing fine on the first 2 hours, but no
matched
> > > pairs
> > > >> for
> > > >> > > later hours.
> > > >> > > I'm not sure what's wrong with it? the time is not
matched?
> > > >> > >
> > > >> > > see the log file
> > > >> > >
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210414145101
> > > >> > > config file
/gpfs/fs1/work/zhuming/METplus4/PointStat2.conf
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Zhuming
> > > >> > >
> > > >> > > On Wed, Apr 14, 2021 at 1:56 PM George McCabe via RT <
> > > >> met_help at ucar.edu>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi Zhuming,
> > > >> > > >
> > > >> > > > Just following up on this ticket. Were you able to get
this
> > > >> resolved?
> > > >> > Let
> > > >> > > > me know if you need more assistance.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > George
> > > >> > > >
> > > >> > > > On Thu Apr 08 10:04:44 2021, mccabe wrote:
> > > >> > > > > Hi Zhuming,
> > > >> > > > >
> > > >> > > > > Typically directories that contain data for an entire
day
> are
> > > >> named
> > > >> > > > > with
> > > >> > > > > YYYYMMDD, so your filename template tag would be
> > > >> {valid?fmt=%Y%m%d},
> > > >> > > > > however that may not always be the case, as you are
seeing
> > here.
> > > >> > There
> > > >> > > > > are
> > > >> > > > > a couple ways to handle this.
> > > >> > > > >
> > > >> > > > > If you know that the directory will always be
YYYYMMDD with
> 00
> > > at
> > > >> the
> > > >> > > > > end,
> > > >> > > > > you can simply put 00 in the template instead of
adding %H
> in
> > > the
> > > >> > tag,
> > > >> > > > > i.e.
> > > >> > > > >
> > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > >> > > > > {valid?fmt=%Y%m%d}00/wrfout_d02_{valid?fmt=%Y-%m-
%d_%H}:00:
> > > >> 00_cf.nc
> > > >> > > > >
> > > >> > > > > If the HH value can vary, for instance if there are 2
> > > directories
> > > >> > > > > created
> > > >> > > > > per day (i.e. 2015071400 and 2015071412), then you
can use
> > the
> > > >> > > > > truncate
> > > >> > > > > keyword to handle it.
> > > >> > > > >
> > > >> > > > > If directories are created every 24 hours:
> > > >> > > > >
> > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > >> > > > >
> {valid?fmt=%Y%m%d%H?truncate=24H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > >> > > > > %d_%H}:00:
> > > >> > > > > 00_cf.nc
> > > >> > > > >
> > > >> > > > > If directories are created every 12 hours:
> > > >> > > > >
> > > >> > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > >> > > > >
> {valid?fmt=%Y%m%d%H?truncate=12H}/wrfout_d02_{valid?fmt=%Y-%m-
> > > >> > > > > %d_%H}:00:
> > > >> > > > > 00_cf.nc
> > > >> > > > >
> > > >> > > > > Let me know if that doesn't work for you.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > George
> > > >> > > > >
> > > >> > > > > On Thu, Apr 8, 2021 at 9:46 AM zhuming at ucar.edu via
RT
> > > >> > > > > <met_help at ucar.edu>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > >
> > > >> > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >
> > > >> > > > > >
> > > >> > > > > > Hi George,
> > > >> > > > > >
> > > >> > > > > > Processing Obs data using "PB2NC" are right now.
> > > >> > > > > >
> > > >> > > > > > Now I'm using "PointStat" , I set VALID_BEG =
2015071500,
> > > >> > VALID_END =
> > > >> > > > > > 2015071501. It looks for the forecast data
2015071500/
> > > >> > > > > > wrfout_d02_2015-07-15_00:00:00_cf.nc, and
2015071501/
> > > >> > > > > > wrfout_d02_2015-07-15_01:00:00_cf.nc.
> > > >> > > > > > But my forecast data are all put in dir of
2015071500/.
> > > >> > > > > > So how could I set FCST_POINT_STAT_INPUT_TEMPLATE
to get
> > the
> > > >> right
> > > >> > > > > > fcst
> > > >> > > > > > data?
> > > >> > > > > >
> > > >> > > > > > FCST_POINT_STAT_INPUT_TEMPLATE =
> > > >> > > > > >
> {valid?fmt=%Y%m%d%H}/wrfout_d02_{valid?fmt=%Y-%m-%d_%H}:00:
> > > >> > 00_cf.nc
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > config file:
> > /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > >> > > > > >
> > > >> > > > > > log file:
> > > >> > > > > >
> > > >> > >
> > >
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210408085450
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > >
> > > >> > > > > > Zhuming
> > > >> > > > > >
> > > >> > > > > > On Wed, Apr 7, 2021 at 5:43 PM George McCabe via RT
> > > >> > > > > > <met_help at ucar.edu>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi Zhuming,
> > > >> > > > > > >
> > > >> > > > > > > Looking at the log output, it looks like the data
is
> > stored
> > > in
> > > >> > the
> > > >> > > > > > prepBUFR
> > > >> > > > > > > files differently that you may have thought:
> > > >> > > > > > >
> > > >> > > > > > > DEBUG 2: PrepBufr Time Center:
20150715_000000
> > > >> > > > > > > DEBUG 2: Searching Time Window:
20150715_030000
> to
> > > >> > > > > > 20150715_040000
> > > >> > > > > > > 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60%
65% 70%
> 75%
> > > 80%
> > > >> > 85%
> > > >> > > > > > > 90%
> > > >> > > > > > 95%
> > > >> > > > > > > 100%
> > > >> > > > > > >
> > > >> > > > > > > DEBUG 2: Total Messages processed =
590836
> > > >> > > > > > > DEBUG 2: Rejected based on message type =
0
> > > >> > > > > > > DEBUG 2: Rejected based on station id =
0
> > > >> > > > > > > DEBUG 2: Rejected based on valid time =
590815
> > > >> > > > > > > DEBUG 2: Rejected based on masking grid =
20
> > > >> > > > > > > DEBUG 2: Rejected based on masking polygon =
0
> > > >> > > > > > > DEBUG 2: Rejected based on elevation =
0
> > > >> > > > > > > DEBUG 2: Rejected based on pb report type =
1
> > > >> > > > > > > DEBUG 2: Rejected based on input report type =
0
> > > >> > > > > > > DEBUG 2: Rejected based on instrument type =
0
> > > >> > > > > > > DEBUG 2: Rejected based on zero observations =
0
> > > >> > > > > > > DEBUG 2: Total Messages retained =
0
> > > >> > > > > > > DEBUG 2: Total observations retained or derived =
0
> > > >> > > > > > > DEBUG 1: Obs time between *20150714_210000 and
> > > >> 20150715_030000*
> > > >> > > > > > > WARNING:
> > > >> > > > > > > WARNING: process_pbfile() -> No PrepBufr messages
> retained
> > > >> from
> > > >> > > > > > > file:
> > > >> > > > > > > /glade/collections/rda/data/ds337.0/prepnr/2015/
> > > >> > > > > > > prepbufr.gdas.2015071500.nr
> > > >> > > > > > > WARNING:
> > > >> > > > > > > ERROR :
> > > >> > > > > > > ERROR : write_netcdf_hdr_data() -> No PrepBufr
messages
> > > >> > retained.
> > > >> > > > > > Nothing
> > > >> > > > > > > to write.
> > > >> > > > > > > ERROR :
> > > >> > > > > > >
> > > >> > > > > > > The prepbufr.gdas.2015071500.nr file actually
contains
> > data
> > > >> from
> > > >> > > > > > > 20150714_210000 to 20150715_030000, not
20150715_000000
> to
> > > >> > > > > > 20150715_060000.
> > > >> > > > > > >
> > > >> > > > > > > To get the correct file for each run, you will
have to
> > shift
> > > >> the
> > > >> > > > > > > valid
> > > >> > > > > > time
> > > >> > > > > > > before truncating it so that it gets the correct
file
> for
> > > each
> > > >> > run.
> > > >> > > > > > Instead
> > > >> > > > > > > of setting:
> > > >> > > > > > >
> > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > >> > > > > > >
> > > >> >
> {valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr
> > > >> > > > > > >
> > > >> > > > > > > You will have to also shift forward 3 hours,
i.e.:
> > > >> > > > > > >
> > > >> > > > > > > PB2NC_INPUT_TEMPLATE =
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?shift=3H?truncate=21600}.nr
> > > >> > > > > > >
> > > >> > > > > > > This says to add 3 hours to the current valid
time
> before
> > > >> > > > > > > truncating to
> > > >> > > > > > the
> > > >> > > > > > > nearest 6 hours.
> > > >> > > > > > >
> > > >> > > > > > > Valid: 20150715 0Z + 3 = 3Z truncate 6H = 0Z
> > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > >> > > > > > >
> > > >> > > > > > > Valid: 20150715 1Z + 3 = 4Z truncate 6H = 0Z
> > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > >> > > > > > >
> > > >> > > > > > > Valid: 20150715 2Z + 3 = 5Z truncate 6H = 0Z
> > > >> > > > > > > File: prepbufr.gdas.2015071500.nr
> > > >> > > > > > >
> > > >> > > > > > > Valid: 20150715 3Z + 3 = 6Z truncate 6H = 6Z
> > > >> > > > > > > File: prepbufr.gdas.2015071506.nr
> > > >> > > > > > >
> > > >> > > > > > > Let me know if that doesn't work.
> > > >> > > > > > >
> > > >> > > > > > > Thanks,
> > > >> > > > > > > George
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Wed, Apr 7, 2021 at 5:32 PM zhuming at ucar.edu
via RT
> <
> > > >> > > > > > met_help at ucar.edu>
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > <URL:
> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >> > >
> > > >> > > > > > > >
> > > >> > > > > > > > Hi George,
> > > >> > > > > > > >
> > > >> > > > > > > > Now makes me confused is why processed obs of
several
> > > times
> > > >> are
> > > >> > > > > > missing.
> > > >> > > > > > > > There has files of time at 00,01,02, but no
03,04,05;
> > has
> > > >> > > > > > > > 06,07,08, but
> > > >> > > > > > > no
> > > >> > > > > > > > 09,10,11, has 12,13,14, no 15,16,17; has
18,19,20,
> but
> > > with
> > > >> > > > > > > > 21,no
> > > >> > > > > > 22,23.
> > > >> > > > > > > > From yours and mines:
> > > >> > > > > > > >
> > > >> > > > > > > > ls -l /glade/u/home/mccabe/data/out/pb2nc1/
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 10141339 Apr 6 17:52
> > > >> > pbs.2015071500.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5413907 Apr 6 17:56
> > > >> > pbs.2015071501.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4923711 Apr 6 18:00
> > > >> > pbs.2015071502.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5724447 Apr 6 18:14
> > > >> > pbs.2015071506.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5280463 Apr 6 18:18
> > > >> > pbs.2015071507.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 4706119 Apr 6 18:22
> > > >> > pbs.2015071508.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 9878263 Apr 6 18:36
> > > >> > pbs.2015071512.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 5392527 Apr 6 18:40
> > > >> > pbs.2015071513.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:40
> > > >> > pbs.2015071514.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 mccabe ral 96 Apr 6 18:42
> > > >> > pbs.2015071515.nc
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
cheyenne3:/glade/scratch/zhuming/metplus4/pb2nc1>l
> > > >> > > > > > > >
> > > >> > > > > > > > total 96112
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11550172 Apr 7 15:11
> > > >> > > pbs.2015071500.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6615956 Apr 7 15:14
> > > >> > > pbs.2015071501.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5984328 Apr 7 15:18
> > > >> > > pbs.2015071502.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6980712 Apr 7 15:33
> > > >> > > pbs.2015071506.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6455824 Apr 7 15:36
> > > >> > > pbs.2015071507.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5749432 Apr 7 15:40
> > > >> > > pbs.2015071508.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11284936 Apr 7 15:55
> > > >> > > pbs.2015071512.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6600888 Apr 7 15:59
> > > >> > > pbs.2015071513.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 5868168 Apr 7 16:03
> > > >> > > pbs.2015071514.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 7273092 Apr 7 16:18
> > > >> > > pbs.2015071518.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6656344 Apr 7 16:22
> > > >> > > pbs.2015071519.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 6109788 Apr 7 16:27
> > > >> > > pbs.2015071520.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 58808 Apr 7 16:31
> > > >> > > pbs.2015071521.nc
> > > >> > > > > > > >
> > > >> > > > > > > > -rw-r--r-- 1 zhuming ncar 11132440 Apr 7 16:43
> > > >> > > pbs.2015071600.nc
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks,
> > > >> > > > > > > >
> > > >> > > > > > > > Zhuming
> > > >> > > > > > > >
> > > >> > > > > > > > On Tue, Apr 6, 2021 at 6:34 PM George McCabe
via RT
> > > >> > > > > > > > <met_help at ucar.edu
> > > >> > > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi Zhuming,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks for the information. You will need to
use the
> > > >> truncate
> > > >> > > > > > > > > keyword
> > > >> > > > > > > to
> > > >> > > > > > > > > find the files, then set the valid beg/end
range to
> 1
> > > >> hour.
> > > >> > > > > > > > > This is
> > > >> > > > > > > what
> > > >> > > > > > > > I
> > > >> > > > > > > > > changed from your PB2NC1.conf config file:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *PB2NC_VALID_BEGIN =
> > > {valid?fmt=%Y%m%d_%H}PB2NC_VALID_END
> > > >> =
> > > >> > > > > > > > > {valid?fmt=%Y%m%d_%H?shift=1H}*
> > > >> > > > > > > > >
> > > >> > > > > > > > > *PB2NC_INPUT_TEMPLATE =
> > > >> > > > > > > > >
> > > >> > > >
> > >
{valid?fmt=%Y}/prepbufr.gdas.{valid?fmt=%Y%m%d%H?truncate=21600}.nr*
> > > >> > > > > > > > >
> > > >> > > > > > > > > My config file is on cheyenne here:
> > > >> > > > > > > > > /glade/u/home/mccabe/PB2NC1.conf
> > > >> > > > > > > > >
> > > >> > > > > > > > > I just started running it and the output is
being
> > > written
> > > >> > here:
> > > >> > > > > > > > > /glade/u/home/mccabe/data/out/pb2nc1
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > For PointStat, you will want to loop over the
same
> > > times.
> > > >> I
> > > >> > > > > > > > > noticed
> > > >> > > > > > you
> > > >> > > > > > > > > specify message type in the VAR<n>_OPTIONS,
but you
> do
> > > not
> > > >> > need
> > > >> > > > > > > > > to do
> > > >> > > > > > > > this
> > > >> > > > > > > > > since there is a POINT_STAT_MESSAGE_TYPE
variable
> that
> > > >> > defines
> > > >> > > > > > > > > this.
> > > >> > > > > > > This
> > > >> > > > > > > > > is what I changed in your PointStat1.conf
file:
> > > >> > > > > > > > >
> > > >> > > > > > > > > *VALID_END = 2015071600*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *VALID_INCREMENT = 1HPOINT_STAT_OBS_VALID_BEG
> > > >> > > > > > =POINT_STAT_OBS_VALID_BEG
> > > >> > > > > > > > =*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *POINT_STAT_MESSAGE_TYPE = ADPSFC*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *#OBS_VAR1_OPTIONS = message_type =
> > > >> > > > > > > > > ["ADPSFC"];#OBS_VAR2_OPTIONS =
> > > >> > > > > > > > > message_type = ["ADPSFC"];#OBS_VAR3_OPTIONS =
> > > >> message_type =
> > > >> > > > > > > > > ["ADPSFC"];#OBS_POINT_STAT_INPUT_DIR =
> > > >> > > > > > > > > /glade/scratch/zhuming/metplus4/pb2nc1*
> > > >> > > > > > > > >
> > > >> > > > > > > > > *OBS_POINT_STAT_INPUT_DIR =
{OUTPUT_BASE}/pb2nc1*
> > > >> > > > > > > > >
> > > >> > > > > > > > > Or you can view my file in the same
directory. I
> > usually
> > > >> try
> > > >> > to
> > > >> > > > > > > > > test
> > > >> > > > > > > > with a
> > > >> > > > > > > > > single run time to make sure it is working
properly,
> > so
> > > I
> > > >> set
> > > >> > > > > > VALID_END
> > > >> > > > > > > > > back to 2015071500. I ran it and it produced
an
> output
> > > >> file,
> > > >> > > > > > > > > but
> > > >> > > > > > there
> > > >> > > > > > > > were
> > > >> > > > > > > > > no values besides the header. I checked the
log file
> > and
> > > >> saw
> > > >> > > > > > > > > this:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *DEBUG 2: Processing T_sfc(0,*,*) versus
TMP/Z2, for
> > > >> > > > > > > > > observation type
> > > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > > >> BILIN(4),
> > > >> > > > > > > > > using 0
> > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs =
> > > 0DEBUG
> > > >> 3:
> > > >> > > > > > > Observations
> > > >> > > > > > > > > processed = 393373DEBUG 3: Rejected:
station id
> > > =
> > > >> > > > > > > > > 0DEBUG 3:
> > > >> > > > > > > > > Rejected: obs type = 390528DEBUG 3:
Rejected:
> > > valid
> > > >> > time
> > > >> > > > > > =
> > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG
3:
> > > Rejected:
> > > >> off
> > > >> > > > > > > > > the
> > > >> > > > > > grid
> > > >> > > > > > > > =
> > > >> > > > > > > > > 2586DEBUG 3: Rejected: topography =
0DEBUG 3:
> > > >> Rejected:
> > > >> > > > > > > > > level
> > > >> > > > > > > > mismatch
> > > >> > > > > > > > > = 259DEBUG 3: Rejected: quality marker =
0DEBUG 3:
> > > >> Rejected:
> > > >> > > > > > message
> > > >> > > > > > > > type
> > > >> > > > > > > > > = 0DEBUG 3: Rejected: masking region =
0DEBUG 3:
> > > >> Rejected:
> > > >> > bad
> > > >> > > > > > fcst
> > > >> > > > > > > > > value = 0DEBUG 3: Rejected: bad climo mean
=
> 0DEBUG
> > 3:
> > > >> > > > > > > > > Rejected:
> > > >> > > > > > bad
> > > >> > > > > > > > > climo stdev = 0DEBUG 3: Rejected: duplicates
=
> > > 0DEBUG
> > > >> > > > > > > > > 2:DEBUG 2:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > >> > > > > > > > > 2:DEBUG 2: Processing u_10m_gr(0,*,*) versus
> UGRD/Z10,
> > > for
> > > >> > > > > > observation
> > > >> > > > > > > > type
> > > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > > >> BILIN(4),
> > > >> > > > > > > > > using 0
> > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs =
> > > 0DEBUG
> > > >> 3:
> > > >> > > > > > > Observations
> > > >> > > > > > > > > processed = 393373*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *DEBUG 3: Rejected: station id = 0DEBUG
3:
> > > Rejected:
> > > >> obs
> > > >> > > > > > > > > type
> > > >> > > > > > > > =
> > > >> > > > > > > > > 382965DEBUG 3: Rejected: valid time =
0DEBUG 3:
> > > >> > Rejected:
> > > >> > > > > > > > > bad
> > > >> > > > > > obs
> > > >> > > > > > > > > value = 0DEBUG 3: Rejected: off the grid
=
> > > 9240DEBUG
> > > >> 3:
> > > >> > > > > > Rejected:
> > > >> > > > > > > > > topography = 0DEBUG 3: Rejected: level
mismatch
> > =
> > > >> > > > > > > > > 1168DEBUG 3:
> > > >> > > > > > > > > Rejected: quality marker = 0DEBUG 3:
Rejected:
> > message
> > > >> type
> > > >> > > > > > > > > =
> > > >> > > > > > > 0DEBUG
> > > >> > > > > > > > 3:
> > > >> > > > > > > > > Rejected: masking region = 0DEBUG 3:
Rejected: bad
> > fcst
> > > >> > value
> > > >> > > > > > > > > =
> > > >> > > > > > > 0DEBUG
> > > >> > > > > > > > 3:
> > > >> > > > > > > > > Rejected: bad climo mean = 0DEBUG 3:
Rejected: bad
> > > climo
> > > >> > stdev
> > > >> > > > > > > > > =
> > > >> > > > > > > 0DEBUG
> > > >> > > > > > > > 3:
> > > >> > > > > > > > > Rejected: duplicates = 0DEBUG 2:DEBUG 2:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > >> > > > > > > > > 2:DEBUG 2: Processing v_10m_gr(0,*,*) versus
> VGRD/Z10,
> > > for
> > > >> > > > > > observation
> > > >> > > > > > > > type
> > > >> > > > > > > > > ADPSFC, over region FULL, for interpolation
method
> > > >> BILIN(4),
> > > >> > > > > > > > > using 0
> > > >> > > > > > > > > matched pairs.DEBUG 3: Number of matched
pairs =
> > > 0DEBUG
> > > >> 3:
> > > >> > > > > > > Observations
> > > >> > > > > > > > > processed = 393373DEBUG 3: Rejected:
station id
> > > =
> > > >> > > > > > > > > 0DEBUG 3:
> > > >> > > > > > > > > Rejected: obs type = 382965DEBUG 3:
Rejected:
> > > valid
> > > >> > time
> > > >> > > > > > =
> > > >> > > > > > > > > 0DEBUG 3: Rejected: bad obs value = 0DEBUG
3:
> > > Rejected:
> > > >> off
> > > >> > > > > > > > > the
> > > >> > > > > > grid
> > > >> > > > > > > > =
> > > >> > > > > > > > > 9240DEBUG 3: Rejected: topography =
0DEBUG 3:
> > > >> Rejected:
> > > >> > > > > > > > > level
> > > >> > > > > > > > mismatch
> > > >> > > > > > > > > = 1168DEBUG 3: Rejected: quality marker =
0DEBUG 3:
> > > >> > Rejected:
> > > >> > > > > > message
> > > >> > > > > > > > > type = 0DEBUG 3: Rejected: masking region
=
> 0DEBUG
> > > 3:
> > > >> > > > > > > > > Rejected:
> > > >> > > > > > bad
> > > >> > > > > > > > > fcst value = 0DEBUG 3: Rejected: bad climo
mean =
> > > >> 0DEBUG 3:
> > > >> > > > > > Rejected:
> > > >> > > > > > > > bad
> > > >> > > > > > > > > climo stdev = 0*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > *DEBUG 3: Rejected: duplicates = 0DEBUG
2:DEBUG
> > 2:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > >> > > > > > > > > 2:DEBUG 1: Output file:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/glade/u/home/mccabe/data/out/point_stat1/point_stat_000000L_20150715_000000V.stat*
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > This describes the reason why each
observation was
> > > >> rejected.
> > > >> > > > > > > > > Some
> > > >> > > > > > > > > observations didn't match the message type
you
> > > requested,
> > > >> > some
> > > >> > > > > > > > > were
> > > >> > > > > > not
> > > >> > > > > > > > > within the forecast grid, and some did not
match the
> > > >> vertical
> > > >> > > > > > > > > level.
> > > >> > > > > > > You
> > > >> > > > > > > > > will have to make adjustments to your
settings so
> that
> > > the
> > > >> > data
> > > >> > > > > > matches
> > > >> > > > > > > > up
> > > >> > > > > > > > > properly. Maybe the grid you are using in the
PB2NC
> > > config
> > > >> > file
> > > >> > > > > > > > > does
> > > >> > > > > > > not
> > > >> > > > > > > > > match the forecast grid and you are missing
values.
> If
> > > >> that
> > > >> > is
> > > >> > > > > > > > > the
> > > >> > > > > > > case,
> > > >> > > > > > > > > you will have to reconfigure and rerun PB2NC
to get
> > > >> output on
> > > >> > > > > > > > > the
> > > >> > > > > > > correct
> > > >> > > > > > > > > gr5id. Maybe the observation points don't
line up
> with
> > > the
> > > >> > > > > > > > > vertical
> > > >> > > > > > > level
> > > >> > > > > > > > > of the model fields. You can use a range of
level
> > values
> > > >> for
> > > >> > > > > > > > > the
> > > >> > > > > > > > > observation data to allow data within that
range,
> i.e.
> > > >> Z0-10.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Let me know if you are still having issues
getting
> > this
> > > to
> > > >> > work
> > > >> > > > > > > > > and I
> > > >> > > > > > > can
> > > >> > > > > > > > > help out.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks,
> > > >> > > > > > > > > George
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Tue, Apr 6, 2021 at 5:17 PM
zhuming at ucar.edu via
> > RT
> > > <
> > > >> > > > > > > > met_help at ucar.edu>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > <URL:
> > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Hi George,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks for your reply!
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > I'm still confused, I need to output hourly
obs
> data
> > > to
> > > >> do
> > > >> > > > > > > > > > hourly
> > > >> > > > > > > > > > verification.
> > > >> > > > > > > > > > The original GDAS Prepbufr data has 4 files
one
> day,
> > > >> should
> > > >> > > > > > > > > > with 6
> > > >> > > > > > > > hours
> > > >> > > > > > > > > > obs data in one file.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > After "PB2NC", I can see pbs.2015071500.nc
(obs
> > time
> > > is
> > > >> > 00?
> > > >> > > > > > > > > > or 6
> > > >> > > > > > > times
> > > >> > > > > > > > > of
> > > >> > > > > > > > > > 00,01,02,03,04,05?), pbs.2015071506.nc...
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > my config file :
> > > >> > > > > >
cheyenne:/gpfs/fs1/work/zhuming/METplus4/PB2NC1.conf
> > > >> > > > > > > > > > and
> /gpfs/fs1/work/zhuming/METplus4/PointStat1.conf
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks,
> > > >> > > > > > > > > > Zhuming
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Tue, Apr 6, 2021 at 4:23 PM George
McCabe via
> RT
> > <
> > > >> > > > > > > met_help at ucar.edu
> > > >> > > > > > > > >
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi Zhuming,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > To run the tools hourly you would set the
> > > >> VALID_INCREMENT
> > > >> > > > > > > > > > > value
> > > >> > > > > > for
> > > >> > > > > > > > 1H.
> > > >> > > > > > > > > > > Here is an example that runs once per
hour for a
> > day
> > > >> from
> > > >> > > > > > > 2015-07-15
> > > >> > > > > > > > 0Z
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > 2017-07-16 0Z:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > VALID_TIME_FMT = %Y%m%d%H
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > VALID_BEG = 2015071500
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > VALID_END = 2015071600
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > VALID_INCREMENT = 1H
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > LEAD_SEQ = 0
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Each run time will use the
PB2NC_INPUT_TEMPLATE
> to
> > > >> find
> > > >> > > > > > > > > > > files
> > > >> > > > > > whose
> > > >> > > > > > > > > > > filenames match the time you need.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > To read prepBUFR observations for 12
hours, you
> > will
> > > >> want
> > > >> > > > > > > > > > > to set
> > > >> > > > > > > > > > > PB2NC_VALID_BEGIN and PB2NC_VALID_END to
specify
> > the
> > > >> > range
> > > >> > > > > > > > > > > around
> > > >> > > > > > > the
> > > >> > > > > > > > > > valid
> > > >> > > > > > > > > > > time to process. These values depend on
what is
> > > >> contained
> > > >> > > > > > > > > > > inside
> > > >> > > > > > > your
> > > >> > > > > > > > > > > prepBUFR files. If you want to use data
from the
> > > valid
> > > >> > time
> > > >> > > > > > > > > > > of
> > > >> > > > > > the
> > > >> > > > > > > > > > filename
> > > >> > > > > > > > > > > backwards 12 hours, you would set:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > PB2NC_VALID_BEGIN =
> > {valid?fmt=%Y%m%d_%H?shift=-12H
> > > }
> > > >> > > > > > > > > > > PB2NC_VALID_END = {valid?fmt=%Y%m%d_%H}
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > For a runtime of 2015071512, this would
be
> > > substituted
> > > >> > as:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > valid_beg: 20150715_00
> > > >> > > > > > > > > > > valid_end: 20150715_12
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > If you want to create an output file for
each
> hour
> > > in
> > > >> > PB2NC
> > > >> > > > > > > > > > > you
> > > >> > > > > > > would
> > > >> > > > > > > > > > need
> > > >> > > > > > > > > > > to make sure that the hour is specified
in the
> > > >> > > > > > > PB2NC_OUTPUT_TEMPLATE.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > PointStat has similar config variables to
set
> the
> > > >> valid
> > > >> > beg
> > > >> > > > > > > > > > > and
> > > >> > > > > > end
> > > >> > > > > > > > > range
> > > >> > > > > > > > > > > to use (POINT_STAT_OBS_VALID_BEG and
> > > >> > > > > > > > > > > POINT_STAT_OBS_VALID_END).
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > If your prepBUFR files are separated out
into
> > > multiple
> > > >> > > > > > > > > > > files that
> > > >> > > > > > > > need
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > be read into PB2NC, you can set a file
window
> > around
> > > >> the
> > > >> > > > > > > > > > > valid
> > > >> > > > > > time
> > > >> > > > > > > > to
> > > >> > > > > > > > > > > allow files to be passed into the app.
More
> > > >> information
> > > >> > on
> > > >> > > > > > > > > > > that
> > > >> > > > > > > > > > > functionality is here:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#using-
> > > >> > > > > > windows-to-find-valid-files
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > If you are still having trouble getting
this
> > > >> configured,
> > > >> > > > > > > > > > > you
> > > >> > > > > > could
> > > >> > > > > > > > send
> > > >> > > > > > > > > > me
> > > >> > > > > > > > > > > the paths of the files you are trying to
read
> and
> > > what
> > > >> > data
> > > >> > > > > > > > > > > is
> > > >> > > > > > > > > contained
> > > >> > > > > > > > > > in
> > > >> > > > > > > > > > > them so I can give you suggestions on
what to
> set
> > > for
> > > >> the
> > > >> > > > > > > > > configuration.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > George
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Tue, Apr 6, 2021 at 2:17 PM Julie
Prestopnik
> > via
> > > >> RT <
> > > >> > > > > > > > > > met_help at ucar.edu>
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Tue Apr 06 14:16:42 2021: Request 99429
was
> > acted
> > > >> upon.
> > > >> > > > > > > > > > > > Transaction: Given to mccabe (George
McCabe)
> by
> > > >> jpresto
> > > >> > > > > > > > > > > > Queue: met_help
> > > >> > > > > > > > > > > > Subject: How to set time for
hourly 12-24
> > > hours
> > > >> > data
> > > >> > > > > > > > > > > > ?
> > > >> > > > > > > > > > > > Owner: mccabe
> > > >> > > > > > > > > > > > Requestors: zhuming at ucar.edu
> > > >> > > > > > > > > > > > Status: new
> > > >> > > > > > > > > > > > Ticket <URL:
> > > >> > > > > > > > >
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99429
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > This transaction appears to have no
content
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > --
> > > >> > > > > > > > > > > George McCabe - Software Engineer III
> > > >> > > > > > > > > > > National Center for Atmospheric Research
> > > >> > > > > > > > > > > Research Applications Laboratory
> > > >> > > > > > > > > > > 303-497-2768
> > > >> > > > > > > > > > > ---
> > > >> > > > > > > > > > > My working day may not be your working
day.
> Please
> > > do
> > > >> not
> > > >> > > > > > > > > > > feel
> > > >> > > > > > > > obliged
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > reply to this email outside of your
normal
> working
> > > >> hours.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > --
> > > >> > > > > > > > > George McCabe - Software Engineer III
> > > >> > > > > > > > > National Center for Atmospheric Research
> > > >> > > > > > > > > Research Applications Laboratory
> > > >> > > > > > > > > 303-497-2768
> > > >> > > > > > > > > ---
> > > >> > > > > > > > > My working day may not be your working day.
Please
> do
> > > not
> > > >> > feel
> > > >> > > > > > obliged
> > > >> > > > > > > to
> > > >> > > > > > > > > reply to this email outside of your normal
working
> > > hours.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > --
> > > >> > > > > > > George McCabe - Software Engineer III
> > > >> > > > > > > National Center for Atmospheric Research
> > > >> > > > > > > Research Applications Laboratory
> > > >> > > > > > > 303-497-2768
> > > >> > > > > > > ---
> > > >> > > > > > > My working day may not be your working day.
Please do
> not
> > > feel
> > > >> > > > > > > obliged to
> > > >> > > > > > > reply to this email outside of your normal
working
> hours.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > George McCabe - Software Engineer III
> > > >> > National Center for Atmospheric Research
> > > >> > Research Applications Laboratory
> > > >> > 303-497-2768
> > > >> > ---
> > > >> > My working day may not be your working day. Please do not
feel
> > obliged
> > > >> to
> > > >> > reply to this email outside of your normal working hours.
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list