[Met_help] [rt.rap.ucar.edu #81437] History for Help with generating unique point_stat file names
John Halley Gotway via RT
met_help at ucar.edu
Thu Aug 3 10:39:34 MDT 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I'm trying to run MET automatically via a python programming. This program
passes in dates and hours, and passes these variables into the PB2NC and
POINT_STAT tools. Also, I'm extracting out bins of data according to the
hour from a PrepBufr, which I use in the POINT_STAT tool. When I tested
these ideas on the command line, they worked properly, as I expected, but,
when I put them into my automated program, the point_stat_* file names were
not correct.
So, as an example, I'm trying to run this for the 06z cycle. My first time
through my loop, I'm pulling out data from 05z - 06:59:59z, and my PB2NC
looks like:
Hourly: running /opt/MET6/met6.0/bin/pb2nc
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
uv_wind_pb_2017080206_06.nc
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
-valid_beg 20170802_050000 -valid_end 20170802_065959 -v 2
DEBUG 1: Default Config File:
/opt/MET6/met6.0/share/met/config/PB2NCConfig_default
DEBUG 1: User Config File:
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
DEBUG 1: Creating NetCDF File:
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
uv_wind_pb_2017080206_06.nc
DEBUG 1: Processing PrepBufr File:
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
DEBUG 1: Blocking PrepBufr file to: /tmp/tmp_pb2nc_blk_50115_0
DEBUG 2: PrepBufr Time Center: 20170802_060000
DEBUG 2: Searching Time Window: 20170802_050000 to 20170802_065959
And, POINT_STAT looks like:
running /opt/MET6/met6.0/bin/point_stat
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.pgrb2.0p25.f000
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
uv_wind_pb_2017080206_06.nc
/opc_test/home/opc_test/all_opc/tables/met_verif/PointStatConfig_working
-outdir /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/ -v 2
And, generate files like:
DEBUG 1: Output file:
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly//point_stat_000000L_20170802_060000V.stat
DEBUG 1: Output file:
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly//point_stat_000000L_20170802_060000V_ctc.txt
DEBUG 1: Output file:
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly//point_stat_000000L_20170802_060000V_cts.txt
etc.
But, then, when I try and pull out my next bin of data (06z - 07:59:59z),
the PB2NC looks like this:
Hourly: running /opt/MET6/met6.0/bin/pb2nc
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
uv_wind_pb_2017080206_07.nc
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
-valid_beg 20170802_060000 -valid_end 20170802_075959 -v 2
DEBUG 1: Default Config File:
/opt/MET6/met6.0/share/met/config/PB2NCConfig_default
DEBUG 1: User Config File:
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
DEBUG 1: Creating NetCDF File:
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
uv_wind_pb_2017080206_07.nc
DEBUG 1: Processing PrepBufr File:
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
DEBUG 1: Blocking PrepBufr file to: /tmp/tmp_pb2nc_blk_53440_0
DEBUG 2: PrepBufr Time Center: 20170802_060000
DEBUG 2: Searching Time Window: 20170802_060000 to 20170802_075959
And, the POINT_STAT looks like:
running /opt/MET6/met6.0/bin/point_stat
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.pgrb2.0p25.f000
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
uv_wind_pb_2017080206_07.nc
/opc_test/home/opc_test/all_opc/tables/met_verif/PointStatConfig_working
-outdir /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/ -v 2
But, the point_stat files look like:
DEBUG 1: Output file:
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly//point_stat_000000L_20170802_060000V.stat
DEBUG 1: Output file:
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly//point_stat_000000L_20170802_060000V_ctc.txt
DEBUG 1: Output file:
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly//point_stat_000000L_20170802_060000V_cts.txt
etc.
So, here's the questions:
1) Shouldn't they look like:
mprfile =
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/point_stat_000000L_20170802_070000V_mpr.txt
cntfile =
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/point_stat_000000L_20170802_070000V_cnt.txt
When I ran this on the command line, they ended up looking like something
like this.
2) And, if not, and they will look like they do in the above DEBUG
statement, how can I make the name of the point_stat files for each bin of
data unique so the files don't overwrite themselves?
3) Shouldn't this debug statement:
DEBUG 2: Searching Time Window: 20170802_060000 to 20170802_075959
be centered at 20170802_070000, and not:
DEBUG 2: PrepBufr Time Center: 20170802_060000?
Thanks!
Roz
--
Rosalyn MacCracken
Support Scientist
Ocean Applilcations Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Help with generating unique point_stat file names
From: John Halley Gotway
Time: Wed Aug 02 17:07:01 2017
Roz,
I see that you have questions about setting up routine evaluation
using
pb2nc and point_stat. In particular, you're having issues generating
unique file names. There are a few important points to make...
(1) When you run Point-Stat, you pass it a single forecast file. It
figures out the valid time and lead time of that forecast (i.e.
valid_time
= init_time + lead_time), and uses the those times in the output file
names
it creates.
(2) If you run Point-Stat multiple times on the same file, the
filenames
will clobber each other unless you do something about it.
(3) In the MET config file, the "output_prefix" string (empty by
default)
is inserted into the output file names in order to make them unique.
We
often run Point-Stat separately for surface and upper-air verification
and
set "output_prefix" to ADPSFC and ADPUPA to differentiate the output
file
names.
(4) Next, PREPBUFR files usually contains observations collected over
a 3
or 6 hour time window. I see from the commands you sent that you're
running PB2NC once for 06z and a second time for 07z. But you're
comparing
both the 06Z and 07Z observations to same forecast file. I don't
really
understand why you'd want to do that.
Usually, the verification logic is based on the forecast times being
evaluated. You'd loop over the forecast hours you'd like to evaluate
and
call Point-Stat once for each time, passing to it all the observations
you'd like to use.
Getting back to your PREPBUFR runs, consider this file:
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.pgrb2.0p25.f000
And this DEBUG line:
DEBUG 2: PrepBufr Time Center: 20170802_060000
I'm guessing that this PREPBUFR file contains observations +/- 3 hours
around that center time. Or maybe it's +/- 1.5 hours around it. I'm
really not sure. When you run PB2NC, the config file contains the
following:
obs_window = {
beg = -5400; // seconds
end = 5400; // seconds
}
This tell PB2NC what observations to retain... the center time of the
file
+ beg to the center time + end. However, the -valid_beg and
-valid_end
command line options that you used override these settings. Either
way of
specifying the time window is fine... sometimes its more convenient to
do
it one way or the other.
Hopefully that helps fill in the details.
Please let me know what other questions you have.
Thanks,
John Halley Gotway
On Wed, Aug 2, 2017 at 9:21 AM, Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> Wed Aug 02 09:21:38 2017: Request 81437 was acted upon.
> Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> Queue: met_help
> Subject: Help with generating unique point_stat file names
> Owner: Nobody
> Requestors: rosalyn.maccracken at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81437 >
>
>
> Hi,
>
> I'm trying to run MET automatically via a python programming. This
program
> passes in dates and hours, and passes these variables into the PB2NC
and
> POINT_STAT tools. Also, I'm extracting out bins of data according
to the
> hour from a PrepBufr, which I use in the POINT_STAT tool. When I
tested
> these ideas on the command line, they worked properly, as I
expected, but,
> when I put them into my automated program, the point_stat_* file
names were
> not correct.
>
> So, as an example, I'm trying to run this for the 06z cycle. My
first time
> through my loop, I'm pulling out data from 05z - 06:59:59z, and my
PB2NC
> looks like:
>
> Hourly: running /opt/MET6/met6.0/bin/pb2nc
> /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> uv_wind_pb_2017080206_06.nc
> /opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> -valid_beg 20170802_050000 -valid_end 20170802_065959 -v 2
> DEBUG 1: Default Config File:
> /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> DEBUG 1: User Config File:
> /opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> DEBUG 1: Creating NetCDF File:
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> uv_wind_pb_2017080206_06.nc
> DEBUG 1: Processing PrepBufr File:
> /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> DEBUG 1: Blocking PrepBufr file to: /tmp/tmp_pb2nc_blk_50115_0
> DEBUG 2: PrepBufr Time Center: 20170802_060000
> DEBUG 2: Searching Time Window: 20170802_050000 to
20170802_065959
>
> And, POINT_STAT looks like:
>
> running /opt/MET6/met6.0/bin/point_stat
>
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.pgrb2.0p25.f000
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> uv_wind_pb_2017080206_06.nc
>
/opc_test/home/opc_test/all_opc/tables/met_verif/PointStatConfig_working
> -outdir /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/ -v 2
>
> And, generate files like:
> DEBUG 1: Output file:
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> point_stat_000000L_20170802_060000V.stat
> DEBUG 1: Output file:
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> point_stat_000000L_20170802_060000V_ctc.txt
> DEBUG 1: Output file:
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> point_stat_000000L_20170802_060000V_cts.txt
> etc.
>
> But, then, when I try and pull out my next bin of data (06z -
07:59:59z),
> the PB2NC looks like this:
> Hourly: running /opt/MET6/met6.0/bin/pb2nc
> /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> uv_wind_pb_2017080206_07.nc
> /opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> -valid_beg 20170802_060000 -valid_end 20170802_075959 -v 2
> DEBUG 1: Default Config File:
> /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> DEBUG 1: User Config File:
> /opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> DEBUG 1: Creating NetCDF File:
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> uv_wind_pb_2017080206_07.nc
> DEBUG 1: Processing PrepBufr File:
> /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> DEBUG 1: Blocking PrepBufr file to: /tmp/tmp_pb2nc_blk_53440_0
> DEBUG 2: PrepBufr Time Center: 20170802_060000
> DEBUG 2: Searching Time Window: 20170802_060000 to
20170802_075959
>
> And, the POINT_STAT looks like:
> running /opt/MET6/met6.0/bin/point_stat
>
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.pgrb2.0p25.f000
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> uv_wind_pb_2017080206_07.nc
>
/opc_test/home/opc_test/all_opc/tables/met_verif/PointStatConfig_working
> -outdir /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/ -v 2
>
> But, the point_stat files look like:
> DEBUG 1: Output file:
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> point_stat_000000L_20170802_060000V.stat
> DEBUG 1: Output file:
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> point_stat_000000L_20170802_060000V_ctc.txt
> DEBUG 1: Output file:
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> point_stat_000000L_20170802_060000V_cts.txt
> etc.
>
> So, here's the questions:
> 1) Shouldn't they look like:
> mprfile =
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> point_stat_000000L_20170802_070000V_mpr.txt
> cntfile =
> /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> point_stat_000000L_20170802_070000V_cnt.txt
> When I ran this on the command line, they ended up looking like
something
> like this.
>
> 2) And, if not, and they will look like they do in the above DEBUG
> statement, how can I make the name of the point_stat files for each
bin of
> data unique so the files don't overwrite themselves?
>
> 3) Shouldn't this debug statement:
> DEBUG 2: Searching Time Window: 20170802_060000 to
20170802_075959
> be centered at 20170802_070000, and not:
> DEBUG 2: PrepBufr Time Center: 20170802_060000?
>
> Thanks!
>
> Roz
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applilcations Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Help with generating unique point_stat file names
From: Rosalyn MacCracken - NOAA Affiliate
Time: Thu Aug 03 05:29:49 2017
Hi John,
I suspected that it was overwriting the point_stat file each time it
was
created. I didn't notice this when I ran my script with my original
loop,
since I stopped it before it could get to that point. It was looping
on
lead times first, f000, f024, f048, etc, so it was creating unique
names.
It was when I switched my loop to doing all the processing for a
particular
lead time (say, f000), that this problem popped up.
Oh, and the reason why it looks like I'm comparing the 06z and 07z to
the
same forecast file is because the loop is wrong. It's not passing in
the
correct variables...something I noticed after I sent the email. It
should
be comparing 06z to the f000, and 07z to f001, 08z to f002, etc from
the
t06z cycle.
Anyway, I don't know if that prefix will help in this case. I think
that
the answer is to change the file name as soon as it's created.
Ok, thanks for helping me and answering my question. I guess you can
close
this ticket.
Roz
On Wed, Aug 2, 2017 at 7:07 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Roz,
>
> I see that you have questions about setting up routine evaluation
using
> pb2nc and point_stat. In particular, you're having issues
generating
> unique file names. There are a few important points to make...
>
> (1) When you run Point-Stat, you pass it a single forecast file. It
> figures out the valid time and lead time of that forecast (i.e.
valid_time
> = init_time + lead_time), and uses the those times in the output
file names
> it creates.
>
> (2) If you run Point-Stat multiple times on the same file, the
filenames
> will clobber each other unless you do something about it.
>
> (3) In the MET config file, the "output_prefix" string (empty by
default)
> is inserted into the output file names in order to make them unique.
We
> often run Point-Stat separately for surface and upper-air
verification and
> set "output_prefix" to ADPSFC and ADPUPA to differentiate the output
file
> names.
>
> (4) Next, PREPBUFR files usually contains observations collected
over a 3
> or 6 hour time window. I see from the commands you sent that you're
> running PB2NC once for 06z and a second time for 07z. But you're
comparing
> both the 06Z and 07Z observations to same forecast file. I don't
really
> understand why you'd want to do that.
>
> Usually, the verification logic is based on the forecast times being
> evaluated. You'd loop over the forecast hours you'd like to
evaluate and
> call Point-Stat once for each time, passing to it all the
observations
> you'd like to use.
>
> Getting back to your PREPBUFR runs, consider this file:
>
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.pgrb2.0p25.f000
> And this DEBUG line:
> DEBUG 2: PrepBufr Time Center: 20170802_060000
>
> I'm guessing that this PREPBUFR file contains observations +/- 3
hours
> around that center time. Or maybe it's +/- 1.5 hours around it.
I'm
> really not sure. When you run PB2NC, the config file contains the
> following:
> obs_window = {
> beg = -5400; // seconds
> end = 5400; // seconds
> }
>
> This tell PB2NC what observations to retain... the center time of
the file
> + beg to the center time + end. However, the -valid_beg and
-valid_end
> command line options that you used override these settings. Either
way of
> specifying the time window is fine... sometimes its more convenient
to do
> it one way or the other.
>
> Hopefully that helps fill in the details.
>
> Please let me know what other questions you have.
>
> Thanks,
> John Halley Gotway
>
>
> On Wed, Aug 2, 2017 at 9:21 AM, Rosalyn MacCracken - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Aug 02 09:21:38 2017: Request 81437 was acted upon.
> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > Queue: met_help
> > Subject: Help with generating unique point_stat file names
> > Owner: Nobody
> > Requestors: rosalyn.maccracken at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81437 >
> >
> >
> > Hi,
> >
> > I'm trying to run MET automatically via a python programming.
This
> program
> > passes in dates and hours, and passes these variables into the
PB2NC and
> > POINT_STAT tools. Also, I'm extracting out bins of data according
to the
> > hour from a PrepBufr, which I use in the POINT_STAT tool. When I
tested
> > these ideas on the command line, they worked properly, as I
expected,
> but,
> > when I put them into my automated program, the point_stat_* file
names
> were
> > not correct.
> >
> > So, as an example, I'm trying to run this for the 06z cycle. My
first
> time
> > through my loop, I'm pulling out data from 05z - 06:59:59z, and my
PB2NC
> > looks like:
> >
> > Hourly: running /opt/MET6/met6.0/bin/pb2nc
> >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > uv_wind_pb_2017080206_06.nc
> >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > -valid_beg 20170802_050000 -valid_end 20170802_065959 -v 2
> > DEBUG 1: Default Config File:
> > /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> > DEBUG 1: User Config File:
> >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > DEBUG 1: Creating NetCDF File:
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > uv_wind_pb_2017080206_06.nc
> > DEBUG 1: Processing PrepBufr File:
> >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > DEBUG 1: Blocking PrepBufr file to: /tmp/tmp_pb2nc_blk_50115_0
> > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > DEBUG 2: Searching Time Window: 20170802_050000 to
20170802_065959
> >
> > And, POINT_STAT looks like:
> >
> > running /opt/MET6/met6.0/bin/point_stat
> >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.pgrb2.0p25.f000
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > uv_wind_pb_2017080206_06.nc
> >
/opc_test/home/opc_test/all_opc/tables/met_verif/PointStatConfig_working
> > -outdir /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/ -v
2
> >
> > And, generate files like:
> > DEBUG 1: Output file:
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > point_stat_000000L_20170802_060000V.stat
> > DEBUG 1: Output file:
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > point_stat_000000L_20170802_060000V_ctc.txt
> > DEBUG 1: Output file:
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > point_stat_000000L_20170802_060000V_cts.txt
> > etc.
> >
> > But, then, when I try and pull out my next bin of data (06z -
07:59:59z),
> > the PB2NC looks like this:
> > Hourly: running /opt/MET6/met6.0/bin/pb2nc
> >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > uv_wind_pb_2017080206_07.nc
> >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > -valid_beg 20170802_060000 -valid_end 20170802_075959 -v 2
> > DEBUG 1: Default Config File:
> > /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> > DEBUG 1: User Config File:
> >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > DEBUG 1: Creating NetCDF File:
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > uv_wind_pb_2017080206_07.nc
> > DEBUG 1: Processing PrepBufr File:
> >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > DEBUG 1: Blocking PrepBufr file to: /tmp/tmp_pb2nc_blk_53440_0
> > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > DEBUG 2: Searching Time Window: 20170802_060000 to
20170802_075959
> >
> > And, the POINT_STAT looks like:
> > running /opt/MET6/met6.0/bin/point_stat
> >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.pgrb2.0p25.f000
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > uv_wind_pb_2017080206_07.nc
> >
/opc_test/home/opc_test/all_opc/tables/met_verif/PointStatConfig_working
> > -outdir /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/ -v
2
> >
> > But, the point_stat files look like:
> > DEBUG 1: Output file:
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > point_stat_000000L_20170802_060000V.stat
> > DEBUG 1: Output file:
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > point_stat_000000L_20170802_060000V_ctc.txt
> > DEBUG 1: Output file:
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > point_stat_000000L_20170802_060000V_cts.txt
> > etc.
> >
> > So, here's the questions:
> > 1) Shouldn't they look like:
> > mprfile =
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > point_stat_000000L_20170802_070000V_mpr.txt
> > cntfile =
> > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > point_stat_000000L_20170802_070000V_cnt.txt
> > When I ran this on the command line, they ended up looking like
something
> > like this.
> >
> > 2) And, if not, and they will look like they do in the above
DEBUG
> > statement, how can I make the name of the point_stat files for
each bin
> of
> > data unique so the files don't overwrite themselves?
> >
> > 3) Shouldn't this debug statement:
> > DEBUG 2: Searching Time Window: 20170802_060000 to
20170802_075959
> > be centered at 20170802_070000, and not:
> > DEBUG 2: PrepBufr Time Center: 20170802_060000?
> >
> > Thanks!
> >
> > Roz
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applilcations Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applilcations Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with generating unique point_stat file names
From: John Halley Gotway
Time: Thu Aug 03 10:19:23 2017
Roz,
Point-Stat includes the forecast lead time and the valid time in the
output
file name. The only time it'll overwrite output files is when you're
passing to point-stat multiple forecast files with the same lead and
valid
times. I don't fully understand the details of the system you're
setting
up, but if file names are being over-written, you should be able to
use
output_prefix to fix the problem.
Changing the output file names after they're created will likely be
messier
logic and much more prone to error.
I would suggest using an environment variable to set output_prefix.
For
example, perhaps you want to include the model name and current
configuration you're running in the output file names...
output_prefix = "${MODEL}_${CONFIG}";
Then in your script that calls point_stat, set the MODEL and CONFIG
environment variables before calling Point-Stat. And of course, set
the
output_prefix in whatever way makes the file names unique.
Hope that helps.
Thanks,
John
On Thu, Aug 3, 2017 at 5:29 AM, Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81437 >
>
> Hi John,
>
> I suspected that it was overwriting the point_stat file each time it
was
> created. I didn't notice this when I ran my script with my original
loop,
> since I stopped it before it could get to that point. It was
looping on
> lead times first, f000, f024, f048, etc, so it was creating unique
names.
> It was when I switched my loop to doing all the processing for a
particular
> lead time (say, f000), that this problem popped up.
>
> Oh, and the reason why it looks like I'm comparing the 06z and 07z
to the
> same forecast file is because the loop is wrong. It's not passing
in the
> correct variables...something I noticed after I sent the email. It
should
> be comparing 06z to the f000, and 07z to f001, 08z to f002, etc from
the
> t06z cycle.
>
> Anyway, I don't know if that prefix will help in this case. I think
that
> the answer is to change the file name as soon as it's created.
>
> Ok, thanks for helping me and answering my question. I guess you
can close
> this ticket.
>
> Roz
>
> On Wed, Aug 2, 2017 at 7:07 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Roz,
> >
> > I see that you have questions about setting up routine evaluation
using
> > pb2nc and point_stat. In particular, you're having issues
generating
> > unique file names. There are a few important points to make...
> >
> > (1) When you run Point-Stat, you pass it a single forecast file.
It
> > figures out the valid time and lead time of that forecast (i.e.
> valid_time
> > = init_time + lead_time), and uses the those times in the output
file
> names
> > it creates.
> >
> > (2) If you run Point-Stat multiple times on the same file, the
filenames
> > will clobber each other unless you do something about it.
> >
> > (3) In the MET config file, the "output_prefix" string (empty by
default)
> > is inserted into the output file names in order to make them
unique. We
> > often run Point-Stat separately for surface and upper-air
verification
> and
> > set "output_prefix" to ADPSFC and ADPUPA to differentiate the
output file
> > names.
> >
> > (4) Next, PREPBUFR files usually contains observations collected
over a 3
> > or 6 hour time window. I see from the commands you sent that
you're
> > running PB2NC once for 06z and a second time for 07z. But you're
> comparing
> > both the 06Z and 07Z observations to same forecast file. I don't
really
> > understand why you'd want to do that.
> >
> > Usually, the verification logic is based on the forecast times
being
> > evaluated. You'd loop over the forecast hours you'd like to
evaluate and
> > call Point-Stat once for each time, passing to it all the
observations
> > you'd like to use.
> >
> > Getting back to your PREPBUFR runs, consider this file:
> > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> pgrb2.0p25.f000
> > And this DEBUG line:
> > DEBUG 2: PrepBufr Time Center: 20170802_060000
> >
> > I'm guessing that this PREPBUFR file contains observations +/- 3
hours
> > around that center time. Or maybe it's +/- 1.5 hours around it.
I'm
> > really not sure. When you run PB2NC, the config file contains the
> > following:
> > obs_window = {
> > beg = -5400; // seconds
> > end = 5400; // seconds
> > }
> >
> > This tell PB2NC what observations to retain... the center time of
the
> file
> > + beg to the center time + end. However, the -valid_beg and
-valid_end
> > command line options that you used override these settings.
Either way
> of
> > specifying the time window is fine... sometimes its more
convenient to do
> > it one way or the other.
> >
> > Hopefully that helps fill in the details.
> >
> > Please let me know what other questions you have.
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> > On Wed, Aug 2, 2017 at 9:21 AM, Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Aug 02 09:21:38 2017: Request 81437 was acted upon.
> > > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > > Queue: met_help
> > > Subject: Help with generating unique point_stat file names
> > > Owner: Nobody
> > > Requestors: rosalyn.maccracken at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81437
> >
> > >
> > >
> > > Hi,
> > >
> > > I'm trying to run MET automatically via a python programming.
This
> > program
> > > passes in dates and hours, and passes these variables into the
PB2NC
> and
> > > POINT_STAT tools. Also, I'm extracting out bins of data
according to
> the
> > > hour from a PrepBufr, which I use in the POINT_STAT tool. When
I
> tested
> > > these ideas on the command line, they worked properly, as I
expected,
> > but,
> > > when I put them into my automated program, the point_stat_* file
names
> > were
> > > not correct.
> > >
> > > So, as an example, I'm trying to run this for the 06z cycle. My
first
> > time
> > > through my loop, I'm pulling out data from 05z - 06:59:59z, and
my
> PB2NC
> > > looks like:
> > >
> > > Hourly: running /opt/MET6/met6.0/bin/pb2nc
> > >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > uv_wind_pb_2017080206_06.nc
> > >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > > -valid_beg 20170802_050000 -valid_end 20170802_065959 -v 2
> > > DEBUG 1: Default Config File:
> > > /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> > > DEBUG 1: User Config File:
> > >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > > DEBUG 1: Creating NetCDF File:
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > uv_wind_pb_2017080206_06.nc
> > > DEBUG 1: Processing PrepBufr File:
> > >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > > DEBUG 1: Blocking PrepBufr file to:
/tmp/tmp_pb2nc_blk_50115_0
> > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > > DEBUG 2: Searching Time Window: 20170802_050000 to
> 20170802_065959
> > >
> > > And, POINT_STAT looks like:
> > >
> > > running /opt/MET6/met6.0/bin/point_stat
> > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> pgrb2.0p25.f000
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > uv_wind_pb_2017080206_06.nc
> > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> PointStatConfig_working
> > > -outdir /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
-v 2
> > >
> > > And, generate files like:
> > > DEBUG 1: Output file:
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > point_stat_000000L_20170802_060000V.stat
> > > DEBUG 1: Output file:
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > point_stat_000000L_20170802_060000V_ctc.txt
> > > DEBUG 1: Output file:
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > point_stat_000000L_20170802_060000V_cts.txt
> > > etc.
> > >
> > > But, then, when I try and pull out my next bin of data (06z -
> 07:59:59z),
> > > the PB2NC looks like this:
> > > Hourly: running /opt/MET6/met6.0/bin/pb2nc
> > >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > uv_wind_pb_2017080206_07.nc
> > >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > > -valid_beg 20170802_060000 -valid_end 20170802_075959 -v 2
> > > DEBUG 1: Default Config File:
> > > /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> > > DEBUG 1: User Config File:
> > >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > > DEBUG 1: Creating NetCDF File:
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > uv_wind_pb_2017080206_07.nc
> > > DEBUG 1: Processing PrepBufr File:
> > >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > > DEBUG 1: Blocking PrepBufr file to:
/tmp/tmp_pb2nc_blk_53440_0
> > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > > DEBUG 2: Searching Time Window: 20170802_060000 to
> 20170802_075959
> > >
> > > And, the POINT_STAT looks like:
> > > running /opt/MET6/met6.0/bin/point_stat
> > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> pgrb2.0p25.f000
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > uv_wind_pb_2017080206_07.nc
> > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> PointStatConfig_working
> > > -outdir /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
-v 2
> > >
> > > But, the point_stat files look like:
> > > DEBUG 1: Output file:
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > point_stat_000000L_20170802_060000V.stat
> > > DEBUG 1: Output file:
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > point_stat_000000L_20170802_060000V_ctc.txt
> > > DEBUG 1: Output file:
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > point_stat_000000L_20170802_060000V_cts.txt
> > > etc.
> > >
> > > So, here's the questions:
> > > 1) Shouldn't they look like:
> > > mprfile =
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > point_stat_000000L_20170802_070000V_mpr.txt
> > > cntfile =
> > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > point_stat_000000L_20170802_070000V_cnt.txt
> > > When I ran this on the command line, they ended up looking like
> something
> > > like this.
> > >
> > > 2) And, if not, and they will look like they do in the above
DEBUG
> > > statement, how can I make the name of the point_stat files for
each bin
> > of
> > > data unique so the files don't overwrite themselves?
> > >
> > > 3) Shouldn't this debug statement:
> > > DEBUG 2: Searching Time Window: 20170802_060000 to
> 20170802_075959
> > > be centered at 20170802_070000, and not:
> > > DEBUG 2: PrepBufr Time Center: 20170802_060000?
> > >
> > > Thanks!
> > >
> > > Roz
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applilcations Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applilcations Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Help with generating unique point_stat file names
From: Rosalyn MacCracken - NOAA Affiliate
Time: Thu Aug 03 10:26:00 2017
Hi John,
I actually had a separate error in the forecast lead time loop. The
one
that you pointed out in the earlier email. When I fixed that error, I
starting generating files with unique names, with forecast leads and
corresponding valid times:
point_stat_010000L_20170803_070000V_mpr.txt
point_stat_020000L_20170803_080000V_mpr.txt
point_stat_030000L_20170803_090000V_mpr.txt
So, that fixed that issue. I have unrelated other issues now...but, I
think I can fix them pretty easily.
And, actually, I think I may use that PREFIX, since I'm evaluating the
GFS
and Forecast Grids, and I should keep the files separate. Thanks for
that
tip!
Roz
On Thu, Aug 3, 2017 at 12:19 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Roz,
>
> Point-Stat includes the forecast lead time and the valid time in the
output
> file name. The only time it'll overwrite output files is when
you're
> passing to point-stat multiple forecast files with the same lead and
valid
> times. I don't fully understand the details of the system you're
setting
> up, but if file names are being over-written, you should be able to
use
> output_prefix to fix the problem.
>
> Changing the output file names after they're created will likely be
messier
> logic and much more prone to error.
>
> I would suggest using an environment variable to set output_prefix.
For
> example, perhaps you want to include the model name and current
> configuration you're running in the output file names...
>
> output_prefix = "${MODEL}_${CONFIG}";
>
> Then in your script that calls point_stat, set the MODEL and CONFIG
> environment variables before calling Point-Stat. And of course, set
the
> output_prefix in whatever way makes the file names unique.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> On Thu, Aug 3, 2017 at 5:29 AM, Rosalyn MacCracken - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81437 >
> >
> > Hi John,
> >
> > I suspected that it was overwriting the point_stat file each time
it was
> > created. I didn't notice this when I ran my script with my
original
> loop,
> > since I stopped it before it could get to that point. It was
looping on
> > lead times first, f000, f024, f048, etc, so it was creating unique
names.
> > It was when I switched my loop to doing all the processing for a
> particular
> > lead time (say, f000), that this problem popped up.
> >
> > Oh, and the reason why it looks like I'm comparing the 06z and 07z
to the
> > same forecast file is because the loop is wrong. It's not passing
in the
> > correct variables...something I noticed after I sent the email.
It
> should
> > be comparing 06z to the f000, and 07z to f001, 08z to f002, etc
from the
> > t06z cycle.
> >
> > Anyway, I don't know if that prefix will help in this case. I
think that
> > the answer is to change the file name as soon as it's created.
> >
> > Ok, thanks for helping me and answering my question. I guess you
can
> close
> > this ticket.
> >
> > Roz
> >
> > On Wed, Aug 2, 2017 at 7:07 PM, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Roz,
> > >
> > > I see that you have questions about setting up routine
evaluation using
> > > pb2nc and point_stat. In particular, you're having issues
generating
> > > unique file names. There are a few important points to make...
> > >
> > > (1) When you run Point-Stat, you pass it a single forecast file.
It
> > > figures out the valid time and lead time of that forecast (i.e.
> > valid_time
> > > = init_time + lead_time), and uses the those times in the output
file
> > names
> > > it creates.
> > >
> > > (2) If you run Point-Stat multiple times on the same file, the
> filenames
> > > will clobber each other unless you do something about it.
> > >
> > > (3) In the MET config file, the "output_prefix" string (empty by
> default)
> > > is inserted into the output file names in order to make them
unique.
> We
> > > often run Point-Stat separately for surface and upper-air
verification
> > and
> > > set "output_prefix" to ADPSFC and ADPUPA to differentiate the
output
> file
> > > names.
> > >
> > > (4) Next, PREPBUFR files usually contains observations collected
over
> a 3
> > > or 6 hour time window. I see from the commands you sent that
you're
> > > running PB2NC once for 06z and a second time for 07z. But
you're
> > comparing
> > > both the 06Z and 07Z observations to same forecast file. I
don't
> really
> > > understand why you'd want to do that.
> > >
> > > Usually, the verification logic is based on the forecast times
being
> > > evaluated. You'd loop over the forecast hours you'd like to
evaluate
> and
> > > call Point-Stat once for each time, passing to it all the
observations
> > > you'd like to use.
> > >
> > > Getting back to your PREPBUFR runs, consider this file:
> > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > pgrb2.0p25.f000
> > > And this DEBUG line:
> > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > >
> > > I'm guessing that this PREPBUFR file contains observations +/- 3
hours
> > > around that center time. Or maybe it's +/- 1.5 hours around it.
I'm
> > > really not sure. When you run PB2NC, the config file contains
the
> > > following:
> > > obs_window = {
> > > beg = -5400; // seconds
> > > end = 5400; // seconds
> > > }
> > >
> > > This tell PB2NC what observations to retain... the center time
of the
> > file
> > > + beg to the center time + end. However, the -valid_beg and
-valid_end
> > > command line options that you used override these settings.
Either way
> > of
> > > specifying the time window is fine... sometimes its more
convenient to
> do
> > > it one way or the other.
> > >
> > > Hopefully that helps fill in the details.
> > >
> > > Please let me know what other questions you have.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > >
> > > On Wed, Aug 2, 2017 at 9:21 AM, Rosalyn MacCracken - NOAA
Affiliate via
> > RT
> > > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Aug 02 09:21:38 2017: Request 81437 was acted upon.
> > > > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > > > Queue: met_help
> > > > Subject: Help with generating unique point_stat file
names
> > > > Owner: Nobody
> > > > Requestors: rosalyn.maccracken at noaa.gov
> > > > Status: new
> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=81437
> > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > > I'm trying to run MET automatically via a python programming.
This
> > > program
> > > > passes in dates and hours, and passes these variables into the
PB2NC
> > and
> > > > POINT_STAT tools. Also, I'm extracting out bins of data
according to
> > the
> > > > hour from a PrepBufr, which I use in the POINT_STAT tool.
When I
> > tested
> > > > these ideas on the command line, they worked properly, as I
expected,
> > > but,
> > > > when I put them into my automated program, the point_stat_*
file
> names
> > > were
> > > > not correct.
> > > >
> > > > So, as an example, I'm trying to run this for the 06z cycle.
My
> first
> > > time
> > > > through my loop, I'm pulling out data from 05z - 06:59:59z,
and my
> > PB2NC
> > > > looks like:
> > > >
> > > > Hourly: running /opt/MET6/met6.0/bin/pb2nc
> > > >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > uv_wind_pb_2017080206_06.nc
> > > >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > > > -valid_beg 20170802_050000 -valid_end 20170802_065959 -v 2
> > > > DEBUG 1: Default Config File:
> > > > /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> > > > DEBUG 1: User Config File:
> > > >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > > > DEBUG 1: Creating NetCDF File:
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > uv_wind_pb_2017080206_06.nc
> > > > DEBUG 1: Processing PrepBufr File:
> > > >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > > > DEBUG 1: Blocking PrepBufr file to:
/tmp/tmp_pb2nc_blk_50115_0
> > > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > > > DEBUG 2: Searching Time Window: 20170802_050000 to
> > 20170802_065959
> > > >
> > > > And, POINT_STAT looks like:
> > > >
> > > > running /opt/MET6/met6.0/bin/point_stat
> > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > pgrb2.0p25.f000
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > uv_wind_pb_2017080206_06.nc
> > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> > PointStatConfig_working
> > > > -outdir
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/ -v 2
> > > >
> > > > And, generate files like:
> > > > DEBUG 1: Output file:
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > point_stat_000000L_20170802_060000V.stat
> > > > DEBUG 1: Output file:
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > point_stat_000000L_20170802_060000V_ctc.txt
> > > > DEBUG 1: Output file:
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > point_stat_000000L_20170802_060000V_cts.txt
> > > > etc.
> > > >
> > > > But, then, when I try and pull out my next bin of data (06z -
> > 07:59:59z),
> > > > the PB2NC looks like this:
> > > > Hourly: running /opt/MET6/met6.0/bin/pb2nc
> > > >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > uv_wind_pb_2017080206_07.nc
> > > >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > > > -valid_beg 20170802_060000 -valid_end 20170802_075959 -v 2
> > > > DEBUG 1: Default Config File:
> > > > /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> > > > DEBUG 1: User Config File:
> > > >
/opc_test/home/opc_test/all_opc/tables/met_verif/PB2NCConfig_working
> > > > DEBUG 1: Creating NetCDF File:
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > uv_wind_pb_2017080206_07.nc
> > > > DEBUG 1: Processing PrepBufr File:
> > > >
/common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.prepbufr.nr
> > > > DEBUG 1: Blocking PrepBufr file to:
/tmp/tmp_pb2nc_blk_53440_0
> > > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > > > DEBUG 2: Searching Time Window: 20170802_060000 to
> > 20170802_075959
> > > >
> > > > And, the POINT_STAT looks like:
> > > > running /opt/MET6/met6.0/bin/point_stat
> > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > pgrb2.0p25.f000
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > uv_wind_pb_2017080206_07.nc
> > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> > PointStatConfig_working
> > > > -outdir
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/ -v 2
> > > >
> > > > But, the point_stat files look like:
> > > > DEBUG 1: Output file:
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > point_stat_000000L_20170802_060000V.stat
> > > > DEBUG 1: Output file:
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > point_stat_000000L_20170802_060000V_ctc.txt
> > > > DEBUG 1: Output file:
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > point_stat_000000L_20170802_060000V_cts.txt
> > > > etc.
> > > >
> > > > So, here's the questions:
> > > > 1) Shouldn't they look like:
> > > > mprfile =
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > point_stat_000000L_20170802_070000V_mpr.txt
> > > > cntfile =
> > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > point_stat_000000L_20170802_070000V_cnt.txt
> > > > When I ran this on the command line, they ended up looking
like
> > something
> > > > like this.
> > > >
> > > > 2) And, if not, and they will look like they do in the above
DEBUG
> > > > statement, how can I make the name of the point_stat files for
each
> bin
> > > of
> > > > data unique so the files don't overwrite themselves?
> > > >
> > > > 3) Shouldn't this debug statement:
> > > > DEBUG 2: Searching Time Window: 20170802_060000 to
> > 20170802_075959
> > > > be centered at 20170802_070000, and not:
> > > > DEBUG 2: PrepBufr Time Center: 20170802_060000?
> > > >
> > > > Thanks!
> > > >
> > > > Roz
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applilcations Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applilcations Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applilcations Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with generating unique point_stat file names
From: John Halley Gotway
Time: Thu Aug 03 10:36:25 2017
Great, glad you got that all figured out!
John
On Thu, Aug 3, 2017 at 10:26 AM, Rosalyn MacCracken - NOAA Affiliate
via RT
<met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81437 >
>
> Hi John,
>
> I actually had a separate error in the forecast lead time loop. The
one
> that you pointed out in the earlier email. When I fixed that error,
I
> starting generating files with unique names, with forecast leads and
> corresponding valid times:
> point_stat_010000L_20170803_070000V_mpr.txt
> point_stat_020000L_20170803_080000V_mpr.txt
> point_stat_030000L_20170803_090000V_mpr.txt
>
> So, that fixed that issue. I have unrelated other issues now...but,
I
> think I can fix them pretty easily.
>
> And, actually, I think I may use that PREFIX, since I'm evaluating
the GFS
> and Forecast Grids, and I should keep the files separate. Thanks
for that
> tip!
>
> Roz
>
> On Thu, Aug 3, 2017 at 12:19 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Roz,
> >
> > Point-Stat includes the forecast lead time and the valid time in
the
> output
> > file name. The only time it'll overwrite output files is when
you're
> > passing to point-stat multiple forecast files with the same lead
and
> valid
> > times. I don't fully understand the details of the system you're
setting
> > up, but if file names are being over-written, you should be able
to use
> > output_prefix to fix the problem.
> >
> > Changing the output file names after they're created will likely
be
> messier
> > logic and much more prone to error.
> >
> > I would suggest using an environment variable to set
output_prefix. For
> > example, perhaps you want to include the model name and current
> > configuration you're running in the output file names...
> >
> > output_prefix = "${MODEL}_${CONFIG}";
> >
> > Then in your script that calls point_stat, set the MODEL and
CONFIG
> > environment variables before calling Point-Stat. And of course,
set the
> > output_prefix in whatever way makes the file names unique.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Aug 3, 2017 at 5:29 AM, Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81437 >
> > >
> > > Hi John,
> > >
> > > I suspected that it was overwriting the point_stat file each
time it
> was
> > > created. I didn't notice this when I ran my script with my
original
> > loop,
> > > since I stopped it before it could get to that point. It was
looping
> on
> > > lead times first, f000, f024, f048, etc, so it was creating
unique
> names.
> > > It was when I switched my loop to doing all the processing for a
> > particular
> > > lead time (say, f000), that this problem popped up.
> > >
> > > Oh, and the reason why it looks like I'm comparing the 06z and
07z to
> the
> > > same forecast file is because the loop is wrong. It's not
passing in
> the
> > > correct variables...something I noticed after I sent the email.
It
> > should
> > > be comparing 06z to the f000, and 07z to f001, 08z to f002, etc
from
> the
> > > t06z cycle.
> > >
> > > Anyway, I don't know if that prefix will help in this case. I
think
> that
> > > the answer is to change the file name as soon as it's created.
> > >
> > > Ok, thanks for helping me and answering my question. I guess
you can
> > close
> > > this ticket.
> > >
> > > Roz
> > >
> > > On Wed, Aug 2, 2017 at 7:07 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu
> > > > wrote:
> > >
> > > > Roz,
> > > >
> > > > I see that you have questions about setting up routine
evaluation
> using
> > > > pb2nc and point_stat. In particular, you're having issues
generating
> > > > unique file names. There are a few important points to
make...
> > > >
> > > > (1) When you run Point-Stat, you pass it a single forecast
file. It
> > > > figures out the valid time and lead time of that forecast
(i.e.
> > > valid_time
> > > > = init_time + lead_time), and uses the those times in the
output file
> > > names
> > > > it creates.
> > > >
> > > > (2) If you run Point-Stat multiple times on the same file, the
> > filenames
> > > > will clobber each other unless you do something about it.
> > > >
> > > > (3) In the MET config file, the "output_prefix" string (empty
by
> > default)
> > > > is inserted into the output file names in order to make them
unique.
> > We
> > > > often run Point-Stat separately for surface and upper-air
> verification
> > > and
> > > > set "output_prefix" to ADPSFC and ADPUPA to differentiate the
output
> > file
> > > > names.
> > > >
> > > > (4) Next, PREPBUFR files usually contains observations
collected over
> > a 3
> > > > or 6 hour time window. I see from the commands you sent that
you're
> > > > running PB2NC once for 06z and a second time for 07z. But
you're
> > > comparing
> > > > both the 06Z and 07Z observations to same forecast file. I
don't
> > really
> > > > understand why you'd want to do that.
> > > >
> > > > Usually, the verification logic is based on the forecast times
being
> > > > evaluated. You'd loop over the forecast hours you'd like to
evaluate
> > and
> > > > call Point-Stat once for each time, passing to it all the
> observations
> > > > you'd like to use.
> > > >
> > > > Getting back to your PREPBUFR runs, consider this file:
> > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > > pgrb2.0p25.f000
> > > > And this DEBUG line:
> > > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > > >
> > > > I'm guessing that this PREPBUFR file contains observations +/-
3
> hours
> > > > around that center time. Or maybe it's +/- 1.5 hours around
it. I'm
> > > > really not sure. When you run PB2NC, the config file contains
the
> > > > following:
> > > > obs_window = {
> > > > beg = -5400; // seconds
> > > > end = 5400; // seconds
> > > > }
> > > >
> > > > This tell PB2NC what observations to retain... the center time
of the
> > > file
> > > > + beg to the center time + end. However, the -valid_beg and
> -valid_end
> > > > command line options that you used override these settings.
Either
> way
> > > of
> > > > specifying the time window is fine... sometimes its more
convenient
> to
> > do
> > > > it one way or the other.
> > > >
> > > > Hopefully that helps fill in the details.
> > > >
> > > > Please let me know what other questions you have.
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > >
> > > > On Wed, Aug 2, 2017 at 9:21 AM, Rosalyn MacCracken - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Aug 02 09:21:38 2017: Request 81437 was acted upon.
> > > > > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: Help with generating unique point_stat file
names
> > > > > Owner: Nobody
> > > > > Requestors: rosalyn.maccracken at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=81437
> > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm trying to run MET automatically via a python
programming. This
> > > > program
> > > > > passes in dates and hours, and passes these variables into
the
> PB2NC
> > > and
> > > > > POINT_STAT tools. Also, I'm extracting out bins of data
according
> to
> > > the
> > > > > hour from a PrepBufr, which I use in the POINT_STAT tool.
When I
> > > tested
> > > > > these ideas on the command line, they worked properly, as I
> expected,
> > > > but,
> > > > > when I put them into my automated program, the point_stat_*
file
> > names
> > > > were
> > > > > not correct.
> > > > >
> > > > > So, as an example, I'm trying to run this for the 06z cycle.
My
> > first
> > > > time
> > > > > through my loop, I'm pulling out data from 05z - 06:59:59z,
and my
> > > PB2NC
> > > > > looks like:
> > > > >
> > > > > Hourly: running /opt/MET6/met6.0/bin/pb2nc
> > > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> prepbufr.nr
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > uv_wind_pb_2017080206_06.nc
> > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> PB2NCConfig_working
> > > > > -valid_beg 20170802_050000 -valid_end 20170802_065959 -v 2
> > > > > DEBUG 1: Default Config File:
> > > > > /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> > > > > DEBUG 1: User Config File:
> > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> PB2NCConfig_working
> > > > > DEBUG 1: Creating NetCDF File:
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > uv_wind_pb_2017080206_06.nc
> > > > > DEBUG 1: Processing PrepBufr File:
> > > > > /common/data/model/com/gfs/prod/gfs.2017080206/
> gfs.t06z.prepbufr.nr
> > > > > DEBUG 1: Blocking PrepBufr file to:
/tmp/tmp_pb2nc_blk_50115_0
> > > > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > > > > DEBUG 2: Searching Time Window: 20170802_050000 to
> > > 20170802_065959
> > > > >
> > > > > And, POINT_STAT looks like:
> > > > >
> > > > > running /opt/MET6/met6.0/bin/point_stat
> > > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > > pgrb2.0p25.f000
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > uv_wind_pb_2017080206_06.nc
> > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> > > PointStatConfig_working
> > > > > -outdir
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> -v 2
> > > > >
> > > > > And, generate files like:
> > > > > DEBUG 1: Output file:
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > point_stat_000000L_20170802_060000V.stat
> > > > > DEBUG 1: Output file:
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > point_stat_000000L_20170802_060000V_ctc.txt
> > > > > DEBUG 1: Output file:
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > point_stat_000000L_20170802_060000V_cts.txt
> > > > > etc.
> > > > >
> > > > > But, then, when I try and pull out my next bin of data (06z
-
> > > 07:59:59z),
> > > > > the PB2NC looks like this:
> > > > > Hourly: running /opt/MET6/met6.0/bin/pb2nc
> > > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> prepbufr.nr
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > uv_wind_pb_2017080206_07.nc
> > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> PB2NCConfig_working
> > > > > -valid_beg 20170802_060000 -valid_end 20170802_075959 -v 2
> > > > > DEBUG 1: Default Config File:
> > > > > /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> > > > > DEBUG 1: User Config File:
> > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> PB2NCConfig_working
> > > > > DEBUG 1: Creating NetCDF File:
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > uv_wind_pb_2017080206_07.nc
> > > > > DEBUG 1: Processing PrepBufr File:
> > > > > /common/data/model/com/gfs/prod/gfs.2017080206/
> gfs.t06z.prepbufr.nr
> > > > > DEBUG 1: Blocking PrepBufr file to:
/tmp/tmp_pb2nc_blk_53440_0
> > > > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > > > > DEBUG 2: Searching Time Window: 20170802_060000 to
> > > 20170802_075959
> > > > >
> > > > > And, the POINT_STAT looks like:
> > > > > running /opt/MET6/met6.0/bin/point_stat
> > > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > > pgrb2.0p25.f000
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > uv_wind_pb_2017080206_07.nc
> > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> > > PointStatConfig_working
> > > > > -outdir
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> -v 2
> > > > >
> > > > > But, the point_stat files look like:
> > > > > DEBUG 1: Output file:
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > point_stat_000000L_20170802_060000V.stat
> > > > > DEBUG 1: Output file:
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > point_stat_000000L_20170802_060000V_ctc.txt
> > > > > DEBUG 1: Output file:
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > point_stat_000000L_20170802_060000V_cts.txt
> > > > > etc.
> > > > >
> > > > > So, here's the questions:
> > > > > 1) Shouldn't they look like:
> > > > > mprfile =
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > point_stat_000000L_20170802_070000V_mpr.txt
> > > > > cntfile =
> > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > point_stat_000000L_20170802_070000V_cnt.txt
> > > > > When I ran this on the command line, they ended up looking
like
> > > something
> > > > > like this.
> > > > >
> > > > > 2) And, if not, and they will look like they do in the
above DEBUG
> > > > > statement, how can I make the name of the point_stat files
for each
> > bin
> > > > of
> > > > > data unique so the files don't overwrite themselves?
> > > > >
> > > > > 3) Shouldn't this debug statement:
> > > > > DEBUG 2: Searching Time Window: 20170802_060000 to
> > > 20170802_075959
> > > > > be centered at 20170802_070000, and not:
> > > > > DEBUG 2: PrepBufr Time Center: 20170802_060000?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Roz
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applilcations Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applilcations Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applilcations Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Help with generating unique point_stat file names
From: Rosalyn MacCracken - NOAA Affiliate
Time: Thu Aug 03 10:38:21 2017
So am I! Thanks for your help!!
Roz
On Thu, Aug 3, 2017 at 12:36 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Great, glad you got that all figured out!
>
> John
>
> On Thu, Aug 3, 2017 at 10:26 AM, Rosalyn MacCracken - NOAA Affiliate
via RT
> <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81437 >
> >
> > Hi John,
> >
> > I actually had a separate error in the forecast lead time loop.
The one
> > that you pointed out in the earlier email. When I fixed that
error, I
> > starting generating files with unique names, with forecast leads
and
> > corresponding valid times:
> > point_stat_010000L_20170803_070000V_mpr.txt
> > point_stat_020000L_20170803_080000V_mpr.txt
> > point_stat_030000L_20170803_090000V_mpr.txt
> >
> > So, that fixed that issue. I have unrelated other issues
now...but, I
> > think I can fix them pretty easily.
> >
> > And, actually, I think I may use that PREFIX, since I'm evaluating
the
> GFS
> > and Forecast Grids, and I should keep the files separate. Thanks
for
> that
> > tip!
> >
> > Roz
> >
> > On Thu, Aug 3, 2017 at 12:19 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Roz,
> > >
> > > Point-Stat includes the forecast lead time and the valid time in
the
> > output
> > > file name. The only time it'll overwrite output files is when
you're
> > > passing to point-stat multiple forecast files with the same lead
and
> > valid
> > > times. I don't fully understand the details of the system
you're
> setting
> > > up, but if file names are being over-written, you should be able
to use
> > > output_prefix to fix the problem.
> > >
> > > Changing the output file names after they're created will likely
be
> > messier
> > > logic and much more prone to error.
> > >
> > > I would suggest using an environment variable to set
output_prefix.
> For
> > > example, perhaps you want to include the model name and current
> > > configuration you're running in the output file names...
> > >
> > > output_prefix = "${MODEL}_${CONFIG}";
> > >
> > > Then in your script that calls point_stat, set the MODEL and
CONFIG
> > > environment variables before calling Point-Stat. And of course,
set
> the
> > > output_prefix in whatever way makes the file names unique.
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Thu, Aug 3, 2017 at 5:29 AM, Rosalyn MacCracken - NOAA
Affiliate via
> > RT
> > > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81437
>
> > > >
> > > > Hi John,
> > > >
> > > > I suspected that it was overwriting the point_stat file each
time it
> > was
> > > > created. I didn't notice this when I ran my script with my
original
> > > loop,
> > > > since I stopped it before it could get to that point. It was
looping
> > on
> > > > lead times first, f000, f024, f048, etc, so it was creating
unique
> > names.
> > > > It was when I switched my loop to doing all the processing for
a
> > > particular
> > > > lead time (say, f000), that this problem popped up.
> > > >
> > > > Oh, and the reason why it looks like I'm comparing the 06z and
07z to
> > the
> > > > same forecast file is because the loop is wrong. It's not
passing in
> > the
> > > > correct variables...something I noticed after I sent the
email. It
> > > should
> > > > be comparing 06z to the f000, and 07z to f001, 08z to f002,
etc from
> > the
> > > > t06z cycle.
> > > >
> > > > Anyway, I don't know if that prefix will help in this case. I
think
> > that
> > > > the answer is to change the file name as soon as it's created.
> > > >
> > > > Ok, thanks for helping me and answering my question. I guess
you can
> > > close
> > > > this ticket.
> > > >
> > > > Roz
> > > >
> > > > On Wed, Aug 2, 2017 at 7:07 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu
> > > > > wrote:
> > > >
> > > > > Roz,
> > > > >
> > > > > I see that you have questions about setting up routine
evaluation
> > using
> > > > > pb2nc and point_stat. In particular, you're having issues
> generating
> > > > > unique file names. There are a few important points to
make...
> > > > >
> > > > > (1) When you run Point-Stat, you pass it a single forecast
file.
> It
> > > > > figures out the valid time and lead time of that forecast
(i.e.
> > > > valid_time
> > > > > = init_time + lead_time), and uses the those times in the
output
> file
> > > > names
> > > > > it creates.
> > > > >
> > > > > (2) If you run Point-Stat multiple times on the same file,
the
> > > filenames
> > > > > will clobber each other unless you do something about it.
> > > > >
> > > > > (3) In the MET config file, the "output_prefix" string
(empty by
> > > default)
> > > > > is inserted into the output file names in order to make them
> unique.
> > > We
> > > > > often run Point-Stat separately for surface and upper-air
> > verification
> > > > and
> > > > > set "output_prefix" to ADPSFC and ADPUPA to differentiate
the
> output
> > > file
> > > > > names.
> > > > >
> > > > > (4) Next, PREPBUFR files usually contains observations
collected
> over
> > > a 3
> > > > > or 6 hour time window. I see from the commands you sent
that
> you're
> > > > > running PB2NC once for 06z and a second time for 07z. But
you're
> > > > comparing
> > > > > both the 06Z and 07Z observations to same forecast file. I
don't
> > > really
> > > > > understand why you'd want to do that.
> > > > >
> > > > > Usually, the verification logic is based on the forecast
times
> being
> > > > > evaluated. You'd loop over the forecast hours you'd like to
> evaluate
> > > and
> > > > > call Point-Stat once for each time, passing to it all the
> > observations
> > > > > you'd like to use.
> > > > >
> > > > > Getting back to your PREPBUFR runs, consider this file:
> > > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > > > pgrb2.0p25.f000
> > > > > And this DEBUG line:
> > > > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > > > >
> > > > > I'm guessing that this PREPBUFR file contains observations
+/- 3
> > hours
> > > > > around that center time. Or maybe it's +/- 1.5 hours around
it.
> I'm
> > > > > really not sure. When you run PB2NC, the config file
contains the
> > > > > following:
> > > > > obs_window = {
> > > > > beg = -5400; // seconds
> > > > > end = 5400; // seconds
> > > > > }
> > > > >
> > > > > This tell PB2NC what observations to retain... the center
time of
> the
> > > > file
> > > > > + beg to the center time + end. However, the -valid_beg and
> > -valid_end
> > > > > command line options that you used override these settings.
Either
> > way
> > > > of
> > > > > specifying the time window is fine... sometimes its more
convenient
> > to
> > > do
> > > > > it one way or the other.
> > > > >
> > > > > Hopefully that helps fill in the details.
> > > > >
> > > > > Please let me know what other questions you have.
> > > > >
> > > > > Thanks,
> > > > > John Halley Gotway
> > > > >
> > > > >
> > > > > On Wed, Aug 2, 2017 at 9:21 AM, Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Wed Aug 02 09:21:38 2017: Request 81437 was acted upon.
> > > > > > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: Help with generating unique point_stat file
names
> > > > > > Owner: Nobody
> > > > > > Requestors: rosalyn.maccracken at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=81437
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm trying to run MET automatically via a python
programming.
> This
> > > > > program
> > > > > > passes in dates and hours, and passes these variables into
the
> > PB2NC
> > > > and
> > > > > > POINT_STAT tools. Also, I'm extracting out bins of data
> according
> > to
> > > > the
> > > > > > hour from a PrepBufr, which I use in the POINT_STAT tool.
When I
> > > > tested
> > > > > > these ideas on the command line, they worked properly, as
I
> > expected,
> > > > > but,
> > > > > > when I put them into my automated program, the
point_stat_* file
> > > names
> > > > > were
> > > > > > not correct.
> > > > > >
> > > > > > So, as an example, I'm trying to run this for the 06z
cycle. My
> > > first
> > > > > time
> > > > > > through my loop, I'm pulling out data from 05z -
06:59:59z, and
> my
> > > > PB2NC
> > > > > > looks like:
> > > > > >
> > > > > > Hourly: running /opt/MET6/met6.0/bin/pb2nc
> > > > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > prepbufr.nr
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > > uv_wind_pb_2017080206_06.nc
> > > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> > PB2NCConfig_working
> > > > > > -valid_beg 20170802_050000 -valid_end 20170802_065959 -v 2
> > > > > > DEBUG 1: Default Config File:
> > > > > > /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> > > > > > DEBUG 1: User Config File:
> > > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> > PB2NCConfig_working
> > > > > > DEBUG 1: Creating NetCDF File:
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > > uv_wind_pb_2017080206_06.nc
> > > > > > DEBUG 1: Processing PrepBufr File:
> > > > > > /common/data/model/com/gfs/prod/gfs.2017080206/
> > gfs.t06z.prepbufr.nr
> > > > > > DEBUG 1: Blocking PrepBufr file to:
/tmp/tmp_pb2nc_blk_50115_0
> > > > > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > > > > > DEBUG 2: Searching Time Window: 20170802_050000 to
> > > > 20170802_065959
> > > > > >
> > > > > > And, POINT_STAT looks like:
> > > > > >
> > > > > > running /opt/MET6/met6.0/bin/point_stat
> > > > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > > > pgrb2.0p25.f000
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > > uv_wind_pb_2017080206_06.nc
> > > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> > > > PointStatConfig_working
> > > > > > -outdir
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > -v 2
> > > > > >
> > > > > > And, generate files like:
> > > > > > DEBUG 1: Output file:
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > > point_stat_000000L_20170802_060000V.stat
> > > > > > DEBUG 1: Output file:
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > > point_stat_000000L_20170802_060000V_ctc.txt
> > > > > > DEBUG 1: Output file:
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > > point_stat_000000L_20170802_060000V_cts.txt
> > > > > > etc.
> > > > > >
> > > > > > But, then, when I try and pull out my next bin of data
(06z -
> > > > 07:59:59z),
> > > > > > the PB2NC looks like this:
> > > > > > Hourly: running /opt/MET6/met6.0/bin/pb2nc
> > > > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > prepbufr.nr
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > > uv_wind_pb_2017080206_07.nc
> > > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> > PB2NCConfig_working
> > > > > > -valid_beg 20170802_060000 -valid_end 20170802_075959 -v 2
> > > > > > DEBUG 1: Default Config File:
> > > > > > /opt/MET6/met6.0/share/met/config/PB2NCConfig_default
> > > > > > DEBUG 1: User Config File:
> > > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> > PB2NCConfig_working
> > > > > > DEBUG 1: Creating NetCDF File:
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > > uv_wind_pb_2017080206_07.nc
> > > > > > DEBUG 1: Processing PrepBufr File:
> > > > > > /common/data/model/com/gfs/prod/gfs.2017080206/
> > gfs.t06z.prepbufr.nr
> > > > > > DEBUG 1: Blocking PrepBufr file to:
/tmp/tmp_pb2nc_blk_53440_0
> > > > > > DEBUG 2: PrepBufr Time Center: 20170802_060000
> > > > > > DEBUG 2: Searching Time Window: 20170802_060000 to
> > > > 20170802_075959
> > > > > >
> > > > > > And, the POINT_STAT looks like:
> > > > > > running /opt/MET6/met6.0/bin/point_stat
> > > > > > /common/data/model/com/gfs/prod/gfs.2017080206/gfs.t06z.
> > > > pgrb2.0p25.f000
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > > uv_wind_pb_2017080206_07.nc
> > > > > > /opc_test/home/opc_test/all_opc/tables/met_verif/
> > > > PointStatConfig_working
> > > > > > -outdir
/opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > -v 2
> > > > > >
> > > > > > But, the point_stat files look like:
> > > > > > DEBUG 1: Output file:
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > > point_stat_000000L_20170802_060000V.stat
> > > > > > DEBUG 1: Output file:
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > > point_stat_000000L_20170802_060000V_ctc.txt
> > > > > > DEBUG 1: Output file:
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly//
> > > > > > point_stat_000000L_20170802_060000V_cts.txt
> > > > > > etc.
> > > > > >
> > > > > > So, here's the questions:
> > > > > > 1) Shouldn't they look like:
> > > > > > mprfile =
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > > point_stat_000000L_20170802_070000V_mpr.txt
> > > > > > cntfile =
> > > > > > /opc_test/home/opc_test/data/met_verif/GFS/data/hourly/
> > > > > > point_stat_000000L_20170802_070000V_cnt.txt
> > > > > > When I ran this on the command line, they ended up looking
like
> > > > something
> > > > > > like this.
> > > > > >
> > > > > > 2) And, if not, and they will look like they do in the
above
> DEBUG
> > > > > > statement, how can I make the name of the point_stat files
for
> each
> > > bin
> > > > > of
> > > > > > data unique so the files don't overwrite themselves?
> > > > > >
> > > > > > 3) Shouldn't this debug statement:
> > > > > > DEBUG 2: Searching Time Window: 20170802_060000 to
> > > > 20170802_075959
> > > > > > be centered at 20170802_070000, and not:
> > > > > > DEBUG 2: PrepBufr Time Center: 20170802_060000?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Roz
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applilcations Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applilcations Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applilcations Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applilcations Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list