[Met_help] [rt.rap.ucar.edu #88604] History for Issue with GriB2 Headers and Forecast Hours

Tue Jul 9 12:07:08 MDT 2019

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi,

I'm verifying a GFS ensemble member's 72 hour total cloud amount forecast against a 72-hour WWMCA ground truth.  When grid_stat generates its cnt.txt output file, the forecast hour reported is 66 instead of 72.

In trying to figure out why it did this, I went into the header for the GriB2 data and found the following:

30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave fcst:TCDC Total Cloud Cover [%]:ENS=+1
    ndata=259920:undef=0:mean=52.3893:min=0:max=100
    grid_template=0:winds(N/S):
        lat-lon grid:(720 x 361) units 1e-06 input WE:NS output WE:SN res 48
        lat 90.000000 to -90.000000 by 0.500000
        lon 0.000000 to 359.500000 by 0.500000 #points=259920

It looks like MET sees the range of forecast hours ("66-72" in this case) and uses the first one to calculate the forecast hour.

If I use "-VT" as an option when I degrib this model file, the valid forecast time reported back is 72 hours, not 66:

30:5224497:vt=20181208000000

Ignoring the issue of trying to figure out how to verify a 66 to 72 hour average cloud forecast with a single WWMCA file, is there any way to get grid_stat to look at the valid time instead of (presumably) the header so that I get 72 in the cnt.txt output rather than 66?

I haven't looked at ensemble_stat yet, but I do know that the other two models in GEPS are NAVGEM and CMC, and both of them have "72 hour fcst" in their header for the same field.  I'm not sure if ensemble_stat will allow us to match up members from GFS with NAVGEM and CMC if MET thinks they don't have a common forecast hour amongst them.

Thanks,
Matt

Matthew C. Sittel, Meteorologist
University Corporation for Atmospheric Research
Matthew.Sittel.ctr at us.af.mil

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Issue with GriB2 Headers and Forecast Hours
From: John Halley Gotway
Time: Thu Jan 24 10:01:48 2019

Matt,

That's a good question.  I'm surprised that it's reporting the
beginning of
the averaging period (66) instead of the end (72).  I wanted to get
some
data for testing so I pulled a recent GFS file:

*wget
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/gfs.t00z.pgrb2.0p25.f072
<http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/gfs.t00z.pgrb2.0p25.f072>*

And then I ran a plot_data_plane command to plot TCDC using the -v 4
command line option to dump out more detailed info, including the
timing
information:

*plot_data_plane gfs.t00z.pgrb2.0p25.f072 TCDC_entire_atmos.ps
'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*

I chose GRIB_lvl_typ = 10 to get TCDC for the "entire atmosphere"
based on
this GRIB2 code table (
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
5.shtml
).

And here's the timing info that was dumped out:
DEBUG 4: Data plane information:
DEBUG 4:       plane min: 0
DEBUG 4:       plane max: 100
DEBUG 4:       scan mode: 0
DEBUG 4:      valid time: 20190127_000000
DEBUG 4:       lead time: 720000
DEBUG 4:       init time: 20190124_000000
DEBUG 4:     bitmap flag: 255

One confusing bit is that when I reran with met-6.1, the GRIB level
codes
were reported as different values.

But that wrinkle aside, I'm not see the discrepancy in timing info
that you
describe.

Any thoughts?

Thanks,
John

On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:

>
> Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
>        Queue: met_help
>      Subject: Issue with GriB2 Headers and Forecast Hours
>        Owner: Nobody
>   Requestors: matthew.sittel.ctr at us.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
>
> Hi,
>
> I'm verifying a GFS ensemble member's 72 hour total cloud amount
forecast
> against a 72-hour WWMCA ground truth.  When grid_stat generates its
cnt.txt
> output file, the forecast hour reported is 66 instead of 72.
>
> In trying to figure out why it did this, I went into the header for
the
> GriB2 data and found the following:
>
> 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave fcst:TCDC
Total
> Cloud Cover [%]:ENS=+1
>     ndata=259920:undef=0:mean=52.3893:min=0:max=100
>     grid_template=0:winds(N/S):
>         lat-lon grid:(720 x 361) units 1e-06 input WE:NS output
WE:SN res
> 48
>         lat 90.000000 to -90.000000 by 0.500000
>         lon 0.000000 to 359.500000 by 0.500000 #points=259920
>
> It looks like MET sees the range of forecast hours ("66-72" in this
case)
> and uses the first one to calculate the forecast hour.
>
> If I use "-VT" as an option when I degrib this model file, the valid
> forecast time reported back is 72 hours, not 66:
>
> 30:5224497:vt=20181208000000
>
> Ignoring the issue of trying to figure out how to verify a 66 to 72
hour
> average cloud forecast with a single WWMCA file, is there any way to
get
> grid_stat to look at the valid time instead of (presumably) the
header so
> that I get 72 in the cnt.txt output rather than 66?
>
> I haven't looked at ensemble_stat yet, but I do know that the other
two
> models in GEPS are NAVGEM and CMC, and both of them have "72 hour
fcst" in
> their header for the same field.  I'm not sure if ensemble_stat will
allow
> us to match up members from GFS with NAVGEM and CMC if MET thinks
they
> don't have a common forecast hour amongst them.
>
> Thanks,
> Matt
>
> Matthew C. Sittel, Meteorologist
> University Corporation for Atmospheric Research
> Matthew.Sittel.ctr at us.af.mil
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with GriB2 Headers and Forecast Hours
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Jan 24 10:16:58 2019

Hey John,

What does the header look like for TCDC when you degrib that single
record?  Does it have that "66-72" field or just 72?

Maybe I should send you the model and observation files so you can try
running grid_stat on it?

Matt

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, January 24, 2019 11:02 AM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
<matthew.sittel.ctr at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with
GriB2 Headers and Forecast Hours

Matt,

That's a good question.  I'm surprised that it's reporting the
beginning of the averaging period (66) instead of the end (72).  I
wanted to get some data for testing so I pulled a recent GFS file:

*wget
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/gfs.t00z.pgrb2.0p25.f072
<http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/gfs.t00z.pgrb2.0p25.f072>*

And then I ran a plot_data_plane command to plot TCDC using the -v 4
command line option to dump out more detailed info, including the
timing
information:

*plot_data_plane gfs.t00z.pgrb2.0p25.f072 TCDC_entire_atmos.ps
'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*

I chose GRIB_lvl_typ = 10 to get TCDC for the "entire atmosphere"
based on this GRIB2 code table (
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
5.shtml
).

And here's the timing info that was dumped out:
DEBUG 4: Data plane information:
DEBUG 4:       plane min: 0
DEBUG 4:       plane max: 100
DEBUG 4:       scan mode: 0
DEBUG 4:      valid time: 20190127_000000
DEBUG 4:       lead time: 720000
DEBUG 4:       init time: 20190124_000000
DEBUG 4:     bitmap flag: 255

One confusing bit is that when I reran with met-6.1, the GRIB level
codes were reported as different values.

But that wrinkle aside, I'm not see the discrepancy in timing info
that you describe.

Any thoughts?

Thanks,
John

On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:

>
> Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
>        Queue: met_help
>      Subject: Issue with GriB2 Headers and Forecast Hours
>        Owner: Nobody
>   Requestors: matthew.sittel.ctr at us.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> >
>
>
> Hi,
>
> I'm verifying a GFS ensemble member's 72 hour total cloud amount
> forecast against a 72-hour WWMCA ground truth.  When grid_stat
> generates its cnt.txt output file, the forecast hour reported is 66
instead of 72.
>
> In trying to figure out why it did this, I went into the header for
> the
> GriB2 data and found the following:
>
> 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave fcst:TCDC
> Total Cloud Cover [%]:ENS=+1
>     ndata=259920:undef=0:mean=52.3893:min=0:max=100
>     grid_template=0:winds(N/S):
>         lat-lon grid:(720 x 361) units 1e-06 input WE:NS output
WE:SN
> res
> 48
>         lat 90.000000 to -90.000000 by 0.500000
>         lon 0.000000 to 359.500000 by 0.500000 #points=259920
>
> It looks like MET sees the range of forecast hours ("66-72" in this
> case) and uses the first one to calculate the forecast hour.
>
> If I use "-VT" as an option when I degrib this model file, the valid
> forecast time reported back is 72 hours, not 66:
>
> 30:5224497:vt=20181208000000
>
> Ignoring the issue of trying to figure out how to verify a 66 to 72
> hour average cloud forecast with a single WWMCA file, is there any
way
> to get grid_stat to look at the valid time instead of (presumably)
the
> header so that I get 72 in the cnt.txt output rather than 66?
>
> I haven't looked at ensemble_stat yet, but I do know that the other
> two models in GEPS are NAVGEM and CMC, and both of them have "72
hour
> fcst" in their header for the same field.  I'm not sure if
> ensemble_stat will allow us to match up members from GFS with NAVGEM
> and CMC if MET thinks they don't have a common forecast hour amongst
them.
>
> Thanks,
> Matt
>
> Matthew C. Sittel, Meteorologist
> University Corporation for Atmospheric Research
> Matthew.Sittel.ctr at us.af.mil
>
>
>

------------------------------------------------
Subject: Issue with GriB2 Headers and Forecast Hours
From: John Halley Gotway
Time: Thu Jan 24 11:16:53 2019

Here it is:

[johnhg at number5]% wgrib2 gfs.t00z.pgrb2.0p25.f072 | grep TCDC

317:161736166:d=2019012400:TCDC:low cloud layer:66-72 hour ave fcst:

318:162225671:d=2019012400:TCDC:middle cloud layer:66-72 hour ave
fcst:

319:162562999:d=2019012400:TCDC:high cloud layer:66-72 hour ave fcst:

320:162972761:d=2019012400:TCDC:entire atmosphere:66-72 hour ave fcst:

332:171785845:d=2019012400:TCDC:convective cloud layer:72 hour fcst:

333:172076594:d=2019012400:TCDC:boundary layer cloud layer:66-72 hour
ave
fcst:

On Thu, Jan 24, 2019 at 10:17 AM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
> Hey John,
>
> What does the header look like for TCDC when you degrib that single
> record?  Does it have that "66-72" field or just 72?
>
> Maybe I should send you the model and observation files so you can
try
> running grid_stat on it?
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 24, 2019 11:02 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with
GriB2
> Headers and Forecast Hours
>
> Matt,
>
> That's a good question.  I'm surprised that it's reporting the
beginning
> of the averaging period (66) instead of the end (72).  I wanted to
get some
> data for testing so I pulled a recent GFS file:
>
> *wget
>
>
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/gfs.t00z.pgrb2.0p25.f072
> <
>
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/gfs.t00z.pgrb2.0p25.f072
> >*
>
> And then I ran a plot_data_plane command to plot TCDC using the -v 4
> command line option to dump out more detailed info, including the
timing
> information:
>
> *plot_data_plane gfs.t00z.pgrb2.0p25.f072 TCDC_entire_atmos.ps
> 'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*
>
> I chose GRIB_lvl_typ = 10 to get TCDC for the "entire atmosphere"
based on
> this GRIB2 code table (
> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
5.shtml
> ).
>
> And here's the timing info that was dumped out:
> DEBUG 4: Data plane information:
> DEBUG 4:       plane min: 0
> DEBUG 4:       plane max: 100
> DEBUG 4:       scan mode: 0
> DEBUG 4:      valid time: 20190127_000000
> DEBUG 4:       lead time: 720000
> DEBUG 4:       init time: 20190124_000000
> DEBUG 4:     bitmap flag: 255
>
> One confusing bit is that when I reran with met-6.1, the GRIB level
codes
> were reported as different values.
>
> But that wrinkle aside, I'm not see the discrepancy in timing info
that
> you describe.
>
> Any thoughts?
>
> Thanks,
> John
>
>
>
>
> On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
> >
> > Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> >        Queue: met_help
> >      Subject: Issue with GriB2 Headers and Forecast Hours
> >        Owner: Nobody
> >   Requestors: matthew.sittel.ctr at us.af.mil
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > >
> >
> >
> > Hi,
> >
> > I'm verifying a GFS ensemble member's 72 hour total cloud amount
> > forecast against a 72-hour WWMCA ground truth.  When grid_stat
> > generates its cnt.txt output file, the forecast hour reported is
66
> instead of 72.
> >
> > In trying to figure out why it did this, I went into the header
for
> > the
> > GriB2 data and found the following:
> >
> > 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave
fcst:TCDC
> > Total Cloud Cover [%]:ENS=+1
> >     ndata=259920:undef=0:mean=52.3893:min=0:max=100
> >     grid_template=0:winds(N/S):
> >         lat-lon grid:(720 x 361) units 1e-06 input WE:NS output
WE:SN
> > res
> > 48
> >         lat 90.000000 to -90.000000 by 0.500000
> >         lon 0.000000 to 359.500000 by 0.500000 #points=259920
> >
> > It looks like MET sees the range of forecast hours ("66-72" in
this
> > case) and uses the first one to calculate the forecast hour.
> >
> > If I use "-VT" as an option when I degrib this model file, the
valid
> > forecast time reported back is 72 hours, not 66:
> >
> > 30:5224497:vt=20181208000000
> >
> > Ignoring the issue of trying to figure out how to verify a 66 to
72
> > hour average cloud forecast with a single WWMCA file, is there any
way
> > to get grid_stat to look at the valid time instead of (presumably)
the
> > header so that I get 72 in the cnt.txt output rather than 66?
> >
> > I haven't looked at ensemble_stat yet, but I do know that the
other
> > two models in GEPS are NAVGEM and CMC, and both of them have "72
hour
> > fcst" in their header for the same field.  I'm not sure if
> > ensemble_stat will allow us to match up members from GFS with
NAVGEM
> > and CMC if MET thinks they don't have a common forecast hour
amongst
> them.
> >
> > Thanks,
> > Matt
> >
> > Matthew C. Sittel, Meteorologist
> > University Corporation for Atmospheric Research
> > Matthew.Sittel.ctr at us.af.mil
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with GriB2 Headers and Forecast Hours
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Jan 24 12:11:30 2019

Interesting.  Did you run it through grid_stat or just
plot_data_plane?

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, January 24, 2019 12:17 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
<matthew.sittel.ctr at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with
GriB2 Headers and Forecast Hours

Here it is:

[johnhg at number5]% wgrib2 gfs.t00z.pgrb2.0p25.f072 | grep TCDC

On Thu, Jan 24, 2019 at 10:17 AM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
> Hey John,
>
> What does the header look like for TCDC when you degrib that single
> record?  Does it have that "66-72" field or just 72?
>
> Maybe I should send you the model and observation files so you can
try
> running grid_stat on it?
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 24, 2019 11:02 AM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with
> GriB2 Headers and Forecast Hours
>
> Matt,
>
> That's a good question.  I'm surprised that it's reporting the
> beginning of the averaging period (66) instead of the end (72).  I
> wanted to get some data for testing so I pulled a recent GFS file:
>
> *wget
>
>
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/gfs
> .t00z.pgrb2.0p25.f072
> <
>
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/gfs
> .t00z.pgrb2.0p25.f072
> >*
>
> And then I ran a plot_data_plane command to plot TCDC using the -v 4
> command line option to dump out more detailed info, including the
> timing
> information:
>
> *plot_data_plane gfs.t00z.pgrb2.0p25.f072 TCDC_entire_atmos.ps
> 'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*
>
> I chose GRIB_lvl_typ = 10 to get TCDC for the "entire atmosphere"
> based on this GRIB2 code table (
> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
5.
> shtml
> ).
>
> And here's the timing info that was dumped out:
> DEBUG 4: Data plane information:
> DEBUG 4:       plane min: 0
> DEBUG 4:       plane max: 100
> DEBUG 4:       scan mode: 0
> DEBUG 4:      valid time: 20190127_000000
> DEBUG 4:       lead time: 720000
> DEBUG 4:       init time: 20190124_000000
> DEBUG 4:     bitmap flag: 255
>
> One confusing bit is that when I reran with met-6.1, the GRIB level
> codes were reported as different values.
>
> But that wrinkle aside, I'm not see the discrepancy in timing info
> that you describe.
>
> Any thoughts?
>
> Thanks,
> John
>
>
>
>
> On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> >        Queue: met_help
> >      Subject: Issue with GriB2 Headers and Forecast Hours
> >        Owner: Nobody
> >   Requestors: matthew.sittel.ctr at us.af.mil
> >       Status: new
> >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > >
> >
> >
> > Hi,
> >
> > I'm verifying a GFS ensemble member's 72 hour total cloud amount
> > forecast against a 72-hour WWMCA ground truth.  When grid_stat
> > generates its cnt.txt output file, the forecast hour reported is
66
> instead of 72.
> >
> > In trying to figure out why it did this, I went into the header
for
> > the
> > GriB2 data and found the following:
> >
> > 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave
fcst:TCDC
> > Total Cloud Cover [%]:ENS=+1
> >     ndata=259920:undef=0:mean=52.3893:min=0:max=100
> >     grid_template=0:winds(N/S):
> >         lat-lon grid:(720 x 361) units 1e-06 input WE:NS output
> > WE:SN res
> > 48
> >         lat 90.000000 to -90.000000 by 0.500000
> >         lon 0.000000 to 359.500000 by 0.500000 #points=259920
> >
> > It looks like MET sees the range of forecast hours ("66-72" in
this
> > case) and uses the first one to calculate the forecast hour.
> >
> > If I use "-VT" as an option when I degrib this model file, the
valid
> > forecast time reported back is 72 hours, not 66:
> >
> > 30:5224497:vt=20181208000000
> >
> > Ignoring the issue of trying to figure out how to verify a 66 to
72
> > hour average cloud forecast with a single WWMCA file, is there any
> > way to get grid_stat to look at the valid time instead of
> > (presumably) the header so that I get 72 in the cnt.txt output
rather than 66?
> >
> > I haven't looked at ensemble_stat yet, but I do know that the
other
> > two models in GEPS are NAVGEM and CMC, and both of them have "72
> > hour fcst" in their header for the same field.  I'm not sure if
> > ensemble_stat will allow us to match up members from GFS with
NAVGEM
> > and CMC if MET thinks they don't have a common forecast hour
amongst
> them.
> >
> > Thanks,
> > Matt
> >
> > Matthew C. Sittel, Meteorologist
> > University Corporation for Atmospheric Research
> > Matthew.Sittel.ctr at us.af.mil
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: Issue with GriB2 Headers and Forecast Hours
From: John Halley Gotway
Time: Thu Jan 24 13:40:36 2019

Matt,

I had just run plot_data_plane, so I ran Grid-Stat as well... just
comparing the same field to itself:

/usr/local/met-6.1/bin/grid_stat \
gfs.t00z.pgrb2.0p25.f072 \
gfs.t00z.pgrb2.0p25.f072 \
GridStatConfig -outdir out -v 4 -log run_gs.log

DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V.stat
DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V_cnt.txt
DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V_pairs.nc

Perhaps there's something different about the way the files you're
using
are encoded?  Or perhaps there's a bug in the script that generates
the
call to Grid-Stat?

If you can isolate the call to Grid-Stat that's resulting in the
unexpected
behavior, and then send me sample data, I'd be happy to take a closer
look
and try to debug.

Thanks,
John

On Thu, Jan 24, 2019 at 12:12 PM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
> Interesting.  Did you run it through grid_stat or just
plot_data_plane?
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 24, 2019 12:17 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> GriB2 Headers and Forecast Hours
>
> Here it is:
>
> [johnhg at number5]% wgrib2 gfs.t00z.pgrb2.0p25.f072 | grep TCDC
>
>
> On Thu, Jan 24, 2019 at 10:17 AM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> >
> > Hey John,
> >
> > What does the header look like for TCDC when you degrib that
single
> > record?  Does it have that "66-72" field or just 72?
> >
> > Maybe I should send you the model and observation files so you can
try
> > running grid_stat on it?
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 24, 2019 11:02 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > matthew.sittel.ctr at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with
> > GriB2 Headers and Forecast Hours
> >
> > Matt,
> >
> > That's a good question.  I'm surprised that it's reporting the
> > beginning of the averaging period (66) instead of the end (72).  I
> > wanted to get some data for testing so I pulled a recent GFS file:
> >
> > *wget
> >
> >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/gfs
> > .t00z.pgrb2.0p25.f072
> > <
> >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/gfs
> > .t00z.pgrb2.0p25.f072
> > >*
> >
> > And then I ran a plot_data_plane command to plot TCDC using the -v
4
> > command line option to dump out more detailed info, including the
> > timing
> > information:
> >
> > *plot_data_plane gfs.t00z.pgrb2.0p25.f072 TCDC_entire_atmos.ps
> > 'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*
> >
> > I chose GRIB_lvl_typ = 10 to get TCDC for the "entire atmosphere"
> > based on this GRIB2 code table (
> >
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.
> > shtml
> > ).
> >
> > And here's the timing info that was dumped out:
> > DEBUG 4: Data plane information:
> > DEBUG 4:       plane min: 0
> > DEBUG 4:       plane max: 100
> > DEBUG 4:       scan mode: 0
> > DEBUG 4:      valid time: 20190127_000000
> > DEBUG 4:       lead time: 720000
> > DEBUG 4:       init time: 20190124_000000
> > DEBUG 4:     bitmap flag: 255
> >
> > One confusing bit is that when I reran with met-6.1, the GRIB
level
> > codes were reported as different values.
> >
> > But that wrinkle aside, I'm not see the discrepancy in timing info
> > that you describe.
> >
> > Any thoughts?
> >
> > Thanks,
> > John
> >
> >
> >
> >
> > On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > >        Queue: met_help
> > >      Subject: Issue with GriB2 Headers and Forecast Hours
> > >        Owner: Nobody
> > >   Requestors: matthew.sittel.ctr at us.af.mil
> > >       Status: new
> > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > > >
> > >
> > >
> > > Hi,
> > >
> > > I'm verifying a GFS ensemble member's 72 hour total cloud amount
> > > forecast against a 72-hour WWMCA ground truth.  When grid_stat
> > > generates its cnt.txt output file, the forecast hour reported is
66
> > instead of 72.
> > >
> > > In trying to figure out why it did this, I went into the header
for
> > > the
> > > GriB2 data and found the following:
> > >
> > > 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave
fcst:TCDC
> > > Total Cloud Cover [%]:ENS=+1
> > >     ndata=259920:undef=0:mean=52.3893:min=0:max=100
> > >     grid_template=0:winds(N/S):
> > >         lat-lon grid:(720 x 361) units 1e-06 input WE:NS output
> > > WE:SN res
> > > 48
> > >         lat 90.000000 to -90.000000 by 0.500000
> > >         lon 0.000000 to 359.500000 by 0.500000 #points=259920
> > >
> > > It looks like MET sees the range of forecast hours ("66-72" in
this
> > > case) and uses the first one to calculate the forecast hour.
> > >
> > > If I use "-VT" as an option when I degrib this model file, the
valid
> > > forecast time reported back is 72 hours, not 66:
> > >
> > > 30:5224497:vt=20181208000000
> > >
> > > Ignoring the issue of trying to figure out how to verify a 66 to
72
> > > hour average cloud forecast with a single WWMCA file, is there
any
> > > way to get grid_stat to look at the valid time instead of
> > > (presumably) the header so that I get 72 in the cnt.txt output
rather
> than 66?
> > >
> > > I haven't looked at ensemble_stat yet, but I do know that the
other
> > > two models in GEPS are NAVGEM and CMC, and both of them have "72
> > > hour fcst" in their header for the same field.  I'm not sure if
> > > ensemble_stat will allow us to match up members from GFS with
NAVGEM
> > > and CMC if MET thinks they don't have a common forecast hour
amongst
> > them.
> > >
> > > Thanks,
> > > Matt
> > >
> > > Matthew C. Sittel, Meteorologist
> > > University Corporation for Atmospheric Research
> > > Matthew.Sittel.ctr at us.af.mil
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with GriB2 Headers and Forecast Hours
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Jan 24 13:44:33 2019

That’s weird.  I can send you the offending model file and
observation, both are 1/2-degree grids.  What method is Bob Craig
using to send you files?

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, January 24, 2019 2:41 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
<matthew.sittel.ctr at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with
GriB2 Headers and Forecast Hours

Matt,

I had just run plot_data_plane, so I ran Grid-Stat as well... just
comparing the same field to itself:

/usr/local/met-6.1/bin/grid_stat \
gfs.t00z.pgrb2.0p25.f072 \
gfs.t00z.pgrb2.0p25.f072 \
GridStatConfig -outdir out -v 4 -log run_gs.log

DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V.stat
DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V_cnt.txt
DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V_pairs.nc

Perhaps there's something different about the way the files you're
using are encoded?  Or perhaps there's a bug in the script that
generates the call to Grid-Stat?

If you can isolate the call to Grid-Stat that's resulting in the
unexpected behavior, and then send me sample data, I'd be happy to
take a closer look and try to debug.

Thanks,
John

On Thu, Jan 24, 2019 at 12:12 PM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
> Interesting.  Did you run it through grid_stat or just
plot_data_plane?
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 24, 2019 12:17 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> GriB2 Headers and Forecast Hours
>
> Here it is:
>
> [johnhg at number5]% wgrib2 gfs.t00z.pgrb2.0p25.f072 | grep TCDC
>
>
> On Thu, Jan 24, 2019 at 10:17 AM SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> >
> > Hey John,
> >
> > What does the header look like for TCDC when you degrib that
single
> > record?  Does it have that "66-72" field or just 72?
> >
> > Maybe I should send you the model and observation files so you can
> > try running grid_stat on it?
> >
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 24, 2019 11:02 AM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > matthew.sittel.ctr at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with
> > GriB2 Headers and Forecast Hours
> >
> > Matt,
> >
> > That's a good question.  I'm surprised that it's reporting the
> > beginning of the averaging period (66) instead of the end (72).  I
> > wanted to get some data for testing so I pulled a recent GFS file:
> >
> > *wget
> >
> >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/g
> > fs
> > .t00z.pgrb2.0p25.f072
> > <
> >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/g
> > fs
> > .t00z.pgrb2.0p25.f072
> > >*
> >
> > And then I ran a plot_data_plane command to plot TCDC using the -v
4
> > command line option to dump out more detailed info, including the
> > timing
> > information:
> >
> > *plot_data_plane gfs.t00z.pgrb2.0p25.f072 TCDC_entire_atmos.ps
> > 'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*
> >
> > I chose GRIB_lvl_typ = 10 to get TCDC for the "entire atmosphere"
> > based on this GRIB2 code table (
> >
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.
> > shtml
> > ).
> >
> > And here's the timing info that was dumped out:
> > DEBUG 4: Data plane information:
> > DEBUG 4:       plane min: 0
> > DEBUG 4:       plane max: 100
> > DEBUG 4:       scan mode: 0
> > DEBUG 4:      valid time: 20190127_000000
> > DEBUG 4:       lead time: 720000
> > DEBUG 4:       init time: 20190124_000000
> > DEBUG 4:     bitmap flag: 255
> >
> > One confusing bit is that when I reran with met-6.1, the GRIB
level
> > codes were reported as different values.
> >
> > But that wrinkle aside, I'm not see the discrepancy in timing info
> > that you describe.
> >
> > Any thoughts?
> >
> > Thanks,
> > John
> >
> >
> >
> >
> > On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > >        Queue: met_help
> > >      Subject: Issue with GriB2 Headers and Forecast Hours
> > >        Owner: Nobody
> > >   Requestors: matthew.sittel.ctr at us.af.mil
> > >       Status: new
> > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > > >
> > >
> > >
> > > Hi,
> > >
> > > I'm verifying a GFS ensemble member's 72 hour total cloud amount
> > > forecast against a 72-hour WWMCA ground truth.  When grid_stat
> > > generates its cnt.txt output file, the forecast hour reported is
> > > 66
> > instead of 72.
> > >
> > > In trying to figure out why it did this, I went into the header
> > > for the
> > > GriB2 data and found the following:
> > >
> > > 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave
> > > fcst:TCDC Total Cloud Cover [%]:ENS=+1
> > >     ndata=259920:undef=0:mean=52.3893:min=0:max=100
> > >     grid_template=0:winds(N/S):
> > >         lat-lon grid:(720 x 361) units 1e-06 input WE:NS output
> > > WE:SN res
> > > 48
> > >         lat 90.000000 to -90.000000 by 0.500000
> > >         lon 0.000000 to 359.500000 by 0.500000 #points=259920
> > >
> > > It looks like MET sees the range of forecast hours ("66-72" in
> > > this
> > > case) and uses the first one to calculate the forecast hour.
> > >
> > > If I use "-VT" as an option when I degrib this model file, the
> > > valid forecast time reported back is 72 hours, not 66:
> > >
> > > 30:5224497:vt=20181208000000
> > >
> > > Ignoring the issue of trying to figure out how to verify a 66 to
> > > 72 hour average cloud forecast with a single WWMCA file, is
there
> > > any way to get grid_stat to look at the valid time instead of
> > > (presumably) the header so that I get 72 in the cnt.txt output
> > > rather
> than 66?
> > >
> > > I haven't looked at ensemble_stat yet, but I do know that the
> > > other two models in GEPS are NAVGEM and CMC, and both of them
have
> > > "72 hour fcst" in their header for the same field.  I'm not sure
> > > if ensemble_stat will allow us to match up members from GFS with
> > > NAVGEM and CMC if MET thinks they don't have a common forecast
> > > hour amongst
> > them.
> > >
> > > Thanks,
> > > Matt
> > >
> > > Matthew C. Sittel, Meteorologist
> > > University Corporation for Atmospheric Research
> > > Matthew.Sittel.ctr at us.af.mil
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: Issue with GriB2 Headers and Forecast Hours
From: John Halley Gotway
Time: Thu Jan 24 14:06:43 2019

Bob sends me data through ARL SAFE and I get an email telling me to go
pick
them up.

John

On Thu, Jan 24, 2019 at 1:45 PM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
> That’s weird.  I can send you the offending model file and
observation,
> both are 1/2-degree grids.  What method is Bob Craig using to send
you
> files?
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 24, 2019 2:41 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> GriB2 Headers and Forecast Hours
>
> Matt,
>
> I had just run plot_data_plane, so I ran Grid-Stat as well... just
> comparing the same field to itself:
>
> /usr/local/met-6.1/bin/grid_stat \
> gfs.t00z.pgrb2.0p25.f072 \
> gfs.t00z.pgrb2.0p25.f072 \
> GridStatConfig -outdir out -v 4 -log run_gs.log
>
> DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V.stat
> DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V_cnt.txt
> DEBUG 1: Output file:
out/grid_stat_720000L_20190127_000000V_pairs.nc
>
> Perhaps there's something different about the way the files you're
using
> are encoded?  Or perhaps there's a bug in the script that generates
the
> call to Grid-Stat?
>
> If you can isolate the call to Grid-Stat that's resulting in the
> unexpected behavior, and then send me sample data, I'd be happy to
take a
> closer look and try to debug.
>
> Thanks,
> John
>
> On Thu, Jan 24, 2019 at 12:12 PM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> >
> > Interesting.  Did you run it through grid_stat or just
plot_data_plane?
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 24, 2019 12:17 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > matthew.sittel.ctr at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> > GriB2 Headers and Forecast Hours
> >
> > Here it is:
> >
> > [johnhg at number5]% wgrib2 gfs.t00z.pgrb2.0p25.f072 | grep TCDC
> >
> >
> > On Thu, Jan 24, 2019 at 10:17 AM SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> > >
> > > Hey John,
> > >
> > > What does the header look like for TCDC when you degrib that
single
> > > record?  Does it have that "66-72" field or just 72?
> > >
> > > Maybe I should send you the model and observation files so you
can
> > > try running grid_stat on it?
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Thursday, January 24, 2019 11:02 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > matthew.sittel.ctr at us.af.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> > > GriB2 Headers and Forecast Hours
> > >
> > > Matt,
> > >
> > > That's a good question.  I'm surprised that it's reporting the
> > > beginning of the averaging period (66) instead of the end (72).
I
> > > wanted to get some data for testing so I pulled a recent GFS
file:
> > >
> > > *wget
> > >
> > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/g
> > > fs
> > > .t00z.pgrb2.0p25.f072
> > > <
> > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400/g
> > > fs
> > > .t00z.pgrb2.0p25.f072
> > > >*
> > >
> > > And then I ran a plot_data_plane command to plot TCDC using the
-v 4
> > > command line option to dump out more detailed info, including
the
> > > timing
> > > information:
> > >
> > > *plot_data_plane gfs.t00z.pgrb2.0p25.f072 TCDC_entire_atmos.ps
> > > 'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*
> > >
> > > I chose GRIB_lvl_typ = 10 to get TCDC for the "entire
atmosphere"
> > > based on this GRIB2 code table (
> > >
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.
> > > shtml
> > > ).
> > >
> > > And here's the timing info that was dumped out:
> > > DEBUG 4: Data plane information:
> > > DEBUG 4:       plane min: 0
> > > DEBUG 4:       plane max: 100
> > > DEBUG 4:       scan mode: 0
> > > DEBUG 4:      valid time: 20190127_000000
> > > DEBUG 4:       lead time: 720000
> > > DEBUG 4:       init time: 20190124_000000
> > > DEBUG 4:     bitmap flag: 255
> > >
> > > One confusing bit is that when I reran with met-6.1, the GRIB
level
> > > codes were reported as different values.
> > >
> > > But that wrinkle aside, I'm not see the discrepancy in timing
info
> > > that you describe.
> > >
> > > Any thoughts?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > > On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF AFWA
16
> > > WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> > > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > >        Queue: met_help
> > > >      Subject: Issue with GriB2 Headers and Forecast Hours
> > > >        Owner: Nobody
> > > >   Requestors: matthew.sittel.ctr at us.af.mil
> > > >       Status: new
> > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > > I'm verifying a GFS ensemble member's 72 hour total cloud
amount
> > > > forecast against a 72-hour WWMCA ground truth.  When grid_stat
> > > > generates its cnt.txt output file, the forecast hour reported
is
> > > > 66
> > > instead of 72.
> > > >
> > > > In trying to figure out why it did this, I went into the
header
> > > > for the
> > > > GriB2 data and found the following:
> > > >
> > > > 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave
> > > > fcst:TCDC Total Cloud Cover [%]:ENS=+1
> > > >     ndata=259920:undef=0:mean=52.3893:min=0:max=100
> > > >     grid_template=0:winds(N/S):
> > > >         lat-lon grid:(720 x 361) units 1e-06 input WE:NS
output
> > > > WE:SN res
> > > > 48
> > > >         lat 90.000000 to -90.000000 by 0.500000
> > > >         lon 0.000000 to 359.500000 by 0.500000 #points=259920
> > > >
> > > > It looks like MET sees the range of forecast hours ("66-72" in
> > > > this
> > > > case) and uses the first one to calculate the forecast hour.
> > > >
> > > > If I use "-VT" as an option when I degrib this model file, the
> > > > valid forecast time reported back is 72 hours, not 66:
> > > >
> > > > 30:5224497:vt=20181208000000
> > > >
> > > > Ignoring the issue of trying to figure out how to verify a 66
to
> > > > 72 hour average cloud forecast with a single WWMCA file, is
there
> > > > any way to get grid_stat to look at the valid time instead of
> > > > (presumably) the header so that I get 72 in the cnt.txt output
> > > > rather
> > than 66?
> > > >
> > > > I haven't looked at ensemble_stat yet, but I do know that the
> > > > other two models in GEPS are NAVGEM and CMC, and both of them
have
> > > > "72 hour fcst" in their header for the same field.  I'm not
sure
> > > > if ensemble_stat will allow us to match up members from GFS
with
> > > > NAVGEM and CMC if MET thinks they don't have a common forecast
> > > > hour amongst
> > > them.
> > > >
> > > > Thanks,
> > > > Matt
> > > >
> > > > Matthew C. Sittel, Meteorologist
> > > > University Corporation for Atmospheric Research
> > > > Matthew.Sittel.ctr at us.af.mil
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with GriB2 Headers and Forecast Hours
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Thu Jan 24 14:28:26 2019

Okay, I have sent you both files.  I sent the first email too early;
it had only one file in it.  The second one has the two files in the
same email.  Please disregard the first email with only one
attachment.

Thanks,
Matt

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, January 24, 2019 3:07 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
<matthew.sittel.ctr at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with
GriB2 Headers and Forecast Hours

Bob sends me data through ARL SAFE and I get an email telling me to go
pick them up.

John

On Thu, Jan 24, 2019 at 1:45 PM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
> That’s weird.  I can send you the offending model file and
> observation, both are 1/2-degree grids.  What method is Bob Craig
> using to send you files?
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 24, 2019 2:41 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> GriB2 Headers and Forecast Hours
>
> Matt,
>
> I had just run plot_data_plane, so I ran Grid-Stat as well... just
> comparing the same field to itself:
>
> /usr/local/met-6.1/bin/grid_stat \
> gfs.t00z.pgrb2.0p25.f072 \
> gfs.t00z.pgrb2.0p25.f072 \
> GridStatConfig -outdir out -v 4 -log run_gs.log
>
> DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V.stat
> DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V_cnt.txt
> DEBUG 1: Output file:
out/grid_stat_720000L_20190127_000000V_pairs.nc
>
> Perhaps there's something different about the way the files you're
> using are encoded?  Or perhaps there's a bug in the script that
> generates the call to Grid-Stat?
>
> If you can isolate the call to Grid-Stat that's resulting in the
> unexpected behavior, and then send me sample data, I'd be happy to
> take a closer look and try to debug.
>
> Thanks,
> John
>
> On Thu, Jan 24, 2019 at 12:12 PM SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> >
> > Interesting.  Did you run it through grid_stat or just
plot_data_plane?
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 24, 2019 12:17 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > matthew.sittel.ctr at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
> > with
> > GriB2 Headers and Forecast Hours
> >
> > Here it is:
> >
> > [johnhg at number5]% wgrib2 gfs.t00z.pgrb2.0p25.f072 | grep TCDC
> >
> >
> > On Thu, Jan 24, 2019 at 10:17 AM SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> > >
> > > Hey John,
> > >
> > > What does the header look like for TCDC when you degrib that
> > > single record?  Does it have that "66-72" field or just 72?
> > >
> > > Maybe I should send you the model and observation files so you
can
> > > try running grid_stat on it?
> > >
> > > Matt
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Thursday, January 24, 2019 11:02 AM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > matthew.sittel.ctr at us.af.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> > > GriB2 Headers and Forecast Hours
> > >
> > > Matt,
> > >
> > > That's a good question.  I'm surprised that it's reporting the
> > > beginning of the averaging period (66) instead of the end (72).
I
> > > wanted to get some data for testing so I pulled a recent GFS
file:
> > >
> > > *wget
> > >
> > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400
> > > /g
> > > fs
> > > .t00z.pgrb2.0p25.f072
> > > <
> > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400
> > > /g
> > > fs
> > > .t00z.pgrb2.0p25.f072
> > > >*
> > >
> > > And then I ran a plot_data_plane command to plot TCDC using the
-v
> > > 4 command line option to dump out more detailed info, including
> > > the timing
> > > information:
> > >
> > > *plot_data_plane gfs.t00z.pgrb2.0p25.f072 TCDC_entire_atmos.ps
> > > 'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*
> > >
> > > I chose GRIB_lvl_typ = 10 to get TCDC for the "entire
atmosphere"
> > > based on this GRIB2 code table (
> > >
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.
> > > shtml
> > > ).
> > >
> > > And here's the timing info that was dumped out:
> > > DEBUG 4: Data plane information:
> > > DEBUG 4:       plane min: 0
> > > DEBUG 4:       plane max: 100
> > > DEBUG 4:       scan mode: 0
> > > DEBUG 4:      valid time: 20190127_000000
> > > DEBUG 4:       lead time: 720000
> > > DEBUG 4:       init time: 20190124_000000
> > > DEBUG 4:     bitmap flag: 255
> > >
> > > One confusing bit is that when I reran with met-6.1, the GRIB
> > > level codes were reported as different values.
> > >
> > > But that wrinkle aside, I'm not see the discrepancy in timing
info
> > > that you describe.
> > >
> > > Any thoughts?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > > On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> > > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > >        Queue: met_help
> > > >      Subject: Issue with GriB2 Headers and Forecast Hours
> > > >        Owner: Nobody
> > > >   Requestors: matthew.sittel.ctr at us.af.mil
> > > >       Status: new
> > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > > I'm verifying a GFS ensemble member's 72 hour total cloud
amount
> > > > forecast against a 72-hour WWMCA ground truth.  When grid_stat
> > > > generates its cnt.txt output file, the forecast hour reported
is
> > > > 66
> > > instead of 72.
> > > >
> > > > In trying to figure out why it did this, I went into the
header
> > > > for the
> > > > GriB2 data and found the following:
> > > >
> > > > 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave
> > > > fcst:TCDC Total Cloud Cover [%]:ENS=+1
> > > >     ndata=259920:undef=0:mean=52.3893:min=0:max=100
> > > >     grid_template=0:winds(N/S):
> > > >         lat-lon grid:(720 x 361) units 1e-06 input WE:NS
output
> > > > WE:SN res
> > > > 48
> > > >         lat 90.000000 to -90.000000 by 0.500000
> > > >         lon 0.000000 to 359.500000 by 0.500000 #points=259920
> > > >
> > > > It looks like MET sees the range of forecast hours ("66-72" in
> > > > this
> > > > case) and uses the first one to calculate the forecast hour.
> > > >
> > > > If I use "-VT" as an option when I degrib this model file, the
> > > > valid forecast time reported back is 72 hours, not 66:
> > > >
> > > > 30:5224497:vt=20181208000000
> > > >
> > > > Ignoring the issue of trying to figure out how to verify a 66
to
> > > > 72 hour average cloud forecast with a single WWMCA file, is
> > > > there any way to get grid_stat to look at the valid time
instead
> > > > of
> > > > (presumably) the header so that I get 72 in the cnt.txt output
> > > > rather
> > than 66?
> > > >
> > > > I haven't looked at ensemble_stat yet, but I do know that the
> > > > other two models in GEPS are NAVGEM and CMC, and both of them
> > > > have
> > > > "72 hour fcst" in their header for the same field.  I'm not
sure
> > > > if ensemble_stat will allow us to match up members from GFS
with
> > > > NAVGEM and CMC if MET thinks they don't have a common forecast
> > > > hour amongst
> > > them.
> > > >
> > > > Thanks,
> > > > Matt
> > > >
> > > > Matthew C. Sittel, Meteorologist University Corporation for
> > > > Atmospheric Research Matthew.Sittel.ctr at us.af.mil
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: Issue with GriB2 Headers and Forecast Hours
From: John Halley Gotway
Time: Thu Jan 24 15:06:35 2019

Matt,

I don't know how long it takes to come through, but I haven't received
an
ARL SAFE email yet.

You could try sending it to johnhg at ucar.edu.  Or if that doesn't work,
Bob
has sent it to met_help at ucar.edu.  That automatically creates a new
support
ticket.  But then I just go delete that ticket... which isn't a big
deal.

John

On Thu, Jan 24, 2019 at 2:28 PM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
> Okay, I have sent you both files.  I sent the first email too early;
it
> had only one file in it.  The second one has the two files in the
same
> email.  Please disregard the first email with only one attachment.
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 24, 2019 3:07 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> GriB2 Headers and Forecast Hours
>
> Bob sends me data through ARL SAFE and I get an email telling me to
go
> pick them up.
>
> John
>
> On Thu, Jan 24, 2019 at 1:45 PM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> >
> > That’s weird.  I can send you the offending model file and
> > observation, both are 1/2-degree grids.  What method is Bob Craig
> > using to send you files?
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 24, 2019 2:41 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > matthew.sittel.ctr at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> > GriB2 Headers and Forecast Hours
> >
> > Matt,
> >
> > I had just run plot_data_plane, so I ran Grid-Stat as well... just
> > comparing the same field to itself:
> >
> > /usr/local/met-6.1/bin/grid_stat \
> > gfs.t00z.pgrb2.0p25.f072 \
> > gfs.t00z.pgrb2.0p25.f072 \
> > GridStatConfig -outdir out -v 4 -log run_gs.log
> >
> > DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V.stat
> > DEBUG 1: Output file:
out/grid_stat_720000L_20190127_000000V_cnt.txt
> > DEBUG 1: Output file:
out/grid_stat_720000L_20190127_000000V_pairs.nc
> >
> > Perhaps there's something different about the way the files you're
> > using are encoded?  Or perhaps there's a bug in the script that
> > generates the call to Grid-Stat?
> >
> > If you can isolate the call to Grid-Stat that's resulting in the
> > unexpected behavior, and then send me sample data, I'd be happy to
> > take a closer look and try to debug.
> >
> > Thanks,
> > John
> >
> > On Thu, Jan 24, 2019 at 12:12 PM SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> > >
> > > Interesting.  Did you run it through grid_stat or just
plot_data_plane?
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Thursday, January 24, 2019 12:17 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > matthew.sittel.ctr at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
> > > with
> > > GriB2 Headers and Forecast Hours
> > >
> > > Here it is:
> > >
> > > [johnhg at number5]% wgrib2 gfs.t00z.pgrb2.0p25.f072 | grep TCDC
> > >
> > >
> > > On Thu, Jan 24, 2019 at 10:17 AM SITTEL, MATTHEW C CTR USAF AFWA
16
> > > WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
>
> > > >
> > > > Hey John,
> > > >
> > > > What does the header look like for TCDC when you degrib that
> > > > single record?  Does it have that "66-72" field or just 72?
> > > >
> > > > Maybe I should send you the model and observation files so you
can
> > > > try running grid_stat on it?
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: Thursday, January 24, 2019 11:02 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > > matthew.sittel.ctr at us.af.mil>
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> > > > GriB2 Headers and Forecast Hours
> > > >
> > > > Matt,
> > > >
> > > > That's a good question.  I'm surprised that it's reporting the
> > > > beginning of the averaging period (66) instead of the end
(72).  I
> > > > wanted to get some data for testing so I pulled a recent GFS
file:
> > > >
> > > > *wget
> > > >
> > > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400
> > > > /g
> > > > fs
> > > > .t00z.pgrb2.0p25.f072
> > > > <
> > > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2019012400
> > > > /g
> > > > fs
> > > > .t00z.pgrb2.0p25.f072
> > > > >*
> > > >
> > > > And then I ran a plot_data_plane command to plot TCDC using
the -v
> > > > 4 command line option to dump out more detailed info,
including
> > > > the timing
> > > > information:
> > > >
> > > > *plot_data_plane gfs.t00z.pgrb2.0p25.f072 TCDC_entire_atmos.ps
> > > > 'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*
> > > >
> > > > I chose GRIB_lvl_typ = 10 to get TCDC for the "entire
atmosphere"
> > > > based on this GRIB2 code table (
> > > >
> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
5.
> > > > shtml
> > > > ).
> > > >
> > > > And here's the timing info that was dumped out:
> > > > DEBUG 4: Data plane information:
> > > > DEBUG 4:       plane min: 0
> > > > DEBUG 4:       plane max: 100
> > > > DEBUG 4:       scan mode: 0
> > > > DEBUG 4:      valid time: 20190127_000000
> > > > DEBUG 4:       lead time: 720000
> > > > DEBUG 4:       init time: 20190124_000000
> > > > DEBUG 4:     bitmap flag: 255
> > > >
> > > > One confusing bit is that when I reran with met-6.1, the GRIB
> > > > level codes were reported as different values.
> > > >
> > > > But that wrinkle aside, I'm not see the discrepancy in timing
info
> > > > that you describe.
> > > >
> > > > Any thoughts?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> > > > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > > >        Queue: met_help
> > > > >      Subject: Issue with GriB2 Headers and Forecast Hours
> > > > >        Owner: Nobody
> > > > >   Requestors: matthew.sittel.ctr at us.af.mil
> > > > >       Status: new
> > > > >  Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > > > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm verifying a GFS ensemble member's 72 hour total cloud
amount
> > > > > forecast against a 72-hour WWMCA ground truth.  When
grid_stat
> > > > > generates its cnt.txt output file, the forecast hour
reported is
> > > > > 66
> > > > instead of 72.
> > > > >
> > > > > In trying to figure out why it did this, I went into the
header
> > > > > for the
> > > > > GriB2 data and found the following:
> > > > >
> > > > > 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave
> > > > > fcst:TCDC Total Cloud Cover [%]:ENS=+1
> > > > >     ndata=259920:undef=0:mean=52.3893:min=0:max=100
> > > > >     grid_template=0:winds(N/S):
> > > > >         lat-lon grid:(720 x 361) units 1e-06 input WE:NS
output
> > > > > WE:SN res
> > > > > 48
> > > > >         lat 90.000000 to -90.000000 by 0.500000
> > > > >         lon 0.000000 to 359.500000 by 0.500000
#points=259920
> > > > >
> > > > > It looks like MET sees the range of forecast hours ("66-72"
in
> > > > > this
> > > > > case) and uses the first one to calculate the forecast hour.
> > > > >
> > > > > If I use "-VT" as an option when I degrib this model file,
the
> > > > > valid forecast time reported back is 72 hours, not 66:
> > > > >
> > > > > 30:5224497:vt=20181208000000
> > > > >
> > > > > Ignoring the issue of trying to figure out how to verify a
66 to
> > > > > 72 hour average cloud forecast with a single WWMCA file, is
> > > > > there any way to get grid_stat to look at the valid time
instead
> > > > > of
> > > > > (presumably) the header so that I get 72 in the cnt.txt
output
> > > > > rather
> > > than 66?
> > > > >
> > > > > I haven't looked at ensemble_stat yet, but I do know that
the
> > > > > other two models in GEPS are NAVGEM and CMC, and both of
them
> > > > > have
> > > > > "72 hour fcst" in their header for the same field.  I'm not
sure
> > > > > if ensemble_stat will allow us to match up members from GFS
with
> > > > > NAVGEM and CMC if MET thinks they don't have a common
forecast
> > > > > hour amongst
> > > > them.
> > > > >
> > > > > Thanks,
> > > > > Matt
> > > > >
> > > > > Matthew C. Sittel, Meteorologist University Corporation for
> > > > > Atmospheric Research Matthew.Sittel.ctr at us.af.mil
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with GriB2 Headers and Forecast Hours
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Mon Jan 28 06:04:05 2019

Hmmm, I sent it to met_help.  I wonder if my being a contractor was an
issue?  I never got any messages saying it wouldn't work.

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, January 24, 2019 4:07 PM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
<matthew.sittel.ctr at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with
GriB2 Headers and Forecast Hours

Matt,

I don't know how long it takes to come through, but I haven't received
an ARL SAFE email yet.

You could try sending it to johnhg at ucar.edu.  Or if that doesn't work,
Bob has sent it to met_help at ucar.edu.  That automatically creates a
new support ticket.  But then I just go delete that ticket... which
isn't a big deal.

John

On Thu, Jan 24, 2019 at 2:28 PM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
> Okay, I have sent you both files.  I sent the first email too early;
> it had only one file in it.  The second one has the two files in the
> same email.  Please disregard the first email with only one
attachment.
>
> Thanks,
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 24, 2019 3:07 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> GriB2 Headers and Forecast Hours
>
> Bob sends me data through ARL SAFE and I get an email telling me to
go
> pick them up.
>
> John
>
> On Thu, Jan 24, 2019 at 1:45 PM SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> >
> > That’s weird.  I can send you the offending model file and
> > observation, both are 1/2-degree grids.  What method is Bob Craig
> > using to send you files?
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 24, 2019 2:41 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > matthew.sittel.ctr at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
> > with
> > GriB2 Headers and Forecast Hours
> >
> > Matt,
> >
> > I had just run plot_data_plane, so I ran Grid-Stat as well... just
> > comparing the same field to itself:
> >
> > /usr/local/met-6.1/bin/grid_stat \
> > gfs.t00z.pgrb2.0p25.f072 \
> > gfs.t00z.pgrb2.0p25.f072 \
> > GridStatConfig -outdir out -v 4 -log run_gs.log
> >
> > DEBUG 1: Output file: out/grid_stat_720000L_20190127_000000V.stat
> > DEBUG 1: Output file:
out/grid_stat_720000L_20190127_000000V_cnt.txt
> > DEBUG 1: Output file:
> > out/grid_stat_720000L_20190127_000000V_pairs.nc
> >
> > Perhaps there's something different about the way the files you're
> > using are encoded?  Or perhaps there's a bug in the script that
> > generates the call to Grid-Stat?
> >
> > If you can isolate the call to Grid-Stat that's resulting in the
> > unexpected behavior, and then send me sample data, I'd be happy to
> > take a closer look and try to debug.
> >
> > Thanks,
> > John
> >
> > On Thu, Jan 24, 2019 at 12:12 PM SITTEL, MATTHEW C CTR USAF AFWA
16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> > >
> > > Interesting.  Did you run it through grid_stat or just
plot_data_plane?
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Thursday, January 24, 2019 12:17 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > matthew.sittel.ctr at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
> > > with
> > > GriB2 Headers and Forecast Hours
> > >
> > > Here it is:
> > >
> > > [johnhg at number5]% wgrib2 gfs.t00z.pgrb2.0p25.f072 | grep TCDC
> > >
> > >
> > > On Thu, Jan 24, 2019 at 10:17 AM SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
>
> > > >
> > > > Hey John,
> > > >
> > > > What does the header look like for TCDC when you degrib that
> > > > single record?  Does it have that "66-72" field or just 72?
> > > >
> > > > Maybe I should send you the model and observation files so you
> > > > can try running grid_stat on it?
> > > >
> > > > Matt
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: Thursday, January 24, 2019 11:02 AM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > > matthew.sittel.ctr at us.af.mil>
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
> > > > with
> > > > GriB2 Headers and Forecast Hours
> > > >
> > > > Matt,
> > > >
> > > > That's a good question.  I'm surprised that it's reporting the
> > > > beginning of the averaging period (66) instead of the end
(72).
> > > > I wanted to get some data for testing so I pulled a recent GFS
file:
> > > >
> > > > *wget
> > > >
> > > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20190124
> > > > 00
> > > > /g
> > > > fs
> > > > .t00z.pgrb2.0p25.f072
> > > > <
> > > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20190124
> > > > 00
> > > > /g
> > > > fs
> > > > .t00z.pgrb2.0p25.f072
> > > > >*
> > > >
> > > > And then I ran a plot_data_plane command to plot TCDC using
the
> > > > -v
> > > > 4 command line option to dump out more detailed info,
including
> > > > the timing
> > > > information:
> > > >
> > > > *plot_data_plane gfs.t00z.pgrb2.0p25.f072 TCDC_entire_atmos.ps
> > > > 'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*
> > > >
> > > > I chose GRIB_lvl_typ = 10 to get TCDC for the "entire
atmosphere"
> > > > based on this GRIB2 code table (
> > > >
> https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-
5.
> > > > shtml
> > > > ).
> > > >
> > > > And here's the timing info that was dumped out:
> > > > DEBUG 4: Data plane information:
> > > > DEBUG 4:       plane min: 0
> > > > DEBUG 4:       plane max: 100
> > > > DEBUG 4:       scan mode: 0
> > > > DEBUG 4:      valid time: 20190127_000000
> > > > DEBUG 4:       lead time: 720000
> > > > DEBUG 4:       init time: 20190124_000000
> > > > DEBUG 4:     bitmap flag: 255
> > > >
> > > > One confusing bit is that when I reran with met-6.1, the GRIB
> > > > level codes were reported as different values.
> > > >
> > > > But that wrinkle aside, I'm not see the discrepancy in timing
> > > > info that you describe.
> > > >
> > > > Any thoughts?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> > > > > Transaction: Ticket created by matthew.sittel.ctr at us.af.mil
> > > > >        Queue: met_help
> > > > >      Subject: Issue with GriB2 Headers and Forecast Hours
> > > > >        Owner: Nobody
> > > > >   Requestors: matthew.sittel.ctr at us.af.mil
> > > > >       Status: new
> > > > >  Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > > > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm verifying a GFS ensemble member's 72 hour total cloud
> > > > > amount forecast against a 72-hour WWMCA ground truth.  When
> > > > > grid_stat generates its cnt.txt output file, the forecast
hour
> > > > > reported is
> > > > > 66
> > > > instead of 72.
> > > > >
> > > > > In trying to figure out why it did this, I went into the
> > > > > header for the
> > > > > GriB2 data and found the following:
> > > > >
> > > > > 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave
> > > > > fcst:TCDC Total Cloud Cover [%]:ENS=+1
> > > > >     ndata=259920:undef=0:mean=52.3893:min=0:max=100
> > > > >     grid_template=0:winds(N/S):
> > > > >         lat-lon grid:(720 x 361) units 1e-06 input WE:NS
> > > > > output WE:SN res
> > > > > 48
> > > > >         lat 90.000000 to -90.000000 by 0.500000
> > > > >         lon 0.000000 to 359.500000 by 0.500000
#points=259920
> > > > >
> > > > > It looks like MET sees the range of forecast hours ("66-72"
in
> > > > > this
> > > > > case) and uses the first one to calculate the forecast hour.
> > > > >
> > > > > If I use "-VT" as an option when I degrib this model file,
the
> > > > > valid forecast time reported back is 72 hours, not 66:
> > > > >
> > > > > 30:5224497:vt=20181208000000
> > > > >
> > > > > Ignoring the issue of trying to figure out how to verify a
66
> > > > > to
> > > > > 72 hour average cloud forecast with a single WWMCA file, is
> > > > > there any way to get grid_stat to look at the valid time
> > > > > instead of
> > > > > (presumably) the header so that I get 72 in the cnt.txt
output
> > > > > rather
> > > than 66?
> > > > >
> > > > > I haven't looked at ensemble_stat yet, but I do know that
the
> > > > > other two models in GEPS are NAVGEM and CMC, and both of
them
> > > > > have
> > > > > "72 hour fcst" in their header for the same field.  I'm not
> > > > > sure if ensemble_stat will allow us to match up members from
> > > > > GFS with NAVGEM and CMC if MET thinks they don't have a
common
> > > > > forecast hour amongst
> > > > them.
> > > > >
> > > > > Thanks,
> > > > > Matt
> > > > >
> > > > > Matthew C. Sittel, Meteorologist University Corporation for
> > > > > Atmospheric Research Matthew.Sittel.ctr at us.af.mil
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: Issue with GriB2 Headers and Forecast Hours
From: John Halley Gotway
Time: Mon Jan 28 10:08:08 2019

Matt,

Sorry, still haven't seen anything.  Is the data you're using publicly
available anywhere?  If so, you could send me a URL to the ftp site.
Or,
if possible, you could post it to our anonymous ftp site:
   https://dtcenter.org/met/users/support/met_help.php#ftp

But I believe Bob has had trouble doing that in the past.

Or you could run a wgrib2 command to extract out a single GRIB record
to
make it small and try sending it as an attachment.  You may need to
add a
".txt" suffix to get passed the email filters.

Here's an example of running wgrib2 to select out record number 320
and
write it to a new output file:
   wgrib2 -d 320 gfs.t00z.pgrb2.0p25.f072 -grib_out rec320.grib2

John

On Mon, Jan 28, 2019 at 6:04 AM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
> Hmmm, I sent it to met_help.  I wonder if my being a contractor was
an
> issue?  I never got any messages saying it wouldn't work.
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 24, 2019 4:07 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> GriB2 Headers and Forecast Hours
>
> Matt,
>
> I don't know how long it takes to come through, but I haven't
received an
> ARL SAFE email yet.
>
> You could try sending it to johnhg at ucar.edu.  Or if that doesn't
work,
> Bob has sent it to met_help at ucar.edu.  That automatically creates a
new
> support ticket.  But then I just go delete that ticket... which
isn't a big
> deal.
>
> John
>
> On Thu, Jan 24, 2019 at 2:28 PM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> >
> > Okay, I have sent you both files.  I sent the first email too
early;
> > it had only one file in it.  The second one has the two files in
the
> > same email.  Please disregard the first email with only one
attachment.
> >
> > Thanks,
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 24, 2019 3:07 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > matthew.sittel.ctr at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> > GriB2 Headers and Forecast Hours
> >
> > Bob sends me data through ARL SAFE and I get an email telling me
to go
> > pick them up.
> >
> > John
> >
> > On Thu, Jan 24, 2019 at 1:45 PM SITTEL, MATTHEW C CTR USAF AFWA 16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> > >
> > > That’s weird.  I can send you the offending model file and
> > > observation, both are 1/2-degree grids.  What method is Bob
Craig
> > > using to send you files?
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Thursday, January 24, 2019 2:41 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > matthew.sittel.ctr at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
> > > with
> > > GriB2 Headers and Forecast Hours
> > >
> > > Matt,
> > >
> > > I had just run plot_data_plane, so I ran Grid-Stat as well...
just
> > > comparing the same field to itself:
> > >
> > > /usr/local/met-6.1/bin/grid_stat \
> > > gfs.t00z.pgrb2.0p25.f072 \
> > > gfs.t00z.pgrb2.0p25.f072 \
> > > GridStatConfig -outdir out -v 4 -log run_gs.log
> > >
> > > DEBUG 1: Output file:
out/grid_stat_720000L_20190127_000000V.stat
> > > DEBUG 1: Output file:
out/grid_stat_720000L_20190127_000000V_cnt.txt
> > > DEBUG 1: Output file:
> > > out/grid_stat_720000L_20190127_000000V_pairs.nc
> > >
> > > Perhaps there's something different about the way the files
you're
> > > using are encoded?  Or perhaps there's a bug in the script that
> > > generates the call to Grid-Stat?
> > >
> > > If you can isolate the call to Grid-Stat that's resulting in the
> > > unexpected behavior, and then send me sample data, I'd be happy
to
> > > take a closer look and try to debug.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jan 24, 2019 at 12:12 PM SITTEL, MATTHEW C CTR USAF AFWA
16
> > > WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
>
> > > >
> > > > Interesting.  Did you run it through grid_stat or just
> plot_data_plane?
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: Thursday, January 24, 2019 12:17 PM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > > matthew.sittel.ctr at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604]
Issue
> > > > with
> > > > GriB2 Headers and Forecast Hours
> > > >
> > > > Here it is:
> > > >
> > > > [johnhg at number5]% wgrib2 gfs.t00z.pgrb2.0p25.f072 | grep TCDC
> > > >
> > > >
> > > > On Thu, Jan 24, 2019 at 10:17 AM SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> > > > >
> > > > > Hey John,
> > > > >
> > > > > What does the header look like for TCDC when you degrib that
> > > > > single record?  Does it have that "66-72" field or just 72?
> > > > >
> > > > > Maybe I should send you the model and observation files so
you
> > > > > can try running grid_stat on it?
> > > > >
> > > > > Matt
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: Thursday, January 24, 2019 11:02 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > > > matthew.sittel.ctr at us.af.mil>
> > > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
> > > > > with
> > > > > GriB2 Headers and Forecast Hours
> > > > >
> > > > > Matt,
> > > > >
> > > > > That's a good question.  I'm surprised that it's reporting
the
> > > > > beginning of the averaging period (66) instead of the end
(72).
> > > > > I wanted to get some data for testing so I pulled a recent
GFS
> file:
> > > > >
> > > > > *wget
> > > > >
> > > > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20190124
> > > > > 00
> > > > > /g
> > > > > fs
> > > > > .t00z.pgrb2.0p25.f072
> > > > > <
> > > > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20190124
> > > > > 00
> > > > > /g
> > > > > fs
> > > > > .t00z.pgrb2.0p25.f072
> > > > > >*
> > > > >
> > > > > And then I ran a plot_data_plane command to plot TCDC using
the
> > > > > -v
> > > > > 4 command line option to dump out more detailed info,
including
> > > > > the timing
> > > > > information:
> > > > >
> > > > > *plot_data_plane gfs.t00z.pgrb2.0p25.f072
TCDC_entire_atmos.ps
> > > > > 'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*
> > > > >
> > > > > I chose GRIB_lvl_typ = 10 to get TCDC for the "entire
atmosphere"
> > > > > based on this GRIB2 code table (
> > > > >
> >
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.
> > > > > shtml
> > > > > ).
> > > > >
> > > > > And here's the timing info that was dumped out:
> > > > > DEBUG 4: Data plane information:
> > > > > DEBUG 4:       plane min: 0
> > > > > DEBUG 4:       plane max: 100
> > > > > DEBUG 4:       scan mode: 0
> > > > > DEBUG 4:      valid time: 20190127_000000
> > > > > DEBUG 4:       lead time: 720000
> > > > > DEBUG 4:       init time: 20190124_000000
> > > > > DEBUG 4:     bitmap flag: 255
> > > > >
> > > > > One confusing bit is that when I reran with met-6.1, the
GRIB
> > > > > level codes were reported as different values.
> > > > >
> > > > > But that wrinkle aside, I'm not see the discrepancy in
timing
> > > > > info that you describe.
> > > > >
> > > > > Any thoughts?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF
AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> > > > > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > > > > >        Queue: met_help
> > > > > >      Subject: Issue with GriB2 Headers and Forecast Hours
> > > > > >        Owner: Nobody
> > > > > >   Requestors: matthew.sittel.ctr at us.af.mil
> > > > > >       Status: new
> > > > > >  Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm verifying a GFS ensemble member's 72 hour total cloud
> > > > > > amount forecast against a 72-hour WWMCA ground truth.
When
> > > > > > grid_stat generates its cnt.txt output file, the forecast
hour
> > > > > > reported is
> > > > > > 66
> > > > > instead of 72.
> > > > > >
> > > > > > In trying to figure out why it did this, I went into the
> > > > > > header for the
> > > > > > GriB2 data and found the following:
> > > > > >
> > > > > > 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave
> > > > > > fcst:TCDC Total Cloud Cover [%]:ENS=+1
> > > > > >     ndata=259920:undef=0:mean=52.3893:min=0:max=100
> > > > > >     grid_template=0:winds(N/S):
> > > > > >         lat-lon grid:(720 x 361) units 1e-06 input WE:NS
> > > > > > output WE:SN res
> > > > > > 48
> > > > > >         lat 90.000000 to -90.000000 by 0.500000
> > > > > >         lon 0.000000 to 359.500000 by 0.500000
#points=259920
> > > > > >
> > > > > > It looks like MET sees the range of forecast hours ("66-
72" in
> > > > > > this
> > > > > > case) and uses the first one to calculate the forecast
hour.
> > > > > >
> > > > > > If I use "-VT" as an option when I degrib this model file,
the
> > > > > > valid forecast time reported back is 72 hours, not 66:
> > > > > >
> > > > > > 30:5224497:vt=20181208000000
> > > > > >
> > > > > > Ignoring the issue of trying to figure out how to verify a
66
> > > > > > to
> > > > > > 72 hour average cloud forecast with a single WWMCA file,
is
> > > > > > there any way to get grid_stat to look at the valid time
> > > > > > instead of
> > > > > > (presumably) the header so that I get 72 in the cnt.txt
output
> > > > > > rather
> > > > than 66?
> > > > > >
> > > > > > I haven't looked at ensemble_stat yet, but I do know that
the
> > > > > > other two models in GEPS are NAVGEM and CMC, and both of
them
> > > > > > have
> > > > > > "72 hour fcst" in their header for the same field.  I'm
not
> > > > > > sure if ensemble_stat will allow us to match up members
from
> > > > > > GFS with NAVGEM and CMC if MET thinks they don't have a
common
> > > > > > forecast hour amongst
> > > > > them.
> > > > > >
> > > > > > Thanks,
> > > > > > Matt
> > > > > >
> > > > > > Matthew C. Sittel, Meteorologist University Corporation
for
> > > > > > Atmospheric Research Matthew.Sittel.ctr at us.af.mil
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with GriB2 Headers and Forecast Hours
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Mon Jan 28 10:25:20 2019

I sent it a second time this morning.  Hopefully it will show up.

FTP connection is refused-I wish that worked!

-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, January 28, 2019 11:08 AM
To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
<matthew.sittel.ctr at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue with
GriB2 Headers and Forecast Hours

Matt,

Sorry, still haven't seen anything.  Is the data you're using publicly
available anywhere?  If so, you could send me a URL to the ftp site.
Or, if possible, you could post it to our anonymous ftp site:
   https://dtcenter.org/met/users/support/met_help.php#ftp

But I believe Bob has had trouble doing that in the past.

Or you could run a wgrib2 command to extract out a single GRIB record
to make it small and try sending it as an attachment.  You may need to
add a ".txt" suffix to get passed the email filters.

Here's an example of running wgrib2 to select out record number 320
and write it to a new output file:
   wgrib2 -d 320 gfs.t00z.pgrb2.0p25.f072 -grib_out rec320.grib2

John

On Mon, Jan 28, 2019 at 6:04 AM SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN via RT <met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
>
> Hmmm, I sent it to met_help.  I wonder if my being a contractor was
an
> issue?  I never got any messages saying it wouldn't work.
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, January 24, 2019 4:07 PM
> To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> matthew.sittel.ctr at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
with
> GriB2 Headers and Forecast Hours
>
> Matt,
>
> I don't know how long it takes to come through, but I haven't
received
> an ARL SAFE email yet.
>
> You could try sending it to johnhg at ucar.edu.  Or if that doesn't
work,
> Bob has sent it to met_help at ucar.edu.  That automatically creates a
> new support ticket.  But then I just go delete that ticket... which
> isn't a big deal.
>
> John
>
> On Thu, Jan 24, 2019 at 2:28 PM SITTEL, MATTHEW C CTR USAF AFWA 16
> WS/WXN via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> >
> > Okay, I have sent you both files.  I sent the first email too
early;
> > it had only one file in it.  The second one has the two files in
the
> > same email.  Please disregard the first email with only one
attachment.
> >
> > Thanks,
> > Matt
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, January 24, 2019 3:07 PM
> > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > matthew.sittel.ctr at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
> > with
> > GriB2 Headers and Forecast Hours
> >
> > Bob sends me data through ARL SAFE and I get an email telling me
to
> > go pick them up.
> >
> > John
> >
> > On Thu, Jan 24, 2019 at 1:45 PM SITTEL, MATTHEW C CTR USAF AFWA 16
> > WS/WXN via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604 >
> > >
> > > That’s weird.  I can send you the offending model file and
> > > observation, both are 1/2-degree grids.  What method is Bob
Craig
> > > using to send you files?
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Thursday, January 24, 2019 2:41 PM
> > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > matthew.sittel.ctr at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
> > > with
> > > GriB2 Headers and Forecast Hours
> > >
> > > Matt,
> > >
> > > I had just run plot_data_plane, so I ran Grid-Stat as well...
just
> > > comparing the same field to itself:
> > >
> > > /usr/local/met-6.1/bin/grid_stat \
> > > gfs.t00z.pgrb2.0p25.f072 \
> > > gfs.t00z.pgrb2.0p25.f072 \
> > > GridStatConfig -outdir out -v 4 -log run_gs.log
> > >
> > > DEBUG 1: Output file:
out/grid_stat_720000L_20190127_000000V.stat
> > > DEBUG 1: Output file:
> > > out/grid_stat_720000L_20190127_000000V_cnt.txt
> > > DEBUG 1: Output file:
> > > out/grid_stat_720000L_20190127_000000V_pairs.nc
> > >
> > > Perhaps there's something different about the way the files
you're
> > > using are encoded?  Or perhaps there's a bug in the script that
> > > generates the call to Grid-Stat?
> > >
> > > If you can isolate the call to Grid-Stat that's resulting in the
> > > unexpected behavior, and then send me sample data, I'd be happy
to
> > > take a closer look and try to debug.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jan 24, 2019 at 12:12 PM SITTEL, MATTHEW C CTR USAF AFWA
> > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
>
> > > >
> > > > Interesting.  Did you run it through grid_stat or just
> plot_data_plane?
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: Thursday, January 24, 2019 12:17 PM
> > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > > matthew.sittel.ctr at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604]
Issue
> > > > with
> > > > GriB2 Headers and Forecast Hours
> > > >
> > > > Here it is:
> > > >
> > > > [johnhg at number5]% wgrib2 gfs.t00z.pgrb2.0p25.f072 | grep TCDC
> > > >
> > > >
> > > > On Thu, Jan 24, 2019 at 10:17 AM SITTEL, MATTHEW C CTR USAF
AFWA
> > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > > > > >
> > > > >
> > > > > Hey John,
> > > > >
> > > > > What does the header look like for TCDC when you degrib that
> > > > > single record?  Does it have that "66-72" field or just 72?
> > > > >
> > > > > Maybe I should send you the model and observation files so
you
> > > > > can try running grid_stat on it?
> > > > >
> > > > > Matt
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > > Sent: Thursday, January 24, 2019 11:02 AM
> > > > > To: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL <
> > > > > matthew.sittel.ctr at us.af.mil>
> > > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #88604] Issue
> > > > > with
> > > > > GriB2 Headers and Forecast Hours
> > > > >
> > > > > Matt,
> > > > >
> > > > > That's a good question.  I'm surprised that it's reporting
the
> > > > > beginning of the averaging period (66) instead of the end
(72).
> > > > > I wanted to get some data for testing so I pulled a recent
GFS
> file:
> > > > >
> > > > > *wget
> > > > >
> > > > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.201901
> > > > > 24
> > > > > 00
> > > > > /g
> > > > > fs
> > > > > .t00z.pgrb2.0p25.f072
> > > > > <
> > > > >
http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.201901
> > > > > 24
> > > > > 00
> > > > > /g
> > > > > fs
> > > > > .t00z.pgrb2.0p25.f072
> > > > > >*
> > > > >
> > > > > And then I ran a plot_data_plane command to plot TCDC using
> > > > > the -v
> > > > > 4 command line option to dump out more detailed info,
> > > > > including the timing
> > > > > information:
> > > > >
> > > > > *plot_data_plane gfs.t00z.pgrb2.0p25.f072
TCDC_entire_atmos.ps
> > > > > 'name="TCDC"; level="L0"; GRIB_lvl_typ=10;' -v 4*
> > > > >
> > > > > I chose GRIB_lvl_typ = 10 to get TCDC for the "entire
atmosphere"
> > > > > based on this GRIB2 code table (
> > > > >
> >
https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.
> > > > > shtml
> > > > > ).
> > > > >
> > > > > And here's the timing info that was dumped out:
> > > > > DEBUG 4: Data plane information:
> > > > > DEBUG 4:       plane min: 0
> > > > > DEBUG 4:       plane max: 100
> > > > > DEBUG 4:       scan mode: 0
> > > > > DEBUG 4:      valid time: 20190127_000000
> > > > > DEBUG 4:       lead time: 720000
> > > > > DEBUG 4:       init time: 20190124_000000
> > > > > DEBUG 4:     bitmap flag: 255
> > > > >
> > > > > One confusing bit is that when I reran with met-6.1, the
GRIB
> > > > > level codes were reported as different values.
> > > > >
> > > > > But that wrinkle aside, I'm not see the discrepancy in
timing
> > > > > info that you describe.
> > > > >
> > > > > Any thoughts?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jan 23, 2019 at 10:31 AM SITTEL, MATTHEW C CTR USAF
> > > > > AFWA
> > > > > 16 WS/WXN via RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Wed Jan 23 10:31:00 2019: Request 88604 was acted upon.
> > > > > > Transaction: Ticket created by
matthew.sittel.ctr at us.af.mil
> > > > > >        Queue: met_help
> > > > > >      Subject: Issue with GriB2 Headers and Forecast Hours
> > > > > >        Owner: Nobody
> > > > > >   Requestors: matthew.sittel.ctr at us.af.mil
> > > > > >       Status: new
> > > > > >  Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88604
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm verifying a GFS ensemble member's 72 hour total cloud
> > > > > > amount forecast against a 72-hour WWMCA ground truth.
When
> > > > > > grid_stat generates its cnt.txt output file, the forecast
> > > > > > hour reported is
> > > > > > 66
> > > > > instead of 72.
> > > > > >
> > > > > > In trying to figure out why it did this, I went into the
> > > > > > header for the
> > > > > > GriB2 data and found the following:
> > > > > >
> > > > > > 30:5224497:vt=2018120800:entire atmosphere:66-72 hour ave
> > > > > > fcst:TCDC Total Cloud Cover [%]:ENS=+1
> > > > > >     ndata=259920:undef=0:mean=52.3893:min=0:max=100
> > > > > >     grid_template=0:winds(N/S):
> > > > > >         lat-lon grid:(720 x 361) units 1e-06 input WE:NS
> > > > > > output WE:SN res
> > > > > > 48
> > > > > >         lat 90.000000 to -90.000000 by 0.500000
> > > > > >         lon 0.000000 to 359.500000 by 0.500000
> > > > > > #points=259920
> > > > > >
> > > > > > It looks like MET sees the range of forecast hours ("66-
72"
> > > > > > in this
> > > > > > case) and uses the first one to calculate the forecast
hour.
> > > > > >
> > > > > > If I use "-VT" as an option when I degrib this model file,
> > > > > > the valid forecast time reported back is 72 hours, not 66:
> > > > > >
> > > > > > 30:5224497:vt=20181208000000
> > > > > >
> > > > > > Ignoring the issue of trying to figure out how to verify a
> > > > > > 66 to
> > > > > > 72 hour average cloud forecast with a single WWMCA file,
is
> > > > > > there any way to get grid_stat to look at the valid time
> > > > > > instead of
> > > > > > (presumably) the header so that I get 72 in the cnt.txt
> > > > > > output rather
> > > > than 66?
> > > > > >
> > > > > > I haven't looked at ensemble_stat yet, but I do know that
> > > > > > the other two models in GEPS are NAVGEM and CMC, and both
of
> > > > > > them have
> > > > > > "72 hour fcst" in their header for the same field.  I'm
not
> > > > > > sure if ensemble_stat will allow us to match up members
from
> > > > > > GFS with NAVGEM and CMC if MET thinks they don't have a
> > > > > > common forecast hour amongst
> > > > > them.
> > > > > >
> > > > > > Thanks,
> > > > > > Matt
> > > > > >
> > > > > > Matthew C. Sittel, Meteorologist University Corporation
for
> > > > > > Atmospheric Research Matthew.Sittel.ctr at us.af.mil
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>

------------------------------------------------