[Met_help] [rt.rap.ucar.edu #84952] History for ceiling and visibility verification in MET
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:03:35 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi MET Help,
Several of us at EMC are interested in doing some verification of ceiling
and visibility using corresponding METAR observations in prepbufr files.
Speaking specifically, we would like to perform categorical statistics
based upon a variety of aviation thresholds (e.g. IFR, LIFR, etc.). Is this
possible in MET v7?
Thanks!
Jacob
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: ceiling and visibility verification in MET
From: John Halley Gotway
Time: Tue May 01 13:41:37 2018
Jacob,
Yes, I certainly think that should be possible using met-7.0. If I
remember correctly, Perry and I worked through this not too long ago.
If you'd like, I could work up an example of calling pb2nc and
point_stat
to demonstrate it. But I'd need you to point me to a sample model
output
file and PREPBUFR file that you'd like to use.
Thanks,
John
On Tue, May 1, 2018 at 12:33 PM, Jacob Carley - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Tue May 01 12:33:24 2018: Request 84952 was acted upon.
> Transaction: Ticket created by jacob.carley at noaa.gov
> Queue: met_help
> Subject: ceiling and visibility verification in MET
> Owner: Nobody
> Requestors: jacob.carley at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
>
>
> Hi MET Help,
>
> Several of us at EMC are interested in doing some verification of
ceiling
> and visibility using corresponding METAR observations in prepbufr
files.
> Speaking specifically, we would like to perform categorical
statistics
> based upon a variety of aviation thresholds (e.g. IFR, LIFR, etc.).
Is this
> possible in MET v7?
>
> Thanks!
> Jacob
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Tue May 01 13:52:09 2018
Hi John,
[cc'd Runhua Yang, as she's working on this project as well].
Wonderful - that would incredibly helpful.
I've put a sample grib2 file and an associated analysis file here on
gyre
(wcoss phase2):
/meso2/noscrub/Jacob.Carley/methelp
It's from a system that runs every 15 minutes, which hopefully isn't
an
issue for MET or METViewer (this particular file is valid at 1215Z
from
this morning).
Thanks again!
Jacob
On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Jacob,
>
> Yes, I certainly think that should be possible using met-7.0. If I
> remember correctly, Perry and I worked through this not too long
ago.
>
> If you'd like, I could work up an example of calling pb2nc and
point_stat
> to demonstrate it. But I'd need you to point me to a sample model
output
> file and PREPBUFR file that you'd like to use.
>
> Thanks,
> John
>
> On Tue, May 1, 2018 at 12:33 PM, Jacob Carley - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue May 01 12:33:24 2018: Request 84952 was acted upon.
> > Transaction: Ticket created by jacob.carley at noaa.gov
> > Queue: met_help
> > Subject: ceiling and visibility verification in MET
> > Owner: Nobody
> > Requestors: jacob.carley at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> >
> >
> > Hi MET Help,
> >
> > Several of us at EMC are interested in doing some verification of
ceiling
> > and visibility using corresponding METAR observations in prepbufr
files.
> > Speaking specifically, we would like to perform categorical
statistics
> > based upon a variety of aviation thresholds (e.g. IFR, LIFR,
etc.). Is
> this
> > possible in MET v7?
> >
> > Thanks!
> > Jacob
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: John Halley Gotway
Time: Tue May 01 14:50:59 2018
Jacob and Runhua,
Great, thanks. Here's the steps I took with this data:
(1) Run pb2nc using the "-index" option to the list observation
variable
names for each message type... takes a few minutes to run:
/usr/local/met-7.0/bin/pb2nc \
rtma2p5_ru.t1215z.prepbufr \
rtma2p5_ru.t1215z.prepbufr.nc \
PB2NCConfig_CandV \
-index -log index.log
I've attached the index.log file but here's a selection of the output
from
that run:
DEBUG 1: HOVI: HORIZONTAL
VISIBILITY
types: ADPSFC SFCSHP
DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF HOCB - HGT OF CLOUD
BASE) types: ADPSFC
You mentioned using METAR observations, but in that PREPBUFR file, I
only
see CEILING and visibility (HOVI) listed for the surface message
types.
(2) Configure PB2NC to retain the observation for C and V. See
attached
file PB2NC_CandV.
// Only keep surface message types
message_type = [ "ONLYSF" ];
// Only keep ceiling and visibility obs
obs_bufr_var = [ "CEILING", "HOVI" ];
// Out of convenience to get OBS to match GRIB2 variable names, rename
CEILING as CEIL and HOVI as VIS
obs_bufr_map = [
{ key = "CEILING"; val = "CEIL"; },
{ key = "HOVI"; val = "VIS"; }
];
(3) Rerun pb2nc to actually process these obs:
/usr/local/met-7.0/bin/pb2nc \
rtma2p5_ru.t1215z.prepbufr \
rtma2p5_ru.t1215z.prepbufr.nc \
PB2NCConfig_CandV
(4) Configure and run point_stat to do the vx. See attached
point_stat
config file. Since LIFR, IFR, MVFR, and VFR are defined in
feet/miles, I
used the convert() function to convert meters to those units prior to
verifying. You certainly don't need to do that... you could just
specify
the thresholds in meters instead:
/usr/local/met-7.0/bin/point_stat \
rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
rtma2p5_ru.t1215z.prepbufr.nc \
PointStatConfig_CandV \
-outdir out -v 3 -log run_ps.log
This yields 3214 matched pairs for ceiling and 3622 for visibility.
(5) I configured point_stat to dump the matched pairs (MPR) output
line
type. So I called the plot_mpr.R Rscript included with MET to see
some
plots of the data:
Rscript /usr/local/met-7.0/share/met/Rscripts/plot_mpr.R
out/point_stat_000000L_20180501_121500V_mpr.txt
The resulting PDF file is attached.
Looking at the scatter plot, there's a curious band of observation
values
around 65000 ft for ceiling... and around 10 miles for visibility.
You
should probably spend some time to understand this data and decide how
you
want to process it for verification.
Looking at the CTS output line type, I see that the equitable threat
scores
(i.e. GSS column) range between .14895 and .54148.
Hopefully that helps get you going.
Thanks,
John
On Tue, May 1, 2018 at 1:52 PM, Jacob Carley - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
>
> Hi John,
>
> [cc'd Runhua Yang, as she's working on this project as well].
>
> Wonderful - that would incredibly helpful.
>
> I've put a sample grib2 file and an associated analysis file here on
gyre
> (wcoss phase2):
>
> /meso2/noscrub/Jacob.Carley/methelp
>
> It's from a system that runs every 15 minutes, which hopefully isn't
an
> issue for MET or METViewer (this particular file is valid at 1215Z
from
> this morning).
>
> Thanks again!
> Jacob
>
>
>
>
>
>
>
> On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Jacob,
> >
> > Yes, I certainly think that should be possible using met-7.0. If
I
> > remember correctly, Perry and I worked through this not too long
ago.
> >
> > If you'd like, I could work up an example of calling pb2nc and
point_stat
> > to demonstrate it. But I'd need you to point me to a sample model
output
> > file and PREPBUFR file that you'd like to use.
> >
> > Thanks,
> > John
> >
> > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue May 01 12:33:24 2018: Request 84952 was acted upon.
> > > Transaction: Ticket created by jacob.carley at noaa.gov
> > > Queue: met_help
> > > Subject: ceiling and visibility verification in MET
> > > Owner: Nobody
> > > Requestors: jacob.carley at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952
> >
> > >
> > >
> > > Hi MET Help,
> > >
> > > Several of us at EMC are interested in doing some verification
of
> ceiling
> > > and visibility using corresponding METAR observations in
prepbufr
> files.
> > > Speaking specifically, we would like to perform categorical
statistics
> > > based upon a variety of aviation thresholds (e.g. IFR, LIFR,
etc.). Is
> > this
> > > possible in MET v7?
> > >
> > > Thanks!
> > > Jacob
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: John Halley Gotway
Time: Tue May 01 14:50:59 2018
VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END
OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR
OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL BASER BASER_NCL BASER_NCU
BASER_BCL BASER_BCU FMEAN FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU
ACC ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS FBIAS_BCL FBIAS_BCU
PODY PODY_NCL PODY_NCU PODY_BCL PODY_BCU PODN PODN_NCL PODN_NCU
PODN_BCL PODN_BCU POFD POFD_NCL POFD_NCU POFD_BCL POFD_BCU FAR
FAR_NCL FAR_NCU FAR_BCL FAR_BCU CSI CSI_NCL CSI_NCU CSI_BCL
CSI_BCU GSS GSS_BCL GSS_BCU HK HK_NCL HK_NCU HK_BCL HK_BCU
HSS HSS_BCL HSS_BCU ODDS ODDS_NCL ODDS_NCU ODDS_BCL
ODDS_BCU LODDS LODDS_NCL LODDS_NCU LODDS_BCL LODDS_BCU ORSS
ORSS_NCL ORSS_NCU ORSS_BCL ORSS_BCU EDS EDS_NCL EDS_NCU EDS_BCL
EDS_BCU SEDS SEDS_NCL SEDS_NCU SEDS_BCL SEDS_BCU EDI EDI_NCL
EDI_NCU EDI_BCL EDI_BCU SEDI SEDI_NCL SEDI_NCU SEDI_BCL SEDI_BCU
BAGSS BAGSS_BCL BAGSS_BCU
V7.0 RTMA2p5 NA 000000 20180501_121500 20180501_121500 000000
20180501_104500 20180501_134500 CEIL L0 CEIL L0
ONLYSF FULL NEAREST 1 <500 <500 NA
0.05 CTS 3214 0.046049 0.039329 0.053852 NA NA
0.064095 0.056137 0.073093 NA NA 0.94151 0.93285
0.9491 NA NA 1.39189 NA NA 0.56081 0.54359
0.57789 NA NA 0.95988 0.95253 0.96614 NA NA
0.040117 0.033864 0.047469 NA NA 0.59709 0.58002 0.61392
NA NA 0.30627 0.29058 0.32243 NA NA 0.28111 NA
NA 0.52069 0.43981 0.60157 NA NA 0.43885 NA NA
30.55272 21.07484 44.29305 NA NA 3.41945 3.04808
3.79083 NA NA 0.93661 0.91382 0.95941 NA
NA 0.68364 0.61799 0.74929 NA NA 0.59321 0.53109 0.65533
NA NA 0.69514 0.61682 0.77346 NA NA 0.73409 0.61844
0.77183 NA NA 0.2553 NA NA
V7.0 RTMA2p5 NA 000000 20180501_121500 20180501_121500 000000
20180501_104500 20180501_134500 CEIL L0 CEIL L0
ONLYSF FULL NEAREST 1 <1000 <1000 NA
0.05 CTS 3214 0.10143 0.091465 0.11235 NA NA
0.16366 0.15127 0.17685 NA NA 0.8911 0.87986
0.90141 NA NA 1.6135 NA NA 0.76994 0.75507
0.78416 NA NA 0.90478 0.89414 0.91445 NA NA
0.095222 0.085552 0.10586 NA NA 0.52281 0.50553 0.54004
NA NA 0.41764 0.4007 0.43478 NA NA 0.3609 NA
NA 0.67472 0.62667 0.72276 NA NA 0.53039 NA NA
31.79942 23.88274 42.34033 NA NA 3.45945 3.17316
3.74574 NA NA 0.93902 0.9221 0.95595 NA
NA 0.79493 0.75316 0.8367 NA NA 0.60731 0.5699 0.64471
NA NA 0.79989 0.74381 0.85597 NA NA 0.82713 0.74842
0.85136 NA NA 0.32908 NA NA
V7.0 RTMA2p5 NA 000000 20180501_121500 20180501_121500 000000
20180501_104500 20180501_134500 CEIL L0 CEIL L0
ONLYSF FULL NEAREST 1 <3000 <3000 NA
0.05 CTS 3214 0.22931 0.2151 0.24416 NA NA
0.30741 0.29169 0.32358 NA NA 0.85034 0.83759
0.86226 NA NA 1.34057 NA NA 0.84396 0.83101
0.8561 NA NA 0.85224 0.83955 0.86409 NA NA
0.14776 0.13591 0.16045 NA NA 0.37045 0.35391 0.38729
NA NA 0.56392 0.54671 0.58097 NA NA 0.45119 NA
NA 0.6962 0.66637 0.72604 NA NA 0.62182 NA NA
31.19606 24.84076 39.1773 NA NA 3.44029 3.21249
3.6681 NA NA 0.93788 0.92417 0.95159 NA
NA 0.79341 0.75951 0.82731 NA NA 0.61494 0.58442 0.64547
NA NA 0.83702 0.79648 0.87756 NA NA 0.83923 0.79686
0.87718 NA NA 0.4241 NA NA
V7.0 RTMA2p5 NA 000000 20180501_121500 20180501_121500 000000
20180501_104500 20180501_134500 CEIL L0 CEIL L0
ONLYSF FULL NEAREST 1 >=3000 >=3000 NA
0.05 CTS 3214 0.77069 0.75584 0.7849 NA NA
0.69259 0.67642 0.70831 NA NA 0.85034 0.83759
0.86226 NA NA 0.89867 NA NA 0.85224 0.83955
0.86409 NA NA 0.84396 0.83101 0.8561 NA NA
0.15604 0.1439 0.16899 NA NA 0.051662 0.044531
0.059864 NA NA 0.81443 0.80062 0.82749 NA NA
0.45119 NA NA 0.6962 0.66702 0.72538 NA NA 0.62182
NA NA 31.19606 24.84076 39.1773 NA NA 3.44029
3.21249 3.6681 NA NA 0.93788 0.92417 0.95159
NA NA 0.23928 0.19094 0.28762 NA NA 0.49345 0.43519
0.55171 NA NA 0.8415 0.81911 0.8639 NA NA
0.83923 0.81888 0.86412 NA NA 0.55303 NA
NA
V7.0 RTMA2p5 NA 000000 20180501_121500 20180501_121500 000000
20180501_104500 20180501_134500 VIS Z0 VIS Z0
ONLYSF FULL NEAREST 1 <1 <1 NA
0.05 CTS 3622 0.017118 0.013376 0.021882 NA NA
0.019326 0.015326 0.024346 NA NA 0.9735 0.96774
0.97825 NA NA 1.12903 NA NA 0.29032 0.27577
0.30532 NA NA 0.98539 0.98094 0.98882 NA NA
0.014607 0.011182 0.01906 NA NA 0.74286 0.72837 0.75683
NA NA 0.15789 0.14638 0.17013 NA NA 0.14895 NA
NA 0.27572 0.16439 0.38704 NA NA 0.25928 NA NA
27.5979 14.95139 50.94135 NA NA 3.31774 2.7048
3.93067 NA NA 0.93006 0.8887 0.97143 NA
NA 0.53369 0.42116 0.64621 NA NA 0.51081 0.39996 0.62165
NA NA 0.54723 0.42381 0.67064 NA NA 0.56999 0.4181
0.67635 NA NA 0.14202 NA NA
V7.0 RTMA2p5 NA 000000 20180501_121500 20180501_121500 000000
20180501_104500 20180501_134500 VIS Z0 VIS Z0
ONLYSF FULL NEAREST 1 <3 <3 NA
0.05 CTS 3622 0.033959 0.028537 0.040369 NA NA
0.029818 0.024757 0.035874 NA NA 0.96825 0.96202
0.97348 NA NA 0.87805 NA NA 0.47154 0.45533
0.48782 NA NA 0.98571 0.9813 0.98909 NA NA
0.01429 0.010907 0.018702 NA NA 0.46296 0.44677 0.47923
NA NA 0.33526 0.32007 0.3508 NA NA 0.32086 NA
NA 0.45725 0.36882 0.54569 NA NA 0.48584 NA NA
61.55138 39.21317 96.6148 NA NA 4.11987 3.66901
4.57073 NA NA 0.96803 0.95384 0.98221 NA
NA 0.63634 0.5623 0.71039 NA NA 0.6678 0.59233 0.74327
NA NA 0.6993 0.62568 0.77292 NA NA 0.7289 0.62532
0.77328 NA NA 0.34162 NA NA
V7.0 RTMA2p5 NA 000000 20180501_121500 20180501_121500 000000
20180501_104500 20180501_134500 VIS Z0 VIS Z0
ONLYSF FULL NEAREST 1 <5 <5 NA
0.05 CTS 3622 0.054942 0.047982 0.062845 NA NA
0.057151 0.05005 0.06519 NA NA 0.96853 0.96232
0.97373 NA NA 1.0402 NA NA 0.73367 0.71903
0.74781 NA NA 0.98218 0.97733 0.986 NA NA
0.017821 0.013995 0.022668 NA NA 0.29469 0.28006 0.30974
NA NA 0.56154 0.54532 0.57762 NA NA 0.54148 NA
NA 0.71585 0.65308 0.77861 NA NA 0.70255 NA NA
151.82555 101.40397 227.3185 NA NA 5.02273 4.61911
5.42635 NA NA 0.98691 0.98167 0.99216 NA
NA 0.80711 0.76 0.85422 NA NA 0.79484 0.74805 0.84163
NA NA 0.85719 0.81375 0.90063 NA NA 0.88458 0.81864
0.89573 NA NA 0.53436 NA NA
V7.0 RTMA2p5 NA 000000 20180501_121500 20180501_121500 000000
20180501_104500 20180501_134500 VIS Z0 VIS Z0
ONLYSF FULL NEAREST 1 >=5 >=5 NA
0.05 CTS 3622 0.94506 0.93715 0.95202 NA NA
0.94285 0.93481 0.94995 NA NA 0.96853 0.96232
0.97373 NA NA 0.99766 NA NA 0.98218 0.97733
0.986 NA NA 0.73367 0.71903 0.74781 NA NA
0.26633 0.25219 0.28097 NA NA 0.01552 0.011977
0.020089 NA NA 0.9672 0.96089 0.97253 NA NA
0.54148 NA NA 0.71585 0.695 0.7367 NA NA 0.70255
NA NA 151.82555 101.40397 227.3185 NA NA 5.02273
4.61911 5.42635 NA NA 0.98691 0.98167 0.99216
NA NA 0.51722 0.42531 0.60913 NA NA 0.54863 0.45482
0.64244 NA NA 0.97318 0.96157 0.9848 NA NA
0.88458 0.93161 1.01476 NA NA 0.55245 NA
NA
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Tue May 01 15:15:21 2018
Hi John,
Thanks so much! This is hugely helpful.
ADPSFC encompasses METAR observations, so that's correct.
This should be more than enough to get us rolling.
Thanks again!
Jacob
On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Jacob and Runhua,
>
> Great, thanks. Here's the steps I took with this data:
>
> (1) Run pb2nc using the "-index" option to the list observation
variable
> names for each message type... takes a few minutes to run:
>
> /usr/local/met-7.0/bin/pb2nc \
> rtma2p5_ru.t1215z.prepbufr \
> rtma2p5_ru.t1215z.prepbufr.nc \
> PB2NCConfig_CandV \
> -index -log index.log
>
> I've attached the index.log file but here's a selection of the
output from
> that run:
> DEBUG 1: HOVI: HORIZONTAL
> VISIBILITY
> types: ADPSFC SFCSHP
> DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF HOCB - HGT OF
CLOUD
> BASE) types: ADPSFC
>
> You mentioned using METAR observations, but in that PREPBUFR file, I
only
> see CEILING and visibility (HOVI) listed for the surface message
types.
>
> (2) Configure PB2NC to retain the observation for C and V. See
attached
> file PB2NC_CandV.
>
> // Only keep surface message types
> message_type = [ "ONLYSF" ];
> // Only keep ceiling and visibility obs
> obs_bufr_var = [ "CEILING", "HOVI" ];
> // Out of convenience to get OBS to match GRIB2 variable names,
rename
> CEILING as CEIL and HOVI as VIS
> obs_bufr_map = [
> { key = "CEILING"; val = "CEIL"; },
> { key = "HOVI"; val = "VIS"; }
> ];
>
> (3) Rerun pb2nc to actually process these obs:
>
> /usr/local/met-7.0/bin/pb2nc \
> rtma2p5_ru.t1215z.prepbufr \
> rtma2p5_ru.t1215z.prepbufr.nc \
> PB2NCConfig_CandV
>
> (4) Configure and run point_stat to do the vx. See attached
point_stat
> config file. Since LIFR, IFR, MVFR, and VFR are defined in
feet/miles, I
> used the convert() function to convert meters to those units prior
to
> verifying. You certainly don't need to do that... you could just
specify
> the thresholds in meters instead:
>
> /usr/local/met-7.0/bin/point_stat \
> rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> rtma2p5_ru.t1215z.prepbufr.nc \
> PointStatConfig_CandV \
> -outdir out -v 3 -log run_ps.log
>
> This yields 3214 matched pairs for ceiling and 3622 for visibility.
>
> (5) I configured point_stat to dump the matched pairs (MPR) output
line
> type. So I called the plot_mpr.R Rscript included with MET to see
some
> plots of the data:
>
> Rscript /usr/local/met-7.0/share/met/Rscripts/plot_mpr.R
> out/point_stat_000000L_20180501_121500V_mpr.txt
>
> The resulting PDF file is attached.
>
> Looking at the scatter plot, there's a curious band of observation
values
> around 65000 ft for ceiling... and around 10 miles for visibility.
You
> should probably spend some time to understand this data and decide
how you
> want to process it for verification.
>
> Looking at the CTS output line type, I see that the equitable threat
scores
> (i.e. GSS column) range between .14895 and .54148.
>
> Hopefully that helps get you going.
>
> Thanks,
> John
>
>
> On Tue, May 1, 2018 at 1:52 PM, Jacob Carley - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> >
> > Hi John,
> >
> > [cc'd Runhua Yang, as she's working on this project as well].
> >
> > Wonderful - that would incredibly helpful.
> >
> > I've put a sample grib2 file and an associated analysis file here
on gyre
> > (wcoss phase2):
> >
> > /meso2/noscrub/Jacob.Carley/methelp
> >
> > It's from a system that runs every 15 minutes, which hopefully
isn't an
> > issue for MET or METViewer (this particular file is valid at 1215Z
from
> > this morning).
> >
> > Thanks again!
> > Jacob
> >
> >
> >
> >
> >
> >
> >
> > On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Jacob,
> > >
> > > Yes, I certainly think that should be possible using met-7.0.
If I
> > > remember correctly, Perry and I worked through this not too long
ago.
> > >
> > > If you'd like, I could work up an example of calling pb2nc and
> point_stat
> > > to demonstrate it. But I'd need you to point me to a sample
model
> output
> > > file and PREPBUFR file that you'd like to use.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue May 01 12:33:24 2018: Request 84952 was acted upon.
> > > > Transaction: Ticket created by jacob.carley at noaa.gov
> > > > Queue: met_help
> > > > Subject: ceiling and visibility verification in MET
> > > > Owner: Nobody
> > > > Requestors: jacob.carley at noaa.gov
> > > > Status: new
> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=84952
> > >
> > > >
> > > >
> > > > Hi MET Help,
> > > >
> > > > Several of us at EMC are interested in doing some verification
of
> > ceiling
> > > > and visibility using corresponding METAR observations in
prepbufr
> > files.
> > > > Speaking specifically, we would like to perform categorical
> statistics
> > > > based upon a variety of aviation thresholds (e.g. IFR, LIFR,
etc.).
> Is
> > > this
> > > > possible in MET v7?
> > > >
> > > > Thanks!
> > > > Jacob
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: John Halley Gotway
Time: Tue May 01 15:51:13 2018
Jacob,
Great, I'll go ahead and resolve this ticket. Just let us know if
anything
else comes up.
Thanks,
John
On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
>
> Hi John,
>
> Thanks so much! This is hugely helpful.
>
> ADPSFC encompasses METAR observations, so that's correct.
>
> This should be more than enough to get us rolling.
>
> Thanks again!
> Jacob
>
> On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Jacob and Runhua,
> >
> > Great, thanks. Here's the steps I took with this data:
> >
> > (1) Run pb2nc using the "-index" option to the list observation
variable
> > names for each message type... takes a few minutes to run:
> >
> > /usr/local/met-7.0/bin/pb2nc \
> > rtma2p5_ru.t1215z.prepbufr \
> > rtma2p5_ru.t1215z.prepbufr.nc \
> > PB2NCConfig_CandV \
> > -index -log index.log
> >
> > I've attached the index.log file but here's a selection of the
output
> from
> > that run:
> > DEBUG 1: HOVI: HORIZONTAL
> > VISIBILITY
> > types: ADPSFC SFCSHP
> > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF HOCB - HGT OF
CLOUD
> > BASE) types: ADPSFC
> >
> > You mentioned using METAR observations, but in that PREPBUFR file,
I only
> > see CEILING and visibility (HOVI) listed for the surface message
types.
> >
> > (2) Configure PB2NC to retain the observation for C and V. See
attached
> > file PB2NC_CandV.
> >
> > // Only keep surface message types
> > message_type = [ "ONLYSF" ];
> > // Only keep ceiling and visibility obs
> > obs_bufr_var = [ "CEILING", "HOVI" ];
> > // Out of convenience to get OBS to match GRIB2 variable names,
rename
> > CEILING as CEIL and HOVI as VIS
> > obs_bufr_map = [
> > { key = "CEILING"; val = "CEIL"; },
> > { key = "HOVI"; val = "VIS"; }
> > ];
> >
> > (3) Rerun pb2nc to actually process these obs:
> >
> > /usr/local/met-7.0/bin/pb2nc \
> > rtma2p5_ru.t1215z.prepbufr \
> > rtma2p5_ru.t1215z.prepbufr.nc \
> > PB2NCConfig_CandV
> >
> > (4) Configure and run point_stat to do the vx. See attached
point_stat
> > config file. Since LIFR, IFR, MVFR, and VFR are defined in
feet/miles, I
> > used the convert() function to convert meters to those units prior
to
> > verifying. You certainly don't need to do that... you could just
specify
> > the thresholds in meters instead:
> >
> > /usr/local/met-7.0/bin/point_stat \
> > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> > rtma2p5_ru.t1215z.prepbufr.nc \
> > PointStatConfig_CandV \
> > -outdir out -v 3 -log run_ps.log
> >
> > This yields 3214 matched pairs for ceiling and 3622 for
visibility.
> >
> > (5) I configured point_stat to dump the matched pairs (MPR) output
line
> > type. So I called the plot_mpr.R Rscript included with MET to see
some
> > plots of the data:
> >
> > Rscript /usr/local/met-7.0/share/met/Rscripts/plot_mpr.R
> > out/point_stat_000000L_20180501_121500V_mpr.txt
> >
> > The resulting PDF file is attached.
> >
> > Looking at the scatter plot, there's a curious band of observation
values
> > around 65000 ft for ceiling... and around 10 miles for visibility.
You
> > should probably spend some time to understand this data and decide
how
> you
> > want to process it for verification.
> >
> > Looking at the CTS output line type, I see that the equitable
threat
> scores
> > (i.e. GSS column) range between .14895 and .54148.
> >
> > Hopefully that helps get you going.
> >
> > Thanks,
> > John
> >
> >
> > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > >
> > > Hi John,
> > >
> > > [cc'd Runhua Yang, as she's working on this project as well].
> > >
> > > Wonderful - that would incredibly helpful.
> > >
> > > I've put a sample grib2 file and an associated analysis file
here on
> gyre
> > > (wcoss phase2):
> > >
> > > /meso2/noscrub/Jacob.Carley/methelp
> > >
> > > It's from a system that runs every 15 minutes, which hopefully
isn't an
> > > issue for MET or METViewer (this particular file is valid at
1215Z from
> > > this morning).
> > >
> > > Thanks again!
> > > Jacob
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu
> > > > wrote:
> > >
> > > > Jacob,
> > > >
> > > > Yes, I certainly think that should be possible using met-7.0.
If I
> > > > remember correctly, Perry and I worked through this not too
long ago.
> > > >
> > > > If you'd like, I could work up an example of calling pb2nc and
> > point_stat
> > > > to demonstrate it. But I'd need you to point me to a sample
model
> > output
> > > > file and PREPBUFR file that you'd like to use.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley - NOAA Affiliate
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue May 01 12:33:24 2018: Request 84952 was acted upon.
> > > > > Transaction: Ticket created by jacob.carley at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: ceiling and visibility verification in MET
> > > > > Owner: Nobody
> > > > > Requestors: jacob.carley at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=84952
> > > >
> > > > >
> > > > >
> > > > > Hi MET Help,
> > > > >
> > > > > Several of us at EMC are interested in doing some
verification of
> > > ceiling
> > > > > and visibility using corresponding METAR observations in
prepbufr
> > > files.
> > > > > Speaking specifically, we would like to perform categorical
> > statistics
> > > > > based upon a variety of aviation thresholds (e.g. IFR, LIFR,
etc.).
> > Is
> > > > this
> > > > > possible in MET v7?
> > > > >
> > > > > Thanks!
> > > > > Jacob
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Mon May 21 14:52:28 2018
Hi John,
A quick follow up to this ticket - I have an additional variable I'd
like
to add and wanted to go the route of running pb2nc with '-index' as
you did
in this example. However when I do so, I get far too much output that
I
can sensibly parse (log files well over 1G). Is this typical? I'm
looking
to see if I can pull out wind gust (the prepbufr mnemonic is 'MXGS').
I
figure I must not have something set quite right in my pb2nc config
file.
Thanks,
Jacob
On Tue, May 1, 2018 at 5:51 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Jacob,
>
> Great, I'll go ahead and resolve this ticket. Just let us know if
anything
> else comes up.
>
> Thanks,
> John
>
> On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> >
> > Hi John,
> >
> > Thanks so much! This is hugely helpful.
> >
> > ADPSFC encompasses METAR observations, so that's correct.
> >
> > This should be more than enough to get us rolling.
> >
> > Thanks again!
> > Jacob
> >
> > On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Jacob and Runhua,
> > >
> > > Great, thanks. Here's the steps I took with this data:
> > >
> > > (1) Run pb2nc using the "-index" option to the list observation
> variable
> > > names for each message type... takes a few minutes to run:
> > >
> > > /usr/local/met-7.0/bin/pb2nc \
> > > rtma2p5_ru.t1215z.prepbufr \
> > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > PB2NCConfig_CandV \
> > > -index -log index.log
> > >
> > > I've attached the index.log file but here's a selection of the
output
> > from
> > > that run:
> > > DEBUG 1: HOVI: HORIZONTAL
> > > VISIBILITY
> > > types: ADPSFC SFCSHP
> > > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF HOCB - HGT OF
CLOUD
> > > BASE) types: ADPSFC
> > >
> > > You mentioned using METAR observations, but in that PREPBUFR
file, I
> only
> > > see CEILING and visibility (HOVI) listed for the surface message
types.
> > >
> > > (2) Configure PB2NC to retain the observation for C and V. See
> attached
> > > file PB2NC_CandV.
> > >
> > > // Only keep surface message types
> > > message_type = [ "ONLYSF" ];
> > > // Only keep ceiling and visibility obs
> > > obs_bufr_var = [ "CEILING", "HOVI" ];
> > > // Out of convenience to get OBS to match GRIB2 variable names,
rename
> > > CEILING as CEIL and HOVI as VIS
> > > obs_bufr_map = [
> > > { key = "CEILING"; val = "CEIL"; },
> > > { key = "HOVI"; val = "VIS"; }
> > > ];
> > >
> > > (3) Rerun pb2nc to actually process these obs:
> > >
> > > /usr/local/met-7.0/bin/pb2nc \
> > > rtma2p5_ru.t1215z.prepbufr \
> > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > PB2NCConfig_CandV
> > >
> > > (4) Configure and run point_stat to do the vx. See attached
point_stat
> > > config file. Since LIFR, IFR, MVFR, and VFR are defined in
> feet/miles, I
> > > used the convert() function to convert meters to those units
prior to
> > > verifying. You certainly don't need to do that... you could
just
> specify
> > > the thresholds in meters instead:
> > >
> > > /usr/local/met-7.0/bin/point_stat \
> > > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > PointStatConfig_CandV \
> > > -outdir out -v 3 -log run_ps.log
> > >
> > > This yields 3214 matched pairs for ceiling and 3622 for
visibility.
> > >
> > > (5) I configured point_stat to dump the matched pairs (MPR)
output line
> > > type. So I called the plot_mpr.R Rscript included with MET to
see some
> > > plots of the data:
> > >
> > > Rscript /usr/local/met-7.0/share/met/Rscripts/plot_mpr.R
> > > out/point_stat_000000L_20180501_121500V_mpr.txt
> > >
> > > The resulting PDF file is attached.
> > >
> > > Looking at the scatter plot, there's a curious band of
observation
> values
> > > around 65000 ft for ceiling... and around 10 miles for
visibility. You
> > > should probably spend some time to understand this data and
decide how
> > you
> > > want to process it for verification.
> > >
> > > Looking at the CTS output line type, I see that the equitable
threat
> > scores
> > > (i.e. GSS column) range between .14895 and .54148.
> > >
> > > Hopefully that helps get you going.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952
>
> > > >
> > > > Hi John,
> > > >
> > > > [cc'd Runhua Yang, as she's working on this project as well].
> > > >
> > > > Wonderful - that would incredibly helpful.
> > > >
> > > > I've put a sample grib2 file and an associated analysis file
here on
> > gyre
> > > > (wcoss phase2):
> > > >
> > > > /meso2/noscrub/Jacob.Carley/methelp
> > > >
> > > > It's from a system that runs every 15 minutes, which hopefully
isn't
> an
> > > > issue for MET or METViewer (this particular file is valid at
1215Z
> from
> > > > this morning).
> > > >
> > > > Thanks again!
> > > > Jacob
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu
> > > > > wrote:
> > > >
> > > > > Jacob,
> > > > >
> > > > > Yes, I certainly think that should be possible using met-
7.0. If I
> > > > > remember correctly, Perry and I worked through this not too
long
> ago.
> > > > >
> > > > > If you'd like, I could work up an example of calling pb2nc
and
> > > point_stat
> > > > > to demonstrate it. But I'd need you to point me to a sample
model
> > > output
> > > > > file and PREPBUFR file that you'd like to use.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley - NOAA
Affiliate via
> > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Tue May 01 12:33:24 2018: Request 84952 was acted upon.
> > > > > > Transaction: Ticket created by jacob.carley at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: ceiling and visibility verification in MET
> > > > > > Owner: Nobody
> > > > > > Requestors: jacob.carley at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=84952
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi MET Help,
> > > > > >
> > > > > > Several of us at EMC are interested in doing some
verification of
> > > > ceiling
> > > > > > and visibility using corresponding METAR observations in
prepbufr
> > > > files.
> > > > > > Speaking specifically, we would like to perform
categorical
> > > statistics
> > > > > > based upon a variety of aviation thresholds (e.g. IFR,
LIFR,
> etc.).
> > > Is
> > > > > this
> > > > > > possible in MET v7?
> > > > > >
> > > > > > Thanks!
> > > > > > Jacob
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: John Halley Gotway
Time: Mon May 21 15:38:17 2018
Jacob,
There's something odd going on here. You shouldn't see a 1G log file.
When I ran that command was 11k. And indeed, I do see MXGS listed:
DEBUG 1: MXGS: MAXIMUM WIND SPEED
(GUSTS) types: ADPSFC SFCSHP MSONET
I've attached the log file to this email for reference.
Here's the command I ran to generate this index file:
/usr/local/met-7.0/bin/pb2nc \
rtma2p5_ru.t1215z.prepbufr \
rtma2p5_ru.t1215z.prepbufr.nc \
PB2NCConfig_CandV \
-index -log index.log
When you run with the "-index" option, no NetCDF output is actually
generated. It just search through the input file, figures out what
observations are present, and then prints them to the screen. And
also a
log file since we used "-log index.log".
Are you seeing behavior that differs from this?
Thanks,
John
On Mon, May 21, 2018 at 2:52 PM, Jacob Carley - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
>
> Hi John,
>
> A quick follow up to this ticket - I have an additional variable I'd
like
> to add and wanted to go the route of running pb2nc with '-index' as
you did
> in this example. However when I do so, I get far too much output
that I
> can sensibly parse (log files well over 1G). Is this typical? I'm
looking
> to see if I can pull out wind gust (the prepbufr mnemonic is
'MXGS'). I
> figure I must not have something set quite right in my pb2nc config
file.
>
> Thanks,
> Jacob
>
>
>
> On Tue, May 1, 2018 at 5:51 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Jacob,
> >
> > Great, I'll go ahead and resolve this ticket. Just let us know if
> anything
> > else comes up.
> >
> > Thanks,
> > John
> >
> > On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > >
> > > Hi John,
> > >
> > > Thanks so much! This is hugely helpful.
> > >
> > > ADPSFC encompasses METAR observations, so that's correct.
> > >
> > > This should be more than enough to get us rolling.
> > >
> > > Thanks again!
> > > Jacob
> > >
> > > On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu
> > > > wrote:
> > >
> > > > Jacob and Runhua,
> > > >
> > > > Great, thanks. Here's the steps I took with this data:
> > > >
> > > > (1) Run pb2nc using the "-index" option to the list
observation
> > variable
> > > > names for each message type... takes a few minutes to run:
> > > >
> > > > /usr/local/met-7.0/bin/pb2nc \
> > > > rtma2p5_ru.t1215z.prepbufr \
> > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > PB2NCConfig_CandV \
> > > > -index -log index.log
> > > >
> > > > I've attached the index.log file but here's a selection of the
output
> > > from
> > > > that run:
> > > > DEBUG 1: HOVI: HORIZONTAL
> > > > VISIBILITY
> > > > types: ADPSFC SFCSHP
> > > > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF HOCB - HGT
OF
> CLOUD
> > > > BASE) types: ADPSFC
> > > >
> > > > You mentioned using METAR observations, but in that PREPBUFR
file, I
> > only
> > > > see CEILING and visibility (HOVI) listed for the surface
message
> types.
> > > >
> > > > (2) Configure PB2NC to retain the observation for C and V.
See
> > attached
> > > > file PB2NC_CandV.
> > > >
> > > > // Only keep surface message types
> > > > message_type = [ "ONLYSF" ];
> > > > // Only keep ceiling and visibility obs
> > > > obs_bufr_var = [ "CEILING", "HOVI" ];
> > > > // Out of convenience to get OBS to match GRIB2 variable
names,
> rename
> > > > CEILING as CEIL and HOVI as VIS
> > > > obs_bufr_map = [
> > > > { key = "CEILING"; val = "CEIL"; },
> > > > { key = "HOVI"; val = "VIS"; }
> > > > ];
> > > >
> > > > (3) Rerun pb2nc to actually process these obs:
> > > >
> > > > /usr/local/met-7.0/bin/pb2nc \
> > > > rtma2p5_ru.t1215z.prepbufr \
> > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > PB2NCConfig_CandV
> > > >
> > > > (4) Configure and run point_stat to do the vx. See attached
> point_stat
> > > > config file. Since LIFR, IFR, MVFR, and VFR are defined in
> > feet/miles, I
> > > > used the convert() function to convert meters to those units
prior to
> > > > verifying. You certainly don't need to do that... you could
just
> > specify
> > > > the thresholds in meters instead:
> > > >
> > > > /usr/local/met-7.0/bin/point_stat \
> > > > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > PointStatConfig_CandV \
> > > > -outdir out -v 3 -log run_ps.log
> > > >
> > > > This yields 3214 matched pairs for ceiling and 3622 for
visibility.
> > > >
> > > > (5) I configured point_stat to dump the matched pairs (MPR)
output
> line
> > > > type. So I called the plot_mpr.R Rscript included with MET to
see
> some
> > > > plots of the data:
> > > >
> > > > Rscript /usr/local/met-7.0/share/met/Rscripts/plot_mpr.R
> > > > out/point_stat_000000L_20180501_121500V_mpr.txt
> > > >
> > > > The resulting PDF file is attached.
> > > >
> > > > Looking at the scatter plot, there's a curious band of
observation
> > values
> > > > around 65000 ft for ceiling... and around 10 miles for
visibility.
> You
> > > > should probably spend some time to understand this data and
decide
> how
> > > you
> > > > want to process it for verification.
> > > >
> > > > Looking at the CTS output line type, I see that the equitable
threat
> > > scores
> > > > (i.e. GSS column) range between .14895 and .54148.
> > > >
> > > > Hopefully that helps get you going.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley - NOAA Affiliate
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > [cc'd Runhua Yang, as she's working on this project as
well].
> > > > >
> > > > > Wonderful - that would incredibly helpful.
> > > > >
> > > > > I've put a sample grib2 file and an associated analysis file
here
> on
> > > gyre
> > > > > (wcoss phase2):
> > > > >
> > > > > /meso2/noscrub/Jacob.Carley/methelp
> > > > >
> > > > > It's from a system that runs every 15 minutes, which
hopefully
> isn't
> > an
> > > > > issue for MET or METViewer (this particular file is valid at
1215Z
> > from
> > > > > this morning).
> > > > >
> > > > > Thanks again!
> > > > > Jacob
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu
> > > > > > wrote:
> > > > >
> > > > > > Jacob,
> > > > > >
> > > > > > Yes, I certainly think that should be possible using met-
7.0.
> If I
> > > > > > remember correctly, Perry and I worked through this not
too long
> > ago.
> > > > > >
> > > > > > If you'd like, I could work up an example of calling pb2nc
and
> > > > point_stat
> > > > > > to demonstrate it. But I'd need you to point me to a
sample
> model
> > > > output
> > > > > > file and PREPBUFR file that you'd like to use.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley - NOAA
Affiliate
> via
> > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Tue May 01 12:33:24 2018: Request 84952 was acted upon.
> > > > > > > Transaction: Ticket created by jacob.carley at noaa.gov
> > > > > > > Queue: met_help
> > > > > > > Subject: ceiling and visibility verification in MET
> > > > > > > Owner: Nobody
> > > > > > > Requestors: jacob.carley at noaa.gov
> > > > > > > Status: new
> > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > Ticket/Display.html?id=84952
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi MET Help,
> > > > > > >
> > > > > > > Several of us at EMC are interested in doing some
verification
> of
> > > > > ceiling
> > > > > > > and visibility using corresponding METAR observations in
> prepbufr
> > > > > files.
> > > > > > > Speaking specifically, we would like to perform
categorical
> > > > statistics
> > > > > > > based upon a variety of aviation thresholds (e.g. IFR,
LIFR,
> > etc.).
> > > > Is
> > > > > > this
> > > > > > > possible in MET v7?
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Jacob
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Mon May 21 16:09:34 2018
Hi John,
I'm running with 7.1 beta3 build on WCOSS, perhaps that's the culprit.
When I run it with -index here is what it prints to the screen (and in
the
log file) in what I think is an infinite loop:
DEBUG 1: User Config File: PB2NCConfig_gust
DEBUG 1: Pre-processing BUFR File for variable names from
rtma2p5_ru.t1545z.prepbufr
DEBUG 1: process_pbfile_metadata() -> the number of records: 114311
DEBUG 1:
DEBUG 1: Header variables:
DEBUG 1: SAID: SATELLITE IDENTIFIER (SATELLITE REPORTS ONLY)
DEBUG 1: XOB: LONGITUDE
DEBUG 1: YOB: LATITUDE
DEBUG 1: YEAR:
DEBUG 1: MNTH:
DEBUG 1: DAYS:
DEBUG 1: HOUR:
DEBUG 1: MINU:
DEBUG 1: SECO:
DEBUG 1:
DEBUG 1: Observation variables:
DEBUG 1: EEQF: PERCENT CONFIDENCE BASED ON NESDIS EXPECTED
ERROR types:
SATWND
DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL INDX
W/O
FCST types: SATWND
DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL INDX
W/ FCST types:
SATWND
DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS RECURSIVE
FILTER
FCN types: SATWND
DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT ASSIMILATED)
types:
SATWND
DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT PROGRAM
CODE types:
SATWND ADPSFC SFCSHP MSONET ASCATW
DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT ASSIMILATED)
types:
SATWND ADPSFC SFCSHP MSONET
DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
types:
SATWND ADPSFC SFCSHP MSONET
DEBUG 1: ZQM: HEIGHT (QUALITY) MARKER
types:
SATWND ADPSFC SFCSHP MSONET
DEBUG 1: ZOB: HEIGHT OBSERVATION
types:
SATWND ADPSFC SFCSHP MSONET
DEBUG 1: CAT: PREPBUFR DATA LEVEL CATEGORY
types:
SATWND ADPSFC SFCSHP MSONET ASCATW
*DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL INDX
W/O
FCST types: SATWND*
DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL INDX
W/ FCST types:
SATWND
DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS RECURSIVE
FILTER
FCN types: SATWND
DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT ASSIMILATED)
types:
SATWND
DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT PROGRAM
CODE types:
SATWND ADPSFC SFCSHP MSONET ASCATW
DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT ASSIMILATED)
types:
SATWND ADPSFC SFCSHP MSONET
DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
types:
SATWND ADPSFC SFCSHP MSONET
Which is pretty much just printed infinitely. I bolded the line above
that
shows when it begins to repeat
Here's the command I use:
pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
PB2NCConfig_gust -index -log index.log
I just switched over to 7.0 on WCOSS (non-beta) and I get the same
behavior
with that as well.
Does it matter what's in the config file when running with the '-
index'
option aside from some basic stuff like the time window?
Thanks!
Jacob
On Mon, May 21, 2018 at 5:38 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Jacob,
>
> There's something odd going on here. You shouldn't see a 1G log
file.
> When I ran that command was 11k. And indeed, I do see MXGS listed:
> DEBUG 1: MXGS: MAXIMUM WIND SPEED
> (GUSTS) types: ADPSFC SFCSHP
MSONET
> I've attached the log file to this email for reference.
>
> Here's the command I ran to generate this index file:
> /usr/local/met-7.0/bin/pb2nc \
> rtma2p5_ru.t1215z.prepbufr \
> rtma2p5_ru.t1215z.prepbufr.nc \
> PB2NCConfig_CandV \
> -index -log index.log
>
> When you run with the "-index" option, no NetCDF output is actually
> generated. It just search through the input file, figures out what
> observations are present, and then prints them to the screen. And
also a
> log file since we used "-log index.log".
>
> Are you seeing behavior that differs from this?
>
> Thanks,
> John
>
>
> On Mon, May 21, 2018 at 2:52 PM, Jacob Carley - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> >
> > Hi John,
> >
> > A quick follow up to this ticket - I have an additional variable
I'd like
> > to add and wanted to go the route of running pb2nc with '-index'
as you
> did
> > in this example. However when I do so, I get far too much output
that I
> > can sensibly parse (log files well over 1G). Is this typical?
I'm
> looking
> > to see if I can pull out wind gust (the prepbufr mnemonic is
'MXGS'). I
> > figure I must not have something set quite right in my pb2nc
config file.
> >
> > Thanks,
> > Jacob
> >
> >
> >
> > On Tue, May 1, 2018 at 5:51 PM, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Jacob,
> > >
> > > Great, I'll go ahead and resolve this ticket. Just let us know
if
> > anything
> > > else comes up.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952
>
> > > >
> > > > Hi John,
> > > >
> > > > Thanks so much! This is hugely helpful.
> > > >
> > > > ADPSFC encompasses METAR observations, so that's correct.
> > > >
> > > > This should be more than enough to get us rolling.
> > > >
> > > > Thanks again!
> > > > Jacob
> > > >
> > > > On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu
> > > > > wrote:
> > > >
> > > > > Jacob and Runhua,
> > > > >
> > > > > Great, thanks. Here's the steps I took with this data:
> > > > >
> > > > > (1) Run pb2nc using the "-index" option to the list
observation
> > > variable
> > > > > names for each message type... takes a few minutes to run:
> > > > >
> > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > PB2NCConfig_CandV \
> > > > > -index -log index.log
> > > > >
> > > > > I've attached the index.log file but here's a selection of
the
> output
> > > > from
> > > > > that run:
> > > > > DEBUG 1: HOVI: HORIZONTAL
> > > > > VISIBILITY
> > > > > types: ADPSFC SFCSHP
> > > > > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF HOCB -
HGT OF
> > CLOUD
> > > > > BASE) types: ADPSFC
> > > > >
> > > > > You mentioned using METAR observations, but in that PREPBUFR
file,
> I
> > > only
> > > > > see CEILING and visibility (HOVI) listed for the surface
message
> > types.
> > > > >
> > > > > (2) Configure PB2NC to retain the observation for C and V.
See
> > > attached
> > > > > file PB2NC_CandV.
> > > > >
> > > > > // Only keep surface message types
> > > > > message_type = [ "ONLYSF" ];
> > > > > // Only keep ceiling and visibility obs
> > > > > obs_bufr_var = [ "CEILING", "HOVI" ];
> > > > > // Out of convenience to get OBS to match GRIB2 variable
names,
> > rename
> > > > > CEILING as CEIL and HOVI as VIS
> > > > > obs_bufr_map = [
> > > > > { key = "CEILING"; val = "CEIL"; },
> > > > > { key = "HOVI"; val = "VIS"; }
> > > > > ];
> > > > >
> > > > > (3) Rerun pb2nc to actually process these obs:
> > > > >
> > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > PB2NCConfig_CandV
> > > > >
> > > > > (4) Configure and run point_stat to do the vx. See attached
> > point_stat
> > > > > config file. Since LIFR, IFR, MVFR, and VFR are defined in
> > > feet/miles, I
> > > > > used the convert() function to convert meters to those units
prior
> to
> > > > > verifying. You certainly don't need to do that... you could
just
> > > specify
> > > > > the thresholds in meters instead:
> > > > >
> > > > > /usr/local/met-7.0/bin/point_stat \
> > > > > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > PointStatConfig_CandV \
> > > > > -outdir out -v 3 -log run_ps.log
> > > > >
> > > > > This yields 3214 matched pairs for ceiling and 3622 for
visibility.
> > > > >
> > > > > (5) I configured point_stat to dump the matched pairs (MPR)
output
> > line
> > > > > type. So I called the plot_mpr.R Rscript included with MET
to see
> > some
> > > > > plots of the data:
> > > > >
> > > > > Rscript /usr/local/met-7.0/share/met/Rscripts/plot_mpr.R
> > > > > out/point_stat_000000L_20180501_121500V_mpr.txt
> > > > >
> > > > > The resulting PDF file is attached.
> > > > >
> > > > > Looking at the scatter plot, there's a curious band of
observation
> > > values
> > > > > around 65000 ft for ceiling... and around 10 miles for
visibility.
> > You
> > > > > should probably spend some time to understand this data and
decide
> > how
> > > > you
> > > > > want to process it for verification.
> > > > >
> > > > > Looking at the CTS output line type, I see that the
equitable
> threat
> > > > scores
> > > > > (i.e. GSS column) range between .14895 and .54148.
> > > > >
> > > > > Hopefully that helps get you going.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley - NOAA
Affiliate via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > [cc'd Runhua Yang, as she's working on this project as
well].
> > > > > >
> > > > > > Wonderful - that would incredibly helpful.
> > > > > >
> > > > > > I've put a sample grib2 file and an associated analysis
file here
> > on
> > > > gyre
> > > > > > (wcoss phase2):
> > > > > >
> > > > > > /meso2/noscrub/Jacob.Carley/methelp
> > > > > >
> > > > > > It's from a system that runs every 15 minutes, which
hopefully
> > isn't
> > > an
> > > > > > issue for MET or METViewer (this particular file is valid
at
> 1215Z
> > > from
> > > > > > this morning).
> > > > > >
> > > > > > Thanks again!
> > > > > > Jacob
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > wrote:
> > > > > >
> > > > > > > Jacob,
> > > > > > >
> > > > > > > Yes, I certainly think that should be possible using
met-7.0.
> > If I
> > > > > > > remember correctly, Perry and I worked through this not
too
> long
> > > ago.
> > > > > > >
> > > > > > > If you'd like, I could work up an example of calling
pb2nc and
> > > > > point_stat
> > > > > > > to demonstrate it. But I'd need you to point me to a
sample
> > model
> > > > > output
> > > > > > > file and PREPBUFR file that you'd like to use.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley - NOAA
Affiliate
> > via
> > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Tue May 01 12:33:24 2018: Request 84952 was acted
upon.
> > > > > > > > Transaction: Ticket created by jacob.carley at noaa.gov
> > > > > > > > Queue: met_help
> > > > > > > > Subject: ceiling and visibility verification in
MET
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: jacob.carley at noaa.gov
> > > > > > > > Status: new
> > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > Ticket/Display.html?id=84952
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi MET Help,
> > > > > > > >
> > > > > > > > Several of us at EMC are interested in doing some
> verification
> > of
> > > > > > ceiling
> > > > > > > > and visibility using corresponding METAR observations
in
> > prepbufr
> > > > > > files.
> > > > > > > > Speaking specifically, we would like to perform
categorical
> > > > > statistics
> > > > > > > > based upon a variety of aviation thresholds (e.g. IFR,
LIFR,
> > > etc.).
> > > > > Is
> > > > > > > this
> > > > > > > > possible in MET v7?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Jacob
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: John Halley Gotway
Time: Mon May 21 16:40:44 2018
Jacob,
No, the contents of the config file don't matter, at least to my
knowledge. I'm not able to replicate this behavior on the sample file
that
I have.
The big question is whether you've found a data file that puts pb2nc
into a
bad loop... or whether we're getting different behavior on different
platforms. Hopefully it's the first problem since that'd be a lot
easier
to fix.
Where can I get a hold of the input file you're using:
rtma2p5_ru.t1545z.prepbufr
If you put it up on theia or an ftp site somewhere, I can grab it, and
try
to replicate the behavior here. And then Howard can debug what's
actually
going on.
Thanks,
John
On Mon, May 21, 2018 at 4:09 PM, Jacob Carley - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
>
> Hi John,
>
> I'm running with 7.1 beta3 build on WCOSS, perhaps that's the
culprit.
> When I run it with -index here is what it prints to the screen (and
in the
> log file) in what I think is an infinite loop:
>
> DEBUG 1: User Config File: PB2NCConfig_gust
> DEBUG 1: Pre-processing BUFR File for variable names from
> rtma2p5_ru.t1545z.prepbufr
> DEBUG 1: process_pbfile_metadata() -> the number of records: 114311
> DEBUG 1:
> DEBUG 1: Header variables:
> DEBUG 1: SAID: SATELLITE IDENTIFIER (SATELLITE REPORTS ONLY)
> DEBUG 1: XOB: LONGITUDE
> DEBUG 1: YOB: LATITUDE
> DEBUG 1: YEAR:
> DEBUG 1: MNTH:
> DEBUG 1: DAYS:
> DEBUG 1: HOUR:
> DEBUG 1: MINU:
> DEBUG 1: SECO:
> DEBUG 1:
> DEBUG 1: Observation variables:
> DEBUG 1: EEQF: PERCENT CONFIDENCE BASED ON NESDIS EXPECTED
ERROR
> types:
> SATWND
> DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL INDX
W/O
> FCST types: SATWND
> DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL INDX
> W/ FCST types:
> SATWND
> DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS RECURSIVE
FILTER
> FCN types: SATWND
> DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT ASSIMILATED)
> types:
> SATWND
> DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT PROGRAM
CODE
> types:
> SATWND ADPSFC SFCSHP MSONET ASCATW
> DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT ASSIMILATED)
> types:
> SATWND ADPSFC SFCSHP MSONET
> DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> types:
> SATWND ADPSFC SFCSHP MSONET
> DEBUG 1: ZQM: HEIGHT (QUALITY) MARKER
> types:
> SATWND ADPSFC SFCSHP MSONET
> DEBUG 1: ZOB: HEIGHT OBSERVATION
> types:
> SATWND ADPSFC SFCSHP MSONET
> DEBUG 1: CAT: PREPBUFR DATA LEVEL CATEGORY
> types:
> SATWND ADPSFC SFCSHP MSONET ASCATW
> *DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL
INDX W/O
> FCST types: SATWND*
> DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL INDX
> W/ FCST types:
> SATWND
> DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS RECURSIVE
FILTER
> FCN types: SATWND
> DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT ASSIMILATED)
> types:
> SATWND
> DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT PROGRAM
CODE
> types:
> SATWND ADPSFC SFCSHP MSONET ASCATW
> DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT ASSIMILATED)
> types:
> SATWND ADPSFC SFCSHP MSONET
> DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> types:
> SATWND ADPSFC SFCSHP MSONET
>
> Which is pretty much just printed infinitely. I bolded the line
above that
> shows when it begins to repeat
>
> Here's the command I use:
>
> pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> PB2NCConfig_gust -index -log index.log
>
>
> I just switched over to 7.0 on WCOSS (non-beta) and I get the same
behavior
> with that as well.
>
> Does it matter what's in the config file when running with the '-
index'
> option aside from some basic stuff like the time window?
>
> Thanks!
> Jacob
>
> On Mon, May 21, 2018 at 5:38 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Jacob,
> >
> > There's something odd going on here. You shouldn't see a 1G log
file.
> > When I ran that command was 11k. And indeed, I do see MXGS
listed:
> > DEBUG 1: MXGS: MAXIMUM WIND SPEED
> > (GUSTS) types: ADPSFC SFCSHP
MSONET
> > I've attached the log file to this email for reference.
> >
> > Here's the command I ran to generate this index file:
> > /usr/local/met-7.0/bin/pb2nc \
> > rtma2p5_ru.t1215z.prepbufr \
> > rtma2p5_ru.t1215z.prepbufr.nc \
> > PB2NCConfig_CandV \
> > -index -log index.log
> >
> > When you run with the "-index" option, no NetCDF output is
actually
> > generated. It just search through the input file, figures out
what
> > observations are present, and then prints them to the screen. And
also a
> > log file since we used "-log index.log".
> >
> > Are you seeing behavior that differs from this?
> >
> > Thanks,
> > John
> >
> >
> > On Mon, May 21, 2018 at 2:52 PM, Jacob Carley - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > >
> > > Hi John,
> > >
> > > A quick follow up to this ticket - I have an additional variable
I'd
> like
> > > to add and wanted to go the route of running pb2nc with '-index'
as you
> > did
> > > in this example. However when I do so, I get far too much
output that
> I
> > > can sensibly parse (log files well over 1G). Is this typical?
I'm
> > looking
> > > to see if I can pull out wind gust (the prepbufr mnemonic is
'MXGS').
> I
> > > figure I must not have something set quite right in my pb2nc
config
> file.
> > >
> > > Thanks,
> > > Jacob
> > >
> > >
> > >
> > > On Tue, May 1, 2018 at 5:51 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu
> > > > wrote:
> > >
> > > > Jacob,
> > > >
> > > > Great, I'll go ahead and resolve this ticket. Just let us
know if
> > > anything
> > > > else comes up.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA Affiliate
via RT
> <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Thanks so much! This is hugely helpful.
> > > > >
> > > > > ADPSFC encompasses METAR observations, so that's correct.
> > > > >
> > > > > This should be more than enough to get us rolling.
> > > > >
> > > > > Thanks again!
> > > > > Jacob
> > > > >
> > > > > On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu
> > > > > > wrote:
> > > > >
> > > > > > Jacob and Runhua,
> > > > > >
> > > > > > Great, thanks. Here's the steps I took with this data:
> > > > > >
> > > > > > (1) Run pb2nc using the "-index" option to the list
observation
> > > > variable
> > > > > > names for each message type... takes a few minutes to run:
> > > > > >
> > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > PB2NCConfig_CandV \
> > > > > > -index -log index.log
> > > > > >
> > > > > > I've attached the index.log file but here's a selection of
the
> > output
> > > > > from
> > > > > > that run:
> > > > > > DEBUG 1: HOVI: HORIZONTAL
> > > > > > VISIBILITY
> > > > > > types: ADPSFC SFCSHP
> > > > > > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF HOCB -
HGT OF
> > > CLOUD
> > > > > > BASE) types: ADPSFC
> > > > > >
> > > > > > You mentioned using METAR observations, but in that
PREPBUFR
> file,
> > I
> > > > only
> > > > > > see CEILING and visibility (HOVI) listed for the surface
message
> > > types.
> > > > > >
> > > > > > (2) Configure PB2NC to retain the observation for C and V.
See
> > > > attached
> > > > > > file PB2NC_CandV.
> > > > > >
> > > > > > // Only keep surface message types
> > > > > > message_type = [ "ONLYSF" ];
> > > > > > // Only keep ceiling and visibility obs
> > > > > > obs_bufr_var = [ "CEILING", "HOVI" ];
> > > > > > // Out of convenience to get OBS to match GRIB2 variable
names,
> > > rename
> > > > > > CEILING as CEIL and HOVI as VIS
> > > > > > obs_bufr_map = [
> > > > > > { key = "CEILING"; val = "CEIL"; },
> > > > > > { key = "HOVI"; val = "VIS"; }
> > > > > > ];
> > > > > >
> > > > > > (3) Rerun pb2nc to actually process these obs:
> > > > > >
> > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > PB2NCConfig_CandV
> > > > > >
> > > > > > (4) Configure and run point_stat to do the vx. See
attached
> > > point_stat
> > > > > > config file. Since LIFR, IFR, MVFR, and VFR are defined
in
> > > > feet/miles, I
> > > > > > used the convert() function to convert meters to those
units
> prior
> > to
> > > > > > verifying. You certainly don't need to do that... you
could just
> > > > specify
> > > > > > the thresholds in meters instead:
> > > > > >
> > > > > > /usr/local/met-7.0/bin/point_stat \
> > > > > > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > PointStatConfig_CandV \
> > > > > > -outdir out -v 3 -log run_ps.log
> > > > > >
> > > > > > This yields 3214 matched pairs for ceiling and 3622 for
> visibility.
> > > > > >
> > > > > > (5) I configured point_stat to dump the matched pairs
(MPR)
> output
> > > line
> > > > > > type. So I called the plot_mpr.R Rscript included with
MET to
> see
> > > some
> > > > > > plots of the data:
> > > > > >
> > > > > > Rscript /usr/local/met-7.0/share/met/Rscripts/plot_mpr.R
> > > > > > out/point_stat_000000L_20180501_121500V_mpr.txt
> > > > > >
> > > > > > The resulting PDF file is attached.
> > > > > >
> > > > > > Looking at the scatter plot, there's a curious band of
> observation
> > > > values
> > > > > > around 65000 ft for ceiling... and around 10 miles for
> visibility.
> > > You
> > > > > > should probably spend some time to understand this data
and
> decide
> > > how
> > > > > you
> > > > > > want to process it for verification.
> > > > > >
> > > > > > Looking at the CTS output line type, I see that the
equitable
> > threat
> > > > > scores
> > > > > > (i.e. GSS column) range between .14895 and .54148.
> > > > > >
> > > > > > Hopefully that helps get you going.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley - NOAA
Affiliate via
> > RT
> > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > [cc'd Runhua Yang, as she's working on this project as
well].
> > > > > > >
> > > > > > > Wonderful - that would incredibly helpful.
> > > > > > >
> > > > > > > I've put a sample grib2 file and an associated analysis
file
> here
> > > on
> > > > > gyre
> > > > > > > (wcoss phase2):
> > > > > > >
> > > > > > > /meso2/noscrub/Jacob.Carley/methelp
> > > > > > >
> > > > > > > It's from a system that runs every 15 minutes, which
hopefully
> > > isn't
> > > > an
> > > > > > > issue for MET or METViewer (this particular file is
valid at
> > 1215Z
> > > > from
> > > > > > > this morning).
> > > > > > >
> > > > > > > Thanks again!
> > > > > > > Jacob
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Jacob,
> > > > > > > >
> > > > > > > > Yes, I certainly think that should be possible using
met-7.0.
> > > If I
> > > > > > > > remember correctly, Perry and I worked through this
not too
> > long
> > > > ago.
> > > > > > > >
> > > > > > > > If you'd like, I could work up an example of calling
pb2nc
> and
> > > > > > point_stat
> > > > > > > > to demonstrate it. But I'd need you to point me to a
sample
> > > model
> > > > > > output
> > > > > > > > file and PREPBUFR file that you'd like to use.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley - NOAA
> Affiliate
> > > via
> > > > > RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Tue May 01 12:33:24 2018: Request 84952 was acted
upon.
> > > > > > > > > Transaction: Ticket created by jacob.carley at noaa.gov
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: ceiling and visibility verification in
MET
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: jacob.carley at noaa.gov
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > > Ticket/Display.html?id=84952
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi MET Help,
> > > > > > > > >
> > > > > > > > > Several of us at EMC are interested in doing some
> > verification
> > > of
> > > > > > > ceiling
> > > > > > > > > and visibility using corresponding METAR
observations in
> > > prepbufr
> > > > > > > files.
> > > > > > > > > Speaking specifically, we would like to perform
categorical
> > > > > > statistics
> > > > > > > > > based upon a variety of aviation thresholds (e.g.
IFR,
> LIFR,
> > > > etc.).
> > > > > > Is
> > > > > > > > this
> > > > > > > > > possible in MET v7?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Jacob
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Tue May 22 07:52:36 2018
Hi John,
Sure thing. I"m able to reproduce the issue on Theia as well. Though
the
repeating prints are a bit different (it gets hung up on QOB, TOB, and
UOB).
On Theia, see the following directory:
/scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET
To run I did:
module load met/7.0
pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
PB2NCConfig_RTMA2P5_RU -index -log index.log
I left my index.log file in that directory from a most recent test in
case
that is helpful (it's ~600M).
Thanks!
Jacob
On Mon, May 21, 2018 at 6:40 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Jacob,
>
> No, the contents of the config file don't matter, at least to my
> knowledge. I'm not able to replicate this behavior on the sample
file that
> I have.
>
> The big question is whether you've found a data file that puts pb2nc
into a
> bad loop... or whether we're getting different behavior on different
> platforms. Hopefully it's the first problem since that'd be a lot
easier
> to fix.
>
> Where can I get a hold of the input file you're using:
> rtma2p5_ru.t1545z.prepbufr
>
> If you put it up on theia or an ftp site somewhere, I can grab it,
and try
> to replicate the behavior here. And then Howard can debug what's
actually
> going on.
>
> Thanks,
> John
>
> On Mon, May 21, 2018 at 4:09 PM, Jacob Carley - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> >
> > Hi John,
> >
> > I'm running with 7.1 beta3 build on WCOSS, perhaps that's the
culprit.
> > When I run it with -index here is what it prints to the screen
(and in
> the
> > log file) in what I think is an infinite loop:
> >
> > DEBUG 1: User Config File: PB2NCConfig_gust
> > DEBUG 1: Pre-processing BUFR File for variable names from
> > rtma2p5_ru.t1545z.prepbufr
> > DEBUG 1: process_pbfile_metadata() -> the number of records:
114311
> > DEBUG 1:
> > DEBUG 1: Header variables:
> > DEBUG 1: SAID: SATELLITE IDENTIFIER (SATELLITE REPORTS
ONLY)
> > DEBUG 1: XOB: LONGITUDE
> > DEBUG 1: YOB: LATITUDE
> > DEBUG 1: YEAR:
> > DEBUG 1: MNTH:
> > DEBUG 1: DAYS:
> > DEBUG 1: HOUR:
> > DEBUG 1: MINU:
> > DEBUG 1: SECO:
> > DEBUG 1:
> > DEBUG 1: Observation variables:
> > DEBUG 1: EEQF: PERCENT CONFIDENCE BASED ON NESDIS EXPECTED
ERROR
> > types:
> > SATWND
> > DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL
INDX W/O
> > FCST types: SATWND
> > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL
INDX
> > W/ FCST types:
> > SATWND
> > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS RECURSIVE
FILTER
> > FCN types: SATWND
> > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
ASSIMILATED)
> > types:
> > SATWND
> > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT PROGRAM
CODE
> > types:
> > SATWND ADPSFC SFCSHP MSONET ASCATW
> > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT ASSIMILATED)
> > types:
> > SATWND ADPSFC SFCSHP MSONET
> > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> > types:
> > SATWND ADPSFC SFCSHP MSONET
> > DEBUG 1: ZQM: HEIGHT (QUALITY) MARKER
> > types:
> > SATWND ADPSFC SFCSHP MSONET
> > DEBUG 1: ZOB: HEIGHT OBSERVATION
> > types:
> > SATWND ADPSFC SFCSHP MSONET
> > DEBUG 1: CAT: PREPBUFR DATA LEVEL CATEGORY
> > types:
> > SATWND ADPSFC SFCSHP MSONET ASCATW
> > *DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL
INDX W/O
> > FCST types: SATWND*
> > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL
INDX
> > W/ FCST types:
> > SATWND
> > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS RECURSIVE
FILTER
> > FCN types: SATWND
> > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
ASSIMILATED)
> > types:
> > SATWND
> > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT PROGRAM
CODE
> > types:
> > SATWND ADPSFC SFCSHP MSONET ASCATW
> > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT ASSIMILATED)
> > types:
> > SATWND ADPSFC SFCSHP MSONET
> > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> > types:
> > SATWND ADPSFC SFCSHP MSONET
> >
> > Which is pretty much just printed infinitely. I bolded the line
above
> that
> > shows when it begins to repeat
> >
> > Here's the command I use:
> >
> > pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> > PB2NCConfig_gust -index -log index.log
> >
> >
> > I just switched over to 7.0 on WCOSS (non-beta) and I get the same
> behavior
> > with that as well.
> >
> > Does it matter what's in the config file when running with the '-
index'
> > option aside from some basic stuff like the time window?
> >
> > Thanks!
> > Jacob
> >
> > On Mon, May 21, 2018 at 5:38 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Jacob,
> > >
> > > There's something odd going on here. You shouldn't see a 1G log
file.
> > > When I ran that command was 11k. And indeed, I do see MXGS
listed:
> > > DEBUG 1: MXGS: MAXIMUM WIND SPEED
> > > (GUSTS) types: ADPSFC SFCSHP
MSONET
> > > I've attached the log file to this email for reference.
> > >
> > > Here's the command I ran to generate this index file:
> > > /usr/local/met-7.0/bin/pb2nc \
> > > rtma2p5_ru.t1215z.prepbufr \
> > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > PB2NCConfig_CandV \
> > > -index -log index.log
> > >
> > > When you run with the "-index" option, no NetCDF output is
actually
> > > generated. It just search through the input file, figures out
what
> > > observations are present, and then prints them to the screen.
And
> also a
> > > log file since we used "-log index.log".
> > >
> > > Are you seeing behavior that differs from this?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Mon, May 21, 2018 at 2:52 PM, Jacob Carley - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952
>
> > > >
> > > > Hi John,
> > > >
> > > > A quick follow up to this ticket - I have an additional
variable I'd
> > like
> > > > to add and wanted to go the route of running pb2nc with '-
index' as
> you
> > > did
> > > > in this example. However when I do so, I get far too much
output
> that
> > I
> > > > can sensibly parse (log files well over 1G). Is this typical?
I'm
> > > looking
> > > > to see if I can pull out wind gust (the prepbufr mnemonic is
'MXGS').
> > I
> > > > figure I must not have something set quite right in my pb2nc
config
> > file.
> > > >
> > > > Thanks,
> > > > Jacob
> > > >
> > > >
> > > >
> > > > On Tue, May 1, 2018 at 5:51 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu
> > > > > wrote:
> > > >
> > > > > Jacob,
> > > > >
> > > > > Great, I'll go ahead and resolve this ticket. Just let us
know if
> > > > anything
> > > > > else comes up.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA
Affiliate via
> RT
> > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Thanks so much! This is hugely helpful.
> > > > > >
> > > > > > ADPSFC encompasses METAR observations, so that's correct.
> > > > > >
> > > > > > This should be more than enough to get us rolling.
> > > > > >
> > > > > > Thanks again!
> > > > > > Jacob
> > > > > >
> > > > > > On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > wrote:
> > > > > >
> > > > > > > Jacob and Runhua,
> > > > > > >
> > > > > > > Great, thanks. Here's the steps I took with this data:
> > > > > > >
> > > > > > > (1) Run pb2nc using the "-index" option to the list
observation
> > > > > variable
> > > > > > > names for each message type... takes a few minutes to
run:
> > > > > > >
> > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > PB2NCConfig_CandV \
> > > > > > > -index -log index.log
> > > > > > >
> > > > > > > I've attached the index.log file but here's a selection
of the
> > > output
> > > > > > from
> > > > > > > that run:
> > > > > > > DEBUG 1: HOVI: HORIZONTAL
> > > > > > > VISIBILITY
> > > > > > > types: ADPSFC SFCSHP
> > > > > > > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF HOCB
- HGT
> OF
> > > > CLOUD
> > > > > > > BASE) types: ADPSFC
> > > > > > >
> > > > > > > You mentioned using METAR observations, but in that
PREPBUFR
> > file,
> > > I
> > > > > only
> > > > > > > see CEILING and visibility (HOVI) listed for the surface
> message
> > > > types.
> > > > > > >
> > > > > > > (2) Configure PB2NC to retain the observation for C and
V. See
> > > > > attached
> > > > > > > file PB2NC_CandV.
> > > > > > >
> > > > > > > // Only keep surface message types
> > > > > > > message_type = [ "ONLYSF" ];
> > > > > > > // Only keep ceiling and visibility obs
> > > > > > > obs_bufr_var = [ "CEILING", "HOVI" ];
> > > > > > > // Out of convenience to get OBS to match GRIB2 variable
names,
> > > > rename
> > > > > > > CEILING as CEIL and HOVI as VIS
> > > > > > > obs_bufr_map = [
> > > > > > > { key = "CEILING"; val = "CEIL"; },
> > > > > > > { key = "HOVI"; val = "VIS"; }
> > > > > > > ];
> > > > > > >
> > > > > > > (3) Rerun pb2nc to actually process these obs:
> > > > > > >
> > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > PB2NCConfig_CandV
> > > > > > >
> > > > > > > (4) Configure and run point_stat to do the vx. See
attached
> > > > point_stat
> > > > > > > config file. Since LIFR, IFR, MVFR, and VFR are defined
in
> > > > > feet/miles, I
> > > > > > > used the convert() function to convert meters to those
units
> > prior
> > > to
> > > > > > > verifying. You certainly don't need to do that... you
could
> just
> > > > > specify
> > > > > > > the thresholds in meters instead:
> > > > > > >
> > > > > > > /usr/local/met-7.0/bin/point_stat \
> > > > > > > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > PointStatConfig_CandV \
> > > > > > > -outdir out -v 3 -log run_ps.log
> > > > > > >
> > > > > > > This yields 3214 matched pairs for ceiling and 3622 for
> > visibility.
> > > > > > >
> > > > > > > (5) I configured point_stat to dump the matched pairs
(MPR)
> > output
> > > > line
> > > > > > > type. So I called the plot_mpr.R Rscript included with
MET to
> > see
> > > > some
> > > > > > > plots of the data:
> > > > > > >
> > > > > > > Rscript /usr/local/met-7.0/share/met/Rscripts/plot_mpr.R
> > > > > > > out/point_stat_000000L_20180501_121500V_mpr.txt
> > > > > > >
> > > > > > > The resulting PDF file is attached.
> > > > > > >
> > > > > > > Looking at the scatter plot, there's a curious band of
> > observation
> > > > > values
> > > > > > > around 65000 ft for ceiling... and around 10 miles for
> > visibility.
> > > > You
> > > > > > > should probably spend some time to understand this data
and
> > decide
> > > > how
> > > > > > you
> > > > > > > want to process it for verification.
> > > > > > >
> > > > > > > Looking at the CTS output line type, I see that the
equitable
> > > threat
> > > > > > scores
> > > > > > > (i.e. GSS column) range between .14895 and .54148.
> > > > > > >
> > > > > > > Hopefully that helps get you going.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=84952
> > >
> > > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > [cc'd Runhua Yang, as she's working on this project as
well].
> > > > > > > >
> > > > > > > > Wonderful - that would incredibly helpful.
> > > > > > > >
> > > > > > > > I've put a sample grib2 file and an associated
analysis file
> > here
> > > > on
> > > > > > gyre
> > > > > > > > (wcoss phase2):
> > > > > > > >
> > > > > > > > /meso2/noscrub/Jacob.Carley/methelp
> > > > > > > >
> > > > > > > > It's from a system that runs every 15 minutes, which
> hopefully
> > > > isn't
> > > > > an
> > > > > > > > issue for MET or METViewer (this particular file is
valid at
> > > 1215Z
> > > > > from
> > > > > > > > this morning).
> > > > > > > >
> > > > > > > > Thanks again!
> > > > > > > > Jacob
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway via
RT <
> > > > > > > > met_help at ucar.edu
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Jacob,
> > > > > > > > >
> > > > > > > > > Yes, I certainly think that should be possible using
> met-7.0.
> > > > If I
> > > > > > > > > remember correctly, Perry and I worked through this
not too
> > > long
> > > > > ago.
> > > > > > > > >
> > > > > > > > > If you'd like, I could work up an example of calling
pb2nc
> > and
> > > > > > > point_stat
> > > > > > > > > to demonstrate it. But I'd need you to point me to
a
> sample
> > > > model
> > > > > > > output
> > > > > > > > > file and PREPBUFR file that you'd like to use.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley - NOAA
> > Affiliate
> > > > via
> > > > > > RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Tue May 01 12:33:24 2018: Request 84952 was acted
upon.
> > > > > > > > > > Transaction: Ticket created by
jacob.carley at noaa.gov
> > > > > > > > > > Queue: met_help
> > > > > > > > > > Subject: ceiling and visibility verification
in MET
> > > > > > > > > > Owner: Nobody
> > > > > > > > > > Requestors: jacob.carley at noaa.gov
> > > > > > > > > > Status: new
> > > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > Ticket/Display.html?id=84952
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi MET Help,
> > > > > > > > > >
> > > > > > > > > > Several of us at EMC are interested in doing some
> > > verification
> > > > of
> > > > > > > > ceiling
> > > > > > > > > > and visibility using corresponding METAR
observations in
> > > > prepbufr
> > > > > > > > files.
> > > > > > > > > > Speaking specifically, we would like to perform
> categorical
> > > > > > > statistics
> > > > > > > > > > based upon a variety of aviation thresholds (e.g.
IFR,
> > LIFR,
> > > > > etc.).
> > > > > > > Is
> > > > > > > > > this
> > > > > > > > > > possible in MET v7?
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > Jacob
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: John Halley Gotway
Time: Tue May 22 09:26:05 2018
Jacob,
Could you please modify the permissions on the prepbufr file? It's
not
group-readable:
[John.H.Gotway at tfe05 MET]$ ls -la
/scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET/rtma2p5_ru.t1545z.prepbufr
-rw-r----- 1 Jacob.Carley meso 9700664 Apr 19 21:22
/scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET/rtma2p5_ru.t1545z.prepbufr
Thanks,
John
On Tue, May 22, 2018 at 7:52 AM, Jacob Carley - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
>
> Hi John,
>
> Sure thing. I"m able to reproduce the issue on Theia as well.
Though the
> repeating prints are a bit different (it gets hung up on QOB, TOB,
and
> UOB).
>
> On Theia, see the following directory:
>
> /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET
>
> To run I did:
>
> module load met/7.0
> pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> PB2NCConfig_RTMA2P5_RU -index -log index.log
>
> I left my index.log file in that directory from a most recent test
in case
> that is helpful (it's ~600M).
>
> Thanks!
> Jacob
>
>
>
> On Mon, May 21, 2018 at 6:40 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Jacob,
> >
> > No, the contents of the config file don't matter, at least to my
> > knowledge. I'm not able to replicate this behavior on the sample
file
> that
> > I have.
> >
> > The big question is whether you've found a data file that puts
pb2nc
> into a
> > bad loop... or whether we're getting different behavior on
different
> > platforms. Hopefully it's the first problem since that'd be a lot
easier
> > to fix.
> >
> > Where can I get a hold of the input file you're using:
> > rtma2p5_ru.t1545z.prepbufr
> >
> > If you put it up on theia or an ftp site somewhere, I can grab it,
and
> try
> > to replicate the behavior here. And then Howard can debug what's
> actually
> > going on.
> >
> > Thanks,
> > John
> >
> > On Mon, May 21, 2018 at 4:09 PM, Jacob Carley - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > >
> > > Hi John,
> > >
> > > I'm running with 7.1 beta3 build on WCOSS, perhaps that's the
culprit.
> > > When I run it with -index here is what it prints to the screen
(and in
> > the
> > > log file) in what I think is an infinite loop:
> > >
> > > DEBUG 1: User Config File: PB2NCConfig_gust
> > > DEBUG 1: Pre-processing BUFR File for variable names from
> > > rtma2p5_ru.t1545z.prepbufr
> > > DEBUG 1: process_pbfile_metadata() -> the number of records:
114311
> > > DEBUG 1:
> > > DEBUG 1: Header variables:
> > > DEBUG 1: SAID: SATELLITE IDENTIFIER (SATELLITE REPORTS
ONLY)
> > > DEBUG 1: XOB: LONGITUDE
> > > DEBUG 1: YOB: LATITUDE
> > > DEBUG 1: YEAR:
> > > DEBUG 1: MNTH:
> > > DEBUG 1: DAYS:
> > > DEBUG 1: HOUR:
> > > DEBUG 1: MINU:
> > > DEBUG 1: SECO:
> > > DEBUG 1:
> > > DEBUG 1: Observation variables:
> > > DEBUG 1: EEQF: PERCENT CONFIDENCE BASED ON NESDIS
EXPECTED ERROR
> > > types:
> > > SATWND
> > > DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL
INDX
> W/O
> > > FCST types: SATWND
> > > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL
INDX
> > > W/ FCST types:
> > > SATWND
> > > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS
RECURSIVE
> FILTER
> > > FCN types: SATWND
> > > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
ASSIMILATED)
> > > types:
> > > SATWND
> > > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT
PROGRAM CODE
> > > types:
> > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT
ASSIMILATED)
> > > types:
> > > SATWND ADPSFC SFCSHP MSONET
> > > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> > > types:
> > > SATWND ADPSFC SFCSHP MSONET
> > > DEBUG 1: ZQM: HEIGHT (QUALITY) MARKER
> > > types:
> > > SATWND ADPSFC SFCSHP MSONET
> > > DEBUG 1: ZOB: HEIGHT OBSERVATION
> > > types:
> > > SATWND ADPSFC SFCSHP MSONET
> > > DEBUG 1: CAT: PREPBUFR DATA LEVEL CATEGORY
> > > types:
> > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > *DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL
INDX
> W/O
> > > FCST types: SATWND*
> > > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT QUAL
INDX
> > > W/ FCST types:
> > > SATWND
> > > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS
RECURSIVE
> FILTER
> > > FCN types: SATWND
> > > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
ASSIMILATED)
> > > types:
> > > SATWND
> > > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT
PROGRAM CODE
> > > types:
> > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT
ASSIMILATED)
> > > types:
> > > SATWND ADPSFC SFCSHP MSONET
> > > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> > > types:
> > > SATWND ADPSFC SFCSHP MSONET
> > >
> > > Which is pretty much just printed infinitely. I bolded the line
above
> > that
> > > shows when it begins to repeat
> > >
> > > Here's the command I use:
> > >
> > > pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> > > PB2NCConfig_gust -index -log index.log
> > >
> > >
> > > I just switched over to 7.0 on WCOSS (non-beta) and I get the
same
> > behavior
> > > with that as well.
> > >
> > > Does it matter what's in the config file when running with the
'-index'
> > > option aside from some basic stuff like the time window?
> > >
> > > Thanks!
> > > Jacob
> > >
> > > On Mon, May 21, 2018 at 5:38 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Jacob,
> > > >
> > > > There's something odd going on here. You shouldn't see a 1G
log
> file.
> > > > When I ran that command was 11k. And indeed, I do see MXGS
listed:
> > > > DEBUG 1: MXGS: MAXIMUM WIND SPEED
> > > > (GUSTS) types: ADPSFC SFCSHP
> MSONET
> > > > I've attached the log file to this email for reference.
> > > >
> > > > Here's the command I ran to generate this index file:
> > > > /usr/local/met-7.0/bin/pb2nc \
> > > > rtma2p5_ru.t1215z.prepbufr \
> > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > PB2NCConfig_CandV \
> > > > -index -log index.log
> > > >
> > > > When you run with the "-index" option, no NetCDF output is
actually
> > > > generated. It just search through the input file, figures out
what
> > > > observations are present, and then prints them to the screen.
And
> > also a
> > > > log file since we used "-log index.log".
> > > >
> > > > Are you seeing behavior that differs from this?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > On Mon, May 21, 2018 at 2:52 PM, Jacob Carley - NOAA Affiliate
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > A quick follow up to this ticket - I have an additional
variable
> I'd
> > > like
> > > > > to add and wanted to go the route of running pb2nc with '-
index' as
> > you
> > > > did
> > > > > in this example. However when I do so, I get far too much
output
> > that
> > > I
> > > > > can sensibly parse (log files well over 1G). Is this
typical? I'm
> > > > looking
> > > > > to see if I can pull out wind gust (the prepbufr mnemonic is
> 'MXGS').
> > > I
> > > > > figure I must not have something set quite right in my pb2nc
config
> > > file.
> > > > >
> > > > > Thanks,
> > > > > Jacob
> > > > >
> > > > >
> > > > >
> > > > > On Tue, May 1, 2018 at 5:51 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu
> > > > > > wrote:
> > > > >
> > > > > > Jacob,
> > > > > >
> > > > > > Great, I'll go ahead and resolve this ticket. Just let us
know
> if
> > > > > anything
> > > > > > else comes up.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA
Affiliate via
> > RT
> > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Thanks so much! This is hugely helpful.
> > > > > > >
> > > > > > > ADPSFC encompasses METAR observations, so that's
correct.
> > > > > > >
> > > > > > > This should be more than enough to get us rolling.
> > > > > > >
> > > > > > > Thanks again!
> > > > > > > Jacob
> > > > > > >
> > > > > > > On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Jacob and Runhua,
> > > > > > > >
> > > > > > > > Great, thanks. Here's the steps I took with this data:
> > > > > > > >
> > > > > > > > (1) Run pb2nc using the "-index" option to the list
> observation
> > > > > > variable
> > > > > > > > names for each message type... takes a few minutes to
run:
> > > > > > > >
> > > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > PB2NCConfig_CandV \
> > > > > > > > -index -log index.log
> > > > > > > >
> > > > > > > > I've attached the index.log file but here's a
selection of
> the
> > > > output
> > > > > > > from
> > > > > > > > that run:
> > > > > > > > DEBUG 1: HOVI: HORIZONTAL
> > > > > > > > VISIBILITY
> > > > > > > > types: ADPSFC SFCSHP
> > > > > > > > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF
HOCB - HGT
> > OF
> > > > > CLOUD
> > > > > > > > BASE) types: ADPSFC
> > > > > > > >
> > > > > > > > You mentioned using METAR observations, but in that
PREPBUFR
> > > file,
> > > > I
> > > > > > only
> > > > > > > > see CEILING and visibility (HOVI) listed for the
surface
> > message
> > > > > types.
> > > > > > > >
> > > > > > > > (2) Configure PB2NC to retain the observation for C
and V.
> See
> > > > > > attached
> > > > > > > > file PB2NC_CandV.
> > > > > > > >
> > > > > > > > // Only keep surface message types
> > > > > > > > message_type = [ "ONLYSF" ];
> > > > > > > > // Only keep ceiling and visibility obs
> > > > > > > > obs_bufr_var = [ "CEILING", "HOVI" ];
> > > > > > > > // Out of convenience to get OBS to match GRIB2
variable
> names,
> > > > > rename
> > > > > > > > CEILING as CEIL and HOVI as VIS
> > > > > > > > obs_bufr_map = [
> > > > > > > > { key = "CEILING"; val = "CEIL"; },
> > > > > > > > { key = "HOVI"; val = "VIS"; }
> > > > > > > > ];
> > > > > > > >
> > > > > > > > (3) Rerun pb2nc to actually process these obs:
> > > > > > > >
> > > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > PB2NCConfig_CandV
> > > > > > > >
> > > > > > > > (4) Configure and run point_stat to do the vx. See
attached
> > > > > point_stat
> > > > > > > > config file. Since LIFR, IFR, MVFR, and VFR are
defined in
> > > > > > feet/miles, I
> > > > > > > > used the convert() function to convert meters to those
units
> > > prior
> > > > to
> > > > > > > > verifying. You certainly don't need to do that... you
could
> > just
> > > > > > specify
> > > > > > > > the thresholds in meters instead:
> > > > > > > >
> > > > > > > > /usr/local/met-7.0/bin/point_stat \
> > > > > > > > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > PointStatConfig_CandV \
> > > > > > > > -outdir out -v 3 -log run_ps.log
> > > > > > > >
> > > > > > > > This yields 3214 matched pairs for ceiling and 3622
for
> > > visibility.
> > > > > > > >
> > > > > > > > (5) I configured point_stat to dump the matched pairs
(MPR)
> > > output
> > > > > line
> > > > > > > > type. So I called the plot_mpr.R Rscript included
with MET
> to
> > > see
> > > > > some
> > > > > > > > plots of the data:
> > > > > > > >
> > > > > > > > Rscript /usr/local/met-
7.0/share/met/Rscripts/plot_mpr.R
> > > > > > > > out/point_stat_000000L_20180501_121500V_mpr.txt
> > > > > > > >
> > > > > > > > The resulting PDF file is attached.
> > > > > > > >
> > > > > > > > Looking at the scatter plot, there's a curious band of
> > > observation
> > > > > > values
> > > > > > > > around 65000 ft for ceiling... and around 10 miles for
> > > visibility.
> > > > > You
> > > > > > > > should probably spend some time to understand this
data and
> > > decide
> > > > > how
> > > > > > > you
> > > > > > > > want to process it for verification.
> > > > > > > >
> > > > > > > > Looking at the CTS output line type, I see that the
equitable
> > > > threat
> > > > > > > scores
> > > > > > > > (i.e. GSS column) range between .14895 and .54148.
> > > > > > > >
> > > > > > > > Hopefully that helps get you going.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=84952
> > > >
> > > > > > > > >
> > > > > > > > > Hi John,
> > > > > > > > >
> > > > > > > > > [cc'd Runhua Yang, as she's working on this project
as
> well].
> > > > > > > > >
> > > > > > > > > Wonderful - that would incredibly helpful.
> > > > > > > > >
> > > > > > > > > I've put a sample grib2 file and an associated
analysis
> file
> > > here
> > > > > on
> > > > > > > gyre
> > > > > > > > > (wcoss phase2):
> > > > > > > > >
> > > > > > > > > /meso2/noscrub/Jacob.Carley/methelp
> > > > > > > > >
> > > > > > > > > It's from a system that runs every 15 minutes, which
> > hopefully
> > > > > isn't
> > > > > > an
> > > > > > > > > issue for MET or METViewer (this particular file is
valid
> at
> > > > 1215Z
> > > > > > from
> > > > > > > > > this morning).
> > > > > > > > >
> > > > > > > > > Thanks again!
> > > > > > > > > Jacob
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway
via RT <
> > > > > > > > > met_help at ucar.edu
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Jacob,
> > > > > > > > > >
> > > > > > > > > > Yes, I certainly think that should be possible
using
> > met-7.0.
> > > > > If I
> > > > > > > > > > remember correctly, Perry and I worked through
this not
> too
> > > > long
> > > > > > ago.
> > > > > > > > > >
> > > > > > > > > > If you'd like, I could work up an example of
calling
> pb2nc
> > > and
> > > > > > > > point_stat
> > > > > > > > > > to demonstrate it. But I'd need you to point me
to a
> > sample
> > > > > model
> > > > > > > > output
> > > > > > > > > > file and PREPBUFR file that you'd like to use.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley -
NOAA
> > > Affiliate
> > > > > via
> > > > > > > RT <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Tue May 01 12:33:24 2018: Request 84952 was
acted upon.
> > > > > > > > > > > Transaction: Ticket created by
jacob.carley at noaa.gov
> > > > > > > > > > > Queue: met_help
> > > > > > > > > > > Subject: ceiling and visibility
verification in
> MET
> > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > Requestors: jacob.carley at noaa.gov
> > > > > > > > > > > Status: new
> > > > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > > Ticket/Display.html?id=84952
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi MET Help,
> > > > > > > > > > >
> > > > > > > > > > > Several of us at EMC are interested in doing
some
> > > > verification
> > > > > of
> > > > > > > > > ceiling
> > > > > > > > > > > and visibility using corresponding METAR
observations
> in
> > > > > prepbufr
> > > > > > > > > files.
> > > > > > > > > > > Speaking specifically, we would like to perform
> > categorical
> > > > > > > > statistics
> > > > > > > > > > > based upon a variety of aviation thresholds
(e.g. IFR,
> > > LIFR,
> > > > > > etc.).
> > > > > > > > Is
> > > > > > > > > > this
> > > > > > > > > > > possible in MET v7?
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > > Jacob
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Tue May 22 09:46:17 2018
Hi John,
Sorry about that - all should be good now.
-Jacob
On Tue, May 22, 2018 at 11:26 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Jacob,
>
> Could you please modify the permissions on the prepbufr file? It's
not
> group-readable:
>
> [John.H.Gotway at tfe05 MET]$ ls -la
> /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET/rtma2p5_
> ru.t1545z.prepbufr
> -rw-r----- 1 Jacob.Carley meso 9700664 Apr 19 21:22
> /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET/rtma2p5_
> ru.t1545z.prepbufr
>
> Thanks,
> John
>
> On Tue, May 22, 2018 at 7:52 AM, Jacob Carley - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> >
> > Hi John,
> >
> > Sure thing. I"m able to reproduce the issue on Theia as well.
Though the
> > repeating prints are a bit different (it gets hung up on QOB, TOB,
and
> > UOB).
> >
> > On Theia, see the following directory:
> >
> > /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET
> >
> > To run I did:
> >
> > module load met/7.0
> > pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> > PB2NCConfig_RTMA2P5_RU -index -log index.log
> >
> > I left my index.log file in that directory from a most recent test
in
> case
> > that is helpful (it's ~600M).
> >
> > Thanks!
> > Jacob
> >
> >
> >
> > On Mon, May 21, 2018 at 6:40 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Jacob,
> > >
> > > No, the contents of the config file don't matter, at least to my
> > > knowledge. I'm not able to replicate this behavior on the
sample file
> > that
> > > I have.
> > >
> > > The big question is whether you've found a data file that puts
pb2nc
> > into a
> > > bad loop... or whether we're getting different behavior on
different
> > > platforms. Hopefully it's the first problem since that'd be a
lot
> easier
> > > to fix.
> > >
> > > Where can I get a hold of the input file you're using:
> > > rtma2p5_ru.t1545z.prepbufr
> > >
> > > If you put it up on theia or an ftp site somewhere, I can grab
it, and
> > try
> > > to replicate the behavior here. And then Howard can debug
what's
> > actually
> > > going on.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, May 21, 2018 at 4:09 PM, Jacob Carley - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952
>
> > > >
> > > > Hi John,
> > > >
> > > > I'm running with 7.1 beta3 build on WCOSS, perhaps that's the
> culprit.
> > > > When I run it with -index here is what it prints to the screen
(and
> in
> > > the
> > > > log file) in what I think is an infinite loop:
> > > >
> > > > DEBUG 1: User Config File: PB2NCConfig_gust
> > > > DEBUG 1: Pre-processing BUFR File for variable names from
> > > > rtma2p5_ru.t1545z.prepbufr
> > > > DEBUG 1: process_pbfile_metadata() -> the number of records:
114311
> > > > DEBUG 1:
> > > > DEBUG 1: Header variables:
> > > > DEBUG 1: SAID: SATELLITE IDENTIFIER (SATELLITE REPORTS
ONLY)
> > > > DEBUG 1: XOB: LONGITUDE
> > > > DEBUG 1: YOB: LATITUDE
> > > > DEBUG 1: YEAR:
> > > > DEBUG 1: MNTH:
> > > > DEBUG 1: DAYS:
> > > > DEBUG 1: HOUR:
> > > > DEBUG 1: MINU:
> > > > DEBUG 1: SECO:
> > > > DEBUG 1:
> > > > DEBUG 1: Observation variables:
> > > > DEBUG 1: EEQF: PERCENT CONFIDENCE BASED ON NESDIS
EXPECTED
> ERROR
> > > > types:
> > > > SATWND
> > > > DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL INDX
> > W/O
> > > > FCST types: SATWND
> > > > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL INDX
> > > > W/ FCST types:
> > > > SATWND
> > > > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS
RECURSIVE
> > FILTER
> > > > FCN types: SATWND
> > > > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
ASSIMILATED)
> > > > types:
> > > > SATWND
> > > > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT
PROGRAM
> CODE
> > > > types:
> > > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT
ASSIMILATED)
> > > > types:
> > > > SATWND ADPSFC SFCSHP MSONET
> > > > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> > > > types:
> > > > SATWND ADPSFC SFCSHP MSONET
> > > > DEBUG 1: ZQM: HEIGHT (QUALITY) MARKER
> > > > types:
> > > > SATWND ADPSFC SFCSHP MSONET
> > > > DEBUG 1: ZOB: HEIGHT OBSERVATION
> > > > types:
> > > > SATWND ADPSFC SFCSHP MSONET
> > > > DEBUG 1: CAT: PREPBUFR DATA LEVEL CATEGORY
> > > > types:
> > > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > > *DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL INDX
> > W/O
> > > > FCST types: SATWND*
> > > > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL INDX
> > > > W/ FCST types:
> > > > SATWND
> > > > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS
RECURSIVE
> > FILTER
> > > > FCN types: SATWND
> > > > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
ASSIMILATED)
> > > > types:
> > > > SATWND
> > > > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT
PROGRAM
> CODE
> > > > types:
> > > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT
ASSIMILATED)
> > > > types:
> > > > SATWND ADPSFC SFCSHP MSONET
> > > > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> > > > types:
> > > > SATWND ADPSFC SFCSHP MSONET
> > > >
> > > > Which is pretty much just printed infinitely. I bolded the
line
> above
> > > that
> > > > shows when it begins to repeat
> > > >
> > > > Here's the command I use:
> > > >
> > > > pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> > > > PB2NCConfig_gust -index -log index.log
> > > >
> > > >
> > > > I just switched over to 7.0 on WCOSS (non-beta) and I get the
same
> > > behavior
> > > > with that as well.
> > > >
> > > > Does it matter what's in the config file when running with the
> '-index'
> > > > option aside from some basic stuff like the time window?
> > > >
> > > > Thanks!
> > > > Jacob
> > > >
> > > > On Mon, May 21, 2018 at 5:38 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Jacob,
> > > > >
> > > > > There's something odd going on here. You shouldn't see a 1G
log
> > file.
> > > > > When I ran that command was 11k. And indeed, I do see MXGS
listed:
> > > > > DEBUG 1: MXGS: MAXIMUM WIND SPEED
> > > > > (GUSTS) types: ADPSFC
SFCSHP
> > MSONET
> > > > > I've attached the log file to this email for reference.
> > > > >
> > > > > Here's the command I ran to generate this index file:
> > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > PB2NCConfig_CandV \
> > > > > -index -log index.log
> > > > >
> > > > > When you run with the "-index" option, no NetCDF output is
actually
> > > > > generated. It just search through the input file, figures
out what
> > > > > observations are present, and then prints them to the
screen. And
> > > also a
> > > > > log file since we used "-log index.log".
> > > > >
> > > > > Are you seeing behavior that differs from this?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > On Mon, May 21, 2018 at 2:52 PM, Jacob Carley - NOAA
Affiliate via
> > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > A quick follow up to this ticket - I have an additional
variable
> > I'd
> > > > like
> > > > > > to add and wanted to go the route of running pb2nc with '-
index'
> as
> > > you
> > > > > did
> > > > > > in this example. However when I do so, I get far too much
output
> > > that
> > > > I
> > > > > > can sensibly parse (log files well over 1G). Is this
typical?
> I'm
> > > > > looking
> > > > > > to see if I can pull out wind gust (the prepbufr mnemonic
is
> > 'MXGS').
> > > > I
> > > > > > figure I must not have something set quite right in my
pb2nc
> config
> > > > file.
> > > > > >
> > > > > > Thanks,
> > > > > > Jacob
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, May 1, 2018 at 5:51 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > wrote:
> > > > > >
> > > > > > > Jacob,
> > > > > > >
> > > > > > > Great, I'll go ahead and resolve this ticket. Just let
us know
> > if
> > > > > > anything
> > > > > > > else comes up.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=84952
> > >
> > > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > Thanks so much! This is hugely helpful.
> > > > > > > >
> > > > > > > > ADPSFC encompasses METAR observations, so that's
correct.
> > > > > > > >
> > > > > > > > This should be more than enough to get us rolling.
> > > > > > > >
> > > > > > > > Thanks again!
> > > > > > > > Jacob
> > > > > > > >
> > > > > > > > On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway via
RT <
> > > > > > > > met_help at ucar.edu
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Jacob and Runhua,
> > > > > > > > >
> > > > > > > > > Great, thanks. Here's the steps I took with this
data:
> > > > > > > > >
> > > > > > > > > (1) Run pb2nc using the "-index" option to the list
> > observation
> > > > > > > variable
> > > > > > > > > names for each message type... takes a few minutes
to run:
> > > > > > > > >
> > > > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > > PB2NCConfig_CandV \
> > > > > > > > > -index -log index.log
> > > > > > > > >
> > > > > > > > > I've attached the index.log file but here's a
selection of
> > the
> > > > > output
> > > > > > > > from
> > > > > > > > > that run:
> > > > > > > > > DEBUG 1: HOVI: HORIZONTAL
> > > > > > > > > VISIBILITY
> > > > > > > > > types: ADPSFC SFCSHP
> > > > > > > > > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF
HOCB -
> HGT
> > > OF
> > > > > > CLOUD
> > > > > > > > > BASE) types: ADPSFC
> > > > > > > > >
> > > > > > > > > You mentioned using METAR observations, but in that
> PREPBUFR
> > > > file,
> > > > > I
> > > > > > > only
> > > > > > > > > see CEILING and visibility (HOVI) listed for the
surface
> > > message
> > > > > > types.
> > > > > > > > >
> > > > > > > > > (2) Configure PB2NC to retain the observation for C
and V.
> > See
> > > > > > > attached
> > > > > > > > > file PB2NC_CandV.
> > > > > > > > >
> > > > > > > > > // Only keep surface message types
> > > > > > > > > message_type = [ "ONLYSF" ];
> > > > > > > > > // Only keep ceiling and visibility obs
> > > > > > > > > obs_bufr_var = [ "CEILING", "HOVI" ];
> > > > > > > > > // Out of convenience to get OBS to match GRIB2
variable
> > names,
> > > > > > rename
> > > > > > > > > CEILING as CEIL and HOVI as VIS
> > > > > > > > > obs_bufr_map = [
> > > > > > > > > { key = "CEILING"; val = "CEIL"; },
> > > > > > > > > { key = "HOVI"; val = "VIS"; }
> > > > > > > > > ];
> > > > > > > > >
> > > > > > > > > (3) Rerun pb2nc to actually process these obs:
> > > > > > > > >
> > > > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > > PB2NCConfig_CandV
> > > > > > > > >
> > > > > > > > > (4) Configure and run point_stat to do the vx. See
> attached
> > > > > > point_stat
> > > > > > > > > config file. Since LIFR, IFR, MVFR, and VFR are
defined in
> > > > > > > feet/miles, I
> > > > > > > > > used the convert() function to convert meters to
those
> units
> > > > prior
> > > > > to
> > > > > > > > > verifying. You certainly don't need to do that...
you
> could
> > > just
> > > > > > > specify
> > > > > > > > > the thresholds in meters instead:
> > > > > > > > >
> > > > > > > > > /usr/local/met-7.0/bin/point_stat \
> > > > > > > > > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > > PointStatConfig_CandV \
> > > > > > > > > -outdir out -v 3 -log run_ps.log
> > > > > > > > >
> > > > > > > > > This yields 3214 matched pairs for ceiling and 3622
for
> > > > visibility.
> > > > > > > > >
> > > > > > > > > (5) I configured point_stat to dump the matched
pairs (MPR)
> > > > output
> > > > > > line
> > > > > > > > > type. So I called the plot_mpr.R Rscript included
with MET
> > to
> > > > see
> > > > > > some
> > > > > > > > > plots of the data:
> > > > > > > > >
> > > > > > > > > Rscript /usr/local/met-
7.0/share/met/Rscripts/plot_mpr.R
> > > > > > > > > out/point_stat_000000L_20180501_121500V_mpr.txt
> > > > > > > > >
> > > > > > > > > The resulting PDF file is attached.
> > > > > > > > >
> > > > > > > > > Looking at the scatter plot, there's a curious band
of
> > > > observation
> > > > > > > values
> > > > > > > > > around 65000 ft for ceiling... and around 10 miles
for
> > > > visibility.
> > > > > > You
> > > > > > > > > should probably spend some time to understand this
data and
> > > > decide
> > > > > > how
> > > > > > > > you
> > > > > > > > > want to process it for verification.
> > > > > > > > >
> > > > > > > > > Looking at the CTS output line type, I see that the
> equitable
> > > > > threat
> > > > > > > > scores
> > > > > > > > > (i.e. GSS column) range between .14895 and .54148.
> > > > > > > > >
> > > > > > > > > Hopefully that helps get you going.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley - NOAA
> Affiliate
> > > via
> > > > > RT
> > > > > > <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=84952
> > > > >
> > > > > > > > > >
> > > > > > > > > > Hi John,
> > > > > > > > > >
> > > > > > > > > > [cc'd Runhua Yang, as she's working on this
project as
> > well].
> > > > > > > > > >
> > > > > > > > > > Wonderful - that would incredibly helpful.
> > > > > > > > > >
> > > > > > > > > > I've put a sample grib2 file and an associated
analysis
> > file
> > > > here
> > > > > > on
> > > > > > > > gyre
> > > > > > > > > > (wcoss phase2):
> > > > > > > > > >
> > > > > > > > > > /meso2/noscrub/Jacob.Carley/methelp
> > > > > > > > > >
> > > > > > > > > > It's from a system that runs every 15 minutes,
which
> > > hopefully
> > > > > > isn't
> > > > > > > an
> > > > > > > > > > issue for MET or METViewer (this particular file
is valid
> > at
> > > > > 1215Z
> > > > > > > from
> > > > > > > > > > this morning).
> > > > > > > > > >
> > > > > > > > > > Thanks again!
> > > > > > > > > > Jacob
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, May 1, 2018 at 3:41 PM, John Halley Gotway
via
> RT <
> > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Jacob,
> > > > > > > > > > >
> > > > > > > > > > > Yes, I certainly think that should be possible
using
> > > met-7.0.
> > > > > > If I
> > > > > > > > > > > remember correctly, Perry and I worked through
this not
> > too
> > > > > long
> > > > > > > ago.
> > > > > > > > > > >
> > > > > > > > > > > If you'd like, I could work up an example of
calling
> > pb2nc
> > > > and
> > > > > > > > > point_stat
> > > > > > > > > > > to demonstrate it. But I'd need you to point me
to a
> > > sample
> > > > > > model
> > > > > > > > > output
> > > > > > > > > > > file and PREPBUFR file that you'd like to use.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley -
NOAA
> > > > Affiliate
> > > > > > via
> > > > > > > > RT <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Tue May 01 12:33:24 2018: Request 84952 was
acted
> upon.
> > > > > > > > > > > > Transaction: Ticket created by
jacob.carley at noaa.gov
> > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > Subject: ceiling and visibility
verification in
> > MET
> > > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > > Requestors: jacob.carley at noaa.gov
> > > > > > > > > > > > Status: new
> > > > > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > > > Ticket/Display.html?id=84952
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi MET Help,
> > > > > > > > > > > >
> > > > > > > > > > > > Several of us at EMC are interested in doing
some
> > > > > verification
> > > > > > of
> > > > > > > > > > ceiling
> > > > > > > > > > > > and visibility using corresponding METAR
observations
> > in
> > > > > > prepbufr
> > > > > > > > > > files.
> > > > > > > > > > > > Speaking specifically, we would like to
perform
> > > categorical
> > > > > > > > > statistics
> > > > > > > > > > > > based upon a variety of aviation thresholds
(e.g.
> IFR,
> > > > LIFR,
> > > > > > > etc.).
> > > > > > > > > Is
> > > > > > > > > > > this
> > > > > > > > > > > > possible in MET v7?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > > Jacob
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: John Halley Gotway
Time: Tue May 22 11:47:02 2018
Jacob,
Thanks for opening up the permissions.
I was able to grab it. Bad news, at least from my perspective, I'm
not
able to replicate the behavior on my machine here. It runs in 4
minutes
and produces a log file with 101 lines in it. So we'll need to debug
some
more on theia.
John
On Tue, May 22, 2018 at 9:46 AM, Jacob Carley - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
>
> Hi John,
>
> Sorry about that - all should be good now.
>
> -Jacob
>
> On Tue, May 22, 2018 at 11:26 AM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Jacob,
> >
> > Could you please modify the permissions on the prepbufr file?
It's not
> > group-readable:
> >
> > [John.H.Gotway at tfe05 MET]$ ls -la
> > /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET/rtma2p5_
> > ru.t1545z.prepbufr
> > -rw-r----- 1 Jacob.Carley meso 9700664 Apr 19 21:22
> > /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET/rtma2p5_
> > ru.t1545z.prepbufr
> >
> > Thanks,
> > John
> >
> > On Tue, May 22, 2018 at 7:52 AM, Jacob Carley - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > >
> > > Hi John,
> > >
> > > Sure thing. I"m able to reproduce the issue on Theia as well.
Though
> the
> > > repeating prints are a bit different (it gets hung up on QOB,
TOB, and
> > > UOB).
> > >
> > > On Theia, see the following directory:
> > >
> > > /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET
> > >
> > > To run I did:
> > >
> > > module load met/7.0
> > > pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> > > PB2NCConfig_RTMA2P5_RU -index -log index.log
> > >
> > > I left my index.log file in that directory from a most recent
test in
> > case
> > > that is helpful (it's ~600M).
> > >
> > > Thanks!
> > > Jacob
> > >
> > >
> > >
> > > On Mon, May 21, 2018 at 6:40 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Jacob,
> > > >
> > > > No, the contents of the config file don't matter, at least to
my
> > > > knowledge. I'm not able to replicate this behavior on the
sample
> file
> > > that
> > > > I have.
> > > >
> > > > The big question is whether you've found a data file that puts
pb2nc
> > > into a
> > > > bad loop... or whether we're getting different behavior on
different
> > > > platforms. Hopefully it's the first problem since that'd be a
lot
> > easier
> > > > to fix.
> > > >
> > > > Where can I get a hold of the input file you're using:
> > > > rtma2p5_ru.t1545z.prepbufr
> > > >
> > > > If you put it up on theia or an ftp site somewhere, I can grab
it,
> and
> > > try
> > > > to replicate the behavior here. And then Howard can debug
what's
> > > actually
> > > > going on.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, May 21, 2018 at 4:09 PM, Jacob Carley - NOAA Affiliate
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > I'm running with 7.1 beta3 build on WCOSS, perhaps that's
the
> > culprit.
> > > > > When I run it with -index here is what it prints to the
screen (and
> > in
> > > > the
> > > > > log file) in what I think is an infinite loop:
> > > > >
> > > > > DEBUG 1: User Config File: PB2NCConfig_gust
> > > > > DEBUG 1: Pre-processing BUFR File for variable names from
> > > > > rtma2p5_ru.t1545z.prepbufr
> > > > > DEBUG 1: process_pbfile_metadata() -> the number of records:
114311
> > > > > DEBUG 1:
> > > > > DEBUG 1: Header variables:
> > > > > DEBUG 1: SAID: SATELLITE IDENTIFIER (SATELLITE
REPORTS ONLY)
> > > > > DEBUG 1: XOB: LONGITUDE
> > > > > DEBUG 1: YOB: LATITUDE
> > > > > DEBUG 1: YEAR:
> > > > > DEBUG 1: MNTH:
> > > > > DEBUG 1: DAYS:
> > > > > DEBUG 1: HOUR:
> > > > > DEBUG 1: MINU:
> > > > > DEBUG 1: SECO:
> > > > > DEBUG 1:
> > > > > DEBUG 1: Observation variables:
> > > > > DEBUG 1: EEQF: PERCENT CONFIDENCE BASED ON NESDIS
EXPECTED
> > ERROR
> > > > > types:
> > > > > SATWND
> > > > > DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL
> INDX
> > > W/O
> > > > > FCST types: SATWND
> > > > > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL
> INDX
> > > > > W/ FCST types:
> > > > > SATWND
> > > > > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS
RECURSIVE
> > > FILTER
> > > > > FCN types: SATWND
> > > > > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
> ASSIMILATED)
> > > > > types:
> > > > > SATWND
> > > > > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT
PROGRAM
> > CODE
> > > > > types:
> > > > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > > > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT
ASSIMILATED)
> > > > > types:
> > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> > > > > types:
> > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > DEBUG 1: ZQM: HEIGHT (QUALITY) MARKER
> > > > > types:
> > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > DEBUG 1: ZOB: HEIGHT OBSERVATION
> > > > > types:
> > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > DEBUG 1: CAT: PREPBUFR DATA LEVEL CATEGORY
> > > > > types:
> > > > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > > > *DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL
> INDX
> > > W/O
> > > > > FCST types: SATWND*
> > > > > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL
> INDX
> > > > > W/ FCST types:
> > > > > SATWND
> > > > > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS
RECURSIVE
> > > FILTER
> > > > > FCN types: SATWND
> > > > > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
> ASSIMILATED)
> > > > > types:
> > > > > SATWND
> > > > > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT
PROGRAM
> > CODE
> > > > > types:
> > > > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > > > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT
ASSIMILATED)
> > > > > types:
> > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> > > > > types:
> > > > > SATWND ADPSFC SFCSHP MSONET
> > > > >
> > > > > Which is pretty much just printed infinitely. I bolded the
line
> > above
> > > > that
> > > > > shows when it begins to repeat
> > > > >
> > > > > Here's the command I use:
> > > > >
> > > > > pb2nc rtma2p5_ru.t1545z.prepbufr
rtma2p5_ru.t1545z.prepbufr.nc
> > > > > PB2NCConfig_gust -index -log index.log
> > > > >
> > > > >
> > > > > I just switched over to 7.0 on WCOSS (non-beta) and I get
the same
> > > > behavior
> > > > > with that as well.
> > > > >
> > > > > Does it matter what's in the config file when running with
the
> > '-index'
> > > > > option aside from some basic stuff like the time window?
> > > > >
> > > > > Thanks!
> > > > > Jacob
> > > > >
> > > > > On Mon, May 21, 2018 at 5:38 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Jacob,
> > > > > >
> > > > > > There's something odd going on here. You shouldn't see a
1G log
> > > file.
> > > > > > When I ran that command was 11k. And indeed, I do see
MXGS
> listed:
> > > > > > DEBUG 1: MXGS: MAXIMUM WIND SPEED
> > > > > > (GUSTS) types: ADPSFC
SFCSHP
> > > MSONET
> > > > > > I've attached the log file to this email for reference.
> > > > > >
> > > > > > Here's the command I ran to generate this index file:
> > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > PB2NCConfig_CandV \
> > > > > > -index -log index.log
> > > > > >
> > > > > > When you run with the "-index" option, no NetCDF output is
> actually
> > > > > > generated. It just search through the input file, figures
out
> what
> > > > > > observations are present, and then prints them to the
screen.
> And
> > > > also a
> > > > > > log file since we used "-log index.log".
> > > > > >
> > > > > > Are you seeing behavior that differs from this?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > > On Mon, May 21, 2018 at 2:52 PM, Jacob Carley - NOAA
Affiliate
> via
> > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > A quick follow up to this ticket - I have an additional
> variable
> > > I'd
> > > > > like
> > > > > > > to add and wanted to go the route of running pb2nc with
> '-index'
> > as
> > > > you
> > > > > > did
> > > > > > > in this example. However when I do so, I get far too
much
> output
> > > > that
> > > > > I
> > > > > > > can sensibly parse (log files well over 1G). Is this
typical?
> > I'm
> > > > > > looking
> > > > > > > to see if I can pull out wind gust (the prepbufr
mnemonic is
> > > 'MXGS').
> > > > > I
> > > > > > > figure I must not have something set quite right in my
pb2nc
> > config
> > > > > file.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jacob
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 1, 2018 at 5:51 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Jacob,
> > > > > > > >
> > > > > > > > Great, I'll go ahead and resolve this ticket. Just
let us
> know
> > > if
> > > > > > > anything
> > > > > > > > else comes up.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=84952
> > > >
> > > > > > > > >
> > > > > > > > > Hi John,
> > > > > > > > >
> > > > > > > > > Thanks so much! This is hugely helpful.
> > > > > > > > >
> > > > > > > > > ADPSFC encompasses METAR observations, so that's
correct.
> > > > > > > > >
> > > > > > > > > This should be more than enough to get us rolling.
> > > > > > > > >
> > > > > > > > > Thanks again!
> > > > > > > > > Jacob
> > > > > > > > >
> > > > > > > > > On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway
via RT <
> > > > > > > > > met_help at ucar.edu
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Jacob and Runhua,
> > > > > > > > > >
> > > > > > > > > > Great, thanks. Here's the steps I took with this
data:
> > > > > > > > > >
> > > > > > > > > > (1) Run pb2nc using the "-index" option to the
list
> > > observation
> > > > > > > > variable
> > > > > > > > > > names for each message type... takes a few minutes
to
> run:
> > > > > > > > > >
> > > > > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > > > PB2NCConfig_CandV \
> > > > > > > > > > -index -log index.log
> > > > > > > > > >
> > > > > > > > > > I've attached the index.log file but here's a
selection
> of
> > > the
> > > > > > output
> > > > > > > > > from
> > > > > > > > > > that run:
> > > > > > > > > > DEBUG 1: HOVI: HORIZONTAL
> > > > > > > > > > VISIBILITY
> > > > > > > > > > types: ADPSFC SFCSHP
> > > > > > > > > > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE OF
HOCB -
> > HGT
> > > > OF
> > > > > > > CLOUD
> > > > > > > > > > BASE) types: ADPSFC
> > > > > > > > > >
> > > > > > > > > > You mentioned using METAR observations, but in
that
> > PREPBUFR
> > > > > file,
> > > > > > I
> > > > > > > > only
> > > > > > > > > > see CEILING and visibility (HOVI) listed for the
surface
> > > > message
> > > > > > > types.
> > > > > > > > > >
> > > > > > > > > > (2) Configure PB2NC to retain the observation for
C and
> V.
> > > See
> > > > > > > > attached
> > > > > > > > > > file PB2NC_CandV.
> > > > > > > > > >
> > > > > > > > > > // Only keep surface message types
> > > > > > > > > > message_type = [ "ONLYSF" ];
> > > > > > > > > > // Only keep ceiling and visibility obs
> > > > > > > > > > obs_bufr_var = [ "CEILING", "HOVI" ];
> > > > > > > > > > // Out of convenience to get OBS to match GRIB2
variable
> > > names,
> > > > > > > rename
> > > > > > > > > > CEILING as CEIL and HOVI as VIS
> > > > > > > > > > obs_bufr_map = [
> > > > > > > > > > { key = "CEILING"; val = "CEIL"; },
> > > > > > > > > > { key = "HOVI"; val = "VIS"; }
> > > > > > > > > > ];
> > > > > > > > > >
> > > > > > > > > > (3) Rerun pb2nc to actually process these obs:
> > > > > > > > > >
> > > > > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > > > PB2NCConfig_CandV
> > > > > > > > > >
> > > > > > > > > > (4) Configure and run point_stat to do the vx.
See
> > attached
> > > > > > > point_stat
> > > > > > > > > > config file. Since LIFR, IFR, MVFR, and VFR are
defined
> in
> > > > > > > > feet/miles, I
> > > > > > > > > > used the convert() function to convert meters to
those
> > units
> > > > > prior
> > > > > > to
> > > > > > > > > > verifying. You certainly don't need to do that...
you
> > could
> > > > just
> > > > > > > > specify
> > > > > > > > > > the thresholds in meters instead:
> > > > > > > > > >
> > > > > > > > > > /usr/local/met-7.0/bin/point_stat \
> > > > > > > > > > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> > > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > > > PointStatConfig_CandV \
> > > > > > > > > > -outdir out -v 3 -log run_ps.log
> > > > > > > > > >
> > > > > > > > > > This yields 3214 matched pairs for ceiling and
3622 for
> > > > > visibility.
> > > > > > > > > >
> > > > > > > > > > (5) I configured point_stat to dump the matched
pairs
> (MPR)
> > > > > output
> > > > > > > line
> > > > > > > > > > type. So I called the plot_mpr.R Rscript included
with
> MET
> > > to
> > > > > see
> > > > > > > some
> > > > > > > > > > plots of the data:
> > > > > > > > > >
> > > > > > > > > > Rscript /usr/local/met-
7.0/share/met/Rscripts/plot_mpr.R
> > > > > > > > > > out/point_stat_000000L_20180501_121500V_mpr.txt
> > > > > > > > > >
> > > > > > > > > > The resulting PDF file is attached.
> > > > > > > > > >
> > > > > > > > > > Looking at the scatter plot, there's a curious
band of
> > > > > observation
> > > > > > > > values
> > > > > > > > > > around 65000 ft for ceiling... and around 10 miles
for
> > > > > visibility.
> > > > > > > You
> > > > > > > > > > should probably spend some time to understand this
data
> and
> > > > > decide
> > > > > > > how
> > > > > > > > > you
> > > > > > > > > > want to process it for verification.
> > > > > > > > > >
> > > > > > > > > > Looking at the CTS output line type, I see that
the
> > equitable
> > > > > > threat
> > > > > > > > > scores
> > > > > > > > > > (i.e. GSS column) range between .14895 and .54148.
> > > > > > > > > >
> > > > > > > > > > Hopefully that helps get you going.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley -
NOAA
> > Affiliate
> > > > via
> > > > > > RT
> > > > > > > <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > Ticket/Display.html?id=84952
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi John,
> > > > > > > > > > >
> > > > > > > > > > > [cc'd Runhua Yang, as she's working on this
project as
> > > well].
> > > > > > > > > > >
> > > > > > > > > > > Wonderful - that would incredibly helpful.
> > > > > > > > > > >
> > > > > > > > > > > I've put a sample grib2 file and an associated
analysis
> > > file
> > > > > here
> > > > > > > on
> > > > > > > > > gyre
> > > > > > > > > > > (wcoss phase2):
> > > > > > > > > > >
> > > > > > > > > > > /meso2/noscrub/Jacob.Carley/methelp
> > > > > > > > > > >
> > > > > > > > > > > It's from a system that runs every 15 minutes,
which
> > > > hopefully
> > > > > > > isn't
> > > > > > > > an
> > > > > > > > > > > issue for MET or METViewer (this particular file
is
> valid
> > > at
> > > > > > 1215Z
> > > > > > > > from
> > > > > > > > > > > this morning).
> > > > > > > > > > >
> > > > > > > > > > > Thanks again!
> > > > > > > > > > > Jacob
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 1, 2018 at 3:41 PM, John Halley
Gotway via
> > RT <
> > > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Jacob,
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, I certainly think that should be possible
using
> > > > met-7.0.
> > > > > > > If I
> > > > > > > > > > > > remember correctly, Perry and I worked through
this
> not
> > > too
> > > > > > long
> > > > > > > > ago.
> > > > > > > > > > > >
> > > > > > > > > > > > If you'd like, I could work up an example of
calling
> > > pb2nc
> > > > > and
> > > > > > > > > > point_stat
> > > > > > > > > > > > to demonstrate it. But I'd need you to point
me to a
> > > > sample
> > > > > > > model
> > > > > > > > > > output
> > > > > > > > > > > > file and PREPBUFR file that you'd like to use.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > John
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 1, 2018 at 12:33 PM, Jacob Carley
- NOAA
> > > > > Affiliate
> > > > > > > via
> > > > > > > > > RT <
> > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Tue May 01 12:33:24 2018: Request 84952 was
acted
> > upon.
> > > > > > > > > > > > > Transaction: Ticket created by
> jacob.carley at noaa.gov
> > > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > > Subject: ceiling and visibility
verification
> in
> > > MET
> > > > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > > > Requestors: jacob.carley at noaa.gov
> > > > > > > > > > > > > Status: new
> > > > > > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > > > > Ticket/Display.html?id=84952
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi MET Help,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Several of us at EMC are interested in doing
some
> > > > > > verification
> > > > > > > of
> > > > > > > > > > > ceiling
> > > > > > > > > > > > > and visibility using corresponding METAR
> observations
> > > in
> > > > > > > prepbufr
> > > > > > > > > > > files.
> > > > > > > > > > > > > Speaking specifically, we would like to
perform
> > > > categorical
> > > > > > > > > > statistics
> > > > > > > > > > > > > based upon a variety of aviation thresholds
(e.g.
> > IFR,
> > > > > LIFR,
> > > > > > > > etc.).
> > > > > > > > > > Is
> > > > > > > > > > > > this
> > > > > > > > > > > > > possible in MET v7?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > Jacob
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Tue May 22 14:50:07 2018
Hi John,
Thanks for taking a look. Were you able to reproduce the issue on
Theia?
-Jacob
On Tue, May 22, 2018 at 1:47 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Jacob,
>
> Thanks for opening up the permissions.
>
> I was able to grab it. Bad news, at least from my perspective, I'm
not
> able to replicate the behavior on my machine here. It runs in 4
minutes
> and produces a log file with 101 lines in it. So we'll need to
debug some
> more on theia.
>
> John
>
> On Tue, May 22, 2018 at 9:46 AM, Jacob Carley - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> >
> > Hi John,
> >
> > Sorry about that - all should be good now.
> >
> > -Jacob
> >
> > On Tue, May 22, 2018 at 11:26 AM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Jacob,
> > >
> > > Could you please modify the permissions on the prepbufr file?
It's not
> > > group-readable:
> > >
> > > [John.H.Gotway at tfe05 MET]$ ls -la
> > > /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET/rtma2p5_
> > > ru.t1545z.prepbufr
> > > -rw-r----- 1 Jacob.Carley meso 9700664 Apr 19 21:22
> > > /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET/rtma2p5_
> > > ru.t1545z.prepbufr
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, May 22, 2018 at 7:52 AM, Jacob Carley - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952
>
> > > >
> > > > Hi John,
> > > >
> > > > Sure thing. I"m able to reproduce the issue on Theia as well.
Though
> > the
> > > > repeating prints are a bit different (it gets hung up on QOB,
TOB,
> and
> > > > UOB).
> > > >
> > > > On Theia, see the following directory:
> > > >
> > > > /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET
> > > >
> > > > To run I did:
> > > >
> > > > module load met/7.0
> > > > pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> > > > PB2NCConfig_RTMA2P5_RU -index -log index.log
> > > >
> > > > I left my index.log file in that directory from a most recent
test in
> > > case
> > > > that is helpful (it's ~600M).
> > > >
> > > > Thanks!
> > > > Jacob
> > > >
> > > >
> > > >
> > > > On Mon, May 21, 2018 at 6:40 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Jacob,
> > > > >
> > > > > No, the contents of the config file don't matter, at least
to my
> > > > > knowledge. I'm not able to replicate this behavior on the
sample
> > file
> > > > that
> > > > > I have.
> > > > >
> > > > > The big question is whether you've found a data file that
puts
> pb2nc
> > > > into a
> > > > > bad loop... or whether we're getting different behavior on
> different
> > > > > platforms. Hopefully it's the first problem since that'd be
a lot
> > > easier
> > > > > to fix.
> > > > >
> > > > > Where can I get a hold of the input file you're using:
> > > > > rtma2p5_ru.t1545z.prepbufr
> > > > >
> > > > > If you put it up on theia or an ftp site somewhere, I can
grab it,
> > and
> > > > try
> > > > > to replicate the behavior here. And then Howard can debug
what's
> > > > actually
> > > > > going on.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, May 21, 2018 at 4:09 PM, Jacob Carley - NOAA
Affiliate via
> > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > I'm running with 7.1 beta3 build on WCOSS, perhaps that's
the
> > > culprit.
> > > > > > When I run it with -index here is what it prints to the
screen
> (and
> > > in
> > > > > the
> > > > > > log file) in what I think is an infinite loop:
> > > > > >
> > > > > > DEBUG 1: User Config File: PB2NCConfig_gust
> > > > > > DEBUG 1: Pre-processing BUFR File for variable names from
> > > > > > rtma2p5_ru.t1545z.prepbufr
> > > > > > DEBUG 1: process_pbfile_metadata() -> the number of
records:
> 114311
> > > > > > DEBUG 1:
> > > > > > DEBUG 1: Header variables:
> > > > > > DEBUG 1: SAID: SATELLITE IDENTIFIER (SATELLITE
REPORTS
> ONLY)
> > > > > > DEBUG 1: XOB: LONGITUDE
> > > > > > DEBUG 1: YOB: LATITUDE
> > > > > > DEBUG 1: YEAR:
> > > > > > DEBUG 1: MNTH:
> > > > > > DEBUG 1: DAYS:
> > > > > > DEBUG 1: HOUR:
> > > > > > DEBUG 1: MINU:
> > > > > > DEBUG 1: SECO:
> > > > > > DEBUG 1:
> > > > > > DEBUG 1: Observation variables:
> > > > > > DEBUG 1: EEQF: PERCENT CONFIDENCE BASED ON NESDIS
EXPECTED
> > > ERROR
> > > > > > types:
> > > > > > SATWND
> > > > > > DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL
> > INDX
> > > > W/O
> > > > > > FCST types: SATWND
> > > > > > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL
> > INDX
> > > > > > W/ FCST types:
> > > > > > SATWND
> > > > > > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS
> RECURSIVE
> > > > FILTER
> > > > > > FCN types: SATWND
> > > > > > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
> > ASSIMILATED)
> > > > > > types:
> > > > > > SATWND
> > > > > > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT
> PROGRAM
> > > CODE
> > > > > > types:
> > > > > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > > > > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT
> ASSIMILATED)
> > > > > > types:
> > > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> > > > > > types:
> > > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > > DEBUG 1: ZQM: HEIGHT (QUALITY) MARKER
> > > > > > types:
> > > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > > DEBUG 1: ZOB: HEIGHT OBSERVATION
> > > > > > types:
> > > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > > DEBUG 1: CAT: PREPBUFR DATA LEVEL CATEGORY
> > > > > > types:
> > > > > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > > > > *DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON
EUMETSAT QUAL
> > INDX
> > > > W/O
> > > > > > FCST types: SATWND*
> > > > > > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON EUMETSAT
QUAL
> > INDX
> > > > > > W/ FCST types:
> > > > > > SATWND
> > > > > > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS
> RECURSIVE
> > > > FILTER
> > > > > > FCN types: SATWND
> > > > > > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
> > ASSIMILATED)
> > > > > > types:
> > > > > > SATWND
> > > > > > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB) EVENT
> PROGRAM
> > > CODE
> > > > > > types:
> > > > > > SATWND ADPSFC SFCSHP MSONET ASCATW
> > > > > > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT
> ASSIMILATED)
> > > > > > types:
> > > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
> > > > > > types:
> > > > > > SATWND ADPSFC SFCSHP MSONET
> > > > > >
> > > > > > Which is pretty much just printed infinitely. I bolded
the line
> > > above
> > > > > that
> > > > > > shows when it begins to repeat
> > > > > >
> > > > > > Here's the command I use:
> > > > > >
> > > > > > pb2nc rtma2p5_ru.t1545z.prepbufr
rtma2p5_ru.t1545z.prepbufr.nc
> > > > > > PB2NCConfig_gust -index -log index.log
> > > > > >
> > > > > >
> > > > > > I just switched over to 7.0 on WCOSS (non-beta) and I get
the
> same
> > > > > behavior
> > > > > > with that as well.
> > > > > >
> > > > > > Does it matter what's in the config file when running with
the
> > > '-index'
> > > > > > option aside from some basic stuff like the time window?
> > > > > >
> > > > > > Thanks!
> > > > > > Jacob
> > > > > >
> > > > > > On Mon, May 21, 2018 at 5:38 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Jacob,
> > > > > > >
> > > > > > > There's something odd going on here. You shouldn't see
a 1G
> log
> > > > file.
> > > > > > > When I ran that command was 11k. And indeed, I do see
MXGS
> > listed:
> > > > > > > DEBUG 1: MXGS: MAXIMUM WIND SPEED
> > > > > > > (GUSTS) types: ADPSFC
SFCSHP
> > > > MSONET
> > > > > > > I've attached the log file to this email for reference.
> > > > > > >
> > > > > > > Here's the command I ran to generate this index file:
> > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > PB2NCConfig_CandV \
> > > > > > > -index -log index.log
> > > > > > >
> > > > > > > When you run with the "-index" option, no NetCDF output
is
> > actually
> > > > > > > generated. It just search through the input file,
figures out
> > what
> > > > > > > observations are present, and then prints them to the
screen.
> > And
> > > > > also a
> > > > > > > log file since we used "-log index.log".
> > > > > > >
> > > > > > > Are you seeing behavior that differs from this?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > > On Mon, May 21, 2018 at 2:52 PM, Jacob Carley - NOAA
Affiliate
> > via
> > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=84952
> > >
> > > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > A quick follow up to this ticket - I have an
additional
> > variable
> > > > I'd
> > > > > > like
> > > > > > > > to add and wanted to go the route of running pb2nc
with
> > '-index'
> > > as
> > > > > you
> > > > > > > did
> > > > > > > > in this example. However when I do so, I get far too
much
> > output
> > > > > that
> > > > > > I
> > > > > > > > can sensibly parse (log files well over 1G). Is this
> typical?
> > > I'm
> > > > > > > looking
> > > > > > > > to see if I can pull out wind gust (the prepbufr
mnemonic is
> > > > 'MXGS').
> > > > > > I
> > > > > > > > figure I must not have something set quite right in my
pb2nc
> > > config
> > > > > > file.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jacob
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, May 1, 2018 at 5:51 PM, John Halley Gotway via
RT <
> > > > > > > > met_help at ucar.edu
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Jacob,
> > > > > > > > >
> > > > > > > > > Great, I'll go ahead and resolve this ticket. Just
let us
> > know
> > > > if
> > > > > > > > anything
> > > > > > > > > else comes up.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA
> Affiliate
> > > via
> > > > > RT
> > > > > > <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=84952
> > > > >
> > > > > > > > > >
> > > > > > > > > > Hi John,
> > > > > > > > > >
> > > > > > > > > > Thanks so much! This is hugely helpful.
> > > > > > > > > >
> > > > > > > > > > ADPSFC encompasses METAR observations, so that's
correct.
> > > > > > > > > >
> > > > > > > > > > This should be more than enough to get us rolling.
> > > > > > > > > >
> > > > > > > > > > Thanks again!
> > > > > > > > > > Jacob
> > > > > > > > > >
> > > > > > > > > > On Tue, May 1, 2018 at 4:50 PM, John Halley Gotway
via
> RT <
> > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Jacob and Runhua,
> > > > > > > > > > >
> > > > > > > > > > > Great, thanks. Here's the steps I took with this
data:
> > > > > > > > > > >
> > > > > > > > > > > (1) Run pb2nc using the "-index" option to the
list
> > > > observation
> > > > > > > > > variable
> > > > > > > > > > > names for each message type... takes a few
minutes to
> > run:
> > > > > > > > > > >
> > > > > > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > > > > PB2NCConfig_CandV \
> > > > > > > > > > > -index -log index.log
> > > > > > > > > > >
> > > > > > > > > > > I've attached the index.log file but here's a
selection
> > of
> > > > the
> > > > > > > output
> > > > > > > > > > from
> > > > > > > > > > > that run:
> > > > > > > > > > > DEBUG 1: HOVI: HORIZONTAL
> > > > > > > > > > > VISIBILITY
> > > > > > > > > > > types: ADPSFC SFCSHP
> > > > > > > > > > > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE
OF
> HOCB -
> > > HGT
> > > > > OF
> > > > > > > > CLOUD
> > > > > > > > > > > BASE) types: ADPSFC
> > > > > > > > > > >
> > > > > > > > > > > You mentioned using METAR observations, but in
that
> > > PREPBUFR
> > > > > > file,
> > > > > > > I
> > > > > > > > > only
> > > > > > > > > > > see CEILING and visibility (HOVI) listed for the
> surface
> > > > > message
> > > > > > > > types.
> > > > > > > > > > >
> > > > > > > > > > > (2) Configure PB2NC to retain the observation
for C and
> > V.
> > > > See
> > > > > > > > > attached
> > > > > > > > > > > file PB2NC_CandV.
> > > > > > > > > > >
> > > > > > > > > > > // Only keep surface message types
> > > > > > > > > > > message_type = [ "ONLYSF" ];
> > > > > > > > > > > // Only keep ceiling and visibility obs
> > > > > > > > > > > obs_bufr_var = [ "CEILING", "HOVI" ];
> > > > > > > > > > > // Out of convenience to get OBS to match GRIB2
> variable
> > > > names,
> > > > > > > > rename
> > > > > > > > > > > CEILING as CEIL and HOVI as VIS
> > > > > > > > > > > obs_bufr_map = [
> > > > > > > > > > > { key = "CEILING"; val = "CEIL"; },
> > > > > > > > > > > { key = "HOVI"; val = "VIS"; }
> > > > > > > > > > > ];
> > > > > > > > > > >
> > > > > > > > > > > (3) Rerun pb2nc to actually process these obs:
> > > > > > > > > > >
> > > > > > > > > > > /usr/local/met-7.0/bin/pb2nc \
> > > > > > > > > > > rtma2p5_ru.t1215z.prepbufr \
> > > > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > > > > PB2NCConfig_CandV
> > > > > > > > > > >
> > > > > > > > > > > (4) Configure and run point_stat to do the vx.
See
> > > attached
> > > > > > > > point_stat
> > > > > > > > > > > config file. Since LIFR, IFR, MVFR, and VFR are
> defined
> > in
> > > > > > > > > feet/miles, I
> > > > > > > > > > > used the convert() function to convert meters to
those
> > > units
> > > > > > prior
> > > > > > > to
> > > > > > > > > > > verifying. You certainly don't need to do
that... you
> > > could
> > > > > just
> > > > > > > > > specify
> > > > > > > > > > > the thresholds in meters instead:
> > > > > > > > > > >
> > > > > > > > > > > /usr/local/met-7.0/bin/point_stat \
> > > > > > > > > > > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
> > > > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
> > > > > > > > > > > PointStatConfig_CandV \
> > > > > > > > > > > -outdir out -v 3 -log run_ps.log
> > > > > > > > > > >
> > > > > > > > > > > This yields 3214 matched pairs for ceiling and
3622 for
> > > > > > visibility.
> > > > > > > > > > >
> > > > > > > > > > > (5) I configured point_stat to dump the matched
pairs
> > (MPR)
> > > > > > output
> > > > > > > > line
> > > > > > > > > > > type. So I called the plot_mpr.R Rscript
included with
> > MET
> > > > to
> > > > > > see
> > > > > > > > some
> > > > > > > > > > > plots of the data:
> > > > > > > > > > >
> > > > > > > > > > > Rscript /usr/local/met-7.0/share/met/
> Rscripts/plot_mpr.R
> > > > > > > > > > > out/point_stat_000000L_20180501_121500V_mpr.txt
> > > > > > > > > > >
> > > > > > > > > > > The resulting PDF file is attached.
> > > > > > > > > > >
> > > > > > > > > > > Looking at the scatter plot, there's a curious
band of
> > > > > > observation
> > > > > > > > > values
> > > > > > > > > > > around 65000 ft for ceiling... and around 10
miles for
> > > > > > visibility.
> > > > > > > > You
> > > > > > > > > > > should probably spend some time to understand
this data
> > and
> > > > > > decide
> > > > > > > > how
> > > > > > > > > > you
> > > > > > > > > > > want to process it for verification.
> > > > > > > > > > >
> > > > > > > > > > > Looking at the CTS output line type, I see that
the
> > > equitable
> > > > > > > threat
> > > > > > > > > > scores
> > > > > > > > > > > (i.e. GSS column) range between .14895 and
.54148.
> > > > > > > > > > >
> > > > > > > > > > > Hopefully that helps get you going.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley -
NOAA
> > > Affiliate
> > > > > via
> > > > > > > RT
> > > > > > > > <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > > Ticket/Display.html?id=84952
> > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi John,
> > > > > > > > > > > >
> > > > > > > > > > > > [cc'd Runhua Yang, as she's working on this
project
> as
> > > > well].
> > > > > > > > > > > >
> > > > > > > > > > > > Wonderful - that would incredibly helpful.
> > > > > > > > > > > >
> > > > > > > > > > > > I've put a sample grib2 file and an associated
> analysis
> > > > file
> > > > > > here
> > > > > > > > on
> > > > > > > > > > gyre
> > > > > > > > > > > > (wcoss phase2):
> > > > > > > > > > > >
> > > > > > > > > > > > /meso2/noscrub/Jacob.Carley/methelp
> > > > > > > > > > > >
> > > > > > > > > > > > It's from a system that runs every 15 minutes,
which
> > > > > hopefully
> > > > > > > > isn't
> > > > > > > > > an
> > > > > > > > > > > > issue for MET or METViewer (this particular
file is
> > valid
> > > > at
> > > > > > > 1215Z
> > > > > > > > > from
> > > > > > > > > > > > this morning).
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks again!
> > > > > > > > > > > > Jacob
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 1, 2018 at 3:41 PM, John Halley
Gotway
> via
> > > RT <
> > > > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Jacob,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, I certainly think that should be
possible
> using
> > > > > met-7.0.
> > > > > > > > If I
> > > > > > > > > > > > > remember correctly, Perry and I worked
through this
> > not
> > > > too
> > > > > > > long
> > > > > > > > > ago.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If you'd like, I could work up an example of
> calling
> > > > pb2nc
> > > > > > and
> > > > > > > > > > > point_stat
> > > > > > > > > > > > > to demonstrate it. But I'd need you to
point me
> to a
> > > > > sample
> > > > > > > > model
> > > > > > > > > > > output
> > > > > > > > > > > > > file and PREPBUFR file that you'd like to
use.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > John
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, May 1, 2018 at 12:33 PM, Jacob
Carley -
> NOAA
> > > > > > Affiliate
> > > > > > > > via
> > > > > > > > > > RT <
> > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Tue May 01 12:33:24 2018: Request 84952
was acted
> > > upon.
> > > > > > > > > > > > > > Transaction: Ticket created by
> > jacob.carley at noaa.gov
> > > > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > > > Subject: ceiling and visibility
verification
> > in
> > > > MET
> > > > > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > > > > Requestors: jacob.carley at noaa.gov
> > > > > > > > > > > > > > Status: new
> > > > > > > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > > > > > Ticket/Display.html?id=84952
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi MET Help,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Several of us at EMC are interested in
doing some
> > > > > > > verification
> > > > > > > > of
> > > > > > > > > > > > ceiling
> > > > > > > > > > > > > > and visibility using corresponding METAR
> > observations
> > > > in
> > > > > > > > prepbufr
> > > > > > > > > > > > files.
> > > > > > > > > > > > > > Speaking specifically, we would like to
perform
> > > > > categorical
> > > > > > > > > > > statistics
> > > > > > > > > > > > > > based upon a variety of aviation
thresholds (e.g.
> > > IFR,
> > > > > > LIFR,
> > > > > > > > > etc.).
> > > > > > > > > > > Is
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > possible in MET v7?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > Jacob
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Wed May 23 11:51:08 2018
Hi John,
I think I found one bug, not sure if it's related. But the prepbufr
file
we're processing is associated with a system that runs every 15
minutes and
so it has prepbufr output available every 15 minutes as well.
Therefore
the center time of the prepbufr file is not always at the top of the
hour.
In the example we have been using it's at 1545Z. When I run pb2nc in
verbose mode, with a +- 900s obs window, I see the following output:
DEBUG 1: Processing Bufr File: rtma2p5_ru.t1545z.prepbufr
DEBUG 1: Blocking Bufr file to: /tmp/tmp_pb2nc_blk_31951_0
*DEBUG 2: PrepBufr Time Center: 20180419_150000*
*DEBUG 2: Searching Time Window: 20180419_144500 to
20180419_151500*
I discovered this as I ran point_stat later and was wondering why all
the
output verification files were empty. When I looked more closely it
seemed
as though everything was getting tossed since it was outside the time
window, which seemed strange to me. When I went back and looked at my
config files they seemed fine. So I ran pb2nc with a high verbosity
level
and that's where I discovered this issue.
So the above *bold blue *lines should actually read the following way
for a
1545Z obs file with a +- 900s window:
*DEBUG 2: PrepBufr Time Center: 20180419_154500*
*DEBUG 2: Searching Time Window: 20180419_153000 to
20180419_160000*
Do you see the same behavior? Let me know if you need additional
info.
Thanks!
Jacob
On Tue, May 22, 2018 at 4:49 PM, Jacob Carley - NOAA Federal <
jacob.carley at noaa.gov> wrote:
> Hi John,
>
> Thanks for taking a look. Were you able to reproduce the issue on
Theia?
>
> -Jacob
>
> On Tue, May 22, 2018 at 1:47 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Jacob,
>>
>> Thanks for opening up the permissions.
>>
>> I was able to grab it. Bad news, at least from my perspective, I'm
not
>> able to replicate the behavior on my machine here. It runs in 4
minutes
>> and produces a log file with 101 lines in it. So we'll need to
debug some
>> more on theia.
>>
>> John
>>
>> On Tue, May 22, 2018 at 9:46 AM, Jacob Carley - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
>> >
>> > Hi John,
>> >
>> > Sorry about that - all should be good now.
>> >
>> > -Jacob
>> >
>> > On Tue, May 22, 2018 at 11:26 AM, John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > > Jacob,
>> > >
>> > > Could you please modify the permissions on the prepbufr file?
It's
>> not
>> > > group-readable:
>> > >
>> > > [John.H.Gotway at tfe05 MET]$ ls -la
>> > > /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET/rtma2p5_
>> > > ru.t1545z.prepbufr
>> > > -rw-r----- 1 Jacob.Carley meso 9700664 Apr 19 21:22
>> > > /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET/rtma2p5_
>> > > ru.t1545z.prepbufr
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Tue, May 22, 2018 at 7:52 AM, Jacob Carley - NOAA Affiliate
via RT
>> <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952
>
>> > > >
>> > > > Hi John,
>> > > >
>> > > > Sure thing. I"m able to reproduce the issue on Theia as
well.
>> Though
>> > the
>> > > > repeating prints are a bit different (it gets hung up on QOB,
TOB,
>> and
>> > > > UOB).
>> > > >
>> > > > On Theia, see the following directory:
>> > > >
>> > > > /scratch4/NCEPDEV/meso/save/Jacob.Carley/rtma/MET
>> > > >
>> > > > To run I did:
>> > > >
>> > > > module load met/7.0
>> > > > pb2nc rtma2p5_ru.t1545z.prepbufr
rtma2p5_ru.t1545z.prepbufr.nc
>> > > > PB2NCConfig_RTMA2P5_RU -index -log index.log
>> > > >
>> > > > I left my index.log file in that directory from a most recent
test
>> in
>> > > case
>> > > > that is helpful (it's ~600M).
>> > > >
>> > > > Thanks!
>> > > > Jacob
>> > > >
>> > > >
>> > > >
>> > > > On Mon, May 21, 2018 at 6:40 PM, John Halley Gotway via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > > Jacob,
>> > > > >
>> > > > > No, the contents of the config file don't matter, at least
to my
>> > > > > knowledge. I'm not able to replicate this behavior on the
sample
>> > file
>> > > > that
>> > > > > I have.
>> > > > >
>> > > > > The big question is whether you've found a data file that
puts
>> pb2nc
>> > > > into a
>> > > > > bad loop... or whether we're getting different behavior on
>> different
>> > > > > platforms. Hopefully it's the first problem since that'd
be a lot
>> > > easier
>> > > > > to fix.
>> > > > >
>> > > > > Where can I get a hold of the input file you're using:
>> > > > > rtma2p5_ru.t1545z.prepbufr
>> > > > >
>> > > > > If you put it up on theia or an ftp site somewhere, I can
grab it,
>> > and
>> > > > try
>> > > > > to replicate the behavior here. And then Howard can debug
what's
>> > > > actually
>> > > > > going on.
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > On Mon, May 21, 2018 at 4:09 PM, Jacob Carley - NOAA
Affiliate via
>> > RT <
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
>> > > > > >
>> > > > > > Hi John,
>> > > > > >
>> > > > > > I'm running with 7.1 beta3 build on WCOSS, perhaps that's
the
>> > > culprit.
>> > > > > > When I run it with -index here is what it prints to the
screen
>> (and
>> > > in
>> > > > > the
>> > > > > > log file) in what I think is an infinite loop:
>> > > > > >
>> > > > > > DEBUG 1: User Config File: PB2NCConfig_gust
>> > > > > > DEBUG 1: Pre-processing BUFR File for variable names from
>> > > > > > rtma2p5_ru.t1545z.prepbufr
>> > > > > > DEBUG 1: process_pbfile_metadata() -> the number of
records:
>> 114311
>> > > > > > DEBUG 1:
>> > > > > > DEBUG 1: Header variables:
>> > > > > > DEBUG 1: SAID: SATELLITE IDENTIFIER (SATELLITE
REPORTS
>> ONLY)
>> > > > > > DEBUG 1: XOB: LONGITUDE
>> > > > > > DEBUG 1: YOB: LATITUDE
>> > > > > > DEBUG 1: YEAR:
>> > > > > > DEBUG 1: MNTH:
>> > > > > > DEBUG 1: DAYS:
>> > > > > > DEBUG 1: HOUR:
>> > > > > > DEBUG 1: MINU:
>> > > > > > DEBUG 1: SECO:
>> > > > > > DEBUG 1:
>> > > > > > DEBUG 1: Observation variables:
>> > > > > > DEBUG 1: EEQF: PERCENT CONFIDENCE BASED ON NESDIS
>> EXPECTED
>> > > ERROR
>> > > > > > types:
>> > > > > > SATWND
>> > > > > > DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON
EUMETSAT QUAL
>> > INDX
>> > > > W/O
>> > > > > > FCST types: SATWND
>> > > > > > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON
EUMETSAT QUAL
>> > INDX
>> > > > > > W/ FCST types:
>> > > > > > SATWND
>> > > > > > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS
>> RECURSIVE
>> > > > FILTER
>> > > > > > FCN types: SATWND
>> > > > > > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
>> > ASSIMILATED)
>> > > > > > types:
>> > > > > > SATWND
>> > > > > > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB)
EVENT
>> PROGRAM
>> > > CODE
>> > > > > > types:
>> > > > > > SATWND ADPSFC SFCSHP MSONET ASCATW
>> > > > > > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT
>> ASSIMILATED)
>> > > > > > types:
>> > > > > > SATWND ADPSFC SFCSHP MSONET
>> > > > > > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
>> > > > > > types:
>> > > > > > SATWND ADPSFC SFCSHP MSONET
>> > > > > > DEBUG 1: ZQM: HEIGHT (QUALITY) MARKER
>> > > > > > types:
>> > > > > > SATWND ADPSFC SFCSHP MSONET
>> > > > > > DEBUG 1: ZOB: HEIGHT OBSERVATION
>> > > > > > types:
>> > > > > > SATWND ADPSFC SFCSHP MSONET
>> > > > > > DEBUG 1: CAT: PREPBUFR DATA LEVEL CATEGORY
>> > > > > > types:
>> > > > > > SATWND ADPSFC SFCSHP MSONET ASCATW
>> > > > > > *DEBUG 1: QIFN: PERCENT CONFIDENCE BASED ON
EUMETSAT QUAL
>> > INDX
>> > > > W/O
>> > > > > > FCST types: SATWND*
>> > > > > > DEBUG 1: QIFY: PERCENT CONFIDENCE BASED ON
EUMETSAT QUAL
>> > INDX
>> > > > > > W/ FCST types:
>> > > > > > SATWND
>> > > > > > DEBUG 1: RFFL: PERCENT CONFIDENCE BASED ON NESDIS
>> RECURSIVE
>> > > > FILTER
>> > > > > > FCN types: SATWND
>> > > > > > DEBUG 1: FFO: WIND SPEED OBSERVATION (kts) (NOT
>> > ASSIMILATED)
>> > > > > > types:
>> > > > > > SATWND
>> > > > > > DEBUG 1: WPC: U-, V-COMPONENT WIND (UOB/VOB)
EVENT
>> PROGRAM
>> > > CODE
>> > > > > > types:
>> > > > > > SATWND ADPSFC SFCSHP MSONET ASCATW
>> > > > > > DEBUG 1: DDO: WIND DIRECTION OBSERVATION (NOT
>> ASSIMILATED)
>> > > > > > types:
>> > > > > > SATWND ADPSFC SFCSHP MSONET
>> > > > > > DEBUG 1: ZPC: HEIGHT EVENT PROGRAM CODE
>> > > > > > types:
>> > > > > > SATWND ADPSFC SFCSHP MSONET
>> > > > > >
>> > > > > > Which is pretty much just printed infinitely. I bolded
the line
>> > > above
>> > > > > that
>> > > > > > shows when it begins to repeat
>> > > > > >
>> > > > > > Here's the command I use:
>> > > > > >
>> > > > > > pb2nc rtma2p5_ru.t1545z.prepbufr
rtma2p5_ru.t1545z.prepbufr.nc
>> > > > > > PB2NCConfig_gust -index -log index.log
>> > > > > >
>> > > > > >
>> > > > > > I just switched over to 7.0 on WCOSS (non-beta) and I get
the
>> same
>> > > > > behavior
>> > > > > > with that as well.
>> > > > > >
>> > > > > > Does it matter what's in the config file when running
with the
>> > > '-index'
>> > > > > > option aside from some basic stuff like the time window?
>> > > > > >
>> > > > > > Thanks!
>> > > > > > Jacob
>> > > > > >
>> > > > > > On Mon, May 21, 2018 at 5:38 PM, John Halley Gotway via
RT <
>> > > > > > met_help at ucar.edu> wrote:
>> > > > > >
>> > > > > > > Jacob,
>> > > > > > >
>> > > > > > > There's something odd going on here. You shouldn't see
a 1G
>> log
>> > > > file.
>> > > > > > > When I ran that command was 11k. And indeed, I do see
MXGS
>> > listed:
>> > > > > > > DEBUG 1: MXGS: MAXIMUM WIND SPEED
>> > > > > > > (GUSTS) types: ADPSFC
SFCSHP
>> > > > MSONET
>> > > > > > > I've attached the log file to this email for reference.
>> > > > > > >
>> > > > > > > Here's the command I ran to generate this index file:
>> > > > > > > /usr/local/met-7.0/bin/pb2nc \
>> > > > > > > rtma2p5_ru.t1215z.prepbufr \
>> > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
>> > > > > > > PB2NCConfig_CandV \
>> > > > > > > -index -log index.log
>> > > > > > >
>> > > > > > > When you run with the "-index" option, no NetCDF output
is
>> > actually
>> > > > > > > generated. It just search through the input file,
figures out
>> > what
>> > > > > > > observations are present, and then prints them to the
screen.
>> > And
>> > > > > also a
>> > > > > > > log file since we used "-log index.log".
>> > > > > > >
>> > > > > > > Are you seeing behavior that differs from this?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > John
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, May 21, 2018 at 2:52 PM, Jacob Carley - NOAA
Affiliate
>> > via
>> > > > RT <
>> > > > > > > met_help at ucar.edu> wrote:
>> > > > > > >
>> > > > > > > >
>> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/Tic
>> ket/Display.html?id=84952
>> > >
>> > > > > > > >
>> > > > > > > > Hi John,
>> > > > > > > >
>> > > > > > > > A quick follow up to this ticket - I have an
additional
>> > variable
>> > > > I'd
>> > > > > > like
>> > > > > > > > to add and wanted to go the route of running pb2nc
with
>> > '-index'
>> > > as
>> > > > > you
>> > > > > > > did
>> > > > > > > > in this example. However when I do so, I get far too
much
>> > output
>> > > > > that
>> > > > > > I
>> > > > > > > > can sensibly parse (log files well over 1G). Is this
>> typical?
>> > > I'm
>> > > > > > > looking
>> > > > > > > > to see if I can pull out wind gust (the prepbufr
mnemonic is
>> > > > 'MXGS').
>> > > > > > I
>> > > > > > > > figure I must not have something set quite right in
my pb2nc
>> > > config
>> > > > > > file.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Jacob
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Tue, May 1, 2018 at 5:51 PM, John Halley Gotway
via RT <
>> > > > > > > > met_help at ucar.edu
>> > > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Jacob,
>> > > > > > > > >
>> > > > > > > > > Great, I'll go ahead and resolve this ticket. Just
let us
>> > know
>> > > > if
>> > > > > > > > anything
>> > > > > > > > > else comes up.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > > John
>> > > > > > > > >
>> > > > > > > > > On Tue, May 1, 2018 at 3:15 PM, Jacob Carley - NOAA
>> Affiliate
>> > > via
>> > > > > RT
>> > > > > > <
>> > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
>> > > Ticket/Display.html?id=84952
>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > Hi John,
>> > > > > > > > > >
>> > > > > > > > > > Thanks so much! This is hugely helpful.
>> > > > > > > > > >
>> > > > > > > > > > ADPSFC encompasses METAR observations, so that's
>> correct.
>> > > > > > > > > >
>> > > > > > > > > > This should be more than enough to get us
rolling.
>> > > > > > > > > >
>> > > > > > > > > > Thanks again!
>> > > > > > > > > > Jacob
>> > > > > > > > > >
>> > > > > > > > > > On Tue, May 1, 2018 at 4:50 PM, John Halley
Gotway via
>> RT <
>> > > > > > > > > > met_help at ucar.edu
>> > > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Jacob and Runhua,
>> > > > > > > > > > >
>> > > > > > > > > > > Great, thanks. Here's the steps I took with
this data:
>> > > > > > > > > > >
>> > > > > > > > > > > (1) Run pb2nc using the "-index" option to the
list
>> > > > observation
>> > > > > > > > > variable
>> > > > > > > > > > > names for each message type... takes a few
minutes to
>> > run:
>> > > > > > > > > > >
>> > > > > > > > > > > /usr/local/met-7.0/bin/pb2nc \
>> > > > > > > > > > > rtma2p5_ru.t1215z.prepbufr \
>> > > > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
>> > > > > > > > > > > PB2NCConfig_CandV \
>> > > > > > > > > > > -index -log index.log
>> > > > > > > > > > >
>> > > > > > > > > > > I've attached the index.log file but here's a
>> selection
>> > of
>> > > > the
>> > > > > > > output
>> > > > > > > > > > from
>> > > > > > > > > > > that run:
>> > > > > > > > > > > DEBUG 1: HOVI: HORIZONTAL
>> > > > > > > > > > > VISIBILITY
>> > > > > > > > > > > types: ADPSFC SFCSHP
>> > > > > > > > > > > DEBUG 1: CEILING: CLOUD CEILING (DERIVATIVE
OF
>> HOCB -
>> > > HGT
>> > > > > OF
>> > > > > > > > CLOUD
>> > > > > > > > > > > BASE) types: ADPSFC
>> > > > > > > > > > >
>> > > > > > > > > > > You mentioned using METAR observations, but in
that
>> > > PREPBUFR
>> > > > > > file,
>> > > > > > > I
>> > > > > > > > > only
>> > > > > > > > > > > see CEILING and visibility (HOVI) listed for
the
>> surface
>> > > > > message
>> > > > > > > > types.
>> > > > > > > > > > >
>> > > > > > > > > > > (2) Configure PB2NC to retain the observation
for C
>> and
>> > V.
>> > > > See
>> > > > > > > > > attached
>> > > > > > > > > > > file PB2NC_CandV.
>> > > > > > > > > > >
>> > > > > > > > > > > // Only keep surface message types
>> > > > > > > > > > > message_type = [ "ONLYSF" ];
>> > > > > > > > > > > // Only keep ceiling and visibility obs
>> > > > > > > > > > > obs_bufr_var = [ "CEILING", "HOVI" ];
>> > > > > > > > > > > // Out of convenience to get OBS to match GRIB2
>> variable
>> > > > names,
>> > > > > > > > rename
>> > > > > > > > > > > CEILING as CEIL and HOVI as VIS
>> > > > > > > > > > > obs_bufr_map = [
>> > > > > > > > > > > { key = "CEILING"; val = "CEIL"; },
>> > > > > > > > > > > { key = "HOVI"; val = "VIS"; }
>> > > > > > > > > > > ];
>> > > > > > > > > > >
>> > > > > > > > > > > (3) Rerun pb2nc to actually process these obs:
>> > > > > > > > > > >
>> > > > > > > > > > > /usr/local/met-7.0/bin/pb2nc \
>> > > > > > > > > > > rtma2p5_ru.t1215z.prepbufr \
>> > > > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
>> > > > > > > > > > > PB2NCConfig_CandV
>> > > > > > > > > > >
>> > > > > > > > > > > (4) Configure and run point_stat to do the vx.
See
>> > > attached
>> > > > > > > > point_stat
>> > > > > > > > > > > config file. Since LIFR, IFR, MVFR, and VFR
are
>> defined
>> > in
>> > > > > > > > > feet/miles, I
>> > > > > > > > > > > used the convert() function to convert meters
to those
>> > > units
>> > > > > > prior
>> > > > > > > to
>> > > > > > > > > > > verifying. You certainly don't need to do
that... you
>> > > could
>> > > > > just
>> > > > > > > > > specify
>> > > > > > > > > > > the thresholds in meters instead:
>> > > > > > > > > > >
>> > > > > > > > > > > /usr/local/met-7.0/bin/point_stat \
>> > > > > > > > > > > rtma2p5_ru.t1215z.2dvaranl_ndfd.grb2 \
>> > > > > > > > > > > rtma2p5_ru.t1215z.prepbufr.nc \
>> > > > > > > > > > > PointStatConfig_CandV \
>> > > > > > > > > > > -outdir out -v 3 -log run_ps.log
>> > > > > > > > > > >
>> > > > > > > > > > > This yields 3214 matched pairs for ceiling and
3622
>> for
>> > > > > > visibility.
>> > > > > > > > > > >
>> > > > > > > > > > > (5) I configured point_stat to dump the matched
pairs
>> > (MPR)
>> > > > > > output
>> > > > > > > > line
>> > > > > > > > > > > type. So I called the plot_mpr.R Rscript
included
>> with
>> > MET
>> > > > to
>> > > > > > see
>> > > > > > > > some
>> > > > > > > > > > > plots of the data:
>> > > > > > > > > > >
>> > > > > > > > > > > Rscript /usr/local/met-7.0/share/met/R
>> scripts/plot_mpr.R
>> > > > > > > > > > > out/point_stat_000000L_20180501_121500V_mpr.txt
>> > > > > > > > > > >
>> > > > > > > > > > > The resulting PDF file is attached.
>> > > > > > > > > > >
>> > > > > > > > > > > Looking at the scatter plot, there's a curious
band of
>> > > > > > observation
>> > > > > > > > > values
>> > > > > > > > > > > around 65000 ft for ceiling... and around 10
miles for
>> > > > > > visibility.
>> > > > > > > > You
>> > > > > > > > > > > should probably spend some time to understand
this
>> data
>> > and
>> > > > > > decide
>> > > > > > > > how
>> > > > > > > > > > you
>> > > > > > > > > > > want to process it for verification.
>> > > > > > > > > > >
>> > > > > > > > > > > Looking at the CTS output line type, I see that
the
>> > > equitable
>> > > > > > > threat
>> > > > > > > > > > scores
>> > > > > > > > > > > (i.e. GSS column) range between .14895 and
.54148.
>> > > > > > > > > > >
>> > > > > > > > > > > Hopefully that helps get you going.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > > John
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Tue, May 1, 2018 at 1:52 PM, Jacob Carley -
NOAA
>> > > Affiliate
>> > > > > via
>> > > > > > > RT
>> > > > > > > > <
>> > > > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
>> > > > > Ticket/Display.html?id=84952
>> > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Hi John,
>> > > > > > > > > > > >
>> > > > > > > > > > > > [cc'd Runhua Yang, as she's working on this
project
>> as
>> > > > well].
>> > > > > > > > > > > >
>> > > > > > > > > > > > Wonderful - that would incredibly helpful.
>> > > > > > > > > > > >
>> > > > > > > > > > > > I've put a sample grib2 file and an
associated
>> analysis
>> > > > file
>> > > > > > here
>> > > > > > > > on
>> > > > > > > > > > gyre
>> > > > > > > > > > > > (wcoss phase2):
>> > > > > > > > > > > >
>> > > > > > > > > > > > /meso2/noscrub/Jacob.Carley/methelp
>> > > > > > > > > > > >
>> > > > > > > > > > > > It's from a system that runs every 15
minutes, which
>> > > > > hopefully
>> > > > > > > > isn't
>> > > > > > > > > an
>> > > > > > > > > > > > issue for MET or METViewer (this particular
file is
>> > valid
>> > > > at
>> > > > > > > 1215Z
>> > > > > > > > > from
>> > > > > > > > > > > > this morning).
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks again!
>> > > > > > > > > > > > Jacob
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Tue, May 1, 2018 at 3:41 PM, John Halley
Gotway
>> via
>> > > RT <
>> > > > > > > > > > > > met_help at ucar.edu
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Jacob,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Yes, I certainly think that should be
possible
>> using
>> > > > > met-7.0.
>> > > > > > > > If I
>> > > > > > > > > > > > > remember correctly, Perry and I worked
through
>> this
>> > not
>> > > > too
>> > > > > > > long
>> > > > > > > > > ago.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > If you'd like, I could work up an example
of
>> calling
>> > > > pb2nc
>> > > > > > and
>> > > > > > > > > > > point_stat
>> > > > > > > > > > > > > to demonstrate it. But I'd need you to
point me
>> to a
>> > > > > sample
>> > > > > > > > model
>> > > > > > > > > > > output
>> > > > > > > > > > > > > file and PREPBUFR file that you'd like to
use.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > John
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Tue, May 1, 2018 at 12:33 PM, Jacob
Carley -
>> NOAA
>> > > > > > Affiliate
>> > > > > > > > via
>> > > > > > > > > > RT <
>> > > > > > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Tue May 01 12:33:24 2018: Request 84952
was
>> acted
>> > > upon.
>> > > > > > > > > > > > > > Transaction: Ticket created by
>> > jacob.carley at noaa.gov
>> > > > > > > > > > > > > > Queue: met_help
>> > > > > > > > > > > > > > Subject: ceiling and visibility
>> verification
>> > in
>> > > > MET
>> > > > > > > > > > > > > > Owner: Nobody
>> > > > > > > > > > > > > > Requestors: jacob.carley at noaa.gov
>> > > > > > > > > > > > > > Status: new
>> > > > > > > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
>> > > > > > > > > > > Ticket/Display.html?id=84952
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi MET Help,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Several of us at EMC are interested in
doing
>> some
>> > > > > > > verification
>> > > > > > > > of
>> > > > > > > > > > > > ceiling
>> > > > > > > > > > > > > > and visibility using corresponding METAR
>> > observations
>> > > > in
>> > > > > > > > prepbufr
>> > > > > > > > > > > > files.
>> > > > > > > > > > > > > > Speaking specifically, we would like to
perform
>> > > > > categorical
>> > > > > > > > > > > statistics
>> > > > > > > > > > > > > > based upon a variety of aviation
thresholds
>> (e.g.
>> > > IFR,
>> > > > > > LIFR,
>> > > > > > > > > etc.).
>> > > > > > > > > > > Is
>> > > > > > > > > > > > > this
>> > > > > > > > > > > > > > possible in MET v7?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thanks!
>> > > > > > > > > > > > > > Jacob
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: John Halley Gotway
Time: Wed May 23 12:52:15 2018
Jacob,
Thanks for looking closely at the output. This whole notion of
picking a
"center time" and defining a retention time window around it using
"obs_window" is inherited from our original processing of convention
PREPBUFR files. That logic assumes that all messages in the file
contain
the same center time. And perhaps that's not true for the RTMA data?
Howard looked into it and found the following...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
pb2nc gets the time by calling IREADNS which returns IDATE (from the
first message and the first subset).
C SUBPROGRAM: IREADNS
C PRGMMR: WOOLLEN ORG: NP20 DATE: 1994-01-06
C
C ABSTRACT: THIS FUNCTION CALLS BUFR ARCHIVE LIBRARY SUBROUTINE READNS
C AND PASSES BACK ITS RETURN CODE. SEE READNS FOR MORE DETAILS.
C
...
C OUTPUT ARGUMENT LIST:
C SUBSET - CHARACTER*8: TABLE A MNEMONIC FOR BUFR MESSAGE
C CONTAINING SUBSET BEING READ
C IDATE - INTEGER: DATE-TIME STORED WITHIN SECTION 1 OF BUFR
C MESSAGE CONTAINING SUBSET BEING READ, IN FORMAT OF
C EITHER YYMMDDHH OR YYYYMMDDHH, DEPENDING ON DATELEN()
C VALUE
C IREADNS - INTEGER: RETURN CODE:
C 0 = normal return
C -1 = there are no more subsets in the BUFR file
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
A workaround already exists for this. You can explicitly specify the
retention time window on the command line for PB2NC:
"-valid_beg time" in YYYYMMDD[_HH[MMSS]] sets the
beginning of the retention time window (optional).
"-valid_end time" in YYYYMMDD[_HH[MMSS]] sets the end
of the retention time window (optional).
That time window setting overrides the obs_window setting from the
config file. So you could use:
-valid_beg 20180419_153000 -valid_end 20180419_160000
So that'd solve the immediate problem.
But long term, we should make sense of the "center time" logic. Do
you have any additional details about these files that would help us
determine if/how the center time logic should change?
Thanks,
John
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Wed May 23 13:20:13 2018
Hi John,
These files do all have the same "center time", which really is the
analysis/cycle time. The obs contained within are all stored relative
to
that time. I suppose the issue is just ensuring the correct time
analysis/cycletime is used (i.e. we get the minutes part).
I did some poking around, and here's an example we do to get the
minutes
portion of the analysis/cycle time after grabbing the 10 digit idate:
call openbf(unit_in,'IN',unit_in)
call dxdump(unit_in,unit_table)
call datelen(10)
nmsg=0
MMcyc0=-1
msg_report: do while (ireadmg(unit_in,subset,idate) == 0)
idate_hold=idate ! idate gets lost/clobbered w/ last, failed
ireadmg call
nmsg=nmsg+1
ntb = 0
! get minutes portion of analysis/cycle time
MMcyc=*IUPVS01*(unit_in,'MINU')
if (MMcyc.ne.MMcyc0) then
MMcyc0=MMcyc
write(*,'(42x,a,$)') '-- analysis time (dTime ref): '
write(*,'(2x,i8,i3.2,":",i2.2)') idate/100, mod(idate,100),MMcyc !
CCYYMMDD HH:MM
endif
[Note I've not pasted the rest of the loop here]
The *IUPVS01 *function will get the minutes of the cycle time. I
*think*
that should do the trick.
I did try running with:
pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
PB2NCConfig_gust -v 5 -log pb2nc.log -valid_beg 20180419_153000
-valid_end
20180419_160000
But pb2nc didn't output anything, here's a summary of some of the
output:
DEBUG 2: PrepBufr Time Center: 20180419_150000
DEBUG 2: Searching Time Window: 20180419_153000 to
20180419_160000
DEBUG 2: Processing 114311 PrepBufr messages...
DEBUG 2: Total PrepBufr Messages processed = 114311
DEBUG 2: Rejected based on message type = 108874
DEBUG 2: Rejected based on station id = 0
DEBUG 2: Rejected based on valid time = 5437
DEBUG 2: Rejected based on masking grid = 0
DEBUG 2: Rejected based on masking polygon = 0
DEBUG 2: Rejected based on elevation = 0
DEBUG 2: Rejected based on pb report type = 0
DEBUG 2: Rejected based on input report type = 0
DEBUG 2: Rejected based on instrument type = 0
DEBUG 2: Rejected based on zero observations = 0
DEBUG 2: Total PrepBufr Messages retained = 0
DEBUG 2: Total observations retained or derived = 0
If pb2nc is caluclating the absolute time of the ob using the DHR in
association with the 10 digit idate then this would probably cause
this
issue. Thus even adding the -valid_beg and -valid_end flags wouldn't
help
since pb2nc is not calculating the valid time of the observations
correctly
as the minutes part of the center time is not accounted for.
Thanks!
Jacob
On Wed, May 23, 2018 at 2:52 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Jacob,
>
> Thanks for looking closely at the output. This whole notion of
picking a
> "center time" and defining a retention time window around it using
> "obs_window" is inherited from our original processing of convention
> PREPBUFR files. That logic assumes that all messages in the file
contain
> the same center time. And perhaps that's not true for the RTMA
data?
>
> Howard looked into it and found the following...
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> pb2nc gets the time by calling IREADNS which returns IDATE (from the
> first message and the first subset).
>
>
> C SUBPROGRAM: IREADNS
> C PRGMMR: WOOLLEN ORG: NP20 DATE: 1994-01-06
> C
> C ABSTRACT: THIS FUNCTION CALLS BUFR ARCHIVE LIBRARY SUBROUTINE
READNS
> C AND PASSES BACK ITS RETURN CODE. SEE READNS FOR MORE DETAILS.
> C
> ...
> C OUTPUT ARGUMENT LIST:
> C SUBSET - CHARACTER*8: TABLE A MNEMONIC FOR BUFR MESSAGE
> C CONTAINING SUBSET BEING READ
> C IDATE - INTEGER: DATE-TIME STORED WITHIN SECTION 1 OF BUFR
> C MESSAGE CONTAINING SUBSET BEING READ, IN FORMAT OF
> C EITHER YYMMDDHH OR YYYYMMDDHH, DEPENDING ON
DATELEN()
> C VALUE
> C IREADNS - INTEGER: RETURN CODE:
> C 0 = normal return
> C -1 = there are no more subsets in the BUFR
file
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> A workaround already exists for this. You can explicitly specify
the
> retention time window on the command line for PB2NC:
> "-valid_beg time" in YYYYMMDD[_HH[MMSS]] sets the
> beginning of the retention time window (optional).
> "-valid_end time" in YYYYMMDD[_HH[MMSS]] sets the
end
> of the retention time window (optional).
>
> That time window setting overrides the obs_window setting from the
> config file. So you could use:
>
> -valid_beg 20180419_153000 -valid_end 20180419_160000
>
> So that'd solve the immediate problem.
>
> But long term, we should make sense of the "center time" logic. Do
> you have any additional details about these files that would help us
> determine if/how the center time logic should change?
>
>
> Thanks,
> John
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Wed May 23 13:23:10 2018
I should mentioned that the *IUPVS01 *function comes from bufrlib and
you
declare it, as well as the other date/time variables, as:
INTEGER :: IUPVS01, MMcyc, MMcyc0, idate_hold, idate
-Jacob
On Wed, May 23, 2018 at 3:19 PM, Jacob Carley - NOAA Federal <
jacob.carley at noaa.gov> wrote:
> Hi John,
>
> These files do all have the same "center time", which really is the
> analysis/cycle time. The obs contained within are all stored
relative to
> that time. I suppose the issue is just ensuring the correct time
> analysis/cycletime is used (i.e. we get the minutes part).
>
> I did some poking around, and here's an example we do to get the
minutes
> portion of the analysis/cycle time after grabbing the 10 digit
idate:
>
> call openbf(unit_in,'IN',unit_in)
> call dxdump(unit_in,unit_table)
> call datelen(10)
> nmsg=0
> MMcyc0=-1
> msg_report: do while (ireadmg(unit_in,subset,idate) == 0)
> idate_hold=idate ! idate gets lost/clobbered w/ last, failed
ireadmg
> call
> nmsg=nmsg+1
> ntb = 0
> ! get minutes portion of analysis/cycle time
> MMcyc=*IUPVS01*(unit_in,'MINU')
> if (MMcyc.ne.MMcyc0) then
> MMcyc0=MMcyc
> write(*,'(42x,a,$)') '-- analysis time (dTime ref): '
> write(*,'(2x,i8,i3.2,":",i2.2)') idate/100, mod(idate,100),MMcyc !
> CCYYMMDD HH:MM
> endif
>
> [Note I've not pasted the rest of the loop here]
>
> The *IUPVS01 *function will get the minutes of the cycle time. I
*think*
> that should do the trick.
>
> I did try running with:
>
> pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> PB2NCConfig_gust -v 5 -log pb2nc.log -valid_beg 20180419_153000
-valid_end
> 20180419_160000
>
> But pb2nc didn't output anything, here's a summary of some of the
output:
>
> DEBUG 2: PrepBufr Time Center: 20180419_150000
> DEBUG 2: Searching Time Window: 20180419_153000 to
20180419_160000
> DEBUG 2: Processing 114311 PrepBufr messages...
> DEBUG 2: Total PrepBufr Messages processed = 114311
> DEBUG 2: Rejected based on message type = 108874
> DEBUG 2: Rejected based on station id = 0
> DEBUG 2: Rejected based on valid time = 5437
> DEBUG 2: Rejected based on masking grid = 0
> DEBUG 2: Rejected based on masking polygon = 0
> DEBUG 2: Rejected based on elevation = 0
> DEBUG 2: Rejected based on pb report type = 0
> DEBUG 2: Rejected based on input report type = 0
> DEBUG 2: Rejected based on instrument type = 0
> DEBUG 2: Rejected based on zero observations = 0
> DEBUG 2: Total PrepBufr Messages retained = 0
> DEBUG 2: Total observations retained or derived = 0
>
>
> If pb2nc is caluclating the absolute time of the ob using the DHR in
> association with the 10 digit idate then this would probably cause
this
> issue. Thus even adding the -valid_beg and -valid_end flags
wouldn't help
> since pb2nc is not calculating the valid time of the observations
correctly
> as the minutes part of the center time is not accounted for.
>
> Thanks!
> Jacob
>
> On Wed, May 23, 2018 at 2:52 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Jacob,
>>
>> Thanks for looking closely at the output. This whole notion of
picking a
>> "center time" and defining a retention time window around it using
>> "obs_window" is inherited from our original processing of
convention
>> PREPBUFR files. That logic assumes that all messages in the file
contain
>> the same center time. And perhaps that's not true for the RTMA
data?
>>
>> Howard looked into it and found the following...
>>
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> pb2nc gets the time by calling IREADNS which returns IDATE (from
the
>> first message and the first subset).
>>
>>
>> C SUBPROGRAM: IREADNS
>> C PRGMMR: WOOLLEN ORG: NP20 DATE: 1994-01-06
>> C
>> C ABSTRACT: THIS FUNCTION CALLS BUFR ARCHIVE LIBRARY SUBROUTINE
READNS
>> C AND PASSES BACK ITS RETURN CODE. SEE READNS FOR MORE DETAILS.
>> C
>> ...
>> C OUTPUT ARGUMENT LIST:
>> C SUBSET - CHARACTER*8: TABLE A MNEMONIC FOR BUFR MESSAGE
>> C CONTAINING SUBSET BEING READ
>> C IDATE - INTEGER: DATE-TIME STORED WITHIN SECTION 1 OF BUFR
>> C MESSAGE CONTAINING SUBSET BEING READ, IN FORMAT OF
>> C EITHER YYMMDDHH OR YYYYMMDDHH, DEPENDING ON
DATELEN()
>> C VALUE
>> C IREADNS - INTEGER: RETURN CODE:
>> C 0 = normal return
>> C -1 = there are no more subsets in the BUFR
file
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> A workaround already exists for this. You can explicitly specify
the
>> retention time window on the command line for PB2NC:
>> "-valid_beg time" in YYYYMMDD[_HH[MMSS]] sets the
>> beginning of the retention time window (optional).
>> "-valid_end time" in YYYYMMDD[_HH[MMSS]] sets the
end
>> of the retention time window (optional).
>>
>> That time window setting overrides the obs_window setting from the
>> config file. So you could use:
>>
>> -valid_beg 20180419_153000 -valid_end 20180419_160000
>>
>> So that'd solve the immediate problem.
>>
>> But long term, we should make sense of the "center time" logic. Do
>> you have any additional details about these files that would help
us
>> determine if/how the center time logic should change?
>>
>>
>> Thanks,
>> John
>>
>>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: John Halley Gotway
Time: Thu May 24 09:07:57 2018
Jacob,
This is all great info! Thanks for clarifying. Yes, we'll need to
patch
this up and get it working.
Is it sufficient to add this to the next met-7.1 beta release or will
we
need a patch for the met-7.0 release as well?
Thanks,
John
On Wed, May 23, 2018 at 1:23 PM, Jacob Carley - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
>
> I should mentioned that the *IUPVS01 *function comes from bufrlib
and you
> declare it, as well as the other date/time variables, as:
>
> INTEGER :: IUPVS01, MMcyc, MMcyc0, idate_hold, idate
>
> -Jacob
>
> On Wed, May 23, 2018 at 3:19 PM, Jacob Carley - NOAA Federal <
> jacob.carley at noaa.gov> wrote:
>
> > Hi John,
> >
> > These files do all have the same "center time", which really is
the
> > analysis/cycle time. The obs contained within are all stored
relative to
> > that time. I suppose the issue is just ensuring the correct time
> > analysis/cycletime is used (i.e. we get the minutes part).
> >
> > I did some poking around, and here's an example we do to get the
minutes
> > portion of the analysis/cycle time after grabbing the 10 digit
idate:
> >
> > call openbf(unit_in,'IN',unit_in)
> > call dxdump(unit_in,unit_table)
> > call datelen(10)
> > nmsg=0
> > MMcyc0=-1
> > msg_report: do while (ireadmg(unit_in,subset,idate) == 0)
> > idate_hold=idate ! idate gets lost/clobbered w/ last, failed
ireadmg
> > call
> > nmsg=nmsg+1
> > ntb = 0
> > ! get minutes portion of analysis/cycle time
> > MMcyc=*IUPVS01*(unit_in,'MINU')
> > if (MMcyc.ne.MMcyc0) then
> > MMcyc0=MMcyc
> > write(*,'(42x,a,$)') '-- analysis time (dTime ref): '
> > write(*,'(2x,i8,i3.2,":",i2.2)') idate/100, mod(idate,100),MMcyc
!
> > CCYYMMDD HH:MM
> > endif
> >
> > [Note I've not pasted the rest of the loop here]
> >
> > The *IUPVS01 *function will get the minutes of the cycle time. I
*think*
> > that should do the trick.
> >
> > I did try running with:
> >
> > pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> > PB2NCConfig_gust -v 5 -log pb2nc.log -valid_beg 20180419_153000
> -valid_end
> > 20180419_160000
> >
> > But pb2nc didn't output anything, here's a summary of some of the
output:
> >
> > DEBUG 2: PrepBufr Time Center: 20180419_150000
> > DEBUG 2: Searching Time Window: 20180419_153000 to
> 20180419_160000
> > DEBUG 2: Processing 114311 PrepBufr messages...
> > DEBUG 2: Total PrepBufr Messages processed = 114311
> > DEBUG 2: Rejected based on message type = 108874
> > DEBUG 2: Rejected based on station id = 0
> > DEBUG 2: Rejected based on valid time = 5437
> > DEBUG 2: Rejected based on masking grid = 0
> > DEBUG 2: Rejected based on masking polygon = 0
> > DEBUG 2: Rejected based on elevation = 0
> > DEBUG 2: Rejected based on pb report type = 0
> > DEBUG 2: Rejected based on input report type = 0
> > DEBUG 2: Rejected based on instrument type = 0
> > DEBUG 2: Rejected based on zero observations = 0
> > DEBUG 2: Total PrepBufr Messages retained = 0
> > DEBUG 2: Total observations retained or derived = 0
> >
> >
> > If pb2nc is caluclating the absolute time of the ob using the DHR
in
> > association with the 10 digit idate then this would probably cause
this
> > issue. Thus even adding the -valid_beg and -valid_end flags
wouldn't
> help
> > since pb2nc is not calculating the valid time of the observations
> correctly
> > as the minutes part of the center time is not accounted for.
> >
> > Thanks!
> > Jacob
> >
> > On Wed, May 23, 2018 at 2:52 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Jacob,
> >>
> >> Thanks for looking closely at the output. This whole notion of
picking
> a
> >> "center time" and defining a retention time window around it
using
> >> "obs_window" is inherited from our original processing of
convention
> >> PREPBUFR files. That logic assumes that all messages in the file
> contain
> >> the same center time. And perhaps that's not true for the RTMA
data?
> >>
> >> Howard looked into it and found the following...
> >>
> >>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >> pb2nc gets the time by calling IREADNS which returns IDATE (from
the
> >> first message and the first subset).
> >>
> >>
> >> C SUBPROGRAM: IREADNS
> >> C PRGMMR: WOOLLEN ORG: NP20 DATE: 1994-01-06
> >> C
> >> C ABSTRACT: THIS FUNCTION CALLS BUFR ARCHIVE LIBRARY SUBROUTINE
READNS
> >> C AND PASSES BACK ITS RETURN CODE. SEE READNS FOR MORE
DETAILS.
> >> C
> >> ...
> >> C OUTPUT ARGUMENT LIST:
> >> C SUBSET - CHARACTER*8: TABLE A MNEMONIC FOR BUFR MESSAGE
> >> C CONTAINING SUBSET BEING READ
> >> C IDATE - INTEGER: DATE-TIME STORED WITHIN SECTION 1 OF
BUFR
> >> C MESSAGE CONTAINING SUBSET BEING READ, IN FORMAT
OF
> >> C EITHER YYMMDDHH OR YYYYMMDDHH, DEPENDING ON
DATELEN()
> >> C VALUE
> >> C IREADNS - INTEGER: RETURN CODE:
> >> C 0 = normal return
> >> C -1 = there are no more subsets in the BUFR
file
> >>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >> A workaround already exists for this. You can explicitly specify
the
> >> retention time window on the command line for PB2NC:
> >> "-valid_beg time" in YYYYMMDD[_HH[MMSS]] sets the
> >> beginning of the retention time window (optional).
> >> "-valid_end time" in YYYYMMDD[_HH[MMSS]] sets the
end
> >> of the retention time window (optional).
> >>
> >> That time window setting overrides the obs_window setting from
the
> >> config file. So you could use:
> >>
> >> -valid_beg 20180419_153000 -valid_end 20180419_160000
> >>
> >> So that'd solve the immediate problem.
> >>
> >> But long term, we should make sense of the "center time" logic.
Do
> >> you have any additional details about these files that would help
us
> >> determine if/how the center time logic should change?
> >>
> >>
> >> Thanks,
> >> John
> >>
> >>
> >
>
>
------------------------------------------------
Subject: ceiling and visibility verification in MET
From: Jacob Carley - NOAA Affiliate
Time: Thu May 24 09:13:44 2018
Hi John,
Since we're spoiled having all the beta releases easily accessible on
WCOSS/Theia, I'm 100% fine with this being put into a beta release. :-
)
Send me a note when you have something for us to play with and I'll
happily
give it a go!
Thanks for your help and assistance!
-Jacob
On Thu, May 24, 2018 at 11:07 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Jacob,
>
> This is all great info! Thanks for clarifying. Yes, we'll need to
patch
> this up and get it working.
>
> Is it sufficient to add this to the next met-7.1 beta release or
will we
> need a patch for the met-7.0 release as well?
>
> Thanks,
> John
>
> On Wed, May 23, 2018 at 1:23 PM, Jacob Carley - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84952 >
> >
> > I should mentioned that the *IUPVS01 *function comes from bufrlib
and you
> > declare it, as well as the other date/time variables, as:
> >
> > INTEGER :: IUPVS01, MMcyc, MMcyc0, idate_hold, idate
> >
> > -Jacob
> >
> > On Wed, May 23, 2018 at 3:19 PM, Jacob Carley - NOAA Federal <
> > jacob.carley at noaa.gov> wrote:
> >
> > > Hi John,
> > >
> > > These files do all have the same "center time", which really is
the
> > > analysis/cycle time. The obs contained within are all stored
relative
> to
> > > that time. I suppose the issue is just ensuring the correct
time
> > > analysis/cycletime is used (i.e. we get the minutes part).
> > >
> > > I did some poking around, and here's an example we do to get the
> minutes
> > > portion of the analysis/cycle time after grabbing the 10 digit
idate:
> > >
> > > call openbf(unit_in,'IN',unit_in)
> > > call dxdump(unit_in,unit_table)
> > > call datelen(10)
> > > nmsg=0
> > > MMcyc0=-1
> > > msg_report: do while (ireadmg(unit_in,subset,idate) == 0)
> > > idate_hold=idate ! idate gets lost/clobbered w/ last, failed
ireadmg
> > > call
> > > nmsg=nmsg+1
> > > ntb = 0
> > > ! get minutes portion of analysis/cycle time
> > > MMcyc=*IUPVS01*(unit_in,'MINU')
> > > if (MMcyc.ne.MMcyc0) then
> > > MMcyc0=MMcyc
> > > write(*,'(42x,a,$)') '-- analysis time (dTime ref): '
> > > write(*,'(2x,i8,i3.2,":",i2.2)') idate/100,
mod(idate,100),MMcyc !
> > > CCYYMMDD HH:MM
> > > endif
> > >
> > > [Note I've not pasted the rest of the loop here]
> > >
> > > The *IUPVS01 *function will get the minutes of the cycle time.
I
> *think*
> > > that should do the trick.
> > >
> > > I did try running with:
> > >
> > > pb2nc rtma2p5_ru.t1545z.prepbufr rtma2p5_ru.t1545z.prepbufr.nc
> > > PB2NCConfig_gust -v 5 -log pb2nc.log -valid_beg 20180419_153000
> > -valid_end
> > > 20180419_160000
> > >
> > > But pb2nc didn't output anything, here's a summary of some of
the
> output:
> > >
> > > DEBUG 2: PrepBufr Time Center: 20180419_150000
> > > DEBUG 2: Searching Time Window: 20180419_153000 to
> > 20180419_160000
> > > DEBUG 2: Processing 114311 PrepBufr messages...
> > > DEBUG 2: Total PrepBufr Messages processed = 114311
> > > DEBUG 2: Rejected based on message type = 108874
> > > DEBUG 2: Rejected based on station id = 0
> > > DEBUG 2: Rejected based on valid time = 5437
> > > DEBUG 2: Rejected based on masking grid = 0
> > > DEBUG 2: Rejected based on masking polygon = 0
> > > DEBUG 2: Rejected based on elevation = 0
> > > DEBUG 2: Rejected based on pb report type = 0
> > > DEBUG 2: Rejected based on input report type = 0
> > > DEBUG 2: Rejected based on instrument type = 0
> > > DEBUG 2: Rejected based on zero observations = 0
> > > DEBUG 2: Total PrepBufr Messages retained = 0
> > > DEBUG 2: Total observations retained or derived = 0
> > >
> > >
> > > If pb2nc is caluclating the absolute time of the ob using the
DHR in
> > > association with the 10 digit idate then this would probably
cause this
> > > issue. Thus even adding the -valid_beg and -valid_end flags
wouldn't
> > help
> > > since pb2nc is not calculating the valid time of the
observations
> > correctly
> > > as the minutes part of the center time is not accounted for.
> > >
> > > Thanks!
> > > Jacob
> > >
> > > On Wed, May 23, 2018 at 2:52 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Jacob,
> > >>
> > >> Thanks for looking closely at the output. This whole notion of
> picking
> > a
> > >> "center time" and defining a retention time window around it
using
> > >> "obs_window" is inherited from our original processing of
convention
> > >> PREPBUFR files. That logic assumes that all messages in the
file
> > contain
> > >> the same center time. And perhaps that's not true for the RTMA
data?
> > >>
> > >> Howard looked into it and found the following...
> > >>
> > >>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>
> > >> pb2nc gets the time by calling IREADNS which returns IDATE
(from the
> > >> first message and the first subset).
> > >>
> > >>
> > >> C SUBPROGRAM: IREADNS
> > >> C PRGMMR: WOOLLEN ORG: NP20 DATE: 1994-01-06
> > >> C
> > >> C ABSTRACT: THIS FUNCTION CALLS BUFR ARCHIVE LIBRARY SUBROUTINE
READNS
> > >> C AND PASSES BACK ITS RETURN CODE. SEE READNS FOR MORE
DETAILS.
> > >> C
> > >> ...
> > >> C OUTPUT ARGUMENT LIST:
> > >> C SUBSET - CHARACTER*8: TABLE A MNEMONIC FOR BUFR MESSAGE
> > >> C CONTAINING SUBSET BEING READ
> > >> C IDATE - INTEGER: DATE-TIME STORED WITHIN SECTION 1 OF
BUFR
> > >> C MESSAGE CONTAINING SUBSET BEING READ, IN
FORMAT OF
> > >> C EITHER YYMMDDHH OR YYYYMMDDHH, DEPENDING ON
DATELEN()
> > >> C VALUE
> > >> C IREADNS - INTEGER: RETURN CODE:
> > >> C 0 = normal return
> > >> C -1 = there are no more subsets in the
BUFR file
> > >>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>
> > >> A workaround already exists for this. You can explicitly
specify the
> > >> retention time window on the command line for PB2NC:
> > >> "-valid_beg time" in YYYYMMDD[_HH[MMSS]] sets
the
> > >> beginning of the retention time window (optional).
> > >> "-valid_end time" in YYYYMMDD[_HH[MMSS]] sets
the end
> > >> of the retention time window (optional).
> > >>
> > >> That time window setting overrides the obs_window setting from
the
> > >> config file. So you could use:
> > >>
> > >> -valid_beg 20180419_153000 -valid_end 20180419_160000
> > >>
> > >> So that'd solve the immediate problem.
> > >>
> > >> But long term, we should make sense of the "center time" logic.
Do
> > >> you have any additional details about these files that would
help us
> > >> determine if/how the center time logic should change?
> > >>
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >>
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list