[Met_help] [rt.rap.ucar.edu #49166] History for sfc data problems in PREPBUFR files
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Fri Aug 26 14:56:35 MDT 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear Met Help,
I have looked more closely at the contents of the GDAS PREPBUFR obs files, both on NOMADS and the NCAR site. In both files, the standard gdas prepbufr files do not appear to contain any ADPSFC type of observation data. I tried a couple different large subset regions using the data from the NCAR site, and in both cases, there weren't any ADPSFC data. (e.g. gdas1.t00z.prepbufr.nr from NOMADS, and prepbufr.gdas.20110813.t00z.nr from NCAR)
Now, on the NOMADS/NCAR servers, I can see specific bufr files with adpsfc as part of the filename (e.g. gdas1.t00z.adpsfc.tm00.bufr_d.nr on http://nomads.ncdc.noaa.gov/data/gdas/201108/20110813/)
The problem I am having is that this datafile won't run in pb2nc. I receive the following error:
> $MET/bin/pb2nc gdas_20110813.t00z.adpsfc.tm00.bufr_d.nr test.nc PB2NCConfig_SFC_MESOAMERICA
Reading Config File: PB2NCConfig_SFC_MESOAMERICA
Creating NetCDF File: test.nc
Processing PrepBufr File: gdas_20110813.t00z.adpsfc.tm00.bufr_d.nr
ERROR: process_pbfile() -> the observation time should remain the same for all PrepBufr messages: 1313186400 != 1313182800
So, even though these files appear to have the obs data I need, I can't use them as is in pb2nc.
Please advise.
Thanks for the help!
Jonathan
--------------------------------------------------------------------------------------------------
Jonathan Case, ENSCO Inc.
NASA Short-term Prediction Research and Transition Center (aka SPoRT Center)
320 Sparkman Drive, Room 3062
Huntsville, AL 35805
Voice: 256.961.7504
Fax: 256.961.7788
Emails: Jonathan.Case-1 at nasa.gov / case.jonathan at ensco.com
--------------------------------------------------------------------------------------------------
"Whether the weather is cold, or whether the weather is hot, we'll weather
the weather whether we like it or not!"
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: sfc data problems in PREPBUFR files
From: John Halley Gotway
Time: Fri Aug 26 13:52:44 2011
Jon,
Sorry for the delay in getting back to you one this.
Two issues here...
(1) Why can't MET read that files you tried?
Unfortunately, PB2NC is currently only compatible with PREPBUFR files.
I believe the file to which you're referring
(http://nomads.ncdc.noaa.gov/data/gdas/201108/20110813/gdas1.t00z.adpsfc.tm00.bufr_d.nr)
is in BUFR format which differs somewhat with the PREPBUFR format. On
the off chance that it might work, I did
try running it through PB2NC anyway, skipping over the checking of the
time. When it got to actually reading data from that BUFR file, PB2NC
just read garbage. PB2NC completed with no apparent
error, but had no valid observation data to write, since it didn't
read the BUFR data properly.
(2) How can you get ADPSFC observations out of GDAS PREPBUFR files?
Those GDAS files actually do contain ADPSFC observations! I suspect
that there's something in your PB2NC configuration which is preventing
them from being retained - probably the quality_mark_thresh
is set too high. For example, I grabbed the following file:
http://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.20110825/gdas1.t00z.prepbufr.nr
And I ran it through PB2NC using the following command line and
attached config file:
METv3.0.1/bin/pb2nc gdas1.t00z.prepbufr.nr gdas1.t00z.prepbufr.nc
PB2NCConfig -v 2
I did have to wait several minutes for it to process, but it
eventually finished and retained 108679 ADPSFC messages:
Reading Config File: PB2NCConfig
Creating NetCDF File: gdas1.t00z.prepbufr.nc
Processing PrepBufr File: gdas1.t00z.prepbufr.nr
Blocking PrepBufr file to: /tmp/tmp_pb2nc_blk_3274_0
PrepBufr Time Center: 20110825_000000
Searching Time Window: 20110824_180000 to 20110825_060000
Processing 406443 PrepBufr messages...
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90%
95% 100%
Total PrepBufr Messages processed = 406443
Rejected based on message type = 295773
Rejected based on station id = 0
Rejected based on valid time = 0
Rejected based on masking grid = 0
Rejected based on masking polygon = 0
Rejected based on elevation = 0
Rejected based on pb report type = 0
Rejected based on input report type = 0
Rejected based on instrument type = 0
Rejected based on zero observations = 1991
Total PrepBufr Messages retained = 108679
Total observations retained or derived = 480609
Next, I ran the NetCDF observation file through plot_point_obs to plot
the locations for the ADPSFC temperature observations. The resulting
image is attached.
Lastly, there are two gotcha's you should know about...
(1) When dealing with PREPBUFR observations, there was a significant
bug for METv3.0 in PB2NC in the handling of the event stack flag. The
fix was posted on 2/15/11
(http://www.dtcenter.org/met/users/support/known_issues/METv3.0/index.php).
I think you already know this, but if not, be sure you're using the
latest set of patches.
(2) I suspect that you have the "quality_mark_thresh" set too tightly
in the PB2NC config file, and that's why you're getting 0
observations. The GDAS PREPBUFR observations are used in data
assimilation, and the model developers have figured out that if they
assimilate too many obs, the performance suffers. Therefore, they
limit the number of observations - particularly ADPSFC
observations - that they assimilate. Unfortunately, the way they
limit them is by changing the quality mark from something good, like
2, to something bad, like 9. So even though the observations
might really be fine, they have a pretty bad quality mark of 9.
This issue was actually masked by that event stack bug I mentioned.
But once the bug was resolved, this issue came to light.
So I'd suggest rerunning with a quality_mark_thresh = 9, see what
ADPSFC observations you get, and go from there.
Hope that helps.
John Halley Gotway
met_help at ucar.edu
On 08/24/2011 09:45 AM, RAL HelpDesk {for Case, Jonathan[ENSCO INC]}
wrote:
>
> Wed Aug 24 09:45:28 2011: Request 49166 was acted upon.
> Transaction: Ticket created by jonathan.case-1 at nasa.gov
> Queue: met_help
> Subject: sfc data problems in PREPBUFR files
> Owner: Nobody
> Requestors: jonathan.case-1 at nasa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49166 >
>
>
> Dear Met Help,
>
>
> I have looked more closely at the contents of the GDAS PREPBUFR obs
files, both on NOMADS and the NCAR site. In both files, the standard
gdas prepbufr files do not appear to contain any ADPSFC type of
observation data. I tried a couple different large subset regions
using the data from the NCAR site, and in both cases, there weren't
any ADPSFC data. (e.g. gdas1.t00z.prepbufr.nr from NOMADS, and
prepbufr.gdas.20110813.t00z.nr from NCAR)
>
>
>
> Now, on the NOMADS/NCAR servers, I can see specific bufr files with
adpsfc as part of the filename (e.g. gdas1.t00z.adpsfc.tm00.bufr_d.nr
on http://nomads.ncdc.noaa.gov/data/gdas/201108/20110813/)
>
>
>
> The problem I am having is that this datafile won't run in pb2nc. I
receive the following error:
>
>
>
>> $MET/bin/pb2nc gdas_20110813.t00z.adpsfc.tm00.bufr_d.nr test.nc
PB2NCConfig_SFC_MESOAMERICA
>
> Reading Config File: PB2NCConfig_SFC_MESOAMERICA
>
> Creating NetCDF File: test.nc
>
> Processing PrepBufr File:
gdas_20110813.t00z.adpsfc.tm00.bufr_d.nr
>
>
>
>
>
> ERROR: process_pbfile() -> the observation time should remain the
same for all PrepBufr messages: 1313186400 != 1313182800
>
>
>
> So, even though these files appear to have the obs data I need, I
can't use them as is in pb2nc.
>
> Please advise.
>
>
>
> Thanks for the help!
>
> Jonathan
>
>
>
--------------------------------------------------------------------------------------------------
> Jonathan Case, ENSCO Inc.
> NASA Short-term Prediction Research and Transition Center (aka SPoRT
Center)
> 320 Sparkman Drive, Room 3062
> Huntsville, AL 35805
> Voice: 256.961.7504
> Fax: 256.961.7788
> Emails: Jonathan.Case-1 at nasa.gov / case.jonathan at ensco.com
>
--------------------------------------------------------------------------------------------------
>
> "Whether the weather is cold, or whether the weather is hot, we'll
weather
> the weather whether we like it or not!"
>
------------------------------------------------
Subject: sfc data problems in PREPBUFR files
From: John Halley Gotway
Time: Fri Aug 26 13:52:44 2011
////////////////////////////////////////////////////////////////////////////////
//
// Default pb2nc configuration file
//
////////////////////////////////////////////////////////////////////////////////
//
// Stratify the observation data in the PrepBufr files in the
following
// ways:
// (1) by message type: supply a list of PrepBufr message types
// to retain (i.e. AIRCFT)
// (2) by station id: supply a list of observation stations to retain
// (3) by valid time: supply starting and ending times in form
// YYYY-MM-DD HH:MM:SS UTC
// (4) by location: supply either an NCEP masking grid, a masking
// lat/lon polygon or a file to a mask lat/lon polygon
// (5) by elevation: supply min/max elevation values
// (6) by report type (typ): supply a list of report types to retain
// (7) by instrument type (itp): supply a list of instrument type to
// retain
// (8) by vertical level: supply min/max vertical levels
// (9) by variable type: supply a list of variable types to retain
// P, Q, T, Z, U, V
// (11) by quality mark: supply a quality mark threshold
// (12) Flag to retain values for all quality marks, or just the first
// quality mark (highest)
// (13) by data level category: supply a list of category types to
// retain.
//
// 0 - Surface level (mass reports only)
// 1 - Mandatory level (upper-air profile reports)
// 2 - Significant temperature level (upper-air profile reports)
// 2 - Significant temperature and winds-by-pressure level
// (future combined mass and wind upper-air reports)
// 3 - Winds-by-pressure level (upper-air profile reports)
// 4 - Winds-by-height level (upper-air profile reports)
// 5 - Tropopause level (upper-air profile reports)
// 6 - Reports on a single level
// (e.g., aircraft, satellite-wind, surface wind,
// precipitable water retrievals, etc.)
// 7 - Auxiliary levels generated via interpolation from spanning
levels
// (upper-air profile reports)
//
//
// Specify a comma-separated list of PrepBufr message type strings to
retain.
// An empty list indicates that all should be retained.
// List of valid message types:
// ADPUPA AIRCAR AIRCFT ADPSFC ERS1DA GOESND GPSIPW
// MSONET PROFLR QKSWND RASSDA SATEMP SATWND SFCBOG
// SFCSHP SPSSMI SYNDAT VADWND
// ANYAIR (= AIRCAR, AIRCFT)
// ANYSFC (= ADPSFC, SFCSHP, ADPUPA, PROFLR)
// ONLYSF (= ADPSFC, SFCSHP)
//
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
//
// e.g. message_type[] = [ "ADPUPA", "AIRCAR" ];
//
message_type[] = [ "ADPSFC" ];
//
// Specify a comma-separated list of station ID strings to retain.
// An empty list indicates that all should be retained.
//
// e.g. station_id[] = [ "KDEN" ];
//
station_id[] = [];
//
// Beginning and ending time offset values in seconds for observations
// to retain. The valid time window for retaining observations is
// defined in reference to the observation time. So observations with
// a valid time falling in the window [obs_time+beg_ds,
obs_time+end_ds]
// will be retained.
//
beg_ds = -21600;
end_ds = 21600;
//
// Specify the name of a single grid to be used in masking the data.
// An empty string indicates that no grid should be used. The
standard
// NCEP grids are named "GNNN" where NNN indicates the three digit
grid number.
//
// http://www.nco.ncep.noaa.gov/pmb/docs/on388/tableb.html
//
// e.g. mask_grid = "G212";
//
mask_grid = "";
//
// Specify a single ASCII file containing a lat/lon polygon.
// Latitude in degrees north and longitude in degrees east.
// By default, the first and last polygon points are connected.
//
// The lat/lon polygon file should contain a name for the polygon
// followed by a space-separated list of lat/lon points:
// "name lat1 lon1 lat2 lon2... latn lonn"
//
// MET_BASE may be used in the path for the lat/lon polygon file.
//
// e.g. mask_poly = "MET_BASE/data/poly/EAST.poly";
//
mask_poly = "";
//
// Beginning and ending elevation values in meters for observations
// to retain.
//
beg_elev = -1000;
end_elev = 100000;
//
// Specify a comma-separated list of PrepBufr report type values to
retain.
// An empty list indicates that all should be retained.
//
//
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_4.htm
//
// e.g. pb_report_type[] = [ 120, 133 ];
//
pb_report_type[] = [];
//
// Specify a comma-separated list of input report type values to
retain.
// An empty list indicates that all should be retained.
//
//
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_6.htm
//
// e.g. in_report_type[] = [ 11, 22, 23 ];
//
in_report_type[] = [];
//
// Specify a comma-separated list of instrument type values to retain.
// An empty list indicates that all should be retained.
//
// e.g. instrument_type[] = [ 52, 87 ];
//
instrument_type[] = [];
//
// Beginning and ending vertical levels to retain.
//
beg_level = 1;
end_level = 255;
//
// Specify a comma-separated list of strings containing grib codes or
// corresponding grib code abbreviations to retain or be derived from
// the available observations.
//
// Grib Codes to be RETAINED:
// SPFH or 51 for Specific Humidity in kg/kg
// TMP or 11 for Temperature in K
// HGT or 7 for Height in meters
// UGRD or 33 for the East-West component of the wind in m/s
// VGRD or 34 for the North-South component of the wind in m/s
//
// Grib Codes to be DERIVED:
// DPT or 17 for Dewpoint Temperature in K
// WIND or 32 for Wind Speed in m/s
// RH or 52 for Relative Humidity in %
// MIXR or 53 for Humidity Mixing Ratio in kg/kg
// PRMSL or 2 for Pressure Reduced to Mean Sea Level in Pa
//
// http://www.nco.ncep.noaa.gov/pmb/docs/on388/table2.html
//
// e.g. obs_grib_code[] = [ "TMP", "UGRD", "VGRD", "WIND" ];
//
obs_grib_code[] = [ "SPFH", "TMP", "HGT", "UGRD", "VGRD",
"DPT", "WIND", "RH", "MIXR" ];
//
// Quality mark threshold to indicate which observations to retain.
// Observations with a quality mark equal to or LESS THAN this
threshold
// will be retained, while observations with a quality mark GREATER
THAN
// this threshold will be discarded.
//
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_7.htm
//
quality_mark_thresh = 9;
//
// Flag to indicate whether observations should be drawn from the top
// of the event stack (most quality controlled) or the bottom of the
// event stack (most raw). A value of 1 indicates that the top of the
// event stack should be used while a value of zero indicates that the
// bottom should be used.
//
event_stack_flag = 1;
//
// Space comma-separated list of data level categorie values to
retain,
// where a value of:
// 0 = Surface level (mass reports only)
// 1 = Mandatory level (upper-air profile reports)
// 2 = Significant temperature level (upper-air profile reports)
// 2 = Significant temperature and winds-by-pressure level
// (future combined mass and wind upper-air reports)
// 3 = Winds-by-pressure level (upper-air profile reports)
// 4 = Winds-by-height level (upper-air profile reports)
// 5 = Tropopause level (upper-air profile reports)
// 6 = Reports on a single level
// (e.g., aircraft, satellite-wind, surface wind,
// precipitable water retrievals, etc.)
// 7 = Auxiliary levels generated via interpolation from spanning
levels
// (upper-air profile reports)
// An empty list indicates that all should be retained.
//
//
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
//
// e.g. level_category[] = [ 0, 1 ];
//
level_category[] = [];
//
// Directory where temp files should be written by the PB2NC tool
//
tmp_dir = "/tmp";
//
// Indicate a version number for the contents of this configuration
file.
// The value should generally not be modified.
//
version = "V3.0.1";
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #49166] sfc data problems in PREPBUFR files
From: Case, Jonathan[ENSCO INC]
Time: Fri Aug 26 13:58:38 2011
Wow. Thanks for the helpful info!! I'm encouraged to know that it
was likely user-error in not seeing any ADPSFC obs in the PB2NC
output. I'll try the QC threshold adjustment you suggest.
BTW, I
did come across the PB2NC bug, so we have the patches all installed
for V3. Thanks for the reminder, however.
Good news prior to the
weekend,
Jonathan
-----Original Message-----
From: RAL HelpDesk
{for John Halley Gotway} [mailto:met_help at ucar.edu]
Sent: Friday,
August 26, 2011 2:53 PM
To: Case, Jonathan (MSFC-VP61)[ENSCO INC]
Subject: Re: [rt.rap.ucar.edu #49166] sfc data problems in PREPBUFR
files
Jon,
Sorry for the delay in getting back to you one this.
Two issues here...
(1) Why can't MET read that files you tried?
Unfortunately, PB2NC is currently only compatible with PREPBUFR files.
I believe the file to which you're referring
(http://nomads.ncdc.noaa.gov/data/gdas/201108/20110813/gdas1.t00z.adpsfc.tm00.bufr_d.nr)
is in BUFR format which differs somewhat with the PREPBUFR format. On
the off chance that it might work, I did try running it through PB2NC
anyway, skipping over the checking of the time. When it got to
actually reading data from that BUFR file, PB2NC just read garbage.
PB2NC completed with no apparent error, but had no valid observation
data to write, since it didn't read the BUFR data properly.
(2) How
can you get ADPSFC observations out of GDAS PREPBUFR files?
Those
GDAS files actually do contain ADPSFC observations! I suspect that
there's something in your PB2NC configuration which is preventing them
from being retained - probably the quality_mark_thresh is set too
high. For example, I grabbed the following file:
http://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.20110825/gdas1.t00z.prepbufr.nr
And I ran it through PB2NC using the following command line and
attached config file:
METv3.0.1/bin/pb2nc gdas1.t00z.prepbufr.nr
gdas1.t00z.prepbufr.nc PB2NCConfig -v 2
I did have to wait several
minutes for it to process, but it eventually finished and retained
108679 ADPSFC messages:
Reading Config File: PB2NCConfig
Creating
NetCDF File: gdas1.t00z.prepbufr.nc
Processing PrepBufr File:
gdas1.t00z.prepbufr.nr
Blocking PrepBufr file to:
/tmp/tmp_pb2nc_blk_3274_0
PrepBufr Time Center: 20110825_000000
Searching Time Window: 20110824_180000 to 20110825_060000 Processing
406443 PrepBufr messages...
5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Total PrepBufr Messages
processed = 406443
Rejected based on message type =
295773
Rejected based on station id = 0
Rejected based on
valid time = 0
Rejected based on masking grid = 0
Rejected based on masking polygon = 0
Rejected based on
elevation = 0
Rejected based on pb report type = 0
Rejected based on input report type = 0
Rejected based on
instrument type = 0
Rejected based on zero observations =
1991
Total PrepBufr Messages retained = 108679
Total
observations retained or derived = 480609
Next, I ran the NetCDF
observation file through plot_point_obs to plot the locations for the
ADPSFC temperature observations. The resulting image is attached.
Lastly, there are two gotcha's you should know about...
(1) When
dealing with PREPBUFR observations, there was a significant bug for
METv3.0 in PB2NC in the handling of the event stack flag. The fix was
posted on 2/15/11
(http://www.dtcenter.org/met/users/support/known_issues/METv3.0/index.php).
I think you already know this, but if not, be sure you're using the
latest set of patches.
(2) I suspect that you have the
"quality_mark_thresh" set too tightly in the PB2NC config file, and
that's why you're getting 0 observations. The GDAS PREPBUFR
observations are used in data assimilation, and the model developers
have figured out that if they assimilate too many obs, the performance
suffers. Therefore, they limit the number of observations -
particularly ADPSFC observations - that they assimilate.
Unfortunately, the way they limit them is by changing the quality mark
from something good, like 2, to something bad, like 9. So even though
the observations might really be fine, they have a pretty bad quality
mark of 9.
This issue was actually masked by that event stack bug I
mentioned. But once the bug was resolved, this issue came to light.
So I'd suggest rerunning with a quality_mark_thresh = 9, see what
ADPSFC observations you get, and go from there.
Hope that helps.
John Halley Gotway
met_help at ucar.edu
On 08/24/2011 09:45 AM, RAL
HelpDesk {for Case, Jonathan[ENSCO INC]} wrote:
>
> Wed Aug 24
09:45:28 2011: Request 49166 was acted upon.
> Transaction: Ticket
created by jonathan.case-1 at nasa.gov
> Queue: met_help
>
Subject: sfc data problems in PREPBUFR files
> Owner: Nobody
> Requestors: jonathan.case-1 at nasa.gov
> Status: new
>
Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49166
> >
>
>
> Dear Met Help,
>
>
> I have looked more closely at
the contents of the GDAS PREPBUFR obs
> files, both on NOMADS and
the NCAR site. In both files, the standard
> gdas prepbufr files do
not appear to contain any ADPSFC type of
> observation data. I tried
a couple different large subset regions
> using the data from the
NCAR site, and in both cases, there weren't
> any ADPSFC data. (e.g.
gdas1.t00z.prepbufr.nr from NOMADS, and
>
prepbufr.gdas.20110813.t00z.nr from NCAR)
>
>
>
> Now, on the
NOMADS/NCAR servers, I can see specific bufr files with
> adpsfc as
part of the filename (e.g. gdas1.t00z.adpsfc.tm00.bufr_d.nr
> on
http://nomads.ncdc.noaa.gov/data/gdas/201108/20110813/)
>
>
>
>
The problem I am having is that this datafile won't run in pb2nc. I
receive the following error:
>
>
>
>> $MET/bin/pb2nc
gdas_20110813.t00z.adpsfc.tm00.bufr_d.nr test.nc
>>
PB2NCConfig_SFC_MESOAMERICA
>
> Reading Config File:
PB2NCConfig_SFC_MESOAMERICA
>
> Creating NetCDF File: test.nc
>
> Processing PrepBufr File:
gdas_20110813.t00z.adpsfc.tm00.bufr_d.nr
>
>
>
>
>
> ERROR:
process_pbfile() -> the observation time should remain the same
>
for all PrepBufr messages: 1313186400 != 1313182800
>
>
>
> So,
even though these files appear to have the obs data I need, I can't
use them as is in pb2nc.
>
> Please advise.
>
>
>
> Thanks
for the help!
>
> Jonathan
>
>
>
----------------------------------------------------------------------
> ----------------------------
> Jonathan Case, ENSCO Inc.
> NASA
Short-term Prediction Research and Transition Center (aka SPoRT
>
Center) 320 Sparkman Drive, Room 3062 Huntsville, AL 35805
> Voice:
256.961.7504
> Fax: 256.961.7788
> Emails: Jonathan.Case-1 at nasa.gov
/ case.jonathan at ensco.com
>
----------------------------------------------------------------------
> ----------------------------
>
> "Whether the weather is cold, or
whether the weather is hot, we'll weather
> the weather whether we
like it or not!"
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #49166] sfc data problems in PREPBUFR files
From: John Halley Gotway
Time: Fri Aug 26 14:56:28 2011
Jon,
Sure. Happy to help. I'll go ahead and resolve this ticket. Just
let us know if additional issues arise.
Thanks,
John
On 08/26/2011 01:58 PM, RAL HelpDesk {for Case, Jonathan[ENSCO INC]}
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49166 >
>
> Wow. Thanks for the helpful info!! I'm encouraged to know that it
was likely user-error in not seeing any ADPSFC obs in the PB2NC
output. I'll try the QC threshold adjustment you suggest.
>
> BTW, I did come across the PB2NC bug, so we have the patches all
installed for V3. Thanks for the reminder, however.
>
> Good news prior to the weekend,
> Jonathan
>
> -----Original Message-----
> From: RAL HelpDesk {for John Halley Gotway}
[mailto:met_help at ucar.edu]
> Sent: Friday, August 26, 2011 2:53 PM
> To: Case, Jonathan (MSFC-VP61)[ENSCO INC]
> Subject: Re: [rt.rap.ucar.edu #49166] sfc data problems in PREPBUFR
files
>
> Jon,
>
> Sorry for the delay in getting back to you one this.
>
> Two issues here...
>
> (1) Why can't MET read that files you tried?
>
> Unfortunately, PB2NC is currently only compatible with PREPBUFR
files. I believe the file to which you're referring
>
(http://nomads.ncdc.noaa.gov/data/gdas/201108/20110813/gdas1.t00z.adpsfc.tm00.bufr_d.nr)
is in BUFR format which differs somewhat with the PREPBUFR format. On
the off chance that it might work, I did try running it through PB2NC
anyway, skipping over the checking of the time. When it got to
actually reading data from that BUFR file, PB2NC just read garbage.
PB2NC completed with no apparent error, but had no valid observation
data to write, since it didn't read the BUFR data properly.
>
> (2) How can you get ADPSFC observations out of GDAS PREPBUFR files?
>
> Those GDAS files actually do contain ADPSFC observations! I suspect
that there's something in your PB2NC configuration which is preventing
them from being retained - probably the quality_mark_thresh is set too
high. For example, I grabbed the following file:
>
http://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.20110825/gdas1.t00z.prepbufr.nr
>
> And I ran it through PB2NC using the following command line and
attached config file:
> METv3.0.1/bin/pb2nc gdas1.t00z.prepbufr.nr gdas1.t00z.prepbufr.nc
PB2NCConfig -v 2
>
> I did have to wait several minutes for it to process, but it
eventually finished and retained 108679 ADPSFC messages:
> Reading Config File: PB2NCConfig
> Creating NetCDF File: gdas1.t00z.prepbufr.nc
> Processing PrepBufr File: gdas1.t00z.prepbufr.nr
> Blocking PrepBufr file to: /tmp/tmp_pb2nc_blk_3274_0
> PrepBufr Time Center: 20110825_000000
> Searching Time Window: 20110824_180000 to 20110825_060000
Processing 406443 PrepBufr messages...
> 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85%
90% 95% 100%
> Total PrepBufr Messages processed = 406443
> Rejected based on message type = 295773
> Rejected based on station id = 0
> Rejected based on valid time = 0
> Rejected based on masking grid = 0
> Rejected based on masking polygon = 0
> Rejected based on elevation = 0
> Rejected based on pb report type = 0
> Rejected based on input report type = 0
> Rejected based on instrument type = 0
> Rejected based on zero observations = 1991
> Total PrepBufr Messages retained = 108679
> Total observations retained or derived = 480609
>
> Next, I ran the NetCDF observation file through plot_point_obs to
plot the locations for the ADPSFC temperature observations. The
resulting image is attached.
>
> Lastly, there are two gotcha's you should know about...
>
> (1) When dealing with PREPBUFR observations, there was a significant
bug for METv3.0 in PB2NC in the handling of the event stack flag. The
fix was posted on 2/15/11
(http://www.dtcenter.org/met/users/support/known_issues/METv3.0/index.php).
I think you already know this, but if not, be sure you're using the
latest set of patches.
>
> (2) I suspect that you have the "quality_mark_thresh" set too
tightly in the PB2NC config file, and that's why you're getting 0
observations. The GDAS PREPBUFR observations are used in data
assimilation, and the model developers have figured out that if they
assimilate too many obs, the performance suffers. Therefore, they
limit the number of observations - particularly ADPSFC observations -
that they assimilate. Unfortunately, the way they limit them is by
changing the quality mark from something good, like 2, to something
bad, like 9. So even though the observations might really be fine,
they have a pretty bad quality mark of 9.
>
> This issue was actually masked by that event stack bug I mentioned.
But once the bug was resolved, this issue came to light.
>
> So I'd suggest rerunning with a quality_mark_thresh = 9, see what
ADPSFC observations you get, and go from there.
>
> Hope that helps.
>
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 08/24/2011 09:45 AM, RAL HelpDesk {for Case, Jonathan[ENSCO INC]}
wrote:
>>
>> Wed Aug 24 09:45:28 2011: Request 49166 was acted upon.
>> Transaction: Ticket created by jonathan.case-1 at nasa.gov
>> Queue: met_help
>> Subject: sfc data problems in PREPBUFR files
>> Owner: Nobody
>> Requestors: jonathan.case-1 at nasa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=49166
>>>
>>
>>
>> Dear Met Help,
>>
>>
>> I have looked more closely at the contents of the GDAS PREPBUFR obs
>> files, both on NOMADS and the NCAR site. In both files, the
standard
>> gdas prepbufr files do not appear to contain any ADPSFC type of
>> observation data. I tried a couple different large subset regions
>> using the data from the NCAR site, and in both cases, there weren't
>> any ADPSFC data. (e.g. gdas1.t00z.prepbufr.nr from NOMADS, and
>> prepbufr.gdas.20110813.t00z.nr from NCAR)
>>
>>
>>
>> Now, on the NOMADS/NCAR servers, I can see specific bufr files with
>> adpsfc as part of the filename (e.g.
gdas1.t00z.adpsfc.tm00.bufr_d.nr
>> on http://nomads.ncdc.noaa.gov/data/gdas/201108/20110813/)
>>
>>
>>
>> The problem I am having is that this datafile won't run in pb2nc.
I receive the following error:
>>
>>
>>
>>> $MET/bin/pb2nc gdas_20110813.t00z.adpsfc.tm00.bufr_d.nr test.nc
>>> PB2NCConfig_SFC_MESOAMERICA
>>
>> Reading Config File: PB2NCConfig_SFC_MESOAMERICA
>>
>> Creating NetCDF File: test.nc
>>
>> Processing PrepBufr File:
gdas_20110813.t00z.adpsfc.tm00.bufr_d.nr
>>
>>
>>
>>
>>
>> ERROR: process_pbfile() -> the observation time should remain the
same
>> for all PrepBufr messages: 1313186400 != 1313182800
>>
>>
>>
>> So, even though these files appear to have the obs data I need, I
can't use them as is in pb2nc.
>>
>> Please advise.
>>
>>
>>
>> Thanks for the help!
>>
>> Jonathan
>>
>>
>>
----------------------------------------------------------------------
>> ----------------------------
>> Jonathan Case, ENSCO Inc.
>> NASA Short-term Prediction Research and Transition Center (aka
SPoRT
>> Center) 320 Sparkman Drive, Room 3062 Huntsville, AL 35805
>> Voice: 256.961.7504
>> Fax: 256.961.7788
>> Emails: Jonathan.Case-1 at nasa.gov / case.jonathan at ensco.com
>>
----------------------------------------------------------------------
>> ----------------------------
>>
>> "Whether the weather is cold, or whether the weather is hot, we'll
weather
>> the weather whether we like it or not!"
>>
>
------------------------------------------------
More information about the Met_help
mailing list