[Met_help] pb2nc: -pbfile option not working right

[John Halley Gotway]

Jonathan,

Hmmm, that's a good question.  Unfortunately, I don't have a good answer
for you.

I've been involved primarily in method and code development for MET and
not actually running it on a regular basis with real data.  I can say that
this is the first I've heard of such an error in that dataset.

I'll pass this along to others in our group to see if they have anything
to add.

John

> Hi John,
>
> I'm re-compiling currently and will test shortly.
> In other news, I noticed a problem in one of the PREPBUFR files I got
> from the NOMADS server.  The 18z 2 June 2008 gdas1 prepbufr file had
> dates from 1 June instead of 2 June for all the sfc obs I extracted.  I
> re-downloaded the dataset in case my script made a mistake.  I also
> grabbed the already blocked data -- same issue.
>
> How much faith do you have in the archived PREPBUFR datasets?  Did I
> just happen to find a fluke dataset with the wrong data/dates?
>
> Thanks once again,
> Jonathan
>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu]
>> Sent: Thursday, February 12, 2009 1:33 PM
>> To: Case, Jonathan (MSFC)[]; met_help at ucar.edu
>> Subject: Re: [Met_help] pb2nc: -pbfile option not working right
>>
>> Jonathan,
>>
>> Please copy the attached file into your METv1.1 build and recompile
>> MET:
>>
>> -> cp pb2nc.cc METv1.1/src/pb2nc/pb2nc.cc make clean make
>>
>> This should fix both issues you were seeing:
>> (1) the error you saw using multiple PrepBufr files
>> (2) the issue of -valid_beg and -valid_end not working right (this was
>> fixed in the development version of the code, but we'd never posted a
>> bug fix for it)
>>
>> Please try using -valid_beg and -valid_end to verify that they seem to
>> work right.
>>
>> And please let me know if these changes did the trick for you.  Thanks
>> again for finding these problems.
>>
>> Thanks,
>> John
>>
>> John Halley Gotway wrote:
>> > Jonathan,
>> >
>> > Good news.  I found a bug in PB2NC which I think is causing this
>> problem.
>> >
>> > When PB2NC reads the first PrepBufr file, it figures out how many
>> > messages that file contains.  Then it attempts to read that number
> of
>> messages from all subsequent PrepBufr files, rather than figuring out
>> how many messages are actually in those files.  I think the error
>> you're seeing is PB2NC attempting to read more messages than are
>> actually in the file!
>> >
>> > I'll work up a fix for this, post it to the MET website, and send it
>> to you.
>> >
>> > Thanks for finding this issue, and sorry for the inconvenience.
> I'll
>> write you when I have the fix ready.
>> >
>> > Thanks,
>> > John
>> >
>> > Case, Jonathan (MSFC)[] wrote:
>> >> John,
>> >>
>> >>
>> >>
>> >> I'm having issues trying to process multiple PREPBUFR files into a
>> >> single daily netcdf file using pb2nc.
>> >>
>> >>
>> >>
>> >> So far, I've only had one successfully attempt when I first
>> processed
>> >> the 12z gdas1 file, then the 00z, 06z, 18z, and next day's 00z,
> each
>> >> using the -pbfile option.  [I had to specify the beg_ds and end_ds
>> to
>> >> be
>> >> +/- 12 hours (43200 seconds) in order for this to work.  Smaller
>> >> +values
>> >> assigned to beg_ds and end_ds resulted in a failure.]  Even so, it
>> >> does not seem as if the -valid_begin and -valid_end options have
> any
>> effect.
>> >> In my data, I had a few OBS times from before the specified
>> >> -valid_begin time of 20080531_234500:
>> >>
>> >>
>> >>
>> >> lc2:/raid1/casejl/MET/PB2NC > pb2nc
>> >> gdas1.20080601.t12z.prepbufr.blk.nr
>> >> test_daily.nc PB2NCConfig_SFONLY_WRF -valid_begin 20080531_234500
>> >> -valid_end 20080601_234500 -pbfile
>> >> gdas1.20080601.t00z.prepbufr.blk.nr
>> >> -pbfile gdas1.20080601.t06z.prepbufr.blk.nr -pbfile
>> >> gdas1.20080601.t18z.prepbufr.blk.nr -pbfile
>> >> gdas1.20080602.t00z.prepbufr.blk.nr
>> >>
>> >> Reading Config File:    PB2NCConfig_SFONLY_WRF
>> >>
>> >> Creating NetCDF File:   test_daily.nc
>> >>
>> >> Reading PrepBufr File:  gdas1.20080601.t12z.prepbufr.blk.nr
>> >>
>> >> Reading PrepBufr File:  gdas1.20080601.t00z.prepbufr.blk.nr
>> >>
>> >> Reading PrepBufr File:  gdas1.20080601.t06z.prepbufr.blk.nr
>> >>
>> >> Reading PrepBufr File:  gdas1.20080601.t18z.prepbufr.blk.nr
>> >>
>> >> Reading PrepBufr File:  gdas1.20080602.t00z.prepbufr.blk.nr
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Then I tried processing the files in temporal order as follows
>> >> (keeping everything else the same):
>> >>
>> >>
>> >>
>> >> lc2:/raid1/casejl/MET/PB2NC > pb2nc
>> >> gdas1.20080601.t00z.prepbufr.blk.nr
>> >> test_daily.nc PB2NCConfig_SFONLY_WRF -valid_begin 20080531_234500
>> >> -valid_end 20080601_234500 -pbfile
>> >> gdas1.20080601.t06z.prepbufr.blk.nr
>> >> -pbfile gdas1.20080601.t12z.prepbufr.blk.nr -pbfile
>> >> gdas1.20080601.t18z.prepbufr.blk.nr -pbfile
>> >> gdas1.20080602.t00z.prepbufr.blk.nr
>> >>
>> >> Reading Config File:    PB2NCConfig_SFONLY_WRF
>> >>
>> >> Creating NetCDF File:   test_daily.nc
>> >>
>> >> Reading PrepBufr File:  gdas1.20080601.t00z.prepbufr.blk.nr
>> >>
>> >> Reading PrepBufr File:  gdas1.20080601.t06z.prepbufr.blk.nr
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ERROR: process_pbfile() -> the observation time should remain the
>> >> same for all PrepBufr messages: -62169984000 != 1212300000
>> >>
>> >> Your advice is much appreciated,
>> >>
>> >> Jon
>> >>
>> >>
>> >>
>> >> ***********************************************************
>> >> Jonathan Case, ENSCO, Inc.
>> >> Aerospace Sciences & Engineering Division Short-term Prediction
>> >> Research and Transition Center 320 Sparkman Drive, Room 3062
>> >> Huntsville, AL 35805-1912
>> >> Voice: (256) 961-7504   Fax: (256) 961-7788
>> >> Emails: Jonathan.Case-1 at nasa.gov
>> >>
>> >>              case.jonathan at ensco.com
>> >>
>> >> ***********************************************************
>> >>
>> >>
>> >>
>> >>
>> > _______________________________________________
>> > Met_help mailing list
>> > Met_help at mailman.ucar.edu
>> > http://mailman.ucar.edu/mailman/listinfo/met_help
>